← Back to System Internals

Designing a High-Volume Telemetry Ingestion Pipeline

A system design interview guide to the pipeline that swallows a firehose of telemetry from heterogeneous sources and lands it, reliably and losslessly, into stores that serve both real-time analytics and LLM/RAG querying.

Behind every observability platform is an ingestion pipeline doing the unglamorous, load-bearing work: accepting telemetry from tens of thousands of sources in a dozen formats, buffering it durably so nothing is lost when a downstream store hiccups, transforming and validating it in flight, and routing it to whichever stores need it. The modern twist is that there are now two very different consumers of that data — the traditional real-time analytics path (dashboards, alerts, ad-hoc queries) and a newer semantic path that embeds telemetry into a vector store so an LLM can answer questions over it via retrieval-augmented generation. A good design serves both from one pipeline while keeping the operational and analytical stores cleanly separated. This guide builds it: the durable buffer at the center, stream processing for parse/enrich/route, the delivery guarantees (exactly-once, dead-letter, PII scrubbing), and the dual-sink fan-out.

1. Requirements & Scope

The pipeline sits between producers of telemetry and the stores that consume it. Pin down what it must guarantee.

• Heterogeneous ingest. Accept logs, metrics, traces, and events from many source types (apps, hosts, network devices, third parties) in many formats, and normalize them.
• Durability & no loss. Once accepted, a record must not be lost even if every downstream store is temporarily unavailable. The buffer decouples producers from consumers.
• Throughput & backpressure. Sustain millions of records per second and degrade gracefully under spikes rather than dropping data or stalling producers.
• Transform in flight. Parse, validate against a schema, enrich with context, scrub PII, and route each record to the right sink(s).
• Dual consumers. Feed a real-time analytical path and a semantic (embedding/vector) path for LLM/RAG querying, keeping operational and analytical stores separated.
• Delivery semantics. At-least-once by default, exactly-once where correctness demands it, with a dead-letter path for records that cannot be processed.

2. Back-of-the-Envelope Estimates

Assume 10M records/second at peak, averaging 1 KB each.

The numbers force two conclusions: the buffer must be partitioned into the low thousands to get the parallelism to keep up, and you cannot embed everything — the semantic path is fed a deliberately routed, filtered subset, which is why routing at processing time matters.

3. Data Model, Schema & Validation

Heterogeneous sources produce a mess of formats, so the first job is to normalize everything into a common envelope while validating structure. A typical normalized record carries a timestamp, a source/type, a set of resource attributes, the payload, and correlation IDs.

# normalized telemetry record (envelope + payload)
{
  "ts":        1751779200000,        # event time, ms
  "source":    "svc-checkout",       # who produced it
  "type":      "log" | "metric" | "trace" | "event",
  "resource":  { "host": ..., "region": ..., "env": ... },
  "trace_id":  "4bf92f...",          # correlation, if in a request
  "body":      { ... },              # the actual payload
  "schema_ver": 3                    # for evolution
}

Schema & validation. Use a schema registry (Avro/Protobuf/JSON Schema) so producers and consumers agree on structure and it can evolve compatibly. Validate at the edge: a record that fails schema validation is not silently dropped and not allowed to poison the pipeline — it is routed to a dead-letter queue for later inspection. Schema-on-write (validate at ingest) gives clean downstream data; schema-on-read (store raw, interpret later) gives flexibility. High-volume telemetry usually does light validation on write plus schema-on-read for the flexible payload, so a new field from a source does not break ingestion.

4. High-Level Architecture

The pipeline is a fan-in to a durable buffer, then stream processing, then a fan-out to separated sinks.

The shape is deliberate. Agents at the edge batch and compress so producers are cheap and the network is efficient. Kafka in the middle decouples everything: producers write as fast as they like, consumers read at their own pace, and a slow or dead sink never backs pressure onto the producers. Stream processing is where all the transformation and routing happens. And the fan-out separates concerns: the operational store is tuned for low-latency reads, the analytical store for cheap batch scans, and the vector store for semantic retrieval — three different access patterns, three different stores, one pipeline.

5. The Durable Buffer: Kafka

Kafka (or a Kafka-compatible log) is the heart of the design. It is a distributed, durable, replayable commit log: producers append records to topics, which are split into partitions that are replicated across brokers, and consumers read at their own offset.

• Partitioning. A topic is split into partitions for parallelism — each partition is an ordered log consumed by one member of a consumer group at a time. Choosing the partition key matters: key by something that keeps related records together (e.g. trace_id so all spans of a trace land in order) while spreading load evenly. A bad key (low cardinality, or a hot tenant) creates a hot partition that becomes the bottleneck.
• Durability. Each partition is replicated (typically 3×); a write is acknowledged only once enough in-sync replicas have it (acks=all), so an acknowledged record survives broker loss.
• Backpressure via decoupling. Because Kafka retains data for a configured window, a downstream outage does not lose data — consumers simply fall behind and then catch up by replaying from their last offset. This is the mechanism that makes the pipeline resilient. Retention is sized (section 2) to absorb the longest outage you must survive.
• Replay. The same durability enables reprocessing: to backfill a new sink or fix a processing bug, you rewind consumer offsets and replay history.

6. Stream Processing & Guarantees

Between the buffer and the sinks sits a stream processor (Flink, Kafka Streams, or similar) that reads from Kafka, transforms, and writes to the sinks. This is where correctness is won or lost.

Delivery semantics

Three guarantees, in increasing strength. At-most-once may drop records (unacceptable for telemetry). At-least-once never drops but may duplicate on retry (the sensible default). Exactly-once neither drops nor duplicates — achieved either with Kafka’s transactional producer (idempotent producer + transactions that atomically commit consumer offsets and output writes) or, more cheaply, with at-least-once delivery plus idempotent sinks that dedupe on a record key. Exactly-once is expensive, so reserve it for sinks where a duplicate is a correctness bug (e.g. a metric counter) and use at-least-once + idempotency elsewhere.

Dead-letter queue

A single malformed “poison” record must never block a partition or crash the processor in a loop. Records that fail parsing, validation, or repeated processing are diverted to a dead-letter queue with the error and original payload, so the main flow keeps moving and the failures can be inspected, fixed, and replayed.

PII scrubbing

Telemetry routinely captures sensitive data by accident — emails in a log line, tokens in a URL. Scrub PII in the processor, before it lands in any queryable store: detect sensitive fields (patterns + tagged schema fields) and mask, hash, or drop them. Doing it in-stream, once, is far safer than trying to redact it in every downstream store after the fact.

for record in kafka_consume(topic, group):
  rec = parse(record)                          # normalize format
  if not schema.valid(rec):
    dead_letter.send(rec, reason="schema"); continue   # do not poison
  rec = enrich(rec)                            # add resource / geo / tenant
  rec = scrub_pii(rec)                         # mask before any store
  for sink in route(rec):                      # fan out to matching sinks
    sink.write_idempotent(rec, key=rec.id)     # dedupe on retry
  commit_offsets_atomically()                  # exactly-once boundary

7. Dual Sinks: Operational, Analytical, Semantic

The routing step fans each record to the sinks that need it, and the sinks are kept cleanly separated because they serve fundamentally different access patterns.

Operational vs analytical separation is the classic move: the operational store is optimized for freshness and low-latency reads and kept small, while the analytical store holds cheap, long-retained columnar data for scans and ML. Keeping them apart means a heavy analytical scan never contends with the low-latency operational path.

The semantic / RAG path

The newer sink supports natural-language querying of telemetry (“why did checkout error rates spike last night?”). A routed subset of records — typically errors, anomalies, and notable events, not the full firehose — is passed through an embedding model that turns each into a vector capturing its meaning, and the vectors are stored in a vector store (with an ANN index like HNSW). At query time, the user’s question is embedded, the vector store returns the nearest telemetry, and that context is fed to an LLM to generate a grounded answer — retrieval-augmented generation. Crucially this is a separate sink: embedding is expensive, so only a filtered subset flows here, and it never sits on the operational hot path.

8. Bottlenecks & Tradeoffs

• Partition key skew. The most common real bottleneck — a poor key funnels a hot tenant or hot service onto one partition, capping throughput. Key for even spread while preserving needed ordering.
• Exactly-once vs throughput. Transactional exactly-once adds coordination overhead and latency. Use it only where duplicates are a correctness bug; prefer at-least-once + idempotent sinks elsewhere.
• Buffer retention vs cost. Longer Kafka retention means surviving longer outages and more replay flexibility, but more storage. Size it to your worst tolerable downstream outage.
• Schema-on-write vs on-read. Strict validation gives clean data but rejects anything unexpected and couples producers tightly; loose validation is flexible but pushes correctness downstream. Most pipelines do light write validation + schema-on-read.
• Backpressure vs loss. Under sustained overload beyond buffer capacity you must eventually choose: shed low-value telemetry (sampling) or risk falling permanently behind. Decide the shedding policy deliberately, by signal priority.

9. Summary

A high-volume telemetry pipeline is organized around a durable, replayable buffer that decouples producers from consumers, with in-stream processing that validates, enriches, scrubs, and routes to cleanly separated sinks — including a semantic path for LLM/RAG.