System Design: How Do You Build an OpenTelemetry Log Pipeline with Trace Correlation?

Prompt and scope

An interviewer may ask: “Design an OpenTelemetry log pipeline that supports trace correlation. Explain the data model, backpressure, tenant isolation, and recovery.”

The signal is whether you can turn application events into governed telemetry. The OpenTelemetry Logs Data Model separates Timestamp, TraceId, SpanId, SeverityNumber, Body, Resource, and Attributes; OTLP allows logs to travel hop by hop through agents, Collectors, and backends. The challenge is preserving semantics, capacity, and security boundaries, not drawing a single Kafka line.

What the interviewer is testing

• Whether you separate Resource, Attributes, Body, and Trace Context responsibilities.
• Whether you can design a reliable path across applications, agents, Collectors, queues, and backends.
• Whether you handle burst backpressure, batching, retries, priority drops, and disk buffering.
• Whether you prevent cross-tenant exposure and redact secrets and personal data.
• Whether you explain duplicates, ordering, sampling, and the query experience when TraceId is absent.

Clarifying questions

• Are sources SDKs, existing files, container stdout, or a mixture?
• What are the tenant count, throughput, retention, and query-latency targets?
• Does the application inject TraceId consistently, and is a correlation key allowed when it is absent?
• Which fields contain personal data, secrets, or business-sensitive content, and where must redaction happen?
• During backend failure, which levels may be dropped and how much replay is required after recovery?

A 30-second answer

You can say:

I would use a layered path from application or file receivers to a local Collector, then to a queue and backend. Each record keeps time, severity, Body, Resource, and Attributes, and includes TraceId and SpanId when context exists. Collectors batch, rate-limit, redact, and route; a queue and disk buffer absorb backend jitter. Tenant identity enters trusted Resource attributes and authorization indexes. During a surge, drop or sample debug first, then manage retries, duplicates, and recovery reads.

Step-by-step reasoning

Define one record contract

Do not make an unstructured line the only contract. A logical record can look like this:

{
  "timestamp": "2026-08-01T10:00:00Z",
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId": "00f067aa0ba902b7",
  "severityNumber": 17,
  "severityText": "ERROR",
  "body": {"message": "payment declined", "code": "CARD_DECLINED"},
  "resource": {"service.name": "checkout", "tenant.id": "t-7"},
  "attributes": {"region": "us-east-1"}
}

Resource describes the emitting entity, Attributes describe an event occurrence, and Body preserves structured content. Compare severity with SeverityNumber while retaining source SeverityText for display.

Design collection and transport

An SDK or file receiver converts records to OTLP. A node agent batches and applies initial limits; a Collector parses, enriches Resource data, redacts, routes, and exports. OTLP supports intermediate Collectors, so long paths need explicit timeouts, authentication, and compression. A queue is optional, but it can provide durable buffering and a tenant quota boundary when backend throughput is unstable.

Handle backpressure and data classes

Set per-tenant and per-service rate, batch, memory, and disk limits. When the backend slows, pause low-priority exports while preserving error, audit, and security events; sample or drop debug. Retries need exponential backoff and a maximum retention time to avoid retry storms. Record failed batch ranges for replay and use idempotency or backend deduplication to reduce duplicates.

Secure, isolate, and correlate queries

Remove secrets, tokens, and personal data early at the edge or Collector. Inject tenant identity from a trusted Resource source; do not accept arbitrary client overrides. Partition indexes by tenant and time, with optional TraceId and SpanId correlation indexes. Logs without TraceId remain queryable by service, time, and request identifier; never invent a plausible TraceId.

Model high-quality answer

I would split the design into record contract, collection, buffering, processing, and query layers. Records follow the OpenTelemetry Logs Data Model for time, severity, Body, Resource, and Attributes, adding TraceId and SpanId when the application has context. An SDK or filelog receiver sends to a node Collector; the Collector batches, redacts, enforces tenant quotas, and routes over OTLP, with a durable per-tenant queue when needed. Backend jitter is absorbed by memory and disk buffers; when budgets are exceeded, drop in the order of debug, info, error, and audit while measuring drop rate and oldest-record age. Query indexes are tenant- and time-isolated, and TraceId correlation accelerates investigation without fabricating context. Recovery uses batch ranges, backoff, and idempotent deduplication; audit and security logs have separate retention and access policies.

Common mistakes

• Putting every field in Body and losing Resource, Attributes, and Trace Context semantics.
• Drawing only an application-to-backend line without an agent, Collector, queue, or disk buffer.
• Retrying forever during backend failure until memory is exhausted and the outage cascades.
• Letting clients submit tenant attributes directly and contaminating cross-tenant indexes.
• Generating random trace IDs for logs that lack context and misleading investigation.
• Discussing throughput without priority drops, redaction, duplicate, and recovery metrics.

Follow-up questions and responses

1. What do you do when TraceId is missing?

Keep the original log and mark context as missing. Use verifiable service, time, and request identifiers for correlation. Fill TraceId only when the application or trusted proxy supplies it; never fabricate one.

2. How do you prevent secret leakage?

Apply field rules and pattern redaction early in the SDK, agent, or Collector, reject obvious secret formats, and enforce tenant-level backend access. Any sampled raw retention needs stricter isolation and audit.

3. When is dropping logs acceptable?

Define business classes first: security, audit, and critical errors are usually preserved; debug and high-cardinality diagnostic fields can be sampled. Record the reason, tenant, time range, and count for every drop so capacity pressure is explainable and alertable.