Prompt and context
A worker consumes 100 events from a queue and makes one downstream API call after aggregation. Each event may belong to a different Trace, and the worker may also be triggered by a scheduler or replay. Model the trace so every source remains discoverable without pretending that unrelated requests form one parent-child tree.
OpenTelemetry describes Spans as operations that can form a tree and allows each Span to contain zero or more Links. Its overview names batch processing initiated by multiple incoming Spans as a typical Links use case.
What the interviewer is testing
The candidate should know that a parent is one current context while a Link represents related causality without parentage. They should control link count, sampling, and high-cardinality attributes, while preserving useful metric and log correlation.
Clarifying questions to ask first
- Can a batch contain multiple tenants, security levels, or business types?
- Is the downstream call one aggregate operation, or can it remain per-event?
- Is the goal per-event accountability, latency analysis, or batch throughput?
- Is sampling decided at ingress, or can the worker retain selected source contexts?
- May Link attributes contain event IDs, tenant IDs, or sensitive fields?
30-second answer framework
“The batch-processing Span uses the worker or scheduler context as its parent. The 100 incoming SpanContexts become Links because they jointly caused one batch operation without forming one parent-child chain. I keep only necessary low-cardinality attributes, enforce a link limit, and count truncation. Batch metrics cover size, queue wait, processing, downstream latency, failures, and retries; logs correlate with a batch ID and event hash. Sensitive tenant fields are filtered, and failed or replayed batches have a sampling safety path.”
Step-by-step deep dive
Step 1: Separate parent from Link
A parent says which single Span the current operation continues, forming a Trace tree and inheriting its TraceId. A Link records one related SpanContext from the same or another Trace. When sources are peers in a batch, choosing the first event as parent creates a false tree.
Batch-processing Span
parent: worker / scheduler context
links: event-1 SpanContext ... event-100 SpanContextStep 2: Preserve SpanContext across the boundary
Extract TraceContext from message headers, validate its format and sampling flag, and create a Link. Do not put the complete message, user input, or raw token in Link attributes. For a missing context, count “no source context” instead of inventing a TraceId.
Step 3: Bound link size and cost
One hundred links is only the example limit; production batches may be larger. Configure the SDK link count limit or an application cap, retain representative error, retry, replay, or priority-tenant sources, and record dropped-link count. Truncation must be visible in batch metrics and logs.
Step 4: Define the batch Span lifecycle
The Span covers batch wait, deserialization, aggregation, downstream call, and commit. Add phase events or metrics. Every created Span must end on success, failure, cancellation, or partial commit. A single slow event must not hide the batch’s phase boundaries.
Step 5: Make sampling and failures diagnosable
Ingress sampling may discard source Traces, so the worker needs a safety policy for failures, retries, dead letters, and manual replay. Links present at Span creation may influence sampling; Links added later may not. Make that order and fallback explicit.
Step 6: Give traces, metrics, and logs different jobs
Traces explain one batch’s causality. Metrics carry batch size, queue wait, processing latency, success/failure, and truncation. Logs use batch ID, event hash, and replay ID to locate a controlled sample. Do not use event ID as an unbounded metric label.
Step 7: Isolate tenants and privacy
For multi-tenant batches, use irreversible hashes or internal references in Links and logs, and filter attributes before export. If security levels cannot mix, split the batch by tenant or permission so a low-privilege reader cannot traverse another tenant’s context.
Step 8: Verify queries and failures
Test one source, mixed Traces, missing context, link overflow, sampling loss, downstream retry, partial failure, dead letter, and replay. Verify that a batch Span jumps to retained source Traces and that metrics identify truncated or unsampled batches.
Model high-quality answer
“The batch Span uses the worker or scheduler as parent, and each message SpanContext is a Link. That preserves the fact that 100 events jointly caused one downstream call without inventing a parent-child tree. I bound links and count drops, filter tenant and sensitive attributes, and use metrics for batch size, queue wait, downstream latency, failures, and retries. Logs correlate with batch and replay IDs, and failed batches receive sampling protection. Tests cover missing context, mixed Traces, overflow, and replay.”
Common mistakes
- Choosing the first message as parent → false causality → use a common worker parent and Links for sources.
- Putting the full message in a Link → privacy and cost leak → keep necessary sanitized low-cardinality fields only.
- Adding Links without a limit → unbounded Span and export cost → cap and measure truncation.
- Ending a Span only on success → failures and cancellations leave open Spans → end in a finally or scope.
- Using event ID as a metric label → cardinality explosion → keep detail in logs or traces.
- Assuming late Links always affect sampling → important sources disappear → prepare sampling context before Span creation.
Follow-up questions and strong responses
Follow-up 1: Does a one-message batch need a Link?
It can use that message context as the parent. If the worker has an independent lifetime, a worker parent plus one Link is also valid; choose based on whether the batch is a direct child operation.
Follow-up 2: Do Links merge different Traces?
No. They express relatedness and preserve each TraceId. The query system must provide navigation from the Link to the source Trace.
Follow-up 3: Which sources survive truncation?
Prefer errors, retries, replays, priority tenants, or deterministic samples; record total and dropped counts. Silently keeping the first N introduces bias.
Follow-up 4: How do you diagnose a failed batch?
Give failure and dead-letter replays independent batch/replay IDs, retain controlled error-source Links or summaries, and put failure type in metrics.
Follow-up 5: May a Link contain tenant.id?
Only after access, cardinality, and privacy review. Cross-tenant export usually hashes or removes it, and it should not become a high-volume metric label.