Prompt and scope
Context propagation can make a request understandable across service boundaries, but every downstream service may receive and log the propagated values. OpenTelemetry documents Baggage as name/value context adjacent to trace context and warns about security implications. The core skill is distributed-boundary design, so this is a system-design question.
What interviewers assess
Strong answers separate trace context from application metadata, define an allowlist and ownership for fields, and prevent sensitive values from crossing untrusted boundaries. They cover header size, canonical encoding, sampling, retries, asynchronous messages, and what happens when context is malformed or missing. They also include metrics that prove propagation is useful without turning it into an uncontrolled data channel.
Questions to clarify first
- Which protocols carry context: HTTP, gRPC, queues, or scheduled jobs?
- Which fields are diagnostic, which affect behavior, and who owns each field?
- What trust boundaries exist between tenants, regions, and third-party services?
- Are values allowed in logs, metrics labels, or only traces?
- What are maximum header size, latency, and availability budgets?
- Should malformed context be rejected, stripped, or replaced with a new root?
30-second answer framework
“I would define a small versioned envelope with separate trace context and approved business fields. At each boundary, a policy library validates names, size, encoding, tenant scope, and destination trust; it strips or rejects disallowed values and never forwards secrets. Propagation must work for synchronous and asynchronous edges, with bounded headers and clear missing-context behavior. I would measure extraction failures, truncation, propagation coverage, cross-tenant violations, and trace join rate.”
Step-by-step answer
Step 1: Define the context contract
Create a registry of fields with owner, type, maximum length, sensitivity, retention, and allowed destinations. Keep trace identifiers in the tracing protocol and put only approved business metadata in a separate baggage-like carrier. Version the envelope so consumers can reject unknown critical fields.
Step 2: Enforce boundary policy
Use a shared middleware or sidecar policy that parses the carrier, validates encoding and size, checks tenant and trust zone, and emits a sanitized carrier. Never copy arbitrary inbound headers into outbound requests. Treat third-party and cross-tenant calls as new trust roots unless an explicit policy permits forwarding.
Step 3: Handle transport and retries
Define equivalent carriers for HTTP, gRPC metadata, and message attributes. Persist only the fields required to correlate an asynchronous job; do not serialize secrets into queues. On retries, preserve the original trace relationship while preventing duplicate or stale business decisions from being trusted without revalidation.
Step 4: Design failure behavior
Malformed or oversized context should be stripped or rejected according to endpoint risk, while the request itself remains observable with a new local trace root when safe. Expose reason-coded counters, not raw values. Make policy version and enforcement result visible to operators.
Step 5: Operate and prove value
Track trace join rate, extraction and injection failures, bytes added, truncation, policy rejects, cross-boundary violations, and queue serialization errors. Sample safely and avoid unbounded high-cardinality metric labels. Add contract tests for every supported protocol and a canary policy rollout before enforcing rejection.
Model answer
“I would treat propagated context as an untrusted data channel. A versioned registry defines which diagnostic and business fields may travel, their sensitivity and size. Middleware validates and sanitizes each hop, with stricter rules for third-party and cross-tenant calls; secrets never enter carriers or queues. Trace context remains separate from business baggage. I would support HTTP, gRPC, and asynchronous attributes, define stripping versus rejection, and measure join rate, policy failures, bytes, and violations. A canary rollout and reason-coded metrics let us tighten policy without losing observability.”
Common mistakes
- Forwarding every inbound header → untrusted data crosses boundaries → use an allowlist and sanitizer.
- Putting tokens or PII in baggage → downstream logs and services can expose them → keep sensitive data out.
- Using baggage as metric labels → cardinality and cost explode → record bounded reason codes and dimensions.
- Ignoring queues and retries → asynchronous work loses or trusts stale context → define transport-specific contracts and revalidate.
- Rejecting every malformed request → observability and availability suffer → choose stripping or rejection by endpoint risk.
- No size budget → headers cause proxy failures → cap each field and the total carrier.
Follow-up questions
Follow-up 1: Should tenant ID be propagated?
Only when the destination is authorized for that tenant and the value is validated against authenticated identity. It should not become an authority by itself.
Follow-up 2: What is the difference between trace context and baggage?
Trace context connects spans and propagation state. Baggage carries application-defined name/value data and therefore needs stricter sensitivity, ownership, and destination policy.
Follow-up 3: How do you prevent header growth?
Set per-field and total budgets, reject or truncate with reason-coded metrics, and prefer a bounded reference to server-side state when larger context is unavoidable.
Follow-up 4: What happens at an untrusted boundary?
Strip non-approved fields, create or continue only the trace information permitted by policy, and log the policy decision without recording the sensitive value.