Prompt and context
The latency histogram for a multi-tenant API shows a p99 regression during peak traffic. Adding user_id, the full URL, or request parameters as metric labels would create unbounded time series and cost. Looking only at aggregate buckets cannot identify one slow request. Design metrics exemplars that attach a small number of trace references to metric samples so an investigator can move from a chart to a trace, then to logs and dependencies.
The question tests whether you separate metric aggregation semantics from an external exemplar reference. OpenTelemetry defines an exemplar as a recorded value associated with a metric event; it may carry trace_id, span_id, observation time, and filtered attributes. OpenMetrics likewise requires a value, label set, and timestamp. The main metric must remain low-cardinality; an exemplar is not a hidden label system.
What the interviewer evaluates
- You know exemplars do not change histogram buckets, count, or sum; they reference an observation outside the aggregate.
- You can choose trace-based or probabilistic sampling and explain the trade-off among rate, tail-latency coverage, and cost.
- You use stable references such as
trace_idandspan_id, without putting full requests, tokens, or personal data into metrics. - You handle timestamps, reordering, backend drops, retention, and cross-tenant authorization rather than drawing only a link.
- You use low-cardinality metrics to find a time window, then use exemplars to validate a representative trace and follow logs, dependencies, and build versions.
Questions to clarify first
- Which metric backend, trace backend, and visualization tool are used, and do they support exemplar queries and deep links?
- Which instruments are in scope: Histogram, Counter, or Gauge? Is p99 measured at the service or gateway?
- Should sampling target every error, tail latency, or a tenant and version budget? Is there one cross-region policy?
- What retention, access-control, redaction, and tenant-isolation rules constrain references?
- What memory, network, and storage budgets apply, and must the primary metric remain available when exemplars are dropped?
A 30-second answer
“I keep a low-cardinality latency histogram and attach trace_id, span_id, the observed value, and timestamp only for a small sampled set. Sampling prioritizes errors and tail latency; user identifiers stay out of metric labels. The metric backend stores a short-lived reference, and a permission check precedes the trace deep link. I verify that bucket statistics are unchanged, the time window returns exemplars, and drops do not affect the primary metric. I gate on hit rate, link success, memory overhead, and sensitive-field scans.”
Step-by-step solution
Step 1: Set the metric/exemplar boundary
Keep histogram labels to bounded dimensions such as service, route_template, region, and status_class. The observation still contributes to bucket_counts, count, and sum; the exemplar stores one traceable reference and must not create a new time series per trace.
Step 2: Sample for tail coverage
Decide trace sampling in the request context, then attach an observation to the relevant Histogram sample when it is sampled and matches an error, latency threshold, or per-dimension budget. Use a fixed capacity per service and metric, such as a reservoir or ring buffer. Version sampling changes so hit counts are not mistaken for traffic volume.
Step 3: Encode references and time
An exemplar contains a numeric value, a label set, and an observation time; trace references should use trace_id and span_id. The timestamp should be close to observation time and align with the metric sample window. Receivers may truncate labels or discard exemplars, so the query path must support “metric present, exemplar absent.”
latency_seconds_bucket{route="/checkout",le="1"} 982
# {trace_id="4f8...",span_id="91a...",build="2026.07.31"} 1.42 1753938000000Step 4: Bound privacy, cardinality, and cost
Never put raw URLs, bodies, email addresses, authorization tokens, or user IDs into an exemplar. Optional attributes such as build and region need an allowlist and length limits. OpenTelemetry notes that attributes removed from a metric stream by a View may still be exported as exemplar filtered attributes, so redaction must be configured separately. Estimate samples per second, bytes per record, ring-buffer memory, remote writes, and trace-query cost; reduce exemplar sampling before polluting metric labels.
Step 5: Implement a cross-backend jump
The dashboard queries exemplars for a selected time range, then builds a controlled trace-backend link from trace_id. The link service checks tenant, region, and authorization, and returns an explicit state for missing, expired, or cross-environment traces. Logs and spans share Trace Context, but the three signals do not need one storage system.
Step 6: Test degradation and release gates
Replay fixed traffic and compare buckets, count, sum, and p99 before and after enabling exemplars to prove aggregation is unchanged. Inject slow and error requests to check hit rate, timestamps, trace jumps, and tenant isolation. Delay, truncate, or disable the exemplar backend and confirm metric queries still work. Gates should cover sensitive-field scans, memory ceilings, remote-write failure rate, jump success, and sampling fairness by service and version.
A strong sample answer
“I first bound metric labels to route templates, status classes, and region, leaving the histogram responsible for p99 aggregation. Requests carry Trace Context; when a sampled trace matches an error or tail-latency policy, I attach trace_id, span_id, value, and timestamp to the Histogram sample. Exemplar attributes are allowlisted, contain no user identity or body, and are redacted again before View export.”
“Prometheus/OpenMetrics queries return exemplars only for the selected window. A permission check precedes the trace link; if the trace has expired, the metric remains visible with an expired-reference state. Load tests compare bucket/count/sum/p99, and drills cover backend drops, reordering, cross-tenant access, and exhausted budgets. Hit rate, time to first trace, memory, write cost, and sensitive-field scans determine the sampling budget; high-cardinality labels never enter the metric.”
Common mistakes
- Put
trace_idin metric labels → time-series explosion → keep it in the exemplar reference. - Use only fixed-probability sampling → p99 or errors can be absent for long periods → add tail, error, and budgeted sampling.
- Ignore exemplar timestamps → the trace is outside the chart window → record observation time and validate the window.
- Assume filtered attributes are safe → sensitive data may still be exported with exemplars → use a separate allowlist, redaction, and authorization check.
- Treat missing exemplars as metric failure → aggregation becomes coupled to references → keep metrics available and monitor exemplar loss separately.
- Retain references without bounds → memory and cost grow without control → use fixed capacity, retention, and sampling budgets.
Follow-up questions and answers
Does an exemplar change p99?
No. Its value is already included in the Histogram bucket, count, and sum. It adds context and a reference; it must not create another metric series or count the observation twice.
Why not put the full URL in an exemplar attribute?
Full URLs can contain user data, tokens, and unbounded cardinality. Use an allowlisted route template and version; inspect parameters in an authorized, redacted trace or log instead.
What happens when every exemplar is dropped?
Metric queries and alerts continue to use aggregate series. Investigators lose the shortcut from a metric to a trace, so monitor exemplar receive and jump-success rates; the reference path must never block metric ingestion.
How do you prove sampling is not biased toward one tenant?
Compare request, sampled, and hit counts by tenant, region, version, and result class. Set minimum guarantees and maximum budgets, and compare error and tail-latency hit rates. Correct bias with stratified sampling rather than raising the global rate.