Data interview: How would you design an OpenTelemetry exponential-histogram pipeline with cost controls?
Prompt
Design a latency-metrics pipeline for multi-region services: SDKs emit OpenTelemetry ExponentialHistogram data, a Collector aggregates it, and a backend writes Prometheus native histograms. Explain semantics, downgrade paths, and cost controls.
Scenario and constraints
Latency spans microseconds to minutes and services have high-cardinality labels. Some backends support only classic buckets, batches can be lost, and queries need p50, p95, and cross-region aggregation. Metrics must not expose tenant data or create unbounded time series.
What this tests
The test covers exponential boundaries, scale, positive and negative buckets, zero count, count/sum, temporality, and mergeability across processes. OpenTelemetry compresses exponential boundaries for high dynamic range at small relative error; Prometheus native histograms can map the model, but conversion and queries must preserve meaning.
Reference approach
Record only approved business dimensions and observations in the SDK. Merge the same schema in the Collector by service, region, and fixed window. Convert explicitly to bounded classic buckets for unsupported backends and record precision loss. Limit scale, bucket count, label sets, and per-tenant budgets; sample down or reject new labels at the limit instead of silently truncating.
Critical details
Check temporality, aggregation temporality, schema, positive and negative buckets, and time range before merging; different schemas cannot be added directly. Query p95 as an approximation and show sample count, error, and downgrade markers. Use batch sequence numbers and retries to avoid counting a batch twice.
Common traps
Calling exponential buckets exact quantiles; adding different temporalities; allowing unbounded labels; converting everything to the finest scale; ignoring negatives, zero buckets, and retry duplication; and measuring storage without query amplification.
Evaluation rubric
Strong answers define SDK, Collector, remote-write, query, and budget-control boundaries; state merge invariants and downgrade policy; and explain p95 error. “Use histogram_quantile” without the data model or cost analysis is insufficient.
Follow-up questions
Why cannot different exponential schemas be merged directly?
Their bucket indexes map to different boundaries, so addition would place observations in the wrong ranges. Remap to a compatible schema according to the specification and record the precision change.
How do you handle cumulative and delta temporality together?
Convert explicitly in the Collector while retaining each stream's start and previous values. If continuity is missing, drop or mark the incomplete window; never add cumulative values as deltas.
When should you fall back to classic histograms?
Use bounded classic buckets when the backend lacks native support, compliance requires fixed ranges, or the query ecosystem cannot interpret exponential buckets. Publish the error and cost trade-off instead of making consumers guess.