Prompt and context
A service is adopting OpenTelemetry and the team wants userid, requestid, and full URLs as metric attributes. Design a cardinality budget, explain what stays, what happens at the limit, and how you prove alerts remain useful.
OpenTelemetry metrics create time series from attribute combinations. An SDK cardinality limit is a hard limit on the metric points tracked for a metric during a collection cycle. High-cardinality fields multiply memory, export, storage, and query costs, so a budget must protect both cost and diagnostic value.
What the interviewer is testing
The interviewer is testing whether you separate metric, log, and trace dimensions, understand combinations rather than single-field counts, design hard limits and degradation, and make service graphs, alerts, sampling, and privacy constraints operational.
Clarifying questions
Confirm metric purpose, query window, alert latency, tenant count, endpoint count, collection interval, backend retention, and budget. Ask whether trace or log correlation fields already exist, which identity fields are privacy-sensitive, and whether the limit should favor new values, old values, or aggregate accuracy.
30-second answer
“I would define allowed dimensions by metric purpose instead of putting all context into metrics. Keep stable, low-cardinality fields that support grouped alerts, such as service, region, route template, and status class; put userid, requestid, and raw URLs in traces or logs. Set a cardinality limit, cost alert, and saturation signal per metric, with an explicit aggregation or drop policy. Finally, replay historical traffic to measure alert recall, false positives, query cost, and privacy.”
Deep-dive answer
Step 1: Define the metric question
Write down the question: Is the error rate rising, which route is affected, which region is degraded, or what happened in one user request? The first cases fit metrics; the last belongs in traces or logs.
Step 2: Estimate combination cardinality
Cardinality is the unique set of attribute combinations, not the sum of values in each field. Estimate the product of service, region, route template, status, method, and tenant tier, then correct it with real distributions, long tails, and bursts.
Step 3: Layer the fields
Keep stable aggregatable fields; put users, requests, and complete URLs in traces or logs. For high-cardinality tenant identifiers, use bucketing, hashing, or sampling only when access control and investigation needs remain satisfied. Never create unbounded series from raw path parameters.
Step 4: Configure SDK and backend budgets
Set budgets in the SDK, Collector, time-series backend, and query layer rather than truncating only at the end. The OpenTelemetry metric cardinality limit is a hard cap, and the retained attribute set must be observable in implementation and alerts.
Step 5: Define limit behavior
Specify which combinations survive, whether an overflow bucket is used, whether new sets are dropped, and how the decision is counted. The policy must be stable and explainable, with a saturation signal; silent loss is not acceptable.
Step 6: Put correlation in the right signal
Use traceid for a request, logs for a raw URL or userid, and exemplars or links to jump from a metric to a sample. Metrics provide trends and alerts; they do not replace per-request diagnosis.
Step 7: Budget service graphs and cost
Service graphs and auto-instrumentation can create many metric families. Set separate budgets for edges, clients, servers, and error states, and control export frequency, retention, and high-cardinality attributes. Attribute cost reports to services and metrics rather than looking only at the total.
Step 8: Validate alert quality and privacy
Replay traffic and inject failures to compare recall, false positives, latency, and query cost with and without the budget. Check redaction, access control, deletion requests, and tenant isolation, and make saturation, configuration changes, and data loss traceable.
Model answer
I would separate trend alerting from single-request investigation. Service, region, route template, method, and status class are usually stable, low-cardinality metric attributes; userid, requestid, and parameterized URLs belong in traces, logs, and exemplars. I would estimate the product of attribute combinations and correct it with long-tail traffic, then configure cardinality and cost budgets in the SDK, Collector, and backend. At the limit, use an explicit overflow or drop policy and record saturation rather than failing silently. Budget service-graph edges, clients, and error states independently, controlling collection and retention. Before launch, replay traffic and inject high-cardinality and failure cases, comparing alert recall, false positives, query cost, privacy, and tenant isolation.
Common mistakes
Counting each field independently
Series come from combinations. Several medium-cardinality fields can multiply into an explosion, so estimate combinations, long tails, and bursts.
Putting user_id in every metric
User-level investigation belongs in traces and logs. Identity dimensions in metrics add cost, privacy risk, and query noise.
Silently dropping data at the limit
Silent loss makes alerts look healthy. Record saturation, retention rules, and configuration version, then test degraded diagnosis.
Follow-up questions and answers
How should route templates differ from full URLs?
Metrics use normalized route templates so path parameters do not create new series. Full URLs go to controlled logs or traces with privacy redaction.
Should a limit preserve old or new attribute sets?
It depends on the alert and aggregator, but the rule must be fixed, explainable, and observable. Count overflow so instances do not behave inconsistently.
How do you correlate metrics, traces, and logs?
Use traceid, spanid, or exemplars to link samples. Keep stable dimensions in metrics, request context in logs and traces, and enforce the access boundary when navigating.
Why does a service graph need its own budget?
Automatically generated edges, clients, and error dimensions multiply series. Split budget by edge type, collection frequency, and retention so the graph cannot crowd out business metrics.
How do you prove the budget is not hiding failures?
Replay real traffic and inject high-cardinality, error spikes, and tenant long tails. Compare recall, false positives, latency, and query results before and after the budget while watching saturation events.
When should you add logs instead of metrics?
When the question needs one request, raw input, or user context, use controlled logs or traces. Metrics should carry aggregatable trends and alert dimensions.