Problem and scope
Discuss single-request latency distributions, composed requests, and service objectives. Assume 100,000 requests per minute, 80 ms average, 180 ms p95, and 2 s p99. These are interview assumptions, not industry benchmarks. Focus on reasoning and verification rather than a specific vendor tool.
What the interviewer is evaluating
The interviewer wants to see that an average describes total work, not distribution shape. Tail requests may cluster by tenant, route, dependency, region, request size, or retry path. Strong answers define measurement boundaries, aggregation dimensions, user impact, server causes, and an actionable SLO.
Clarifying questions
- Is latency measured to first byte or complete response?
- Is p99 per instance, route, region, or the whole service, and over what window?
- Are timeouts, cancellations, retries, and cache hits included?
- Do request size, tenant, dependency calls, or errors change in the tail?
- Is the goal to improve everyone or protect a critical-path tail budget?
A 30-second answer framework
“An 80 ms average can coexist with a 2 s p99; the slowest 1% can still trigger user and upstream timeouts. I would fix the boundary and window, then split the distribution by route, region, instance, tenant, request size, and dependency. I would inspect queues, pools, GC, disk, locks, downstream calls, and retries. The fix should match the bottleneck: bounded queues, sensible timeouts, batching or caching, fewer serial dependencies, noisy-tenant isolation, and finite retry budgets. Validate with p95/p99 SLOs and an error budget.”
Step-by-step deep design
Define observation points: client total time, edge-to-service, service processing, downstream wait, and response transfer. Choose first-byte or complete-response semantics, use a monotonic clock, and document sampling. Count timeouts and cancellations separately; link retries to the original request so one user action is not counted as multiple independent successes.
Percentiles must come from request samples; averaging instance p99 values is invalid. Break down percentiles and sample counts by route, status, region, instance, tenant, request size, and dependency. Short windows reveal regressions but jitter; long windows stabilize SLOs but can hide a bad release. Show both.
Composed requests amplify tails. Serial calls add their stages; parallel calls finish near the maximum child latency. Assign budgets to each stage, record trace spans, and inspect fan-out. A service average cannot predict end-to-end page experience.
Diagnose correlation before guessing. Rising queue age suggests capacity or admission pressure; pool wait indicates concurrency limits; GC, disk, and CPU jitter create tails; a slow downstream p99 propagates upstream. Compare successful and timed-out samples by request attributes and retain controlled context for slow traces.
Match the remedy to the cause. Fewer serial calls, caching, or batching reduce fixed overhead; bounded concurrency, backpressure, and tenant isolation prevent congestion; finite timeouts and retries with jitter avoid retry storms; asynchronous work removes unpredictable tasks from the synchronous path. More threads or replicas alone can lengthen the tail.
Express an SLO as the fraction of qualifying requests meeting a threshold, such as 99.9% under 300 ms. An error budget combines latency violations and errors in release decisions. State its ingress, statuses, cache semantics, and window. An average-only target can remain green while the slowest users suffer.
Validate distribution changes in releases and drills. Compare p50, p95, p99, maximum, timeout rate, and sample count before and after. Inject dependency latency, create a large-tenant request, and exhaust a pool to verify isolation and degradation. After recovery, inspect queue drain and retry amplification, not just the mean.
High-quality sample answer
“The 80 ms average and 2 s p99 mean a clear long tail; the slowest 1% can trigger user or upstream timeouts. I would standardize timing boundaries, separate first byte, complete response, cancellation, and retries, then split by route, region, instance, tenant, request size, and dependency. A page also sums serial calls and takes the maximum of parallel calls, so trace spans need per-stage budgets.
After identifying queues, pools, GC, disk, or downstream latency, I would use bounded concurrency, caching or batching, fewer serial dependencies, tenant isolation, and finite retries. I would validate with a 99.9%-under-300-ms SLO, an error budget, and percentile comparisons across releases, ensuring the tail improves without hiding errors or correctness problems.”
Common mistakes
- Report only the average → tail users disappear → report percentiles, samples, and timeouts.
- Average instance p99 values → percentiles are not linearly averaged → aggregate samples or histogram buckets correctly.
- Count retries as new user requests → load and experience are misrepresented → link attempts to the original.
- Measure server time only → network, queue, and transfer vanish → define an end-to-end boundary.
- Add threads without bounds → contention and queueing lengthen the tail → use bounded concurrency and backpressure.
- Retry every timeout blindly → a retry storm collapses dependencies → use budgets, jitter, and deadlines.
- Set one global SLO → a tenant or region can degrade silently → slice by critical dimensions.
- Watch only the mean after release → regressions stay hidden → compare multiple percentiles and windows.
Follow-up questions and answers
Follow-up 1: Why is p99 not the slowest request?
p99 means about 99% of samples are no slower than that value; roughly 1% are slower. The maximum is sensitive to one anomaly and sample size, so both have different uses.
Follow-up 2: A page has five parallel calls. How do you estimate experience?
The parallel stage is near the maximum child latency plus client scheduling and transfer. Measure page total time and trace which child most often becomes the maximum.
Follow-up 3: When is higher tail latency acceptable?
Offline exports or low-frequency background jobs may trade latency for throughput; interactive critical paths usually cannot. Define SLOs per request class.
Follow-up 4: Why are histograms better than averages?
Histograms retain buckets and distribution shape, allowing several percentile estimates and tail inspection. Aggregation still needs attention to bucket boundaries and sample count.
Follow-up 5: How do you separate queue tail from downstream tail?
Record wait, processing, and downstream spans separately. Queue and service time rising together suggests capacity; only downstream growth points to the dependency or its retries.
Follow-up 6: How does an error budget help latency work?
Count threshold violations as budget consumption. Rapid burn pauses risky releases and prioritizes tail fixes; remaining budget allows controlled experiments.