1. Question
An asynchronous order-processing service gets slower at peak traffic. Monitoring shows stable throughput near 200 req/s and end-to-end average latency near 150 ms. Use Little's Law to estimate in-flight requests, explain the relationship between latency, throughput, and queues, and propose engineering actions that prevent an unstable backlog.
2. Constraints and clarifications
- Little's Law describes a long-run average for a stable system:
L = λW, where L is average work in the system, λ is average throughput, and W is average time in the system. - State the measurement window, request boundary, and units; a short burst or an unstable system cannot be treated as a long-run average.
- Separate service time, queue time, and end-to-end residence time, or concurrency and thread-pool sizing will be understated.
- Clarify capacity limits, timeout policy, priorities, and work that may be dropped.
3. Core derivation
Multiplying 200 req/s by 0.15 s gives L = 30 requests in the system on average. This is not a maximum of 30 requests and not p99 concurrency; it is the average inventory for that window. If throughput stays constant while average residence time doubles, average in-flight work also doubles, usually indicating a growing queue or slower dependency.
4. Reference analysis
lambda = 200 # requests / second
W = 0.150 # seconds / request
L = lambda * W # 30 requests in the system on average
if arrival_rate > sustainable_service_rate:
queue grows without a stable bound
apply_admission_control_or_scale_out()
capacity = concurrency_limit / target_latencyMeasure when work enters a queue, starts processing, and completes. Little's Law can estimate a rough capacity ceiling: with a concurrency limit of 100 and a 200 ms target average residence time, stable throughput is about 500 req/s. Leave headroom for tail latency, bursts, and dependency jitter.
5. Overload cases and trade-offs
When arrival rate stays above service rate, the queue grows, W increases, and L increases, creating a feedback loop of timeouts and retries. An unbounded queue only delays failure; work may be useless when it finally completes. Use bounded queues, fail-fast behavior, priorities, load shedding, backpressure, or scale-out. Each policy must state what work is dropped and how callers are informed.
6. Verification and observability
- Record arrival rate, completion rate, in-flight work, and average plus p95/p99 latency by time window.
- Cross-check three independent measures of
L,λ, andWto catch unit or sampling-boundary errors. - Run a controlled load test that increases arrival rate gradually and observes queue length, timeout rate, and recovery time.
- Alert on queue depth, age, concurrency, rejection rate, and retries; verify how quickly they fall after scaling or shedding load.
7. Common mistakes
- Treating average L as a hard concurrency limit and ignoring bursts, tail latency, and queue distributions.
- Using service time instead of end-to-end W and missing network, lock, and dependency waits.
- Inferring long-run capacity from a short sample before the system is stable.
- Scaling consumers without limiting producers, leaving shared dependencies or downstream queues overloaded.
8. Interview scoring points
Substitutes into the formula correctly
The candidate keeps units consistent, computes 200 × 0.15 = 30, and explains that this is average in-flight work rather than a limit.
Defines time boundaries
The candidate separates queue, service, and end-to-end time and states the sampling window and stability assumption.
Recognizes overload feedback
The candidate explains how arrival rate above service rate amplifies queue, latency, retries, and concurrency, then proposes bounded controls.
Validates capacity with data
The candidate uses load tests, p95/p99, queue age, rejection rate, and recovery time instead of reporting one average number.