Backend Interview: How Do You Protect a Service from Overload?

Prompt and When This Question Applies

An order service has a load-tested sustainable capacity of 3,000 requests per second at a p99 latency of 250 ms. During a promotion, arrivals rise to 8,000 requests per second. An inventory dependency slows down, the number of in-flight requests and queue age both increase, clients retry, and autoscaling needs three minutes before new instances are ready. Critical order submissions, interactive availability checks, and internal batch reconciliation share the service.

Design an overload-protection policy that keeps the process responsive and preserves as much useful work as possible. Explain detection signals, concurrency and queue bounds, admission control, request priority, graceful degradation, retry behavior, recovery, and validation. The figures are interview assumptions, not a claim about a particular production system.

This is a backend question because the core decision is how one service protects its CPU, memory, threads, connections, and downstream calls. A distributed rate limiter enforces traffic policy across tenants or time windows; this question starts after permitted traffic is still greater than the service's current capacity. Timeouts, retries, and circuit breakers protect individual dependency calls, while overload control decides which new work the service can afford to admit.

What the Interviewer Is Evaluating

A strong answer identifies saturation from local work, not from request rate alone. A fixed request-per-second threshold fails when one request becomes five times more expensive or a dependency slowdown holds connections longer. In-flight work, queue age, available worker or connection slots, CPU, memory pressure, and deadline slack expose the resource that is actually running out.

The next signal is a bounded system. Unbounded queues turn excess demand into memory growth and old requests. A candidate should cap concurrent work at the bottleneck, keep the queue small or eliminate it, and reject cheaply before expensive parsing or downstream calls. The objective is useful successful work, not accepting every request or maximizing raw attempts.

The interviewer also wants a business-aware policy. Critical order submissions can receive a reserved share; availability checks may use a short-lived cache; batch reconciliation can pause. Priority still needs per-tenant fairness so one large customer cannot consume every reserved slot. Finally, the design needs a closed recovery loop: suppress retry amplification, scale as a slower capacity response, ramp admission back gradually, and test the degraded path before an incident.

Questions to Clarify Before Answering

• What resource saturates first? CPU saturation favors a local concurrency or cost limit. A slow dependency calls for a separate connection/concurrency bulkhead. Memory growth or queue age may require a smaller queue and earlier rejection. Each bottleneck changes the admission signal.
• Which operations are essential and what can degrade? Here, submitting an order is critical, availability can tolerate slightly stale data, and batch reconciliation can pause. If every operation is legally required to use fresh inventory, the cache fallback is invalid and explicit failure is safer.
• What is the end-to-end deadline? Queue time consumes the same deadline as execution. The service should discard work that can no longer finish and propagate cancellation downstream.
• Are requests equally expensive? A single token per request works only when costs are similar. Expensive endpoints may need weighted permits or separate pools so cheap critical work is not trapped behind costly batch jobs.
• Can clients safely retry? A rejected read may be retried later. An order write needs an idempotency key and outcome lookup. The service must distinguish “retry later” from “do not retry,” and all layers must share a retry budget.
• Is traffic shift available? Healthy-region failover can absorb work only if the target has verified spare capacity. Blindly moving overload can create a second failure.

30-Second Answer Framework

“I would protect the bottleneck with a small closed loop. First, use load tests to set per-instance concurrency and queue bounds, then watch local in-flight work, queue age, CPU, memory, connection pools, and remaining deadlines. When saturation rises, reject new work before expensive processing, reserve capacity for order submissions, preserve fairness within each priority, serve an approved cached response for availability, and pause batch work. Rejections use a clear temporary-overload response, while clients retry only idempotent operations with backoff, jitter, and a shared retry budget. Autoscaling adds capacity but is too slow to be the first defense. I would recover with hysteresis and a gradual admission ramp, then verify bounded memory, useful throughput, critical success, fairness, retry amplification, and recovery under overload tests.”

Step-by-Step Deep Dive

Start with a measured capacity envelope. The 3,000 requests-per-second figure is valid only for the tested request mix, dependency latency, instance count, and 250 ms p99 target. Record the corresponding per-instance in-flight work, CPU, memory, worker utilization, and downstream connection use. Request rate is an input; the admission decision should follow the resource closest to failure. For example, when inventory slows, the same arrival rate creates more in-flight calls, so a rate-only controller reacts too late.

Place independent concurrency limits around scarce resources. The order handler needs an overall cap, while inventory calls and batch jobs get smaller bulkheads. A permit is acquired before allocating expensive work and released on success, failure, timeout, or cancellation. If request costs differ materially, use weighted permits or separate endpoint pools. A hard-coded limit is a safe starting point from load tests; an adaptive limit may improve utilization, but it requires stable feedback, guardrails, and a quick rollback.

Keep queueing explicit and bounded. A short queue can absorb a known burst, but its limit should come from deadline slack rather than available memory. Reject when the queue is full or when estimated queue delay leaves too little time to finish. An unbounded queue cannot create capacity; it increases tail latency, retains memory, and makes clients retry requests that are already waiting. Monitor both queue depth and the age of the oldest item because a small queue of expensive work can still be stale.

Admission control should fail cheaply and consistently. At the gateway, enforce contractual tenant quotas and coarse traffic limits. At each service instance, use local saturation to protect the task that owns the resource. If a downstream microservice is overloaded, reject upstream before performing work that will be discarded later. Carry request criticality along the call path so the same order is not admitted by one layer and randomly rejected after consuming work at another.

Use a priority policy with explicit capacity reservations:

Priority without fairness can starve small tenants. Apply tenant quotas or fair scheduling inside a class, and keep a minimum share for recovery or control traffic. Avoid dozens of priority levels: operators must be able to predict which request is admitted during an incident.

When the limit is reached, return before database or dependency work. HTTP 503 represents temporary overload and may include Retry-After; it does not grant permission for every client to retry simultaneously. Clients use capped exponential backoff with jitter, a deadline, and a retry budget. Retry at one suitable layer instead of every layer. Order creation reuses an idempotency key and queries the previous outcome after an ambiguous timeout. Requests whose caller deadline has expired are cancelled so the server does not complete useless work.

Graceful degradation reduces cost rather than merely rejecting. Availability can skip optional enrichment or use a bounded-age cache if the product contract allows it. Batch consumers can stop pulling messages. A fallback that reports stale stock as current is dishonest; when freshness is mandatory, return an explicit unavailable result. Exercise degradation continuously on a small fraction of traffic because an unused emergency path is likely to fail when needed.

Autoscaling, regional spillover, and capacity increases remain useful, but they operate after admission control. Scaling on raw request count can add instances during cheap traffic and lag during expensive traffic; include concurrency, queue age, or resource saturation. New instances should warm connections before receiving a full share. Failover needs a destination capacity check. Neither mechanism justifies removing local bounds.

Recovery uses a lower exit threshold than the entry threshold. After in-flight work, queue age, and dependency health remain below that threshold for a hold period, increase admitted load in steps. Keep priority reservations until normal latency is stable. This hysteresis and ramp prevent the system from toggling between open and overloaded states or flooding a dependency that has only partly recovered.

Validate the policy beyond the nominal capacity test. Replay the production request-cost mix, then raise arrivals from 3,000 to 8,000 requests per second while slowing inventory and delaying autoscaling for three minutes. Add synchronized client retries, one abusive tenant, expired deadlines, and recovery of the dependency. Assert bounded queue and memory, stable worker and connection usage, cheap rejection, order success within its reservation, tenant fairness, honest degradation, controlled retry amplification, and a gradual return to normal. Measure completed useful operations separately from accepted requests and downstream attempts.

High-Quality Sample Answer

“The sustainable 3,000 requests per second is a load-test result for one request mix, so I would first identify the resource at that boundary. During the inventory slowdown, in-flight calls and connection occupancy are more useful than request rate. I would set a tested per-instance concurrency cap for the service, a smaller bulkhead for inventory, and a small queue whose wait still fits the request deadline. Once either bound is reached, the service rejects before expensive work.

I would classify traffic into order submission, availability, and reconciliation. Orders receive reserved capacity but still have a hard limit. Availability may use a short-lived cache only if the API exposes that freshness contract. Reconciliation pauses and resumes from durable progress. Within every class I would enforce tenant fairness, so one customer cannot take the whole reservation. I would also propagate the priority downstream to avoid spending work on a request that a later service randomly drops.

Temporary overload returns a recognizable 503 and, when we can estimate it, Retry-After. Clients still need capped backoff, jitter, deadlines, and a shared retry budget. Only one layer retries, and order writes reuse an idempotency key. Expired callers cancel downstream work.

Autoscaling is the slower capacity loop because instances need three minutes. I would scale on saturation signals and warm new instances, while local admission control keeps the existing fleet alive. Recovery requires lower exit thresholds and a gradual ramp.

The proof is an overload test at 8,000 requests per second with slow inventory, delayed scaling, retries, mixed request costs, and a noisy tenant. I expect bounded memory and queues, stable useful throughput, the promised order reservation and fairness, low-cost rejection, no retry storm, and controlled recovery after inventory returns.”

Common Mistakes

• Raise the queue limit until errors disappear → Accepted requests wait longer, consume memory, expire, and trigger retries without adding execution capacity → Bound queueing from deadline slack and reject early.
• Detect overload only from requests per second → Request cost and dependency latency change, so the same rate can be safe or catastrophic → Use local in-flight work, queue age, resource saturation, and dependency pools.
• Let autoscaling be the first defense → The three-minute delay allows queues and retries to destabilize the current fleet → Keep local admission bounds, then scale to restore headroom.
• Give critical traffic unlimited priority → It can exhaust the same resource and starve recovery work → Reserve capacity but retain a hard cap and fairness.
• Randomly shed at every microservice → Upstream work is consumed before a later random rejection, reducing end-to-end useful success → Propagate criticality and reject as early as the bottleneck is known.
• Return 503 and let every client retry → Synchronized retries multiply the excess load → Use jitter, deadlines, one retry layer, and a retry budget.
• Serve an unmarked stale fallback → The system appears available while violating inventory semantics → Expose the freshness contract or fail explicitly.
• Restore full traffic immediately → The recovering dependency is overloaded again → Use hysteresis, bounded probes, and a gradual admission ramp.
• Track only accepted traffic → A high acceptance rate can hide timeouts, wasted work, and retries → Measure useful completions, rejection cost, deadline waste, and attempt amplification.

Follow-Up Questions and Responses

Follow-up 1: Why not solve this with a distributed rate limiter?

A distributed rate limiter is useful for contractual quotas, abuse control, and traffic shaping before requests reach the service. It cannot by itself observe that inventory latency has increased the cost of each permitted request. Keep the gateway limit, then add local concurrency and queue protection at the resource owner. If request costs are stable and the service has one bottleneck, a conservative rate limit may be the simpler sufficient solution.

Follow-up 2: How would you choose the concurrency limit?

Start from a load test that uses the production request mix and find the highest concurrency that still meets the latency and resource targets with headroom. Repeat it with the dependency slowdown. A steady-state relationship between concurrency, throughput, and time can provide a sanity check, but it does not guarantee capacity under bursty arrivals or mixed costs. Roll out the limit in observe-only mode, canary rejection, and then enforcement; review it when code, instance size, or dependencies change.

Follow-up 3: What if 90% of traffic is marked critical?

Then the label no longer makes a useful admission decision. Define criticality from business operations, authenticate who may set it, cap each class, and reserve only a measured share. Within the critical class, use tenant fairness or a user-stable priority so overload does not arbitrarily favor the noisiest client. If truly critical demand exceeds physical capacity, some critical work must still fail; the contract should state how.

Follow-up 4: Could adaptive concurrency make the design better?

It can follow changing service time more closely than a static cap, but noisy latency and delayed feedback can cause oscillation or false shedding. Begin with the tested static bound. Add adaptive control only with minimum and maximum limits, smoothed signals, hysteresis, a stable fallback value, and replay tests covering fast deterioration and slow recovery.

Follow-up 5: Where should load shedding happen in a deep call graph?

The resource owner needs a local last line of defense. Once that service signals overload, upstream layers should stop work earlier and preserve the same criticality decision along the call path. Shedding only at the edge lacks precise downstream state; shedding only at the leaf wastes upstream work. The practical design combines a coarse edge policy, local protection, and an overload signal that upstream callers can act on.

Follow-up 6: Which production metrics show that the policy is working?

Track useful completions and latency by operation and priority; admitted, queued, degraded, and rejected requests; oldest queue age; in-flight work; CPU, memory, worker, and connection saturation; deadline-expired work; per-tenant fairness; physical attempts per logical request; autoscaling readiness delay; and time spent in overload mode. Alert on lost critical reservation, sustained overload, rejection cost approaching normal request cost, or a recovery ramp that repeatedly falls back.