Prompt and context
An API request calls user, catalog, and recommendation gRPC services in sequence. The user may navigate away while a recommendation call is slow. The system must bound end-to-end latency, release server resources, avoid retry amplification, and keep writes queryable when cancellation happens after an external side effect.
The gRPC guides define a deadline as the client’s upper bound for waiting and describe DEADLINE_EXCEEDED; cancellation terminates an RPC that is no longer needed. A strong answer separates budget propagation, retry limits, stream cleanup, and unknown outcomes instead of assigning every layer an independent five-second timeout.
What the interviewer is assessing
Look for an absolute deadline or remaining budget, propagation into child RPCs and databases, retries that consume the same total budget, and a refusal to infer “no response” means “write did not happen.” The candidate should also name observable fields and a failure-injection test plan.
Clarifying questions
- What are the top-level SLO, client deadline, and partial-result policy?
- Are the three calls read-only, or do they include payments and other side effects?
- Which statuses are retryable, and is there an idempotency key?
- Can stream messages be replayed, and is there a cursor or sequence number?
- Do the database and external APIs accept cancellation?
- After cancellation, is the contract status lookup, compensation, or reconciliation?
30-second answer
“Create one absolute deadline at the edge; children inherit only the remaining budget. Propagate cancellation to every RPC, database, and cancellable worker. Retry only safe transient failures with an attempt cap, backoff, jitter, and the same total deadline. Writes use an operation ID or status lookup, not a new blind retry. Streaming calls close iterators and connections, and logs record remaining budget, cancellation reason, attempt, and final status.”
Step-by-step solution
Step 1: Define the end-to-end time budget
Treat the client deadline as an upper bound. Each service computes remaining = deadline - now, sets child deadlines no later than that point, and reserves time for queueing, serialization, and response handling.
parent_deadline = 2.0s from request start
child_deadline = min(parent_deadline, now + remaining_budget)Do not give every layer another two seconds: a three-hop chain could exceed six seconds. Use a monotonic clock locally for queue and work measurements while the RPC framework carries the deadline.
Step 2: Propagate cancellation and stop work
Navigation, explicit cancel, and deadline expiry must reach the server handler. Pass the handler’s cancellation token to databases, HTTP clients, queue waits, and stream iterators. Returning an error while background work continues is a resource leak.
Cancellation should release connections, temporary files, and locks; stop producing new messages; and record completed stages. It cannot undo a committed transaction, so later status lookup or compensation is required.
Step 3: Retry inside the remaining budget
Retry only explicitly transient errors such as UNAVAILABLE, with maximum attempts, exponential backoff, and jitter. Every attempt uses the same deadline; expiry skips remaining retries.
for attempt in 1..maxAttempts:
if remaining(deadline) <= backoff: stop
result = call(child, deadline, cancellation)
if result is success or permanent_error: return result
sleep(jittered_backoff, cancellation)
return DEADLINE_EXCEEDEDChoose one retry owner across the call chain. Track retry amplification and per-layer attempts so a proxy and a client do not multiply work.
Step 4: Separate reads from writes
Reads may retry when the contract allows it. Writes need an operation ID, a unique constraint, and a status lookup. If cancellation follows an external write, the caller has an UNKNOWN result and must not create a new ID immediately.
PENDING -> CONFIRMED
\-> FAILED
\-> UNKNOWN (reconcile before retry)The status endpoint returns the authority’s result. A deadline controls waiting; it does not create an atomic rollback across systems.
Step 5: Handle streaming RPC lifecycles
Set a total deadline and, where appropriate, an idle timeout. Check cancellation on every read. When the client leaves, stop generation and close the database cursor or subscription. Reconnect with a cursor or sequence number rather than replaying side effects from the beginning.
Record last-message time, cancellation reason, reconnects, and backlog. During graceful shutdown, let active streams finish within their remaining deadline or return a recognizable cancellation status.
Step 6: Verify propagation and resource limits
Test parent cancel, each child timeout, slow database work, transient UNAVAILABLE, permanent errors, stream interruption, response loss after a write, multi-layer retries, and graceful server stop. Assert children stop before the parent deadline, cancellation leaves no background work, and attempts stay within budget.
Record RPC method, trace ID, remaining deadline, attempt, status, cancellation source, and operation ID without sensitive payloads. Load tests must observe connections, queues, retry amplification, CPU, and tail latency rather than average success alone.
Model answer
“I create an absolute deadline at the edge and propagate the remaining budget to all three services. Every handler passes cancellation to its database, HTTP call, and stream iterator and releases resources before returning. Read calls retry only safe transient errors, with one total deadline and bounded jitter.”
“Writes carry an operation ID; after cancellation I mark the result UNKNOWN and query status. Streams use a total deadline, cursor, and reconnect policy. I inject parent cancellation, child timeouts, lost responses, and graceful stop, then verify no leaks or retry amplification.”
Common mistakes
- Reset a full timeout at every layer → the chain exceeds the edge SLO → propagate one absolute deadline.
- Return from the handler and call it cancelled → downstream work still runs → propagate cancellation and clean up.
- Retry every error → load amplifies → retry by semantics, budget, and attempt cap.
- Create a new write ID after timeout → the first write may have succeeded → query authority or compensate.
- Reconnect a stream from the beginning → messages and effects duplicate → use a cursor, sequence, and idempotent consumer.
- Log only the final error → no layer explains budget exhaustion → log remaining deadline and attempt per hop.
Follow-ups and responses
Follow-up 1: May a child service extend the parent deadline?
No. If the operation needs longer, make it asynchronous and return an operation ID. Quietly extending a synchronous request defeats the edge SLO.
Follow-up 2: Can cancellation undo a committed database transaction?
Not reliably. It stops work that has not started or completed; committed work needs status lookup, compensation, or reconciliation. The API must expose observable state.
Follow-up 3: How do you prevent double retries?
Assign one retry owner and let other layers propagate errors. Share attempt and total-budget fields, cap attempts, and monitor amplification at every layer.
Follow-up 4: Should an idle stream always be cancelled?
Distinguish an intentional long-lived stream from an abandoned one. Use a total deadline, idle timeout, or heartbeat and return a reconnectable state instead of holding resources forever.
Follow-up 5: What if the server keeps computing after the deadline?
Make the handler and downstream work cancellation-aware. Move work that cannot be cancelled into a bounded background queue with an operation ID and resource limit, rather than keeping the request thread occupied.