Prompt and scope
A client sends a create-order command. The server may commit the order, then lose the connection before sending a response. The client cannot know the outcome and retries with the same Idempotency-Key. Design the request contract, durable record, concurrency control, and recovery flow, and state which failures are safe to retry. The core skills are side-effect boundaries, atomicity, and explainable failure states, so this belongs to backend.
What the interviewer evaluates
A strong answer separates “request never arrived,” “completed but response was lost,” and “still processing,” then chooses replay, status lookup, or an in-progress response. It also covers same-key/different-parameters, concurrent first requests, storage failure, expiry, multi-region routing, and downstream effects instead of only saying “put the key in Redis.”
Questions to clarify first
- Is the key generated by the client for one logical command, or derived by the server from business fields?
- Which request fields are bound to the key, and what error represents a mismatch?
- Are the order write and idempotency record in one database transaction? Does payment have its own idempotency contract?
- How long may the client wait, how long is the key retained, and can expiry create a second order?
- Is the service single-region or multi-region, and which store is authoritative for every ingress path?
30-second answer framework
“The client creates one stable key for one logical command and reuses it on retries. The server enforces a durable uniqueness constraint over tenant, operation, and key, storing a request fingerprint, state, and final response. The first request atomically claims processing and runs the order transaction; same-key/same-parameters requests wait or replay, while a mismatch is rejected. After a crash, recover from transaction and downstream evidence. Query unknown side effects before compensating, never blindly retry. Retention must cover the retry window, and metrics must prove zero duplicate side effects.”
Step-by-step solution
The contract should require a non-empty, bounded key that remains unchanged across retries of one logical command. The server stores a unique (tenantid, operation, idempotencykey) together with a normalized request hash, state, resource ID, response code, response body, and expiry. The hash prevents accidental reuse of one key for a different command.
The first request should insert a durable processing record and create the order in one local transaction, or use a transaction log that unambiguously links the two. On a uniqueness conflict, read the existing record: replay the saved response for succeeded, return the same business error for failed, and wait briefly or return in-progress for processing. An in-memory lock alone fails on restart and across replicas.
Concurrency is decided by the uniqueness constraint and conditional updates. Only the request that owns the creation record may move processing to succeeded; include a version or expected state in the update so two workers cannot commit. If the order transaction succeeded before a process crash, a retry reads succeeded and replays it. If the transaction rolled back, a retry can execute safely.
Downstream effects create a window where the local result is unknown while the remote effect may have succeeded. Payments, shipping, and messages should use the same business operation ID with a downstream idempotency contract, and persist request and result. Without downstream idempotency, query reconciliation or publish through an outbox and retry from a worker; never charge again just because the local call timed out.
An in-progress record needs recovery. Store a lease and heartbeat; a recovery worker checks the order transaction, downstream status, or transaction log before advancing to a terminal state. If evidence is insufficient, mark unknown and send it to reconciliation rather than treating it as failure. The state machine must not move succeeded back to processing.
Expiry must match business risk. Retain the record through the maximum client retry, network retry queue, and compensation window. Reusing an expired key should return an explicit key_expired, not silently create a second order. Old response bodies can be compacted only if an audit digest and resource-level uniqueness remain.
In multi-region deployments, route a logical key to one authoritative store or enforce a global uniqueness constraint with synchronous replication. Do not execute again on another replica merely because a processing read is delayed. Measure key conflicts, parameter conflicts, in-progress timeouts, replay responses, unknown states, duplicate-resource blocks, and reconciliation differences.
Model high-quality answer
“I define an idempotency key as the stable identity of one logical create command, not a new request ID on every HTTP retry. The server enforces a durable uniqueness constraint over tenant, operation, and key, storing the request hash, state, resource ID, response, and expiry. The first request atomically claims processing and creates the order; same-key/same-parameters requests replay or wait, and a mismatch is rejected.
I keep the order write and idempotency record in one transaction and pass the same operation ID to payment or other downstream effects. If the response is lost, a retry reads the stored result; if the local state is unknown, I query downstream and reconciliation records rather than guessing with another POST. A leased worker recovers stuck processing, and conditional transitions lead only to succeeded, failed, or unknown. Retention covers retries and compensation. I inject concurrent first requests, process crashes, lost responses, and cross-region delay, and assert zero duplicate orders or charges.”
Common mistakes
- Generate a new key for every retry → the server cannot identify one logical command → reuse the original key.
- Use only an in-memory or single-host lock → restart and replicas lose deduplication → enforce durable uniqueness.
- Replay a result for a different payload → hide a client bug → store a fingerprint and reject the mismatch.
- Call downstream immediately after recording
processing→ a crash leaves the effect unknowable → use a transaction, outbox, or downstream idempotency. - Charge again after a timeout → the remote charge may have succeeded → query and reconcile first.
- Treat a stuck record as failed and rerun → create a second resource → gather evidence in recovery.
- Expire keys too quickly → delayed retries create duplicates → align retention with risk and retry windows.
- Test only serial calls → races still double-write → test same-key concurrency, crashes, and multi-region routing.
Follow-up questions and responses
Follow-up 1: Why not use the order ID as the unique key?
The order ID usually exists only after the server creates the resource, so it cannot cover the window before the first response. The idempotency key exists before the side effect and binds retries to one logical command.
Follow-up 2: What if the payload changes with the same key?
Hash the normalized body and relevant headers. A different fingerprint returns a parameter-conflict error without a new side effect; the client must create a new key for a new command.
Follow-up 3: What if the first request stays in processing forever?
Use leases, heartbeats, and timeout scans. A recovery worker checks local transactions, downstream results, and message logs; only sufficient evidence advances the state, otherwise it becomes unknown for reconciliation.
Follow-up 4: Can Redis be the only idempotency store?
If the database owns the side effect, Redis eviction, failure, or replication lag can remove the safety boundary. Redis can coordinate short-lived work, but final state and uniqueness belong in durable storage consistent with the business write.
Follow-up 5: What should happen after key expiry?
Do not silently accept the old key. Return an expiry error and direct the client to query the original order or create a new command; resource-level business uniqueness should provide another guard.
Follow-up 6: How do you prove duplicate effects are absent?
Send the same key concurrently, kill the process before and after commit, lose responses, and time out downstream calls. Check resource uniqueness, operation IDs, replayed responses, state-transition logs, and reconciliation. The assertion is at most one successful side effect per logical key.
Follow-up 7: Is an idempotency key exactly-once processing?
No. It makes one service recognize duplicate commands; it does not make a cross-service network exactly-once. Transactions, outbox delivery, downstream idempotency, queries, and reconciliation are still required, with explicit unknown outcomes.