Prompt and context
Order tasks are written to a Redis Stream and consumed by several workers in one consumer group. A worker can crash after an external API succeeds but before XACK, leaving the message in the Pending Entries List (PEL). The interviewer asks you to recover it without duplicate side effects, infinite retries, or silent loss.
This tests stream-processing delivery semantics and recovery design. XREADGROUP records delivered but unacknowledged messages in the PEL; XACK only acknowledges processing to the group. XAUTOCLAIM transfers idle pending messages to a consumer, but it does not make a business operation idempotent.
What the interviewer is evaluating
- Whether you explain new entries, pending entries, PEL, and consumer ownership.
- Whether you assign the right roles to
XACK,XPENDING,XCLAIM, andXAUTOCLAIM. - Whether claim, side effect, and acknowledgment form a retryable sequence.
- Whether idempotency keys, retry counts, and a dead-letter stream handle poison messages.
- Whether you monitor idle time, delivery count, PEL size, and recovery latency.
Questions to clarify first
- What side effect does one message cause? Charges, fulfillment, and notifications have different duplicate risks.
- Is there an idempotency key and authoritative state store? Without them, “at least once” is not safe.
- How long may a worker be idle before reclaim? The threshold must exceed normal processing and network jitter.
- Can the Stream be trimmed or deleted? Missing payloads need their own metric and alert.
- Are you using Redis 8.4
XREADGROUP CLAIM, or the compatible scan-and-claim flow for older versions?
A 30-second answer framework
“I would define at-least-once delivery and keep business idempotency outside the Stream. Workers read with XREADGROUP, commit the idempotent business state, and only then XACK. A recovery worker scans the PEL and uses XAUTOCLAIM for entries idle beyond a safe threshold. It limits retries by delivery count and error type, sends poison messages to a dead-letter Stream, and monitors PEL, idle time, claim count, acknowledgment latency, and duplicate suppression.”
Deep-dive answer
Draw the message lifecycle
XADD appends an entry to the Stream. XREADGROUP delivers it and records it in that consumer’s PEL. After business processing succeeds, XACK removes it from the group’s PEL. If the worker crashes first, another worker may claim it after the idle condition is met.
Set a safe idle threshold
min-idle-time for XAUTOCLAIM should exceed normal processing p99, reasonable dependency retries, and network jitter. Otherwise a slow but healthy worker can be reclaimed while still working. Scan with the returned cursor until 0-0, then start another cycle from the beginning because entries that were too young can become eligible later. The threshold is an operational parameter, not a universal Redis default.
Order claim and acknowledgment
After a recovery worker claims an entry, check state by message ID or business idempotency key before invoking the external side effect. Write the successful result and completed state, then XACK. If the state write and external call cannot share a transaction, record intent, outcome, and a compensating task. Admit that retries can repeat a call; do not present XACK as proof that the business commit happened.
Handle duplicates and concurrent claims
Several recovery workers may scan at once, and network retries can race with claims. Enforce idempotency on order ID, payment request ID, or a business key. Allow only valid state transitions such as pending to processing to completed. A duplicate that reads completed can be acknowledged and counted as suppressed without charging again. Delivery count alone does not identify a business duplicate.
Isolate poison messages
Each delivery increments a delivery count. Persistent failure may come from malformed payloads, a permanent business rule, or an unavailable dependency. Classify errors: malformed input can go directly to dead letter; temporary dependencies use backoff; entries that exceed a retry threshold move to a dead-letter Stream with original ID, last error, attempts, and business key. Give the dead-letter flow an owner and a compensation procedure.
Handle trimmed or deleted entries
If a pending entry’s Stream payload was trimmed or deleted with XDEL, XAUTOCLAIM may remove the ID from the PEL without redelivering a payload. Recovery metrics must distinguish “retried and completed” from “payload no longer exists.” For orders, plan retention, archival, or an external payload store so a cleaned ID is not reported as business success.
Monitor recovery quality
Monitor PEL size, maximum idle, claim rate, delivery-count distribution, XACK latency, dead-letter volume, and idempotency-suppression hits per group. Alerts should include stream, group, consumer, and business key. Kill a worker during a controlled drill and verify that the side effect completes once and the entry is eventually acknowledged or dead-lettered; command success alone is not enough.
Model high-quality answer
“I would use at-least-once delivery and store order idempotency state in the business store. A worker reads with XREADGROUP, marks the order processing, calls the dependency, writes the result and completed state, and only then sends XACK. A crash between those steps leaves the entry in the PEL.
A recovery worker uses XAUTOCLAIM for entries idle beyond normal p99 plus jitter. It checks the order or payment request key: completed entries are acknowledged and counted as suppressed; unfinished entries continue through the workflow. Malformed and permanent errors do not loop forever; temporary dependency errors back off, and entries over the delivery threshold go to a dead-letter Stream with their original ID and error context.
I would monitor PEL, idle time, claim count, acknowledgment latency, dead letters, and duplicate suppression, then drill trimming, worker crashes, and dependency timeouts. If XAUTOCLAIM cleans an ID whose payload is gone, Redis only says the payload is unavailable; it does not prove the order succeeded. Retention or archival must cover that case.”
Common mistakes
- Calling Redis Streams exactly once: acknowledgment does not include an external side effect → state at-least-once and add business idempotency.
- Acknowledging before completion: a crash can silently lose work → acknowledge after the successful business state.
- Setting idle shorter than p99: healthy workers are reclaimed → tune from processing distribution and jitter.
- Calling
XAUTOCLAIMforever: a poison entry burns resources → classify errors and use retry and dead-letter policies. - Using delivery count as duplicate detection: one business action can have different message IDs → enforce a business key and state machine.
- Ignoring trimmed pending IDs: cleanup is reported as success → alert on missing payloads and retain an external archive.
- Running recovery workers without a plan: claims and side effects race → use idempotent state, leases, or bounded concurrency.
- Watching Stream length only: a blocked PEL is invisible → monitor PEL, idle, acknowledgment latency, and dead letters.
Follow-up questions and answers
Follow-up 1: Why not use XCLAIM directly?
XCLAIM requires the caller to know which message IDs to claim. XAUTOCLAIM scans the PEL by minimum idle time and advances a cursor, which suits recovery workers. Neither replaces business idempotency or error isolation.
Follow-up 2: Does a 0-0 cursor mean there are no old or new messages?
It means this scan reached the end of the PEL cursor range. Start another cycle from the beginning because entries that were previously too young may now be idle, and new PEL entries may have appeared.
Follow-up 3: What if charging succeeds but XACK times out?
The entry can be delivered again. The idempotency key must make the second attempt read the completed state and avoid a second charge. Record one business result plus an acknowledgment retry; do not interpret acknowledgment timeout as charge failure.
Follow-up 4: How do you choose min-idle-time?
Use normal p99 processing, the longest allowed dependency retry, and network jitter as the baseline, then add safety margin. Validate with false-claim rate, recovery latency, and PEL growth. One fixed number cannot fit every task.
Follow-up 5: How many times should malformed input retry?
Parse failures and immutable business-rule violations usually go directly to dead letter; temporary dependency failures back off and retry. Set thresholds by error type, cost, and recoverability, while preserving the original ID, payload summary, and last error.
Follow-up 6: What changes with Redis 8.4 XREADGROUP CLAIM?
It combines reading new entries and claiming idle pending entries in one command, reducing the multi-command loop required by older versions. PEL semantics, acknowledgment ordering, idempotency, compensation, and poison-message handling still belong to the consumer design.