1. Question
An order-event stream is parsed, windowed, aggregated, and scored for risk. Results are written to a warehouse and a downstream notification is triggered. Workers can crash, networks can time out, and events can be late. Explain the boundary of exactly-once and design a flow that does not create duplicate charges during retries.
2. Constraints and clarifications
- Separate three layers: message delivery, in-pipeline results, and external side effects.
- Events may be duplicated, out of order, or late; a processing log is not proof of a committed result.
- Results must be replayable, while notifications and other external calls need idempotency or deduplication.
- Clarify latency, lateness windows, and dropping policy before choosing an implementation.
3. Core concept
Exactly-once processing usually means a record's pipeline result is reflected at most once in durable output while the system still avoids losing the record. It does not mean every user function runs once, and it does not automatically cover an HTTP, email, or database call. At-least-once input plus checkpoints, deterministic replay, and result deduplication can produce a verifiable output guarantee.
4. Reference flow
onEvent(event):
key = stableEventId(event)
state = readCheckpointOrState(key)
result = deterministicTransform(event, state)
writeTransactionalResult(key, result) # unique(key)
commitCheckpointAfterResult(key)
onExternalSideEffect(result):
idempotencyKey = result.eventId + ":" + result.version
callOrOutbox(idempotencyKey, result.payload)First write the result and event ID to storage with a uniqueness constraint or transaction, then advance the checkpoint. Route external notifications through an idempotent API or an outbox and a separate sender. The sender may retry, while the receiver accepts a given idempotency key only once.
5. Failure cases and trade-offs
If a worker crashes after an external call succeeds but before its checkpoint commits, replay calls the external service again. Without an idempotency key, the runner alone cannot remove that duplicate side effect. Window results also depend on late data and watermarks, so the correction boundary must be explicit. Stronger end-to-end guarantees add deduplication state, transaction coordination, and storage cost; if duplicates are acceptable, at-least-once may provide lower latency.
6. Verification and observability
- Inject crashes, timeouts, duplicate messages, and out-of-order events; inspect the final result for one business key.
- Record input event IDs, attempt counts, commit versions, deduplication hits, and external-call outcomes.
- Reconcile four counts: received, processed, committed, and notified. Worker logs alone are insufficient.
- Monitor duplicate rate, lateness, checkpoint age, deduplication-state size, and replay backlog.
7. Common mistakes
- Treating exactly-once delivery, exactly-once processing, and exactly-once side effects as one promise.
- Assuming a framework switch gives arbitrary custom code and external APIs one-time effects.
- Deduplicating with a timestamp instead of a stable event ID, producing different keys on retry or replay.
- Ignoring late events, version conflicts, and deduplication-record retention.
8. Interview scoring points
Draws the guarantee boundary
The candidate separates delivery, pipeline results, and external side effects, and identifies which layer the framework actually covers.
Designs a replayable flow
The candidate uses a stable event ID, deterministic transformation, transactional result commit, and checkpoint ordering, then explains crash recovery.
Handles external side effects
The candidate proposes an idempotency key, uniqueness constraint, or outbox and explains how sender and receiver jointly prevent duplicates.
Verifies with fault injection
The candidate covers duplicates, reordering, lateness, timeouts, and worker crashes, using committed data and business reconciliation to validate the claim.