Prompt and scope
This data-engineering design question tests the full lifecycle of a rejected record: detection, quarantine, repair, replay, and closure. The answer should protect throughput on the healthy path while preserving enough evidence to reproduce a decision.
What the interviewer is testing
- Distinguishing schema, business-rule, duplicate, late, and poison-pill failures.
- Designing a quarantine record with payload evidence, source position, and reason.
- Using idempotency, versioned rules, and auditable state for replay.
- Covering backpressure, alerting, privacy, retention, and ownership.
Clarifying questions to ask
Clarify whether input is batch, streaming, or both; whether valid records may commit independently; whether events have a stable event_id, business timestamp, and source offset; whether replay reads the original event or refetches the source; and what privacy and retention requirements apply.
The 30-second answer
I would separate an immutable raw layer, a validation router, a valid-data path, and a quarantine path. Every failed record keeps its evidence, source position, rule version, reason, and state; valid records are written idempotently by event_id. After a fix, a replay batch runs the same transformation and write boundary, with a controlled rule version and reconciliation metrics proving that records were neither silently lost nor duplicated.
Step-by-step deep dive
1. Preserve facts before routing
Write the incoming event to immutable storage or a replayable log first, including source, partition/offset, receive time, event_id, and a payload hash. A parse or schema failure should be routed with an error code to quarantine rather than discarded. Passing records continue through the normal path, so one bad record does not stall an unrelated batch or partition.
2. Make quarantine actionable
Store the payload or a controlled reference, failed fields, rule name and version, first-failure time, source position, retry count, repair batch, and state such as open, readyforreplay, replayed, or rejected. Encrypt or minimize sensitive fields and enforce a retention policy. A quarantine table is an operational queue, not an unbounded archive.
3. Give replay the same correctness boundary
Version the repair rule and never overwrite the original event. A replay job selects an approved state and rule version, validates a small sample or shadow target, then calls the same transformation and write path used by live traffic. Use event_id plus business version as the idempotency key; define whether a conflict is an upsert, a no-op, or a new version. Replaying twice must converge to the same result.
4. Handle duplicates, lateness, and poison pills
Detect duplicates with event_id, source position, or a documented deduplication window; an offset alone is not a business identity. Route late events by business time and state how watermarks or backfills affect downstream results. Cap retries for poison pills and hand them to an owner or a terminal rejection state, preventing one record from consuming all worker capacity.
5. Prove health with observability
Track valid-pass rate, quarantine count by source and rule, oldest unresolved age, replay success rate, duplicate-write conflicts, end-to-end latency, and freshness. Alert on thresholds or isolate one source before stopping the whole pipeline. Reconcile raw, valid, quarantined, replayed, and rejected counts for each batch or offset range; unexplained differences are incidents.
A strong sample answer
I would first clarify event identity and consistency requirements. Then I would persist the raw event immutably, validate schema, business rules, duplicates, and ordering, and route failures to a quarantine store instead of dropping them. The quarantine record keeps a payload reference, source offset, rule version, detailed reason, and lifecycle state. Valid records and replayed records share the same idempotent write path keyed by event_id and business version. Repairs create an approved replay batch without mutating the original event; I would test a small sample, cap poison-pill retries, and make late-event behavior explicit. Finally, I would monitor quarantine age, replay success, write conflicts, freshness, and count reconciliation, while protecting sensitive fields and enforcing retention.
Common mistakes
- Dropping failed rows or logging only an error string.
- Omitting source position, event identity, or rule version.
- Implementing replay with a separate transformation that can diverge from live traffic.
- Retrying poison pills without a bound or owner.
- Treating quarantine as a garbage dump with no states, retention, or closure condition.
- Watching only an aggregate success rate instead of source, rule, and age dimensions.
Follow-up questions and responses
How do you protect personal data in quarantine?
Keep only the minimum fields needed for diagnosis, encrypt sensitive payloads, restrict access, and retrieve the original through a controlled reference. Audit reads and enforce deletion at the retention deadline.
What if new data arrives during replay?
Use a separate replay batch and explicit event versions. Merge by the idempotency key; if ordering matters, define a partition or entity boundary and record conflict decisions.
When would you stop the whole pipeline?
Only for a breaking schema, an unavailable target, or corruption risk that can contaminate valid data. A single bad source or rule should normally be isolated while other paths continue.
How do you prove there was no loss?
Create a ledger for each input batch or offset range and reconcile raw, valid, quarantined, replayed, and rejected counts. Sample event_id sets and include unresolved age in the quality SLO.