Prompt and context
You compute five-minute order amount and order-count metrics. Offline clients, retries, and multi-region delivery make events arrive late, and the same order may be sent more than once. The business wants an initial value within one minute and corrections available to the report within 24 hours.
Distinguish event time from processing time, define when a window emits, and show where late events go and how consumers identify corrections.
What the interviewer is testing
Time semantics
A strong answer defines event timestamps, processing time, window boundaries, and time zones before explaining that a watermark is an estimate that most data for a window has arrived.
Result lifecycle
Separate early results, the on-time result, late corrections, and data past the cutoff. One emission is not automatically the permanent truth.
State and cost
Discuss state retention, allowed lateness, recomputation scope, hot keys, and checkpoints. Waiting forever makes state and cost unbounded.
Observability
Track watermark lag, lateness distribution, correction ratio, dropped events, duplicate ratio, and result freshness.
Clarifying questions to ask
- Are metrics grouped by event time or arrival time?
- How much error is acceptable for the first result, and when is the final cutoff?
- Must late events correct reports that were already published?
- Does every event carry a stable event_id for deduplication?
- What are the peak rate per key and the maximum state budget?
- Should events past the cutoff be dropped, quarantined, or recomputed offline?
A 30-second answer framework
“I will use event-time windows and require an event_id plus event timestamp. The stream processor uses a watermark to emit an initial result and a bounded allowed-lateness period for corrections; corrections use the same window key with a revision. Events past the cutoff go to quarantine and batch recomputation instead of silently rewriting history. I will monitor watermark lag, lateness percentiles, correction rate, drops, and state size.”
Step-by-step deep dive
Step 1: Define windows and time
Use UTC event time and fixed windows such as [10:00, 10:05). Processing time is for operational alerts and early triggers, not business grouping. Include tenant, product, or region in the key.
Step 2: Advance the watermark
Each partition advances from observed event timestamps, while a global policy takes a safe lower bound. Detect idle partitions; a silent partition must not hold the entire pipeline hostage.
Step 3: Trigger and accumulate
Emit an approximation with a processing-time trigger, then emit the on-time result when the watermark passes the window end. Choose accumulating or discarding panes and include windowend, revision, and isfinal in every output.
Step 4: Handle late and duplicate events
Within allowed lateness, deduplicate by event_id, update state, and emit a new revision. A replay must not add amount twice. Retain state until the business cutoff, then clean it up.
Step 5: Define the late-data fallback
Write events past allowed lateness to quarantine with the reason and original payload. A batch job recomputes the last 24 hours and applies an idempotent upsert or higher revision to the correction store.
Step 6: Test and publish
Use controlled timestamps to test reordering, duplicates, idle partitions, restart recovery, and boundary lateness. Consumers deduplicate (metric, window_end, revision) and use final or cutoff-approved revisions for reports.
High-quality sample answer
“I first make event time, windows, and cutoffs an explicit contract. Every event carries a stable eventid, eventtime, and schema_version. The processor windows by tenant and metric for five minutes. Partition watermarks account for idle partitions, and the global watermark is a conservative estimate of window completion.
The system emits an early revision immediately, emits the on-time revision after the watermark passes the end, and accepts late events for 30 minutes. Every output includes window bounds, a revision, and a final flag, so downstream writes are idempotent. Events later than 30 minutes enter quarantine; a 24-hour recomputation job produces a higher revision. After the report cutoff, we audit the event but do not silently rewrite the business ledger.
I monitor watermark lag, p50/p95/p99 lateness, correction and drop rates, duplicates, state bytes, recomputation backlog, and final-result delay. Capacity is active windows times state per window, bounded by checkpoints and state TTL.”
Common mistakes
- Replacing event time with processing time → offline events land in the wrong window → keep event_time and reserve processing time for operations.
- Treating the watermark as truth → late events disappear silently → document it as an estimate and configure allowed lateness plus quarantine.
- Emitting one immutable result → corrections cannot propagate → use revisions and a final flag for idempotent updates.
- Waiting forever → state and cost have no bound → set a business cutoff and recompute offline after it.
- Deduplicating only by payload → retry order changes double-counting → use a stable event_id and durable dedupe state.
- Ignoring idle partitions → watermark stalls and alerts lie → detect idle partitions and temporarily exclude them from the lower bound.
- Testing only ordered input → boundary failures appear in production → inject reordering, duplicates, late data, restarts, and recovery.
Follow-up questions and responses
Follow-up 1: Why can a watermark stall?
A partition may be silent, disconnected, or conservatively estimating progress. Combine idle timeouts, partition heartbeats, and watermark-lag alerts to distinguish silence from failure.
Follow-up 2: How do you choose allowed lateness?
Use historical lateness, the business cutoff, and the state budget. Start with a p99 lateness estimate and validate it by replay. Longer is not automatically more correct; it increases state and correction cost.
Follow-up 3: How do you prevent a correction storm?
Micro-batch late events, cap corrections per window, and keep only the newest revision downstream. During spikes, downgrade non-critical metrics to batch correction.
Follow-up 4: Can events past the cutoff be dropped?
Never drop them silently. Record quarantine and audit data and measure business impact. Accounting or compliance requirements may require recomputation or manual handling.
Follow-up 5: How do you prevent results from moving backward after restart?
Checkpoint window state, dedupe state, and watermark. Use monotonic revisions, reject older revisions downstream, replay the log, and run consistency checks after recovery.
Source 1: Apache Beam Programming Guide
Beam defines watermarks, triggers, allowed lateness, and accumulation modes, including how late data can produce new panes.
Source 2: Apache Kafka Streams Core Concepts
Kafka Streams documents the grace period for out-of-order records and the discard semantics after window end plus grace.
Source 3: Dataford streaming interview question
The public interview prompt treats watermarks, late-event routing, recomputation, and monitoring as the depth probes for this scenario.