Data Engineering Interview: When Should You Use Batch, Micro-Batch, or Stream Processing?

Problem and Scope

An e-commerce platform writes order events to a durable replayable log and retains immutable raw data. The steady rate is unspecified; the peak is 20,000 events per second, with brief promotional bursts up to ten times that rate. Inventory anomalies need an alert within 10 seconds of event creation. An operations dashboard must update within 5 minutes. Finance closes the books the next day and needs the same business day to be rerunnable and reconcilable. About 2% of events are late, by up to 6 hours.

Those rates, burst multiplier, late percentage, and deadlines are problem assumptions, not performance claims about an engine. Assume each event has a stable eventid, orderid, event_time, and schema version. Finance uses a business day in an explicit time zone. The four-person data team already operates a warehouse and scheduler reliably but does not yet have mature on-call operations for stateful streaming jobs.

The problem does not require one processing mode for all three consumers. The goal is to derive the smallest sufficient design from action deadlines, result completeness, recovery, and maintenance cost. This belongs in data because its core is processing semantics, freshness, and data-quality tradeoffs rather than cross-domain platform architecture.

What Interviewers Evaluate

The first signal is whether the candidate works backward from the last useful business action instead of defaulting to streaming after seeing a message queue. An unbounded source only says that data keeps arriving. It does not require every downstream result to process one record at a time. Next-day finance can read a bounded snapshot of the same log and gain easier reproducibility from batch processing.

The second signal is separating processing latency from action latency. An engine may compute in 500 milliseconds, but a sink that refreshes every 5 minutes prevents a second-level dashboard. A strong answer budgets ingestion, queueing, computation, writes, cache refresh, and alert delivery separately, then measures each segment.

The third signal is a precise correctness boundary. Exactly-once processing in a streaming engine does not prove that late data is complete, and it does not automatically protect external side effects. Rerunnable batch jobs are not automatically safe either. Without a defined input range, business key, snapshot version, and atomic publication, a rerun can duplicate or mix definitions.

Finally, the interviewer is testing operational judgment. Continuous streaming needs sustained capacity, backlog recovery, state, checkpoints, safe deployment, and on-call ownership. If micro-batches reliably meet a 5-minute deadline, record-at-a-time state adds failure modes without changing a decision. Conversely, hourly batch processing destroys the value of a 10-second alert that actually triggers replenishment or a sales stop.

Questions to Clarify Before Answering

• Where does the deadline start and end? Here it starts when the source system commits the event and ends when the alert arrives or the query result is visible. A promise that ends at warehouse ingestion ignores sink and cache delay.
• Does the 10-second alert trigger an automatic action? Automatic inventory changes, sales stops, or notifications require deduplication, idempotency, and audit. An observational alert can tolerate more false positives or duplicates.
• Is the 5-minute dashboard an estimate or a complete number? Micro-batch is sufficient for a revisionable estimate with an “as of” time. If every display must include a 6-hour late tail, the 5-minute freshness and completeness requirements conflict and the product contract must change.
• When does finance freeze a close, and can it adjust later? Producing a next-day draft followed by adjustment entries differs from waiting six hours before freezing. The answer determines the batch cutoff and revision protocol.
• Can the three uses share a raw layer and transformation definitions? They can share normalized events, business keys, and test fixtures. Independently copying filters, time zones, and amount rules will eventually create batch-stream drift.
• Can the team operate a stateful stream around the clock? Without backlog alerts, checkpoint recovery drills, and safe deployment, continuous streaming should be restricted to the smallest path that truly needs a second-level action.

30-Second Answer

“I would split consumers by business action deadline rather than apply one mode to the source. The 10-second inventory anomaly path uses continuous streaming. The 5-minute operations dashboard uses one- or two-minute micro-batches, leaving budget for writes and cache refresh. Next-day finance uses a business-day snapshot in batch and remains the reconciliation source. All three share immutable raw events, normalization rules, and business keys.

Streaming results are revisionable views, not claims that late data is complete. External actions are idempotent by event and rule version. Before cutover, I replay the same history through batch, micro-batch, and stream paths, shadow-compare counts, amounts, and tail latency, and enable actions only for the seconds-level path. The rule is to choose the simplest mode that meets the action SLO and whose recovery converges.”

Step-by-Step Deep Dive

Step 1: Write three result contracts

Define the key, deadline, completeness, and revision behavior of each output before selecting Spark, Flink, or another product.

One event can serve three different contracts. The message queue is an input transport and cannot force all three rows into the same execution mode.

Step 2: Allocate an end-to-end action budget

A testable first budget for the 10-second alert might assign 2 seconds each to source commit and transport, queueing, computation, sink and rule action, and alert delivery. Load tests must replace those provisional allocations, but their total cannot exceed the business deadline. Record p95, p99, and maximum backlog age for every segment. Operator time alone hides a slow sink.

The 5-minute dashboard can start a micro-batch every 1 or 2 minutes. With a 2-minute slice, schedule wait is about 2 minutes at worst, computation and writing receive 1 minute each, and cache refresh receives the last minute. If a tenfold burst pushes computation beyond the budget, first increase parallelism, shorten the slice, or defer low-value dimensions. An average runtime does not prove the tail SLO.

Finance is constrained by reproducibility and a frozen definition, not minimum latency. The batch reads an explicit snapshot or offset range, writes results to a staging area labeled with run_id, validates them, and publishes atomically. Repeating the same input and rule version should produce the same result.

Step 3: Share facts without creating dual-implementation debt

Raw events first enter a replayable log or object store with the original payload, schema version, ingestion time, and source position. A normalization layer handles schema evolution, time zones, amount units, cancellation status, and business keys consistently. Each execution mode reads after that boundary.

Shared facts do not require line-for-line code sharing among three engines. A more useful boundary is a shared data contract, golden fixtures, and deterministic rules. If batch and stream must implement window logic separately, run equivalence tests over the same fixtures. Finance can retain an independent stronger check; code reuse should not eliminate a control.

Step 4: Design duplicates, late data, and side effects separately

The inventory stream deduplicates by event_id, computes short event-time windows, and emits the rule and result versions. A stop-sale, replenishment, or notification uses a stable action idempotency key. A successful checkpoint does not prove that an external HTTP call occurred once, so retries must recognize an action already performed.

The operations micro-batch reads fixed half-open source-offset ranges and replaces a versioned target window instead of appending another total. Late events enter later batches and raise the result version. The dashboard shows an “as of” time, making it clear that 5-minute freshness does not mean the 6-hour tail is complete.

The finance batch reads a frozen input after the agreed cutoff. Later events enter an adjustment run that retains the original run, input range, rule version, and discrepancy approval. Exactly-once stream processing can prevent duplicate persisted processing results in some boundaries, but it does not prove completeness in the presence of late data or extend automatically to external side effects.

Step 5: Make capacity and cost reproducible

The stated peak is 20,000 events per second, so a tenfold burst is 200,000 per second. If a normalized event is illustratively 1 KiB, peak ingress is about 195 MiB per second. This is a capacity estimate only; production sizing must recalculate it from compressed and uncompressed size distributions.

Continuous streaming needs sustained capacity for both peak traffic and backlog recovery. Ten minutes of failure at 20,000 events per second creates 12 million queued events. If recovered capacity merely equals current input, the backlog never clears. A micro-batch must finish one slice before the next one arrives. Batch can concentrate resources in cheaper periods, but very small frequent batches make startup, commit, and small-file overhead dominant.

The comparison includes continuous compute, state and checkpoint storage, sink writes, scans, on-call work, and deployment complexity. Cloud charges alone omit the cost of a four-person team maintaining three similar implementations. The recommended design limits continuous streaming to inventory alerts, avoiding around-the-clock state for finance and the dashboard.

Step 6: Migrate with shadow replay instead of one cutover

Freeze a history segment that includes normal traffic, tenfold bursts, duplicates, disorder, and the 6-hour late tail. Use the old batch result as a baseline while new micro-batch and stream jobs run in shadow mode without triggering inventory actions. Compare counts, amounts, versions, and late-revision histories for every business key, and explain every discrepancy.

Increase exposure in stages: write only shadow tables, open an internal dashboard, then permit the stream alert to trigger reversible actions. Fault injection covers crashes before and after checkpoints, sink timeouts, stalled partitions, schema changes, and backlog catch-up. Acceptance metrics include end-to-end p99, maximum backlog age, micro-batch completion time, late-revision rate, duplicate action count, batch-stream amount differences, and recovery time.

Define exit criteria too. If micro-batches repeatedly miss 5 minutes, inspect scheduling, skew, sinks, and burst capacity before considering continuous streaming. If a 10-second alert no longer drives an action, downgrade it to micro-batch to reduce on-call cost. A processing mode is a verifiable business choice, not a permanent identity.

Strong Sample Answer

“I separate an unbounded input from its processing modes. These consumers have different action deadlines, so I would not force them all into streaming for technical uniformity. The inventory anomaly really must act within 10 seconds, so I use a continuous stream, allocate budgets across source, queue, compute, write, and delivery, and make each action idempotent by eventid and rule version. The operations dashboard needs only 5 minutes, so I start with one- or two-minute micro-batches that replace results by window and version and expose an as-of time. The financial close reads a frozen business-day snapshot in batch, stages by runid, validates, then publishes atomically; late data enters adjustment runs.

The three paths share replayable raw events, a normalized contract, and golden fixtures, while finance keeps an independent control. At 20,000 events per second with a brief tenfold burst, I test 200,000-per-second ingress and the catch-up ratio after failures, not just steady state. Stream checkpoints do not replace external-action idempotency, and exactly-once processing does not prove that the 6-hour late tail is complete.

For migration, I replay the same history and compare key-level results and revisions from batch, micro-batch, and stream paths in shadow tables. Continuous streaming starts with observational alerts. It triggers automatic actions only after p99 is below 10 seconds, duplicate actions are zero, and backlog clears within its target. My decision rule is to choose the simplest mode that meets the action deadline, completeness, and recovery contract. Lower latency earns its state and on-call cost only when it changes a business decision.”

Common Mistakes

• Moving everything to streaming because Kafka is present → continuous input does not make every result record-at-a-time, so finance and dashboards inherit state and operations cost without benefit → choose per consumer action deadline.
• Saying only that streaming is low latency → sinks, caches, and notification delivery can consume the entire budget → measure the end-to-end path and each tail.
• Calling a 5-minute view final → events can still arrive up to 6 hours late → define estimate, revision, freeze, and adjustment semantics.
• Treating checkpoints as exactly-once external actions → retries can repeat inventory or notification calls → use action idempotency keys, an audit record, and replay tests.
• Appending a new total from every micro-batch → the same window is counted repeatedly → atomically replace by window and version or define a retractable-delta protocol.
• Copying business rules into batch and stream independently → time-zone, cancellation, and amount definitions drift → share contracts and fixtures, then compare at the key level.
• Sizing only for steady input → tenfold bursts and failure backlogs cannot clear within the deadline → test peak, net recovery rate, and sink capacity together.
• Enabling real actions on first deployment → a semantic discrepancy directly changes inventory → shadow-write and observe alerts before gradually enabling reversible actions.

Follow-up Questions and Answers

Follow-up 1: Why not use continuous streaming for the 5-minute dashboard?

If one- or two-minute micro-batches finish reliably under peak and recovery conditions, with final writes and cache refresh still inside 5 minutes, continuous streaming does not change an operations decision. It adds long-lived state, checkpoints, backlog recovery, and deployment cost. If the deadline later becomes 30 seconds, or scheduling and startup consume most of the micro-batch budget, replay the same history to compare continuous streaming. Evidence against the SLO, not the label “real time,” triggers the upgrade.

Follow-up 2: Could one stream produce alerts, dashboards, and financial results?

It could, but one checkpoint, schema change, or bad window must not block all three uses. An alert can read the normalized stream and maintain short state, a dashboard can read versioned aggregates, and finance can still recompute from immutable history at a frozen snapshot. Share inputs and definitions while isolating deployments and failure domains. Combine runtime units only after proving that failure impact, backfill, and audit remain acceptable.

Follow-up 3: Does exactly-once processing make stream output suitable for the financial close?

That label alone is insufficient. Processing semantics can keep committed output from duplicating on retries, but they do not guarantee that late records have arrived and do not automatically cover side effects. Finance still needs a frozen input range, rule version, repeatable run, ledger constraints, and discrepancy approval. Stream output can be an early estimate. It becomes a close input only if those audit controls also hold and long-running reconciliation proves equivalence.

Follow-up 4: How do you prove recovery from a tenfold burst?

Record burst duration and failure duration, calculate queued events, and measure net drain rate. If current input is 20,000 per second and consumers process 30,000 per second, net drain is only 10,000 per second; 12 million queued events take about 20 minutes. Acceptance also observes end-to-end alert latency, sink limits, state growth, and autoscaling delay. A peak-throughput number without net recovery math hides a long-tail breach.

Follow-up 5: When should continuous streaming be downgraded to micro-batch?

Downgrade when the business no longer acts on second-level output, micro-batch meets the new deadline, or streaming on-call and state cost persistently exceed the loss it prevents. First shadow-run micro-batch and compare results and latency. Then disable real actions while retaining replayable input and a rollback window. Continue monitoring late revisions and peak completion time after the change so that cost savings do not hide stale data.