Data engineering interview: what does Kafka exactly-once semantics actually guarantee?

Prompt and context

This data-engineering and stream-processing question targets data engineers, streaming-platform engineers, and backend data-infrastructure roles. The consumer may crash or rebalance, and a producer may retry after losing a response; results first go back to Kafka. The answer must separate Kafka's atomic processing from arbitrary external side effects instead of treating exactly-once as a system-wide magic guarantee.

What the interviewer assesses

• Whether you can split exactly-once into output visibility and atomic advancement of the input offset.
• Whether you can correctly use a transactional producer, transactional.id, read_committed, and manual offset commits.
• Whether you can explain abort, restart, fencing, and rebalance recovery.
• Whether you recognize that a database, search engine, or HTTP service needs its own transaction, idempotency key, or reconciliation.

Clarifying questions before answering

First confirm whether the output remains in Kafka, whether Kafka Streams is used, whether the consumer group rolls during deployment, and whether an external system must commit in the same transaction as Kafka. Then distinguish “one visible output per input” from “one external side effect”; the latter requires cooperation from the destination. Ask about latency, transaction batch size, retry window, and acceptable lag.

A 30-second answer framework

I would put the input records, transformed output, and consumer offsets in one Kafka transaction. The producer enables transactional.id; the consumer disables auto-commit and uses read_committed. A successful commit makes all three visible together, while an abort hides the output and leaves the offset before the transaction. A stable unique transaction ID fences the old instance after restart. This covers Kafka reads and writes only; an external database needs the result and offset in one storage transaction, or idempotency, an outbox, and reconciliation.

Step-by-step solution

1. State the guarantee boundary

Kafka's design describes topic-to-topic exactly-once as atomically updating the output records and the consumer position in one transaction. It does not mean the function runs once or that an arbitrary HTTP request arrives once. State this boundary before discussing configuration.

2. Atomically write output and offsets with a transactional producer

Disable auto-commit, process a batch, send the output, and submit that batch's offsets as part of the transaction. The core pseudocode is:

producer.initTransactions();
while (running) {
  ConsumerRecords<String, Order> records = consumer.poll(timeout);
  producer.beginTransaction();
  try {
    for (ConsumerRecord<String, Order> r : records) {
      producer.send(new ProducerRecord<>("orders-enriched", r.key(), enrich(r.value())));
    }
    producer.sendOffsetsToTransaction(offsets(records), groupMetadata);
    producer.commitTransaction();
  } catch (AbortableException e) {
    producer.abortTransaction();
    consumer.seekToCommitted();
  }
}

Committing output and offsets together prevents visible duplicates from output-before-offset failure and prevents loss from offset-before-output failure. Handle the client's exception classes according to the deployed version; blindly retrying every exception is unsafe.

3. Make consumers see committed transactions only

Set isolation.level=read_committed and keep enable.auto.commit=false. read_uncommitted exposes records from aborted transactions, allowing a consumer to propagate output that should have been rolled back. read_committed uses transaction markers to align visibility with commit results.

4. Handle restart, fencing, and rebalance

Give each active consumer instance a stable, cluster-wide unique transactional.id. When a new instance registers with the same ID, Kafka aborts the old instance's in-flight transaction and fences it, preventing both from committing. After an abort, the application must recreate or explicitly rewind the consumer position and reprocess the batch; it must not continue from a local cursor that advanced inside the aborted transaction. Partition assignment ensures one group member owns a partition at a time.

5. Explain why external systems must cooperate

If the result goes to PostgreSQL, a Kafka transaction does not automatically include the database commit. A database-first crash leaves the Kafka offset uncommitted, so a retry needs a unique key or conditional version update; an offset-first commit can lose the write. A stronger design puts result and offset in one database transaction, or writes a replayable outbox/connector record and lets the destination deduplicate and reconcile. Without destination cooperation, promise at-least-once execution with detectable duplicates, not end-to-end exactly-once.

6. Verify with fault injection, not configuration snapshots

Crash before and after sendOffsetsToTransaction, lose the commit response, trigger rebalance, fence the old instance, expire a transaction, and make the sink unavailable. Consume outputs with read_committed and check visible output count per input ID, final offsets, post-restart lag, and aborted-transaction records. For an external sink, test unique-key conflicts, replay, and reconciliation separately. Monitor abort rate, consumer lag, processing latency, and fencing count.

High-quality sample answer

I would first bound the claim. If both input and output are Kafka, I would use Kafka Streams or an equivalent transactional consume-transform-produce loop. The consumer disables auto-commit; the producer has a stable unique transactional.id; and each batch's outputs plus offsets are committed with sendOffsetsToTransaction. Downstream consumers use read_committed, so aborted output is invisible. On restart, the same transaction ID fences the old instance, and an abort requires rewinding to the last committed offset. If the destination is PostgreSQL, I would not expand Kafka's guarantee into an end-to-end claim: I would put the result and offset in one database transaction, or use an outbox, idempotency key, and reconciliation. I would then inject crashes, lost responses, rebalances, and fencing and verify visible outputs, offsets, and destination state per input ID.

Common mistakes

• Calling an idempotent producer exactly-once → it mainly prevents duplicate log entries on producer retries and does not make output plus offset atomic → add transactions and offset commits.
• Only setting read_committed → it changes visibility but does not commit transactions or recover crashes → implement the full transaction lifecycle with auto-commit off.
• Sharing one transactional.id across active instances → instances fence one another and destabilize throughput → assign a stable unique ID per active instance.
• Continuing from the local position after abort → that position may be inside the uncommitted batch → reload the committed offset or seek explicitly.
• Claiming Kafka makes a database side effect happen once → the two systems lack an automatic atomic commit → use a destination transaction, idempotent write, outbox, or reconciliation.

Follow-ups and responses

Does exactly-once mean the business function runs once?

No. The function can run and then run again after a commit failure; the guarantee is that Kafka's visible output and offset commit agree. Keep the function free of external side effects where possible, or make those effects repeatable and reconcilable.

Why can read_committed still add latency?

The consumer must skip aborted records and wait for transactions to complete. Open or long transactions increase visibility latency and lag. Bound batch size and timeout, monitor transaction duration, and weigh the cost against at-least-once processing for low-latency paths.

What happens when a rebalance occurs mid-transaction?

The new owner resumes from the last committed offset. The current transaction should abort; the old instance releases or is fenced; and the new instance reprocesses the uncommitted batch. Do not commit offsets unless the current member still owns the partition.

How would you send Kafka output to PostgreSQL?

Write the business result and consumer position in one PostgreSQL transaction, or write an outbox row with a unique event ID and let a reliable relay deliver it. If they cannot share a transaction, choose at-least-once delivery, unique constraints, conditional version updates, and reconciliation; do not market it as exactly-once.

Which metrics prove the design works?

Count unique visible read_committed outputs per input event ID, then correlate committed offsets, abort and fence counts, consumer lag, replay volume after restarts, and duplicate-key conflicts at the sink. Keep samples before and after fault injection; a configuration snapshot cannot prove the invariant.