Prompt and context
A multi-tenant Kafka cluster still runs ZooKeeper. The team wants to migrate to KRaft and upgrade to Kafka 4.2. The cluster uses transactional producers, Kafka Streams, and Connect, and cannot tolerate message loss, duplicate commits, or a long outage. Design preflight checks, rolling upgrades, metadata-version gates, client compatibility, failure recovery, and rollback.
What the interviewer tests
- Recognizing that Kafka 4.2 is KRaft-only rather than treating it as a normal broker replacement.
- Separating software version,
metadata.version, and migration state. - Choosing 4.2.1 after accounting for the 4.2.0 Streams offline-migration and transactional-producer upgrade fixes.
- Covering controllers, brokers, clients, Connect, and Streams in verification.
- Knowing when rollback is supported and when metadata changes require forward recovery.
Questions to clarify
- What are the Kafka, ZooKeeper, client, and Streams versions, and do they meet KRaft migration prerequisites?
- How are transactional idempotence, transaction timeouts, and open transactions observed?
- Is there cross-region replication, MirrorMaker, or a verified recovery cluster?
- Do workloads use the Streams Rebalance Protocol, Share Groups, or other 4.2 features?
- What rolling window, consumer rebalance, and rollback time can the business accept?
30-second answer
I would treat 4.2 as an architecture migration: a ZooKeeper cluster must migrate to KRaft before it can upgrade. I would target 4.2.1 to avoid the 4.2.0 Streams offline-migration defect and the transactional-producer rolling-upgrade issue. Before migration, freeze high-risk changes and inventory clients, transactions, Streams state, and recovery replicas. Roll the software first, observe behavior and performance, and raise metadata.version separately. Each step checks end-to-end messages, transaction commits, consumer lag, controller quorum, and Streams state. If a step fails, stop before metadata activation or recover through a supported compatible path instead of treating metadata changes as a reversible switch.
Step-by-step deep dive
1. Draw the version and state matrix
Kafka 4.2 supports KRaft only, so ZooKeeper mode must migrate first. Software version, metadata version, and controller quorum state are independent dimensions. Confirm prerequisites and record broker, controller, client, Connect, Streams, and protocol versions in an auditable checklist.
ZooKeeper -> KRaft migration -> rolling broker/controller upgrade -> verify -> raise metadata.version
| | | |
+-- blocked --+----------------------+----------> stop and recover2. Choose the fixed version and order
Use 4.2.1 as the target. The official upgrade guide lists a fix for transactional-producer rolling upgrades that could produce UnsupportedVersionException and a fix for the Streams Rebalance Protocol offline migration defect; 4.2.0 should not run the affected classic-to-streams-group migration. Upgrade tooling and client matrices first, then roll one broker at a time to avoid changing several failure domains together.
3. Gate metadata and controller changes
After the rolling software upgrade, observe cluster behavior and performance before raising metadata.version with kafka-features.sh. Gates include controller-quorum stability, leader elections, metadata propagation latency, ISR, and disk health. Kafka 4.2 documents downgrade support when there are no metadata changes, but every target version needs its own metadata compatibility check.
4. Verify transactions, Streams, and Connect
Transaction tests cover producer epochs, commits, aborts, restarts, duplicates, and timeouts. Streams tests cover state-store restoration, rebalances, changelogs, processing semantics, and the offline migration path. Connect tests cover offsets, task restarts, and external-system idempotence. Run end-to-end produce/consume checks for each client class; broker health alone is insufficient.
5. Observe and rehearse failures
Record controller elections, metadata version, broker errors, request failures, transaction state, consumer lag, Streams restoration, Connect tasks, and disk growth. Rehearse broker restart, controller loss, interrupted transactions, failed Streams migration, and incompatible clients; each scenario needs a stop-upgrade and recovery condition.
6. Define rollback and recovery boundaries
Before raising metadata.version, keep old binaries, configs, snapshots, and a recovery cluster, and define a read-only verification window. A rolling-upgrade failure can stop at a compatible version and restore brokers. Once an unsupported metadata change occurs, rollback becomes compatible-snapshot restoration or rebuilding a new cluster; do not force a binary downgrade. Preserve transaction and offset consistency before restoring throughput.
Model answer
I would first prove this is not a routine version upgrade: Kafka 4.2 removes ZooKeeper support, so the existing cluster must migrate to KRaft. I would choose 4.2.1 because the official guide names fixes for transactional-producer rolling upgrades and Streams offline migration. Beforehand, inventory brokers, controllers, clients, Connect, Streams, transactions, and recovery replicas in a version/state matrix.
Roll one broker at a time, verify controller quorum, ISR, latency, and behavior, then raise metadata.version. Transaction tests cover epochs, commit, abort, restart, and duplicates; Streams tests cover state stores, changelogs, rebalances, and migration; Connect tests cover offsets and task recovery. Record metadata version, elections, lag, and errors. A pre-metadata failure can stop at a compatible stage; after an unsupported metadata change, rebuild from a snapshot or recovery cluster instead of forcing a binary downgrade.
Common mistakes
- Replacing ZooKeeper brokers directly with Kafka 4.2 without a KRaft migration.
- Checking only broker liveness instead of transactions, Streams state, Connect offsets, and end-to-end messages.
- Raising
metadata.versionimmediately after the rolling software upgrade. - Running the known-risk Streams offline migration on 4.2.0.
- Treating
metadata.versionlike an ordinary setting that can always be downgraded. - Having no recovery cluster, snapshots, or explicit stop-upgrade gates.
Follow-up questions and responses
Why cannot a ZooKeeper cluster upgrade directly to Kafka 4.2?
Kafka 4.2 is KRaft-only and removed ZooKeeper mode. The cluster must migrate and verify controller quorum before entering the 4.2 rolling-upgrade path.
When should metadata.version be raised?
After all broker/controller software is upgraded and stable, and quorum, ISR, client errors, and performance pass. Raising it separately distinguishes code problems from protocol activation.
What if Streams state restoration fails after metadata activation?
Stop further feature activation, preserve the evidence and logs, and recover from a supported snapshot or rebuild path. Do not hide inconsistency by deleting changelogs or forcing a binary downgrade.