Data Engineering Interview: How Do You Evolve Schemas Without Breaking Event Consumers?

Prompt and Applicable Context

orders.events is an Avro-encoded order event stream whose schemas are managed in a registry. Old and new writers coexist during rolling producer deployments. Twelve independent consumers deploy on different schedules, and the slowest team can release only once per week. Traffic averages 8,000 events per second and peaks at 25,000. Messages are retained for 90 days, and consumers may replay from any retained position.

The current OrderPlaced event stores the order amount in a required totalcents field. Every old consumer interprets it as US-dollar cents. The business now needs multiple currencies and wants to replace it with amountminor and currency. Design the workflow from proposal, compatibility checking, and deployment order through data validation and retirement of the old version. Cover these failures:

• a new writer is live while one old consumer has not upgraded;
• a new consumer starts replaying messages from 60 days ago;
• the registry reports structural compatibility even though field meaning changed;
• the same business event appears in v1 and v2 during a dual-stream migration;
• the new fields are populated incorrectly and the release must stop or roll back.

The throughput, consumer count, deployment cadence, and retention period are interview assumptions, not published limits of any platform. The core skill is event-contract evolution and consumer data correctness, so this question belongs to data. Kafka, Avro, and a registry make the scenario concrete. A strong answer explains writer/reader relationships first and then shows how tools enforce that contract.

What the Interviewer Is Evaluating

First, can the candidate define “backward compatible” precisely? A new reader consuming old-writer data and an old reader consuming new-writer data are different directions. Reciting BACKWARD, FORWARD, and FULL without naming reader and writer cannot determine a safe deployment order.

Second, can the candidate separate structural compatibility from semantic compatibility? A registry can check fields, types, defaults, and format rules. It does not know that total_cents implied US dollars. Adding nullable currency may pass a structural check while an old consumer silently treats euro cents as dollars.

Third, does the answer put rolling deployments, independent consumers, and historical replay on one timeline? Comparing only the latest two schema versions misses 90 days of history and long-lived consumers. Compile-time compatibility also does not prove that producers populate new fields correctly.

Fourth, does the candidate create a new compatibility boundary for a breaking change? Field removal, semantic change, key change, or altered enum behavior often cannot be completed safely as one in-place schema update. A strong answer explains when to retain an old field, introduce a major event stream, bridge versions, and prevent double consumption.

Finally, the rollout must be operable. Every phase needs entry criteria, observable gates, an abort action, and a recoverable state. Consumer inventory, schema versions, field population, deserialization failures, quarantine volume, lag, and old-versus-new result differences should be queryable. Asking in a group chat whether everyone upgraded is not a retirement control.

Clarifying Questions Before Answering

• Which encoding and registry are in use? Avro, Protobuf, and JSON Schema have different compatibility details. This answer derives the main flow with Avro and calls out Protobuf field-number rules separately.
• Can producers and consumers deploy independently? Rolling writers and 12 independent consumers rule out an atomic upgrade. Ask for the slowest cadence, supported rollback versions, and any unowned consumers.
• How long is history retained and replayed? A new reader must understand every writer schema in the 90-day window. Compatibility checks and schema retention must cover that window, not only the latest version.
• Is the change structural or semantic? A rename can use an additive transition. Generalizing US-dollar cents to arbitrary currency changes what an old consumer believes and needs a major event version or strict routing.
• Which consumers must receive multi-currency orders? If the legacy path is USD-only, v1 can remain USD-only while v2 carries all orders. Data that v1 cannot interpret must not be inserted silently.
• Is there a stable event ID? Dual streams, retries, and bridges require a stable event_id. Two topic offsets do not identify the same business fact across streams.
• Can incompatible records be quarantined and replayed? Quarantine needs the original bytes, writer schema ID, source position, and reason so a fixed reader can restore the missing data.
• Who owns the contract and retirement decision? Producer ownership is not unilateral permission to remove fields. Identify the contract owner, consumer registration process, exception approval, deprecation period, and retirement evidence.

30-Second Answer Framework

“I would express every change as a writer/reader compatibility matrix and include the 90-day replay window plus the slowest consumer in the support period. The registry blocks structurally incompatible schemas in CI under a policy that covers retained versions. Contract fixtures, shadow consumption, and field-quality metrics catch semantic changes that the registry cannot see.

I would not change the amount meaning in place. First add defaulted fields, deploy new readers that fall back to totalcents for old messages, and then have writers populate both representations. Multi-currency is a breaking semantic change, so I would create orders.v2, dual-publish or bridge with the same eventid, and keep v1 on its old meaning. Consumers migrate one at a time and reconcile results. Only after a 90-day replay test and the deprecation gates pass would I stop v1. A failed gate leaves the system in a compatible phase. Rolling code back does not erase already published events.”

Step-by-Step Deep Dive

Step 1: Establish the Consumer Inventory and Invariants

Register the owner, writers, consumers, schema format, supported versions, maximum lag, replay scope, and release contact for each event type. Discover consumer groups, schema IDs, deserialization failures, and last-read timestamps automatically, but do not equate “no recent traffic” with “retired.” Batch and disaster-recovery jobs can remain quiet for long periods.

This migration needs at least four invariants:

1. No registered reader fails to deserialize a message in the supported window.
2. One event_id represents one business fact in v1 and v2; bridging or retrying cannot create a second order.
3. v1 total_cents always means US-dollar cents. It never carries another currency's minor units under the old name.
4. A new reader replaying 90 days can retrieve the original writer schema and derive an unambiguous amount meaning.

These conditions are closer to data correctness than “the schema registered successfully,” and they create testable release and retirement criteria.

Step 2: Determine Direction with a Writer/Reader Matrix

Write the four combinations before choosing compatibility vocabulary:

Avro reading uses both writer and reader schemas. A writer field absent from the reader is ignored. When the reader expects a field absent from the writer, the reader schema must supply a default or resolution fails. That default fills a missing field during reading; it does not make a required field optional when the writer encodes a record.

The first phase can therefore add defaulted fields but cannot remove total_cents in place. This abbreviated schema shows the relevant fields:

{
  "type": "record",
  "name": "OrderPlaced",
  "fields": [
    { "name": "event_id", "type": "string" },
    { "name": "total_cents", "type": "long" },
    { "name": "amount_minor", "type": ["null", "long"], "default": null },
    { "name": "currency", "type": ["null", "string"], "default": null }
  ]
}

null is the first union branch and matches the default. A new reader sees null for an old message and executes an explicit legacy fallback. A new writer still supplies total_cents, so an old reader continues to work. This structure enables a transition; it does not make multi-currency semantics safe automatically.

Step 3: Put Compatibility Gates in the Proposal and CI

Every event change should include a machine-readable schema, compatibility direction, semantic description, example records, affected consumers, migration steps, deprecation date, and rollback points. CI validates the format and calls the compatibility check for the target subject. A schema registered under NONE in development cannot flow directly into production.

This scenario has mixed clients and 90-day replay, so use FULLTRANSITIVE as the interview policy for this Avro subject. New readers must understand every retained writer version, while supported old readers must understand new writers. TRANSITIVE compares with all historical versions rather than only the latest. This is not a universal setting. A system that enforces consumer-first upgrades and only requires new readers to consume old data may choose BACKWARDTRANSITIVE instead.

Run three contract-test classes beyond the registry gate:

• feed examples from every retained writer schema to the new reader;
• feed new-writer examples to every still-supported old reader;
• assert business behavior for amount, currency, event ID, unknown enums, and null combinations.

Pin the schema ID, compatibility configuration, and generated-code version into the release artifact. A runtime writer must not silently auto-register an unreviewed schema.

Step 4: Expand Readers Before Writers Populate Both Fields

Deploy the new reader first. It uses the new representation only when both amountminor and currency exist. When both are absent, it interprets totalcents as USD under the explicit v1 contract. If only one new field exists, or the two USD amounts disagree, it quarantines the record rather than guessing.

Run the new reader in shadow mode over fixtures and a production slice, comparing its USD results with the old logic. Once that passes, canary the writer so each USD event includes both representations:

event_id     = stable business event identity
total_cents  = 2599
amount_minor = 2599
currency     = "USD"

Gates include paired-field population, USD amount agreement, schema-ID distribution, old-reader errors, quarantine volume, and end-to-end consumer lag. Start with one writer canary and expand gradually. Do not remove the old read path as soon as the first new fields appear.

Even if FULL_TRANSITIVE permits more than one structural deployment order, readers-first makes semantic observation and rollback easier. Schema compatibility says “can parse”; deployment order must also preserve “can process correctly.”

Step 5: Create a v2 Boundary for the Breaking Meaning

Multi-currency events cannot go into a v1 stream consumed by every old reader. An old reader does not understand currency and would interpret 2,599 Japanese yen as USD 25.99. A registry cannot detect that error from the byte structure.

Create orders.v2 with a separate subject whose first version makes the meaning explicit:

{
  "type": "record",
  "name": "OrderPlacedV2",
  "fields": [
    { "name": "event_id", "type": "string" },
    { "name": "amount_minor", "type": "long" },
    { "name": "currency", "type": "string" },
    { "name": "schema_major", "type": "int" }
  ]
}

During migration, all orders go to v2. Only orders that preserve legacy USD meaning also go to v1. One local transaction can append a canonical business event to an outbox, followed by a replayable publisher that renders both representations. A replayable bridge may instead derive v1 from the canonical log. Two unrelated requests must not construct the records independently because one failure would leave a cross-stream gap.

Reconcile by the same eventid. One consumer job must not count its v1 and v2 copies as two orders. A consumer can shadow v2, compare outputs, then stop v1 and start v2 at recorded offset/time boundaries. If overlap is necessary, deduplicate by eventid in durable state whose retention covers both overlap and replay.

Step 6: Preserve Historical Schemas, Defaults, and Replay Meaning

With 90 days of message retention, the corresponding writer schemas must remain retrievable for at least that period plus a disaster-recovery margin. Deleting the subject, mutating an old schema, or recycling field identity can make archived bytes uninterpretable. Long-term archives should retain the writer schema or a durable schema-ID mapping with the data.

A default is a compatibility mechanism, not the result of a business migration. When a new reader sees currency = null in a v1 message, the explicit v1 contract permits it to infer USD. It must not infer dollars for data with unknown provenance. Units, time zones, unknown-enum policy, and null meaning all belong in semantic contract tests.

For Protobuf, preserve field numbers as well. After deleting a field, reserve its tag and name. Never assign an old tag to a new meaning, and avoid changing a type on the existing tag. Old binaries can remain in logs, so tag reuse can reinterpret historical data. Formats can share a release workflow, but they cannot safely share one oversimplified allowed-change table.

Step 7: Define Abortable Gates and Observability

Give every phase entry and exit criteria:

1. Proposal approved: owner, consumer inventory, matrix, examples, and retirement plan exist.
2. New reader ready: it replays 90-day fixtures and handles legacy fallback correctly.
3. Writer canary: paired fields are populated, USD representations agree, and old consumers show no new failures.
4. v2 shadow period: counts, amounts, currencies, and key aggregates reconcile by event_id with the source of truth.
5. Consumer cutover: each owner records version, offset, lag, result difference, and rollback point.
6. v1 retirement: no active or scheduled consumers remain, the replay window has passed, and audit evidence is retained.

Monitor message volume per schema ID, reader-version distribution, unknown schemas, deserialization failures, semantic-validation failures, oldest quarantined record, lag per consumer, v1/v2 event-set differences, null rates, and currency distributions. Slice by producer, consumer, schema version, and region so a low global error rate cannot hide one completely broken consumer.

Rollback actions also depend on phase. If the writer canary fails, stop writing the new schema; the old field still exists, so historical backfill is unnecessary. If v2 semantics are wrong, pause v2 and retain canonical events for corrected replay. A code rollback cannot retract bad events already published. Repair requires correction events, versioned replay, or downstream compensation with the original audit trail preserved.

Step 8: Verify Version Combinations and Inject Failures

The test matrix covers old writer/old reader, new writer/old reader, old writer/new reader, and new writer/new reader. Perform serialization and resolution tests for every retained schema version. Add cases with missing fields, unknown fields, unknown enums, only one new field, disagreeing amounts, and unsupported currencies.

Integration drills should include a producer rollback halfway through a rolling deployment, temporary registry unavailability, a new consumer replaying from 60 days ago, v1 success while v2 retries, duplicate bridge delivery, a consumer crash around its cutover offset, repeated replay after fixing quarantine, and CI rejection of a bad schema. Passing means event sets, business amounts, schema IDs, offset continuity, and downstream side effects reconcile; a process merely staying up is insufficient.

Retain a full replay drill after rollout. Historical compatibility is proven only when the new reader consumes from the earliest retained position to real time, reconciles results, and encounters no unknown schema.

High-Quality Sample Answer

“I would inventory the 12 consumers, their reader versions, maximum lag, and 90-day replay requirements, then write the change as four writer/reader combinations. An Avro new reader consumes old data using the original writer schema and reader defaults. An old reader consuming new data requires the new writer to retain every field it needs. Because versions coexist and history is replayed, I would enforce compatibility across retained schemas and add semantic contract examples outside CI's structural check.

The first phase is additive. I add nullable amountminor and currency, then deploy the new reader first. It interprets legacy totalcents explicitly as USD. It quarantines records with only one new field or disagreeing USD representations. After shadow verification, a writer canary fills both old and new fields while I monitor schema IDs, population, amount differences, deserialization errors, and consumer lag.

Multi-currency changes the implied meaning of totalcents, so yen and euro events cannot continue to reach old v1 readers. I create orders.v2 under a separate subject and render it from a canonical business event. v1 receives only events that still satisfy its USD meaning. Both representations share a stable eventid. Consumers shadow, reconcile event sets and amounts, then cut over at recorded offsets. Any overlap deduplicates by event ID.

Finally, the registry and original writer schemas remain available for at least the replay window. Every consumer proves it can read from the earliest retained position to real time. v1 retirement requires no active consumer, an expired deprecation window, successful v2 reconciliation, and recorded rollback evidence. A failed canary stops new writes while the old contract remains valid. If bad semantics were already published, I issue a correction or replay repaired events; rolling code back does not repair history.”

Common Mistakes

• Saying only “keep it backward compatible” → the answer never identifies new reader/old writer or old reader/new writer → draw all four version combinations before choosing registry policy and deployment order.
• Shipping when the registry check passes → structural compatibility cannot detect a change from USD cents to arbitrary currency units → add semantic fixtures, shadow consumption, and business reconciliation.
• Adding defaults and immediately deleting the old field → old readers still need it, and historical writers may not supply enough information for new readers → expand and dual-populate first; use a new major stream or complete support window for breaking contraction.
• Putting old and new meanings in the old topic → old consumers parse successfully while silently producing wrong amounts → keep v1 semantics fixed and place multi-currency data under an explicit v2 contract.
• Generating different event IDs during dual publication → consumers cannot prove that two topic records describe one fact → derive both representations from a canonical event and retain the same stable event_id.
• Comparing only with the latest schema → a new reader replaying 90 days may hit an older incompatible writer → use transitive checks over the support window and test every retained writer version.
• Treating a default as permission for writers to omit required fields → Avro defaults resolve missing reader fields and do not change writer encoding requirements → test writer validation and reader resolution separately.
• Reusing a deleted Protobuf tag → archived binaries or old services may parse the new meaning as the old field → reserve deleted numbers and names and assign a fresh tag.
• Confirming upgrades in a group chat → quiet batch jobs, disaster-recovery jobs, and unowned consumers are easy to miss → maintain queryable inventory, version telemetry, and retirement approval.
• Treating an application rollback as historical data recovery → bad events remain in the log and downstream state → pause publication, retain originals, and repair through corrections, replay, and reconciliation.

Follow-Up Questions and Responses

Follow-up 1: Why use FULL_TRANSITIVE here instead of the default BACKWARD?

Old readers must consume new writers during rolling deployment, while new readers must replay every retained old writer. The 90-day history also requires comparison with every supported version. FULLTRANSITIVE is a conservative fit for these assumptions. If a real system enforces consumer-first upgrades so old readers never see new writers, BACKWARDTRANSITIVE may suffice, but deployment order must become a hard gate.

Follow-up 2: Can an alias rename totalcents to amountminor directly?

An Avro alias can help some readers resolve an old structural name. It cannot make deployed old readers understand new business meaning, and it does not solve multi-currency semantics. Aliases can participate in a structural rename after verifying the implementation, direction, and historical schemas. This semantic change still needs a v2 boundary.

Follow-up 3: How do two topics avoid missing or duplicate events?

Commit one canonical event or outbox record in a local transaction, then let a replayable publisher render v1 and v2. Track delivery to each target independently and retry, so temporary success on only one target does not lose the other. Stable event_id plus idempotent consumer writes absorbs duplicates. Reconcile event sets, not only aggregate counts.

Follow-up 4: What should a producer do while the registry is unavailable?

It may continue using an already approved, cached schema ID under an explicit expiry and safety policy. It must not auto-register an unknown schema around the outage. If the writer schema cannot be established, critical events should queue, throttle, or fail closed while backlog is monitored. After recovery, validate configuration and schema IDs before replaying in original order. Schema-less JSON is not a safe temporary escape hatch.

Follow-up 5: What if one consumer can never upgrade?

Confirm whether it still has a business owner and requirement. If it must remain, treat v1 as a supported compatibility product with explicit capability, cost, and end conditions; its bridge emits only records it can interpret correctly. It should neither block v2 indefinitely nor be cut off without business approval. An unowned consumer with no verifiable reads enters an audited retirement process.

Follow-up 6: How do you verify semantics rather than successful deserialization?

Use fixed golden fixtures, a production shadow stream, and independent source-of-truth reconciliation. Compare old and new amount, currency, order count, and critical downstream results by event_id, including zero, maximum, refund, unknown-currency, and replay cases. Null rates and amount/currency distributions help detect drift, but final gates should be explainable invariants and event-level differences.