Prompt and context
This data-engineering question tests long-term event-contract evolution. The point is not to recite backward or forward definitions, but to derive a safe plan from real readers, writers, serialization format, historical data, and rollout order. You need to explain compatibility goals, field semantics, registration and testing, dual-version operation, and the conditions for final cleanup.
What the interviewer is assessing
- Whether you distinguish writer schema, reader schema, and deployment-time combinations.
- Whether you choose backward, forward, full, or transitive constraints from consumer rollout order.
- Whether you handle required fields, enum changes, defaults, unknown values, and historical replay.
- Whether registry governance, contract tests, monitoring, rollback, and ownership are part of the release process.
Clarifying questions to ask
List the event format, topic or table, every producer and consumer, external or cross-team subscribers, and each consumer’s upgrade speed. Clarify whether fulfillment_mode is new meaning or can be losslessly derived from the old status, and whether the new enum permits unknown values. Also ask whether historical events must be replayed, whether the lake stores raw payloads, who owns compatibility policy, and whether two versions can run briefly in parallel.
A 30-second answer framework
I would build the consumer inventory and reader/writer rollout order, choose the compatibility direction, and enforce it in a registry. Add the new field as optional or with a stable default, use an extensible enum representation, and have producers write old and new fields while consumers migrate by capability. Define mappings and unknown-value behavior for replay, then validate with contract tests, replay samples, and runtime monitoring. Remove old fields or versions only after every consumer has migrated and the retention window is closed.
Step-by-step deep dive
1. Draw the reader/writer matrix and compatibility goal
List old writer with old reader, old writer with new reader, new writer with old reader, and new writer with new reader. If consumers may upgrade first, you need forward compatibility; if producers upgrade first, you need backward compatibility; rolling deployment usually needs both during the transition. Historical replay across many versions also calls for transitive checks rather than comparing only with the latest schema.
2. Make field semantics safely extensible
Do not make a new field immediately required for every old reader. With formats such as Avro, use a nullable union or a default where appropriate; with JSON, define the difference between missing, null, and unknown fields. When extending an enum, ensure old consumers have a safe unknown branch instead of assuming only today’s values arrive. If old status cannot be mapped without loss, add a new event version or parallel field rather than silently changing meaning.
3. Use a registry and contract tests to block incompatible releases
Register schemas by subject or event type and define naming, compatibility level, and owner. Check new schemas during producer builds. In consumer CI, deserialize real old and new samples and assert business behavior. Test defaults, unknown enums, time and unit semantics, nulls, and behavior after field removal, not only parse success. A failed check blocks release instead of becoming an online consumer error.
4. Roll out with dual writes and stages
Release consumers that can read both formats first. Then have producers populate the new field while retaining the old one. Watch consumer versions, parse errors, default usage, and event lag; data pipelines must also verify table types, partitions, and backfills. For an incompatible enum or structure, create a new event type or topic, use a bridge to publish both, and give each path an owner and retirement date.
5. Define replay, rollback, and cleanup conditions
Parse historical events with their writer schema, then use an explicit versioned mapping to create the current model. Do not quietly apply today’s default to yesterday’s fact. Retain old and new schemas, conversion code, and sample snapshots. During producer rollback, verify that the old version can read events already written. Delete old fields or bridge topics only after all consumers migrate, old-field reads reach zero, replay and quality checks pass, and the notification window closes.
Example of a strong answer
I would inventory the dozen consumers and several data pipelines, draw the four reader/writer combinations during rolling deployment, and determine whether fulfillment_mode is losslessly derived from the old status or is a new fact. If it is derivable, I would make the field optional with a stable default and retain the old status; the enum would include a safe unknown branch. I would release consumers that read both formats, then have producers dual-write while the registry enforces compatibility. CI would test old and new events, missing fields, unknown enums, and historical replay, while production monitoring tracks parse failures, default usage, consumer versions, and table quality. If the structure is truly incompatible, I would create a new event version and bridge both paths. Only after every consumer migrates, old-field reads reach zero, replay validation passes, and the rollback window closes would I remove the old field and bridge.
Common mistakes
- Saying “enable backward compatibility” without listing producer and consumer rollout order.
- Adding a required field without a default, missing-field rule, or safe path for old readers.
- Extending an enum so old consumers throw or treat an unknown value as the wrong business state.
- Checking only whether a schema registers instead of testing real old samples, replay, table types, and business results.
- Overwriting field meaning so historical replay is interpreted incorrectly.
- Keeping dual writes and bridge code forever without an owner, monitor, retirement date, or rollback condition.
Follow-up questions and answers
How do you choose backward, forward, or full compatibility?
Use rollout order and consumption behavior. Producer-first rollout requires old readers to read new data, so emphasize backward compatibility. Consumer-first rollout requires new readers to read old data, so emphasize forward compatibility. If order is uncontrolled or both directions must work, use full. For all historical versions, use transitive and confirm the format’s actual rules.
Why should a new field usually have a default?
Old events do not contain it, so reading old data needs explicit behavior. A default lets the reader parse, but it must mean unknown or not provided rather than pretending to be historical fact. Without a meaningful default, use a nullable type, a new event version, or an explicit backfill.
How do you handle a new enum during historical replay?
Preserve the writer schema, parse the original semantics first, and map into the current model with versioned conversion. Route unmappable values to quarantine or manual handling with a reason; do not drop the event or silently apply the current default state.
When can you remove the old field?
After every producer and consumer migrates, old-field reads and writes are zero, replay, quality, and contract tests pass, customer or external-subscriber notice windows close, and rollback and audit materials remain available. Remove it in stages.