Data Engineering Interview: How Do You Evolve a Data Contract Without Breaking Consumers?

Question and scenario

This is a mid-to-senior data-engineering schema-evolution prompt. The event feeds real-time metrics and a lakehouse; consumers run different versions and cannot all upgrade on one day. Assume at-least-once delivery and replayable events. The contract must cover field types, semantics, quality, freshness, ownership, and security boundaries.

What the interviewer is testing

• A strong answer separates “the field parses” from “the business meaning is unchanged.”
• Can you build a producer-consumer compatibility matrix instead of saying that adding a field is always safe?
• Do you use lineage and usage evidence to identify affected columns, queries, and dashboards?
• Can you enforce the contract in CI, release gates, and runtime while keeping a reversible migration?

Clarifying questions to ask first

Confirm the event format and registry, the current unit and range of amount, whether consumers reject unknown fields, whether old messages are replayed, and what missing currency means. A unit or rounding change is semantic breakage even if the wire type parses; an optional metadata field has a different migration. Also ask whether consumer versions are observable, whether a temporary dual-version topic is acceptable, and what freshness and backfill windows apply.

A 30-second answer framework

I would define the contract as schema, field semantics, quality rules, freshness, ownership, and security constraints, then inventory lineage and consumer capabilities. I would not silently change amount; I would publish a version or new fields, keep the old projection readable, run compatibility CI and shadow validation, and migrate consumers in batches. Runtime validation would reject or quarantine violations with versioned evidence. After migration I would keep a measured deprecation window, rebuild old projections from retained events, and prove equivalence with reconciliation and replay.

Step-by-step deep dive

1. Write the contract boundary. Record units, precision, nullability, enums, keys, event time, freshness target, PII labels, owners, and approval for breaking changes in addition to names and types. OpenMetadata models schema, semantics, SLA, security, quality tests, and ownership as one governed data-contract object.
2. Classify the change. Adding an optional field is often backward compatible for tolerant readers; deletion, type changes, narrowed ranges, unit changes, or optional-to-required changes are initially breaking. Changing integer cents into decimal money changes semantics, so add a normalized field or version instead of silently replacing the old one.
3. Perform impact analysis. Use OpenLineage Dataset, Job, Run, and Schema Facet metadata to find jobs, downstream tables, column lineage, and recent runs. For all 40 consumers, record parser version, field usage, replay behavior, and migration owner in a change-by-consumer matrix.
4. Design the migration. For a bounded period publish amount_minor and amount_decimal, or publish a v2 event. Old consumers keep reading the old projection; new consumers shadow-read the new fields. Publish currency as optional only when a default can be proven not to change business meaning.
5. Set gates. CI compares the candidate contract with the registered version for type, requiredness, enum, and semantic changes. Then run sample replays, quality assertions, and consumer contract tests. The production ingress validates event versions; invalid messages go to quarantine with producer, contract version, and reason.
6. Switch and roll back. Migrate consumers in batches while watching parse errors, missingness, monetary reconciliation, latency, and replay differences. If the new projection is wrong, stop new-version writes and restore the old read path; retained events can rebuild the old projection. Do not remove old fields until the last consumer and replay window pass the deprecation line.
7. Make it attributable. OpenLineage run events describe jobs, runs, inputs, and outputs; a Schema Facet records dataset fields. Put contract version, Git revision, and validation result into lineage events so a later investigation can identify which release changed which consumer result.

High-quality sample answer

I would not treat this as an ordinary field addition. First I would put the unit, precision, and rounding rules for amount in the contract, then inspect lineage to learn whether the 40 consumers use it as integer cents, a display value, or an aggregation key. OpenMetadata’s data-contract model covers schema, semantics, SLAs, security, quality tests, and ownership, which prevents a “the parser accepts it” check from being mistaken for a business guarantee.

I would register a v2 or compatible dual-field version: retain amount_minor, add amount_decimal with explicit precision, and add optional currency. CI would run compatibility checks; consumer contract tests would cover unknown fields, missing currency, old-message replay, and precision boundaries. I would shadow-compute the new projection, migrate consumers in batches, and reject events without a valid contract version at ingress, quarantining failures with an alert.

During the switch I would watch monetary reconciliation, field missingness, parse errors, latency, and replay differences. Any discrepancy stops new-version writes, restores the old read path, and rebuilds from retained events. Only after every consumer migrates, the replay window closes, and deprecation metrics reach zero would I remove the old field. OpenLineage Job, Run, Dataset, and Schema Facet metadata would carry contract versions and validation results so impact analysis and audit remain reproducible.

Common mistakes

• Checking only whether JSON parses → treating type compatibility as semantic compatibility → put units, precision, nullability, and ranges in the contract and review them separately.
• Publishing because it is “just a new field” → strict parsers or required-field checks fail → inventory consumers and use a version or dual field when needed.
• Watching logs only after production → bad data is already hard to recover → gate in CI, replay samples, and quarantine at runtime.
• Deleting the old field immediately → replaying or lagging consumers lose their read path → set a deprecation window that covers consumers and replay.
• Treating lineage as a static catalog → you cannot answer which run was affected → connect Job, Run, Dataset, Schema Facet, version, and validation evidence.

Follow-up questions and responses

What if a legacy consumer cannot upgrade?

Keep a compatible old projection or translation layer so new events also produce the old view. Give that consumer an owner, deadline, and error budget. Do not freeze the contract forever or let the translator silently change monetary meaning.

What if currency is missing and cannot be inferred safely?

Treat it as a contract violation or explicit unknown state; do not invent a plausible default. Quarantine the event and notify the producer. If the business allows it, publish an explicit “unspecified currency” value and exclude or group it in downstream metrics.

How do you prove rollback did not double-count money?

Use event ID, contract version, and projection version as idempotency keys. Replay both projections and compare aggregates by order and currency. Preserve difference samples, rounding rules, and input snapshots; resume new-version writes only after reconciliation passes.