Prompt and scope
This question tests whether a data engineer can turn a cross-team data agreement into a verifiable, evolvable interface. Assume a payments team publishes order events used by finance dashboards, risk features, and operations reports; field types, business meaning, quality thresholds, and availability windows are not agreed, so a small change can become a downstream incident.
It fits data engineers, data platform engineers, and data-product governance roles. Focus on contract boundaries, ownership, version compatibility, validation, and failure handling rather than a particular Kafka, warehouse, or vendor choice. State what the producer guarantees, what consumers may assume, and how to block or degrade when the guarantee cannot be met.
What the interviewer is assessing
A strong answer separates a data contract from a schema alone: the contract also describes field semantics, quality assertions, sensitive data, service levels, ownership, and change rules. It derives a minimum contract from consumer needs, explains the boundary between pre-release checks and runtime monitoring, and uses compatibility policies for additions, deprecations, and semantic changes. It also names the notification owner, isolation boundary, rollback path, and replay strategy when a contract is violated.
Clarifying questions to ask
- Is the asset an event stream, table, file, or model feature? What freshness and latency are allowed?
- Which consumers depend on it, and what fields, time semantics, precision, and retention do they need?
- What does “correct” mean here: requiredness, uniqueness, ranges, enums, cross-field rules, and reconciliation?
- Which fields contain personal or financial data, and who approves access, masking, and permitted use?
- Is the change backward compatible, does it require dual writes, or must a release gate reject it? Can a violation be delayed, isolated, or rolled back?
A 30-second answer framework
“I would have consumers state their business needs first, then split the contract into structure, semantics, quality, freshness, security, and responsibility. Each contract has an owner, version, status, and change policy. Producers run schema and quality checks before release; consumers monitor freshness and availability at ingestion. Compatible changes ship directly, while breaking changes use a new version, dual-track migration, and a deprecation window. A failed check isolates bad data, notifies the owner, and preserves replayable raw input. I would validate the design with change-defect rate, where violations are first detected, and recovery time.”
Step-by-step answer
Step 1: Bound the contract from consumer use cases
List the fields and decisions consumers actually need instead of copying every column from a producer table. For each field, record business meaning, unit, timezone, nullability, and source; for example, whether totalamount is cents or dollars and whether createdat is event time or write time. Keep internal implementation details outside the shared promise.
Step 2: Define structure, semantics, and quality assertions
Structure covers field names, types, requiredness, and nesting. Semantics cover enums, units, time windows, and calculation definitions. Quality assertions cover nulls, uniqueness, ranges, distributions, and cross-field relationships. OpenMetadata separates schema, semantics, security, business assertions, SLA, and status; that layering prevents “the field exists” from being mistaken for “the data is trustworthy.”
Step 3: Make freshness, security, and ownership explicit
Define refresh frequency, maximum latency, retention, and availability windows. Name a producer owner, backup contact, and consumer support channel. Add classification labels, access policies, and permitted-use constraints for sensitive fields. The boundary must be enforceable: the producer guarantees contract compliance, while a consumer still validates its own business derivations.
Step 4: Choose versioning and compatibility rules
Separate contract version from data-product implementation version. Adding an optional field is often compatible; deleting a field, changing its type, shrinking an enum, or changing time semantics is high risk. Publish a new version or translation layer for breaking changes, dual-write while consumers migrate, set a deadline, and retire the old version only after usage and the deprecation window reach zero. A schema registry compatibility mode cannot hide a semantic change.
Step 5: Put the contract in the release gate
Store the contract in version control and review it in pull requests. CI should validate the contract format, then run structure, quality, and compatibility checks against samples or shadow data before publishing. Confluent data contracts can express integrity constraints, metadata, migration rules, and sensitive-data tags, illustrating why executable rules are stronger than documentation reminders. A failed gate blocks release or routes bad data to quarantine.
Step 6: Design runtime protection and recovery
Monitor freshness, missingness, enum drift, latency, consumer errors, and contract status after release. Preserve the raw payload, version, and validation result for replay. A dashboard or feature service may temporarily use the last trusted snapshot with a staleness label; settlement, authorization, and other irreversible actions should stop and alert. Alerts must point to an owner and escalation path, not only say “data issue.”
Step 7: Exercise the design with one change
Suppose an order event changes amount from integer cents to an object with amount and currency. Identify consumers, publish a new version, dual-write, replay samples, and reconcile old and new results before deprecating the old version. If refunded changes from “refund initiated” to “refund completed,” it is a breaking semantic change even though its type is unchanged. The exercise proves the contract covers meaning, not just field shape.
Step 8: Review whether the contract works
Track the share of breaking changes blocked by gates, where incidents are first detected, violation-to-recovery time, remaining old-version consumers, and datasets without owners. If incidents still surface first in downstream dashboards, checks are too late or assertions are too weak. If the contract contains many unused fields, narrow its boundary. Version and maintain it as business use and consumers change; it is not a one-time document.
Design trade-offs and boundaries
More detail can provide stronger protection, but it also raises the change cost for producers and consumers. I would put cross-team assumptions that affect business decisions or cannot be replayed into the mandatory contract, while keeping experimental fields optional. Set quality thresholds by use case: settlement data may require strict completeness, while exploratory analysis may accept delay with an explicit confidence or freshness label. Do not duplicate every governance rule in one file; link access approval, retention requirements, and data-product documentation to avoid parallel sources of truth.
When should a release be rejected?
I would refuse to mark a version active when field meaning is unclear, a critical owner is missing, a breaking change lacks a migration plan, or a quality assertion cannot run at any boundary. It can remain in draft while consumers confirm assumptions and the producer supplies samples and replay evidence.
How do you resolve consumer conflicts?
Separate shared guarantees from consumer-specific needs. Keep the shared contract small, stable, and verifiable. If one consumer needs higher precision or lower latency, provide a derived data product or a tiered SLA instead of forcing every consumer to pay the same cost. Every exception needs an owner, expiry date, and exit condition.
Rollout plan and evidence
Start with a high-impact dataset that has a limited consumer set: register consumers, write the minimum contract, run sample checks in CI, then add freshness and quality monitoring at the production boundary. After the first migration, review blocked changes, false positives, recovery time, and consumer feedback before expanding to more topics or tables. The evidence tunes assertions before a large governance platform is installed without users.
What are the pilot exit criteria?
Define exit criteria: several release cycles pass the gate, violations are found before downstream contamination, consumers complete migration, and the owner responds within the agreed window. If the criteria are not met, narrow the scope or revise the contract instead of presenting a failed pilot as platform success.
How do you show that the gain is real?
Compare pre- and post-pilot breaking changes, first-detection location, rollback count, and recovery time, segmented by dataset and consumer. If incidents fall while release frequency also falls, include change throughput and waiting time. If gates block many changes but false positives are high, improve assertions and samples rather than disabling the gate.
Common mistakes and follow-ups
Calling field types a data contract
Types and requiredness protect shape, not currency, time semantics, quality, freshness, access, or ownership. Without those guarantees, consumers can receive data that is structurally valid but semantically wrong.
Making consumers absorb every dispute
Consumers can monitor their own outcomes, but they cannot own the producer’s shared definition and release responsibility. The contract must state producer guarantees, consumer assumptions, and the escalation path between them.
Alerting only at runtime
By the time bad data reaches a shared layer, several downstream systems may be contaminated. Shift structure and compatibility checks left; use runtime monitoring and quarantine for business assertions that cannot be decided before release.
How do you classify and migrate a breaking change?
Judge consumer impact, not only a schema diff. Removing a field, changing a type, narrowing an enum, or changing a unit or time definition can break consumers. Publish a new version or translation layer, validate both tracks, notify owners, and remove the old version only after usage is zero and the deprecation window has passed.
The check fails but the business must continue. What do you do?
Separate reversible display from irreversible actions. A display path may use the last trusted snapshot with an explicit staleness label. Settlement, authorization, and risk decisions should quarantine bad data, pause the affected action, and preserve raw input. Replay, reconcile, and obtain consumer acceptance before lifting quarantine.
How do you stop a contract becoming an abandoned document?
Put it in code review, CI gates, catalog metadata, and runtime status. Bind every contract to an owner, backup contact, consumer list, and last validation time. Review violation rate, detection location, and recovery time, and retire entries that have no consumer or cannot be enforced.