Prompt and when it applies
An interviewer may ask, “How would you evolve an OpenLineage event schema without breaking downstream consumers?” This fits data-platform, data-infrastructure, and lineage-system roles. It tests whether you can turn JSON Schema, Facet extensions, generated clients, event versions, and consumer compatibility into a release process.
What the interviewer is assessing
The key is not memorizing field names; it is recognizing change boundaries. OpenLineage documents its spec as JSON Schema and requires a version bump when an existing JSON file changes; Java and Python clients are generated from it. The interviewer also expects you to distinguish RunEvent, JobEvent, and DatasetEvent, and to know that custom Facets need a unique prefix and an immutable versioned schema URL.
Clarifying questions to ask yourself
Clarify whether the change affects a core object, an existing Facet, or a new custom Facet. Which producers emit the event and which consumers parse it? Are there old clients, replay jobs, or cross-language SDKs? Is the compatibility goal reading old events, dual-writing versions, or a one-time cutover? Also ask whether a field is required, optional, or changing meaning, and how failed events are handled.
A 30-second answer framework
Use five steps:
- Inventory producers, consumers, event types, and current schema versions.
- Prefer an optional field or new Facet over changing existing semantics.
- Bump the version, update examples and generated clients, and build a compatibility matrix.
- Validate with replay and shadow traffic, then canary producers while monitoring parse failures and missing fields.
- Set a deprecation window, rollback path, and consumer-migration exit criteria.
Step-by-step deep answer
1. Map event and dependency boundaries
The OpenLineage object model contains Jobs, Runs, and Datasets. RunEvent represents runtime state, while JobEvent and DatasetEvent represent design-time metadata. Confirm which event, Facet, and clients are affected. Map the schema repository, generated code, message bus, indexes, and query APIs so the change is not limited to the producer.
2. Choose a compatible evolution
Adding an optional field is usually safer than deleting, changing type, or redefining an existing field. If the meaning changes, add a new field or Facet and dual-write for a period. Use a project-specific prefix for a custom Facet to avoid collisions; a same-named Facet replaces the previous instance on an entity, so names and versions must remain stable.
3. Change the version with code generation
OpenLineage requires a file-version bump when an existing JSON Schema changes, and the version URL should point to an immutable version. Generate Java and Python clients, run their tests, and verify every producer and consumer is pinned to the intended version. Do not update documentation or copy types by hand only.
4. Make a compatibility matrix explicit
At minimum test a new producer with an old consumer, an old producer with a new consumer, and both versions against replayed data. For each field record whether it may be missing, whether unknown fields are ignored, whether new enum values are safe, and whether conversion is reversible. If an old consumer rejects unknown fields, do not expand production traffic directly.
5. Validate with examples, replay, and shadow traffic
Keep minimal, complete, and invalid examples for every Facet. Replay historical events through the new parser and compare structured output and query indexes. Then copy events from the new producer to a shadow topic without changing the live lineage graph. Monitor parse failures, unknown Facets, version distribution, and end-to-end latency; stop the canary on anomalies.
6. Design deprecation and rollback
Publish the old-version cutoff, migration owner, and consumer list. Producers can dual-write first and stop old fields after consumers upgrade. Rollback must retain the old schema, clients, and replay capability; deleting old-version events would restore code but not the ability to interpret data.
High-quality sample answer
This fictional answer must be replaced with your event types and organizational constraints:
I would inventory producers, consumers, client versions, and replay jobs for RunEvent, JobEvent, and DatasetEvent, then identify whether the change is in the core schema or a custom Facet. I would prefer a backward-compatible optional field; if the meaning changes, I would add a field or Facet with a project-specific prefix. I would bump the JSON Schema version, generate Java and Python clients, and update minimal, complete, and invalid examples. Validation would cover a new producer with an old consumer, an old producer with a new consumer, and historical replay, followed by a shadow-traffic canary. I would monitor parse failures, unknown Facets, version distribution, and latency. After consumers meet migration criteria, I would end dual-writing during a documented deprecation window. The old schema, clients, and replay path remain available so rollback does not erase event meaning.
Common mistakes
Saying that adding a field is always compatible
Optionality, unknown-field behavior, and generated-client updates change the result. Give a compatibility matrix and concrete fixtures.
Changing a schema without bumping its version
The version URL lets consumers identify semantics. Missing the bump can break code generation or make different consumers assume the old definition.
Treating a custom Facet as arbitrary JSON
Custom Facets need a unique prefix and an immutable versioned schema URL. A naming collision can silently replace an entity’s previous Facet instance.
Testing only new events, not replay
Lineage systems often replay history. Without replay tests, missing fields, mixed versions, and index migrations remain hidden.
Follow-ups and advanced practice
An old consumer fails on an unknown Facet. How do you release?
First make the consumer ignore unknown Facets or route the new Facet to shadow traffic. Canary the producer only after parser and metric behavior is safe; never assume every JSON consumer is permissive.
When would you use a new Facet instead of a core field?
Use a Facet for context that can evolve independently. Consider a core schema change only when the information changes the identity or lifecycle of a Job, Run, or Dataset. Explain why query and ownership boundaries matter more than field count.
Java generation passes but Python fails. What do you do?
Pause the release, compare how the generators handle optional fields, enums, and unknown properties, then fix the schema or templates and run both client suites. One language passing is not a migration exit criterion.
Old producers remain after the migration window. How do you handle them?
List the remaining sources by producer and team, restrict old-version writes, and provide an explicit error or degradation path. If a hard stop would lose critical lineage, extend the window with recorded risk; do not delete old events or schemas.