Prompt and context
An Apache Iceberg fact table stores 18 months of orders while Spark batch jobs, Flink streams, and Trino queries run concurrently. The team wants to split customername into firstname and last_name and change partitioning from day to month. Historical files cannot all be rewritten, old queries must keep working, streaming writes cannot stop, and a failed rollout must be reversible.
The interview tests whether a candidate separates schema compatibility, field identity, partition layout, snapshot commits, and reader/writer upgrades. Iceberg documents stable field IDs and independent schema and partition evolution; a strong answer turns those guarantees into a staged migration instead of saying “alter the table and backfill.”
What the interviewer is assessing
Look for a distinction between names and field identity, between schema evolution and historical backfill, and between a new partition spec and physically moving old files. The candidate should explain snapshot atomicity, engine compatibility, and rollback evidence, including what happens when a writer races with the migration.
Clarifying questions
- Which Spark, Flink, Trino, catalog, and Iceberg versions read and write the table?
- Can
customer_namebe parsed reliably across languages, single names, and privacy restrictions? - Do old clients depend on positions,
SELECT *, or serialized schemas? - Can readers mix partition specs and still apply date predicate pushdown?
- How are streaming checkpoints and schema versions coordinated?
- Does the catalog support atomic commits, snapshot retention, and rollback by ID?
30-second answer
“I would inventory engine compatibility and column usage, add the new columns using Iceberg field IDs, and keep the old column during an observation period. Partition evolution is separate: new files use monthly transforms while old files remain readable. I would roll out compatible readers, then writers, then consumers, validate queries, checkpoints, and concurrent commits, and keep the previous snapshot for rollback. Historical backfill is a versioned job, not an implicit schema change.”
Step-by-step solution
Step 1: Define a compatibility matrix and change contract
Record the schema, field IDs, current partition specs, snapshot retention, writer versions, and reader versions. Keep separate contracts for column semantics and partition layout so parsing, renaming, and physical rewrites do not become one opaque commit.
Treat the split as additive first: add firstname and lastname, keep customer_name, and define rules for whitespace, mononyms, multilingual names, and parse failures. Do not silently assign a new meaning to the old column. Plan removal only after every consumer has migrated.
Step 2: Use field IDs, not column positions
Iceberg binds fields by stable IDs. A change record can look like this:
old: id=7 customer_name:string
new: id=21 first_name:string, id=22 last_name:string
old id=7 remains until consumers migrateNever reuse ID 7 for a different meaning after deletion. Check nested struct, map, and list child IDs too. CI should compare the before/after ID map; a name-only rename is not proof of safe evolution.
Step 3: Separate backfill from online writes
Let batch and streaming writers populate the new columns first and publish parse-quality metrics. Backfill historical files per partition later. Read a fixed snapshot or watermark, and record input snapshot, code version, row counts, parse failures, and checksums in a manifest. If a concurrent Iceberg commit conflicts, reread the latest snapshot and retry; never overwrite another writer's commit.
If a historical name cannot be parsed reliably, keep null plus nameparsestatus rather than inventing identity data. A backfill is a separate versioned computation, not a requirement of the schema command itself.
Step 4: Evolve the partition spec independently
Add a partition spec using a month transform for new data while old files keep the day spec. The planner should identify each file's spec and prune it correctly.
spec-0: day(ts) -> existing files
spec-1: month(ts) -> new filesTest scan volume, pruning, and small-file behavior in a shadow table. Consumers should read metadata through the catalog rather than constructing object-storage paths from directory names.
Step 5: Stage reader and writer releases
Deploy readers that tolerate both columns first, then writers that populate the new columns, and only then consumers that require the new fields. Validate streaming checkpoint recovery against both snapshots. Record engine-specific support for renames, nested types, type promotion, transforms, and mixed specs; use a view or an engine upgrade where support is missing.
Make each change a small snapshot and record the writer, catalog, ID map, and partition spec. Avoid combining column deletion, type changes, and a large rewrite in the same observation window.
Step 6: Gate commits and preserve rollback
Run structural checks on IDs, types, and specs; data checks on row counts, null rates, parse failures, aggregates, and date slices; and behavior checks on old/new SQL, checkpoint recovery, commit conflicts, and pruning. Keep unparsed samples for review.
Save previoussnapshotid and newsnapshotid before release. If a gate fails, point the catalog back to the old snapshot, retain new files and evidence, pause consumers that require the new columns, and rerun affected partitions from fixed inputs.
Model answer
“I would first inventory Spark, Flink, Trino, and catalog support for field IDs, then add the two new columns while retaining customername. A deterministic parser and nameparse_status quantify quality; a historical backfill uses a fixed snapshot, code version, and a manifest. New files use month(ts) while old files use day(ts); the table metadata handles both.”
“I would upgrade compatible readers, then writers, then consumers. Before release I would validate IDs, nulls, aggregates, old and new queries, stream recovery, and concurrent commits. I would retain the previous snapshot and roll the catalog pointer back if any hard gate fails.”
Common mistakes
- Use column position as identity → old bytes are misread → validate stable field IDs.
- Treat rename as a new semantic column → old values acquire a new meaning → add, deprecate, then remove.
- Move every old file after changing the spec → huge commits and weak rollback → let specs coexist and rewrite selectively.
- Upgrade a new-column-only reader first → old writers emit nulls → compatible readers, writers, then consumers.
- Only test the schema command → queries or stream recovery still fail → run structural, data, and behavior gates.
- Backfill directly into the live snapshot → there is no clean recovery point → use a versioned output and snapshot IDs.
Follow-ups and responses
Follow-up 1: Why must a deleted field ID never be reused?
Old data files still contain the original ID. Reusing it makes readers interpret old bytes as a new semantic field, so allocate a new ID and retain the old field until consumers and historical replays migrate.
Follow-up 2: Can old and new partition specs coexist safely?
Yes, if the metadata records each file's spec and the actual engines apply transforms and predicates correctly. Verify pruning, time-zone behavior, and small-file counts with production-like queries instead of inferring behavior from directory names.
Follow-up 3: How do you recover from a concurrent commit conflict?
Read the latest snapshot, confirm the input remains valid, recompute only affected partitions, and submit again. Never force an old metadata file over another writer's snapshot; repeated conflicts call for lower concurrency or smaller slices.
Follow-up 4: When can the old column be dropped?
After readers, writers, replays, audits, and exports have migrated, the observation period is stable, and retained snapshots cover the rollback window. Inventory hidden consumers before submitting the delete.
Follow-up 5: What should happen to unparseable names?
Write null plus a reason and a controlled reference to the original value, monitor by language and format, and avoid putting a guess into an identity field. Correct it in a later version or a human review workflow.