Data engineering interview: How does Iceberg v3 row lineage preserve row identity?

Prompt and scope

An Iceberg v3 event table must support CDC, recomputation, and cross-snapshot auditing. The team wants a traceable row identity and the commit that last updated each row. Explain row-lineage fields, inheritance timing, commit retries, equality-delete limits, and verification.

What the interviewer is testing

• Whether you understand inheritance of rowid and lastupdatedsequencenumber instead of inventing values at write time.
• Whether you can connect first-row-id, next-row-id, and manifests.
• Whether you handle optimistic-commit retries, concurrent writers, and old readers.
• Whether you separate row lineage, business keys, equality deletes, and physical cleanup.

Clarifying questions to ask

1. Do we need physical row identity, business-entity identity, or both?
2. Do every reader support Iceberg v3, and what is the old-engine fallback?
3. Are updates copy-on-write, merge-on-read, or equality deletes?
4. Is auditing across snapshots required, or only the current table version?

30-second answer framework

Iceberg v3 uses rowid for table row identity and lastupdatedsequencenumber for the last update commit. Readers inherit them from a data file's first-row-id, row position, and manifest sequence number. Writers can emit null fields before commit, so a retry must reread metadata and assign a new first-row-id. Row lineage is not a business key, and equality deletes do not preserve the old row ID. Establish a v2/v3 capability matrix, then test snapshots, conflicts, and retries.

Step-by-step deep dive

1. Separate the identities

A business key says what entity a record represents; rowid identifies a row in this table. If an entity is deleted and written again, its business key may recur while its row lineage should not be treated as a permanent entity ID. Auditing should retain the business key, snapshot ID, and row ID together.

2. Understand field inheritance

A new row's rowid and lastupdatedsequencenumber may be null in the data file. On read, row ID is derived from the file's first-row-id plus row position, while the update sequence comes from the manifest entry. This lets writers avoid rewriting data files before a commit succeeds.

row_id = data_file.first_row_id + row_position
last_updated = manifest_entry.data_sequence_number

3. Handle commit retries

After an optimistic conflict, the table's next-row-id may have changed. A retry must reread current metadata, allocate a new first-row-id, and rebuild the manifest list; it cannot reuse the first attempt's range. Record the attempt, conflict reason, and final snapshot ID.

4. Equality-delete boundary

An equality-delete engine commonly writes changes without reading old data rows, so it cannot provide the replaced row's original ID. The specification treats such an update as removing the old row and adding a unique new row. Preserve business keys and change events separately when entity identity matters.

5. Compatibility and reads

Row lineage is a v3 capability, and old readers may not understand its reserved fields. Before upgrading, build an engine matrix and verify whether an old reader sees nulls, ignores fields, or fails. Exports should not depend on every reader deriving row IDs; a v3-aware service can materialize audit fields first.

6. Verification, rollback, and cleanup

Test a single write, concurrent commits, conflict retries, snapshot reads, delete-and-rewrite, and compaction. Verify row-ID uniqueness in each snapshot, no reuse of a stale range, and alignment between sequence numbers and the final snapshot. Rollback changes the snapshot reference rather than rewriting historical IDs; physical cleanup still belongs to snapshot expiration and orphan cleanup.

Model high-quality answer

I would keep business keys separate from Iceberg row lineage. In v3, rowid and lastupdatedsequencenumber provide table-row identity and commit order, inherited from first-row-id, row position, and manifest sequence number at read time. Writers leave fields null before commit; conflict retries reread metadata and allocate a new range rather than reuse an old manifest. Equality deletes do not guarantee the old row ID, so audit data also stores business keys, snapshot IDs, and change events. Before rollout, test v2/v3 readers, conflicts, snapshot reads, compaction, and cleanup, with rollback limited to switching snapshot references.

Common mistakes

• Treat row ID as a business key → delete-and-rewrite semantics break → retain both identities.
• Allocate permanent row IDs while writing files → retries can duplicate or waste ranges → rely on commit-time inheritance.
• Reuse a conflicted manifest → it contains a stale next-row-id → reread metadata on every retry.
• Assume equality deletes preserve row IDs → the specification does not guarantee it → track entities with business events.
• Validate only the current query → snapshots and old readers remain risky → test cross-snapshot behavior and capabilities.

Follow-up questions and responses

Why not replace rowid with a business key?

Business keys can repeat, change, or be non-unique across tables. Row lineage is table-row identity; the two answer different audit questions and should both be retained.

Can conflict retries leave gaps in row IDs?

Unused ranges may exist, depending on implementation. The safety rule is never to reuse IDs exposed by a successful snapshot and to keep IDs unique in the final snapshot.

Does compaction change row IDs?

A correct implementation preserves identity through lineage inheritance. Verify the audit mapping before and after compaction for the same snapshot semantics; file paths alone are insufficient.