Prompt and scope
An Apache Iceberg table stores user records and must support GDPR erasure as well as frequent corrections. The team plans to upgrade from v2 to v3 and is considering deletion vectors. Explain the trade-offs among the three row-level delete formats and how you would keep old readers, concurrent writers, rollback, and compaction safe.
This question tests a table-format read/write protocol, not a feature toggle. The Iceberg specification defines a deletion vector (DV) as a position bitmap for one referenced data file; its scope, metadata, and maintenance obligations differ from equality and position delete files.
What the interviewer evaluates
- Whether you distinguish value-based, file-position, and bitmap deletes.
- Whether you know that deletion vectors are an Iceberg v3 capability and are not newly supported in v2.
- Whether you can explain the file-path, partition, and sequence-number conditions for snapshot reads.
- Whether you account for at most one DV per data file and merging older position deletes.
- Whether you connect format choice to read amplification, write amplification, delete density, compatibility, and maintenance budget.
- Whether you design observability, rollback drills, and a safe path for older readers.
Clarifications to ask first
Confirm:
- Is the table Iceberg v2 or v3, and which versions of each reader and writer are deployed?
- Do delete requests contain business key values, or do they already identify a data file and row position?
- What are the delete density, update rate, query latency target, and compaction budget?
- Are read-only v2 engines still present, and are time travel and snapshot rollback required?
- Must deletes feed a downstream CDC stream, or is current-snapshot invisibility sufficient?
If details are missing, assume most readers support v3, a small v2 reader population remains, and a successful commit must make the deletion visible in the committed snapshot.
Thirty-second answer framework
I would choose by semantics: equality deletes match column values, position deletes identify a file and row position and can serve v2 compatibility, and a v3 table can consolidate frequent position deletes for one data file into a deletion vector. Readers must validate the referenced file, partition, and sequence-number scope rather than looking only at the bitmap.
Writers must keep at most one DV per data file in a snapshot and merge existing position deletes into it. Commits use Iceberg snapshot concurrency checks. Since v2 readers cannot interpret DVs, I would build a capability matrix, dual-read comparison, and rollback plan before enabling writes. Acceptance tests cover delete visibility, old snapshots, conflicts, compaction, and performance.
Step-by-step deep dive
1. Define the semantics of each format
An equality delete matches rows in any applicable data file by one or more column values, such as id = 5. A position delete identifies a file path and a zero-based row position. A deletion vector stores a position bitmap for one referenced data file; a set bit means that row is deleted.
These are not merely compression levels. Equality deletes fit key-based events but require predicate matching during scans. Position deletes are precise and v2-compatible but can accumulate many files. DVs consolidate many positions for one data file into a directly addressable binary object.
2. Handle versions and reader capability
Iceberg places row-level deletes after v1, while deletion vectors are added in v3. A v3 table must not add new position delete files, but existing position deletes from an upgraded v2 table remain valid and must be merged when a DV is created.
Therefore a catalog upgrade is insufficient. Inventory whether every reader can parse v3 metadata, the deletion-vector-v1 Puffin blob, and delete manifests. An unsupported reader needs a compatible snapshot or a completed migration; it cannot be expected to silently ignore newly written DVs.
3. Respect snapshot read scope
A reader applies a DV to a data file only when the data-file path equals referenceddatafile, the data-file sequence number is less than or equal to the DV sequence number, and the partition spec and values match. Matching only the path could apply a delete to a rewritten file; ignoring sequence numbers breaks ordering semantics.
Delete metadata also records the containing file, blob offset, and length. Readers must locate the blob through the snapshot metadata, not treat an object-store file with a familiar name as authoritative.
4. Design writer merging and concurrent commits
A snapshot allows at most one DV for a data file. When appending deletes, a writer reads the current delete state, merges new positions with the old DV and position-delete files, and writes a replacement DV. If the data file is removed, applicable DV entries must be removed from delete manifests.
Commits still use Iceberg snapshot optimistic concurrency. A conflict retry must reread current metadata and delete manifests; it cannot reuse a bitmap generated during the first attempt. Record retry counts, conflict causes, and the final snapshot id.
5. Balance read and write amplification
For sparse, distributed deletes, equality deletes avoid locating original files but add matching work during scans. For frequent deletes concentrated in a few files, DVs can reduce position-delete file management and merge work. For a broad delete or a file already due for rewrite, rewriting data and cleaning delete files may be cheaper.
Do not claim that DVs are always faster. Compare scanned bytes, delete-file count, bitmap size, manifest-read time, compaction CPU, and end-to-end latency at several delete densities.
6. Plan maintenance, rollback, and compliance evidence
Maintenance should merge old delete files, rewrite data files with high delete density, and remove unreferenced blobs in a safe window. Retention rules for time travel must be respected, or historical snapshots become unreadable. For GDPR, prove that the target row is absent from the current valid snapshot and downstream copies.
Rollback drills should cover a rollback after a DV commit, continued availability of its Puffin file, application of older position deletes, and deletion guarantees in the new snapshot. Keep request id, committed snapshot id, and verification results in the audit record without putting personal data in logs.
7. Establish compatibility and observability gates
Before release, build a matrix for v2 readers, v3 readers, batch jobs, streaming readers, and maintenance tools. Test equality, position, and DV behavior for each. Track DV application failures, unmatched data files, delete-manifest growth, conflict retries, old-snapshot failures, and compaction backlog.
During rollout, compare results from old and new readers on the same snapshot: row counts, key sets, and sampled values. If an old reader cannot interpret a v3 deletion, stop DV writes or switch to a compatible format instead of expanding the blast radius.
High-quality sample answer
I would start with delete semantics and the reader matrix. Key-based deletes use equality deletes. If the request already identifies a data file and row position and v2 compatibility is required, position deletes are appropriate. Once the table is v3 and deletes are concentrated in one data file, I would consolidate the positions into a deletion vector. A DV is a per-file bitmap, with at most one DV for that data file in a snapshot, and creating a new one must merge existing position deletes.
Readers cannot rely on the file path alone. They must validate referenceddatafile, partition spec and values, and that the data-file sequence number is no greater than the DV sequence number; the blob offset and length must come from the delete manifest. Writers use snapshot optimistic concurrency, reread current metadata on conflicts, and never retry with a stale bitmap.
Before migration I would verify v3, Puffin, and DV support in every reader and keep a kill switch for DV writes. Acceptance covers current-snapshot visibility, time travel, concurrent deletes, rollback, compaction, old-reader comparison, and compliance evidence. Performance compares scanned bytes, manifest time, DV size, compaction CPU, and end-to-end latency. At high delete density, rewriting the data file may beat accumulating more DVs.
Common mistakes
- Treating a DV as a compressed equality delete and ignoring matching semantics.
- Claiming Iceberg v2 can write deletion vectors.
- Applying a DV by file path only, without partition and sequence-number checks.
- Keeping multiple DVs for one data file or failing to merge older position deletes.
- Upgrading the catalog without testing every reader and maintenance tool.
- Using one compaction run as a universal answer without comparing read and write amplification.
- Cleaning delete manifests in a way that breaks retained time-travel snapshots.
- Putting personal data in logs and calling it deletion evidence.
Follow-up questions and responses
Follow-up 1: Why not let v2 readers ignore DVs?
Ignoring delete metadata can return rows that were already deleted, creating a silent correctness failure. Complete a capability matrix and result comparison first, then choose migration, compatible writes, or a pause on DV writes.
Follow-up 2: When should you rewrite a file whose deletes keep increasing?
Set a threshold using delete density, bitmap size, scan CPU, query latency, and compaction budget. Above it, rewrite the data file, remove its applicable DV in a new snapshot, and verify the time-travel retention policy.
Follow-up 3: What is easiest to miss during a conflict retry?
Rereading the latest delete manifests and data sequence number. Every retry must merge against current table metadata and record the conflict plus final snapshot id.
Follow-up 4: How do you prove regulatory deletion?
Keep the request scope, committed snapshot id, current-snapshot query result, downstream-copy checks, and maintenance status. State the time-travel retention period explicitly and keep personal data out of logs.
Follow-up 5: When is an equality delete preferable?
When events contain only business keys, files are frequently rewritten, or one predicate must cover many files, equality deletes are more direct. Still measure scan matching cost and define a rewrite policy.