Data engineering interview: How would you use Iceberg deletion vectors for row-level deletes?

Prompt and context

An event lake appends billions of rows daily and receives continuous user-deletion requests. The team wants Iceberg v3 deletion vectors to avoid rewriting data files for every delete. Explain how they differ from position and equality deletes, and design snapshot consistency, reader compatibility, and eventual physical cleanup.

What the interviewer evaluates

• Understanding that a deletion vector is a logical row-position marker, not immediate erasure of object bytes.
• Comparing the three delete representations by write amplification, read cost, and boundary.
• Designing atomic snapshot commits, concurrent merging, old-reader fallback, and compaction.
• Connecting privacy evidence with query correctness, backups, and replication retention.

Clarifying questions

1. Do every writer, catalog, query engine, and SDK support Iceberg v3 and deletion vectors?
2. Are deletes identified by stable positions, business keys, or matches across files?
3. How long may logical deletes remain, and when do object versions, backups, and replicas expire?
4. How does the engine load delete files, and does its cache key include the snapshot ID?
5. Can compaction race streaming writes, snapshot expiration, or privacy deletion?

30-second answer

I would treat a deletion vector as a logical layer within a snapshot: each data file may reference one vector that marks deleted row positions; reads filter those rows, while the physical file is rewritten later under controlled compaction. Position deletes also identify positions but commonly use separate delete files. Equality deletes match column values, which is flexible but can scan more data. I would verify v3 support, use atomic snapshot commits, provide an old-reader compatibility path, and align compaction, expiration, backups, and replicas to one deletion SLA.

Deep-dive answer

Step 1: Define delete semantics

A deletion vector is a bitmap or equivalent structure associated with a data file and marks deleted positions. It hides rows from a logical snapshot but does not prove that object-storage bytes are erased, so privacy deletion also needs rewrite, expiration, and backup governance.

Step 2: Compare the three representations

Position deletes name a file position and fit writers that already know the file and row. Equality deletes match column values and fit CDC or business-key deletes, but readers may scan more files. A deletion vector concentrates position marks per data file, reducing many small delete files while moving filtering and vector-maintenance cost into the read path.

Step 3: Establish the snapshot boundary

Vector references are committed atomically with an Iceberg snapshot and record data-file path, vector location, size, checksum, and format version. A generator reads a fixed input snapshot and never mutates an existing vector in place. On a concurrent conflict it merges from the latest snapshot instead of overwriting another delete.

Step 4: Design the read path

The planner reads manifests and snapshot metadata, then loads applicable vectors. If a vector is missing, corrupt, or unsupported, the safe result is to reject the snapshot or fall back to a trusted delete representation; treating the error as “no deletes” leaks rows. A cache key includes table, data file, snapshot ID, and vector version.

Step 5: Plan compaction and cleanup

When vector density, random-read amplification, or delete ratio crosses a measured threshold, rewrite surviving rows into new data files and remove old files and vectors in a new snapshot. Object-storage lifecycle, backups, replicas, and snapshot expiration must meet the same deletion SLA; deleting a catalog pointer is not physical erasure evidence.

Step 6: Migrate old readers

Inventory each engine's format version, delete-file support, and cache behavior. An old reader can temporarily consume compatibility snapshots represented by position or equality deletes, or use a materialized view. Do not publish a vector-bearing snapshot to a reader that cannot interpret it.

Step 7: Verify correctness and compliance

Test concurrent and duplicate deletes, update-after-delete, snapshot rollback, corrupt vectors, and interrupted compaction. For each snapshot compare result hashes with and without the optimization, sample that deleted rows are invisible, and record the final retention time of data files, backups, and replicas.

Model answer

I would first prove that every reader parses Iceberg v3 and deletion vectors. A delete transaction fixes a baseline snapshot, builds a row-position vector per data file, and atomically commits the references with a new snapshot; a conflict rereads and merges the latest snapshot. Reads load vectors by snapshot ID. Missing or unsupported vectors stop publication or fall back to a trusted delete file, never to an empty vector. Once density crosses a measured threshold, compaction rewrites surviving rows, and old files, snapshots, backups, and replicas expire under one deletion SLA. Acceptance covers concurrent deletes, corrupt vectors, rollback, and interrupted recovery, compares result hashes, and produces physical-cleanup evidence.

Common mistakes

• Treating a deletion vector as immediate object-storage erasure.
• Overwriting an existing vector in place without version and snapshot control.
• Letting a v2-only reader consume a vector-bearing snapshot and hoping it ignores it.
• Declaring compliant deletion after removing only a catalog pointer.
• Running compaction without checking concurrent snapshots, losing writes or deletes.

Follow-up questions and answers

Follow-up 1: Why not always use equality deletes?

They express business-key deletion well, but reads may match across many files. When physical positions are known and deletes are frequent, vectors can reduce delete-file overhead. Measure reader support and query cost before choosing.

Follow-up 2: May a corrupt vector return unfiltered data?

No. That would expose deleted rows. Validate checksum and version, reject the snapshot or fall back to a trusted representation, and alert for repair.

Follow-up 3: How do vectors interact with updates?

An update commonly writes a new data file and marks the old row deleted. Commit the new file and delete reference in one snapshot so readers see either the old or new row, never both.

Follow-up 4: How do you set a compaction threshold?

Measure delete ratio, vector size, random-read amplification, scan latency, and storage cost under representative workloads. Do not choose a threshold from file count alone.

Follow-up 5: How do you prove privacy deletion?

Provide row-level invisibility checks, expired-snapshot records, rewritten-file manifests, object-version deletion results, backup and replica retention, and sampled scans with no matches.