Prompt and scope
This question tests lifecycle design and proof, not a single DELETE. Trace a subject identifier through raw objects, table files, derived results, caches, features, exports, logs, snapshots, and backups. Define when the system may honestly claim completion.
The prompt does not decide legal obligations. State that privacy and legal owners define retention duties, exceptions, and deadlines; engineering turns them into executable scope, states, and evidence.
What the interviewer is evaluating
- A data inventory and lineage covering copies, derived values, logs, snapshots, caches, and backups.
- An idempotent, retryable, pausable request that cannot be reintroduced by ingestion or replay.
- Clear boundaries between logical invisibility, physical erasure, snapshot expiration, and backup expiry.
- Isolation of queries and training during deletion, with tenant and sensitive-data protection.
- Evidence that proves every scope was processed rather than one successful boolean.
Recommended answer structure
Define the subject key, request state, and completion contract. Map domains and lineage. Assign a stable erasure_id, block ingestion and replay for the subject, delete or rebuild each domain, then run independent checks and retention cleanup. Persist input versions, counts, failures, and cursors so recovery resumes without duplicate effects.
Deep dive: from request to proof
Define scope and identity first
Map account, device, order, and external identifiers to an immutable subject key. List raw objects, Iceberg rows, aggregates, features, indexes, exports, caches, logs, and backups. Unknown or untracked data becomes an explicit risk item.
Make erasure mutually exclusive with ingestion
The erasure service registers an erasure_id and subject state. Ingestion, replay, and derivation jobs check the deletion marker before writing. A tombstone or subject partition barrier blocks old events, while versions or leases prevent an older snapshot from committing after erasure.
Separate logical deletion from physical cleanup
Iceberg equality or position deletes can hide rows from reads while old files, snapshots, and orphan files still exist. Schedule compaction, snapshot expiration, and orphan cleanup within the retention window. Delete raw objects from an inventory. For backups, record controlled restore rules and an expiry deletion task.
Derived data cannot be fixed by deleting only inputs
Recompute traceable aggregates by subject. For aggregates that cannot be traced back, retain subject-level intermediate state or rebuild the partition. Feature, cache, and export owners must clean their versions; pause publication during erasure and rebuild from sanitized inputs.
Define completion with evidence
For every domain record scan scope, match count, deletion version, physical-cleanup job, failures, and verification time. Independent query, export, and replay paths must return no target subject. Missing evidence means partial completion, not success.
Sample answer
“I would first confirm the subject key, retention exceptions, and deadline with the privacy owner. The erasure service creates an idempotent erasure_id, blocks ingestion and replay, and writes a work item for each domain. Raw objects are deleted from an inventory. Iceberg receives an equality delete first, then compaction and snapshot expiration; aggregates and features are rebuilt by subject, while cache and export owners clean their versions. Backups permit only controlled restore and replay the tombstone before release. Each step records versions, counts, and failures. Only zero matches across all domains and no unresolved failures moves the request to complete.”
Common failure modes and fixes
- Deleting only the primary table → Enumerate raw, derived, cache, export, log, and backup copies.
- Calling delete files physical erasure → Explain snapshots, old files, compaction, and orphan cleanup.
- Ignoring concurrent writes → Add barriers, tombstones, version checks, and replay gates.
- Using one success flag → Persist per-domain scope, counts, versions, and independent checks.
- Promising immediate backup rewrite → State retention, access controls, expiry cleanup, and approved exceptions.
Scoring rubric and self-check
Strong answers include stable identity, complete lineage, an idempotent state machine, write and replay barriers, logical/physical boundaries, derived rebuilds, backup handling, failure recovery, per-domain evidence, zero-match verification, tenant isolation, and audit minimization.
Ask yourself: Do I know every copy’s owner? Can an old event return? Which data is hidden versus erased? How does replay consume a tombstone? How does failure resume? What evidence changes the completion state?
Follow-ups and extensions
What if the Iceberg table is too large for immediate file rewrites?
Write a row-level delete so reads exclude the subject immediately, then schedule compaction by delete density and retention deadline. Restrict snapshot and export access meanwhile, and keep physical-cleanup time in the state.
What if deletion and a new event arrive together?
Use a subject barrier or monotonic version to reject or quarantine new events until erasure completes. Replay must check the tombstone; only an explicitly newer, permitted event can pass after release.
What if a derived metric cannot be traced to users?
Mark the domain as unprovable, pause publication, and choose subject-level intermediate state, partition rebuild, or an approved alternative. Without lineage evidence, do not claim deletion.
How do you keep the audit trail from leaking personal data?
Store an irreversible subject reference, request ID, scope, and counts, not names, emails, or event contents. Protect audit access separately and align its retention with the privacy policy.