Prompt and context
After a tenant closes, the system receives a deletion request. Tenant data is spread across a primary database, object storage, search indexes, caches, and backups. Deletion must not block the request thread or silently lose work when one downstream is unavailable. The interview focuses on orchestration, idempotency, recovery, and the evidence behind a “complete” state.
What the interviewer is testing
The interviewer wants to see deletion semantics defined before components are drawn. A strong design covers the data catalog, state machine, task partitioning, idempotency keys, retries and dead letters, concurrency control, observability, and final proof. Object-storage documentation distinguishes immediate deletion, versions, and lifecycle rules; replicas and asynchronous deletion mean that primary-database success is not proof that every copy has been processed.
Clarifying questions to ask first
Ask whether the scope is one tenant, one user, or selected resources; how hard delete, delayed delete, and retention windows are defined; whether backups are erased immediately or expire after a window; which audit records must remain without copying raw personal data; and what consistency, completion-time, scale, and policy boundaries apply. Do not invent legal deadlines that have not been supplied.
A 30-second answer framework
Say: “I would build a tenant data catalog and deletion policy, then orchestrate versioned deletion jobs through storage adapters. The request creates one idempotent job and returns quickly; a queue drives phases, and each phase records evidence, retries, and its last error. The verifier marks the job complete only when primary data, derived indexes, caches, and policy-allowed backups meet their states; otherwise it schedules retry or human handling.”
Step-by-step deep analysis
1. Build the data catalog and policy
For each resource record tenant ownership, location, derivation, deletion method, retention window, and verification query. Mark backups or logs that cannot be deleted immediately as policy exceptions, including expiry, approval, and how they remain inaccessible to product reads during retention.
2. Submit an idempotent deletion job
The close flow creates a globally unique deletionJobId and uses a tenant generation or closure-event version as an idempotency condition. The API returns an accepted status quickly; duplicate requests return the same job. Persist a snapshot of the catalog version so later catalog edits do not change the job boundary.
3. Split retryable work by dependency
Freeze writes or switch the tenant state first, then process primary records and objects, followed by search indexes, caches, and derived files. Each adapter uses (deletionJobId, resourceId, generation) as its idempotency key. Success, already-absent, and safely retryable errors need distinct outcomes; non-retryable errors go to a dead-letter queue with a human recovery path.
4. Handle replicas, versions, and backups
Object storage may have versions, lifecycle rules, and cross-region replicas; a successful delete API call does not prove that asynchronous copies are immediately gone. Record request time, observed version or delete marker, and the last check for each replica. Let backups move through their retention policy and verify irrecoverability after expiry rather than pretending that erasure is instantaneous.
5. Design the verifier and completion states
The verifier queries the catalog and checks primary reads, index searches, cache reads, and object listings. The state machine should distinguish at least ACCEPTED, RUNNING, WAITING_RETRY, BLOCKED, VERIFIED, and FAILED. Only required resources with evidence and explicit policy exceptions may reach VERIFIED; conditional writes prevent an old retry from overwriting a newer result.
6. Add observability, isolation, and audit
Track latency, success rate, retries, dead letters, downstream throttling, and remaining resources by tenant and job. Use dedicated credentials and least privilege to prevent cross-tenant deletion. Audit records keep only the job ID, resource type, policy version, actor, and result summary, never raw data. Alerts distinguish one blocked tenant from a global downstream outage.
High-quality sample answer
“I would start with a catalog of tables, object prefixes, indexes, cache keys, and backup policies, each with tenant ownership, a deletion method, and a verification query. Closing a tenant writes one unique deletionJobId, freezes new writes, and enqueues the job. The orchestrator follows dependencies: primary data and objects first, then indexes and caches; every adapter is idempotent by resource generation. Replica and object-version evidence is recorded, while backups follow their retention policy rather than being claimed as instantly erased. A verifier rechecks primary, search, cache, and object listings and moves to VERIFIED only when every required resource is evidenced. Downstream failures use exponential backoff and then a dead-letter path with a named owner. Audits store policy version, a result summary, and the job ID; credentials are tenant-isolated. The client receives queryable status and evidence, not an unsupported ‘deleted’ string.”
Common mistakes and improvements
- Deleting only the primary rows: Draw the data catalog and derivations, then define deletion and verification for each.
- Calling every downstream synchronously: Use a job queue and state machine to isolate request latency from outages.
- Treating duplicate work as an exception: Define a per-resource idempotency key and distinguish already-absent from failure.
- Promising instant backup erasure: State the retention window, lifecycle, and verifiable expiry state.
Follow-up questions and responses
What should the client see when a downstream keeps failing?
Return queryable job state, phase, error class, and next retry time without exposing internal credentials. After the retry budget, move to BLOCKED or FAILED, notify the owner, and retain a safe resume path.
How do you stop an old job from deleting new writes?
Freeze writes or use a tenant generation. Every delete carries the generation captured at creation, and the storage condition rejects a mismatch. Catalog and status updates also use versioned conditional writes.
How do you test that VERIFIED cannot be a false positive?
Inject duplicate messages, delayed replicas, index rebuilds, cache repopulation, and backup restores. If a simulated query can still read data, the verifier must remain incomplete and record the missing evidence.
What if audit logs themselves contain tenant information?
Store only an irreversible tenant reference or hash, job ID, policy version, and result summary. Restrict access and give the audit record its own retention policy. Auditing should prove processing without copying the deleted data.