Prompt and scope
A B2B product needs full and incremental exports. One export can exceed request timeouts and single-file limits, and a client may retry after a broken download. The interview tests export boundaries, asynchronous state, checkpoints, file visibility, authorization, and resource quotas.
What the interviewer is testing
The interviewer wants to see whether you can turn a long job into an observable, resumable protocol. Google’s Data Portability API treats export as a dedicated capability and requires review for sensitive scopes; Zendesk incremental exports save a cursor or time boundary for the next starting point and explicitly avoid a recent-write race; Oracle FHIR Bulk Export exposes a job location for polling. A strong answer also states snapshot consistency, deletion semantics, and tenant isolation.
Questions to clarify first
Confirm the resources, full versus incremental mode, maximum rows and retention; whether cross-table consistency is required; whether output is JSONL, CSV, or compressed shards; whether clients need cancellation and range downloads; who may create and download; and each tenant’s concurrency, bandwidth, and storage budget. Do not treat “export complete” and “file downloaded” as one state.
30-second answer framework
Open with: “I’ll expose create, status, manifest/download, and cancel operations. Creation fixes the authorization scope and snapshot boundary and returns a job ID. Workers read shards and persist checkpoints, then publish an immutable file manifest. The client polls the job location and receives only short-lived download credentials after completion. Checkpoints and idempotency keys make worker retries safe; expiry removes files and releases quota. Incremental exports use a cursor or sealed high-water mark instead of a moving clock boundary.”
Step-by-step deep dive
Step 1: Create the job and freeze its boundary
POST /exports validates tenant authorization, resource scope, filters, and quota, then creates an unguessable exportId. The job stores an authorization snapshot, format, compression, creation time, expiry, and snapshotWatermark. If the database supports a consistent snapshot, read the watermark in one transaction; otherwise state that this is a per-resource approximate snapshot, not strict cross-table consistency.
Step 2: Make status and creation idempotent
Accept an Idempotency-Key; the same tenant and key return the same job. GET /exports/{id} reports QUEUED, RUNNING, SUCCEEDED, FAILED, CANCELLED, or EXPIRED, plus processed shards and the next action. The state machine only moves forward. A conditional update decides a cancel-versus-complete race, and retries must not charge quota twice.
Step 3: Read shards and persist checkpoints
Workers read fixed batches by a stable primary key or a database snapshot cursor and write temporary objects. Each shard records (exportId, partition, cursor, rowCount, checksum, objectKey); checkpoint and shard-metadata commits are idempotent. After a crash, the worker reruns from the last committed cursor. Deterministic shard versions or conditional object writes overwrite duplicates, while the final manifest references each shard once.
Step 4: Define full and incremental boundaries
The full export reads at or before snapshotWatermark; later inserts and updates belong to a later incremental run. An incremental run starts from the saved cursor or high-water mark, not “now minus one second.” Zendesk documents that a cursor can be reused for the next page and next export, with a safety delay to avoid recent-write races. Put the delay, duplicate handling, and repair window in the contract.
Step 5: Publish a manifest and download safely
After every shard is complete and verified, atomically move the job from RUNNING to SUCCEEDED and create an immutable manifest containing files, sizes, checksums, and ranges. The download endpoint returns credentials scoped to the tenant and objects for a short lifetime; they cannot access arbitrary paths. A failed download retrieves the same manifest instead of rerunning the export. Files remain read-only until expiry.
Step 6: Protect resources, cancel, and clean up
Creation and scheduling enforce per-tenant limits on concurrent jobs, scanned rows, CPU, object storage, and egress. Large jobs use a queue and global concurrency cap; online requests use a separate resource pool. Cancellation stops new shards, marks the job, and asynchronously deletes temporary objects. A retryable cleaner reclaims EXPIRED jobs and records orphan bytes, failure reasons, and cleanup delay. Audit events capture the actor, scope, and downloads.
Model high-quality answer
“I would create a job with POST /exports, validate tenant authorization and quota, and persist the scope, format, expiry, snapshot watermark, and idempotency key. The response returns only exportId; the client polls GET /exports/{id}. Workers read batches from a snapshot cursor, persisting each shard’s cursor, row count, and checksum. A retry resumes from the last committed checkpoint, and (exportId, partition, cursor) makes duplicate writes idempotent. Full exports use one fixed watermark; incremental exports save a cursor or high-water mark plus a safety delay. After all shards verify, an immutable manifest is published and the job atomically becomes SUCCEEDED; downloads use short-lived tenant-bound credentials. Cancellation and expiry stop new work, remove temporary objects, and release quota. Rate limits, isolated resources, and audit logs protect online traffic.”
Common mistakes and improvements
- Scanning the whole database in one HTTP request: Use a queued job and pollable state so connections do not time out.
- Using the current clock as the incremental start: Save a cursor or sealed high-water mark and define the safety delay and duplicate policy.
- Creating a new file on every retry: Make shards and the manifest idempotent; retries fill only missing work.
- Issuing permanent download links: Bind credentials to tenant, object, and expiry; a failed download retrieves the same file.
Follow-up questions and answers
What if a record is deleted during the export?
Define the product semantics first. A snapshot export includes records visible before its watermark; an incremental stream should carry tombstones or change types. Retain tombstones until every consumer’s repair window has passed.
What if a worker writes a file and crashes before committing its checkpoint?
On restart, trust the database checkpoint and rerun the batch. Conditional writes or deterministic shard versions safely overwrite the object, and the manifest accepts only committed, verified shards, so partial work is never downloadable.
How do you stop one tenant from harming the online database?
Use read replicas, snapshots, or a dedicated query pool; cap scan concurrency and rows per second. Schedule fairly by tenant and monitor replica lag, lock waits, and egress; slow or pause exports past thresholds.
How can a client resume a broken download?
Keep the export immutable and support HTTP Range or shard downloads. The client stores the manifest, checksums, and completed ranges; refreshing credentials does not change the file version or rerun the export.