Prompt and scope
Assume 20,000 tenants and 1 million operations per hour, with 10x short bursts and a p95 authorization decision under 100 ms. Customers buy credits before running operations. Each operation has a variable cost, requests can arrive concurrently, and delivery of usage events is at least once. The design must prevent negative balances and double charging while preserving an audit trail. This is a purchased-value ledger, not a rate limiter or a recurring subscription-billing system.
What the interviewer is testing
- Separating immutable credit movements from a fast balance read model.
- Making authorize, capture, release, refund, and grant operations idempotent.
- Handling concurrent requests, delayed events, retries, expiry, and reconciliation.
- Explaining consistency boundaries, hot tenants, observability, and recovery.
Clarifying questions to ask
Ask whether credits expire, whether a failed operation may reserve then release, how refunds are approved, whether cost is known before execution, and whether one tenant can spend in multiple regions. Confirm the required decision latency and whether temporary over-authorization is allowed.
The 30-second answer
I would use an append-only, immutable ledger of grants, reservations, captures, releases, refunds, and expirations. A transactional balance projection supports fast reads, while an idempotency key makes every mutation repeatable. Authorization atomically checks available credits and creates a reservation with an expiry; successful work captures it, failures release it, and a reclaimer handles abandoned reservations. An outbox, replayable events, and reconciliation compare the projection with the ledger and repair drift with auditable compensating entries.
Step-by-step deep dive
1. Model the credit state
Keep one tenant/account key, integer credit units, a ledger sequence, and a projection containing granted, reserved, captured, released, refunded, and expired totals. Store each mutation with operationid, idempotencykey, reason, actor, and timestamp. Never edit an old entry; corrections append a compensating entry.
2. Authorize without double spending
Use one transaction or a linearizable conditional update for the tenant’s hot key: create a reservation only when available credits are at least the requested cost. A repeated idempotency key returns the original decision; a reused key with different parameters is rejected. Keep reservation expiry and status transitions conditional so capture and release cannot both win.
3. Connect execution and events
The business request carries the reservation identifier. Capture follows confirmed success; release follows a failed or cancelled operation. Usage events may be duplicated or delayed, so consumers deduplicate by event_id and reconcile unknown reservations instead of blindly decrementing a counter. An outbox publishes committed ledger changes after the transaction.
4. Scale, alternatives, and recover
Partition by tenant, route one tenant to a home writer, and isolate exceptionally hot tenants with admission control or a serialized command stream. A relational transaction is the simpler fit when each tenant’s write rate is moderate; an event-sourced command stream is preferable for very hot tenants when replay and ordering matter more than query simplicity. Reads can use replicas only when their staleness is shown. A sweeper expires abandoned reservations, and replay rebuilds the projection from the ledger. Alerts cover negative availability, capture-after-expiry, replay lag, and reconciliation differences.
A strong sample answer
I would clarify expiry, refund authority, multi-region spending, and whether cost is known before execution. The source of truth is an immutable per-tenant ledger; the balance table is a rebuildable projection. authorize atomically reserves cost under an idempotency key and expiry, capture converts that reservation after success, and release or expiry returns it. Duplicate or delayed events are deduplicated and reconciled. Tenant-affine writes avoid cross-region double spending; if a region must operate independently, it receives a bounded allocation whose debt is reconciled explicitly. Every correction is an append-only compensating entry.
Common mistakes
- Mutable counter only → retries and refunds cannot be audited → append an immutable entry with an operation identity.
- Charge on request receipt → failed work still spends credits → reserve first and capture only after success.
- Retry treated as new consumption → one operation is charged twice → reuse the same idempotency key.
- Reservation never reclaimed → crashed workers lock credits forever → add expiry and a conditional sweeper.
- Eventually consistent cross-region read → two writers can overspend → use one writer or an explicit regional budget.
- Edit old rows to fix history → the audit trail becomes untrustworthy → append a compensating entry.
Follow-up questions and responses
The client timed out after authorization. What happens?
Query the reservation or retry the next transition with the same idempotency key. Do not authorize again until the original reservation is captured, released, or expired.
How do you process a refund after the month closes?
Append a refund entry referencing the original capture and an approval record. The projection increases available credits, while the ledger preserves both events and their ordering.
Can two regions spend the same account concurrently?
Prefer a single writer or tenant home region for strict no-double-spend semantics. If bounded local spending is required, allocate explicit regional budgets and reconcile the debt; state the maximum temporary overage.
How do you prove the balance is correct?
Periodically replay the ledger, compare the derived available amount with the projection, and compare captures with business-operation outcomes. Emit an idempotent compensating entry and an audit record for every repair.