System design interview: Design a multi-tenant quota and usage metering service

1. Question and context

A SaaS platform serves API requests, concurrent jobs, and storage for many tenants. A tenant may have project-, organization-, and resource-level quotas; some reset per minute while others accumulate over a billing period. The business wants a fast admission decision and trustworthy usage for alerts, audit, and billing. Design the quota-check and metering service, including hard limits, soft thresholds, overage policy, and multi-region failure behavior.

2. What the interviewer evaluates

• Whether you distinguish allocation, rate, and concurrent limits and define their scope and reset semantics.
• Whether you separate synchronous admission from asynchronous usage events, billing aggregation, and reconciliation.
• Whether you handle atomic reservation, idempotency keys, duplicate or out-of-order events, hot tenants, and configuration versions.
• Whether you explain cross-region consistency, degradation, audit trails, alerts, and safe manual changes.

3. Clarifications to ask before answering

1. Are we limiting request rate, simultaneous jobs, storage capacity, or cumulative usage in a billing period?
2. Is the scope organization, tenant, project, user, or resource instance, and do limits inherit down the hierarchy?
3. On exhaustion, must the system reject, queue, degrade, or allow overage for billing?
4. Is strong multi-region consistency required, and how much event delay is acceptable before reconciliation?

4. A 30-second answer framework

I would split the system into a quota catalog, synchronous admission engine, reservation ledger, asynchronous usage-event pipeline, aggregate queries, and audit alerts. A request carries tenant, project, resource type, and an idempotent request ID. The admission engine checks rate, concurrency, and cumulative quotas under a configuration version and atomically reserves when needed; completion or cancellation releases the reservation and emits a usage event. Events carry a unique ID, event time, and dimensions; consumers aggregate idempotently, then reconcile the period with the ledger and billing input. For each resource, choose authoritative writes or bounded oversell across regions; protect hard limits during failure and return a retryable signal when appropriate.

5. Step-by-step deep answer

Step 1: Define quota models and scopes

The quota catalog stores resource type, scope, unit, window, limit, adjustability, and configuration version. A rate quota limits consumption in a time window, a concurrent quota limits operations running at once, and an allocation quota limits resources already allocated. An organization limit can be the ceiling; tenant and project reservations must fit within parent remaining capacity so independent checks cannot oversell the total.

Step 2: Design synchronous checks and atomic reservations

The synchronous path owns facts that affect admission: current-window counters, active reservations, and configuration version. Use sharded hot keys or tenant-partitioned strongly consistent storage for a hot tenant. A reservation must check remaining capacity and increase the reserved amount atomically, preventing two concurrent requests from using the same balance. Long jobs receive a reservation ID; completion, cancellation, and expiry are idempotent.

checkAndReserve(tenant, dimensions, amount, requestId, configVersion)
  verify configVersion is active
  if requestId already committed: return previous decision
  atomically check remaining quota and add reservation
  persist reservation with expiry and requestId
  return reservationId and retryAfter

Step 3: Decouple usage events from metering aggregation

Success, failure, cancellation, and expiry should emit events. Each event carries tenant, project, resource, quantity, unit, event time, and a unique event ID. With at-least-once delivery, consumers deduplicate by event ID; out-of-order events use an event-time window or replayable ledger. Aggregates serve queries, threshold alerts, and billing input, but must not overwrite the synchronous reservation ledger.

Step 4: Handle overage, quota changes, and fairness

Reject or queue when a hard quota is exhausted; a soft threshold should trigger an alert. Whether overage is allowed must be an explicit product setting with an authorized actor and price rule. Before lowering a quota, inspect existing reservations so accepted work does not suddenly lose capacity. A hot tenant must not consume a shared shard; tenant concurrency caps, token buckets, and fair queues can protect other tenants. Increase requests should follow approval or automation rules and retain old versions for audit.

Step 5: Multi-region behavior, recovery, and reconciliation

If a hard limit must be globally exact, route the resource to one authority or use consensus-backed storage. If availability matters more, allocate regional budgets and state the maximum oversell. During a regional partition, reject hard-limit decisions that cannot be made safely; a soft limit may temporarily degrade to alerting. After recovery, replay immutable events and reservation records, compare admission decisions, aggregates, and billing results, and repair differences with compensating events instead of editing history.

6. High-quality sample answer

I would separate a quota catalog, synchronous admission engine, reservation ledger, asynchronous metering pipeline, and audit queries. The catalog defines organization, tenant, and project scopes plus rate, concurrent, and allocation limits. Admission uses tenant dimensions and an idempotency ID for atomic check-and-reserve; long jobs get a reservation ID and make completion, cancellation, and expiry idempotent. Success and failure events enter an at-least-once pipeline; consumers deduplicate by event ID and aggregate by event time for alerts and billing, without overwriting the admission ledger. Per resource, multi-region behavior is either authoritative or bounded-oversell. During failure, protect hard quotas; after recovery, replay and reconcile, with every change and compensation auditable.

7. Common mistakes

• Using one shared counter → a hot tenant slows everyone → shard by tenant and enforce fair caps.
• Letting an asynchronous aggregate decide admission → event lag causes oversell → keep an atomic reservation ledger as admission truth.
• Discussing only rate limits → long jobs and storage consume unlimited capacity → model rate, concurrent, and allocation limits.
• Deduplicating only request IDs → retried events are counted twice → use separate idempotency keys for reservations and usage events.
• Overwriting balance when lowering a quota → accepted work is suddenly rejected → version configuration and inspect existing reservations.

8. Follow-up questions and responses

Follow-up 1: Why not rely only on a Redis counter?

A counter is useful for low-latency window statistics, but cross-resource atomic reservations, job expiry, configuration versions, and audits need a fuller ledger. A counter can be a fast path only when an authoritative record and rebuild process exist.

Follow-up 2: How do you prevent duplicate usage events from billing twice?

Give every event a stable unique ID. The consumer writes the deduplication record and aggregate in one transaction, or uses an equivalent atomic operation. Aggregates remain recomputable; billing consumes a confirmed aggregate version and keeps compensating events.

Follow-up 3: Do you continue admitting work during a multi-region partition?

Pause or route resources to the authority for hard quotas that cannot oversell. For a soft quota with a bounded oversell budget, admit from regional allocations and record the maximum risk. The choice follows business loss and consistency objectives.

Follow-up 4: How do you test the quota service?

Load-test a hot tenant, concurrent reservations, duplicate and out-of-order events, configuration downgrade, regional partition, consumer replay, and period rollover. Assert that hard limits never break, idempotent operations are stable, and ledger, aggregates, and billing converge after recovery.