System Design Interview: Design a Multi-Region API Gateway

Problem and Use Cases

Design the public entry point for 300 backend services. It peaks at 500,000 requests per second across three active regions and holds about 20,000 route definitions. The gateway terminates TLS, authenticates callers, applies coarse authorization and quotas, normalizes requests, selects an upstream, supports weighted canaries, and emits metrics and traces. Assume a 99.99% availability target and less than 10 milliseconds of gateway-added p99 latency. These are interview inputs, not claims about a product.

Route and policy changes should reach healthy gateways within 30 seconds. Emergency credential revocations need a faster path. A bad configuration must not take all regions down, and an unavailable control plane must not stop gateways from serving with the last known good configuration. Backend business logic, response composition, long-running workflow orchestration, and application-specific authorization remain outside the gateway.

This is a senior system-design question because the critical work sits between components: defining the hot request path, separating configuration from traffic, bounding failure behavior, and proving that a globally shared entrance is not a single point of failure.

What the Interviewer Is Evaluating

A strong answer first separates the data plane from the control plane. Regional gateway processes serve requests from an immutable local snapshot. The control plane validates, versions, stores, and distributes route and policy changes. If a candidate puts a database or remote configuration lookup on every request, neither the latency target nor control-plane failure isolation closes.

The interviewer also expects a boundary for gateway responsibilities. TLS termination, identity verification, route matching, coarse policy, rate limiting, headers, timeouts, and telemetry are cross-cutting. Inventory checks, payment decisions, and object-level permissions belong to services. A universal gateway that executes arbitrary business plugins becomes difficult to test and dangerous to release.

Capacity should be recalculable. At 500,000 requests per second, an average 2 KiB inbound request is about 500,000 × 2 KiB ≈ 0.95 GiB/s before protocol overhead. If one benchmarked gateway instance safely sustains Q requests per second at the required p99, the fleet needs at least ceil(500,000 / Q) instances, then additional capacity for a region loss. Losing one of three equal regions raises each surviving region from roughly 167,000 to 250,000 requests per second, a 50% increase. Capacity planning must include that state.

Finally, a complete answer covers configuration safety, regional routing, overload propagation, retries, observability, and explicit degradation. Saying “deploy it in multiple regions” does not explain how traffic moves, how state changes, or what remains available during a partition.

Clarifying Questions Before Answering

• Who are the clients? Assume public web, mobile, and partner clients using HTTP APIs. Internal service-to-service traffic can use a separate ingress or service mesh policy.
• What is the availability unit? A region contains gateway instances across at least three failure zones. The three regions are active-active, and global routing removes an unhealthy region.
• Are routes independent by hostname, path, and method? Yes. Matching order must be deterministic, and ambiguous route definitions are rejected before publication.
• Which policies run synchronously? Signature verification, route matching, a coarse authorization check, quotas, request limits, and header transformation. Business data is never fetched by the gateway to make a decision.
• How fresh must configuration be? Normal changes converge within 30 seconds. Gateways report their applied version. Security revocations use short token lifetimes or a small separately distributed deny list rather than forcing a full snapshot rebuild.
• How exact are quotas across regions? The default design uses regional quotas allocated from a global budget. A truly strict global counter would add a cross-region dependency to every request and must be justified separately.
• May the gateway retry? Only idempotent requests, with a small retry budget, after proving the first attempt was not accepted. Non-idempotent writes require an application idempotency key or no automatic retry.
• What is excluded? DDoS scrubbing before the gateway, service implementation, database replication, and application-level transactions are separate systems.

30-Second Answer Framework

“I would place a stateless gateway fleet in each of three active regions behind health-aware global routing. Each request stays in one region: the gateway terminates TLS, verifies identity from a local key cache, matches an immutable route snapshot, applies coarse policy and a regional quota, then selects a healthy same-region upstream. A separate control plane validates each configuration, assigns a version, canaries it to a small gateway cohort, and atomically activates it; gateways keep serving the last known good snapshot if that plane fails. I would size every region for the 50% load increase after one of three equal regions fails, bound retries and queues to avoid overload amplification, and monitor added latency, route outcomes, upstream health, configuration versions, and regional failover.”

Step-by-Step Deep Dive

The system has four layers. Global traffic management sends a client to a healthy nearby region. A regional load balancer spreads traffic over stateless gateway instances in multiple zones. The data plane proxies requests to same-region service endpoints. The control plane stores desired configuration, compiles it into versioned snapshots, and distributes those snapshots independently of the request path. Microsoft documents gateways as centralized entry points for routing and cross-cutting concerns, and its managed and self-hosted gateway model likewise separates centrally managed configuration from distributed runtime traffic.

The normal request flow is:

1. Global DNS or anycast selects a healthy region. The regional load balancer selects a gateway instance.
2. The gateway terminates TLS, enforces request-size and connection limits, and attaches a request ID.
3. It verifies a signed token using cached issuer metadata and public keys. It never calls the identity provider for every request.
4. A compiled matcher uses host, method, and normalized path to select a route from the active snapshot.
5. The policy chain applies coarse scopes, regional quota, headers, timeout, and optional canary selection.
6. The gateway picks a healthy endpoint from locally cached service discovery and forwards the request with a bounded deadline.
7. It records outcome, latency, route ID, upstream cluster, configuration version, and trace context, then streams telemetry off the hot path.

A route definition can remain declarative:

Route {
  id: string
  host: string
  methods: string[]
  path_template: string
  upstream_cluster: string
  auth_policy_id: string
  quota_policy_id: string
  timeout_ms: uint32
  retry_policy: { max_attempts, retryable_statuses }
  traffic_split: [{ revision, weight }]
}

The control API accepts a desired revision with an idempotency key and returns a revision ID. Validation checks schema, conflicting matches, referenced clusters, certificate ownership, policy limits, and unsafe retry combinations. A compiler produces an immutable snapshot with prebuilt match structures. Publication proceeds through validation, shadow comparison, a small canary cohort, one zone, one region, and then all regions. Each gateway downloads the snapshot, verifies checksum and signature, builds it off the request thread, and switches one atomic pointer. In-flight requests finish on the old snapshot. Failed health checks roll the cohort back to the previous version.

The deepest bottleneck is configuration safety at fleet scale. Sending individual mutable updates can leave route, policy, and certificate versions inconsistent. A versioned snapshot makes the activation unit explicit. Gateways persist the latest verified snapshot on local disk or durable node storage, retain at least one previous version, and expose desiredversion, downloadedversion, and active_version. A control-plane outage freezes changes but not traffic. A corrupt or incomplete snapshot is rejected. An emergency revocation should not wait for all 20,000 routes to recompile: short-lived credentials limit exposure, while a small signed deny list has its own fast distribution channel and expiry.

For capacity, benchmark the complete policy chain rather than an empty reverse proxy. If a tested instance sustains 8,000 requests per second at the target p99, peak traffic needs ceil(500,000 / 8,000) = 63 instances before headroom. With three equal regions, one-region failure leaves 250,000 requests per second per survivor, or 32 instances at that benchmark; deploying 40–45 per region gives operational headroom. The number is illustrative and must be replaced by measurements using real TLS, token verification, payload sizes, logging, and upstream latency.

The gateway must shed load instead of accumulating it. Put limits on concurrent requests, connections, request bodies, per-route queues, and telemetry buffers. Propagate deadlines to upstreams. Circuit breaking prevents a failing service from consuming every gateway connection. Retry only a narrow set of idempotent failures, use jitter, and charge every attempt to a retry budget. When an upstream is saturated, returning 503 promptly is safer than building an unbounded queue that exhausts the shared gateway.

Authentication uses local verification for signed tokens. Public keys refresh asynchronously, overlap during rotation, and keep a last known valid set for a bounded period. If a token is opaque and introspection is mandatory, cache short positive results and define fail-open or fail-closed by route risk; that dependency changes the availability calculation. Object ownership and business authorization stay in the service because the gateway lacks authoritative domain state.

Quotas are regional on the hot path. A global allocator divides a tenant's budget into short leases for regions, and regional data stores make atomic decisions. This keeps a partition from multiplying an unlimited global quota, but an isolated region can spend only its lease. If independent regions continue admitting and reconcile later, availability improves while over-admission becomes possible. The product owner must choose that bound. The detailed token-bucket implementation is delegated to the rate-limiter subsystem rather than rebuilt inside every gateway.

AWS documents both active-passive failover routing and active-active weighted routing for multi-region gateways. Here, active-active reduces cold-failover risk. Health is hierarchical: instance health removes one process, zonal signals drain a zone, and synthetic end-to-end probes plus regional error rates remove a region. Traffic shifting is gradual when possible. Regional backends and their data stores must also be ready; moving only the gateway cannot make an unavailable service healthy.

Observability needs gateway-added p50/p95/p99 latency, request and error rates by route, TLS and authentication failures, quota outcomes, active connections, queue depth, retries, circuit state, upstream latency, endpoint health, and configuration-version lag. Logs sample successful traffic but retain security and error events under a bounded budget. Traces preserve the incoming context and start a gateway span. Alerts distinguish gateway failure from upstream failure so operators do not roll back a healthy gateway for a service incident.

The main alternative is a single universal gateway versus gateways or BFFs by client or domain. One fleet simplifies external entry and governance, but enlarges the blast radius and encourages policy accumulation. Multiple gateways isolate teams and client-specific behavior, but duplicate operations and require consistent global controls. Start with one thin platform gateway plus domain-owned services; add a BFF only when a client genuinely needs aggregation or a distinct contract. A service mesh complements this design for east-west traffic and does not replace the public north-south gateway.

Verification includes route-table property tests for deterministic matching, snapshot compatibility tests, load tests at normal and one-region-failed capacity, shadow comparison of old and new revisions, and fault injection for lost control-plane connectivity, stale keys, slow upstreams, telemetry backpressure, zonal loss, and regional evacuation. A design is complete when each failure has an observable state and a bounded response.

High-Quality Sample Answer

“I will run stateless gateway fleets across three active regions and multiple zones. Global routing sends clients to a healthy nearby region; a regional load balancer spreads traffic over gateway instances. The hot path terminates TLS, verifies signed identity from a local key cache, matches a compiled route, applies coarse authorization and a regional quota, chooses a healthy same-region endpoint, and forwards with a bounded timeout. Domain authorization remains in the service.

The request path never reads the configuration database. A separate control plane validates desired route and policy changes, rejects ambiguous matches and unsafe retries, compiles an immutable signed snapshot, and rolls it out through shadow, cohort, zone, and region stages. Each instance builds the new snapshot off-thread and atomically swaps it in. It persists the last known good version, so a control-plane outage stops changes without stopping traffic. Version lag and automatic rollback make partial rollout visible and reversible.

At 500,000 requests per second, 2 KiB of inbound data is about 0.95 GiB/s before overhead. I will benchmark the full policy chain to get safe per-instance throughput Q and deploy ceil(500,000 / Q) plus failure headroom. Since losing one of three equal regions increases each survivor's load by 50%, regional capacity and load tests include that state.

I will bound connections, concurrency, request bodies, queues, and retry attempts. Idempotent retries use a small budget; non-idempotent writes require an idempotency key or no automatic retry. Global quotas are allocated as short regional leases, making the partition trade-off explicit. Finally, I will monitor gateway-added latency, route outcomes, upstream health, retry and circuit behavior, active configuration versions, and synthetic regional probes, then exercise control-plane loss, bad configuration, zonal loss, and regional failover before launch.”

Common Mistakes

• Reading routes or policies from a database on every request → latency and availability depend on the control plane → serve from a validated local snapshot and keep last-known-good state.
• Putting business logic in gateway plugins → releases gain a global blast radius and domain ownership becomes unclear → keep the gateway declarative and move business decisions to services or a justified BFF.
• Publishing individual mutable changes → routes, policies, and certificates can activate in incompatible combinations → compile one versioned snapshot and switch it atomically.
• Treating multiple instances as sufficient resilience → one bad revision can break every instance at once → canary by cohort, zone, and region, with automated health gates and rollback.
• Sizing for ordinary peak only → surviving regions overload during evacuation → calculate and test the 50% per-region increase after one of three equal regions fails.
• Retrying all failures → retries amplify overload and can duplicate writes → retry only bounded idempotent cases and require application idempotency for writes.
• Using an identity-provider call on every request → authentication inherits a synchronous remote dependency → verify signed tokens locally and refresh keys asynchronously with bounded staleness.
• Giving every region a full global quota → a tenant can multiply its allowance by region count → allocate regional leases or state the accepted over-admission bound.
• Moving gateway traffic without checking backends → the selected region may have no healthy service or data capacity → make end-to-end regional probes and backend readiness part of failover.
• Logging every successful payload → telemetry consumes hot-path resources and may leak sensitive data → emit structured metadata asynchronously, sample successes, and redact by policy.

Follow-up Questions and Responses

Follow-up 1: How do you roll back a bad route configuration without dropping requests?

Keep immutable snapshots and at least one previously verified version. A gateway builds a candidate away from request threads, validates its checksum and references, and atomically swaps the active pointer. Existing requests retain their old snapshot until completion. If cohort health regresses, the control plane marks the revision failed and gateways switch back to the previous pointer. Rollback changes configuration state; it does not restart the entire fleet.

Follow-up 2: What happens if the control plane is unavailable for an hour?

Traffic continues on the persisted last known good snapshot. Gateways expose the age and version of that snapshot and stop accepting unverified changes. Certificate and key rotations need overlapping validity long enough for this condition. Emergency revocations use short token lifetimes or a separately signed, compact deny list. Operators lose change capability during the outage, so alerts fire on distribution delay well before existing material expires.

Follow-up 3: How do you avoid duplicating a POST when a gateway retries?

The gateway does not automatically retry a non-idempotent request merely because the upstream connection failed; the backend may have committed before the response was lost. For operations that must be retryable, the client supplies an idempotency key, and the owning service stores and returns the first result. The gateway can retry only within the request deadline and a small retry budget. Connection errors before any bytes are accepted can be treated separately if the transport proves that state.

Follow-up 4: Would you choose global DNS or anycast for regional routing?

Either can satisfy the architecture. Health-aware DNS is operationally simpler but caches make evacuation gradual. Anycast can steer traffic more quickly, but it requires stronger network operations and still needs application health signals. I would choose the mechanism the platform already operates, measure failover time, and keep clients tolerant of endpoint changes. The regional gateway design does not depend on pretending DNS changes are instantaneous.

Follow-up 5: When should you split one gateway into multiple gateways?

Split when isolation or contracts genuinely differ: regulated traffic, independently operated domains, or mobile and web clients that require materially different aggregation. Do not split only to mirror every microservice, because clients then recover internal topology and operations multiply. Shared policy schemas, identity rules, telemetry, and release safety can remain platform capabilities even when runtime fleets are isolated.