Prompt and use cases
This is a multi-region availability design problem. The useful answer explains traffic decisions, health-signal propagation, capacity and data boundaries during failure, and how failover and failback are rehearsed.
What the interviewer evaluates
- Whether DNS, edge proxies, and application routing have distinct responsibilities and time scales.
- Whether health uses layered signals from multiple observers instead of one endpoint.
- Whether failover capacity, connections, and rate limits are calculated.
- Whether stateful data has explicit write ownership, replication lag, and regional constraints.
- Whether split brain, flapping, and failback cascades are controlled.
- Whether RTO, RPO, SLO, and exercise evidence are defined.
Clarifications before answering
- Which regions, traffic peak, and single-region failure model apply?
- What are the RTO, RPO, residency, and compliance constraints?
- Are requests mostly stateless reads, or do they include writes and long-lived connections?
- Is steering done by DNS, Anycast, an edge proxy, or a service mesh?
- Which user journeys and dependencies must health checks cover?
- What do client DNS TTL, connection migration, and failback windows require?
30-second answer framework
“I would define the regional failure, peak load, and RTO/RPO first. DNS or an edge entry point can choose latency-based candidates, but only regions passing layered health checks with spare capacity receive traffic. During overload, we rate-limit and degrade by priority so the backup region survives. Writes follow a data-ownership boundary with explicit replication lag and retry semantics. A damped controller shifts weight gradually, and we rehearse, warm, and verify metrics before failback.”
Step-by-step deep dive
Step 1: Define failure and objectives. State the region, dependency, network, and control-plane failure scope, then quantify RTO, RPO, peak, and degradation levels.
Step 2: Separate decision layers. DNS or a global entry point handles coarse region choice; edge or service layers handle live weights, connections, and local limits. Do not make one control plane carry every failure action.
Step 3: Build health signals. Combine probes from multiple locations, critical user journeys, dependency state, error rate, and capacity. Consecutive windows, recovery windows, and quorum-like decisions reduce flapping.
Step 4: Protect capacity. Reserve failover headroom and set per-region concurrency, queue, and rate limits. During failure, shed low-priority work before critical traffic causes a cascade.
Step 5: Define data boundaries. Explain replica lag, write ownership, conflict handling, idempotency keys, and cross-region retries. Routing cannot make unavailable writes consistent.
Step 6: Execute failover. The controller records reason, version, and approval, lowers the failed region's weight, and shifts a small cohort before expanding. Long-lived connections need backoff and session recovery.
Step 7: Rehearse failback. Warm the recovered region, verify stable metrics and data checks, then fail back. Inject regional, dependency, and control-plane failures regularly and preserve measured RTO/RPO evidence.
Model high-quality answer
“I would split each region into an entry layer, stateless services, and a data unit. The global entry point selects latency candidates, but a region receives traffic only when multi-location journey checks, error thresholds, and capacity headroom pass. The controller applies a minimum dwell time and cooldown, first sending 5% to the backup. If backup concurrency reaches its limit, it protects login and writes while pausing low-priority reports. Tenant ownership determines write location; cross-region retries carry idempotency keys and expose replication lag. Exercises validate RTO, RPO, reconnect success, and data checks for both failover and failback.”
Common mistakes
- Draw DNS and two regions only → capacity and data are missing → add controller guardrails and write ownership.
- Remove a region after one failed probe → traffic flaps → use windows, cooldowns, and multiple signals.
- Assume backup accepts all traffic → failover overloads it → calculate headroom and staged degradation.
- Treat TTL as completion time → clients still cache old answers → include resolver, connection, and edge delays.
- Say active-active without conflict rules → writes are undefined → state ownership, replication, and idempotency.
Follow-up questions and responses
Follow-up 1: What if DNS TTL is long?
Use the edge for faster weight changes and include TTL, recursive caches, and connection lifetime in the RTO budget; do not promise instant movement.
Follow-up 2: What if health checking fails?
Use independent control paths and probes from multiple locations, track freshness, and hold the last safe state or enter a protected manual mode when checks expire.
Follow-up 3: What if backup capacity is insufficient?
Reserve and warm capacity, then rate-limit, degrade, or queue by priority. Prove the single-region ceiling with load tests.
Follow-up 4: How do you prevent flapping?
Use different failure and recovery thresholds, minimum dwell time, cooldown, and approval, with a reason recorded for each weight change.
Follow-up 5: How do you handle cross-region write conflicts?
Assign ownership by tenant or key and use version or idempotency conditions. If multi-writer is required, define conflict rules and non-mergeable data.
Follow-up 6: How do you prove it works?
Exercise regional, dependency, network, and control-plane failures and measure RTO, RPO, errors, recovered capacity, reconnects, and data checks.
Follow-up 7: When should you avoid multi-region?
If residency, replication semantics, operating capability, or cost cannot meet the target, use a tested single-region design with disaster recovery first. Multi-region is not automatic.