System Design Interview: Design a Multi-Region Failover Coordinator

Question and when it applies

Design a coordinator that evaluates regional health, shifts traffic when rules are met, and safely restores traffic after recovery. Explain control and data planes, signals, data state, approvals, audit, and drills.

AWS ARC describes routing controls as reliable data-plane switches and stresses that failover mechanisms must work during a disaster. Kubernetes highlights replicas, topology spread, and disruption budgets. Amazon’s software interview topics emphasize applying knowledge to solve system problems.

This differs from designing a multi-region API gateway: the focus is the decision and recovery workflow, including false positives, split brain, data catch-up, and failback.

What interviewers assess

They look for multi-signal health, resilient routing controls, explicit RTO/RPO and replication, fencing against concurrent decisions, auditability, drills, and a measurable failback plan.

Questions to clarify before answering

• What are RTO, RPO, availability, and human-intervention targets?
• Is the failure regional reachability, a dependency, latency, deployment, or data corruption?
• Is the standby hot, warm, or cold?
• Which data is replicated and which writes may be rejected?
• How do DNS, a gateway, Anycast, or clients receive new routing?
• Who can take over if automation fails?
• How is dual writing prevented?
• What health and catch-up conditions gate failback?
• Is the scope one cloud or an abstract design?

30-second answer framework

“I define RTO, RPO, and standby readiness first. Each region runs its data plane; the coordinator publishes only validated routing state. Health combines business probes, dependencies, capacity, and replication lag with quorum-like signals and dampening. Before switching, I fence or degrade writes, verify the standby data threshold, and use a highly available routing control. Every state change is audited and reversible. Failback waits for health and data catch-up, then starts with a small traffic slice.”

Deep answer, step by step

Step 1: Define failure models and targets

Separate zone impairment, regional partition, dependency failure, bad deployment, and data corruption. Map RTO/RPO to standby cost, replication, and automation.

Step 2: Separate control and data planes

The data plane serves requests and executes routing. The control plane computes recommendations, approvals, and records. Failover execution should not depend on an unavailable control plane.

Step 3: Build health decisions

Combine business success, dependencies, latency, replication lag, capacity, and operator signals. One failed probe creates a candidate state; sustained thresholds advance the workflow.

Step 4: Execute a safe switch

Use states such as ready, candidate, approved, switching, stable, and failback. Leases or fencing versions prevent concurrent operations. Define write fencing, maximum switch time, and stop actions.

Step 5: Handle consistency and split brain

Choose a single writer, global sequencing, or conflict resolution. During a partition, reject unsafe writes rather than accepting unmergeable writes in both regions. Record the cutover point and data gap.

Step 6: Propagate routes and protect capacity

Explain DNS, gateway, or edge propagation, including TTL, caches, and client retries. Warm standby capacity and define throttling and degraded modes.

Step 7: Observe, audit, and hand over

Record health evidence, rule version, operator, state transitions, and route result. Alert on switch time, errors, lag, capacity, and dual-write signs. Manual actions must be authorized and idempotent.

Step 8: Drill and fail back safely

Inject failures and test staged traffic. Before failback, verify primary health, data catch-up, dependency stability, and capacity. Shift a small percentage first and retain a quick revert.

High-quality sample answer

“I would design two regions with one writer and a five-minute RTO and thirty-second RPO. Both regions have a complete data plane; the standby keeps warm caches and capacity.

Health combines business success, dependencies, replication lag, and capacity. Three consecutive bad windows create a candidate; a lease and monotonic fencing version allow only one coordinator to advance state. Before switching, writes are fenced, the standby lag is checked, and a highly available routing control moves traffic. Errors and latency are observed, with an immediate revert threshold.

Audits store evidence, rules, approvals, and route outcomes. Monthly drills test DNS caching, client retries, and standby capacity. Failback waits for catch-up, shifts five percent first, and stops if dual writes or latency appear.”

Common mistakes

• Showing two regions without a decision owner.
• Using one ping or infrastructure metric as business health.
• Automating without fencing, dampening, approval, or revert.
• Ignoring TTL, caches, and client retries.
• Allowing dual writes without conflict handling.
• Forgetting standby warm-up and capacity.
• Discussing failover but not failback or drills.
• Stating RTO/RPO without mapping them to replication and cost.

Follow-ups and how to answer

Follow-up 1: How do you handle false positives?

Use multiple signals, sustained windows, and dampening. A low-confidence failure should observe or degrade before switching.

Follow-up 2: What if the control plane is down?

Keep execution switches and required reads on a highly available data plane, pre-distribute safe state, and limit audited manual operations.

Follow-up 3: What if both regions become primary?

Use leases, fencing tokens, or an external arbiter. If single-writer ownership is uncertain, reject writes.

Follow-up 4: What if standby capacity is insufficient?

Reserve capacity, use degradation and throttling, and accept only critical traffic when the full load cannot be served.

Follow-up 5: Why not automate everything?

Automate reversible, high-confidence failures. Require human confirmation for corruption, split brain, and low-confidence events.

Follow-up 6: How do you prove drills work?

Track detection-to-recovery time, RPO, errors, capacity, manual steps, and revert results; turn gaps into owned improvements.