How do you design a disaster recovery strategy with explicit RPO and RTO?

Question and scope

Design recovery for a single-region SaaS after a region-wide outage. The business requires a maximum of one hour of data loss (RPO) and service restoration within four hours (RTO). Assume 1,000 requests per second at peak is an interview assumption, not a measured production fact; the application is stateless, the primary database is relational, and object storage contains user uploads.

The design must restore data, application code, configuration, secrets, routing, and operator access. It must also prevent a corrupted primary from being copied blindly into the recovery region. “We have backups” is not a complete answer: the backup cadence, restore time, failover authority, and evidence from drills must match the objectives.

What the interviewer is testing

The system-design interview handbook frames disaster recovery as an interview-style RPO/RTO problem and expects explicit failure domains, recovery choices, and operational proof. AWS defines RPO and RTO as recovery objectives set by business needs, then recommends defined strategies, testing, configuration-drift management, and automation.

A strong answer:

• translates RPO into replication or backup frequency and measures lag;
• translates RTO into restore, promotion, deployment, routing, and verification budgets;
• separates data plane recovery from control-plane actions that may be unavailable;
• chooses pilot light, warm standby, or active/active based on cost and target, rather than naming multi-region by default;
• rehearses restore and failover, including corrupted data and stale configuration.

Clarifying questions before answering

1. Does RPO apply equally to payments, profiles, and uploads? Tiering can give critical records a tighter objective than analytics.
2. Is the disaster a regional loss, a bad deployment, or logical corruption? Replication helps the first case but can copy the second; point-in-time backups are required for rollback.
3. Must writes continue during failover? If yes, choose a write authority and conflict policy; if no, a read-only recovery window simplifies correctness.
4. Is a four-hour RTO measured to health-check success, customer traffic, or full feature parity? The runbook must name the milestone.
5. Can the recovery account and control plane be operated independently? Pre-provisioned quotas, credentials, images, and DNS access may determine whether the target is achievable.

30-second answer framework

“I would tier the workload, make the business RPO/RTO measurable, and use asynchronous cross-region database and object replication plus point-in-time backups. A pilot-light region with infrastructure-as-code fits a four-hour RTO if restore and promotion stay within the budget. The runbook promotes a known-good data point, deploys the application, switches routing, runs integrity and smoke checks, and communicates status. Quarterly drills measure each step, verify backup restoration, detect configuration drift, and update the objective if the measured recovery no longer fits.”

Step-by-step deep answer

1. Turn objectives into a recovery budget

For RPO ≤ 1 hour, the maximum accepted replication lag or backup interval is one hour. Set an alert below that limit, for example when lag approaches 45 minutes, and keep point-in-time snapshots to recover from corruption. Do not claim a universal replication frequency; measure the source workload and choose a margin.

For RTO ≤ 4 hours, allocate a concrete budget: detection and declaration, data promotion, application deployment, routing, smoke tests, and a reserve. If the runbook needs 20 minutes to restore a database, 30 minutes to deploy images, and 10 minutes to switch traffic, those are assumptions to verify in drills, not guarantees.

2. Choose the recovery topology

Backup and restore is cheapest but requires rebuilding infrastructure and usually has the largest RTO. Pilot light keeps replicated data and core recovery resources ready while application capacity is off; it is a reasonable starting point for a four-hour target. Warm standby keeps a scaled-down functional stack running and shortens recovery at higher cost. Active/active minimizes failover time but adds cross-region write conflicts, traffic steering complexity, and the largest operational surface.

The choice follows the objective: do not pay active/active cost for a four-hour target unless business impact justifies it. Keep infrastructure-as-code, container images, schemas, and configuration versioned in a recovery account so deployment is reproducible.

3. Protect and recover data

Replicate relational changes asynchronously to a separate region and monitor lag. Promote only a replica whose log position and integrity checks satisfy the RPO. Store immutable, point-in-time backups in a separate account or region to handle accidental deletion, ransomware, and corrupted writes. Replicate object data with versioning and verify object counts and checksums.

The runbook records the selected recovery timestamp, freezes or drains writes when needed, promotes the data store, and reconciles any accepted write loss. If payment records require zero data loss, they need a different tier and possibly synchronous replication; do not claim the one-hour objective covers every domain.

4. Recover the serving path

Deploy the application from immutable images and infrastructure code. Pre-create DNS or global routing health checks, secrets access, service quotas, and observability in the recovery region. Route traffic only after database connectivity, migrations, authentication, and a representative read/write smoke test pass.

Keep failover control paths minimal. AWS notes that data-plane operations are typically more available than control-plane operations during a disaster; a design that depends on creating every resource through a control plane after the outage may miss its RTO.

5. Make the runbook safe to execute

Define who can declare the disaster, who promotes data, and who approves traffic. Use a monotonic step log with an idempotent command for each action, a stop condition, and a rollback or failback plan. Publish a customer status update before and after routing changes. During recovery, disable non-critical features if capacity is reduced and preserve an audit trail of the chosen recovery point.

6. Validate with drills and metrics

Run scheduled restore tests, regional game days, and configuration-drift checks. Measure detection, decision, restore, promotion, deployment, routing, smoke-test, and stabilization durations separately. Verify restored row counts, checksums, permissions, queues, and background jobs. Inject a failed backup, stale secret, insufficient quota, and partial replica lag; each should produce an actionable alert or a documented stop.

The acceptance test is evidence: the latest drill's measured RPO and RTO, unresolved gaps, and owners. A document without a restore is a hypothesis.

High-quality sample answer

“For a one-hour RPO and four-hour RTO, I would use asynchronous cross-region database and object replication, immutable point-in-time backups, and a pilot-light region managed by infrastructure-as-code. I would reserve time for detection, data promotion, application deployment, DNS or global routing, and smoke tests, then prove those timings in quarterly drills. The failover runbook selects a known-good recovery point, promotes only after integrity checks, deploys immutable images, validates authentication and representative writes, and then shifts traffic. If the primary has logical corruption, replication alone is unsafe, so the runbook restores an earlier point in time. Critical payment data would be a separate tier with stricter objectives. I would report measured lag, recovery durations, drift findings, and the next corrective owner.”

Common mistakes

• Say “multi-region” without objectives → cost and complexity are unbounded → derive topology from RPO/RTO and service tier.
• Treat replication as backup → corruption or deletion can replicate instantly → keep immutable point-in-time copies and test restore.
• Ignore code and configuration → recovered data has no serving stack → version images, schemas, secrets access, quotas, and IaC in the recovery account.
• Count DNS change as recovery → traffic reaches an unhealthy or unverified system → include promotion, smoke tests, and stabilization in RTO.
• Depend on an unavailable control plane → automation cannot create the target during the outage → pre-provision critical recovery resources and exercise the path.
• Never run a drill → backups, quotas, and runbooks can be stale → measure restore and failover regularly with injected failures.

Follow-ups and responses

What changes for an RPO of zero?

Asynchronous replication no longer satisfies the target. Define a synchronous write authority or accept a product-level write stop; then re-evaluate latency, quorum, and regional partition behavior.

How do you recover from logical corruption?

Stop promotion of the corrupted replica, identify a clean point-in-time backup, restore into an isolated environment, validate invariants, and replay only approved changes. A healthy region does not prove healthy data.

Why choose pilot light over warm standby?

Pilot light lowers steady-state cost and fits a four-hour target when deployment and scale-up are measured. Warm standby is justified when the RTO budget cannot absorb provisioning or when reduced-capacity traffic must start immediately.

How do you prevent configuration drift?

Render the recovery region from versioned IaC, compare desired and observed resources, alert on image, schema, secret, quota, and routing differences, and include drift checks in every drill.