Prompt and context
A team wants to expose a new version to a small slice of real requests before increasing traffic gradually. The service must configure weights, compare the canary with the stable version, and automatically stop or roll back when signals degrade. Design the control and data planes, including SLOs, metric windows, durable state, permissions, idempotency, and behavior when the controller is disconnected.
This fits system design, platform engineering, and SRE roles. The core skill is turning “safe release” into a recoverable state machine, not listing Kubernetes, a service mesh, and a monitoring product. The answer should cover routing, analysis jobs, decision thresholds, human intervention, and backward compatibility with the old version.
What the interviewer is testing
A strong answer defines exposure, duration, and rollback target before separating the rollout controller, router, metric queries, and decision logic. It distinguishes success, failure, and insufficient evidence so missing metrics do not become an accidental pass. It uses durable state and idempotent actions to survive controller restarts, matches analysis windows to rollout steps, and addresses small-sample noise, delayed metrics, and rollback storms.
Questions to clarify first
- Is the target one service, a multi-service orchestration, or only Kubernetes workloads?
- Is traffic split by random request, stable user hash, region, or tenant, and must users remain sticky?
- Which SLOs matter: error rate, latency, business conversion, or cost, and how is the stable baseline chosen?
- How long may each step run, how many failures trigger rollback, and should uncertain analysis pause for a human?
- Are database schemas, message formats, and external APIs backward compatible, and is rollback of the old version safe?
A 30-second answer framework
“I would split the service into a rollout controller, traffic-routing adapter, metric analyzer, and durable state store. Every rollout has stable and candidate versions, staged weights, an analysis window, and explicit success, failure, and inconclusive conditions. The controller expands only after success, returns traffic to stable on failure, and pauses with an alert when evidence is inconclusive. Commands carry a version and idempotency key, and state is persisted before the next step. During a short router or metrics outage, keep the last safe weight and resume from durable state after recovery.”
Step-by-step solution
Step 1: define goals and scale
Assume one region handles 200 releases per day, each lasting at most 40 minutes, while the controller processes tens of state and metric events per second; business requests remain on the existing gateway and services. This order-of-magnitude estimate supports a highly available controller and queue rather than a workflow instance per request.
The safety goal is to limit blast radius. A configurable sequence such as 1% → 5% → 25% → 50% → 100% is illustrative, not a universal threshold. Each step needs a minimum observation time and a maximum wait so a missing signal cannot occupy a release slot forever.
Step 2: separate control and data planes
The control plane stores the release specification, version digests, current step, target weight, analysis results, and actor. The data plane uses a gateway or service mesh to route requests to the stable or canary ReplicaSet. Argo Rollouts’ architecture separates the Rollout, two versioned ReplicaSets, Services/Ingress, and AnalysisTemplate/AnalysisRun, demonstrating boundaries that can evolve independently.
The metric adapter only executes queries against providers such as Prometheus and returns observations with their time window. The decision engine applies policy and does not edit routes directly, so replacing a metrics backend does not change the rollout state machine.
Step 3: build a recoverable state machine
Use states such as Draft → Running → Paused → Promoting → Succeeded, with Running or Paused able to enter Aborting → RolledBack. Every transition carries a rollout_id, desired version, step number, and idempotency key. A unique constraint or compare-and-set prevents two controllers from advancing the same rollout.
Running + analysis=success -> Promoting(next_weight)
Running + analysis=failure -> Aborting(weight=0)
Running + analysis=inconclusive -> Paused(reason=insufficient_signal)
Paused + operator=resume -> Running
Aborting + route=stable -> RolledBackAfter a restart, the controller replays unfinished actions from the last committed state. Route updates and state writes cannot form one cross-system atomic transaction, so actions must be repeatable: setting the same weight twice has no extra effect, and the controller reads the actual route before choosing the next step.
Step 4: choose metrics and windows
Each step should have at least one reliability signal and one business signal, such as the canary-versus-stable error-rate delta, P95 latency delta, and key-request success rate. Fix the query window, denominator, and filters so canary retries are not compared with stable’s original requests.
The window must cover collection delay and stay within the step timeout. Google’s SRE canary guidance compares the canary with a control and warns that a metric period longer than a short canary stage produces a muddled signal. Mark small samples inconclusive; pausing is safer than expanding on weak evidence.
Step 5: handle traffic splits and stickiness
Random request splitting fits stateless APIs. For a consistent experience, use a stable hash of user or tenant and record the routed version. Percentage routing must handle an unhealthy candidate, a weight that was not applied, regional divergence, and cache-key collisions.
The routing adapter returns the effective weight and version digest. If the desired and actual values differ, the controller pauses and alerts; a successful routing API response is not proof that traffic has switched.
Step 6: design rollback, pause, and human intervention
On failure, stop increasing canary traffic and reduce it to zero or a safe weight. Keep the old version runnable, and use an expand/contract order for database migrations so rollback can still read the schema. Rollback itself needs a timeout and retry limit to avoid an infinite loop when the router is unavailable.
When analysis is Inconclusive, pause with the evidence: query, sample count, version, window, and threshold must be auditable. Argo Rollouts documents Inconclusive as a paused outcome for human judgment, which is safer than treating missing data as success.
Step 7: reliability, permissions, and audit
Use leader election or a lease for the controller. Queue delivery may be at least once, so consumers deduplicate by idempotency key. Write release specs, policy changes, and approvals to an immutable audit log. Only release owners change weights; metric credentials come from a secret manager; rollback permission is separate from ordinary promotion.
Track control-plane SLOs: transition latency, stuck rollouts, rollback duration, actual-versus-desired weight, and metric-query failures. When the control plane fails, keep the last safe route and page an operator instead of automatically sending the canary to 100%.
Step 8: verify and load-test
Inject faults for rising error rate, no data, delayed data, router timeouts, controller restarts, duplicate messages, and an incompatible database schema. Check the final state and alert for each fault, not only the happy path.
Replay historical releases to measure analysis-query cost and queue backlog, and run N+1 controller-failure tests. Use one hundred concurrent rollouts as an example load ceiling, observe state-store lock contention, metrics-provider QPS, and route-update rate, then set a concurrency limit.
Trade-offs and boundaries
Automation removes human delay, but a bad threshold can turn noise into rollback or a real regression into a pass. An absolute error-rate threshold is easy to explain and suits low-traffic services; a stable comparison or stratified baseline handles daily traffic changes better but needs more careful statistics and sample alignment. High-risk services can require human approval after automated analysis.
Blue-green gives a fast cutover and simple rollback but usually needs double capacity. Canary reduces exposure but requires traffic splitting and analysis windows. A feature flag can separate a feature launch from a binary release, yet it cannot replace compatibility checks for binaries, dependencies, or schemas. Choose according to rollback cost, traffic shape, and SLO risk.
Rollout plan and evidence
Start with read-only analysis and manual pause for one stateless service. Verify stable/canary labels, metric grouping, and audit fields; then enable automatic rollback; finally add multi-region signals, business metrics, and a concurrency cap. Keep a human kill switch and an explicit owner at every stage.
Google SRE defines canarying as a time-limited, partial deployment plus evaluation and requires that evaluation to feed the release process. Argo Rollouts provides AnalysisTemplate/AnalysisRun, metric thresholds, and success, failure, and inconclusive outcomes. Public system-design interview material also treats traffic splitting, guardrail evaluation, and automatic rollback as canary design points. This article therefore focuses on a recoverable control-plane state machine rather than a glossary of deployment strategies.
Common mistakes and follow-ups
Only saying “watch 10% of traffic”
Without the population, duration, denominator, and failure action, safety is unproven. Add a stable baseline, window, threshold, pause, and rollback path.
Treating missing metrics as a pass
A collector outage can fabricate a healthy signal. Mark no data, NaN, delay, and insufficient samples inconclusive; pause and alert.
Rolling back a Deployment but not the route
Healthy old pods do not prove requests left the canary. Verify the effective weight, service selectors, and cache or session stickiness.
Making the database migration irreversible
The old version may not read the new schema, so switching the route back cannot restore safety. Use backward-compatible expand/contract changes and make migration state a rollout gate.
What if the metrics provider is down?
Keep the last safe weight, stop automatic expansion, record the unfinished analysis, and notify the owner. After recovery, resume from durable state; never fill the gap with a default “pass.”