Prompt and scope
This is a release-control-plane problem, not merely changing a Deployment's replica count several times. The controller records desired version, current step, traffic weight, analysis result, and human decisions, then reflects actions reliably in the data plane. Google SRE defines a canary as a partial, time-limited deployment and evaluation. Kubernetes RollingUpdate supplies basic availability; progressive delivery adds traffic-based analysis, pauses, approvals, and automated rollback.
What the interviewer is testing
- Separate control-plane, workload, routing, and metric-analysis state.
- Model promotion as a durable idempotent state machine rather than an unrecoverable script.
- Define comparable canary/control windows, sample sizes, guardrails, and metric delay.
- Handle a newer release superseding an older one, controller restarts, unavailable metrics, and partial regional success.
- Preserve who approved, why a rollout paused or aborted, and which version became stable.
Questions to clarify first
- Is traffic divided by request, user, region, or replica? Is stable bucketing required?
- Which metrics are hard gates and which are observations? How are error budget and minimum sample defined?
- Does rollback only shift traffic, or also stop and scale down the canary? How are database and message formats compatible?
- Are steps automatic or manually approved? How is approval authorized?
- Do regions advance together, independently, or does one regional failure stop the global rollout?
A 30-second answer
“I would model a release as a durable state machine with steps, target weights, pause policy, analysis template, timeout, and rollback version. An idempotent reconcile loop applies desired state to workloads and routing, then reads actual state and version-scoped metrics. Promotion requires sufficient samples, a complete window, passing error, tail-latency, and business guardrails; a missing metric source pauses by default. Every action carries a release and step version, so restarts converge safely. Approvals, pauses, rollbacks, and routing changes become an audit trail.”
Deep-dive answer
Step 1: Define resources and states
A release resource contains release_id, candidate and stable versions, steps, current step, target weight, analysis template, pause reason, timeout, and rollback policy. States can be PENDING, RUNNING, PAUSED, PROMOTING, ABORTING, SUCCEEDED, and FAILED. Each transition needs explicit preconditions and idempotent effects.
Step 2: Separate control and data planes
The control plane stores desired state and analysis conclusions; the data plane runs Pods, Services, Ingress, or a service mesh. A successful API write is not rollout success. Observe available replicas, actual weight, readiness, and version labels. Kubernetes maxUnavailable and maxSurge constrain replacement, not request-level canary traffic.
Step 3: Design stable traffic allocation
Use a consistent request or user key so one user does not jump between canary and control. The routing layer reports actual weights and version-hit counts. For multiple regions, store target and actual weight per region; a global average must not hide one region at 100% failure.
Step 4: Define analysis windows and guardrails
An analysis template declares queries, sampling period, minimum sample, tolerance, consecutive failures, and maximum wait. Metrics cover availability, tail latency, resource saturation, and critical business outcomes, each tagged by version, region, and traffic denominator. An incomplete window or missing data pauses; missing is not success.
Step 5: Implement recoverable reconcile
The controller periodically reads the release, workload, route, and analysis result to compute one next action. External writes include release_id and step version, so retries do not duplicate rules or approvals. After a restart, it converges from persisted and observed state. If actual weight diverges, pause and repair before promotion.
Step 6: Handle pauses, approvals, and timeouts
Steps may pause automatically, for a duration, or indefinitely for approval. Approval carries identity, scope, and the current step version; an old approval cannot promote a new release. A timeout pauses or aborts by policy rather than expanding traffic. Force-promote requires authorization and a reason.
Step 7: Design rollback and compatibility boundaries
Rollback normally shifts traffic to the stable version first, then decides whether to stop or scale down the canary. Database migrations, event schemas, and cache formats need an overlap window for both versions; reverting a binary cannot undo irreversible writes. Rollback itself must be idempotent, observable, and retain the previous stable version.
Step 8: Verify, audit, and rehearse
Test promotion, routing weights, metric grouping, pauses, controller restart, metric outage, regional outage, duplicate webhooks, and a newer release superseding the current one. Audit desired and actual state, actor, time, reason, and metric snapshot. The exercise must prove a bad signal stops expansion, not merely that an API returned 200.
Trade-offs and boundaries
Native RollingUpdate versus a progressive controller
RollingUpdate fits services needing gradual replica replacement and readiness checks. Traffic, business metrics, approvals, and automated rollback require additional controller or platform capability; replica percentage is not request percentage.
Automatic rollback versus human decisions
Hard gates fit high-confidence, quickly detectable failures. For delayed or ambiguous business metrics, automation should pause and notify an owner. The policy must name the final stop authority.
Global versus regional progression
Global progression is simpler but amplifies regional risk. Independent progression is safer but requires more state and capacity. Choose from traffic isolation, data residency, and failure-domain boundaries.
Failure drills and evolution
Failure: treat canary replica ratio as traffic ratio
Replica count is not request count; connection reuse and regional traffic skew actual weight. Allocate at the routing layer and record hits.
Failure: promote when metrics are missing
Query delay, label errors, or small samples can produce empty results. Pause on missing data and rebuild a complete window after recovery.
Failure: roll back only the application image
Irreversible schema, event, or cache writes can make the old version unreadable. Add compatibility gates before rollout and shift traffic before data remediation.
Common mistakes and follow-ups
Mistake: keep the state machine only in controller memory
A restart loses steps, approvals, and rollback version. Persist release resources and audit events; memory is only a cache.
Follow-up: how do you stop an old controller overwriting a new release?
Use resource and step versions with conditional updates. Re-read before writes and stop stale actions when the version changed.
Follow-up: how do you handle two releases at once?
Use a service or route mutex, or explicitly budget traffic across candidates. Two controllers must not independently mutate one weight.
Follow-up: why can p99 regress while average latency is normal?
The average hides a small set of severe slow requests. Compare tail latency and errors with the same version, region, and denominator.
Follow-up: what if the metrics system is down?
Pause and mark ANALYSIS_UNAVAILABLE, keeping the current weight. Re-run the window after recovery; missing data is not success.
Follow-up: how do you measure the controller itself?
Track step dwell time, actual-versus-target weight error, false rollback rate, detection delay, recovery time, audit completeness, and manual overrides.