Problem and context
Kubernetes v1.35 introduces restartPolicyRules for containers when the RestartAllContainers action is enabled. The rule can map an exit code to a restart action, while the Pod still has a lifecycle and readiness contract. Treat the feature as failure classification, not as a replacement for probes or alerting.
Assume a worker Pod has three containers with different failure domains. The fetcher may recover from a transient credential refresh failure; a proxy configuration error should page an operator; a reporter may need the whole Pod restarted when its local state is corrupt.
What interviewers evaluate
Interviewers look for a precise failure taxonomy, awareness of feature-gate and version prerequisites, and a plan that prevents restart loops from masking incidents. Strong answers connect exit codes to ownership, readiness, backoff, metrics, and rollout safety.
An ordinary answer adds restartPolicyRules to every container. A strong answer explains which codes are stable API semantics, which are accidental process details, and how to prove that a restart is safe for each container’s state.
Questions to clarify first
- Which Kubernetes versions and feature gates are guaranteed in every cluster?
- Are exit codes controlled by the application contract or emitted by a runtime and shell wrapper?
- Is container state disposable, checkpointed, or coupled to another container in the Pod?
- What should happen when the same code repeats: backoff, Pod replacement, or escalation?
- Which signals define user readiness while one container is restarting?
If codes are not part of a tested application contract, prefer a uniform restart policy and improve the process contract first. If state is coupled, restarting one container may create a split-brain interaction with its peers.
A 30-second answer
“I would first verify the Kubernetes version and feature gate, then define exit-code semantics in each container’s contract. I would restart only transient, stateless failures; let permanent configuration failures become visible; and use Pod replacement for corrupted shared state. I would instrument rule matches, restart counts, backoff, readiness, and repeated-code alerts, canary the policy, and roll it back if error rates or restart loops increase.”
Step-by-step design
- Check prerequisites. Confirm v1.35 behavior,
RestartAllContainers, admission policy, and whether the cluster’s controller and observability stack understand the new fields. - Define exit-code ownership. Reserve a small documented set such as transient refresh, permanent configuration, and unrecoverable state. Test wrappers so signals are not remapped accidentally.
- Map the smallest safe action. Restart a stateless fetcher for transient codes. Do not restart a proxy for a bad configuration; surface it as NotReady and alert. Replace the Pod when shared state is invalid.
- Protect dependencies. Gate readiness on the dependency contract, coordinate shutdown hooks, and avoid accepting traffic while a restarted container lacks warmed state.
- Control repetition. Combine restart counts, exponential backoff, and a repeated-code alert. A rule that keeps restarting must eventually become an operator-visible failure.
- Roll out gradually. Canary one workload, compare restart-loop rate, recovery time, error rate, and Pod churn with the previous policy, then expand only when guardrails hold.
Alternatives include a supervisor inside one container, separate Deployments for independent failure domains, or a controller that replaces the Pod. Choose the simplest boundary that preserves state and makes failure visible.
Example answer
“For the fetcher, exit code 42 means a short-lived credential refresh failure and is safe to restart because it has no durable local state. A configuration error remains NotReady and pages the owner; restarting it would only repeat the same failure. If the reporter detects corrupted local checkpoints, I would terminate the Pod so a clean volume or replacement can recover consistently. I would ship the rules behind the feature gate to one canary, alert on repeated matches and restart loops, and remove the policy if recovery time or error rate regresses.”
Common mistakes
- Error: Treating every non-zero exit as transient → Why it fails: permanent faults become silent restart loops → Fix: define tested exit-code semantics.
- Error: Ignoring feature-gate and cluster skew → Why it fails: manifests behave differently across environments → Fix: add admission and rollout checks.
- Error: Restarting one container with coupled state → Why it fails: peers retain incompatible state → Fix: replace the Pod or coordinate recovery.
- Error: Monitoring only container restarts → Why it fails: users may still see errors while restarts look healthy → Fix: pair restart metrics with readiness and service SLOs.
Follow-up questions and responses
What if exit code 42 is emitted by a shell wrapper and changes after an image update?
Treat the code as an API. Pin and test the wrapper, document ownership, and fail the rollout when the contract changes. Do not silently inherit runtime-specific codes.
How do you prevent a restart loop from hiding an outage?
Alert on repeated rule matches, restart rate, backoff saturation, and readiness loss. After a bounded number of attempts, escalate to Pod replacement or an operator-visible failure.
When is restarting the whole Pod safer?
Use Pod replacement when state is shared, initialization order matters, or one container’s corruption can invalidate its peers. A broader restart is preferable to inconsistent partial recovery.
How would you roll back the feature?
Disable the policy at the workload template, restore the prior restart behavior, and verify that old replicas converge. Keep the exit-code contract and dashboards so the rollback remains observable.