System design interview: How do Kubernetes topology spread constraints improve availability?

Prompt and context

A stateless service runs across multiple zones and scales from 2 to 30 replicas. During node failures, scaling, and rolling releases, the team wants to avoid concentrating replicas in one failure domain without making a new release impossible to schedule. Design topologySpreadConstraints and explain when to use DoNotSchedule, ScheduleAnyway, or anti-affinity.

What the interviewer evaluates

• Whether you turn availability into explicit node, zone, and region failure-domain goals.
• Whether you correctly explain maxSkew, minDomains, topologyKey, and whenUnsatisfiable.
• Whether you catch selector, missing topology-label, and small-replica edge cases.
• Whether placement, PDBs, rolling updates, autoscaling, and observability work as one design.

Clarifying questions

1. Are zones independent in power, networking, and capacity, and is a regional failure in scope?
2. How many replicas must remain after losing one failure domain?
3. Should an unschedulable Pod wait, reduce availability, or tolerate temporary skew?
4. Are workload labels, node topology labels, and default constraints platform-managed?
5. Do old and new versions use the same labelSelector during a rolling update?

30-second answer

I would define failure domains and minimum capacity first, then set separate placement goals for nodes and zones. Critical services use bounded DoNotSchedule; ordinary or elastic workloads use ScheduleAnyway as a soft goal. maxSkew limits the difference for one selector across eligible domains, while minDomains prevents misleading calculations when domains are missing. Before rollout I would test labels, scaling, rolling updates, and single-domain failure; in production I would monitor replicas per domain, Pending time, available capacity, and the PDB disruption budget.

Deep-dive answer

Step 1: Define domains and capacity

Map nodes, zones, and regions to distinct topologyKey values. If the service must survive a zone loss, keep enough replicas in at least two zones; with only two replicas, you cannot promise even placement across three zones and two healthy replicas after losing one. State the capacity invariant before choosing strictness.

Step 2: Choose maxSkew and minDomains

maxSkew is the permitted difference between a target domain and the global minimum; with DoNotSchedule, exceeding it leaves the Pod Pending. minDomains expresses how many eligible domains are required and prevents an undersized domain set from being treated as a valid distribution. Test small-replica services so the constraint does not block releases.

Step 3: Separate hard and soft constraints

Control-plane or payment services can use zone-level DoNotSchedule and turn placement failure into a capacity alert. Batch or deferrable work can use ScheduleAnyway, asking the scheduler to reduce skew while allowing temporary imbalance. Hard anti-affinity fits a simple “do not co-locate” rule; multi-level topology and measurable skew are usually clearer with spread constraints.

Step 4: Make selectors and labels trustworthy

The constraint labelSelector must match the actual Pod template, or new Pods may be counted against the wrong set. Nodes need stable zone, region, and hostname labels; a node without the topology key does not participate correctly in that domain calculation. Admission checks should validate selectors, labels, and defaults before teams copy a broken manifest.

Step 5: Coordinate releases, PDBs, and scaling

A rolling update must account for old replicas, new replicas, and maxUnavailable together. A PDB limits voluntary disruption; it does not replace cross-domain placement. The autoscaler should understand Pending Pods and per-domain capacity, otherwise strict constraints can wait forever without adding usable nodes.

Step 6: Define failure and degradation actions

Exercise a node loss, a zone loss, and missing node labels, then observe whether new Pods are rejected, skewed, or Pending. Critical services can pause low-priority releases, add capacity to healthy domains, or enter read-only mode. Do not remove a hard constraint in production without recording the availability risk. Version and roll back degradation actions.

Step 7: Verify with distribution metrics

Record replicas, skew, Pending duration, scheduling reasons, available capacity, PDB disruption, and request errors by workload, version, and topology domain. Load tests should cover 2-to-30 replicas, uneven domain capacity, rolling updates, and autoscaling. The goal is to prove that remaining capacity meets the SLO after a domain loss, not merely that replicas landed on different nodes.

Model answer

I would model nodes, zones, and regions as three failure-domain layers and first set the minimum replicas required after losing a zone. Service Pods use a selector matching the template; constraints apply separately to hostname and zone. Critical services use a small zone maxSkew with DoNotSchedule, while batch work uses ScheduleAnyway. If eligible domains fall below minDomains, raise a capacity alert rather than silently accepting a misleading distribution. Validate PDB, maxUnavailable, and old/new selectors during rolling updates, and make the autoscaler observe Pending reasons and domain capacity. Roll out in observation mode, then enable hard constraints gradually with per-domain replicas, Pending time, failure-domain drills, and business SLOs as rollback gates.

Common mistakes

• Saying “deploy across zones” without naming the topologyKey or capacity target.
• Treating maxSkew as an absolute per-domain replica limit.
• Ignoring a selector mismatch and therefore counting the wrong Pod set.
• Treating a PDB as a scheduler constraint or as protection from every node failure.
• Promising even placement across three zones and no degradation after a zone loss with only two replicas.
• Removing DoNotSchedule to clear Pending without recording the availability trade-off.

Follow-up questions

Follow-up 1: Does ScheduleAnyway still help availability?

Yes. It makes lower skew a scheduling preference while allowing a workload to run when capacity is constrained. It fits deferrable work or workloads with other redundancy; critical services can scale ahead and use a hard constraint.

Follow-up 2: Why not use only pod anti-affinity?

Anti-affinity says not to co-locate with selected Pods. It expresses multiple topology levels, numeric skew, and minimum domain requirements less directly. Spread constraints describe cross-domain imbalance, while anti-affinity remains useful for one exclusion rule.

Follow-up 3: What happens when minDomains is wrong?

With too few eligible domains, the global minimum used for skew can make a constraint appear satisfied or keep Pods Pending. Include domain availability in admission checks and capacity alerts, and verify field behavior for the cluster version.

Follow-up 4: Why can a rolling release break distribution?

Old and new versions may use different selectors, or maxUnavailable and maxSurge may temporarily add replicas in one domain. Simulate intermediate states and inspect skew by version and domain before release.

Follow-up 5: What if a node lacks a zone label?

It will not participate correctly in the requested topology calculation. Repair or isolate the node; do not count an unlabeled node as an independent failure domain.

Follow-up 6: How do you prove the SLO survives a zone failure?

Run an isolation drill and verify schedulable capacity, healthy replicas, request errors, and recovery time in the remaining domains. Record PDB status, Pending reasons, and scale-up time to validate the full path from scheduling to business metrics.