System Design Interview: How Would You Design a Maintenance Orchestrator That Respects PDB and Topology Spread?

Prompt and scope

A cluster needs a rolling node upgrade while workloads use PodDisruptionBudget and topology spread constraints. Design the maintenance orchestrator, covering candidate nodes, eviction concurrency, unschedulable replacements, capacity reservation, recovery, and observability.

Kubernetes defines a PDB to limit Pods affected by voluntary disruptions; maintenance tools should use the Eviction API so the budget participates in admission. Topology spread constraints control skew across failure domains such as zones and nodes. A strong design puts both constraints in one control loop instead of checking a Pod count once and deleting a batch.

What the interviewer evaluates

• Distinguishing voluntary disruption from involuntary failure and its guarantee boundary.
• Using the Eviction API instead of deleting Pods directly.
• Evaluating maxSkew, whenUnsatisfiable, and label selectors from topology spread.
• Designing concurrency and capacity reservations per node and failure domain.
• Handling PDB blocking, unready Pods, retries, timeouts, and rollback.
• Explaining decisions through events, metrics, and audit records.

Clarifying questions to ask

1. Is this an OS upgrade, instance replacement, or emergency repair? Emergency failure may bypass PDB protection.
2. What are replica counts, PDB settings, topology domains, and spare capacity?
3. Is the priority total duration, minimum user risk, or strict redundancy in each zone?
4. Is there a Cluster Autoscaler, temporary node pool, or cross-zone capacity limit?
5. How long may the controller wait on a PDB before pausing, degrading, or requesting approval?

30-second answer

I would build a declarative controller that reads nodes, Pods, PDBs, topology constraints, and schedulable capacity. It would simulate the impact of one eviction on healthy replicas and topology skew, then execute small batches through the Eviction API. Node and failure-domain tokens define concurrency; the controller waits for a replacement Pod to become Ready before continuing. PDB conflicts or unsatisfied topology constraints pause the batch with a reason instead of forcing deletion. Every operation is idempotent and observable, with timeout, rollback, and a separate risk path for involuntary failure.

Step-by-step deep dive

1. Define state and constraint inputs

The task contains target nodes, upgrade batches, maximum concurrency, deadlines, and rollback policy. Cache node labels, Pod owners, readiness, PDB disruptionsAllowed, topology constraints, and spare capacity, but reread critical state before each decision rather than trusting an old snapshot.

2. Choose candidate nodes and safe batches

Filter cordoned nodes, nodes carrying critical system Pods, and nodes without replacement capacity. Group candidates by zone, rack, and workload. A batch must stay within each workload's PDB allowance and simulate the topology skew after replacement. Prefer nodes with spare capacity that do not turn a failure domain into a single point.

3. Use the Eviction API

For voluntary maintenance, call the Eviction API and let Kubernetes evaluate the PDB. Record the specific PDB, workload, and retry time on conflict or rejection. Direct deletion bypasses the budget and should be reserved for an explicitly approved destructive emergency path.

read constraints -> choose one node -> create eviction request
                -> PDB allows? no: back off and re-evaluate
                -> yes: wait for Ready replacement and topology recovery
                -> success: mark node complete; failure: pause and recover

4. React to topology and capacity feedback

Eviction is not completion. A replacement can remain Pending because of DoNotSchedule, a missing topology key, affinity, or insufficient capacity. Watch scheduler events; expand a temporary pool or change batch order when appropriate, but do not silently relax workload constraints. ScheduleAnyway still requires recording actual skew and cost.

5. Design concurrency, leases, and idempotency

Use a task lease for a node and controller generation so multiple maintenance workers cannot evict the same target. Start with a small per-zone limit or a workload token bucket. Acquire the next token only after the replacement is Ready, the PDB budget recovers, and topology skew is within the intended bound. Treat a repeated eviction as already handled and recover after controller restarts from API state.

6. Failure, timeout, and rollback

If a Pod remains Pending, a PDB stays at zero, draining times out, or a new node is unhealthy, pause later batches and remove unnecessary cordons. When an upgraded node cannot be reverted to an old image, retain an old pool or switch to a validated image. Rollback restores service redundancy; it does not force a maintenance percentage to 100%.

7. Observability and drills

Record selected nodes, affected workloads, PDB allowance before and after, topology counts, eviction responses, Pending reasons, and duration. Measure successful evictions, PDB blocking time, maximum skew, Ready latency, cross-zone traffic, and retries. Drill single-zone capacity shortage, zero PDB budget, scheduler failure, controller restart, and sudden node loss.

High-quality sample answer

I would model the orchestrator as a declarative controller that advances by node and failure domain. It reads Pod owners, readiness, PDBs, topology spread, node labels, and schedulable capacity; it simulates one eviction's effect on healthy replicas and maxSkew, then submits a small Eviction API batch. Each zone has a concurrency token, and the next node waits for a Ready replacement, a recovered PDB budget, and satisfied topology constraints.

PDB conflicts, Pending replacements, capacity shortage, or timeout pause the batch and record a reason. The controller may expand a temporary pool or reorder candidates, but it never deletes Pods directly or silently relaxes constraints. Leases and generations make retries idempotent, and restarts recover from API state. Metrics cover budget blocking, maximum skew, Ready latency, retries, and cross-zone cost before the batch is enlarged.

Common mistakes

• Deleting Pods to drain faster → bypasses PDB → use the Eviction API for voluntary maintenance.
• Looking only at disruptionsAllowed → replacements may violate topology or lack capacity → simulate skew and placement first.
• Evicting replicas in several zones at once → loses failure-domain redundancy → use zone and workload concurrency tokens.
• Editing the PDB when it blocks → transfers risk to users → pause, add capacity, or request approval.
• Waiting only for deletion → a service may have no Ready replacement → drive state with readiness, scheduler events, and deadlines.
• Omitting controller leases → workers repeat operations → persist leases, generations, and idempotent state.

Follow-up questions and responses

Can a PDB protect against a node suddenly disappearing?

Not completely. A PDB primarily limits voluntary disruptions; hardware failure and resource exhaustion can remove replicas directly. Redundancy still requires multiple failure domains, replicas, and spare capacity.

Why not use kubectl delete pod?

Direct deletion bypasses the PDB admission check. A maintenance controller should call the Eviction API so the API server returns an allow or conflict decision.

What if the PDB allows eviction but the replacement stays Pending?

Pause the batch, inspect scheduler events and topology counts, and add capacity or choose another node. Continuing to evict would create a larger redundancy gap.

Can ScheduleAnyway be treated as ignoring skew?

No. It tells the scheduler to prefer nodes that reduce skew, but skew may remain. Record the actual distribution, cost, and redundancy risk.

How does a controller restart avoid duplicate evictions?

Use task leases, node locks, controller generations, and persisted state. On recovery, trust Pod, Eviction, and node state from the API rather than replaying an in-memory queue.

When is forced deletion acceptable?

Only for an explicitly approved emergency when the voluntary path cannot handle the risk, with elevated authorization and a record that PDB and redundancy may be violated.