System Design Interview: How Would You Design a Kubernetes Node Readiness Controller?

Prompt and scope

After a node boots, kubelet may report Ready even though a GPU driver, CNI, storage plugin, or local agent is not usable. If the scheduler places Pods too early, workloads fail repeatedly or degrade silently. Design a Node Readiness Controller that declares extra prerequisites, blocks scheduling until they hold, and handles a dependency failing later in the node lifecycle.

This fits platform engineering, SRE, and Kubernetes controller interviews. Public Kubernetes interview material covers NotReady nodes, taints and tolerations, and Pending-Pod troubleshooting. The official Node Readiness Controller project describes a NodeReadinessRule API, automated taint management, bootstrap-only and continuous modes, and dry run. The design below is a reasoned interview answer, not a claimed company question.

What the interviewer evaluates

The interviewer wants you to separate “the node is Ready” from “the node is suitable for this workload class.” A strong answer defines condition sources, rule scope, taint idempotency, restart recovery, observable state, and protection against accidental fleet-wide blocking.

A weak answer writes a DaemonSet that checks everything. A strong answer explains why condition reporting is decoupled from policy enforcement, how bootstrap-only differs from continuous enforcement, how dry run estimates blast radius, and how to handle stale conditions, controller partitions, and conflicting rules.

Clarifying questions before answering

• Should the controller protect only node bootstrap, or also stop new scheduling when a driver fails later? This determines the enforcement mode and recovery action.
• Who produces conditions: Node Problem Detector, a device plugin, a CNI agent, or a custom DaemonSet? Untrusted sources need identity and freshness checks.
• Must all conditions be true, or may any one condition pass? GPU, network, and storage usually need different node pools and rules.
• Does a false condition block new Pods or evict existing Pods? The former is reversible; the latter needs stronger evidence and a separate eviction policy.

30-second answer framework

“I would separate condition reporting, rule evaluation, and taint execution. A NodeReadinessRule selects nodes and lists conditions that must be True; the controller adds or removes a NoSchedule taint according to bootstrap-only or continuous enforcement. Writes must be idempotent, observable, and dry-run capable. Rules and condition status live in the API server so state can be rebuilt after a restart. I would log impact first, enable one node pool, and on failure stop writes or roll back the rule instead of making the entire cluster unschedulable.”

Step-by-step answer

Start with the boundary. The controller does not probe GPUs or networks; it consumes Node Conditions. Node Problem Detector, a device plugin, or a custom agent reports facts. This reuses the existing probe ecosystem and separates “the check failed” from “scheduling is allowed.” A condition should carry type, status, update time, source, and an observation generation; a stale condition cannot keep authorizing workloads.

The rule model contains a node selector, condition set, target taint, and enforcement mode. The official project requires every listed condition to be satisfied before removing the taint and supports selecting heterogeneous nodes by labels. bootstrap-only stops evaluating that rule after initialization succeeds. continuous re-adds the taint when a critical condition becomes False later. The first mode fits image pre-pulls or hardware setup; the second fits dependencies that must remain healthy.

The control loop reads rules, nodes, and conditions, computes desired taints, and applies updates using resource versions. Writes must be idempotent and retried on conflict: manage only taints owned by this controller, never remove an administrator’s or another controller’s taint. If two rules claim the same taint key with different meaning, admission validation should reject the conflict; otherwise one rule can accidentally remove another rule’s protection.

Make failure paths explicit. When a reporter stops updating, fail-closed or fail-open is a product choice: a critical security dependency can fail closed, while low-risk bootstrap can fail open with a short TTL. A disconnected controller must not clear existing taints; it reconciles after recovery. If a node is deleted or its labels change, mark the rule as not applicable so stale state cannot block a replacement node.

Observability should include matched nodes per rule, missing or stale conditions, desired-versus-actual taint drift, latency from a condition becoming True to schedulability, and gated-Pod count. Status should expose the failed condition and last evaluation time without logging credentials or sensitive device data. To protect the control plane, queue by node and rule, coalesce rapid condition changes, and avoid an API-server write for every heartbeat.

Roll out with dry run. The official project’s dry run records intended actions and updates rule status without applying taints. Observe affected nodes and Pending Pods, then enable one node pool. Rollback pauses new rule evaluation, preserves existing protection taints, fixes the condition source, and reconciles again; it does not delete every NoSchedule taint with one command.

Alternatives include adding nodeSelector to every workload, using Pod schedulingGates, or adding taints from a bootstrap script. Workload selectors fit a small set of known consumers but miss future Pods. A Pod gate protects a Pod, while this problem protects a node. Scripts lack shared state and recovery. A controller fits heterogeneous nodes and shared clusters, at the cost of a CRD, a control loop, and another failure domain.

High-quality sample answer

I would build a declarative controller responsible only for “may new Pods be scheduled here?” Node Problem Detector, device plugins, or CNI agents write conditions; the controller does not duplicate their probes. A rule selects nodes, lists conditions that must be True, names a NoSchedule taint, and chooses a mode. Bootstrap work uses bootstrap-only: once the GPU driver and network agent are ready, the taint is removed. Continuous dependencies use continuous: a later failure adds the taint again, but does not evict existing Pods automatically.

The controller owns and reconciles only taints carrying its owner marker, using resource versions and conflict retries for idempotency. Conflicting claims for the same taint key are rejected. Rule status reports matched nodes, missing or stale conditions, taint drift, and evaluation time. I would deploy dry run first, compare projected impact with Pending Pods, and canary one node pool. During controller loss, keep existing protection; after recovery, reconcile. Rollback pauses evaluation and repairs the condition source rather than clearing every cluster taint.

Common mistakes

• Mistake → Make the controller SSH into every node; failure → it bypasses Kubernetes condition sources and widens the permission boundary; fix → let specialized agents report conditions and keep policy evaluation in the controller.
• Mistake → Use continuous for one-time bootstrap; failure → completed nodes keep flapping and remain unschedulable; fix → use bootstrap-only and record completion.
• Mistake → Delete any taint with the same key; failure → an administrator or another controller’s safety protection can disappear; fix → use ownership markers, conflict validation, and field-level updates.
• Mistake → Clear all taints after a controller restart; failure → workloads land on unready nodes during the recovery window; fix → preserve state and reconcile with resource-version checks.

Follow-up questions and responses

What if the condition reporter stops updating: fail-open or fail-closed?

Classify the dependency. GPU drivers, crypto modules, and cross-zone networking can use fail-closed with an expiry time, accepting temporary capacity loss. Low-risk one-time bootstrap can fail open with a successful record and short TTL. In both cases, measure “unknown” separately from False so missing telemetry is not mistaken for health.

Two rules match a node and claim the same taint key. What do you do?

Reject the semantic conflict during admission or rule compilation, requiring distinct keys or one explicit aggregate owner. The controller keeps desired state for each rule and removes an aggregate taint only when every owner satisfies its release condition. The last rule to reconcile must not delete it alone.

How do you prove dry run will not exhaust cluster capacity?

Put matched nodes, available schedulable capacity, Pod requests, and topology spread into one impact report. Simulate one node pool, observe gated Pods, autoscaler queues, and scheduling latency, then expand. If critical workloads lack headroom, change the node pool or rule before enforcement.

Why not evict existing Pods when a node fails continuous readiness?

“No new Pods” and “existing Pods are unsafe” are separate decisions. NoSchedule preserves continuity while blocking new placement; eviction needs its own PDB, graceful termination, and data-safety policy. A separate eviction controller should act only when evidence says existing workloads are unsafe.