System design interview: How would you roll out Kubernetes Node Declared Features safely?

Prompt and context

Your cluster is adopting Kubernetes Node Declared Features: kubelets report managed node capabilities in Node status, the scheduler filters Pods accordingly, and an admission controller validates Pod updates. Design rollout, observability, failure handling, and rollback for a mixed-version cluster. Assume only controlled node features are used; application teams cannot write arbitrary capability names.

What the interviewer evaluates

• Whether capability declarations are treated as node-state facts rather than user-controlled labels.
• Whether the dependency order among kubelet, kube-apiserver, kube-scheduler, and admission control is explicit.
• Whether old nodes, stale status, Pod updates, and feature-gate rollback are covered.
• Whether safety is demonstrated with declaration completeness and scheduling rejection, not just process startup.

Clarifying questions

• Which kubelet version produces the capability, and may old nodes keep serving ordinary Pods?
• Is the goal to protect placement only, or also updates after a Pod is bound?
• Are there custom schedulers, multiple API servers, or a multi-region control plane?
• On a false report, should new Pods stop while existing Pods continue running?

30-second answer

I would treat this as a cross-component fact chain: kubelets publish status.declaredFeatures, the scheduler plugin infers Pod requirements in PreFilter and filters nodes, and admission validation protects later updates. I would enable it first on a reversible node pool and a small workload slice, verify matching feature-gate configuration on the API server, scheduler, and kubelets, and then expand. I would monitor report completeness, scheduling rejection reasons, update rejections, and version skew. Any inconsistency stops workloads that require the new capability and preserves the ordinary scheduling path for other Pods.

Step-by-step solution

1. Define the source of truth. At startup, the kubelet detects managed features and writes them to Node.status.declaredFeatures; application labels are not equivalent. Capability names must come from managed feature gates or an explicit component contract.
2. Make dependencies explicit. Kubernetes requires the NodeDeclaredFeatures gate on kube-apiserver, kube-scheduler, and kubelet. Verify control-plane versions and configuration before upgrading kubelets; enabling one side alone creates a field nobody consumes or a scheduler that expects reports nodes cannot produce.
3. Design the scheduling path. The scheduler plugin infers required features from the PodSpec in PreFilter and compares them with the node declaration in Filter. A node without a required declaration is unschedulable for that Pod. A custom scheduler using the field must preserve the same default and failure semantics.
4. Protect the update path. The NodeDeclaredFeatureValidator admission controller checks Pod updates against the bound node, preventing a later update from bypassing the capability constraint. Surface a clear rejection instead of silently degrading.
5. Handle version skew. An old kubelet may omit the field or not know a new capability. Treat an undeclared feature as unsupported, leaving dependent Pods pending while ordinary Pods use compatible nodes. Do not hand-edit Node status to bypass the skew.
6. Roll out in stages. Enable the gate on a reversible node pool, place a probe workload requiring the capability, and expand gradually. Stop when report-missing rate, scheduling rejection, Pod-update rejection, or scheduler latency exceeds its baseline.
7. Observe and audit. Collect node declaration version, a digest of capability sets, scheduler filter reasons, admission rejection reasons, and gate configuration fingerprints, sliced by pool and Kubernetes version. Avoid writing full Node objects into high-cardinality logs.
8. Rollback deliberately. Stop creating Pods that depend on the feature, restore ordinary scheduling, and close the gate in component order after pending dependent work is handled. If the business already depends on the capability, migrate workloads or retain compatible nodes before making declarations disappear.

Model answer

I would first identify the source of truth and the protection boundary. Kubelets publish managed status.declaredFeatures, the scheduler’s NodeDeclaredFeatures plugin filters placement, and NodeDeclaredFeatureValidator protects updates after binding. Because Kubernetes requires the gate on kube-apiserver, scheduler, and kubelet, I would align versions and configuration before enabling it on one reversible pool. A probe Pod verifies capability reporting and placement; report-missing rate, filter rejection, update rejection, and scheduling latency become expansion gates. An old node that does not declare the feature is unsupported, while ordinary Pods keep the old path. For rollback I would stop new dependent workloads, migrate existing ones, close the gate, and confirm pending counts and ordinary scheduling recover.

Common mistakes

• Treating declaredFeatures as an ordinary label → users can forge capability and defeat scheduling safety → accept only the kubelet-managed set.
• Enabling the gate only in the scheduler → nodes do not publish the field → check API server, scheduler, and kubelet configuration together.
• Treating “undeclared” as supported → mixed-version nodes may receive incompatible Pods → undeclared means unsupported.
• Testing only initial placement → a Pod update can bypass the constraint → enable and observe admission validation too.
• Turning it on cluster-wide → failures cannot be attributed to a pool, version, or workload → use probes, stages, and stop conditions.
• Disabling the gate immediately → dependent Pods can become unrecoverable → stop creation and migrate dependent workloads first.

Follow-ups and responses

A node reported a capability but its status is stale. What should the scheduler do?

Make report age an operational signal. A Pod requiring the feature should wait or move to a fresh node rather than rely on an old declaration. After the freshness deadline, isolate the pool and require operator confirmation.

What can go wrong with a custom scheduler that ignores the official plugin?

It may allow a Pod that the default scheduler rejects. The custom path must implement equivalent PreFilter, Filter, default, and version semantics; run conformance tests before allowing workloads to select it.

How do you avoid a configuration window while three components change the gate?

Put a configuration fingerprint in the rollout check, keep ordinary workloads serving, and roll through the control plane, scheduler, and node pools in a controlled order. Any mismatch stops expansion; process startup alone is not success.

The business depends on the feature, but rollback finds old nodes do not support it. What now?

Keep a pool of nodes that declare the capability, migrate or shrink dependent workloads, and only then close the gate. If migration is impossible, pause rollback and add compatible capacity instead of suddenly removing the scheduling constraint.