Prompt and use case
A batch platform creates thousands of Pods at once, but images, quotas, topology, or external resources are not ready. Design a mechanism that creates Pods without immediately sending them to the Kubernetes scheduler, then releases them when conditions are satisfied. Explain how to avoid wasting scheduler and Cluster Autoscaler work on invalid Pending Pods, and how to handle gate timeouts, controller failure, and tenant isolation.
What the interviewer is testing
- Separating the created state from the schedulable state.
- Understanding creation, removal, and the no-addition constraint of
spec.schedulingGates. - Designing idempotent release, timeout, authorization, and recovery flows.
- Distinguishing SchedulingGated, Unschedulable, and runtime failure.
- Proving that the system cannot silently stall with metrics, events, and audit records.
Questions to clarify first
- Who adds each gate, and who confirms its condition?
- Is the condition Pod-, batch-, or tenant-scoped, and is gang scheduling required?
- What are the release latency, expiry policy, concurrency limit, and cost target?
- How should controller restarts, API retries, node scaling, and revoked permission behave?
Thirty-second answer
I would create the Pod with namespace-scoped schedulingGates and make image warming, quota, and external-resource readiness observable conditions. A controller only removes gates it owns and never adds a new gate after creation; it rechecks tenant quota and batch policy before release. Metrics distinguish gated Pods from genuinely unschedulable Pods, and expiry enters an explicit failure or human-review state with versioned conditions and audit records.
Deep-dive answer, step by step
1. Define the state machine
Separate Created, SchedulingGated, ReadyToSchedule, Unschedulable, Running, and Expired. The API can still read a Pod, but the scheduler does not attempt a Pod whose gate list is non-empty.
2. Choose gate names
Each gate is a string condition, such as batch.example.com/image-ready. Put namespace, batch, and controller versions in labels or annotations; keep the gate name as a statement of an unmet condition rather than a container for dynamic data.
3. Set gates at creation
A gate can be initialized at Pod creation by the client or an admission mutator. Existing gates may be removed in any order after creation, but a new gate cannot be added, so every possible blocking condition must be enumerated at creation time.
4. Design the release controller
The controller watches Pods, quota, image warming, and external-resource events, computes the condition set, and applies a resource-versioned patch. Duplicate events must converge to the same result; each controller removes only its own gates so controllers cannot overwrite one another.
5. Handle batches and concurrency
Record batch-level resources in a separate object, while a Pod gate waits for the batch controller's decision. Remove gates in tenant-, priority-, and concurrency-aware batches so the scheduler and autoscaler do not receive a sudden pressure spike.
6. Design expiry and human intervention
Persist creation time, last condition progress, and the deadline. On expiry, do not silently remove a gate; record a reason, emit an event, and choose cancellation, retry, or a human queue. Retries need backoff and a maximum count.
7. Observe the real queues
Kubernetes exposes the gated label on schedulerpendingpods to distinguish explicitly not-ready Pods from Pods that were tried and found unschedulable. Combine it with Pod conditions, events, controller queue length, gate-age percentiles, and autoscaler activity to locate the bottleneck.
8. Constrain authorization and recover
Admission policy limits who may create gates, and the controller may remove only gates with an allowed namespace and prefix. After a restart, rebuild state from the API; on a patch conflict, reread and compare the resource version. If the controller remains unavailable, on-call staff need a safe cancel or release procedure backed by the audit trail.
Trade-offs and boundaries
Scheduling gates control whether a Pod enters scheduling; they do not replace node affinity, resource requests, topology constraints, or runtime readiness. Too many gates hide real capacity problems in an application layer; too few send unprepared Pods to the scheduler. Keep unmet conditions distinct from a cluster with no capacity, and do not treat a gate as a general cross-object transaction lock.
Rollout plan and evidence
- Record each gate's owner, condition source, expiry, and removal permission.
- Validate gate prefixes, tenant quota, and the complete creation-time condition set in admission.
- Load-test the scheduler, autoscaler, API server, and controller patch conflicts with small batches first.
- Monitor
schedulerpendingpods{queue="gated"}, gate age, expiry rate, release throughput, and per-tenant concurrency. - Exercise controller restart, network partition, permission revocation, batch cancellation, and duplicate patches, then verify the audit log.
Common mistakes and follow-ups
Mistake 1: Adding a gate after creation
The API allows gates only at creation and allows removal afterwards. Dynamic conditions must be aggregated before creation or waited on through a separate object before creating the Pod.
Mistake 2: Calling SchedulingGated a scheduling failure
A Pod with non-empty gates has not been tried by the scheduler. Distinguish gated, Unschedulable, ImagePullBackOff, and application readiness.
Mistake 3: Using gate removal as quota control
Removing a gate only permits a scheduling attempt; it does not guarantee resources. Recheck quota, priority, and batch concurrency before release.
Mistake 4: Omitting gate-age and expiry alerts
Without age percentiles, a silent stall is invisible. Record the condition, responsible controller, last progress, and an explicit cancel path.
Mistake 5: Ignoring autoscaler cost
A large set of unprepared Pods can trigger pointless scale-up evaluation. Use gated-queue metrics and release throttling to verify that cost actually falls.