System design interview: How would you use Kubernetes PodGroup for gang scheduling?

Prompt

Kubernetes v1.35 documents Scheduling Group as an alpha capability. Design a batch platform where interdependent workers start only when at least minCount Pods can be placed together. Explain how PodGroup, the scheduler, controllers, autoscaling, and failure handling cooperate. State the version and feature-gate assumptions.

What the interviewer is testing

The core test is whether you can lift feasibility from one Pod to a group while keeping API, scheduling, and runtime state consistent. A strong answer distinguishes basic from gang, handles a missing PodGroup, capacity shortage, member failure, preemption, and observability, and avoids presenting an alpha API as universally production-ready.

Clarifying questions

1. Must every worker start together, or is a lower concurrent threshold enough?
2. Are members one-shot Jobs or long-running services?
3. Do they need GPUs, topology constraints, or cross-cluster placement? The documented reference is a PodGroup in the same namespace.
4. What is the waiting deadline, and may a job queue, borrow capacity, or downgrade?

A 30-second framework

Start with the boundary: Scheduling Group and PodGroup policies are alpha in v1.35, disabled by default, and require the GenericWorkload feature gate. Then cover four layers: API contract, group scheduling, lifecycle and operations. A Workload creates the PodGroup; Pods reference it; the scheduler applies the policy and minCount; controllers own timeout, retry, and cleanup; metrics and rollback protect rollout.

Step-by-step design

1. Model and invariants

The controller creates one PodGroup per run with desired members, policy, version, and tenant. Each Pod sets spec.schedulingGroup.podGroupName to a PodGroup in the same namespace. The field is immutable, so moving a Pod means creating a new set. For a gang, no member binds until the minCount invariant is met.

2. Choose the policy

Use basic when members can run independently and grouping is mainly for management and observability. Use gang for tightly coupled training or batch work; the group is feasible only when at least minCount members can be scheduled simultaneously. Setting minCount below the total allows elasticity; setting it equal to the total requires all members.

3. Submission and waiting state machine

Create the PodGroup before creating Pods so references resolve. If the referenced object is absent, Pods remain Pending and the scheduler retries after the group appears. Track states such as PendingGroup, WaitingCapacity, Feasible, Bound, Running, Failed, and Cancelled, each with a generation, reason, and timestamp.

4. Scheduling and capacity coordination

The scheduler first evaluates candidate nodes using filters, topology, devices, and priority, then checks group feasibility. Bind operations must be retryable and idempotent, and occur only after the candidate count reaches minCount. The autoscaler should consume the group resource shape and scale for the whole request instead of adding capacity for the first Pod only.

5. Preemption, deadlines, and fairness

Give a group consistent priority and queue weight so preemption does not leave an unusable half-group. On a deadline, cancel the group and release reservations; retries use a new generation so stale members cannot rejoin. Tenant quotas, maximum group size, and queue aging prevent large gangs from monopolizing a cluster.

6. Member failure and rollback

After startup, a crashed member must not be treated as evidence that the group is currently healthy. Depending on job semantics, restart one member or terminate the group; training jobs commonly restore a checkpoint and create a new generation. Deleting a Workload should clean up owned Pods and the PodGroup while preserving terminal events for audit.

7. Observability and safety boundaries

Expose group queue latency, feasible-member count, minCount, scale-up and preemption counts, and failure reasons. An admission webhook validates namespace, quota, group size, and feature-gate availability. Because the API is alpha, use an explicit rollout switch, compatibility tests, and a fast rollback path.

Example of a strong answer

“I would have a Workload controller create a PodGroup in the same namespace with the gang policy and a minCount equal to the minimum workers needed to make progress. Pods reference it through the immutable schedulingGroup field. The controller creates the group first; unresolved references stay Pending. The scheduler evaluates resources, topology, and priority for all members and binds only when the feasible count reaches minCount. The autoscaler scales from the group resource shape. A deadline cancels the whole group and releases capacity; a failed training run gets a new generation restored from checkpoint. The deployment manifest explicitly records v1.35 alpha and GenericWorkload, and rollout starts in an isolated cluster.”

Common failure modes

• Calling basic all-or-nothing even though its members can schedule independently.
• Using only labels without explaining the same-namespace PodGroup reference and immutable field.
• Ignoring a missing group, scale-up, deadlines, preemption, or retry generations.
• Omitting the alpha status and GenericWorkload feature gate.
• Discussing a successful bind without group-level cancellation and metrics.

Follow-up directions

What if the PodGroup is created after the Pod?

The Pod remains Pending, and the scheduler reconsiders it after the PodGroup is created. The controller should still create the group first to shorten the window.

How do you choose minCount?

Use the application’s minimum useful parallelism, per-Pod resources, and acceptable queue time, then enforce quotas and an upper bound.

How do you avoid gang starvation?

Combine fair queues, aging, group-size limits, tenant quotas, and deadline cancellation while monitoring group wait time.

How is this different from scheduling gates?

A gate controls when one Pod enters the schedulable queue. Scheduling Group makes the scheduler evaluate a set through a PodGroup. They can be combined, but their metrics and failure semantics should stay separate.

When would you delay rollout?

Delay when the cluster version, feature gate, scheduler plugins, or autoscaler are incompatible, or when alpha upgrade and rollback have not been tested. Use an explicit queue controller as an interim design.

References

• Kubernetes documentation: “Scheduling Group”.
• Kubernetes documentation: “PodGroup Scheduling Policies”.
• Kubernetes documentation: “Scheduling API Reference”.