System Design Interview: Explainable Device Preferences and Fallbacks with Kubernetes DRA

Prompt and context

You own a Kubernetes cluster for training and inference jobs. Nodes contain H100s, A100s, and smaller accelerators. A workload wants H100 first, then A100, then another compatible device when capacity is unavailable. Design the Dynamic Resource Allocation (DRA) request and scheduling flow, covering determinism, fairness, failures, observability, migration, and rollback. Answer as a senior engineer who can make architectural trade-offs.

What the interviewer tests

• Whether preference is a verifiable policy rather than client-side polling or hard-coded claiming.
• Understanding of the ResourceClaim, ResourceSlice, driver, and scheduler lifecycles.
• Deterministic tie-breaking, health changes, retries, and timeouts.
• Fair capacity allocation, tenant isolation, metrics, and auditability.
• A reversible migration path from device plugins to DRA.

Questions to clarify

1. Is any available fallback acceptable, or must memory, architecture, and driver capability remain hard constraints?
2. Can one Pod accept different device models? Are topology, NUMA, and network bandwidth hard constraints?
3. Does the business require a strict model guarantee, or can it trade model preference for success rate and cost?
4. Do tenants have quotas, priorities, and preemption rules? When is an unhealthy device removed from candidates?
5. Can existing workloads adopt ResourceClaims, and must old device plugins coexist during migration?

30-second answer

I would encode device preference as ordered candidates plus hard constraints in a DRA request, letting the scheduler match resources during the claim lifecycle instead of having clients poll nodes. The scheduler filters driver, architecture, topology, and tenant-quota constraints first, then evaluates H100, A100, and other compatible devices in order. Each tier uses a stable tie-breaker, and events and metrics record candidates, rejection reasons, and the final choice. Health changes or binding failures trigger reevaluation only while the claim is retryable. Queue quotas and tenant weights provide fairness. Migration uses dual-track canaries, versioned ResourceClaim templates, and a switch back to the old plugin.

Step-by-step deep dive

1. Define the preference model

Treat model order as a soft preference. Treat driver version, architecture, memory floor, topology, and isolation as hard constraints. The request may say “H100 first, A100 second,” but fallback must not bypass memory or tenant quota. Include a versioned policy identifier for audit and rollback.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
spec:
  spec:
    devices:
      requests:
      - name: accelerator
        exactly:
          deviceClassName: gpu
          selectors:
          - cel: "device.attributes['model'] == 'H100'"
          - cel: "device.attributes['model'] == 'A100'"

The ordered candidates express business preference; deployment must still match the DRA API version and driver capabilities supported by the cluster.

2. Resource claim and scheduling flow

The Pod uses a ResourceClaimTemplate to create a claim. A DRA driver publishes ResourceSlices containing device attributes, capacity, and health. During pre-filtering, the scheduler reads claims and slices, verifies hard constraints, and searches candidates in order. After binding, the driver exposes the allocation to the container. Claims must be idempotent so retries cannot create a second allocation that cannot be reclaimed.

3. Deterministic choice and explanation

When a tier has multiple devices, use a stable order based on resource pool, ResourceSlice name, and device identifier, or an explicit capacity and topology ranking. Make the rule part of the interface contract. Record candidates, filters, rejection reasons, final device, and policy version. Replaying the same inputs should produce the same result, and an operator can explain why an H100 was skipped.

4. Failure, health, and retry behavior

New claims should skip devices marked unhealthy; already-bound workloads follow the runtime and controller policy for termination, migration, or recreation. Distinguish binding conflicts, stale slices, and node loss as retryable or terminal. Retries need backoff and an idempotency key to avoid repeated scans amplifying scheduler load. Falling back to A100 is valid only when hard constraints still hold, and the actual model must be visible in Pod status.

5. Fairness, capacity, and observability

Preference must not become a pass that reserves H100s indefinitely. Queue-level arbitration uses tenant quota, weight, and wait time; device-level choice follows candidate order. Track request volume by model, fallback rate, wait time, binding failures, health changes, tenant usage, and policy hit rate. Keep a readable decision chain in events while avoiding sensitive tenant data in logs.

6. Migration and rollback

Start with DRA classes and ResourceClaim templates for a small workload slice while old device plugins serve the rest. Compare success rate, fallback rate, scheduling latency, and GPU utilization before expanding. Version templates and policies. If a driver or scheduling issue appears, stop publishing the new template and switch workloads back to the old plugin; handle already-bound jobs under the stated termination policy instead of changing claims and device allocation simultaneously.

Model answer

I would separate preference, constraints, and allocation proof. Preference is an ordered candidate list. Constraints cover model capability, memory, driver, topology, tenant quota, and isolation. After the Pod creates a ResourceClaim, the DRA driver publishes ResourceSlices and the scheduler filters hard constraints before selecting candidates in order. A stable resource-pool and device-identifier order resolves ties, making replay deterministic. Events include candidates, rejection reasons, policy version, and the actual model.

Failure paths distinguish unhealthy devices, binding conflicts, stale claims, and node loss. Only retryable errors use backoff; fallback never crosses a hard constraint, and the final model is written to status. Tenant quota, queue weight, and wait time protect fairness so H100 preference does not starve other tenants. Migration runs old plugins and DRA in a canary, with versioned templates and a tested rollback switch.

Common mistakes

• Saying only “sort by model” without hard constraints or an in-tier tie-breaker.
• Letting clients scan nodes or claim devices directly, bypassing claims and the scheduler.
• Treating health changes, binding conflicts, and unsatisfiable constraints as infinite retries.
• Optimizing only H100 hit rate while ignoring tenant fairness, quota, and wait time.
• Removing old device plugins in one step, leaving no fast rollback.
• Recording only the final device and losing candidate and rejection evidence.

Follow-up questions and responses

If both H100 and A100 satisfy constraints, why not choose randomly?

Random choice weakens replayability and diagnosis. Capacity balancing can follow a stable order, but the rule must remain explainable and observable; a random seed should not become a hidden contract.

What if a device becomes unhealthy after pre-filtering but before binding?

Recheck version and health at binding. Return a classified error and retry with backoff only when the claim remains valid and candidates exist; otherwise expose a clear unsatisfiable reason on the Pod.

How do you prove fallback did not harm fairness?

Measure wait time, usage, fallback rate, and queue share by tenant and model, then compare replay and canary cohorts. If a tenant rarely gets its first choice, adjust quota or weight instead of continuously raising request priority.