System design interview: How would you use Kubernetes device health to govern GPU failures?

Prompt and context

A multi-tenant GPU inference platform uses Kubernetes Device Plugins and Dynamic Resource Allocation (DRA). When a driver detects a disconnected card, the old system exposes the error only in node logs; the Pod restarts and repeatedly claims the same device. Design failure governance around allocatedResourcesStatus in Pod .status: operators and controllers should see device health, quarantine bad cards, handle Unknown safely, and avoid mass deletion from a transient signal.

Kubernetes v1.36 promotes resource health status to Beta. The official release notes say that Pod status reports the health of allocated devices and that kubectl describe pod can expose Unhealthy or Unknown; the mechanism covers both traditional Device Plugins and DRA paths.

What the interviewer is testing

• Can you distinguish allocation success, device health, container readiness, and business SLOs?
• Can you draw the data flow among the driver, kubelet, Pod status, controllers, scheduler, and alerting?
• Can you treat Unhealthy and Unknown differently instead of turning missing observation into a hard failure?
• Can you design idempotent quarantine, leases, retries, rate limits, and re-entry after recovery?
• Can you explain that status is a diagnostic signal, not an automatic replacement for scheduling policy or business probes?

Questions to clarify first

• Which drivers produce health, and what are their update delays and heartbeat periods?
• A Pod may hold several cards; if one fails, can the job degrade partially or must it migrate as a unit?
• Does Unknown mean a transient disconnect, a node outage, or unsupported driver behavior? What is the tolerance window?
• Is quarantine per device, node, ResourceClaim, or tenant workload, and who can release it?
• Do jobs have checkpoints, idempotent commits, and a retry budget? How much GPU churn can migration cause?

30-second answer

"I would treat device health as state input, not as a delete-Pod command. After the driver and kubelet update allocatedResourcesStatus, a controller aggregates by device ID: Unhealthy enters quarantine and alerting, while Unknown first gets a heartbeat tolerance window. Idempotency keys and leases block new allocations, and checkpointed jobs migrate with per-node rate limits. Recovery runs a probe and a small canary before release. I would validate state age, false quarantines, migration success, and business SLOs."

Step-by-step deep dive

1. Define the state model. Record device identity, Pod, container, ResourceClaim, node, health value, message, observation time, and status version. Keep Unhealthy (confirmed failure) separate from Unknown (unconfirmed) so one Boolean does not drive every automation.

1. Build the data flow. DRA or a Device Plugin allocates a device to a Pod; kubelet writes driver-reported health into Pod status. A status observer watches Pod changes, deduplicates by device key, and writes a replayable device directory. Alerting and controllers read that directory instead of independently scanning the API.

1. Quarantine new allocations. For a confirmed unhealthy device, create an internal quarantine record and exclude it from scheduler extensions, ResourceClaim selection, or node-capacity views. Do not edit Pod status or immediately turn a one-off event into Node NotReady. The quarantine action needs a reason, actor, and expiry.

1. Handle running jobs. The controller checks checkpoints and idempotent commits before acquiring a migration lease. A recoverable job stops new requests, saves a checkpoint, releases the device, and rebuilds on a healthy device. An unrecoverable job preserves evidence and notifies the tenant. Only one migration flow may own a device and job.

1. Guard Unknown. Unknown may come from kubelet, the driver, or a node network interruption. Apply a heartbeat-based tolerance window and exponential backoff; alert during the window and restrict allocation only after it expires. For a whole-node outage, use node leases and existing failure detection rather than inferring that every device is broken from one Pod status.

1. Recover and verify. When a device reports healthy again, run a driver probe, a small job, and a stability window before releasing quarantine. Track status age, Unknown duration, false quarantine rate, migration success, idle GPU time, and business errors. Replay duplicate and out-of-order events and restart the controller to test recovery.

Device health event
  -> Pod status observer
  -> deduplicate by (node, deviceID, statusVersion)
  -> device quarantine record with lease and expiry
  -> scheduler/claim filter excludes unhealthy device
  -> checkpointed workload migration
  -> probe + canary
  -> release quarantine

Model answer

I would create a device-level status directory linking the driver, kubelet, Pod, ResourceClaim, and job. Unhealthy is a confirmed failure, so I create a lease-backed quarantine with an expiry, block new allocations, and alert. Unknown first enters a heartbeat tolerance window and only restricts allocation after the window expires. The observer deduplicates by device ID, and a restarted controller rebuilds from persisted events and records rather than in-memory flags.

Migration depends on checkpoints, idempotent commits, and tenant priority. Recoverable jobs checkpoint, release the device, and rebuild on a healthy one; unrecoverable jobs preserve evidence. Recovery starts with a driver probe and a small canary. Scheduler extensions, DRA selection, and capacity views consume the quarantine record, while status remains a diagnostic input rather than readiness or a business SLO. I would prove the design with state age, false positives, migration success, and business error metrics.

Common mistakes

• Symptom: Delete every Pod when status is Unknown → Why it fails: A transient observation gap becomes cluster churn → Fix: Use a lease, heartbeat window, and rate limits.
• Symptom: Quarantine only the node and omit device identity → Why it fails: One bad card takes healthy devices with it → Fix: Key records by device and ResourceClaim, escalating to the node only when justified.
• Symptom: Edit Pod status to make a device look healthy → Why it fails: The source of truth is corrupted and recovery may be wrong → Fix: Preserve kubelet status and create an auditable quarantine state separately.
• Symptom: Return a recovered device to full load immediately → Why it fails: A transient recovery or unstable driver can fail again → Fix: Probe, canary, then ramp gradually.

Follow-up questions and responses

One GPU failed but the Pod owns others. Should only part of the job migrate?

First verify that the framework supports dynamic shrink and rebinding. If the model requires fixed device topology, migrate the whole job. If it can shard, move only the affected shard while preserving ResourceClaim and checkpoint semantics for the remaining devices.

The value is Healthy but the status message is old. What do you do?

Separate health value from status age. After the heartbeat limit, transition to Unknown rather than treating stale Healthy as current proof; alerting and scheduling guardrails use the age.

How do you stop multiple controllers from migrating the same job?

Use an idempotency key built from device ID, job ID, and status version, with an expiring lease or optimistic version update. A controller that loses the lease stops, and a replacement resumes from the persisted record.

When can the device return to the capacity pool?

A healthy driver report is necessary but insufficient. Require a device probe, a small job, a stability window, and a closed quarantine reason; any failure extends quarantine and blocks manual force-release.

References

• Kubernetes v1.36: Haru
• Kubernetes v1.36: More Drivers, New Features, and the Next Era of DRA
• Kubernetes v1.34: Pods Report DRA Resource Health
• Dynamic Resource Allocation documentation

Interview checklist

Draw the driver-to-Pod-status, observer, quarantine, and scheduling-filter flow. Separate Unhealthy from Unknown, then add leases, checkpoints, rate limits, a recovery canary, and metrics.

One-sentence takeaway

Device health makes hardware failures observable in Kubernetes, but safe recovery still requires device-level quarantine, Unknown guardrails, idempotent migration, and staged reuse.