Prompt and context
A Kubernetes controller reads objects from an Informer cache and writes desired state to the API server. Under load, watch delay, or restart, the cache can lag behind the API server; the controller may repeat writes, scale incorrectly, or treat an old Lease as expired. Design staleness detection and mitigation that blocks only affected objects.
This is relevant to platform engineering, SRE, and cloud-native controller roles. Kubernetes v1.36 and KEP 5647 describe AtomicFIFO, LastStoreSyncResourceVersion(), tracking the resource version of writes, skipping and re-queuing stale keys, and onboarding DaemonSet, StatefulSet, ReplicaSet, and Job controllers. This is a public-source design exercise, not a claim about any company’s interview bank.
What the interviewer evaluates
The interviewer wants to see whether you understand the boundary of an eventually consistent cache and can turn “fresh enough” into a testable resource-version condition. A strong answer covers cache sync, read-after-write, per-key queues, backoff, restarts, monitoring, and feature gates. A weak answer makes every reconcile call the API server directly and ignores load and consistency cost.
Clarifying questions
- Which decisions are destructive: deleting Pods, scaling down, leader changes, or ordinary status updates?
- What stale window is acceptable, and can a critical operation afford one API-server read?
- Do we need read-after-write for one object or causal ordering across several objects?
- After restart, watch reconnect, or API-server failure, should the controller wait conservatively or allow bounded degradation?
A 30-second answer
“I would keep the Informer cache as the default read path and use AtomicFIFO plus the latest observed resource version to measure progress. After writing a critical object, the controller records its target version; until the cache catches up, it skips only that key and re-queues with exponential backoff. Destructive actions add a bounded live read or circuit breaker. Metrics expose cache lag, skipped reconciles, and queue age. On restart, keep protection in place, complete cache sync, then resume.”
Step-by-step solution
Define staleness first. An Informer Store is populated by watch events that can be delayed, reordered, or temporarily incomplete while a cache rebuilds. Kubernetes v1.36’s AtomicFIFO makes the initial list batch atomic relative to incremental events, avoiding an inconsistent cache caused by interleaving them. LastStoreSyncResourceVersion() exposes the latest version observed by the Store.
For each critical write, keep objectKey -> resourceVersion. When a DaemonSet updates a Pod, record the version returned by the API server. Each Pod informer event advances the highest observed version. The DaemonSet can run the next reconcile that depends on new state only when the observed version reaches the last write. Otherwise re-queue that key with the normal exponential backoff; never stop the whole worker pool.
Writes and checks must be idempotent. Use resource-version conflict retries, and ensure a repeated reconcile has no extra side effect. A restart loses in-memory mappings, so startup waits for informer cache sync before rebuilding protected state from object status or queue events. If the mapping is incomplete, delay destructive work rather than assuming freshness.
For time-sensitive decisions, add a circuit breaker. Use the cache for a fast path; when the target version is unknown or lag exceeds a threshold, perform one bounded live read from the API server. If it fails, pause only that key and record the reason. Making live reads the default would amplify API-server QPS, latency, and failure radius.
Expose informer resource version, target write version, lag duration, skipped reconcile count, queue age, live-read success rate, and circuit-breaker count per controller. Logs include the object key, versions, and action, never Secret values. Alerts distinguish API-server slowness, broken watches, a hot key, and slow controller processing.
Release behind a feature gate and a canary. Enable it for one high-contention controller and a small node pool, then verify convergence after skips, restart recovery, and absence of queue starvation. Expand only after the API-server load is acceptable. Roll back by disabling the new consistency path while retaining metrics and object state; do not bulk-delete protection mappings.
Model answer
I would implement four layers: cache-first reads, a resource-version gate, per-key requeue, and a circuit breaker for critical actions. AtomicFIFO keeps initial-list processing consistent with incremental events, and the Store reports its latest observed resource version. After a write, the controller records the target version; it processes that key only after the cache reaches it, otherwise it re-queues with backoff.
Deletes, scale-downs, and Lease decisions use a bounded live read when cache freshness is unknown or above the threshold; failure pauses that key. Metrics cover version lag, skipped work, queue age, and live-read ratio. Restart waits for cache sync before restoring mappings. Canary rollout validates convergence, recovery, and API-server load; protection state is never globally cleared.
Common mistakes
- Mistake → live-read the API server on every reconcile; Why it fails → QPS and latency increase and the cache loses its purpose; Fix → live-read only for critical actions or unknown versions.
- Mistake → pause every worker when one object is stale; Why it fails → one hot key creates a global outage; Fix → skip and re-queue by object key.
- Mistake → compare only local timestamps; Why it fails → clock drift cannot prove watch causality; Fix → compare API resource versions.
- Mistake → clear all protection after restart; Why it fails → decisions may run while caches rebuild; Fix → wait for sync and restore state with version checks.
Follow-up questions
Why is a resource version safer than a local timestamp?
It comes from the API server’s object-change sequence and can show that the cache has observed a particular write. Local time is affected by clock drift, network delay, and process pauses. A resource version is not a transaction sequence across objects, so multi-object consistency still needs an explicit design.
How do you avoid starvation when one key stays stale?
Cap exponential backoff and alert on maximum wait while allowing other keys to run. After the threshold, switch to low-frequency probing or one live read. Count consecutive skips so rapid retries cannot overload the API server.
When should an operation be rejected?
For deletion, scale-down, failover, or Lease-expiry decisions, pause the key and keep protection when freshness is unknown and the live read fails. Ordinary status reporting can continue within an explicit stale window, but must mark the degraded state in status and metrics.