Problem and scope
This is a control-plane system-design question for platform and Kubernetes engineers. Kubernetes v1.36 introduces server-side sharded List and Watch as an alpha feature behind the ShardedListAndWatch gate. Treat the feature as optional and design a controller that remains correct when the gate is disabled.
What the interviewer evaluates
- A shard ownership model that covers the keyspace exactly once.
- Correct initial List, resource-version handling, and Watch reconnect behavior.
- Detection of an API server that ignored the selector.
- Resharding, replica churn, and the trade-off between overlap and gaps.
- Capacity reasoning for API server CPU, network, informer memory, and recovery storms.
What is the ownership key?
The alpha implementation supports object.metadata.uid and object.metadata.namespace. UID hashing gives stable distribution; namespace hashing can preserve tenant locality but may be skewed. The choice changes balancing, security boundaries, and migration cost.
What is the correctness target?
Ask whether at-least-once reconciliation is acceptable and whether duplicate processing is idempotent. If a business side effect cannot tolerate duplicates, add a durable work key and fencing instead of relying on an event stream alone.
Can every API server and client upgrade together?
Assume mixed versions unless the platform team proves otherwise. The controller must inspect the response metadata and fall back to client-side filtering without silently claiming reduced load.
30-second answer framework
“I would partition a deterministic 64-bit hash space into non-overlapping ranges and assign each range to one controller replica. Each informer sends a shard selector on both List and Watch, then verifies the response contains matching shard metadata. A missing acknowledgment triggers a safe full-stream fallback or disables the optimization. Resharding uses a generation and a handoff protocol with overlap, idempotent reconciliation, and metrics for coverage, duplicates, lag, and fallback traffic.”
Step-by-step design
Partition and assignment
Represent ownership as half-open ranges [start, end) over a 64-bit ring. Store a generation, owner, and lease for each range. A two-replica deployment can split the ring in half; more replicas use a controller-managed range table. Do not derive ownership from replica ordinal alone because a restart can otherwise create gaps.
Replica A: [0000..., 8000...)
Replica B: [8000..., 1000...)Make List and Watch one protocol
The initial List and every Watch reconnect must carry the same selector and a resource version. The informer replaces its local store only after the initial list is complete, then starts the watch from the returned resource version. A 410 Gone or expired version causes a fresh list for that shard, not a blind replay of another range.
Verify server support
The v1.36 blog describes a shardInfo response field that echoes the applied selector. If it is absent, assume the server returned the full collection. The client can filter locally for correctness, but must emit a fallback metric and apply backpressure so an unsupported server does not multiply memory use across replicas.
Reshard without gaps
Create a new generation of ranges. During handoff, the old owner keeps reconciling while the new owner warms a list and waits for a watch at a known resource version. Commit the ownership change only after both sides report readiness; duplicate events are acceptable when reconciliation is idempotent. If readiness fails, expire the lease and keep the old owner.
Capacity and failure paths
Server-side filtering reduces bytes and deserialization for discarded objects, but the API server now performs hashing and selector evaluation. Protect it with a feature gate, per-controller concurrency limits, and rollout canaries. On API-server overload, throttle reconnects and use exponential backoff; never let every replica relist at once.
High-quality sample answer
“I would use a generation-stamped range table over a deterministic hash of UID. Informers include that selector on List and Watch and require shardInfo before counting the optimization as active. A missing field falls back to local filtering with a global concurrency budget. Resharding creates a new generation, warms the incoming owner from a resource version, and switches leases only after readiness; reconciliation is idempotent so overlap is safe. I would canary the feature gate and compare API CPU, bytes, list latency, watch reconnects, shard coverage, duplicates, and fallback rate.”
Common mistakes
- Assigning shards by replica index → Restarts change ownership and create gaps → persist a generation-stamped range table.
- Applying the selector only to Watch → The initial List still overloads the server and can disagree with the stream → use the same selector and resource-version rules for both.
- Trusting an unsupported server → Every replica receives the full stream → require
shardInfoand expose fallback traffic. - Moving a range instantly → Events can be missed during handoff → overlap owners, fence with generations, and make reconciliation idempotent.
- Measuring only controller CPU → API-server hashing or reconnect storms remain invisible → monitor control-plane and client metrics together.
Scoring rubric and self-check
Score partition correctness, list-watch semantics, fallback, resharding, capacity, and observability. A strong answer states the invariant: every object belongs to one active range generation, and every handoff has a resource-version boundary. It also explains why duplicates are safer than gaps when reconciliation is idempotent.
Follow-ups and extensions
What if one namespace is much larger than others?
Prefer UID hashing for balance, or split hot namespaces into multiple UID ranges. Measure range load instead of assuming equal object counts.
How do you detect a silent gap?
Compare sampled global object counts with the union of shard counts, record selector generations, and run periodic synthetic objects through every range. Alert on lag or coverage drift.
What happens during an API-server upgrade?
Keep the feature gate disabled until canaries observe shardInfo from every serving endpoint. Mixed responses trigger fallback and a bounded reconnect rate.
Can a controller process the same object twice?
Yes during overlap or reconnect. Use object version and a durable work key to make side effects idempotent; never trade duplicate reconciliation for an unbounded risk of missed state.