Prompt and scope
An HTTP service runs in a Deployment. After a Pod is deleted, some requests still reach the old instance, while long polling and background jobs are interrupted. Explain the termination timeline across Kubernetes, the Service, load balancers, and the process, then design draining, cancellation, retries, and verification. The core skill is bounding connections and side effects during shutdown, so this belongs to backend.
What the interviewer evaluates
A strong answer separates Pod deletion, EndpointSlice updates, load-balancer propagation, preStop, SIGTERM, and SIGKILL. Receiving SIGTERM does not mean traffic has stopped. Cover long-lived connections, queue jobs, idempotent retries, unexpected node power loss, and an insufficient grace period.
Questions to clarify first
- Is the service short request, SSE, WebSocket, or long polling? Can connections migrate?
- What do readiness, liveness, and startup probes check, and does the application expose drain state?
- Is preStop a sleep, an HTTP hook, or an application drain endpoint? Who performs final exit?
- Does the Pod run workers, and how are leases and duplicate deliveries handled?
- What are the propagation delays of the ingress load balancer, service mesh, and cloud LB?
30-second answer framework
“First mark the application not-ready and stop accepting new work, then wait for endpoint and load-balancer propagation. Enter draining: finish short requests, close or migrate long-lived connections, and stop workers from claiming new jobs while releasing leases they cannot finish. preStop is only a coordination window; SIGTERM performs cleanup. The grace period must cover p99 propagation, draining, and cleanup, with SIGKILL only after it expires. Test releases with connection and job metrics to prove no lost requests or duplicate effects.”
Step-by-step solution
Draw the timeline: the controller deletes the Pod, kubelet begins termination, EndpointSlice and load balancers remove the endpoint asynchronously, and the container may run preStop before SIGTERM. Because propagation is asynchronous, the application must reject new work as soon as it knows it is draining, even if traffic still arrives at the old address.
Keep separate ready and draining states. The shutdown path sets draining, fails readiness, and stops accepting new connections or returns a retryable response. Existing short requests finish. SSE, WebSocket, and long polling should receive a close signal, a cursor, or a resume token so the client can reconnect safely.
preStop sleep cannot replace application logic. It only buys propagation time and consumes the grace period. Prefer an application drain endpoint that records start time and remaining budget; the SIGTERM handler stops new work, closes the listener, waits for active requests, and releases connection pools.
Design worker draining separately from HTTP draining. Stop claiming new messages. Protect active jobs with a lease or visibility timeout, and release them when the remaining budget is insufficient so another worker can retry. Job writes and external effects need idempotency keys; a killed Pod must not charge or ship twice.
Choose terminationGracePeriodSeconds from measured p99 values: ingress propagation, the longest allowed request, connection close, worker cleanup, and log flushing, plus jitter margin. Once the deadline expires kubelet can force termination; unfinished requests and in-process state cannot rely on finally executing.
Node maintenance is different from Pod deletion. Kubernetes graceful node shutdown gives kubelet and Pods a planned window, but power loss, kernel panic, and forced host restart do not. Critical jobs need durable checkpoints, multiple replicas, and queue-based recovery.
Verify rolling releases, manual deletion, node drain, slow LB propagation, long connections, and timeout after SIGTERM. Check 5xx, disconnects, completion rate, draining rejections, job retries, duplicate effects, endpoint-removal delay, and forced Pod kills, all correlated to the release version.
Model high-quality answer
“I would not treat preStop: sleep 30 as graceful shutdown. On the shutdown signal, the application marks itself draining, fails readiness, and stops accepting new work. Endpoint and LB updates propagate asynchronously, so the old instance still rejects new requests. Short requests finish; SSE and WebSocket connections receive a close or resume token; workers stop claiming jobs and release leases they cannot finish.
preStop only buys propagation time. The SIGTERM handler closes the listener, waits for active requests, releases pools, and exits before one deadline. I size the grace period from measured p99 propagation, longest request, worker cleanup, and log flushing. Tests cover rolling release, node drain, long connections, crash, and force kill, checking for no lost requests, duplicate effects, or unrecovered jobs.”
Common mistakes
- Wait only for SIGTERM → propagated traffic still reaches the old Pod → fail readiness and enter draining first.
- Use a fixed sleep instead of draining → changing propagation or request duration breaks it → drive logic from remaining deadline.
- Stop the Pod but keep claiming queue jobs → forced kill duplicates work → stop claiming and release leases.
- Treat long connections as ordinary requests → clients cannot resume → send close signals and cursors.
- Size grace from average latency → p99 requests get SIGKILL → measure tail latency and propagation.
- Assume
finallyalways runs → force kill and power loss skip cleanup → persist critical state. - Watch only Pod status → miss slow LB propagation and disconnects → correlate endpoints, requests, and release metrics.
- Fail liveness while draining → trigger restart storms → readiness means traffic eligibility; liveness means process health.
Follow-up questions and responses
Follow-up 1: Why can requests arrive after readiness fails?
Endpoint, service-mesh, and cloud load-balancer updates propagate with delay, and existing connections are not migrated automatically. The draining application must reject new work and close existing connections safely.
Follow-up 2: How long should preStop sleep?
Do not guess. Measure endpoint and LB propagation p99, combine it with request and worker cleanup budgets, and treat sleep as a bounded coordination window.
Follow-up 3: How do you close WebSockets gracefully?
Stop accepting new connections and send a close code with reconnect guidance. Let clients resume with a session or cursor, and persist enough state to avoid replaying side effects.
Follow-up 4: What if a Pod dies halfway through a job?
Persist job state and a lease. A new worker claims it after visibility timeout; idempotency keys or status queries protect external effects. Do not rely on in-process finally.
Follow-up 5: Can node power loss be graceful?
Not reliably. Planned node-shutdown handling covers observable maintenance flows; power loss and kernel crashes require replicas, durable checkpoints, and retryable jobs.
Follow-up 6: How do you choose the grace period?
Add endpoint propagation, maximum request, long-connection close, lease release, connection cleanup, and log flush, then validate with measured p99 and margin rather than averages.
Follow-up 7: How do you avoid a restart storm while draining?
Fail readiness while keeping liveness healthy. Separate probe responsibilities so “temporarily not accepting traffic” is not mistaken for a crashed process.