Prompt and context
This system design question tests whether you can turn low-level TCP keep-alive parameters into a governed platform capability. The challenge is not simply sending probes more often. Separate transport liveness from application health, then design tenant isolation, policy versions, connection lifecycle, network cost, and recovery.
What the interviewer is testing
- Explain the relationship among idle time, probe interval, maximum failures, and false positives.
- Design safe defaults, tenant quotas, permissions, and configuration validation.
- Apply a policy at an appropriate connection lifecycle point without causing churn during hot updates.
- Provide auditability, metrics, rollout, rollback, and overload protection.
Clarifying questions
Confirm connection types, client and server control, shared nodes, NAT, mobile networks, proxies, and the business target for detecting a dead connection. Check kernel support, whether keep-alive is disabled by default, existing application heartbeats, per-tenant connection counts and budgets, and whether changes must be immediate. A TCP probe only establishes transport response; it does not replace an application-level health check.
A 30-second answer framework
I would build a versioned tenant policy service with conservative global defaults and tenant limits, validating the combination of idle time, probe interval, and failure count. New connections read and bind a policy version; runtime refresh is controlled to avoid resetting huge numbers of sockets. Publication requires authorization, audit, staged rollout, and automatic rollback, while the data plane records probes, disconnects, and false positives. I would cap the total probe budget, keep using the last valid version when the control plane is unavailable, and reserve application heartbeats for business semantics.
Step-by-step design
1. Define the policy model and safe defaults
Include idle-time, probe-interval, max-probes, connection scope, version, expiry, and source. RFC 9293 requires keep-alive to be switchable per connection and disabled by default; RFC 9643 recommends conservative intervals because probes consume resources. Use a sufficiently long default idle time and interval, and let tenants shorten them only within a resource budget.
2. Separate control plane and data plane
The control plane stores tenant policies, versions, and audit records, and exposes validation, approval, publication, rollout, and rollback. After connection setup or handshake, the data plane reads a signed snapshot, applies it to the controllable socket, and reports the effective version. A short control-plane outage should not fail every connection; cache the last valid snapshot and define expiry behavior.
3. Validate combinations and quotas
Reject probe intervals below a safe floor, cap per-tenant connection counts and probe rates, and check that failure count matches the detection-time target. Compute budgets by node, zone, and egress path so one tenant cannot concentrate probes on one instance group. Apply a second limit based on real-time node load even after configuration validation.
4. Distribute versions and handle updates
Bind a policy version to each connection, with new connections using the latest published version. Existing connections can refresh in batches, on natural reconnect, or at the next idle cycle; do not update millions of sockets at once. Stage each release to a small set of tenants and nodes, compare detection time, false positives, CPU, bandwidth, and connection churn, then expand.
5. Build observability and failure classes
Record policy version, probes sent, response rate, consecutive failures, final disconnect reason, reconnect success, and per-tenant resource use. Distinguish peer close, packet loss, node overload, expired policy, and application-heartbeat failure. A missing keep-alive response alone does not prove death; alerts should combine repeated probes with application outcomes.
6. Design rollback and overload protection
Keep the previous stable version and an emergency global policy. If a rollout raises disconnects, probe traffic, or CPU above thresholds, stop publication and restore the old version; if updates fail, continue with a validated snapshot. Add rate limits, tenant circuit breakers, and queue backpressure so a policy storm cannot overwhelm the connection manager.
High-quality sample answer
I would implement keep-alive policies as a versioned control-plane service containing idle time, probe interval, maximum probes, connection scope, and audit data. The platform would use a disabled, conservative global default and cap tenant settings by connection and probe budgets. New connections would read a signed snapshot and bind its version; existing connections would update only through batches or natural reconnects. The rollout would require approval, staged metrics, and automatic rollback, tracking probe rate, responses, disconnect causes, false positives, reconnects, and node load. Keep-alive would cover transport liveness while application heartbeats confirm business health. If probe traffic or disconnects rose unexpectedly, I would stop propagation and restore the stable version.
Common mistakes
- Treating keep-alive as an application health check because an ACK arrived.
- Letting tenants shorten intervals without node, egress, and global probe budgets.
- Walking every connection immediately after a policy change and causing synchronized churn.
- Omitting versions, signatures, audit records, and a last-known-good snapshot.
- Looking only at disconnect counts without classifying loss, peer close, overload, and false positives.
- Publishing globally without a staged rollout or rollback path.
Follow-up questions and answers
Why not set the probe interval to a few seconds?
Frequent probes consume network, CPU, and battery resources and can amplify congestion. Calculate parameters from the detection target, connection count, and resource budget, and prefer an application heartbeat when business semantics are required.
A tenant wants an immediate change. What do you do?
Let new connections use the approved version immediately and update existing connections through bounded batches with per-node rate limits. Emergency security changes can receive higher priority, but still require an approver, scope, expected churn, and rollback condition.
What happens when the control plane is down?
The data plane continues using the last unexpired signed snapshot for a bounded period. After expiry it falls back to a safe global default, not an empty configuration. Once recovered, reconcile versions without refreshing every connection at once.
How do you detect a keep-alive false positive?
Compare consecutive probe failures with application heartbeats, reconnect outcomes, and path metrics. A single missed ACK is insufficient; require the configured failure count and corroborating business or application signals.
How do you stop one large tenant consuming the probe budget?
Use hierarchical tenant, node, zone, and global quotas based on connection count and probe rate. Above quota, lengthen intervals, reject aggressive settings, or require a quota upgrade while preserving a minimum service level for others.