TCP Connection Lifecycle: Handshake, Teardown, and TIME_WAIT

Question and Scope

A Linux API client does not reuse connections and opens 5,000 short TCP connections per second to one upstream. It then reports connection timeouts and a large TIMEWAIT population. Explain how TCP opens and closes, which endpoint enters TIMEWAIT, why the state exists, and how you would diagnose and fix the incident.

Use explicit baseline assumptions. The client has one source IP, and the upstream IP and port are fixed. The ephemeral-port range is the Linux-documented default of 32768–60999, with no reserved ports. A packet capture shows that the client sends the first FIN, and observed TIME_WAIT entries last about 60 seconds. The last two facts are evidence supplied by the scenario, not universal operating-system constants.

This question fits backend, infrastructure, SRE, client, and general software engineering interviews. The real task is not to recite “three-way handshake and four-way teardown.” It is to use TCP states, four-tuple capacity, and packet evidence to separate normal cleanup from ephemeral-port exhaustion, packet loss, and upstream overload.

What the Interviewer Is Testing

First, can the candidate explain that the handshake synchronizes both initial sequence numbers? A SYN consumes one sequence number, and the acknowledgment identifies the next sequence number expected. The third message confirms that the initiator received the peer's initial sequence number. Listing SYN, SYN+ACK, and ACK without this purpose is incomplete.

Second, can the candidate map full-duplex close semantics to states? The two directions can stop sending independently, so a normal close is commonly shown as FIN → ACK → FIN → ACK. An ACK can be combined with a FIN, so a capture does not always contain four separate packets. A strong answer explains what FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, LAST-ACK, and TIME-WAIT are waiting for.

Third, can the candidate identify the TIMEWAIT owner? It is usually the endpoint that actively closes and sends the final ACK. That endpoint can be the client or the server. If both endpoints actively close at the same time, both can enter TIMEWAIT. Client and server labels alone do not decide the state.

Fourth, can the candidate avoid diagnosing every large TIME_WAIT count as a failure? Many short connections naturally produce many entries. Port exhaustion requires a relationship among tuple supply, connection creation rate, error behavior, and packet evidence.

Fifth, can the candidate propose fixes in increasing order of risk? Reduce connection creation first, inspect capacity and the network path next, and consider kernel settings last. Truncating states, lowering a bucket limit, or using RST to bypass a normal close can turn an obvious capacity problem into intermittent correctness failures.

Questions to Clarify First

• Which endpoint actively closes? The endpoint that sends the first FIN normally follows the active-close path. If the upstream closes first, client-side TIME_WAIT needs another explanation.
• What does “connection timeout” mean in the error data? Local port allocation can fail immediately with errors such as EADDRNOTAVAIL. A connection that remains in SYN-SENT and retransmits points more toward the path, a filter, listen backlog pressure, or an unresponsive upstream. Error type and duration change the investigation.
• Can the protocol reuse connections? HTTP keep-alive, a connection pool, HTTP/2 multiplexing, or another persistent transport can remove most handshakes and closes. Capacity expansion becomes relevant only when one-request-per-connection behavior is genuinely required.
• Where is the port constraint? The host has an ephemeral-port range. A NAT, proxy, or load balancer has its own public-address and port mapping pool. Spare host ports do not prove that the egress device has spare tuples.
• Is the destination fixed? A TCP connection is identified by source address, source port, destination address, and destination port. The same local port can be used for different destinations, so concentration on one upstream matters more than the machine-wide connection total.
• What is the live system configuration? Port ranges, reserved ports, TCP timestamps, reuse policy, and kernel version affect capacity. The scenario's defaults support an initial estimate, while a production conclusion requires reading actual values.

30-Second Answer Framework

“The three-way handshake exchanges and confirms both initial sequence numbers. The client sends SYN, the server's SYN+ACK acknowledges the client's number and supplies its own, and the client ACKs that number. A normal close has two independent directions, so it is commonly FIN, ACK, FIN, ACK. The active closer that sends the final ACK normally enters TIME_WAIT so it can acknowledge a retransmitted FIN and keep delayed duplicates from an old connection away from a new use of the same tuple.

A high TIMEWAIT count is not automatically a leak. I would inspect the error type, SYN-SENT count, packet capture, and port range. One source IP to one upstream has 28,232 ports in the stated default range. At 5,000 new connections per second with about 60 seconds observed in TIMEWAIT, demand is roughly 300,000 recent tuples, far above supply. I would first add pooling or multiplexing, then inspect NAT and tuple capacity, and only then consider a wider range or more source addresses. I would not begin by deleting TIMEWAIT or changing tcptw_reuse blindly.”

Step-by-Step Deep Dive

Step 1: Explain the handshake with sequence numbers

Let the client's initial sequence number be x and the server's be y:

| Message | Important fields | State transition | What it establishes | |---|---|---|---| | Client → server | SYN, seq=x | Client enters SYN-SENT | Client requests a connection and supplies its initial sequence number | | Server → client | SYN, ACK, seq=y, ack=x+1 | Server enters SYN-RECEIVED | Server received the client SYN and supplies its own initial sequence number | | Client → server | ACK, ack=y+1 | Both reach ESTABLISHED | Client received the server SYN; sequence synchronization is complete |

A SYN consumes one sequence number, which is why the acknowledgment is x+1 or y+1. A pure ACK consumes no sequence space. The handshake gives each endpoint the peer's initial sequence number and confirmation that its own was received. It also reduces the chance that an old duplicate connection request will be mistaken for a new connection.

After only two messages, the initiator has seen the server's confirmation, but the server has not received confirmation that the client accepted the server's initial sequence number. “It checks that the network works in both directions” is an imprecise shortcut. TCP is establishing acknowledged, retransmittable sequence space and shared connection state.

Step 2: Derive teardown from two independent directions

TCP is a full-duplex byte stream. Closing one sending direction means “I have no more data to send,” while that endpoint may continue receiving data that the peer has not finished sending. A normal close therefore terminates two directions:

| Active closer | Passive closer | Meaning | |---|---|---| | Sends FIN, enters FIN-WAIT-1 | Receives FIN, ACKs, enters CLOSE-WAIT | Active side stops sending; passive application may still send remaining data | | Receives ACK, enters FIN-WAIT-2 | Application finishes, sends FIN, enters LAST-ACK | Active side waits for the peer to close the other direction | | Receives FIN, sends final ACK, enters TIME-WAIT | Receives final ACK, enters CLOSED | Both directions have closed normally |

A persistent CLOSE-WAIT population usually means that the application was told about a peer close but did not close its own socket promptly. That is a different failure from TIME_WAIT. Persistent FIN-WAIT-2 means the active closer's FIN was acknowledged, but the peer has not sent its FIN. Calling all of these “connections that were not released” loses the state machine's diagnostic value.

The four-message description is a useful normal case, not a fixed packet count. A passive closer with no remaining data can combine its ACK and FIN. Simultaneous active close can use CLOSING and can leave both endpoints in TIME-WAIT.

Step 3: Explain the two jobs of TIME_WAIT

The endpoint that sends the final ACK cannot immediately forget the connection for two main reasons.

First, the final ACK can be lost. The passive closer remains in LAST-ACK and retransmits its FIN. An active closer that still has TIME_WAIT state can acknowledge that FIN again. If the state disappeared immediately, the retransmitted FIN could receive an RST and the normal close would lose its reliability property.

Second, delayed or duplicated segments from the old connection may still exist in the network. TCP identifies a connection by its four-tuple. If the same tuple is immediately assigned to a new connection, an old segment can overlap the new sequence space. The TCP standard requires an actively closed connection to linger for 2 × MSL, giving old segments time to expire and preserving the ability to acknowledge a retransmitted FIN. RFC 1337 explains how terminating this protection early can reintroduce old-data acceptance, desynchronization, and erroneous-ACK hazards.

TIME_WAIT is therefore a correctness mechanism, not a synonym for a memory leak. The first questions should be why the application creates so many connections, whether tuple supply is sufficient, and which endpoint closes actively—not how to make the count zero.

Step 4: Estimate ephemeral-port pressure

The Linux-documented default iplocalport_range is 32768–60999. With the scenario's assumption of no reserved ports, the automatic allocation pool contains:

60999 - 32768 + 1 = 28232

For one source IP, one destination IP, and one destination port, there are about 28,232 source-port slots when old tuples cannot yet be safely reused. At 5,000 new connections per second, the application creates that many tuples in about 5.65 seconds:

28232 / 5000 ≈ 5.65 seconds

The scenario also supplies an observed TIME_WAIT duration of roughly 60 seconds. The rough recent-close demand is:

5000 × 60 = 300000 recent connections

Three hundred thousand is far above 28,232, so “one source IP, one upstream, no connection reuse” is a clear capacity risk. This does not prove that failure occurs precisely at 5.65 seconds. Safe kernel reuse, established connections, reserved ports, connection duration, and NAT behavior all change the live result. The estimate proves an order-of-magnitude mismatch and tells the investigator what evidence to collect next.

The same source port can serve connections to different destinations, so a machine-wide new-connection total cannot be divided blindly by the port count. With NAT, the scarce resource may instead be the public NAT address's mapping tuples toward the destination.

Step 5: Separate causes with states, errors, and packets

Start with evidence that does not change system behavior:

ss -s
ss -Htan state syn-sent | wc -l
ss -Htan state time-wait | wc -l
ss -Htan state close-wait | wc -l
cat /proc/sys/net/ipv4/ip_local_port_range
cat /proc/sys/net/ipv4/ip_local_reserved_ports
sysctl net.ipv4.tcp_tw_reuse

Then branch on the evidence:

| Evidence | More likely direction | Next step | |---|---|---| | connect() quickly returns a local-address error and no SYN leaves the host | Ephemeral-port or local bind resource exhaustion | Count new connections by destination; inspect range, reserved ports, source IPs, and NAT | | Many SYN-SENT sockets and repeated SYNs with no SYN+ACK | Packet loss, ACL, unresponsive upstream, or listen backlog pressure | Capture at client and upstream to locate where packets disappear | | Many CLOSE-WAIT sockets | Local application did not complete passive close | Inspect file descriptors, request cancellation, and exception paths | | Many TIME_WAIT sockets, no failures, and sufficient port headroom | Normal consequence of short connections | Observe; do not tune only to reduce the count | | Host ports are available but many instances behind one NAT fail together | Egress NAT mapping pool may be exhausted | Inspect NAT metrics, public source-address count, and destination concentration |

A capture should answer three questions: who sends the first FIN, whether a failed connection emits a SYN at all, and where retransmission occurs. A total from ss cannot prove port exhaustion, and an application log containing the word “timeout” cannot prove packet loss.

Step 6: Fix in increasing order of risk

The first fix is to create fewer connections. Configure a connection pool, HTTP keep-alive, HTTP/2 multiplexing, or another suitable persistent channel so requests share established connections. This reduces handshake latency, CPU, ephemeral-port use, and TIME_WAIT together, addressing the cause rather than the symptom.

Next, correct the connection lifecycle. Do not actively close a healthy connection after every request. Give the pool bounded concurrency, an appropriate idle timeout, a maximum lifetime, and upstream concurrency limits. If the server closes too aggressively, inspect its keep-alive settings, load-balancer idle timeout, and deployment behavior. The goal is to remove pointless connection churn, not merely move active close to the other endpoint.

If reuse is still insufficient, expand tuple supply. After checking reserved ports, evaluate a wider ephemeral range. A single-destination hotspot can also use more source IPs or more destination addresses. If NAT is present, expand or shard the public NAT address pool as well; adding source IPs inside the application network may have no effect after translation.

Only then evaluate kernel reuse settings. Linux documents tcptwreuse as reusing TIMEWAIT sockets only when the protocol considers it safe and warns against changing it without expert advice. tcpmaxtwbuckets is a defensive limit against simple denial-of-service conditions; the documentation explicitly says not to lower it artificially. Exceeding it destroys time-wait sockets immediately and logs a warning. Any change needs kernel-version, timestamp, peer-behavior, load-test, and rollback evidence.

Do not use RST as a general optimization. RST aborts the connection and discards state immediately; data that the application has not safely accounted for can be lost. “Make the server close first” is also not a universal fix. It moves TIME_WAIT pressure and may create server-side port, memory, or connection-state pressure.

Step 7: Verify the fix

Verification must cover capacity, correctness, and network evidence:

1. Replay the same traffic and compare new TCP connections per second, reuse ratio, and request throughput.
2. Track SYN-SENT, TIME_WAIT, CLOSE-WAIT, file descriptors, and local-port use over time rather than at one instant.
3. Capture failed samples to confirm whether SYN leaves, whether SYN+ACK returns, and which endpoint actively closes.
4. Inspect client, NAT, load balancer, and upstream transport metrics so the egress bottleneck is not missed.
5. Run a sustained test and check for truncated data, higher RST counts, worse tail latency, or stale pooled connections.

Success means that the same business throughput creates far fewer new connections, errors disappear, port headroom stays stable, and request correctness does not regress. A lower TIME_WAIT count is a consequence of those improvements, not the only objective.

Strong Sample Answer

“I would first establish two facts from the incident: which endpoint sends the first FIN, and whether ‘timeout’ means an immediate local error or a SYN that receives no response. TIME_WAIT normally belongs to the active closer that sends the final ACK, while several unrelated failures can look like a connection timeout.

During establishment, the client sends SYN with initial sequence number x. The server's SYN+ACK acknowledges x+1 and supplies its own number y. The client then acknowledges y+1. Both endpoints have now exchanged and confirmed sequence space, and the process can reject old duplicate connection attempts. Teardown closes the two sending directions independently, so the common sequence is FIN, ACK, FIN, ACK. The active closer moves through FIN-WAIT-1 and FIN-WAIT-2, ACKs the peer's FIN, and enters TIME_WAIT. That state lets it re-ACK a retransmitted FIN if the final ACK was lost and prevents delayed duplicates from an old connection from contaminating immediate reuse of the same four-tuple.

The scenario's capacity is suspicious. Ports 32768 through 60999 give 28,232 default ephemeral ports. One source IP creates 5,000 connections per second to the same destination, and the observed TIME_WAIT duration is about 60 seconds, implying roughly 300,000 recent closes—far more than the port slots. I would not stop at the estimate. If connect fails immediately with a local-address error and no SYN, I would investigate host and NAT tuple supply. If many sockets remain in SYN-SENT and retransmit, I would investigate the path, ACL, listen backlog, and upstream.

I would first add pooling, keep-alive, or HTTP/2 multiplexing to reduce new connections. Then I would check pool idle timeouts, upstream close behavior, and NAT capacity. If the workload still needs a high establishment rate, I would consider a wider ephemeral range, more source addresses, or destination distribution. tcptwreuse is only safe under protocol conditions and Linux warns against casual changes. I would not begin by shortening TIME_WAIT, lowering the bucket limit, or forcing RST closes. Finally, I would validate new-connection rate, errors, every relevant TCP state, packet captures, and NAT metrics under the same load.”

Common Mistakes

• Describing the handshake as “saying hello” → It omits sequence and acknowledgment semantics and cannot explain why two messages are insufficient → derive it from both initial sequence numbers and the third confirmation.
• Assuming the client always owns TIME_WAIT → Active-close role determines the state, not the client label → identify the first FIN and final ACK.
• Treating teardown as exactly four packets → ACK and FIN can combine, and simultaneous close exists → describe two independent directions and their state transitions.
• Calling every large TIME_WAIT count a leak → Short connections naturally create the state → combine creation rate, range, error type, and packet evidence.
• Blaming every connect timeout on port exhaustion → SYN loss, ACLs, upstream overload, and listen backlog pressure can also time out → separate immediate local failure from SYN retransmission.
• Lowering tcpmaxtw_buckets first → Exceeding the limit destroys state early, and Linux explicitly warns against lowering it artificially → reduce churn before expanding capacity.
• Using RST instead of a normal close → Immediate state deletion can discard data that was not safely delivered → keep FIN-based close for normal completion and reserve RST for genuine aborts.
• Only widening the application's port range → The real limit may be the NAT public mapping pool → check tuple capacity at the source host, egress device, and upstream.

Follow-up Questions

Follow-up 1: How does TIME_WAIT recover when the final ACK is lost?

The passive closer remains in LAST-ACK waiting for acknowledgment of its FIN. If the final ACK is missing, it retransmits the FIN. The active closer still has TIME_WAIT state, recognizes the retransmitted FIN, sends the ACK again, and restarts the wait timer. If the active endpoint had forgotten the connection, that FIN could trigger an RST and reliable normal close would no longer be preserved.

Follow-up 2: How is a large CLOSE_WAIT population different?

CLOSE-WAIT means the peer sent FIN and the local TCP stack notified the application, but the application has not closed its own sending direction; the local application may still send remaining data. It usually points to application lifecycle, exception handling, or file-descriptor management. TIME_WAIT means the active closer completed the exchange and is protecting final-ACK recovery and old-segment isolation. Both appear during closure, but their causes, ability to carry remaining data, and fixes differ.

Follow-up 3: How can one source port belong to multiple connections?

The complete four-tuple identifies a connection. If the destination address or destination port differs, the same local port can identify another connection. Ephemeral-port pressure must therefore be analyzed by source IP and destination hotspot. Traffic spread across many destinations can support more total connections than one port range suggests, while concentration on one upstream exhausts usable tuples faster.

Follow-up 4: Will scaling the client to ten instances solve exhaustion?

Not necessarily. Independent source IPs increase host-side tuple supply, but if all instances share a NAT with only a few public IPs, the connections converge into the NAT's mapping pool. The upstream may also rate-limit by source IP, and extra concurrent connections may exceed its listen backlog, file descriptors, or load-balancer capacity.

Follow-up 5: When is tcptwreuse worth considering?

First prove that connection reuse, pool settings, the port range, and NAT capacity remain insufficient. Then check the exact kernel version, TCP timestamp behavior, and peer characteristics. Linux allows reuse only when it is safe from the protocol's perspective and explicitly warns against changing the setting casually. Test reconnection and data correctness under load, monitor RSTs, failures, and latency, and prepare a rollback. It is not a substitute for application-level connection reuse.