TCP: the reliable byte stream
What TCP actually guarantees, how a connection starts and ends, and the two windows that keep it from drowning the other side or the path.
On this page
The network underneath you drops packets, shuffles them, and sometimes delivers the same one twice. TCP hides that. You write bytes in; they come out the other end, complete and in order, or the connection fails. This page is the picture you want in your head: open, stream, close, and the two brakes TCP uses along the way.
The map
Read this first when short on time. Every branch is a section below.
A reliable ordered byte stream
TCP is a connection that carries a stream of bytes. They arrive complete, in the order you sent them, or the connection dies trying. It does not know about messages. Two writes of 50 bytes can arrive as one read of 100. If you need messages, you put a length on them yourself.
It sits on top of IP. IP is allowed to drop, delay, duplicate or reorder. TCP's job is to hide that. The cost is a handshake before the first byte, and a stall if one packet in the middle goes missing.
UDP is the same layer without any of that. You fire a datagram. Maybe it arrives. DNS, games and HTTP/3 pick UDP on purpose. HTTP/1.1, HTTP/2, Postgres, Redis and SSH pick TCP.
| TCP | UDP | |
|---|---|---|
| What you get | A connection and a byte stream | Separate datagrams |
| Setup | Three-way handshake | None |
| If a packet is lost | TCP sends it again, in order | That datagram is just gone |
| Typical uses | HTTP, databases, SSH, TLS | DNS, HTTP/3, video, games |
- A browser tab opens TCP (then TLS) to port 443. HTTP/2 still sits on that one connection.
- PostgreSQL and Redis each speak their own framing over TCP. The database protocol is messages; TCP is just the pipe.
- HTTP/3 left TCP for QUIC over UDP, so one lost packet does not freeze every request on the connection. That is in the networking and edge article.
- Segment
- A TCP packet: a header plus a slice of the byte stream.
- 4-tuple
- Source IP, source port, destination IP, destination port. That is one connection.
Opening: the three-way handshake
Nothing useful happens until both sides agree the connection exists. That takes three packets.
sequenceDiagram autonumber participant C as Client participant S as Server C->>S: SYN seq=x S->>C: SYN-ACK seq=y ack=x+1 C->>S: ACK ack=y+1 Note over C,S: both sides ESTABLISHED
- The client sends SYN with a random starting sequence number
x. - The server replies SYN-ACK: "I heard
x, here is my start numbery." - The client sends ACK of
y. Now both sides are ESTABLISHED.
Two packets would not be enough. Both sides have to publish a start number, and the client has to prove it saw the server's. Without that last ACK, a delayed SYN from last week could open a connection nobody asked for.
The start number is random so an old packet from a previous connection cannot land in this one, and so a stranger on the internet cannot guess the next number and inject a reset.
- Every HTTPS request pays this round trip (plus TLS) unless the connection is already open. Keep-alive and HTTP/2 exist so you do not pay it per request.
- AWS NLB forwards the handshake through. The server's kernel still owns it. An L4 balancer does not terminate TCP.
A flood of SYNs with fake source addresses can fill the server's waiting list before any handshake finishes. Edges like Cloudflare catch this before it reaches you. A health check that only connect()s proves the kernel is up, not the app.
- ISN
- Initial sequence number. The random starting point for this direction of the connection.
Closing: FIN, RST and TIME-WAIT
A clean close takes four packets, because each direction of the stream is shut on its own. An abort takes one RST and throws away whatever was still in the air.
sequenceDiagram autonumber participant C as Client participant S as Server C->>S: FIN S->>C: ACK Note over S: CLOSE-WAIT, app must close S->>C: FIN C->>S: ACK Note over C: TIME-WAIT, about 60 s
FIN means "I will send no more." The other side can still talk back. That is a half-close. When it is done, it sends its own FIN. RST means "drop this now." There is no wait. Use it when something is wrong, not as a shortcut for close.
The side that sent FIN first waits in TIME-WAIT for about a minute on Linux. Two reasons: stray packets from this connection need to die before the same ports are reused, and if the last ACK was lost the other side can still get an answer instead of a reset. The cost is a 4-tuple stuck for a minute. Lots of short connections burn through ports. The real fix is to reuse connections, not to abort them.
Names you will actually see in ss or netstat:
- ESTABLISHED — data can flow.
- TIME-WAIT — you closed first. Usually churn, not a leak.
- CLOSE-WAIT — the other side already hung up, and your process has not called
close. That is an application bug.
- HAProxy on a busy front door piles up TIME-WAIT if it is the one sending FIN first. Connection reuse on the backend is the fix.
- Browsers keep connections alive so they do not handshake and TIME-WAIT on every click.
Do not "fix" TIME-WAIT by resetting on close. You lose the last response and you hand the other side a connection error. Reuse the connection.
How lost packets come back
Every byte has a sequence number. The receiver says, with an ACK, "I have everything up to here; send me the next byte." If that next byte never arrives, TCP sends it again.
sequenceDiagram autonumber participant C as Sender participant S as Receiver C->>S: seq 1000 C-xS: seq 1500 C->>S: seq 2000 S->>C: ACK 1500 C->>S: retransmit 1500 S->>C: ACK 2500
If later packets arrive while one in the middle is missing, the receiver keeps ACKing the hole. Three of those repeated ACKs and the sender resends immediately, without waiting for a timer. If nothing comes back at all, a timer fires and it tries again, slower each time.
Modern TCP can also name the chunks that did arrive after the hole, so the sender does not resend those. You do not need the option name. The idea is: ACK points at the gap, and extra information lists what is already there.
- Wireshark is how you see this: sequence numbers going up, ACKs pointing at the next expected byte, a retransmit when a number repeats.
- Linux
ss -ishows live retransmits on a connection. A path with a lot of them is the network, not your code.
Two windows: the receiver and the path
TCP will not send the whole file at once. It keeps two limits, and it sends at most the smaller of the two.
| Flow control | Congestion control | |
|---|---|---|
| Protects | The other process's buffer | The links and queues in between |
| Who sets it | The receiver, in every ACK | The sender, from loss and delay |
| If it fails | One process is overrun | A router drops everyone's packets |
Flow control is simple: the receiver advertises how many bytes of buffer it has left. The sender is not allowed to have more than that in flight. ACKs slide the left edge forward. A bigger advertisement opens the right edge. If the application stops reading, the window goes to zero and the sender pauses.
Congestion control is the sender being polite to the rest of the internet. It starts with a small window, grows it quickly while ACKs come back, then much more slowly. If packets go missing, it cuts the window and tries again. You do not need the kernel's algorithm name. The idea is: probe for capacity, back off on trouble.
- A browser download advertises a large window so a fat pipe can fill. If a tab stops reading the socket, the window shrinks and the server pauses.
- Redis on one connection: a client that stops reading replies will zero the window, and the server looks stuck.
- gRPC has its own stream window on top of TCP. If TCP's window is blocked, every stream on that connection waits.
- In flight
- Bytes you have sent that have not been ACKed yet. Capped by both windows.
When TCP feels slow
A correct connection can still feel laggy. Two reasons show up over and over: TCP holding a small write, and one lost packet freezing everything behind it.
Nagle holds a tiny send if there is already unacked data, hoping to fill a full packet. Delayed ACK waits a moment (up to about 200 ms) before sending a pure ACK, hoping to piggyback it on a reply. Together they wait for each other. Chatty RPCs pick up a 200 ms tax. The fix is TCP_NODELAY, which turns Nagle off. Leave it on for bulk copies. This is the same idea as batching in the scalability article, just inside TCP.
sequenceDiagram autonumber participant C as Client participant S as Server C->>S: first 40 bytes Note over C: Nagle holds next write Note over S: delayed ACK, 200 ms S->>C: ACK C->>S: second 40 bytes
TCP_NODELAY sends the second write immediately.Head-of-line blocking is the other one. TCP is one ordered stream. If packet N is lost, packets after it sit in the kernel until N is retransmitted, even if they belong to a different HTTP/2 request. HTTP/2 fixed this at the HTTP layer and then hit it again at TCP. HTTP/3 moved to QUIC over UDP so a loss stalls only its own stream. That comparison is in the networking and edge article.
- Redis and gRPC set
TCP_NODELAYby default. A chatty RPC without it feels like every call took an extra 200 ms. - HTTP/2 multiplexes many requests on one TCP connection and still stalls them all on a single loss. That is the reason HTTP/3 exists.
If small payloads work and large ones hang, the path's packet-size discovery is probably broken (a firewall eating the "too big" replies). That is a network problem, not your code.
Recap
- TCP is a connection that delivers a byte stream, complete and in order. UDP is datagrams with no handshake and no retry.
- There are no message boundaries. If you need messages, you frame them.
- Open is three packets: SYN, SYN-ACK, ACK. Both sides pick a random start number. Two packets would let a delayed SYN open a ghost connection.
- Close is four packets (FIN and ACK each way). RST aborts. The side that sent FIN first waits in TIME-WAIT so old packets and a lost last ACK are safe.
- CLOSE-WAIT in
ssmeans your process has not closed a socket the peer already finished. TIME-WAIT on the closer is usually just churn. - Every byte has a sequence number. An ACK names the next byte wanted. A hole is retransmitted; three repeated ACKs resend without waiting for the timer.
- Flow control protects the receiver (it advertises buffer space). Congestion control protects the path (the sender starts small, grows, cuts back on loss). In flight is the minimum of the two.
- Nagle plus delayed ACK adds ~200 ms to chatty RPCs: set
TCP_NODELAY. One lost TCP packet stalls the whole stream, which is why HTTP/3 left TCP.
Questions
Try answering each one out loud before opening it. Lead with the one-line answer, then a couple of points, then one extra detail.
What does TCP give you that UDP does not?
A connection that delivers bytes reliably and in order, or fails; UDP sends datagrams with no handshake, no ACK and no retry.
- TCP has no message boundaries: two writes can arrive as one read.
- The price is a handshake, extra delay, and a stall if one packet in the middle is lost.
HTTP, Postgres and Redis take the guarantees. DNS and HTTP/3 take UDP and build what they need.
Walk through the three-way handshake. Why not two packets?
Client SYN with start number x, server SYN-ACK with y and ack x+1, client ACK of y; the third packet proves the client saw the server's number, so a delayed old SYN cannot open a connection nobody asked for.
- Both sides must publish a start number, so you already need a SYN each way.
- Each SYN consumes one sequence number, so the ACK is always that number plus one.
The start number is random so old packets and guessed resets cannot land in the new stream.
How does a connection close, and what is RST for?
Each side sends FIN and gets an ACK, four packets in the usual case; RST aborts immediately and drops whatever was still in flight.
- FIN is "I will send no more." The other side can still reply. That is a half-close.
- The side that sent FIN first is the one that ends in TIME-WAIT.
Use RST when the connection is garbage, not as a faster close.
Why does TIME-WAIT exist, and why does CLOSE-WAIT mean something else?
TIME-WAIT is the closer waiting about a minute so stray packets die and a lost last ACK can still be answered; CLOSE-WAIT means the peer already hung up and this process has not called close.
- TIME-WAIT is usually churn on a client or proxy that closes first. Reuse connections rather than shortening it.
- A pile of CLOSE-WAIT is an application leak, not a network problem.
ss -t plus the owning process is how you tell which of the two you are looking at.
How does TCP notice a lost packet and get it back?
Every byte has a sequence number, the ACK names the next byte wanted, and anything not ACKed is sent again: immediately after a few repeated ACKs, or after a timer if the path goes quiet.
- Later packets can already be at the receiver; the application does not see them until the gap is filled.
- The receiver can also list chunks it already has after the hole, so those are not resent.
Three duplicate ACKs are the fast path. The timer is the slow, painful one.
What is the difference between flow control and congestion control?
Flow control protects the other process's buffer: the receiver advertises how much space it has. Congestion control protects the path: the sender starts small, grows, and cuts back when packets go missing. In flight is the minimum of the two.
- If flow control fails, one process is overrun. If congestion control fails, a router drops packets for everyone on that link.
- A zero window means the application stopped reading, not that the network died.
gRPC and HTTP/2 add another window on top; TCP's two windows still sit underneath and can block every stream at once.
How does the sliding window work, in one picture?
The sender may transmit from the first unacked byte up to that byte plus the advertised receive window; ACKs slide the left edge, a larger window slides the right edge, and congestion control is a second cap on the same region.
- In flight is the middle: sent, not yet ACKed.
- If the window is zero, the sender pauses until the receiver opens it again.
You never send the whole file at once. You send a window, wait, slide, repeat.
How does congestion control behave, without naming the kernel algorithm?
Start with a small window, grow it quickly while ACKs come back, then more slowly; if packets go missing, cut the window and probe again.
- The fast growth at the beginning is why a new connection takes a few round trips to fill a fat pipe.
- A timeout is the ugly path: the sender treats the path as nearly dead and starts small again.
The kernel has a named algorithm for the exact curve. The story you need is probe, grow, back off.
How do Nagle and delayed ACK interact, and when do you set TCP_NODELAY?
Nagle holds a small write while data is unacked, delayed ACK waits up to about 200 ms to send a pure ACK, and together they stall a chatty protocol; set TCP_NODELAY on RPCs, leave Nagle on for bulk transfers.
- Each optimisation is fine alone. The bug is both at once on small request/response traffic.
- Redis and gRPC disable Nagle by default.
If a trace looks like "one RTT plus 200 ms," this is the first thing to check.
What is TCP head-of-line blocking, and why did HTTP/3 leave TCP?
TCP is one ordered stream, so a lost packet stalls every byte behind it; HTTP/3 moved to QUIC over UDP so a loss stalls only its own stream.
- HTTP/2 multiplexed many requests on one TCP connection and still hit this.
- That is a property of TCP, not of HTTP.
The edge often speaks HTTP/3 to browsers and older HTTP to origins, where the extra wait is small.