Linux systems · 4 of 5

Sockets, IPC and network interfaces

The socket API, Unix domain sockets, epoll, pipes, shared memory and message queues, and how the kernel sees interfaces, routes and packet filters.

Updated 2026-09-08

On this page

Processes are walled off from each other on purpose. This page is about the doors the kernel provides: sockets for talking across a network or across the same machine, pipes and shared memory for processes that live together, and, underneath the sockets, the interfaces, routes and filters that decide where a packet goes.

The map

Read this first when short on time. Every branch is a section below.

Figure 1. The whole article on one page. Every branch is a section below; fold what you know, open what you do not.

Sockets: a file descriptor with a protocol behind it

A socket is an endpoint for communication that the kernel hands to you as a file descriptor. You read and write it like a file. What is on the other end, a program across the internet or one on the same machine, depends on the address family you asked for.

socket(family, type, protocol) creates one. The family is the address kind: AF_INET for IPv4, AF_INET6 for IPv6, AF_UNIX for a path on this machine. The type is the shape of the conversation: SOCK_STREAM for a reliable byte stream (TCP), SOCK_DGRAM for separate messages (UDP). What TCP itself does with that stream, the handshake, the windows and the retransmits, is in the TCP article. This page is about the API around it.

sequenceDiagram
  autonumber
  participant S as Server
  participant K as Kernel
  participant C as Client
  S->>K: socket, bind(0.0.0.0:8080), listen(128)
  S->>K: accept, blocks
  C->>K: socket, connect(server:8080)
  K->>K: TCP handshake, connection queued
  K-->>S: accept returns fd 5 for this client
  C->>K: write("GET / ...")
  K-->>S: read on fd 5 returns the bytes
  S->>K: write(fd 5, response)
  K-->>C: read returns the response
  S->>K: close(5)

Figure 2. One connection, both sides. The listening socket never carries data; every accepted client gets its own descriptor.

The server side

socket makes the descriptor.
bind gives it a local address and port. 0.0.0.0 means every interface; 127.0.0.1 means only this machine. Ports below 1024 need root or CAP_NET_BIND_SERVICE.
listen marks it as accepting connections and sets the backlog, how many completed handshakes may wait before the server picks them up. The kernel caps it at net.core.somaxconn, 4096 on current kernels. When the queue is full, new clients see a slow connect or a reset.
accept blocks until a connection is ready and returns a new descriptor for it. The listening socket stays as it was. A server with a thousand clients has a thousand and one sockets.
read and write (or recv and send, which take flags) move bytes. close ends the connection, sending TCP's FIN.

The client side

The client calls socket and then connect with the server's address. The kernel picks a local ephemeral port from the range in net.ipv4.ip_local_port_range (32768 to 60999 by default), does the handshake, and returns when the connection is open. Then it is read and write like the server.

import socket

srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)   # reuse the port right after a restart
srv.bind(("0.0.0.0", 8080))
srv.listen(128)

while True:
    conn, addr = srv.accept()          # a new socket per client
    with conn:
        data = conn.recv(4096)         # up to 4096 bytes, may be a partial message
        conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")

Two options come up constantly. SO_REUSEADDR lets a server bind a port that still has connections in TIME-WAIT from its previous run, so a restart does not fail with "address already in use". SO_REUSEPORT lets several processes bind the same port and has the kernel spread connections across them, which is how nginx and Envoy scale accept across workers.

UDP is the same calls minus the connection: socket with SOCK_DGRAM, bind, then recvfrom and sendto with an address on every call. No listen, no accept, and each call is one datagram.

In the wild

ss -tlnp lists every listening TCP socket with its process; ss -tnp lists the connected ones. It reads /proc/net/tcp and the netlink interface.
Postgres listens on TCP 5432 and, at the same time, on a Unix socket in /var/run/postgresql; local clients get the socket.
A full backlog is a classic outage: the server is alive but slow to accept, the queue fills, and clients see connection timeouts. ss -ltn shows the queue length against the limit in its Recv-Q and Send-Q columns for listening sockets.

Watch out

A stream socket has no message boundaries. One send of 1000 bytes may arrive as several recvs, and two sends may arrive as one. Every protocol on TCP frames its own messages, with a length prefix or a delimiter, and every client must loop on recv until it has a whole one.

Socket: A communication endpoint exposed as a file descriptor.
Address family: The kind of address a socket uses: IPv4, IPv6, or a Unix path.
Backlog: The queue of completed connections waiting for accept.
Ephemeral port: A temporary local port the kernel picks for an outgoing connection.

Unix domain sockets: the same API on a path

A Unix domain socket is a socket whose address is a path in the filesystem instead of an IP and port. Two processes on the same machine talk through it with the same calls, and the kernel skips the whole network stack because the data never leaves memory.

	TCP on 127.0.0.1	Unix domain socket
Address	IP and port	A path such as `/run/app.sock`, or an abstract name starting with a NUL byte
Path taken	Full TCP/IP stack, loopback interface	Kernel buffer to kernel buffer
Speed	Fast	Faster, lower latency, fewer copies
Access control	Anyone who can reach the port	The socket file's permission bits, plus the peer's uid on request
Reachable from	Any machine, if bound wider than loopback	This machine only, and only inside the same mount namespace
Extras	—	Can pass open file descriptors and credentials

Two abilities are unique to it. SCM_RIGHTS lets one process send an open file descriptor to another through the socket; the receiver gets its own descriptor for the same open file, as if it had called open itself. That is how a privileged helper can open a port or a file and hand it to an unprivileged worker. SO_PEERCRED tells a server the user ID, group ID and PID of the process on the other end, verified by the kernel, so a daemon can allow or refuse based on who connected without any password.

Because the socket is a file, chmod and chown on it control who can connect. That is the entire security model of the Docker socket: being in the docker group means being able to write /run/docker.sock, which means being able to start a privileged container, which means root.

In the wild

Docker listens on /run/docker.sock; the CLI is an HTTP client speaking to it over the Unix socket.
systemd talks to services over Unix sockets, journald collects logs through one, and D-Bus, the desktop and system message bus, runs on one too.
Wayland and X11 connect applications to the display server over a Unix socket, and pass GPU buffers as file descriptors with SCM_RIGHTS.
MySQL and Postgres clients use the socket automatically when connecting to "localhost" and TCP when given an IP.

Unix domain socket: A local socket addressed by a filesystem path. AF_UNIX.
SCM_RIGHTS: The message type that carries file descriptors between processes over a Unix socket.
SO_PEERCRED: A socket option that returns the connected peer's uid, gid and PID.

Handling many connections: blocking, epoll and io_uring

A plain read on a socket blocks until data arrives. With one client that is fine. With ten thousand, the server needs a way to wait on all of them at once and be told which are ready. That is what epoll does.

flowchart TD
  A["epoll_create1: one descriptor for the whole set"] --> B["epoll_ctl ADD: register each socket with the events you want"]
  B --> C["epoll_wait: sleep until at least one is ready"]
  C --> D["Returns only the ready descriptors"]
  D --> E["read or write each one without blocking"]
  E --> C

Figure 3. The event loop every modern server runs. The cost of a wait does not grow with the number of idle connections, because the kernel tracks readiness as it happens.

The old way was a thread per connection: simple to write, and each thread costs a stack and a context switch whenever it wakes. The older waiting calls, select and poll, take the whole list of descriptors on every call and scan it, so they slow down as the list grows. epoll keeps the list in the kernel. You register a descriptor once, and epoll_wait returns just the ones with something to do. Ten thousand idle connections cost nothing per wait.

Sockets in the loop are set non-blocking (O_NONBLOCK) so a read with nothing to read returns EAGAIN instead of stalling the loop. epoll has two modes. Level-triggered, the default, reports a descriptor every time it is ready, so if you read half the data you are told again. Edge-triggered reports only on the change, so you must drain the socket until EAGAIN or you will not hear about it again. Edge-triggered saves wakeups and is easier to get wrong.

io_uring, added in kernel 5.1, goes a step further: instead of being told what is ready and then making a call per operation, the program puts operations (read this, write that, accept) into a ring buffer shared with the kernel and picks up completions from another ring. Many operations per system call, and it covers disk I/O too, which epoll never did.

In the wild

nginx runs one worker per CPU, each an epoll loop, which is how a small machine serves tens of thousands of connections.
Node.js is one epoll loop (through libuv) plus a thread pool for the things that cannot be made non-blocking, such as file I/O and DNS.
Go hides epoll inside the runtime: a goroutine blocked on a socket is parked, the netpoller waits with epoll, and you write blocking-looking code that scales.
Redis is a single epoll thread for commands, which is why one slow command stalls every client.

Watch out

Anything that blocks inside an epoll loop blocks every connection: a synchronous DNS lookup, a disk read, a slow log write, a lock. The symptom is all clients stalling together. That is why event-loop runtimes push such work to a thread pool.

epoll: The Linux call for waiting on many descriptors, with the set kept in the kernel.
Non-blocking: A descriptor that returns EAGAIN instead of waiting when it cannot proceed.
Edge-triggered: Notify only when readiness changes. The reader must drain the descriptor.
io_uring: Submission and completion rings shared with the kernel for batched asynchronous I/O.

Pipes and FIFOs: a byte stream between two processes

A pipe is a one-way byte stream held in a kernel buffer, with a write end and a read end, each a file descriptor. It is the oldest form of interprocess communication on Unix and still the most used, because every shell pipeline is one.

flowchart TD
  P["Shell: pipe() gives fd 3 read, fd 4 write"] --> F1["fork: child 1"]
  P --> F2["fork: child 2"]
  F1 --> D1["dup2(4, 1): stdout is the pipe. exec ls"]
  F2 --> D2["dup2(3, 0): stdin is the pipe. exec grep"]
  D1 -- writes --> B["Kernel buffer, 64 KB"]
  D2 -- reads --> B

Figure 4. ls | grep foo. The shell creates the pipe, forks twice, and each child wires one end onto its standard input or output before exec. The programs never know.

pipe() returns two descriptors. Whatever is written to the second can be read from the first, in order. The buffer is 64 KB by default. A writer with a full buffer blocks until the reader catches up, which is the back-pressure that keeps cat hugefile | slowprogram from using all the memory. A reader with an empty buffer blocks until something arrives, or gets end-of-file once every write end has been closed. Writing to a pipe whose read ends have all closed sends SIGPIPE, which kills the writer by default; that is how yes | head ends.

A pipe has no name, so only processes that inherit the descriptors, through fork, can use it. A FIFO, or named pipe, is the same buffer with a name in the filesystem, made with mkfifo. Any process that can open the path can use it. Opening one end blocks until the other end is opened too.

Writes of up to PIPE_BUF bytes (4096) are atomic: several writers can share a pipe and their small messages will not interleave. Larger writes can.

In the wild

Every shell pipeline is pipe, fork, dup2, exec. strace -f sh -c 'ls | wc -l' shows all of it.
Git hooks and CGI pass data to a child program through its stdin, which is a pipe from the parent.
logger and log shippers read a FIFO that an application writes to, so the application does not need to know about syslog.
xargs and parallel read a pipe and spawn workers, so a pipeline can fan out.

Pipe: A one-way in-kernel byte buffer with a read descriptor and a write descriptor.
FIFO: A pipe with a filesystem name, so unrelated processes can share it.
SIGPIPE: The signal sent to a process writing to a pipe or socket with no reader.

Shared memory, message queues and semaphores

Pipes and sockets copy every byte through the kernel. Shared memory does not: two processes map the same physical pages and read and write them directly. It is the fastest way to move data between processes and the only one where the kernel is not involved in each transfer, which also means the kernel is not there to keep them from stepping on each other.

Shared memory

The modern way is shm_open("/name", ...), which creates a file under /dev/shm (a tmpfs), then ftruncate to size it and mmap with MAP_SHARED in each process. The pages are the same in both address spaces, as the memory article described. memfd_create does the same without a name, and the descriptor can be passed over a Unix socket. Mapping an ordinary file with MAP_SHARED works too, and the page cache is the shared memory. The old System V calls (shmget, shmat) still exist and Postgres used them for decades.

Because writes are just memory stores, two processes updating the same structure at once will corrupt it. Something has to order them. A semaphore is a counter the kernel protects: sem_wait blocks until it is positive and decrements it, sem_post increments it. A mutex placed inside the shared region with PTHREAD_PROCESS_SHARED works across processes. Underneath both is futex, a system call that only runs when there is contention, so the uncontended case is a single atomic instruction in user space and never enters the kernel.

Message queues

A POSIX message queue (mq_open, mq_send, mq_receive) is a kernel-held queue of discrete messages, each with a priority, delivered highest priority first. Unlike a pipe it preserves message boundaries, and the receiver can be told when a message arrives through a signal or a thread. The queue appears under /dev/mqueue. System V message queues (msgget) are the older equivalent with a stranger API. Both are little used today; a Unix socket with a small framing protocol does the same job with tools that understand it.

A few small descriptors round out the set. eventfd is a counter you can wait on with epoll, used to wake an event loop from another thread. signalfd delivers signals as bytes on a descriptor. timerfd does the same for timers. Together they let one epoll loop handle sockets, signals, timers and cross-thread pokes with one wait.

In the wild

PostgreSQL keeps its buffer pool in shared memory that every backend process maps, protected by lightweight locks built on atomics and futexes.
Chrome moves rendered frames between renderer, GPU and browser processes through shared memory, with the descriptors passed over Unix sockets.
Android's Binder is an IPC driver of its own, but large payloads still travel as shared memory (ashmem, now memfd) referenced from the Binder message.
Every pthread mutex in every program is a futex; strace on a contended lock shows futex(...) calls, and on an uncontended one shows nothing.

Watch out

Shared memory outlives the process that created it. A crash leaves the segment in /dev/shm with whatever half-written state it had, and the next start reads it. Name segments carefully, unlink them on clean exit, and treat the contents as untrusted on start. ipcs and ls /dev/shm show what is lying around.

Shared memory: Physical pages mapped into more than one process, so writes are visible without copying.
Semaphore: A kernel-protected counter for ordering access between processes.
futex: Fast user space mutex. The system call locks use only when they actually have to wait.
Message queue: A kernel queue of discrete, prioritised messages.

Choosing an IPC mechanism

The choice comes down to the shape of the data, whether the two sides are related, and how much of the kernel you want in between.

Mechanism	Shape	Between	Kernel copies	Use it for
Pipe	One-way byte stream	Parent and child	Yes	Feeding a child program, shell pipelines
FIFO	One-way byte stream	Anyone with the path	Yes	A pipe between unrelated programs
Unix socket	Two-way stream or datagrams	Anyone with the path	Yes	Local client and server, fd passing, most daemons
TCP socket	Two-way byte stream	Any machine	Yes	Anything that may move to another host
Shared memory	A region of bytes	Anyone who maps it	No	Large or high-rate data: frames, buffers, caches
Message queue	Discrete prioritised messages	Anyone with the name	Yes	Rare today; small command messages
Signal	One number, no data	Anyone with permission	—	Stop, reload, notify. Described in the processes article
eventfd	A counter	Threads, or via fd passing	—	Waking an event loop

The usual answer for a new local service is a Unix socket: it is two-way, it works with epoll, it has permissions, and every language has a client. Pipes win when one process spawns the other. Shared memory wins only when copying is measurably the bottleneck, and it costs you a locking design.

In the wild

D-Bus is a message bus on a Unix socket that systemd, NetworkManager and the desktop use to call each other's methods.
gRPC and HTTP over TCP are used even between processes on one host when the services might be split across machines later; the loopback cost is accepted for the flexibility.
The Docker CLI and kubectl's local proxies are HTTP over a Unix socket, an ordinary protocol on a local transport.

Interfaces: how the kernel sees a network card

A network interface is the kernel's object for one way in and out of the machine: a physical card, or a virtual device that behaves like one. Sockets sit on top; interfaces are where packets actually leave.

$ ip -brief link
lo        UNKNOWN  00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp3s0    UP       3c:7c:3f:12:ab:cd <BROADCAST,MULTICAST,UP,LOWER_UP>
docker0   UP       02:42:9a:1f:00:01 <BROADCAST,MULTICAST,UP,LOWER_UP>
veth1a2b  UP       6e:11:2f:8b:44:0e <BROADCAST,MULTICAST,UP,LOWER_UP>

$ ip -brief addr
lo        UNKNOWN  127.0.0.1/8 ::1/128
enp3s0    UP       192.168.1.20/24 fe80::3e7c:3fff:fe12:abcd/64
docker0   UP       172.17.0.1/16

lo is the loopback: packets to 127.0.0.1 go out through it and straight back in without touching hardware. enp3s0 is the Ethernet card; the name encodes where it is on the bus (PCI bus 3, slot 0) so it stays stable across reboots, replacing the old eth0 that could swap with another card. Wireless cards show as wlp2s0 or wlan0.

Each interface has a MAC address, an MTU (1500 bytes on Ethernet; the biggest packet it will send), a state (UP, and LOWER_UP when a cable is actually connected), and counters. ip -s link and /sys/class/net/enp3s0/statistics/ show bytes, packets, errors and drops. ethtool talks to the driver for link speed and hardware offload settings.

Virtual interfaces

The kernel can create interfaces that have no hardware at all, and containers and VPNs are built out of them.

flowchart TD
  subgraph C1["Container A"]
    E1["eth0 172.17.0.2"]
  end
  subgraph C2["Container B"]
    E2["eth0 172.17.0.3"]
  end
  E1 --- V1["veth1a2b"]
  E2 --- V2["veth3c4d"]
  V1 --- BR["docker0 bridge 172.17.0.1"]
  V2 --- BR
  BR --> NAT["Netfilter: masquerade"]
  NAT --> P["enp3s0 192.168.1.20"]
  P --> I["Internet"]

Figure 5. Docker's default network. Each container's eth0 is one end of a veth pair; the other end is plugged into the docker0 bridge on the host, and NAT rewrites the source address on the way out.

Type	What it is	Used by
veth	A pair of interfaces joined by a virtual cable; what goes in one comes out the other	Every container's network connection
bridge	A virtual switch; interfaces plugged into it can reach each other at layer 2	`docker0`, `virbr0` for VMs, `cni0` in Kubernetes
tun / tap	An interface whose other end is a user space program (tun carries IP packets, tap carries Ethernet frames)	OpenVPN, Tailscale, QEMU
wireguard	An in-kernel encrypted tunnel interface	`wg0`
bond	Several physical interfaces acting as one, for redundancy or bandwidth	Servers with two uplinks
vlan	A tagged sub-interface, `enp3s0.100`, on one physical link	Data centres
macvlan / ipvlan	Extra MAC or IP addresses on a physical interface, no bridge needed	Containers that need their own LAN address

A packet leaving a socket goes: socket buffer, IP layer (pick a route, choose the interface), netfilter hooks, the interface's queue, the driver, the wire. Arriving is the reverse, with the card raising an interrupt and the driver pulling frames off a ring buffer as the interrupts section described. The interface counters show drops on either path, and tcpdump -i enp3s0 taps in just above the driver.

In the wild

Docker creates a veth pair per container and plugs one end into docker0; docker network create makes a new bridge.
Kubernetes delegates this to a CNI plugin, which does the same veth-and-bridge trick (Flannel, Calico) or replaces it with eBPF routing (Cilium).
Tailscale and WireGuard appear as an ordinary interface with an address, and the rest of the system routes to it like any other.
Cloud VMs see a virtual NIC (virtio-net or ENA) that is itself a veth-like device into the hypervisor's switch.

Interface: The kernel object for one network device, physical or virtual.
MTU: Maximum transmission unit. The largest packet an interface sends. 1500 on Ethernet.
veth: Virtual Ethernet. A pair of interfaces connected back to back, used to connect namespaces.
Bridge: A virtual switch inside the kernel.

Routing and name resolution: where a packet goes

Once a packet has a destination IP, the kernel consults the routing table to pick an interface and a next hop, and then ARP to turn that next hop's IP into a MAC address. Before any of that, the program had to turn a name into an IP.

flowchart TD
  A["connect to api.example.com:443"] --> N["Resolve the name: /etc/hosts, then DNS from /etc/resolv.conf"]
  N --> R["Routing table: longest prefix match on the IP"]
  R --> L{"Local subnet?"}
  L -- yes --> H["Next hop is the destination itself"]
  L -- no --> G["Next hop is the gateway from the default route"]
  H --> M["ARP: what MAC has that IP?"]
  G --> M
  M --> O["Send the frame out of the chosen interface"]

Figure 6. From a name to a frame on the wire. Every step has a cache: nsswitch and the resolver, the route cache, the neighbour table.

The routing table

$ ip route
default via 192.168.1.1 dev enp3s0 proto dhcp metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.1.0/24 dev enp3s0 proto kernel scope link src 192.168.1.20

Each line says: for this prefix, use this interface, and send to this gateway if there is one. The kernel picks the longest prefix that matches the destination. 192.168.1.50 matches the /24 line, so it goes straight out of enp3s0 with no gateway. 8.8.8.8 matches nothing but default, so the frame is addressed to the router at 192.168.1.1, which forwards it on. The scope link lines were added automatically when the addresses were configured.

A machine forwards packets between its interfaces only if net.ipv4.ip_forward is 1. It is 0 on a laptop and 1 on a router, and Docker turns it on because the host must route between docker0 and the outside.

Neighbours

Ethernet delivers to MAC addresses, so the kernel needs the MAC of the next hop. ARP broadcasts "who has 192.168.1.1?" and caches the answer in the neighbour table; ip neigh shows it, with entries aging out after a while. IPv6 does the same with Neighbour Discovery. A wrong or stale entry is the classic cause of "it pings from one machine but not the other".

Names

Resolution is not in the kernel. libc's getaddrinfo reads /etc/nsswitch.conf to learn the order (usually files then dns), checks /etc/hosts, then sends DNS queries to the servers in /etc/resolv.conf. On systemd machines that file usually names 127.0.0.53, a stub run by systemd-resolved that caches and forwards to the real servers; resolvectl status shows which. Inside a Docker container, /etc/resolv.conf points at Docker's own DNS, which answers with other containers' names on the same network.

In the wild

AWS VPC route tables are the same idea one level up: a prefix, a target (internet gateway, NAT gateway, peering), longest prefix wins.
Kubernetes gives every pod an IP and makes routes for them on each node; in the simplest setups, ip route on a node lists one line per other node's pod range.
Alpine's musl resolver behaves differently from glibc's: it queries all servers in parallel and handles search domains and ndots differently, a recurring source of "DNS works everywhere except in this container".

Route: A rule mapping a destination prefix to an interface and optional gateway.
Default route: The route for everything that matches nothing more specific.
ARP: Address Resolution Protocol. Finds the MAC address for an IP on the local network.
nsswitch: The libc configuration for where names (hosts, users, groups) are looked up and in what order.

Netfilter: the kernel's packet filter

Netfilter is a set of hooks in the network stack where rules can inspect, drop, or rewrite packets as they pass. Firewalls, NAT and port forwarding are all rules attached to these hooks.

There are five hooks, and a packet passes through some of them depending on where it is going:

prerouting: just arrived, before the routing decision. Where destination NAT happens.
input: addressed to this machine, on its way to a local socket. Where the firewall for local services lives.
forward: passing through this machine to another. Router and Docker rules.
output: created by a local process, on its way out.
postrouting: about to leave, after routing. Where source NAT (masquerade) happens.

Rules are managed with nftables (the nft command) on current distributions, or the older iptables, which on most systems is now a compatibility front end that writes nftables rules. Both organise rules into tables (filter, nat) and chains attached to hooks. A rule matches on addresses, ports, interface, connection state, and says accept, drop, reject, or jump to another chain.

# A minimal host firewall in nftables
nft add table inet filter
nft add chain inet filter input '{ type filter hook input priority 0; policy drop; }'
nft add rule inet filter input ct state established,related accept
nft add rule inet filter input iif lo accept
nft add rule inet filter input tcp dport { 22, 80, 443 } accept
nft list ruleset

# What Docker adds for  -p 8080:80  (shown in iptables form)
iptables -t nat -A DOCKER -p tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:80
iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

The ct state established,related rule is the one that makes stateful firewalls possible. conntrack keeps a table of every connection it has seen, so a reply packet is recognised as belonging to something already allowed and does not need its own rule. conntrack -L lists the table; on a busy NAT box it can hold hundreds of thousands of entries, and running out of room (nf_conntrack_max) drops new connections silently.

NAT rewrites addresses. Masquerade in postrouting replaces a container's private source address with the host's, and conntrack remembers the mapping so replies can be rewritten back. DNAT in prerouting sends traffic arriving on the host's port 8080 to the container's port 80. That pair of rules is all a "published port" is.

In the wild

Docker writes the rules above on every -p; iptables -t nat -L -n on a Docker host shows one DNAT line per published port.
Kubernetes kube-proxy in its default mode implements every Service as a set of NAT rules that pick a pod at random, which is why a cluster with thousands of Services has tens of thousands of rules and why IPVS and eBPF modes exist.
ufw and firewalld are front ends that generate these rules from simpler commands.
Cloudflare and Facebook filter and load-balance with XDP, an eBPF hook that runs in the driver before netfilter is even reached; the tracing article introduces eBPF.

Watch out

Docker's rules are inserted ahead of yours. A port published with -p 3306:3306 is reachable from the whole network even if a host firewall rule says otherwise, because the DNAT happens in prerouting before the input chain sees it. Bind published ports to 127.0.0.1:3306:3306 when they are meant to be local.

Netfilter: The kernel's packet filtering framework: five hooks in the network stack that rules attach to.
nftables: The current rule language and tool for netfilter. iptables is the older one.
conntrack: Connection tracking. The table that lets rules match on connection state and lets NAT reverse its rewrites.
NAT: Network address translation: rewriting source (masquerade) or destination (DNAT) addresses in flight.

Recap

A socket is a file descriptor with a protocol behind it. Servers do socket, bind, listen, accept; clients do socket, connect; both then read and write.
accept returns a new descriptor per client; the listening socket only queues connections, up to the backlog.
Stream sockets have no message boundaries; frame your own.
A Unix domain socket is the same API on a path: faster, local, protected by file permissions, and able to pass descriptors and prove the peer's identity.
epoll waits on thousands of descriptors and returns only the ready ones. Sockets in the loop are non-blocking. Anything that blocks in the loop stalls every client.
A pipe is a one-way 64 KB kernel buffer; the shell wires one between processes with fork and dup2. A FIFO is a pipe with a name.
Shared memory maps the same pages into several processes: no copies, but you must lock. Futexes make uncontended locks free.
Default choice for a local service: a Unix socket. Pipes for parent and child. Shared memory when copying is proven to be the bottleneck.
An interface is one way in and out. veth pairs and bridges connect containers; tun, tap and wireguard connect VPNs.
The routing table picks an interface and gateway by longest prefix; ARP turns the next hop into a MAC; libc, not the kernel, resolves names.
Netfilter has five hooks. nftables rules on them make firewalls; conntrack makes them stateful; masquerade and DNAT make published container ports.
Docker's NAT rules run before the host's input firewall. Bind local-only ports to 127.0.0.1.

Questions

Try answering each one out loud before opening it. Lead with the one-line answer, then a couple of points, then one extra detail.

Walk through the system calls a TCP server makes to serve one client.

socket creates the descriptor, bind attaches the address and port, listen makes it accept connections with a backlog, accept blocks and returns a new descriptor for the client, then read and write on that descriptor, and close.

The listening socket never carries data; each client gets its own.
The client side is socket and connect, with the kernel choosing an ephemeral port.

SO_REUSEADDR before bind is what lets a restarted server take a port still holding TIME-WAIT connections.

What is the listen backlog and what happens when it fills?

The queue of connections whose handshake has completed but which the server has not yet accepted; when it is full the kernel stops completing new handshakes, so clients see slow connects or resets even though the server is up.

The value passed to listen is capped by net.core.somaxconn.
ss -ltn shows the current queue against the limit for each listening socket.

A full backlog usually means the accept loop is stalled, not that the machine is out of capacity.

When would you use a Unix domain socket instead of TCP on localhost?

Whenever both ends are on the same machine and will stay there: it skips the network stack, is faster with lower latency, is protected by file permissions, and can pass file descriptors and the peer's verified uid.

Docker, systemd, Postgres, MySQL, Wayland and D-Bus all use one.
It is invisible from other machines and from other mount namespaces, which is a feature.

Choose TCP when the other side might move to another host later.

How can one process give another an open file descriptor?

Send it as ancillary data with SCM_RIGHTS over a Unix domain socket; the receiver gets a new descriptor number pointing at the same open file description.

Used by privilege separation: a root helper opens the resource and hands it to an unprivileged worker.
Display servers pass GPU buffers this way, and systemd passes listening sockets to services on activation.

The descriptor can be for anything: a file, a socket, a memfd of shared memory.

Why does epoll scale where select and poll do not?

select and poll pass the whole descriptor list on every call and the kernel scans it, so cost grows with the number of connections; epoll registers descriptors once, tracks readiness as events happen, and each wait returns only the ready ones.

Idle connections cost nothing per wait.
Descriptors are non-blocking so a read with no data returns EAGAIN instead of stalling the loop.

Edge-triggered mode cuts wakeups further but requires draining each descriptor until EAGAIN.

What is the risk of blocking inside an event loop?

Every connection stalls at once, because the single thread that would service them is stuck in the blocking call: a synchronous DNS lookup, a disk read, a lock, a slow log write.

Node.js and libuv push file I/O and DNS to a thread pool for this reason.
Redis's single-threaded command loop means one slow command delays all clients.

Go avoids the problem by parking goroutines on the netpoller instead of blocking OS threads.

How does the shell implement ls | grep foo?

It calls pipe to get a read and write descriptor, forks twice, and in each child uses dup2 to put the write end on stdout of ls and the read end on stdin of grep, then execs; the programs just read and write descriptors 0 and 1.

The kernel buffer is 64 KB; a full buffer blocks the writer, giving back-pressure.
The reader sees EOF only when every write end is closed, so the shell must close its own copies.

Writing to a pipe with no reader raises SIGPIPE, which is how yes | head terminates.

Pipe versus FIFO?

Same kernel buffer and same rules, but a pipe has no name and is shared only by inheritance across fork, while a FIFO is created with mkfifo as a path that any process with permission can open.

Opening a FIFO blocks until the other end is opened too.
Writes up to 4096 bytes are atomic on both, so several writers can share one.

Both are one-way; two-way needs two of them or a socket.

How does shared memory work between processes, and what does it cost you?

Each process maps the same physical pages with mmap and MAP_SHARED on a file, a shm_open object under /dev/shm, or a memfd, so a write in one is immediately visible in the other with no copy; the cost is that you must provide all the synchronisation yourself.

Semaphores, process-shared mutexes and atomics order the access; all block through futex only when contended.
The segment outlives a crashed process and holds its half-written state.

Postgres's buffer pool and Chrome's frame passing are both shared memory with descriptors passed over Unix sockets.

What is a futex?

The fast user space mutex system call: locks do an atomic compare-and-swap in user space and only call futex to sleep or wake when there is actual contention, so an uncontended lock never enters the kernel.

Every pthread mutex, condition variable and most language runtimes' locks sit on it.
strace on a lock-heavy program shows futex calls only when threads are actually waiting.

Process-shared mutexes work because the futex is keyed on the physical page, so shared memory works too.

Given a new local service, which IPC would you pick and why?

A Unix domain socket: two-way, works with epoll and every language, protected by file permissions, can pass descriptors, and any protocol that works on TCP works on it unchanged.

Pipes only when one process spawns the other and one direction is enough.
Shared memory only when measurement shows copying is the bottleneck, and then with a careful locking design.

Using TCP on loopback is a reasonable choice when the service might be split across hosts later.

What is a veth pair, and how does Docker use it?

Two virtual interfaces joined back to back so a frame sent into one comes out the other; Docker puts one end inside the container's network namespace as eth0 and plugs the other into the docker0 bridge on the host.

The bridge is a virtual switch, so containers on it reach each other directly.
Traffic to the outside is masqueraded to the host's address in postrouting.

Kubernetes CNI plugins do the same, or replace the bridge with eBPF routing.

How does the kernel decide which interface a packet leaves on?

It matches the destination IP against the routing table and takes the longest matching prefix; that route names the interface and, if the destination is not directly connected, the gateway to send to.

The default route catches everything with no more specific match.
ARP then turns the next hop's IP into a MAC for the frame.

Forwarding between interfaces needs net.ipv4.ip_forward=1, which Docker enables on the host.

Where does DNS resolution happen on Linux?

In libc, not the kernel: getaddrinfo follows /etc/nsswitch.conf, checks /etc/hosts, then queries the servers in /etc/resolv.conf, which on systemd machines is usually the local stub at 127.0.0.53.

Inside a container the file points at the runtime's DNS, which resolves other containers' names.
musl and glibc resolvers behave differently, a common source of container-only DNS problems.

Because it is a library call, it blocks the calling thread, which is why event loops move it to a thread pool.

Explain how docker run -p 8080:80 works at the packet level.

Docker adds a DNAT rule in the prerouting hook that rewrites packets arriving on the host's port 8080 to the container's IP and port 80, and a masquerade rule in postrouting so the container's outbound traffic carries the host's address; conntrack remembers both mappings to rewrite the replies.

The rules live in the nat table and are visible with iptables -t nat -L -n.
Because DNAT runs before the input chain, a host firewall does not block the published port.

Publishing on 127.0.0.1:8080:80 keeps the port local.

What does conntrack do, and what breaks when its table is full?

It tracks every connection passing through netfilter so rules can match on state (established, related) and NAT can reverse its rewrites on replies; when the table hits nf_conntrack_max, new connections are dropped silently.

The symptom is intermittent connection failures on a busy NAT host or Kubernetes node with plenty of CPU.
conntrack -L | wc -l and dmesg ("nf_conntrack: table full") confirm it.

Raising the limit costs memory per entry; shortening timeouts for closed connections helps too.