Namespaces, cgroups and tracing

There is no such thing as a container in the Linux kernel. There are ordinary processes, a handful of features that change what a process can see, and another that limits what it can use. Put them together and you have Docker. This page takes them apart, then turns to the tools that let you watch any of it happening.

The map

Read this first when short on time. Every branch is a section below.

A container is a process with a narrower view

A container is an ordinary process, running on the host's kernel, that has been given a restricted view of the system and a cap on what it may consume. Nothing about the process itself is special. ps on the host shows it with a PID like any other.

flowchart TD
  P["One process, started by the runtime"] --> N["Namespaces: own PIDs, mounts,<br/>network, hostname, users"]
  P --> C["Cgroup: at most 2 CPUs,<br/>512 MB, 1000 processes"]
  N --> R["Root filesystem: image layers<br/>stacked with overlayfs"]
  C --> S["Capabilities dropped,<br/>seccomp filter on system calls"]
  R --> K["The host kernel, shared with every other container"]
  S --> K

• Namespaces change what the process sees: which other processes exist, what is mounted where, which network interfaces are there, what the hostname is, who root is.
• Cgroups limit what the process and its children can use: CPU time, memory, disk bandwidth, number of processes.
• A root filesystem built from the image's layers, so / inside is the image and not the host.
• Fewer privileges: most of root's capabilities dropped, and a seccomp filter that refuses dangerous system calls.

What a container is not is a virtual machine. There is one kernel. A kernel bug reachable from inside a container is reachable from the host. That is why a VM is still the boundary between customers at a cloud provider, and why Firecracker and gVisor exist for people who want to run untrusted code with container convenience. For your own services on your own machines, the container boundary is plenty.

Container

A process isolated by namespaces, limited by cgroups, given its own root filesystem and reduced privileges. Not a kernel object.

Runtime

The program that sets all of that up and starts the process: runc, crun, and the higher-level containerd and CRI-O.

Namespaces: eight ways to narrow what a process sees

A namespace wraps one global kernel resource so that a group of processes gets its own private copy of it. Processes inside see only their copy and believe it is the whole thing. There are eight kinds, and a container normally uses six of them.

Creating and joining

A process gets a new namespace at creation, with clone and a CLONE_NEWPID, CLONE_NEWNET flag for each kind, or afterwards with unshare. An existing process can join another's namespace with setns, given a descriptor for it. Those descriptors come from /proc/PID/ns/, where each namespace appears as a symlink whose target is its identifier. Two processes are in the same namespace if the links match.

# A container with nothing but the kernel's own tools
sudo unshare --pid --fork --mount-proc --net --uts --ipc --mount bash
hostname box                 # only this uts namespace changes
ps aux                       # bash is PID 1, nothing else visible
ip link                      # only lo, and it is down

# From the host: see and enter it
lsns                         # every namespace and who is in it
ls -l /proc/4242/ns          # pid:[4026532583] and friends
nsenter -t 4242 -n ss -tlnp  # run ss inside that process's network namespace

That unshare line is most of what docker run does before it execs the image's command. Docker adds the root filesystem and the cgroup, described next.

The pid namespace

flowchart TD
  H1["systemd<br/>host PID 1"] --> H2["containerd-shim<br/>host PID 2100"]
  H2 --> H3
  subgraph NS["Container pid namespace"]
    H3["nginx master<br/>host PID 2115, inside PID 1"] --> H4["nginx worker<br/>host PID 2130, inside PID 7"]
  end

PID namespaces are a tree. A process in a child namespace has a PID in it and in every ancestor namespace, so the host can always see and signal container processes, while the container cannot see out. The first process in a new pid namespace is its PID 1, with everything that implies from the processes article: it must reap orphans, and the kernel ignores signals it has no handler for. When that PID 1 exits, the kernel kills every other process in the namespace. /proc has to be mounted fresh inside (--mount-proc) or ps would read the host's.

The user namespace

The user namespace maps a range of IDs inside to a range outside, written in /proc/PID/uid_map. A process can be uid 0 inside, with full capabilities over things in its own namespaces, while being uid 100000 outside with no power over the host at all. It is the one namespace an unprivileged user may create, which is what makes rootless containers possible: Podman, and Docker in rootless mode, run the whole runtime as a normal user. The cost is that files written to a volume show up on the host owned by the mapped ID, which confuses people until they know why.

Namespace

A private copy of one kernel resource, such as the PID table or the network stack, shared by a group of processes.

unshare

The system call and command that move the calling process into new namespaces.

setns / nsenter

Join an existing namespace by descriptor; nsenter is the command form.

uid_map

The file defining how IDs inside a user namespace map to IDs outside.

The root filesystem: image layers and overlayfs

A container's / is not a copy of the image. It is the image's layers stacked on top of each other with overlayfs, plus one empty writable layer for this container. Starting a container copies nothing, which is why it takes milliseconds.

flowchart TD
  M["merged: what the container sees as /"] --> U["upper: this container's writes, deletes as whiteouts"]
  U --> L3["lower 3: COPY app /app"]
  L3 --> L2["lower 2: apt-get install nginx"]
  L2 --> L1["lower 1: debian base"]

An image is a list of layers, each a tar archive of the files that one Dockerfile step added or changed, plus a JSON config with the command, environment and working directory. Layers are identified by the hash of their contents, so two images built from the same base share the base layer on disk and in the registry. docker image history lists them; /var/lib/docker/overlay2/ holds them unpacked.

The runtime mounts the layers as overlayfs's lowerdir (read-only, several of them), adds a fresh upperdir and workdir for the container, and the merged view becomes the root. Reads of untouched files come straight from the shared lower layers, through the same page cache for every container using that image. The first write to a file from a lower layer copies the whole file up, which is why appending one byte to a 1 GB file in the image is slow the first time, and why databases put their data in a volume rather than the container layer.

Making that directory the root is pivot_root: move the process's root to the merged directory and put the old root somewhere it can be unmounted. The old chroot only changes what / resolves to and can be escaped by a process holding a descriptor from outside; pivot_root inside a mount namespace leaves nothing to escape to.

overlayfs

A filesystem that presents several directories as one, with reads falling through the stack and writes landing in the top one.

Layer

One read-only step of an image: a tar of added and changed files, named by its content hash.

Copy-up

Overlayfs copying a file from a lower layer into the upper one before the first write.

pivot_root

Swap the root filesystem of the current mount namespace. Stronger than chroot.

Cgroups: limits and accounting for groups of processes

A control group is a set of processes with resource limits and counters attached. Namespaces decide what a process sees; cgroups decide how much CPU, memory and I/O it and its descendants may use, and measure what they used.

The cgroup tree is a filesystem, mounted at /sys/fs/cgroup. Every directory is a cgroup, every process is in exactly one, and a child process starts in its parent's. Limits are files: write a number, the kernel enforces it. This is cgroup v2, the single unified tree that every current distribution, Docker and Kubernetes use. v1 had a separate tree per controller and is what older documentation describes.

flowchart TD
  R["/sys/fs/cgroup (root)"] --> S["system.slice: services started by systemd"]
  R --> U["user.slice: login sessions"]
  R --> M["machine.slice or docker: containers"]
  S --> S1["nginx.service"]
  S --> S2["docker.service"]
  M --> C1["container a1b2: cpu.max 200000 100000, memory.max 512M"]
  M --> C2["container c3d4: memory.max 2G, pids.max 1000"]

The controllers

# Make a group by hand, limit it, put a shell in it
sudo mkdir /sys/fs/cgroup/demo
echo "50000 100000" | sudo tee /sys/fs/cgroup/demo/cpu.max    # half a CPU
echo 256M           | sudo tee /sys/fs/cgroup/demo/memory.max
echo $$             | sudo tee /sys/fs/cgroup/demo/cgroup.procs # move this shell in

# Watch it
cat /sys/fs/cgroup/demo/cpu.stat        # nr_throttled, throttled_usec
cat /sys/fs/cgroup/demo/memory.events   # oom_kill count
cat /proc/self/cgroup                   # 0::/demo

# What systemd and Docker show
systemd-cgls                            # the tree with processes
systemd-cgtop                           # top, by cgroup
docker stats                            # per container, from the same files

How the limits bite

CPU and memory fail differently. A group over its CPU quota is throttled: its runnable threads are simply not scheduled until the next period starts. The process is alive and slow, and nothing tells it why. cpu.stat's nr_throttled counter is the evidence. A multi-threaded program with a 1-CPU quota can burn its 100 ms allowance in the first 10 ms of a period across ten threads and then stall for 90 ms, which is the classic cause of a latency spike that shows nothing in the application's own metrics.

A group at its memory limit first has its page cache reclaimed and its anonymous pages swapped if swap is allowed, and if that is not enough, the OOM killer runs inside the group and kills its largest process. The host has plenty free; the container dies with exit code 137. Kernel memory used on the group's behalf, such as socket buffers and the page cache for its files, counts against the limit, which is why a container that only reads big files can be killed for memory.

cgroup

Control group. A node in a tree of process groups with resource limits and counters.

Controller

One resource type the cgroup tree can limit: cpu, memory, io, pids, cpuset.

Throttling

Not scheduling a group's threads once it has used its CPU quota for the period.

Slice

systemd's name for a cgroup used to group services or sessions.

Capabilities and seccomp: fewer superpowers

The last ingredient takes away privileges. Even as root inside a container, a process should not be able to load a kernel module, change the clock, or mount arbitrary filesystems, and it should not be able to reach system calls that exist only to be exploited.

Capabilities, introduced in the filesystems article, split root into named pieces. Docker starts a container with 14 of the roughly 40 (enough to change file ownership, bind low ports, send signals, use raw sockets for ping) and drops the rest, notably CAP_SYS_ADMIN, CAP_SYS_MODULE, CAP_SYS_TIME and CAP_NET_ADMIN. --privileged puts them all back and is roughly equivalent to root on the host. The good default for a service is --cap-drop ALL plus the one or two it needs.

seccomp is a filter on system calls. A process installs a small program (written in BPF, the same technology as eBPF below) that the kernel runs on every system call it makes, and the program says allow, return an error, or kill. Once installed it cannot be removed, and it applies to every child. Docker's default profile refuses about 40 of the 400 or so calls, such as reboot, mount, kexec_load and swapon. Together with a user namespace, that is why an attacker who gets a shell in a container has a much harder time than one who gets a shell on the host.

seccomp

Secure computing mode. A per-process filter that decides which system calls are allowed.

Privileged container

One started with all capabilities and no seccomp filter. Close to root on the host.

The first tools to reach for

Before tracing anything, get the shape of the problem: which resource is busy, which is saturated, and whether anything is reporting errors. The standard tools answer that in a minute, and most of them are reading /proc and /sys.

The pattern is: for every resource, check utilisation, saturation and errors, in that order. Utilisation says it is busy; saturation says work is queueing; errors say it is failing. A CPU at 90 percent with no queue is fine. A disk at 30 percent with an await of 200 ms is not.

The /proc/pressure/ files are the newest and most useful addition: they report the percentage of time in the last 10, 60 and 300 seconds that some or all tasks were stalled waiting for that resource. A non-zero full line under memory means the whole machine was stuck reclaiming, which is thrashing described in one number.

Saturation

Work queued for a resource that cannot serve it yet. The signal that utilisation alone hides.

Pressure stall information

The /proc/pressure files: how much time tasks spent stalled on CPU, memory or I/O.

Steal

CPU time a virtual machine wanted but the hypervisor gave elsewhere.

strace: watching a process talk to the kernel

strace prints every system call a process makes, with arguments, return value and error. It answers "what is this program actually doing" when logs do not, and it is the fastest way to see which file a program is really reading, which address it is connecting to, and where it is hanging.

It works through ptrace, the same mechanism debuggers use. The kernel stops the traced process at the entry and exit of every system call and hands control to strace, which reads the registers, prints, and lets it continue. That is two extra context switches per call, so a traced program runs ten to a hundred times slower. Never leave it on a production process for long, and prefer -e trace= to narrow what is stopped.

strace -f -tt -T -o trace.txt ./server        # follow forks, timestamps, time per call, to a file
strace -p 4242 -e trace=network               # attach, only socket calls
strace -e trace=openat,stat -e status=failed cmd   # which files did it look for and not find
strace -c -p 4242                             # count and time per syscall, Ctrl-C for the summary
strace -y -e trace=read,write -p 4242         # -y shows the path or socket behind each fd
strace -s 200 -e trace=write -p 4242          # print 200 bytes of each buffer instead of 32

Reading the output is mostly pattern matching:

openat(AT_FDCWD, "/etc/app/config.yml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/app/config.yml", O_RDONLY) = 3          # found it on the second try
connect(5, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.0.7")}, 16) = 0
poll([{fd=5, events=POLLIN}], 1, 30000                      # hanging here: waiting 30 s on fd 5
futex(0x7f1c2c0010a0, FUTEX_WAIT_PRIVATE, 0, NULL          # hanging here: waiting on a lock
wait4(-1,                                                   # hanging here: waiting for a child

A hung program's last line tells you what it is waiting for. ENOENT and EACCES on an openat tell you which path and which permission. A storm of tiny read calls tells you buffering is off. -c turns a slow program into a table of where its kernel time goes. ltrace is the same idea for calls into shared libraries rather than the kernel.

ptrace

The system call that lets one process observe and control another: stop it, read its registers and memory. strace and gdb use it.

strace -c

Summary mode: a count and total time per system call instead of a line per call.

perf: sampling where the CPU time goes

perf is the Linux profiler. It interrupts the CPU at a fixed rate, records which function was running and how it got there, and turns thousands of those samples into a picture of where time is spent, in the kernel and in user programs alike. Unlike strace it costs almost nothing to run.

perf top                                  # live: hottest functions on the machine right now
perf stat -e cycles,instructions,cache-misses,context-switches ./cmd   # hardware counters for one run
perf record -g -F 99 -p 4242 -- sleep 30  # sample 99 times a second for 30 s with stacks
perf report                               # browse the samples by function and caller
perf record -g -F 99 -a -- sleep 10       # the whole machine, every CPU
perf trace -p 4242                        # strace-like output at a fraction of the cost

perf stat uses the CPU's hardware counters. Its most useful line is instructions per cycle: near 1 or above means the CPU is computing; well below 0.5 means it is stalled, usually on memory, and the cache-misses line says so. perf record is the sampling profiler. With -g it records the call stack at each sample, so perf report can show not just that memcpy is hot but which caller made it hot.

A flame graph is the standard way to read a recording: each function is a box as wide as its share of samples, stacked on its callers. Wide boxes at the top are where time goes; the stack beneath says why. The stackcollapse-perf.pl and flamegraph.pl scripts turn perf script output into an interactive SVG.

Two things make or break it. Stacks need frame pointers or DWARF unwind information; binaries built with -fomit-frame-pointer (which was the default for years) give broken stacks, and --call-graph dwarf is the slower fix. And samples need symbols: a stripped binary shows hex addresses. JIT-compiled languages (Java, Node) need a perf map file so the sampler can name their generated code.

Sampling profiler

One that interrupts at a fixed rate and records what was running, instead of instrumenting every call.

Flame graph

A visualisation of stack samples: width is time, height is call depth.

Frame pointer

A register that makes walking the call stack cheap. Omitted by some compilers for a small speedup.

IPC

Instructions per cycle. Low means the CPU is waiting, usually on memory.

eBPF: safe programs inside the kernel

eBPF lets you load a small program into the running kernel and attach it to an event: a system call, a kernel function, a network packet, a user space function. The program runs when the event fires, collects what you asked for, and hands it to user space. No kernel module, no reboot, no risk of a crash, and low enough overhead to leave on in production.

flowchart TD
  W["Write: bpftrace script, bcc tool, or C with libbpf"] --> V["Kernel verifier: no loops without bounds, no bad memory, must terminate"]
  V --> J["JIT: compiled to native code"]
  J --> A["Attached to an event"]
  A --> E1["kprobe or tracepoint: a kernel function ran"]
  A --> E2["uprobe: a user space function ran"]
  A --> E3["XDP or tc: a packet arrived"]
  E1 --> M["BPF map: shared with user space"]
  E2 --> M
  E3 --> M
  M --> U["User space reads counts, histograms, events"]

Safety comes from the verifier. Before the kernel accepts a program it checks every path through it: no unbounded loops, no reading memory it should not, a bounded instruction count, only calls to approved helper functions. A program that fails is rejected, so a bad eBPF program is a load error, not a kernel panic. Accepted programs are JIT-compiled to native code and run at nearly native speed.

Programs attach to tracepoints (stable hooks the kernel developers placed, such as every system call entry), kprobes (any kernel function, dynamically), uprobes (any function in a user space binary), and network hooks (XDP in the driver, tc at the queue). Results go into maps, which are kernel-side hash tables, arrays and ring buffers that user space reads.

bpftrace is the fastest way to use it: a one-line language for ad-hoc questions.

# Which programs are opening which files, live
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# Histogram of read() latency in microseconds, per process
bpftrace -e 'kprobe:vfs_read { @start[tid] = nsecs; }
             kretprobe:vfs_read /@start[tid]/ { @us[comm] = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]); }'

# Count system calls by name for one PID
bpftrace -e 'tracepoint:raw_syscalls:sys_enter /pid == 4242/ { @[probe] = count(); }'

The bcc collection packages the common questions as commands: execsnoop (every new process), opensnoop (every file open), biolatency (disk I/O latency histogram), tcplife (every TCP connection with its duration and bytes), runqlat (how long tasks wait for a CPU), offcputime (where threads block). Each is what a Linux expert would have written by hand; they are the second thing to reach for after the tools in the table above.

eBPF

A virtual machine and verifier inside the kernel for running user-supplied programs safely at kernel events.

Verifier

The static checker that proves an eBPF program is safe before the kernel loads it.

Tracepoint / kprobe / uprobe

Attachment points: stable kernel hooks, arbitrary kernel functions, arbitrary user functions.

BPF map

A kernel-resident data structure shared between an eBPF program and user space.

XDP

Express data path. An eBPF hook in the network driver, before the kernel's stack sees a packet.

Recap