Filesystems, inodes and permissions

On Linux a file's name and a file's contents are two different things, held in two different places. Once that clicks, hard links, deleted-but-open files, /proc, and half of the permission system stop being surprising. This page starts from the inode and works outward to the directory tree and who is allowed to touch what.

The map

Read this first when short on time. Every branch is a section below.

Inodes: where a file's metadata lives

An inode is the record a filesystem keeps for one file. It holds everything about the file except its name and its contents.

• Type: regular file, directory, symlink, device, socket, FIFO.
• Permission bits and owner (user ID, group ID).
• Size in bytes, and how many blocks are allocated.
• Three timestamps: last access (atime), last content change (mtime), last metadata change (ctime).
• The link count: how many names point at this inode.
• Where the data is: a list of block pointers, or on ext4 and XFS a list of extents (start block, length).

Every inode has a number, unique within its filesystem. ls -i prints it, stat prints all of the above. The name is not there, because the name lives in the directory.

flowchart TD
  D["Directory /var/log: a list of entries"] --> E1["syslog: inode 1310"]
  D --> E2["auth.log: inode 1311"]
  D --> E3["syslog.old: inode 1310"]
  E1 --> I["Inode 1310: mode 0640, owner syslog, size 48 KB, links 2, extents"]
  E3 --> I
  E2 --> J["Inode 1311: mode 0640, owner root, size 12 KB, links 1, extents"]
  I --> B["Data blocks on disk"]
  J --> B

A directory is itself a file whose contents are a list of (name, inode number) pairs. Looking up /var/log/syslog means reading the root directory's inode to find the entry var, then that directory's inode to find log, then that one to find syslog, and finally the inode 1310 with the data. The kernel caches all of this so it happens once.

Each filesystem has a fixed number of inodes, set when it was created. df -i shows how many are used. A disk can be half empty by bytes and still refuse to create a file because a build directory, a mail spool or node_modules used every inode.

Inode

The on-disk record of one file: metadata and data location, but not the name.

Directory entry

A (name, inode number) pair inside a directory. Also called a dentry.

Extent

A contiguous run of blocks described by a start and a length.

Links: two ways to give a file a second name

Because names are separate from inodes, one inode can have several names. Because a file can hold a path, a name can also point at another name. Those are hard links and symbolic links.

Deleting a file is unlink: remove one name and decrement the inode's link count. The data is freed only when the count reaches zero and no process has the file open. That second condition is the source of a classic problem. A process writes to a log, someone deletes the log to free space, and the space does not come back because the process still holds the descriptor. df says the disk is full and du cannot find the files. lsof +L1 lists open files with no name left; restarting the writer, or truncating through /proc/PID/fd/N, frees the space.

The same behaviour is useful on purpose. A program can open a temporary file, unlink it immediately, and keep using it: the file exists only as long as the process does and nobody else can find it. Package upgrades rely on it too. Replacing /usr/lib/libc.so.6 writes a new inode and renames it over the old name; running programs keep the old inode mapped until they exit.

Hard link

An extra directory entry for an existing inode. Indistinguishable from the original.

Symbolic link

A file containing a path. Resolved at every access.

Link count

The number of directory entries naming an inode. Zero plus no open descriptors means the data is freed.

The VFS: one API over many filesystems

The virtual filesystem is the layer in the kernel that makes ext4, XFS, tmpfs, NFS and /proc all answer the same open, read, write and stat. Programs never know which one they are talking to.

flowchart TD
  A["read(fd, buf, 4096)"] --> V["VFS: find the file object, the inode, the offset"]
  V --> C{"Cached?"}
  C -- yes --> R["Copy from the cached page to buf"]
  C -- no --> F["Ask the filesystem driver: ext4, XFS, NFS"]
  F --> B["Block layer: schedule the I/O, wait for the disk"]
  B --> P["Fill the page cache"]
  P --> R

The VFS keeps four kinds of objects in memory: a superblock per mounted filesystem, an inode per file in use, a dentry per path component it has looked up (cached, so repeated path lookups are fast), and a file per open. Each filesystem driver fills in a table of functions for these objects, and the VFS calls through the table.

Mounting attaches one filesystem's root to a directory of another. The result is one tree, with / on one disk, /home maybe on another, /proc and /sys generated by the kernel, and /run in RAM. findmnt draws the tree; /proc/mounts is the raw list. Each container has its own mount table, which the containers article explains.

The page cache and durability

Almost every read and write goes through the page cache, the kernel's pool of file pages in RAM. A read that hits the cache is a memory copy. A write copies the bytes into the cache, marks the page dirty, and returns. The disk has not been touched. The kernel writes dirty pages back in the background, within about 30 seconds by default, or sooner if there are many of them.

That is fast and it is also why a power cut loses the last few seconds of writes. A program that needs the data on disk before it continues calls fsync(fd), which blocks until the file's dirty pages and metadata are written. Databases call it on every commit; that call is most of what makes a commit slow. O_DIRECT skips the cache entirely for programs that manage their own, and O_SYNC makes every write behave like write plus fsync.

Journaling protects the filesystem's own structures, not your data. ext4 writes a description of each metadata change to a journal before applying it, so a crash mid-way leaves either the old state or the new, never a half-updated directory. On a bad shutdown the kernel replays the journal at mount time instead of scanning the whole disk with fsck.

VFS

Virtual filesystem. The kernel layer that gives every filesystem the same interface.

Mount

Attaching a filesystem's root at a directory so it joins the single tree.

Page cache

File pages kept in RAM. Reads hit it; writes land in it and reach disk later.

fsync

Block until a file's dirty pages and metadata are on stable storage.

Journal

A log of metadata changes written before the changes themselves, so a crash leaves the filesystem consistent.

File descriptors: how a process holds a file open

A file descriptor is the small integer open returns. It indexes a table that belongs to the process, and that table points into two more tables shared across the system. Knowing the three levels explains every odd thing about redirection, forks and offsets.

flowchart LR
  subgraph P["fd table"]
    F0["0"]
    F1["1"]
    F3["3"]
    F4["4"]
  end
  F0 --> O1["Open file description<br/>the terminal, O_RDWR"]
  F1 --> O1
  F3 --> O2["Open file description<br/>offset 4096, O_RDONLY"]
  F4 --> O3["Open file description<br/>offset 0, O_RDONLY"]
  O2 --> I["Inode<br/>/var/log/syslog"]
  O3 --> I

1. The fd table is per process: an array from 0 up, each slot pointing at an open file description or empty. 0, 1 and 2 are standard input, output and error by convention only; the kernel does not care.
2. The open file description is system-wide: it holds the current offset, the access mode and flags such as O_APPEND, and a pointer to the inode. One is created per open.
3. The inode is the file itself.

dup and dup2 make a second descriptor pointing at the same open file description, so they share the offset. fork copies the fd table, so parent and child share every description too. That is why a child writing to inherited stdout appends after the parent's output rather than over it. Opening the same file twice creates two descriptions with independent offsets.

Descriptors survive exec unless opened with O_CLOEXEC, which is how a shell can open a file, dup2 it onto 1, and exec a program that then writes to the file without knowing. It is also how descriptors leak into child processes that should not have them, which is why modern code sets O_CLOEXEC by default.

Each process has a limit on open descriptors, ulimit -n, traditionally 1024 as a soft limit. Hitting it gives EMFILE, "Too many open files", and is the standard failure of a busy server or a program leaking descriptors. ls -l /proc/PID/fd shows every open descriptor as a symlink to what it points at, and lsof -p PID decodes them.

File descriptor

A per-process integer indexing an open file. Files, sockets, pipes, devices and timers all get one.

Open file description

The kernel object behind a descriptor: offset, flags, and the inode. Shared by dup and fork.

O_CLOEXEC

Close this descriptor automatically when the process execs a new program.

The directory tree: what goes where

The top-level directories follow a convention, the Filesystem Hierarchy Standard, that every distribution mostly follows. The rule underneath it is simple: separate what the distribution installs, what the administrator configures, and what changes while the machine runs.

The split explains where things go when you deploy a service. Its binary belongs under /usr/local/bin or /opt. Its configuration goes in /etc/myservice/. Its data goes in /var/lib/myservice/, its logs in /var/log/myservice/, and its socket or PID file in /run/myservice/. A backup that takes /etc, /home and /var captures everything the package manager cannot reinstall.

/run replaced /var/run, which is now a symlink to it. It was moved because /var might be on a disk that is not mounted yet during early boot, and PID files were needed before that.

FHS

Filesystem Hierarchy Standard. The convention for what each top-level directory holds.

tmpfs

A filesystem in RAM. /run and /dev/shm are always one; /tmp usually.

/proc and /sys: the kernel as text files

Two directories contain no files on any disk. Their contents are generated by the kernel when you read them, and some of them change kernel state when you write them. They are how ps, top, free and Docker actually work.

/proc

/proc has one directory per process, named by PID, plus a set of files about the machine. /proc/self is a symlink to the reading process's own directory. Files show a size of 0 because there is nothing to size; the text appears on read.

cat /proc/1234/status      # name, state, PID, PPID, UIDs, threads, VmRSS
cat /proc/1234/cmdline     # argv, NUL separated (tr '\0' ' ')
cat /proc/1234/maps        # every memory region: address, perms, backing file
ls -l /proc/1234/fd        # open descriptors as symlinks
ls -l /proc/1234/exe       # the running binary, even if it was deleted
cat /proc/1234/cgroup      # which cgroup it is in
cat /proc/1234/limits      # ulimits in effect
cat /proc/1234/oom_score_adj

cat /proc/meminfo          # what free reads
cat /proc/cpuinfo          # one block per CPU
cat /proc/loadavg          # 0.52 0.58 0.59 2/1042 8123
cat /proc/mounts           # the mount table
cat /proc/net/tcp          # every TCP socket, what ss reads

/proc/sys is different: it is the kernel's tunables, and writing to a file changes the setting immediately. sysctl is a front end for it: sysctl net.ipv4.ip_forward reads /proc/sys/net/ipv4/ip_forward, and sysctl -w or echo 1 > sets it. /etc/sysctl.d/ makes a setting survive a reboot. Common ones: vm.swappiness, vm.overcommit_memory, net.core.somaxconn, fs.file-max, kernel.pid_max.

/sys

/sys is the kernel's view of devices and drivers, one directory per object and one value per file. Where /proc grew organically, /sys is strictly structured. /sys/class/net/eth0/statistics/rx_bytes is the byte counter for one interface; /sys/block/sda/queue/scheduler shows and sets the I/O scheduler; /sys/class/thermal has the temperature sensors. /sys/fs/cgroup is where the cgroup tree lives, and writing there is how a container's limits are set.

/dev

/dev holds device nodes: special files that stand for hardware or a kernel service. Each has a type (character or block), a major number picking the driver, and a minor number picking the device. /dev/sda is a whole disk, /dev/sda1 a partition, /dev/tty1 a console. A few are pure software: /dev/null swallows writes, /dev/zero produces zeros, /dev/urandom produces random bytes, /dev/shm is a tmpfs for shared memory. The nodes are created by the kernel (devtmpfs) and named and given permissions by udev as devices appear.

procfs

The filesystem mounted at /proc: process information and kernel tunables, generated on read.

sysfs

The filesystem mounted at /sys: devices, drivers and kernel subsystems, one value per file.

Device node

A special file in /dev that names a driver by major and minor number.

sysctl

The command and the setting namespace for /proc/sys.

Permissions: who may do what

Every inode carries an owner, a group, and nine permission bits: read, write and execute, each for the owner, the group, and everyone else. The kernel checks them on every open, and the check picks exactly one of the three sets.

$ ls -l /etc/passwd /etc/shadow /usr/bin/passwd /tmp
-rw-r--r-- 1 root root    2847 Sep  1 10:12 /etc/passwd
-rw-r----- 1 root shadow  1424 Sep  1 10:12 /etc/shadow
-rwsr-xr-x 1 root root   68208 Mar 23 14:00 /usr/bin/passwd
drwxrwxrwt 9 root root    4096 Sep  8 09:30 /tmp

Read the first column in groups of three after the type character: rw- for the owner, r-- for the group, r-- for others. The same thing in octal is one digit per group, with r=4, w=2, x=1: rw-r--r-- is 644, rwxr-xr-x is 755. chmod 640 file and chmod g-w file are two spellings of the same operation.

The check goes: if you are the owner, only the owner bits apply. Else if you are in the group, only the group bits. Else the other bits. It stops at the first match, so a file owned by you with mode ---rwxrwx is unreadable to you and open to everyone else.

On a directory the same three bits mean something else. r lets you list the names. x lets you enter it and reach the inodes by name, which is needed to open anything inside. w lets you create, rename and delete entries. Deleting a file is a write to the directory, not to the file: you can delete a file you cannot read, and you cannot delete a file you own if the directory is not writable by you.

New files get their mode from the creating program's request (usually 666 for files, 777 for directories) minus the umask. With the common umask of 022 that yields 644 and 755; with 077 nothing is visible to others. umask is inherited across fork and exec, so a service's umask comes from systemd or its startup script.

Root, user ID 0, skips these checks. It can read and write anything and enter any directory. Only the execute bit is honoured: root cannot run a file with no x bit at all.

Mode

The permission bits plus the type and special bits, shown as -rwxr-xr-x or octal 0755.

umask

The bits removed from every newly created file's mode. 022 by default.

Owner, group, other

The three classes a permission check picks between. The first match wins.

Beyond the nine bits: setuid, sticky, capabilities and ACLs

Nine bits are not enough for everything, so there are three special bits in the mode, a way to split root's power into pieces, and a way to attach extra users and groups to a file.

setuid, setgid and the sticky bit

A program with the setuid bit runs with the file owner's user ID instead of the caller's. /usr/bin/passwd is setuid root (the s in rwsr-xr-x) so an ordinary user can update /etc/shadow, a file they cannot otherwise write. sudo works the same way. Every setuid root binary is a piece of code that must be bug-free, which is why distributions keep the list short and why the kernel ignores the bit on scripts.

setgid on a file does the same for the group. On a directory it means something more useful: new files inside inherit the directory's group rather than the creator's, which is how a shared project directory keeps everything owned by one group.

The sticky bit on a directory (the t in /tmp's rwxrwxrwt) means that even though everyone can create files there, only a file's owner, the directory's owner, or root can delete or rename it. Without it, a world-writable directory lets anyone delete anyone's files.

In octal they are a fourth digit in front: 4 for setuid, 2 for setgid, 1 for sticky. chmod 4755 program, chmod 2775 shared/, chmod 1777 /tmp.

Capabilities

Traditionally a process was either root, able to do everything, or not. Capabilities split root's power into about 40 named pieces so a process can have just the one it needs.

setcap cap_net_bind_service=+ep /usr/local/bin/myserver lets that binary bind port 80 as an ordinary user. getcap shows what a file has, and /proc/PID/status shows a running process's sets. systemd units can set CapabilityBoundingSet= and AmbientCapabilities=, and Docker starts containers with a short default list and drops the rest, so root inside a container is far weaker than root outside.

Access control lists

When one owner and one group are not enough, an ACL attaches extra entries to the inode: this user may read, that group may write. setfacl -m u:alice:rw file adds one; getfacl file lists them. ls -l shows a + after the mode when a file has an ACL, and the group bits then show the ACL mask rather than the group's real permissions, which surprises people. ACLs are stored as extended attributes and are supported by ext4, XFS and Btrfs.

setuid

A mode bit making a program run as the file's owner rather than the caller.

Sticky bit

On a directory: only a file's owner may delete or rename it.

Capability

One named slice of root's privilege that a process or file can hold on its own.

ACL

Access control list. Extra per-user and per-group permission entries on one file.

Recap