/var is full and nothing is in it

Lab machine

A private Linux machine with the problem already set up. Sessions last up to 60 minutes.

Tomas Berg opened INC-2304 at 06:10SEV-2

depot-2 cannot print shipping labels. /var is at 100%, the night shift deleted the 3.9 GB log, and df still says 100%.

depot-2 is the harbour depot; the 06:00 trucks are already waiting for labels. Waybill's health check returns 507 (insufficient storage).

"/var hit 100% around 02:30. trace.log was 3.9G so I deleted it, then compressed the old access logs. df still says 100% and I have no idea why. Leaving this for day shift, sorry." (Tomas)

Engineering also needs the trace lines around the 02:14 panic for the postmortem. Those lines are in the file Tomas deleted. A file is only really gone when nothing has it open any more.

Your task

Save the trace lines around the 02:14 panic under var/incident/ before anything else, then free the space without restarting Waybill blindly, and give the trace a size limit so this cannot happen again.

On the machine

bin/df -h for the simulated 4 GiB /var
lsof +L1, ls -l /proc/PID/fd: open files that were deleted
etc/waybill/trace.json, re-read on bin/hostctl reload waybill
home/nightops/.bash_history

Timeline

02:14Waybill panics; a trace burst starts writing to trace.log.

02:30/var reaches 100%. Label printing fails.

02:41Tomas deletes trace.log (3.9 GB) and compresses access logs. df: still 100%.

06:10Tomas hands over: "no idea why, sorry".

Done when

/var has room again.
Waybill's health check returns 200.
The trace has a size limit, and a rotation drill proves it works.
The 02:14 trace lines are saved under var/incident/.

How it is checked:The checker looks for the 02:14 trace lines under `var/incident/`, checks that no deleted file is still held open, that /var has room and `/healthz` answers 200, then runs a rotation drill to prove the trace size limit works.

Passed. Here is how others fixed it

The fix we expected. Find the descriptor with `lsof +L1` or `ls -l /proc/PID/fd`, copy the head of the deleted trace (for example `head -c 64k /proc/PID/fd/3 | grep -a -A4 TRACE-7f3a > var/incident/trace-0214.txt`), set a bounded `max_bytes` in `etc/waybill/trace.json`, then `bin/hostctl reload waybill` so Waybill re-reads its config and reopens the trace. Recheck `bin/df -h` and the health endpoint.

Other fixes that also pass:

Truncate the deleted file through its descriptor (`: > /proc/PID/fd/N`) after saving the excerpt, then set retention and reload.
Use `cp /proc/PID/fd/N` with `head` limits to copy only the incident window, then restart Waybill once retention is configured.

Fixes that look right but are not:

Repeatedly deleting logs while the process still holds the large file.
Restarting first and losing the only copy of the incident trace.
Copying the whole deleted trace into the same full volume.

Why it broke. Unlinked files consume disk until the last open descriptor closes; deleting more visible files may not help.

How Mara worked it

bin/df -h and du -sh var/*
df and du disagree by gigabytes. Space is held by something du cannot see.
lsof +L1, or ls -l /proc/<waybill pid>/fd
A deleted file that a process still holds open keeps using its space.
head -c 64k /proc/<pid>/fd/<n> | grep -a -A4 TRACE-7f3a > var/incident/trace-0214.txt
Save the 02:14 evidence first. Restarting would destroy the only copy.
Set a bounded max_bytes in etc/waybill/trace.json, then bin/hostctl reload waybill
The reload makes Waybill reopen its trace, which releases the deleted file, and the limit stops it happening again.
bin/df -h and curl the health endpoint
Space is back and Waybill answers 200.

Hints

Hint 1

Compare df and du.

Hint 2

Look for unlinked open files.

Hint 3

A process can hold deleted file blocks.

Hint 4

Inspect `lsof +L1` and rotate/restart safely.

Show the solution

Find the descriptor with `lsof +L1` or `ls -l /proc/PID/fd`, copy the head of the deleted trace (for example `head -c 64k /proc/PID/fd/3 | grep -a -A4 TRACE-7f3a > var/incident/trace-0214.txt`), set a bounded `max_bytes` in `etc/waybill/trace.json`, then `bin/hostctl reload waybill` so Waybill re-reads its config and reopens the trace. Recheck `bin/df -h` and the health endpoint.

Linux Administration

Storage, filesystems, and inodes

Why a disk can be full while the files you can see are small, and the three distinct things "full" can mean.

Devices, filesystems, mounts

A disk (or a cloud volume) appears as a block device such as /dev/nvme0n1. It is divided into partitions or logical volumes. A filesystem such as ext4 or XFS is created on one of them, and it is mounted at a directory. From then on, everything under that directory lives on that filesystem. findmnt and lsblk show the tree. The operational consequence is that "the disk is full" really means "the filesystem holding this path is full". df -h /var/lib/waybill answers for the exact path you care about.

Names, inodes, and blocks

A traditional Unix filesystem separates three things:

Directory entries map a name to an inode number.
Inodes store metadata such as owner, mode, size, and timestamps, plus pointers to data.
Data blocks hold the contents.

A file is deleted when its link count (the number of names) reaches zero and no process has it open. The second condition is the source of a famous incident.

Three different kinds of "full"

1. Bytes really are full. df -h and du -sh broadly agree. Find the consumer by walking down with du -xh --max-depth=1 /var | sort -h. The -x keeps you on one filesystem.

2. Deleted but open. Someone removed a 30 GB log, but the service still has it open and keeps writing. du no longer sees a name, while df still counts the blocks. lsof +L1 lists open files with zero links. The fix is to make the holder close the descriptor: reload or restart the service, or truncate through /proc/<pid>/fd/<n> if a restart is unacceptable. Deleting it "again" does nothing.

3. Inodes are full. Each file needs one inode, and on ext4 the number of inodes is fixed when the filesystem is created. A spool directory with millions of tiny files can use every inode while gigabytes of bytes remain free. df -i shows it. Creating files fails with the same "No space left on device" message, which is why this one gets misdiagnosed. Find the directory with the most entries, not the most bytes.

There is also a misleading kind of full. ext filesystems reserve about 5% of blocks for root, so ordinary users see 100% while root can still write.

Freeing space without making things worse

Space incidents tempt people into broad deletes. A safer order:

Identify which filesystem is full and which kind of full it is.
Find the consumer: a path from du, a process from lsof, or a directory with too many entries.
Check whether the data is evidence or state before removing it. Old logs may explain this incident, and a queue directory may hold undelivered work.
Remove or compress named items and verify df moved.
Fix the cause: rotation without a reopen, a job that never cleans up, a retention policy nobody set.

Growing a filesystem

When the data is legitimate, the fix is capacity. With LVM or cloud volumes you can usually grow the underlying device and then the filesystem online: lvextend, then resize2fs for ext4 or xfs_growfs for XFS. Shrinking is harder (XFS cannot shrink at all), so grow in modest steps and keep the reason in the change record.

Key terms

Mount point: A directory where a filesystem is attached to the tree. Everything below it lives on that filesystem until another mount takes over.
Inode: The on-disk record for a file, holding its metadata and the locations of its blocks. Filenames live in directories, not inodes.
Link count: How many directory entries name an inode. Space is freed when this reaches zero and no process holds the file open.
Reserved blocks: Space ext filesystems hold back for root (5% by default), so df can show 100% for ordinary users while root can still write.

Read further

How Linux Works, 3rd edition, Ch. 4, "Disks and Filesystems" (Purchase)
Partitions, filesystem creation, mounting, /etc/fstab, and the section on inodes and the structure of a traditional filesystem. Note what a directory entry stores and what an inode stores.
UNIX and Linux System Administration Handbook, 5th edition, Ch. 5, "The Filesystem" (Purchase)
The file tree, file types, link counts, and the discussion of removing a file that is still open.
UNIX and Linux System Administration Handbook, 5th edition, Ch. 20, "Storage" (Purchase)
The overview of the storage stack and the parts on logical volume management and filesystem resizing. Skim the hardware sections.

/var is full and nothing is in it INC-2304