No space left, 41% used

Lab machine

A private Linux machine with the problem already set up. Sessions last up to 60 minutes.

Depot-4 shift lead opened INC-2317 at 13:20SEV-2

The depot-4 spooler rejects every new manifest with "No space left on device". df says the spool is 41% full.

depot-4 is the spool hub: every label printed in the north region passes through its spooler. Two restarts changed nothing.

"Spooler says No space left on device but df -h shows the spool at 41%. I restarted it twice. Is the disk lying?" (depot-4 shift lead)

A filesystem can run out of two things: bytes, and inodes, the entries that describe each file. A few million tiny files use almost no bytes and every inode. The retention job has reported success every hour for a month.

Your task

Find out what is using the inodes and why the hourly retention job never removes it, fix the job, and run it once. Keep customer manifests in spool/outbound/ and any retry marker younger than seven days.

On the machine

bin/df -h and bin/df -i for the simulated /spool
opt/spooler/CHANGELOG.md
var/log/spool-cleanup.log
The retention unit and its find command under etc/units/

Timeline

Jul 30Spooler 3.2.0 changes how it names retry markers.

Every hourThe retention job runs, finds nothing to delete, and logs success.

13:02Spooler: "No space left on device".

13:20INC-2317: outbound manifests backing up.

Done when

The spool has free inodes again.
The spooler is writing manifests again.
The hourly retention job now removes expired markers, and only those.

How it is checked:The checker confirms no manifest or recent marker was deleted, inode use has dropped, the spooler is writing manifests again, and then plants expired markers and runs the retention job to prove it removes them.

Passed. Here is how others fixed it

This lab has 3 versions. Reset for a fresh machine, and you may get a different one.

Break it for a friend. Pick the version they get, and the link is copied for you to send: version 1 · version 2 · version 3

Other fixes that also pass:

Run the corrected `find` command by hand once to clear expired markers, then fix the job's pattern so it keeps working.
Match on age only (`-mmin +10080`) inside the marker directory, as long as customer manifests stay excluded.

Fixes that look right but are not:

Deleting customer parcel payloads to free bytes.
Deleting the large archive; bytes were never the constraint.
Wiping all retry markers, including ones the carrier retry loop still needs.
Cleaning up by hand without fixing the job, so the inode table fills again.

Why it broke. A filesystem has a fixed number of inodes, one per file, and can run out of them long before it runs out of bytes. Here the growth comes from retry markers, which are tiny and many. Retention jobs break quietly when the thing they clean up changes shape: a new file name, a new directory layout, or an upgrade that leaves the timer off. The job keeps "succeeding" with "removed 0", which is why nobody noticed. Clean up by fixing the job, so the next hour does the same.

How Mara worked it

bin/df -h against bin/df -i
41% of the bytes, 100% of the inodes. Lots of small files, not one big one.
find spool -type f | cut -d/ -f2 | sort | uniq -c | sort -n
Which directory holds the millions of files.
cat opt/spooler/CHANGELOG.md opt/spooler/bin/cleanup-retry-markers, bin/hostctl list-timers, tail var/log/spool-cleanup.log
What spooler 3.2 changed about markers, and whether the cleanup job still matches it, or runs at all.
Fix the job (its pattern, its depth, or its timer), then bin/hostctl start spool-cleanup.service
Let the job do the cleanup, so it also works next hour.
bin/df -i and bin/hostlog -u spooler
Inodes free, and the spooler writing manifests again.

Hints

Hint 1

`bin/df -i`: a filesystem can run out of files before it runs out of bytes.

Hint 2

Count files per directory. The growth is thousands of tiny files, not one big one.

Hint 3

`var/log/spool-cleanup.log` says the hourly job removes nothing, or stopped running. Since when, and what changed then?

Hint 4

Compare the job's find command and its timer with what `opt/spooler/CHANGELOG.md` says about markers. Fix the job, then run it once.

Show the solution

`bin/df -i` shows the spool out of inodes while bytes are fine, and `spool/retry/` holds thousands of markers. Find out why the hourly job stopped expiring them: compare `opt/spooler/bin/cleanup-retry-markers` with where and how spooler 3.2 writes markers (`opt/spooler/CHANGELOG.md`), and check `bin/hostctl list-timers` and `var/log/spool-cleanup.log` to see whether it runs at all. Fix that (the name pattern, the depth it searches, or the timer), run `bin/hostctl start spool-cleanup.service` once, confirm with `bin/df -i`, and watch the spooler recover in its journal.

Linux Administration

Storage, filesystems, and inodes

Why a disk can be full while the files you can see are small, and the three distinct things "full" can mean.

Devices, filesystems, mounts

A disk (or a cloud volume) appears as a block device such as /dev/nvme0n1. It is divided into partitions or logical volumes. A filesystem such as ext4 or XFS is created on one of them, and it is mounted at a directory. From then on, everything under that directory lives on that filesystem. findmnt and lsblk show the tree. The operational consequence is that "the disk is full" really means "the filesystem holding this path is full". df -h /var/lib/waybill answers for the exact path you care about.

Names, inodes, and blocks

A traditional Unix filesystem separates three things:

Directory entries map a name to an inode number.
Inodes store metadata such as owner, mode, size, and timestamps, plus pointers to data.
Data blocks hold the contents.

A file is deleted when its link count (the number of names) reaches zero and no process has it open. The second condition is the source of a famous incident.

Three different kinds of "full"

1. Bytes really are full. df -h and du -sh broadly agree. Find the consumer by walking down with du -xh --max-depth=1 /var | sort -h. The -x keeps you on one filesystem.

2. Deleted but open. Someone removed a 30 GB log, but the service still has it open and keeps writing. du no longer sees a name, while df still counts the blocks. lsof +L1 lists open files with zero links. The fix is to make the holder close the descriptor: reload or restart the service, or truncate through /proc/<pid>/fd/<n> if a restart is unacceptable. Deleting it "again" does nothing.

3. Inodes are full. Each file needs one inode, and on ext4 the number of inodes is fixed when the filesystem is created. A spool directory with millions of tiny files can use every inode while gigabytes of bytes remain free. df -i shows it. Creating files fails with the same "No space left on device" message, which is why this one gets misdiagnosed. Find the directory with the most entries, not the most bytes.

There is also a misleading kind of full. ext filesystems reserve about 5% of blocks for root, so ordinary users see 100% while root can still write.

Freeing space without making things worse

Space incidents tempt people into broad deletes. A safer order:

Identify which filesystem is full and which kind of full it is.
Find the consumer: a path from du, a process from lsof, or a directory with too many entries.
Check whether the data is evidence or state before removing it. Old logs may explain this incident, and a queue directory may hold undelivered work.
Remove or compress named items and verify df moved.
Fix the cause: rotation without a reopen, a job that never cleans up, a retention policy nobody set.

Growing a filesystem

When the data is legitimate, the fix is capacity. With LVM or cloud volumes you can usually grow the underlying device and then the filesystem online: lvextend, then resize2fs for ext4 or xfs_growfs for XFS. Shrinking is harder (XFS cannot shrink at all), so grow in modest steps and keep the reason in the change record.

Key terms

Mount point: A directory where a filesystem is attached to the tree. Everything below it lives on that filesystem until another mount takes over.
Inode: The on-disk record for a file, holding its metadata and the locations of its blocks. Filenames live in directories, not inodes.
Link count: How many directory entries name an inode. Space is freed when this reaches zero and no process holds the file open.
Reserved blocks: Space ext filesystems hold back for root (5% by default), so df can show 100% for ordinary users while root can still write.

Read further

How Linux Works, 3rd edition, Ch. 4, "Disks and Filesystems" (Purchase)
Partitions, filesystem creation, mounting, /etc/fstab, and the section on inodes and the structure of a traditional filesystem. Note what a directory entry stores and what an inode stores.
UNIX and Linux System Administration Handbook, 5th edition, Ch. 5, "The Filesystem" (Purchase)
The file tree, file types, link counts, and the discussion of removing a file that is still open.
UNIX and Linux System Administration Handbook, 5th edition, Ch. 20, "Storage" (Purchase)
The overview of the storage stack and the parts on logical volume management and filesystem resizing. Skim the hardware sections.

No space left, 41% used INC-2317