A filesystem can refuse a write for more than one reason, and the tools that tell them apart are cheap.
Blocks (bytes). df -h reports allocated blocks and du sums the files it can see. When df says full and du finds little, space is held by something du cannot walk. Most often that is a file that was deleted while a process still had it open. unlink removes the name, not the data. The blocks are freed only when the last file descriptor closes. lsof +L1 lists such files. ls -l /proc/PID/fd shows them as (deleted), and you can still read their contents through that path. That is how you rescue evidence before closing the descriptor.
Inodes (file count). Each file and directory consumes an inode, and the inode table is fixed when many filesystems are created. Millions of empty marker files can exhaust it while df -h shows plenty of room. df -i is the check. find DIR -xdev -type f | cut -d/ -f2 | sort | uniq -c | sort -n shows which subtree holds the entries.
Reserved or quota space can also produce ENOSPC for one user while root still writes.
Recovery order matters:
- Preserve evidence you cannot get back, such as the only copy of a trace, before closing anything.
- Free space with the least disruptive action. Ask the process to reopen its logs (SIGHUP or reload), restart it in a controlled way, or truncate through the descriptor as a last resort.
- Delete only data you are allowed to delete. A retention policy defines what is expired. "Old-looking" does not.
- Fix the growth path. Set bounded log sizes, repair cleanup jobs, and rotate logs with a signal to the writer. Otherwise you are back tomorrow.
A cleanup job that reports "removed 0 files" every hour for weeks is a signal worth alerting on, because silence is not success.