Skip to content

Disk Space and Inode Exhaustion Explained

"No space left on device" has more than one cause. A filesystem can be full of data blocks, full of inodes, or have its space held by files that were deleted but are still open. Each needs a different fix, and df -h alone only shows the first.

About the output on this page

The inode examples were captured from a real run on Ubuntu 22.04: a tiny tmpfs with a deliberately small inode limit, mounted inside an unprivileged user namespace (unshare -rm), so nothing on the host was touched. Mount paths are shortened to /data. The deleted-but-open example is representative.

How it works

What an inode is

An inode stores a file's metadata (owner, mode, timestamps, size, where its data blocks are). The file name lives in a directory entry that points at the inode number. Every file, directory and symlink uses one inode. On ext4 the number of inodes is fixed when the filesystem is created (mkfs.ext4 -i <bytes-per-inode> or -N <count>), by default one inode per 16 KiB of space. XFS allocates inodes dynamically, so it rarely runs out, though it can hit its inode-space limit (imaxpct). Btrfs has no fixed inode table.

So a filesystem with millions of tiny files (mail queues, PHP session files, caches, container layers) can run out of inodes while most of its gigabytes are free.

Telling the two apart

df -h /data      # blocks: size, used, available
df -i /data      # inodes: total, used, free

Real capture of the inode case (a 10 MB tmpfs limited to 200 inodes, filled with small files):

$ df -i .
Filesystem     Inodes IUsed IFree IUse% Mounted on
tmpfs             200   189    11   95% /data

$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            10M     0   10M   0% /data

Space is 0% used, but inodes are 95% used. When the last inode is taken, creating anything fails even though the disk is empty:

touch: cannot touch 'another': No space left on device

Same error text as a full disk. That is why you must check df -i whenever df -h looks fine.

Finding where the inodes went

Count files per top-level directory (works with only coreutils):

$ find . -xdev -type f | cut -d/ -f2 | sort | uniq -c | sort -rn
    120 sessions
     60 cache
      5 logs
      1 new

-xdev keeps find on one filesystem, so it does not walk into other mounts. GNU du can do the same in one command:

$ du --inodes -d1 . | sort -rn
190 .
121 ./sessions
61  ./cache
6   ./logs

Repeat one level deeper inside the biggest directory until you find the culprit. Here it is sessions, and removing the stale session files freed the inodes:

$ rm -f sessions/sess_*
$ df -i .
Filesystem     Inodes IUsed IFree IUse% Mounted on
tmpfs             200    70   130   35% /data

On a real server, run this against the affected mount (/var, /var/lib/docker, /home), starting with the usual suspects: /var/spool, /var/lib/php/sessions, /tmp, mail queues, build caches.

Block exhaustion and finding big things

sudo du -xh --max-depth=1 /var | sort -rh | head
sudo find /var -xdev -type f -size +500M -exec ls -lh {} +

-x (du) and -xdev (find) stay on one filesystem. sort -rh sorts human-readable sizes largest first.

Space held by deleted files

If df says the disk is full but du over the whole filesystem adds up to much less, a process probably still holds a deleted file open. The blocks are released only when the last file descriptor closes:

sudo lsof +L1 /var                          # open files with link count 0 (deleted)
sudo lsof -nP +L1 | awk '$7 > 100000000'    # only those over ~100 MB (column 7 is SIZE/OFF)

Typical case: rm app.log while the application still logs to it. Fix by restarting or signalling the process so it reopens its log (systemctl reload app, or kill -HUP <pid> if the program supports it). To free space without restarting, truncate through the open descriptor: : > /proc/<pid>/fd/<n> (use ls -l /proc/<pid>/fd to find <n>). Prevent it next time by truncating logs (truncate -s 0 file) or using logrotate instead of rm.

Reserved blocks (ext4)

ext4 reserves 5% of blocks for root by default, so ordinary users hit "No space" at about 95% while df still shows some Avail. Check with sudo tune2fs -l /dev/<dev> | grep -i reserved. Lowering it on large data volumes (sudo tune2fs -m 1 /dev/<dev>) is safe on non-root data disks; leave the default on / so system services keep working near-full.

Ubuntu/Debian vs. RHEL-family differences

The tools are identical. The default filesystem differs: RHEL-family systems default to XFS (inodes are dynamic, so df -i exhaustion is rare, and XFS cannot be shrunk), while Ubuntu and Debian default to ext4 (fixed inode count, so exhaustion is a real risk on mail or cache volumes). Check which you have with df -T /path or findmnt -no FSTYPE /path.

Practical exercise

This is safe on any machine: it uses a throwaway tmpfs in your own mount namespace and needs no root.

mkdir -p ~/inode-lab
unshare -rm bash        # new shell in a private mount namespace (user namespaces must be enabled)
mount -t tmpfs -o size=10m,nr_inodes=64 tmpfs ~/inode-lab
cd ~/inode-lab
df -i .
i=0; while touch f$i 2>/dev/null; do i=$((i+1)); done; echo "created $i files"
touch one-more          # fails with "No space left on device"
df -h . ; df -i .
rm -f f*                # free the inodes
touch one-more && echo ok

Cleanup: type exit to leave the namespace shell; the tmpfs disappears with it. Then rmdir ~/inode-lab. If unshare is refused (some hardened systems disable unprivileged user namespaces), use a spare VM and sudo mount -t tmpfs ... instead, then sudo umount.

How to verify you did it correctly

  • df -i . right after mounting shows 64 total inodes and about 1 used.
  • The loop stops at 63 files (64 inodes minus the one used by the mount's root directory), and touch one-more prints No space left on device.
  • df -h . at that moment shows 0% used: the proof that this was not block exhaustion.
  • After rm -f f*, df -i . shows plenty of free inodes and touch one-more succeeds.

Common mistakes

  • Only running df -h and concluding that the error is a mystery. Always pair it with df -i.
  • Running du without -x over / and counting /proc, /sys and other mounts.
  • rm-ing a huge log held open by a process. The space does not come back until the process closes the file. Truncate or rotate instead.
  • Cleaning the wrong layer on Docker hosts. Container layers consume inodes under /var/lib/docker. Use docker system df to see what is reclaimable before pruning.
  • Making a filesystem for millions of small files with defaults. Pick -i/-N at mkfs time (or use XFS), because the inode count of ext4 cannot be raised later without recreating it.

⚠ Dangerous commands

  • rm -rf on a path built from a variable (rm -rf $DIR/* with an empty $DIR runs on /*). Quote it and use ${DIR:?}: rm -rf -- "${DIR:?}"/*.
  • Deleting millions of files with rm -rf on a production disk can saturate I/O and stall the service. Prefer find <dir> -xdev -type f -delete, optionally under ionice -c3 nice.
  • Clearing /var/lib/php/sessions or a cache while the application is live logs everybody out. Delete only files older than the session lifetime: find ... -type f -mmin +1440 -delete.
  • docker system prune -a --volumes removes unused images and volumes with their data. Run docker system df first and never add --volumes on a host with databases.
  • tune2fs -m 0 on the root filesystem: a full / can then break logins and services that need to write.

Review questions

  1. A disk shows 40% used in df -h but nothing can be created. Which command do you run next?
  2. Why can an ext4 filesystem run out of inodes while XFS rarely does?
  3. df reports 100% but du -x / totals only 60% of the disk. What is the likely cause, and how do you find it?
  4. Why is truncating a log safer than deleting it while the service runs?
  5. Which find option stops it from walking into other mounted filesystems?