I recently celebrated 3 years running Arch Linux as my daily driver. Shortly thereafter, I noticed that my 300 GB root file system partition had less than 2 GB of free space. Why was my Arch install consuming so much disk space?
Why is my disk full?
If you already know how to troubleshoot Linux disk space problems, skip ahead to learn how to prune the pacman cache. If you are curious how I found my lost disk space, continue reading.
Troubleshooting Low Disk Space on Linux
Linux provides a variety of command line tools that can be used to troubleshoot low disk space problems. These tools include the df (Disk Free) and du (Disk Usage) commands.
Note that the command outputs below are simplified to emphasize key points.
Listing Partitions with “df”
I first ran “disk free” to look at the available space on each of my disk partitions.
~% sudo df -hT
Filesystem Type Size Used Avail Use% Mounted on
dev devtmpfs 32G 0 32G 0% /dev
run tmpfs 32G 10M 32G 1% /run
/dev/nvme0n1p2 btrfs 300G 297G 1.3G 100% /
tmpfs tmpfs 32G 485M 31G 2% /dev/shm
tmpfs tmpfs 32G 40M 32G 1% /tmp
/dev/nvme0n1p2 btrfs 300G 297G 1.3G 100% /var/cache
/dev/nvme0n1p2 btrfs 300G 297G 1.3G 100% /var/log
/dev/nvme0n1p1 vfat 999M 621M 379M 63% /boot
/dev/nvme0n1p3 btrfs 1.6T 1.1T 446G 72% /opt
/dev/nvme0n1p3 btrfs 1.6T 1.1T 446G 72% /home
My Arch system uses a single SSD (/dev/nvme0n1) with three partitions (p1, p2 and p3). The dev, run, efivarfs and tmpfs lines are virtual file systems and can be ignored.
Partition 1 (/dev/nvme0n1p1) is my boot partition and has 37% of allocated space available. My /home and /opt directories are located on partition 3 (/dev/nvme0n1p2), which has 28% of allocated space available.
Partition 2 (/dev/nvme0n1p2) contains the root file system / as well as two separate BTRFS subvolumes for cache and log files. Partition 2 shows 100% of space used, with only 1.3G available. This is a problem!
List Top-Level Directories with “ls”
Let’s list all of the top-level directories in the root file system ("/"):
~% ls /
bin dev home lib64 opt root sbin sys tmp var
boot etc lib mnt proc run srv timeshift-btrfs usr
Ignore Directories Not on Full Partition
We can ignore the /home and /opt directories since they are located on partition 3. We can ignore the /boot directory as it is located on partition 1. We can ignore /dev and /run as they are virtual file systems. Finally, we can ignore /proc and /sys as these are also virtual file systems (although they are not listed as such by the df command).
One (or more) of the remaining directories is consuming lots of disk space. Let’s investigate to figure out which is the culprit.
Determine Disk Space Usage for Top-Level Directories
How much space is used by each of the top-level directories on partition 2? To find out, I use the du command as follows:
~% sudo du -h --max-depth=1 --exclude=/home --exclude=/opt --exclude=/boot --exclude=/run --exclude=/proc --exclude=/sys /
201G /var
130M /dev
14M /etc
24K /tmp
11G /usr
0 /mnt
1.6G /root
0 /srv
0 /timeshift-btrfs
0 /snapshots
214G /
The “disk usage” command shown above uses the following parameters:
- -h: displays output in human-readable format
- –max-depth=1: only looks at top-level directories
- -exclude: ignores specified directories (those not located in partition 2)
Ouch! The /var directory is using 201G?? Why?
Determine Disk Space Usage for “/var”
Let’s refine our du command to explore the /var directory space usage:
~% sudo du -h --max-depth=1 /var
82G /var/cache
119G /var/lib
1.1G /var/log
0 /var/empty
0 /var/games
0 /var/local
0 /var/opt
484K /var/spool
0 /var/tmp
8.0K /var/db
201G /var
The /var/cache directory uses 82G, and the /var/lib directory uses 119G! These two directories are using 2/3 of the 300G I allocated for the entire root file system! Why?
Determine Disk Space Usage for “/var/cache”
Let’s explore /var/cache:
~% sudo du -h --max-depth=1 /var/cache
0 /var/cache/lightdm
5.2M /var/cache/man
0 /var/cache/private
460K /var/cache/cups
82G /var/cache/pacman
124K /var/cache/ldconfig
6.5M /var/cache/fontconfig
0 /var/cache/libvirt
20M /var/cache/swcatalog
0 /var/cache/lxc
17M /var/cache/lxd
0 /var/cache/powertop
82G /var/cache
Wow – /var/cache/pacman is using 82G! For readers not familiar with Arch, pacman is the software update tool for Arch Linux (and other Arch-based distributions such as Manjaro). Exploring the /var/cache/pacman directory using ls, I see that it contains packages from 3 years ago when I first installed Arch! The pacman cache apparently retains all downloaded packages.
Prune pacman Cache with paccache
Pacman has a feature that allows users to “rollback” a bad package update by restoring a previous version from its cache. To implement this feature, it saves all versions of previously downloaded packages. Rollback is a useful feature, but will I ever need to rollback to a 3 year old package?
What is a Rolling Release?
Arch Linux uses a “rolling release” model. This means that updated packages are released immediately rather than packaged together into discrete version updates. The advantage of this is that Arch provides updated packages (bug fixes and new features) shortly after they are released by the upstream maintainers. With non-rolling Linux distros, updates may not be released until months after the upstream maintainers have created them. I update my Arch install every week, and I’ve been running Arch for three years. Some packages (including the Linux kernel) are updated quite often. My pacman cache contains over a hundred versions of some frequently updated packages. No wonder my pacman cache uses 82G of disk space!
Using “paccache” to Prune Pacman Cache
Arch provides a utility called paccache for pruning the pacman cache. Let’s use this utility to keep only the last three updates for each package, and delete all older packages. It is highly unlikely that I’ll ever need to roll back a package to a version before the last three updates.
First let’s do a “dry run” (does not delete any package files):
~% sudo paccache -dk2
==> finished dry run: 5907 candidates (disk space saved: 75.78 GiB)
Now let’s actually run the command (deletes old package files):
sudo paccache -rvk2
==> finished: 5907 packages removed (disk space saved: 75.78 GiB)
How Much Space did “paccache” Reclaim?
Let’s run the df command again to make sure we actually freed up 76G of space:
~% df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/nvme0n1p2 300G 221G 77G 75% /
/dev/nvme0n1p2 300G 221G 77G 75% /var/cache
/dev/nvme0n1p2 300G 221G 77G 75% /var/log
...
We now have 77G of free disk space just by pruning the pacman cache!
Deleting Unneeded Packages
paccache can also delete all versions of packages that are no longer needed (i.e. obsolete):
~%sudo paccache -ruvk3
Closing Thoughts
Astute readers may have several questions:
- Why is “/var/lib” using 119G?
- Why did “du /” show 214G disk space used when “df” showed 297G disk space used?
- Why did I put the “/var/cache” and “/var/log” directories into BTRFS subvolumes?
- Why did I put “/home” and “/opt” onto a separate partition from “/”?
- Won’t the pacman cache fill up your disk again?
Why is “/var/lib” using 119G of disk space?
Using the du command, I discovered that 102G of disk space was used by /var/lib/lxd. LXD is a tool for creating and running Linux containers and virtual machines. This probably means that I have stored my LXD virtual machines on my SSD’s small partition 2 (300G) instead of its large partition 3 (1.6T). Fixing this will be the topic of another blog post.
Why did “du /” show 214G disk space used when “df” showed 297G disk space used?
Excellent question! I use the Linux timeshift utility, which creates a BTRFS snapshot before each system update. These snapshots are not mounted into the root file system (and thus not seen when executing du -h, but still consume disk space in partition 2 which is seen when executing df -h. At least I think that is what is happening.
Why did I put the “/var/cache” and “/var/log” directories into BTRFS subvolumes?
As previously mentioned, I use timeshift to create BTRFS snapshots before each system update. A BTRFS snapshot only “saves” files within its volume. It does not snapshot files within subvolumes. So, by putting /var/cache and /var/log into BTRFS subvolumes, I prevent these from being included in the snapshot of / created by timeshift before each system update. I consider the contents of /var/cache and /var/log to contain temporary info that changes frequently, and thus shouldn’t be included in system snapshots.
Why did I put “/home” and “/opt” onto a separate partition from “/”?
Similar to the previous answer, I don’t want /home and /opt to be included in system update snapshots. If /home and /opt were subdirectories of /, they would be included in BTRFS snapshots of /. But since they are subvolumes of /, they are not included in snapshots of /. This means that I can snapshot them whenever I feel is appropriate, rather than each time I do a system update.
Won’t the pacman cache fill up your disk again?
Yes it will! I see several potential solutions to this problem:
- Install a cron job that periodically prunes the cache using
paccache - Create a systemd time to periodically prune the cache
- Create a pacman callback to prune the cache each time a system update is performed
- Enter a recurring task in Todoist, my favorite task manager, to manually prune the pacman cache every six months.
The Software Engineer in me wants to implement 1, 2, or 3 (or maybe all three!). The Software Manager in me says that the frequency of problem occurence and time required to manually prune doesn’t warrant spending time on items 1, 2 or 3. Hmmm… which side of me will win this argument?
Hint: I’m no longer a software manager, but I’ll always be a software engineer ;))
Thanks for reading!