Reducing storage usage for Docker in LXC

Docker containers (like onions) have layers. In your Dockerfile, each new RUN, COPY or ADD line creates a new layer (so do the others, but not ones which affect the filesystem). Each layer contains only the files which changed from the previous layer, which allows layers to be shared between containers, reducing download size and disk usage. It’s possible to access individual layers though, so don’t go adding extra layers to delete secrets in a futile attempt at security.

On a normal system, Docker transparently merges the required layers together with a Union filesystem (similar to MergerFS, if you’ve heard of it). Docker refers to this as overlay2 (which is its default storage driver). This creates a single filesystem, compiled from multiple layers without duplicating files. The container sees a single filesystem, but in reality it’s multiple stitched together.

Running Docker inside LXC is a great deployment method, but the default overlay2 driver doesn’t work. This is because the LXC doesn’t have permission to mount these magic overlay FSs (they’re a kernel construct). Instead, it defaults to vfs, which whilst significantly simpler, also has some major issues. Rather than using an overlay to only store the changes on disk, vfs does a deep copy of the entire previous filesystem to build the next layer, on top of the previous. This means mounting a container uses significantly more storage, as many of the base OS files are duplicated with each layer, massively increasing disk usage. Because of this, it also means that multiple containers can’t share the same underlying layers. Forcing overlay2 doesn’t make things easier:

There is however a solution to this, in the form of FUSE. There exists a userspace implementation of overlayfs, known helpfully as fuse-overlayfs. Even more helpfully, docker has native support for it. Switching to it is simple:

1. Ensure that fuse is enabled on the container. Either set features: fuse=1 in the LXC config file, or check “fuse” under options in Proxmox. After changing, restart the container.
2. Install fuse-overlayfs using your package manager of choice
3. Modify /etc/docker/daemon.json and set storage-driver to fuse-overlayfs
4. Restart Docker
5. Profit?

Note that because I don’t use native docker volumes for anything important, I didn’t back up /var/lib/docker at all in the above. If you are, or want to be extra cautious, it’s probably worth doing.

Now, once docker restarts, it’s likely none of your containers will be running. This is normal. Now you’ll just need to pull and start the previously running containers (you’re using something like docker-compose to manage your configuration, right?). Containers should start in exactly the same way, and work in exactly the same way. If everything went well, you’ll be up and back to normal.

Now, to see the differences. Inside /var/lib/docker, Docker stores separate directories based on the storage driver it’s using. du -hs <directory> will show you the size of each, so you can see how much storage you saved. When I performed this, the vfs directory was 18GB, whilst fuse-overlayfs was a mere 5GB. 4x saving by doing practically nothing - just how I like it!