Snowy mountains & self storage in Logan, Utah. TK Secure Storage of Brigham City is located in areas surrounded by seasonal change and panoramic views.

City Storage Idaho Falls

Reducing storage usage for Docker in LXC

Docker containers (like onions) have layers. In your Dockerfile, each new RUN, COPY or ADD line creates a new layer (so do the others, but not ones which affect the filesystem). Each layer contains only the files which changed from the previous layer, which allows layers to be shared between containers, reducing download size and disk usage. It’s possible to access individual layers though, so don’t go adding extra layers to delete secrets in a futile attempt at security.

On a normal system, Docker transparently merges the required layers together with a Union filesystem (similar to MergerFS, if you’ve heard of it). Docker refers to this as overlay2 (which is its default storage driver). This creates a single filesystem, compiled from multiple layers without duplicating files. The container sees a single filesystem, but in reality it’s multiple stitched together.

Running Docker inside LXC is a great deployment method, but the default overlay2 driver doesn’t work. This is because the LXC doesn’t have permission to mount these magic overlay FSs (they’re a kernel construct). Instead, it defaults to vfs, which whilst significantly simpler, also has some major issues. Rather than using an overlay to only store the changes on disk, vfs does a deep copy of the entire previous filesystem to build the next layer, on top of the previous. This means mounting a container uses significantly more storage, as many of the base OS files are duplicated with each layer, massively increasing disk usage. Because of this, it also means that multiple containers can’t share the same underlying layers. Forcing overlay2 doesn’t make things easier:

ERRO[2021-09-30T09:24:56.914420548Z] failed to mount overlay: invalid argument     storage-driver=overlay2
ERRO[2021-09-30T09:24:56.914439880Z] [graphdriver] prior storage driver overlay2 failed: driver not supported
failed to start daemon: error initializing graphdriver: driver not supported

There is however a solution to this, in the form of FUSE. There exists a userspace implementation of overlayfs, known helpfully as fuse-overlayfs. Even more helpfully, docker has native support for it. Switching to it is simple:

  1. Ensure that fuse is enabled on the container. Either set features: fuse=1 in the LXC config file, or check “fuse” under options in Proxmox. After changing, restart the container.
  2. Install fuse-overlayfs using your package manager of choice
  3. Modify /etc/docker/daemon.json and set storage-driver to fuse-overlayfs
  4. Restart Docker
  5. Profit?

Note that because I don’t use native docker volumes for anything important, I didn’t back up /var/lib/docker at all in the above. If you are, or want to be extra cautious, it’s probably worth doing.

Now, once docker restarts, it’s likely none of your containers will be running. This is normal. Now you’ll just need to pull and start the previously running containers (you’re using something like docker-compose to manage your configuration, right?). Containers should start in exactly the same way, and work in exactly the same way. If everything went well, you’ll be up and back to normal.

Now, to see the differences. Inside /var/lib/docker, Docker stores separate directories based on the storage driver it’s using. du -hs <directory> will show you the size of each, so you can see how much storage you saved. When I performed this, the vfs directory was 18GB, whilst fuse-overlayfs was a mere 5GB. 4x saving by doing practically nothing - just how I like it!

Share this page

Similar content

View all →


Docker in LXC

Docker is a great containerization technology for running applications. It keeps multiple applications completely isolated from each other, only allowing connections exactly when you tell them to. But what if you’re on a hypervisor? You want your host OS to be as lean as possible (else it defeats the point),…


LXC vs Docker

5 minutes

Both LXC and Docker are great containerization technologies, brought to you by the powers of the Linux kernel. At their core, they’re pretty similar, but the further out you look, the differences increase massively. At their heart, they’re both still containers - understanding the differences between the 2 takes a…

Freight Containers on a Ship

Backing up and restoring Docker containers

You should back up your data, properly! If you’re not, you’re playing a dangerous game with fate. Computers are pretty reliable, but they also go wrong, often. You should always backup your files, but backing up a containerized application isn’t quite as simple. A container is 3 things:ConfigurationVolumesNetworking The point…