Advanced Docker — Docker storage drivers and file systems

In this post we will be talking about advanced docker concepts. We are going to see how docker stores data and how it manages file system of containers. Please read Docker for the absolute beginner post before you continue if you haven’t already.

File systems

When you install docker on a system, it creates /var/lib/docker folder structure to store all data related with images and containers running on the docker host by default.

So, how exactly does docker store the files of an image and a container?

To understand this, we need to understand Layered architecture of docker containers first.

Layered architecture

In a previous post we saw how to build a docker image. When docker build images, it builds in a layered architecture. Each lines of instruction in the Dockerfile creates a new layer in the docker image with just the changes from the previous layer.

Lets take the Dockerfile from the previous post as an example.

The first layer is a Base ubuntu layer followed by a second instruction that creates a second layer which installs the apt packages going to the fifth layer that Updates the Entrypoint.

Since each layer only stores changes from the previous layer, it is reflected on the size as well.

To understand the advantages of the layered architecture, lets consider another application.

The new application is the same as the previous app only with different source code(app2.py).

When you run the docker build command to build an image for this application, since the first three layers of the application are the same, docker is not going to build them. It reuses them from the cache. This way docker build images faster and efficiently saves disk space.

The image layers are read only. To edit them, you must rebuild your image again.

When you create a container based on your image using docker build command, docker creates a new writable layer (Container layer) on top of the image layers. This layer is used to store data created by the container.

When the container is destroyed, the layer and all the files stored in it are also destroyed.

Volume

So, what if you wished to persist this data? For example, if you were working with a database, and you want to preserve the data created by the container, you can add a persistent volume to the container.

To do this, first create a volume using the command

sudo docker create volume data_volume

This creates a data_volumes folder inside /var/lib/docker/volumes directory.

Then, wen you run the container, you can mount this volume inside the write layer with either of the two commands below. The — mount command is the recommended way as it is more verbose. Read more here.

docker run -d -v data_volume:/var/lib/mysql mysql

docker run -d — mount type=volume, source=data_volume, target=/var/lib/mysql mysql

This method is called Volume mounting.

What if we had our data already at another location? For example, in /data/mysql?

We can provide the full path to the folder just like the volume.

docker run -d -v /data/mysql:/var/lib/mysql mysql

docker run -d — mount type=bind, source= /data/mysql, target=/var/lib/mysql mysql

This method is called Bind mounting.

We have two types of mounts:

Volume mount: mounts a volume from the volumes directory.

Bind mount: mounts a directory from any location on the docker host.

If you start a container with a volume that does not yet exist, Docker creates the volume for you.

So who is responsible for all of this operations?

Maintaining the layered architecture, creating a writable layer, moving files across layers and etc. is the responsibility of the Storage drivers.

Docker uses storage drivers to enable layered architecture. Some of the common storage drivers are AUFS, ZFS, BTRFS, Device Mapper, Overlay, Overlay2.

The selection of the storage drivers relies on the underlying operating system (OS). For example, the default storage driver for ubuntu is AUFS. Docker will choose the best storage driver available based on your OS.

Well, that’s it for now. I hope this helps. More advanced topics about Docker, Docker compose, Docker swarm… coming soon! Follow me not to miss them.