It’s literally the best article on Docker I have ever read! It explains the anatomy of Docker in great detail and gives a nice overview of its ecosystem. It took me forever to understand the relationship between docker, containerd, and nerdctl before reading the article, but everything seems so clear now. Thank you!
Kind of an unreasonable request, but do you have any plan to write a similar article on Kubernetes (and its friends like K3s and Rancher)? Container orchestration systems are complex behemoths so I believe many people will benefit from your skillful dissection :)
Re: Kubernetes follow-up, you’re not the first person that’s requested that; I’m considering it but it’ll probably take me a while to actually put it together, since I have much less hands-on experience with that side of things.
This one is informative as always (I finally understand what a Kubernetes distribution is). I just submitted a story so that crustaceans can discuss it!
By the way, what in your opinion leads to Docker’s wide adoption? As you said, its predecessors already have bind mounts, control groups, and namespaces, so apparently Docker’s only novelty is that it encourages “application containers” instead of “system containers”, but I guess it’s also possible to create lightweight LXC images by removing unnecessary stuff like systemd and the package manager. Does Docker’s philosophy give it any advantage over LXC… or maybe Docker is successful simply because it’s backed by Google? :)
I think it was really all about the user and developer experience. It doesn’t take much effort at all to run a Docker container and very little additional effort to create one. Docker made sure they had a library of popular applications pre-packaged and ready to go on Docker Hub. “Deploy nginx to your server with this one command without worrying about blowing up the rest of the system with the package manager, no matter what distro you’re using” and “Take that shell script you use to build and run the app and reproducibly deploy it everywhere” are very attractive stories that were enabled by the combination of application containers, Dockerfiles, and Docker Hub.
That and crossplatform. Even back when we were still on Vagrant+Virtualbox I stopped using lxc as the only Linux user in the team because I had to duplicate some effort. Then with docker it just worked on Linux and Macs.
Cross-platform is a bit of a stretch to describe Docker. The Mac version uses a port of FreeBSD’s hypervisor to run on top of the XNU kernel and deploys a Linux VM to run the actual contain, so you’re running your app on macOS by having macOS run a Linux VM that then runs the application. That’s very different from having your app run natively on macOS.
The underling OCI container infrastructure supports native Windows and (almost supports) FreeBSD containers, but you can’t run these on other platforms without a VM layer.
When we want to run “a service using e.g. the Debian RabbitMQ package” then a docker container with a debian image with the rabbitmq package is 100% “usable cross-platform”, from the developer standpoint of “I have this thing running locally”. It exactly does not matter what the host is, you just run that image. I don’t care about “native or not” in that scenario, just that it runs at all. Purely an end user point of view.
Stacker is an OCI image builder that uses YAML-based build files, and you write your filesystem changes with a shell script instead of a new syntax. We use it heavily at Cisco.
I remember those from a 2016 paper, so it’s cool that there are open source implementations of it.
Although my pet peeve is that I think there shouldn’t be such things as container registries. We should just have general purpose storage with differential compression, which can store containers or other data.
For example pulling containers on sourcehut 10x slower because I guess there are no container caches on the local network, like there are in most clouds. But I think that is the “fault” of the whole tightly coupled container architecture.
Does anyone store containers in a place other than a registry? It looks like there is an optional IPFS backend, and I’m interested if anyone uses it, although my suspicion is that it probably works very poorly and slowly (since I mostly hear bad things about how well IPFS works)
I would like to experiment with storing container metadata in git and then the layers on a plain HTTP server (e.g. served by Nginx). I don’t see any reason that’s not possible? What does the container registry really add?
Tools to construct a read-only container ‘registry’ served by plain Nginx
Oh nice that’s basically what I’m looking for ! It doesn’t seem widely used, which makes me think there must be some downside like performance … but it’s worth checking out and understanding why, thanks.
edit: I guess the obvious downside is being read-only, which makes sense since plain HTTP has no way to persist data. So you can’t do docker push. But yeah I still think there is some potential to get rid of docker registries.
One bummer here is that it doesn’t seem to be fully static; it relies on a bit of custom nginx config to do URL rewriting and to munge error pages;. It doesn’t result in something you could just dump into an S3 bucket or moral equivalent - I’m not sure if such a thing is even possible, or the endpoints required by the clients are such that at least a little logic is required.
The registries add the distribution API. Most registries implement anything beyond the API calls that aren’t static (e.g. /v2/<name>/tags/, various forms of auth, the push flow) by redirecting to static storage.
Container registries are general purpose storage if you squint (and there are people working on making it “official”), with the added benefit of being able to programmatically understand how the pieces are related.
Oh, hey, I wrote this.
It’s literally the best article on Docker I have ever read! It explains the anatomy of Docker in great detail and gives a nice overview of its ecosystem. It took me forever to understand the relationship between docker, containerd, and nerdctl before reading the article, but everything seems so clear now. Thank you!
Kind of an unreasonable request, but do you have any plan to write a similar article on Kubernetes (and its friends like K3s and Rancher)? Container orchestration systems are complex behemoths so I believe many people will benefit from your skillful dissection :)
Thanks so much!
Re: Kubernetes follow-up, you’re not the first person that’s requested that; I’m considering it but it’ll probably take me a while to actually put it together, since I have much less hands-on experience with that side of things.
I wrote a follow-up: https://lwn.net/SubscriberLink/905164/c9cf0b577773df10/
This one is informative as always (I finally understand what a Kubernetes distribution is). I just submitted a story so that crustaceans can discuss it!
By the way, what in your opinion leads to Docker’s wide adoption? As you said, its predecessors already have bind mounts, control groups, and namespaces, so apparently Docker’s only novelty is that it encourages “application containers” instead of “system containers”, but I guess it’s also possible to create lightweight LXC images by removing unnecessary stuff like systemd and the package manager. Does Docker’s philosophy give it any advantage over LXC… or maybe Docker is successful simply because it’s backed by Google? :)
I think it was really all about the user and developer experience. It doesn’t take much effort at all to run a Docker container and very little additional effort to create one. Docker made sure they had a library of popular applications pre-packaged and ready to go on Docker Hub. “Deploy nginx to your server with this one command without worrying about blowing up the rest of the system with the package manager, no matter what distro you’re using” and “Take that shell script you use to build and run the app and reproducibly deploy it everywhere” are very attractive stories that were enabled by the combination of application containers, Dockerfiles, and Docker Hub.
That and crossplatform. Even back when we were still on Vagrant+Virtualbox I stopped using lxc as the only Linux user in the team because I had to duplicate some effort. Then with docker it just worked on Linux and Macs.
Cross-platform is a bit of a stretch to describe Docker. The Mac version uses a port of FreeBSD’s hypervisor to run on top of the XNU kernel and deploys a Linux VM to run the actual contain, so you’re running your app on macOS by having macOS run a Linux VM that then runs the application. That’s very different from having your app run natively on macOS.
The underling OCI container infrastructure supports native Windows and (almost supports) FreeBSD containers, but you can’t run these on other platforms without a VM layer.
I think you’re talking about something else.
When we want to run “a service using e.g. the Debian RabbitMQ package” then a docker container with a debian image with the rabbitmq package is 100% “usable cross-platform”, from the developer standpoint of “I have this thing running locally”. It exactly does not matter what the host is, you just run that image. I don’t care about “native or not” in that scenario, just that it runs at all. Purely an end user point of view.
Since some of the discussion of this is about dockerfiles and building containers, take a look at Stacker: https://github.com/project-stacker/stacker
Stacker is an OCI image builder that uses YAML-based build files, and you write your filesystem changes with a shell script instead of a new syntax. We use it heavily at Cisco.
Great article!
FWIW I’m using docker and podman with the same images and it works well. Our CI runs on two configurations the same way:
There were some minor permissions issues on the mounts, but it wasn’t hard to work around.
So I’m glad that there are two implementations, so I don’t get locked into Docker.
Some of the links led me to these lazy pulling projects with associated file formats.
https://github.com/containerd/stargz-snapshotter
https://github.com/containerd/nerdctl
I remember those from a 2016 paper, so it’s cool that there are open source implementations of it.
Although my pet peeve is that I think there shouldn’t be such things as container registries. We should just have general purpose storage with differential compression, which can store containers or other data.
For example pulling containers on sourcehut 10x slower because I guess there are no container caches on the local network, like there are in most clouds. But I think that is the “fault” of the whole tightly coupled container architecture.
Does anyone store containers in a place other than a registry? It looks like there is an optional IPFS backend, and I’m interested if anyone uses it, although my suspicion is that it probably works very poorly and slowly (since I mostly hear bad things about how well IPFS works)
I would like to experiment with storing container metadata in git and then the layers on a plain HTTP server (e.g. served by Nginx). I don’t see any reason that’s not possible? What does the container registry really add?
https://github.com/NicolasT/static-container-registry is a thing but I haven’t gotten around to trying it so no idea if it’s any good
Oh nice that’s basically what I’m looking for ! It doesn’t seem widely used, which makes me think there must be some downside like performance … but it’s worth checking out and understanding why, thanks.
edit: I guess the obvious downside is being read-only, which makes sense since plain HTTP has no way to persist data. So you can’t do docker push. But yeah I still think there is some potential to get rid of docker registries.
One bummer here is that it doesn’t seem to be fully static; it relies on a bit of custom nginx config to do URL rewriting and to munge error pages;. It doesn’t result in something you could just dump into an S3 bucket or moral equivalent - I’m not sure if such a thing is even possible, or the endpoints required by the clients are such that at least a little logic is required.
There are some tools in https://github.com/ostreedev/ostree-rs-ext/ for importing and exporting container images from an ostree repo; it’s on my list of things to try out.
The registries add the distribution API. Most registries implement anything beyond the API calls that aren’t static (e.g.
/v2/<name>/tags/, various forms of auth, the push flow) by redirecting to static storage.Container registries are general purpose storage if you squint (and there are people working on making it “official”), with the added benefit of being able to programmatically understand how the pieces are related.
Check also cvmfs https://cvmfs.readthedocs.io/en/latest/cpt-containers.html