I’m not sure why this is surprising. Docker specifically and container runtimes in general all need to set up virtual networks and enable port forwarding between them and the host. There’s no way that it can do this without hooking into your firewall in some way.
This causes some minor issues if you use WSL2 with Ubuntu on Windows because WSL2 uses a Microsoft-provided kernel and a distro-provided userland and the kernel is too old for the default firewall in the most recent Ubuntu. On FreeBSD, the container runtime packages generally require pf and ask you to add a hook in your pf.conf for them to use.
About half of the surprise came from having hosted software in more traditional ways for quite a long time, and from this being my first experiment with putting a container somewhere I might eventually care about. (To be clear… this was part of a pre-flight test. There was no data there yet.) When I write Listen 8000 in my apache httpd.conf file while my firewall is set to default deny, I don’t expect httpd to punch a hole for me. I didn’t expect ports: 8000:8000 to do that, either, even though I understand your and others’ point that there’s a decent reason for it.
Upon reflection, the more jarring half of the surprise was ubuntu’s default tool, ufw, completely failing to report the change.
It’s not surprising (to me) but a major reason why I dislike docker. If you operate some tiny VPS with docker on it, but like to use NFTables to lockdown stuff, it’s very annoying how docker essentially doesn’t integrate into that. Even more when you’re not only running docker stuff.
I’ve only ever tried networked containers on FreeBSD, it my understanding was that the tooling is a lot more mature on Linux so this surprises me. The default on FreeBSD is to specify the interface to use for the containers. In the simple case, this is your public network interface but if you want to lock things down then you use a virtual interface that has a firewall between it and the rest of your system, and then explicitly open ports for containers that you’ve deployed.
As I understand it, most of the network configuration is normally driven by the CNI plugin, which is mostly intended for cloud deployments where the host is doing nothing other than providing a container runtime and then network needs to be dynamically configured based on the containers that you’ve pushed to it.
+1. On one of my machines I had carefully written nftables rules. Then at some point I installed docker and ran some application using docker. This resulted in my firewall rules breaking in odd ways, and IIRC I ran a dns server in a VM that suddenly wasn’t available. After I discovered the cause I purged docker from the machine, and am now more hesitant to use docker.
I think the worst part of the surprise came from the intersection of docker and ufw. I’ve gotten used to using ufw on ubuntu hosts. And the way that docker changed the firewall rules did not trigger any report in the ufw status verbose command.
Also, I’d have caught this in my own testing before I invited anyone to use my web app. Because I always do check my database and app server from a machine that shouldn’t be able to connect to them. But I’m impressed that the internet found its way in before I got ’round to checking that.
Not to rub salt in the wound, but this is one of the things that docker networks are for (don’t have to expose a single port to the host other than caddy!) As for the post itself, it’s never good if we are surprising users when it comes to their security posture. I would definitely recommend just running caddy in your compose file though, it gives you DNS
This is necessary, as others have pointed out, but I agree that it shouldn’t be surprising to users.
In particular, this one tends to catch people out when they restart iptables for some reason. A restart flushes all the firewall rules, meaning that it breaks Docker’s networking.
If you don’t know that Docker is managing firewall rules, it can look quite mysterious until you figure out that Docker wants restarting so it will reapply its rules.
It can also interact badly with tools like iptables-save / iptables-load, although vanilla Docker isn’t as bad for this as services like Kubernetes’ kube-proxy. If the created rules refer to ephemeral objects (for example, specific container interfaces) that aren’t there on a reboot, the saved rule set will fail parsing and will not be loaded.
I’ve not found a “standard” way out of that second one yet, sadly.
iptables requires that the firewall rules are taken in and out of the kernel in one atomic blob. There’s no way for two programs to agree on “my rules” and “not my rules”.
nftables solves this by providing a default set of chains which different rules providers can “hook” into. Now firewalld can manage its firewalld hook, libvirt can manage its libvirt hook, and docker can manage its docker hook. They can all coexist happily without treading on each others toes.
I am vaguely aware that some container software has not adopted nftables yet, partly because they chose to work on their own userspace implementation of things instead.
I am vaguely aware that some container software has not adopted nftables yet, partly because they chose to work on their own userspace implementation of things instead.
This is the problem, yes - as far as I’m aware, Docker doesn’t natively support nftables so if you want to take this approach, you’re left doing a fair amount for yourself. It’s possible, but it’s not standardised!
Although it’s not directly to the point of the article, kube-proxy et al are in a similar boat and potentially worse due to the amount of ephemeral configuration they manage. I’ve approached this in the past by using iptables rule comments to filter out rules that shouldn’t be saved, but it’s not a good answer.
I’m not sure why this is surprising. Docker specifically and container runtimes in general all need to set up virtual networks and enable port forwarding between them and the host. There’s no way that it can do this without hooking into your firewall in some way.
This causes some minor issues if you use WSL2 with Ubuntu on Windows because WSL2 uses a Microsoft-provided kernel and a distro-provided userland and the kernel is too old for the default firewall in the most recent Ubuntu. On FreeBSD, the container runtime packages generally require
pfand ask you to add a hook in yourpf.conffor them to use.About half of the surprise came from having hosted software in more traditional ways for quite a long time, and from this being my first experiment with putting a container somewhere I might eventually care about. (To be clear… this was part of a pre-flight test. There was no data there yet.) When I write
Listen 8000in my apache httpd.conf file while my firewall is set to default deny, I don’t expect httpd to punch a hole for me. I didn’t expectports: 8000:8000to do that, either, even though I understand your and others’ point that there’s a decent reason for it.Upon reflection, the more jarring half of the surprise was ubuntu’s default tool,
ufw, completely failing to report the change.It’s not surprising (to me) but a major reason why I dislike docker. If you operate some tiny VPS with docker on it, but like to use NFTables to lockdown stuff, it’s very annoying how docker essentially doesn’t integrate into that. Even more when you’re not only running docker stuff.
I’ve only ever tried networked containers on FreeBSD, it my understanding was that the tooling is a lot more mature on Linux so this surprises me. The default on FreeBSD is to specify the interface to use for the containers. In the simple case, this is your public network interface but if you want to lock things down then you use a virtual interface that has a firewall between it and the rest of your system, and then explicitly open ports for containers that you’ve deployed.
As I understand it, most of the network configuration is normally driven by the CNI plugin, which is mostly intended for cloud deployments where the host is doing nothing other than providing a container runtime and then network needs to be dynamically configured based on the containers that you’ve pushed to it.
+1. On one of my machines I had carefully written nftables rules. Then at some point I installed docker and ran some application using docker. This resulted in my firewall rules breaking in odd ways, and IIRC I ran a dns server in a VM that suddenly wasn’t available. After I discovered the cause I purged docker from the machine, and am now more hesitant to use docker.
One thing I’ve appreciated about using Jail-based tools such as Bastille is the lack of assumptions on how exactly your firewall rules need to look.
I think the worst part of the surprise came from the intersection of docker and ufw. I’ve gotten used to using ufw on ubuntu hosts. And the way that docker changed the firewall rules did not trigger any report in the
ufw status verbosecommand.Also, I’d have caught this in my own testing before I invited anyone to use my web app. Because I always do check my database and app server from a machine that shouldn’t be able to connect to them. But I’m impressed that the internet found its way in before I got ’round to checking that.
Yeah, you have to explicily set
127.0.0.1:8000:8000(or some other available ip on the host) in order to limit it.Always love hearing about folks’ idea of unexpected behavior though!
Regardless if one thinks something like this is good or bad; A program should say something onto stderr if it edits a security setting like this.
Anyone know if Podman also does this?
I will check this afternoon and update.
Looks like podman now has moved towards nftables - it still does something a little similar AFAIK:
https://www.redhat.com/sysadmin/podman-new-network-stack
Let’s make it worse!
Docker + CNI bridges edit the NAT table making it non trivial to find if you don’t know where you’re looking.
Not to rub salt in the wound, but this is one of the things that docker networks are for (don’t have to expose a single port to the host other than caddy!) As for the post itself, it’s never good if we are surprising users when it comes to their security posture. I would definitely recommend just running caddy in your compose file though, it gives you DNS
In my experience docker is difficult to make work with non-docker services. How do you make sure the docker network exists when caddy is started?
run caddy in docker and only expose the caddy ports to the host
I start Docker using this script to avoid it polluting my iptables: https://github.com/jafarlihi/dotfiles/blob/master/scripts/bin/dockstart
Then I just do PAT using this script if I want the container to be able to reach Internet: https://github.com/jafarlihi/dotfiles/blob/master/scripts/bin/patinit
Not sure how well it cleans up for the recent versions, haven’t used it in a while. They might have added more crap.
This is necessary, as others have pointed out, but I agree that it shouldn’t be surprising to users.
In particular, this one tends to catch people out when they restart iptables for some reason. A restart flushes all the firewall rules, meaning that it breaks Docker’s networking.
If you don’t know that Docker is managing firewall rules, it can look quite mysterious until you figure out that Docker wants restarting so it will reapply its rules.
It can also interact badly with tools like
iptables-save/iptables-load, although vanilla Docker isn’t as bad for this as services like Kubernetes’ kube-proxy. If the created rules refer to ephemeral objects (for example, specific container interfaces) that aren’t there on a reboot, the saved rule set will fail parsing and will not be loaded.I’ve not found a “standard” way out of that second one yet, sadly.
The way out is nftables.
iptables requires that the firewall rules are taken in and out of the kernel in one atomic blob. There’s no way for two programs to agree on “my rules” and “not my rules”.
nftables solves this by providing a default set of chains which different rules providers can “hook” into. Now firewalld can manage its firewalld hook, libvirt can manage its libvirt hook, and docker can manage its docker hook. They can all coexist happily without treading on each others toes.
I am vaguely aware that some container software has not adopted nftables yet, partly because they chose to work on their own userspace implementation of things instead.
This is the problem, yes - as far as I’m aware, Docker doesn’t natively support nftables so if you want to take this approach, you’re left doing a fair amount for yourself. It’s possible, but it’s not standardised!
Although it’s not directly to the point of the article, kube-proxy et al are in a similar boat and potentially worse due to the amount of ephemeral configuration they manage. I’ve approached this in the past by using iptables rule comments to filter out rules that shouldn’t be saved, but it’s not a good answer.
On my laptop docker disconnects wifi…