1. 44
  1.  

  2. 18

    The catch is that ufw status verbose implies that there is no open port which makes this even more dangerous, as it gives you a false sense of security if you’re used to dealing only with ufw.

    So from what I understand after reading a bit more over on the Orange Site, Docker adds its rule to the ipfilter DOCKER-USER chain, which is applied before the filter chain, which many people are using to filter out traffic. So even if you’re not using ufw, you might be affected by this footgun. For instance, fail2ban by default operates on the filter chain, which means docker containers are not being filtered the way you might expect.

    I also noticed a useful tool which plugs at least this ufw-specific hole: ufw-docker

    1. 17

      I always found firewalls on Linux in general a pain. ipfilter is hard to grok and use for many, and the official documentation is essentially just in a “reference guide” format with very little hand-holding. That most people are using frontends like ufw to generate the config demonstrates its shortcomings IMO and always reminded me of sendmail.cf.

      To be fair, nftables is better than iptables, though the nft CLI has gpg/git-level UX (or maybe even worse?)

      ufw is the CLI/interface iptables should have had in the first place, and without this extra layer sandwiched in-between the scope of the problem would be greatly reduced.

      This entire thing is really a combination of problems: iptables’ byzantine UI, Docker’s surprising frobbing of it without clear communication, MongoDB not using auth by default and binding to localhost instead of a socket (but hey, at least it no longer binds on all ports by default), and finally the sysadmin not setting up MongoDB auth (as they mentioned in the article).

      1. 2

        I always found firewalls on Linux in general a pain. ipfilter is hard to grok and use for many, and the official documentation is essentially just in a “reference guide” format with very little hand-holding. That most people are using frontends like ufw to generate the config demonstrates its shortcomings IMO and always reminded me of sendmail.cf.

        I have too, but a lesson here IMO is to not trust yourself given that and test test test :)

    2. 9

      First, kudos to the author at NewsBlur for their radical candor in this write-up. I love post mortems with nitty gritty details to dig into :)

      Second, something I didn’t see addressed was why this wasn’t addressed in test? One of the big wins Docker brings to the table if I understand correctly is that it makes it trivial to stand up a test cluster of your infrastructure, even if it’s all running on your own machine.

      Third, sounds like after the build the test infra mentioned above they might consider adding some sort of basic security check to their CI loop. Having a MongoDB exposed to the world raised my hackles the second I heard about it given the rampant ransomware-ing of MongoDB instances.

      1. 2

        It’s hard for me to express how awesome Sam Clay is. I’ve been a newsblur user since 2012, and the fact that he’s been able to single handedly run such a massively complex site and communicate to his customers in a really open and transparent way is something that everyone who calls themselves a web entrepeneur should strive to emulate.

        1. 1

          That’s great to hear. I’ve been using and liking Feedly but I’ll give them a look.

      2. 4

        Something like Consul service mesh would fix this, I imagine. Although it’s fixing it by adding more stuff, which is slightly distasteful given how much stuff we already have on top of the other stuff.

        I’m reminded of a Monty Python sketch: https://www.youtube.com/watch?v=LFrdqQZ8FFc – I choose to believe that they very fully cognizant in predicting the software world of the 2020s.

        1. 4

          I think on the security side this boils down to: Don’t use unauthenticated database setups and don’t use software that is either badly designed or not well understood.

          The latter seems to be a common pattern when it comes to setups by people that evidently have put work in. Choosing solutions one can comprehend can both help on the stability and security side of things.

          1. 15

            Don’t use unauthenticated database setups and don’t use software that is either badly designed or not well understood.

            You just removed 70% of software from the pool.

            1. 22

              Only 70%?! You’re an optimist, I see.

              1. 2

                That’s exactly my point though. Why does it have to be like that in first place and how can we even start talking about security and stability when that’s the case? Maybe it would make sense to thin things out. It appears that for various projects (OpenBSD) that process of reducing the “pool” seems to work, yet the thought seems to rarely occur when thinking about what a company’s stack should consist of.

                At least in my experience it’s common to pull in giant pieces of software (services, libraries, etc.) without even giving it more than a minute of thought, and if there actually is that thought the reasons are on the level of “others do it”, “it’s by someone who worked at Google”, “it has a lot of stars on GitHub”, so not technical reasons and certainly not reasons telling you whether any of fitting the criteria.

                1. 4

                  I think the reason this happens is because of how business pressures are: just shovel something out the door as fast as you can and deal with issues later. 9 times out of 10 there are no issues (or only very deeply hidden ones that surface after a long time) so you can just invoice the customer and move on to the next thing.

                  This kind of thinking and behaviour carries over to personal projects too. Once you’re used to working in a certain way that’s going to influence everything you do.

                  EDIT: Also see the article on the front page today about the GitHub copilot - shoveling code in bulk at an even faster pace!

            2. 4

              Rather than publishing ports on any address using Docker (or, the same with podman), I publish to localhost or to a private subnet address (e.g. a virtual network for a virtual machine on a dedicated server, or usually a WireGuard tunnel), then I use a reverse proxy (e.g. nginx or HAproxy) on the front server (if different). This way I’m certain of what to whitelist and rate limit on my stateful firewall setup.

              In the case of a database, it doesn’t need to be accessible to the Internet, so I can just bind into a WireGuard tunnel (though I have not yet looked into WireGuard failover which is important for replicated databases).

              1. 2

                So not strictly related to the content of this specific article but: what’s with the term “footgun”?

                The phrase “shooting yourself in the foot” does not mean there’s some magical gun always aimed at your foot called a “footgun” that goes off unexpectedly. It means you mishandled something and made a mistake. The term “footgun” makes it sound like the interpretation is “ah, there’s literally nothing I could’ve done to prevent this ¯\_(ツ)_/¯” when that is, in general, not the case.

                1. 16

                  A footgun is a common term, it’s for when something seems to be explicitly designed for a user to shoot themselves in the foot with. The design of the gun leads itself to mishandling. In this article, Docker port exposure is a footgun, since in normal usage it bypasses the firewall completely, which is a dangerous default behavior that the user does not expect.

                  1. 4

                    The mental image it gives me is a gun where the barrel curves downwards.

                2. 2

                  Does k8s also suffer from this in the default configuration?

                  1. 3

                    The short answer is ‘no’. This is among the reasons for CNI and why, with Calico say, you often see a boatload of interfaces on the node.

                    The right way for all of this was mentioned in an above comment – create and bind and specifically route a virtual subnet/interface – which is what CNI is automating.

                  2. 1

                    Your server is not private if it has a public IP.

                    I really wish infrastructure providers had the option to not assign a public IP at all at instance creation and only hook it up in a private network.