1. 56
  1. 11

    How do you keep all your software updated? I’m curious about the monthly maintenance needed, how many mailing lists you subscribe to, etc.

    1. 9

      Nice article.

      I know that this is an area that could use improvement; it’s quite manual right now. I’d be interested to hear about some kind of light-weight solution for this that people have come up with.

      You can look at a simple docker-compose setup as managing 67 containers just with docker run can get messy. I’ll say if you want to get a little more sophisticated, you can look at writing a simple ansible playbook that will template out different docker-compose.yml files on the server and use that to manage stuff like image updates, config changes etc.

      1. 10

        To most people, Docker and Ansible are anything but simple.

        1. 6

          Of course it’s relative… to most people, running even a single web app/site is anything but simple.

          But obviously if you can read the article and understand what’s going on, then OP’s suggestion of using docker compose and/or ansible makes sense.

          1. 6

            I say this completely seriously and not intending to throw any shade at the OP. Over the past year I have tried to recognize how and where I use words like “simply” or “just”, particularly when it applies to technical instructions, and kill them off for exactly that reason.

            What I find simple someone else might not. I might find it simple because I have lived it for 10 years and not realize my bias. The person I’m speaking to or writing for might read those words (as you’ve noted) and think “That’s not simple, I can’t do this.” To some, words like those come off as arrogant (even if not intended that way).

            Instead, I now try to explain things in a simple way, if I think it is simple, or give clear examples that step the reader through it. If it really is simple, they can skip the step-by-step and if it isn’t for that reader, they have what they need to start accumulating the experience needed for it to seem simple the next time.

            1. 1

              Sure, but what are the alternatives, genuinely?

              1. 4

                Pyinfra is a great replacement for Ansible for these types of tasks. Nomad would also be a great way to orchestrate some containers with little fuss. So would be Docker Swarm, in my opinion.

            2. 4

              I tried using just docker-compose with a similar project last year but I found it required a fair amount of maintenance to keep up during deploys.

              I’m considering redoing it using Harbormaster (compose file watcher) so that updating the source replicates to the servers.

              https://gitlab.com/stavros/harbormaster

              1. 3

                I’m using systemd-docker at the moment to run docker containers in systemd (at work, in my personal life I don’t like docker). I find this very nice, because then I have all services in a standard systemd environment and can fully use dependencies between systemd units and so on. If I remember correctly, systemd also has some built-in container runtime (nspawn?).

              2. 8

                If someone is able to get root on the host, it’s really game over. If an attacker is somehow able to escape from a container, things don’t look very good either.

                If you do want to take the next step in the security department (not that it’s strictly needed, as you point out), you could consider either non-root containers (pdf page 7) or rootless docker.

                The benefit is that you can keep the server protected even if an attacker gains remote code execution on a single container (pretty common nowadays in the age of log4shell). Containers running as non-root users present a significant hurdle to attackers trying to break out the container since many kernel code paths are not even reachable by unprivileged users. So future zero-day container breakouts like this are much less likely to affect you.

                For rootless mode, containers still run as a user they think is root (UID=0) but that UID is remapped to an unprivileged UID on the host. So an attacker who breaks out of the container ends up becoming an unprivileged user who can’t do much damage.

                I’m selfishly posting this because all the blog posts and stuff on this topic are garbage and the concept is pretty new, so like 99% of Dockerfiles out there run as root for no reason, and I’m hoping more people join the non-root party :)

                1. 3

                  I would switch from Docker to Podman if you want to run rootless containers.

                  1. 2

                    Could you elaborate on the reasons to choose one over the other?

                2. 7

                  I feel Caddy would be a nice fit for automating stuff even more. Auto LE and support for brotli & QUIC.

                  If the author reads this: thoughts?

                  1. 2

                    Yeah I hadn’t looked into Caddy before, but it’s been mentioned by a few people and I agree it looks pretty ideal. The only thing that would hold me back from switching is that I’d have to re-configure all my configs that I’ve built up so far.

                    That being said if I had to do it over again, I’d definitely go with it

                    1. 3

                      It doesn’t seem super supported, but if you wanted to just throw your nginx config at caddy and see if it sticks: https://github.com/caddyserver/nginx-adapter

                    2. 1

                      On a recent personal server update, I looked at Caddy, and decided not to go with it because it’s not in Ubuntu proper yet. I considered building from source, and the Cloudsmith repo / source, but opted for lower potential maintenance headaches.

                      1. 1

                        Alternatively traefik if you’re using docker already. Being able to configure the whole routing automatically using container labels is great. I’ve been enjoying that much more than keeping a global config. Having all image/networking/volumes/proxy config in one place (docker-compose) makes sense to me.

                        1. 2

                          Yeah. Been using traefik myself at times. Two good contestants.

                          1. 1

                            Yeah I hadn’t looked into Caddy before, but it’s been mentioned by a few people and I agree it looks pretty ideal. The only thing that would hold me back from switching is that I’d have to re-configure all my configs that I’ve built up so far.

                            That being said if I had to do it over again, I’d definitely go with it

                            EDIT: Whoops I responded to the wrong comment! Traefik looks cool as well, but I make pretty heavy use of the static fileserver aspect of NGINX so I’d probably have to deploy something else for that as well if I went with Traefik

                            1. 1

                              Traefik isn’t a web server, so it can’t serve files. I’ve wished for a combo between caddy and traefik for a long time. I tend to put nginx behind traefik, but have wanted to try caddy since the licencing change.

                              1. 1

                                there’s a caddy plugin that parses docker labels for config

                                1. 1

                                  Hot dang! Thanks so much!

                          2. 4

                            The only thing missing from this setup is a tool like salt or ansible. I personally prefer salt.

                            The docker run scripts should be written down in a salt controlled file, and checked in with the rest of the states. The setup of nginx should be similarly captured. All the packages you installed, all the one off static groups of files, all of that should end up captured. Suddenly you fix corrupted files whenever you apply your configuration. You also know with certainty the system has not drifted from what you have captured. Every backup script and location is assured present.

                            Being able to describe the entire machine in an idempotent set of applied roles (ansible) or states (salt) becomes a super power when debugging or figuring out what’s wrong with a replacement system.

                            I would consider a tool in this class a minimum for new machines. I even use salt to configure my personal laptop (a recent machine).

                            1. 5
                              1. 3

                                The first step is firewall. I use ufw, a wrapper around iptables, and default to denying all access to everything other than ports 80, 443, and the non-default port I run my SSH server on.

                                Did you test it?
                                https://github.com/moby/moby/issues/4737
                                https://blog.newsblur.com/2021/06/28/story-of-a-hacking/

                                While searching for these links, I stumbled upon https://github.com/chaifeng/ufw-docker which people seem to love.

                                1. 2

                                  I make use of Podman + Openrc on Alpine to run apps (deployed via Ansible). It works surprisingly well, using this openrc script that can just get copied to /usr/local/etc/init.d/{{ image name }} and as long as that podman image exists it will work. There are some tweaks to look into, such as looking into s6 or native openrc cgroups (though it looked a bit sparse), but for the most part, it works surprisingly well.

                                  1. 2

                                    A simple low hanging fruit improvement: name the containers with some convention using the subdomain you want to expose them to, then route requests to them dynamically in your nginx config. This eliminates the unnecessary stop or adding/changing an nginx config each time you add a service.

                                    1. 2

                                      I love this post, I read most of it with much interest.

                                      On the orange site there’s been talk about email hosting and I’ve been thinking about moving mine to a “real” server. I’m currently paying DO about a hundred dollars for several servers…this can be combined to mirror what OP has done.

                                      All this to say I’m gonna investigate my options in the AM, thanks for sharing.

                                      1. 6

                                        I used to rent a dedicated server and I’ve moved entirely to using cloud VMs instead. The problem with dedicated servers is that failure is your responsibility.

                                        If a hard disk fails in an Azure datacenter, then I won’t notice. Even if it’s storing some of my data, something in the background will transparently resilver. If I have a disk die in a dedicated host (which happened to me twice), then I need to buy a new one, pay for remote hands at the hosting company to install it, and then restore from backups if I’m not doing some form of RAID, or wait with degraded performance for the resilver if I am (and hope that the other disk doesn’t fail at the same time).

                                        Similarly, if a CPU or RAM fails catastrophically in a cloud datacenter, my VM will die, the node will be taken offline, my VM will be redeployed immediately and I’ll experience probably a minute of downtime. For other other kinds of fault (including RAM with too many recoverable ECC failures) the hypervisor will migrate my VM to a node that isn’t failing, or will try to remove the faulty resources from any assigned VMs. If I experience hardware issues with a dedicated machine, I have to buy a new one, pay for remote hands to move the disks over, and so on.

                                        On top of that, it’s much easier to pay for what I use with cloud hosting. Buying 4 TiB of cloud storage is more expensive than buying a 4 TiB disk, but if I’m only using 500 GiB now and just want to be able to expand later, I can do that trivially with cloud hosting and pay per GiB, whereas with dedicated hardware I either need to buy the big disks up front or have the same kind of issues as I’d have with hardware failure: needing to buy the new disks, schedule downtime, have remote hands install the new ones, and so on.

                                        1. 2

                                          If a hard disk fails in an Azure datacenter, then I won’t notice. Even if it’s storing some of my data, something in the background will transparently resilver

                                          FWIW this does very wildly depending on your luck. e.g. people I know using AWS and Rackspace Cloud (hey it was fairly good for a while, tbh) have had fairly unceremonious emails to the effect that “your VM and all the data on it is gone, deal with it”.

                                          The fact that I can set up new infra and restore backups to it straight away (provided us-east-1 doesn’t die again) is the useful part.

                                          1. 1

                                            Yeah that’s a very good point. I’ve not had issues with the hard drive or anything else so far, but I know it’s definitely a possibility.

                                        2. 1

                                          I have a house server in my home and have been looking at StaticDeploy quite a bit. Not sure if anyone’s got experience but it seems close to the phost tool from the article in select ways.

                                          1. 1

                                            I host a handful of services on an old junker laptop in my basement, using Ansible for config management: https://github.com/timmc/commapps (Matrix, Sandstorm and everything that runs on it, Jabber)

                                            Those scripts include configuring an LVM + LUKS for backups-friendly encrypted storage. (I use borgbase for hosting backups.)

                                            1. 1

                                              I use an old tool called Munin for monitoring my server and some services. It’s pretty fiddly to configure and can break easily, but it produces clear and useful charts. I might swap this out for Prometheus + Grafana or some other more modern alternative some day if I get bored.

                                              I just set this up the other day. I was previously using netdata, but I found that it used a lot of CPU and memory all the time. One thing I miss is per-process/service/application resource usage. It’s really helpful to be able to notice e.g. memory leaks over time.

                                              My backup solution is simple but effective. I have a variety of shell scripts that run periodically to back up databases and other things and upload them to Google Cloud Storage for backup.

                                              How do you manage backup rotation? I was looking into automated backups recently, and one of the issues I found was that

                                              • Most backup tools expect some kind of block storage, like an external hard drive or a directory to rsync to. Tools which support s3-style APIs are much fewer in number.
                                              • Supporting automatic rotation reduces the field even further.
                                              • Same for encryption, deduplication, and compression.

                                              The only tool I found meeting the above requirements is restic. But it’s a very opinionated tool. I was really impressed with borg, but it has the same “block storage” restriction, so you have to pair it with something like rclone.