1. 21

Since this is my first blog post (and I’m learning it by doing it) I’d be very thankful for some constructive criticism.

Thanks

  1.  

  2. 10

    Congrats on getting your server up and written about!

    The thing that strikes me as kinda odd–maybe I’m just showing my age–is that you seem to have not one, not two, but many webservers:

    • Caddy on the host machine
    • Apache in the Next Cloud (I think?)
    • Nginx in Gitlab (if you’re using the community edition image, which doesn’t look like it’s be updated in two years?)
    • Go server for Hugo (which, as a static site generator, should just be a directory of files to serve directly, no?)
    • Apache for Piwik
    • Node for Bookstack (unavoidable)

    Like, I can’t help but wonder if this is really an efficient use of resources. This sort of thing is why I view container-based solutions for ops with tremendous skepticism.

    1. 3

      Congrats on getting your server up and written about!

      Thank you very much!

      Apache in the Next Cloud (I think?)

      There is a version (tag) of the nextcloud image that does not use a webserver and only exposes the php-fpm port. I’m using that image.

      Thank you for reading my post so thoroughly - You’re mostly right. Except for the nextcloud service every other container has a dedicated web-server built in. This takes some additional memory, but I don’t think it’s relevant cpu-wise (perceived, not factual).

      Nevertheless: Yes, it comes with some overhead. And this surely is not a solution for everyone. But in my case, I’m very happy to be able to isolate all the services with containerization; The pleasure of having easy updates and clean isolation far outweighs the (IMO) slight overhead in computation. Although: With some extra effort I’d be able to remove the web-servers in most of the containers.

      1. 1

        Although: With some extra effort I’d be able to remove the web-servers in most of the containers.

        I’m curious - how would you do this and still keep gitlab isolated? You’d still be running it with thin/puma/unicorn rather than spawning with passenger, right?

        1. 2

          I’d configure gitlab to not use the integrated nginx server and configure caddy to serve gitlab accordingly :). I’ve not figured out the required caddy settings, but that’s on my agenda.

          All other daemons that are required for running gitlab you’ve mentioned would still run inside the docker container. Thus gitlab would be isolated, but without the nginx server.

      2. 3

        This is unfortunate, but also rather necessary for isolation. Often, apps depend on specific webserver settings (especially in the PHP world). If you’re going to pull those settings outside the container, that means you have to be aware of any changes during an upgrade.

        For what it’s worth, in our case at least, there’s only 2 webservers in the request path, and our apps that need their own webserver always use nginx. These instances of nginx have all caching and buffering disabled, and are at the very bottom in memory usage on the system.

        I’m not sure of the processing overhead per request of an extra webserver, but we’ve currently not hit any issues. The idea is to cache as much as possible at the front proxy, and the remainder that goes through the stack is heavier any way. My gut says the overhead is probably small compared to the rest of the app logic in PHP.

      3. 2

        This last 12 months of actions and reactions by the developers of Caddy make me wonder why anyone would still use it.

        1. 3

          It’s very useful and even open source, why not?

          1. 1

            In may when LE had an outage, Caddy servers with valid certificates in the renewable period would refuse to start. This was not an oversight, it was intended behaviour and it took a lot of complaints before they relented and adjusted the configuration. It will still refuse to start if a certificate is very close to expiry and LE is down, from memory.

            Later in the year they started injecting ads into repsonse headers for the downloaded “free” binary before again, relenting under a wave of backlash.

            Whenever the main developer is involved in a discussion about it where there is criticism or particularly comparison to alternative open source tools, his responses make it seem like he thinks requesting/renewing a certificate via letsencrypt is some kind of secret sauce that he alone provides to the world.

            Theres also that whole “separation of concerns” thing but that’s not specific to the last 12 months

            1. 3

              You could just set the certs explicitly in the caddyfile to work around that issue. And I guarantee you I could code, build, and deploy a hotfix removing that behavior in 15 minutes or less if necessary. But I can’t say the same for Apache or nginx. Actually I’d probably just shoot myself if I had to hotfix Apache.

              As for separation of concerns, I think a web server that handles all aspects of being a web server is a great idea. Certs in particular cause loads of trouble for amateur users.

              1. 1

                In no other TLS terminating program do you need to “deploy a hotfix” for fucking stupid behaviour.

                The problem with caddy is two fold: it tries to be “magic” and the “wizard” in control thinks he knows better than anyone else.

                1. 2

                  In no other TLS terminating program do you need to “deploy a hotfix” for fucking stupid behaviour.

                  Sorry, are you jokin my ass? You’ve NEVER heard of anyone having to deploy a config hotfix to Apache or nginx for stupid behavior? Hah, good one.

                  1. 1

                    Changing a config value isnt “code build and deploy a hotfix”.

                    1. 1

                      First sentence of my comment:

                      You could just set the certs explicitly in the caddyfile to work around that issue.

                      1. 1

                        Literally the next sentence of your comment:

                        I guarantee you I could code, build, and deploy a hotfix removing that behavior in 15 minutes or less if necessary

                        Changing the config to statically reference a certificate file isn’t a long term solution, because it will then turn off Caddy’s renewal of said certificate. So, what, you either keep manually adjusting the config whenever Caddy won’t start, or you have to modify the source and re-build the whole program? All to work around fucking stupid behaviour that was intentional.

                        1. 1

                          Yes, I could do either one. The point of the second sentence was not only is a config change easy, Caddy is so easy to work with I could do a code change instead if I wanted. Or roll out a config change first and a code change later. Or use an external LE client. I’ve had worse problems with other servers that were a lot harder to fix, and bitching about this one like it’s the end of the world sounds really amateur to me. And yes, the many problems in other servers are “fucking stupid behaviors that are intentional,” and still aren’t fixed.

                          Besides, how would you even hit this problem? Why are all of your frontend web servers restarting at the same time? If you only have one, why is it restarting unattended? I can’t imagine any skilled team having a second of downtime because of this.

                          1. 3

                            Why shouldn’t I restart my server? There’s no reason for me to expect it won’t come back.

                            1. 1

                              You shouldn’t make any unattended changes to production. If you’re restarting manually, you can just fix the issue with a config change trivially in < 1 minute. If you’re a company making an infrastructure change, you better have some real process that would catch any problem, not just this one.

                              1. 0

                                You can fix it “trivially” if you know how to fix it (which is highly unlikey given that the use-case for caddy is “you dont have to worry about certificates”) and if you’re manually restarting.

                                What if the software crashes and is restarted by your init?

                                What if the server has to restart due to a failure of some kind?

                                Its a fucking stupid concept and it was intentional. That should tell anyone all they need to know about how this project is run.

                                1. 2

                                  That should tell anyone all they need to know about how this project is run.

                                  I think you’re too willing to make black and white judgements, but ok.

                            2. 1

                              A skilled team wouldnt find certbot and haproxy/similar “too hard to configure” and turn to caddy.

                              I don’t know the exact situations that triggered the problem because I dont use the fucking thing, but it had enough people seeing the issue that the github issue was like a townhall meeting.

                              1. 1

                                No, they wouldn’t. But they might use Caddy if they didn’t want to spend the time doing so. Time is a finite resource.

            2. 1

              I literally only know about the header thing, where they relented. What else happened?

              1. 2

                Oh, the header is gone? I only remember adding it, missed the retraction.

                1. 6

                  Yep, it died.

                  To answer the grandparent, I’m going to be really blunt: Caddy has made some poor decisions that I disagree with and that make me disinclined to trust them. But so has Mozilla, as we’ve discussed recently. And, very recently, so has Apple, with the battery situation. Caddy, at least, was very straightforward and transparent in what they were doing, whereas these other companies were not. And in the modern context, where most open-source projects are sponsored by major companies, transparency in what’s going on is close to the best I can ask for. Toss in that building Caddy without that header at any point required, let’s be honest, the bare minimum of effort, whereas e.g. building Firefox without the Mr. Robot plugin did not, means I don’t personally have any trouble continuing to use them.

                  1. 3

                    I didn’t know that either, good to know. And I mostly agree. As I see it, Matt is trying to make a living out of a great piece of software. I respect his attempt, although I’m really glad he dropped the propriety headers.

              2. 1

                What actions and reactions?

              3. 1

                Indeed Docker and docker swarm mode are great, I’ve recently finished a similar setup, I’ve replaced caddy with traefik which is able to listen to docker events and redirect automatically to the desired service without constantly editing a conf file (an alternative could be to use consul template). Also I’ve setup keepalive/haproxy in front of traefik to provide HA to traefik through a virtual IP. I’ll clean up the ansible recipes a little bit before uploading to Internet, although surely there are already similar setups online.

                https://imgur.com/a/pFT1U

                1. 1

                  This all seems sensible and nice. :)

                  One tiny nitpick: where you used the word “abstract” here, I’d have used the word “isolate”.

                  1. 2

                    Makes sense. Will remember it; thank you!

                  2. 1

                    Great setup, thank you for sharing it. I like the transparency of it. I’m currently using dokku, but it keeps evolving in ways I don’t understand very well.

                    1. 2

                      Great to hear! I’m going to write more detailed notes and instructions soon(ish), maybe those will inspire you to try it out.

                    2. 1

                      What is the hardware context? Physical machine at home? Cloud provider? Shared hosting? Rented root server? I would assume the last one.

                      1. 2

                        I started with a VM at vultr.com (2 core, 4G RAM), but then switched to a root server at Hetzner. Performance was good on both systems, I switched because GitLab recommends at least 4 GB dedicated RAM (although it worked fine, I personally like to fulfill those recommendations).

                        1. 2

                          And I have a similar setup at home (with less potent hardware). There’s currently one container running (a Minio server) that takes my restic backups.

                          That machine could easily take my other containers if I’d have to migrate for some reason.

                      2. 1

                        I have been using Dokku for a while on my personal server. This gives you a similar setup as to the one in your post (swapping out caddy for nginx) but spares you the configuration of the proxy and gives you the option to deploy using git + heroku buildpacks next to Dockerfile deploys.

                        1. 1

                          I really like this setup. Unique and different compared to what I typically hear people running. my so far is pygopherd for gopher web server apache for normal http so that let’s encrypt is easier to setup hugo for the site

                          1. 1

                            Thank you very much!

                          2. 0

                            With respect to blogging:

                            I find the conversational style (“Hi” and “Thanks” etc) superfluous, but that is your choice.

                            I would have started with my requirements (blog, GitLab, https, etc) before diving into implementation details (docker, caddy, etc).