My experience with jails is that they are much harder to use as the tooling around them is not as extensive or fully featured. It may well be possible but I imagine it would require much more manual effort.
Having used both docker and jails, the big difference for me is the Dockerfile. A single place with a single command to build my “server” is a killer feature. A jail in my mind is more like a VM than a container.
Hey @wezm, excellent write up and superbly timed as I’m about to go about building my own home infra on an Intel NUC after reading articles from Jessie Frazelle and Carolyn Van Slyck. I especially appreciated your thoughts on rejecting certbot which I’ve always just blindly used for being ‘too magical’. We should all take that approach more often I think.
A couple things that you didn’t explain:
Why you chose hitch as your reverse proxy over say Traefik or Caddy?
Did you consider writing a separate docker-compose file per service? Why did you end up going with a ‘monolith’ docker-compose file?
As well, did you consider creating a postgres instance per service?
Why LuaDNS over say CloudFlare? While having DNS history via your git history seems like a good pro, having to do a commit and push to make changes seems a bit unwieldy, no?
Why you chose hitch as your reverse proxy over say Traefik or Caddy?
Mostly momentum and personal preference. I was already using varnish on the old infrastructure and I like its, “Swiss Army Knife of HTTP”, nature. Hitch is by the varnish folks and is their recommended way to add TLS support.
Did you consider writing a separate docker-compose file per service? Why did you end up going with a ‘monolith’ docker-compose file?
No I never really considered a docker-compose per service. I guess my thinking all along was that this docker-compose file would describe all the services on the server. I think that probably makes expressing the dependencies easier (E.g. varnish depends on all the downstream services), but that’s just a guess.
As well, did you consider creating a postgres instance per service?
As you can probably tell from the article, I kind of like compactness and efficiency (although this is not a strict requirement). The idea of running three database servers to do what one can easily handle never really crossed my mind.
Why LuaDNS over say CloudFlare? While having DNS history via your git history seems like a good pro, having to do a commit and push to make changes seems a bit unwieldy, no?
I like the idea of having the config in text files that I can edit in my editor of choice instead of having to log into an admin UI and click around. Additionally LuaDNS makes it easy to share chunks of records with templates. So I only had to write the fastmail stuff once, then just include it in the other domains that need the same records.
I was using CloudFront before the move for linkedlist.org. They are ok, but they’re also just another tech giant, gaining more control over the internet. Basically I’m a sucker for the underdog.
I ended up going with Dokku for my setup. I find it provides a bit more structure than hand rolling everything, and it’s API is heavily inspired by Heroku which I’ve always found to work well. I did a write up on my setup here. There’s also Flynn which is similar to Dokku but with support for clustering.
Nice summary. I’ve built something similar previously also re-using docker-compose. It works quite well for small setups. One thing that I was missing was zero-downtime switching to new version of the app so I started new docker-compose project each time a new version was deployed, switched traffic to that one using iptables (it was a long time ago) and shut down the old one. That was triggered by a webhook from GitHub.
You can create a local Dockerfile and then build and use that Dockerfile’s image without a registry at all. The built image gets added to your local machines image pool. See https://docs.docker.com/compose/reference/build/
This tripped me up at first when using docker-compose as well (unnecessarily publishing images to Docker Hub).
There might be. As I was writing the section on the registry up I remembered that you can point the docker tools at remote machines so maybe it’s possible to push from one to the other.
Thanks for the excellent write-up. In case anyone else needs it, Gitlab offers a free easy to use docker container registry that I have found convenient for a similar application.
I built alpine-based docker images for a few websites, then pushed to the authenticated Gitlab docker registry. On the target host I had simple automation to docker pull the images & start the containers. Very much like the main article.
Ah neat. Do you know about Docker Machine? I’m curious how people who are pushing to multiple servers handle that and if there are reasons to prefer one approach over the other. I’m still just playing with this at home on one server.
Was there a reason you couldn’t use FreeBSD’s Jails for this? It seems you could have accomplished pretty much the same setup.
My experience with jails is that they are much harder to use as the tooling around them is not as extensive or fully featured. It may well be possible but I imagine it would require much more manual effort.
Can you provide an example? Because I pretty much always find jails less opaque/hand-wayvy/etc than docker.
Having used both docker and jails, the big difference for me is the Dockerfile. A single place with a single command to build my “server” is a killer feature. A jail in my mind is more like a VM than a container.
Hey @wezm, excellent write up and superbly timed as I’m about to go about building my own home infra on an Intel NUC after reading articles from Jessie Frazelle and Carolyn Van Slyck. I especially appreciated your thoughts on rejecting
certbotwhich I’ve always just blindly used for being ‘too magical’. We should all take that approach more often I think.A couple things that you didn’t explain:
hitchas your reverse proxy over say Traefik or Caddy?docker-composefile per service? Why did you end up going with a ‘monolith’docker-composefile?postgresinstance per service?githistory seems like a good pro, having to do a commit and push to make changes seems a bit unwieldy, no?Lastly, we should grab coffee sometime in MEL!
Thanks for reading!
Mostly momentum and personal preference. I was already using varnish on the old infrastructure and I like its, “Swiss Army Knife of HTTP”, nature. Hitch is by the varnish folks and is their recommended way to add TLS support.
No I never really considered a docker-compose per service. I guess my thinking all along was that this docker-compose file would describe all the services on the server. I think that probably makes expressing the dependencies easier (E.g. varnish depends on all the downstream services), but that’s just a guess.
As you can probably tell from the article, I kind of like compactness and efficiency (although this is not a strict requirement). The idea of running three database servers to do what one can easily handle never really crossed my mind.
I like the idea of having the config in text files that I can edit in my editor of choice instead of having to log into an admin UI and click around. Additionally LuaDNS makes it easy to share chunks of records with templates. So I only had to write the fastmail stuff once, then just include it in the other domains that need the same records.
I was using CloudFront before the move for linkedlist.org. They are ok, but they’re also just another tech giant, gaining more control over the internet. Basically I’m a sucker for the underdog.
Sure!
OT: very defensive disclaimer
Hah my gf said the same thing when proofreading it.
Edit: I toned the disclaimer down.
I ended up going with Dokku for my setup. I find it provides a bit more structure than hand rolling everything, and it’s API is heavily inspired by Heroku which I’ve always found to work well. I did a write up on my setup here. There’s also Flynn which is similar to Dokku but with support for clustering.
Nice summary. I’ve built something similar previously also re-using
docker-compose. It works quite well for small setups. One thing that I was missing was zero-downtime switching to new version of the app so I started new docker-compose project each time a new version was deployed, switched traffic to that one usingiptables(it was a long time ago) and shut down the old one. That was triggered by a webhook from GitHub.Is there a way to push Docker images directly to a remote host thus avoiding an external registry?
docker save myimage > myimage.tar and then scp or whatever
You can create a local Dockerfile and then build and use that Dockerfile’s image without a registry at all. The built image gets added to your local machines image pool. See https://docs.docker.com/compose/reference/build/
This tripped me up at first when using docker-compose as well (unnecessarily publishing images to Docker Hub).
There might be. As I was writing the section on the registry up I remembered that you can point the docker tools at remote machines so maybe it’s possible to push from one to the other.
Thanks for the excellent write-up. In case anyone else needs it, Gitlab offers a free easy to use docker container registry that I have found convenient for a similar application.
How did you make use of this?
I built alpine-based docker images for a few websites, then pushed to the authenticated Gitlab docker registry. On the target host I had simple automation to docker pull the images & start the containers. Very much like the main article.
Ah neat. Do you know about Docker Machine? I’m curious how people who are pushing to multiple servers handle that and if there are reasons to prefer one approach over the other. I’m still just playing with this at home on one server.