Static sites are easy to deploy: rsync to some directory served up by NGINX seems to work well. At the other end of the spectrum, there are big web apps deployed as containers in a Docker Swarm, Kubernetes or whatever. I’m interested in the middle ground: mostly static sites with maybe a contact form, or a searchable image gallery.
How do people deploy sites like these? I’m talking about the sort of thing you’d serve from a single VPS: cheap and cheerful. Not a high traffic or high value corporate site.
Back in the dark ages, when I last did web development, CGI was pretty much the only option. Is CGI still a reasonable option? I find it hard to believe that starting a process to serve a request is actually too slow for most websites… but maybe I’m just hopelessly out of touch. Would rsync of static assets and a CGI binary be a bad way to deploy a website in 2019?
How do you lot deploy your personal or hobby websites?
My deploy is a target in a
make deploy) that builds system-installable packages (for archlinux, because that’s what I deploy on, but it could reasonably be adapted for anything else if the tooling exists), copies the packages over to my machine via
scp, and uses
sshto both install the packages and restart the necessary services on the target.
This has worked really well for me for quite some time. Though, to be fair, my website isn’t hugely complex.
All the best,
I do all of this except I don’t do system packages anymore because I want a repo of my deployed built artifacts. Also I use a CI/CD system.
This sounds interesting, thank you. When you say that you “restart the necessary services on the target”, are those services part of the package that you’ve just installed?
So you have some services continuously running, serving HTTP or FastCGI or something behind nginx? And you use the OS’s service management to restart them when you deploy? Is that systemd on ArchLinux these days?
A little more detail should clear things up. My website is written in plain C using a library called lwan. I authored some very small
.servicefiles (for systemd, because that’s what Arch uses) which control the site(s). One of the packages my
deploytarget builds includes all the files for the site (assets,
.servicefiles, and the C programs themselves). The services that are being restarted are the website itself.
So, in answer to your specific questions: yes, the service files are provided by one of the packages. The services are not FastCGI, but they are indeed running an HTTP server. And yes, I use
systemd, because of Arch.
This is a topic for another thread and another time, but I may be on the cusp of switching away from archlinux, and the place I move will likely not have systemd (though that’s not the reason I am switching), so part of this infrastructure may need to be rewritten. Though, admittedly, not much: really just the service files themselves and this
deployprocedure to operate with the new, analogous tooling.
All the best,
Is that Lwan Web Server that you use as a library? I forgot the web site said it can be used that way. Looking at minimalist and securable servers, it was one of the most exciting things I found. I keep mentioning it in different threads so more people try or review it.
If that’s it, what’s its setup difficulty and performance like compared to Nginx? And I mean Nginx if you install a package followed by one of the setup guides out there that try to make it easy.
@nickpsecurity, indeed it is. I do not know if the official website documents this functionality. Actually, one of the only things that I do not love about lwan is its lack of documentation (for the most part, it is more useful to read the code).
I think it is excellent for performance, and it did quite well in the techempower benchmarks; though, I should say explicitly that I have not done exhaustive performance tests or comparisons with nginx. If using the binary form, it is quite simple to use (and configurable via lua, which is a usability plus in many ways). When used as a library, it is a bit more work, but the result is that you only have installed exactly what you need (which is something that speaks to me more and more).
If you are interested in how I set my site up, you can find the site itself at https://halosgho.st and the source-code (including the
deploytarget) on my github. I also idle on freenode much of the time (freenode account
halosghost); please feel free to ping me if you wish to have a more thorough discussion.
Appreciate it. Yeah, I like the install just what you need philosophy. Better docs is actually an easy area for folks to contribute. I’ll try to remember your offer if I do anything with it. :)
Thank you for going into detail - much appreciated!
I wouldn’t be so quick to write off static websites. The use cases you listed are entirely do-able with a static site, and modern generators (hugo, jekyll, pelican) are miles ahead of where static site generation was even a few short years ago.
I serve my blog with Github Pages, so deploying is
git push(wrapped in a script that does some other stuff like re-generating the static files). Things like comments (or photo galleries) can be embedded if you really want to go that route. Though there are often clever ways to accomplish stuff like this anyway (e.g staticman for comments).
I’m curious what innovations you’ve seen in static site generators in the past few years? I haven’t seen anything fundamentally different, but I’m curious if there’s something I’m missing!
I used to use and love Pelican for my blog, and I still have the utmost respect for that project and its maintainers (The code is solid and very approachable - something I really value.)
But I have zero web design talent and couldn’t get it looking the way I wanted, and also posting from a mobile device is pretty painful, so I turned back to the dark side and now use wordpress.com.
I use Caddy to serve static files. I couldn’t be happier with it: totally takes care of https which was a pain even with nginx and let’s encrypt.
Digital ocean droplet + namecheap = best money I’ve ever spent.
Edit: Oh yeah and rsync for actually moving the files. (Way faster than scp since it does diffing)
rsync -aPv --del. This is pretty typical for most of my little projects - I put it in a script because I know I’m going to forget steps after a week or two. Builds the site, minimizes assets, any random odds and ends, rsync. Could be a bash script, but there’s always that one thing I want a conditional or a regexp for and Ruby’s more comfortable for me.
The dynamic stuff is always PHP. I never love it, but it’s reliable, doesn’t want to be a framework that takes over the site, and there’s always a snippet or answer one google search away.
Thanks - a few people have said that PHP would suit this situation (i.e. mostly static with just a couple of dynamic pages) very well.
I have a VCS for the source, and a VCS for all of my websites deployed HTML/binaries/etc served on a single host(which is basically all of my personal stuff)
So my CI/CD will take commits from each individual src repo, build the code and then commit it to a subdir of a different VCS repo. Then my host runs a VCS checkout every 5 minutes from the combined deployable repo, and runs a script in that repo that will restart anything if needed.
so an example:
VCS blah.com sources in repo A VCS example.com sources in repo B deployable code for example.com and blah.com in repo C
when I push a commit for blah.com in repo A, my CI/CD will pull it down, and pull down repo c. then build and do any tests, and then push the built code into repo C under a blah.com/ dir.
Then every 5 mins my VPS host pulls down repo C and runs Make that will restart anything if needed. (under a special www-cron user)
This way repo C has all of my deployable websites and history of what has been deployed. Also, this way my CI/CD doesn’t get direct access to my VPS host. Since I don’t run the CI/CD (it’s a SAAS ) I don’t have SSH keys out there waiting to be gobbled up by the latest hack.
Using VCS to track the history of deployments is an interesting idea. I usually try to keep build artefacts out of VCS… but this is an interesting use case, thank you.
I agree, you don’t want built code/html/etc out in your VCS normally, and it’s only in the deployable www VCS that I put it, not in the source repo(s).
I do it this way for the reasons above, but also so that I can very easily go back and see what my website was on X day, it’s my own personal version of archive.org, done the lazy, lazy way.
Another way to do this would build system packages of your deployed code, and archive the built packages.. but using a VCS is actually a win here, since only the diff’s are stored (usually, depending on VCS).
slowcgi(8) on httpd(8) on OpenBSD works for me :~)
My static sites are currently served by httpd(8) on OpenBSD. I have been thinking about using slowcgi(8)! That’s pretty much what prompted this question, actually :)
CGI (and fastcgi) still exists, but even very simple frameworks often use http now, with nginx serving as a proxy.
So if I use one of these frameworks to create an HTTP server, how do I manage it? daemontools? runit? Something like that?
A nice thing about CGI is that you don’t have to worry about service management or monitoring. If I have something continuously running and serving HTTP, do I need to worry about restarting it if it crashes, cleanly restarting when I deploy a new version, etc. Or is this not really very difficult?
For my that-kind-of-scale website: Also small http servers behind nginx. And those were initially running from tmux, later I set up some user systemd services (because systemd was what’s available on that machine). Although I remember some trouble getting logging of user services to work well.
Thank you - this is the sort of thing I just don’t know about. A small HTTP server sounds great for local development - I just worry about the sysadmin effort required to keep it going reliably on a real server. For small projects, especially unpaid ones, I really don’t want unnecessary sysadmin work!
This commit seems very timely: https://marc.info/?l=openbsd-cvs&m=155931636824866
SNI support in relayd(8) make this a much simpler option for me. Thank you for suggesting it.
Webserver for both is nginx on a Debian box.
I host Wordpress for a friend on a FreeBSD VM, that’s using nginx & php-fpm behind it via fastcgi. Did similar when I was hosting a forum for my mother a few years back too.
I dump my files into
/var/www/html/for Apache and that’s the end of it. I write all of the HTML, SVG, and whatnot by hand and when I want to add a new article I copy the HEAD and all of that from a previous article. Then, I add the article to
index.htmland that’s the end of it.
There is one question I have, however. I’ve recently wanted to add the ability to comment on my website and the current mechanism is sending me an email, as explained here, but this seems a high enough barrier that I’ll receive very few, as I’ve received none so far. I figured it would be sufficient to add a form to that page, but I don’t know how to attach an arbitrary program to a form. As I’ll be doing this all manually, it would even be enough to simply have Apache log the POST requests sent to a certain URL, but every option I’ve come across so far requires me to either install something or perform some heavy configuration, both of which I’m reluctant to do.
I suppose I could tell it to vomit the POST to a port I have listening, but surely there’s a better way, right?
This is actually a great use case for php. Since you already have Apache set up, you would use it with mod_php and direct the post request to a php controller. From there you can do whatever you want in process, probably the simplest thing would just have the server itself email you and then optionally log the data or action somewhere. So no messing around with sockets or other processes on the host side or spinning up and managing arbitrary interpreters as the module handles those details.
You can also write cgi modules for python and your favorite language, though php is basically made for this. So you have choices that can also lead to “easy to write scripts for single endpoints”
This is close to the sort of thing I’m talking about. Static assets are easy - but what do you do when you just want to add a small dynamic part, like a contact form? A full “web app” with the accompanying sysadmin headaches just seems like massive overkill.
Really, I’m actually surprised this is so complicated, all to add a single form that wouldn’t even create a dynamic page.
It sounds like you want cgi scripts then. https://httpd.apache.org/docs/2.4/howto/cgi.html which still take a touch of config.
Or you want to log the post data and just manually grep it later, this page gives you three options, two of which are modules and the other one is use an application layer: https://www.simplified.guide/apache/log-post#log-post-request-data-in-apache-in-application.
On the bright side you definitely don’t need js to send a simple post so that’s good.
I have a git
post-receivehook on my webserver that checks out the latest master to
/var/www. It’s a little naive at the moment because it only handles content changes (rather than code changes, which would require the hook to re-start a node process).
It’s only for my personal site, but I’ve been honestly surprised at how well it has worked for the past few years. Just
git push web masterand the site is deployed.
This is pretty much exactly what I have at the moment for my static sites. It’s the management of the dynamic parts - I guess that’s restarting the node process in your case - that I’m not sure about. Perhaps this can all be solved by daemontools or runit or similar.
Any process manager should work. I use pm2 myself, which allows defining your config in a json file. I use that file to bounce the service, like ‘pm2 restart app.json’, where app.json is my pm2 config. And then I can reuse that script unchanged for any project as long as I name the config file the same.
And FWIW, I’m on Hetzner’s cloud servers, which are 1/4 the price of DO (under $3/mo for a 2gb server) with the trade-off of being located in Europe.
Thanks. I am also located in Europe, so maybe I should give Hetzner a try. I use Vultr at the moment.
All of my current stuff is in Ruby. I set it up so that I can git push from my local dev environment to the server. I then have a shell script on the server that I ssh in and run that updates the bundle, runs any migrations, builds the new assets, and restarts the Puma server process. It’s been pretty simple and painless so far.
Thank you - so the Puma server sits behind nginx or something similar? Is there anything monitoring Puma? If your Ruby app crashes - there’s an uncaught exception or something - does it get restarted?
Yup, it’s behind Nginx. The Puma server and a DelayedJobs worker are running as SystemD-managed services, so SystemD handles keeping them running through system restarts and that sort of thing. I’m pretty sure it would restart it if the whole process crashed, but that’s never happened.
Personally, I really like Netlify for publish-and-forget types of sites. You point it to a Git repo, and configure it to build your static site from it - be it with a static site generator or by copying some files. Regarding small dynamic parts: they offer ways to include dynamic elements, like a submit form, and take care of handling the data for you. I haven’t tried it myself but it looks quite simple. So, deploying a static site like my blog to Netlify simply consists of committing some changes and pushing to master.
For the “middle ground”: I use Gunicorn (a WSGI server which runs a few threads continuously and runs your Python code when a new request arrives) behind Nginx. Systemd takes care of keeping them running and restarting if needed.
Deploying to this setup is easy as with most low-traffic sites: you pull the latest changes to a repo on the server, and restart the systemd unit responsible for Gunicorn. All of this can be automated using Fabric, so there’s no need to SSH to the server manually.
Also handy, but behind a bit of a learning curve: using Zappa or the Serverless Framework to run your dynamic code in something like AWS Lambda - it’s very cost effective, and you don’t need to do any of the sysadmin work described above.
Thank you. Your “middle ground” solution is something that I’ve heard a lot recently: gunicorn/uwsgi/puma behind nginx with systemd to restart as needed. This seems to be a popular solution! I think I’ll go for runit rather than systemd but I guess it makes sense to use systemd if it’s there on your OS anyway. Thank you for sharing.
post-receive git hook
I use to work with Drupal, a php/sql cms/framework, and as a result I still use it years later (funny how that works). While professionally I dealt with CI/CD around this subject, I didn’t want that much overhead for my personal blog so it’s effectively a docker container with my site’s code mounted from the host’s filesystem. I don’t run a machine or vm dedicated to it, so there’s other virtualhosts at play too. I have a few containers with the following:
What I like about this setup is that it gives me both the flexibility of docker for dependency management and the simplicity of “classic” web hosting techniques. My git repo for the site’s assets also has a scripts folder with things like
I of course use Let’s Encrypt, and using that centralized nginx instance let me have a little fun with how I manage that. Since it’s centralized, I was able to route all requests concerning the
/.well-known/path prefix used by Let’s Encrypt to one instance of a custom LE client. Obviously, this is all rather pointless now that LE supports DNS challenges too, but this setup predates that.
Now if only I had as much patience for writing content for the blog as I did building the infrastructure - https://dpedu.io/
Edit: OP asked about deployment, and I suppose I didn’t answer that directly. The custom docker images flow through a self-hosted registry. The machine running them is configured by puppet. And the site’s code is manually uploaded with rsync.
When I started I ran a small blog hosted on Heroku so I got used to ‘git push heroku master’. I wanted to cut costs from the $7/mo Heroku was charging since I wasn’t making any money, so I switched to a $5 DO droplet but wanted to keep the simplicity of ‘git push xxx master’ so I installed dokku just to have a Heroku-like interface.
After that I realized my $5 droplet had extra capacity for some other side projects so deploying those to the same server was simple thanks to the decision to use dokku.
What’s your purpose? What’s the velocity of changes to the content of the website? Who needs to make changes or submit content? Can the editor(s) use Git? Can you use a static website that consumes dynamic APIs, e.g. comments from Disqus or similar or something self-hosted?
Unless you’re learning or adamant about running your own server, don’t. It’ll detract from your main goal, which likely is to produce content. There are enough free hosting options out there now for static content that running your own static content engine should be an exercise served for learning.
I already have a few static websites. I have a VPS running OpenBSD and I just use the httpd included in the base OS. I like it because there’s absolutely no sysadmin burden: it has run for years with absolutely no problems. Apart from upgrading to new OS releases, which is fast and painless and takes place twice a year at most, I don’t have to do anything at all. This is exactly as it should be for websites that I host for free.
However, some of the sites now have a need for dynamic components, such as a searchable image gallery. I am really unwilling to do anything that will increase my maintenance burden. I have, in the dim and distant past, written CGI programs in C and Perl… but I thought there might be another option these days.
I’ve been meaning to write this up, but I have two personal static websites (one is a blog). They both use git and GitHub for source control and repository management, build with Hugo inside CodeBuild, and deploy to S3 and CloudFront with ACM certificates. (I work for AWS, which is part of why I decided to use a bunch of AWS services, but this is my own paid personal account.)
It took me a few months to do this, but I’ve now documented how the blog works.
I’ve been using Dokku to deploy my site.
I edit the Lua code and then I do
kill -s HUP $OPENRESTY-PID
Is it just stock Lua or are you using a templating library or framework?
I got hooked on Lua through PICO-8 not very popular among this crowd because it’s not FLOSS, but I love it and grew to love the simplicity of the language.
I use OpenResty.
I didn’t know this existed. Very cool! What drew you to it?
I like LuaJIT. Perhaps too much…..
For usesthis.com, I push a commit to master on GitHub, which fires a webhook to my server, which then pulls from the repo and generates the site with my own static site generator.
My personal site is completely static and hosted on GitHub despite incorporating a small webapp. Aside from using GitHub for hosting, I use Cloudflare for my domain name, DNS, and CDN.
I use sitecopy for uploading my personal pages, and php for scripting, it is available practically everywhere, and easy to use.
I just ssh into it and overwrite the old rust binary/php stuff. This way I know that there’s only one way to change stuff: get SSH access. No repo-overtake etc. Also nginx in front of everything, as it’s easier to manage TLS this way and remove possible attack surface. Nginx will also cache if required and serve static stuff. I have some lighttpd-reverse-proxy stuff going on as lighttpd can do per-host traffic limiting ( downloads..)
It automatically downloads and installs on target host an uptodate version of NGINX
It allows me to me to specify where my site files are (usually in /var/www/
It allows me to configure a bunch of settings (time outs, how to handle URL’s without www, and so on – all by passing arguments to that NGINX role)
It allows to specify where your certificate/private keys files are (I usually put mine in /etc/letsencrypt/ )
I created my own ansible role that I call ‘local_build’ that basically runs a build of my webapp, and deposits it into a predetermined location on my build machine.
Finally, my 3rd ansible role copies certificates and then webapp files (from 2) onto the target host(s)
so The playbook executes (1), (2), (3)… also I can manually execute just (3) (update certificate).
The NGINX role works well on any of the Linux distros, there is also support for FreeBSD (and maybe OpenBSD), there will be some updates to the role – probably shortly to fix some issues on FreeBSD.
I rarely have have a need to login into the target host, only if I need to debug (although we are pre-prod still).
A linode + namecheap and nginx. Not really much else to say.
I have a cheap VPS from Contabo that hosts all of my hobby and personal sites. They’re all Elixir applications, deployed using edeliver