1. 9

The recent story reminded me of this question! Suppose I am a developer with a webapp that I want to run cheaply, on-prem or using Linux hosting (droplets on DO, Linode, etc.), but I am not a proper sysadmin. I also want to avoid using proprietary cloud services to avoid vendor lock-in. What are some books or tools of trade that would be useful to know about to properly run webapp for production?

  1. 11

    but I am not a proper sysadmin

    Tip 1: run from the imposter syndrome. The only difference between a “proper” sysadmin and not is what, experience? You will gain the experience :)

    Tip 2: depend on only things present in your OS (ie: Debian or Guix) and only on the versions present in your OS wherever possible. This will simplify your life immensely. Note that this only matters for runtime dependencies, compile-time dependencies you can use whatever and it won’t matter for deployment of course.

    Tip 3: you mention graphite, but you’ll need some way to feed data in there. I suggest looking at collectd if you don’t already have a favourite. It doubles as a fully-functional statsd as well.

    Tip 4: if you’re using a dynamic language likely to throw many exceptions in production, use something like sentry (freedomware you can self host if you like, though they have a cloud version as well) to track fatal errors in production – gathers more context and easier to work with IME than just a log.

    Tip 5: Make sure your logs rotate. Anything set up by your system will automatically, but your custom app stuff may not unless you set it up properly. Best way to do this honestly is to just log to your local syslog and let it handle such things for you, but no matter what you must rotate lest you run out of disk space and be sad.

    1. 2

      Relatedly to Tip 5, rotate SSL certificates before they expire (probably the most frequent cause of outages I’ve seen in self-hosted setups). LetsEncrypt (or similar ACME cert issuer) have elegant ways to do this.

    2. 4

      Find a linux or BSD environment.

      Run the app kind of like you would the app in development. If anything warns you about being in development mode, find out what setting you need to change. Then run it from your init system (systemd, runit, BSD init, OpenRC, what have you). Then, put it behind a Real Web Server like nginx or its many rivals. Set up https here with letsencrypt. If you have lots of static content, look at varnish for caching that.

      Pick a programs to keep track of disk, CPU, and network usage. I like Prometheus + graphana. Keep half an eye on them. When your disks exceed 80 percent, find out why and figure out how to reduce that (sometimes: larger disk). When your network is getting saturated, consider ways to scale. When your cpu usage is too high, find out why.

      You can do without the monitoring. It just makes life easier later when something crashes and you don’t know what.

      Finally sit back, relax, and make sure the credit card on your domain name is current.

      By building your own solution, you are presently locked in to those solutions. However by building your own solution, you have gained the knowledge necessary to be able to switch solutions when the need arises. Just leave yourself comments as to what you did and why you chose the config options you did, and you’ll be off to a very good start.

      1. 2

        Here’s a tip. If your web app occasionally hangs in PROD but never TEST and while it is hung, resource usage is low, but it’s really acting like it is in some kinda IOWAIT, check if your developers used a secure hash function instead of a regular hash function and maybe the only real difference between the two is that the former hits /dev/random instead of /dev/urandom … Where was I? Oh. Run cat /dev/random > /dev/null on TEST and see if it suddenly acts just like PROD.

        And have fun! :)

        1. 2

          I am not a proper sysadmin

          Pfft, who is?

          I would suggest going with easy to google things, DO themselves have good guides for common use cases. If you can google it and follow a tutorial, you’re doing fine. More than fine, you’re doing better than anyone that didn’t bother with that step 😁

          If you’re getting stuck in the quagmire of “oh fuck, what do I choose”. Start with ubuntu. nginx or apache. configure it by hand, but keep notes on what you did. Get it working, someone like DO is cheap, easy and not stupidly expensive. 🤘🏻

          1. 2

            My personal tip is to use Docker Compose. Some people might disagree here but I think containers make it easier at the end (you do not depend on the host for almost anything). Just be sure you can execute your app in your dev computer using Docker Compose, and build/move the same containers to the servers. You are also free to use whatever service you want. Also, if you use Docker Compose you don’t need to learn/configure systemd, … because it will manage that for you. If you want automatic restarts on app crashes, you might need Swarm, which is very similar to Compose.

            Also, prepare a script (I use Ansible, but even Bash could work) for automatic backups of the data (storing it on Azure or Amazon S3 is simple and not expensive, checkout the Cold Storage/Glacier options if they suit you).

            1. 1

              “Properly” is a relative term.

              Forewarning, this may be viewed as a biased approach, due to my company’s business model.

              I would suggest (if you can) to pay for some time for someone with experience to discuss what you plan to do, alternatives, etc. S/he should be able to identify potential pitfalls in your planned approach and recommend possible solutions to (a) those pitfalls; (b) things you haven’t got a plan for.

              A more expensive but possibly quicker solution is to pay someone to handle the infra for you, and “learn in reverse” from the working system/documentation/etc.

              1. 1

                You probably know some people that have more experience in this area than you do. I’d say that it is probably great to involve them at some point. For example, you could do something that you think would work for production, learn a lot doing so, then discuss with someone about your setup to benchmark it. Most people I know in Ops are happy to send 20minutes to help :)

                1. 1

                  As an example of type of answers I would enjoy the most: graphite (http://graphiteapp.org/), because you’d probably want to collect a lot of various metrics, from host CPU/MEM to app metrics (e.g. time for database query).