1. 25

I have a couple of servers at home, that are now running services that I use more an more when I’m not at home. One notable example is Nextcloud. However, it has happened a couple of times so far that the machine didn’t come back online (and I didn’t bother to check), or router went down. Thus, I’d like to start using some external monitoring that could send me an email if the machine is not pinging. It would also be nice to keep track of performance and active connections throughout the day. At previous work we used Datadog, which has a free tier, however, it doesn’t support FreeBSD. Now I saw that InfluxDB has a cloud offering with a free tier and support for FreeBSD. I’m wondering what others are using?

  1.  

  2. 18

    I used to have a pretty fancy home setup where I had metrics dashboards (using graphite, grafana, etc[1]), host stats, uptime monitoring, the whole 9 yards. It ended up being more of a pain to manage all that than it was worth. Over time I realize I tended to hardly ever check the dashboards, and if a server was down, I could pretty quickly tell it was down.

    Later, I redid my home network and jettisoned all that metrics overhead. Now I just use a few very simple alerts – mostly crontabs with hchk.io+pushover alerts. It works like this:

    # crontab
    40  2  *  *  *  root  some-backup-script && curl -fsS --retry 3 https://hc-ping.com/SOME_UUID_HERE
    

    My hchk.io alerts invoke pushover to send me an alert if it doesn’t fire in a certain period of time.
    In practice, it works out pretty well, and is pleasantly low resource as well as low touch.

    [1]: This was many years ago already. If I had (or desired) to do the full metrics+dashboard stuff again today, I’d probably just use prometheus+grafana. A far more simplified stack, which might be easier to manage.

    1. 2

      I love the simplicity of this service. Thanks for the pointer.

      1. 1

        My setup is identical! Love healthchecks.io.

        1. 1

          Woah, TIL about healthchecks.io. That seems like a really cool service.

          Why do you use healthchecks.io + pushover? They seem to basically do the same thing (send messages)?

          1. 2

            Why do you use healthchecks.io + pushover? They seem to basically do the same thing (send messages)?

            healthchecks.io has a few mechanisms (called “integrations”) by which you can send alerts. One is email, another is pushover – there are quite a few others too. I use email or pushover (or both!), depending on the check in question.

          2. 1

            Thanks! Just learned about healthchecks.io and it seems to perfectly fit the bill!

          3. 4

            I run Prometheus at home, though I’m obviously a bit biased there.

            1. 3

              I’m not biased and I run Prometheus at home, and elsewhere. Blackbox Exporter running on RPis in various physical/network locations, with ZeroTier. Most of the Blackbox Exporter target configuration uses DNS service discovery. Alert Manager for alerting. I’ve used many different monitoring systems and recommend Prometheus+Grafana, with Netdata for some low-level monitoring.

            2. 3

              If my bouncer is up.

              1. 4

                Simple, really: I use the things. Usually this works and everybody is happy. Sometimes it doesn’t and people complain, in which case I try to SSH into the server in question to see what’s amiss and fix it if I can (i.e. if it is a ‘soft’ problem). If fixed, goto start. If not, eventually I go home and find out where the magic smoke escaped.

                I have some off-site remote monitoring in place, i.e. my parents and my brother. They are quick enough to tell me their mail doesn’t work or the web thing doesn’t web or the media server doesn’t mediate.

                1. 2

                  This works for e.g. my weechat bouncer. But I don’t actually “use” my personal home page – I’ve in the past been notified by other people telling me it’s down, but I suppose some might get a bad first impression from pages being down :-) Now I have a cron job on a different server that emails me when it’s down …

                2. 2

                  I wanted a dashboard with monitors but not compromising on maintenance and load as well. netdata fit the bill perfectly.

                  It provides all of the hardware/application stats for a quick overview. You can also configure alarms. I haven’t explored push/email notifications yet - not sure if its possible by default. But definitely a worthy addition to your home server stack.

                  Sample net-data dashboard

                  If you want something super-lightweight, you can look at linux-dash too.

                  1. 1

                    Just set it up on my machine, that’s been something I’ve been looking for for a long time – I’ve been just swoon. It is so easy to set up and run, and the dashboards it shows are really well done. Had troubles finding how to enable their cloud service, tho – it’s not obvious that one has to have a running daemon first. Thanks for the pointer!

                    1. 1

                      Glad it helped :)

                  2. 3

                    I used to use VGA but nowadays prefer DisplayPort.

                    1. 1

                      If I don’t use them enough that I won’t notice they are down, they probably shouldn’t be running in the first place. :)

                      1. 2

                        What about cases when you need documents while you’re out, yet machine was down and you could’ve fix it while at home? You might not necessarily need the service while at your computer at home, but once you step out and go to work/government agencies, you might need something you haven’t thought of initially, but now really need. It’s the same as with backups, we don’t know if we need some things until we really need them.

                        1. 2

                          That’s why I’m always torn between self-hosting and hosted, especially with services like e-mail. It usually works all fine until I go on vacation for two weeks and only have limited time and equipment with me to fix it. That’s when I want to go for a hosted solution, when I’m back from vacation I want to manage it by myself again…

                          1. 2

                            Agreed. I was partly saying what I said in jest. There is of course cases where things happen. But other comments here pretty much cover when you need to actually monitor something. It’s not hard, from a ping -c 3 to nagios or some service. All of these things are fairly straightforward, each with their own tradeoffs.

                            I think there is also a larger question here, about should I keep service X or Y running? Do they continue to provide value for the resources I use to keep them alive? Sure running X or Y alone doesn’t take a lot of resources, and $10/month or whatever for a cloud instance or $1/m in power costs for your server @ home may not be a lot of resources. But not a lot of resources, consistently, adds up over time to a significant amount.

                            My initial comment was a snarky/jokey/pithy way of saying all of this.

                        2. 1

                          I don’t 🙄

                          1. 1

                            monit

                            1. 1

                              Pingdom is enough.

                              1. 1

                                We are really missing a 0-configuration solution to deploy, configure, monitor and share data across small groups of personal servers.

                                1. 0

                                  After looking a bit into netdata, it seems that it could be just that, when paired with NixOS or other systems which allow easy configuration for most of the services.