1. 16
  1.  

  2. 8

    Everybody’s written a version of this, right?

    Our house version is called errorwatch, and it now boasts the following ridiculous yet eventually necessary set of options:

    errorwatch -p “PROGRAM” -s success@somewhere -f failure@somewhere -i -r “regexp” –failregexp “FAILED” –successregexp “” -l logdir -m “message” -t “title” -T timeout -d -q -e

    -p is required; everything else has a plausible default.

    It looks at errorcodes, it can look at stdout and stderr for regexps, it can expire on a timeout, and it sends variable sizes of mail and logs. It would be nice if it could report that it hasn’t run, but that’s too meta even for Perl.

    1. 6

      Make a cron job that watches for its emails and if it doesn’t see any, reports via a different email service.. Then set errorwatch to watch this first cron job. That way both have to fail independently at the same time for notifications to vanish.

      …I wish I knew whether this was a joke or not…

      1. 7

        Not a joke, my important cron commands always post to a dead man’s snitch-style service on success.

        https://deadmanssnitch.com/

        1. 3

          We actually have an end-to-end check to see if mail is working which does a negative feedback: it accumulates mail into a directory, and if the most recent timestamp is too far in the past, it goes off.

        2. 1

          . It would be nice if it could report that it hasn’t run

          And when it doesn’t finish running and loops instead of slowly progressing, thanks

        3. 8

          Or use systemd timers. They work as expected and are more flexible to use.

          1. 3

            Not everything uses systemd.

            1. 3

              But if your system does, then systemd timers are probably a better choice. Cronic looks great for the other systems.

          2. 3

            I use this on my Debian machines, but OpenBSD has the -n flag on crontab(5) which does the same.

            1. 1

              You probably use the similarly named (and well established) chronic, part of moreutils. this tool seems to be an unwitting reimplementation.

              1. 1

                I’ve used both! Just kinda settled with cronic

            2. 3

              I wrote a tool to send my cron jobs to HealthChecks.io, so I get error notifications and late job notifications. https://github.com/spotlightpa/kristy

              1. 2

                I’ve been using HealthChecks also, but I add the “&& curl …” manually. I will definitely check your repo out.

              2. 2

                Not the same thing, but I wrote my own cron agent that just exports prometheus metrics so i can get alerts:

                https://github.com/andrewchambers/promcron

                1. 2

                  One of the best features of cron is its automatic email

                  I don’t run e-mail on any of my machines (attack surface) and Cron is the only program that would need it, so this hasn’t been Cron’s best feature for decades for me. I’d rather Cron log to the system facilities like any other program.

                  1. 1

                    You mean you don’t send e-mail out from any of your machines?

                    1. 1

                      My e-mail client connects to an external IMAP server and besides cron that wants to send e-mail there’s nothing else on my machines that needs it.

                  2. 1

                    Assuming that any failed command would return a positive status code, I think the following would suffice:

                    command 1>file 2>&1 || cat file