1. 11
  1.  

  2. 1

    Not sure why the DB has to be on the servers (to be monitored), and not on the client/monitoring server. Personally I try to keep the monitoring workload on the servers as minimal as possible, and do all the aggregation ++ on a separate server.

    If they had done that, they also could have used snmpd from the base system, without the need for writing a collector daemon.

    1. 3

      It’s on both. The database is really just a set of circular buffers that’s transmitted in totality. But it’s on the system being monitored because hey, you might turn off the monitor. And then your history is gone. Also, even if the monitor is running, the network might fail.

      1. 3

        This is something I wish the Prometheus people had factored in to their design. I’d love to be able to restart the collectors or have network disruptions without losing data points. A locally spooled catch-up ring buffer is a great idea.

        1. 1

          A locally spooled catch-up ring buffer is a great idea.

          Would you persist it to disk? Just in case the monitored server crashes while the buffer is not empty.

          1. 2

            Persisting to disk would definitely be something I’d like to explore. I think the downside might be that it induces extra I/O load and storage pressure on what might already be a busy system.

            1. 1

              I’m not sure you’re referring to slant, but it leaves the database on the disc. The point of slant is to be a simple tool for normative hosts: if your systems are so significantly under load, you probably want to do something like snmp.

              That being said, I’m happy to work with folks with patches to make slant more performant. I know that ksql has lots of room for improvement!

              1. 2

                I’m not sure you’re referring to slant, but it leaves the database on the disc.

                Sorry, I was talking more in the abstract – we’re using the Prometheus ecosystem pretty heavily at the moment.

                It’s neat that slant persists to disk!