1. 14
  1.  

  2. 5

    That is why there is OpenTelemetry initiative (this is merge of 2 earlier projects - OpenCensus and OpenTracing) and in near future there is plan to create “agent” that could work locally and provide all metrics and traces via built in, local, basic dashboard. The only missing part in such case would be logging, but I think that console is “good enough” for that (unless you need distributed logs, then it would be nice to provide some kind of local service for storing logs in in-memory buffer).

    1. 3

      Speaking as a Prometheus developer, it’s very easy to run Prometheus locally and I’ve done this in the past to debug both Prometheus (https://www.robustperception.io/optimising-go-allocations-using-pprof) and other applications. Most of the time I’m debugging the sort of issue that metrics aren’t suitable for though, so I’ll be print lining.

      1. 1

        True, and his only example here was basically HTTP transport, which is but 1 aspect of dev observability that’s horrid.

        1. 1

          There’s also jaeger that came out of Uber, I think there are a fair few projects using it and it’s pretty stable and has a fairly nice web-based UI.

          1. 1

            Try running Elasticsearch + Filebeat rather than the full ELK stack for a more lightweight approach. Ingest pipelines negate most of the need for Logstash and Kibana can be spun up when needed. You can also change the processors config option to make Elasticsearch reduce the processors it uses. NB I work at Elastic.

            1. 1

              With open census you get zpages which does pretty much exactly what the author is talking about.

              https://opencensus.io/zpages/

              1. 3

                Yes, but OpenCensus is going to be shut down (probably this year) and moved to the OpenTelemetry.

                1. 2

                  OpenTelemetry is nice but you can’t use it yet. Well maybe you can in Java but not any of the other languages. OpenCensus is here right now and the API will be compatible with OpenTelemetry.

                  1. 2

                    Will be similar, but it isn’t yet decided whether it will be 1:1 compatible (source, I am part of team that implement OC/OT in Erlang and we are discussing it).

                    1. 1

                      The site states

                      We are still working on the first production-ready release of OpenTelemetry. For those who want to start instrumenting production code immediately, use either OpenCensus or OpenTracing. OpenTelemetry will provide compatibility bridges and there is no need to wait for production-ready OpenTelemetry APIs in your language

                      So even if it’s not 1:1, the compat layer should be fine.

              2. 1

                what’s people’s perspective on ephemeral clusters designated for development, with laptops essentially thin clients the cluster? it’s something I’ve thought might be useful, but would be a dramatic workflow shift.

                i.e., the above problem gets turned into a zero because the cluster is where work happens.