I love this post for the tools and real world graphs shown. And it also answers my fear when using tracing on actix-web: Yes you can actually leak memory if the tracing isn’t ended after every request.
Edit: What I’m missing though is a good frontend for tracing. I’ve seen jaeger for opentelemetry, but it doesn’t show as much info as I’d like. (How much time does function X take over multiple calls for example.)
In the beginning of the “What Do You Mean You Leak a Bunch of Memory” section, it mentions that the Go garbage collector is generational. That’s actually not true.
What happens in Go is that the compiler uses escape analysis to place a bunch of allocations on the stack, which means that the GC doesn’t have to look at any of those allocations.
This is the second tracing-related memory leak post I’ve seen recently (I don’t have the other one handy but it was due to holding spans over await points IIRC). Makes me a bit nervous to dig into tracing when regular logging seems to “just work”.
I’d say look up how to do tracing with your framework of choice, ask some people on irc discord and you’ll be fine. Just don’t ship a custom tracer, unless you’re willing to deal with that.
I am a big fan of tracing, and I’m not to spooked by the possibility of leaks. As long as you have a graph of your memory usage, and you can overlay a deploy event bar on that graph you can spot a big leak right away. At least in NodeJS land, upgrading to a new anything ever may introduce a leak, and dd-trace specifically (datadog’s library) is fraught with this issue. So, we stare at the graph once in a while and track down the source. Not too bad.
I love this post for the tools and real world graphs shown. And it also answers my fear when using tracing on actix-web: Yes you can actually leak memory if the tracing isn’t ended after every request.
Edit: What I’m missing though is a good frontend for tracing. I’ve seen jaeger for opentelemetry, but it doesn’t show as much info as I’d like. (How much time does function X take over multiple calls for example.)
Take a look at Grafana Tempo.
In the beginning of the “What Do You Mean You Leak a Bunch of Memory” section, it mentions that the Go garbage collector is generational. That’s actually not true.
What happens in Go is that the compiler uses escape analysis to place a bunch of allocations on the stack, which means that the GC doesn’t have to look at any of those allocations.
This is the second tracing-related memory leak post I’ve seen recently (I don’t have the other one handy but it was due to holding spans over await points IIRC). Makes me a bit nervous to dig into tracing when regular logging seems to “just work”.
I’d say look up how to do tracing with your framework of choice, ask some people on
ircdiscord and you’ll be fine. Just don’t ship a custom tracer, unless you’re willing to deal with that.I am a big fan of tracing, and I’m not to spooked by the possibility of leaks. As long as you have a graph of your memory usage, and you can overlay a deploy event bar on that graph you can spot a big leak right away. At least in NodeJS land, upgrading to a new anything ever may introduce a leak, and dd-trace specifically (datadog’s library) is fraught with this issue. So, we stare at the graph once in a while and track down the source. Not too bad.