There’s even simpler… one could say, codeless technique of poor man’s sample profiling.
Just break-&-resume the program a dozen times, each time glancing at the stacks, in a debugger, by hand.
Chances are, if the debuggee has a bottleneck — you’ll notice its stack in most of the “samples”.
Automating that like shown is the natural next step… but you can “get the food” even without it.
Yup. Done that a couple of times in my life!
If one scrapes a few coins together one can take this poor mans approach to the next level…
Haha, yes, the flamegraph is an excellent invention of Brendan Gregg. Got me into seeing more of his work. This gem? https://www.youtube.com/watch?v=tDacjrSCeq4 — I could never imagine.
I think flamegraphs should be the standard representation of profiler output. It’s great both to get the picture yourself, and to showcase the data to whomever you need to.
Whenever I look at what the hardware engineers are doing, I say Hardware Engineering is exactly like Software Engineering…. except all the basic primitives were designed by satan.
So a lot of what hardware (and software) is doing is wrapping. ie. Wrapping something almost reliable enough to be useful, in enough checks and retries to make it reliable enough to be useful.
So anything (like shouting!) that makes that fundamentally dodgy hardware slightly more dodgy shows up as latency.
A lot of equipment is designed and tested in near ideal conditions… shipped to the noisy real world…. and magically the throughput and latency numbers aren’t quite as advertised…
Whether this is RF noise, audio noise, vibration, electrical supply noise, thermal stress, …
There is a lovely list of embedded facts somewhere, my favourite one is “All sensors are temperature sensors… some of them also measure other things.” and “Everything else is also a temperature sensor”.
The whole industry is plastered over with “As If”’s..
Y works “As If” X was true. The fact that sometimes X isn’t true shows up as Y taking (almost) imperceptibly longer.
Various flavours of Meltdown and Spectre are almost fundamental to what the whole industry does.