I would say these days my primary method of exploring what a piece of code does is writing a test then attaching a debugger to it. Otherwise you’re just reading the code and running some kind of virtual debugger in your head.
I wrote this a while ago about print debugging and about how it’s fine. See lobsters discussion. So I have some observations.
It’s interesting how the author went with print debugging being fine for many years, had reasons why that was good and fine, and then finally ended up in a context where he needed a debugger, and now has changed his mind and considers debugger use as the new baseline.
Hinted at but not discussed in detail is what in code necessitates use of a debugger. Code size isn’t it. “Lots of callbacks” make it important, apparently and “complex systems”.
Is it only complexity or also how that complexity is organized: i.e. architecture? There’s a relationship between software architecture and how easy is it to debug with print statements, and how easy is it to test. I think we should strive for islands in our codebase as big as possible that are easy to test automatically, and thus probably also be debuggable with simple tools like print. But that’s not always feasible - parts of your code may have a structure that make more advanced debugging tools helpful, just like in some cases it’s actually easier to test by hand than writing an automated test.
That said, it’s worthwhile to see how far you can push a codebase so it’s easier to test and debug again. See for instance recent discussions about sans-IO
I’ve found structured log based print debugging with a file:lineno pair to be super helpful. It gives you something to click that takes you to where the log message came from, which works wonders for rapid development. Basically, invest in it if you find yourself constantly slowed down by grepping the codebase for the log message– especially if you end up with a lot of logs that have a generic message format like “Failed to do $thing”.
This can be annoying to implement properly, depending on the languages ability to introspect the call stack, but if you get it working it’s been worth it, for me. Need to worry about things like nesting log emit behind helper functions, you need a way of telling the underlying file:lineno logic “go up by N from callsite”.
I’ll still drop in breakpoints for the surgical work, but when debugging a system holistically, you may need to see a stream of logs as things are happening in real time to get the full picture.
For me lately I hardly work on code that is not parallelized in one form or another. Reasoning about how that code runs strictly based on print debugging is difficult.
Sometimes it’s also difficult to do in a debugger, but having a stack and a memory state to look at is better than anything that could be done in print in my opinion.
For me a debugger is most useful in those circumstances where you don’t know what code you have to instrument or put debug printfs in. In that case, throw some hardware watchpoints around and see what falls out, look at the backtrace, etc.
Sometimes, depending on what I’m working on. Test fixtures, REPLs, profilers, linters, logs, printlns, and even “debuggers” have their advantages and disadvantages.
If everyone you work with thinks one specific tool or technique is indispensable, you may not have a very diverse team, or a very diverse set of problems to tackle. (Or both!)
I treat stepping in a debugger as a last resort solution, when the bug is really WTF.
Use of debuggers (and manual testing) feels like lost time to me, because once I find the cause of the bug, I don’t get anything tangible for it. I just spent time on it, and will spend time again when there’s a similar bug in the future. OTOH if I catch the bug with tests or assertions, that code will stay and catch regressions in the future.
Right click the line you want to see the program state at in your IDE, click “set breakpoint”
Run debugger, see program state at that point including all class & static variables, execute arbitrary expressions, etc.
Is there anything that could actually be faster? Every time I encounter this attitude it is just about not wanting to pay the fixed learning cost to set up a debugger in your development environment. Pair this with the principle that a regression test should be written for every bug.
You’re describing an easy case. If I know the line to look at, I may be able to spot the bug without even running the program. If I know which expressions to inspect, I can printf them. In easy cases most methods are similar, so it’s just a matter of preference.
It’s the mysteries like “1% of responses is missing an http header that was supposed to be there”. Then I have to wonder, do I trace a whole request start to finish to see where it goes wrong? Will I need to trace it 100 times? If it’s a timing issue, will it even happen under the debugger at all?
I don’t understand why people have negative thoughts about using debuggers. There isn’t a “tests or debugger” dichotomy, and there’s no reason not to use both. They’re different tools for solving different problems.
I can’t imagine getting up to speed and making changes on a large code base without using a debugger.
Probably because raw gdb and lld are a pain to use. Unless you adopt a full fledged IDE to do your work there are hardly any good standalone UX debuggers for linux.
I generally prefer developing on Linux, but I have to admit Visual Studio’s C++ debugger is a lot easier to work with. I’m not a fan of the UI hangs and questionable default settings, but it’s quick and easy to use and the learning curve is much gentler than GDB’s.
i worked on debuggers and i use them in sometimes weird creative ways. for instance i recently turned rr into a bash profiler, by recording the execution of the process and then injecting calls to dump the current line and function name at various points in the recording. this technique of recording the process and extracting a profile from the recording was invented by a dear friend i met when working on a proprietary debugger. it’s a really powerful idea because the program can run unmodified and it doesn’t require support from the system/interpreter/runtime, thus collecting the profile adds basically zero overhead (the overhead of the debugger isn’t necessarily zero, but it doesn’t modify the state of bash, unlike the overhead of printing this information from within bash which can be very significant in certain optimised programs, rendering the profile meaningless)
Debuggers have fallen out of fashion because programming tools have become really inconsistent. In the 90s everything had a debugger. You often didn’t have tests, you just debugged everything. I used to dog food my apps by running them with a debugger permanently attached, when it crashed or hit a bad case you could jump in and sometimes even fix it on the fly.
Good luck doing that these days. Everything is in the cloud and you can’t get access to anything. Just reboot it and hope for the best
You are correct and it is extremely annoying. I really wish there were a better way to do it. Every time I modify the CI it is guaranteed to take the entire day.
I would say these days my primary method of exploring what a piece of code does is writing a test then attaching a debugger to it. Otherwise you’re just reading the code and running some kind of virtual debugger in your head.
I wrote this a while ago about print debugging and about how it’s fine. See lobsters discussion. So I have some observations.
It’s interesting how the author went with print debugging being fine for many years, had reasons why that was good and fine, and then finally ended up in a context where he needed a debugger, and now has changed his mind and considers debugger use as the new baseline.
Hinted at but not discussed in detail is what in code necessitates use of a debugger. Code size isn’t it. “Lots of callbacks” make it important, apparently and “complex systems”.
Is it only complexity or also how that complexity is organized: i.e. architecture? There’s a relationship between software architecture and how easy is it to debug with print statements, and how easy is it to test. I think we should strive for islands in our codebase as big as possible that are easy to test automatically, and thus probably also be debuggable with simple tools like
print. But that’s not always feasible - parts of your code may have a structure that make more advanced debugging tools helpful, just like in some cases it’s actually easier to test by hand than writing an automated test.That said, it’s worthwhile to see how far you can push a codebase so it’s easier to test and debug again. See for instance recent discussions about sans-IO
I’ve found structured log based print debugging with a file:lineno pair to be super helpful. It gives you something to click that takes you to where the log message came from, which works wonders for rapid development. Basically, invest in it if you find yourself constantly slowed down by grepping the codebase for the log message– especially if you end up with a lot of logs that have a generic message format like “Failed to do $thing”.
This can be annoying to implement properly, depending on the languages ability to introspect the call stack, but if you get it working it’s been worth it, for me. Need to worry about things like nesting log emit behind helper functions, you need a way of telling the underlying file:lineno logic “go up by N from callsite”.
I’ll still drop in breakpoints for the surgical work, but when debugging a system holistically, you may need to see a stream of logs as things are happening in real time to get the full picture.
For me lately I hardly work on code that is not parallelized in one form or another. Reasoning about how that code runs strictly based on print debugging is difficult.
Sometimes it’s also difficult to do in a debugger, but having a stack and a memory state to look at is better than anything that could be done in print in my opinion.
For me a debugger is most useful in those circumstances where you don’t know what code you have to instrument or put debug printfs in. In that case, throw some hardware watchpoints around and see what falls out, look at the backtrace, etc.
Sometimes, depending on what I’m working on. Test fixtures, REPLs, profilers, linters, logs, printlns, and even “debuggers” have their advantages and disadvantages.
If everyone you work with thinks one specific tool or technique is indispensable, you may not have a very diverse team, or a very diverse set of problems to tackle. (Or both!)
I treat stepping in a debugger as a last resort solution, when the bug is really WTF.
Use of debuggers (and manual testing) feels like lost time to me, because once I find the cause of the bug, I don’t get anything tangible for it. I just spent time on it, and will spend time again when there’s a similar bug in the future. OTOH if I catch the bug with tests or assertions, that code will stay and catch regressions in the future.
Is there anything that could actually be faster? Every time I encounter this attitude it is just about not wanting to pay the fixed learning cost to set up a debugger in your development environment. Pair this with the principle that a regression test should be written for every bug.
You’re describing an easy case. If I know the line to look at, I may be able to spot the bug without even running the program. If I know which expressions to inspect, I can
printfthem. In easy cases most methods are similar, so it’s just a matter of preference.It’s the mysteries like “1% of responses is missing an http header that was supposed to be there”. Then I have to wonder, do I trace a whole request start to finish to see where it goes wrong? Will I need to trace it 100 times? If it’s a timing issue, will it even happen under the debugger at all?
I don’t understand why people have negative thoughts about using debuggers. There isn’t a “tests or debugger” dichotomy, and there’s no reason not to use both. They’re different tools for solving different problems.
I can’t imagine getting up to speed and making changes on a large code base without using a debugger.
Probably because raw gdb and lld are a pain to use. Unless you adopt a full fledged IDE to do your work there are hardly any good standalone UX debuggers for linux.
I’m keeping my eye on uscope which looks promising https://calabro.io/uscope
I think I tried it a couple of weeks ago and it wasn’t usable in wayland with some pretty bad windowing bugs. :(
Yeah, I’d agree with that.
I generally prefer developing on Linux, but I have to admit Visual Studio’s C++ debugger is a lot easier to work with. I’m not a fan of the UI hangs and questionable default settings, but it’s quick and easy to use and the learning curve is much gentler than GDB’s.
i worked on debuggers and i use them in sometimes
weirdcreative ways. for instance i recently turned rr into a bash profiler, by recording the execution of the process and then injecting calls to dump the current line and function name at various points in the recording. this technique of recording the process and extracting a profile from the recording was invented by a dear friend i met when working on a proprietary debugger. it’s a really powerful idea because the program can run unmodified and it doesn’t require support from the system/interpreter/runtime, thus collecting the profile adds basically zero overhead (the overhead of the debugger isn’t necessarily zero, but it doesn’t modify the state of bash, unlike the overhead of printing this information from within bash which can be very significant in certain optimised programs, rendering the profile meaningless)here’s the version for doing that with rr https://gist.github.com/izabera/ac954b99e7022ddd157a5034dfbd04c1 and it’s trivially adaptable to other languages
Two things specifically about tool-assisted print injection in comparison to stepping/inspection.
I cannot write C or C++ without a debugger, but Python, R, HTML/CSS/JS I don’t think I’ve ever used a debugger. So I guess it’s language dependent
Debuggers have fallen out of fashion because programming tools have become really inconsistent. In the 90s everything had a debugger. You often didn’t have tests, you just debugged everything. I used to dog food my apps by running them with a debugger permanently attached, when it crashed or hit a bad case you could jump in and sometimes even fix it on the fly.
Good luck doing that these days. Everything is in the cloud and you can’t get access to anything. Just reboot it and hope for the best
CI is a good example of this. You can’t debug that shit. You just poke the config and hope for the best. Over, and over again. So unproductive
You are correct and it is extremely annoying. I really wish there were a better way to do it. Every time I modify the CI it is guaranteed to take the entire day.
It is bad and nobody is doing anything.
CI is bad if you can’t run the pipeline locally to test. Even if locally is in a container, but preferably NOT.