One area that print shines compared to interactive debuggers is seeing a sequence of states. With a debugger, you often have to keep the sequence of events in your head, while a log of output will show you all the events and their states at the same time.
This pattern can be enhanced where needed by including a stack trace in your print output, which is pretty easy in Python.
This is also why I really want all systems to have an extraordinarily cheap-and-off-by-default logging layer. Because these uses of print-debugging can often be productively leveled up to logging and then shared for future debugging sessions by you and others.
And even when using an interactive debugger, this sequence of states providing context can be invaluable. It can end up being both, not either-or.
I think of print debugging as the worst implementation of trace-based debugging. I use it because it’s often the only available implementation.
I really miss the stream traces we got from the first CHERI prototypes. I wrote some tooling that let you reconstruct the state of the register file at any instruction and show the disassembly in context, so I could find a faulting instruction and search backwards to find the last store to a memory location or last update of a particular register.
DTrace is usually better than printf, but it isn’t universally available.
I don’t look down on print debugging but I do look down on people who can’t be bothered to learn their tools.
Also I’m not sure the Brian Kernighan quote is relevant anymore. I’m sure he would have been astounded at the debugging capabilities of a tool like Visual Studio back in 1979.
Exactly. Who looks down on tools that work? Whether the tool is a console output, some amazing UI around a debugger, or custom-built data structures built just to debug a single hard-to-find problem? The result is what’s important.
the argument generally goes that if print debugging can’t help you figure out the problem then the code is probably too hard to follow and it’s time to refactor. The person I have worked with that was the most knowledgeable about using a step debugger also happened to write the worst code I’ve ever encountered, because invariably he could always fine the exact place something was going wrong, and then he’d … write a conditional right there instead of rethinking the design.
a) haven’t had an opportunity to discover that a tool exists or is applicable, or
b) have tried to learn the tool but struggled with the educational material available.
I think this is often mistaken for “can’t be bothered to learn their tools”, but rather than a reason to look down on them, is really an opportunity to share knowledge and awareness. It might be any of you reading this who has the right opportunity to spread awareness of a tool, or the right approach to teaching how to use it effectively.
I really encourage people to assume (a) or (b) rather than get frustrated or look down on people.
You’ll never get a debugger in production, so no matter how good you get with that tool you still need to learn how to produce quality logs and read them
Sure you can. Attaching a debugger to remote, running process is a totally normal thing that decent debuggers can do. You just have to deploy a debug build, which a lot of places do for this exact reason. And even if you don’t attach to it, you can often get a dump which you can open in a debugger to examine what happened.
You don’t even have to go debug build (which do have a substantial observer effect much akin to a ‘I just add some printf’ recompiles) - many, particularly proprietary or in-house, projects don’t.
You generate the debug metadata on the side and keep it on a symbol server. Remote attach-debugging or crash-dump to triage server combined with the current state of decompilers in production, is often more than sufficient.
I feel like a majority of people use print debugging because they’ve never learnt to use a proper debugger (I was once one of those people).
It’s frustrating pairing with poeple where the feedback loop is so much slower because they’re adjusting print statements to include additional context that could readibly be explored within the debugger.
Tracing and snapshot debugging are two different dimensions of debugging (over time and over space respectively) and it is a mistake to try and call one of them “proper”.
Debuggers that work over both time and space are record and replay debuggers like rr — which are fantastic but have their own compromises (e.g single threaded, limited platform support).
It’s frustrating pairing with poeple where the feedback loop is so much slower because they’re adjusting print statements to include additional context that could readibly be explored within the debugger.
I have a question for you then: after over 17 years on the job, I am still one of those people. How do I shorten my feedback loop?
A bit of context first: I program mostly in C and C++, with bit of Python and Ocaml. I have used visual debuggers when available, but in my practice, the one tool that really saved me was Valgrind, not the debugger. And to this day, I still use printf() as my main debugging tool. I tried learn a bit of GDB, but so far it has been underwhelming and insufficient: I can run my program, set up break points, step in/over/out, but I can’t visualise my program state to save my life: scalars are okay, but something as simple as a byte buffer, no clue yet.
To make it more concrete, let’s take a specific example: this week I implemented the SSCPv2 protocol from specs. It’s an old school (MAC then encrypt) cryptographic protocol devised I think around 2022, meant to secure RS232 and RS485 links. It’s mostly used to secure buildings, and the threat model includes physical access to the wire. To complete my assignment I ripped off BearSSL’s AES CBC, stole a free SHA256 implementation, adapted my own implementation of HMAC, and constructed the rest of the protocol from there. I used C, which I am most familiar with for this domain (and I needed a C API anyway to talk to an existing Dart implementation).
Yes, I implemented my own crypto for a real customer that must have actual security on production. In 5 days. In C. Sue me. We were not aware of a freely available implementation for Linux hosts, and the only other alternative was OSDP.
My validation tests typically compared my output to test vectors, and failed if there was a mismatch. Obviously, I made tons of mistakes I needed to debug. My method was looking up at various values, mostly byte buffers, and look for the point where I started to diverge. My main tool for this was this:
void print_vector(const char *name, const u8 *buf, size_t size)
{
printf("%s: ", name);
for (size_t i = 0 i < size; i++) {
printf("%02x", buf[i]);
}
printf("\n");
}
Whenever something went wrong I sprinkled my program with calls to that function, and after waiting 300ms for the whole thing to recompile and my test suite to run, I had a visualisation of what I thought the relevant buffers at the relevant times. When that wasn’t enough, I added some more print statements until I saw the light and fixed my bug.
What I know of GDB, or even a visual debugger from an IDE, doesn’t feel faster. I’m totally willing to blame a skill issue there. Maybe I would have taken 4 days instead of 5 if I knew the proper tools. As things stand though, I am fairly sceptical. So, for this specific gig I just completed, do you know of a debug method and associated tools that, had I taken the time to properly learn them, would have been likely to speed me up?
The only thing I can think of (especially since you mentioned C) is the liberal use of assert(). I typically use it to ensure parameters are valid (because in my opinion, if they aren’t, it’s a bug), but I have also used them to check the validity of structures upon function entry and (if need be) function exit. They can also help you solidify your own assumptions about the code.
I’ve been programming for 40 years, and under LInux for about 25, and so far, I’ve found it much easier to use printf debugging, although I have learned enough about gdb to help the debugging process (conditional breakpoints for the win). Often times, I don’t even have access to a debugger thanks to either the system or language being used.
The hardest bug I’ve had to find wasn’t amenable to a debugger or printf debugging, but lots of thought and careful inspection of the code (Unix signal handlers are a bitch and I learned the hard way that there’s not much one can safely do in a signal handler).
I’ve learned how to use debuggers 25 years ago, have used probably 15 different ones in several languages and always go back to print debug.
Your assumption is wrong. You assume that a debugger is faster because you don’t waste time setting up print statements. That is at best debatable, but I will even claim that the opposite is true. Strategically place a couple of print statements and you will get a clear birds eye view of what is happening and where something is messed up. If you use a debugger, you have to guess where to stop. It is impossible to know ahead of time because the location of the bug is what you’re after. So people end up needing to run the program in debug mode a few times till they get it to stop at a useful place. Often times frenetically stepping over a dozen times till they realize they need to find a better breakpoint.
By this time, a seasoned print debugging programmer is already working in fixing the bug.
It does require more practice than a debugger. But in some dimension it can be more powerful.
If you use a debugger, you have to guess where to stop. It is impossible to know ahead of time because the location of the bug is what you’re after.
I can flip that around and say you have to guess what to print and when. It’s impossible to know ahead of time because the point where the invariant is violated is what you’re after.
For the record I use both techniques, often at the same time. The flow is something like: add tracing via prints, then once I narrow down the source add a breakpoint to investigate further. If I don’t need to narrow down the candidates for a breakpoint I go straight for the debugger.
I can flip that around and say you have to guess what to print and when. It’s impossible to know ahead of time because the point where the invariant is violated is what you’re after.
No, you can’t. That’s my point. A debugger is what forces you to wager on a breakpoint location that might be helpful or useless. Print debugging let’s the program execute as usual, even if your print statements are position too early or too late I. The execution path.
The reason why I usually don’t bother with debuggers is because my debugging workflow is exactly as you describe. I also used debugging once I narrowed down the approximate location of the offending code. But with practice this becomes an unnecessary last step.
How is breaking somewhere useless any different from printing a value that offers no insight? Why is adding a useless breakpoint any more of a wager than adding a useless print? You can obtain (or not) the same information from both.
Because the useless print statement does not stop the execution and the useful information could be in further calls that not the first few. And you can also place multiple at once, not constrained by the risk of stopping to soon.
It’s frustrating pairing with poeple where the feedback loop is so much slower because they’re adjusting print statements to include additional context that could readibly be explored within the debugger.
that really depends on the platform, language & debugger.
here’s an example I just recorded: iteration loop from “adding a printf to the code” to “print executes” is like 1-2 seconds max. In contrast debuggers take at least 4/5 seconds to load, and gdb even crashes. https://streamable.com/k8f6v3
UNIX: program crashed, better recompile with print statements as we have no debugger.
LISP: program crashed, here’s a debugger. Restart when you fixed your mess.
So in an alternative timeline we all have debuggers always available all the time and printf is lost to time. It’s not that I look down on printf debugging, it’s that I long for an alternate history where we are more parenthetically inclined.
print debugging is so popular in lisp circles that one of the popular modern tools for developing common lisp programs, Sly, literally has it integrated. you can select any expression to have its value captured instead of having to manually insert a print.
Calling Sly’s stickers feature “print debugging” is absurd. It’s like saying that using GDB is print-debugging because you use the print command to inspect values.
Unless you set them to break, all they do is capture values and then let you go through them in order. Seems functionally identical to print debugging to me.
By a similar logic any value inspection a debugger is print debugging. Which it isn’t, printing involves many things, it produces output (and a system call or two, generally), is irreversible, and so on.
I wish SLY had proper step debugging like edebug, though. Many imperative IDEs let you step statement-by-statement and see how variables behave, sexp step debugging is something I wish was more widespread. To date I’ve only seen edebug (for emacs lisp) and cider-debug (for Clojure).
Amusingly, the BASIC on the Atari 2600 gaming system can trace substeps in an expression! That’s about the only good thing about programming on a system with 128 bytes (yes, BYTES) of RAM.
Hot code reload is cool, it allows you to retry with prints inserted quickly and without disrupting slow-to-setup semi-persistent state! (My Common Lisp code was multithreaded and subject to external timeouts, so I was just writing macros to do print debugging more comfortably rather than figuring out how to use the debugger in such a context; and trace was often worse fit than slapping print-every-expression on a crucial let*)
The thing that has really made me unsatisfied with print debugging recently is not having much structure in the output. When you start using observability tooling to solve some issues, suddenly everything looks like a poor man’s trace.
Having context with your ‘simple’ print statements (And ideally being able to collapse stuff) means you really can see the state of your program across the board. The moral equivalent to opening up the CPU and just looking as it’s doing stuff.
Debugging tools to poke around at values is nice, but the aggregative effects of building up a logging universe on a branch you’re working on means that you end up seeing “all” the important values!
Really though, I think I want something like dtrace but with breakpoint support. Let me write logging statements all over my program, let me easily filter out the logging statements and mess with it while the program is running. And let me jump into debugging when I hit a certain kind of structure in a certain kind of way.
I think debugger vs print debugging is a lot about what sort of application you’re writing.
Writing a web server that’s primarily driven by network requests with lots of async logic? Use print debugging.
Writing a C++ desktop application where compile times are slow and everything run in a single process? Use a debugger.
Writing a JavaScript web application? It depends. If you’re debugging the UI, maybe use a debugger. If you’re debugging request handling, maybe use print statements.
For me, it really just depends on the problem space.
I strongly hold a belief that every program ideally should offer debug logging detailed enough to allow the operator to get a sense of what exactly is going on and why it’s not going as they expect, even without inspecting the source code. An added benefit is that the maintainer can ask users to send them debug logs in case of complicated bugs they cannot reproduce.
It’s not always possible, but I find it really helpful when it is, both as a user and as a maintainer.
Exactly. I “debug print” by fleshing out my logging down to the TRACE level because it means I don’t have to do it a second time if I need to debug stuff in the future.
Having done far too much historical debugging in production, when I see people deleting log lines while using a logger supporting level, I am always much confused. Sure, make the statement well-formed and intelligible. Sure, avoid string-formatting overhead which a decent logging framework will help with. But if you ran into an issue trying to figure this out in development, well, it could crop up as an issue in prod with that corner case you didn’t observe. Leave the log-line in there and off most of the time (but on in the debug environment), so when we are trying to trace through a production issue, it’s as simple as turning the log line on.
Not mentioned in the article: you can collect megabytes or even gigabytes of printf output from your program run and then use all manner of tools to quickly and powerfully analyse it, ranging from simple grep to awk or perl or even lisp. Don’t single-step your program or jump from breakpoint to breakpoint manually looking at key values … let your computer do the work for you – it’s billions of times faster than you are. Analysing logs is the kind of thing where the more expressive the language and the faster you can iterate your analysis and re-run it the better.
This is especially important for things where different runs of your program can produce different results because of timing differences, communications, different real-world input coming in etc. You can capture a single run and analyse it.
You may want to execute a particular printf in your program only when certain complex conditions are detected – maybe then set a flag to trigger other printfs. You can do that in a debugger, but the logic to do it in your actual compiled program code runs a thousand times faster than conditional breakpoints and other triggers in a debugger.
If carefully designed and not excessive, printf/logging from your program doesn’t slow it down enough to affect real-time interactions with the world. Sitting at a breakpoint in a debugger for seconds or minutes often completely changes what is happening, can even timeout network connections etc.
An interactive debugger is useful for beginners who don’t yet understand their programming language, don’t understand how loops and other logic work: simply don’t understand their data structures and algorithms.. Or for the more experienced programmers if you are trying to dive into a program that you’re completely unfamiliar with.
Also, sometimes setting a breakpoint and examining stuff right at that time just doesn’t work. For example, when you’re waiting on a remote program to trigger the faulty code path and can’t have it pause for too long or the remote side will time out and hang up.
In these cases I have pickled all the possibly relevant objects and loaded them in an interactive python shell afterwards to examine their contents. Sometimes, this doesn’t work; iirc Flask gets unhappy when you attempt to load a pickled request object (because they are doing something a little too clever).
I’ve always freely used the spectrum of techniques from fancy debugger to printf/logging. Lately I’ve been working on an embedded project for fun that involves a kind of homegrown TDMA radio protocol, so it requires accurate and precise timing (to the tens of microseconds) across multiple devices. The best way I’ve found to debug it is “one-bit printf”: probing lots of single-bit GPIO outputs with a logic analyzer, and moving the bit set/clear statements to different places in the code. So the spectrum is even wider than I realized!
Also, long ago I got a little desperate debugging a HyperCard stack, and inserted code to play tones at different pitches as a form of “printf”. That was fun.
One can get a port 80 display for the PC architecture. It’s used (was used?) by the BIOS code to indicate where it was, and if there were an error, a diagnostic code could be displayed on two LED 7-segment displays.
Horses for courses. Debuggers are good during development. You usually don’t get that debugger in Production so you’ve hopefully put in a bunch of structured log statements or plain old printf’s and an external trigger to cause it to emit them. If you’ve really been around the block you have a ring buffer and log into that, dumping on unexpected conditions (the, “this can never happen” clauses) and when that external trigger is flipped.
This strikes me as similar to the woodworking community “debates” around ‘I stick to hand planers and chisel thankyouverymuch’ versus those with a full on machine shop – although I suspect that a majority of the poor-man’s tracer (that’s printf) fans has much less intuition about the downsides to the tool they’re using versus someone with a much beloved hand planer collection.
As an exercise, write down the side effects of you think whatever you have that fills the ‘print’ role. Dig down one system layer or two and compare that to the actual implementation instead of the pragmatist ‘I called printf(), I got thing on stderr, that’s good enough’. No matter how you slice it, trace output by synthesising strings and writing to a file is full of ‘shaking the baby’ energy.
I don’t recall the first use of it, but at least ‘the science of debugging (2021)’ (ignoring the lack of science in it) leaned hard into prebugging: make conscious design choices towards ensuring your program will be debuggable and keep the implementation that way; complex data model? sneak an on-demand snapshot and report generator into that bad boy; complex flow and/or tight performance requirements? there’s plenty of production friendly tracing frameworks that know how to sample and exfiltrate. suspect dependencies? be sure to decouple and be able to link in placeholders, proxies alternate implementations etc.
Thankfully having to implement is more the exception than the norm by now (as long as you’re not in embedded). The tracing situation is in a good place, much more-so than symbolic debuggers, and need little in terms of manual work even: strace built with stacktraces enabled and some choice -e filters, dtrace if you’re so lucky, any interpreted language VM of repute, …
When actually needing to annotating software specific knowledge, having LOG macros match format so you can slot in Tracy.hpp is awesome in both UI, minimising overhead, dealing with threading quirks. Remote access support to boot. The only downside for me have been merging traces from multiple processes on multiple devices. Even if it is specialised around graphics it generalises well for anything with timeline and ordering needs.
Print debugging can be useful together with GDB as well.
If theres some part where youre not exactly sure where to place a breakpoint, add some prints and add breakpoints to those. You get to see the print and then you can start stepping around the code.
Great post. Good job destroying silly dogmas that are devoided of reason.
I am glad such discussions are popping up, we need to bring back critical thinking to software engineering. Waaaayyy to many pre-made ideas that don’t get questioned enough. Good post about best practices too.
One area that print shines compared to interactive debuggers is seeing a sequence of states. With a debugger, you often have to keep the sequence of events in your head, while a log of output will show you all the events and their states at the same time.
This pattern can be enhanced where needed by including a stack trace in your print output, which is pretty easy in Python.
+100
This is also why I really want all systems to have an extraordinarily cheap-and-off-by-default logging layer. Because these uses of print-debugging can often be productively leveled up to logging and then shared for future debugging sessions by you and others.
And even when using an interactive debugger, this sequence of states providing context can be invaluable. It can end up being both, not either-or.
I think of print debugging as the worst implementation of trace-based debugging. I use it because it’s often the only available implementation.
I really miss the stream traces we got from the first CHERI prototypes. I wrote some tooling that let you reconstruct the state of the register file at any instruction and show the disassembly in context, so I could find a faulting instruction and search backwards to find the last store to a memory location or last update of a particular register.
DTrace is usually better than printf, but it isn’t universally available.
Another use case is any sort of bulk data that you’d want to display but is only available as singular local variables, and not aggregated anywhere.
I don’t look down on print debugging but I do look down on people who can’t be bothered to learn their tools.
Also I’m not sure the Brian Kernighan quote is relevant anymore. I’m sure he would have been astounded at the debugging capabilities of a tool like Visual Studio back in 1979.
Exactly. Who looks down on tools that work? Whether the tool is a console output, some amazing UI around a debugger, or custom-built data structures built just to debug a single hard-to-find problem? The result is what’s important.
the argument generally goes that if print debugging can’t help you figure out the problem then the code is probably too hard to follow and it’s time to refactor. The person I have worked with that was the most knowledgeable about using a step debugger also happened to write the worst code I’ve ever encountered, because invariably he could always fine the exact place something was going wrong, and then he’d … write a conditional right there instead of rethinking the design.
A lot of folks I encounter either:
I think this is often mistaken for “can’t be bothered to learn their tools”, but rather than a reason to look down on them, is really an opportunity to share knowledge and awareness. It might be any of you reading this who has the right opportunity to spread awareness of a tool, or the right approach to teaching how to use it effectively.
I really encourage people to assume (a) or (b) rather than get frustrated or look down on people.
You’ll never get a debugger in production, so no matter how good you get with that tool you still need to learn how to produce quality logs and read them
Sure you can. Attaching a debugger to remote, running process is a totally normal thing that decent debuggers can do. You just have to deploy a debug build, which a lot of places do for this exact reason. And even if you don’t attach to it, you can often get a dump which you can open in a debugger to examine what happened.
You don’t even have to go debug build (which do have a substantial observer effect much akin to a ‘I just add some printf’ recompiles) - many, particularly proprietary or in-house, projects don’t.
You generate the debug metadata on the side and keep it on a symbol server. Remote attach-debugging or crash-dump to triage server combined with the current state of decompilers in production, is often more than sufficient.
Funny, it often feels like the opposite sentiment is much more common: “Debuggers are a waste of time, we only need print debugging”
I feel like a majority of people use print debugging because they’ve never learnt to use a proper debugger (I was once one of those people).
It’s frustrating pairing with poeple where the feedback loop is so much slower because they’re adjusting print statements to include additional context that could readibly be explored within the debugger.
Tracing and snapshot debugging are two different dimensions of debugging (over time and over space respectively) and it is a mistake to try and call one of them “proper”.
Debuggers that work over both time and space are record and replay debuggers like rr — which are fantastic but have their own compromises (e.g single threaded, limited platform support).
I have a question for you then: after over 17 years on the job, I am still one of those people. How do I shorten my feedback loop?
A bit of context first: I program mostly in C and C++, with bit of Python and Ocaml. I have used visual debuggers when available, but in my practice, the one tool that really saved me was Valgrind, not the debugger. And to this day, I still use
printf()as my main debugging tool. I tried learn a bit of GDB, but so far it has been underwhelming and insufficient: I can run my program, set up break points, step in/over/out, but I can’t visualise my program state to save my life: scalars are okay, but something as simple as a byte buffer, no clue yet.To make it more concrete, let’s take a specific example: this week I implemented the SSCPv2 protocol from specs. It’s an old school (MAC then encrypt) cryptographic protocol devised I think around 2022, meant to secure RS232 and RS485 links. It’s mostly used to secure buildings, and the threat model includes physical access to the wire. To complete my assignment I ripped off BearSSL’s AES CBC, stole a free SHA256 implementation, adapted my own implementation of HMAC, and constructed the rest of the protocol from there. I used C, which I am most familiar with for this domain (and I needed a C API anyway to talk to an existing Dart implementation).
Yes, I implemented my own crypto for a real customer that must have actual security on production. In 5 days. In C. Sue me. We were not aware of a freely available implementation for Linux hosts, and the only other alternative was OSDP.
My validation tests typically compared my output to test vectors, and failed if there was a mismatch. Obviously, I made tons of mistakes I needed to debug. My method was looking up at various values, mostly byte buffers, and look for the point where I started to diverge. My main tool for this was this:
Whenever something went wrong I sprinkled my program with calls to that function, and after waiting 300ms for the whole thing to recompile and my test suite to run, I had a visualisation of what I thought the relevant buffers at the relevant times. When that wasn’t enough, I added some more print statements until I saw the light and fixed my bug.
What I know of GDB, or even a visual debugger from an IDE, doesn’t feel faster. I’m totally willing to blame a skill issue there. Maybe I would have taken 4 days instead of 5 if I knew the proper tools. As things stand though, I am fairly sceptical. So, for this specific gig I just completed, do you know of a debug method and associated tools that, had I taken the time to properly learn them, would have been likely to speed me up?
I can’t say much about the current state of it but that seems up DDDs alley - https://www.gnu.org/software/ddd/
Personally I use a hybrid between https://www.youtube.com/watch?v=WBsv9IJpkDw (the first part is n mutations of a buffer from a memory dumper with xor and a few variants on that) + https://videos.files.wordpress.com/Pqo1Hwx8/clip_8_debug_mp4_hd.mp4 + https://arcan-fe.com/2024/09/16/a-spreadsheet-and-a-debugger-walks-into-a-shell + some mutation and snapshotting / distribution scripts to see how buffer versus test vectors mutate, but the turnkey for that is not public and won’t be now post LLM-scrappy-times.
The only thing I can think of (especially since you mentioned C) is the liberal use of
assert(). I typically use it to ensure parameters are valid (because in my opinion, if they aren’t, it’s a bug), but I have also used them to check the validity of structures upon function entry and (if need be) function exit. They can also help you solidify your own assumptions about the code.I’ve been programming for 40 years, and under LInux for about 25, and so far, I’ve found it much easier to use printf debugging, although I have learned enough about gdb to help the debugging process (conditional breakpoints for the win). Often times, I don’t even have access to a debugger thanks to either the system or language being used.
The hardest bug I’ve had to find wasn’t amenable to a debugger or printf debugging, but lots of thought and careful inspection of the code (Unix signal handlers are a bitch and I learned the hard way that there’s not much one can safely do in a signal handler).
[Comment removed by author]
I’ve learned how to use debuggers 25 years ago, have used probably 15 different ones in several languages and always go back to print debug.
Your assumption is wrong. You assume that a debugger is faster because you don’t waste time setting up print statements. That is at best debatable, but I will even claim that the opposite is true. Strategically place a couple of print statements and you will get a clear birds eye view of what is happening and where something is messed up. If you use a debugger, you have to guess where to stop. It is impossible to know ahead of time because the location of the bug is what you’re after. So people end up needing to run the program in debug mode a few times till they get it to stop at a useful place. Often times frenetically stepping over a dozen times till they realize they need to find a better breakpoint. By this time, a seasoned print debugging programmer is already working in fixing the bug.
It does require more practice than a debugger. But in some dimension it can be more powerful.
I can flip that around and say you have to guess what to print and when. It’s impossible to know ahead of time because the point where the invariant is violated is what you’re after.
For the record I use both techniques, often at the same time. The flow is something like: add tracing via prints, then once I narrow down the source add a breakpoint to investigate further. If I don’t need to narrow down the candidates for a breakpoint I go straight for the debugger.
No, you can’t. That’s my point. A debugger is what forces you to wager on a breakpoint location that might be helpful or useless. Print debugging let’s the program execute as usual, even if your print statements are position too early or too late I. The execution path.
The reason why I usually don’t bother with debuggers is because my debugging workflow is exactly as you describe. I also used debugging once I narrowed down the approximate location of the offending code. But with practice this becomes an unnecessary last step.
How is breaking somewhere useless any different from printing a value that offers no insight? Why is adding a useless breakpoint any more of a wager than adding a useless print? You can obtain (or not) the same information from both.
Because the useless print statement does not stop the execution and the useful information could be in further calls that not the first few. And you can also place multiple at once, not constrained by the risk of stopping to soon.
That’s exactly what I’d debate (in my particular domain), so let’s agree to disagree.
that really depends on the platform, language & debugger. here’s an example I just recorded: iteration loop from “adding a printf to the code” to “print executes” is like 1-2 seconds max. In contrast debuggers take at least 4/5 seconds to load, and gdb even crashes. https://streamable.com/k8f6v3
Exactly. It shouldn’t be an either-or. Each has its appropriate use cases.
Historical analysis (fun, not accurate)
UNIX: program crashed, better recompile with print statements as we have no debugger. LISP: program crashed, here’s a debugger. Restart when you fixed your mess.
So in an alternative timeline we all have debuggers always available all the time and printf is lost to time. It’s not that I look down on printf debugging, it’s that I long for an alternate history where we are more parenthetically inclined.
print debugging is so popular in lisp circles that one of the popular modern tools for developing common lisp programs, Sly, literally has it integrated. you can select any expression to have its value captured instead of having to manually insert a print.
Calling Sly’s stickers feature “print debugging” is absurd. It’s like saying that using GDB is print-debugging because you use the
printcommand to inspect values.Unless you set them to break, all they do is capture values and then let you go through them in order. Seems functionally identical to print debugging to me.
By a similar logic any value inspection a debugger is print debugging. Which it isn’t, printing involves many things, it produces output (and a system call or two, generally), is irreversible, and so on.
I wish SLY had proper step debugging like edebug, though. Many imperative IDEs let you step statement-by-statement and see how variables behave, sexp step debugging is something I wish was more widespread. To date I’ve only seen edebug (for emacs lisp) and cider-debug (for Clojure).
Amusingly, the BASIC on the Atari 2600 gaming system can trace substeps in an expression! That’s about the only good thing about programming on a system with 128 bytes (yes, BYTES) of RAM.
Hot code reload is cool, it allows you to retry with prints inserted quickly and without disrupting slow-to-setup semi-persistent state! (My Common Lisp code was multithreaded and subject to external timeouts, so I was just writing macros to do print debugging more comfortably rather than figuring out how to use the debugger in such a context; and
tracewas often worse fit than slapping print-every-expression on a cruciallet*)The thing that has really made me unsatisfied with print debugging recently is not having much structure in the output. When you start using observability tooling to solve some issues, suddenly everything looks like a poor man’s trace.
Having context with your ‘simple’ print statements (And ideally being able to collapse stuff) means you really can see the state of your program across the board. The moral equivalent to opening up the CPU and just looking as it’s doing stuff.
Debugging tools to poke around at values is nice, but the aggregative effects of building up a logging universe on a branch you’re working on means that you end up seeing “all” the important values!
Really though, I think I want something like dtrace but with breakpoint support. Let me write logging statements all over my program, let me easily filter out the logging statements and mess with it while the program is running. And let me jump into debugging when I hit a certain kind of structure in a certain kind of way.
i wannaaaaaaa https://gu.outerproduct.net/debug.html STRICTLY BETTER but no one bothered to implement it
I think debugger vs print debugging is a lot about what sort of application you’re writing.
Writing a web server that’s primarily driven by network requests with lots of async logic? Use print debugging.
Writing a C++ desktop application where compile times are slow and everything run in a single process? Use a debugger.
Writing a JavaScript web application? It depends. If you’re debugging the UI, maybe use a debugger. If you’re debugging request handling, maybe use print statements.
For me, it really just depends on the problem space.
I strongly hold a belief that every program ideally should offer debug logging detailed enough to allow the operator to get a sense of what exactly is going on and why it’s not going as they expect, even without inspecting the source code. An added benefit is that the maintainer can ask users to send them debug logs in case of complicated bugs they cannot reproduce.
It’s not always possible, but I find it really helpful when it is, both as a user and as a maintainer.
Exactly. I “debug print” by fleshing out my logging down to the TRACE level because it means I don’t have to do it a second time if I need to debug stuff in the future.
Having done far too much historical debugging in production, when I see people deleting log lines while using a logger supporting level, I am always much confused. Sure, make the statement well-formed and intelligible. Sure, avoid string-formatting overhead which a decent logging framework will help with. But if you ran into an issue trying to figure this out in development, well, it could crop up as an issue in prod with that corner case you didn’t observe. Leave the log-line in there and off most of the time (but on in the debug environment), so when we are trying to trace through a production issue, it’s as simple as turning the log line on.
Not mentioned in the article: you can collect megabytes or even gigabytes of printf output from your program run and then use all manner of tools to quickly and powerfully analyse it, ranging from simple grep to awk or perl or even lisp. Don’t single-step your program or jump from breakpoint to breakpoint manually looking at key values … let your computer do the work for you – it’s billions of times faster than you are. Analysing logs is the kind of thing where the more expressive the language and the faster you can iterate your analysis and re-run it the better.
This is especially important for things where different runs of your program can produce different results because of timing differences, communications, different real-world input coming in etc. You can capture a single run and analyse it.
You may want to execute a particular printf in your program only when certain complex conditions are detected – maybe then set a flag to trigger other printfs. You can do that in a debugger, but the logic to do it in your actual compiled program code runs a thousand times faster than conditional breakpoints and other triggers in a debugger.
If carefully designed and not excessive, printf/logging from your program doesn’t slow it down enough to affect real-time interactions with the world. Sitting at a breakpoint in a debugger for seconds or minutes often completely changes what is happening, can even timeout network connections etc.
An interactive debugger is useful for beginners who don’t yet understand their programming language, don’t understand how loops and other logic work: simply don’t understand their data structures and algorithms.. Or for the more experienced programmers if you are trying to dive into a program that you’re completely unfamiliar with.
Also, sometimes setting a breakpoint and examining stuff right at that time just doesn’t work. For example, when you’re waiting on a remote program to trigger the faulty code path and can’t have it pause for too long or the remote side will time out and hang up.
In these cases I have pickled all the possibly relevant objects and loaded them in an interactive python shell afterwards to examine their contents. Sometimes, this doesn’t work; iirc Flask gets unhappy when you attempt to load a pickled
requestobject (because they are doing something a little too clever).I’ve always freely used the spectrum of techniques from fancy debugger to printf/logging. Lately I’ve been working on an embedded project for fun that involves a kind of homegrown TDMA radio protocol, so it requires accurate and precise timing (to the tens of microseconds) across multiple devices. The best way I’ve found to debug it is “one-bit printf”: probing lots of single-bit GPIO outputs with a logic analyzer, and moving the bit set/clear statements to different places in the code. So the spectrum is even wider than I realized!
Also, long ago I got a little desperate debugging a HyperCard stack, and inserted code to play tones at different pitches as a form of “printf”. That was fun.
One can get a port 80 display for the PC architecture. It’s used (was used?) by the BIOS code to indicate where it was, and if there were an error, a diagnostic code could be displayed on two LED 7-segment displays.
Horses for courses. Debuggers are good during development. You usually don’t get that debugger in Production so you’ve hopefully put in a bunch of structured log statements or plain old printf’s and an external trigger to cause it to emit them. If you’ve really been around the block you have a ring buffer and log into that, dumping on unexpected conditions (the, “this can never happen” clauses) and when that external trigger is flipped.
This strikes me as similar to the woodworking community “debates” around ‘I stick to hand planers and chisel thankyouverymuch’ versus those with a full on machine shop – although I suspect that a majority of the poor-man’s tracer (that’s printf) fans has much less intuition about the downsides to the tool they’re using versus someone with a much beloved hand planer collection.
As an exercise, write down the side effects of you think whatever you have that fills the ‘print’ role. Dig down one system layer or two and compare that to the actual implementation instead of the pragmatist ‘I called printf(), I got thing on stderr, that’s good enough’. No matter how you slice it, trace output by synthesising strings and writing to a file is full of ‘shaking the baby’ energy.
I don’t recall the first use of it, but at least ‘the science of debugging (2021)’ (ignoring the lack of science in it) leaned hard into prebugging: make conscious design choices towards ensuring your program will be debuggable and keep the implementation that way; complex data model? sneak an on-demand snapshot and report generator into that bad boy; complex flow and/or tight performance requirements? there’s plenty of production friendly tracing frameworks that know how to sample and exfiltrate. suspect dependencies? be sure to decouple and be able to link in placeholders, proxies alternate implementations etc.
When is the time to implement some kind of tracing framework in your program? Probably now. :grin:
Thankfully having to implement is more the exception than the norm by now (as long as you’re not in embedded). The tracing situation is in a good place, much more-so than symbolic debuggers, and need little in terms of manual work even: strace built with stacktraces enabled and some choice -e filters, dtrace if you’re so lucky, any interpreted language VM of repute, …
I have only recently started playing around with it but, so far, https://github.com/open-telemetry/opentelemetry-ebpf-profiler is excellent.
When actually needing to annotating software specific knowledge, having LOG macros match format so you can slot in Tracy.hpp is awesome in both UI, minimising overhead, dealing with threading quirks. Remote access support to boot. The only downside for me have been merging traces from multiple processes on multiple devices. Even if it is specialised around graphics it generalises well for anything with timeline and ordering needs.
Print debugging can be useful together with GDB as well.
If theres some part where youre not exactly sure where to place a breakpoint, add some prints and add breakpoints to those. You get to see the print and then you can start stepping around the code.
Both are very useful.
Great post. Good job destroying silly dogmas that are devoided of reason. I am glad such discussions are popping up, we need to bring back critical thinking to software engineering. Waaaayyy to many pre-made ideas that don’t get questioned enough. Good post about best practices too.