cgdb built from master was the only usable linux debugger I found when I looked a few years ago, providing an experience similar to that of 22 year old Visual Studio but with more crashes. Other gdb wrappers either don’t work or are unusable. lldb is unusable too.

It’s unfortunate, but really not any different to the rest of linux userspace…

It also takes over the arrow keys, which makes it much harder to edit and run commands in the bottom half of the window.

The prompt supports readline shortcuts, so you can Ctrl+p for previous command, ctrl+n for next, etc. You navigate through words/characters with readline shortcuts as well (Ctrl+b, ctrl+f, alt+f, alt+b)

Readline cheatsheet

Hm how was that binary built? Is it a Zig binary?

I haven’t had problems building a C++ binary myself with shell scripts, and then debugging it with gdb console, Eclipse, or Clion – on Ubuntu 16.04. I can set the stack, set breakpoints, and it jumps to the right source locations, etc.

I think you basically just have to build it with -g, and if you strip it, you have to set the ELF header metadata to point to the debug symbols.

I’m impressed with how many things you tried, but I think there are a whole bunch of different things going on here, and they could probably be teased out a little more.

I think if you create a simple C or C++ test binary you might get different results (though it’s not clear exactly what you’re trying to do from the blog post).

These days, I usually use a combination of lldb and radare2 on HardenedBSD. Very occasionally, I use Ghidra.

What else should I try?

Netbeans. They have quite nice GDB support.

(unfortunate you need Netbeans version 8 or install C/C++ plugins from 8 to current Apache Netbeans, because they have not included the C/C++ support yet)

A better title would be, “the state of linux debugger frontends.” gdb itself is quite reliable[1], but it’s an enormously complex piece of software and interoperating with complex software is difficult.

I think debuggers are a poor power-complexity trade-off for most debugging. They do have a place, but for most simple things, tools amplifying debug-by-print feel much more economic. Here’s the tool I debug with these days: https://github.com/akkartik/mu/blob/main/tools/browse_trace.readme.md It’s quite independent of my project and surprisingly versatile. All you have to do is emit prints in a specific format with a depth at the front.

Here is a description of the big picture of debugging that this tool fits in: https://github.com/akkartik/mu/blob/main/subx_debugging.md

All this is possible with an outlay of a couple of hundred lines of C (so fewer lines in a higher-level language). I think we programmers need to expand our horizons on the extent to which we can help ourselves and not need to rely on complex, specialist tools.

[1] As long as I don’t try to watch variables. Or travel backwards in time.

Putting on my Zig hat here, the thing that jumps out to me is this:

assert(meta.deepEqual(expected.items, actual.items));

If we had a better facility for testing deep equality, there would be no need for a debugger at all! We have this for string comparison. With a line like this:

testing.assertStringsEqual(a, b);

The output for one example values of a and b looks like:

Test [1/1] test "example"... 
====== expected this output: =========
one
two
three␃

======== instead found this: =========
one
three␃

======================================
First difference occurs on line 2:
expected:
two
 ^
found:
three
 ^
test failure

I know the point of the blog post is trying to use this as an example to examine the state of linux debuggers, but this was my personal takeaway :)

I agree that the GDB TUI is terrible. I use GDB in Emacs and to get the stacktrace and print variables I type the commands in the GUD command window. It’s not optimal, but it works. I’ve never had the graphical interface work as advertised.

Having a decent GUI interface with good memory views is a real boon to debugging low-level problems. It’s too bad that it doesn’t seem to work.

The console interface to gdb works, but it’s inefficient:

1. I can’t easily see the surrounding code
2. I have to manually request information rather than just glancing at the display

That’s why they called Visual Studio “visual”, isn’t it? GUI-style interaction offers you options to pick from, instead of requiring that you know how to ask for what you need.

What else should I try?

Not exactly what you’re looking for, and certainly problematic for multithreaded code, I’ve had success using edb when working with my single threaded assembly programs.

I have personally stopped using debuggers for crashes altogether. I rarely find them useful anymore. I compile my programs with asan (AddressSanitizer) and ubsan (UndefinedBehaviourSanitizer), which insert an enormous amount of checks into the binary directly. The overhead is not insignificant (2-7x), but with that knowledge it can not only detect corruptions that only cause a crash later, it can also pinpoint the exact type of error (use after free, buffer overflow, null reference, overflow, etc), summarize the layout of your data in memory (“read of size 4 at 8 bytes right of array x”) and track the lifecycle of objects and where they were allocated.

With memory bugs taken care of, the major class of problems left for me is then application bugs. For this I often use bpftrace, which lets you attach lightweight probes to collect data when a function is entered or exited without having to restart the binary. It is especially useful with lower level software, as you can use it to collect information from both kernel space and arbitrarily many different user space processes at the same time.

I’ve been using ugdb and loving it.