1. 44
  1. 11

    cgdb built from master was the only usable linux debugger I found when I looked a few years ago, providing an experience similar to that of 22 year old Visual Studio but with more crashes. Other gdb wrappers either don’t work or are unusable. lldb is unusable too.

    It’s unfortunate, but really not any different to the rest of linux userspace…

    1. 1

      I keep wishing RemedyBG would come to linux.

    2. 4

      It also takes over the arrow keys, which makes it much harder to edit and run commands in the bottom half of the window.

      The prompt supports readline shortcuts, so you can Ctrl+p for previous command, ctrl+n for next, etc. You navigate through words/characters with readline shortcuts as well (Ctrl+b, ctrl+f, alt+f, alt+b)

      Readline cheatsheet

      1. 3

        Hm how was that binary built? Is it a Zig binary?

        I haven’t had problems building a C++ binary myself with shell scripts, and then debugging it with gdb console, Eclipse, or Clion – on Ubuntu 16.04. I can set the stack, set breakpoints, and it jumps to the right source locations, etc.

        I think you basically just have to build it with -g, and if you strip it, you have to set the ELF header metadata to point to the debug symbols.

        I’m impressed with how many things you tried, but I think there are a whole bunch of different things going on here, and they could probably be teased out a little more.

        I think if you create a simple C or C++ test binary you might get different results (though it’s not clear exactly what you’re trying to do from the blog post).

        1. 2

          Hm how was that binary built? Is it a Zig binary?

          It’s a zig binary built in debug mode.

          You can see from the transcript at the beginning that debugging zig stuff in gdb itself works fine, so I was hoping that the various gdb frontends would also just work.

          Notably, each thing I tried in that session worked in at least one of the frontends. So if the bugs are zig-related, they’re also different zig-related bugs per frontend.

          I think if you create a simple C or C++ test binary you might get different results

          It’s possible. It would be a useful thing to do for science to track down where exactly the problems are in each frontend. But I don’t much feel like doing that at the moment, I just want to find a debugger that works on my project.

          I got a ton of comments and emails with suggestions, so I’ll likely have a followup post in a few days.

          1. 4

            Fair enough, but I don’t think the title is really accurate. I would guess a lot of the suggestions you got glossed over the fact that you’re running Zig and Nix. They are technically mentioned in the post, but it’s not clear.

            I would say debugging C/C++ on Ubuntu or Debian is more representative of the state of debuggers on Linux, and I’ve never had a problem getting source locations to work with ANY debugger. I tried at least 5.

            When I hear “Zig and Nix”, I think “probably does not work”. Nix kinda scares me because it completely changes the file system layout (with checksums, etc.). So I would strongly suspect anything to do with finding stuff via paths to be broken unless the package author fixed it, and debug symbols rely on file system paths.

            The package author of GDB may have fixed it, but not the other ones. I like the idea of Nix, but I’m not that interested in using it due to what I’ve seen of its internals [1]

            Also, DWARF is an extremely complicated file format (e.g. it has a Turing complete VM, so you may need to run arbitrary code to compute line numbers [2]). As far as I understand, there’s not just “one way” to encode line numbers. So it wouldn’t surprise me either if Zig’s data tickled a bug here or there.

            That is I think that unconstraining one of the variables Zig and Nix would go a long way to narrowing down the problem. I would guess that the variable “which GDB front end” is less relevant …

            Although to be fair, I found the Eclipse GUI to work roughly 70% on C++/Ubuntu. The symbols worked, but lots of other stuff was broken inexplicably. So I switched to CLion (via free open source license) or gdb console.

            If I wanted the best chance of debugging a Zig binary with a good UI, I’d probably try CLion on Ubuntu/Debian (e.g. copy it to a VM as an initial test).

            [1] e.g. it uses bash in more abusive ways than any Linux distro I’ve seen, and that’s saying a lot: https://github.com/oilshell/oil/issues/26 . My experience is that package authors on all distros often “hack it until it works”, and when there is this kind of complexity, they give up.

            [2] https://kristerw.blogspot.com/2016/01/more-turing-completeness-in-surprising.html

            1. 2

              I’ve never had a problem getting source locations to work with ANY debugger. I tried at least 5.

              I tried 13 and only 1 had trouble with source locations, so I guess we have compatible success rates :)

              The package author of GDB may have fixed it, but not the other ones. debug symbols rely on file system paths.

              Most of the frontends launch gdb from a bash command that you give them, so they’re all using the same binary. The exceptions that I know of are code-lldb, which didn’t work when patched, and intellij, which found debug symbols but still refused to show me variables at the failed assert.

              DWARF is an extremely complicated file format

              To what extent gdb frontends actually have to deal with it though? It looks like they mostly just send print commands to gdb and get strings back. Maybe for understanding the structure of types?

              due to what I’ve seen of its internals

              I’m in more or less the same boat. I think there are a lot of design decisions in nix that make things much buggier than they needed to be. But stuff that works tends to stay working, whereas my most-of-a-decade experience with ubuntu was that stuff would just mysteriously break at random times and be impossible to rollback.

              I am tempted to try guix, since it seems to be much better designed, but that’s even more niche.

              My main interest is just finding something that works for this project and I did get a lot of interesting suggestions to test out.

              The title was off the cuff and wasn’t intended to be a big statement about anything. I might just change it to “Looking for debugger” or similar.

              1. 3

                Well it looks like you had problems getting backtraces in multiple debuggers. I consider that the most basic functionality: after a crash, get a backtrace with function names, then click to go to source location.

                What I read seems significantly more broken than I’ve experienced, although I won’t argue with the general point that debugger UIs can be flaky. I tried a Python web UI as well and didn’t have good luck with it. Eclipse had some weird problems respecting my breakpoints, but was generally usable and I did fix many bugs with it.

                I think another source of difficulty could be the GDB MI version (protocol version). I’m not versed in the details, but some debuggers use a newer protocol and some use and older one. That seems heavily distro and version dependent.

                Generally when I look at a distro, I look at the size of the package definitions. For every line I assume a pretty large probability of having a bug. Stuff like this is why “minimalist distros” are appealing (although I use Ubuntu for my desktop, since it’s tested by brute force in the happy paths at least).

                1. 3

                  A few months ago I went through a similar list of debuggers on Debian, and encountered very similar errors. Many debuggers refuse to work properly when they haven’t built the binary themselves, or they can’t find the C/C++ source. While I won’t call Debian a “minimalist distro”, the fact I also encountered doesn’t make these seem like the fault of Nix.

        2. 3

          These days, I usually use a combination of lldb and radare2 on HardenedBSD. Very occasionally, I use Ghidra.

          1. 3

            What else should I try?

            Netbeans. They have quite nice GDB support.

            (unfortunate you need Netbeans version 8 or install C/C++ plugins from 8 to current Apache Netbeans, because they have not included the C/C++ support yet)

            1. 3

              A better title would be, “the state of linux debugger frontends.” gdb itself is quite reliable[1], but it’s an enormously complex piece of software and interoperating with complex software is difficult.

              I think debuggers are a poor power-complexity trade-off for most debugging. They do have a place, but for most simple things, tools amplifying debug-by-print feel much more economic. Here’s the tool I debug with these days: https://github.com/akkartik/mu/blob/main/tools/browse_trace.readme.md It’s quite independent of my project and surprisingly versatile. All you have to do is emit prints in a specific format with a depth at the front.

              Here is a description of the big picture of debugging that this tool fits in: https://github.com/akkartik/mu/blob/main/subx_debugging.md

              All this is possible with an outlay of a couple of hundred lines of C (so fewer lines in a higher-level language). I think we programmers need to expand our horizons on the extent to which we can help ourselves and not need to rely on complex, specialist tools.

              [1] As long as I don’t try to watch variables. Or travel backwards in time.

              1. 2

                FWIW you might like uftrace for C/C++ code:


                It’s significantly less complex than GDB but also quite powerful. I used it to measure performance, but you can also use it for correctness, etc.

                This kind of tracing is a nice shell-ish paradigm that even Python doesn’t have!

              2. 3

                Putting on my Zig hat here, the thing that jumps out to me is this:

                assert(meta.deepEqual(expected.items, actual.items));

                If we had a better facility for testing deep equality, there would be no need for a debugger at all! We have this for string comparison. With a line like this:

                testing.assertStringsEqual(a, b);

                The output for one example values of a and b looks like:

                Test [1/1] test "example"... 
                ====== expected this output: =========
                ======== instead found this: =========
                First difference occurs on line 2:
                test failure

                I know the point of the blog post is trying to use this as an example to examine the state of linux debuggers, but this was my personal takeaway :)

                1. 2

                  This is great and I would love to have more of it. I feel like I’ve ended up implementing test diffing in every language I’ve used.

                  Does it always print the whole string? A few tests up I’m comparing strings, but they’re ~1gb each :)

                  I know the point of the blog post is trying to use this as an example to examine the state of linux debuggers,

                  The point was actually just to get people to recommend me a debugger that works. I just made a very poor choice of title.

                  1. 2

                    Just checked - it has no detection of long strings! That would be a nice contributor-friendly improvement to make :-)

                    1. 1

                      Touche :)

                  2. 1

                    I prefer the approach of using the same infrastructure you have for debug-printing values, then just using your normal string-diffing equality comparison.

                    The nice thing about this is you can build an expect-test workflow on top of it where failing tests produce a diff which you can use to update your tests to pass.

                  3. 2

                    I agree that the GDB TUI is terrible. I use GDB in Emacs and to get the stacktrace and print variables I type the commands in the GUD command window. It’s not optimal, but it works. I’ve never had the graphical interface work as advertised.

                    Having a decent GUI interface with good memory views is a real boon to debugging low-level problems. It’s too bad that it doesn’t seem to work.

                    1. 1

                      It’s a real pity.

                      For as long as I’ve known about the TUI it has been in bad shape, and yet the demand for a better debug UI is very high. I don’t understand why the former, given the latter.

                      1. 2

                        I’m sure like every other foundational open source project, GDB doesn’t have enough people working on it.

                        Also, the philosophy is to let people make UIs on top of it, which lessens the need for the core to cater to everyone’s needs. Almost every debugger in the blog post is built on top of GDB! (or maybe EVERY single one?) So you can choose the one you want.

                        In my case, I noticed that the proprietary one (CLion) has the best GUI, which isn’t a surprise.

                        1. 2

                          I wouldn’t be surprised if there weren’t enough people working on GDB or LLDB, and yet GCC and LLVM seem to have no such problems, even though I’d argue that improving the debuggers have much higher utility.

                          My guess is that compilers are more exciting and/or higher status work, and attract their own share of PhD students. I have the exact same problem with the Ocaml ecosystem, where the compiler sees plenty of work, but attempts to improve debugging have barely gone anywhere.

                      2. 1


                        He is under using GUD,

                        See documentation on gdb-many-windows command.

                        I have on one frame the gdb command line, the local vars, the source code, the stack frames, the i/o, the break points and threads.

                      3. 2

                        The console interface to gdb works, but it’s inefficient:

                        1. I can’t easily see the surrounding code
                        2. I have to manually request information rather than just glancing at the display

                        That’s why they called Visual Studio “visual”, isn’t it? GUI-style interaction offers you options to pick from, instead of requiring that you know how to ask for what you need.

                        1. 2

                          What else should I try?

                          Not exactly what you’re looking for, and certainly problematic for multithreaded code, I’ve had success using edb when working with my single threaded assembly programs.

                          1. 2

                            I have personally stopped using debuggers for crashes altogether. I rarely find them useful anymore. I compile my programs with asan (AddressSanitizer) and ubsan (UndefinedBehaviourSanitizer), which insert an enormous amount of checks into the binary directly. The overhead is not insignificant (2-7x), but with that knowledge it can not only detect corruptions that only cause a crash later, it can also pinpoint the exact type of error (use after free, buffer overflow, null reference, overflow, etc), summarize the layout of your data in memory (“read of size 4 at 8 bytes right of array x”) and track the lifecycle of objects and where they were allocated.

                            With memory bugs taken care of, the major class of problems left for me is then application bugs. For this I often use bpftrace, which lets you attach lightweight probes to collect data when a function is entered or exited without having to restart the binary. It is especially useful with lower level software, as you can use it to collect information from both kernel space and arbitrarily many different user space processes at the same time.

                            1. 2

                              I’ve been using ugdb and loving it.