On macOS you can have the best of both worlds by running sample thatprocess 2 in another shell. It’s not much more effort to type and it gives you a full profile, not just a single stack frame.
Sweet. But does it work with a scripting language, many of which print a callstack upon Ctrl-C? Does it work with -fomit-frame-pointer? How do you read the output - if there are many call stacks all spending time doing json parsing in the end with a big subtree of calls for that, what view will tell you as quickly as 2-3 Ctrl-Cs?
(Like I said in TFA, I implemented profiles myself, I like them and you definitely need them at times; Ctrl-C is just really quick & useful in a bunch of cases)
Well, 99% of the time for me, the thing being slow is either written in native code or else has a signal handler so it exits cleanly, so Ctrl-C doesn’t do anything but stop it.
(In Go programs there’s a different key, I think Ctrl-], to dump stacks.)
I want something similar for LLDB. My usual flow of debugging a C++ crash is:
lldb -o r ./program to boot up LLDB
Program crashes
bt to print backtrace
Cool, crash is obvious, time to patch the crash site
q to quit
y to confirm that yes I would like to kill the process please
This is sort of annoying. I much prefer the Rust flow of RUST_BACKTRACE=1 cargo run, and I wish there was a similar tool for C++ (external binary, because I don’t want to link platform-specific backtrace support into my binary.)
And not to mention if the process ends up running fine, I still have to run q in LLDB explicitly to exit the debugger. Annoying.
A backtrace ./program utility would be super nice to have.
I am not super familiar with LLDB, but this is pretty easy to do with gdb and I imagine it wouldn’t be harder for LLDB:
$ ./backtrace ./p
hello, world!
$ ./backtrace ./p q
#0 0x00007fffff066c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fffff06a028 in __GI_abort () at abort.c:89
#2 0x0000000000400597 in main (argc=2, argv=0x7ffffffee448) at p.c:7
$ cat backtrace
#!/bin/bash
thisfile="$(readlink -s -f "$0")"
thisdir="$(dirname "$thisfile")"
commands="$thisdir/stackonsig.gdb"
gdb --batch --quiet -x "$commands" --args "$@"
$ cat stackonsig.gdb
set logging file /dev/null
set logging redirect on
set logging on
set width 0
set height 0
set pagination no
catch signal SIGABRT SIGSEGV SIGFPE SIGBUS SIGILL SIGQUIT SIGSYS
commands
set logging off
bt
end
run
if on windows, you can use SetUnhandledExceptionFilter at the start of the program with a simple function that catches most of your exceptions and print a simple stack trace:
https://git.sr.ht/~lessa/foundation/tree/master/item/src/foundation/windows.c#L118
(this version even generates a minidump which you can later inspect using visual studio)
This assumes frame pointers, doesn’t it? (Not near a computer to check but it says it uses ELF for symbol names unless you stripped the binary; I assume this implies that it never looks at DWARF at all, either for source line numbers or for the info it would need to get the stack under -fomit-frame-pointer)
Then you will learn about MIPS “relocation relaxation” (used also in RISC-V AFAIK.) The compiler “assumes the worst” and generates code loading a function address into a register, and then jumping to the address stored in that register.
It isn’t linker relaxation. The maximum distance of branch instructions is limited.
Since MIPS64 does not use range extension thunks / veneers as AArch32/AArch64/PowerPC do, the only choice is to load the address to a register and perform an indirect jump.
Apparently gcc generates some DWARF data that gdb is slow to handle. The GNU linker fixes this data, so that gdb doesn’t end up handling it slowly. LLD refuses to emulate this behavior of the GNU linker, because it’s gcc’s fault to have produced that DWARF data in the first place. And gdb refuses to handle LLD’s output efficiently, because it’s LLD’s fault to not have handled gcc’s output the way the GNU linker does.
Right. I still hope someone will fix this in gdb.
The resolved value of the relocation referencing a symbol in a discard section is sometimes called a tombstone value.
The value can be customized using -z dead-reloc-in-nonalloc=. I haven’t checked whether gdb is happy with other tombstone values.
Pretty sure they call it relocation relaxation (eg LLD 10 release notes call it relaxing jr and jalr instructions), but, if not, what should changing these instructions marked with R_MIPS_JALR be called instead?
I sometimes do something similar in Emacs: M-x toggle-debug-on-quit and then interrupt the long running thing with C-g. Or sending Emacs a SIGUSR2 is another way, that one doesn’t need any setup.
But Emacs has an easy to use built-in profiler too: M-x profile-start and profile-report.
http://poormansprofiler.org/
On macOS you can have the best of both worlds by running
sample thatprocess 2in another shell. It’s not much more effort to type and it gives you a full profile, not just a single stack frame.Sweet. But does it work with a scripting language, many of which print a callstack upon Ctrl-C? Does it work with -fomit-frame-pointer? How do you read the output - if there are many call stacks all spending time doing json parsing in the end with a big subtree of calls for that, what view will tell you as quickly as 2-3 Ctrl-Cs?
(Like I said in TFA, I implemented profiles myself, I like them and you definitely need them at times; Ctrl-C is just really quick & useful in a bunch of cases)
Well, 99% of the time for me, the thing being slow is either written in native code or else has a signal handler so it exits cleanly, so Ctrl-C doesn’t do anything but stop it.
(In Go programs there’s a different key, I think Ctrl-], to dump stacks.)
Is there some utility that can dump a stack trace of a process, and do nothing more?
99% of my usages of gdb is just “give me backtrace”, and I wish I could ask for that directly, without going through the rest of gdb’s interface.
On illumos systems, pstack(1) does this – either for a live process by PID, or for a core file.
There are a few I am aware of (that aren’t just scripts wrapping gdb):
I want something similar for LLDB. My usual flow of debugging a C++ crash is:
lldb -o r ./programto boot up LLDBbtto print backtraceqto quityto confirm that yes I would like to kill the process pleaseThis is sort of annoying. I much prefer the Rust flow of
RUST_BACKTRACE=1 cargo run, and I wish there was a similar tool for C++ (external binary, because I don’t want to link platform-specific backtrace support into my binary.)And not to mention if the process ends up running fine, I still have to run
qin LLDB explicitly to exit the debugger. Annoying.A
backtrace ./programutility would be super nice to have.I am not super familiar with LLDB, but this is pretty easy to do with gdb and I imagine it wouldn’t be harder for LLDB:
if on windows, you can use
SetUnhandledExceptionFilterat the start of the program with a simple function that catches most of your exceptions and print a simple stack trace: https://git.sr.ht/~lessa/foundation/tree/master/item/src/foundation/windows.c#L118 (this version even generates a minidump which you can later inspect using visual studio)gstack does exactly this.
This assumes frame pointers, doesn’t it? (Not near a computer to check but it says it uses ELF for symbol names unless you stripped the binary; I assume this implies that it never looks at DWARF at all, either for source line numbers or for the info it would need to get the stack under -fomit-frame-pointer)
IME gstack is a script over gdb, that disables pagination and just runs
thread apply all btIt isn’t linker relaxation. The maximum distance of branch instructions is limited. Since MIPS64 does not use range extension thunks / veneers as AArch32/AArch64/PowerPC do, the only choice is to load the address to a register and perform an indirect jump.
MIPS32 has some tricks https://maskray.me/blog/2023-09-04-toolchain-notes-on-mips#mno-shared-for-o32n32-non-pic
Right. I still hope someone will fix this in gdb. The resolved value of the relocation referencing a symbol in a discard section is sometimes called a tombstone value. The value can be customized using
-z dead-reloc-in-nonalloc=. I haven’t checked whether gdb is happy with other tombstone values.Pretty sure they call it relocation relaxation (eg LLD 10 release notes call it relaxing jr and jalr instructions), but, if not, what should changing these instructions marked with R_MIPS_JALR be called instead?
It seems that people call it “relax … JALR” or “JALR relaxation”.
x86 invented -Wa,-mrelax-relocations=yes
OK. I think “relocation relaxation” is fine.
https://en.wikipedia.org/wiki/Profiling_(computer_programming)#Statistical_profilers
I sometimes do something similar in Emacs:
M-x toggle-debug-on-quitand then interrupt the long running thing withC-g. Or sending Emacs aSIGUSR2is another way, that one doesn’t need any setup.But Emacs has an easy to use built-in profiler too:
M-x profile-startandprofile-report.