That it is not common practice to ship debug symbols with production binaries is, in my opinion, quite sad and insensible. One argument is obfuscation. That never stopped any reverse engineer. Another is binary sizes. My bin directories are a few piddling gigabytes; they can bear to grow a bit in exchange for a better user experience.
That it is not common practice to ship debug symbols with production binaries is, in my opinion, quite sad
Agreed.
Another is binary sizes […] they can bear to grow a bit in exchange for a better user experience.
Unfortunately this doesn’t apply to everyone. At $WORK, stripping all of the binaries contained in the package we ship to customers results in saving almost a gigabyte. Our customers sometimes try to download our product from behind awful corporate firewalls/antivirii that will randomly interrupt downloads, so shipping the tiniest package we can is very important. A couple of my remote colleagues are also on terribly slow networks where that extra gigabyte would be a real pain when attempting to bisect the whole package itself.
Another example was Debian’s firefox-dbgsym package, which IIRC was also a multi-gigabyte monstrosity. Clearly not something that should be shipped to everyone.
Was coming here to say basically this. Storage is cheap, but transfer can be expensive, and time can be very expensive. We made our life much better at $WORK last year when we split our software into data and code and turned a 2.5 GB download into a 2 GB download that seldom changes and a 0.5 GB download that changes all the time.
You may want to experiment with stripping debug sections and debug symbols but keeping other symbol names. In my experience, that results in a binary that’s almost as small as a stripped binary, but you get function names rather than byte offsets in stack traces. It’s not as nice to work with in a debugger as a binary with full debug info, but it’s much nicer than a completely stripped binary.
With the strip tool, use --strip-debug to strip debug symbols and debug sections but keep other symbols.
Another is binary sizes. My bin directories are a few piddling gigabytes; they can bear to grow a bit in exchange for a better user experience.
In what way is the user experience better? At best, users get better stack traces when things crash, but if you’re recording those stack traces for debugging and know the package version then you can symbolise them later. Unless you’re a developer, you are unlikely to have any use for debug symbols. If you are a developer then you can
Debug symbols are usually 2-10 times larger than the whole of the rest of the binaries. This doesn’t just impact disk size (though it has a huge impact on container start times) it also impacts downloads. You may have FTTP, but for a lot of users, downloading an extra 10-100 MiBs of debug symbols for each package when they update is a big impact.
I just had a look at a FreeBSD system (FreeBSD ships with separate debug info). The /usr/lib directory on this system contains 9.9 MiBs of shared libraries, 27 MiBs of debug info. That’s three times as much disk space and network bandwidth consumed to distribute the debug symbols as the libraries. I’m very happy that that’s an optional install so I can install it on dev systems but not production ones.
I don’t really care about debug symbols, and I wouldn’t want to need +700% of additional disk space used for something I will never use, and without which everything works perfectly fine:
-rwxrwxr-x 1 x x 25M lis 24 13:41 app.without-symbols*
-rwxrwxr-x 1 x x 184M lis 24 13:41 app.with-symbols*
In rare cases where something crashes, I can download debug symbols later (if they’re available), in case I need them.
One argument is obfuscation. That never stopped any reverse engineer.
This is a protection based on cost to reward ratio. It seeds out less determined reverse engineers, but doesn’t affect more determined ones. So it does stop some reverse engineers, just not everyone. It increases the cost of reversing to a point where some of reversers bail out, because cost is higher than the reward.
How often do users need the debug symbols? My educated guess is “almost never”. We complain about bloat all the time, why bloat things further with unnecessary things, then?
Making debug symbols easily and readily available benefits both the end-user (who doesn’t need, nor want them), and the developer (who does, and can install them trivially) is, in my experience, a much nicer practice, with almost the same benefits, at the fraction of the cost.
I don’t. And I don’t find that it is ‘trivial’ to install debug symbols (case in point, else-thread: ‘it seems that packed is currently unusable on Linux’). Something l have noticed other people say they value is open-source contributions; wouldn’t it be a great help if it were as easy as possible to debug, modify, and contribute to open source software?
If it were a simple matter of changing a setting in my system package manager’s configuration file, to get debug symbols, source code, and dirty build artifacts—with the default being as it currently is—that would be one thing. But this is not a feature anyone implements, as far as I know.
And I don’t find that it is ‘trivial’ to install debug symbols
Then that should be fixed, rather than shipping debug symbols by default, which would be useless 99% of the time.
If it were a simple matter of changing a setting in my system package manager’s configuration file, to get debug symbols, source code, and dirty build artifacts—with the default being as it currently is—that would be one thing. But this is not a feature anyone implements, as far as I know.
Debian has a debuginfod server, which gives you trivial access to debug symbols: all you have to do is either have elfutils installed, or export a single environment variable (more information). While making source code similarly easy to access isn’t a thing yet, that’s also simple to automate, and it’s usually an apt-get source away.
It’s a bit more work, yes, but you won’t get gigabytes of near useless stuff by default, and that’s a big win in my book.
Don’t forget about apt-get install somepackage-dbg! (which might require enabling the debug symbols repo? can’t recall.)
Debugging software from Debian repos is a pretty painless experienc. I’ve been much more willing to debug minor bugs in random software since switching to it.
I agree with the sentiment. It would probably make more sense to have the common practice be that package maintainers release software exactly as is described in the article: the stripped binary, and an addon containing all of the debug symbols. Best of both worlds!
This is standard practice on macOS (i believe it’s literally the default behaviour out of the standard toolchains), I’m surprised it isn’t happening by default on linux as it is superior for most use cases.
I think macOS does it a slightly different way. As I recall, their toolchains use a different format for the debug info that is linked by a custom tool. Symbols referenced by debug info are marked to not be discarded, but you can be very lazy about linking the debug info. Xcode, I think, starts running linked binaries immediately for testing and links the separate debug info files in the background.
On ELF platforms, the same thing is built using existing relocation types and file formats. Apple was doing it for about 10 years before ELF toolchains got similar features (Windows has also done it for a long time).
I wonder if you could do this with Go or Rust. It seems to be super useful to store the debug info else where and use it to help troubleshoot production binaries.
While I’m hoping situation will improve in not too distant future, after reading lots of comments and discussion it seems that packed is currently unusable on Linux.
That it is not common practice to ship debug symbols with production binaries is, in my opinion, quite sad and insensible. One argument is obfuscation. That never stopped any reverse engineer. Another is binary sizes. My bin directories are a few piddling gigabytes; they can bear to grow a bit in exchange for a better user experience.
Agreed.
Unfortunately this doesn’t apply to everyone. At $WORK, stripping all of the binaries contained in the package we ship to customers results in saving almost a gigabyte. Our customers sometimes try to download our product from behind awful corporate firewalls/antivirii that will randomly interrupt downloads, so shipping the tiniest package we can is very important. A couple of my remote colleagues are also on terribly slow networks where that extra gigabyte would be a real pain when attempting to bisect the whole package itself.
Another example was Debian’s firefox-dbgsym package, which IIRC was also a multi-gigabyte monstrosity. Clearly not something that should be shipped to everyone.
Was coming here to say basically this. Storage is cheap, but transfer can be expensive, and time can be very expensive. We made our life much better at $WORK last year when we split our software into data and code and turned a 2.5 GB download into a 2 GB download that seldom changes and a 0.5 GB download that changes all the time.
You may want to experiment with stripping debug sections and debug symbols but keeping other symbol names. In my experience, that results in a binary that’s almost as small as a stripped binary, but you get function names rather than byte offsets in stack traces. It’s not as nice to work with in a debugger as a binary with full debug info, but it’s much nicer than a completely stripped binary.
With the
striptool, use--strip-debugto strip debug symbols and debug sections but keep other symbols.In what way is the user experience better? At best, users get better stack traces when things crash, but if you’re recording those stack traces for debugging and know the package version then you can symbolise them later. Unless you’re a developer, you are unlikely to have any use for debug symbols. If you are a developer then you can
Debug symbols are usually 2-10 times larger than the whole of the rest of the binaries. This doesn’t just impact disk size (though it has a huge impact on container start times) it also impacts downloads. You may have FTTP, but for a lot of users, downloading an extra 10-100 MiBs of debug symbols for each package when they update is a big impact.
I just had a look at a FreeBSD system (FreeBSD ships with separate debug info). The /usr/lib directory on this system contains 9.9 MiBs of shared libraries, 27 MiBs of debug info. That’s three times as much disk space and network bandwidth consumed to distribute the debug symbols as the libraries. I’m very happy that that’s an optional install so I can install it on dev systems but not production ones.
I don’t really care about debug symbols, and I wouldn’t want to need +700% of additional disk space used for something I will never use, and without which everything works perfectly fine:
In rare cases where something crashes, I can download debug symbols later (if they’re available), in case I need them.
This is a protection based on cost to reward ratio. It seeds out less determined reverse engineers, but doesn’t affect more determined ones. So it does stop some reverse engineers, just not everyone. It increases the cost of reversing to a point where some of reversers bail out, because cost is higher than the reward.
How often do users need the debug symbols? My educated guess is “almost never”. We complain about bloat all the time, why bloat things further with unnecessary things, then?
Making debug symbols easily and readily available benefits both the end-user (who doesn’t need, nor want them), and the developer (who does, and can install them trivially) is, in my experience, a much nicer practice, with almost the same benefits, at the fraction of the cost.
I don’t. And I don’t find that it is ‘trivial’ to install debug symbols (case in point, else-thread: ‘it seems that
packedis currently unusable on Linux’). Something l have noticed other people say they value is open-source contributions; wouldn’t it be a great help if it were as easy as possible to debug, modify, and contribute to open source software?If it were a simple matter of changing a setting in my system package manager’s configuration file, to get debug symbols, source code, and dirty build artifacts—with the default being as it currently is—that would be one thing. But this is not a feature anyone implements, as far as I know.
Then that should be fixed, rather than shipping debug symbols by default, which would be useless 99% of the time.
Debian has a debuginfod server, which gives you trivial access to debug symbols: all you have to do is either have
elfutilsinstalled, or export a single environment variable (more information). While making source code similarly easy to access isn’t a thing yet, that’s also simple to automate, and it’s usually anapt-get sourceaway.It’s a bit more work, yes, but you won’t get gigabytes of near useless stuff by default, and that’s a big win in my book.
Don’t forget about
apt-get install somepackage-dbg! (which might require enabling the debug symbols repo? can’t recall.)Debugging software from Debian repos is a pretty painless experienc. I’ve been much more willing to debug minor bugs in random software since switching to it.
I agree with the sentiment. It would probably make more sense to have the common practice be that package maintainers release software exactly as is described in the article: the stripped binary, and an addon containing all of the debug symbols. Best of both worlds!
This is standard practice on macOS (i believe it’s literally the default behaviour out of the standard toolchains), I’m surprised it isn’t happening by default on linux as it is superior for most use cases.
I think macOS does it a slightly different way. As I recall, their toolchains use a different format for the debug info that is linked by a custom tool. Symbols referenced by debug info are marked to not be discarded, but you can be very lazy about linking the debug info. Xcode, I think, starts running linked binaries immediately for testing and links the separate debug info files in the background.
On ELF platforms, the same thing is built using existing relocation types and file formats. Apple was doing it for about 10 years before ELF toolchains got similar features (Windows has also done it for a long time).
I wonder if you could do this with Go or Rust. It seems to be super useful to store the debug info else where and use it to help troubleshoot production binaries.
This seems to do something along the same lines for Rust. (See the docs for more info)
While I’m hoping situation will improve in not too distant future, after reading lots of comments and discussion it seems that
packedis currently unusable on Linux.https://github.com/MaterializeInc/materialize/pull/23410#issuecomment-1824758123
https://github.com/rust-lang/rust/issues/105991