One more thing I have experienced on the Windows platform: when you UPX pack your binary, the various virus scanners are much more likely to flag you as malware, mostly because most malware is also UPX packed.
It’s true that runtime packers always remove the possibility of sharing the memory with other instances, but so does relocations and different versions of app runtime libraries (e.g. two apps with different CEF frameworks won’t share anything).
So I’m not sure that it’s a valid argument to say that disqualifies the use of UPX, because for sure it doesn’t disqualify relocations nor different runtime versions from use (although it’s also true that everyone agrees that we should minimize relocations for this sole reason, if it’s feasable).
Sometimes the difference between packed vs non-packed is really large, depending on how the executable was built, and sometimes we expect only one instance of the executable to be launched anyway. So it’s a bad idea to UPX the i.e. system-wide java.exe, but what would be wrong with UPX’ing i.e. a game executable? Only one game will need to be executed at a time, and it’s not that important if it will load in 10 seconds or 13 seconds. On the other hand, the executable is stored on the hard disk for the whole time, so what is the point of allocating that much space for it, if it’s used a few times a week at most?
The malware argument is not wrong, but in reality AV engines contain UPX decompression code, so those engines see right through UPX (as well as lots of other runtime packers and non-password-protected archive formats). There’s no need to flag an executable as malware only because it uses UPX. The problem with AV vendors and packed executables is when a strong compressor is used, like VMProtect or PELock. Strong executable protections are something different than simple packers like UPX, so those might trigger heuristic AV detections (because often times AV can’t see what’s inside the executable file).
Well, kind of. Most modern executable formats are designed to avoid this, by placing things that need dynamic relocations in different pages. For example, in PIC libraries and PIE programs, all local symbols are referenced using PC-relative displacements so they don’t need dynamic relocations and everything else is referenced via a global offset table, which is on a separate page. You can read-only map the majority of the binary into multiple processes and just need private copies of the GOT pages and any pages for the data section. In addition to sharing, this also means that you can avoid reading all of the binary from disk or you can evict any of the code pages when memory pressure is high because they can always be reloaded from disk lazily later.
and different versions of app runtime libraries
That’s true but the key case here is sharing between multiple instances of the same program. Whether that matters depends a lot on the program in question. For example, a GUI app may have a single instance running at any given time, whereas something run in build systems, cron jobs, or in response to network events may have tens or more of instances running simultaneously.
Only one game will need to be executed at a time, and it’s not that important if it will load in 10 seconds or 13 seconds
I know game developers who would strongly disagree and would kill for an easy 10% win in load times. For games, the binary is likely to be large and being able to page the hot paths in lazily is probably more of a win.
That’s true but the key case here is sharing between multiple instances of the same program.
I understand what was the key point, but I was trying to point out that if an executable is expected to run only once at a time for most cases, then optimization for the “multiple concurrent instances” use case is often pointless, considering that it’s not free, and costs disk space. My point is that optimizing an executable for concurrent use that runs only once at a time, but costs a lot of disk space, is similarily wasteful than runtime packing an executable that is expected to be executed a lot of times concurrently (although the second case has a potential to waste more memory than the first case I guess).
I know game developers who would strongly disagree and would kill for an easy 10% win in load times.
I seriously doubt that UPX can ever do a 10% difference between application load times; I haven’t measured it, but I would guess it would be more similar to 0.1%. The amount of data needed to load a game often dwarfs the size of the executable without compression, plus the majority of game developers don’t operate on optimizations on this level. The game data is compressed anyway, so if developers would really value easy speedup, then they would distribute games with some assets uncompressed, and they don’t do that.
My point is that optimizing an executable that runs only once at a time but costs a lot of disk space is similarily wasteful
Modern file systems can do transparent compression. This means that the pages are still shareable in the buffer cache, so you get the same advantages in terms of sharing (and often faster loads, since zstd decompression on even a fairly slow CPU is faster than a fast SSD). I don’t see why I’d want an application to roll a bespoke solution that integrates badly with other OS features, unless the compression that UPX gives is much better than my filesystem can do (from the numbers quoted in the article, it isn’t).
Modern file systems can do transparent compression.
Except cases where the filesystem doesn’t support it, but it’s still widely used (fat32), or an application is distributed to different platforms, each with their own set of filesystems (ntfs? ext? btrfs? hfs+? apfs?), and an installer is suddenly needed to apply the compression flag on some specific files.
How many use case are meet all of the following requirements:
So constrained in space that a small saving in binary size is desirable
Not using a filesystem configured with compression by default for binaries
Have enough spare RAM that decompressing an entire binary in memory in addition to its space in the buffer cache is not an issue
My guess would be approximately none. Even on servers with huge disks, I turn on disk encryption by default because it makes things faster and zstd has early-abort for thing that are too high entropy (things that are already compressed or encrypted) and so doesn’t cost me anything noticeable even on things like MPEG-4 videos.
I’m not sure why an installer would do it, since it’s something that’s easy for a user to configure if they care. For most *NIX systems, I’d expect it to be the default for wherever you install packages. I’m not sure if Apple defaults to something sensible for binaries but it’s the kind of thing that they could easily control with a system-wide policy.
As you said elsewhere, for most things where I care about the space consumption of a program, that space doesn’t come from the executable, it comes from other assets.
I.e. portable software stored on an USB stick can fit this criteria.
Also filesystem compression is not a magic solution to the problem, as it also has its costs, like increased file fragmentation, each filesystem has its own rules of enabling it (that’s why I’ve mentioned that an installer is needed if the vendor wants it applied to their software by default), and it requires the user knowledgeable enough to be able to enable it. We can transfer the authority of managing compressed files to the system owner, but that won’t solve the problem if the vendor wants it automatically. Which I guess is not the focus on *NIX systems at all (as it’s expected that software installation is managed by the OS).
I.e. portable software stored on an USB stick can fit this criteria.
That might count, but the cheapest USB flash stick these days is 8 GiB and 128 GiB is pretty common. That said, they tend to be slow, so compression can be a win, and they often use FAT or exFAT where per-application compression is the only option. I can see UPX being a win here but it’s far from the default and (unless I’m shipping a portable app bundle) not the case I’d expect to optimise for in software distribution.
Also filesystem compression is not a magic solution to the problem, as it also has its costs, like increased file fragmentation
I’d love to see a source for that. It certainly doesn’t match my experience. The only case I can see if would happen is if it’s not a CoW filesystem and the file is frequently modified, which doesn’t apply for files that are part of an application distribution (whether they’re the program itself or other assets).
each filesystem has its own rules of enabling it (that’s why I’ve mentioned that an installer is needed if the vendor wants it applied to their software by default), and it requires the user knowledgeable enough to be able to enable it.
True enough. I mostly come to this from a ZFS perspective, where it’s just on by default and I never think about it.
We can transfer the authority of managing compressed files to the system owner, but that won’t solve the problem if the vendor wants it automatically. Which I guess is not the focus on *NIX systems at all (as it’s expected that software installation is managed by the OS).
We’re already transferring the authority for installing software to the system owner. They can make a choice to trade off memory and disk. In most systems I’ve used in the past decade, memory is a more scarce resource than disk, so optimising for disk space at the expense of memory seems like exactly the wrong default, though I can imagine situations where you’d want to give users the tools to make the other choice.
One more thing I have experienced on the Windows platform: when you UPX pack your binary, the various virus scanners are much more likely to flag you as malware, mostly because most malware is also UPX packed.
Thanks for mentioning this! I’ll add this to the conclusion as an additional tidbit people should know.
It’s true that runtime packers always remove the possibility of sharing the memory with other instances, but so does relocations and different versions of app runtime libraries (e.g. two apps with different CEF frameworks won’t share anything).
So I’m not sure that it’s a valid argument to say that disqualifies the use of UPX, because for sure it doesn’t disqualify relocations nor different runtime versions from use (although it’s also true that everyone agrees that we should minimize relocations for this sole reason, if it’s feasable).
Sometimes the difference between packed vs non-packed is really large, depending on how the executable was built, and sometimes we expect only one instance of the executable to be launched anyway. So it’s a bad idea to UPX the i.e. system-wide java.exe, but what would be wrong with UPX’ing i.e. a game executable? Only one game will need to be executed at a time, and it’s not that important if it will load in 10 seconds or 13 seconds. On the other hand, the executable is stored on the hard disk for the whole time, so what is the point of allocating that much space for it, if it’s used a few times a week at most?
The malware argument is not wrong, but in reality AV engines contain UPX decompression code, so those engines see right through UPX (as well as lots of other runtime packers and non-password-protected archive formats). There’s no need to flag an executable as malware only because it uses UPX. The problem with AV vendors and packed executables is when a strong compressor is used, like VMProtect or PELock. Strong executable protections are something different than simple packers like UPX, so those might trigger heuristic AV detections (because often times AV can’t see what’s inside the executable file).
Well, kind of. Most modern executable formats are designed to avoid this, by placing things that need dynamic relocations in different pages. For example, in PIC libraries and PIE programs, all local symbols are referenced using PC-relative displacements so they don’t need dynamic relocations and everything else is referenced via a global offset table, which is on a separate page. You can read-only map the majority of the binary into multiple processes and just need private copies of the GOT pages and any pages for the data section. In addition to sharing, this also means that you can avoid reading all of the binary from disk or you can evict any of the code pages when memory pressure is high because they can always be reloaded from disk lazily later.
That’s true but the key case here is sharing between multiple instances of the same program. Whether that matters depends a lot on the program in question. For example, a GUI app may have a single instance running at any given time, whereas something run in build systems, cron jobs, or in response to network events may have tens or more of instances running simultaneously.
I know game developers who would strongly disagree and would kill for an easy 10% win in load times. For games, the binary is likely to be large and being able to page the hot paths in lazily is probably more of a win.
I understand what was the key point, but I was trying to point out that if an executable is expected to run only once at a time for most cases, then optimization for the “multiple concurrent instances” use case is often pointless, considering that it’s not free, and costs disk space. My point is that optimizing an executable for concurrent use that runs only once at a time, but costs a lot of disk space, is similarily wasteful than runtime packing an executable that is expected to be executed a lot of times concurrently (although the second case has a potential to waste more memory than the first case I guess).
I seriously doubt that UPX can ever do a 10% difference between application load times; I haven’t measured it, but I would guess it would be more similar to 0.1%. The amount of data needed to load a game often dwarfs the size of the executable without compression, plus the majority of game developers don’t operate on optimizations on this level. The game data is compressed anyway, so if developers would really value easy speedup, then they would distribute games with some assets uncompressed, and they don’t do that.
Modern file systems can do transparent compression. This means that the pages are still shareable in the buffer cache, so you get the same advantages in terms of sharing (and often faster loads, since zstd decompression on even a fairly slow CPU is faster than a fast SSD). I don’t see why I’d want an application to roll a bespoke solution that integrates badly with other OS features, unless the compression that UPX gives is much better than my filesystem can do (from the numbers quoted in the article, it isn’t).
Except cases where the filesystem doesn’t support it, but it’s still widely used (fat32), or an application is distributed to different platforms, each with their own set of filesystems (ntfs? ext? btrfs? hfs+? apfs?), and an installer is suddenly needed to apply the compression flag on some specific files.
How many use case are meet all of the following requirements:
My guess would be approximately none. Even on servers with huge disks, I turn on disk encryption by default because it makes things faster and zstd has early-abort for thing that are too high entropy (things that are already compressed or encrypted) and so doesn’t cost me anything noticeable even on things like MPEG-4 videos.
I’m not sure why an installer would do it, since it’s something that’s easy for a user to configure if they care. For most *NIX systems, I’d expect it to be the default for wherever you install packages. I’m not sure if Apple defaults to something sensible for binaries but it’s the kind of thing that they could easily control with a system-wide policy.
As you said elsewhere, for most things where I care about the space consumption of a program, that space doesn’t come from the executable, it comes from other assets.
I.e. portable software stored on an USB stick can fit this criteria.
Also filesystem compression is not a magic solution to the problem, as it also has its costs, like increased file fragmentation, each filesystem has its own rules of enabling it (that’s why I’ve mentioned that an installer is needed if the vendor wants it applied to their software by default), and it requires the user knowledgeable enough to be able to enable it. We can transfer the authority of managing compressed files to the system owner, but that won’t solve the problem if the vendor wants it automatically. Which I guess is not the focus on *NIX systems at all (as it’s expected that software installation is managed by the OS).
That might count, but the cheapest USB flash stick these days is 8 GiB and 128 GiB is pretty common. That said, they tend to be slow, so compression can be a win, and they often use FAT or exFAT where per-application compression is the only option. I can see UPX being a win here but it’s far from the default and (unless I’m shipping a portable app bundle) not the case I’d expect to optimise for in software distribution.
I’d love to see a source for that. It certainly doesn’t match my experience. The only case I can see if would happen is if it’s not a CoW filesystem and the file is frequently modified, which doesn’t apply for files that are part of an application distribution (whether they’re the program itself or other assets).
True enough. I mostly come to this from a ZFS perspective, where it’s just on by default and I never think about it.
We’re already transferring the authority for installing software to the system owner. They can make a choice to trade off memory and disk. In most systems I’ve used in the past decade, memory is a more scarce resource than disk, so optimising for disk space at the expense of memory seems like exactly the wrong default, though I can imagine situations where you’d want to give users the tools to make the other choice.