You missed a method! I’ve been meaning to write about a version of this forever but the basic idea is that the total size of the executing binary is stored within the ELF data, and ELF files do not give a shit about what comes after them.
Thus, you can simply do:
$ cat foo.txt >> program.bin
this shifts the difficulty to:
getting the ELF header of the currently executing program
I will cheerfully admit that I hadn’t thought of that :) Though it would make it hard (without adding some sort of header) to add more than one binary blob. There’s also a secondary problem, which is that the newly added blob will be outside the ALLOC segment, so one would probably have to mmap that part of the executable in manually. It’s definitely doable, albeit somewhat laborious. Still, a neat trick!
Note that, depending on the sandboxing technology that you use, the application may not be able to access its own binary file. This may also interfere with code signing flows.
However, perhaps surprisingly, xxd is part of Vim
I was very surprised by this. I went to refute this claim but it looks as if that’s where my copy comes from. Checking the history, it looks as if the original author of vim made some small tweaks to xxd and bundled it. I’m fairly sure that it was a separate package when I first encountered it, but that was more than 20 years ago now.
Mind you, I’ve never used a *NIX system that hasn’t had it installed - xxd is one of my go-to tools for a bunch of tasks and I’ve never needed to know which package it came from before.
The performance issue that you encounter comes from the fact that clang generates an AST node for every single one of the initialiser elements. This is a lot faster if you insert it as a C string ("\x23\x42" instead of 0x23, 0x42), though you need to be careful of the extra null byte that the compiler will insert at the end of a C-string literal. I wrote a very small (about 4 lines of code) C++ tool for a project to generate C/C++ headers with string literals from arbitrary binary files a few years ago. The compiler happily passes them through and you can use things like section attributes to put them in specific locations in the output.
This is a lot faster if you insert it as a C string (”\x23\x42” instead of 0x23, 0x42), though you need to be careful of the extra null byte that the compiler will insert at the end of a C-string literal.
Ah, interesting! I must admit that I hadn’t thought to try this.
I wrote a C tool, that converts binary data into C strings https://github.com/mulle-nat/mulle-c-string-escape, which I use for just this purpose. The created .inc file can then be easily include like so:
We used this technique in the nineties. It was fine except that the compiler would use much memory per byte, so sometimes the compile would run out of RAM on even large boxes (of that time). Of course computers have more memory now, I’ve 64GB on my desk now and had 64MB then, but perhaps the included blobs are bigger too.
With clang 14.0.6 and mulle-c-string-escape I can get a turnaround of 3s on my system for a file of 100 MB (like the author used). The compiler takes 2.1s and 912 MB RAM and mulle-c-string-escape uses 0.9s and 1.4MB RAM. (measured with /usr/bin/time -v). For comparison: the author sees 15s for hexdump and 90s for clang. My machine takes 12s for the 100 MB hexdump.
So I would estimate it would be 4s vs 105s.
I was very surprised by this. I went to refute this claim but it looks as if that’s where my copy comes from. Checking the history, it looks as if the original author of vim made some small tweaks to xxd and bundled it. I’m fairly sure that it was a separate package when I first encountered it, but that was more than 20 years ago now.
“This version is based on V1.10 27oct98 by Juergen Weigert.
Between 2000 and 2011 it accumulated patches by the Vim team and
others. In 2013 it was pulled out of the Vim tree, had a couple
of features added and was pronounced version 1.11.”
– https://github.com/ConorOG/xxd
Lack of access is a real problem a case I’ve come across.
Some postscript drivers generate code that reads the, uhm, the part of the postscript file that’s after the postscript program. The postscript parser parses the first part of the foo.ps file until the syntactical EOF, and the parsed postscript code reads and uses the rest.
This collides unprettily with boxes that accept input from many client systems, count pages, assign costs to the right team/department, and send postscript onwards to the right printer.
Though it would make it hard (without adding some sort of header) to add more than one binary blob.
the header format already exists. try concatenating a zipfile to the end of an executable. unzip will still manage to extract, and the file will execute fine. self-extracting archives work this way.
This works on Windows too! Taking advantage of it is part of the reason why the ZIP file format is designed with the dictionary of filenames and offsets at the end of the file instead of the beginning. :)
(You can skip all the work of parsing the ELF header if you do something like putting the length of the file as the last 4 bytes of the file.)
I’ve always liked this method and the fact that it works on windows executables is great too. I was wondering if there’s a similar trick that will work with shared objects – is there a straightforward substitute for argv[0]?
I am usually using a secondary tool (similar to xxd) but it is checked into the same source control and then invoked first with CMake, then within each target that needs it, we utilize the CMake command add_custom_command to invoke it and generate the necessary files (.h/.cpp). This is portable enough to Unix, macOS, and Windows. The files are typically small and not more than a couple of kilobytes, usually for small icons or maybe a small image in the “About” dialog box of a GUI application. We may eventually switch to #embed but it’s not really a “pain” point at the moment.
I guess if you’re asking for the most portable way, and are not tied to C, Go supports this out of the box on every platform
Same in rust where you can use include_bytes!.
This feels too simple. Including binary data used to be a skill.
So was flint-knapping, blacksmithing and professional abacus use. ¯_(ツ)_/¯
This was just on here a few days ago 🤷🏻♀️
You missed a method! I’ve been meaning to write about a version of this forever but the basic idea is that the total size of the executing binary is stored within the ELF data, and ELF files do not give a shit about what comes after them.
Thus, you can simply do:
this shifts the difficulty to:
I will cheerfully admit that I hadn’t thought of that :) Though it would make it hard (without adding some sort of header) to add more than one binary blob. There’s also a secondary problem, which is that the newly added blob will be outside the ALLOC segment, so one would probably have to
mmapthat part of the executable in manually. It’s definitely doable, albeit somewhat laborious. Still, a neat trick!Note that, depending on the sandboxing technology that you use, the application may not be able to access its own binary file. This may also interfere with code signing flows.
I was very surprised by this. I went to refute this claim but it looks as if that’s where my copy comes from. Checking the history, it looks as if the original author of vim made some small tweaks to xxd and bundled it. I’m fairly sure that it was a separate package when I first encountered it, but that was more than 20 years ago now.
Mind you, I’ve never used a *NIX system that hasn’t had it installed - xxd is one of my go-to tools for a bunch of tasks and I’ve never needed to know which package it came from before.
The performance issue that you encounter comes from the fact that clang generates an AST node for every single one of the initialiser elements. This is a lot faster if you insert it as a C string (
"\x23\x42"instead of0x23, 0x42), though you need to be careful of the extra null byte that the compiler will insert at the end of a C-string literal. I wrote a very small (about 4 lines of code) C++ tool for a project to generate C/C++ headers with string literals from arbitrary binary files a few years ago. The compiler happily passes them through and you can use things like section attributes to put them in specific locations in the output.Include the size indicator, cf
char c[5] = "abcde", and it goes away. As I recall, this does not work in c++ for some reason.Ah, interesting! I must admit that I hadn’t thought to try this.
I wrote a C tool, that converts binary data into C strings https://github.com/mulle-nat/mulle-c-string-escape, which I use for just this purpose. The created .inc file can then be easily include like so:
This is what
#embedis defined to be as-if, by the way.We used this technique in the nineties. It was fine except that the compiler would use much memory per byte, so sometimes the compile would run out of RAM on even large boxes (of that time). Of course computers have more memory now, I’ve 64GB on my desk now and had 64MB then, but perhaps the included blobs are bigger too.
With clang 14.0.6 and mulle-c-string-escape I can get a turnaround of 3s on my system for a file of 100 MB (like the author used). The compiler takes 2.1s and 912 MB RAM and mulle-c-string-escape uses 0.9s and 1.4MB RAM. (measured with /usr/bin/time -v). For comparison: the author sees 15s for hexdump and 90s for clang. My machine takes 12s for the 100 MB hexdump. So I would estimate it would be 4s vs 105s.
The repository I found when writing the “Related work” section of my Cedro pre-processor (you know about it), states the following:
I too didn’t know which package it came from.
See Rui Ueyama’s suggestion for avoiding this. It’s quite clever and, I think, portable (but I might be wrong!).
Lack of access is a real problem a case I’ve come across.
Some postscript drivers generate code that reads the, uhm, the part of the postscript file that’s after the postscript program. The postscript parser parses the first part of the foo.ps file until the syntactical EOF, and the parsed postscript code reads and uses the rest.
This collides unprettily with boxes that accept input from many client systems, count pages, assign costs to the right team/department, and send postscript onwards to the right printer.
the header format already exists. try concatenating a zipfile to the end of an executable. unzip will still manage to extract, and the file will execute fine. self-extracting archives work this way.
This works on Windows too! Taking advantage of it is part of the reason why the ZIP file format is designed with the dictionary of filenames and offsets at the end of the file instead of the beginning. :)
(You can skip all the work of parsing the ELF header if you do something like putting the length of the file as the last 4 bytes of the file.)
I’ve always liked this method and the fact that it works on windows executables is great too. I was wondering if there’s a similar trick that will work with shared objects – is there a straightforward substitute for argv[0]?
I am usually using a secondary tool (similar to
xxd) but it is checked into the same source control and then invoked first with CMake, then within each target that needs it, we utilize the CMake commandadd_custom_commandto invoke it and generate the necessary files (.h/.cpp). This is portable enough to Unix, macOS, and Windows. The files are typically small and not more than a couple of kilobytes, usually for small icons or maybe a small image in the “About” dialog box of a GUI application. We may eventually switch to #embed but it’s not really a “pain” point at the moment.I documented a few other methods: You can combine the incbin method with file-level inline assembly:
Only really suitable if you’re only targeting GCC/clang. iPXE uses this technique.
On Windows you can also use Windows resource files to embed binaries.
Well, the most portable way had better be #embed within a few years.
I recall a discussion about this on the other site (Embedding binary objects in C – about a post on Ted Ungangst’s site), with loads of interesting methods.