1. 63
  1.  

  2. 22

    Author here. Ask me anything.

    1. 4

      Out of curiosity, where did you learn all this stuff?

      1. 12

        Through trial/error and reading inspiring works such as:

        • SVR4 source code
        • W. Richard Stevens books
        • Lions’ Commentary on UNIX
        • Linux Kernel, XNU, FreeBSD, OpenBSD, and NetBSD codebases

        I also received outstanding training during the six years I worked at Google.

        1. 4

          Can’t speak for the OP, but linkers and loaders is a well-known book about—well, linkers and loaders.

          If you want to learn assembly, there’s no book I can recommend more than this one.

          1. 4

            That book covers the basics. Where assembly gets super fun is once you start using Richard Stallman’s Math 55 inline asm() notation. I had to dig through decades of primary materials to bring you this reference: https://gist.github.com/jart/fe8d104ef93149b5ba9b72912820282c

            1. 1

              Inline asm is harmful and should be avoided. It’s really complicated and easy to get wrong (as your monster of a reference shows). And if performance is critical, then the whole loop should be written in assembly.

              1. 3

                Math 55 is harmful and should be avoided. It’s really complicated and easy to get wrong.

        2. 3

          Hi! As I mentioned in some nested comment on HN, I recently did some hacking on building .apk files from scratch; that’s a much simpler thing than what you did, so I’m not sure I can be of any help, but in case you were interested in talking some more details than what I said in the video, I’m happy to discuss. I’m not an expert (e.g. not really sure how .apks could be useful for commandline apps at all) though, and I’ll for sure not try to work on this myself unfortunately, as I’m already spread waaaay to thin on tons of hobby projects; but .apks are just zips, so just saying I’m there in case this feels tempting to you, if just for the sake of it ;P I’m happy to be contacted via lobste.rs “messages”.

          1. 2

            This is amazing, great job.

          2. 4

            Doesn’t work for me on x86_64 Debian linux. Just outputs:

            Failed to execute process './redbean-2021-02-25.com'. Reason:
            exec: Exec format error
            The file './redbean-2021-02-25.com' is marked as an executable but could not be run by the operating system.
            

            Looks like this is because I use fish as my default shell, which doesn’t do the (slightly insane) Bourne shell thing of “dunno what this file is, let’s just try executing it as a shell script”. Running the thing using Bash works.

            Also, despite the immense cleverness, I’m not sure that writing tools using self-modifying executable code is something we should encourage even as a joke.

            1. 8

              I’m not sure that writing tools using self-modifying executable code is something we should encourage even as a joke.

              Honestly, I’m afraid in my eyes the “even as a joke” phrase here changes a reasonable and objective question into something that sounds uncharitably and completely unnecessarily condescending, and makes me sorry for the author to have to hear that. I know by myself that it’s sometimes hard to resist smug quips, yet personally I find lobste.rs to be teaching me how to strive for sky high civility, and love the community for that.

              As to the actual matter at hand, personally I assume it by default to be the case of a practical approach I see as required in programming: if we can, we do stuff pretty way; but if we can’t, if the ugly way is the only way, buckling up and getting the hands dirty is IMO a much needed display of skill and maturity as well. Then to add to that, trailblazing a new path is often an even dirtier job. Sure, when it’s done, I welcome the cleaners to come and search for what can be improved and perfected. I totally like to do that myself at various times. But in a “hackathon mode” of PoC-ing, my explicit hard rule is to push forward like crazy and don’t look back - the only exception being when the mess distracts my own thoughts too much, stopping my forward motion.

              Based on that - if someone finds a way to achieve the APE goal with no self-modifying code, that would for sure be an interesting improvement, esp. from a W^X security perspective I guess. But if that’s the only way to do that, I find it a perfectly fine solution - another tool in the toolbelt (and one of the most beautiful ones this), with some specific set of compromises as each one of them is. Edit: yeah, and and as I actually suspected, the author doesn’t especially love this approach either, but that’s the only way she could find to do that.

              Uhh… sorry for the rant, I just find the job and the author so amazing, I really felt the need to put down a strong voice of support that I’m backing with all my self.

              1. 3

                I’m not sure that writing tools using self-modifying executable code is something we should encourage even as a joke.

                Why deny ourselves some of the most interesting bits of programming? I suppose you aren’t a fan of currying or lambdas or partial application either then, eh?

                1. 2

                  I suppose you aren’t a fan of currying or lambdas or partial application either then, eh?

                  I was assuming it was a security issue more than an issue of expressivity. That said, it’s not totally clear what icefox is worried about.

                  1. 2

                    There are different degrees of self-modifying code. When I hear that phrase, I’m thinking along the lines of “okay, patch this conditional jump to switch the sorting method from ascending to descending” or “compile a custom bitblt routine for this object at this position and call it.” There are security issues, sure (pages for code have to be writable) but there are other aspects to consider—any speed increases might be negated by the instruction cache being invalidated, or possible crashes in a multithreaded application. It can also make debugging difficult because the source doesn’t match the code being run.

                    With all that said, JIT is a form of self-modifying code, but to me, it feels a bit different since it’s more “run time recompilation of code” than “run time modification of code”.

                  2. 2

                    Also, despite the immense cleverness, I’m not sure that writing tools using self-modifying executable code is something we should encourage even as a joke.

                    Could you elaborate?

                    1. 5

                      On Linux, as far as I can tell, the executable runs with bash on the first run, alters its own header to be a valid ELF file on the first execution, and then executes itself.

                      Any security or robustness issues aside, it means that after running it, you can’t pick up the executable and then give it to someone else and have it have the same properties of “you can run it on anywhere”. After running on Linux, it is no longer a universal executable, it is a Linux executable.

                      I’m sorry for coming off as more critical than I really intended, but I was quite unpleasantly surprised to discover that you have to run it in bash or sh (doing /bin/bash redbean doesn’t work for some reason), that file reports a different file type for the thing before and after running, and that it doesn’t work properly when set read-only or on a read-only filesystem. The executable format is extremely clever, but apparently (currently) comes with a lot of hidden gotcha’s, and I hate hidden gotcha’s. Whether or not they’re gotchas one cares about is up to the user.

                      1. 1

                        That should be avoidable. On Linux, you could create a new shared memory object from bash by creating a file in /dev/shm, then copy the file contents there, edit it, and exec it.

                        1. 1

                          I’m sorry for coming off as more critical than I really intended…

                          I think this might be more directed at jart and/or akavel than me. For my part, I sometimes see people on the cusp of saying something interesting, but they stop for whatever reason. So I wasn’t trying to criticize you; I was just trying to get you to say whatever seemed to be on your mind. :]

                          Any security or robustness issues aside, it means that after running it, you can’t pick up the executable and then give it to someone else and have it have the same properties of “you can run it on anywhere”. After running on Linux, it is no longer a universal executable, it is a Linux executable.

                          I hadn’t actually tried to run this. It’s still a very neat feature, but maybe the name is a bit misleading. (Though Actually Portable (Once) Executable is probably not as catchy.)

                      2. 2

                        I upstreamed a patch with zsh. Looks like I need to upstream a patch with fish too. https://github.com/zsh-users/zsh/commit/326d9c203b3980c0f841bc62b06e37134c6e51ea

                      3. 3

                        Polyglots are really fun! There’s a lot of inspiring research in this area.

                        1. 2

                          Go 1.16 got a new feature that can replicate a similar behavior: https://golang.org/pkg/embed/

                          1. 2

                            I don’t think that this is really comparable, you still need to compile the Go binary for each ARCH/OS combination and adding or adjusting resources will require a recompilation.

                          2. 2

                            I like the idea, except it doesn’t work for me on Linux. It starts in WINE and listens on 8080, but never returns any replies.

                            1. 3

                              If your system is configured to use binfmt_misc then you need to run this command:

                              sudo sh -c "echo ':APE:M::MZqFpD::/bin/sh:' >/proc/sys/fs/binfmt_misc/register"
                              
                              1. 1

                                Do you execute it from the command line?

                                1. 1

                                  You probably have a binary file loader that looks for PE executable headers and lauches wine in order to run them. Maybe wine doesn’t like the binary for some reason?

                                2. 2

                                  This is both a) a lovely hack (the portable executable thing) and b) actually useful.

                                  I love it.

                                  1. 1

                                    Ooh! Cool to see someone is actually using APE out in the wild.

                                    I thought it was a novelty at the time but maybe it’ll be more than that?

                                    1. 15

                                      Well, to be fair, the author of this tool is the person who created APE as well.

                                      1. 3

                                        Wasm3 is using it to provide a portable, universal WebAssembly engine (I’m one of the authors).