1. 25
    1. 9

      This was one of my favorite interview questions for systems engineers: “What happens when you run a ‘hello world’ program, starting from when you hit the “enter” key at the shell prompt. Go as deep as you can.” No surprise, but jvns more than cleared my bar. :-)

      I didn’t know about debugfs, that’s a neat-sounding tool to add to the toolbox. She asked about set_tid_addrss and arch_prctl

      set_tid_address: the TID address of a thread is a futex that the kernel pokes when the thread dies. It’s used to implement pthread_join.

      arch_prctl: well, these are architecture-specific. In this case, though, it’s probably being used to set the %fs register on an x86_64 platform, which is used as the Thread Local Storage base address. See Ulrich Drepper’s document, https://www.uclibc.org/docs/tls.pdf . Technically, with new x86 parts, you don’t need the arch_prctl to do this, the %fs register (not sure about %gs) can be set directly from userspace.

      I’d looked a bit at set_robust_list and rt_sigaction, and I think, at this point, it has something to do with futexes and interruptible system calls, related to thread clean-up, but I could well be mistaken. The glibc code in this area is not very easy to follow, this was a nice little rant about it: https://github.com/m4b/dryad/issues/5#issuecomment-257228730

      Note that these are called as libpthread.so is loaded. What she’s observing is the main thread used for program entry being adapted to run in a multi-threaded environment.

      1. 3

        Do you know how libraries like libpthread add code to be run before main when loaded? Would love to add that to the post

        1. 2

          Probably something like constructor attribute: https://stackoverflow.com/a/2053078

          1. 2

            That’s correct. Interesting to note that these functions are called with the same argc, argv, and env arguments as main is, I just verified that

            int shlib_entry(int argc, const char **argv, const char **env)
              int i;
              for (i = 0; i < argc; i++) {
                printf("shlib: [%d/%d]: %s\n", i, argc, argv[i]);
              for (i = 0; env[i]; i++) {
                printf("shlib: env: %s\n", env[i]);
              return 0;

            worked as expected.

            As for how it works at runtime, constructor functions are stored in the .init_array section in ELF binaries, and the dynamic linker walks through these as the library is loaded. In my example, that looked like:

            $ readelf -a libentry_demo.so
              [17] .init_array       INIT_ARRAY       0000000000003df0  00002df0
                   0000000000000010  0000000000000008  WA       0     0     8
            $ objcopy -j .init_array -O binary libentry_demo.so init_array.out; hexdump -C init_array.out
            00000000  00 11 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

            That doesn’t actually include the shlib_entry pointer I was expecting, but it’s also 16 bytes on disk, only 8 of which have a valid pointer (for something related to transactional memory support?). There’s a relocation in the ELF object that fills in the second entry:

            $ readelf -a libentry_demo.so
            000000003df8  000600000001 R_X86_64_64       0000000000001109 shlib_entry + 0

            If you’re interested in debugging this sort of thing, it’s not well known, but you can run the ELF interpreter directly on the command-line, and it’ll behave in a very similar (not exactly the same) way as it does when loaded by the kernel. In trying to figure out why .init_array didn’t include the entry I thought it did, I ran gdb as follows:

            $ LD_PRELOAD=$PWD/libentry_demo.so gdb --args /lib64/ld-linux-x86-64.so.2 /bin/echo hello

            which lets me easily single step through the dynamic linker’s own logic, in a way that bare gdb on /bin/echo didn’t work (for me).

    2. 2

      This is great! I love these style articles that take the things we use every day and deep dive them. If you’re interested in this type of thing, and want to explore the dynamic linking and ELF sections in more detail but similarly light writing style I can’t recommend enough this series: https://fasterthanli.me/series/making-our-own-executable-packer

    3. 1

      supposedly you can use dtruss or dtrace on mac instead of strace but I’ve never been brave enough to turn off system integrity protection to get it to work

      What are the security implications of disabling SIP? I mean, you are already doomed when someone can run arbitrary code as root on your system, so how much does disabling SIP make the situation worse?

      1. 1

        I mean, you are already doomed when someone can run arbitrary code as root on your system

        Yes probably as far as /Users is concerned but with SIP enabled system files/paths are read-only, even to root, which provides some protection against tampering/malware.

        Before System Integrity Protection (introduced in OS X El Capitan), the root user had no permission restrictions, so it could access any system folder or app on your Mac. Software obtained root-level access when you entered your administrator name and password to install the software. That allowed the software to modify or overwrite any system file or app.

        System Integrity Protection includes protection for these parts of the system:

        • /System
        • /usr
        • /bin
        • /sbin
        • /var
        • Apps that are pre-installed with the Mac operating system


        1. 1

          Before System Integrity Protection (introduced in OS X El Capitan), the root user had no permission restrictions, so it could access any system folder or app on your Mac.

          Actually we can also protect files from the root account by using the legacy of the BSD family: https://apple.stackexchange.com/questions/282339/protect-hosts-file