This was one of my favorite interview questions for systems engineers: “What happens when you run a ‘hello world’ program, starting from when you hit the “enter” key at the shell prompt. Go as deep as you can.” No surprise, but jvns more than cleared my bar. :-)
I didn’t know about debugfs, that’s a neat-sounding tool to add to the toolbox. She asked about set_tid_addrss and arch_prctl…
set_tid_address: the TID address of a thread is a futex that the kernel pokes when the thread dies. It’s used to implement pthread_join.
arch_prctl: well, these are architecture-specific. In this case, though, it’s probably being used to set the %fs register on an x86_64 platform, which is used as the Thread Local Storage base address. See Ulrich Drepper’s document, https://www.uclibc.org/docs/tls.pdf . Technically, with new x86 parts, you don’t need the arch_prctl to do this, the %fs register (not sure about %gs) can be set directly from userspace.
I’d looked a bit at set_robust_list and rt_sigaction, and I think, at this point, it has something to do with futexes and interruptible system calls, related to thread clean-up, but I could well be mistaken. The glibc code in this area is not very easy to follow, this was a nice little rant about it: https://github.com/m4b/dryad/issues/5#issuecomment-257228730
Note that these are called as libpthread.so is loaded. What she’s observing is the main thread used for program entry being adapted to run in a multi-threaded environment.
That’s correct. Interesting to note that these functions are called with the same argc, argv, and env arguments as main is, I just verified that
__attribute__((constructor))
int shlib_entry(int argc, const char **argv, const char **env)
{
int i;
for (i = 0; i < argc; i++) {
printf("shlib: [%d/%d]: %s\n", i, argc, argv[i]);
}
for (i = 0; env[i]; i++) {
printf("shlib: env: %s\n", env[i]);
}
return 0;
}
worked as expected.
As for how it works at runtime, constructor functions are stored in the .init_array section in ELF binaries, and the dynamic linker walks through these as the library is loaded. In my example, that looked like:
$ readelf -a libentry_demo.so
...
[17] .init_array INIT_ARRAY 0000000000003df0 00002df0
0000000000000010 0000000000000008 WA 0 0 8
...
That doesn’t actually include the shlib_entry pointer I was expecting, but it’s also 16 bytes on disk, only 8 of which have a valid pointer (for something related to transactional memory support?). There’s a relocation in the ELF object that fills in the second entry:
If you’re interested in debugging this sort of thing, it’s not well known, but you can run the ELF interpreter directly on the command-line, and it’ll behave in a very similar (not exactly the same) way as it does when loaded by the kernel. In trying to figure out why .init_array didn’t include the entry I thought it did, I ran gdb as follows:
This is great! I love these style articles that take the things we use every day and deep dive them. If you’re interested in this type of thing, and want to explore the dynamic linking and ELF sections in more detail but similarly light writing style I can’t recommend enough this series: https://fasterthanli.me/series/making-our-own-executable-packer
supposedly you can use dtruss or dtrace on mac instead of strace but I’ve never been brave enough to turn off system integrity protection to get it to work
What are the security implications of disabling SIP? I mean, you are already doomed when someone can run arbitrary code as root on your system, so how much does disabling SIP make the situation worse?
I mean, you are already doomed when someone can run arbitrary code as root on your system
Yes probably as far as /Users is concerned but with SIP enabled system files/paths are read-only, even to root, which provides some protection against tampering/malware.
Before System Integrity Protection (introduced in OS X El Capitan), the root user had no permission restrictions, so it could access any system folder or app on your Mac. Software obtained root-level access when you entered your administrator name and password to install the software. That allowed the software to modify or overwrite any system file or app.
System Integrity Protection includes protection for these parts of the system:
/System
/usr
/bin
/sbin
/var
Apps that are pre-installed with the Mac operating system
Before System Integrity Protection (introduced in OS X El Capitan), the root user had no permission restrictions, so it could access any system folder or app on your Mac.
This was one of my favorite interview questions for systems engineers: “What happens when you run a ‘hello world’ program, starting from when you hit the “enter” key at the shell prompt. Go as deep as you can.” No surprise, but jvns more than cleared my bar. :-)
I didn’t know about
debugfs, that’s a neat-sounding tool to add to the toolbox. She asked aboutset_tid_addrssandarch_prctl…set_tid_address: the TID address of a thread is a futex that the kernel pokes when the thread dies. It’s used to implementpthread_join.arch_prctl: well, these are architecture-specific. In this case, though, it’s probably being used to set the%fsregister on an x86_64 platform, which is used as the Thread Local Storage base address. See Ulrich Drepper’s document, https://www.uclibc.org/docs/tls.pdf . Technically, with new x86 parts, you don’t need thearch_prctlto do this, the%fsregister (not sure about%gs) can be set directly from userspace.I’d looked a bit at
set_robust_listandrt_sigaction, and I think, at this point, it has something to do with futexes and interruptible system calls, related to thread clean-up, but I could well be mistaken. The glibc code in this area is not very easy to follow, this was a nice little rant about it: https://github.com/m4b/dryad/issues/5#issuecomment-257228730Note that these are called as
libpthread.sois loaded. What she’s observing is the main thread used for program entry being adapted to run in a multi-threaded environment.Do you know how libraries like libpthread add code to be run before main when loaded? Would love to add that to the post
Probably something like
constructorattribute: https://stackoverflow.com/a/2053078That’s correct. Interesting to note that these functions are called with the same
argc,argv, andenvarguments asmainis, I just verified thatworked as expected.
As for how it works at runtime,
constructorfunctions are stored in the.init_arraysection in ELF binaries, and the dynamic linker walks through these as the library is loaded. In my example, that looked like:That doesn’t actually include the
shlib_entrypointer I was expecting, but it’s also 16 bytes on disk, only 8 of which have a valid pointer (for something related to transactional memory support?). There’s a relocation in the ELF object that fills in the second entry:If you’re interested in debugging this sort of thing, it’s not well known, but you can run the ELF interpreter directly on the command-line, and it’ll behave in a very similar (not exactly the same) way as it does when loaded by the kernel. In trying to figure out why
.init_arraydidn’t include the entry I thought it did, I ran gdb as follows:which lets me easily single step through the dynamic linker’s own logic, in a way that bare gdb on
/bin/echodidn’t work (for me).This is great! I love these style articles that take the things we use every day and deep dive them. If you’re interested in this type of thing, and want to explore the dynamic linking and ELF sections in more detail but similarly light writing style I can’t recommend enough this series: https://fasterthanli.me/series/making-our-own-executable-packer
What are the security implications of disabling SIP? I mean, you are already doomed when someone can run arbitrary code as root on your system, so how much does disabling SIP make the situation worse?
Yes probably as far as
/Usersis concerned but with SIP enabled system files/paths are read-only, even toroot, which provides some protection against tampering/malware.https://support.apple.com/en-us/HT204899
Actually we can also protect files from the root account by using the legacy of the BSD family: https://apple.stackexchange.com/questions/282339/protect-hosts-file