The original problem was that our test suite started hanging on Mac OS X.
fork-without-exec has never been safe on Darwin-based OSs. Besides the deal-breaker about threads, Mach ports are not copied by a fork — they kind of can’t be, since a port has a matching port in another process, like a pipe — and many system services use Mach ports for IPC.
What does that mean, exactly? If you’re calling fork without exec ion a cross-platform application which doesn’t use “ports” (I’d never heard that term before in this context), is that something you’d have to worry about?
Mach Ports are an IPC mechanism that was used to build a rich RPC layer on Mach and then inherited by the XNU kernel. It’s used extensively on all XNU-based operating systems (macOS, iOS, iPadOS, tvOS, and watchOS) for a lot of system services. Just because you’re not using them explicitly doesn’t mean that they won’t be used in the underlying implementation of libraries that you link to.
In particular, every Mach Task (roughly, process) has a task port that defines the identity of the task and provide interfaces for debugging and so on. If something has been authorised for your parent’s identity, it may suddenly stop working after fork.
hm, that sounds pretty broken - doesn’t that violate POSIX? I guess there are no tests that target this in the UNIX compliance, otherwise I don’t see how Mac OS could get UNIX certified.
The spec for fork says that, in a multithreaded program, you may call only async-signal-safe functions in between fork and execve. If you link third-party libraries then you need to assume that threads might exist. This is a stricter set of requirements than vfork, which can call any function but must ensure that any memory allocated after vfork is either freed before execve (or after, in the parent).
Note that it’s possible for libraries to spawn threads in global constructors, and so, of the three options, only the first and third are actually safe. The safe variant of the second is to audit all of your libraries to ensure that you have no threads created at the point where you fork.
For ‘threads created’ read ‘resources that are not properly shared across fork’. On macOS, as @snej points out, this includes Mach ports. On Android, it includes the Binder handle. I am not sure what happens with io_uring across fork, but it might be surprising. In general, file descriptors (including their seek offset) are shared across fork, which may cause some surprise, but in a few cases (particularly certain device nodes) they will be effectively closed and reopened and so will have different sharing properties.
There are a few places where you can safely use fork but they are few and far between. In general, you want vfork + execve, you don’t want fork.
Fork’s semantics have infected the design of each new API that creates process state. The POSIX specification now lists 25 special cases in how the parent’s state is copied to the child [63]: file locks, timers, asynchronous IO operations, tracing, etc. In addition, numerous system call flags control fork’s behaviour with respect to memory mappings (Linux madvise() flags MADV_DONTFORK/DOFORK/WIPEONFORK, etc.), file descriptors (O_CLOEXEC, FD_CLOEXEC) and threads (pthread_atfork()). Any non-trivial OS facility must document its behaviour across a fork, and user-mode libraries must be prepared for their state to be forked at any time.
Yeah, it convinces me even more that this UNIX(TM) certification is just a bullshit business. You can get certified for strict standard libc-using programs and yet have all sorts of problems for anything that strays beyond that (even indirectly).
Of course, but not if those APIs interfere with normal operation. If I understand @david_chisnall’s comment correctly, libraries may use these APIs and then mess up the process’ state. But I guess those libraries aren’t libc, so strictly speaking a regular portable application shouldn’t run into this. Of course, portable libraries may decide to use these APIs on MacOS though.
Pretty much any non-trivial program uses libraries outside of POSIX. The Darwin ports of some of those libraries (especially if they are system libraries provided by Apple) may well depend on Mach ports. For example, anything using Keychain to store secrets will depend on Mach ports, anything using libdispatch for concurrency will as well.
The spec for fork explicitly states that you can only use APIs that are explicitly stated as being safe to use after fork. So you’d need to go through every API you call to verify that if it’s a POSIX API it’s an API that states fork-safety is mandated in the spec, any non-POSIX API should be assumed to be non-fork safe, any libraries you use should be assumed to use non-fork-safe APIs, and so be transitively unsafe.
A lot of “low level”/“basic” APIs on Darwin are not fork safe.
If you stick to basic APIs you may be OK, but there’s not really an easy way to tell what APIs may be using Mach IPC under the hood, and it can change over time. Using CoreFoundation almost certainly involves some port-based connections for access to things like the pasteboard and LaunchServices. Even doing DNS lookups, like getaddrinfo, probably calls out to the system resolver process.
I am pretty sure Apple has a tech note about this.
What is a “basic API”? To quote the fork Darwin man page:
“All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. “
fork-without-exec has never been safe on Darwin-based OSs. Besides the deal-breaker about threads, Mach ports are not copied by a fork — they kind of can’t be, since a port has a matching port in another process, like a pipe — and many system services use Mach ports for IPC.
What does that mean, exactly? If you’re calling fork without exec ion a cross-platform application which doesn’t use “ports” (I’d never heard that term before in this context), is that something you’d have to worry about?
Mach Ports are an IPC mechanism that was used to build a rich RPC layer on Mach and then inherited by the XNU kernel. It’s used extensively on all XNU-based operating systems (macOS, iOS, iPadOS, tvOS, and watchOS) for a lot of system services. Just because you’re not using them explicitly doesn’t mean that they won’t be used in the underlying implementation of libraries that you link to.
In particular, every Mach Task (roughly, process) has a task port that defines the identity of the task and provide interfaces for debugging and so on. If something has been authorised for your parent’s identity, it may suddenly stop working after fork.
hm, that sounds pretty broken - doesn’t that violate POSIX? I guess there are no tests that target this in the UNIX compliance, otherwise I don’t see how Mac OS could get UNIX certified.
The spec for
forksays that, in a multithreaded program, you may call only async-signal-safe functions in betweenforkandexecve. If you link third-party libraries then you need to assume that threads might exist. This is a stricter set of requirements thanvfork, which can call any function but must ensure that any memory allocated aftervforkis either freed beforeexecve(or after, in the parent).Note that it’s possible for libraries to spawn threads in global constructors, and so, of the three options, only the first and third are actually safe. The safe variant of the second is to audit all of your libraries to ensure that you have no threads created at the point where you
fork.For ‘threads created’ read ‘resources that are not properly shared across fork’. On macOS, as @snej points out, this includes Mach ports. On Android, it includes the Binder handle. I am not sure what happens with
io_uringacrossfork, but it might be surprising. In general, file descriptors (including their seek offset) are shared across fork, which may cause some surprise, but in a few cases (particularly certain device nodes) they will be effectively closed and reopened and so will have different sharing properties.There are a few places where you can safely use
forkbut they are few and far between. In general, you wantvfork+execve, you don’t wantfork.A relevant paper; https://www.microsoft.com/en-us/research/publication/a-fork-in-the-road/
Makes me wonder if that question whether FreeBSD is a real UNIX is being asked about a wrong OS indeed. :)
Yeah, it convinces me even more that this UNIX(TM) certification is just a bullshit business. You can get certified for strict standard libc-using programs and yet have all sorts of problems for anything that strays beyond that (even indirectly).
You can have extra APIs outside of the specification and still be compliant
Of course, but not if those APIs interfere with normal operation. If I understand @david_chisnall’s comment correctly, libraries may use these APIs and then mess up the process’ state. But I guess those libraries aren’t
libc, so strictly speaking a regular portable application shouldn’t run into this. Of course, portable libraries may decide to use these APIs on MacOS though.Pretty much any non-trivial program uses libraries outside of POSIX. The Darwin ports of some of those libraries (especially if they are system libraries provided by Apple) may well depend on Mach ports. For example, anything using Keychain to store secrets will depend on Mach ports, anything using libdispatch for concurrency will as well.
The spec for fork explicitly states that you can only use APIs that are explicitly stated as being safe to use after fork. So you’d need to go through every API you call to verify that if it’s a POSIX API it’s an API that states fork-safety is mandated in the spec, any non-POSIX API should be assumed to be non-fork safe, any libraries you use should be assumed to use non-fork-safe APIs, and so be transitively unsafe.
A lot of “low level”/“basic” APIs on Darwin are not fork safe.
If you stick to basic APIs you may be OK, but there’s not really an easy way to tell what APIs may be using Mach IPC under the hood, and it can change over time. Using CoreFoundation almost certainly involves some port-based connections for access to things like the pasteboard and LaunchServices. Even doing DNS lookups, like getaddrinfo, probably calls out to the system resolver process.
I am pretty sure Apple has a tech note about this.
What is a “basic API”? To quote the fork Darwin man page:
“All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. “