Nix is only as deterministic as it’s builders and compilers that’s true. It gives you the scaffolding you need but you still have to ensure you follow the rules. If you use a builder that creates random data somewhere then you still won’t have a deterministic output. That’s true of pretty much anything that tries to solve this problem though. They are working with tools that make it a little too easy to violate the rules unexpectedly.
Yeah I was actually surprised by this, from some brief experience with a shell.nix
It seems like it is extremely easy to write Nix files “wrong” and break the properties of Nix
Whereas with Bazel, the build breaks if you do something wrong. It helps you write a correct configuration
Similar issue with Make – Make provides you zero help determining dependencies, so basically all Makefiles have subtle bugs. To be fair Ninja also doesn’t strictly enforce things, but it’s easy to read and debug, and has a few tools like a dep graph exporter.
nix-shell is a bit of a lukewarm compromise between Nix’s ability to supply arbitrary dependencies and the guarantees of the build sandbox. It’s not really trying to be hermetic, just ~humane.
I would like to hear more about your shell.nix experience. It has some issues, for sure, but I have found the platform to be extremely reliable overall.
Is that really relevant? A determined user could flash their BIOS with garbage and break anything they want. Does it matter that you can go out of your way to break your own NixOS system? The point is that as a user you’re not supposed to mess with your system manually, but you’re supposed to make changes by changing your configuration.nix. This is not the case for most other distros where you make changes to files under /etc side by side with files maintained by the distro packages.
The point is that I don’t think anyone is checking. IDK how many options in nixpkgs break this. I don’t think we need to stop the user from purposely breaking it but we should make “official” flags not break it and make it easy for users to not break this.
However, I do think there is a place for classical distributions, as there is a place for distributions like Nix, Guix, and even something like GoboLinux.
Pretty much every concern you share in that post is solved by NixOS. Some points
You don’t modify files in /etc, you use the NixOS configuration to do so
Changes are idempotent
Changes are reversible, there’s a history of config revisions called generations, or you can just remove the config and apply to roll forward
System configuration is reproducible and deterministic
Services are systemd units declared through the configuration
Packages reference their direct dependencies allowing for multiple versions of the same lib/package/etc
NixOS isn’t perfect by any means, but it is a solution you seem to be looking for. Guix probably as well, but it’s a smaller and more limited distro.
I took the multiple comments that “NixOS may do this but I’ll have to research” to mean you haven’t tried it. And this entire blog post screams to me that NixOS is a solution for you.
You don’t modify files in /etc, you use the NixOS configuration to do so
In a way, that’s second system syndrome. Suse Linux was bashed for doing something like that with YaST for a long time…
In case of NixOS, I found that new language (not the syntax, the semantics) changed too fast for my liking…
System configuration is reproducible and deterministic
With a few caveats: Software might still get updated (I suppose you can lock that down to specific revisions, but who does?). From a bugfix/security perspective this may be desirable, but it’s not entirely reproducible, and updates can always introduce new bugs or incompatibilities, too.
You’ll get the same version with the same Nixpkgs, every time. buildInputs = [ bash ] will always point at the same bash package for some given Nixpkgs. Package versioning is a different issue.
I think you’re misunderstanding the way nixpkgs works with versions and what the versions actually mean there. Check it out in practice. Unless you update the channels/flakes or whatever you use between runs, nothing will change - there’s no explicit pinning required.
The way I manage this is using several different versions of nixpkgs; a “known good” version for each package I want to lock, so for agda I would have nixpkgsAgda, the regular nixpkgs which is pinned to a stable release, and nixpkgsLatest which is pinned to master.
Most of my packages are on stable nixpkgs. Every now and then when I run nix flake update, it pulls in new versions of software. Things pinned to stable have never broken so far. Things pinned to latest are updated to their latest versions, and things pinned to specific packages never change.
While it does involve pulling in several versions of nixpkgs, I build a lot of software from source anyway so this doesn’t matter to me very much. I do hope that nixpkgs somehow fixes the growing tarball size in the future…
buildInputs = [ bash ] may or may not point at the same version
That’s a snippet of code written in the Nix expression language. bash is not a keyword in that language, it’s just some arbitrary variable name (indeed, so is buildInputs). Just like any other language, if that bash variable is defined/computed to be some pure, constant value, then it will always evaluate to that same thing. If it’s instead defined/computed to be some impure, under-determined value then it may vary between evaluations. Here’s an example of the former:
This evaluates to { buildInputs = [ «derivation /nix/store/6z1cb92fmxq2svrq3i68wxjmd6vvf904-bash-5.2-p15.drv» ]; } and (as far as I’m aware) always will do; even if github.com disappears, it may still live on if that sha256 appears in a cache!
Here’s an example of an impure value, which varies depending on the OS and CPU architecture, on the presence/contents of a NIX_PATH environment variable, the presence/contents of a ~/.config/nixpkgs/overlays.nix file, etc.
PS: I’ve avoided the word “version”, since that concept doesn’t really exist in Nix. It’s more precise to focus on the .drv file’s path, since that includes a hash (e.g. 6z1cb92fmxq2svrq3i68wxjmd6vvf904) which is the root of a Merkle tree that precisely defines that derivation and all of its transitive dependencies; whereas “version” usually refers to an optional, semi-numerical suffix on the file name (e.g. 5.2-p15) which (a) is easy to spoof, and (b) gives no indication of the chosen dependencies, compiler flags, etc. which may have a large effect on the resulting behaviour (which is ultimately what we care about).
In case of NixOS, I found that new language (not the syntax, the semantics) changed too fast for my liking…
Do you happen to have an example of such an semantic change at hand? Curious as nix is often regarded as rather conservative and trying to keep backwards-compatibility as good as possible.
Part of the reason Nix/Guix take their particular approach is specifically because the global mutable state of a traditional FHS style distribution is nigh-impossible to manipulate
in a deterministic way; not just because of the manipulation by the package manager, but because there is no delineation between mutations by the package manager and those by the user. A great example is /etc/passwd, which contains both locally created users as well as “service accounts” the distro maintainers make to separate out some unprivileged services.
it needs a lot of work to enforce 1) determinism/reproducibility, 2) non-hysterisis (when you stretch an object and it doesn’t return /exactly/ back… like hidden side effects in ansible), 3) ease of use, & 4) acceptance testing.
Indeed, after reading the landing page, it seems Michael has reached (way before me) to some of the same conclusions (he has more articles on the topic that I’ll have to read).
However, distri is clearly an experimental distribution (say so on the landing page), thus unfortunately still not a proper production-ready solution.
I don’t think this actually pertains to the problem I describe. I don’t think we need a new OS (as in kernel), I think we need to clean up our distributions packaging. (Do you have a link where they describe the way packaging and installation is done?)
Genode supports several kernels, including Linux, but generally not preferable, as third generation microkernels such as seL4 are simply that much better.
By default, it works with perfectly reproduce-able scenarios similar to nix. They have a decent webbrowser as well as 3d-acceleration so Linux is seldom needed. When it is, they have a working port of Virtualbox.
The Genode Foundations handbook (linked in their frontpage) is the go-to for describing the architecture; Chapter 4.6 and 5.5 and 6 is where I’d look at for this, but the intro chapter is worth reading.
Remove: Instead of the container technology nudging people into creating small deployment bases with only the minimum required files to support the use-case at hand, it has helped “devops engineers” bundle even the proverbial kitchen sink with their applications.
Add: Container technology has not nudged people to small, minimal deployments. Many applications get deployed with the proverbial kitchen sink, enabled by containerization to minimize the fallout of sloppy decisions.
The Yocto / Bitbake system is often used to create a Linux filesystem (and kernels, bootloaders, etc.) from a set of recipes. These recipes usually specify a released tarball or git commit for the software being built. Primarily for embedded systems.
If you have all your recipes and such in version control, then theoretically you can exactly reproduce a system image.
And of course they do care quite a bit about reproducibility, since it’s so core to bitbake’s functionality. The output of each build step is cached and hashed. If the hash hasn’t changed, then its outputs are reused. This is applied recursively, so you can make a change and be assured that the output is the same as if you had built everything from scratch, even though only a few build steps were ran. It’s common to share state (sstate) within an organization, which can greatly speed up builds.
My suggestion is of course a bit tongue-in-cheek, since yocto (or rather openembedded) is sort of like a cross-compile-only version of gentoo. In their quest for reproducibility, they don’t even use the host toolchain (or any utilities). While you certainly could run yocto in yocto, it would be pretty masochistic. Plus, many of the defaults are oriented towards producing small, efficient binaries. While you can certainly enable all the bells and whistles, it’s not really the intention of the distro.
While I generally agree with what you are saying, I was hedging a little bit. Mostly with regards to creating two bit-exact build images when you have two different machines building from scratch from the same set of recipes. It is easy, for example, to have timestamps squirreled away inside the build image. Though of course those two build images should be functionally identical. I’m not so worried about bit-for-bit identical builds, but I am under the impression that some people are.
Well, that’s why we have SOURCE_DATE_EPOCH and the like.
It actually is quite important to have bit-for-bit identical images in bitbake. For example, say that you modify a dependency of some package and do_build has to be re-run. If that dependency change doesn’t affect the build output, then the sstate hash for that build step will be the same. Then, since the hash of do_build didn’t change, all the dependent build steps can be reused. In this way, changes to variables which might otherwise require large rebuilds can result in much smaller effort. But if you don’t have a bit-for-bit reproducibility, then everything else in the dependency cone will need to be rebuilt. I’m not sure if this is as feasible on multi-machine setups, but it certainly is useful for me.
Systemd has a lot of functionality to reduce the need for package manager hooks. The Fitting it all together post by Lennnart Pottering explains their vision for an Immutable OS built around Systemd.
You can get pretty far just creating drop in files for the various systemd components (sysusers, tmpfiles, network, repart, etc). Then the unit files have a lot of directives for limiting host requirements for daemons (DynamicUser, The *Directory directives, socket activation).
The Systemd tool mkosi is also pretty great for making signed images. Everything is built from traditional distro packages but a distro that uses Systemd should be using hooks sparingly.
You still need hooks to tell systemd to reload configuration files, but I suppose a package manager could do that based on what paths the package created.
This is an interesting article, but I feel that it leans too heavily on a few jargon terms that are not as widely understood as the author perhaps suspects, and doesn’t look hard enough at the existing systems out there.
It assumes more knowledge of how existing packaging tools work than I’ve ever needed to acquire. I have never built a package. What’s a hook? Why does it exist? What does it do? Is it needed? What for?
It also assumes more foreknowledge of container systems than I’ve ever needed. I predicted the rise of containers two years before Docker, but that doesn’t mean I’ve ever built or deployed one. That’s not my job.
What precisely does “deterministic” mean in this context? I interviewed the CEO of Rocky Linux and he talked a lot about deterministic compilation, and when I asked him what that was and what it meant, he floundered. He expected me to know and to care and was unable to clearly explicate what it was when I did not.
For comparison, I have written about packaging systems too; here is part 1 and part 2.
You are right that I’m assuming from the reader quite a bit of familiarity with OS package management. (The container stuff was tangent to the subject, thus not essential.)
Regarding the hooks, I’ve tried to briefly describe what they are (“scripts that …”), but I’ve focused more on the functionality they provide.
Regarding “determinism” I’ve succinctly described it as “in the cryptographic sense of identical checksums”, and I’ve stated that, starting from the same “base-image” (say an existing installation), installing (in one operation, or the same sequence of multiple installations) the same set of packages (given in any order), regardless of how many times (or the phase of the moon), should yield an identical “image”. (Here by “image” I actually mean the same directories, the same files, with the same contents, the same timestamps, the same owners, etc. If one creates a reproducible tar after each install, the md5sum of those tar archives would be identical.)
We need everything to be deterministic, or at least to be repeatable after a non-deterministic choice is made. That is, non-deterministic choices at least need to be captured and able to be used to reproduce determinism.
You already have your solution, but you haven’t tried it. NixOS.
Note that Nix isn’t quite completely deterministic due to the activation script which is a giant pile of shell commands.
(It maybe be mostly deterministic, but as a user you can add basically whatever you want to it and break this.)
Nix is only as deterministic as it’s builders and compilers that’s true. It gives you the scaffolding you need but you still have to ensure you follow the rules. If you use a builder that creates random data somewhere then you still won’t have a deterministic output. That’s true of pretty much anything that tries to solve this problem though. They are working with tools that make it a little too easy to violate the rules unexpectedly.
Yeah I was actually surprised by this, from some brief experience with a shell.nix
It seems like it is extremely easy to write Nix files “wrong” and break the properties of Nix
Whereas with Bazel, the build breaks if you do something wrong. It helps you write a correct configuration
Similar issue with Make – Make provides you zero help determining dependencies, so basically all Makefiles have subtle bugs. To be fair Ninja also doesn’t strictly enforce things, but it’s easy to read and debug, and has a few tools like a dep graph exporter.
nix-shell is a bit of a lukewarm compromise between Nix’s ability to supply arbitrary dependencies and the guarantees of the build sandbox. It’s not really trying to be hermetic, just ~humane.
I would like to hear more about your shell.nix experience. It has some issues, for sure, but I have found the platform to be extremely reliable overall.
I linked some comments from this page - https://github.com/oilshell/oil/wiki/Can-Oil-Use-Nix%3F
Is that really relevant? A determined user could flash their BIOS with garbage and break anything they want. Does it matter that you can go out of your way to break your own NixOS system? The point is that as a user you’re not supposed to mess with your system manually, but you’re supposed to make changes by changing your
configuration.nix. This is not the case for most other distros where you make changes to files under/etcside by side with files maintained by the distro packages.The point is that I don’t think anyone is checking. IDK how many options in nixpkgs break this. I don’t think we need to stop the user from purposely breaking it but we should make “official” flags not break it and make it easy for users to not break this.
I did try Nix (briefly).
However, I do think there is a place for classical distributions, as there is a place for distributions like Nix, Guix, and even something like GoboLinux.
Pretty much every concern you share in that post is solved by NixOS. Some points
NixOS isn’t perfect by any means, but it is a solution you seem to be looking for. Guix probably as well, but it’s a smaller and more limited distro.
I took the multiple comments that “NixOS may do this but I’ll have to research” to mean you haven’t tried it. And this entire blog post screams to me that NixOS is a solution for you.
In a way, that’s second system syndrome. Suse Linux was bashed for doing something like that with YaST for a long time…
In case of NixOS, I found that new language (not the syntax, the semantics) changed too fast for my liking…
With a few caveats: Software might still get updated (I suppose you can lock that down to specific revisions, but who does?). From a bugfix/security perspective this may be desirable, but it’s not entirely reproducible, and updates can always introduce new bugs or incompatibilities, too.
All packages are locked in Nix. So With a few caveats: Software might still get updated is not a problem there.
https://www.software.ac.uk/blog/2017-10-05-reproducible-environments-nix offers “download a specific release tarball of buildInput specs” to improve matters. Other than that,
buildInputs = [ bash ]may or may not point at the same version. https://github.com/NixOS/nixpkgs/issues/93327 discusses this for 3 years and there doesn’t seem to be a resolution.You’ll get the same version with the same Nixpkgs, every time.
buildInputs = [ bash ]will always point at the samebashpackage for some given Nixpkgs. Package versioning is a different issue.I count package versioning as part of the system configuration: Sure, your package names are always the same, but the content might differ wildly.
The content will be the same every time with the same system configuration, assuming you’ve pinned your Nixpkgs version.
Or using flakes.
I think you’re misunderstanding the way nixpkgs works with versions and what the versions actually mean there. Check it out in practice. Unless you update the channels/flakes or whatever you use between runs, nothing will change - there’s no explicit pinning required.
The way I manage this is using several different versions of nixpkgs; a “known good” version for each package I want to lock, so for agda I would have
nixpkgsAgda, the regular nixpkgs which is pinned to a stable release, andnixpkgsLatestwhich is pinned to master.Most of my packages are on stable nixpkgs. Every now and then when I run
nix flake update, it pulls in new versions of software. Things pinned to stable have never broken so far. Things pinned to latest are updated to their latest versions, and things pinned to specific packages never change.While it does involve pulling in several versions of nixpkgs, I build a lot of software from source anyway so this doesn’t matter to me very much. I do hope that nixpkgs somehow fixes the growing tarball size in the future…
That’s a snippet of code written in the Nix expression language.
bashis not a keyword in that language, it’s just some arbitrary variable name (indeed, so isbuildInputs). Just like any other language, if thatbashvariable is defined/computed to be some pure, constant value, then it will always evaluate to that same thing. If it’s instead defined/computed to be some impure, under-determined value then it may vary between evaluations. Here’s an example of the former:This evaluates to
{ buildInputs = [ «derivation /nix/store/6z1cb92fmxq2svrq3i68wxjmd6vvf904-bash-5.2-p15.drv» ]; }and (as far as I’m aware) always will do; even ifgithub.comdisappears, it may still live on if thatsha256appears in a cache!Here’s an example of an impure value, which varies depending on the OS and CPU architecture, on the presence/contents of a
NIX_PATHenvironment variable, the presence/contents of a~/.config/nixpkgs/overlays.nixfile, etc.I recommend sticking to the former ;)
PS: I’ve avoided the word “version”, since that concept doesn’t really exist in Nix. It’s more precise to focus on the
.drvfile’s path, since that includes a hash (e.g.6z1cb92fmxq2svrq3i68wxjmd6vvf904) which is the root of a Merkle tree that precisely defines that derivation and all of its transitive dependencies; whereas “version” usually refers to an optional, semi-numerical suffix on the file name (e.g.5.2-p15) which (a) is easy to spoof, and (b) gives no indication of the chosen dependencies, compiler flags, etc. which may have a large effect on the resulting behaviour (which is ultimately what we care about).Do you happen to have an example of such an semantic change at hand? Curious as nix is often regarded as rather conservative and trying to keep backwards-compatibility as good as possible.
Part of the reason Nix/Guix take their particular approach is specifically because the global mutable state of a traditional FHS style distribution is nigh-impossible to manipulate in a deterministic way; not just because of the manipulation by the package manager, but because there is no delineation between mutations by the package manager and those by the user. A great example is
/etc/passwd, which contains both locally created users as well as “service accounts” the distro maintainers make to separate out some unprivileged services.Relevant writing from Lennart Poettering on state in Linux distributions and how to accomplish a factory reset mechanism.
There is a place for “classical” (read: legacy) distributions, yes, in the same way there is a place for collecting stamps.
I want to switch to NixOS so badly, but I also want SELinux support.
it needs a lot of work to enforce 1) determinism/reproducibility, 2) non-hysterisis (when you stretch an object and it doesn’t return /exactly/ back… like hidden side effects in ansible), 3) ease of use, & 4) acceptance testing.
How about something like distri?
I’ll post the link to the main site and GitHub of the project: https://distr1.org/ and https://github.com/distr1/distri.
Indeed, after reading the landing page, it seems Michael has reached (way before me) to some of the same conclusions (he has more articles on the topic that I’ll have to read).
However,
distriis clearly an experimental distribution (say so on the landing page), thus unfortunately still not a proper production-ready solution.Ever looked into Genode?
I don’t think this actually pertains to the problem I describe. I don’t think we need a new OS (as in kernel), I think we need to clean up our distributions packaging. (Do you have a link where they describe the way packaging and installation is done?)
Genode supports several kernels, including Linux, but generally not preferable, as third generation microkernels such as seL4 are simply that much better.
By default, it works with perfectly reproduce-able scenarios similar to nix. They have a decent webbrowser as well as 3d-acceleration so Linux is seldom needed. When it is, they have a working port of Virtualbox.
The Genode Foundations handbook (linked in their frontpage) is the go-to for describing the architecture; Chapter 4.6 and 5.5 and 6 is where I’d look at for this, but the intro chapter is worth reading.
Recommendation:
Remove: Instead of the container technology nudging people into creating small deployment bases with only the minimum required files to support the use-case at hand, it has helped “devops engineers” bundle even the proverbial kitchen sink with their applications.
Add: Container technology has not nudged people to small, minimal deployments. Many applications get deployed with the proverbial kitchen sink, enabled by containerization to minimize the fallout of sloppy decisions.
the solution is clearly yocto :^)
Could you please elaborate more?
The Yocto / Bitbake system is often used to create a Linux filesystem (and kernels, bootloaders, etc.) from a set of recipes. These recipes usually specify a released tarball or git commit for the software being built. Primarily for embedded systems.
https://www.yoctoproject.org/
If you have all your recipes and such in version control, then theoretically you can exactly reproduce a system image.
And of course they do care quite a bit about reproducibility, since it’s so core to bitbake’s functionality. The output of each build step is cached and hashed. If the hash hasn’t changed, then its outputs are reused. This is applied recursively, so you can make a change and be assured that the output is the same as if you had built everything from scratch, even though only a few build steps were ran. It’s common to share state (sstate) within an organization, which can greatly speed up builds.
My suggestion is of course a bit tongue-in-cheek, since yocto (or rather openembedded) is sort of like a cross-compile-only version of gentoo. In their quest for reproducibility, they don’t even use the host toolchain (or any utilities). While you certainly could run yocto in yocto, it would be pretty masochistic. Plus, many of the defaults are oriented towards producing small, efficient binaries. While you can certainly enable all the bells and whistles, it’s not really the intention of the distro.
While I generally agree with what you are saying, I was hedging a little bit. Mostly with regards to creating two bit-exact build images when you have two different machines building from scratch from the same set of recipes. It is easy, for example, to have timestamps squirreled away inside the build image. Though of course those two build images should be functionally identical. I’m not so worried about bit-for-bit identical builds, but I am under the impression that some people are.
Well, that’s why we have
SOURCE_DATE_EPOCHand the like.It actually is quite important to have bit-for-bit identical images in bitbake. For example, say that you modify a dependency of some package and
do_buildhas to be re-run. If that dependency change doesn’t affect the build output, then the sstate hash for that build step will be the same. Then, since the hash ofdo_builddidn’t change, all the dependent build steps can be reused. In this way, changes to variables which might otherwise require large rebuilds can result in much smaller effort. But if you don’t have a bit-for-bit reproducibility, then everything else in the dependency cone will need to be rebuilt. I’m not sure if this is as feasible on multi-machine setups, but it certainly is useful for me.Neat! I didn’t know that reproducible builds had been taken that far with Yocto. I’ve still got a lot to learn.
[Comment removed by author]
Systemd has a lot of functionality to reduce the need for package manager hooks. The Fitting it all together post by Lennnart Pottering explains their vision for an Immutable OS built around Systemd.
You can get pretty far just creating drop in files for the various systemd components (sysusers, tmpfiles, network, repart, etc). Then the unit files have a lot of directives for limiting host requirements for daemons (DynamicUser, The *Directory directives, socket activation).
The Systemd tool mkosi is also pretty great for making signed images. Everything is built from traditional distro packages but a distro that uses Systemd should be using hooks sparingly.
You still need hooks to tell systemd to reload configuration files, but I suppose a package manager could do that based on what paths the package created.
If you haven’t seen it Michael Stapelberg wrote about hooks making package managers slow and built an experimental distro around the ideas from that post.
This is an interesting article, but I feel that it leans too heavily on a few jargon terms that are not as widely understood as the author perhaps suspects, and doesn’t look hard enough at the existing systems out there.
It assumes more knowledge of how existing packaging tools work than I’ve ever needed to acquire. I have never built a package. What’s a hook? Why does it exist? What does it do? Is it needed? What for?
It also assumes more foreknowledge of container systems than I’ve ever needed. I predicted the rise of containers two years before Docker, but that doesn’t mean I’ve ever built or deployed one. That’s not my job.
What precisely does “deterministic” mean in this context? I interviewed the CEO of Rocky Linux and he talked a lot about deterministic compilation, and when I asked him what that was and what it meant, he floundered. He expected me to know and to care and was unable to clearly explicate what it was when I did not.
For comparison, I have written about packaging systems too; here is part 1 and part 2.
You are right that I’m assuming from the reader quite a bit of familiarity with OS package management. (The container stuff was tangent to the subject, thus not essential.)
Regarding the hooks, I’ve tried to briefly describe what they are (“scripts that …”), but I’ve focused more on the functionality they provide.
Regarding “determinism” I’ve succinctly described it as “in the cryptographic sense of identical checksums”, and I’ve stated that, starting from the same “base-image” (say an existing installation), installing (in one operation, or the same sequence of multiple installations) the same set of packages (given in any order), regardless of how many times (or the phase of the moon), should yield an identical “image”. (Here by “image” I actually mean the same directories, the same files, with the same contents, the same timestamps, the same owners, etc. If one creates a reproducible
tarafter each install, themd5sumof thosetararchives would be identical.)Conceded, yes. But the things are this:
Have you?
Why is that (there being scripts) a problem?
OK, which switches the requirement from “knowing about packaging systems” to “knowing about cryptography”.
Which effectively doubles the prior knowledge needed. That’s not helping, that’s making it worse.
For very small values of “need”.
We need everything to be deterministic, or at least to be repeatable after a non-deterministic choice is made. That is, non-deterministic choices at least need to be captured and able to be used to reproduce determinism.
[Comment removed by author]
hence the reproducible builds project