Great explanation, now I get what stdenv is – I guess it’s a little like build-essential in Debian, a set of packages / tools that are used to build other packages. Obviously there is the huge difference that variations on it are created as expressions in functional programming language. But the idea is very simple.
Also see this gist by abathur on the recent thread:
The 1600-line setup.sh script is a bit of a tragedy. It is absolutely SCREAMING for a shell with JSON-like data structures.
The JSON derivation file shows that.
It’s not doing anything particularly interesting or advanced – all it boils down is making a list of commands to execute, their environments, and file system, and running them.
Yet it contains metaprogramming head-scratchers like
local hooksSlice="${hookName%Hook}Hooks[@]"
And
if [[ -d "$dir" && "${!varName:+${delimiter}${!varName}${delimiter}}" \
!= *"${delimiter}${dir}${delimiter}"* ]]; then
export "${varName}=${!varName:+${!varName}${delimiter}}${dir}"
fi
Also, I believe this stuff is NOT confined to the setup.sh script – it leaks into package definitions, with string-ish metaprogramming hacks.
The irony of version pinning and determinism is that it allows you to write extremely bad code that happens to work. It allows you depend on bugs in specific versions of bash, like interactions between empty arrays, set -u, and var ref / name ref.
The semantics of those features aren’t documented, and they change based on the winds of the bash codebase.
(This phenomena also emerges in Google’s monorepo with Bazel – except there are full-time engineers that clean up the code there. They mainly clean up C++ and Java, e.g. as part of doing toolchain upgrades.)
Nix uses some of the worst shell I’ve ever seen, and I’ve looked at many distros. I implemented a bunch of features for it in Oil, starting way back in 2017, prompted by Nix users:
It looks fine and simple (for a shell script). Somewhere along the way every last bash misfeature was added to that script, and I fail to see any reason why. The code is also structured pretty poorly.
If anyone wants to do something about this mess, rather than piling more of a mess on top, you could implement nocasematch in OSH (the most bash-compatible shell by a mile), which is basically some flags to fnmatch() and glob(), aided by extensive tests:
This is a neat runthrough! It takes a look at the same concepts of Nix Pill 7 (Working Derivation) from another angle and with a more complex build - I think your post does a better job at making it understandable though.
This is a nice write up, I learned quite a few things as a long time Nix user! I just have a nit to pick:
It turns out that derivation is a function in the Nix language
That’s true, derivation is a function, but a derivation is not a function, it’s that data structure you see in that .drv file, it’s a recipe for building something. derivation is a function that constructs a derivation.
I wonder if that 1600 lines setup.sh script could be simplified? Or at least improved in a way that it would be possible to explain what it does and why.
Yes! This is the good stuff. This is what Nix needs more of - explanations of how to make new things with it, rather than how to use things that have already been made. Tutorials about how to use nix run program are not really Nix tutorials; they don’t teach you how to use Nix, they teach you how to invoke it. It’s easy to invoke things, but it’s much harder to build invokable things that did not already exist.
Good for insight into what Nix “does”! I admit to never really digging into the build scripts this deeply!
However, I think there is a lot of cool stuff in those derivation files that are important. The Nix store is a really important part of how Nix works!
A key thing about Nix is that those derivation files contain references to all the inputs for a build so a hash of that file will point to a unique build and any changes to the build will have a different hash (I think that’s the hash in the path). The system knows it only has to rebuild things when the drv files change and old derivations will continue to point at their old inputs so you can actually have two versions of a package on disk and they can even depend on different versions of a library without conflicting.
Great explanation, now I get what stdenv is – I guess it’s a little like build-essential in Debian, a set of packages / tools that are used to build other packages. Obviously there is the huge difference that variations on it are created as expressions in functional programming language. But the idea is very simple.
Also see this gist by abathur on the recent thread:
https://gist.github.com/abathur/917d722a16c65824663d762cdd2f4d22
https://lobste.rs/s/eru89e/some_notes_on_using_nix#c_ssszkf
The 1600-line setup.sh script is a bit of a tragedy. It is absolutely SCREAMING for a shell with JSON-like data structures.
The JSON derivation file shows that.
It’s not doing anything particularly interesting or advanced – all it boils down is making a list of commands to execute, their environments, and file system, and running them.
Yet it contains metaprogramming head-scratchers like
And
Also, I believe this stuff is NOT confined to the
setup.shscript – it leaks into package definitions, with string-ish metaprogramming hacks.The irony of version pinning and determinism is that it allows you to write extremely bad code that happens to work. It allows you depend on bugs in specific versions of bash, like interactions between empty arrays,
set -u, and var ref / name ref.The semantics of those features aren’t documented, and they change based on the winds of the bash codebase.
(This phenomena also emerges in Google’s monorepo with Bazel – except there are full-time engineers that clean up the code there. They mainly clean up C++ and Java, e.g. as part of doing toolchain upgrades.)
Nix uses some of the worst shell I’ve ever seen, and I’ve looked at many distros. I implemented a bunch of features for it in Oil, starting way back in 2017, prompted by Nix users:
https://github.com/oilshell/oil/issues/26
Also in 2021 there was another effort, prompted by Nix users:
http://www.oilshell.org/blog/2021/11/recent-progress.html#release-093-extended-globs-and-nix
I don’t think any of it is fundamental. Through git blame, I stumbled across a ~600 line version of the script from 2006 or so:
https://github.com/NixOS/nixpkgs/commit/f1166e0bbbc5b2ab0b39a622499223dcfe9b467a
It looks fine and simple (for a shell script). Somewhere along the way every last bash misfeature was added to that script, and I fail to see any reason why. The code is also structured pretty poorly.
If anyone wants to do something about this mess, rather than piling more of a mess on top, you could implement
nocasematchin OSH (the most bash-compatible shell by a mile), which is basically some flags tofnmatch()andglob(), aided by extensive tests:https://github.com/oilshell/oil/issues/701
FWIW Oil has a nascent component Hay that allows you to define interleaved structured data and shell code – exactly what you need to building distros.
https://www.oilshell.org/release/latest/doc/hay.html
OSH implements most of the bash string nonsense so that … one day … in a distant future … we can jettison it :-)
This is a neat runthrough! It takes a look at the same concepts of Nix Pill 7 (Working Derivation) from another angle and with a more complex build - I think your post does a better job at making it understandable though.
This is a nice write up, I learned quite a few things as a long time Nix user! I just have a nit to pick:
That’s true,
derivationis a function, but a derivation is not a function, it’s that data structure you see in that.drvfile, it’s a recipe for building something.derivationis a function that constructs a derivation.I loved how the “magic” of stdenv was explained.
I wonder if that 1600 lines setup.sh script could be simplified? Or at least improved in a way that it would be possible to explain what it does and why.
Yes! This is the good stuff. This is what Nix needs more of - explanations of how to make new things with it, rather than how to use things that have already been made. Tutorials about how to use
nix run programare not really Nix tutorials; they don’t teach you how to use Nix, they teach you how to invoke it. It’s easy to invoke things, but it’s much harder to build invokable things that did not already exist.Good for insight into what Nix “does”! I admit to never really digging into the build scripts this deeply!
However, I think there is a lot of cool stuff in those derivation files that are important. The Nix store is a really important part of how Nix works!
A key thing about Nix is that those derivation files contain references to all the inputs for a build so a hash of that file will point to a unique build and any changes to the build will have a different hash (I think that’s the hash in the path). The system knows it only has to rebuild things when the drv files change and old derivations will continue to point at their old inputs so you can actually have two versions of a package on disk and they can even depend on different versions of a library without conflicting.