I was complaining yesterday about how there are no Nix packaging tutorials, and lo and behold, someone blogging their Nix experience that actually has packaging in it!
Replace the first statement with import <nixpkgs> {}; I don’t know why you have to do this, but it works.
I basically just wrote this by copying and pasting from other packages in the nixpkgs repository.
I figured out how to set installFlags just by running rg PREFIX in the nixpkgs repository
replace with stdenv.lib to with lib for some reason.
I figured out how to run this by running rg 'mv ' in the nixpkgs repository and just copying and modifying something that seemed related.
Yeah, I guess that’s what I should have expected.
Random thought, but I wonder if Nix needs to have fewer defaults in stdenv. Currently, it seems like all the Nix packaging tutorials use hello or some other simple C program because stdenv is C-oriented. This makes it look like it has fewer moving parts than it really does, which does new users a disservice. It’s harder to understand, precisely, what things Nix is doing when all the builder steps are hidden, and understanding those is critical to having generalizable knowledge about using Nix to build software. This is doubly important when new users try to package something in a language other than C, because then they have to set up things themselves that were not only never explained, but never even appeared in the mkDerivation calls in their tutorials.
Of course, the die has been cast, since the source code in “search for source code that solves an adjacent problem” is already written with the existing defaults.
I think part of the problem with all of these are that they assume any familiarity with existing software norms around building/installing/packaging software in different ecosystems.
Disemboweling the stdenv and showing the gore in the derivation would make it much clearer what’s going on for people who know how to read the entrails, but I imagine it’s still going to be greek to someone who doesn’t know the make/configure/automake patterns, compiler/linker flags, build system, etc. in use.
I guess there’s tension between baiting people in by making golden-path things easy, and intimidating them from ever trying by requiring an understanding of the toolchains for the basics. That understanding is obviously helpful once the leaky abstractions leak, and I too have spent plenty of time rage-flailing my way through problems with projects that don’t square with undiscussed/taken-for-granted patterns.
I’ve had a thought that might be related, which is whether it’d be a good time investment for the Nix community to ~compile a broad resource on toolchain/packaging patterns for well-behaved software in all ecosystems, both as a resource for people in the community who need to understand the patterns (and some anti-patterns…) to package something, and maybe as a resource to help project maintainers in many ecosystems quickly figure out how to structure their project to “go with the flow”?
It might be cool to have something a bit like explainshell for a Nix derivation that can create an interactively-explorable hierarchy, from the expression level down to the shell functions the builder will be using, so that people can drill down into what it’s doing and why. Sounds like a lot of work though :)
Disemboweling the stdenv and showing the gore in the derivation would make it much clearer what’s going on for people who know how to read the entrails, but I imagine it’s still going to be greek to someone who doesn’t know the make/configure/automake patterns, compiler/linker flags, build system, etc. in use.
I just want to compliment your choice of metaphor here.
You’re right that someone with no packaging experience, whose problem requires adding a compiler flag, will likely be just as confused by “where do I add the compiler flag in mkDerivation” as “what is a compiler flag” and “which compiler flag do I need”. But maybe a tutorial for “How to package with Nix” should resign itself to teaching the “how to package” parts as well as the “with nix” part. There’s got to be enough general teachable material about building, installing, libraries, documentation, etc. to cover without having to get into area-specific bits of wisdom.
It might be cool to have something a bit like explainshell for a Nix derivation
I think I’d settle for any kind of intellisense. Maybe I haven’t found the right extensions, but I feel like the most assistance I get writing Nix is syntax coloring, which just feels bad in 2023.
I’d personally love to see a guide to nix packaging that starts by going into the guts. I’m not an autotools expert or anything, but I’ve compiled many things in my life and it would be very helpful to see what’s going on under the hood. I think there are a lot of people out there who know what a compiler flag and a linker flag are but do not understand nix :)
I think ianthehenry’s series of learning Nix with only the manuals and experimentation did a fair job of diving into internals when he needed to understand something, though it’s far from being a systematic walk through all the aspects of nix packaging. I don’t think I could have figured out what he did without asking someone for help.
I think it would be reasonable to require autotools-based packages to have to use a pkgs.buildAutotoolsPackage wrapper (a la pkgs.buildPythonPackage) instead of having stdenv just magically do things when using mkDerivation. That way, you can direct new packagers to see how buildAutotoolsPackage is implemented using the mkDerivation primitive.
Or, maybe setup hooks should be replaced with a “mixin” system so they aren’t magically called when you add a package to buildInputs, and the existing magic in stdenv should be moved to separate mixins. (e.g. pkgs.stdenv.mkDerivation { mixins = [ pkgs.autotools.passthru.mixin ]; })
I think part of the problem with all of these are that they assume any familiarity with existing software norms around building/installing/packaging software in different ecosystems.
one of the tings that really helped me with getting how to build packages with nix was learning about autotools in depth and the norms they implement, which definitely supports this idea.
Unfortunately, https://www.gnu.org/software/automake/manual/automake.html as the first step to getting nix builds is quite far from appetizing.
I sort of got that idea from some comments, but not sure if it’s exactly true.
If it’s true, it would probably help my mental model a little.
I agree that the core distro-building patterns, aside from Nix, are extremely confusing themselves … I think a big mystery to me has been pkg-config. It seems like an informal set of conventions.
And I just built a tool uftrace from scratch. It uses pkg-config , but that’s actually WRONG on Debian/Ubuntu, because it gets libpython3.7.so, while the actual library name is libpython3.7m.so.1.0 (note both the m for pymalloc, and 1.0 version, which I guess is ABI version)
Instead you are supposed to use python3-config on those distros ! Not even sure where that comes from, or when it arose
I patched this when building from source, but I need to file a bug about this …
So I have been using Python for ~20 years now, and this still utterly confuses me … i.e. loading the Python interpreter as a shared library on a distro. When you add in Nix, it gets even more confusing
a console log where I “built” oil (it just fetched from cache instead of building and this created a result symlink in my current directory to the oil in the nix store)
a derivation.json that shows the output of nix show-derivation result/
a default-builder.sh that’s a copy of what you can see specified at the top of the derivation in the args/builder keys
a stdenv_setup.sh (a bit of a fictional name for gist restrictions and syntax highlighting) that shows what the default-builder.sh would be sourcing
This isn’t the whole story. The rest of the derivation also factors into how the environment is set up before running, for example.
Thanks, that’s very helpful. Definitely should be a blog post :) I have never seen that despite years of Nix-related stories on lobste.rs.
Some people might not be able to read shell, but there are many who can.
Though the setup script is a bit messy, it doesn’t look as bad as I thought. I would say it’s closer to generating JSON than generating shell, though both are involved.
I can sort of see the JSON is used to set up what’s “visible” to the build.
It also points to the tiny default-builder.sh, which sources some attributes, then sources $stdenv/setup – and I guess the latter is fixed across all packages?
And I guess you can choose genericBuild or some other function to call from it, on a per package basis.
And there are lots of other hooks to override in the package.
I’m actually building a mini containerized “semi-distro” now for Oil’s build deps :-) And I’ve look at a lot of other distros.
It also points to the tiny default-builder.sh, which sources some attributes, then sources $stdenv/setup – and I guess the latter is fixed across all packages?
It isn’t fixed, though I think it is probably close in practice? Most derivations don’t explicitly set their own builder, but there are still a fair number that do.
Of those, most still source $stdenv/setup and use at least part of it. But I could also find at least two special cases that didn’t without looking too hard.
The author mentioned switching to Nix primarily because Homebrew is too slow. This was also a complaint of mine until I discovered HOMEBREW_INSTALL_FROM_API, an environment variable that, when set, causes Homebrew to forego the git repo and use a JSON package index fetched over HTTP. This makes Homebrew substantially faster, and is also the default in Homebrew 4.0 (released in the last few weeks).
There is also an option to disable Homebrew from automatically upgrading dependent packages when another package is upgraded. Setting that helped with performance a lot too.
After I discovered these options all of my complaints about Homebrew went away. I’m now very happy with it, and spend approximately zero time fighting with my package manager. I cannot say the same about my experience with Nix.
The Nix programming language is a purely-functional programming language, not a configuration language. It’s used to program the Nix package manager in the same way as any other programming language would. It just happens to be good at configuration :^)
I actually made the opposite move a few days ago. I switched over to Nix on my work laptop so I could access fresh packages with a faster install process, but after the recent performance improvements in the Homebrew 4.0.0 release, I ended up switching back. Don’t get me wrong, I find the Nix language truly delightful, I think flakes are brilliant, and I don’t mind the new Nix CLI, but I rarely find myself reaching for any of Nix’s special features. Even when I inevitably end up needing an ancient version of some random package for a work project, I usually end up using an environment manager like Conda/Mamba or a container manager like Docker/Podman so that I can more easily share my configuration with my Nix-averse colleagues. Moreover, I’m not a big fan of how often Nix installs unique versions of the same dependency for every new package I install, and I’ve found that the Homebrew equivalents of most packages tend to be kept better up-to-date and, more importantly, ship with better defaults (e.g., compilation flags, environment variables, and so forth). Similarly, I tried getting into Home Manager, but I failed to grok the advantage over Git-versioned dotfiles.
For context, at home and on personal servers, I satisfy my desire for fresh packages and speedy installations with pacman, either via Arch Linux or JuNest, and I scarcely require ancient software given how I manage dependencies in my personal projects (i.e., I’m happy to rewrite code to accommodate breaking changes).
I love Nix, and I want to need it, but I just don’t see how it improves my workflow.
Similarly, I tried getting into Home Manager, but I failed to grok the advantage over Git-versioned dotfiles.
+1 (and I do use NixOS). I can see a fundamental benefit of Home Manager for multi-user setup, as it allows declarative management of per-use environments. But, in typical desktop systems with single user, just stuff packages into system environment?
A non-fundamental benefit of HM is that a bunch of people use it, so it probably has convenient configuration models for all kinds of software, which is a benefit, even if you don’t care about user packages per se.
The reason I use home-manager is to have the same setup among all the machines I work on. It’s really great that my settings and favorite tools follow me.
I don’t think I’d use NixOS as a daily driver (not the least because my work uses a proprietary VPN client that doesn’t work on Linux), but I do run it on my home servers and it’s really great at that. I can install and uninstall services I want to try and feel confident that I’m not leaving config gunk lying around. I think I’d use a Nix profile on Ubuntu, Mint, or Arch if I wanted to get more into it.
I was complaining yesterday about how there are no Nix packaging tutorials, and lo and behold, someone blogging their Nix experience that actually has packaging in it!
Yeah, I guess that’s what I should have expected.
Random thought, but I wonder if Nix needs to have fewer defaults in
stdenv. Currently, it seems like all the Nix packaging tutorials usehelloor some other simple C program becausestdenvis C-oriented. This makes it look like it has fewer moving parts than it really does, which does new users a disservice. It’s harder to understand, precisely, what things Nix is doing when all the builder steps are hidden, and understanding those is critical to having generalizable knowledge about using Nix to build software. This is doubly important when new users try to package something in a language other than C, because then they have to set up things themselves that were not only never explained, but never even appeared in themkDerivationcalls in their tutorials.Of course, the die has been cast, since the source code in “search for source code that solves an adjacent problem” is already written with the existing defaults.
I think part of the problem with all of these are that they assume any familiarity with existing software norms around building/installing/packaging software in different ecosystems.
Disemboweling the stdenv and showing the gore in the derivation would make it much clearer what’s going on for people who know how to read the entrails, but I imagine it’s still going to be greek to someone who doesn’t know the make/configure/automake patterns, compiler/linker flags, build system, etc. in use.
I guess there’s tension between baiting people in by making golden-path things easy, and intimidating them from ever trying by requiring an understanding of the toolchains for the basics. That understanding is obviously helpful once the leaky abstractions leak, and I too have spent plenty of time rage-flailing my way through problems with projects that don’t square with undiscussed/taken-for-granted patterns.
I’ve had a thought that might be related, which is whether it’d be a good time investment for the Nix community to ~compile a broad resource on toolchain/packaging patterns for well-behaved software in all ecosystems, both as a resource for people in the community who need to understand the patterns (and some anti-patterns…) to package something, and maybe as a resource to help project maintainers in many ecosystems quickly figure out how to structure their project to “go with the flow”?
It might be cool to have something a bit like explainshell for a Nix derivation that can create an interactively-explorable hierarchy, from the expression level down to the shell functions the builder will be using, so that people can drill down into what it’s doing and why. Sounds like a lot of work though :)
I just want to compliment your choice of metaphor here.
You’re right that someone with no packaging experience, whose problem requires adding a compiler flag, will likely be just as confused by “where do I add the compiler flag in
mkDerivation” as “what is a compiler flag” and “which compiler flag do I need”. But maybe a tutorial for “How to package with Nix” should resign itself to teaching the “how to package” parts as well as the “with nix” part. There’s got to be enough general teachable material about building, installing, libraries, documentation, etc. to cover without having to get into area-specific bits of wisdom.I think I’d settle for any kind of intellisense. Maybe I haven’t found the right extensions, but I feel like the most assistance I get writing Nix is syntax coloring, which just feels bad in 2023.
I’d personally love to see a guide to nix packaging that starts by going into the guts. I’m not an autotools expert or anything, but I’ve compiled many things in my life and it would be very helpful to see what’s going on under the hood. I think there are a lot of people out there who know what a compiler flag and a linker flag are but do not understand nix :)
I tried to look through this 1600-line setup.sh bash script that seems to be at the heart of nix builds, but it’s a lot to process.
I think ianthehenry’s series of learning Nix with only the manuals and experimentation did a fair job of diving into internals when he needed to understand something, though it’s far from being a systematic walk through all the aspects of nix packaging. I don’t think I could have figured out what he did without asking someone for help.
If you want to understand how (and why) things work under the hook your best bet is to go over the “Nix pills” -> https://nixos.org/guides/nix-pills/
I think it would be reasonable to require autotools-based packages to have to use a
pkgs.buildAutotoolsPackagewrapper (a lapkgs.buildPythonPackage) instead of havingstdenvjust magically do things when usingmkDerivation. That way, you can direct new packagers to see howbuildAutotoolsPackageis implemented using themkDerivationprimitive.Or, maybe setup hooks should be replaced with a “mixin” system so they aren’t magically called when you add a package to
buildInputs, and the existing magic instdenvshould be moved to separate mixins. (e.g.pkgs.stdenv.mkDerivation { mixins = [ pkgs.autotools.passthru.mixin ]; })I think this mixins idea is how Guix does it, IIRC, and it seems much more explicit and sane.
one of the tings that really helped me with getting how to build packages with nix was learning about autotools in depth and the norms they implement, which definitely supports this idea. Unfortunately, https://www.gnu.org/software/automake/manual/automake.html as the first step to getting nix builds is quite far from appetizing.
Does a Nix derivation evaluate to a shell script that calls some pre-baked shell functions? i.e. the ones pointed at below:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh
I sort of got that idea from some comments, but not sure if it’s exactly true.
If it’s true, it would probably help my mental model a little.
I agree that the core distro-building patterns, aside from Nix, are extremely confusing themselves … I think a big mystery to me has been
pkg-config. It seems like an informal set of conventions.And I just built a tool uftrace from scratch. It uses
pkg-config, but that’s actually WRONG on Debian/Ubuntu, because it getslibpython3.7.so, while the actual library name islibpython3.7m.so.1.0(note both themfor pymalloc, and1.0version, which I guess is ABI version)Instead you are supposed to use
python3-configon those distros ! Not even sure where that comes from, or when it aroseI patched this when building from source, but I need to file a bug about this …
So I have been using Python for ~20 years now, and this still utterly confuses me … i.e. loading the Python interpreter as a shared library on a distro. When you add in Nix, it gets even more confusing
Pretty much. Here’s a gist (https://gist.github.com/abathur/917d722a16c65824663d762cdd2f4d22) with:
resultsymlink in my current directory to the oil in the nix store)nix show-derivation result/This isn’t the whole story. The rest of the derivation also factors into how the environment is set up before running, for example.
Thanks, that’s very helpful. Definitely should be a blog post :) I have never seen that despite years of Nix-related stories on lobste.rs.
Some people might not be able to read shell, but there are many who can.
Though the setup script is a bit messy, it doesn’t look as bad as I thought. I would say it’s closer to generating JSON than generating shell, though both are involved.
I can sort of see the JSON is used to set up what’s “visible” to the build.
It also points to the tiny default-builder.sh, which sources some attributes, then sources
$stdenv/setup– and I guess the latter is fixed across all packages?And I guess you can choose genericBuild or some other function to call from it, on a per package basis.
And there are lots of other hooks to override in the package.
I’m actually building a mini containerized “semi-distro” now for Oil’s build deps :-) And I’ve look at a lot of other distros.
Some parts of the oft-linked Nix Pills overlap with this. Particularly:
It isn’t fixed, though I think it is probably close in practice? Most derivations don’t explicitly set their own builder, but there are still a fair number that do.
Of those, most still source $stdenv/setup and use at least part of it. But I could also find at least two special cases that didn’t without looking too hard.
Julia of course turned around and wrote a very approachable version of this :)
https://lobste.rs/s/isligm
Yup, great post! I think the difference is that it shows all the error messages you get if you DON’T do the right thing, which helps comprehension
I’m used to seeing all those same errors when building software from scratch
The author mentioned switching to Nix primarily because Homebrew is too slow. This was also a complaint of mine until I discovered HOMEBREW_INSTALL_FROM_API, an environment variable that, when set, causes Homebrew to forego the git repo and use a JSON package index fetched over HTTP. This makes Homebrew substantially faster, and is also the default in Homebrew 4.0 (released in the last few weeks).
There is also an option to disable Homebrew from automatically upgrading dependent packages when another package is upgraded. Setting that helped with performance a lot too.
After I discovered these options all of my complaints about Homebrew went away. I’m now very happy with it, and spend approximately zero time fighting with my package manager. I cannot say the same about my experience with Nix.
My shower thought on Nix the other day was “they’re trying to boil three different oceans, and I really only want them to boil one.”
can you elaborate on what this means?
boil the ocean
Which is the one you want?
Packaging binaries. The other oceans are cool too, but I’m sort of in Julia’s camp of just wanting to install some stuff.
The Nix programming language is a purely-functional programming language, not a configuration language. It’s used to program the Nix package manager in the same way as any other programming language would. It just happens to be good at configuration :^)
You can get Nixpkgs to run on any other Linux distro (and on Mac too), so you don’t need to boil that third ocean if you don’t want to.
Most likely an old version of the profile still holds a reference to it. There’s a store query command to check for those.
I actually made the opposite move a few days ago. I switched over to Nix on my work laptop so I could access fresh packages with a faster install process, but after the recent performance improvements in the Homebrew 4.0.0 release, I ended up switching back. Don’t get me wrong, I find the Nix language truly delightful, I think flakes are brilliant, and I don’t mind the new Nix CLI, but I rarely find myself reaching for any of Nix’s special features. Even when I inevitably end up needing an ancient version of some random package for a work project, I usually end up using an environment manager like Conda/Mamba or a container manager like Docker/Podman so that I can more easily share my configuration with my Nix-averse colleagues. Moreover, I’m not a big fan of how often Nix installs unique versions of the same dependency for every new package I install, and I’ve found that the Homebrew equivalents of most packages tend to be kept better up-to-date and, more importantly, ship with better defaults (e.g., compilation flags, environment variables, and so forth). Similarly, I tried getting into Home Manager, but I failed to grok the advantage over Git-versioned dotfiles.
For context, at home and on personal servers, I satisfy my desire for fresh packages and speedy installations with pacman, either via Arch Linux or JuNest, and I scarcely require ancient software given how I manage dependencies in my personal projects (i.e., I’m happy to rewrite code to accommodate breaking changes).
I love Nix, and I want to need it, but I just don’t see how it improves my workflow.
+1 (and I do use NixOS). I can see a fundamental benefit of Home Manager for multi-user setup, as it allows declarative management of per-use environments. But, in typical desktop systems with single user, just stuff packages into system environment?
A non-fundamental benefit of HM is that a bunch of people use it, so it probably has convenient configuration models for all kinds of software, which is a benefit, even if you don’t care about user packages per se.
The reason I use home-manager is to have the same setup among all the machines I work on. It’s really great that my settings and favorite tools follow me.
I do the same, but without home manager:
~, I store then in the same repo as flake.nixI don’t think I’d use NixOS as a daily driver (not the least because my work uses a proprietary VPN client that doesn’t work on Linux), but I do run it on my home servers and it’s really great at that. I can install and uninstall services I want to try and feel confident that I’m not leaving config gunk lying around. I think I’d use a Nix profile on Ubuntu, Mint, or Arch if I wanted to get more into it.