This looks great! I’m firmly in the camp that believes that the lack of an official, non-experimental way to cache the eval-time outputs of Nix expressions is the biggest roadblock to Nix’s wider adoption.
Most Nix users will never feel the need for it and it’s a challenge even to describe what it is exactly that one can’t currently do without going into deep experimental territory, but as this post points out, it’s a blocker for the seamless implementation of tools like ${LANG}2nix that everyone would use if they didn’t have so much friction.
That said, I can’t tell what exactly this new experimental dynamic-derivations feature enables that one couldn’t get from recursive-nix before. Sure, you couldn’t output a .drv file from recursive-nix evaluations, but you COULD output the outputs of recursively built derivations. Here’s an example flake I’ve written for my previous job in order to wrap haskell.nix derivations in such a way that they could be fetched from the binary cache without having to fetch and evaluate the eval-time dependencies of haskell.nix (more context). In particular, you can see on line 159 that we’re able to symlink $out to the output of the inner nix-build:
ln -s $(nix-build default.nix) $out
Don’t get me wrong, my example does very shady stuff to work around recursive-nix‘s restrictions and it’s prone to triggering some weird Nix bugs around recursive-nix when used too creatively. But those issues are all about recursive-nix and not the restriction around emitting .drv files as outputs. So what was it that you couldn’t do by forwarding the outputs of recursive derivations and now you can do by forwarding the .drvs of recursive derivations?
I feel like I’m missing something. A lot of IFD or pre-generation works something like:
Generate a Nix overlay or attrset from Cargo.{lock,nix}/poetry.lock/whatever (or just JSON that can be loaded into Nix).
Load the pre-generated set (or use IFD) during eval.
Compose the overlay with another overlay with overrides. This bit is really important, since packages often don’t build without additional build inputs (e.g. additional libraries).
Turn into an overridable set.
IFD delays evaluation because generating the Nix is work (and might involve something like running cargo metadata). As far as I understand, this moves the expensive step from evaluation to the build. However, how would this work if the thing we are generating isn’t some text file that we just want to copy to the output, but Nix code that we need to evaluate? Shouldn’t the generation be part of the evaluation phase somehow?
This is for many things not an alternative to IFD, since we still can not build derivations and then use them at evaluation time, meaning that you can’t have an attribute set whose contents are determined by some build, and then access that attribute set outside of build that dependens on that derivation.
This looks great! I’m firmly in the camp that believes that the lack of an official, non-experimental way to cache the eval-time outputs of Nix expressions is the biggest roadblock to Nix’s wider adoption.
Most Nix users will never feel the need for it and it’s a challenge even to describe what it is exactly that one can’t currently do without going into deep experimental territory, but as this post points out, it’s a blocker for the seamless implementation of tools like
${LANG}2nixthat everyone would use if they didn’t have so much friction.That said, I can’t tell what exactly this new experimental
dynamic-derivationsfeature enables that one couldn’t get fromrecursive-nixbefore. Sure, you couldn’t output a.drvfile fromrecursive-nixevaluations, but you COULD output the outputs of recursively built derivations. Here’s an example flake I’ve written for my previous job in order to wraphaskell.nixderivations in such a way that they could be fetched from the binary cache without having to fetch and evaluate the eval-time dependencies ofhaskell.nix(more context). In particular, you can see on line 159 that we’re able to symlink$outto the output of the innernix-build:Don’t get me wrong, my example does very shady stuff to work around
recursive-nix‘s restrictions and it’s prone to triggering some weird Nix bugs aroundrecursive-nixwhen used too creatively. But those issues are all aboutrecursive-nixand not the restriction around emitting.drvfiles as outputs. So what was it that you couldn’t do by forwarding the outputs of recursive derivations and now you can do by forwarding the.drvs of recursive derivations?This is neat. Hopefully it doesn’t go the way of other nix experimental features.
I assume you mean flakes and nix-command? There was a roadmap discussed at planetnix to stop these from being experiments this year.
I feel like I’m missing something. A lot of IFD or pre-generation works something like:
Cargo.{lock,nix}/poetry.lock/whatever (or just JSON that can be loaded into Nix).IFD delays evaluation because generating the Nix is work (and might involve something like running
cargo metadata). As far as I understand, this moves the expensive step from evaluation to the build. However, how would this work if the thing we are generating isn’t some text file that we just want to copy to the output, but Nix code that we need to evaluate? Shouldn’t the generation be part of the evaluation phase somehow?Hmmm, from the RFC:
https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dynamism.md