This looks great! I’m firmly in the camp that believes that the lack of an official, non-experimental way to cache the eval-time outputs of Nix expressions is the biggest roadblock to Nix’s wider adoption.
Most Nix users will never feel the need for it and it’s a challenge even to describe what it is exactly that one can’t currently do without going into deep experimental territory, but as this post points out, it’s a blocker for the seamless implementation of tools like ${LANG}2nix that everyone would use if they didn’t have so much friction.
That said, I can’t tell what exactly this new experimental dynamic-derivations feature enables that one couldn’t get from recursive-nix before. Sure, you couldn’t output a .drv file from recursive-nix evaluations, but you COULD output the outputs of recursively built derivations. Here’s an example flake I’ve written for my previous job in order to wrap haskell.nix derivations in such a way that they could be fetched from the binary cache without having to fetch and evaluate the eval-time dependencies of haskell.nix (more context). In particular, you can see on line 159 that we’re able to symlink $out to the output of the inner nix-build:
ln -s $(nix-build default.nix) $out
Don’t get me wrong, my example does very shady stuff to work around recursive-nix‘s restrictions and it’s prone to triggering some weird Nix bugs around recursive-nix when used too creatively. But those issues are all about recursive-nix and not the restriction around emitting .drv files as outputs. So what was it that you couldn’t do by forwarding the outputs of recursive derivations and now you can do by forwarding the .drvs of recursive derivations?
Yes, you can just build with recursive Nix. But that makes for less nice caching, parallelism, dry-rub etc. in many cases. Returning derivations gives the scheduler more latitude while being just as higher order.
Put another way, returning derivations is more declarative.
Thank you for the response, I think I understand a little better now, but I suspect that the post does dynamic-derivations injustice by bringing recursive-nix into the picture. Because recursive-nix is a nuclear option that has a lot of rough edges (both while writing the derivations, but also while consuming them) and it already makes a superset of dynamic-derivations’s use cases possible.
I think a better demonstration of the benefits of dynamic-derivations would be to assemble a .drv file as part of the build without using nix-instantiate, but using other command-line applications (like a language’s package manager). In that case dynamic-derivations would allow us to do something that we couldn’t do before (Namely, using a non-nix executable to define a derivation) using neither IFD nor recursive-nix, which both have problems in different ways.
I feel like I’m missing something. A lot of IFD or pre-generation works something like:
Generate a Nix overlay or attrset from Cargo.{lock,nix}/poetry.lock/whatever (or just JSON that can be loaded into Nix).
Load the pre-generated set (or use IFD) during eval.
Compose the overlay with another overlay with overrides. This bit is really important, since packages often don’t build without additional build inputs (e.g. additional libraries).
Turn into an overridable set.
IFD delays evaluation because generating the Nix is work (and might involve something like running cargo metadata). As far as I understand, this moves the expensive step from evaluation to the build. However, how would this work if the thing we are generating isn’t some text file that we just want to copy to the output, but Nix code that we need to evaluate? Shouldn’t the generation be part of the evaluation phase somehow?
This is for many things not an alternative to IFD, since we still can not build derivations and then use them at evaluation time, meaning that you can’t have an attribute set whose contents are determined by some build, and then access that attribute set outside of build that dependens on that derivation.
This looks great! I’m firmly in the camp that believes that the lack of an official, non-experimental way to cache the eval-time outputs of Nix expressions is the biggest roadblock to Nix’s wider adoption.
Most Nix users will never feel the need for it and it’s a challenge even to describe what it is exactly that one can’t currently do without going into deep experimental territory, but as this post points out, it’s a blocker for the seamless implementation of tools like
${LANG}2nixthat everyone would use if they didn’t have so much friction.That said, I can’t tell what exactly this new experimental
dynamic-derivationsfeature enables that one couldn’t get fromrecursive-nixbefore. Sure, you couldn’t output a.drvfile fromrecursive-nixevaluations, but you COULD output the outputs of recursively built derivations. Here’s an example flake I’ve written for my previous job in order to wraphaskell.nixderivations in such a way that they could be fetched from the binary cache without having to fetch and evaluate the eval-time dependencies ofhaskell.nix(more context). In particular, you can see on line 159 that we’re able to symlink$outto the output of the innernix-build:Don’t get me wrong, my example does very shady stuff to work around
recursive-nix‘s restrictions and it’s prone to triggering some weird Nix bugs aroundrecursive-nixwhen used too creatively. But those issues are all aboutrecursive-nixand not the restriction around emitting.drvfiles as outputs. So what was it that you couldn’t do by forwarding the outputs of recursive derivations and now you can do by forwarding the.drvs of recursive derivations?Yes, you can just build with recursive Nix. But that makes for less nice caching, parallelism, dry-rub etc. in many cases. Returning derivations gives the scheduler more latitude while being just as higher order.
Put another way, returning derivations is more declarative.
Thank you for the response, I think I understand a little better now, but I suspect that the post does
dynamic-derivationsinjustice by bringingrecursive-nixinto the picture. Becauserecursive-nixis a nuclear option that has a lot of rough edges (both while writing the derivations, but also while consuming them) and it already makes a superset ofdynamic-derivations’s use cases possible.I think a better demonstration of the benefits of
dynamic-derivationswould be to assemble a.drvfile as part of the build without usingnix-instantiate, but using other command-line applications (like a language’s package manager). In that casedynamic-derivationswould allow us to do something that we couldn’t do before (Namely, using a non-nixexecutable to define a derivation) using neither IFD norrecursive-nix, which both have problems in different ways.This is neat. Hopefully it doesn’t go the way of other nix experimental features.
I assume you mean flakes and nix-command? There was a roadmap discussed at planetnix to stop these from being experiments this year.
I feel like I’m missing something. A lot of IFD or pre-generation works something like:
Cargo.{lock,nix}/poetry.lock/whatever (or just JSON that can be loaded into Nix).IFD delays evaluation because generating the Nix is work (and might involve something like running
cargo metadata). As far as I understand, this moves the expensive step from evaluation to the build. However, how would this work if the thing we are generating isn’t some text file that we just want to copy to the output, but Nix code that we need to evaluate? Shouldn’t the generation be part of the evaluation phase somehow?Hmmm, from the RFC:
https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dynamism.md