How would replacing bash with nushell play with bootstrapping of nix and nixpkgs? When comparing guix and nix, guix did quite a good job on that topic and there is really a minimal set of packages to build everything from scratch. I’m wondering if bringing Rust in, just to build nushell, just to build stdenv based on it, would make bootstrapping nearly impossible.
Adopting this as a default stdenv would require to push this massive bindist to the Nixpkgs bootstrap seed. That seed is already massive compared to what Guix has, I don’t think we want to degrade this further.
Rust is definitely source-bootstrapable, as a matter of fact, Guix manages to do it, there’s no reason we can’t do the same. The bootstrap chain is pretty long though. On top of what we already bootstrap (gcc, openssl, etc.), we’d need to bootstrap llvm, mrustc, then rust 1.54 -> 55 -> 56 -> 57 -> 58 -> 60 -> 61 -> 62 -> 63 -> 64 -> 65.
So yeah, pushing this to stdenv would considerably degrade the bootstrap story in any case.
From my perspective, Bash is a local optimum, I personally wouldn’t change it, it’s certainly a good balance between a language that is easy to bootstrap and a good-enough expressiveness to express builds.
If we really want to move to something more modern, Oil could be a more serious contender, they seem to take bootstrapping seriously. There’s a drafted RFC wrt. Oil adoption.
[Edit]: The question is eluded, but I don’t think the author expects this to replace stdenv, at least it’s not mentionned in the article. Don’t take this comment as an overwhelming negative “this project is worthless”. Cool hack!
This made me realize that Rust is fundamentally non-bootstrapable. It’s definitely going to produce a release every six weeks for quite a number of years, and Rust’s release N needs release N-1 to build, so the bootstrap chain, by design, grows quickly and linearly with time. So it seems that, in the limit, it is really a choice between:
Is there a reference interpreter, perhaps? I imagine that that can’t be a complete solution since LLVM is a dependency, but it would allow pure-Rust toolchains to periodically adjust their bootstraps, so to speak.
There is mrustc which is written in C++ and allows you to bootstrap Rust. There is also GCC Rust implementation in the works, that will allow bootstrapping.
In my defense, I do use the term “experimental” several times and I don’t make any suggestion of replacing stdenv. I could be wrong, but I think that flakes are going to decentralize the Nix ecosystem quite a bit. While the Nixpkgs/stdenv relationship is seemingly ironclad, I don’t see why orgs or subsets of the Nix community couldn’t adopt alternative builders. Any given Nix closure can in principle have N builders involved; they’re all just producing filesystem state after all.
As for bootstrapping, yes, the cost of something like Nushell/Nuenv is certainly higher than Bash, but it’s worth considering that (a) you don’t need things like curl, jq, and coreutils and (b) one could imagine using Rust feature flags to produce lighter-weight distributions of Nushell that cut out things that aren’t germane to a Nix build environment (like DataFrames support).
The resulting binary is about 1.3 MB now. I seem to recall that the nushell binary is something like 10 MB or 50 MB, which is typical for Rust binaries. rustc is probably much larger.
There was some debate about whether Oil’s code gen needs to be bootstrapped. That is possible, but lots of people didn’t seem to realize that the bash build in Nix is not.
It includes yacc’s generated output, which is less readable than most of Oil’s generated code.
Nushell is certainly larger! Although I would be curious how much smaller Nushell could be made. Given that it’s Rust, you could in principle use feature flags to bake in only those features that a Nix builder would be likely to use and leave out things like DataFrames support (which is quite likely the “heaviest” part of Nushell).
For sure it would make bootstrapping much harder on things like OpenBSD. Good luck if you are on an arch that doesn’t have rust or LLVM. That said, I don’t think this would replace stdenv.. for sure not any time soon!
Also the article does mention exactly what you are pointing out:
it’s nowhere near the range of systems that can run Bash.
My question is orthogonal to this, and maybe I should have specify what I mean by bootstrapping. It’s “how many things I have to build first, before I can have working nixpkgs and build things users ask for”. So if we assume that nushell runs wherever bash runs, how much more effort is to build nushell (and rust and llvm) than bash? I would guess order of magnitude more, thus really complicating the initial setup of nixpkgs (or at least getting them to install without any caches).
How would replacing bash with nushell play with bootstrapping of nix and nixpkgs? When comparing guix and nix, guix did quite a good job on that topic and there is really a minimal set of packages to build everything from scratch. I’m wondering if bringing Rust in, just to build nushell, just to build stdenv based on it, would make bootstrapping nearly impossible.
100% agree, this article completely eludes this central question. Bash is semi-trivial to bootstrap!
Nixpkgs does not bootstrap rustc, we’re currently using a binary distribution: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/rust/rustc.nix#L28
Adopting this as a default stdenv would require to push this massive bindist to the Nixpkgs bootstrap seed. That seed is already massive compared to what Guix has, I don’t think we want to degrade this further.
Rust is definitely source-bootstrapable, as a matter of fact, Guix manages to do it, there’s no reason we can’t do the same. The bootstrap chain is pretty long though. On top of what we already bootstrap (gcc, openssl, etc.), we’d need to bootstrap llvm, mrustc, then rust 1.54 -> 55 -> 56 -> 57 -> 58 -> 60 -> 61 -> 62 -> 63 -> 64 -> 65.
So yeah, pushing this to stdenv would considerably degrade the bootstrap story in any case.
From my perspective, Bash is a local optimum, I personally wouldn’t change it, it’s certainly a good balance between a language that is easy to bootstrap and a good-enough expressiveness to express builds.
If we really want to move to something more modern, Oil could be a more serious contender, they seem to take bootstrapping seriously. There’s a drafted RFC wrt. Oil adoption.
[Edit]: The question is eluded, but I don’t think the author expects this to replace stdenv, at least it’s not mentionned in the article. Don’t take this comment as an overwhelming negative “this project is worthless”. Cool hack!
This made me realize that Rust is fundamentally non-bootstrapable. It’s definitely going to produce a release every six weeks for quite a number of years, and Rust’s release N needs release N-1 to build, so the bootstrap chain, by design, grows quickly and linearly with time. So it seems that, in the limit, it is really a choice between:
Is there a reference interpreter, perhaps? I imagine that that can’t be a complete solution since LLVM is a dependency, but it would allow pure-Rust toolchains to periodically adjust their bootstraps, so to speak.
There is mrustc which is written in C++ and allows you to bootstrap Rust. There is also GCC Rust implementation in the works, that will allow bootstrapping.
In my defense, I do use the term “experimental” several times and I don’t make any suggestion of replacing stdenv. I could be wrong, but I think that flakes are going to decentralize the Nix ecosystem quite a bit. While the Nixpkgs/stdenv relationship is seemingly ironclad, I don’t see why orgs or subsets of the Nix community couldn’t adopt alternative builders. Any given Nix closure can in principle have N builders involved; they’re all just producing filesystem state after all.
As for bootstrapping, yes, the cost of something like Nushell/Nuenv is certainly higher than Bash, but it’s worth considering that (a) you don’t need things like curl, jq, and coreutils and (b) one could imagine using Rust feature flags to produce lighter-weight distributions of Nushell that cut out things that aren’t germane to a Nix build environment (like DataFrames support).
Yes the new Oil C++ tarball is 375 kilobytes / 90K lines of compressed, readable C++ source :)
https://www.oilshell.org/release/0.14.2/
The resulting binary is about 1.3 MB now. I seem to recall that the nushell binary is something like 10 MB or 50 MB, which is typical for Rust binaries. rustc is probably much larger.
There was some debate about whether Oil’s code gen needs to be bootstrapped. That is possible, but lots of people didn’t seem to realize that the bash build in Nix is not.
It includes yacc’s generated output, which is less readable than most of Oil’s generated code.
Nushell is certainly larger! Although I would be curious how much smaller Nushell could be made. Given that it’s Rust, you could in principle use feature flags to bake in only those features that a Nix builder would be likely to use and leave out things like DataFrames support (which is quite likely the “heaviest” part of Nushell).
For sure it would make bootstrapping much harder on things like OpenBSD. Good luck if you are on an arch that doesn’t have rust or LLVM. That said, I don’t think this would replace stdenv.. for sure not any time soon!
Also the article does mention exactly what you are pointing out:
My question is orthogonal to this, and maybe I should have specify what I mean by bootstrapping. It’s “how many things I have to build first, before I can have working nixpkgs and build things users ask for”. So if we assume that nushell runs wherever bash runs, how much more effort is to build nushell (and rust and llvm) than bash? I would guess order of magnitude more, thus really complicating the initial setup of nixpkgs (or at least getting them to install without any caches).
The snippet above does not mention bootstrap.nu, only build.nu. Is it a bug in the blog post, or in my understanding of how it works?