Interesting post. The main issue of Nix is its onboarding curve and lack of simple-to-grok documentation.
There’s a few things in this post worth responding to:
Now, you may ask, how do you get that hash? Try and build the package with an obviously false hash and use the correct one from the output of the build command! That seems safe!
Nix has the prefetch-* commands that can do this for you and output either the hash, or a full Nix expression that refers to that thing.
I could avoid this by making each dependency its own Nix package, but that’s not a productive use of my time.
It depends. My personal view recently has been that Nix should adopt a more Bazel-like model, in which the Nix language is also used for describing the actual software builds rather than just wrapping external package managers.
I have implemented this for Go (buildGo.nix / docs) and Common Lisp (buildLisp.nix), and with the Go one specifically external dependencies can be traversed and automatically transformed into a Nix data structure.
For example, here’s the buildGo.nix packaging of golang.org/x/net (from here):
This makes every subpackage available as an individual Nix derivation, which also means that those builds are cached across different software using those dependencies.
this is at least 200 if not more packages needed for my relatively simple CRUD app that has creative choices in technology
For most mainstream languages generators have been written to wrap 3rd-party package managers automatically. For some languages (e.g. Python), the nixpkgs tree actually contains derivations for all packages already so it’s just a matter of dragging them in.
Oh, even better, the build directory isn’t writable.
This isn’t true for Nix in general. The build directory is explicitly writable and output installation (into /nix/store) usually (in Nix’s standard environment) happens as one of the last steps of a build.
It might be that this was a case of some language-specific tooling implementing such a restriction, in which case there’s probably also a flag for toggling it.
You know, the things that handle STATE, like FILES on the DISK. That’s STATE. GLOBALLY MUTABLE STATE.
The conceptual boundary is drawn differently here. In some sense, we look at the artefacts of realised derivations (i.e. completed “builds”) as a cache. The hashes you see in the /nix/store reference the inputs, not the outputs.
Also, nothing written to the store is mutated afterwards so for any given store there is mutability, but it is append-only.
As a side effect of making its package management system usable by normal users, it exposes the package manager database to corruption by any user mistake, curl2bash or malicious program on the system.
I’m not sure what is meant by this.
Edit 1: Ah, on the above this tweet adds some background (I think):
It doesn’t matter how PURE your definitions are; because the second some goddamn shell script does anything involving open(), you lose. The functional purity of your build is gone.
By default, Nix sandboxes builds which means that this is not a problem. Only explicitly declared dependencies are visible to a builder, and only the build directory and output path are writeable. Users can enable various footguns, such as opting out of sandboxing or whitelisting certain paths for passthrough.
It seems like sandboxing is available on Mac (thanks puck for pointing this out), but for users running Nix in single-user mode (which OP might be doing) there is currently some extra hoop-jumping required to make it work correctly.
In addition, on Linux, builds run in private PID, mount, network, IPC and UTS namespaces to isolate them from other processes in the system (except that fixed-output derivations do not run in private network namespace to ensure they can access the network).
It looks like on mac it’s just a chroot to hide store paths but you can still curl install.sh | bash in your build. I didn’t know it even had that much sandboxing on mac though, so thanks for pointing it.
You know, the things that handle STATE, like FILES on the DISK. That’s STATE. GLOBALLY MUTABLE STATE.
The conceptual boundary is drawn differently here. In some sense, we look at the artefacts of realised derivations (i.e. completed “builds”) as a cache. The hashes you see in the /nix/store reference the inputs, not the outputs.
Also, nothing written to the store is mutated afterwards so for any given store there is mutability, but it is append-only.
I really like this aspect of Nix: it’s like all packages exist in some platonic library of babel, and we copy a few of them into our /nix/store cache as they’re needed. This style of reasoning also fits with the language’s laziness, the design of nixpkgs (one huge set, whose contents are computed on-demand) and common patterns like taking fixed points to allow overrides (e.g. all of the function arguments called self).
A similar idea applies to content-addressable storage like IPFS, which I’m still waiting to be usable with Nix :(
Nix should adopt a more Bazel-like model, in which the Nix language is also used for describing the actual software builds rather than just wrapping external package managers.
Would that involve “recursive Nix” to allow builders to use Nix themselves, in order to build sub-components?
Recursive Nix is not necessary. For some languages this can already been done. E.g. the buildRustCrate function reimplements (most of) Cargo in Nix and does not use Cargo at all. This is in contrast to buildRustPackage, which relies on Cargo to do the builds.
You can convert a Cargo.lock file to a Nix expression with e.g. crate2nix and build crates using buildRustCrate. This has the same benefits as Nix has for other derivations: each compiled crate gets its own store path, so builds are incremental, and crate dependencies with the same version/features can be shared between derivations.
No, I’m not using recursive Nix for these. In my opinion (this might be controversial with some people) recursive Nix is a workaround for performance flaws of the current evaluator and I’d rather address those than add the massive amount of complexity required by recursive Nix.
What’s potentially more important (especially for slow compilers like GHC or rustc) is content-addressed store paths, which allow for early build cutoff if two differing inputs (e.g. changes in comments or minor refactorings) yield the same artefact. Work is already underway towards that.
Can you please edit documentation somewhere to note the existence of the prefetch commands and how to use them?
nix-prefetch-url is part of Nix itself and is documented here, nix-prefetch-git etc. come from another package in nixpkgs and I don’t think there’s any docs for them right now.
Nix has several large documentation issues and this being undocumented is a symptom of them. The two most important ones that I see are that the docs are written in an obscure format (DocBook) that is not conducive to a smooth writing flow and that the docs are an entirely separate tree in the nixpkgs repo, which means that it’s completely unclear where documentation for a given thing should go.
The community disagrees on this to various degrees and there is an in-progress RFC (see here) to determine a different format, but that is only the first step in what is presumably going to be a long and slow improvement process.
Does that buildGo.nix thing support Go modules?
I’ve never used (and probably won’t use) Go modules, but I believe Go programs/libraries written with them have the same directory layout (i.e. are inspectable via go/build) which means they’re supported by buildGo.external.
If your question is whether there’s a generator for translating the Go module definition files to Nix expressions, the answer is currently no (though there’s nothing preventing one from being written).
For local files, you can also just refer to them by their local path (either absolute or relative) and Nix will copy them into the store as appropriate when the expression is evaluated, for example:
{ pkgs ? import <nixpkgs> {} }:
pkgs.runCommand "example" {} ''
# Compute the SHA256 hash of the file "some-file" relative to where
# this expression is located.
${pkgs.openssl}/bin/openssl dgst -sha256 ${./some-file} > $out
''
Edit: Oh also, in case the question is “Can I get this hash without adding the file to the store?” - yes, the nix-hash utility (documented here) does that (and supports various different output formats for the hashes).
Oh, even better, the build directory isn’t writable.
This isn’t true for Nix in general. The build directory is explicitly writable and output installation (into /nix/store) usually (in Nix’s standard environment) happens as one of the last steps of a build.
It might be that this was a case of some language-specific tooling implementing such a restriction, in which case there’s probably also a flag for toggling it.
It’s most likely caused by the derivation either trying to build inside a store path, e.g. cd "${src}" && build, or inside a copy of a store path (which preserves the read-only flags), e.g. cp -a "${src}" src && cd src && build. We can see if that’s the case by looking at the build script in the failing .drv file: they’re plain text files, although they’re horrible to read without a pretty-printer like pretty-derivation. This is probably quicker than trying to get hold of and inspecting the failing derivation in nix repl, since it may be buried a few layers deep in dependencies.
I actually make this mistake a lot when writing build scripts; I usually solve it by putting chmod +w -R after the copy. If someone else has written/generated the build script it may be harder to override; although in that case it would presumably break for every input, so I’d guess the author might be calling it wrong (AKA poor documentation, which is unfortunately common with Nix :( )
It might be a symptom of the Koolaid, but I find this a feature rather than a bug: Nix keeps each download pristine, and forces my fiddling to be done on a copy; although the need to chmod afterwards is admittedly annoying.
Thanks for offering another view on Nix. I do not know much about NixOS and Nix, but I feel that I see This is the perfect solution articles about Nix almost everywhere. And I feel that in our industry I see 99% articles of this is the perfect solution, we all have to switch immediately because all of the old stuff is bad. Thus, it’s refreshing to see articles that highlight problematic aspects of hype-technology from time to time.
I think most of those articles, along with stuff about K8s, is more about validation for the author. I also think that all this tech has a purpose, but a lot of people blow it out of proportion. We all know cadey uses K8s to manage their services, but really, why not something like Ansible instead? Or some shell scripts? (This is not a stab at you cadey)
I think it puts the wrong idea into a lot of people’s heads. As cadey mentions in the article, no one talks about resource usage either. Downloading things four times kills bandwidth to those that are not on an unlimited plan, but also, takes four times as long! Sure, it’s computation time and not human time that’s spent, but what are people doing while waiting? My bet is not much other than checking back to see if said thing is done. It means you can’t do anything while traveling reliably. It means you gotta be strapped in at “HQ” to do any real work.
We need to use this tech to save time, not fill it. We need things to remain simple, if we want to grow more powerful.
To be honest, I use Kubernetes to manage my stuff because I needed to learn it for work. I’ve been idly considering going back to a “boring” setup with Ansible, but I have limited time for my personal projects and infrastructure. At the end of the day, the thing I really want to do is figure out how to make computers less bad by making them more explainable. This may be a futile endeavour, but this way I can’t say I didn’t try.
I think a better approach might be to leverage existing package managers and their cache stores that developers already use and [like|hate]. A more aggressive caching strategy may be a good idea, but a lot of it really boils down to this question: “what the heck are you doing?”. The answers to that question are probably closer to what you need to do than you’d think.
Nothing wrong even if wasn’t for work :) I just think it’s really important people point out things like that though. Remember that you’re a role model to people (even me!).
I’m learning Ansible because I believe it’s actually an improvement to how to manage services, even for 1 person. I believe the total outcome is not just overall a little bit positive, but swings towards the REALLY positive side.
K8s really really does seem like a solution for enterprise, where version mismatch and tech-mixing can get in the way of progress.
Your motivation behind using dhall and k8s makes sense to me now. :) That would absolutely be useful in a work setting, where your coworkers or future workers could potentially mess up configurations.
let APTPkg =
{ Type = { name : Text, state : Text }
, default = { name = "", state = "" }
}
let getPkg =
λ(name : Text)
→ { name = "Install ${name}"
, apt = APTPkg::{ name = name, state = "present" }
}
in [ getPkg "nginx", getPkg "weechat" ]
Oh, I can resonate with this. Nix is just one of many tools I have persevered through frustration (and often defeat) because in the end it feels like something with the potential of being not better, but less bad than the alternatives. Also even though Nix might not be the solution for all software distribution, I must admit I admire the bold ambition of it trying to be. If for nothing else, it’s something we can build on.
I also enjoyed a quote from a sub-comment:
At the end of the day, the thing I really want to do is figure out how to make computers less bad by making them more explainable.
This almost defines what my ambition or mission in software is. And one crucial observation is that “less bad” doesn’t by any means imply simple or easy, at least to me. Occasionally it can be, but more often it is not. I invest my time in learning tools, programming languages, type systems, and what have you. All with this utopia in mind that at one point I just “get it”. No more surprises, long nights tearing my hair out, mind racing over edge-case madness, all because the machine didn’t do its job (because we didn’t tell it how to). Of course I’m not expecting that’s ever gonna happen to me, but at least I can contribute towards making it a reality for future generations of engineers.
because in the end it feels like something with the potential of being not better, but less bad than the alternatives.
This was certainly the case when I tried creating build/runtime environments with a set of dependencies in Python. VirtualEnvs were a huge messy headaches and extraordinarily stateful. Even with the headaches of Nix, it was far preferable to that mess (piled on top of the mess that is Python on MacOS).
Well, the previous developer used Docker and made bash scripts that pip installed everything in a virtualenv for each new build. With Nix, I could eliminate redundancy and let Nix figure out dependencies, not have to deal with messy bash scripts, I could integrate non-Python dependencies seamlessly (which is probably the biggest advantage). It was also cross-platform so the same thing works regardless of whether it’s on my MacOS work laptop or the Linux server. Nix could do it all. There were pain points of course, but fairly few compared to the alternative. I suppose if you’re only dealing with Python it might provide that much of an advantage.
I intentionally hide jokes like that in my more ranty posts, and you are the first one to mention it. You win the prize of knowing you are the first to report it. Congrats!
I never used NixOS, but I’m quite a fan of nix-pkg insofar as it has mostly replaced my use for GNU stow on all Debian (and some Ubuntu 18.04) machines I own. I like a relatively stable base system (that’s why I dislike rolling distros) and I sometimes need or want a new version of some userland tool (often vim or tmux or sadly nodejs for work). With nix I can easily grab that in user space without having to build stuff myself or shove random binaries into ~/bin that I then never update.
Not a fan of the nix expression language but as I see a strong correlation between Haskell users and nix expressions, this makes sense for me :P
I think this is the first time I hear of something about nix that would actually solve a problem for me, thanks! If other people have concrete advantages of nix - not abstract more clean or whatever - then I would love to hear them!
It’s a nice way to package development environments across mac and linux, so that someone joining a company or project can just run nix-shell and immediately have the same versions of all compilers, libraries, tools etc as everyone else, without interfering with anything else on their machine.
The only similar alternative is docker, but mixing in sandboxing and virtualization makes it hard to use tools from outside the environment and interferes with performance testing on mac. With nix it just feels like a normal shell that happens to have some extra packages in the path and you use all the same tools you would normally use.
Docker builds are also not reproducible, so sometimes rebuilding an image to make a minor change will also pull in updates for various dependencies just because that’s what’s in apt today. With nix we can pin versions and update one thing at a time in a controlled way.
I’ve been using Nix (NixOS?) as a replacement for Homebrew and so far it’s been pretty good. Not the best learning curve but so far I’ve been able to install the packages I need (although getting git installed with shell completion took up the better part of an afternoon…). The main reason I like it over Homebrew is that every time I go to install a package I’m not rolling the dice on how long the operation will take. Homebrew operations seem to take 5 seconds or 15 minutes depending on whether it needs to recompile openSSL or not, and you can’t predict this before you run brew install… Good to know that from the packager’s side it’s a bit more tricky.
That issue with the multiple rebuilds is a design issue with the pkgs.buildGoModule function. It creates its own internal fetcher derivation to get the go modules instead of letting you pass one and compose. I find that most of the functions in nixpkgs are designed to be used for package management and don’t lend themselves very well to do incremental development in a monorepo.
I think here, since you are rebuilding exactly the same source, it might make sense to use the multiple-outputs feature and build everything at once instead:
{ pkgs ? import <nixpkgs> { } }:
let
version = "1.2.3";
out = pkgs.buildGoModule {
pname = "x";
version = version;
src = pkgs.fetchFromGitHub {
owner = "Xe";
repo = "x";
rev = "v${version}";
hash = "sha256-iw9WtpuSMl2NeNlcdg2JV618YVoU3UPa/0ED+r5Sj90=";
};
modSha256 = "sha256-mTzZcEoag1/1IAZCSY7MLgNS+lI453GM8SM1aFVLKUU=";
# TODO: remove references to go in the non-default output
allowGoReference = true;
subPackages = [
"cmd/appsluggr"
"cmd/johaus"
"cmd/license"
"cmd/prefix"
];
outputs = [
"out" # keep the default output but don't use it
"appsluger"
"johaus"
"license"
"prefix"
];
installPhase = ''
dir=$GOPATH/bin
install -D $dir/appsluggr $appsluger/bin/appsluggr
install -D $dir/johaus $johaus/bin/johaus
install -D $dir/license $license/bin/license
install -D $dir/prefix $prefix/bin/prefix
# let nix know that everything was fine
touch $out
'';
};
in
# could also just be `out` directly
{
appsluger = out.appsluger;
johaus = out.johaus;
license = out.license;
prefix = out.prefix;
}
BuildStream is a much less well known tool attempting to solve similar kinds of problems as nix, I’d argue it’s simpler, but it doesn’t have all the benefits that nix does either.
Interesting post. The main issue of Nix is its onboarding curve and lack of simple-to-grok documentation.
There’s a few things in this post worth responding to:
Nix has the
prefetch-*commands that can do this for you and output either the hash, or a full Nix expression that refers to that thing.It depends. My personal view recently has been that Nix should adopt a more Bazel-like model, in which the Nix language is also used for describing the actual software builds rather than just wrapping external package managers.
I have implemented this for Go (buildGo.nix / docs) and Common Lisp (buildLisp.nix), and with the Go one specifically external dependencies can be traversed and automatically transformed into a Nix data structure.
For example, here’s the buildGo.nix packaging of
golang.org/x/net(from here):This makes every subpackage available as an individual Nix derivation, which also means that those builds are cached across different software using those dependencies.
For most mainstream languages generators have been written to wrap 3rd-party package managers automatically. For some languages (e.g. Python), the nixpkgs tree actually contains derivations for all packages already so it’s just a matter of dragging them in.
This isn’t true for Nix in general. The build directory is explicitly writable and output installation (into
/nix/store) usually (in Nix’s standard environment) happens as one of the last steps of a build.It might be that this was a case of some language-specific tooling implementing such a restriction, in which case there’s probably also a flag for toggling it.
The conceptual boundary is drawn differently here. In some sense, we look at the artefacts of realised derivations (i.e. completed “builds”) as a cache. The hashes you see in the
/nix/storereference the inputs, not the outputs.Also, nothing written to the store is mutated afterwards so for any given store there is mutability, but it is append-only.
I’m not sure what is meant by this.
Edit 1: Ah, on the above this tweet adds some background (I think):
By default, Nix sandboxes builds which means that this is not a problem. Only explicitly declared dependencies are visible to a builder, and only the build directory and output path are writeable. Users can enable various footguns, such as opting out of sandboxing or whitelisting certain paths for passthrough.
Only on linux, unfortunately. The author seems to be on mac, which is probably why they didn’t know about the sandboxing.
It seems like sandboxing is available on Mac (thanks puck for pointing this out), but for users running Nix in single-user mode (which OP might be doing) there is currently some extra hoop-jumping required to make it work correctly.
I was thinking of this line from https://nixos.org/nix/manual/:
It looks like on mac it’s just a chroot to hide store paths but you can still
curl install.sh | bashin your build. I didn’t know it even had that much sandboxing on mac though, so thanks for pointing it.I really like this aspect of Nix: it’s like all packages exist in some platonic library of babel, and we copy a few of them into our
/nix/storecache as they’re needed. This style of reasoning also fits with the language’s laziness, the design ofnixpkgs(one huge set, whose contents are computed on-demand) and common patterns like taking fixed points to allow overrides (e.g. all of the function arguments calledself).A similar idea applies to content-addressable storage like IPFS, which I’m still waiting to be usable with Nix :(
Would that involve “recursive Nix” to allow builders to use Nix themselves, in order to build sub-components?
Recursive Nix is not necessary. For some languages this can already been done. E.g. the
buildRustCratefunction reimplements (most of) Cargo in Nix and does not use Cargo at all. This is in contrast tobuildRustPackage, which relies on Cargo to do the builds.You can convert a
Cargo.lockfile to a Nix expression with e.g. crate2nix and build crates usingbuildRustCrate. This has the same benefits as Nix has for other derivations: each compiled crate gets its own store path, so builds are incremental, and crate dependencies with the same version/features can be shared between derivations.No, I’m not using recursive Nix for these. In my opinion (this might be controversial with some people) recursive Nix is a workaround for performance flaws of the current evaluator and I’d rather address those than add the massive amount of complexity required by recursive Nix.
What’s potentially more important (especially for slow compilers like GHC or rustc) is content-addressed store paths, which allow for early build cutoff if two differing inputs (e.g. changes in comments or minor refactorings) yield the same artefact. Work is already underway towards that.
Can you please edit documentation somewhere to note the existence of the prefetch commands and how to use them?
Does that buildGo.nix thing support Go modules?
nix-prefetch-url is part of Nix itself and is documented here, nix-prefetch-git etc. come from another package in nixpkgs and I don’t think there’s any docs for them right now.
Nix has several large documentation issues and this being undocumented is a symptom of them. The two most important ones that I see are that the docs are written in an obscure format (DocBook) that is not conducive to a smooth writing flow and that the docs are an entirely separate tree in the nixpkgs repo, which means that it’s completely unclear where documentation for a given thing should go.
The community disagrees on this to various degrees and there is an in-progress RFC (see here) to determine a different format, but that is only the first step in what is presumably going to be a long and slow improvement process.
I’ve never used (and probably won’t use) Go modules, but I believe Go programs/libraries written with them have the same directory layout (i.e. are inspectable via go/build) which means they’re supported by
buildGo.external.If your question is whether there’s a generator for translating the Go module definition files to Nix expressions, the answer is currently no (though there’s nothing preventing one from being written).
Is there a way to get a hash of a file without making it available over HTTP?
Yep!
For local files, you can also just refer to them by their local path (either absolute or relative) and Nix will copy them into the store as appropriate when the expression is evaluated, for example:
Edit: Oh also, in case the question is “Can I get this hash without adding the file to the store?” - yes, the
nix-hashutility (documented here) does that (and supports various different output formats for the hashes).Proxy error(the link obvisouly).Edit: Back up!
Hah, sorry about that - I’m running that web UI on a preemptible GCP instance and usually nobody manages to catch an instance cycling moment :-)
It’s most likely caused by the derivation either trying to build inside a store path, e.g.
cd "${src}" && build, or inside a copy of a store path (which preserves the read-only flags), e.g.cp -a "${src}" src && cd src && build. We can see if that’s the case by looking at the build script in the failing.drvfile: they’re plain text files, although they’re horrible to read without a pretty-printer like pretty-derivation. This is probably quicker than trying to get hold of and inspecting the failing derivation innix repl, since it may be buried a few layers deep in dependencies.I actually make this mistake a lot when writing build scripts; I usually solve it by putting
chmod +w -Rafter the copy. If someone else has written/generated the build script it may be harder to override; although in that case it would presumably break for every input, so I’d guess the author might be calling it wrong (AKA poor documentation, which is unfortunately common with Nix :( )It might be a symptom of the Koolaid, but I find this a feature rather than a bug: Nix keeps each download pristine, and forces my fiddling to be done on a copy; although the need to
chmodafterwards is admittedly annoying.Thanks for offering another view on Nix. I do not know much about NixOS and Nix, but I feel that I see This is the perfect solution articles about Nix almost everywhere. And I feel that in our industry I see 99% articles of this is the perfect solution, we all have to switch immediately because all of the old stuff is bad. Thus, it’s refreshing to see articles that highlight problematic aspects of hype-technology from time to time.
I think most of those articles, along with stuff about K8s, is more about validation for the author. I also think that all this tech has a purpose, but a lot of people blow it out of proportion. We all know cadey uses K8s to manage their services, but really, why not something like Ansible instead? Or some shell scripts? (This is not a stab at you cadey)
I think it puts the wrong idea into a lot of people’s heads. As cadey mentions in the article, no one talks about resource usage either. Downloading things four times kills bandwidth to those that are not on an unlimited plan, but also, takes four times as long! Sure, it’s computation time and not human time that’s spent, but what are people doing while waiting? My bet is not much other than checking back to see if said thing is done. It means you can’t do anything while traveling reliably. It means you gotta be strapped in at “HQ” to do any real work.
We need to use this tech to save time, not fill it. We need things to remain simple, if we want to grow more powerful.
To be honest, I use Kubernetes to manage my stuff because I needed to learn it for work. I’ve been idly considering going back to a “boring” setup with Ansible, but I have limited time for my personal projects and infrastructure. At the end of the day, the thing I really want to do is figure out how to make computers less bad by making them more explainable. This may be a futile endeavour, but this way I can’t say I didn’t try.
I think a better approach might be to leverage existing package managers and their cache stores that developers already use and [like|hate]. A more aggressive caching strategy may be a good idea, but a lot of it really boils down to this question: “what the heck are you doing?”. The answers to that question are probably closer to what you need to do than you’d think.
Nothing wrong even if wasn’t for work :) I just think it’s really important people point out things like that though. Remember that you’re a role model to people (even me!).
I’m learning Ansible because I believe it’s actually an improvement to how to manage services, even for 1 person. I believe the total outcome is not just overall a little bit positive, but swings towards the REALLY positive side.
K8s really really does seem like a solution for enterprise, where version mismatch and tech-mixing can get in the way of progress.
Your motivation behind using dhall and k8s makes sense to me now. :) That would absolutely be useful in a work setting, where your coworkers or future workers could potentially mess up configurations.
Some food for thought:
Using dhall with Ansible was the first thing I thought while reading your post on dhall and k8s :)
Oh, I can resonate with this. Nix is just one of many tools I have persevered through frustration (and often defeat) because in the end it feels like something with the potential of being not better, but less bad than the alternatives. Also even though Nix might not be the solution for all software distribution, I must admit I admire the bold ambition of it trying to be. If for nothing else, it’s something we can build on.
I also enjoyed a quote from a sub-comment:
This almost defines what my ambition or mission in software is. And one crucial observation is that “less bad” doesn’t by any means imply simple or easy, at least to me. Occasionally it can be, but more often it is not. I invest my time in learning tools, programming languages, type systems, and what have you. All with this utopia in mind that at one point I just “get it”. No more surprises, long nights tearing my hair out, mind racing over edge-case madness, all because the machine didn’t do its job (because we didn’t tell it how to). Of course I’m not expecting that’s ever gonna happen to me, but at least I can contribute towards making it a reality for future generations of engineers.
This was certainly the case when I tried creating build/runtime environments with a set of dependencies in Python. VirtualEnvs were a huge messy headaches and extraordinarily stateful. Even with the headaches of Nix, it was far preferable to that mess (piled on top of the mess that is Python on MacOS).
Could you elaborate? I am quite happy with creating a virtualenv and pip installing everything.
Well, the previous developer used Docker and made bash scripts that pip installed everything in a virtualenv for each new build. With Nix, I could eliminate redundancy and let Nix figure out dependencies, not have to deal with messy bash scripts, I could integrate non-Python dependencies seamlessly (which is probably the biggest advantage). It was also cross-platform so the same thing works regardless of whether it’s on my MacOS work laptop or the Linux server. Nix could do it all. There were pain points of course, but fairly few compared to the alternative. I suppose if you’re only dealing with Python it might provide that much of an advantage.
Did anyone else laugh at “This makes Nix packages a kind of blockchain.”?
(I assume that it was meant as a joke)
I intentionally hide jokes like that in my more ranty posts, and you are the first one to mention it. You win the prize of knowing you are the first to report it. Congrats!
NB: Noticed it but didn’t find it funny. Blockchains are serious business!11
I never used NixOS, but I’m quite a fan of nix-pkg insofar as it has mostly replaced my use for GNU stow on all Debian (and some Ubuntu 18.04) machines I own. I like a relatively stable base system (that’s why I dislike rolling distros) and I sometimes need or want a new version of some userland tool (often vim or tmux or sadly nodejs for work). With nix I can easily grab that in user space without having to build stuff myself or shove random binaries into
~/binthat I then never update.Not a fan of the nix expression language but as I see a strong correlation between Haskell users and nix expressions, this makes sense for me :P
I think this is the first time I hear of something about nix that would actually solve a problem for me, thanks! If other people have concrete advantages of nix - not abstract more clean or whatever - then I would love to hear them!
It’s a nice way to package development environments across mac and linux, so that someone joining a company or project can just run
nix-shelland immediately have the same versions of all compilers, libraries, tools etc as everyone else, without interfering with anything else on their machine.The only similar alternative is docker, but mixing in sandboxing and virtualization makes it hard to use tools from outside the environment and interferes with performance testing on mac. With nix it just feels like a normal shell that happens to have some extra packages in the path and you use all the same tools you would normally use.
Docker builds are also not reproducible, so sometimes rebuilding an image to make a minor change will also pull in updates for various dependencies just because that’s what’s in apt today. With nix we can pin versions and update one thing at a time in a controlled way.
That does sound interesting. What should I read if I want to know how to set that up?
I would love to say the documentation, but it’s incredibly confusing as a beginner. https://nixos.org/nix/manual/ and https://nixos.org/nixpkgs/manual/ are somewhat helpful, but I also end up reading the source in https://github.com/NixOS/nixpkgs/ a lot and also just learning through painful trial and error. Perhaps someone can recommend a good intro talk or tutorial.
I’m planning to take a look at guix at some point, because it’s similar in concept to nix but is often described as having much better ux and docs.
Christine’s writing style in posts like this really remind me of Beryl Markham. She seems, to me, to have an uncanny ability to “condense fact from the vapor of nuance.”
I also have very much enjoyed the follow up discussion and documentation links!
I’ve been using Nix (NixOS?) as a replacement for Homebrew and so far it’s been pretty good. Not the best learning curve but so far I’ve been able to install the packages I need (although getting
gitinstalled with shell completion took up the better part of an afternoon…). The main reason I like it over Homebrew is that every time I go to install a package I’m not rolling the dice on how long the operation will take. Homebrew operations seem to take 5 seconds or 15 minutes depending on whether it needs to recompile openSSL or not, and you can’t predict this before you runbrew install… Good to know that from the packager’s side it’s a bit more tricky.That issue with the multiple rebuilds is a design issue with the
pkgs.buildGoModulefunction. It creates its own internal fetcher derivation to get the go modules instead of letting you pass one and compose. I find that most of the functions in nixpkgs are designed to be used for package management and don’t lend themselves very well to do incremental development in a monorepo.I think here, since you are rebuilding exactly the same source, it might make sense to use the multiple-outputs feature and build everything at once instead:
Can you please document this somewhere?
I’m not sure what you want exactly. Nix has quite a lot to it. It’s certainly more complicated than just a Dockerfile.
The multiple outputs are explained over here, I don’t know how easy that is to consume: https://nixos.org/nixpkgs/manual/#sec-multiple-outputs-
Can you see about making that into multiple pages? Like one per section or something. That page is just too large and makes me gloss over things.
BuildStream is a much less well known tool attempting to solve similar kinds of problems as nix, I’d argue it’s simpler, but it doesn’t have all the benefits that nix does either.