Looks like nix-shell is a handy tool to provide project-level reproducibility, and I think it’s much more useful compared to NixOS, which seems to offer workstation-level reproducibility. Most people replace their workstations very infrequently, and when they do, chances are that there exists some migration assistant (e.g. Time Machine or perhaps dd(1)). I don’t think I need to “deploy” my workstation-level configuration anywhere; it’s only meant for me to begin with.
Tangentially, as a research assistant, I need to share a gigantic computing cluster with everyone affiliated with the university, and I don’t think I can convince the sysadmin into installing Nix on it (especially when the installer is a scary curl -L https://nixos.org/nix/install | sh). I know a root-level /nix directory is required to make use of the binary cache since the absolute path to dynamic libraries is embedded in the cached binaries, but there must be some workaround. Like, why not just scan the relocation table of each cached binary and replace the prefix of the paths?
I’m currently using NixOS as my main daily driver. The advantage for me isn’t workspace migrations, but workspace versioning. There is a lot of cruft that builds up on workspaces overtime. Packages that you download to try out, work-arounds that stick, etc. These are all versioned & documented in the gitlog. It also lets me do stupid things without having to really worry about the consequences on my machine, as I can rollback the changes easily.
I tried to configure the whole workspace with Nix on macOS, but it turns out that I cannot even install Firefox. This gives me the impression that nixpkgs is currently not mature/popular enough (at least on macOS), and that at some point I will be forced to install Homebrew/MacPorts/Anaconda or run curl shiny.tool/install.sh | sh to get some niche package, and suddenly I have packages outside of version control.
Also, nix-env -qaP some_package is already ridiculously slow, and with more packages in the repositroy, it will probably become even slower. More importantly, even a huge package repository cannot include everything, so from time to time users must write Nix expressions themselves, which I don’t think would be trivial (if it was, then Nix would have already automated that).
I’m not complaining, but that’s the reason I’m not bold enough to use Nix as my daily driver. I guess I should donate some money to Nix to facilitate its growth.
I don’t disagree with the facts that you wrote, but I thought I’d comment since I cash some of them out differently… For a little context, I first took the dive into NixOS in early 2018, when my Windows desktop’s motherboard flamed out. It was rocky (a mix of Nix, plus it being my first desktop Linux), but I started using nix and nix-darwin when I replaced my macbook air in early 2019.
I tried to configure the whole workspace with Nix on macOS, but it turns out that I cannot even install Firefox. This gives me the impression that nixpkgs is currently not mature/popular enough (at least on macOS), and that at some point I will be forced to install Homebrew
A linux-first package manager’s ability to manage all software including desktop apps on macOS is a very stringent ruler. (Don’t get me wrong–it’ll be good if it works some day–but it’s a high bar, and Nix can be very useful without clearing it.)
Yes–keep homebrew. TBH, I think forcing desktop apps, many of which already want to auto-update, into the Nix paradigm is a bit weird. I, to be frank, find Nix on macOS, with nix-darwin, to be a very nice compromise over the purity of these apps on NixOS.
I actually find it more ergonomic to let these apps update as they have new security patches, and update my Nixpkgs channel less frequently. Since early 2019, I think I’ve only twice used Homebrew to install a package–I’ve used it almost exclusively for casks, and the packages were really just to play with something that was in Homebrew to decide if it was worth porting to Nix (neither was). Once again, I’d say the freedom to do this is a really nice compromise over purity in NixOS.
suddenly I have packages outside of version control.
You can still version-control a .Brewfile if it is for apps. It’s obviously not the same level of reproducibility, but if I’m trying to rebuild the software I had 3 years ago I’m generally not doing it for Chrome, Firefox, Steam, etc. I added a section to my backup script to barf out a report on whether I have installed anything with brew that isn’t in my brewfile. If I really cared, I think I could script it to uninstall those packages every day to force the issue.
If it’s for a smaller package and you care about version-controlled reproducibility this much, you’ll generally have enough motivation to port it to Nix. (In my experience it has been true, but I recognize that the practicality of this will depend on the scope of the package in question…)
More importantly, even a huge package repository cannot include everything, so from time to time users must write Nix expressions themselves, which I don’t think would be trivial
This is the proverbial two-edged sword. So, yes, yes, this can happen. I am personally very conservative when it comes to recommending Nix to anyone who isn’t open to learning the language. I think it can be okay, narrowly, as a tool with no knowledge of the language. Learning it can be frustrating. But:
Nix can be a really big lever. I’m not sure if this is of much use to people who don’t program, but I feel like my time was well spent (even if it was a bigger investment than it needed to be).
A lot of the difficulty of learning to write Nix packages has honestly just been my near-complete lack of experience with the processes for going from source to installed software in Unix-alikes. If you already know a lot about this, it’ll be mostly about the language.
It is fairly easy to manage/integrate private packages (whether that’s from scratch, or just overrides to use PRs I took the time to submit that the upstream is ignoring). I end up writing a Nix package for most of the unreleased/private software on my system (even when I could use it without doing so).
(if it was, then Nix would have already automated that).
Have any other package managers automated the generation of package expressions? I’m not terribly knowledgeable on prior art, here. If so, you’ve probably got a point. IME most of the work of writing package expressions is human knowledge going into understanding the software’s own packaging assumptions and squaring them with Nix. I’d be a little surprised if this is highly automatable. I can imagine translation of existing packages from other languages being more tractable, but IDK.
A lot of things that are using standard frameworks can be mostly automated. Debian/Ubuntu are doing pretty well with dh. Nix already has templates https://github.com/jonringer/nix-template
It’s not perfect, but as long as you’re using common rolling, packaging is not terribly hard.
As to -qaP, unfortunately you learn to not do it; instead I’d recommend to use Nix 2.0’s nix search, as it has some caching (and was in fact introduced primarily to solve the problem with -qaP); and then e.g. nix-env -iA nixpkgs.some_package. Or, alternatively, maybe nix profile install, though I haven’t experimented with it yet myself.
As to getting some niche package: yes, at some point you’ll either have to do this, or write your own Nix expression to wrap it. FWIW, not every byte in the world is wrapped by Nix, and This Is Just A FOSS Project Run By Good-Willing Volunteers In Their Spare Time, and You Can (Try To) Contribute, and going into another rabbit hole trying to wrap Your Favourite Package™ in a Nix expression is probably something of a rite of passage. I like to draw a parallel between Nix and earlier days of Linux, before Ubuntu, when you had to put a lot of time into it to have some things. A.k.a. your typical day at the bleeding edge of technology (which Nix is actually already not so much bleeding as it was just a few years ago). And, actually, people say Nixpkgs are kinda at the top on https://repology.org. But you know, at least when a package is broken on Nixpkgs, it doesn’t send your whole OS crashing & burning into some glitchy state from which you’ll not recover for the next couple years… because as soon as you manage to get back to some working terminal & disk access (assuming things went really bad on NixOS), or in worst case restart to GRUB, you’re just one nix-rebuild generation away from your last known good state. And with flakes, you nearly certainly even have the source of the last known good generation in a git repo.
I think the biggest problem is that for Nix to be useful, we must achieve nearly complete package coverage, i.e. almost all packages must be installable via Nix. Covering 90% of the most popular packages is still not good enough, because even a single non-reproducible package in the whole dependency graph will ruin everything. It’s an all-or-nothing deal, and assuming package usage follows a power-law distribution, we will have a very hard time covering the last few bits. This is very different from Linux, where implementing 90% of the functionality makes a pretty useful system.
Since you mentioned Linux, I’d like to note that packages from the system repository of most distributions are outdated, and users are encouraged to install from source or download the latest release from elsewhere (e.g. on Ubuntu 20.04, you must use the project-specific “PostgreSQL Apt Repository” to install PostgreSQL 13, which was released over a year ago). I guess some people took the effort to package something they want to use, but lack the incentive to keep maintaining it. While it’s perfectly fine to sidestep apt or yum and run make && make install instead, you can never sidestep Nix because otherwise you would lose reproducibility. How can the Nix community keep nearly all packages roughly up to date? I have no clue.
What I’m trying to answer to this, is to think small and “egoistically”: instead of thinking how Nix is doomed from a “whole world” perspective, try to focus just on your own localised use case and how much reproducibility you need yourself: if you must have 100% reproducibility, it means you either have enough motivation and resources to wrap the (finite number of) specific dependencies that you need and are not yet wrapped, or it’s apparently more of a would like to than must have, i.e. you have other, higher priorities overriding that. If the latter, you can still use Nix for those packages it provides, and add a few custom scripts over that doing a bit of curl+make or whatsit that you’ve been doing all the time till now (though once you learn to write Nix expressions for packages, you may realize they’re not really much different from that). Unless you go full NixOS (which I don’t recommend for starters), your base distro stays the same as it was (you said you don’t have root access anyway, right?) and you still can do all of what you did before. If some parts of your system are reproducible and some are not (yet), is it worse than if none are? Or maybe it is actually an improvement? And with some luck and/or persistence, eventually others may start helping you with wrapping the “last mile” of their personal pet packages (ideally, when they start noticing the benefits, i.e. their Nix-wrapped colleagues’ projects always building successfully and reproducibly and not breaking, and thus them being able to “just focus on the science/whatsit-they’re-paid-for”).
That makes sense. IMHO Nix can and should convince package authors to wrap their own stuff in Nix. It can because the Nix language is cross-platform (this is not the case for apt/yum/brew/pkg); it should because only authors can make sure the Nix derivations are always up-to-date (with a CI/CD pipeline or something) while minimizing the risk of a supply chain attack.
That is not how the open-source movement works. You don’t get to tell people what they should do; you rather take with humbleness and gratitude what they created, try to help them by contributing back (yet humbly enough to be ready to accept and respect if they might not take your contribution for some reason - though knowing you’re also free to fork), and yes, this means to possibly also contribute back ideas, but with the same caveat of them possibly not being taken - in this even more often, given that ideas are a dime a dozen. And notably, through contributing back some high quality code, you might earn some recognition that might give you a tiny bit more attention when sharing ideas. Ah, and/or you can also try to follow up on your ideas with ownership and actions, this tends to have the highest chance of success (though still not 100% guaranteed).
That said, I see this thread now as veering off on a tangent from the original topic, and as such I think I will take a break and refrain from contributing to making it a digression train (however I love digressions and however tempting this is), whether I agree or not with any further replies :) thanks, cheers and wish you great Holidays! :)
I think the biggest problem is that for Nix to be useful, we must achieve nearly complete package coverage, i.e. almost all packages must be installable via Nix.
Why do you think so? I’m wondering why this applies to Nix but not to Homebrew or apt or yum or the likes? One can still build a package manually by setting up the dependencies in a nix shell – that’s no different from building something that package managers of other systems still don’t have.
From my understanding, Nix aims to provide a reproducible environment/build, so it must exhaustively know about every piece of dependency. Homebrew, apt, and yum don’t have such an ambition; they just install packages, and can thus happily co-exist with other package managers and user-installed binaries.
Nix-build, yes; nix-shell, no. In a nix-shell env, you still see all of your pre-existing system, plus what nix-shell provides as an “overlay” (not in docker filesystem sense, just extra entries in PATH etc.). It reproducibly provides you the dependencies you asked it to provide, but it doesn’t guarantee reproducibility of what you do over that. So you could start with nix-shell (or nix develop IIRC in case of flakes).
This gives me the impression that nixpkgs is currently not mature/popular enough (at least on macOS)
Have no idea about Mac, but my understanding is that on Linux the amount of packaged stuff for NixOS is just ridiculously high: https://repology.org/repositories/graphs. Anecdotally, everything I need on daily basis is there.
Still, there are cases when you do want to try a random binary from the internet (happened to me last time when I wanted to try JetBrains Fleet first public release), and yeah, NixOS does make those cases painful.
NixOS […] seems to offer workstation-level reproducibility.
Don’t forget about server reproducibility! This is especially nice when you need to spin up multiple servers for a single project that need similar configuration.
In which case I either have docker/kurbernetes/podman or I’m using ansible to be fast and productive. Sure you may get some stuff done much more precisely in nixos, but that’s definitely not worth the hassle. That said: Best of luck to nixos, hopefully it’ll be stable enough one day.
Wait, what hassle? And what about it isn’t “stable?” Citation needed? It’s stable enough for every NixOS user I know on server and desktop and it’s stable enough for me. Hassle is a very vague word that could mean a lot of things, so if not for the fact that exactly zero of those possible meanings make what you said a true statement, I wouldn’t know how to respond to this. What is it about NixOS you think is a hassle?
Heh, my previous company went from NixOS to “let’s deploy using ansible on Ubuntu boxes, everyone knows those”. Productivity and velocity just went down the drain. Oh, the horrors, oh the PTSD… But everyone has different experiences, sometimes some tools work, sometimes they don’t.
I don’t know how you could get any more precise than NixOS. It specifies everything by hash, all the way down to a linker. I’ve never seen anybody do anything like that with any other system.
Looks like nix-shell is a handy tool to provide project-level reproducibility,
Definitely. Every project I use now has a shell.nix file that pins the tools I use. I switched to that workflow after I was bitten several times by brew replacing python, so virtual environments were not working, or completely forgetting what tools I need in a project (after returning to it a year later). shell.nix is acting both as a bill of materials and recipe for fetching the right tools.
For me that work-around is ‘containers’. As in nix generates the containers (reproducible!), the clusters run the containers, and in the containers it’s /nix.
If you manage to get it, personally, I would heartily recommend trying to get into “Nix Flakes” as soon as possible. They’re theoretically still “experimental”, and even harder to find reliable documentation about than Nix itself (which is somewhat infamously not-easy already), but IMO they seem to kinda magically solve a lot of auxillary pain points I had with “classic” Nix.
Also, as a side note, the nix-shell stuff was apparently much earlier than NixOS. The original thesis and project was just Nix, and NixOS started later as another guy’s crazy experiment, AFAIU.
EDIT 2: Finally, although with much less fancy features, there are still some interesting encapsulation aspects in 0install.net: “0install also has some interesting features not often found in traditional package managers. For example, while it will share libraries whenever possible, it can always install multiple versions of a package in parallel when there are conflicting requirements. Installation is always side-effect-free (each package is unpacked to its own directory and will not touch shared directories such as /usr/bin), making it ideal for use with sandboxing technologies and virtualisation.”
This article did sent me on search for “how do you flakes and nix-shell at the same time” and apparently there’s “nix develop” now. This link has some details for both system wide and local setups: https://www.tweag.io/blog/2020-07-31-nixos-flakes/
asdf works fine for pinning runtimes until you have system libraries, etc that extensions link against which aren’t versioned with asdf. Then you’re back in the same boat as you are with brew, etc where upgrading might break existing workdirs.
It sounds like Nix can also build your containers for you based on your project definition?
Yep basically just something like this, lots of assumptions given with this and that you “want” to containerize the hello world program gnu ships but eh its an example:
There are caveats to using the docker builds (can’t build on macos) and you’ll need to learn the nix programming language at some point but its a rather droll affair IME once you get that its all just data and functions. And before you ask why is it so big, the short answer is everything that hello defined it depends on is included, which includes jq/pigz/jshon/perl/moreutils etc… for some reason. But its basically lifted straight out of the nix store verbatim.
everything that hello defined it depends on is included, which includes jq/pigz/jshon/perl/moreutils etc… for some reason
I recognise this list. These are the dependencies used in the shell scripts which build the Docker image. They shouldn’t be included in the image itself.
Of course if I used musl libc than glibc and its dependencies would go away automatically.
What’s better is that if you use buildLayeredImage each of these is a separate layer so that rebuilding for example the word list, or the binary doesn’t require rebuilding other layers. (This is actually better than docker itself because docker only supports linear layering, so you would have to decide if the word list or the binary is the top layer and rebuilding the lower would force a rebuild of the higher one.)
Getting shell.nix set up for projects is really nice, but I jump around a lot between different codebases in different languages where it’s often not worth it (or wanted by the owners) to set it up. The nix-shell command that I use all the time for one-offs:
$ nix-shell -p <package-name>
…and now I have package-name, and it’s gone when I exit the shell. Well, it’s still in my local nix-store, so it’s almost instant to nix-shell -p it again, but otherwise it isn’t really installed.
Recent one in my shell history: nix-shell -p nodejs. Specific node version: nix-shell -p nodejs-10_x. Wanted to try some python script depending on matplotlib on a mac😱nix-shell -p python3Packages.matplotlib. Even nix-shell -p inkscape works.
I use nix-shell similarly, but never commit shell.nix to the repository. So I get the benefits of having the tools, but I don’t draw ire of other devs :)
New nix has the nix search command. It used to be that nix search <term> would do the search, but now (nix 2.4) it’s nix search nixpkgs <term>.
Looks like
nix-shellis a handy tool to provide project-level reproducibility, and I think it’s much more useful compared to NixOS, which seems to offer workstation-level reproducibility. Most people replace their workstations very infrequently, and when they do, chances are that there exists some migration assistant (e.g. Time Machine or perhapsdd(1)). I don’t think I need to “deploy” my workstation-level configuration anywhere; it’s only meant for me to begin with.Tangentially, as a research assistant, I need to share a gigantic computing cluster with everyone affiliated with the university, and I don’t think I can convince the sysadmin into installing Nix on it (especially when the installer is a scary
curl -L https://nixos.org/nix/install | sh). I know a root-level/nixdirectory is required to make use of the binary cache since the absolute path to dynamic libraries is embedded in the cached binaries, but there must be some workaround. Like, why not just scan the relocation table of each cached binary and replace the prefix of the paths?I’m currently using NixOS as my main daily driver. The advantage for me isn’t workspace migrations, but workspace versioning. There is a lot of cruft that builds up on workspaces overtime. Packages that you download to try out, work-arounds that stick, etc. These are all versioned & documented in the gitlog. It also lets me do stupid things without having to really worry about the consequences on my machine, as I can rollback the changes easily.
I tried to configure the whole workspace with Nix on macOS, but it turns out that I cannot even install Firefox. This gives me the impression that
nixpkgsis currently not mature/popular enough (at least on macOS), and that at some point I will be forced to install Homebrew/MacPorts/Anaconda or runcurl shiny.tool/install.sh | shto get some niche package, and suddenly I have packages outside of version control.Also,
nix-env -qaP some_packageis already ridiculously slow, and with more packages in the repositroy, it will probably become even slower. More importantly, even a huge package repository cannot include everything, so from time to time users must write Nix expressions themselves, which I don’t think would be trivial (if it was, then Nix would have already automated that).I’m not complaining, but that’s the reason I’m not bold enough to use Nix as my daily driver. I guess I should donate some money to Nix to facilitate its growth.
I don’t disagree with the facts that you wrote, but I thought I’d comment since I cash some of them out differently… For a little context, I first took the dive into NixOS in early 2018, when my Windows desktop’s motherboard flamed out. It was rocky (a mix of Nix, plus it being my first desktop Linux), but I started using nix and nix-darwin when I replaced my macbook air in early 2019.
A linux-first package manager’s ability to manage all software including desktop apps on macOS is a very stringent ruler. (Don’t get me wrong–it’ll be good if it works some day–but it’s a high bar, and Nix can be very useful without clearing it.)
Yes–keep homebrew. TBH, I think forcing desktop apps, many of which already want to auto-update, into the Nix paradigm is a bit weird. I, to be frank, find Nix on macOS, with nix-darwin, to be a very nice compromise over the purity of these apps on NixOS.
I actually find it more ergonomic to let these apps update as they have new security patches, and update my Nixpkgs channel less frequently. Since early 2019, I think I’ve only twice used Homebrew to install a package–I’ve used it almost exclusively for casks, and the packages were really just to play with something that was in Homebrew to decide if it was worth porting to Nix (neither was). Once again, I’d say the freedom to do this is a really nice compromise over purity in NixOS.
You can still version-control a .Brewfile if it is for apps. It’s obviously not the same level of reproducibility, but if I’m trying to rebuild the software I had 3 years ago I’m generally not doing it for Chrome, Firefox, Steam, etc. I added a section to my backup script to barf out a report on whether I have installed anything with brew that isn’t in my brewfile. If I really cared, I think I could script it to uninstall those packages every day to force the issue.
If it’s for a smaller package and you care about version-controlled reproducibility this much, you’ll generally have enough motivation to port it to Nix. (In my experience it has been true, but I recognize that the practicality of this will depend on the scope of the package in question…)
This is the proverbial two-edged sword. So, yes, yes, this can happen. I am personally very conservative when it comes to recommending Nix to anyone who isn’t open to learning the language. I think it can be okay, narrowly, as a tool with no knowledge of the language. Learning it can be frustrating. But:
Have any other package managers automated the generation of package expressions? I’m not terribly knowledgeable on prior art, here. If so, you’ve probably got a point. IME most of the work of writing package expressions is human knowledge going into understanding the software’s own packaging assumptions and squaring them with Nix. I’d be a little surprised if this is highly automatable. I can imagine translation of existing packages from other languages being more tractable, but IDK.
A lot of things that are using standard frameworks can be mostly automated. Debian/Ubuntu are doing pretty well with
dh. Nix already has templates https://github.com/jonringer/nix-templateIt’s not perfect, but as long as you’re using common rolling, packaging is not terribly hard.
As to
-qaP, unfortunately you learn to not do it; instead I’d recommend to use Nix 2.0’snix search, as it has some caching (and was in fact introduced primarily to solve the problem with-qaP); and then e.g.nix-env -iA nixpkgs.some_package. Or, alternatively, maybenix profile install, though I haven’t experimented with it yet myself.As to getting some niche package: yes, at some point you’ll either have to do this, or write your own Nix expression to wrap it. FWIW, not every byte in the world is wrapped by Nix, and This Is Just A FOSS Project Run By Good-Willing Volunteers In Their Spare Time, and You Can (Try To) Contribute, and going into another rabbit hole trying to wrap Your Favourite Package™ in a Nix expression is probably something of a rite of passage. I like to draw a parallel between Nix and earlier days of Linux, before Ubuntu, when you had to put a lot of time into it to have some things. A.k.a. your typical day at the bleeding edge of technology (which Nix is actually already not so much bleeding as it was just a few years ago). And, actually, people say Nixpkgs are kinda at the top on https://repology.org. But you know, at least when a package is broken on Nixpkgs, it doesn’t send your whole OS crashing & burning into some glitchy state from which you’ll not recover for the next couple years… because as soon as you manage to get back to some working terminal & disk access (assuming things went really bad on NixOS), or in worst case restart to GRUB, you’re just one
nix-rebuildgeneration away from your last known good state. And with flakes, you nearly certainly even have the source of the last known good generation in a git repo.I think the biggest problem is that for Nix to be useful, we must achieve nearly complete package coverage, i.e. almost all packages must be installable via Nix. Covering 90% of the most popular packages is still not good enough, because even a single non-reproducible package in the whole dependency graph will ruin everything. It’s an all-or-nothing deal, and assuming package usage follows a power-law distribution, we will have a very hard time covering the last few bits. This is very different from Linux, where implementing 90% of the functionality makes a pretty useful system.
Since you mentioned Linux, I’d like to note that packages from the system repository of most distributions are outdated, and users are encouraged to install from source or download the latest release from elsewhere (e.g. on Ubuntu 20.04, you must use the project-specific “PostgreSQL Apt Repository” to install PostgreSQL 13, which was released over a year ago). I guess some people took the effort to package something they want to use, but lack the incentive to keep maintaining it. While it’s perfectly fine to sidestep
aptoryumand runmake && make installinstead, you can never sidestep Nix because otherwise you would lose reproducibility. How can the Nix community keep nearly all packages roughly up to date? I have no clue.What I’m trying to answer to this, is to think small and “egoistically”: instead of thinking how Nix is doomed from a “whole world” perspective, try to focus just on your own localised use case and how much reproducibility you need yourself: if you must have 100% reproducibility, it means you either have enough motivation and resources to wrap the (finite number of) specific dependencies that you need and are not yet wrapped, or it’s apparently more of a would like to than must have, i.e. you have other, higher priorities overriding that. If the latter, you can still use Nix for those packages it provides, and add a few custom scripts over that doing a bit of curl+make or whatsit that you’ve been doing all the time till now (though once you learn to write Nix expressions for packages, you may realize they’re not really much different from that). Unless you go full NixOS (which I don’t recommend for starters), your base distro stays the same as it was (you said you don’t have root access anyway, right?) and you still can do all of what you did before. If some parts of your system are reproducible and some are not (yet), is it worse than if none are? Or maybe it is actually an improvement? And with some luck and/or persistence, eventually others may start helping you with wrapping the “last mile” of their personal pet packages (ideally, when they start noticing the benefits, i.e. their Nix-wrapped colleagues’ projects always building successfully and reproducibly and not breaking, and thus them being able to “just focus on the science/whatsit-they’re-paid-for”).
That makes sense. IMHO Nix can and should convince package authors to wrap their own stuff in Nix. It can because the Nix language is cross-platform (this is not the case for apt/yum/brew/pkg); it should because only authors can make sure the Nix derivations are always up-to-date (with a CI/CD pipeline or something) while minimizing the risk of a supply chain attack.
That is not how the open-source movement works. You don’t get to tell people what they should do; you rather take with humbleness and gratitude what they created, try to help them by contributing back (yet humbly enough to be ready to accept and respect if they might not take your contribution for some reason - though knowing you’re also free to fork), and yes, this means to possibly also contribute back ideas, but with the same caveat of them possibly not being taken - in this even more often, given that ideas are a dime a dozen. And notably, through contributing back some high quality code, you might earn some recognition that might give you a tiny bit more attention when sharing ideas. Ah, and/or you can also try to follow up on your ideas with ownership and actions, this tends to have the highest chance of success (though still not 100% guaranteed).
That said, I see this thread now as veering off on a tangent from the original topic, and as such I think I will take a break and refrain from contributing to making it a digression train (however I love digressions and however tempting this is), whether I agree or not with any further replies :) thanks, cheers and wish you great Holidays! :)
Why do you think so? I’m wondering why this applies to Nix but not to Homebrew or apt or yum or the likes? One can still build a package manually by setting up the dependencies in a nix shell – that’s no different from building something that package managers of other systems still don’t have.
From my understanding, Nix aims to provide a reproducible environment/build, so it must exhaustively know about every piece of dependency. Homebrew, apt, and yum don’t have such an ambition; they just install packages, and can thus happily co-exist with other package managers and user-installed binaries.
Nix-build, yes; nix-shell, no. In a nix-shell env, you still see all of your pre-existing system, plus what nix-shell provides as an “overlay” (not in docker filesystem sense, just extra entries in PATH etc.). It reproducibly provides you the dependencies you asked it to provide, but it doesn’t guarantee reproducibility of what you do over that. So you could start with nix-shell (or
nix developIIRC in case of flakes).Have no idea about Mac, but my understanding is that on Linux the amount of packaged stuff for NixOS is just ridiculously high: https://repology.org/repositories/graphs. Anecdotally, everything I need on daily basis is there.
Still, there are cases when you do want to try a random binary from the internet (happened to me last time when I wanted to try JetBrains Fleet first public release), and yeah, NixOS does make those cases painful.
Nix on macOS should not even be a thing.
Don’t forget about server reproducibility! This is especially nice when you need to spin up multiple servers for a single project that need similar configuration.
In which case I either have docker/kurbernetes/podman or I’m using ansible to be fast and productive. Sure you may get some stuff done much more precisely in nixos, but that’s definitely not worth the hassle. That said: Best of luck to nixos, hopefully it’ll be stable enough one day.
Wait, what hassle? And what about it isn’t “stable?” Citation needed? It’s stable enough for every NixOS user I know on server and desktop and it’s stable enough for me. Hassle is a very vague word that could mean a lot of things, so if not for the fact that exactly zero of those possible meanings make what you said a true statement, I wouldn’t know how to respond to this. What is it about NixOS you think is a hassle?
Heh, my previous company went from NixOS to “let’s deploy using ansible on Ubuntu boxes, everyone knows those”. Productivity and velocity just went down the drain. Oh, the horrors, oh the PTSD… But everyone has different experiences, sometimes some tools work, sometimes they don’t.
I don’t know how you could get any more precise than NixOS. It specifies everything by hash, all the way down to a linker. I’ve never seen anybody do anything like that with any other system.
Definitely. Every project I use now has a
shell.nixfile that pins the tools I use. I switched to that workflow after I was bitten several times by brew replacing python, so virtual environments were not working, or completely forgetting what tools I need in a project (after returning to it a year later).shell.nixis acting both as a bill of materials and recipe for fetching the right tools.For me that work-around is ‘containers’. As in nix generates the containers (reproducible!), the clusters run the containers, and in the containers it’s /nix.
Check if you can get the following to print “YES”:
or any of the following to print
CONFIG_USER_NS=y(they all check the same thing IIUC, just on various distributions some commands or paths differ):If so, there’s reportedly a chance you might be able to install Nix without root permissions.
If you manage to get it, personally, I would heartily recommend trying to get into “Nix Flakes” as soon as possible. They’re theoretically still “experimental”, and even harder to find reliable documentation about than Nix itself (which is somewhat infamously not-easy already), but IMO they seem to kinda magically solve a lot of auxillary pain points I had with “classic” Nix.
Also, as a side note, the
nix-shellstuff was apparently much earlier than NixOS. The original thesis and project was just Nix, and NixOS started later as another guy’s crazy experiment, AFAIU.EDIT: If that fails, you could still try playing with @ac’s experimental https://github.com/andrewchambers/p2pkgs.
EDIT 2: Finally, although with much less fancy features, there are still some interesting encapsulation aspects in 0install.net: “0install also has some interesting features not often found in traditional package managers. For example, while it will share libraries whenever possible, it can always install multiple versions of a package in parallel when there are conflicting requirements. Installation is always side-effect-free (each package is unpacked to its own directory and will not touch shared directories such as /usr/bin), making it ideal for use with sandboxing technologies and virtualisation.”
This article did sent me on search for “how do you flakes and nix-shell at the same time” and apparently there’s “nix develop” now. This link has some details for both system wide and local setups: https://www.tweag.io/blog/2020-07-31-nixos-flakes/
Yeah, there’s also https://nixos.wiki/wiki/Flakes, and generally that’s the issue, that you need quite some google-fu to find stuff about them :)
Been thinking about standardizing on asdf+direnv. Could anyone offer a quick comparison?
It sounds like Nix can also build your containers for you based on your project definition?
asdf works fine for pinning runtimes until you have system libraries, etc that extensions link against which aren’t versioned with
asdf. Then you’re back in the same boat as you are with brew, etc where upgrading might break existing workdirs.Yep basically just something like this, lots of assumptions given with this and that you “want” to containerize the hello world program gnu ships but eh its an example:
There are caveats to using the docker builds (can’t build on macos) and you’ll need to learn the nix programming language at some point but its a rather droll affair IME once you get that its all just data and functions. And before you ask why is it so big, the short answer is everything that hello defined it depends on is included, which includes jq/pigz/jshon/perl/moreutils etc… for some reason. But its basically lifted straight out of the nix store verbatim.
I recognise this list. These are the dependencies used in the shell scripts which build the Docker image. They shouldn’t be included in the image itself.
They won’t be included in the image if unused.
Have I just been building docker images wrong then this whole time?
Yup. Nix is a fantastic way to build docker images. For example https://gitlab.com/kevincox/dontsayit-api/-/blob/46cbc50038dfd3d76fee2e458a4503c646b8ff2c/default.nix#L23-35 (nd older project but good example because it has more than just a single binary) creates an image with:
Of course if I used musl libc than glibc and its dependencies would go away automatically.
What’s better is that if you use
buildLayeredImageeach of these is a separate layer so that rebuilding for example the word list, or the binary doesn’t require rebuilding other layers. (This is actually better than docker itself because docker only supports linear layering, so you would have to decide if the word list or the binary is the top layer and rebuilding the lower would force a rebuild of the higher one.)There is also https://nixery.dev/ which allows you to build a container with the necessary tools as easy as just properly naming them. For example:
Will bring you in a container that has a shell, git, and htop.
I usually use the web search here: https://search.nixos.org/packages
Getting
shell.nixset up for projects is really nice, but I jump around a lot between different codebases in different languages where it’s often not worth it (or wanted by the owners) to set it up. Thenix-shellcommand that I use all the time for one-offs:…and now I have
package-name, and it’s gone when I exit the shell. Well, it’s still in my local nix-store, so it’s almost instant tonix-shell -pit again, but otherwise it isn’t really installed.Recent one in my shell history:
nix-shell -p nodejs. Specific node version:nix-shell -p nodejs-10_x. Wanted to try some python script depending on matplotlib on a mac😱nix-shell -p python3Packages.matplotlib. Evennix-shell -p inkscapeworks.I use nix-shell similarly, but never commit
shell.nixto the repository. So I get the benefits of having the tools, but I don’t draw ire of other devs :)New nix has the
nix searchcommand. It used to be thatnix search <term>would do the search, but now (nix 2.4) it’snix search nixpkgs <term>.the shebangs are my favourite part of nix-shell, you can do stuff like executable haskell files with all their deps: https://github.com/guibou/nixGL/blob/c4aa5aa15af5d75e2f614a70063a2d341e8e3461/Test.hs#L2