I’m really surprised by the amount of negativity here. This post shows a problem and a solution. It’s short and sweet and demonstrates a concrete application of Nix.
Saying that you can do this with Docker, you can do this with virtualenv, you can rewrite the entire script in Python 3 – what are those statements adding? The article does not claim this is the only way to do this or this is the single morally correct way. The article says “I had a problem and I solved it like this.” Of course there are other ways to solve the same problem. It’s worth discussing the relative merits of different approaches to the same problem, but remember what that problem is: I found a script on the internet and I want to run it.
Anyway: great first post @maxdeviant; hope to see you around more. If the response here did not just turn you off to lobsters completely.
Starting with the Python scaffold script makes this look, I think, a little more intimidating than it has to be. A shell.nix file is great if you’re sharing this with other people, but for just running a one-off script, you can get by with an even-more-ad-hoc approach:
Since you don’t need all the rest of that Python scaffolding to run this script.
You could also put the withPackages bit in the script’s shebang, if you were likely to run it multiple times, so you don’t have to remember all of its dependencies later if you come back to it. (Of course, a shell.nix file gets you the same thing.)
(Splitting this across two lines is required on macOS, but probably not on Linux. Also for some reason you have to use double quotes; nix-shell -i can’t deal with single quotes.)
It’s also worth noting, maybe, that this doesn’t work with nixos-unstable right now. You can pretty cheaply pin by adding -I nixpkgs=https://nixos.org/channels/nixos-21.05/nixexprs.tar.xz to that nix-shell invocation (er, to the second one, in the shebanged version).
Anyway: great first post @maxdeviant; hope to see you around more. If the response here did not just turn you off to lobsters completely.
Thanks, I appreciate the kind words!
And I haven’t been turned off completely. I’ll just remember to post articles with a bit more substance the next time around 😅
In truth, this was primarily a way for me to document this for myself should I need to do something similar again. But I thought it was interesting enough that folks here on Lobsters might enjoy it.
A shell.nix file is great if you’re sharing this with other people, but for just running a one-off script, you can get by with an even-more-ad-hoc approach:
This would have been a much simpler solution! I sort of figured there was a more elegant way to do this, but the shell.nix from the NixOS Wiki was the first thing that turned up in my search.
This post shows a problem and a solution. It’s short and sweet and demonstrates a concrete application of Nix.
Unfortunately, it is a use case where it happens to be easy. In perfect circumstances, Nix can indeed be so simple. However, it paints an overly optimistic picture and is somewhat misleading.
For example, what this post does not tell you is that nixpkgs only packages one version of each Python package (with a very small number of exceptions, such as Tensorflow). There are good reasons for this – Python cannot handle multiple versions in PYTHONPATH and having multiple versions in a transitive closure of dependencies could happen easily. To avoid this issue, nixpkgs only contains one version of each Python package.
If you need a version of a Python package that is not in nixpkgs (which unfortunately happens often, since Python packages rarely conform to semver), then you are off in the woods. You have to package that particular version yourself. If other packages also need the Python package, you need to overlay your package definition and hope/pray that it is compatible with the other derivations in the dependency graph. Or you need to use tools like poetry2nix (hopefully the project uses Poetry) or mach-nix. Sometimes these tools work immediately, but as you can see in the Nix forums they often require a lot of manual overrides, etc. (which in turn requires that you know Nix and nixpkgs pretty well).
If you had instead used a virtual environment + pip, you could have probably gotten the version you need with a single pip install.
Disclaimer: I love Nix and I have contributed hundreds of commits to the nixpkgs tree, but I think it is good to manage expectations.
Not with NixOS. It does not have a traditional file system hierarchy, so Python packages that rely on traditional paths (e.g. compiled C modules that expect libc or other dynamic libraries to be in a location that FHS dictates) will not work.
You can get fairly closely by emulating a FHS using e.g. buildFHSUserEnv.
I recently did a similar thing for a friend’s ancient PHP site while upgrading it to a supported version of the runtime.
That specific PHP 5.6.x release was no longer in nixpkgs. Of course, it being a git repo, allowed me to go back in history, find the last version that had my PHP release in, import it in a nix expression, and BOOM - it just worked.
I did the same for MySQL. Then branched a version of the site that used newer PHP and MySQL. That allowed me to quickly switch between the two environments and test things.
10/10 experience. Few people realize that every nixpkgs commit since the beginning of the project is a full Linux distro that you can modify to suit your needs.
I love reading these Nix success stories and then last night trying to simply install Grub on a system with zfs filesystem being literally impossible with the way my Nixos system was failing to derive something for no discernible reason, with no documentation anywhere and any reportings of that issue completely ignored :D
I should just ruin it all and tell my secret. I don’t run NixOS. Plain nixpkgs on an LTS Ubuntu (boomer edition!) all the way.
I use Nix as a development helper, and, on rare occasions, deploy a service by sticking it in a systemd unit file and nix-copy-closure my way to success. Of course, that’s just for my gear. At $DAYJOB it’s the usual k8s circus.
Nix is one of those technologies I think is amazing, but at the same time is practically unusable because of hard semantics and an inscrutable command line interface. It’s kind of like rust: dipping your toes and getting the tutorial working is easy enough, but the first time you are confronted with a real problem, finding the solution requires so much ancillary knowledge of arcane the minutiae of how the system works, it becomes a waste of time and solving it ‘the way I know’ is easier.
It’s great to see folks talking about the practical immediate benefits you can get from nix. OP I hope you don’t let some of the negativity here get you down too much!
For the detractors: Nix is one of those things where the benefits can be really hard to see at first. So much of what nix does can be done in other ways- so I think sometimes miss out that the real benefit of nix is that it offers a single cohesive and principled way to solve classes of problems that might otherwise require several different dedicated tools. Even narrowing down your view to just the case of version or dependency management, you can do most of what you’d do in nix with virtualenv or go modules or cabal sandboxes or cargo or whatever, but then you are dealing with the individual quirks of each of those different tools. Go up a level and you might also need to deal with alternatives or whatever other mechanism your particular OS uses to select between different system level packages. Up another level and you have to think about versions of docker containers or AMI images. Nix lets you handle all of that configuration coherently, with a single language and set of tools. Buying into nix for a project, your OS, or your entire approach to infrastructure can feel odd at first, but it has a huge force multiplier effect with itself when you buy into it.
What’s going on here? How did this get to the top of lobste.rs with 26 upvotes? I’m happy for the OP that they could get their system to work, but as far as I can tell, the story here is “package manager used to manage packages.” We have been doing that for decades. Is there any way the community can get a lever to push back on thin stories like this one?
Would it change your opinion if the article mentioned that the nix shell being used here is entirely disposable and this process leaves no mark in your OS setup? Also that even if this required some obscure versions of common system dependencies you could drop into such a shell without worrying about version conflicts or messing up your conventional package manager?
I agree that the article is thin in content, but I don’t think you can write this story off as “package manager used to manage packages.” , I think nix shell is very magical in the package management world.
Yes, but then you’d be inside a container, so you’d have to deal with the complexities of that, like mounting drives, routing network traffic etc. With nix shell, you’re not really isolated, you’re just inside a shell session that has the necessary environment variables that provide just the packages you’ve asked for.
Aside from the isolation, the nix shell is also much more composable. It can drop you into a shell that simultaneously has a strange Java, python and Erlang environment all compiled with your personal fork of GCC, and you’d just have to specify your GCC as an override for that to happen.
I get that, but I have to go through the learning curve of nix-shell, while I already know docker, since I need it for my job anyway. I am saying that there are more ways to achieve what the article is talking about. It is fine that the author is happy with their choice of tools, but it is very unremarkable for the title and given how many upvotes that article got.
Why not learn nix and then use it at work as well :) Nix knows how to package up a nix-defined environment into a docker container and produce very small images, and you don’t even need docker itself to do that. That’s what we do at work. I’m happy because as far as I’m concerned Nix is all there is and the DevOps folks are also happy because they get their docker images.
I think that the key takeaway is that with Docker, you’re actually running a container will a full-blown OS inside. I have a bias against it, which is basically just my opinion, so take it with a grain of salt.
I think that once the way to solve the problem of I need to run some specific version of X becomes let’s just virtualize a whole computer and OS because dependency handling is broken anyway, we, as a category simply gave up. It is side-stepping the problem.
Now, the approach with Nix is much more elegant. You have fully reproducible dependency graphs, and with nix-shell you can drop yourself in an environment that is suitable for whatever you need to run regardless of dependency conflicts. It is quite neat, and those shells are disposable. You’re not running in a container, you’re not virtualizing the OS, you’re just loading a different dependency graph in your context.
See, I don’t use Nix at all because I don’t have these needs, but I played with it and was impressed. I dislike our current approach of just run a container, it feels clunky to me. I think Docker has it’s place, specially in DevOps and stuff, but using it to solve the I need to run Python 2.x and stuff conflicts with my Python 3.x install is not the way I’d like to see our ecosystem going.
In the end, from a very high-level, almost stratospheric, point-of-view: both docker and nix-shell workflow will be the developer typing some commands on the terminal, and having what they need running. So from a mechanical standpoint of needing to run something, they’ll both solve the problem. I just don’t like how solving things by doing the evergreen is now the preferred solution.
Just be aware that this is an opinion from someone heavily biased against containers. You should play with both of them and decide for yourself.
This comment is a very good description of why I’ve never tried Docker (and – full disclosure – use Nix for things like this).
But what I’m really asking – although I didn’t make this explicit – is a comparison of the ergonomics. The original post shows the shell.nix file that does this (although as I point out in another comment, there’s a shell one-liner that gets you the same thing). Is there an equivalent Dockerfile?
I was surprised to see Docker brought up at all because my (uninformed) assumption is that making a Docker image would be prohibitively slow or difficult for a one-off like this. I assumed it would be clunky to start a VM just to run a single script with a couple dependencies. But the fact that that was offered as an alternative to nix-shell makes me think that I’m wrong, and that Docker might be appropriate for more ad-hoc things than I expected, which makes me curious what that looks like. It points out a gap in my understanding that I’d like to fill… with as little exertion of effort as possible. :)
But the fact that that was offered as an alternative to nix-shell makes me think that I’m wrong, and that Docker might be appropriate for more ad-hoc things than I expected, which makes me curious what that looks like. It points out a gap in my understanding that I’d like to fill… with as little exertion of effort as possible. :)
I think containers is a perfectly capable solution to this. The closest thing you can use would probably be toolbox.
It would allow you to even provide a standardized environment which would be decoupled from the deployment itself (if that makes sense). It also mount $HOME as well.
I would recommend most people to use Toolbox over nix-shell. With toolbox you can create one-off containers in literally seconds (it’s two commands). After entering the container you can just dnf install whatever you need. Your home directory gets mounted, so you do not have to juggle with volumes, etc. If you need to create the same environment more often, you can create a Dockerfile and build your toolbox containers with podman. The upstream containers that Fedora provides are also just built using Dockerfiles.
The post shows a simple use case, but if you want to do something less trivial, it often entails learning Nix the language and nixpkgs (and all its functions, idioms, etc.). And the Nix learning curve is steep (though it is much simpler if you are familiar with functional programming). This makes the toolbox approach orders of magnitude easier for most people - you basically need to know toolbox create and toolbox enter and you can use all the knowledge that you already have.
However, a very large shortcoming of toolbox/Dockerfiles/etc. is reproducibility. Sure, you can pass around an image and someone else will have the same environment. But Nix allows you to pin all dependencies plus the derivations (e.g. as a git SHA). You can give someone your Nix flake and they will have exactly the same dependency graph and build environment guaranteed.
Another difference is that once you know Nix, it is immensely powerful for defining packages. Nix is a turing-complete functional language, so nixpkgs can provide a lot of powerful abstractions. I dread every time I have to create/modify and RPM spec file, because it is so primitive compared to making a Nix derivation.
tl;dr: most people will want to use something like Toolbox, it is familiar and provides many of the same benefits as e.g. nix-shell (isolated, throw-away environments, with your home directory available). However, if you want strong reproduciblity across systems and a more powerful packaging/configuration language, learning Nix is worth it.
A cool aspect of Docker is that it has a gazillion images already built and available for it. So depending on what you need, you’ll find a ready-made image you can put to good use with a single command. If there are no images that fill your exact need, then you’ll probably find an image that is close enough and can be customised. You don’t need to create images from scratch. You can remix what is already available. In terms of ergonomics, it is friendly and easy to use (for these simple cases).
So, NixPkgs have a steeper learning curve in comparison to dockerfiles. It might be simpler to just run Docker. What I don’t like is what is happening inside Docker, and how the solution for what looks like simple problems involves running a whole OS.
I’m aware that you can have containers without an OS like described in this thread, but that is not something I often see people using in the wild.
Let’s assume nix shell is actual magic — like sourcerer level, wave my hand and airplanes become dragons (or vice versa) magic — well this article just demonstrated that immense power by pulling a coin out of a deeply uncomfortable kid’s ear while pulling on her nose.
I can’t speak for the previous comment’s author, but those extra details, or indeed any meat on the bones, would definitely help justify this article’s otherwise nonsensical ranking.
Yeah, I agree with your assessment. This article could just as well have the title “MacOS is so fragile, I consider this simple thing to be an issue”. The trouble with demonstrating nix shell’s power is that for all the common cases, you have a variety of ad-hoc solutions. And the truly complex cases appear contrived out of context (see my other comment, which you may or may not consider to be turning airplanes into dragons).
nix is not the first thing most devs would think of when faced with that particular problem, so it’s interesting to see reasons to add it to your toolbox.
Good, as it is not supposed to be the first thing. Learning a fringe system with a new syntax just to do something trivial is not supposed to be the first thing at all.
I find it also baffling that this story has more upvotes than the excellent and original code visualization article currently also very high. Probably some nix up vote ring pushing this
I didn’t think this article was amazing, but I found it more interesting than the code visualization one, which lost me at the first, “From this picture, you can immediately see that X,” and I had to search around the picture for longer than it would have taken me to construct a find command to find the X it was talking about.
This article, at least, caused me to say, “Oh, that’s kind of neat, wouldn’t have thought of using that.”
This article is useless. It is way simpler (and the python way) to just create a 2.7 virtualenv and run “pip install psycopg2 graphwiz”. No need to write a nix file, and then write a blog post to convince yourself you didn’t waste your time!
Considering all nix posts get upvoted regardless of content, it’s about time we have a “nix” tag added to the site.
This article is not useless just because you don’t see its value.
I work mainly with Ruby and have to deal with old projects. There are multiple instances where the Ruby way (using a Ruby version manager) did not work because it was unable to install an old Ruby version or gem on my new development machine. Using a nix-shell did the job every time.
just create a 2.7 virtualenv and run “pip install psycopg2 graphwiz”
What do you do if this fails due to some obscure dependency problem?
What do you do if this fails due to some obscure dependency problem?
Arguably you solve it by pinning dependency versions in the pip install invocation or requirements.txt, as any Python developer not already using Nix would do.
This article is not useless just because you don’t see its value.
No, but it is fairly useless because it doesn’t do anything to establish that value, except to the choir.
In my experience there will be a point where your dependencies will fail due to mismatching OpenSSL, glibc versions and so on. No amount of pinning dependencies will protect you against that. The only way out is to update dependencies and the version of your language. But that would just detract from your goal of getting an old project to run or is straight up impossible.
Enter Nix: You pin the entire environment in which your program will run. In addition you don’t pollute your development machine with different versions of libraries.
Arguably that’s just shifting the burden of effort based on a value judgement. If your goal is to get an old project to run while emphasizing the value of incurring zero effort in updating it, then obviously Nix is a solution for you and you’ll instead put the effort into pinning its entire runtime environment. If, however, your value to emphasize is getting the project to run then it may well be a more fruitful choice to put the effort into updating the project.
The article doesn’t talk about any of the hairier details you’re speaking to, it just shows someone taking a slightly out of date Python project and not wanting to put any personal effort into updating it… but updating it by writing a (in this case relatively trivial) Python 3 version and making that publicly available to others would arguably be the “better” solution, at least in terms of the value of contributing back to the community whose work you’re using.
But ultimately my argument isn’t with the idea that Nix is a good solution to a specific problem, it’s that this particular article doesn’t really make that point and certainly doesn’t convincingly demonstrate the value of adding another complex bit of tooling to the toolkit. All the points you’ve raised would certainly help make that argument, but they’re not sadly not present in this particular article.
Just out of curiosity, I’m also dealing with ancient ruby versions and use nix at work but I couldn’t figure out how to get old enough versions, is there something that helps with that?
Do note this method will get you a ruby linked to dependencies from the same checkout. In many cases this is what you want.
If instead you want an older ruby but linked to newer libraries (eg, OpenSSL) there’s a few extra steps, but this is a great jumping off point to finding derivations to fork.
Do note this method will get you a ruby linked to dependencies from the same checkout. In many cases this is what you want.
Plus glibc, OpenSSL and other dependencies with many known vulnerabilities. This is fine for local stuff, but definitely not something you’d want to do for anything that is publicly visible.
Also, note that mixing different nixpkgs versions does not work when an application uses OpenGL, Vulkan, or any GPU-related drivers/libraries. The graphics stack is global state in Nix/NixOS and mixing software with different glibc versions quickly goes awry.
This comment mentions having done something similar with older versions by checking out an older version of the nixpkgs repo that had the version of the language that they needed.
Like others already said you can just pin nixpkgs. Sometimes there is more work involved. For example this is the current shell.nix for a Ruby on Rails project that wasn’t touched for 5 years. I’m in the process of setting up a reproducible development environment to get development going again. As you can see I have to jump through hoops to get Nokogiri play nicely.
this example, perhaps. I recently contributed to a python 2 code base and running it locally was very difficult due to c library dependencies. The best I could do at the time was a Dockerfile (which I contributed with my changes) to encapsulate the environment. However, even with the container standpoint, fetching dependencies is still just as nebulous as “just apt install xyz.” Changes to the base image, an ambiently available dependency or simply turning off the distro package manager services for unsupported versions will break the container build.
In the nix case, it is sort of forced on the user to spell it out completely what the code needs, combine with flakes and I have a lockfile not only for my python dependencies, but effectively the entire shell environment.
More concretely, at work, the powers to be wanted to deploy python to an old armv7 SoC running on a device.
Some of the python code requires c dependencies like openssl, protobuf runtime and other things and it was hard to cross compile this for the target. Yes, for development it works as you describe, you just use a venv, pip install (pipenv, poetry, or whatever as well) and everything is peachy. then comes to deployment:
First you need to make a cross-compiled python interpreter, which involves first building the interpreter for your host triple then rebuilding the same source for the target host triple making sure to tell the build process where the host triple build is. This also ignores that some important python interpreter things may not build, like ctypes.
Learn every environment variable you need to expose to the setup.py or the n-teenth build / packaging solution for the python project you want to deploy, hope it generates a wheel. We will conveniently ignore how every C depending package may use cmake, or make, or meson, etc, etc…
make the wheels available to the image you actually ship.
I was able to crap out a proof-of-concept in a small nix expression that made a shell that ran the python interpreter I wanted with the python dependencies needed on both the host and the target and didn’t even have to think. Nixpkgs even gives you cross compiling capabilities.
Your suggested plan is two years out of date, because CPython 2.7 is officially past its end of life and Python 2 packages are generally no longer supported by upstream developers. This is the power of Nix: Old software continues to be available, as if bitrot were extremely delayed.
CPython 2.7 is available in debian stable (even testing and sid!), centos and rhel. Even on MacOS it is still the default python, that ships witht he system. I don’t know why you think it is no longer available in any distro other than nix.
There are a couple reasons why this is getting such a strong response:
It’s difficult to demonstrate the uses of reproducibility. You probably don’t care about nix’s guarantees unless you’ve done sysadmin-type work. Or perhaps you have some interest in distributed computing, formal verification, OS dev, etc.
There’s a dearth of documentation on nix. It’s rare to see a succinct article with some example code. Normally, you have to hunt through GitHub to figure things out.
Most developers have dealt with package management. It’s foundational to modern software development, so we have established workflows that work for us. We have learned to accept occasional as ‘the cost of getting things done’, so the example in this blog post appears unremarkable.
Anyway, I enjoyed it but I’m a NixOS user, so I’m a sucker for any thoughtfully put-together nix code (as per bullet point #2).
Wouldn’t it have been better to simply re-write it to work in Python 3? The script doesn’t scream crazy outdated Python 2 program to me, it would take a bit of work to get it working, but I don’t think it would be hard. Instead now it’s coupled with a Nix shell, which is a lot harder to share with other people who don’t use Nix or don’t want to start learning how to.
I’m not trying to hate on Nix here, but it’s just strange to me that you went this route instead of updating the script to work with a more recent version of Python. It seems backwards to have to bend to a 10+ year old version of Python instead of simply updating it.
Author here. I don’t regularly use Python and haven’t setup a virtualenv in years, so in this case it seemed more arduous to port the script and try to wrangle the dependencies in a Pythonic way.
I do, however, use Nix regularly, and it took me just a couple minutes to get the script up and running, dump the output I needed, and move on with my work.
The argument that the code is written for python2.7 is pretty poor seems just print statement is requiring python2 adding a pair of parenthesis is not so hard
Its not just the print statement that changed between Python 2 and 3. There’s a number of subtle differences that might affect the semantics of a program, and I wouldn’t want to debug those in someone else’s project I was using to quickly check one thing.
I’m pretty sure I have read a few month ago that Python 2.7 was deprecated once and for all after a more than a decade long agony, in favor of Python 3.x flavour.
Not sure it is a great idea to have a scaffolding file targetting at python 2.7.
Sometimes you don’t have the luxury of forcing people to upgrade random things from the past they’ve forgotten about. Having an option to run it at all is better than not.
Totally agreed that “to run it at all is better than not”.
My concern was rather more about the fact that it was an example file in Nix documentation for Python projects with pip and virtualenv, that is: a quasi-standard common python project.
Only this made me think of the mention of “2.7” worth a comment here.
I’m really surprised by the amount of negativity here. This post shows a problem and a solution. It’s short and sweet and demonstrates a concrete application of Nix.
Saying that you can do this with Docker, you can do this with
virtualenv, you can rewrite the entire script in Python 3 – what are those statements adding? The article does not claim this is the only way to do this or this is the single morally correct way. The article says “I had a problem and I solved it like this.” Of course there are other ways to solve the same problem. It’s worth discussing the relative merits of different approaches to the same problem, but remember what that problem is: I found a script on the internet and I want to run it.Anyway: great first post @maxdeviant; hope to see you around more. If the response here did not just turn you off to lobsters completely.
Starting with the Python scaffold script makes this look, I think, a little more intimidating than it has to be. A
shell.nixfile is great if you’re sharing this with other people, but for just running a one-off script, you can get by with an even-more-ad-hoc approach:Since you don’t need all the rest of that Python scaffolding to run this script.
You could also put the
withPackagesbit in the script’s shebang, if you were likely to run it multiple times, so you don’t have to remember all of its dependencies later if you come back to it. (Of course, ashell.nixfile gets you the same thing.)(Splitting this across two lines is required on macOS, but probably not on Linux. Also for some reason you have to use double quotes;
nix-shell -ican’t deal with single quotes.)It’s also worth noting, maybe, that this doesn’t work with
nixos-unstableright now. You can pretty cheaply pin by adding-I nixpkgs=https://nixos.org/channels/nixos-21.05/nixexprs.tar.xzto thatnix-shellinvocation (er, to the second one, in the shebanged version).Thanks, I appreciate the kind words!
And I haven’t been turned off completely. I’ll just remember to post articles with a bit more substance the next time around 😅
In truth, this was primarily a way for me to document this for myself should I need to do something similar again. But I thought it was interesting enough that folks here on Lobsters might enjoy it.
This would have been a much simpler solution! I sort of figured there was a more elegant way to do this, but the
shell.nixfrom the NixOS Wiki was the first thing that turned up in my search.This post shows a problem and a solution. It’s short and sweet and demonstrates a concrete application of Nix.
Unfortunately, it is a use case where it happens to be easy. In perfect circumstances, Nix can indeed be so simple. However, it paints an overly optimistic picture and is somewhat misleading.
For example, what this post does not tell you is that
nixpkgsonly packages one version of each Python package (with a very small number of exceptions, such as Tensorflow). There are good reasons for this – Python cannot handle multiple versions inPYTHONPATHand having multiple versions in a transitive closure of dependencies could happen easily. To avoid this issue,nixpkgsonly contains one version of each Python package.If you need a version of a Python package that is not in nixpkgs (which unfortunately happens often, since Python packages rarely conform to semver), then you are off in the woods. You have to package that particular version yourself. If other packages also need the Python package, you need to overlay your package definition and hope/pray that it is compatible with the other derivations in the dependency graph. Or you need to use tools like
poetry2nix(hopefully the project uses Poetry) ormach-nix. Sometimes these tools work immediately, but as you can see in the Nix forums they often require a lot of manual overrides, etc. (which in turn requires that you know Nix and nixpkgs pretty well).If you had instead used a virtual environment + pip, you could have probably gotten the version you need with a single
pip install.Disclaimer: I love Nix and I have contributed hundreds of commits to the nixpkgs tree, but I think it is good to manage expectations.
Can’t you simply open up a shell with the desired python version, make a venv, then source activate and pip install?
Not with NixOS. It does not have a traditional file system hierarchy, so Python packages that rely on traditional paths (e.g. compiled C modules that expect libc or other dynamic libraries to be in a location that FHS dictates) will not work.
You can get fairly closely by emulating a FHS using e.g.
buildFHSUserEnv.I recently did a similar thing for a friend’s ancient PHP site while upgrading it to a supported version of the runtime.
That specific PHP 5.6.x release was no longer in nixpkgs. Of course, it being a git repo, allowed me to go back in history, find the last version that had my PHP release in, import it in a nix expression, and BOOM - it just worked.
I did the same for MySQL. Then branched a version of the site that used newer PHP and MySQL. That allowed me to quickly switch between the two environments and test things.
10/10 experience. Few people realize that every nixpkgs commit since the beginning of the project is a full Linux distro that you can modify to suit your needs.
I love reading these Nix success stories and then last night trying to simply install Grub on a system with zfs filesystem being literally impossible with the way my Nixos system was failing to derive something for no discernible reason, with no documentation anywhere and any reportings of that issue completely ignored :D
I should just ruin it all and tell my secret. I don’t run NixOS. Plain nixpkgs on an LTS Ubuntu (boomer edition!) all the way.
I use Nix as a development helper, and, on rare occasions, deploy a service by sticking it in a systemd unit file and nix-copy-closure my way to success. Of course, that’s just for my gear. At $DAYJOB it’s the usual k8s circus.
Setting up Linux to boot from ZFS root is tricky even under the best of circumstances. What was the reported issue?
I’m a huge fan of both NixOS and ZFS, but in the future I might aim for a tmpfs root and use ZFS only for persistent data.
Nix is one of those technologies I think is amazing, but at the same time is practically unusable because of hard semantics and an inscrutable command line interface. It’s kind of like rust: dipping your toes and getting the tutorial working is easy enough, but the first time you are confronted with a real problem, finding the solution requires so much ancillary knowledge of arcane the minutiae of how the system works, it becomes a waste of time and solving it ‘the way I know’ is easier.
It’s great to see folks talking about the practical immediate benefits you can get from nix. OP I hope you don’t let some of the negativity here get you down too much!
For the detractors: Nix is one of those things where the benefits can be really hard to see at first. So much of what nix does can be done in other ways- so I think sometimes miss out that the real benefit of nix is that it offers a single cohesive and principled way to solve classes of problems that might otherwise require several different dedicated tools. Even narrowing down your view to just the case of version or dependency management, you can do most of what you’d do in nix with virtualenv or go modules or cabal sandboxes or cargo or whatever, but then you are dealing with the individual quirks of each of those different tools. Go up a level and you might also need to deal with alternatives or whatever other mechanism your particular OS uses to select between different system level packages. Up another level and you have to think about versions of docker containers or AMI images. Nix lets you handle all of that configuration coherently, with a single language and set of tools. Buying into nix for a project, your OS, or your entire approach to infrastructure can feel odd at first, but it has a huge force multiplier effect with itself when you buy into it.
What’s going on here? How did this get to the top of lobste.rs with 26 upvotes? I’m happy for the OP that they could get their system to work, but as far as I can tell, the story here is “package manager used to manage packages.” We have been doing that for decades. Is there any way the community can get a lever to push back on thin stories like this one?
Would it change your opinion if the article mentioned that the nix shell being used here is entirely disposable and this process leaves no mark in your OS setup? Also that even if this required some obscure versions of common system dependencies you could drop into such a shell without worrying about version conflicts or messing up your conventional package manager?
I agree that the article is thin in content, but I don’t think you can write this story off as “package manager used to manage packages.” , I think nix shell is very magical in the package management world.
I could do that with docker too and it would not leave a trace either
Yes, but then you’d be inside a container, so you’d have to deal with the complexities of that, like mounting drives, routing network traffic etc. With nix shell, you’re not really isolated, you’re just inside a shell session that has the necessary environment variables that provide just the packages you’ve asked for.
Aside from the isolation, the nix shell is also much more composable. It can drop you into a shell that simultaneously has a strange Java, python and Erlang environment all compiled with your personal fork of GCC, and you’d just have to specify your GCC as an override for that to happen.
I get that, but I have to go through the learning curve of nix-shell, while I already know docker, since I need it for my job anyway. I am saying that there are more ways to achieve what the article is talking about. It is fine that the author is happy with their choice of tools, but it is very unremarkable for the title and given how many upvotes that article got.
Why not learn nix and then use it at work as well :) Nix knows how to package up a nix-defined environment into a docker container and produce very small images, and you don’t even need docker itself to do that. That’s what we do at work. I’m happy because as far as I’m concerned Nix is all there is and the DevOps folks are also happy because they get their docker images.
I work in a humongous company where the tools and things are less free to choose from atm, so even if I learned nix, it would be a very tough sell..
As someone who hasn’t used Docker, it would be nice to see what that looks like. I’m curious how the two approaches compare.
I think that the key takeaway is that with Docker, you’re actually running a container will a full-blown OS inside. I have a bias against it, which is basically just my opinion, so take it with a grain of salt.
I think that once the way to solve the problem of I need to run some specific version of X becomes let’s just virtualize a whole computer and OS because dependency handling is broken anyway, we, as a category simply gave up. It is side-stepping the problem.
Now, the approach with Nix is much more elegant. You have fully reproducible dependency graphs, and with nix-shell you can drop yourself in an environment that is suitable for whatever you need to run regardless of dependency conflicts. It is quite neat, and those shells are disposable. You’re not running in a container, you’re not virtualizing the OS, you’re just loading a different dependency graph in your context.
See, I don’t use Nix at all because I don’t have these needs, but I played with it and was impressed. I dislike our current approach of just run a container, it feels clunky to me. I think Docker has it’s place, specially in DevOps and stuff, but using it to solve the I need to run Python 2.x and stuff conflicts with my Python 3.x install is not the way I’d like to see our ecosystem going.
In the end, from a very high-level, almost stratospheric, point-of-view: both docker and nix-shell workflow will be the developer typing some commands on the terminal, and having what they need running. So from a mechanical standpoint of needing to run something, they’ll both solve the problem. I just don’t like how solving things by doing the evergreen is now the preferred solution.
Just be aware that this is an opinion from someone heavily biased against containers. You should play with both of them and decide for yourself.
This comment is a very good description of why I’ve never tried Docker (and – full disclosure – use Nix for things like this).
But what I’m really asking – although I didn’t make this explicit – is a comparison of the ergonomics. The original post shows the
shell.nixfile that does this (although as I point out in another comment, there’s a shell one-liner that gets you the same thing). Is there an equivalent Dockerfile?I was surprised to see Docker brought up at all because my (uninformed) assumption is that making a Docker image would be prohibitively slow or difficult for a one-off like this. I assumed it would be clunky to start a VM just to run a single script with a couple dependencies. But the fact that that was offered as an alternative to
nix-shellmakes me think that I’m wrong, and that Docker might be appropriate for more ad-hoc things than I expected, which makes me curious what that looks like. It points out a gap in my understanding that I’d like to fill… with as little exertion of effort as possible. :)I think containers is a perfectly capable solution to this. The closest thing you can use would probably be toolbox.
https://github.com/containers/toolbox
It would allow you to even provide a standardized environment which would be decoupled from the deployment itself (if that makes sense). It also mount
$HOMEas well.I use Nix, but also have experience with Toolbox.
I would recommend most people to use Toolbox over
nix-shell. With toolbox you can create one-off containers in literally seconds (it’s two commands). After entering the container you can justdnf installwhatever you need. Your home directory gets mounted, so you do not have to juggle with volumes, etc. If you need to create the same environment more often, you can create aDockerfileand build your toolbox containers withpodman. The upstream containers that Fedora provides are also just built usingDockerfiles.The post shows a simple use case, but if you want to do something less trivial, it often entails learning Nix the language and
nixpkgs(and all its functions, idioms, etc.). And the Nix learning curve is steep (though it is much simpler if you are familiar with functional programming). This makes the toolbox approach orders of magnitude easier for most people - you basically need to knowtoolbox createandtoolbox enterand you can use all the knowledge that you already have.However, a very large shortcoming of toolbox/Dockerfiles/etc. is reproducibility. Sure, you can pass around an image and someone else will have the same environment. But Nix allows you to pin all dependencies plus the derivations (e.g. as a git SHA). You can give someone your Nix flake and they will have exactly the same dependency graph and build environment guaranteed.
Another difference is that once you know Nix, it is immensely powerful for defining packages. Nix is a turing-complete functional language, so nixpkgs can provide a lot of powerful abstractions. I dread every time I have to create/modify and RPM spec file, because it is so primitive compared to making a Nix derivation.
tl;dr: most people will want to use something like Toolbox, it is familiar and provides many of the same benefits as e.g. nix-shell (isolated, throw-away environments, with your home directory available). However, if you want strong reproduciblity across systems and a more powerful packaging/configuration language, learning Nix is worth it.
A cool aspect of Docker is that it has a gazillion images already built and available for it. So depending on what you need, you’ll find a ready-made image you can put to good use with a single command. If there are no images that fill your exact need, then you’ll probably find an image that is close enough and can be customised. You don’t need to create images from scratch. You can remix what is already available. In terms of ergonomics, it is friendly and easy to use (for these simple cases).
So, NixPkgs have a steeper learning curve in comparison to dockerfiles. It might be simpler to just run Docker. What I don’t like is what is happening inside Docker, and how the solution for what looks like simple problems involves running a whole OS.
I’m aware that you can have containers without an OS like described in this thread, but that is not something I often see people using in the wild.
Nit-pick: AFAIK one doesn’t really need Alpine or any other distro inside the container. It’s “merely” for convenience. AFAICT it’s entirely possible to e.g. run a Go application in a container without any distro. See e.g. https://www.cloudbees.com/blog/building-minimal-docker-containers-for-go-applications
Let’s assume
nix shellis actual magic — like sourcerer level, wave my hand and airplanes become dragons (or vice versa) magic — well this article just demonstrated that immense power by pulling a coin out of a deeply uncomfortable kid’s ear while pulling on her nose.I can’t speak for the previous comment’s author, but those extra details, or indeed any meat on the bones, would definitely help justify this article’s otherwise nonsensical ranking.
Yeah, I agree with your assessment. This article could just as well have the title “MacOS is so fragile, I consider this simple thing to be an issue”. The trouble with demonstrating nix shell’s power is that for all the common cases, you have a variety of ad-hoc solutions. And the truly complex cases appear contrived out of context (see my other comment, which you may or may not consider to be turning airplanes into dragons).
nix is not the first thing most devs would think of when faced with that particular problem, so it’s interesting to see reasons to add it to your toolbox.
Good, as it is not supposed to be the first thing. Learning a fringe system with a new syntax just to do something trivial is not supposed to be the first thing at all.
I find it also baffling that this story has more upvotes than the excellent and original code visualization article currently also very high. Probably some nix up vote ring pushing this
Or folks just like Nix I guess? 🤷
Nix is cool and people like it.
I didn’t think this article was amazing, but I found it more interesting than the code visualization one, which lost me at the first, “From this picture, you can immediately see that X,” and I had to search around the picture for longer than it would have taken me to construct a
findcommand to find the X it was talking about.This article, at least, caused me to say, “Oh, that’s kind of neat, wouldn’t have thought of using that.”
This article is useless. It is way simpler (and the python way) to just create a 2.7 virtualenv and run “pip install psycopg2 graphwiz”. No need to write a nix file, and then write a blog post to convince yourself you didn’t waste your time!
Considering all nix posts get upvoted regardless of content, it’s about time we have a “nix” tag added to the site.
This article is not useless just because you don’t see its value.
I work mainly with Ruby and have to deal with old projects. There are multiple instances where the Ruby way (using a Ruby version manager) did not work because it was unable to install an old Ruby version or gem on my new development machine. Using a nix-shell did the job every time.
What do you do if this fails due to some obscure dependency problem?
Arguably you solve it by pinning dependency versions in the
pip installinvocation or requirements.txt, as any Python developer not already using Nix would do.No, but it is fairly useless because it doesn’t do anything to establish that value, except to the choir.
In my experience there will be a point where your dependencies will fail due to mismatching OpenSSL, glibc versions and so on. No amount of pinning dependencies will protect you against that. The only way out is to update dependencies and the version of your language. But that would just detract from your goal of getting an old project to run or is straight up impossible.
Enter Nix: You pin the entire environment in which your program will run. In addition you don’t pollute your development machine with different versions of libraries.
Arguably that’s just shifting the burden of effort based on a value judgement. If your goal is to get an old project to run while emphasizing the value of incurring zero effort in updating it, then obviously Nix is a solution for you and you’ll instead put the effort into pinning its entire runtime environment. If, however, your value to emphasize is getting the project to run then it may well be a more fruitful choice to put the effort into updating the project.
The article doesn’t talk about any of the hairier details you’re speaking to, it just shows someone taking a slightly out of date Python project and not wanting to put any personal effort into updating it… but updating it by writing a (in this case relatively trivial) Python 3 version and making that publicly available to others would arguably be the “better” solution, at least in terms of the value of contributing back to the community whose work you’re using.
But ultimately my argument isn’t with the idea that Nix is a good solution to a specific problem, it’s that this particular article doesn’t really make that point and certainly doesn’t convincingly demonstrate the value of adding another complex bit of tooling to the toolkit. All the points you’ve raised would certainly help make that argument, but they’re not sadly not present in this particular article.
Just out of curiosity, I’m also dealing with ancient ruby versions and use nix at work but I couldn’t figure out how to get old enough versions, is there something that helps with that?
Use https://lazamar.co.uk/nix-versions/
Thank you, very helpful!
Do note this method will get you a ruby linked to dependencies from the same checkout. In many cases this is what you want.
If instead you want an older ruby but linked to newer libraries (eg, OpenSSL) there’s a few extra steps, but this is a great jumping off point to finding derivations to fork.
Plus glibc, OpenSSL and other dependencies with many known vulnerabilities. This is fine for local stuff, but definitely not something you’d want to do for anything that is publicly visible.
Also, note that mixing different nixpkgs versions does not work when an application uses OpenGL, Vulkan, or any GPU-related drivers/libraries. The graphics stack is global state in Nix/NixOS and mixing software with different glibc versions quickly goes awry.
This comment mentions having done something similar with older versions by checking out an older version of the
nixpkgsrepo that had the version of the language that they needed.Like others already said you can just pin
nixpkgs. Sometimes there is more work involved. For example this is the current shell.nix for a Ruby on Rails project that wasn’t touched for 5 years. I’m in the process of setting up a reproducible development environment to get development going again. As you can see I have to jump through hoops to get Nokogiri play nicely.There is also a German blog post with shell.nix examples in case you need inspiration.
this example, perhaps. I recently contributed to a python 2 code base and running it locally was very difficult due to c library dependencies. The best I could do at the time was a Dockerfile (which I contributed with my changes) to encapsulate the environment. However, even with the container standpoint, fetching dependencies is still just as nebulous as “just apt install xyz.” Changes to the base image, an ambiently available dependency or simply turning off the distro package manager services for unsupported versions will break the container build. In the nix case, it is sort of forced on the user to spell it out completely what the code needs, combine with flakes and I have a lockfile not only for my python dependencies, but effectively the entire shell environment.
More concretely, at work, the powers to be wanted to deploy python to an old armv7 SoC running on a device. Some of the python code requires c dependencies like openssl, protobuf runtime and other things and it was hard to cross compile this for the target. Yes, for development it works as you describe, you just use a venv, pip install (pipenv, poetry, or whatever as well) and everything is peachy. then comes to deployment:
I was able to crap out a proof-of-concept in a small nix expression that made a shell that ran the python interpreter I wanted with the python dependencies needed on both the host and the target and didn’t even have to think. Nixpkgs even gives you cross compiling capabilities.
Your suggested plan is two years out of date, because CPython 2.7 is officially past its end of life and Python 2 packages are generally no longer supported by upstream developers. This is the power of Nix: Old software continues to be available, as if bitrot were extremely delayed.
CPython 2.7 is available in debian stable (even testing and sid!), centos and rhel. Even on MacOS it is still the default python, that ships witht he system. I don’t know why you think it is no longer available in any distro other than nix.
There are a couple reasons why this is getting such a strong response:
Anyway, I enjoyed it but I’m a NixOS user, so I’m a sucker for any thoughtfully put-together nix code (as per bullet point #2).
Wouldn’t it have been better to simply re-write it to work in Python 3? The script doesn’t scream crazy outdated Python 2 program to me, it would take a bit of work to get it working, but I don’t think it would be hard. Instead now it’s coupled with a Nix shell, which is a lot harder to share with other people who don’t use Nix or don’t want to start learning how to.
I’m not trying to hate on Nix here, but it’s just strange to me that you went this route instead of updating the script to work with a more recent version of Python. It seems backwards to have to bend to a 10+ year old version of Python instead of simply updating it.
Author here. I don’t regularly use Python and haven’t setup a
virtualenvin years, so in this case it seemed more arduous to port the script and try to wrangle the dependencies in a Pythonic way.I do, however, use Nix regularly, and it took me just a couple minutes to get the script up and running, dump the output I needed, and move on with my work.
Fair enough, if you felt it would have saved you more time then that is your decision. :)
The argument that the code is written for python2.7 is pretty poor seems just print statement is requiring python2 adding a pair of parenthesis is not so hard
Its not just the print statement that changed between Python 2 and 3. There’s a number of subtle differences that might affect the semantics of a program, and I wouldn’t want to debug those in someone else’s project I was using to quickly check one thing.
Am I the only one that have eyes hurting when reading “Python 2.7”, like in the this nix documentation example that the article is pointing at? https://nixos.wiki/wiki/Packaging/Python#Pip_and_Virtualenv_enabled_nix-shell
I’m pretty sure I have read a few month ago that Python 2.7 was deprecated once and for all after a more than a decade long agony, in favor of Python 3.x flavour.
Not sure it is a great idea to have a scaffolding file targetting at python 2.7.
Sometimes you don’t have the luxury of forcing people to upgrade random things from the past they’ve forgotten about. Having an option to run it at all is better than not.
Totally agreed that “to run it at all is better than not”.
My concern was rather more about the fact that it was an example file in Nix documentation for Python projects with pip and virtualenv, that is: a quasi-standard common python project.
Only this made me think of the mention of “2.7” worth a comment here.