1. 19
  1.  

  2. 10

    Yesterday I wanted to use my laptop I just recently put NixOS on to resume working on one of my projects that uses pip/requirements.txt to set up dependencies. After realizing that ‘pip install’ doesn’t work in here, I determined that the amount of work required to accomplish this was too great and ended up putting my HDD with Arch back in so I could do some actual work rather than re-solve the already solved (many times over) problem of locally managing python dependencies. The complexity of doing this on NixOS effectively kills using virtualenvs as a quick way to get a special environment up and running.

    Since I’m still very new to NixOS/Nix, deducing the steps in this post from the Nix manual on python was too much for me. This post might encourage me to give it a try though.. but my point of “why is this so ridiculously/needlessly complicated here?” still stands…

    1. 3

      Addressing your point, I like to bring up the difference between complicated and difficult: Nix is difficult, but not very complicated.

      I think it is difficult because we are used to working with package managers that treat the filesystem as a global shared space and that results in a lot of implicit dependencies (and all the problems that come with it). Nix on the other hand makes dependencies explicit which I think is simpler and lets you do things such as having 10 different variations of the Python runtime without resorting to external tools and shell hacks.

      It takes effort plus the official manuals aren’t very welcoming, but it is very rewarding to me.

      1. 1

        external tools and shell hacks

        Thing is, the Python world already has tools like tox which solve this well-known problem, are comparatively well documented, and generally just more stable and better supported for Python than Nix’s “final solution”. For those of us who need (or just prefer) to get some development work done rather than tinker with their system package manager, it’s a significant barrier to adoption.

        1. 1

          I agree with you, and I think the best case scenario is the end user (the persona you described) using Nix through a wrapper so they don’t even know Nix is involved which makes it is more interesting to people working on tooling. I have used Nix to reduce the onboarding of new developers on a project to “install Nix and run nix-shell”. That included multiple programming languages (with their packages), databases, web servers, etc.

          If you think about it (and I will exagerate a bit to make my point), it’s weird that you need to install Python (probably using some package manager), then use pip (another package manager) to install Tox which manages virtualenv instances containing Python installations. Now take that and replicate it for every other programming language/environment out there.

          In the end everything is just files depending on other files.

          1. 2

            In a polyglot project or environment, language-specific dependency management may well be inadequate. Reproducible builds are a good thing, no matter how you get there. But Nix is a pretty big hammer, and it’s probably not a good idea to try to force everyone up that learning curve. If you’re going to make a wrapper around it for a project, there should be someone who can maintain it and respond quickly when that abstraction barrier is breached.

            I was responsible for introducing Nix on a big, ambitious, polyglot project myself. Can’t say for sure that it solved more problems than it created, but I do believe it was a net win. Most of the devs still won’t touch it except to run nix-shell, but that’s OK. But I do have mixed feelings; mostly along the lines of, Nix can make it a little too easy and “safe” to add complexity which will end up biting you anyway, and probably worse than if you’d been forced to deal with it at an earlier stage. YMMV; Nix is good stuff, just no substitute for discretion and other forms of sound engineering judgement. Beware silver bullets.

      2. 2

        The nixpkgs manual has the section “15.17.3.6. How to consume python modules using pip in a virtual environment like I am used to on other Operating Systems?” that explains how to use pip and virtualenv inside Nix. I previously used that approach to work on an existing project with lots of dependencies. It’s not elegant but can be useful in some cases.

        Now I’m using a custom solution (not released yet) to build Nix expressions from Python projects. It’s similar to mach-nix, the one mentioned in the article. Actually, I didn’t know mach-nix existed before reading the article :)

        My recommendation is to start with the pip/virtualenv approach mentioned earlier in order to have a working, good enough development environment. Then you can move to a pure Nix solution in order to guarantee reproducibility.

        1. 2

          I had a similar experience. I was working on a project that required a lot of other new technologies and I thought I would do it on my NixOS install. Only to realize that on top of all of the other technologies I was trying to learn, I would also have to learn about how to use pip in Nix. I spent a while trying the tips on the guide, only to have one required packages not be in the Nix package collection. With this guide I may try to get that project working in Nix again.

          I love the idea of NixOS, not allowing so many small things to work as expected puts a real damper on productivity. And as a beginner to Nix, it is daunting not knowing if I will be able to do a project without any hitches or if I will hit some a multi-hour speed bump that leads me down a rabbit hole with 5 different imperfect solutions to a problem.

        2. 3

          I would use python38.withPackages instead of the IMO needlessly convoluted python38.buildEnv.override approach.

          1. 2

            Agree.

            Nix’s documentation really needs to improve around these language-specific functions. I had a hard time figuring out how to build a Go library because of similar issues.

          2. 2

            You could add packages from above directly to the buildInputs list, but that would install each package in isolate and they would not see each other, and what you want is a Python distribution with the package installed inside it.

            I’m confused about this point. What can go wrong if we include the packages in buildInputs?

            1. 3

              As I mentioned on my other comment, with Nix you make dependencies explicit. When you add a python38 and a python38Packages.invoke side by side on a buildInputs list, you are saying that you want Python and Invoke but you don’t care about how they relate and it’s just a coincidence they are both Python things.

              In this particular case it will just work because 1) Nix puts invoke in $PYTHONPATH and it will be found when you try to import it from Python, and 2) python38.invoke has the default python38 as input so it is “reused”.

              So what can actually go wrong?

              Say you want to customize the Python installation (e.g. a compile flag or a patch). Now you have two variations of Python because Invoke still pulls the default one (and you haven’t told it depends on your new Python). Invoke has it’s own executable script to run tasks, and when you run it you are using default Python, but if you import it into another script you are using your customized Python.