1. 36
  1. 24

    I realize that Docker (or, I suppose, “containers”) provides fewer guarantees and comes with a host of problems, but as a practical matter, it has delivered on (more or less) the same promise, but in a way that is accessible and easy to understand.

    The lab I work in uses Docker for basically everything (even HPC cluster jobs) because scientific software is notoriously finicky and containers make life so, so much simpler. Could we use Nix? Sure, but it takes 5 minutes to teach someone how to use Docker well enough to be productive, while it would take… I don’t even know how long with Nix.

    In fact, I don’t even know how to use Nix well enough to teach someone else, and I actually built a Nix package a few years back! The language was so inscrutable and poorly-documented that I basically just copy-pasted from other packages until it worked, then decided I never wanted to do that again.

    1. 6

      pkgs.dockerTools.buildImage might be a good way to get started with Nix then. In my (admittedly very limited) experience the Nix code ends up being easier to understand than the equivalent Dockerfile. Probably because as long as you keep to simple Nix expressions there are far fewer gotchas. No need to clean up after installing packages, no need for multi-stage builds, no need to worry about putting “volatile” statements later to minimise build times, and much less shell scripting.

      1. 5

        I wonder how feasible it would be to write, say, a python library that lets you take advantage of Nix without having to do the full buy-in of the special language.

        I have similar gripes about Bazel, with “almost-Python” and documentation that really doesn’t want you to do hacky things (despite hacky things being necessary for taking an existing project and bundling it!).

        Docker is a ball of mud generator, but it is also a ball-of-mud capturer. Nix demands you to throw your mud into a centrifuge while you still need the ball to roll around. Whoever can figure out how to give all the powers of both of these will be my savior

        1. 10

          FWIW:

          • @ac is occasionally experimenting with trying to capture the core idea of Nix in a simpler form;
          • the authors of Nix are actually also experimenting with a “simpler” language;
          • not strictly related, but CUE is a configuration language I find very interesting, and I sometimes wonder if it could be used to build some Nix-like system on, and if that could make things more attractive and usable (if still not as easy as docker to learn; but at least giving a static typing system in exchange).
          1. 5

            FWIW I wonder if https://earthly.dev/ is doing this … It’s a container based distributed build system, which in my mind is sort of the middleground between Docker and Nix.

            I mentioned it here in the “curse of NixOS thread” https://lobste.rs/s/psfsfo/curse_nixos#c_cqc27k

            I haven’t used it but I’d be interested in a comparison / evaluation

            We need a “meta-build” system that solves the reproducibility/parallelism/distribution/incrementality problem for everything at once without requiring O(N) rewrites of upstream build systems. I don’t really have any doubt that this will be based on containers.


            I also collected my past comments on this subject pointing to some ideas on composable Unix-y mechanisms to solve this problem:

            https://oilshell.zulipchat.com/#narrow/stream/266575-blog-ideas/topic/Idea.3A.20A.20Middleground.20between.20Docker.20and.20Nix (login with Github)

            • Allow both horizontal layers (like Docker) and vertical slices (like Nix)
            • A hashing HTTP proxy for package managers that don’t have lockfiles and hashes (which is a lot of them).
            • Storage on top of git to separate metadata from data (layers and big blobs)
              • following the “git ops” philosophy but you need something for big files like layers; I have tried git annex
            • A remote process execution abstraction on top of containers, with data dependencies (this is sort of where a distributed shell comes in, i.e. you want to take some shell command, package it up with dependencies, execute it somewhere else, and name the output without necessarily retrieving it)
            • which leads to a coarse grained dependency graph (not fine-grained like Bazel)
            • haven’t figured this part out, but it appears that the static dependency graph and up-front evaluation is a fairly big constraint that makes it harder to write package definitions; a more dynamic notion dependencies might be useful (but also introduces problems I’m sure)

            But now that I list it out, this is a huge multi-year project, even though it is trying to reuse a lot of stuff (git, container runtimes like podman and bubblewrap, HTTP) and combine it in a Unix-style toolkit … The scope is not that surprising since it’s tantamount to developing a distributed OS :-/ (i.e. task execution and storage management on distributed hardware; building any application)

            I’ll also add that this system is “decentralized in the sense of git, BitTorrent, and the web” – i.e. it’s trivial to set up your own disconnected instances. Docker, Nix, and Bazel don’t feel that way; you kind of have buy into a big ecosystem


            But I’d like to hear from anyone who is interested. @akavel I did chat with @ac about this a few years ago … I think my main feedback is that you have to go further to “boil the ocean”.

            That is, it is not enough to come up with a “clean model” … (which Nix and Bazel already provide). You have to provide a bunch of mechanisms that will help people write correct and clean package definitions for a wide array of diverse software (and config language that’s better than a bunch of macros and dynamic parsing on top of shell or Python, which I think Oil is incidentally :-) ). And “kick start” that ecosystem with software that people want to run. (Docker Hub sort of does this; I think there is a lot of room for improvement)

            And you have to avoid the “rewriting upstream” problem which I mentioned in a comment above. For example, that is why I mentioned the HTTP proxy to help making existing package managers more reproducible. A good test case I always bring up is R code, since it has a very deep and functional ecosystem that you don’t want to rewrite

            1. 1

              Nix already (mostly) doesn’t require rewriting the world of everyone’s build systems (in the way that Bazel does require). In fact it requires substantially less rewriting than systems like Bazel or Python wheels or any other non-containerized package manager AFAIK.

              It’s not clear to me that there needs to be any fundamental change in philosophy to make the Nix idea widely usable - it seems the ideas you listed are ultimately things which Nix already does. You might like reading the original Nix thesis from 2006: https://nixos.org/~eelco/pubs/phd-thesis.pdf

              The need, I think, is more just UX and polish.

              1. 1

                It’s true that Nix requires / encourages fewer rewrites, because it’s more coarse grained while Bazel is fine-grained. But in the linked threads multiple users talk about the duplication of Cargo in the Rust ecosystem. So there still is some.

                And for every package you do have to do all the --prefix stuff because of the unconventional /nix/store layout. This leads to the RPATH hacks too. However I tried to find this in the .nix derivations and it seems hidden. So I feel like there is a lot of magic and that’s where people get tripped up, compared to shell where everything is explicit.

                (And I appreciate the huge benefit that /nix/store gives you – containers will help but not eliminate that problem, since most packages aren’t relocatable.)


                Echoing some previous threads, I don’t really have any problem with Nix having its own language per se. I’m surprised that people continue to complain about the syntax – as others pointed out, the language is just lazy “JSON (tuples) with functions”. It looks pretty conventional to me.

                I think the more substantive complaint is lack of documentation about common library functions. And maybe the lazy evaluation rules.

                The main issue I see is the leaky abstraction. At the end of the day Nix is just running a bunch of shell commands, so I think it’s more natural to build a system around shell. The package defs still need a bunch of inline shell to fix up various issues, just like in every other Linux distro.

                FWIW I think you can make a staged execution model in shell like Make / Bazel, etc.:

                http://www.oilshell.org/blog/2021/04/build-ci-comments.html#language-design-staged-execution-models

                Performance is apparently a problem, and I think it’s more natural to debug performance issues in a model like that than a lazy functional language.

                (Yes I read the thesis over 10 years ago and was surprised how similar it was to Bazel!)

              2. 1

                I’d love to read through what you linked and wrote above and to try to process it as it deserves, but unfortunately I’m extremely busy recently. That’s also big part of the reason why practically all my side projects (including the attempts at a Nix wrapper for the Oil build env) are on a definite hiatus now, and I don’t expect this to change much in conceivable future if I’m still employed where I want to be (tradeoffs…). But for sure still generally interested in all of that, so scanning lobster.rs regularly and trying to stay more or less on top of what’s written here, to the limited extent that my brain still has some minimal capacity to consume!

                1. 2

                  Yeah I feel sort of the same way… I made an attempt to write a package manager ~8 years ago, and got an appreciation for how hard a problem it is! It still feels out of reach, since just the shell and the language are a huge project. (Obviously I am envisioning something based around a better shell)

                  So I put those ideas out there in hopes that someone else has 5 or 10 person-years to make it happen :)

              3. 3

                I’d really like that. I played with nix for a bit and I feel like it’s a great experiment where we learned some approaches that could be moved from r&d to a real product. Getting rid of a lot of the flexibility and putting some real boundaries in place would really help.

                I’d love it if we could transform the “here’s a trace through a stack of meta functions, none of which you wrote - deal with it” situation into “this is an expectation of this well supported pattern, you broke it in this place”. Basically make RuntimeException(haha_fail) into DerivationNeedsToBeWithinSystem(package_name). I know there’s dhall, but that doesn’t go far enough.

                Alternatively nix could get LOTS of asserts with “to get past this point, these contracts need to be satisfied”.

                1. 2

                  That’s our approach with anysnake2 (link omitted, I don’t want to shill my pet project here). End-users write a toml file defining python and R ecosystem date and the list of packages, and it creates the necessary flake.nix for a container or a dev shell.

                  Works well, except for days where the ecosystem is broken for the packages you care about. But that’s full reproduciblity for you.

                2. 5

                  Container images are not automatically reproducible; builders must have special support for reproducibility, and Docker does not, but nixpkgs’ dockerTools does. As a practical matter, I have been bitten by the non-reproducibility of Docker containers.

                  1. 2

                    I also work in HPC and most of our core services are Dockerized (even our hard RT service!). There’s been some attempts to use Nix, and one of our guys is a GUIX advocate, but I don’t really see it gaining any traction due to the complexity, lack of good documentation, and general usability woes. It’s hard enough getting physicists to learn basic docker commands…

                    1. 1

                      Nix / Guix are quite orthogonal to Docker. Actually, you can use them to generate a Docker image.

                      In any case, I’m glad HPCs are at least moving to Docker. I’ve used some fairly large European HPCs, and the policy was basically that you should compile whatever you needed! I had some colleagues that would spend a good fortnight just to get things running. A regular company would panic at the amount of employee work, but academia on the other hand…

                      1. 1

                        Sure, you can use Nix/Guix in Docker too. The point is that the usability of those platforms is awful and I wouldn’t encourage any company to use them in production as they are now.

                        1. 1

                          I meant not only inside Docker, but also to generate Docker images.

                          I think Nix is much more usable that what is generally discussed, thanks to the new nix command line, which is not often advertised. Obviously, it’s not an easy language or toolset, just like Haskell is not easy either.

                          But simple usecases are remarkably straightforward once you grasp the basic principles. I find some tasks much easier than Arch or Alpine, which are my goto distributions (and pretty barebones).

                  2. 15
                    • “Nix the idea” - absolutely, thoroughly agree
                    • “Nix the implementation” - I hope not as it’s absolutely terrible!

                    Guix shows promise, but still needs more of a community. I’m hoping for more tools building on the reproducible hermetic environment stuff, but less of the “we hacked together a really terrible config language” approach.

                    1. 10

                      I don’t think Guix will really catch up or overtake Nix as-is for a few reasons:

                      • NixOS is a fairly standard distro past the massive sweeping non-FHS changes; GuixSD has all of these changes plus a GNU-borne insistence on using the niche Shepherd init system and the linux-libre kernel, which is going to be a non-starter for most people.

                      • Nix’s value for things like reproducible tooling and environments over e.g. Docker, is its support for macOS as well as Linux - combined with WSL2, you can have a reproducible environment for basically every OS on the market. Because macOS is a closed-source OS, as a GNU project it will almost certainly never be made available on there.

                      • Nix’s other benefit for tooling and development is that its standard central package “repository” (codebase?) in nixpkgs will package essentially anything that is legal to do so; making use of the expressive nature of Nix package management to allow users to decide the level of unfree software they want to permit; Guix will only package libre software and forces users to go through unofficial channels (nonguix) and official channels prohibit the discussion of the fact that nonguix even exists.

                      1. 2

                        Absolutely. If it didn’t have the terrible language, it would be great for all those reasons. But it’s terrible enough that I’m staying away from it, at least for now, despite how much I want it.

                      2. 8

                        I am always a little torn on this. I agree nix is hard to learn, and am pretty selective about who I recommend it to.

                        But it is also a pretty big lever.

                        So big that (at least for people likely to put the lever to use) I don’t think the decision is comparable to something like choosing a general purpose language to invest in learning.

                        I think it is a little bit more like deciding whether or not to move to a place you know you will love–but you don’t speak the language.

                        If you want to live there, it will be worth learning the language. Even if it is like learning Mandarin or English or Icelandic.

                        1. 4

                          Maybe nonguix if someone actually makes it the the whole system rather than an extra channel. Guix-the-system (not just the tooling) is too idealistic about things and I don’t see it ever catching on in the industry.

                          1. 2

                            I’m curious, in what way is Guix better than Nix?

                            1. 6

                              Two ways:

                              • Starts from an existing language (Guile) and then is effectively a DSL. I’m very pro this approach vs. rolling a new language as it means if you can’t figure out how to do things with the DSL syntax, you can always drop into the parent language to do something, and that’s more likely to have general-purpose language features (e.g. a variety of control flow stuff) than a custom “for this config only” language. See also Chef (which has all of Ruby to play with) v.s. say Ansible YAML loops
                              • Their work on reducing the bootstrap seed is vital for things like this to be reliable.

                              OTOH, Guix basically has the problem of it being a purist GNU project i.e. many of the things you want to have packaged just aren’t because they aren’t free enough, and also the NIH is very strong (e.g. Shepherd).

                              On balance, I’m not overly fond of either of them, but I love the idea of them.

                          2. 2

                            In Nix’s case, however, you’re building the package from source.

                            So Nix is building everything from source? Like Gentoo?

                            1. 8

                              Because you know the exact hash of each derivation’s inputs, you can query a cache for binary packages. This is cache.nixos.org by default, but you can add your own.

                              1. 6

                                So, if the question is about whether ~Nix is building it from source: yes (roughly). If the question is about whether you have to wait for it to build from source: it depends :)

                                1. 7

                                  So, if the question is about whether ~Nix is building it from source:

                                  It is always possible to build from source.

                                  If the question is about whether you have to wait for it to build from source: it depends :)

                                  For almost all cases for most people: no.

                                  1. 4

                                    If the question is about whether you have to wait for it to build from source: it depends :)

                                    For almost all cases for most people: no.

                                    To expand on this, setting up your own cache using Cachix is actually really easy. Two steps in GitHub Actions (cachix/install-nix-action and cachix/cachix-action) and about four lines in GitLab CI. (I don’t think I’m allowed to post links yet.)

                                    1. 1

                                      Cachix, while pretty convenient, is quite pricey for simple use cases, no?

                                      1. 2

                                        From their pricing page:

                                        Users have a free 5 GB limit for open source projects.

                                        I’m just using that, and it’s plenty for a few small projects. Only the derivations which are not available from nixpkgs are uploaded, after all, so you’d need to be doing something fairly complex (or building a bunch of packages differently from nixpkgs) to run into the limit.