1. 60
  1.  

  2. 9

    I like the idea of Nix, even though I haven’t used it much at this point. However, all this extra tooling and setup concerns me. Niv, lorri, direnv, naersk? Separate configuration files for each? Isn’t this what we we’re supposed to be able to avoid by using Nix?

    1. 8

      However, all this extra tooling and setup concerns me.

      There is a lot of extra tooling. However, the nice thing is that if you pass such a project to others, they could still just use nix-build/nix-shell. No need to use direnv or lorri.

      Niv

      niv has some overlap with the proposed the Nix flakes functionality, which is currently under development:

      https://github.com/tweag/rfcs/blob/flakes/rfcs/0049-flakes.md

      With flakes, Nix allows you to lock dependencies, etc.

      1. 3

        The flake branch is also caching the evaluation outputs which makes lorri mostly redundant.

        1. 2

          Nix flakes hasn’t been released yet. Niv has.

          1. 4

            Sure, this was not a criticism. I also use Niv. I just wanted to point out that while you now need an extra tool such as Niv, Nix will have similar functionality in the future. So, Niv will not be necessary as additional tooling in the future.

        2. 4

          They each do one thing very well and work to integrate with eachother.

          • Niv is like git submodules but maintainable and uses the nix store to your advantage
          • Lorri caches nix-shell generations (because making your shell/editor lag for ~5-10 minutes at worst is unacceptable)
          • Direnv exposes lorri’s data into your shell
          • Naersk parses Cargo manifests and puts every rust dependency in the Nix store

          It looks like a lot from the outside, but it’s not actually that much from the inside. It’s about building on top of things in logical increments. Traditional package managers have the same kind of problem (but do a far worse job of handling it in my opinion) where they need libraries and tooling to handle Rust, Go, Haskell and Perl programs. Nix just lets the user handle those things the way they want. I could have used cargo2nix or something, but naersk is significantly less setup/configuration because it parses the cargo manifest/lock file.

          1. 2

            Naersk parses Cargo manifests and puts every rust dependency in the Nix store

            Well, it realizes all the dependencies in together in the Nix store (in a single store path). This is better than buildRustPackage, but it is not optimal, since compiled dependencies cannot be reused between projects and/or of the dependencies of the projects change.

            buildRustCrate realizes every crate (dependency) into a separate store path, meaning that compiled dependencies can be reused between projects. Also, if you change a dependency of a project, only that dependency is (re)compiled. I’d recommend crate2nix to use buildRustCrate. By using IFD, crate2nix can also be used to parse Cargo.lock without explicit code generation.

            The downside of buildRustCrate is that it reimplements Cargo in Nix (it does not use Cargo at all), this can sometimes result in subtle differences in behavior compared to Cargo and some esoteric features are not implemented. But generally it results in very fast builds.

            1. 2

              Thanks for writing this blog post, it reminded me of all the steps we currently have to go through to setup new projects. IMO this is just the beginning and a lot of trimming down will happen. If I had to guess the future:

              • Niv and Lorri become redundant because of Nix flakes
              • Maybe somebody will rewrite a minimal version of direnv that only handles nix projects.
              • Naersk will be part of nixpkgs, or be more easily composable thanks to flakes. If it’s part of the flake registry it will be easier to discover.
              • A nix init command that auto-detects the project layout and fills with sensible defaults.

              Ideally the default surface of nixified projects would be 2 files so it’s not too intrusive.

          2. 7

            Nix seems… weird. I don’t quite grok its value. I mean, Rust pins everything to the Cargo.lock file, what sort of additional reproducibility guarantees does Nix give here? If it’s about the toolchain, if I need to pin a specific rustc version, can’t I ship just the Rust compiler in a Docker container or something?

            1. 8

              It lets you pin versions of native libraries, and also install any tools you need for tests etc. In a large rust project I work on we use Cargo.lock for rust dependencies, but I also have a nix file with:

              buildInputs = [
                  openssl
                  zlib.static
                  rdkafka
                  pkgconfig
                  cmake
                  
                  shellcheck # for linter
                ];
              

              and another for load testing with:

              buildInputs = [
                    confluent-platform
                    mariadb
                    unix-odbc-mariadb
                    chbenchmark
                    sql-exporter
                    prometheus
                    prometheus-pushgateway
                    grafana
              
                    coreutils # provides chmod
                    postgresql # provides psql
                    which
                    curl
                    procps # provides pkill
                    less
                    lsof
              ]
              

              The load test is nice both because it’s reproducible, and because it’s easy to parameterize it across multiple versions of the upstream software.

              1. 4

                can’t I ship just the Rust compiler in a Docker container or something?

                It (or things like Spack) can also be manageable-as-code reproducible ways to build the complicated things that one then encapsulates in a container (or not, if containers don’t actually add any additional value in a given situation).

                If you have a Docker image that you built with a Docker file that starts off with yum update or pip install (without some sort of lockfile) or …, you’ll never be able rebuild exactly that Docker image and it’ll be all kinds of fun trying to figure what’s actually different in the new one.

                This matters a lot when you have an image and need to make “just one small change to it”.

                This turns out to be really useful for reproducible science (which is, I suppose, redundant).

                1. 3

                  You could, but the nix method works on macs too. Rust and Cargo.lock doesn’t pin glibc or other system-level dependencies. I use this in another project of mine to pin sqlite3 for a Discord bot.

                  1. 1

                    Docker works on Macs too though? I figured the issue is when Rust links to glibc, this is where funny things can happen.

                    1. 11

                      Docker works on macs by running a VM. Nix works as a Darwin binary emitting Darwin binaries.

                      1. 1

                        But the VM is means to an end, right? Abstract away all those userland dependencies. I guess Nix makes them explicit, but isolated.

                        Now I get the value though, thanks. If I really want my program to emit native stuff reproducibly Nix makes sense. That is, if I’m building something that needs to work on OSX natively, instead of just running on it.

                        If I’m shipping containerized server apps to run god knows where Docker will be sufficient. As long as the place where those containers are built can be done reproducibly!

                        1. 2

                          I haven’t gone deep into containers, but nixpkgs has a dockerTools.buildImage function that I’ve used to build some really lean images atop alpine. If you end up with a nixified package, I’ve found it a good way to make a container for it.

                          1. 3

                            The nice thing is that you don’t even need a base image such as Alpine. You can just start from scratch and then the Docker image will contain the transitive closure of the package specified in contents. So, the result is a fully self-sufficient container that only contains what is absolutely needed.

                            By the way, in most cases I would recommend dockerTools.buildLayeredImage, since it builds multiple layers based on Nix store paths, meaning that layers are often shared between images. See grahamc’s blog post on how the layers are constructed.

                            1. 2

                              Sometimes people think they might need a shell.

                              1. 2

                                Never needed really needed that, but that can be worked around. Both buildImage and buildLayeredImage take a list for contents, so add bash and perhaps coreutils. I am not denying that there are use cases where you want to use some base image though.

                                I like these small images without bash, coreutils, etc. I know container != security, but if someone manages to exploit the software it’s nice if they don’t have immediate access to a shell, etc.

                                At work, I provide these lean images to the people who deploy them in Kubernetes or whatever they use and I never had a single complaint.

                  2. 1

                    You can pin and rollback your whole deployed OS, or dev env, if you don’t use this, its less valuable.

                  3. 2

                    Off topic: this is a very nice looking blog format/design. I’ve been shopping around for a look like that. What do you use?

                    1. 2

                      I hand rolled a blog engine: https://github.com/Xe/site

                      1. 2

                        Nice! Thanks for the link.

                    2. 1

                      I’m missing a bit of motivation for the single parts, direnv for example. Is it strictly needed? Not sure, I think not. Maybe sprinkling some (optional) would help

                      1. 1

                        direnv isn’t strictly needed, but this is how I do things. It helps by providing the development tools in your shell directly, but you could just use nix-shell for the development tools too.