1. 78
  1.  

  2. 73

    I think it is time for a nix tag, since unix is very broad for this article

    1. 4

      It’s been suggested many times hasn’t it? I do hope it’s considered this time round

      1. 3

        The usual way to make the case for a new tag is to put together a list of existing articles that would fit under it. Feel free! It should be a new meta thread so it doesn’t get missed.

        1. 11

          Here is the last year’s suggestion: https://lobste.rs/s/i5fneg/add_tag_for_nix_nixos (there were more articles since then).

          1. 8
        2. 15

          No more strict separation of evaluation and build phases: Generating Nix data structures from build artefacts (“IFD”) should be supported first-class and not incur significant performance cost.

          I don’t know much about Nix (although I’m about to learn with Oil), but this sounds like the same problem Bazel has, which I mentioned here:

          https://lobste.rs/s/virbxa/papers_i_love_gg

          and linked in this post:

          http://www.oilshell.org/blog/2021/04/build-ci-comments.html#two-problems-with-bazel-and-gg-again

          i.e. if you have to compute all dependencies beforehand, it both makes things slow and clashes with the way that certain packages want to be built. You have a lot of work to do before you can start to parallelize your build.

          Nix also very much has the “rewrite the entire upstream build system” problem that Bazel has.

          It sounds like we still need:

          1. A more dynamic notion of dependencies like gg
          2. A middleground between Docker-like layers and /nix/store composability

          related: https://lobste.rs/s/65xymz/will_nix_overtake_docker#c_tfjfif

          1. 8

            Thanks for linking the gg paper! It’s quite interesting how most concepts in there directly map to concepts that already exist in Nix today, for example:

            gg models computation as a graph of “thunks,” which represent individual steps of computation. Thunks describe the execution of a specific binary on a set of inputs, producing one or more outputs. Thunks are entirely self-contained; any dependencies of the invoked binary must also be explicitly named as dependencies.

            That’s the concept behind Nix derivations.

            Either a primitive value, representing a concrete set of bytes stored into the object store, or

            Fixed-output derivation.

            As the output of a named thunk (referred to using a hash of the serialized thunk object), which must be first executed to produce its output(s).

            That’s a standard derivation.

            You can encode, for instance, “Link(Compile(Preprocess(foo.cc)))” as a sequence of three thunks

            This is where it gets interesting. Nothing in Nix’s model prevents us from doing things at this abstraction level, but it is far more common to write a single derivation that is more like Wrap(EntireBuildSystemOfTheThing). We would like to elevate Nix one level up.

            A gg thunk can return either one or more primitive values, or it can return a new thunk, with its own set of inputs.

            In Nix this is known as import from derivation (IFD). It refers to any time the build graph is extended with things that first need to be computed as the result of derivations. This process is currently a major pain in Nix as you can not “chunk” these parts of the evaluation if you’re doing something like computing an entire repository’s build graph for a CI system.

            Nix also very much has the “rewrite the entire upstream build system” problem that Bazel has.

            I find this statement confusing. Nix rather does the opposite right now - most build systems are wrapped fully in derivations. The one thing that often causes a lot of overhead (e.g. through code generation) is that the majority of build systems either don’t provide Nix with usable hashes of dependencies, or they make it really hard to “inject” “pre-built” artefacts.

            A middleground between Docker-like layers and /nix/store composability

            Can you expand on this?

            1. 2

              Nix also very much has the “rewrite the entire upstream build system” problem that Bazel has.

              I find this statement confusing.

              I believe GP meant “rewrite the upstream package management system”, which is mostly true for Nix; Unless a particular language’s package manager cooperates heavily with Nix and exposes its guts, you can’t reuse it to express your Nix derivations. That’s why we have projects like cabal2nix, node2nix etc. that try to bridge the gap.

              1. 4

                The majority of those tools just pin hashes, because the upstream package managers don’t provide appropriate lockfiles, but it’s still the upstream build system that is being invoked.

                A notable exception is Rust packaging with buildRustCrate, which reimplements a huge chunk of cargo as cargo is impossible to deal with from a perspective of wanting to package individual crates (it doesn’t support injecting these pre-built artefacts in any way).

                On the other end of the spectrum is something like naersk, which uses lockfiles natively.

              2. 1

                Yeah gg and Bazel are both very functional, parallel, and fine-grained. Actually the fine-grained-ness of Bazel is nice at times but it also takes a large amount of time and memory to handle all that file-level metadata.

                It’s also worth looking at Llama, which is an open source experiment inspired by gg: https://blog.nelhage.com/post/building-llvm-in-90s/

                (by the author of the gg post)

                There were a few more blog posts and comments on lobste.rs about llama.


                I think gg’s main contribution is that it’s fast. However it’s also not a production system like Bazel or Nix; it’s research.

                Though it is open source: https://github.com/StanfordSNR/gg

                What stood out to me as being somewhat unreasonable and not-fit-for-production is the idea of “model substitution”, which seems like just a fancy way of saying we have to reimplement a stub of every tool that runs, like GCC.

                https://github.com/StanfordSNR/gg/blob/master/src/models/gcc.cc

                The stub is supposed to find all the dependencies. This seems like a lot of tech debt to me, and limits real world applicability. However gg is fast and that’s interesting! And I do think the notion of dynamic dependencies is important, and it seems like Bazel and Nix suffer there. (FWIW I feel Nix is a pioneering system with many of the right ideas, though it’s not surprising that after >15 years there is opportunity for a lot of improvement. Just because the computing world has grown so much, etc.)


                By rewrite the whole build system I basically mean stuff like PyPI and I have some particular experience with R packages (in Bazel, not Nix).

                As for the middleground, I’m experimenting with that now :) As far as I understand, writing Nix package definitions can be tricky because nothing is ever in a standard place – you always have to do ./configure --prefix. Although I just looked at a few definitions, and I guess that’s hidden? But the actual configure command underneath can never be stock.

                In my experience this is slightly annoying for C code, but when you have say R and Python code which rely on native libraries, it starts to get really tricky and hard to debug. My experience is more with Bazel but I’ve seen some people not liking “100 KLOC” of Nix. I’d prefer package definitions that are simpler with less room for things to go wrong.

                For me a key issue is that Nix was developed in a world before Linux containers and associated mechanisms like OverlayFS, and I think with that new flexibility you might choose something different than the /nix/store/$hash mechanism.

                From a philosophical level, Bazel and Nix both have a very strong model of the world, and they get huge benefits from that. But if you don’t fit in the model, then it tends to fall off a cliff and you struggle to write solid build config / package defs.

                Also I don’t remember the details, but I remember being excited about Nix Flakes, because I thought Nix had some of those guarantees all along but apparently doesn’t.


                Anyway it sounds like a very cool project, and I look forward to future blog posts. BTW it might also be worth checking out the Starlark language from Bazel. It started out as literally Python, but evolved into a language that can be evaluated in parallel quite quickly. This partially mitigates the “two stage” latency of evaluating dependencies and then evaluating the graph. I didn’t see any mention of parallel Nix evaluation but it seems to make sense, especially for a functional language? Starlark is not as functional but the semantics are such that there’s a lot of parallelism.

                1. 2

                  BTW it might also be worth checking out the Starlark language from Bazel. It started out as literally Python, but evolved into a language that can be evaluated in parallel quite quickly.

                  Oh, I’m aware of Starlark - my previous full-time position was in SRE at Google :)

                  I didn’t see any mention of parallel Nix evaluation but it seems to make sense, especially for a functional language

                  Yeah, we want to get to a parallel evaluation model for sure. In fact the lack of that model is a huge problem for current Nix evaluation, as any use of IFD will force a build process and block the rest of the language evaluation completely until it is done. It makes a lot more sense to mark the thunk as being evaluated and continue on a different part of the graph in the meantime.

              3. [Comment removed by author]

                1. 3

                  If you zoom out a lot there’s some conceptual similarities, but in practice they’re pretty much completely different systems with a completely different UX.

              4. 3

                My experience with Nix in the past has been slightly less advanced/dynamic (mainly NixOS and simple packages) but the performance point was a major factor to me. I understand that Flakes are meant to address some of this, but as it stands, Nix evaluations can get really slow. I’d personally love to see something closer to the speed of an apk add.

                I’d be curious if there is a “simpler” version of Nix that could exist which gets speed ups from different constraints. For example, I’ve found please to be faster than most bazel projects, partly due to being written in go and having less of a startup cost, but also because the build setup seems to be simpler.

                I think that the root of the problem might be that Nix is a package build system, not a development build system, and so is build with different assumptions. I wonder if there’s a way to build a good tool that does both package builds (tracks package dependencies, build binary artifacts, has install hooks) and a build tool (tracks file dependencies, has non-build rules such as linting, and caches artifacts for dev not installation). I’m just spitballing but it seems to me like trying to reconcile these two different systems might force a useful set of constraints that results in a fast & simple build system? (though it could just as easily go the other way and become unwieldy and complex).

                1. 10

                  Nix is a package build system, not a development build system

                  Ah, but this is exactly the point :-)

                  There is nothing fundamental about Nix that prevents it from covering both, other than significant performance costs of representing the large build graphs of software itself (rather than a simplified system build graph of wrapped secondary build systems). At TVL we have buildGo and buildLisp as “Bazel-like” build systems written in Nix, and we do use them for our own tools, but evaluation performance suffers significantly and stops us from adding more development-focused features that we would like to see.

                  In fact this was a big driver behind the original motivation that led to us making a Nix fork, and then eventually starting Tvix.

                  1. 6

                    I wonder if there’s a way to build a good tool that does both package builds (tracks package dependencies, build binary artifacts, has install hooks) and a build tool (tracks file dependencies, has non-build rules such as linting, and caches artifacts for dev not installation).

                    I believe there is! It mostly comes from a paper called A Sound and Optimal Incremental Build System with Dynamic Dependencies (PDF), which is not my work (although I’m currently working on an implementation of the ideas).

                    There are three key things needed:

                    1. Dynamic dependencies.
                    2. Flexible “file stamps.”
                    3. Targets can execute arbitrary code, instead of just commands.

                    The first item is needed based on the fact that dependencies can change based on the configuration needed for the build of a package. Say you have a package that needs libcurl, but only if users enable network features.

                    It is also needed to import targets from another build. I’ll use the libcurl example above. If your package’s build target has libcurl as a dependency, then it should be able to import libcurl’s build files and then continue the build making the dependencies of libcurl’s build target dependencies of your package’s build targets.

                    In other words, dynamic dependencies allow a build to properly import the builds of its dependencies.

                    The second item is the secret sauce and is, I believe, the greatest idea from the paper. The paper calls them “file stamps,” and I call them “stampers.” They are basically arbitrary code that returns a Boolean showing whether or not a target needs updating or not.

                    A Make-like target’s stampers would check if the file mtime is less than any of its dependencies. A more sophisticated one might check that any file attributes of a target’s dependencies have changed. Another might hash a file.

                    The third is needed because otherwise, you can’t express some builds, but tying it with dynamic dependencies is also the bridge between building in the large (package managers) and building in the small (“normal” build systems).

                    Why does this tie it all together? Well, first consider trying to implement a network-based caching system. In most build systems, it’s a special thing, but in a build system with the above the things, you just need write a target that:

                    1. Uses a custom stamper that checks the hash of a file, and if it is changed, checks the network for a cached built version of the new version of the file.
                    2. If such a cache version exists, make updating that target mean downloading the cache version; otherwise, make updating the target mean building it as normal.

                    Voila! Caching in the build system with no special code.

                    That, plus being able to import targets from other build files is what ties packages together and what allows the build system to tie package management and software building together.

                    I’ll leave it as an exercise to the reader to figure out how such a design could be used to implement a Nix-like package manager.

                    (By the way, the paper uses special code and a special algorithm for handling circular dependencies. I think this is a bad idea. I think this problem is neatly solved by being able to run arbitrary code. Just put mutually dependent targets into the same target, which means targets need to allow multiple outputs, and loop until they reach a fixed point.)

                    I’m just spitballing but it seems to me like trying to reconcile these two different systems might force a useful set of constraints that results in a fast & simple build system?

                    I think that design is simple, but you can judge for yourself. As to whether it’s fast, I think that comes down to implementation.

                    1. 2

                      If categorised in the terminology of the ‘build systems a la carte’ paper (expanded jfp version from 2020), where would your proposal fit? Though you havn’t mentioned scheduling or rebuilding strategies (page 27).

                      1. 4

                        That is a good question.

                        To be able to do dynamic dependencies, you basically have to have a “Suspending” scheduler strategy, so that’s what mine will have.

                        However, because targets can run arbitrary code, and because stampers can as well, my build system doesn’t actually fit in one category because different stampers could implement different rebuilding strategies. In fact, there could be stampers for all of the rebuilding strategies.

                        So, technically, my build system could fill all four slots under the “Suspending” scheduler strategy in the far right column in table 2 on page 27.

                        In fact, packages will probably be build files that use deep constructive traces, thus making my build system act like Nix for packages, while in-project build files will use any of the other three strategies as appropriate. For example, a massive project run by Google would probably use “Constructive Traces” for caching and farming out to a build farm, medium projects would probably use “Verifying Traces” to ensure the flakiness of mtime didn’t cause unnecessary cleans, and small projects would use “Dirty Bit” because the build would be fast enough that flakiness wouldn’t matter.

                        This will be what makes my build system solve the problem of scaling from the smallest builds to medium builds to the biggest builds. That is, if it actually does solve the scaling problem, which is a BIG “if”. I hope and think it will, but ideas are cheap; execution is everything.

                        Edit: I forgot to add that my build system also will have a feature that allows you to limit its power in a build script along three axes: the power of targets, the power of dependencies, and the power of stampers. Limiting the last one is what causes my build system to fill those four slots under the “Suspending” scheduler strategy, but I forgot about the ability to limit the power of dependencies. Basically, you can turn off dynamic dependencies, which would effectively make my build system use the “Topological” scheduler strategy. Combine that with the ability to fill all four rebuilder strategy slots, and my build system will be able to fill 8 out of the 12.

                        Filling the other four is not necessary because anything you can do with a “Restarting” scheduler you can do with a “Suspending” scheduler. And restarting can be more complicated to implement.

                  2. 3

                    This is so great. The bus factor on the Nix tool is ~1, and this has been causing lots of issues.

                    Is this meant to be flakes-aware?

                    Also, what’s it being implemented in? If there’s any code available, I couldn’t see it - navigating the SourceGraph-based forge on mobile was an exercise in frustration.

                    1. 6

                      Is this meant to be flakes-aware?

                      We’re not planning to support experimental features for now beyond what is required for nixpkgs, but note that experiments like flakes can also be implemented in the Nix language - you don’t need tooling support for them.

                      Also, what’s it being implemented in?

                      The evaluator is in Rust, store/builder are undecided.

                      navigating the SourceGraph-based forge on mobile was an exercise in frustration

                      There’s also a cgit instance at https://code.tvl.fyi - we have a search service that takes a cookie for redirecting to the preferred code browser.

                      1. 1

                        I think it’s a shame that Flakes are still considered experimental.

                        1. 3

                          I disagree, but opinions are split on this :-)

                          1. 4

                            what are the problems you see with flakes?

                      2. 1

                        I think I recall reading on Matrix earlier that whatever current source is available is from the ~failed fork attempt, and that the reimplementation would likely be in rust?

                        (For that matter, I also saw the suggestion that they may be over-estimating how readily ~compatible their effort may be with guix–though I don’t recall for sure if that was on Matrix or orange site.)

                        1. 2

                          how readily ~compatible their effort may be with guix

                          You’re probably referring to this thread. The thing is that we don’t explicitly intend to be compatible with the derivation files, but the fundamental principles of Guix (language evaluation leads to some sort of format for build instructions) has - to our knowledge - not diverged too much to be conceptually compatible.

                          Note that we haven’t really sat down with anyone from Guix to discuss this yet, for now our efforts are focused on Nix (and personally I believe that a functional, lazy language is the right abstraction for this kind of problem).

                      3. 0

                        Unrelated but the name reminded me of the (now delisted) exchange traded fund TVIX