Nix Flakes, Part 1: An introduction and tutorial

29

Nix Flakes, Part 1: An introduction and tutorial ☶ programming tweag.io
via neeasade 3 years ago | archive
Archive.org Archive.today Ghostarchive
| 16 comments

16

Note: This is written by Eelco Dolstra, who is the originator of Nix (cf “The Purely Functional Software Deployment Model” https://nixos.org/~eelco/pubs/phd-thesis.pdf)

edit: shortly after I posted this, the flakes RFC entered the final comment period: https://github.com/NixOS/rfcs/pull/49#issuecomment-635387734

6

ac 3 years ago | link

I wasn’t too happy with some of the design decisions that have been going on with Nix to be honest. Some of the design just seems to be getting too complicated for me.

In the end I wrote my own replacement https://github.com/andrewchambers/hermes.
1. 1
  
  akavel edited 3 years ago | link
  
  Do you use patchelf (or a new tool to the same effect) in hermes? I wasn’t able to find an answer after reading the readme & a few linked articles, and this is a feature of Nix I see as super valuable (as a solution to the “dll hell”).
  
  Do you fully control/jail the build env in a deterministic way similar as Nix? Again, I find this a huge feature of Nix and don’t understand the answer from hermes docs.
  
  If the two above are “yes”, I’d be super interested in hermes. I kinda started to think recently there may be some simpler kernel of a project “hiding” in Nix; also Janet sounds like a cool language from some stuff I saw about it before on lobste.rs.
  
  As a side question, also curious about one thing: do you keep most of the infrastructure shell-based as in Nix, or do you strive to move as much as possible to Janet?
  1. 2
    
    ac edited 3 years ago | link
    
    Do you use patchelf (or a new tool to the same effect) in hermes? I wasn’t able to find an answer after reading the readme & a few linked articles, and this is a feature of Nix I see as super valuable (as a solution to the “dll hell”).
    
    Yes, in general you don’t need patchelf unless you are working around a problem with rpath though. The way packages work is more or less the same though.
    
    Do you fully control/jail the build env in a deterministic way similar as Nix? Again, I find this a huge feature of Nix and don’t understand the answer from hermes docs.
    
    Yes.
    
    As a side question, also curious about one thing: do you keep most of the infrastructure shell-based as in Nix, or do you strive to move as much as possible to Janet?
    
    It is not shell based no. Builders are Janet closures run in the sandbox. I have a DSL for running commands in a more pleasant way I wrote about in my blog: https://acha.ninja/blog/dsl_for_shell_scripting/. This DSL was primarily meant for the hermes stdlib as a way to ‘shell out’ in builders when it makes sense.
    1. 2
      
      akavel edited 3 years ago | link
      
      Awsum! So it is serious, gonna try and take a look at the source & examples then, and maybe try If Time Permits™.
      
      Some further questions that came to my mind, if I may:
      
      what’s the story about reproducibility/determinism? esp. vs. the practical issues in Nix that flakes are trying to solve (among others) - i.e. difficulty of pinning nixpkgs, as well as mixing pkgs from various versions of nixpkgs?
      
      what’s the situation around build artifact cache(s)?
      
      do you plan/support p2p binary caches sharing, e.g. via torrents or ipfs or dat? maybe through some 3rd party plugins? how about signing those binaries? can I put e.g. a public key and torrent magnet links (or dat/hyper:// URIs) on my website, then let users add the key to trusted and auto-download the prebuilt binaries?
      
      what’s the situation around pkgs requiring OpenGL? in nixpkgs this is deemed tricky but there’s a nixGL wrapper that worked for me
      
      is there a way I can wrap Steam/GOG.com games, as well as other non-hermes-aware binary-only apps?
      
      is hermes expected to work on BSDs? darwin? open to a GenodeOS port? how about ARM arch. (e.g. Raspberry Pi) - both cross-compiled and native?
      1. 2
        
        ac edited 3 years ago | link
        
        what’s the story about reproducibility/determinism? esp. vs. the practical issues in Nix that flakes are trying to solve (among others) - i.e. difficulty of pinning nixpkgs, as well as mixing pkgs from various versions of nixpkgs? what’s the situation around build artifact cache(s)?
        
        There is no trouble in hermes, as all imports are relative, you can simply clone hpkgs multiple times. Instead of flakes, the current plan is to setup something more similar to GOPATH. Multiple repos depend on eachother via relative import to a name, e.g. (import ../github.com/me/lib). To manage this we can make a tool, or you can just use git submodules manually, or whatever method you prefer for constructing a directory tree.
        
        do you plan/support p2p binary caches sharing, e.g. via torrents or ipfs or dat? maybe through some 3rd party plugins? how about signing those binaries? can I put e.g. a public key and torrent magnet links (or dat/hyper:// URIs) on my website, then let users add the key to trusted and auto-download the prebuilt binaries?
        
        All hermes stores come with a pub/priv keypair already for signing yes. I want to support decentralized artifact (source) downloads as part of the normal mirroring process.
        
        what’s the situation around pkgs requiring OpenGL? in nixpkgs this is deemed tricky but there’s a nixGL wrapper that worked for me is there a way I can wrap Steam/GOG.com games, as well as other non-hermes-aware binary-only apps?
        
        Too early to say.
        
        is hermes expected to work on BSDs? darwin? open to a GenodeOS port? how about ARM arch. (e.g. Raspberry Pi) - both cross-compiled and native?
        
        There is no reason it won’t work, however I would rather not have a lowest common denominator multi OS package tree. I would rather have 1 package tree per OS. Currently there is only linux.
        
        1
        
        akavel 3 years ago | link
        
        what’s the story about (…) difficulty of pinning nixpkgs, as well as mixing pkgs from various versions of nixpkgs?
        
        There is no trouble in hermes, as all imports are relative, you can simply clone hpkgs multiple times. Instead of flakes, the current plan is to setup something more similar to GOPATH. Multiple repos depend on eachother via relative import to a name, e.g. (import ../github.com/me/lib). To manage this we can make a tool, or you can just use git submodules manually, or whatever method you prefer for constructing a directory tree.
        
        Hmmmmm ok, I think I might be starting to get it: so first of all, I now understand hpkgs seems to be not in any way a special repo (contrary to nixpkgs & nix channels), its only differentiation being basically that it was chronologically the first one ever created, and that it has the same maintainer as hermes itself (as of now). If I want, I can create a “mypkgs” repo containing hpkgs-new/ dir with a git submodule of hpkgs at today’s tip, together with e.g. hpkgs-old/ pointing to a tip from 1 month ago. Then I can add my own “frankenstein.hpkg” file in “mypkgs”, importing some stuff from hpkgs-old/ as well as from hpkgs-new/. (And it works “automagically” because hermes is using relative paths instead of absolute paths as used in nixpkgs.) Now, if I want to add some further tweaks over hpkgs-old/ and hpkgs-new/, I probably have to maintain on my own 2 forks of hpkgs with the tweaks, then use those forks as the submodules in “mypkgs”. Is that more or less correct? Also, if #15 lands in some form at some point in the future, I might be able to drop my 2 forks, and instead apply some “overlays” over hpkgs-old/ and hpkgs-new/, I think?
        
        2
        
        ac 3 years ago | link
        
        Yes correct, these are my current thoughts. Its an open point of design though. If we one day got a lockfile, i imagine it could load desired set of repositories and lay them out on disk, and builds happen as before.
        
        1
        
        danieldk 3 years ago | link
        
        If I want, I can create a “mypkgs” repo containing hpkgs-new/ dir with a git submodule of hpkgs at today’s tip, together with e.g. hpkgs-old/ pointing to a tip from 1 month ago. Then I can add my own “frankenstein.hpkg” file in “mypkgs”, importing some stuff from hpkgs-old/ as well as from hpkgs-new/. (And it works “automagically” because hermes is using relative paths instead of absolute paths as used in nixpkgs.)
        
        You can also use relative paths in Nix? Nothing holds you from making a similar setup where you have a nixpkgs submodule. Not sure why you would do that though, something like niv is much more flexible (until flakes are stabilized).
        
        2
        
        ac 3 years ago | link
        
        In practice and my experience was that a lot of packages used <> style imports to reference nixpkgs breaking this workflow completely.
        
        1
        
        danieldk 3 years ago | link
        
        If you want to have a ‘stand-alone derivation’ (as in one that build with just nix-build), the best is if they take an attribute set like:
        
        { pkgs ? import <nixpkgs> {} }: # ...
        
        When someone just uses nix-build it works with their configured channel. But you can also bring your own nixpkgs by ‘importing’ the derivation with
        
        # callPackage comes from the nixpkgs version that you want foobar = callPackage ../pkgs/some-derivation {};
        
        This works because pkgs is a top-level nixpkgs attribute containing nixpkgs itself. It seems that more and more third-party derivations do something along these lines.
4

Corbin 3 years ago | link

I am excited to adopt flakes. However, from a capability-security point of view, I am not comfortable with the flake registry. I think that this sort of registry is committing the same mistakes regarding discoverability and enumerability as older, capability-unaware systems. I hope that the registry will be sufficiently optional as to not interfere with folks who want to publish hard-to-find flakes or otherwise have a desire to be unregistered.
1. 3
  
  pbronez 3 years ago | link
  
  I’m not familiar with the security implications of the Flake Registry, but it does seem to be an under-specified part of the plan. The RFC mentions it a couple times, but only in passing. The RFC assumes a registry exists and does stuff without explaining exactly what it is or what it does.
  
  The Flakes RFC issue has a couple comments about this but no resolution yet.
2. 2
  
  Irene 3 years ago | link
  
  I agree, and additionally I think it’s really important to understand the central role that a registry like this can end up playing in the ecosystem. A lot has been written about npm and how it came under corporate control largely as a result of being designed around a centralized package registry. Nix is a really important piece of infrastructure and I wouldn’t want to see its ecosystem go the same way.
4

pbronez 3 years ago | link

Interesting read. As a Nix novice, I like the separation of flake.nix and flake.lock. This design would let me depend on a moving target like a branch in a remote git repo (in the flake.nix that I write directly), let my tooling pin that to a specific commit on first build (generating flake.lock automatically), and then manually manage updates (with nix flake update changing flake.lock upon explicit request).

I’ve got Nix and Home-Manager running on MacOS at the moment. I’ve been able to install basic stuff from Nixpkgs and am starting to write my own… Derivations? Documentation is challenging. The Nix manual’s hello world example doesn’t actually work (a known issue ).

My latest accomplishment was building the Bombadillo gopher client. This required discovering buildGoPackage, which wasn’t mentioned in the Nix manual. Double-checking now, apparently it’s (1) part of nixpkgs rather than nix where I was looking for it and (2) it’s obsolete and I should use buildGoModule instead. The journey continues!
4
neeasade 3 years ago | link
As noted elsewhere in this thread, this RFC is vague in places and there are some differing opinions around flakes. Here are some linked comments with mixed thoughts:
1. 1
  
  danieldk edited 3 years ago | link
  
  In general, I am pretty enthusiastic about flakes, it seems like it will improve reproducibility and give projects and package sets a standard structure.
  
  My (perhaps unwarranted) worry is that as long as nixpkgs is a monorepo (which sometimes is a good property) we’ll end up pulling in dozens of different large nixpkgs revisions to build a project, environment, or system.
  
  I am also not sure how well this is going to work for languages where mixing libraries/modules (e.g. Python) leads to incompatibilities. This is currently already a problem – if you do not pin nixpkgs versions, your own packages break fairly often due to upstream changes. If you pin nixpkgs then your e.g. Python module may end up being compiled with a newer version of e.g. libstdc++ than the user’s system and fail because of missing symbols when imported. What makes this is worse is that a lot of third-party package sets are tracking master or nixos-unstable, whereas user machines use a stable version.
  
  Of course, the latter point may be more a criticism of a language such as Python than Nix. But it is something that needs to be dealt with.