IMO, every time this topic is raised someone needs to say:
In order to avoid destroying the great thing that Nix currently is, flakes need to be either (a) ignored or (b) standardised (see @infinisil’s points) AND FULLY DOCUMENTED IN ALL RELEVANT PARTS OF THE OFFICIAL MANUALS.
And the documentation team is already overstretched. So don’t hold your breath.
But (a) is not really possible anymore, flakes are already widely used (I know there is flake-compat). Removing flakes support now is unquestionably going to lead to a fork.
The current flakes situation could be foreseen from the beginning, or latest when the RFC was shot down. But there is clearly a lot of value in it, because it became so widely adopted. Also, the current flake format is going to be around, because nobody wants to break all existing flakes. I fear the only solution is accepting what there is as a v0 and then formalizing a v1 and see if it is possible to provide a backwards compatibility mechanism (akin to the earlier editions proposal).
And the documentation team is already overstretched
I was following what they’re doing and it seems like accepting flakes could clarify a lot of things, but otherwise I think outside support/finances could be gathered to update all documentation.
I appreciate your perspective, but I disagree with your opinions. Specifically:
We don’t need a version resolver. You may want one, but it is clearly not strictly necessary.
flake.nix‘s grammar and semantics are a strict subset of Nix; it’s not inconsistent at all. It is a downgrade in expressive power, and that’s a good thing when we consider POLA and packages-as-capabilities.
system must be explicit for good cross-compilation. Yeah, it’s a little irritating, but the alternative is vaxocentrism, and anybody who survived the 32-to-64-bit transition will agree on the value of “i686” and “amd64” being different strings.
We don’t need a version resolver. You may want one, but it is clearly not strictly necessary.
If FlakeHub is going to bolt on a centralized version resolver, I really want flakes to properly tackle it in a decentralized manner instead. FlakeHub version resolution meshes poorly with input following and machine-scope flake registries. I am generally worried about FlakeHub becoming how the Nix ecosystem deals with flakes; it’s a glaring point of centralization in a feature set designed for decentralization. Flakes currently only have one enforced point of centralization: the global flake registry, managed by the NixOS foundation.
system must be explicit for good cross-compilation. Yeah, it’s a little irritating, but the alternative is vaxocentrism, and anybody who survived the 32-to-64-bit transition will agree on the value of “i686” and “amd64” being different strings.
I agree! System sets also need to be extensible. systems in the centralized flake registry is the only working solution I’ve seen so far, and it’s woefully underused. We need to heavily enforce one convention across the whole flake ecosystem for extensible systems to be usable, or we need to extend Nix with an official flake system registry that’s also extensible.
I’m curious: Don’t you dislike the fact that basically the entire point of flakes is to avoid the Nixpkgs monorepo? The ability to have a monorepo is a key advantage of free software, and the main reason to avoid it is to support proprietary software. I’d rather we keep the monorepo and just educate people better on how to use it, rather than dismantle it with flakes.
I’m not entirely sure that is the point; I have a bunch of github projects that are deployed out of flakes. There is little to no reason for putting them into the nixpkgs monorepo, as they’re basically used by me only. There is no wider interest that would justify putting them into nixpkgs and nixpkgs would only add massive delays to trying to push new updates to my servers.
edit: I would also like to add in a quick edit that nixpkgs contains plenty of non-free software, such as VSCode. Free software does not enable monorepos any more than non-free software does and monorepos can contain both free and non-free software.
Aren’t you just agreeing with me that the point of flakes is to avoid the Nixpkgs monorepo?
Anyway, if your projects don’t have dependencies on each other, then you don’t need flakes. If they do, then maybe the common component should be published as something shareable, which could go into Nixpkgs. Or just put the projects which depend on each other into a single repo.
The project in question is a collection of various web services, they do not make sense in isolation but don’t share much common code (but some do and the flakes in question are pointing the right direction), if they’re even in the same language.
The issue is that again; this is code solely deployed by me that in parts even only makes sense being deployed by me, in other parts simply stuff I deploy to servers for others. There is no reason to pull them into nixpkgs and increase the maintenance burden of everyone there because they have to maintain some random code I’ve barely touched for a few years.
In this matter, I see nix flakes no different to a Dockerfile; it’s a way for a maintainer to bring availability of their code to a wider audience without having to play the maintainer game of thrones. The upstream is in full control of the deployment and determines the defaults and configurations, as it should be (before going for NixOS I used ArchLinux in part for the reason that they have very minimal patches on top of every package).
There is also no reason to put all my projects into a single repo, I do not like the monorepo approach. It’s a repository approach propagated by the likes of Google and Microsoft, it’s hostile to small open source projects due to the increased maintenance burden. Nixpkgs gets away because it’s got plenty of subtree maintainers, I don’t wanna put that on myself just for my own silly bits of code.
And no, flakes are not there to avoid the nixpkgs monorepo, evidenced by the fact that all of them pull it in for common components. That argument would hold up if the majority or a good slice of flakes, didn’t pull in nixpkgs at all purposefully to avoid it. But almost all flakes hit nixpkgs in various ways.
So nixpkgs should be filled with a bunch of “caterns_repos” folder for each developer having a public hobby project? You can surely agree that it doesn’t scale there.
the entire point of flakes is to avoid the Nixpkgs monorepo
It is a point, but I don’t believe “entire point” does the multitude of motivations for flakes justice. I’ll defer to today’s other frontpage Nix post [1.] and provide this list of Flakes features:
- Dependency locking/tracking between Nix expressions (flake.lock)
- Fetching dependencies with a common interface (builtins.fetchTree)
- Harmonized declarative interface
- New CLI interface semantics
- Pure evaluation semantics (not exactly, but currently linked to the Flakes mental model)
I’m not expert enough in Nix/Flakes or the history of Nix to say whether these other are the most salient motivations for flakes, but collectively they constitute a body of work that goes significantly beyond what you describe as avoiding the nixpkgs monorepo.
The ability to have a monorepo is a key advantage of free software, and the main reason to avoid it is to support proprietary software.
Similarly, I believe this statement is too absolute. The main reason to avoid a monorepo depends on a user/project/organization’s motivations. E.g. my main reason to avoid nixpkgs for a silly little di.fm player [2.] that I develop is that I don’t think it belongs in the nixpkgs monorepo. Few Nix users would benefit from it, as it has few users, and never will have a significant number of users. Going through an additional step of review every time I make a release would take away some of the joy I get out of developing the application. This doesn’t mean di-tui has proprietary aspirations. It’s open source and will remain that way.
nixpkgs is massive, and it is absolutely inconceivable to be able to store, say, the whole python library ecosystem, plus the whole of nodejs and whatnot. There is simply a limit on what can sanely fit inside it and it is already bordering on that limit.
I believe the duality of Flakes and a good base monorepo is actually a quite great approach as is: have a rolling release “blob” of mostly working together, common software I can refer to always and let more exotic software be defined and linked from anywhere – you end up with a Flakes file that pins a working nixpkgs.
Disclaimer: I only use nixpkgs on other Distros, I have never installed NixOS on its own, but I did read up a lot over the years and tinkered, and before flakes I could sum it up as “Nice in theory, but completely unusable for the amount of work I am willing to invest.” - with flakes everything seemed easier and made more sense.
No, I don’t care whether a ports tree is confined to a single git repository, especially not at the size of something like nixpkgs. Flakes only act as a federation mechanism; the integration of code still happens in the same fashion either way.
the integration of code still happens in the same fashion either way.
But that’s not true! Federation in this form requires “stable APIs”, a deformity usually induced by the presence of proprietary software which can’t simply be patched and recompiled for new APIs. “Stable APIs” are only needed when implementers can’t just update their users for the new version. See http://catern.com/freetech.html
We might be thinking of different actions. To me, the integration of code is what the Nix daemon is doing when I hand it a build request. That request will involve two git repositories already (nixpkgs and the upstream repo for the target package), so I’m neither bothered by the idea that the second repository will have Nix expressions, nor by the idea that the second repository is fetched prior to the first repository.
At no point here am I advocating for non-free software. I will point out that, by design, a ports tree usually has no problem expressing the integration of free and non-free software; nixpkgs already carries many such expressions, regardless of whether I use them.
Stability of a Flake API is almost entirely irrelevant from this perspective, because the versions of expressions are locked in flake.lock. Lockfiles provide bitemporality, disconnecting the time of integration from the time of locking.
That is a great advantage of flakes though. Because you can make much more than just packages. See home-manager, for example. Installing it as a flake is considerably easier and more reliable than as a channel. It makes developing Nix module sets that wouldn’t really fit in NixOS considerably more practical.
I don’t get the version solver part. Those expressions don’t have versions or version constraints. You refer to some branch or tag and that’s it. An alternative version solver could be built on top, but that would also require flakes themselves to be versioned differently than people do at the moment. (One foo-2.0 doesn’t equal another foo-2.0 when they pull in different refs of the whole system they depend on)
And that’s more of a nixpkgs thing than a flake thing.
Version solvers make a lot of sense in systems like apt or pip, in which you can only have at most one version of a dependency so everyone needs to agree on what that version is.
However, this can have some weird non-local effects. For example, you can have libA depending on libB ^1.2, and in libA’s CI they use libB 1.2.0 to run their tests. Now a downstream project that depends on libA but that also (either directly or via one of its other deps) requires libB ~1.3 would silently use libA with libB 1.3.0, which it was never tested against.
In Nix, everyone can have their own versions of all their deps, and everything is used with the same versions of dependencies when integrated into a larger project as it does as an individual software project.
This makes reasoning nice and local, but it also wastes a lot of disk space and network bandwidth.
I think that’s a decent tradeoff a lot of the time.
Yeah, it also seems to me like a usual case of people recognizing a pattern they’re familiar with and assuming that they can carry over all of their experience and mindset because of the similarity. There are specific reasons why version resolving works well for programming language package managers, but they don’t necessarily apply to nix flakes. In addition to what you said, here are some differences that come to mind:
The public facing interface of a flake is very hard to pinpoint compared to a library in a programming language, especially one with static types. Unless the flake is a Nix-only utility library, pretty much any change would require a major version number bump because of how deeply the consumer might be interacting with your flake and its outputs.
Nix doesn’t have a runtime! It’s very important to think carefully about the backwards compatibility of packages in programming languages, because you’re mostly concerned about what happens at runtime, when you, the programmer don’t have the means to swoop in and fix the issue. Of course it’s important to care about your consumers’ builds not failing unnecessarily, but the real nightmare fuel is what can happen at runtime. In that respect, since Nix only exists at build-time, which is when a developer is at hand to fix the issue, version issues are less critical.
You are much less likely to be forced to upgrade your flake dependencies. With programming language packages, you often need to upgrade a bunch of mutually interacting packages all at once because everything depends intricately on each other. But with Nix flakes, it’s so easy to add a inputs.my-nixpkgs-for-a-very-specific-purpose.url = github:myorg/myrepo/my-ad-hoc-branch and use my-nixpkgs-for-a-very-specific-purpose for that specific purpose without disturbing anything else.
I often see a similar thing happening when people criticize Nix for being Turing-complete or untyped etc. They just carry over reasoning from related but different domains without reevaluating them for the unusual case that Nix is.
Yes and right now it is what channels handle. But with flakes, the whole point is that you only change things when you decide; that is what the lock is about. So now you have to take care of every single version change of your dependencies and how they may not interact well with each other yourself. It kindly becomes unwieldly
I don’t believe that’s a technical problem which can be solved. You always had to do that yourself. The versions constraints in libraries are mostly just indicative anyways. (we’re fixing some random issue from package updates every other week) If you’re depending on two unrelated projects and reference them both in your flake, that means they don’t depend on each other anyway and there would be no constraint anyway, because you can’t realistically do a full compatibility matrix of everything installable.
For example someone writes a Frobnicator which uses /tmp/foo as a lock file and someone else writes a Bloberator which does the same. When they get independent flakes, who would you expect to test all the combinations and where would the constraint of “those two can’t work together” even live? (And how do you express that they can work, but not at the same time)
I’m calling on the Nix team to remove the experimental flag by the end of the year and to thereby open a new chapter in Nix’s history and pave the way for other worthy goals.
I thoroughly agree - I’ve avoided building stuff around flakes specifically because it’s labelled experimental; people keep saying “it’s stable” and then hemming and hawing around why they don’t mark it as stable. If being “experimental” is meaningless then why even have the flag? If it’s meaningful then why should I ignore the flag?
This isn’t Nix-specific, it’s the Kickstarter logic of believe what they tell you through official channels - if they officially say “we give zero guarantees”, but unofficially say “no, we 100% guarantee it, trust us bro”, then listen to the official warnings and treat it like a lottery ticket.
I’m happy to see a unified way to do Nix patterns, even if it’s flawed. Other projects you have so many different cats being skinned to accomplish tasks & it is hard to follow the code. I wish there was less infighting tho as this could be a community-fracturing feature. Instead of getting fixes & proper development, often merge requests & discussions seem to be met with philosophical opposition & grandstanding even if it could improve the very situations (and sometimes legitimate) being stood against.
Flawed or not, flakes are better than not-flakes & it has been helping many projects & teams.
This absolutely does not follow from the above. You (and others) might find them useful; I (and others) might consider them a backwards step. Those are opinions; neither is definitively correct.
I wish there was less infighting tho as this could be a community-fracturing feature.
If you wish there was less infighting, then please don’t inflame discussions. For example, I disagree with the article (e.g. I think flakes should be disentangled from other Nix features before cutting a 1.x version; ideally it would live in a separate tool, so those who prefer “pure” Nix can avoid it); however, I think the article is perfectly reasonable, and wouldn’t mind too much if its approach were actually taken (it wouldn’t affect me much directly, other than the continued bloat and churn; indirectly, I’m happy as long as orthogonal functionality doesn’t depend on flakes). In contrast, you seem to be entirely disregarding those who don’t want flakes; which I don’t find reasonable.
“Pure” Nix has it’s own set of issues of issues managing versions + inter-opting with foreign Nix code in a standardized way + etc. At least flakes are trying to solve some of them. Split it up & call the parts different names if that helps UX features cross the line, but by “flakes are better than not-flakes” I mean I’m in favor of adjusting base Nix to have better usability & flakes seem to be the only option going in that direction.
Content-addressing is where I see the most potential, e.g. for decoupling dependencies from their particular definitions; for a global binary cache (e.g. via IPFS, magnet/torrents, etc.); for more seamless integration with git blobs, RDF/JSONLD URIs, etc.
As for standardising entrypoints/datastructures, that appears to be more suited to linters; rather than making fundamental changes to the tooling, complicating the conceptual model, inventing new syntax (foo:bar/baz#quux.foobar), etc.
I (and others) might consider them a backwards step
I haven’t really seen what Nix without flakes would look like and what the path forward of that would be. All the commands and files that don’t use flakes feel relatively awkward…
I haven’t really seen what Nix without flakes would look like… All the commands and files that don’t use flakes feel relatively awkward…
In my last job, every project was built like this:
$ nix-build
Optionally you can give it a --show-trace flag to help debugging, but I prefer to set that globally in my nix.conf.
That’s literally all you need; everything else can be automated, reproducibly, using Nix expressions. In particular, these are useful:
Default values for function arguments: use these to define all your build’s dependencies; including paths to any source code, compilers, libraries, external repos like Nixpkgs, etc. Such default values will be used by nix-build, but are easily overridable, e.g. if we import this project from another.
fetchGit/fetchTarball: use these to (a) specify (and cryptographically verify) precisely which files you’re depending on, and (b) move as much boilerplate, policy choices, dependency versions, etc. out of individual projects and into a central place. That way, individual projects can get most of what they from a single dependency, like { helpers ? import (fetchGit { ... }) }: ... (using the default value of a function argument, as above)
import-from-derivation: this lets a Nix expression import/read the output of a derivation; since derivations can run arbitrary commands (with network access, if they’re fixed-output), we can write Nix functions that call out to external tools, like dependency solvers. Note that nixpkgs doesn’t allow code that uses import-from-derivation, since it causes builds to happen during evaluation: that makes sense for a large collection like Nixpkgs (don’t want Haskell dependency solvers slowing us down if we only want some Python packages), but it’s fine for standalone projects.
There’s currently work on recursive Nix, which would be a nice complement to import-from-derivation (I used to write recursive Nix derivations, but the sandboxing in Nix 2.x made that less reliable).
Likewise, the following should be avoided:
Environment variables: these are impure, and require external management. Prefer to keep everything in Nix expressions, e.g. using function arguments (and perhaps putting a .override function in the result).
<nixpkgs>: this sort of path is taken from the NIX_PATH env var. Avoid env vars: use a function argument, defaulting to fetchTarball with a specific Nixpkgs revision.
Channels: again, these are impure and require external management. Prefer pure Nix expressions using fetchGit/fetchTarball.
builtins.currentTime: this is a hacky way to invalidate cached derivations. Instead, use builtins.hashFile or builtins.hashString on whatever’s actually invalidating the output. In particular, putting hashes in the name of a fixed-output derivation will cause it to be rebuilt automatically (even if we forget to update the derivation’s output hash).
My problem with flakes is that it goes against this latter advice, by moving more things out of pure Nix expressions, and into commandline arguments, or externally-managed “registries”, or blessed locations like “lock files”, etc. I also don’t like the increasing reliance on centralised, single-points-of-failure; e.g. instead of making it easier to depend on github:foo/bar it’s better to move further to content-addressing (e.g. the ongoing work on the Nix store layer to interoperate better with IPFS, git, dat, hyperdrive, etc.)
Thanks for the lengthy reply but understanding most of this is going to take me several hours of painstakingly tracing and trying out nix commands.
For some reason when I started doing Nix I gravitated directly to flakes and climbing out of that well seems like it would be a lot of effort for an ‘outdated’ way of doing things.
Also my interface with nix is not so much nix build at the moment (which though is nice if it works) as it is nix develop which turns out to be an absurd pain to make work for any non-trivial project that has its own dependencies (python, cargo, node) and then you have to use dream2nix or something which is more or less undocumented and well… I gave up and I’m now back to using system global tools for these three and I’m much happier and much more productive for it.
On iOS safari, this site renders every differently-styled word with a huge amount of horizontal space surrounding it. I’d love to read these posts but this is so distracting that I’m having a hard time doing it in the situations when I’m browsing lobsters.
source explosion is generally mitigated using the follows mechanism built into flakes, which reduces the number of unique Nixpkgs instances involved in an evaluation
Can a flake not declare a nixpkgs input, and make its outputs be curried functions that take nixpkgs arguments explicitly passed by the user of the flake?
I’m not sure if what I’m saying even makes sense but, supposing it does, what would be the tradeoffs compared to the “follows” mechanism?
My assumption is follows makes it easier to create the lockfile as it’s a part of the input’s API. Otherwise the lockfile would need to calculate all the different input nixpkg (etc.) versions, & then override them when evaluated.
IMO, every time this topic is raised someone needs to say:
In order to avoid destroying the great thing that Nix currently is, flakes need to be either (a) ignored or (b) standardised (see @infinisil’s points) AND FULLY DOCUMENTED IN ALL RELEVANT PARTS OF THE OFFICIAL MANUALS.
And the documentation team is already overstretched. So don’t hold your breath.
But (a) is not really possible anymore, flakes are already widely used (I know there is flake-compat). Removing flakes support now is unquestionably going to lead to a fork.
The current flakes situation could be foreseen from the beginning, or latest when the RFC was shot down. But there is clearly a lot of value in it, because it became so widely adopted. Also, the current flake format is going to be around, because nobody wants to break all existing flakes. I fear the only solution is accepting what there is as a v0 and then formalizing a v1 and see if it is possible to provide a backwards compatibility mechanism (akin to the earlier editions proposal).
I was following what they’re doing and it seems like accepting flakes could clarify a lot of things, but otherwise I think outside support/finances could be gathered to update all documentation.
I posted an answer to this on Discourse.
I’d also like to link to this very relevant new blog post, “Nix Flakes is an experiment that did too much at once…”
I appreciate your perspective, but I disagree with your opinions. Specifically:
flake.nix‘s grammar and semantics are a strict subset of Nix; it’s not inconsistent at all. It is a downgrade in expressive power, and that’s a good thing when we consider POLA and packages-as-capabilities.systemmust be explicit for good cross-compilation. Yeah, it’s a little irritating, but the alternative is vaxocentrism, and anybody who survived the 32-to-64-bit transition will agree on the value of “i686” and “amd64” being different strings.If FlakeHub is going to bolt on a centralized version resolver, I really want flakes to properly tackle it in a decentralized manner instead. FlakeHub version resolution meshes poorly with input following and machine-scope flake registries. I am generally worried about FlakeHub becoming how the Nix ecosystem deals with flakes; it’s a glaring point of centralization in a feature set designed for decentralization. Flakes currently only have one enforced point of centralization: the global flake registry, managed by the NixOS foundation.
I agree! System sets also need to be extensible.
systemsin the centralized flake registry is the only working solution I’ve seen so far, and it’s woefully underused. We need to heavily enforce one convention across the whole flake ecosystem for extensible systems to be usable, or we need to extend Nix with an official flake system registry that’s also extensible.I’m curious: Don’t you dislike the fact that basically the entire point of flakes is to avoid the Nixpkgs monorepo? The ability to have a monorepo is a key advantage of free software, and the main reason to avoid it is to support proprietary software. I’d rather we keep the monorepo and just educate people better on how to use it, rather than dismantle it with flakes.
I’m not entirely sure that is the point; I have a bunch of github projects that are deployed out of flakes. There is little to no reason for putting them into the nixpkgs monorepo, as they’re basically used by me only. There is no wider interest that would justify putting them into nixpkgs and nixpkgs would only add massive delays to trying to push new updates to my servers.
edit: I would also like to add in a quick edit that nixpkgs contains plenty of non-free software, such as VSCode. Free software does not enable monorepos any more than non-free software does and monorepos can contain both free and non-free software.
Aren’t you just agreeing with me that the point of flakes is to avoid the Nixpkgs monorepo?
Anyway, if your projects don’t have dependencies on each other, then you don’t need flakes. If they do, then maybe the common component should be published as something shareable, which could go into Nixpkgs. Or just put the projects which depend on each other into a single repo.
The project in question is a collection of various web services, they do not make sense in isolation but don’t share much common code (but some do and the flakes in question are pointing the right direction), if they’re even in the same language.
The issue is that again; this is code solely deployed by me that in parts even only makes sense being deployed by me, in other parts simply stuff I deploy to servers for others. There is no reason to pull them into nixpkgs and increase the maintenance burden of everyone there because they have to maintain some random code I’ve barely touched for a few years.
In this matter, I see nix flakes no different to a Dockerfile; it’s a way for a maintainer to bring availability of their code to a wider audience without having to play the maintainer game of thrones. The upstream is in full control of the deployment and determines the defaults and configurations, as it should be (before going for NixOS I used ArchLinux in part for the reason that they have very minimal patches on top of every package).
There is also no reason to put all my projects into a single repo, I do not like the monorepo approach. It’s a repository approach propagated by the likes of Google and Microsoft, it’s hostile to small open source projects due to the increased maintenance burden. Nixpkgs gets away because it’s got plenty of subtree maintainers, I don’t wanna put that on myself just for my own silly bits of code.
And no, flakes are not there to avoid the nixpkgs monorepo, evidenced by the fact that all of them pull it in for common components. That argument would hold up if the majority or a good slice of flakes, didn’t pull in nixpkgs at all purposefully to avoid it. But almost all flakes hit nixpkgs in various ways.
So nixpkgs should be filled with a bunch of “caterns_repos” folder for each developer having a public hobby project? You can surely agree that it doesn’t scale there.
Nikpkgs does not scale as it is. This is an absurd suggestion.
This is not a reasonable suggestion and does not make sense for really any project I think.
It is a point, but I don’t believe “entire point” does the multitude of motivations for flakes justice. I’ll defer to today’s other frontpage Nix post [1.] and provide this list of Flakes features:
I’m not expert enough in Nix/Flakes or the history of Nix to say whether these other are the most salient motivations for flakes, but collectively they constitute a body of work that goes significantly beyond what you describe as avoiding the nixpkgs monorepo.
Similarly, I believe this statement is too absolute. The main reason to avoid a monorepo depends on a user/project/organization’s motivations. E.g. my main reason to avoid nixpkgs for a silly little di.fm player [2.] that I develop is that I don’t think it belongs in the nixpkgs monorepo. Few Nix users would benefit from it, as it has few users, and never will have a significant number of users. Going through an additional step of review every time I make a release would take away some of the joy I get out of developing the application. This doesn’t mean di-tui has proprietary aspirations. It’s open source and will remain that way.
[1] https://samuel.dionne-riel.com/blog/2023/09/06/flakes-is-an-experiment-that-did-too-much-at-once.html
[2] https://github.com/acaloiaro/di-tui/
nixpkgs is massive, and it is absolutely inconceivable to be able to store, say, the whole python library ecosystem, plus the whole of nodejs and whatnot. There is simply a limit on what can sanely fit inside it and it is already bordering on that limit.
I believe the duality of Flakes and a good base monorepo is actually a quite great approach as is: have a rolling release “blob” of mostly working together, common software I can refer to always and let more exotic software be defined and linked from anywhere – you end up with a Flakes file that pins a working nixpkgs.
Why not? I can and do upstream Nix expressions for Python packages I use. I only use what’s packaged in Nixpkgs. Works fine for me.
There is no scaling limit to Nixpkgs, and I’m wondering what is leading you to believe there is one.
Who approves the PRs? It already takes a lot of time to have to have something merged, because the maintainers are spread so thin.
That doesn’t matter, because you can use your fork while waiting for your PRs to be merged.
As an outsider that’s an interesting statement!
Disclaimer: I only use nixpkgs on other Distros, I have never installed NixOS on its own, but I did read up a lot over the years and tinkered, and before flakes I could sum it up as “Nice in theory, but completely unusable for the amount of work I am willing to invest.” - with flakes everything seemed easier and made more sense.
No, I don’t care whether a ports tree is confined to a single git repository, especially not at the size of something like nixpkgs. Flakes only act as a federation mechanism; the integration of code still happens in the same fashion either way.
But that’s not true! Federation in this form requires “stable APIs”, a deformity usually induced by the presence of proprietary software which can’t simply be patched and recompiled for new APIs. “Stable APIs” are only needed when implementers can’t just update their users for the new version. See http://catern.com/freetech.html
We might be thinking of different actions. To me, the integration of code is what the Nix daemon is doing when I hand it a build request. That request will involve two git repositories already (nixpkgs and the upstream repo for the target package), so I’m neither bothered by the idea that the second repository will have Nix expressions, nor by the idea that the second repository is fetched prior to the first repository.
At no point here am I advocating for non-free software. I will point out that, by design, a ports tree usually has no problem expressing the integration of free and non-free software; nixpkgs already carries many such expressions, regardless of whether I use them.
Stability of a Flake API is almost entirely irrelevant from this perspective, because the versions of expressions are locked in
flake.lock. Lockfiles provide bitemporality, disconnecting the time of integration from the time of locking.That is a great advantage of flakes though. Because you can make much more than just packages. See home-manager, for example. Installing it as a flake is considerably easier and more reliable than as a channel. It makes developing Nix module sets that wouldn’t really fit in NixOS considerably more practical.
Yeah I thoroughly agree with you there. Using a lockfile without version solver and a nice UX to upgrade versions is simply calling for so much pain.
I don’t get the version solver part. Those expressions don’t have versions or version constraints. You refer to some branch or tag and that’s it. An alternative version solver could be built on top, but that would also require flakes themselves to be versioned differently than people do at the moment. (One foo-2.0 doesn’t equal another foo-2.0 when they pull in different refs of the whole system they depend on)
And that’s more of a nixpkgs thing than a flake thing.
Version solvers make a lot of sense in systems like apt or pip, in which you can only have at most one version of a dependency so everyone needs to agree on what that version is.
However, this can have some weird non-local effects. For example, you can have
libAdepending onlibB ^1.2, and in libA’s CI they uselibB1.2.0 to run their tests. Now a downstream project that depends onlibAbut that also (either directly or via one of its other deps) requireslibB ~1.3would silently uselibAwithlibB1.3.0, which it was never tested against.In Nix, everyone can have their own versions of all their deps, and everything is used with the same versions of dependencies when integrated into a larger project as it does as an individual software project. This makes reasoning nice and local, but it also wastes a lot of disk space and network bandwidth.
I think that’s a decent tradeoff a lot of the time.
Yeah, it also seems to me like a usual case of people recognizing a pattern they’re familiar with and assuming that they can carry over all of their experience and mindset because of the similarity. There are specific reasons why version resolving works well for programming language package managers, but they don’t necessarily apply to nix flakes. In addition to what you said, here are some differences that come to mind:
inputs.my-nixpkgs-for-a-very-specific-purpose.url = github:myorg/myrepo/my-ad-hoc-branchand usemy-nixpkgs-for-a-very-specific-purposefor that specific purpose without disturbing anything else.I often see a similar thing happening when people criticize Nix for being Turing-complete or untyped etc. They just carry over reasoning from related but different domains without reevaluating them for the unusual case that Nix is.
Yes and right now it is what channels handle. But with flakes, the whole point is that you only change things when you decide; that is what the lock is about. So now you have to take care of every single version change of your dependencies and how they may not interact well with each other yourself. It kindly becomes unwieldly
I don’t believe that’s a technical problem which can be solved. You always had to do that yourself. The versions constraints in libraries are mostly just indicative anyways. (we’re fixing some random issue from package updates every other week) If you’re depending on two unrelated projects and reference them both in your flake, that means they don’t depend on each other anyway and there would be no constraint anyway, because you can’t realistically do a full compatibility matrix of everything installable.
For example someone writes a Frobnicator which uses /tmp/foo as a lock file and someone else writes a Bloberator which does the same. When they get independent flakes, who would you expect to test all the combinations and where would the constraint of “those two can’t work together” even live? (And how do you express that they can work, but not at the same time)
I thoroughly agree - I’ve avoided building stuff around flakes specifically because it’s labelled experimental; people keep saying “it’s stable” and then hemming and hawing around why they don’t mark it as stable. If being “experimental” is meaningless then why even have the flag? If it’s meaningful then why should I ignore the flag?
This isn’t Nix-specific, it’s the Kickstarter logic of believe what they tell you through official channels - if they officially say “we give zero guarantees”, but unofficially say “no, we 100% guarantee it, trust us bro”, then listen to the official warnings and treat it like a lottery ticket.
I’m happy to see a unified way to do Nix patterns, even if it’s flawed. Other projects you have so many different cats being skinned to accomplish tasks & it is hard to follow the code. I wish there was less infighting tho as this could be a community-fracturing feature. Instead of getting fixes & proper development, often merge requests & discussions seem to be met with philosophical opposition & grandstanding even if it could improve the very situations (and sometimes legitimate) being stood against.
Flawed or not, flakes are better than not-flakes & it has been helping many projects & teams.
That sounds true, from what I’ve heard online.
This absolutely does not follow from the above. You (and others) might find them useful; I (and others) might consider them a backwards step. Those are opinions; neither is definitively correct.
If you wish there was less infighting, then please don’t inflame discussions. For example, I disagree with the article (e.g. I think flakes should be disentangled from other Nix features before cutting a 1.x version; ideally it would live in a separate tool, so those who prefer “pure” Nix can avoid it); however, I think the article is perfectly reasonable, and wouldn’t mind too much if its approach were actually taken (it wouldn’t affect me much directly, other than the continued bloat and churn; indirectly, I’m happy as long as orthogonal functionality doesn’t depend on flakes). In contrast, you seem to be entirely disregarding those who don’t want flakes; which I don’t find reasonable.
“Pure” Nix has it’s own set of issues of issues managing versions + inter-opting with foreign Nix code in a standardized way + etc. At least flakes are trying to solve some of them. Split it up & call the parts different names if that helps UX features cross the line, but by “flakes are better than not-flakes” I mean I’m in favor of adjusting base Nix to have better usability & flakes seem to be the only option going in that direction.
Content-addressing is where I see the most potential, e.g. for decoupling dependencies from their particular definitions; for a global binary cache (e.g. via IPFS, magnet/torrents, etc.); for more seamless integration with git blobs, RDF/JSONLD URIs, etc.
As for standardising entrypoints/datastructures, that appears to be more suited to linters; rather than making fundamental changes to the tooling, complicating the conceptual model, inventing new syntax (
foo:bar/baz#quux.foobar), etc.I haven’t really seen what Nix without flakes would look like and what the path forward of that would be. All the commands and files that don’t use flakes feel relatively awkward…
In my last job, every project was built like this:
Optionally you can give it a
--show-traceflag to help debugging, but I prefer to set that globally in mynix.conf.That’s literally all you need; everything else can be automated, reproducibly, using Nix expressions. In particular, these are useful:
Default values for function arguments: use these to define all your build’s dependencies; including paths to any source code, compilers, libraries, external repos like Nixpkgs, etc. Such default values will be used by
nix-build, but are easily overridable, e.g. if we import this project from another.fetchGit/fetchTarball: use these to (a) specify (and cryptographically verify) precisely which files you’re depending on, and (b) move as much boilerplate, policy choices, dependency versions, etc. out of individual projects and into a central place. That way, individual projects can get most of what they from a single dependency, like{ helpers ? import (fetchGit { ... }) }: ...(using the default value of a function argument, as above)import-from-derivation: this lets a Nix expression import/read the output of a derivation; since derivations can run arbitrary commands (with network access, if they’re fixed-output), we can write Nix functions that call out to external tools, like dependency solvers. Note that nixpkgs doesn’t allow code that uses import-from-derivation, since it causes builds to happen during evaluation: that makes sense for a large collection like Nixpkgs (don’t want Haskell dependency solvers slowing us down if we only want some Python packages), but it’s fine for standalone projects.
There’s currently work on recursive Nix, which would be a nice complement to import-from-derivation (I used to write recursive Nix derivations, but the sandboxing in Nix 2.x made that less reliable).
Likewise, the following should be avoided:
.overridefunction in the result).<nixpkgs>: this sort of path is taken from theNIX_PATHenv var. Avoid env vars: use a function argument, defaulting tofetchTarballwith a specific Nixpkgs revision.fetchGit/fetchTarball.builtins.currentTime: this is a hacky way to invalidate cached derivations. Instead, usebuiltins.hashFileorbuiltins.hashStringon whatever’s actually invalidating the output. In particular, putting hashes in the name of a fixed-output derivation will cause it to be rebuilt automatically (even if we forget to update the derivation’s output hash).My problem with flakes is that it goes against this latter advice, by moving more things out of pure Nix expressions, and into commandline arguments, or externally-managed “registries”, or blessed locations like “lock files”, etc. I also don’t like the increasing reliance on centralised, single-points-of-failure; e.g. instead of making it easier to depend on
github:foo/barit’s better to move further to content-addressing (e.g. the ongoing work on the Nix store layer to interoperate better with IPFS, git, dat, hyperdrive, etc.)Thanks for the lengthy reply but understanding most of this is going to take me several hours of painstakingly tracing and trying out nix commands.
For some reason when I started doing Nix I gravitated directly to flakes and climbing out of that well seems like it would be a lot of effort for an ‘outdated’ way of doing things.
Also my interface with nix is not so much
nix buildat the moment (which though is nice if it works) as it isnix developwhich turns out to be an absurd pain to make work for any non-trivial project that has its own dependencies (python, cargo, node) and then you have to use dream2nix or something which is more or less undocumented and well… I gave up and I’m now back to using system global tools for these three and I’m much happier and much more productive for it.On iOS safari, this site renders every differently-styled word with a huge amount of horizontal space surrounding it. I’d love to read these posts but this is so distracting that I’m having a hard time doing it in the situations when I’m browsing lobsters.
Looks like the culprit is the rule
for whatever that’s worth.
Ohp, thanks! I’ll get this to the team and try and get that solved.
Oh wow, then don’t look at the newer one about schemas (fired up the iPhone because I was curious), this one is very readable :P
Can a flake not declare a nixpkgs input, and make its outputs be curried functions that take nixpkgs arguments explicitly passed by the user of the flake?
I’m not sure if what I’m saying even makes sense but, supposing it does, what would be the tradeoffs compared to the “follows” mechanism?
My assumption is follows makes it easier to create the lockfile as it’s a part of the input’s API. Otherwise the lockfile would need to calculate all the different input nixpkg (etc.) versions, & then override them when evaluated.
It sure could!