Pretty cool. Though I’d only start feeling safe enough to not read the generated code if property tests were supported (otherwise the LLM might just sneak in a case statement making just the tests pass).
Pretty quickly after seeing what the current generation of LLMs was capable of, my first thought was that this makes the idea of formal specification and verification so much more compelling. This is tangentially related to my side project Sligh where I’ve been playing around with test suite generation a la certifying compilers.
Since the output isn’t controlled at all, compiler correctness becomes a huge issue with LLM-generated code. In this paradigm, even if you had an automated test checking for correctness, what would you do if the test failed? LLMs are opaque.
As I understand this, the underlying build tool needs to be outputting its cache artefacts to intermediates and then reading that on every compilation step.
Would love to see a pragmatic example for a familiar build system/language, if such a thing is possible at this stage? Like Go, Make, or npm builds. If it is not too much effort to integrate, this would be pretty exciting!
I doubt it, at least it’s not a sentiment I encountered (here in Germany, where it’d be the native language interpretation). And from other examples I’d say negative reaction is mostly tied to words that cause some emotional response or offense, and that’s not really true of Nix.
Maybe I’m just not seeing this in my relatively brief skim of the site / docs, but it’s not clear to me what the pricing or licensing model is. Are you planning to charge per seat / per cpu / etc? And is the determinate-nixd closed source? I couldn’t find the sources in your github account, but maybe I’m just missing it.
Right now, determinate-nixd is closed source. I know pretty much all of us want to see it be open source, so it probably will be. We have some (legal) ducks to put in a row before we can. We don’t charge for Determinate Nix or determinate-nixd. We do charge for FlakeHub: https://flakehub.com/pricing – $20/user/month in your organization, plus storage and bandwidth.
Yeah, I mean, determinate-nixd is also super useful for users without FlakeHub (like being able to no-touch install Nix on macOS on EC2, managed GC, automatic certificate management, etc.) and you don’t hae to connect it to FlakeHub. But: we created it because we needed that integration layer to make the getting started process for a user be sensible.
Determinate Nix is useful for anyone looking for a guarantee around flake stability, and, eventually, parallel evaluation and improved monorepo support via lazy-trees before the upstream project decides to merge our patches. We offer support and SLAs on it, too, so people can pay for it, but really the point is FlakeHub.
So, determinate-nixd executes the nix daemon underneath it, we don’t link to it. That means if Nix were GPL, we wouldn’t trigger virality. However, Nix is LGPL which explicitly does allow linking to it without virality. All of our patches to Nix itself, of course, are and will continue to be open source, and sent upstream to the Nix project.
I work in an enterprise and have to deal with dev tooling. As soon as Nix is an easy sell, which this gets us closer to, I can’t wait to pitch it as the be-all-end-all solution. Enterprise-compatible Nix installation on macOS is annoying, so having that solved with this release is awesome.
That said, I can’t recommend it until their daemon is open. Soon, though!
Having enterprise turn-key solutions is unfortunately just kind of mandatory in these huge organizations. Things have a bigger cost the less smooth they go just because of the sheer numbers involved.
Thanks! What do you think is most important of what we’ve done here, and what are you hoping gets better next? We’ve got more posts coming (quite soon…) and are evaluating our next few months of work.
This is beautiful. This section (among many others) makes it clear how much more powerful tools can be if APIs follow best practices:
If an API supports both a GET and a PUT for a resource, there is a client-side edit convenience operation which allows you to edit the resource similar to how you might use a PATCH if the API were available.
…
If the API resource includes a $schema then you will also get documentation on hover, completion suggestions, and linting as you type in your editor.
Editing resources will make use of conditional requests if any relevant headers are found on the GET response. For example, if an ETag header is present in the GET response then an If-Match header will be send on the PUT to prevent performing the write operation if the resource was modified by someone else while you are editing.
How do the zero-downtime deploys work (as mentioned on the homepage)? As another commenter mentioned, the complexity of deploying a single NixOS server isn’t too bad, but stuff I want someone else to handle is all the really hard stuff like: high availability, easy backup/restoration, easy linking to external services that handle stateful workloads like DBaaS/object storage providers, vertical and horizontal scaling, secret management, etc.
Another server is spun up, and only when it’s ready does the traffic get redirected. Another advantage of that is that you don’t have to worry about systemd services potentially failing: if they do, the old version stays current, and you can ssh into the failing machine at your leisure to debug.
Would that not support something stateful like a database? While the new server is still booting, you could INSERT a record into the old server, then after the switch, you won’t see the record when you try to SELECT it.
That’s right (see e.g. https://garnix.io/docs/hosting/persistence). The idea is to separate compute from storage as much as possible (a la k8s) so we can both do zero-downtime deploys and scale the compute horizontally. The storage tends to be redeployed much less frequently so in total there’s less downtime. (You can however keep everything in one machine and mark it as persistent to save on machines.)
But for SQL databases in particular, we do have ideas to support zero-downtime deployments too. Just a bit further into the future.
The premise of this is that deploying a NixOS server is always difficult (even for a simple server), and that’s just not true. I’m not a great programmer at the best of times, and I started using NixOS at a time when I particularly didn’t feel like doing anything difficult, and it was just very very easy.
I think if you already have a provisioned server, and that is the only resource you need, and you won’t have to deprovision anything, and you don’t care about a few seconds downtime, it’s true that it’s easy. But otherwise in my experience it is quite annoying. But maybe you found a better way! How do you provision a new server and get NixOS installed on it the first time around? And manage secrets?
Anecdotally I have found it easier to just never install NixOS in the first place – I run it from the RAM, both on my laptop and on production servers. Once your flake generates a nixosConfiguration which produces the bzImage and initrd, you can either netboot or kexec these. That is, your life gets easier once you get rid of a (Linux) bootloader. Some cloud providers (notably Hetzner and Vultr) have ipxe support. To upload the images to the Internet, I have used both Cloudflare tunnels from my laptop, and IPFS through public gateways. For your own networks that you want non-interactive boot, my company did a flake-compatible PXE server, which we will open-source eventually. For computer you have physical access to, it’s easier to put rEFInd with ipxe to a USB stick or some partition where you also place your flake outputs. Once your nodes are up you can iterate on the configuration from your local machine with nixos-rebuild test --flake .#hostname target-host user@host --use-remote-sudo. You can bootstrap secrets with agenix-rekey, because it allows dummy placeholders. Once your node boots, you can ssh-scan its public key, set it manually in your flake, and then do the switch command above, which will allow the new node to decrypt its secrets.
Does anyone know how the shell makes sure children are killed? Not by having a list of children and a signal handler that kills them I presume, since in that case it wouldn’t get a chance to cleanup if it itself is killed with KILL, and I just tried killing a shell with KILL and subprocesses still were terminated.
It uses unix process groups. There’s a great intro to process groups and shells in chapter 8 section 8.2 of Computer Systems: A Programmer’s Perspective if you have access to a copy. Implementing a shell with job control is a common university assignment so there is likely to be lots of good material covering how it works online.
Could anybody knowledgeable clarify for me if interaction combinators are as efficient as interaction nets? What is the overhead of the emulation, in asymptotic terms?
There is now the sense of their being two sides in this debacle, which sets us up for thinking in terms of a battle from which one will emerge winner, and the other the loser.
If instead we think of it in terms of what the community wants, this can maybe still become productive. We could for instance instead call for dissolving the Nix Foundation board and Team, and calling for a vote (among, say, all people who contributed to Nix/nixpkgs in the past year). This is in contrast to calling for Eelco’s departure explicitly, or claiming this whole debate is because of a hostile takeover attempt by leftists: in particular, we commit to a methodology that brings legitimacy rather than to our preferred outcome.
One fun consequence of being able to run every hash in parallel is that you could route traffic to each commit based on a multi-arm bandit optimization. That would mean highly efficient canary deploys for free.
Depending on your traffic volume it could also make most A/B tests (where the hypothesis is “does not decrease metric”) moot. Fewer conditionals in your code base by virtue of your entire code base being under experiment.
Depending on your traffic volume it could also make most A/B tests (where the hypothesis is “does not decrease metric”) moot. Fewer conditionals in your code base by virtue of your entire code base being under experiment.
Yes - in fact, because the instrumentation is so much more thorough, every change can be seen as an A/B test, with its own bucket of metrics!
As you gain confidence in each commit you can ramp traffic to it. The “canary” size would be related to the regret of your algorithm.
The canary would be efficient because routing decisions could happen immediately rather than in some time-period like most canary setups today. If a commit ever serves a single 500 there’s no reason to serve it anymore! The dev workflow would be amazing because there would be virtually 0 cost to just trying things and testing against prod.
Since this was submitted by the author, a question: platform users probably don’t want to wipe their database on every deploy (and may not want to blow away all Redis/memcached-style caches either), so platform users do end up with persistent(-ish) state shared between old and new code. Can you say something about patterns you’ve found to work well?
(Even if this trick “only” solves problems around connections and in-process state, it’s still neat - again, thanks for posting!)
This question comes up a lot in relation to declarative programming techniques in general, there are variations on this posed as “if Haskell is pure, then how does it do IO” or “how does it handle state”. In general, if I’m declaring something that’s timeless and eternal, how do I deal with a mutable world?
I think the best way to approach this question is to imagine that what you’re declaring is a machine that operates on the real world. Think of a Haskell program or a system declared with Nix as the blueprint of a farm equipment. Your blueprint gets manufactured as a piece of equipment that operates on a farm field (the data, state, API endpoints etc.). The field it will operate on is not in the blueprint directly.
That’s why when you’re configuring a NixOS machine, it’s impossible to implement a Nix function that will write a file to the /etc/ folder or some user’s home folder. What you instead do is to declare a systemd service that will do your bidding at runtime, then you use systemd’s capabilities to declare things like what service runs before or after it and what happens if it fails. You’re describing a machine. That said, Nix is very much capable of letting you abstract over the definition of these services.
I have no experience with garnix, but I can imagine what the response would be here. Essentially Garnix wouldn’t concern itself with arranging your DB migrations so that two versions of your backend can run against it simultaneously. It’s your responsibility to set that up. What Garnix would do is to run those different versions simultaneously and make sure that their front ends only talk to their respective backends.
As a side note, you might be thinking that every piece of program is essentially the blueprint for a machine that operates on the real world. And that’s correct, but the distinction is front and center when you’re doing declarative programming (or maybe it’s safe to specialize this statement and say “purely functional programming”)
The persistence part of garnix is still not deployed, though we’ve designed most of it. We will provide support for migrations with simultaneous versions; the way this hooks up to the hash side of things is that the “name” of a DB is both a unique identifier for the data, and the hash of the schema.
This is all quite hand-wavy, but a full description takes another blog post or two…
The way garnix approaches this is a lot like Kubernetes: you have compute, which shouldn’t have any state, and persistence, which should have next to no compute. That way, you can also easily horizontally scale: just add more compute (whereas if you allowed state with compute, there would be problems here). And zero-downtime deployments also become easier.
But the downside is that yes, caches get purged pretty often. It’s less efficient, but it does “test” that you aren’t “seriously” depending on state, which is nice.
A lens-based shell and terminal. The idea is that a lot of unix commands can naturally be both getters and setters, and pipes can preserve that, so you can often make shell output sensibly editable. For example, cat foo shows the output of foo, but if you edit that output (which the terminal would need to support), that’s equivalent to editing foo itself. If you grep that, you get your matches, but if you edit those matches, it’s like using sed.
Emacs wgrep can do this. You grow for some lines and it puts them in a buffer. If you edit the lines it changes the original files. But I understand that your idea is more general than that.
It’s an interesting idea, but if you have “Unavailable commands:” lying there without any context, that’s not exactly good usability because it’s impossible to discover the state machine behind it without trying it out.
The shell has poor discoverability as it is which means many people coming to the command will have read some documentation about it but still from a pure UX perspective, it’s a bad practice to put people into a state without affordance how to get to a different state.
Since your comment is overall negative in tone, I want to emphasize that moving some commands to “Unavailable commands:” without further explanation still makes for more usable --help output than otherwise. If it were all you had time to implement, it would be better than the common practice of listing available and unavailable commands together indiscriminately.
But yes, the problem you describe of being unsure why a command is unavailable does imply the possibility of further improvements to --help output.
I was thinking the same thing.. I mean, you could dump a state transition diagram/dependency graph in --help output, but I’m not sure if that’s.. helpful enough. ;-) One does not usually see such even in more detailed documentation like man pages.
Similarly, even his guiding examples of fd/rg have pros & cons. Users must learn that honoring .gitignore is the default. In the case of rg this is right at the top of a 1000 line --help output (which also has pros & cons, right at the top, but so long its easy to miss; inverse text or color embellishment might help, such as user-config driven color-choice for markup which is a different kind of “missed opportunity based on dynamic state”).
In the case of fd, since unlike greps, finds mostly operate on file names/FS metadata not file contents, it’s a more dubious &| surprising default. (So, while hex coded hash named .pack files might make little sense to search for, a substring of “HEAD” or finding permission problems inside .git or executable files explicitly gitignored might make a lot of sense.) I say this bit only really as a teaser for how much this kind of context-driven stuff could be ripe for abuse leading to issues you highlight from a use-case variability perspective.
FWIW, I would say, as a document usability feature, that hovernote-only text is tricky. I didn’t even realize there was hidden text to hover over until you just said something. You have a lot of good text there - I particularly liked [3].
What was the first Unix tool to introduce subcommands in a way approaching the modern conventions and aesthetics? CVS comes to mind and later on Subversion and eventually Git may have been influenced by this, perhaps via p4 and bk?
I don’t know of anything before SCCS where just one executable exposed multiple entry points and version control systems definitely seem to have been the “popularizers” of this idea.
That said, it bears noting that early VC systems were written in an era before POSIX shells blocked what Zsh now calls setopt PATH_DIRS. So, you could “just” namespace families of commands via directories. Specifically, there was a “system of programs” like the The MH Mail Handling System where often you put programs all under $HOME/bin/mh/ or /usr/bin/mh/ but then - at your preference - not include that in your $PATH but instead type mh/inc, mh/show, ETC.
To those who might counter “but this is totally different!!!”, there are git subcmds which are also available as git-sub instead of being routed via git sub, such as git-shell or git shell. The only real difference is using '-' as a separator. Well, PATH having directories not simply string path prefixes (e.g. /usr/bin:/usr/bin/git-:..."). So, I think it’s a pretty strong analogy. (It does break down a bit at “shared set up code/logic”, but then so does git-shell/git shell duality.) Beyond actual dispatch, the git man pages are all set up with git- prefixed entry points, at least in my installation.
Prefixing as namespacing (as often seen in C code bases like module_submodule_function) likely dates back to the earliest 1960s identifier systems or (imaginably) even “pre-prog.lang ‘technology’/convention”. Also, I’m not advocating it as better/worse - there are pros&cons. E.g., it can be more error prone but it’s also be nice to have an open/user-extensible namespace via git-my-own-command. E.g., C++ began with only class & later added namespace to open things up.
SCCS is an interesting one, because the version shipped by AT&T originally lacked the sccs wrapper. You can see the source from System III at https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/cmd/sccs - take a look at the .mk file and the code in the src subdirectory.
The sccs wrapper was added by Berkeley, to implement some conventions such as SCCS/s. prefixes on history files, and it could act as a setuid wrapper for accessing shared source code. You can see the source from 4.1c BSD at https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/usr/src/ucb/sccs.c
Ah. I was wondering about exact dates. Nice VC spelunking! So, looks like around early 1983 from the Berkeley guys (who were maybe in the same cohort as the 1986 CVS).
That leaves “a lot more daylight” than the 1972 date from that Wikipedia link of mine for other systems to have used subcommands earlier. I’d bet someone did.. It seems as obvious as “foo-bar” or other prefix naming systems. I just don’t know who. :-) I mean, likely “not Aristotle”, but someone before Eric Allman who in the comment to that program text even calls it “for humans”. { Probably Lobsters could profitably add emoji votes so I could just Rocketship this & ROTFL that. }
Your mh example, and pre-berkeley sccs, are interesting in that they don’t try to namespace their commands. (Perhaps the troff suite, pic, tbl, etc., also counts.) I guess unix was small enough in the 1970s that it did not feel much namespace pressure.
The sccs wrapper is also interesting in that it isn’t just, or even primarily, about namespacing.
It’s also worth comparing RCS and CVS: RCS did not namespace its commands. CVS started as a wrapper around RCS, so (like Berkeley multi-user sccs) CVS has cvs co as a multi-user version of RCS’s bare co.
So my guess is that the namespacing aspect started to matter much later; and I guess other version control systems were the first to copy the pattern because the earliest examples were version control systems.
The other thing to look at is multi-mode commands that don’t have subcommands. For example, tar -c vs tar -x. In the 1990s it was (I think) still more common to use options (long as well as short) than subcommands. Subcommands have the advantage of not having to type -- so much, and they can’t be confusingly interspersed with the other options. So (as well as namespacing) subcommands are good for simplifying the command-line syntax of complicated programs.
Agreed 100% namespace pressure was light early on, but my MH example was meant to show a different but very similar { just ‘/’ instead of space or ‘-’ (dash) or ‘_’ (underscore) } mechanism for namespacing. /usr/bin/mh/ (inc|show|prev|next) shows up as exceptions to some file hierarchy standard quite recently (2010s). It may still install there today.
I don’t know them personally, but I’d bet the guys who wrote MH “intended”/’hoped” their mechanism might catch on, but it did not catch on like dash or space or underbar or even camelCase in a command-name setting. So, instead they come off as having squatted on names like “show”. And then POSIX stomped on the mechanism (at least in std shell), but Zsh keeps it alive. :-)
Interestingly, your tar example overlaps with both subcommand syntax and option syntax - just with 1-letter subcommands that can be “combining” like boolean flags with later options! tar cf and tar xpf both still work fine today without a dash (’-’). So, it’s a tricky semantic thing to even say “tar has no subcommands”. ;-) The top of the Linux man page even kind of lays this out as two syntax families.
All the various frozen-in things make it occasionally hard to talk about this stuff. E.g., GNU Compiler Collection (gcc) doesn’t even use GNU getopt, but its own bespoke CL syntax. To my knowledge, the only thing ever standardized by anyone official is classic 1970s single-letter option getopt(3). In my experience, people often mistake the uniformity from one or two CL syntax toolkits they are “used to” for uniformity In The World. (Not saying this is true of you! It just comes up a lot.. Really none of this is intended to be critical - only clarification/exploratory.)
Pretty cool. Though I’d only start feeling safe enough to not read the generated code if property tests were supported (otherwise the LLM might just sneak in a case statement making just the tests pass).
Pretty quickly after seeing what the current generation of LLMs was capable of, my first thought was that this makes the idea of formal specification and verification so much more compelling. This is tangentially related to my side project Sligh where I’ve been playing around with test suite generation a la certifying compilers.
Since the output isn’t controlled at all, compiler correctness becomes a huge issue with LLM-generated code. In this paradigm, even if you had an automated test checking for correctness, what would you do if the test failed? LLMs are opaque.
As I understand this, the underlying build tool needs to be outputting its cache artefacts to
intermediatesand then reading that on every compilation step.Would love to see a pragmatic example for a familiar build system/language, if such a thing is possible at this stage? Like Go, Make, or npm builds. If it is not too much effort to integrate, this would be pretty exciting!
Not the build tool (if that means e.g. make/cargo/cabal/etc), but the build script. There are two examples, for GHC and Make, in the Nix discourse
link returns 404 & it doesn’t seem to be present on the garnix blog at the time of writing.
Fixed!
I sometimes wonder if nix is doing itself a disservice by choosing this name. (nix = nothing, garnix = nothing at all)
On the other hand Nix is particularly popular in Germany, where these words mean that :)
I doubt it, at least it’s not a sentiment I encountered (here in Germany, where it’d be the native language interpretation). And from other examples I’d say negative reaction is mostly tied to words that cause some emotional response or offense, and that’s not really true of Nix.
Determinate CEO here. I’m relieved to start sharing what we’ve put together as a workflow here, and I’m happy to answer any questions :)!
Maybe I’m just not seeing this in my relatively brief skim of the site / docs, but it’s not clear to me what the pricing or licensing model is. Are you planning to charge per seat / per cpu / etc? And is the determinate-nixd closed source? I couldn’t find the sources in your github account, but maybe I’m just missing it.
Right now, determinate-nixd is closed source. I know pretty much all of us want to see it be open source, so it probably will be. We have some (legal) ducks to put in a row before we can. We don’t charge for Determinate Nix or determinate-nixd. We do charge for FlakeHub: https://flakehub.com/pricing – $20/user/month in your organization, plus storage and bandwidth.
Ahh got it. So Determinite Nix[d] makes nix more attractive to the kind of user who would get value from (and pay for) flakehub.
Yeah, I mean, determinate-nixd is also super useful for users without FlakeHub (like being able to no-touch install Nix on macOS on EC2, managed GC, automatic certificate management, etc.) and you don’t hae to connect it to FlakeHub. But: we created it because we needed that integration layer to make the getting started process for a user be sensible.
Determinate Nix is useful for anyone looking for a guarantee around flake stability, and, eventually, parallel evaluation and improved monorepo support via lazy-trees before the upstream project decides to merge our patches. We offer support and SLAs on it, too, so people can pay for it, but really the point is FlakeHub.
Doesn’t it have to be open source if it’s a modification of the nix daemon, given that that’s LGPL?
So, determinate-nixd executes the nix daemon underneath it, we don’t link to it. That means if Nix were GPL, we wouldn’t trigger virality. However, Nix is LGPL which explicitly does allow linking to it without virality. All of our patches to Nix itself, of course, are and will continue to be open source, and sent upstream to the Nix project.
You’re doing god’s work.
In what way?
I work in an enterprise and have to deal with dev tooling. As soon as Nix is an easy sell, which this gets us closer to, I can’t wait to pitch it as the be-all-end-all solution. Enterprise-compatible Nix installation on macOS is annoying, so having that solved with this release is awesome.
That said, I can’t recommend it until their daemon is open. Soon, though!
Having enterprise turn-key solutions is unfortunately just kind of mandatory in these huge organizations. Things have a bigger cost the less smooth they go just because of the sheer numbers involved.
Thanks! What do you think is most important of what we’ve done here, and what are you hoping gets better next? We’ve got more posts coming (quite soon…) and are evaluating our next few months of work.
This is beautiful. This section (among many others) makes it clear how much more powerful tools can be if APIs follow best practices:
How do the zero-downtime deploys work (as mentioned on the homepage)? As another commenter mentioned, the complexity of deploying a single NixOS server isn’t too bad, but stuff I want someone else to handle is all the really hard stuff like: high availability, easy backup/restoration, easy linking to external services that handle stateful workloads like DBaaS/object storage providers, vertical and horizontal scaling, secret management, etc.
Another server is spun up, and only when it’s ready does the traffic get redirected. Another advantage of that is that you don’t have to worry about systemd services potentially failing: if they do, the old version stays current, and you can ssh into the failing machine at your leisure to debug.
Would that not support something stateful like a database? While the new server is still booting, you could INSERT a record into the old server, then after the switch, you won’t see the record when you try to SELECT it.
Or is there some shared disk magic happening?
EDIT: I’m thinking about this from the perspective of Kubernetes: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy
That’s right (see e.g. https://garnix.io/docs/hosting/persistence). The idea is to separate compute from storage as much as possible (a la k8s) so we can both do zero-downtime deploys and scale the compute horizontally. The storage tends to be redeployed much less frequently so in total there’s less downtime. (You can however keep everything in one machine and mark it as persistent to save on machines.)
But for SQL databases in particular, we do have ideas to support zero-downtime deployments too. Just a bit further into the future.
Ah that page was what I was looking for, thanks!
The premise of this is that deploying a NixOS server is always difficult (even for a simple server), and that’s just not true. I’m not a great programmer at the best of times, and I started using NixOS at a time when I particularly didn’t feel like doing anything difficult, and it was just very very easy.
I think if you already have a provisioned server, and that is the only resource you need, and you won’t have to deprovision anything, and you don’t care about a few seconds downtime, it’s true that it’s easy. But otherwise in my experience it is quite annoying. But maybe you found a better way! How do you provision a new server and get NixOS installed on it the first time around? And manage secrets?
Anecdotally I have found it easier to just never install NixOS in the first place – I run it from the RAM, both on my laptop and on production servers. Once your flake generates a nixosConfiguration which produces the bzImage and initrd, you can either netboot or kexec these. That is, your life gets easier once you get rid of a (Linux) bootloader. Some cloud providers (notably Hetzner and Vultr) have ipxe support. To upload the images to the Internet, I have used both Cloudflare tunnels from my laptop, and IPFS through public gateways. For your own networks that you want non-interactive boot, my company did a flake-compatible PXE server, which we will open-source eventually. For computer you have physical access to, it’s easier to put rEFInd with ipxe to a USB stick or some partition where you also place your flake outputs. Once your nodes are up you can iterate on the configuration from your local machine with
nixos-rebuild test --flake .#hostname target-host user@host --use-remote-sudo. You can bootstrap secrets withagenix-rekey, because it allows dummy placeholders. Once your node boots, you can ssh-scan its public key, set it manually in your flake, and then do the switch command above, which will allow the new node to decrypt its secrets.I agree with your caveats. I’m saying that it’s sometimes easy - in fact, very very easy.
Does anyone know how the shell makes sure children are killed? Not by having a list of children and a signal handler that kills them I presume, since in that case it wouldn’t get a chance to cleanup if it itself is killed with KILL, and I just tried killing a shell with KILL and subprocesses still were terminated.
It uses unix process groups. There’s a great intro to process groups and shells in chapter 8 section 8.2 of Computer Systems: A Programmer’s Perspective if you have access to a copy. Implementing a shell with job control is a common university assignment so there is likely to be lots of good material covering how it works online.
There’s also, in the interaction nets/combinators space:
And Futhark/Accelerate/ILGPU for non-IC based higher-level GPU approaches.
Could anybody knowledgeable clarify for me if interaction combinators are as efficient as interaction nets? What is the overhead of the emulation, in asymptotic terms?
There is now the sense of their being two sides in this debacle, which sets us up for thinking in terms of a battle from which one will emerge winner, and the other the loser.
If instead we think of it in terms of what the community wants, this can maybe still become productive. We could for instance instead call for dissolving the Nix Foundation board and Team, and calling for a vote (among, say, all people who contributed to Nix/nixpkgs in the past year). This is in contrast to calling for Eelco’s departure explicitly, or claiming this whole debate is because of a hostile takeover attempt by leftists: in particular, we commit to a methodology that brings legitimacy rather than to our preferred outcome.
Fun fact: Michael Thompson, the original author of “streaming”, is a brilliant philosopher.
Another prominent academic philosopher who moonlights in Haskell-land is John MacFarlane (of pandoc fame).
One fun consequence of being able to run every hash in parallel is that you could route traffic to each commit based on a multi-arm bandit optimization. That would mean highly efficient canary deploys for free.
Depending on your traffic volume it could also make most A/B tests (where the hypothesis is “does not decrease metric”) moot. Fewer conditionals in your code base by virtue of your entire code base being under experiment.
Could you say more about the canary deploys idea?
Yes - in fact, because the instrumentation is so much more thorough, every change can be seen as an A/B test, with its own bucket of metrics!
As you gain confidence in each commit you can ramp traffic to it. The “canary” size would be related to the regret of your algorithm.
The canary would be efficient because routing decisions could happen immediately rather than in some time-period like most canary setups today. If a commit ever serves a single 500 there’s no reason to serve it anymore! The dev workflow would be amazing because there would be virtually 0 cost to just trying things and testing against prod.
This is interesting, thanks for posting this!
Since this was submitted by the author, a question: platform users probably don’t want to wipe their database on every deploy (and may not want to blow away all Redis/memcached-style caches either), so platform users do end up with persistent(-ish) state shared between old and new code. Can you say something about patterns you’ve found to work well?
(Even if this trick “only” solves problems around connections and in-process state, it’s still neat - again, thanks for posting!)
This question comes up a lot in relation to declarative programming techniques in general, there are variations on this posed as “if Haskell is pure, then how does it do IO” or “how does it handle state”. In general, if I’m declaring something that’s timeless and eternal, how do I deal with a mutable world?
I think the best way to approach this question is to imagine that what you’re declaring is a machine that operates on the real world. Think of a Haskell program or a system declared with Nix as the blueprint of a farm equipment. Your blueprint gets manufactured as a piece of equipment that operates on a farm field (the data, state, API endpoints etc.). The field it will operate on is not in the blueprint directly.
That’s why when you’re configuring a NixOS machine, it’s impossible to implement a Nix function that will write a file to the /etc/ folder or some user’s home folder. What you instead do is to declare a systemd service that will do your bidding at runtime, then you use systemd’s capabilities to declare things like what service runs before or after it and what happens if it fails. You’re describing a machine. That said, Nix is very much capable of letting you abstract over the definition of these services.
I have no experience with garnix, but I can imagine what the response would be here. Essentially Garnix wouldn’t concern itself with arranging your DB migrations so that two versions of your backend can run against it simultaneously. It’s your responsibility to set that up. What Garnix would do is to run those different versions simultaneously and make sure that their front ends only talk to their respective backends.
As a side note, you might be thinking that every piece of program is essentially the blueprint for a machine that operates on the real world. And that’s correct, but the distinction is front and center when you’re doing declarative programming (or maybe it’s safe to specialize this statement and say “purely functional programming”)
The persistence part of garnix is still not deployed, though we’ve designed most of it. We will provide support for migrations with simultaneous versions; the way this hooks up to the hash side of things is that the “name” of a DB is both a unique identifier for the data, and the hash of the schema.
This is all quite hand-wavy, but a full description takes another blog post or two…
The way garnix approaches this is a lot like Kubernetes: you have compute, which shouldn’t have any state, and persistence, which should have next to no compute. That way, you can also easily horizontally scale: just add more compute (whereas if you allowed state with compute, there would be problems here). And zero-downtime deployments also become easier.
But the downside is that yes, caches get purged pretty often. It’s less efficient, but it does “test” that you aren’t “seriously” depending on state, which is nice.
Everything is fine until someone commits a hash collision (https://arstechnica.com/information-technology/2017/02/watershed-sha1-collision-just-broke-the-webkit-repository-others-may-follow/) :D
These are sha256 though!
:D
A lens-based shell and terminal. The idea is that a lot of unix commands can naturally be both getters and setters, and pipes can preserve that, so you can often make shell output sensibly editable. For example, cat foo shows the output of foo, but if you edit that output (which the terminal would need to support), that’s equivalent to editing foo itself. If you grep that, you get your matches, but if you edit those matches, it’s like using sed.
Emacs wgrep can do this. You grow for some lines and it puts them in a buffer. If you edit the lines it changes the original files. But I understand that your idea is more general than that.
Dired is also another specific, and emacs, example.
This is quite nice! I didn’t understand apply-to-installable though.
It’s an interesting idea, but if you have “Unavailable commands:” lying there without any context, that’s not exactly good usability because it’s impossible to discover the state machine behind it without trying it out.
The shell has poor discoverability as it is which means many people coming to the command will have read some documentation about it but still from a pure UX perspective, it’s a bad practice to put people into a state without affordance how to get to a different state.
Who said you can’t have the reason why the command is unavailable printed right next to it?
Well, that’s the least you should be doing here.
Since your comment is overall negative in tone, I want to emphasize that moving some commands to “Unavailable commands:” without further explanation still makes for more usable
--helpoutput than otherwise. If it were all you had time to implement, it would be better than the common practice of listing available and unavailable commands together indiscriminately.But yes, the problem you describe of being unsure why a command is unavailable does imply the possibility of further improvements to
--helpoutput.I was thinking the same thing.. I mean, you could dump a state transition diagram/dependency graph in
--helpoutput, but I’m not sure if that’s.. helpful enough. ;-) One does not usually see such even in more detailed documentation like man pages.Similarly, even his guiding examples of fd/rg have pros & cons. Users must learn that honoring
.gitignoreis the default. In the case ofrgthis is right at the top of a 1000 line--helpoutput (which also has pros & cons, right at the top, but so long its easy to miss; inverse text or color embellishment might help, such as user-config driven color-choice for markup which is a different kind of “missed opportunity based on dynamic state”).In the case of
fd, since unlike greps, finds mostly operate on file names/FS metadata not file contents, it’s a more dubious &| surprising default. (So, while hex coded hash named .pack files might make little sense to search for, a substring of “HEAD” or finding permission problems inside.gitor executable files explicitly gitignored might make a lot of sense.) I say this bit only really as a teaser for how much this kind of context-driven stuff could be ripe for abuse leading to issues you highlight from a use-case variability perspective.I’m looking forward to see the first command line tool sprinkle AI in there to try to learn what it is that you actually want to do. (Please no!)
Yeah, as is I agree information is missing (I mention that on the hovernote 6).
FWIW, I would say, as a document usability feature, that hovernote-only text is tricky. I didn’t even realize there was hidden text to hover over until you just said something. You have a lot of good text there - I particularly liked [3].
Thanks! Yeah, maybe Bringhurst-style margin notes would be better. Or at least also listing hovernotes at the end of the text.
What was the first Unix tool to introduce subcommands in a way approaching the modern conventions and aesthetics? CVS comes to mind and later on Subversion and eventually Git may have been influenced by this, perhaps via p4 and bk?
Anything before CVS?
I don’t know of anything before SCCS where just one executable exposed multiple entry points and version control systems definitely seem to have been the “popularizers” of this idea.
That said, it bears noting that early VC systems were written in an era before POSIX shells blocked what Zsh now calls
setopt PATH_DIRS. So, you could “just” namespace families of commands via directories. Specifically, there was a “system of programs” like the The MH Mail Handling System where often you put programs all under$HOME/bin/mh/or/usr/bin/mh/but then - at your preference - not include that in your $PATH but instead typemh/inc,mh/show, ETC.To those who might counter “but this is totally different!!!”, there are git subcmds which are also available as git-sub instead of being routed via
git sub, such asgit-shellorgit shell. The only real difference is using'-'as a separator. Well, PATH having directories not simply string path prefixes (e.g./usr/bin:/usr/bin/git-:..."). So, I think it’s a pretty strong analogy. (It does break down a bit at “shared set up code/logic”, but then so does git-shell/git shell duality.) Beyond actual dispatch, the git man pages are all set up withgit-prefixed entry points, at least in my installation.Prefixing as namespacing (as often seen in C code bases like
module_submodule_function) likely dates back to the earliest 1960s identifier systems or (imaginably) even “pre-prog.lang ‘technology’/convention”. Also, I’m not advocating it as better/worse - there are pros&cons. E.g., it can be more error prone but it’s also be nice to have an open/user-extensible namespace via git-my-own-command. E.g., C++ began with onlyclass& later addednamespaceto open things up.SCCS is an interesting one, because the version shipped by AT&T originally lacked the
sccswrapper. You can see the source from System III at https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/cmd/sccs - take a look at the.mkfile and the code in thesrcsubdirectory.The
sccswrapper was added by Berkeley, to implement some conventions such asSCCS/s.prefixes on history files, and it could act as a setuid wrapper for accessing shared source code. You can see the source from 4.1c BSD at https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/usr/src/ucb/sccs.cAh. I was wondering about exact dates. Nice VC spelunking! So, looks like around early 1983 from the Berkeley guys (who were maybe in the same cohort as the 1986 CVS).
That leaves “a lot more daylight” than the 1972 date from that Wikipedia link of mine for other systems to have used subcommands earlier. I’d bet someone did.. It seems as obvious as “foo-bar” or other prefix naming systems. I just don’t know who. :-) I mean, likely “not Aristotle”, but someone before Eric Allman who in the comment to that program text even calls it “for humans”. { Probably Lobsters could profitably add emoji votes so I could just Rocketship this & ROTFL that. }
Your mh example, and pre-berkeley sccs, are interesting in that they don’t try to namespace their commands. (Perhaps the troff suite, pic, tbl, etc., also counts.) I guess unix was small enough in the 1970s that it did not feel much namespace pressure.
The sccs wrapper is also interesting in that it isn’t just, or even primarily, about namespacing.
It’s also worth comparing RCS and CVS: RCS did not namespace its commands. CVS started as a wrapper around RCS, so (like Berkeley multi-user
sccs) CVS hascvs coas a multi-user version of RCS’s bareco.So my guess is that the namespacing aspect started to matter much later; and I guess other version control systems were the first to copy the pattern because the earliest examples were version control systems.
The other thing to look at is multi-mode commands that don’t have subcommands. For example,
tar -cvstar -x. In the 1990s it was (I think) still more common to use options (long as well as short) than subcommands. Subcommands have the advantage of not having to type--so much, and they can’t be confusingly interspersed with the other options. So (as well as namespacing) subcommands are good for simplifying the command-line syntax of complicated programs.Agreed 100% namespace pressure was light early on, but my MH example was meant to show a different but very similar { just ‘/’ instead of space or ‘-’ (dash) or ‘_’ (underscore) } mechanism for namespacing.
/usr/bin/mh/(inc|show|prev|next) shows up as exceptions to some file hierarchy standard quite recently (2010s). It may still install there today.I don’t know them personally, but I’d bet the guys who wrote MH “intended”/’hoped” their mechanism might catch on, but it did not catch on like dash or space or underbar or even camelCase in a command-name setting. So, instead they come off as having squatted on names like “show”. And then POSIX stomped on the mechanism (at least in std shell), but Zsh keeps it alive. :-)
Interestingly, your
tarexample overlaps with both subcommand syntax and option syntax - just with 1-letter subcommands that can be “combining” like boolean flags with later options!tar cfandtar xpfboth still work fine today without a dash (’-’). So, it’s a tricky semantic thing to even say “tarhas no subcommands”. ;-) The top of the Linux man page even kind of lays this out as two syntax families.All the various frozen-in things make it occasionally hard to talk about this stuff. E.g., GNU Compiler Collection (gcc) doesn’t even use GNU getopt, but its own bespoke CL syntax. To my knowledge, the only thing ever standardized by anyone official is classic 1970s single-letter option
getopt(3). In my experience, people often mistake the uniformity from one or two CL syntax toolkits they are “used to” for uniformity In The World. (Not saying this is true of you! It just comes up a lot.. Really none of this is intended to be critical - only clarification/exploratory.)This whole subthread is great - thanks to everyone involved!