Threads for TyberiusPrime
-
Nice device.
I had an idea of a keybord that you can hold like a ps controller with most keys on the back of the device. I imagine it would have quite a learning curve but maybe it’d be good for avoiding RSI? Or maybe not.
Do you know whether anyone had a similar idea and actually built a prototype?
-
The twiddler was held kind of like that. But it’s a one hand chorded keyboard, and feels about as ergonomic as its shape suggest.
-
-
Really good ciriticsms, I agree with them all. One mitigating aspect of documentation is that Nix configuration is code so you can search on GitHub and hopefully find an example of whatever you want to achieve.
-
I disagree a bit with the last one. Nix has some escape hatches, like FHS user environments.
The major pain point is documentation. The post is totally right about that. Nix is trying to address it with a documentation team funded by some sponsors:
https://discourse.nixos.org/t/documentation-team-flattening-the-learning-curve
I think Nix should try to document the basics much better. IMHO, the basic use cases are simple to learn. Arguably, it’s one of the easiest distros to use if you have very basic needs. And it’s very forgiving. However. it’s getting a reputation for exactly the opposite.
In this regard, there is https://snowflakeos.org, built on top of Nix, which is quite interesting.
-
afaik the funding by Tweag has ended and there is no replacement. A team still seems to be meeting though, recorded here https://discourse.nixos.org/c/dev/documentation/25
Flakes being experimental is a huge problem for the entire nix/nixpkgs ecosystem, including documentation. Official docs will likely avoid flakes as much as possible, but they are a huge improvement in many areas and reduce some of the major pain points users run in to (eg channels). Flakes have problems, but solving the most important ones and stabilizing the schema should be the top priority of nix.
-
afaik the funding by Tweag has ended and there is no replacement.
According to [0], funding is now provided by https://determinate.systems
https://github.com/NixOS/nix.dev/tree/master/maintainers#funding
-
-
-
-
The author’s “must search GitHub repo” argument reads like https://search.nixos.org isn’t as know as it should be.
-
-
I use https://search.nixos.org/ frequently - but this doesn’t help if I don’t already have some idea of what an option does, or if the option is doing something I don’t expect based on its name or documentation (which isn’t always clear anyway), or if I’m trying to use the
nixcommand line tools to do something, or trying to import or create a custom module because the standard nixos-provided modules don’t solve my problem.
-
-
-
AVMs FRITZ!Boxes, which are quite popular in Germany, have been doing the same thing for decades. They use fritz.box as their domain, which was probably pretty safe to use when tlds where limited to countries.
-
Fritz!Boxes use their DNS server, they do not man in the middle port 53. Or at least mine does.
-
-
-
A related thing that I’d like to see added in browsers in a meta tag that force-disables any javascript on the page. This is useful when you’ve got a perfectly fine and functional page that doesn’t use javascript, but someone else decides to inject javascript into it - be it Facebook, American ISPs or whatever. With the meta value set, any tracking junk would be unable to do its thing.
-
Content Security Policies are flexible enough to do this. They can be specified either through a HTTP header or as a meta tag in the page.
-
-
Yes, a sufficiently malicious embed can retrieve the source HTML as a string via network call, strip out any anti-embed headers or tags, and then pass that source HTML on to be rendered.
-
-
-
-
Until they don’t just inject, but also filter your meta tag, I suppose.
-
-
Good grief. VOIP / video conferencing leaking your secret keys !
-
Provided you use them 91 times to sign the same message, if I am reading this correctly.
-
If I understand the paper correctly, they are suggesting that ECDSA is often implemented so that if you supply the same key, the code will calculate the same nonce & you can track that calculation to leak the nonce & therefore the secret key used by the other party. So simply connecting with the same ECDSA parameters is sufficient?
Urgh.
-
-
-
No, in all seriousness, the documentation sucks on this… So I’ll explain it how it was explained to me by a colleague:
Vim stores edit history and undo as a tree. For example, if you do
A,B,C,D, undo, undo,E,F, undo, undo, undo,G, vim will store this history:D F | | G C E | | | | +----+---+ | | | B | | | +----+-----+ | Ag+andg-will navigate between the “tips” of this tree, so betweenD,FandG. One goes back, the other forward. I never know which one is which betweeng+andg-to go the way I want, so I always try both until I realize it’s going the right way :) .-
-
Especially when you throw in a visualizer for it like mundo. Emacs has it as well.
-
-
-
-
-
JetBrains IDEs have a local history feature for this. It’s fantastic, as it shows you a merge view between your current state and the selected history state. So you can still edit the current state and cherry-pick bits from the diff trivially.
-
-
-
Nix works pretty well for the most part, the real friction of it is when you need to do something that doesn’t have its own package/flake or derivation. Though things like Home Manager do definitely help with this as you can easily manage your own derivatives and flakes with it, though most new users won’t actually know Home Manager exists and that’s a sort of community contention point.
-
I’ve found it to be pretty easy if it’s similar enough to something that you can copy-paste the derivation. As a bonus, it’s way more natural to do than it is to package a .deb or whatever. The downside is that if you need to do anything weird, you’re in for a bad time.
-
-
I’ve considered trying Guix because I think Nix is really held back by its language and design (why are packages functions????) but I don’t want to go from a niche distro to an even more niche one. :/
(I also worry about the fact that it’s a GNU project, but that’s a separate issue.)
-
-
I know there’s a “nonguix” package list somewhere… but you’re not allowed to talk about it on official Guix channels, even if someone asks, because it’s a GNU project.
TBH I don’t really care about the choice of language that much, I just really think Nix is a bad language and the heavy use of functions and lazy evaluation was a really bad idea.
-
There are other places to ask things both something like this or just tele/irc/discord/matrix channels that are not the official one.
Here’s how you reproduce your system, first:
guix describe --format=channels > channels.scmthen:guix pull -C channels.scmand do guix system/home reconfigure..The channel system is simpler than what nix does, it’s just a thin wrapper around git afaik. Guix cares a lot about bootstrapping so keeping track of the version of guix that is compatible with your package is possible. In nix when they update the code then old derivations may no longer build… so much for reproducibility.
Guix may actually cure some of the GNU stigma (finally!) … it feels like it is on the doorstep of the future.
-
TBH I still find “we don’t talk about nonguix” annoying. I have similar frustrations with emacs and the fact that you have to explicitly add melpa. Though to be fair, it’s worse with emacs because almost all the really good emacs packages are on MELPA, whereas you could probably get by fine without nonguix as long as your hardware supports it.
Does
guix describeinclude information about the list of installed packages or whatever? It looks to me like it’s just the state of all your channels, which is good, but it’s not what I would call a reproduction.Overall though Guix looks very interesting, and I think more work in the space of declarative package managers/operating systems is always welcome! I look forward to seeing what comes out of it :)
-
Look, I agree that it is kind of annoying to be treated like a child and placing a flat ban on relevant technical discussion but there’s not really an issue.. Guix is free software and there is plenty of alternative (communication) channels where everyone hangs out (so they are just as “official” in many ways). You can easily make your own (guix) channel and package whatever you want (or work with others and establish communal channels like guixrus or nonguix).
Does guix describe include information about the list of installed packages or whatever? It looks to me like it’s just the state of all your channels, which is good, but it’s not what I would call a reproduction.
No, that is what your config is for. Guix describe is there to tell you the state of guix itself, which is (clearly) also an element in reproducing the desired software.
-
Aha. Yeah, it’s frustrating to me that Nix doesn’t reify the channel state nicely by default. I think this is one of the things that flakes solve: when I look at /etc/nixos/flake.lock I get a list of the git hashes of each flake I have and such. I think that the version of nix I use is determined by what the nixpkgs flake gives me, but I’m not sure.
-
-
-
-
-
I’ve only ever used a lisp for emacs config, and it’s actually been alright. You can write some pretty compact DSLs using quoting.
-
-
-
-
-
The second you need to do anything out of the normal it becomes a hellscape.
But that’s pretty much also true if you just want to build anything ‘out of the normal’ just once. On the plus side, nixpkgs has the largest set of packages out there.
-
Well I mean more of trying to make a derivation for anything out of the norm it’s a bit of a hellscape. Say for example I want to create a derivation for some programming language I’m making so I can update it via nix, it’s a bit of a hellscape compared to something like arch’s BUILDPKG where it’s practically just a glorified shell script.
-
From my point of view, a nix derivation is a glorified shell script and some meta data.
-
-
-
-
-
-
What is Home Manager and how does it help? The docs seem pretty focused on how to use it and not what it is helping with: https://nix-community.github.io/home-manager/
-
Think of it like a nix system configuration but for your specific user/home directory. It allows you to have a nix config for stuff like per-user SSH keys, derivations, packages per-user, etc. Pretty handy when you have a multi-user system and want each user to be able to have their dotfile configuration in a similar format to
/etc/nixos/configuration.nix. -
-
-
-
I realize that Docker (or, I suppose, “containers”) provides fewer guarantees and comes with a host of problems, but as a practical matter, it has delivered on (more or less) the same promise, but in a way that is accessible and easy to understand.
The lab I work in uses Docker for basically everything (even HPC cluster jobs) because scientific software is notoriously finicky and containers make life so, so much simpler. Could we use Nix? Sure, but it takes 5 minutes to teach someone how to use Docker well enough to be productive, while it would take… I don’t even know how long with Nix.
In fact, I don’t even know how to use Nix well enough to teach someone else, and I actually built a Nix package a few years back! The language was so inscrutable and poorly-documented that I basically just copy-pasted from other packages until it worked, then decided I never wanted to do that again.
-
pkgs.dockerTools.buildImagemight be a good way to get started with Nix then. In my (admittedly very limited) experience the Nix code ends up being easier to understand than the equivalent Dockerfile. Probably because as long as you keep to simple Nix expressions there are far fewer gotchas. No need to clean up after installing packages, no need for multi-stage builds, no need to worry about putting “volatile” statements later to minimise build times, and much less shell scripting. -
I wonder how feasible it would be to write, say, a python library that lets you take advantage of Nix without having to do the full buy-in of the special language.
I have similar gripes about Bazel, with “almost-Python” and documentation that really doesn’t want you to do hacky things (despite hacky things being necessary for taking an existing project and bundling it!).
Docker is a ball of mud generator, but it is also a ball-of-mud capturer. Nix demands you to throw your mud into a centrifuge while you still need the ball to roll around. Whoever can figure out how to give all the powers of both of these will be my savior
-
FWIW:
- @ac is occasionally experimenting with trying to capture the core idea of Nix in a simpler form;
- the authors of Nix are actually also experimenting with a “simpler” language;
- not strictly related, but CUE is a configuration language I find very interesting, and I sometimes wonder if it could be used to build some Nix-like system on, and if that could make things more attractive and usable (if still not as easy as docker to learn; but at least giving a static typing system in exchange).
-
FWIW I wonder if https://earthly.dev/ is doing this … It’s a container based distributed build system, which in my mind is sort of the middleground between Docker and Nix.
I mentioned it here in the “curse of NixOS thread” https://lobste.rs/s/psfsfo/curse_nixos#c_cqc27k
I haven’t used it but I’d be interested in a comparison / evaluation
We need a “meta-build” system that solves the reproducibility/parallelism/distribution/incrementality problem for everything at once without requiring O(N) rewrites of upstream build systems. I don’t really have any doubt that this will be based on containers.
I also collected my past comments on this subject pointing to some ideas on composable Unix-y mechanisms to solve this problem:
https://oilshell.zulipchat.com/#narrow/stream/266575-blog-ideas/topic/Idea.3A.20A.20Middleground.20between.20Docker.20and.20Nix (login with Github)
- Allow both horizontal layers (like Docker) and vertical slices (like Nix)
- A hashing HTTP proxy for package managers that don’t have lockfiles and hashes (which is a lot of them).
- Storage on top of git to separate metadata from data (layers and big blobs)
- following the “git ops” philosophy but you need something for big files like layers; I have tried git annex
- A remote process execution abstraction on top of containers, with data dependencies (this is sort of where a distributed shell comes in, i.e. you want to take some shell command, package it up with dependencies, execute it somewhere else, and name the output without necessarily retrieving it)
- which leads to a coarse grained dependency graph (not fine-grained like Bazel)
- haven’t figured this part out, but it appears that the static dependency graph and up-front evaluation is a fairly big constraint that makes it harder to write package definitions; a more dynamic notion dependencies might be useful (but also introduces problems I’m sure)
But now that I list it out, this is a huge multi-year project, even though it is trying to reuse a lot of stuff (git, container runtimes like podman and bubblewrap, HTTP) and combine it in a Unix-style toolkit … The scope is not that surprising since it’s tantamount to developing a distributed OS :-/ (i.e. task execution and storage management on distributed hardware; building any application)
I’ll also add that this system is “decentralized in the sense of git, BitTorrent, and the web” – i.e. it’s trivial to set up your own disconnected instances. Docker, Nix, and Bazel don’t feel that way; you kind of have buy into a big ecosystem
But I’d like to hear from anyone who is interested. @akavel I did chat with @ac about this a few years ago … I think my main feedback is that you have to go further to “boil the ocean”.
That is, it is not enough to come up with a “clean model” … (which Nix and Bazel already provide). You have to provide a bunch of mechanisms that will help people write correct and clean package definitions for a wide array of diverse software (and config language that’s better than a bunch of macros and dynamic parsing on top of shell or Python, which I think Oil is incidentally :-) ). And “kick start” that ecosystem with software that people want to run. (Docker Hub sort of does this; I think there is a lot of room for improvement)
And you have to avoid the “rewriting upstream” problem which I mentioned in a comment above. For example, that is why I mentioned the HTTP proxy to help making existing package managers more reproducible. A good test case I always bring up is R code, since it has a very deep and functional ecosystem that you don’t want to rewrite
-
Nix already (mostly) doesn’t require rewriting the world of everyone’s build systems (in the way that Bazel does require). In fact it requires substantially less rewriting than systems like Bazel or Python wheels or any other non-containerized package manager AFAIK.
It’s not clear to me that there needs to be any fundamental change in philosophy to make the Nix idea widely usable - it seems the ideas you listed are ultimately things which Nix already does. You might like reading the original Nix thesis from 2006: https://nixos.org/~eelco/pubs/phd-thesis.pdf
The need, I think, is more just UX and polish.
-
It’s true that Nix requires / encourages fewer rewrites, because it’s more coarse grained while Bazel is fine-grained. But in the linked threads multiple users talk about the duplication of Cargo in the Rust ecosystem. So there still is some.
And for every package you do have to do all the
--prefixstuff because of the unconventional/nix/storelayout. This leads to the RPATH hacks too. However I tried to find this in the .nix derivations and it seems hidden. So I feel like there is a lot of magic and that’s where people get tripped up, compared to shell where everything is explicit.(And I appreciate the huge benefit that /nix/store gives you – containers will help but not eliminate that problem, since most packages aren’t relocatable.)
Echoing some previous threads, I don’t really have any problem with Nix having its own language per se. I’m surprised that people continue to complain about the syntax – as others pointed out, the language is just lazy “JSON (tuples) with functions”. It looks pretty conventional to me.
I think the more substantive complaint is lack of documentation about common library functions. And maybe the lazy evaluation rules.
The main issue I see is the leaky abstraction. At the end of the day Nix is just running a bunch of shell commands, so I think it’s more natural to build a system around shell. The package defs still need a bunch of inline shell to fix up various issues, just like in every other Linux distro.
FWIW I think you can make a staged execution model in shell like Make / Bazel, etc.:
http://www.oilshell.org/blog/2021/04/build-ci-comments.html#language-design-staged-execution-models
Performance is apparently a problem, and I think it’s more natural to debug performance issues in a model like that than a lazy functional language.
(Yes I read the thesis over 10 years ago and was surprised how similar it was to Bazel!)
-
-
I’d love to read through what you linked and wrote above and to try to process it as it deserves, but unfortunately I’m extremely busy recently. That’s also big part of the reason why practically all my side projects (including the attempts at a Nix wrapper for the Oil build env) are on a definite hiatus now, and I don’t expect this to change much in conceivable future if I’m still employed where I want to be (tradeoffs…). But for sure still generally interested in all of that, so scanning lobster.rs regularly and trying to stay more or less on top of what’s written here, to the limited extent that my brain still has some minimal capacity to consume!
-
Yeah I feel sort of the same way… I made an attempt to write a package manager ~8 years ago, and got an appreciation for how hard a problem it is! It still feels out of reach, since just the shell and the language are a huge project. (Obviously I am envisioning something based around a better shell)
So I put those ideas out there in hopes that someone else has 5 or 10 person-years to make it happen :)
-
-
I’d really like that. I played with nix for a bit and I feel like it’s a great experiment where we learned some approaches that could be moved from r&d to a real product. Getting rid of a lot of the flexibility and putting some real boundaries in place would really help.
I’d love it if we could transform the “here’s a trace through a stack of meta functions, none of which you wrote - deal with it” situation into “this is an expectation of this well supported pattern, you broke it in this place”. Basically make RuntimeException(haha_fail) into DerivationNeedsToBeWithinSystem(package_name). I know there’s dhall, but that doesn’t go far enough.
Alternatively nix could get LOTS of asserts with “to get past this point, these contracts need to be satisfied”.
-
That’s our approach with anysnake2 (link omitted, I don’t want to shill my pet project here). End-users write a toml file defining python and R ecosystem date and the list of packages, and it creates the necessary flake.nix for a container or a dev shell.
Works well, except for days where the ecosystem is broken for the packages you care about. But that’s full reproduciblity for you.
-
-
Container images are not automatically reproducible; builders must have special support for reproducibility, and Docker does not, but nixpkgs’ dockerTools does. As a practical matter, I have been bitten by the non-reproducibility of Docker containers.
-
I also work in HPC and most of our core services are Dockerized (even our hard RT service!). There’s been some attempts to use Nix, and one of our guys is a GUIX advocate, but I don’t really see it gaining any traction due to the complexity, lack of good documentation, and general usability woes. It’s hard enough getting physicists to learn basic docker commands…
-
Nix / Guix are quite orthogonal to Docker. Actually, you can use them to generate a Docker image.
In any case, I’m glad HPCs are at least moving to Docker. I’ve used some fairly large European HPCs, and the policy was basically that you should compile whatever you needed! I had some colleagues that would spend a good fortnight just to get things running. A regular company would panic at the amount of employee work, but academia on the other hand…
-
-
I meant not only inside Docker, but also to generate Docker images.
I think Nix is much more usable that what is generally discussed, thanks to the new nix command line, which is not often advertised. Obviously, it’s not an easy language or toolset, just like Haskell is not easy either.
But simple usecases are remarkably straightforward once you grasp the basic principles. I find some tasks much easier than Arch or Alpine, which are my goto distributions (and pretty barebones).
-
-
-
-
-
Note that this can be triggered in running & working Firefox instances, and might or might not disappear (temporarily?) upon restart.
Workaround seems to be to disable http3 in about:config for now.
-
Looks like
nix-shellis a handy tool to provide project-level reproducibility, and I think it’s much more useful compared to NixOS, which seems to offer workstation-level reproducibility. Most people replace their workstations very infrequently, and when they do, chances are that there exists some migration assistant (e.g. Time Machine or perhapsdd(1)). I don’t think I need to “deploy” my workstation-level configuration anywhere; it’s only meant for me to begin with.Tangentially, as a research assistant, I need to share a gigantic computing cluster with everyone affiliated with the university, and I don’t think I can convince the sysadmin into installing Nix on it (especially when the installer is a scary
curl -L https://nixos.org/nix/install | sh). I know a root-level/nixdirectory is required to make use of the binary cache since the absolute path to dynamic libraries is embedded in the cached binaries, but there must be some workaround. Like, why not just scan the relocation table of each cached binary and replace the prefix of the paths?-
I’m currently using NixOS as my main daily driver. The advantage for me isn’t workspace migrations, but workspace versioning. There is a lot of cruft that builds up on workspaces overtime. Packages that you download to try out, work-arounds that stick, etc. These are all versioned & documented in the gitlog. It also lets me do stupid things without having to really worry about the consequences on my machine, as I can rollback the changes easily.
-
I tried to configure the whole workspace with Nix on macOS, but it turns out that I cannot even install Firefox. This gives me the impression that
nixpkgsis currently not mature/popular enough (at least on macOS), and that at some point I will be forced to install Homebrew/MacPorts/Anaconda or runcurl shiny.tool/install.sh | shto get some niche package, and suddenly I have packages outside of version control.Also,
nix-env -qaP some_packageis already ridiculously slow, and with more packages in the repositroy, it will probably become even slower. More importantly, even a huge package repository cannot include everything, so from time to time users must write Nix expressions themselves, which I don’t think would be trivial (if it was, then Nix would have already automated that).I’m not complaining, but that’s the reason I’m not bold enough to use Nix as my daily driver. I guess I should donate some money to Nix to facilitate its growth.
-
I don’t disagree with the facts that you wrote, but I thought I’d comment since I cash some of them out differently… For a little context, I first took the dive into NixOS in early 2018, when my Windows desktop’s motherboard flamed out. It was rocky (a mix of Nix, plus it being my first desktop Linux), but I started using nix and nix-darwin when I replaced my macbook air in early 2019.
I tried to configure the whole workspace with Nix on macOS, but it turns out that I cannot even install Firefox. This gives me the impression that nixpkgs is currently not mature/popular enough (at least on macOS), and that at some point I will be forced to install Homebrew
-
A linux-first package manager’s ability to manage all software including desktop apps on macOS is a very stringent ruler. (Don’t get me wrong–it’ll be good if it works some day–but it’s a high bar, and Nix can be very useful without clearing it.)
-
Yes–keep homebrew. TBH, I think forcing desktop apps, many of which already want to auto-update, into the Nix paradigm is a bit weird. I, to be frank, find Nix on macOS, with nix-darwin, to be a very nice compromise over the purity of these apps on NixOS.
I actually find it more ergonomic to let these apps update as they have new security patches, and update my Nixpkgs channel less frequently. Since early 2019, I think I’ve only twice used Homebrew to install a package–I’ve used it almost exclusively for casks, and the packages were really just to play with something that was in Homebrew to decide if it was worth porting to Nix (neither was). Once again, I’d say the freedom to do this is a really nice compromise over purity in NixOS.
suddenly I have packages outside of version control.
You can still version-control a .Brewfile if it is for apps. It’s obviously not the same level of reproducibility, but if I’m trying to rebuild the software I had 3 years ago I’m generally not doing it for Chrome, Firefox, Steam, etc. I added a section to my backup script to barf out a report on whether I have installed anything with brew that isn’t in my brewfile. If I really cared, I think I could script it to uninstall those packages every day to force the issue.
If it’s for a smaller package and you care about version-controlled reproducibility this much, you’ll generally have enough motivation to port it to Nix. (In my experience it has been true, but I recognize that the practicality of this will depend on the scope of the package in question…)
More importantly, even a huge package repository cannot include everything, so from time to time users must write Nix expressions themselves, which I don’t think would be trivial
This is the proverbial two-edged sword. So, yes, yes, this can happen. I am personally very conservative when it comes to recommending Nix to anyone who isn’t open to learning the language. I think it can be okay, narrowly, as a tool with no knowledge of the language. Learning it can be frustrating. But:
- Nix can be a really big lever. I’m not sure if this is of much use to people who don’t program, but I feel like my time was well spent (even if it was a bigger investment than it needed to be).
- A lot of the difficulty of learning to write Nix packages has honestly just been my near-complete lack of experience with the processes for going from source to installed software in Unix-alikes. If you already know a lot about this, it’ll be mostly about the language.
- Packages aren’t all hard to write. They certainly can be nightmarish, but packages for “well-behaved” software can be fairly simple. Consider something like https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/smenu/default.nix which is almost entirely metadata by volume.
- It is fairly easy to manage/integrate private packages (whether that’s from scratch, or just overrides to use PRs I took the time to submit that the upstream is ignoring). I end up writing a Nix package for most of the unreleased/private software on my system (even when I could use it without doing so).
(if it was, then Nix would have already automated that).
Have any other package managers automated the generation of package expressions? I’m not terribly knowledgeable on prior art, here. If so, you’ve probably got a point. IME most of the work of writing package expressions is human knowledge going into understanding the software’s own packaging assumptions and squaring them with Nix. I’d be a little surprised if this is highly automatable. I can imagine translation of existing packages from other languages being more tractable, but IDK.
-
A lot of things that are using standard frameworks can be mostly automated. Debian/Ubuntu are doing pretty well with
dh. Nix already has templates https://github.com/jonringer/nix-templateIt’s not perfect, but as long as you’re using common rolling, packaging is not terribly hard.
-
-
As to
-qaP, unfortunately you learn to not do it; instead I’d recommend to use Nix 2.0’snix search, as it has some caching (and was in fact introduced primarily to solve the problem with-qaP); and then e.g.nix-env -iA nixpkgs.some_package. Or, alternatively, maybenix profile install, though I haven’t experimented with it yet myself.As to getting some niche package: yes, at some point you’ll either have to do this, or write your own Nix expression to wrap it. FWIW, not every byte in the world is wrapped by Nix, and This Is Just A FOSS Project Run By Good-Willing Volunteers In Their Spare Time, and You Can (Try To) Contribute, and going into another rabbit hole trying to wrap Your Favourite Package™ in a Nix expression is probably something of a rite of passage. I like to draw a parallel between Nix and earlier days of Linux, before Ubuntu, when you had to put a lot of time into it to have some things. A.k.a. your typical day at the bleeding edge of technology (which Nix is actually already not so much bleeding as it was just a few years ago). And, actually, people say Nixpkgs are kinda at the top on https://repology.org. But you know, at least when a package is broken on Nixpkgs, it doesn’t send your whole OS crashing & burning into some glitchy state from which you’ll not recover for the next couple years… because as soon as you manage to get back to some working terminal & disk access (assuming things went really bad on NixOS), or in worst case restart to GRUB, you’re just one
nix-rebuildgeneration away from your last known good state. And with flakes, you nearly certainly even have the source of the last known good generation in a git repo.-
I think the biggest problem is that for Nix to be useful, we must achieve nearly complete package coverage, i.e. almost all packages must be installable via Nix. Covering 90% of the most popular packages is still not good enough, because even a single non-reproducible package in the whole dependency graph will ruin everything. It’s an all-or-nothing deal, and assuming package usage follows a power-law distribution, we will have a very hard time covering the last few bits. This is very different from Linux, where implementing 90% of the functionality makes a pretty useful system.
Since you mentioned Linux, I’d like to note that packages from the system repository of most distributions are outdated, and users are encouraged to install from source or download the latest release from elsewhere (e.g. on Ubuntu 20.04, you must use the project-specific “PostgreSQL Apt Repository” to install PostgreSQL 13, which was released over a year ago). I guess some people took the effort to package something they want to use, but lack the incentive to keep maintaining it. While it’s perfectly fine to sidestep
aptoryumand runmake && make installinstead, you can never sidestep Nix because otherwise you would lose reproducibility. How can the Nix community keep nearly all packages roughly up to date? I have no clue.-
What I’m trying to answer to this, is to think small and “egoistically”: instead of thinking how Nix is doomed from a “whole world” perspective, try to focus just on your own localised use case and how much reproducibility you need yourself: if you must have 100% reproducibility, it means you either have enough motivation and resources to wrap the (finite number of) specific dependencies that you need and are not yet wrapped, or it’s apparently more of a would like to than must have, i.e. you have other, higher priorities overriding that. If the latter, you can still use Nix for those packages it provides, and add a few custom scripts over that doing a bit of curl+make or whatsit that you’ve been doing all the time till now (though once you learn to write Nix expressions for packages, you may realize they’re not really much different from that). Unless you go full NixOS (which I don’t recommend for starters), your base distro stays the same as it was (you said you don’t have root access anyway, right?) and you still can do all of what you did before. If some parts of your system are reproducible and some are not (yet), is it worse than if none are? Or maybe it is actually an improvement? And with some luck and/or persistence, eventually others may start helping you with wrapping the “last mile” of their personal pet packages (ideally, when they start noticing the benefits, i.e. their Nix-wrapped colleagues’ projects always building successfully and reproducibly and not breaking, and thus them being able to “just focus on the science/whatsit-they’re-paid-for”).
-
That makes sense. IMHO Nix can and should convince package authors to wrap their own stuff in Nix. It can because the Nix language is cross-platform (this is not the case for apt/yum/brew/pkg); it should because only authors can make sure the Nix derivations are always up-to-date (with a CI/CD pipeline or something) while minimizing the risk of a supply chain attack.
-
That is not how the open-source movement works. You don’t get to tell people what they should do; you rather take with humbleness and gratitude what they created, try to help them by contributing back (yet humbly enough to be ready to accept and respect if they might not take your contribution for some reason - though knowing you’re also free to fork), and yes, this means to possibly also contribute back ideas, but with the same caveat of them possibly not being taken - in this even more often, given that ideas are a dime a dozen. And notably, through contributing back some high quality code, you might earn some recognition that might give you a tiny bit more attention when sharing ideas. Ah, and/or you can also try to follow up on your ideas with ownership and actions, this tends to have the highest chance of success (though still not 100% guaranteed).
That said, I see this thread now as veering off on a tangent from the original topic, and as such I think I will take a break and refrain from contributing to making it a digression train (however I love digressions and however tempting this is), whether I agree or not with any further replies :) thanks, cheers and wish you great Holidays! :)
-
-
-
I think the biggest problem is that for Nix to be useful, we must achieve nearly complete package coverage, i.e. almost all packages must be installable via Nix.
Why do you think so? I’m wondering why this applies to Nix but not to Homebrew or apt or yum or the likes? One can still build a package manually by setting up the dependencies in a nix shell – that’s no different from building something that package managers of other systems still don’t have.
-
From my understanding, Nix aims to provide a reproducible environment/build, so it must exhaustively know about every piece of dependency. Homebrew, apt, and yum don’t have such an ambition; they just install packages, and can thus happily co-exist with other package managers and user-installed binaries.
-
Nix-build, yes; nix-shell, no. In a nix-shell env, you still see all of your pre-existing system, plus what nix-shell provides as an “overlay” (not in docker filesystem sense, just extra entries in PATH etc.). It reproducibly provides you the dependencies you asked it to provide, but it doesn’t guarantee reproducibility of what you do over that. So you could start with nix-shell (or
nix developIIRC in case of flakes).
-
-
-
-
-
This gives me the impression that nixpkgs is currently not mature/popular enough (at least on macOS)
Have no idea about Mac, but my understanding is that on Linux the amount of packaged stuff for NixOS is just ridiculously high: https://repology.org/repositories/graphs. Anecdotally, everything I need on daily basis is there.
Still, there are cases when you do want to try a random binary from the internet (happened to me last time when I wanted to try JetBrains Fleet first public release), and yeah, NixOS does make those cases painful.
-
-
-
-
-
In which case I either have docker/kurbernetes/podman or I’m using ansible to be fast and productive. Sure you may get some stuff done much more precisely in nixos, but that’s definitely not worth the hassle. That said: Best of luck to nixos, hopefully it’ll be stable enough one day.
-
Wait, what hassle? And what about it isn’t “stable?” Citation needed? It’s stable enough for every NixOS user I know on server and desktop and it’s stable enough for me. Hassle is a very vague word that could mean a lot of things, so if not for the fact that exactly zero of those possible meanings make what you said a true statement, I wouldn’t know how to respond to this. What is it about NixOS you think is a hassle?
-
Heh, my previous company went from NixOS to “let’s deploy using ansible on Ubuntu boxes, everyone knows those”. Productivity and velocity just went down the drain. Oh, the horrors, oh the PTSD… But everyone has different experiences, sometimes some tools work, sometimes they don’t.
-
much more precisely in nixos
I don’t know how you could get any more precise than NixOS. It specifies everything by hash, all the way down to a linker. I’ve never seen anybody do anything like that with any other system.
-
-
-
Looks like nix-shell is a handy tool to provide project-level reproducibility,
Definitely. Every project I use now has a
shell.nixfile that pins the tools I use. I switched to that workflow after I was bitten several times by brew replacing python, so virtual environments were not working, or completely forgetting what tools I need in a project (after returning to it a year later).shell.nixis acting both as a bill of materials and recipe for fetching the right tools. -
For me that work-around is ‘containers’. As in nix generates the containers (reproducible!), the clusters run the containers, and in the containers it’s /nix.
-
Check if you can get the following to print “YES”:
$ unshare --user --pid echo YES YESor any of the following to print
CONFIG_USER_NS=y(they all check the same thing IIUC, just on various distributions some commands or paths differ):$ zgrep CONFIG_USER_NS /proc/config.gz CONFIG_USER_NS=y $ grep CONFIG_USER_NS /boot/config-$(uname -r) CONFIG_USER_NS=yIf so, there’s reportedly a chance you might be able to install Nix without root permissions.
If you manage to get it, personally, I would heartily recommend trying to get into “Nix Flakes” as soon as possible. They’re theoretically still “experimental”, and even harder to find reliable documentation about than Nix itself (which is somewhat infamously not-easy already), but IMO they seem to kinda magically solve a lot of auxillary pain points I had with “classic” Nix.
Also, as a side note, the
nix-shellstuff was apparently much earlier than NixOS. The original thesis and project was just Nix, and NixOS started later as another guy’s crazy experiment, AFAIU.EDIT: If that fails, you could still try playing with @ac’s experimental https://github.com/andrewchambers/p2pkgs.
EDIT 2: Finally, although with much less fancy features, there are still some interesting encapsulation aspects in 0install.net: “0install also has some interesting features not often found in traditional package managers. For example, while it will share libraries whenever possible, it can always install multiple versions of a package in parallel when there are conflicting requirements. Installation is always side-effect-free (each package is unpacked to its own directory and will not touch shared directories such as /usr/bin), making it ideal for use with sandboxing technologies and virtualisation.”
-
This article did sent me on search for “how do you flakes and nix-shell at the same time” and apparently there’s “nix develop” now. This link has some details for both system wide and local setups: https://www.tweag.io/blog/2020-07-31-nixos-flakes/
-
Yeah, there’s also https://nixos.wiki/wiki/Flakes, and generally that’s the issue, that you need quite some google-fu to find stuff about them :)
-
-
-
ggplot2 for R… (though plotnine for python is a pretty decent port)
I’d say the whole tidyverse is the killer feature of R, and it has great docs too:
https://www.rstudio.com/wp-content/uploads/2015/02/data-wrangling-cheatsheet.pdf
The equivalent Python and Julia libraries, including Pandas, are largely copying what tidyverse does, often in a less elegant and composable way.
The origin story of Pandas was basically that the creator couldn’t stand R the language, and wanted to do the same thing in Python. I agree R the language is hard to learn, but tidyverse (including ggplot2) makes it worth it.
I tried to see if ChatGPT could explain R APIs, and at least for my examples it failed horribly.
Whether that says something about the language model, or R’s documentation, is anybody’s guess.
This “Tidy Data” paper by author Hadley Wickam is a great way to understand the tidyverse, which is much cleaner than R’s APIs:
https://vita.had.co.nz/papers/tidy-data.pdf
Also see references in my blog post: What Is a Data Frame? (In Python, R, and SQL)
ChatGPT can spit back common knowledge, but it doesn’t really understand anything, so I wouldn’t expect it to do well on this task.
It frequently makes up answers out of the blue, so if you don’t know something, you can’t rely on it. It’s only useful when you already know the answer but are perhaps having problems recalling it.
If you’re interested: what did you ask it to explain, about which task / package / function signature?
With that information, I can probably tell you whether the function has bad documentation (they definitely exist); if not, it’s a ChatGPT problem. Without that information, the answer is anybody’s guess :-)
The one I recall: using devtools::install_github, how to prevent it from asking about updating other packages.
I think the killer ones for Python are GPU-accelerated NumPy replacements, i.e. PyTorch and JAX. This is what really sets Python apart from other languages in terms of ML support.
I disagree with @hwayne on this one, Pandas is IMHO clunky and slow compared to R data.table or DataFrames.jl. I never found anything special about it.
I am surprised to see no mention of Simulink, which is a big competitive moat for MATLAB. I don’t think there’s anything else comparable in the domain of dynamic system simulation.
Mathematica’s standard library is really broad and well documented. Probably one of the best pieces of proprietary source code out there, both in terms of functionality and quality.
Others have already mentioned OTP, which would be another candidate.
Nit: s/QT/Qt/
QT is used for QuickTime.
Interesting that you mention Eigen. I suppose I didn’t use it to its full potential, just for some vector math and it was… fine? but not outstanding or one of a kind to me.
Seconding numpy (and also scipy!) for Python.
Recently I’ve been writing Rust code that does lots of math in 3D space, and although the performance is excellent (the reason I chose Rust!), I do find some of the math repetitive: I find myself thinking “I shouldn’t have to write an iterator-pipeline/for-loop just to multiply and average these vectors.” In retrospect, maybe I should have used nalgebra, but having never used it before, I have no idea how it compares to numpy.