Threads for catern

  1. 1

    Unfortunately flake.nix is more complicated and harder to use than traditional default.nix, but definitely one should use Nix rather than Docker.

    1. 7

      This is just ridiculous. The author works as a marketer at Pulumi. People call Pulumi “imperative” as an insult, so the author is redefining “imperative” to something completely different so that he can claim Pulumi is “declarative”.

      Personally I like Pulumi because it is imperative. If Pulumi was declarative I would not like it.

      Perhaps the author should focus on explaining how imperative is better (in Pulumi) than declarative, rather than creating his own bizarre definitions for “declarative” and “imperative” and gaslighting people into accepting them.

      1. 4

        Pulumi is no more imperative than Terraform’s CDK. And in terraform’s case it’s quite widely understood that the underlying system works in a declarative way. Making it easier to express the declaration by constructing it in an imperative language does not change that.

      1. 8

        The only problem with lots of custom aliases (or custom keybindings in other programs like editors), is that the muscle memory burns you every time you have to work on a remote machine. I used to go custom-to-the-max with my config, but I’ve gradually shifted back to fewer and fewer aliases except for the most prevalent build/version control commands I run dozens of times each day.

        1. 9

          When I need to remote into machines where I can’t set up my shell profile for whatever reason, I just config ssh to run my preferred shell setup commands (aliases, etc) as I’m connecting.

          My tools work for me, I don’t work for my tools.

          1. 5

            You mean, could single session only? Care to share that lifehack? I’m assuming something in ssh_config?

            1. 2

              Yeah, single session only. There are a bunch of different ways to skin this cat — LocalCommand and RemoteCommand along with ForceTTY in ssh_config can help.

              Conceptually you want to do something like (syntax probably wrong, I’m on my phone)

              scp .mypreferedremoterc me@remote:.tmprc; ssh -t me@remote “bash —rcfile ~/.tmprc -l; rm .tmprc”

              which you could parameterize with a shell function or set up via LocalCommand and RemoteCommand above, or skip the temp file entirely with clever use of an env variable to slurp the rc file in and feed it into the remote bash (with a heredoc or SendEnv/SetEnv)

          2. 2

            every time i have to work on a remote machine i do the commands through ssh or write a script to do it for me.

            1. 2

              naming a meta-archive-extracter, “atool” doesn’t help either. OP used unzip for this but it is overloaded. uncompress also is taken.

              What word would you guys use for aliasing it?

              1. 3

                I use extract as a function that just calls the right whatever based on the filename.

                1. 2

                  I think prezto comes with x alias, and I like it a lot. It’s burns easily into the muscle memory.

                2. 2

                  To defeat muscle memory when changing tools, I make sure the muscle memory command fails:

                  alias unzip = “echo ‘use atool’”

                  It doesn’t take many times to break the muscle memory. Then I remove the alias.

                  1. 1

                    Is atool there by default on Linux boxes?

                    1. 1

                      Nope. At least I’m not aware of any Linux distro installing it by default.

                      But being installed by default is IMHO totally overrated. The main point is that it is available in many Linux distribution’s repos without having to add 3rd party repos—at least in Debian and all derivatives like Devuan, Kali oder Ubuntu.

                      1. 2

                        I understand, but it’s not the same. If I don’t have a shell regularly there, and not my own dotfiles, I likely want to avoid installing and removing system packages on other people’s systems. When stuff breaks, I want the minimum amount of blame :)

                        Not that this is not a useful tool.

                        1. 1

                          Ok, granted. Working as a system administrator it’s usually me who has to fix things anyway. And it happens only very, very seldom that something breaks just because you install a commandline tool. (Saying this with about 25 years of Linux system administration experience.)

                          Only zutils can theoretically have an impact as it renames commandline system tools and replaces them with wrappers. But so far in the past decade, I’ve never seen any system break due to zutils. (I only swa things not working properly because it was not installed. But that was mostly because I’m so used to it that I take it as given that zutils is installed. :-)

                          1. 2

                            Yep, different role. I did some freelance work a long ago, and learned on (fortunately) my predecessor’s mistake: they hired me to do some work, because I guess someone before me updated some stuff, and that broke… probably PHP version? Anyway, their shop didn’t work any more and they were bleeding money till I fixed it. It was one of my early freelance jobs, so that confirmed the age-old BOFH mantra of if it ain’t broke, don’t fix it. So given time, I would always explicitly ask permission to do this or that or install the other, if needed.

                            But I went a different route anyway, so even though I am still better than average, I think, I’m neither good nor professional. But I think old habits die hard, so that’s why I’m saying “if this stuff isn’t there by default, you’ll just have to learn your tar switches” :)

                  2. 2

                    muscle memory burns you every time you have to work on a remote machine

                    Note that this doesn’t apply for eshell as the OP is using: If you cd to a remote machine in eshell, your aliases are still available.

                    1. 1

                      Command history and completion suggestions have really helped me avoid new aliases.

                    1. 7

                      I have been trying to package something with Nix for a couple of days now and the experience is just horrible. The documentation gets worse and more esoteric the deeper you go - thanks for this article, will allow me to un-bodge a bodged part of CI (just to bodge it differently, but still).

                      1. 11

                        Yeah, Nix the language is fine overall. What’s most difficult are all the libraries that ship with nixpkgs. It’s impossible to use them without reading the code. And there are small variations that prevent knowledge from being transferable.

                        Hopefully, the article helped bring back a bit more Dockerfile-like experience.

                        1. 13

                          Even reading the code is laborious due to dynamic typing and utter lack of documentation. If some package function takes an argument and passes it to some other function, and you want to know what type that function is, you need to find every call-site for that function or else go down the call stack and find every usage for that parameter. It’s an absolute grind.

                          1. 3

                            +1 to the nixpkgs libraries being tricky to follow/understand.

                            This was a major motivator for my in-progress switch to Guix (which has its own issues).

                            I find the library aspect of Guix much easier to follow, since they’re written wrt records with decent levels of abstractions (see the operating-system record, for example).

                          2. 4

                            While the whole nixpkgs is complex and lacks good docs, I’ve found it not that deep in the end. There’s a few language-specific abstractions to learn, nix language itself, but then… in the end it’s functions generating shell scripts. The serious problems remain where they are with any packaging - getting a reliable, reproducible environment. So for example dealing with Darwin sdk quickly becomes the biggest problem rather than nix.

                            This is not a disagreement with your experience, i also think it sucks. Just a note that “more esoteric the deeper you go” flattens out surprisingly quickly. I think I “got” nix quicker than deb/dh.

                            1. 4

                              Ah, then can you help me find out how to change the kernel in NixOS to L4Linux? I hit a cognitive wall around cross-compilation, especially trying to introduce a new architecture (L4Linux seemed to require this, or maybe it looked like a reasonable way forward when I tried doing this).

                              1. 1

                                I think that one would deserve a blog post rather than a lobsters comment :-)

                                1. 1

                                  I’m fine whichever way you prefer!

                              2. 1

                                in the end it’s functions generating shell scripts

                                This idea alone scares me

                                1. 2

                                  Most of the packaging is running configure, make, yarn, cargo, install and others. It all reduces to firing up a few processes at some point. Why is it scary?

                              3. 3

                                The best documentation is the implementation, for better or for worse

                                1. 2

                                  Nix documentation is bad

                                  Just how long should it stay like this? This something I’ve been reading for, what, 2 years…

                                  1. 9

                                    The documentation is awful and with the push for Flakes over regular derivations the situation is getting worse. It won’t be better for a while, although what exists is usable. But trying to do most non-trivial things requires finding someone who’s done something similar and using it as an example. I am trying to understand more of the ecosystem so I can start an alternative Nix/NixOS wiki. The current wiki is full of outdated and poorly explained disparate pieces of information and should really be more like ArchWiki.

                                    1. 1

                                      I don’t know how management of these projects work, but can’t they like, just assign 150k for two devs for two years to fix the documentation once and for all?

                                      1. 10

                                        You got 150k laying around and two people who know enough about Nix but don’t care to hack on it directly?

                                        1. 4

                                          You can certainly motivate people using pooled money. Nix macos is being supported through https://opencollective.com/nix-macos so maybe the documentation could be done the same way.

                                          1. 2

                                            Because it’s a technical problem. Documentation is a tough nut to crack. Technical people often don’t even know how to tackle it.

                                    2. 4

                                      Until a hero comes along.

                                      The NixOS project is largely driven by volunteers. Volunteers work on the things they are interested in. And you get to benefit from it for free. That’s the deal.

                                      It’s not some corporation where you allocate Human Resource on the docs. You need to actually get people interested in solving the problem, and it’s hard. Most people in open source are technical people, and they like to solve technical problems.

                                  1. 6

                                    This is great! I’m already fully in agreement with the philosophy for my site http://catern.com/ but I didn’t know some of the tricks about when you could validly omit tags like <html> and <head>! I’m going to start using that right away!

                                    And, it’s obvious in retrospect, but writing tables one-row-per-paragraph makes perfect sense. Previously I’ve been using org-mode radio tables (complicated Emacs stuff) to write my tables, but maybe now I can do away with that!

                                    1. 1

                                      Nice site! I’m delighted I could show you something new :)

                                    1. 4

                                      This list is somewhat dishonest. It starts with:

                                      ship by default “cool” themes like Solarized (it seems it’s very important to have some dark mode these days)

                                      Yes, this is superficial and better left up to the user.

                                      make Emacs more mouse-friendly (e.g. right clicking would open some context menus, there would be more tooltips on mouse hover and so on)

                                      have a fancy configuration wizard to help with the initial setup of Emacs

                                      These are, however, really powerful things that would greatly help both new and old users!

                                      The mouse support in Emacs is already great, but many experienced users turn it off (menu-bar and tool-bar) to reduce clutter in the UI; adding that back in with context menus provides a way to help both experienced and novice users.

                                      And many new users go straight to unnecessary heavy Emacs configuration frameworks, even if just a few basic customizations are all they want. This then makes life difficult for other experienced users who want to help them, but don’t know how to navigate the heavy-weight framework. Encouraging more use of customize and built-in Emacs features helps everyone.

                                      1. 8

                                        Yes, this is superficial and better left up to the user.

                                        Particularly when the author himself is responsible for an absolutely terrible emacs port of Solarized, which is best avoided at all costs. The last thing Emacs needs is new users asking why turning on the solarized theme inflicted a bunch of randomly different font sizes on their poor code when they’re expecting the kind of solarized experience they’d get from any other editor’s solarized port, instead.

                                        1. 5

                                          Well, I’m sorry you feel like this about Solarized and you’re always free to voice your concerns on the issue tracker as well. ;-)

                                          You are missing my point, though - for how many people exactly will bundling a theme like Solarized (or another other popular theme, that has been ported to your standards to Emacs) and making it the default be deciding factor to use or not use Emacs? :-) After all most themes are trivial to install even today. There are 20+ themes in the default Emacs repos today + a few themes that are bundled with Emacs.

                                        2. 6

                                          To be clear - I don’t mind making Emacs more approachable to newcomers at all. I’m just skeptical this is going to make a difference for the broader adoption of Emacs. Also I view user-friendliness and ease of use as being orthogonal to being “modern” (e.g. Emacs can be “dated” and user-friendly at the same time), that’s why in general I don’t like it when those two concepts get conflated.

                                        1. 2

                                          A desktop if you already have one, otherwise an old laptop, or your WiFi router.

                                          1. 2

                                            ^ this

                                            One can bring down the energy consumption of such old computers by downclocking and shut down all unused peripherials. with coreboot (if available) it should even be possible to run without graphics adapter. AFAIK pcengines.ch did this for their SBCs.

                                            1. 1

                                              Parallel suggestion: an old HP Microserver.

                                              I have 2, an N40L and an N54L. The 54 runs TrueNAS fine, with 8GB of RAM and 4×2TB disks, and is responsive and quiet.

                                              The 40 is noisy – I don’t know why – but runs OpenMediaVault happily in 6GB of RAM.

                                              Both cost me about GBP 100 used.

                                            1. 1

                                              Can Standard ML be made so it can write device drivers?

                                              Yes, this is quite like what the Fox Project in the 90s did.

                                              1. 2

                                                Did they produce device drivers? FoxNet was a network stack in SML and that too in userspace. Not exactly device drivers. With the exception of mlton, SML runtimes (and OCaml too) use an uniform value representation therefore use a few bits of the pointer for the type tag. GC also takes away a bit typically which means full width machine words needs to be boxed and that can be inefficient.

                                                ATS, though, is the perfect combination of ML and C, barring the syntax. The metasepi project is using ATS for kernel components[1].

                                                [1] https://metasepi.org/en/tags/ats.html

                                              1. 10

                                                Finally it means you need two separate sets of keys to do almost the same thing inside or outside of Emacs. Pretty nasty and pointless!

                                                You can also solve this by going in the other direction by making Emacs your window manager. I moved to EXWM a few years ago and have never looked back, it’s incredible how much better basically all of my workflows have become. It also doesn’t really need any frequent handholding, I seem to touch this config about once per month.

                                                This post reminds me of another nested version of this problem that some EXWM users have: Applications (especially browsers) with tabs, which they’d prefer to move into Emacs as well.

                                                1. 1

                                                  And even if you don’t use exam - this problem becomes less significant the more you live inside Emacs. So that’s one motivation to do things in Emacs.

                                                  1. 1

                                                    Applications (especially browsers) with tabs, which they’d prefer to move into Emacs as well.

                                                    surf is a handy little browser for this sort of thing

                                                    1. 1

                                                      I tried one-tab-per-window inside exwm with firefox but it was significantly slower to create a new window than creating a new tab, so I ended up having to ditch it. Hadn’t tried it with one of the webkit ones; maybe once I get forced to stop using exwm by wayland I’ll give it a shot. Is it easy to configure surf with emacs keys? When I looked at it years ago it didn’t look straightforward.

                                                      1. 1

                                                        Is it easy to configure surf with emacs keys?

                                                        I don’t use surf, but there are similar patches for dwm. I added emacs keys to tabbed (haven’t published the patch yet), and it was pretty easy there too. I’m guessing surf would be similar to these other suckless projects.

                                                    1. 2

                                                      It’s an interesting article with what seems to be an interesting thrust, but I’m struggling a bit to understand where the author is trying to take it.

                                                      So, yes, modern computers are in and of themselves distributed systems with a myriad of buses, multi-core processors, and arbitrarily complex software systems governing their operation. That in and of itself is an almost miraculous thing worth pondering.

                                                      But it seems like from there the author is asserting that we don’t currently have good abstractions to help us write potentially globe spanning distributed systems, and this is where they and I part ways.

                                                      I work for a Major Cloud Provider, and we operate crazy scale globe spanning distributed systems as a way of life. We’re practically awash in abstractions that help us achieve this. They all have different characteristics depending on precisely what they’re trying to achieve and how they’re choosing to present that, and no doubt as with everything they need to iterate and evolve.

                                                      So, if what they’re really saying is “We need to continue innovating better abstractions that makes running distributed systems easy and safe” then I’m in violent agreement :)

                                                      1. 8

                                                        I work for a Major Cloud Provider, and we operate crazy scale globe spanning distributed systems as a way of life. We’re practically awash in abstractions that help us achieve this.

                                                        Yes, but those don’t abstract away the distributed nature of the system. That’s why they differ from the abstraction in an individual computer.

                                                        1. 2

                                                          There’s a challenge there though, right? Because human brains by design have an incredibly difficult time visualizing parallel tasks.

                                                          Look at the utter sh** show any kind of threaded programming has been for the last 30 years even when we know there are far better models like Actors available the whole time.

                                                          How does one both enable people to build reliable distributed systems that work and NOT hide their distributed nature?

                                                          1. 3

                                                            How does one both enable people to build reliable distributed systems that work and NOT hide their distributed nature?

                                                            I think you may have misunderstood me. Today’s large-scale distributed abstractions don’t hide the distributed nature of the system, and, ostensibly, they’re used to build reliable systems.

                                                            That differs from the abstraction in an individual computer, which does hide the distributed nature of the system, and also, ostensibly, is used to build reliable systems.

                                                            Both kinds of abstractions, “hiding” and “non-hiding”, are heavily used. (Whether one is better than the other, I haven’t said anything about)

                                                            1. 2

                                                              How does one both enable people to build reliable distributed systems that work and NOT hide their distributed nature?

                                                              I think we already have some great abstractions around distributed systems in both theory and practice, especially if you work on a big cloud. However once it comes to an individual computer, we pretend that everything is working synchronously. I think we could author our systems much more effectively by designing with distributed systems abstractions from the beginning.

                                                              1. 1

                                                                Ah you’re absolutely right. We’re still building systems using an architecture designed in the 50s and 60s when computers were implemented using vacuum tubes, paper tape, and raw grit :)

                                                                I do have to wonder though - If we start thinking about building computers based on non Von Neumann architectures, will humans actually be able to reason about them and their internal workings?

                                                                I’d argue that even WITH an essentially serial architecture most people, myself included can’t even begin to truly wrap our brains around everything inside their modern computer.

                                                                It’s one of the reasons I so very much enjoy working with 8 bit era systems like my Atari 800XL. You really CAN understand everything about the machine from ‘tail to snout’ as they say :)

                                                                1. 3

                                                                  Ah you’re absolutely right. We’re still building systems using an architecture designed in the 50s and 60s when computers were implemented using vacuum tubes, paper tape, and raw grit :)

                                                                  Most modern CPUs have a Harvard architecture up to L2 or L3.

                                                                  1. 2

                                                                    Thanks for that. TIL!

                                                                    https://en.wikipedia.org/wiki/Harvard_architecture

                                                                    I’d not heard of the Harvard Architecture but reading about it the issues around contention are certainly widely felt whenever you talk about performance.

                                                                    1. 1

                                                                      While separate L1 instruction and data caches are ubiquitous, yes, I think that’s largely an implementation detail due to circuit-design constraints – they’re still the same address space. Some CPUs, e.g. x86, will even enforce coherence between them, so a store instruction to an address that happens to be present in the I-cache will invalidate that line (though others require manual I-cache invalidation for things like JITs and self-modifying code).

                                                                2. 2

                                                                  Look at the utter sh** show any kind of threaded programming has been for the last 30 years even when we know there are far better models like Actors available the whole time.

                                                                  Better for what? There are plenty of problems where threads are preferable to actors. There’s a reason threads were invented in the first place, despite already having processes which can communicate through message passing.

                                                                  1. 3

                                                                    I would be open to hearing this story. The version I learned is that threads arose because disk I/O was expensive, and sharing the CPU with threads could allow a program to simultaneously wait for a disk and run a computation. Today, we have better options; we can asynchronously manage our I/O.

                                                                    1. 2

                                                                      Actors are an abstraction implemented on top of threads. (I have implemented Actors myself.)

                                                                      Concurrency by means of multiple communicating OS processes tends to be inefficient, because processes are expensive to create and slow to context-switch between. Messaging is expensive too. So lightweight threads in a process were a performance boost as well as easier to use.

                                                                      The advantage of actors is they’re much easier to reason about. But I agree with you that in some cases it’s simpler to use threads as your model and just deal with mutexes.

                                                                      1. 4

                                                                        in some cases it’s simpler to use threads as your model and just deal with mutexes.

                                                                        Also, in a lot of the (IMO) good use cases for threads, “just deal with mutexes” is barely even a concern. If you are processing a bunch of independent units of work with little or no shared state, threads make the flow of control really easy to reason about and the classic pitfalls rarely come up.

                                                                        This is arguably the situation for a pretty big percentage of multithreaded programs. For example, .NET or Java web services, where there are large numbers of framework-managed threads active and there is shared state under the covers (database connection pools, etc.) but the vast majority of the day-to-day application logic can be correctly and safely written with zero attention paid to thread-related issues.

                                                                        1. 1

                                                                          True dat, but what you’re describing is also a good fit for Actors or similar abstractions. Even the “under the covers” part.

                                                                          1. 1

                                                                            Exactly. A lot of network services that work on stateless protocols (looking at you SIP and NNTP…) essentially have no state to mutate except for a backend database. Threads are easy abstractions because you can partition incoming work in exactly the thread/db connection pool patterns that you talk about.

                                                                            For more stateful work (say working on some form of distributed cache), the thread pool model can proved to be much more complicated.

                                                                        2. 1

                                                                          So, you’re right. My comment was off the cuff and not particularly articulate.

                                                                          What I was getting at is that there was a time - mostly the early-ish Java era, where legions of work-a-day business programmers were exposed to threads and concurrency in contexts with which they were unfamiliar.

                                                                          They then proceeded to, in the large, make a giant cock-up of it because without some helpful abstractions to moderate them, threads can be so powerful that they give people more than the rope they need to hang themselves.

                                                                          Later on, the Java community reacted to this and introduced things like ThreadPool and ThreadGroup (I think that was the name. My Java is rusty) which helped people use threads in ways that were much easier to reason about, and they had a much better time.

                                                                          But you’re right, in the hands of a capable programmer with the right experience, threads are an incredibly powerful tool with tremendous potential.

                                                                          1. 2

                                                                            FWIW the abstraction we arrive at doesn’t have to be threads, actors, or anything else. I’m personally very sympathetic to capability-passing style API boundaries (and I know a few folks here are also 😛) but what I’m trying to get at is, the current abstractions we have now in systems design is purely trying to hold onto the old synchronous model of computing. It’s not a thoughtful abstraction to help tame the complexity of modern computers. It’s literally just an accident of history. And it leaves out a lot of important things you could do an API that does anything except pretend like every action is synchronous.

                                                                  1. 31

                                                                    Tl;DR; The Interface Definition Language for Operating Systems is C…. and it’s notoriously bad at being an IDL.

                                                                    Yup. Dead right.

                                                                    If you want something close to a real IDL, you probably should be looking at dbus.

                                                                    Yup. Dbus sucks, but as Poettering points out… it exists, it’s in real world practical use and has a number of other important security characteristics.

                                                                    https://archive.fosdem.org/2014/schedule/event/anatomy_of_kdbus/attachments/slides/460/export/events/attachments/anatomy_of_kdbus/slides/460/kdbus_fosdem.pdf

                                                                    1. 9

                                                                      On Linux, it’s not really C. It’s a bunch of hardcoded syscall numbers and carefully formatted structures passed in-memory, along with a bunch of prose documentation. The userspace interface to Linux is unfortunately not formally defined/described - but the interface is stable, which makes it less of a problem.

                                                                      glibc or musl or other libcs provide a C API to access Linux functionality, in the form of functions like “read” and “write”, but other languages can “use the Linux interface directly” - just make syscalls directly.

                                                                      1. 6

                                                                        Unfortunately, POSIX mandates the existence of the C itnerfaces but nothing else. Some systems, such as macOS or Solaris, don’t provide a stable system call interface, only a stable C interface. Go was bitten by this when a minor update of macOS changed the ABI of the system call used to implement gettimeofday, which the Go runtime called on startu. Go, at the time, bypassed libSystem and issued system calls directly and so every single Go program failed on startup until it was recompiled with an updated compiler.

                                                                        Even on Linux, the situation is quite painful because Linux makes the system call table part of the architecture-specific code and so the system call numbers differ between architectures and sometimes even the arguments for system calls change. This isn’t the case on *BSD (where the system call tables are all part of the architecture-agnostic code and the machine-dependent code just provides a way of getting the arguments out of the trap frame).

                                                                        1. 1

                                                                          I wrote about this: http://catern.com/linux_api.html

                                                                      1. 15

                                                                        In the last few days, a few popular NodeJS libraries (like vue-cli and node-ipc) have been “weaponized” to wipe the users files if the IP-based geo-location indicates the user is from Russia or Belarus.

                                                                        The problem is that such geo-location is not always 100% accurate, thus going forward, especially those living in nearby countries (which might be geo-located by mistake as in one of the targeted countries), should perhaps pay extra care to the new dependencies one adds, or updates, to his software… (Also pay attention to recursive dependencies…)

                                                                        Moreover, given the current geo-political landscape, given the war and sanctions, it’s not unrealistic to believe that the other side might start doing the same and target users from western countries…

                                                                        As one has said in a GitHub issue:

                                                                        […] the Pandora’s box is now opened […]

                                                                        1. 16

                                                                          The problem is that such geo-location is not always 100% accurate, thus going forward, especially those living in nearby countries (which might be geo-located by mistake as in one of the targeted countries), should perhaps pay extra care to the new dependencies one adds, or updates, to his software… (Also pay attention to recursive dependencies…)

                                                                          I guess it goes without saying that these issues apply even more to Russians and Belarusians. It just seems strange to completely leave that out.

                                                                          1. 6

                                                                            In the last few days, a few popular NodeJS libraries (like vue-cli and node-ipc) have been “weaponized” to wipe the users files if the IP-based geo-location indicates the user is from Russia or Belarus.

                                                                            The code to do this was never deployed anywhere.

                                                                            I don’t want to cast aspersions about you specifically, but the fact that people keep repeating this sensational false claim smacks of Russian propaganda efforts.

                                                                            1. 15

                                                                              The code to do this was never deployed anywhere.

                                                                              The fact that the code was or was not deployed anywhere doesn’t mean it doesn’t exist, or that the intention didn’t exist.

                                                                              Just for reference, there is a tag, v10.1.2 (https://github.com/RIAEvangelist/node-ipc/commits/v10.1.2), that does contain the code in question as part of the commit https://github.com/RIAEvangelist/node-ipc/commit/847047cf7f81ab08352038b2204f0e7633449580). Granted this version was not published on NPM, https://www.npmjs.com/package/node-ipc?activeTab=versions, but that doesn’t mean that some automated tool that happens to download the code directly from GitHub, wouldn’t be affected by the issue…

                                                                              Also, at least for node-ipc there exists a CVE on this topic, which is reason enough to be aware of the situation: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23812

                                                                              It’s not like in the last few months, even before the whole war started, there were no open-source developers that decided to yank their repository (left-pad), change their code so that on startup it enters an infinite loop and output raw characters on the console (colors), and many other such cases which perhaps didn’t got through the news mill…

                                                                              As the situation escalates, I bet it will spill into other domains such as open-source…


                                                                              I don’t want to cast aspersions about you specifically, but the fact that people keep repeating this sensational false claim smacks of Russian propaganda efforts.

                                                                              Please do cast suspicions! It’s healthy to be skeptical!

                                                                              What is exactly the “sensational false claim”? That some developers have wrote the code in question, and even tagged it?

                                                                              How does it help the Russian propaganda? By raising awareness that perhaps developers should pay closer attention to how deep their dependency trees are? By raising the awareness that our development tools, which with the exception of Go, will gladly download and run any code they find in their repositories as part of a simple build?

                                                                              It’s not like the the Nvidia driver make file didn’t include rm -rf /usr /lib/nvidia-current/xorg/xorg at one point (https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issues/123)…

                                                                              1. 14

                                                                                Granted this version was not published on NPM, https://www.npmjs.com/package/node-ipc?activeTab=versions,

                                                                                Apparently, at least according to the snyk.io article, the vulnerable versions were indeed published on NPM for some time, then afterwards they were retracted.

                                                                                1. 4

                                                                                  That’s accurate. 9.2.2, 10.1.1 and 10.1.2 were all pulled from npm.

                                                                              2. 12

                                                                                Not everything false on the internet is Russian propaganda. It turns out people have been saying untrue things on the web for a little while without Russia’s help. Also, it was actually published on npm so just looking in from the outside here the person posting misinformation seems to be you?

                                                                                1. 4

                                                                                  which sensational false claim do people keep repeating?

                                                                                  1. 3

                                                                                    The sentence I quoted.

                                                                                    1. 6

                                                                                      So the code was not weaponized because it never made it to the main NPM repo? Really doesn’t help to use the term “false claim” for such ambiguous, subjective issues, especially when you’re making some meta point about propaganda by an official enemy.

                                                                                  2. 3

                                                                                    The code to do this was never deployed anywhere.

                                                                                    What’s your evidence of this? npm doesn’t show removed, malicious versions of packages, so you can’t just trust their site.

                                                                                1. 2

                                                                                  So what does this do that iptables doesn’t do? Redirecting traffic to different ports is a pretty old capability, seems like this should just be a friendly API on iptables.

                                                                                  1. 5

                                                                                    The repository seems to be linked to CloudFlare’s Production ready eBPF, or how we fixed the BSD socket API from a few days ago. (I’ve missed the article, so I’ll have to read it.)

                                                                                    BTW, not long ago they’ve also released a related article How to stop running out of ephemeral ports and start to love long-lived connections.

                                                                                    I think both of them describe how CloudFlare uses the TCP/IP and UDP/IP stacks both for incoming (internet->server) and outgoing (server->internet) flows.

                                                                                    1. 5

                                                                                      Despite it not being a lot of code, what is now tubular has been in the works for quite a while! But indeed it doesn’t predate any of the iptables capabilities you may be referring to. The truth is, Cloudflare has been using those iptables features (though not Destination NAT AFAIK) to deliver new connections to listening sockets. But there are a lot of pain points when trying to achieve what’s needed for the variety of services that run on the servers.

                                                                                      If you’d like to know more about the justification for the bpf_sk_lookup hook, and specifics as to why it’s more suitable than alternatives/previous attempts, this talk presented at Linux Plumber’s Conf on Programmable Socket Lookup explains in more detail.

                                                                                      1. 2

                                                                                        Thanks that talk looks quite interesting.

                                                                                        1. 1

                                                                                          I watched the talk and looked at the patches, but I’m not convinced, and it seemed like several netdev posters weren’t either. When did they get convinced that a TPROXY approach would not work?

                                                                                          This seems like an abuse of BPF. There weren’t any fundamental issues with the TPROXY approach, nothing that couldn’t be solved. Making something fully programmable is good, but here that programmability is just replicating the existing APIs, which is bad - slower, less introspectable.

                                                                                        2. 4

                                                                                          OK, so I’ve read the mentioned article, Production ready eBPF, or how we fixed the BSD socket API, and it at first it does seem to be some sort of local DNAT.

                                                                                          However I don’t think it’s equivalent at least for the following reasons:

                                                                                          • scale – it seems CloudFlare wants to have many small services listening on many IP/port ranges; with iptables one has to create one rule for each such mapping which quickly gets out-of-hand;
                                                                                          • transparency – with iptables DNAT the local process can’t get the original destination IP; the sockets API only returns the IP/port on which the socket was bound;
                                                                                        1. 24

                                                                                          I realize that Docker (or, I suppose, “containers”) provides fewer guarantees and comes with a host of problems, but as a practical matter, it has delivered on (more or less) the same promise, but in a way that is accessible and easy to understand.

                                                                                          The lab I work in uses Docker for basically everything (even HPC cluster jobs) because scientific software is notoriously finicky and containers make life so, so much simpler. Could we use Nix? Sure, but it takes 5 minutes to teach someone how to use Docker well enough to be productive, while it would take… I don’t even know how long with Nix.

                                                                                          In fact, I don’t even know how to use Nix well enough to teach someone else, and I actually built a Nix package a few years back! The language was so inscrutable and poorly-documented that I basically just copy-pasted from other packages until it worked, then decided I never wanted to do that again.

                                                                                          1. 6

                                                                                            pkgs.dockerTools.buildImage might be a good way to get started with Nix then. In my (admittedly very limited) experience the Nix code ends up being easier to understand than the equivalent Dockerfile. Probably because as long as you keep to simple Nix expressions there are far fewer gotchas. No need to clean up after installing packages, no need for multi-stage builds, no need to worry about putting “volatile” statements later to minimise build times, and much less shell scripting.

                                                                                            1. 5

                                                                                              I wonder how feasible it would be to write, say, a python library that lets you take advantage of Nix without having to do the full buy-in of the special language.

                                                                                              I have similar gripes about Bazel, with “almost-Python” and documentation that really doesn’t want you to do hacky things (despite hacky things being necessary for taking an existing project and bundling it!).

                                                                                              Docker is a ball of mud generator, but it is also a ball-of-mud capturer. Nix demands you to throw your mud into a centrifuge while you still need the ball to roll around. Whoever can figure out how to give all the powers of both of these will be my savior

                                                                                              1. 10

                                                                                                FWIW:

                                                                                                • @ac is occasionally experimenting with trying to capture the core idea of Nix in a simpler form;
                                                                                                • the authors of Nix are actually also experimenting with a “simpler” language;
                                                                                                • not strictly related, but CUE is a configuration language I find very interesting, and I sometimes wonder if it could be used to build some Nix-like system on, and if that could make things more attractive and usable (if still not as easy as docker to learn; but at least giving a static typing system in exchange).
                                                                                                1. 5

                                                                                                  FWIW I wonder if https://earthly.dev/ is doing this … It’s a container based distributed build system, which in my mind is sort of the middleground between Docker and Nix.

                                                                                                  I mentioned it here in the “curse of NixOS thread” https://lobste.rs/s/psfsfo/curse_nixos#c_cqc27k

                                                                                                  I haven’t used it but I’d be interested in a comparison / evaluation

                                                                                                  We need a “meta-build” system that solves the reproducibility/parallelism/distribution/incrementality problem for everything at once without requiring O(N) rewrites of upstream build systems. I don’t really have any doubt that this will be based on containers.


                                                                                                  I also collected my past comments on this subject pointing to some ideas on composable Unix-y mechanisms to solve this problem:

                                                                                                  https://oilshell.zulipchat.com/#narrow/stream/266575-blog-ideas/topic/Idea.3A.20A.20Middleground.20between.20Docker.20and.20Nix (login with Github)

                                                                                                  • Allow both horizontal layers (like Docker) and vertical slices (like Nix)
                                                                                                  • A hashing HTTP proxy for package managers that don’t have lockfiles and hashes (which is a lot of them).
                                                                                                  • Storage on top of git to separate metadata from data (layers and big blobs)
                                                                                                    • following the “git ops” philosophy but you need something for big files like layers; I have tried git annex
                                                                                                  • A remote process execution abstraction on top of containers, with data dependencies (this is sort of where a distributed shell comes in, i.e. you want to take some shell command, package it up with dependencies, execute it somewhere else, and name the output without necessarily retrieving it)
                                                                                                  • which leads to a coarse grained dependency graph (not fine-grained like Bazel)
                                                                                                  • haven’t figured this part out, but it appears that the static dependency graph and up-front evaluation is a fairly big constraint that makes it harder to write package definitions; a more dynamic notion dependencies might be useful (but also introduces problems I’m sure)

                                                                                                  But now that I list it out, this is a huge multi-year project, even though it is trying to reuse a lot of stuff (git, container runtimes like podman and bubblewrap, HTTP) and combine it in a Unix-style toolkit … The scope is not that surprising since it’s tantamount to developing a distributed OS :-/ (i.e. task execution and storage management on distributed hardware; building any application)

                                                                                                  I’ll also add that this system is “decentralized in the sense of git, BitTorrent, and the web” – i.e. it’s trivial to set up your own disconnected instances. Docker, Nix, and Bazel don’t feel that way; you kind of have buy into a big ecosystem


                                                                                                  But I’d like to hear from anyone who is interested. @akavel I did chat with @ac about this a few years ago … I think my main feedback is that you have to go further to “boil the ocean”.

                                                                                                  That is, it is not enough to come up with a “clean model” … (which Nix and Bazel already provide). You have to provide a bunch of mechanisms that will help people write correct and clean package definitions for a wide array of diverse software (and config language that’s better than a bunch of macros and dynamic parsing on top of shell or Python, which I think Oil is incidentally :-) ). And “kick start” that ecosystem with software that people want to run. (Docker Hub sort of does this; I think there is a lot of room for improvement)

                                                                                                  And you have to avoid the “rewriting upstream” problem which I mentioned in a comment above. For example, that is why I mentioned the HTTP proxy to help making existing package managers more reproducible. A good test case I always bring up is R code, since it has a very deep and functional ecosystem that you don’t want to rewrite

                                                                                                  1. 1

                                                                                                    Nix already (mostly) doesn’t require rewriting the world of everyone’s build systems (in the way that Bazel does require). In fact it requires substantially less rewriting than systems like Bazel or Python wheels or any other non-containerized package manager AFAIK.

                                                                                                    It’s not clear to me that there needs to be any fundamental change in philosophy to make the Nix idea widely usable - it seems the ideas you listed are ultimately things which Nix already does. You might like reading the original Nix thesis from 2006: https://nixos.org/~eelco/pubs/phd-thesis.pdf

                                                                                                    The need, I think, is more just UX and polish.

                                                                                                    1. 1

                                                                                                      It’s true that Nix requires / encourages fewer rewrites, because it’s more coarse grained while Bazel is fine-grained. But in the linked threads multiple users talk about the duplication of Cargo in the Rust ecosystem. So there still is some.

                                                                                                      And for every package you do have to do all the --prefix stuff because of the unconventional /nix/store layout. This leads to the RPATH hacks too. However I tried to find this in the .nix derivations and it seems hidden. So I feel like there is a lot of magic and that’s where people get tripped up, compared to shell where everything is explicit.

                                                                                                      (And I appreciate the huge benefit that /nix/store gives you – containers will help but not eliminate that problem, since most packages aren’t relocatable.)


                                                                                                      Echoing some previous threads, I don’t really have any problem with Nix having its own language per se. I’m surprised that people continue to complain about the syntax – as others pointed out, the language is just lazy “JSON (tuples) with functions”. It looks pretty conventional to me.

                                                                                                      I think the more substantive complaint is lack of documentation about common library functions. And maybe the lazy evaluation rules.

                                                                                                      The main issue I see is the leaky abstraction. At the end of the day Nix is just running a bunch of shell commands, so I think it’s more natural to build a system around shell. The package defs still need a bunch of inline shell to fix up various issues, just like in every other Linux distro.

                                                                                                      FWIW I think you can make a staged execution model in shell like Make / Bazel, etc.:

                                                                                                      http://www.oilshell.org/blog/2021/04/build-ci-comments.html#language-design-staged-execution-models

                                                                                                      Performance is apparently a problem, and I think it’s more natural to debug performance issues in a model like that than a lazy functional language.

                                                                                                      (Yes I read the thesis over 10 years ago and was surprised how similar it was to Bazel!)

                                                                                                    2. 1

                                                                                                      I’d love to read through what you linked and wrote above and to try to process it as it deserves, but unfortunately I’m extremely busy recently. That’s also big part of the reason why practically all my side projects (including the attempts at a Nix wrapper for the Oil build env) are on a definite hiatus now, and I don’t expect this to change much in conceivable future if I’m still employed where I want to be (tradeoffs…). But for sure still generally interested in all of that, so scanning lobster.rs regularly and trying to stay more or less on top of what’s written here, to the limited extent that my brain still has some minimal capacity to consume!

                                                                                                      1. 2

                                                                                                        Yeah I feel sort of the same way… I made an attempt to write a package manager ~8 years ago, and got an appreciation for how hard a problem it is! It still feels out of reach, since just the shell and the language are a huge project. (Obviously I am envisioning something based around a better shell)

                                                                                                        So I put those ideas out there in hopes that someone else has 5 or 10 person-years to make it happen :)

                                                                                                    3. 3

                                                                                                      I’d really like that. I played with nix for a bit and I feel like it’s a great experiment where we learned some approaches that could be moved from r&d to a real product. Getting rid of a lot of the flexibility and putting some real boundaries in place would really help.

                                                                                                      I’d love it if we could transform the “here’s a trace through a stack of meta functions, none of which you wrote - deal with it” situation into “this is an expectation of this well supported pattern, you broke it in this place”. Basically make RuntimeException(haha_fail) into DerivationNeedsToBeWithinSystem(package_name). I know there’s dhall, but that doesn’t go far enough.

                                                                                                      Alternatively nix could get LOTS of asserts with “to get past this point, these contracts need to be satisfied”.

                                                                                                      1. 2

                                                                                                        That’s our approach with anysnake2 (link omitted, I don’t want to shill my pet project here). End-users write a toml file defining python and R ecosystem date and the list of packages, and it creates the necessary flake.nix for a container or a dev shell.

                                                                                                        Works well, except for days where the ecosystem is broken for the packages you care about. But that’s full reproduciblity for you.

                                                                                                      2. 5

                                                                                                        Container images are not automatically reproducible; builders must have special support for reproducibility, and Docker does not, but nixpkgs’ dockerTools does. As a practical matter, I have been bitten by the non-reproducibility of Docker containers.

                                                                                                        1. 2

                                                                                                          I also work in HPC and most of our core services are Dockerized (even our hard RT service!). There’s been some attempts to use Nix, and one of our guys is a GUIX advocate, but I don’t really see it gaining any traction due to the complexity, lack of good documentation, and general usability woes. It’s hard enough getting physicists to learn basic docker commands…

                                                                                                          1. 1

                                                                                                            Nix / Guix are quite orthogonal to Docker. Actually, you can use them to generate a Docker image.

                                                                                                            In any case, I’m glad HPCs are at least moving to Docker. I’ve used some fairly large European HPCs, and the policy was basically that you should compile whatever you needed! I had some colleagues that would spend a good fortnight just to get things running. A regular company would panic at the amount of employee work, but academia on the other hand…

                                                                                                            1. 1

                                                                                                              Sure, you can use Nix/Guix in Docker too. The point is that the usability of those platforms is awful and I wouldn’t encourage any company to use them in production as they are now.

                                                                                                              1. 1

                                                                                                                I meant not only inside Docker, but also to generate Docker images.

                                                                                                                I think Nix is much more usable that what is generally discussed, thanks to the new nix command line, which is not often advertised. Obviously, it’s not an easy language or toolset, just like Haskell is not easy either.

                                                                                                                But simple usecases are remarkably straightforward once you grasp the basic principles. I find some tasks much easier than Arch or Alpine, which are my goto distributions (and pretty barebones).

                                                                                                        1. 46

                                                                                                          If there’s anything that should be required to use free software, it should be biomedical devices. To have a part of your body only modifiable and controllable by a corporation that keeps the details as proprietary secrets is like something out of a cyberpunk dystopia. And yet, this article is a great example of it happening!

                                                                                                          1. 5

                                                                                                            I guess the counter-argument is that (as a society) we don’t want random people tinkering with medical implants, for the same reasons we don’t want people making snake-oil medicine or amateur surgeons.

                                                                                                            I’d like to think there’s an acceptable middle-ground, but I’m not sure what that would even look like.

                                                                                                            1. 18

                                                                                                              I guess the counter-argument is that (as a society) we don’t want random people tinkering with medical implants, for the same reasons we don’t want people making snake-oil medicine or amateur surgeons.

                                                                                                              There are actually pretty robust communities working towards automated insulin delivery and (IIRC) CPAP machines. And on the medication front you have RADVAC. There’s a lot of interesting exploration going on here.

                                                                                                              I’d like to think there’s an acceptable middle-ground

                                                                                                              One option I heard that appealed to me: the FDA should require code escrow for all medical devices, and release code for any devices that are still in service after the manufacturer ends support.

                                                                                                              1. 7

                                                                                                                Code escrow is a good first step.

                                                                                                                But you need access to all the development tools like JTAG debuggers as well.

                                                                                                                Also, this all breaks down of the device firmware is locked down, and / or requires signed code.

                                                                                                                1. 3

                                                                                                                  Required signed code is fine - even good - but the key for signing code that runs inside your body should be your own private key.

                                                                                                              2. 18

                                                                                                                There’s nothing illegal or immoral about eating snake oil or trying to do surgery on yourself.

                                                                                                                Segfaulting my own cyberware is pretty scary, but somebody else segfaulting it and having no recourse is scarier. The acceptable middle ground is sufficient documentation that a professional is capable of fixing it, and an amateur is capable of hurting themselves, but hopefully wise enough to choose a professional. This is how medicine works on stock hardware already.

                                                                                                                1. 8

                                                                                                                  Requirungy something to be open source doesn’t imply that random people add random code.

                                                                                                                  People also tend to run custom versions of big open source projects. However I think you should have full access on software installed on your body and if you want to should be able to have it changed, just like body modification at large.

                                                                                                                  Will it be exploited by greedy companies? Certainly. But the same is true already for body modification and likely medicine at large. There’s always gonna be someone going as unethical as someone is legally allowed to to maximize profits.

                                                                                                                2. 1

                                                                                                                  there don’t seem to be many companies creating this kind of technology. Adding additional burdens on them doesn’t seem like a promising way to promote these systems.

                                                                                                                  1. 22

                                                                                                                    Medicine has never really been a place where market dynamics did either well or good.

                                                                                                                    1. 12

                                                                                                                      Yeah, this might not be a problem which the free market is capable of solving on its own. Good thing there are ways to fix that too.

                                                                                                                  1. 4

                                                                                                                    This is a nice explanation of sudo and its use of the setuid bit.

                                                                                                                    It’s worth noting that the setuid-bit feature is generally considered bad these days - it’s both essentially impossible to write secure setuid programs, and it greatly complicates other security properties of the system. Ideally, usage of setuid programs such as sudo would be replaced with IPC like s6-sudod or plain old sshd, and the setuid bit can be completely removed from modern Linux distributions.

                                                                                                                    1. 2

                                                                                                                      How could sshd replace usage of sudo and friends? My impression is part of the value of sudo is avoiding passing around credentials for root accounts (on top of everything else). Perhaps I’m missing some important detail there.

                                                                                                                      Though perhaps the general answer is “just don’t have sudo-using flows in your operations”, which is definitely a respectable position!

                                                                                                                      1. 1

                                                                                                                        My impression is part of the value of sudo is avoiding passing around credentials for root accounts (on top of everything else).

                                                                                                                        Plenty of ways to do that; you could configure ssh to only allow root logins from localhost; you could stick the pubkeys of users allowed to log into root in root’s authorized_keys; you could use a centralized authorization scheme like Kerberos and allow certain authenticated users to log in as root; you could do all of those combined, or other things too.

                                                                                                                    1. 7

                                                                                                                      What would be a Pythonic way to implement a singleton-ish pattern?

                                                                                                                      class Borg:
                                                                                                                          _shared_ = {}
                                                                                                                          def __init__(self):
                                                                                                                              self.__dict__ = self._shared
                                                                                                                      

                                                                                                                      You may already know this, but that’s not the usual Python way to implement a singleton - usually a singleton is implemented by overriding __new__. This Borg is much less singleton-y because two Borg instances won’t compare as equal.

                                                                                                                      Also, singletons are bad http://www.object-oriented-security.org/lets-argue/singletons

                                                                                                                      1. 6

                                                                                                                        There are cases where you need something like a singleton, but the right way to do that in Python is just to have one named instance in a module, and its “singleness” comes from the fact that you are importing foo from bar, and imports are a shared global namespace.

                                                                                                                        1. 3

                                                                                                                          The enum module in the standard library is one example where that approach to singletons does not work and you need to override __new__.

                                                                                                                        2. 2

                                                                                                                          You’re absolutely right: Two borgs don’t compare as equal (using id(borg_1) == id(borg_2)) but everything inside is the same.

                                                                                                                          My point was not to make a case for singletons (or borgs) as I outlined in the article:

                                                                                                                          Note that you should use singleton classes only occasionally because they introduce some global state in your program, which makes it hard to test individual components of your program in isolation.

                                                                                                                          Instead, I wanted to illustrate how Python manages attributes.

                                                                                                                        1. 6

                                                                                                                          Remembering to pass that flag each time you run git blame is a bit too much, but you can set a global ignoreRevsFile in your config. By naming the file .git-blame-ignore-revs consistently, it will be used to skip any useless commits in a repo.

                                                                                                                          git config –global blame.ignoreRevsFile .git-blame-ignore-revs

                                                                                                                          Seems like it might be better to not pass –global, and just configure the ignoreRevsFile on a per-repository basis.

                                                                                                                          1. 4

                                                                                                                            The benefit of global is that it automatically works for all repos following the convention. Git doesn’t raise any warning if the file doesn’t exist.

                                                                                                                            1. 1
                                                                                                                              % git --version
                                                                                                                              git version 2.35.1
                                                                                                                              % git blame mime.types
                                                                                                                              fatal: could not open object name list: .git-blame-ignore-revs
                                                                                                                              

                                                                                                                              I just had to revert setting it globally because, at least in the current git release, it fatals out instead of performing the blame operation.

                                                                                                                              1. 1

                                                                                                                                I can swear I tested this before, but it appears that it is in fact fatal. I guess I don’t blame from the CLI often.

                                                                                                                                I found a thread discussing making it optional appears not to have gone anywhere: https://public-inbox.org/git/xmqq5ywehb69.fsf@gitster.g/T/

                                                                                                                          1. 7

                                                                                                                            This is an interesting article!

                                                                                                                            It seems like the “overshadowing” problem would be much better solved as a kernel patch. I doubt that many people desire that behavior, so a new sockopt would be quite useful.