Threads for yossarian
-
these results present a strong case against attempting to “rehabilitate” PGP signatures for PyPI, or any other packaging ecosystem
Is there already an alternative in the works? I don’t know much about this space. I hear about this a lot: https://www.sigstore.dev/
-
The quiet part of Sigstore that’s buried in the docs:
Currently, you can authenticate with Google, GitHub, or Microsoft, which will associate your identity with a short-lived signing key.
Meaning, you cannot participate in this ecosystem as a signer without an account on one of these privacy-invasive, proprietary services. It wouldn’t surprise me if you can’t ever self-host either given that they seem to intend to have a list of OIDC IdP inclusion criteria that new IdPs will have to meet before being manually included.
-
That list is out of date: BuildKite is also supported as an IdP, and (IIRC) GitLab is either supported or close to being supported.
The goal is to support high-quality IdPs that meet Sigstore’s claim and integrity requirements, which necessarily excludes random one-off IdPs run from someone’s home server. That doesn’t make them inherently privacy-compromising, however: there’s no reason why a community-hosted IdP couldn’t meet these requirements, and several of the IdPs currently supported (GitHub and BuildKite at the minimum) aren’t tied to any human identity in the first place.
-
Right, OK, I guess this makes sense - if the entire security of the system rests on the certificate issued by the OIDC authentication process, you want to make sure that authentication process is good. It still makes me uncomfortable though, for a lot of reasons.
Also, doesn’t that mean that anyone with legal authority over the signing IdP can compel the issuance of valid signatures on malicious software? My understanding of Sigstore is pretty hazy (the OIDC stuff I’m only aware of because I was in a talk last year about it) so I could simply be misunderstanding, but that seems like a pretty bad threat model, particularly in a world where governments are increasingly moving to compel software developers to insert backdoors. My understanding is that Sigstore signatures are publicly auditable using append-only ledgers a la Certificate Transparency, but this is still… unideal. (Maybe that’s unfair of me though because the current status quo of individual developers carrying signing keys is also subject to government compulsion, unless developers are willing to go to jail, and isn’t even publicly discoverable?)
-
Two points:
-
Sigstore is a public signing system: that means that anybody who can obtain an identity can sign for anything that they have the input for. That means that malicious people can make legitimate looking signatures (for some identity they control) for malicious things via legitimate IdPs; the scheme doesn’t attempt to enforce that only “good” people can sign. In this sense, Sigstore is very similar to Web PKI, and in particular very similar to Let’s Encrypt. The notion of trusted parties comes at the index or endpoint layers, via TUF, a TOFU setup, or something else[^1].
-
Transparency logs are indeed Sigstore’s main source of auditability, and the primary line of defense against CA compromise. I think there are a lot of legitimate reasons to think this is non-ideal – I think it’s non-ideal! But it’s also a pragmatic decision: CT has been shown to work for the Web PKI, and its distributed auditability has nice “knock-on” qualities (e.g. Firefox not needing to do privacy-compromising CT lookups because other large players do so).
Ultimately, I think it’s correct to point out that Sigstore’s adversarial model doesn’t include developers being forced into inserting backdoors into their own software. I think that’s correct to point out, because no codesigning scheme that I’m aware of can address that. What Sigstore does do is eliminate those adversaries’ stealth factor: if an identity is known to be compromised, the community is able to immediately see everything it signed for.
[^1]: If that sounds vague, it’s because it unfortunately is – it’s an area of active development within Sigstore, and in particular it’s something I’m working on.
-
In this sense, Sigstore is very similar to Web PKI, and in particular very similar to Let’s Encrypt. The notion of trusted parties comes at the index or endpoint layers, via TUF, a TOFU setup, or something else[^1].
Sure, this makes sense to me. It’s also similar to the status quo; given an arbitrary PGP key signing some (possibly tampered-with) software, you don’t know whether that’s the PGP key of the legitimate author of that software or not.
Ultimately, I think it’s correct to point out that Sigstore’s adversarial model doesn’t include developers being forced into inserting backdoors into their own software. I think that’s correct to point out, because no codesigning scheme that I’m aware of can address that. What Sigstore does do is eliminate those adversaries’ stealth factor: if an identity is known to be compromised, the community is able to immediately see everything it signed for.
I’m uncomfortable with Sigstore’s model but I’m having trouble thinking through exactly why, so I apologize for being unclear/making silly points while I essentially think out loud on the internet - this apology goes for this comment and my grandparent comment, lol. :P
But I think actually what I was trying to get at in the grandparent comment is that this is increasing the number of entities that have to be trusted not to be compromised (whether by compulsion or straight up hacking) - you’re adding the IdP but not subtracting the individual developer. However, it is indeed decreasing the amount of trust you have to place in these entities, because of the transparency logs. I’m not sure whether I personally agree with this although I think I’m starting to warm up to it; essentially you’re trading off needing to trust this extra entity for the ability to theoretically catch any entity making bad signatures and serving them to a subset of users, plus the fact that this system is actually usable and PGP isn’t. (Transparency logs aren’t a major advantage unless the attack is targeted because even under PGP, if the malicious signature is visible globally then there’s your answer - the signing key is compromised.)
-
-
-
-
-
-
Thank you for that link. Will there be any alternative for people who do not want to be vulnerable to certain supply chain attacks this doesn’t prevent and do not want to be vulnerable to third parties?
-
-
-
@yossarian FWIW the first link goes to 7.7 “what’s DSA”, I’d assume it was meant to go to 8.2 “how large should my key be?”.
-
That was intentional! The end of the DSA section claims that DSA is “well-regarded,” which it absolutely is not.
(The guidance around RSA 2048 is subject to some debate – I lean on the side of thinking that RSA 2048 is insufficient for signatures that are expected to last beyond 2030, but others believe that RSA 2048’s security margins are sufficient until RSA itself is more generally broken.)
-
-
PGP is an insecure and outdated ecosystem that hasn’t reflected cryptographic best practices in decades.
i feel like this is needlessly inflammatory.
it’s absolutely true that tools like GnuPG have insecure defaults and painful UX, but OpenPGP-the-ecosystem is alive and well.
IMO this is mostly thanks to the Sequoia PGP folks, but that work has been bearing fruit for years at this point.
-
It’s inflammatory, but it’s also not remotely controversial in cryptographic circles:
- https://blog.cryptographyengineering.com/2014/08/13/whats-matter-with-pgp/
- https://latacora.micro.blog/2019/07/16/the-pgp-problem.html
- https://words.filippo.io/giving-up-on-long-term-pgp/
The OpenPGP ecosystem is absolutely not alive and well, if alive and well includes the things that the ecosystem has historically maintained (synchronizing keyservers, a web of trust, etc.). The existence of a memory safe implementation is good, but does very little to address the basic format and cryptographic problems that are baked into OpenPGP’s standards. Fixing those requires standards changes, at which point we’re better off with something without the baggage.
-
when i say “alive and well” i mean that the core & associated technologies are being actively developed and there is a healthy ecosystem with strong support for existing standards and tools.
SKS has (effectively) been completely deprecated by the community in favor of https://keys.openpgp.org; i don’t use web of trust at all and have no strong opinions on it.
competing technologies like
ageare convenient but i have little confidence that they’ll ever see the same degree of support that OpenPGP has in software (e.g.git) or hardware (e.g. Yubikey, Nitrokey).EDIT: i feel like it’d be a little too long-winded to respond to all of those blog posts in a single comment, but just to pick on Matt Green: his criticisms of PGP are a little silly to apply here because he seems to be speaking mostly from the perspective of using it to secure communication (e.g. email and chat).
perfect forward secrecy doesn’t really make sense in the context of OpenPGP when you have other tools for communication that implement cryptographic protocols designed for that purpose.
-
[Matt Green’s] criticisms of PGP are a little silly to apply here because he seems to be speaking mostly from the perspective of using it to secure communication (e.g. email and chat).
To a layman like me, PGP’s primary use case seems to be secure communication (specifically email). So PGP isn’t a good tool for this use case then?
-
it depends entirely on what your threat model is; in its current state i wouldn’t recommend PGP for communication to a layperson, but for a savvy individual with specific threat models & use-cases it is still a best-in-class tool.
for the average software developer/maintainer, however, PGP is probably most useful for authentication & signing (i.e. cryptographic identity) + as a primitive that other tools (with better UIs) can use for encryption operations.
for authentication: i loaded up my PGP keys onto several Yubikeys and use them as my SSH credentials. between that & Secure Enclave on my mobile devices, i have almost completely done away with private keys on durable storage.
for signing: one can use PGP to verify
gitcommits (although this can be done with an SSH key now, not all forges support it).for encryption: PGP sucks to use directly but is fantastic in conjunction with tools like mozilla/sops &
pass(orgopass) for sharing development secrets in a project without relying on 3rd-party infrastructure.
-
-
-
-
The controversial part is that all of these people you linked imply that we should be vulnerable to a few centralized third parties.
-
The criticisms of PGP are not rooted in its lack of centralized ownership. They’re rooted in what a mess it is, both in terms of the user experience and the actual cryptography.
-
I get the impression that the political climate (for want of a better term) has changed in the security community. It used to be heavily invested in privacy, decentralization, and open platforms, and PGP ’s design reflects those values.
The new established wisdom is that centralization is good, open platforms are bad, and multinational corporations are seen as the most competent guardians of consumer privacy.
The arguments against PGP (including the author’s) all read as a disagreement about values, in the guise of a discussion about technical competence.
-
-
There are ways to use PGP that are kinda reasonably secure, and ways to use PGP that are interoperable.
Unfortunately, the ways that are secure are not interoperable, and the ways that are interoperable are not secure. Plenty of critiques, including the ones already linked, cover this in detail – if you want a more-secure setup in PGP, for example, you have to either do it in an interoperable/compatible way which requires letting other parties strip off the extra-security bits, or do it in a way that doesn’t allow stripping off the extra-security bits but is as a result non-compatible/non-interoperable with likely large numbers of PGP users’ configurations.
-
TLS once faced similar issues, but an effort was made to fix it, gradually breaking compatibility with the insecure setups, despite millions of users worldwide being on old, outdated operating systems and browsers, without the ability to update, or even the desire to do so.
PGP’s installed user base is orders of magnitude smaller, technically savvy, and/or heavily invested in security.
-
PGP’s installed user base is orders of magnitude smaller, technically savvy, and/or heavily invested in security.
Unfortunately, I think the developers of PGP implementations and tooling are much more invested in defending the idea that PGP as-is has no security problems that would need to be fixed by ripping out compatibility with ancient crypto algorithms. And even doing that doesn’t really fix all the potential problems with PGP’s design; like a lot of people have said, the preferable well-designed approach is to support one way to do things, and if it gets broken then you increment the protocol version and switch to a different single way of doing things.
-
I’m not so sure. TLS and Signal have the advantage that they deal with ephemeral data. Software signatures have a far longer lifetime (and in fact, most of the authors’ criticisms are related to the signatures being old). I think it’s very easy to get into a situation where you’re supporting multiple protocol versions at the same time, (as for example PASETO does) effectively ending up in the same place.
-
-
-
-
-
-
-
This is a pretty misleading title. They don’t do any curation nor additional checks for publishers. This is simply “Trusted Publishing”, akin to SSO in company setting.
-
As Peter Neumann frequently points out, there is a big difference between ‘trusted’ (you depend on this for core security guarantees) and ‘trustworthy’ (this thing will not compromise your core security guarantees). This seems to be correctly named.
-
I was not commenting on the word “trusted”, but what comes after it. People publish their packages to pypi, thus, “trusted publishers”, in my eyes, refers to those who publish. Yet, what the article describes is more about the process of publishing, not about who does the publishing.
-
In this context, the entity doing the publishing (the “publisher”) is a GitHub Action. Normally, you’d establish that action’s permission to publish by giving it a manually created PyPI token; with “trusted publishers,” the publisher is already trusted to mint a short-lived token against its OIDC identity.
-
-
-
-
yossarian
3 months ago
|
link
| on:
C-rusted: The Advantages of Rust, in C, without the Disadvantages
See also Checked C1, which addresses many of the same vulnerability and bug classes.
As a bit of a rant: I wish academics wouldn’t say things like “the advantages of Rust” when describing these safer C subsets. These are admirable research areas to pursue, but they all lack the primary (IMO) advantage that languages like Rust have: that security is established on a program level by construction, rather than cobbling together individually verified components and hoping that invariants are not broken in an unchecked component. It would be fairer (and more useful!) to say “a subset of C without a subset of spatial memory safety bug classes.”
-
Or even “cobbling together individually verified components and hoping that invariants are not broken in the ‘cobbling together’ process”, as one of the big problems with ‘C the language’ for this stuff is how composition does not preserve the kinds of invariants we are interested in, so even exclusively using individually verified components doesn’t save you.
-
I don’t think your description is accurate. I’ve just read the paper, and my understanding is that those checks do compose. A checked function also keeps tracks of the functions it calls, and if the prototype of the affected functions is not properly annotated you get a warning. In addition, the tool could (maybe does) issue a list of all unchecked functions. We would know which parts need extra scrutiny, same as Rust’s
unsafe.The only real problem here is the same as
constand gradual typing: it’s unsafe by default.-
-
I’ve worked in teams where “Thou shalt obey the Holy Code code Formatter” were religiously enforced. We could likewise enforce that every piece of new code must be annotated, unless you can write a justification that has to be vetted by Q/A or whatever. Considering that the benefits of gradual typing & annotations are both indisputable and substantial (especially at scale), the social component should be relatively easy to assert.
Don’t get me wrong, I still prefer a language that’s safe by default, where the escape hatches read
unsafe_dangerous_hazardin blinking blood letters. -
-
-
They check that each function is internally consistent and can validate that a call graph contains only checked functions. However, they also explicitly flag their analysis as intraprocedural and context-insensitive, meaning that it won’t track invariants that are managed between functions or in visited loop states.
In other words, using this as intended will give you a false sense of security: the paper encourages you to treat these macros as proving properties similar to “safe” Rust, when they really only prove that a function without interaction doesn’t violate its invariants.
-
-
That part is the same as Rust! The difference that allows Rust to get away with it is aliasing and ownership/lifetime semantics: you don’t need to do a heavy interprocedural analysis if you know that writes are exclusive and that every binding has a known lifetime.
It looks like C-rusted can do a subset of that, since its macros allow exclusive and shared ownership. But it’s unclear to me how they’d prevent an interprocedural UAF through an owned pointer, for example, or how they’d enforce the liveness of a shared pointer. Maybe they do that by providing their own annotations on top of
mallocandfreeand other resource allocation primitives, or maybe they expect the user to do that.-
What’s “UAF”?
-
-
Okay, that’s the usual choice between safe and complete type systems: the only way to reject all incorrect programs is to reject some correct ones. That’s why Rust needs
unsafe. it’s okay to reject too many programs, as long as we have a well defined escape hatch. Likewise, we almost certainly can’t just annotate all C programs and make them fully C-Rusted.Use-after-free is one of the problems Rust solves, and C-Rusted is (as far as I can tell) trying to address. I believe both face the same safe/complete dilemma, I’m not sure why you feel Rust has to fare any better in this particular department. Specifically I don’t know what you mean by “total” lifetime information, nor why C-Rusted wouldn’t have that. The C-Rusted model seems pretty simple as far as I understand:
- Shared handles are like immutable borrows.
- Exclusive handles are like mutable borrows.
- Owning handles end the lifetime of the handled resource. I don’t know the Rust equivalent.
So when you call a function and give it some handle, and the annotation of the function says it takes ownership, the caller considers the resource “dead” right after the function call. At least that’s how I understood the
process()example in the paper.Another thing that could happen is that a function you call gives you an owning handle. Just like
malloc(). I guess you must then convince the checker that you either free that handle before returning to the caller (by calling a function likeprocess()orfree()), or return that handle to the caller one way or another. I’d wager the same is true with Rust too.I bet we can also forward an owning handle from input argument to return value, but my guess here is that from the caller’s perspective, the handle they passed on and the handle they get are unrelated. I don’t think we can do better with intra-procedural analysis. Besides, why would we forward references like that? Such functions should take an ordinary exclusive handle/mutable borrow instead, that’d make liveness analysis easier.
It looks like C-rusted can do a subset of that, since its macros allow exclusive and shared ownership.
Wait, what you call “shared” sounds like
std::shared_ptr, while the paper uses the same word to describe immutable borrows.To be honest I have no idea how I might implement
std::shared_ptrin a way that would satisfy a borrow checker — without usingunsafe. I mean, the very point of a shared pointer is to split ownership, and as far as I understand that breaks the linearity I believe is necessary to borrow checkers, so I’d wager you need some of your code to beunsafe. (If you have a reference to Rust safe-only implementation of something like astd::shared_ptr, that means I’m dead wrong, and I’d be highly interested in correcting that error.)-
Owning handles end the lifetime of the handled resource. I don’t know the Rust equivalent.
Rust refers to any value that you hold directly (as opposed to through a reference) as “owned”, and they have similar semantics to what you describe: destructors, if any, are automatically invoked when they go out of scope. If you want an “owned pointer”, that’s
std::boxed::Box, similar to C++’sstd::unique_ptr.Other types, like
VecandString, also act as “owning pointers” to a block of memory;Boxis implemented via black magic for historical reasons, but fundamentally all these types could just be structs with destructors that callfree.To be honest I have no idea how I might implement std::shared_ptr in a way that wou ]ld satisfy a borrow checker — without using unsafe. I mean, the very point of a shared pointer is to split ownership, and as far as I understand that breaks the linearity I believe is necessary to borrow checkers, so I’d wager you need some of your code to be unsafe. (If you have a reference to Rust safe-only implementation of something like a std::shared_ptr, that means I’m dead wrong, and I’d be highly interested in correcting that error.)
Rust’s equivalents here are
std::rc::Rcandstd::sync::Arc(the latter uses an atomic reference count and thus is sharable across threads) and they do use unsafe internally.I believe both face the same safe/complete dilemma, I’m not sure why you feel Rust has to fare any better in this particular department.
It’s hard to tell from the paper, but as a broader comment: most “Rust replacements” I’ve looked at fall short in one of three ways.
- The “safe” subset isn’t fully safe, so you just have two subsets that allow incorrect programs.
- There’s no clear syntactical delineation between the safe and unsafe subsets; you need an encyclopedic knowledge of the syntax and standard library to recognize potentially unsafe code.
- The safe subset is insufficiently expressive to write meaningful programs in.
(To be clear, there are languages I like that fall short on all the above points! But I get annoyed when people bill them as “just as safe as Rust.”)
-
-
-
-
-
-
-
-
Signing binaries and relying on a CA to establish “trust” is already a lost cause.
Did notarization significantly improved the OSX security ? To me it seems it was only a roadblock for OSS software. On Windows there was malware signed with leaked developer certificates.
-
These are pretty different ecosystems: with end-user software signing, you’re relying on intermediate issuers (or, in macOS’s case, just Apple) to only issue certificates to “legitimate” entities. Sigstore is two or three levels below that: it’s intended to help engineers sign packages and distributions that larger programs and systems are made out of. Signing and verification are simpler and easier to automate in that context.
-
-
This 1. Assumes that the integration with the email client works 2. A user can follow instructions beyond clicking a link.
-
It’s also not clear what you’d do in the case where the DMARC policy is not sufficiently strong for you to be able to guarantee that the email is not spoofed. If someone who runs their own mail server hits this, they can fix it. If someone who doesn’t run their own server hits it then they’re stuck. Note that, in addition to the existing constraints, the server should also require DNSSEC. This isn’t a requirement for DMARC normally but this is changing the threat model. Someone who can tamper with a DNS response can tamper with the DMARC keys. Normally, this just causes emails to be rejected or some spam to get through, but with this model a DNS spoofing attack can compromise an account.
It’s a huge shame that XMPP never caught on. Almost 20 years ago, I built a proof-of-concept that used XMPP for login, where you’d get an XMPP message containing a URL and clicking on the URL would log you in. The message was sent as a normal message but, because XMPP messages can contain arbitrary XML, also included metadata saying that this is what it is. I modified a client to allow you to automatically fetch the URL for senders that you put on an allow list, so you could automatically log in while your client was running. The biggest downside of this was that it gave the server your JID, which could be used for cross-site tracking (it’s much easier with email to generate throw-away addresses than it is with XMPP).
-
Right, which is why the scheme proposed in the post doesn’t use DMARC – it essentially ignores DMARC entirely and mandates both SPF and DKIM. Those in turn could be misconfigured, but you could do additional normative filtering from there (e.g. rejecting messages from hosts who allow any IP to send mail).
This admittedly makes the scheme much, much less generally useful :-)
-
Why bother with something broken like SPF when DKIM exists?
-
-
You mean RCPT FROM? Why does that matter? That value is invisible to most mail software anyway.
-
-
-
-
-
-
I agree that it makes assumption (1), but I don’t think it assumes (2) in a meaningful sense – email verification as a whole is based on the latent assumption that the user has an email account and knows how to use it, meaning that the scheme only requires them to understand that they need to click the “send” button.
-
but it also doesn’t resolve your issue of “I have a work, personal, and secret e-mail account/browser/whatever”. I can have gmail in chrome, work email in outlook and all my other mailboxes in thunderbird.
For “muggles” the regular process is the same anyway, they will usually not be logged in two-three sessions per browsers, two-three browsers per desktop session. For e.g. devs like us, the problem is similar just reversed - when I click on a mailto link, what does it get opened in?
-
-
-
DKIM only guarantees that the email was sent from an authorised server, it doesn’t ensure that the headers or body are genuine. I have sent email from my Gmail account with a From: header of larry.page@gmail.com. I don’t know if they still allow that, but I would be unsurprised if other large mail servers allow it.
This scheme also relies on correct use of DKIM by mail servers, which is fine for some of the major players, but there is a long tail of institutional email servers, etc where this will not be true.
Finally, the scheme requires a fair bit more complexity from the implementor because receiving and validating incoming email is quite a bit more complicated than sending mail.
-
I might be misunderstanding, but I think you’re mixing up SPF and DKIM: DKIM ensures that the body and a subset of the headers are authentic; SPF ensures (non-cryptographically) that the message envelope is authentic. The scheme I proposed in the post requires both.
I agree about the “long tail,” however! I have no idea how practical this actually is; it just popped into my head.
-
SPF and DKIM don’t allow authenticating the sender, they allow authenticating the sender’s mail server. If the mail server is properly configured then it will prevent anyone from sending mail as an account that is not theirs but some do.
There’s a bigger problem though: DKIM signs the envelope from, which is set by the server, but this often doesn’t match the email address that the user sets (for example, before they moved to M365, my Cambridge email address had an envelope from of {my unique ID}@hermes.cam.ac.uk, whereas my email address was {my name}@cl.cam.ac.uk. If I’d entered the latter on your web site, you’d have received a confirmation email that would have looked fake. You’d need to do an additional round trip to ensure that I could receive an email from the sent account.
You might be able to use the normal flow for the initial account creation step and this for normal login though: If I receive an email and, rather than clicking on a link, reply to it, then you will know the from header and the envelope from field that are associated with the account and so can then perform your reversed flow for a subsequent login.
Note that, for this to be responsive and reliable, the mail server processing the inbound login emails can’t use any kind of DNSRBL or greytraping. This means that it will end up processing a pretty huge volume of spam. That’s probably not a huge problem (especially if you only care about the subject line in the emails that you receive and can discard the rest).
-
You’re right that I was mixing them up, tho DKIM doesn’t do enough to guarantee what you want: DKIM specifies some metadata and signs it plus the body and some headers of the mail. That can optionally include the “agent identity” and/or the “From:” header. The sending server may choose not to include either, or may choose to sign them but only validate the domain part (the DMARC “domain alignment” policy only looks at the domain part); or it may choose to sign without validating at all.
-
-
-
This parse ambiguity is similar to the ambiguity exploited by the recently-discussed rubygems CVE-2022-29176, though the article reports discovering the Python issue independently. The problem in both cases is looking up a package by the concatenated string
{name}-{version}instead of by the name and the version separately.-
Yeah, these kinds of parsing ambiguities can be extremely pernicious! As best I can tell no modern Python packaging tools will be confused by the “vexing parse” here, but it’s possible that older tooling could be confused into installing
package-someversion==suffixinstead ofpackage==someversion-suffix.
-
-
This is an awesome series. I hadn’t considered interpreting LLVM bitcode before but I’m aware sulong does just that so now you’ve inspired me to give it a shot sometime too.
-
-
lorddimwit
edited
1 year ago
|
link
| on:
In C, how do you know if the dynamic allocation succeeded?
This isn’t a C problem, it’s an operating system problem.
In fact, I’d even argue that, on systems that use this sort of allocation scheme, they’re not in keeping with the C standard, which states that a successful return from
malloc(or related functions) yields a pointer that can be used to access objects or arrays of any type up to the specified size. Crashing on a valid return is not a specified error condition.-
This is a pedantic response, but the pedantic answer is important for why this behavior isn’t incompatible with the C standard:
mallocdoesn’t cause a program to crash when an overcommit is realized. It causes the kernel to abort the program. The latter is perfectly acceptable in the C/C++ memory model.Put another way: from C’s perspective, there’s no real sense in which the
mallocincorrectly fails. It succeeds, and the system decides not to play along.
-
-
Really cool!!!
My aversion to databases in personal projects is largely unfounded: turning the DOCUMERICA dataset into a small SQLite DB gave me exactly the kind of query and state control that I needed. No more hacky state files, unlike my other Twitter bots.
I found exactly the same recently! Databases are cool and I regret not touching them for ten years :)
A small nitpick:
created TEXT, -- the ISO8301 date for the photo's creationIt’s ISO 8601 :)
-
-
I’m working on a third post in my LLVM internals series.
-
This is my opinion as just another Rust programmer in the community, but: encouraging novices to opt into runtime lifetime management and interior mutability sets them up for failure when they eventually have to interact with the majority of the Rust ecosystem. They’re also a performance and abstraction hazard: a novice who leans too heavily on interior mutability will end up designing interfaces that don’t match impedance with the rest of the crate ecosystem.
These techniques are vital to Rust’s ability to express complex memory semantics, but they’re also a barrier to fully understanding the language when you’re a novice. I suspect that the advice in this post would have been a significant impediment to my personal understanding of the language when I was just starting out.
-
I’m continuing to work an a (partial) reimplementation of LLVM’s IR APIs in Rust: https://github.com/woodruffw/mollusc
I’ve been blogging about the different interesting bits of writing a bitcode parser from the ground up here: https://blog.yossarian.net/series#llvm-internals
-
This is a well-written post, but I think I disagree with the author w.r.t. this actually being a problem: effects within closures and iterators are subtle things, and I prefer Rust’s approach of requiring users to explicitly ferry them into the surrounding context.
It also tends to be only a minor hiccup in practice:
collectcan be used to flatten a sequence ofResult<T>s intoResult<Container<T>>, which is the intuitive and intended behavior in almost all circumstances.-
-
I know this thread is long dead, but you actually can do this with
join_all.Or you can use
FuturesUnorderedto pull out results as they complete. Unfortunately it’s a little weird to use: iterating returns a Future to await, which in turn returnsSome(_)orNoneto signal end of iteration. AnIteratorneeds to returnOption, not aFuture<Option>.let mut results: FuturesUnordered<_> = ... while let Some(result) = results.next().await { ... }
-
-
-
I think there are valid arguments on both sides here, but this post doesn’t seem to be grounded in experience.
Practically speaking, users of weird architectures do contribute patches back. Those people eventually become the maintainers. When those people go away, the project drops support for certain architectures. That happened with CPython, e.g. it doesn’t support Mac OS 9 anymore as far as I remember.
It’s sort of a self-fulfilling prophesy – if the code is in C, you will get people who try to compile it for unusual platforms. If it’s in Rust, they won’t be able to try.
I’m not saying which one is better, just that this post misses the point. If you want to use Rust and close off certain options, that’s fine. Those options might not be important to the project. Someone else can start a different project with the goal of portability to more architectures.
Changing languages in the middle of the project is a slightly different case. But that’s why the right to fork exists.
-
Author here: this post is grounded in a couple of years of experience as a packager, and a couple more years doing compiler engineering (mostly C and C++).
Practically speaking, users of weird architectures do contribute patches back. Those people eventually become the maintainers. When those people go away, the project drops support for certain architectures. That happened with CPython, e.g. it doesn’t support Mac OS 9 anymore as far as I remember.
This is the “hobbyist” group mentioned in the post. They do a fantastic job getting complex projects working for their purposes, and their work is critically undervalued. But the assumptions that stem from that work are also dangerous and unfounded: that C has any sort of “compiled is correct” contract, and that you can move larger, critical work to novel architectures just by patching bugs as they pop up.
-
OK I think I see your point now. TBH the post was a little hard to read.
Yes, the people contributing back patches often have a “it works on my machine” attitude. And if it starts “working for others”, the expectation of support can arise.
And those low quality patches could have security problems and tarnish the reputation of the project.
So I would say that there are some projects where having the “weird architectures” off to the side is a good thing, and some where it could be a bad thing. That is valid but I didn’t really get it from the post.
I also take issue with the “no such thing as cross platform C”. I would say it’s very hard to write cross platform C, but it definitely exists. sqlite and Lua are pretty good examples from what I can see.
After hacking on CPython, I was surprised at how much it diverged from that. There are a lot of #ifdefs in CPython making it largely unportable C.
In the ideal world you would have portable C in most files and unportable C in other files. Patches for random architectures should be limited to the latter.
In other words, separate computation from I/O. The computation is very portable; I/O tends to be very unportable. Again, sqlite and Lua are good examples – they are parameterized by I/O (and even memory allocators). They don’t hard-code dependencies, so they’re more portable. They use dependency inversion.
-
TBH the post was a little hard to read.
That’s very fair; I’m not particularly happy with how it came out :-)
I also take issue with the “no such thing as cross platform C”. I would say it’s very hard to write cross platform C, but it definitely exists. sqlite and Lua are pretty good examples from what I can see.
I’ve heard this argument before, and I think it’s true in one important sense: C has a whole bunch of mechanisms for making it easy to get your code compiling on different platforms. OTOH, to your observation about computation being generally portable: I think this is less true than C programmers generally take for granted. A lot of C is implicitly dependent on memory models that happen to be shared by the overwhelming majority of today’s commercial CPUs; a lot of primitive operations in C are under-specified in the interest of embedded domains.
Maybe it’s possible to truly cross-platform C, but it’s my current suspicion that there’s no to verify that for any given program (even shining examples of portability like sqlite). But I admit that that’s moving the goalposts a bit :-)
-
Maybe it’s possible to truly cross-platform C, but it’s my current suspicion that there’s no to verify that for any given program (even shining examples of portability like sqlite).
I think the argument holds up just fine despite the existence of counterexamples like Sqlite and Lua; basically it means that every attempt to write portable and safe code in C can be interpreted as an assertion that the author (and every future contributor) is as capable and experienced as Dr. Hipp!
-
A lot of C is implicitly dependent on memory models that happen to be shared by the overwhelming majority of today’s commercial CPUs
That’s largely a result of the CPU vendors optimising for C, due to its popularity. Which leads to its popularity. Which…
-
A lot of C is implicitly dependent on memory models that happen to be shared by the overwhelming majority of today’s commercial CPUs; a lot of primitive operations in C are under-specified in the interest of embedded domains.
As the author of a C library, I can confirm that fully portable C is possible (I target the intersection of C99 and C++). It wasn’t always easy, but I managed to root out all undefined and unspecified behaviour. All that is left is one instance of implementation defined behaviour: right shift of negative integers. Which I have decided is not a problem, because I don’t know a single platform in current use that doesn’t propagate the sign bit in this case.
The flip side is that I don’t do any I/O, which prevents me from directly accessing the system’s RNG.
Incidentally, I’m a firm believer in the separation of computation and I/O. In practice, I/O makes a relatively small portion of programs. Clearly separating it from the rest turns the majority of the program into “pure computation”, which (i) can be portable, and (ii) is much easier to test than I/O.
-
-
I also take issue with the “no such thing as cross platform C”. I would say it’s very hard to write cross platform C, but it definitely exists. sqlite and Lua are pretty good examples from what I can see.
I see this as parallel to “no such thing as memory-safe C”. Sure, cross-platform C exists in theory, but it’s vanishingly rare in practice, and I’d wager even the examples you cite are likely to have niche platform incompatibilities that haven’t been discovered yet.
-
I’d wager even the examples you cite are likely to have niche platform incompatibilities that haven’t been discovered yet.
Portability in C is hard, but it is simple: no undefined behaviour, no unspecified behaviour, no implementation defined behaviour. If you do that, and there are still are platform incompatibilities, then the platform’s compiler is at fault: it has a bug, fails to implement part of the standard, or simply conforms to the wrong standard (say, C89 where the code was C99).
If we’re confident a given project is free of undefined, unspecified, and implementation defined behaviour, then we can be confident we’ll never discover further niche platform incompatibilities. (Still, achieving such confidence is much harder than it has any right to be.)
-
Portability in C is hard, but it is simple: no undefined behaviour, no unspecified behaviour, no implementation defined behaviour.
That is a very tall order, though. Probably impossibly tall for many (most?) people. I asked how to do this and the answers I would say were mixed at best. Simple isn’t good enough if it’s so hard nobody can actually do it.
-
-
-
-
-
If it’s in Rust, they won’t be able to try.
I think this is the most trenchant point here. If someone wants to maintain a project for their own “weird” architecture, then they need to maintain the toolchain and the project. I’ve been in that position and it sucks. In fact, it’s worse, because they need to maintain the toolchain before they even get to the project.
I’m particularly sensitive to this because I’m typing this on ppc64le. We’re lucky that IBM did a lot of the work for us, but corporate interests shift. There’s no Rust compiler for half the systems in this room.
-
-
-
-
So does that mean we should only care about architectures that have corporate backing? Like I say, this isn’t a situation where it’s only a project port that needs maintainers. The OP puts it well that without a toolchain, they can’t even start on it. If Rust is going to replace C, then it should fill the same niche, not the same niche for systems “we like.”
For the record, my projects are all officially side projects; my day job has nothing officially to do with computing.
-
So does that mean we should only care about architectures that have corporate backing?
Yes, it does. Money talks. Open source is not sustainable without money. I can work on a pet project on the side on evenings and weekends only for a relatively short period of time. After that it’s going to go unmaintained until the next person comes along to pick it up. This is going to happen until someone gets a day job working on the project.
If Rust is going to replace C, then it should fill the same niche, not the same niche for systems “we like.”
C has a four-decade head start on Rust, if no one is allowed to use Rust until it’s caught up to those four decades of porting and standardization effort–for the sake of people’s side projects–then that argument is a non-starter.
-
Yes, it does. Money talks.
In such a world there would be no room for hobbyists, unless they work with what other people are using. Breakage of their interests would be meaningless and unimportant. That’s a non-starter too.
But, as you say, you’re unfamiliar with these systems, so as far as you’re concerned they shouldn’t matter, right?
-
-
-
-
OpenWrt runs on TP-Link TL-WDR4900 WiFi Router. This is a PowerPC system. OpenWrt is nearly a definition of hobbyists working on side projects using mass-market hardware.
-
-
-
-
-
-
-
One point: ARM instructions tend to fixed-width instructions (like UTF-32), vs x86 instructions tend to vary in size (like UTF-8). I always loved that.
I’m intrigued by the Apple Silicon chip, but I can’t give you any one reason it should perform as well as it does, except maybe smaller process size / higher transistor count. I am also curious how well the Rosetta 2 can JIT x86 to native instructions.
-
“Thumb-2 is a variable-length ISA. x86 is a bonkers-length ISA.” :)
-
-
If you really want to split your brain, look up the EPIC architecture on the 64-bit Itaniums. These were an implementation of VLIW (Very Long Instruction Word). In VLIW, you can just pass a huge instruction that tells what individual functional unit should do (essentially moving scheduling to the compiler). I think EPIC batched these in groups of three .. been I while since I read up on it.
-
interestingly by one definition of RISC, this kind of thing makes itanium a RISC machine: The compiler is expect to work out dependencies, functional units to use, etc which was one of the foundational concepts of risc in the beginning. At some point RISC came to mean just “fewer instructions”, “fixed length instructions”, and “no operations directly with memory”.
Honestly at this point I believe it is the latter that most universally distinguishes CISC and RISC at this point.
-
Raymond Chen also wrote a series about the Itanium.
https://devblogs.microsoft.com/oldnewthing/20150727-00/?p=90821
It explains a bit of the architecture behind it.
-
-
My (limited) understanding is that it’s not the instruction size as much as the fact that x86(-64) has piles of prefixes, weird special cases and outright ambiguous encodings. A more hardwarily inclined friend of mine once described the instruction decoding process to me as “you can never tell where an instruction boundary actually is, so just read a byte, try to figure out if you have a valid instruction, and if you don’t then read another byte and repeat”. Dunno if VAX is that pathological or not, but I’d expect most things that are actually designed rather than accreted to be better.
-
The VAX is “read byte, decode, read more if you have to”, but then, most architectures which don’t have fixed sized instructions are like that. The VAX is actually quite nice—each opcode is 1 byte, each operand is 1 to 6 bytes in size, up to 6 operands (most instructions take two operands). Every instruction supports all addressing modes (with the exception of destinations not accepting immediate mode for obvious reasons). The one instruction that can potentially take “all of memory” is the
CASEinstruction, which, yes, implements a jump table.
-
-
-
-
fixed-width instructions (like UTF-32)
Off-topic tangent from a former i18n engineer, which in no way disagrees with your comment: UTF-32 is indeed a fixed-width encoding of Unicode code points but sadly, that leads some people to believe that it is a fixed-width encoding of characters which it isn’t: a single character can be represented by a variable-length sequence of code points.
-
-
I can’t give you any one reason it should perform as well as it does, except maybe smaller process size / higher transistor count.
One big thing: Apple packs an incredible amount of L1D/L1I and L2 cache into their ARM CPUs. Modern x86 CPUs also have beefy caches, but Apple takes it to the next level. For comparison: the current Ryzen family has 32KB L1I and L1D caches for each core; Apple’s M1 has 192KB of L1I and 128KB of L1D. Each Ryzen core also gets 512KB of L2; Apple’s M1 has 12MB of L2 shared across the 4 “performance” cores and another 4MB shared across the 4 “efficiency” cores.
-
-
I’m not an expert but here are some thoughts on what might be going on. In short, the 4 KB minimum page size on x86 puts an upper limit on the number of cache rows you can have.
The calculation at the end is not right and I’d like to know exactly why. I’m pretty sure the A12 chip has 4-way associativity. Maybe the cache lookups are always aligned to 32 bits which is something I didn’t take into account.
-
-
For comparison: the current Ryzen family has 32KB L1I and L1D caches for each core; Apple’s M1 has 192KB of L1I and 128KB of L1D. Each Ryzen core also gets 512KB of L2; Apple’s M1 has 12MB of L2 shared across the 4 “performance” cores
This is somewhat incomplete. The 512KiB L2 on Ryzen is per core. Ryzen CPUs also have L3 cache that is shared by cores. E.g. the Ryzen 3700X has 16MiB L3 cache per core complex (32 MiB in total) and the 3900X has 64MiB in total (also 16MiB per core complex).
-
-
-
-
One small correction: _N is not a prefix.
-
Interesting take on authentication, however perhaps a bit too convoluted?
Let aside the fact it relies on JWT that has a bad reputation of being impossible to get right in implementation, but what caught my eye was how the verification (by PyPI for example) is being done: by applying certain policies over the verified claims in the received JWT.
Because the article doesn’t say, and doesn’t give any screenshots in case of PyPI, I can only wonder how this is actually achieved:
So, I’m left wondering if they haven’t just moved the problem from one place to another by introducing yet another attack surface (which besides JWT is now the policies)?
They have even identified one themselves – that of resurrecting dead or renamed accounts – and they’ve solved it with the intervention of GitHub, which means that this issue has to be solved by each individual provider.
I can’t believe we can’t find simpler approaches than this…
This was meant to be just a technical summary of the architecture. The link at the very top of the post goes to the the PyPI blog, which contains an announcement that’s more specific to the features available at launch-time; it also contains a link to the technical documentation1, which should address some of your questions around policies and security model.
To summarize:
Yes, JWTs at bad. They’re also, practically speaking, the only game in town. OAuth2 and OIDC use them, so we have to use them. There would be no trusted publishing at all without these underlying technologies.
At the moment, PyPI supports just one provider: GitHub. That provider was selected because it accounts for an overwhelming majority of publishing activity, meaning it’s the one that would be most immediately useful to potential users. I’m currently working on support for two more providers: Google’s IdP and GitLab.
It would be fantastic if we could do vendor neutral policy verification, but neither OAuth2 nor OIDC is rigid enough for us to assume a ‘baseline’ of shared relevant claims. Each IdP needs to be added manually, since each has its own claim set and quirks; eliminating that problem is a problem for standards bodies, not relying parties like PyPI.
The “policies” in question are just HTML form inputs for (in GitHub’s case) the username, repository name, workflow name, and an optional actions environment. You don’t have to know a DSL; that’s something we considered early on in design, and rejected for complexity reasons. There are examples of this in the docs (linked above).
Trusted publishing is indeed architecturally complex. But it’s using OIDC as it’s intended, and we’ve tried to abstract nearly all of this complexity from the end user (who only needs to know which
user/repoand workflow they trust). The surface that it exposes is roughly equivalent to granting an attacker access to your CI (including all secrets in it), so I don’t think that part is a significant new risk.I’m not trying to dismiss the work you have done with this integration. It is a first step towards a better solution for a quite hard to solve problem.
However, PyPI is now in the same league as Rust’s
crates.ioin that it requires a GitHub account to be able to “play” in Rust’s (and now Python’s) ecosystem. I don’t have anything against GitHub either, I use them for all my code, however there are developers (and certainly also amongst Python ones) that don’t want to have a GitHub account. Thus for this category of users the problem is still unsolved.However, assuming GitHub is only the first such identity provider, what are the next steps to support multiple independent providers? How would the UI change to support something that doesn’t have GitHub’s particularities? would a separate HTML UI (i.e. form with inputs) be developed for each?
Because each time such a proposal appears, that in theory supports multiple identity providers, it falls short and supports only the largest ones like Google / Facebook / Apple.
To summarize, I understand why PyPI (and
crates.iofor that matter) have chosen to start (and stop) with GitHub: at the moment it has a large user-base and it’s pretty secure (or at least with deeper pockets to throw money at the problem if needed); in a few words it’s a practical approach (that I would have taken myself). But I doubt there would be a time when the “trusted publishing” would expand to other providers…So, if GitHub is the only one, why bother with all the complexity?
GitLab is perhaps similar to GitHub, thus it might have the concepts of repositories, branches, etc. But would you support only the “official hosted” GitLab instance, or will you support independent GitLab deployments?
But what are the equivalents for Google? They’ve killed their
code.google.comlong ago. Are you targeting another of their products?This new feature, which offers an ability to publish from CI with a short-lived token in order to make CI compromises less severe, chose GitHub as the first supported IdP, not the only one that will ever be supported, and you’re literally replying to someone who says they’re working on adding support for others.
And if you don’t have or want a GitHub account, or object to using a GitHub account, or don’t want to publish from CI at all, you can still use plenty of other PyPI publishing workflows to put your packages onto PyPI, all the way down to just running
python -m build && python -m twine upload dist/*manually from your own personal machine. So claiming that PyPI now “requires a GitHub account” is false.I appreciate this concern!
I’ll state it unambiguously: trusted publishing is not intended to deprecate ordinary PyPI API tokens. Users will continue to be able to create API tokens, which can then be used however they like. Trusted publishing is only intended to provide a faster and more convenient alternative, where the platform (i.e. GitHub, and soon others) permit.
Support for independent providers is something that PyPI’s admins will probably consider on a case-by-case basis, as a balancing act between implementation complexity (each IdP needs special handling), the security margins of each new IdP (there’s relatively little value in a random self-hosted IdP versus a normal API token), and the impact to the community (e.g., major CI platforms).
Yes, sorry if this was unclear: in this context, the IdP being supported is the one that provides proofs for machine identities on Google Cloud Build. In other words, similar to GitHub Actions’ machine identities.