1. 25

  2. 8

    I’ve been looking for articles like this. It’s a good article but only covers the outages from one angle. I’d love to see a writeup on what products were affected and why. For instance a bunch of coffeeshops stopped being able to take credit cards because they were using old iPad-based POS terminals that couldn’t handle the certificate change. Things like that broke all over the world, would love to see an analysis.

    1. 7

      My smart TV (purchased in 2020!) no longer connects to my Plex server because of this.

      All it requires is a firmware update that includes the new root certificate from Let’s Encrypt, but we all know how companies are once they’ve got your money.

      1. 1

        It looks like Plex is serving the cross-signed chain. If they can fix that I’d guess that at least some of those devices will work again. I don’t have a device that’s affected, but I’ve tried modifying the chain in my Plex server, and it seems to work and not make anything worse.

        If you want to try that I’m happy to go into what I did. But the real solution would be for Plex to make that change themselves…

        1. 1

          It’s a bit late, have already switched over to Jellyfin because Plex have made it clear they’re not going to do anything to improve the situation.

    2. 4

      Maybe I’m just missing it, but this writeup seems to gloss over the main event, at least as viewed through my filter bubble.

      The problem comes back to a similar concern around the client being outdated in some way and not having the new ISRG Root X1 installed, meaning it can no longer validate certificate chains as it has no Root CA to anchor on.

      The outdated clients (some not outdated by very much; GnuTLS was only patched in June 2020) in many cases did have ISRG Root X1 installed, but ignored it because they preferred the cross-signed version of ISRG Root X1 sent by the server. Removing the cross-signed root from one’s chain would have been enough to fix anything affected in this way. That’s what I did, figuring more people IRC from CentOS than ancient Androids, and it resolved pretty much everyone’s problems; so far I haven’t heard from anyone who needed the cross-signed cert.

      1. 1

        Do you have automation for that? AFAICT if I nuke part of my fullchain it’ll just come back in 90 days

        1. 1

          We have some custom scripts around it anyway to request all the certs centrally and distribute them, so it was easy (if not very pretty) to hack in some awk to cut out the cross-signed root right before we send the certs out. But if you use dehydrated (which I’d recommend anyway) I think you can use --preferred-chain 'ISRG Root X1' to get the non-cross-signed chain straight from LE.

        2. 0

          Yes, that was our (unpleasant) experience as well: updating ca-certificates was not enough, we also had to upgrade openssl and gnutls. In the end we solved the problem by switching to ZeroSSL (we have a ton of older VMs for testing and upgrading them all was not an option). The whole ordeal left quite a bad taste, I doubt we will touch Let’s Encrypt again if we can help it. And their attempt at spinning it as a good thing (“standing on our own feet”) just adds insult to injury.

          1. 14

            The whole ordeal left quite a bad taste, I doubt we will touch Let’s Encrypt again if we can help it.

            I don’t really understand this at all. The bugs were in other software and could just as easily have been triggered by another cert, but you blame LE for it. And I don’t know what they were supposed to do differently. Were you expecting them to somehow have and get away with a perpetual non-expiring root cert?

            1. 4

              I broadly agree—the expiration was manifestly not LE’s fault—but I suspect that if they’d done more testing they might have chosen not to default to the chain with the expired cross-sign. It broke more things than anyone expected.

              But… I didn’t test it either. Apparently hardly anyone did. And given it’s a public benefit doing this for free, I don’t particularly feel that they owed me the testing I couldn’t be bothered to do.

              1. 6

                I guess I just see this as the rehearsal run for the expirations of older and longer-lived certs from “traditional” root CAs. We were all going to have to deal with it sooner or later, and some of the bugs and faulty assumptions turned up by this one have been kind of scary and I think it’s good to be exposing them.

                1. 1

                  I mean, it seems easier to me to get vendors to push a new root CA than figure out the exact mix of cross-signing rules that won’t peeve off a diverse set of implementations to me, anyways.

                  I’m tempted to say “fuck it, just ship a 20 year root certificate and we’ll replace it with all the other certs come 2038, and only sign 1 month certificates in case we need to revoke it”, but I suppose that isn’t security, isn’t it?

              2. 1

                The bugs were in other software and could just as easily have been triggered by another cert, but you blame LE for it.

                Yes, but when a non-trivial portion of the web depends on your service, saying that it’s not our bug and therefore we are going to go ahead and break things is not a good strategy, IMO.

                And I don’t know what they were supposed to do differently.

                I am not an expert (especially when it comes to cross-signing, alternative paths, etc) and I could very well be wrong here (in which case please correct me) but from their initial announcement my take is that they could have looked for a new well-trusted root (probably by going to one of the older ones and perhaps paying them some non-trivial amount of money) but they decided not to.

                And speaking of predictions, I did expect this to happen, I just didn’t expect it to be this bad: it’s one thing to update ca-certificates (which, at least on Debian, you can just copy from any newer version and it will install fine on any older) and another upgrading foundational libraries like libssl and libgnutls (for example, there are no fixed versions for older Debian releases).

          2. 2

            The only thing I’ve had to fix so far: Arcanist not connecting to reviews.freebsd.org. There’s some bizzare code in Arcanist that makes sure that if a root CA bundle path is not explicitly set in php.ini, the bundled bundle (heh) – that is now outdated – would be used, PHP’s built-in finding of the system-wide default CA bundle be damned.

            1. 2

              If someone from Google reads this, it seems that GMail fails to validate ISRG Root X1 when sending emails though a 3rd party SMTP server using the “Add another email address you own” functionality. JFYI

              1. 1

                I found this article by the same author to be a good overview of the Root CA issue, and it mentions the Let’s Encrypt situation in passing (it was written before the post-mortem linked here).