1. 49

  2. 19

    I like that there is now yet another ACME compliant endpoint. What we need next are clients that actually support arbitrary endpoints. There are a lot of management UIs that interface with generic clients but expose Let’s Encrypt as the only option. I want to be able to plug in my own private ACME CA but still get all of the automation benefits.

    1. 2

      99% of them do so that you can use a staging URL, don’t they?

      1. 1

        Not that many even expose the option of staging LE or not. Of those that do, it’s still hardcoded to Let’s Encrypt’s staging environment. Still not generic.

        1. 3

          All of these allow setting the API server:

          The official client does it: https://certbot.eff.org/docs/using.html#changing-the-acme-server

          Acme.sh does it in the article

          Terraform: https://registry.terraform.io/providers/vancluever/acme/latest/docs

          Traefik: https://doc.traefik.io/traefik/https/acme/#caserver

          K8s cert manager: https://cert-manager.io/docs/configuration/acme/

          Which ones have you used that don’t? I get that they probably mostly want sane defaults and don’t want people filling out random MitM API servers or something, but I’ve not found one that doesn’t allow me to change it.

          1. 1

            I’m thinking about those that sit on top of these. For example, setting up ACME in CPanel, OpenWRT, OPNsense. Or commercial software, like a website builder or managed service provider. (Installing wordpress, gitlab, or something else for you.) It has been a while since I’ve checked on these; I’d love it if they are more flexible now.

            The underlying protocol implementations are flexible, indeed. There isn’t really a sysadmin/CLI focused tool that can’t accept an arbitrary endpoint. It’s the layer above that I’m frustrated with.

            1. 1

              Oh! Yeah, if it’s not actually an ACME client, but a client to the client, yeah, I’ve never seen those expose arbitrary endpoints either. CPanel doesn’t even use Let’s Encrypt, it uses it’s own root CA. So you’re kinda stuck trusting CPanel and not even a public entity like Let’s Encrypt.

    2. 8

      I remember learning about other CAs that support ACME several months back from a Fediverse admin. I’m really glad there are alternatives. Mozilla did the right thing by making the entire process open. I feel like this is more important that ever.

      Mozilla has had financial troubles, and although it’s unlikely they would lose funding for LetsEncrypt, they certainly could. Second, Mozilla has made a lot of questionable political decisions, and has made it clearly they care a lot about politics internally within the non-profit. Having alternatives is essentially for the day when Mozilla says, “We refuse to grant you a TLS certificate because of what’s hosted on your domain.”

      1. 15

        Mozilla helped bootstrap Let’s Encrypt, with money, staff and expertise but Let’s Encrypt is a completely independent entity for a while now.

        1. 6

          Mozilla helped, but Linux Foundation did more in terms of staffing.

          Source: Was hired by Linux Foundation to work on LE, back in 2016.

        2. 9

          Mozilla does not own Let’s Encrypt directly, it’s a non-profit.

          The EFF is a sponsor, so denying someone a cert for political reasons will be a hard sell to them.

        3. 3

          I’m a big fan of ZeroSSL for larger organizations for a lot of reasons. While LE is amazing at its mission of getting more of the internet on HTTPS, it lacks some of the features I think are well worth paying for. Having a REST API you can use to integrate internal tooling is really nice, allowing applications to request and manage their own certificates. It also offers email verification for certificates which is great for applications where the lack of IP whitelisting that Let’s Encrypt provides is a problem.

          All that said, if your org uses LE extensively as many do, I don’t think there is a real business usecase for randomizing it. If LE is down for a long period of time, then you might need to switch, but it seems strange to optimize for that edge case.

          1. 1

            Does the email validation mean that you can get a cert with no A record and no DNS control?

            1. 2

              Yup! Let’s Encrypt didn’t want to deal with the headache of managing email at scale to automate that form of domain control, but there are a few RFC-standardized email addresses you can rely on, as zaynetro mentions. But the CA/Browser Forum baseline requirements only require (for “basic”/DV certs, anyways) that you prove you control a domain. There are lots of ways to do that, since that’s a social agreement.

              1. 1

                Sounds kind of crazy from the ACME perspective but email validation is acceptable to the CA/B baseline requirements and is basically the norm for DV certs for non-ACME issuers. The security implications aren’t great, and you need to make sure that e.g. no user can register one of the email addresses that’s acceptable to CA/B for this purpose, but it can be convenient for scenarios like issuing certificates for internal systems (not internet accessible) that use public domain names.

                1. 1

                  it can be convenient for scenarios like issuing certificates for internal systems (not internet accessible) that use public domain names

                  I use DNS challenges for this purpose. Once I got tired of manually creating and cleaning the challenge-response records, I spent a few hours adapting one of the existing plugins to work with my DNS host.

                  I like this better than injecting email into the process.

                2. 1

                  Looks like it: https://help.zerossl.com/hc/en-us/articles/360058295354-Verify-Domains

                  To verify your domains via email, first, select one of the available verification email addresses and make sure you have access to the associated email inbox. Typically, you will be able to choose between the following types of email addresses for your specific domain:

                  admin@domain.com, administrator@domain.com, hostmaster@domain.com, postmaster@domain.com, webmaster@domain.com

              2. 2

                Is there any reason for randomizing, or even rotating, the CA? I don’t understand the reasoning for it. It seems entirely unrelated to the “let’s encrypt can go down” scenario.

                1. 12

                  If you always use LetsEncrypt, that means you won’t ever see if your ssl.com setup is still working. So if and when LetsEncrypt stops working, that’s the first time in years you’ve tested your ssl.com configuration.

                  If you rotate between them, you verify that each setup is working all the time. If one setup has broken, the other one was tested recently, so it’s vastly more likely to still be working.

                  1. 2

                    when LetsEncrypt stops working

                    That’s how I switched to ZeroSSL. I was tweaking my staging deployment relying on a lua/openresty ACME lib running in nginx and Let’sEncrypt decided to rate limit me for something ridiculous like several cert request attempts. I’ve had zero issues with ZeroSSL (pun intended). Unpopular opinion - Let’s Encrypt sucks!

                    1. 5

                      LE does have pretty firm limits; they’re very reasonable (imo) once you’ve got things up and running, but I’ve definitely been burned by “Oops I misconfigured this and it took a few tries to fix it” too. Can’t entirely be mad – being the default for ACME, no doubt they’d manage to get a hilariously high amount of misconfigured re-issue certs if they didn’t add a limit on there, but between hitting limits and ZeroSSL having a REALLY convenient dashboard, I’ve been moving over to ZeroSSL for a lot of my infra.

                    2. 2

                      But he’s shuffling during the request-phase. Wouldn’t it make more sense to request from multiple CAs directly and have more than one cert per each domain instead of ending up with half your servers working?

                      I could see detecting specific errors and recovering from them, but this doesn’t seem to make sense to me :)

                    3. 6

                      It’s probably not a good idea. If you have set up a CAA record for your domain for Let’s Encrypt and have DNSSEC configured then any client that bothers to check will reject any TLS certificate from a provider that isn’t Let’s Encrypt. An attacker would need to compromise the Let’s Encrypt infrastructure to be able to mount a valid MITM attack (without a CAA record, they need to compromise any CA, which is quite easy for some attackers, given how dubious some of the ‘trusted’ CAs are). If you add ssl.com, then now an attacker who can compromise either Let’s Encrypt or ssl.com can create a fake cert for your system. Your security is as strong as the weakest CA that is allowed to generate certificates for your domain.

                      If you’re using ssl.com as fall-back for when Let’s Encrypt is unavailable and generate the CAA records only for the cert that you use, then all an attacker who has compromised ssl.com has to do is drop packets from your system to Let’s Encrypt and now you’ll fall back to the one that they’ve compromised (if they compromised Let’s Encrypt then they don’t need to do anything). The fail-over case is actually really hard to get right: you probably need to set the CAA record to allow both, wait for the length of the old record’s TTL, and then update it to allow only the new one.

                      This matters a bit less if you’re setting up TLSA records as well (and your clients use DANE), but then the value of the CA is significantly reduced. Your DNS provider (which my be you, if you run your own authoritative server) and the owner of the SOA record for your domain are your trust anchors.

                      1. 3

                        There isn’t any reason. The author says they did it only because they can.

                        1. 2

                          I think so. A monoculture is bad in this case. LE never wanted to be the stewards of ACME itself, instead just pushing the idea of automated certificates forward. Easiest way to prove it works is to do it, so they did. Getting more parties involved means the standard outlives the organization, and sysadmins everywhere continue to reap the benefits.

                          1. 2

                            To collect expiration notification emails from all the CAs! :D

                            1. 2

                              The article says “Just because I can and just because I’m interested”.