1. 69
  1.  

  2. 14

    Cloudflare’s competitors rely on a DNS extension that provides some fairly coarse geographical information about clients upstream (the client subnet) in order to provide feedback about the geographical distribution of requests. Cloudflare blocks this with 1.1.1.1 claiming it’s for privacy reasons but that’s just using trigger-talk to justify an anti-competitive practice IMO. The archive.is folks are refusing to support non-EDNS resolvers.

    I’ve noticed a lot of breakage on the Russian internet when using 1.1.1.1 as well.

    In the early days, implementors of this DNS extension would sometimes use a manually configured whitelist to determine who they sent the augmented (and invalid-to-non-supporters) responses to, but it seems like now there are a number of resolvers that just send it to any requesting resolver.

    1. 3

      Cloudflare’s competitors rely on a DNS extension that provides some fairly coarse geographical information about clients upstream (the client subnet) in order to provide feedback about the geographical distribution of requests.

      That’s not entirely true. Archive Today relies on it, and I know they aren’t the only ones, but many of CloudFlare’s actual competitors don’t rely on EDNS.

      CloudFlare, Fastly, and Google all use BGP anycast for geographical distribution of requests. Basically, it’s a deliberate IP address conflict, but since each node is placed in a different location, the “shortest, fastest route” BGP logic does the load balancing. While AWS does use DNS to perform “latency-based” load balancing, they don’t rely on EDNS either: they can guess where the resolver is, geographically, based on which of their data centers wound up getting the request (AWS has lots and lots of peering arrangements, all of which lead to their handful of “region” data centers).

      If someone tried to bring an anti-competitive charge against CloudFlare for this, that’s what their reply would be. Their competitors don’t need EDNS, and neither do they. The websites that are actually impacted by this are ones that are big enough to benefit from a CDN, too small to make BGP peering arrangements, and too stubborn to make a deal with a company for their CDN.

      Killing off EDNS would help all of the big CDN providers, not just CloudFlare. They’re not hurting their competitors, but they are growing the market that they’re in.

      1. 4

        Just because a given provider runs an anycast network doesn’t mean that they don’t also use EDNS-Client-Subnet, too. If it was as useless as you claim, it wouldn’t have seen a rather good adoption.

        In fact, if you look at the responses from Cloudflare CEO, you can see that the big players actually do use ECS, and Cloudflare’s lower number of PoPs does seem to affect them:

        https://news.ycombinator.com/item?id=19828702

        We are working with the small number of networks with a higher network/ISP density than Cloudflare (e.g., Netflix, Facebook, Google/YouTube) to come up with an EDNS IP Subnet alternative that gets them the information they need for geolocation targeting without risking user privacy and security. Those conversations have been productive and are ongoing. If archive.is has suggestions along these lines, we’d be happy to consider them.

        Keep in mind that these are merely the bigger players, and the smaller players, other than archive.today, probably don’t even get a say in the matter at all.


        too stubborn to make a deal with a company for their CDN

        Yeah… No. The logic that everyone should be beyond one of the big CDNs is beyond flawed. Are you seriously blaming a hobbyist project, bootstrapped startup like archive.today, for not wanting to have corporate overloads like Cloudflare dictate what they can and cannot do for their hobby, and have complete control over the viability of the project? To have a dependency on opaque third-party SaaS? Describing this as stubborness is insincere at best.

        1. 4

          Stubborn isn’t necessarily bad. I apologize for using language that had such a negative connotation. Archive Today is not doing anything wrong, and it was bad for me to make it sound like they were. In the future, I’ll avoid using language that comes across so negatively.

          1. 1

            One of the biggest character flaws I see among engineers is a myopic focus on “just making it work”, regardless of “If you should make it work”. I feel that has appeared here, where small players must cater to big players.

            Granted, everyone loves the “big players shouldn’t cater to every small player” mindset, but it’s inherently reductive and leaves us at the mercy of the big players who can control everything.

            Perhaps EDNS is no longer necessary. However, I really think a form of “Be conservative in what you send, be liberal in what you accept” should apply to Big players most of all.

      2. 6

        Cloudflare’s CEO response to this issue (5 months old): https://news.ycombinator.com/item?id=19828317

        TL;DR: it’s Archive.is’ authoritative nameservers who return bad results (something in the 127.0.0.0/8 block) when 1.1.1.1 queries them. They have discussed internally to band-aid this with some workaround, but they decided it “would violate the integrity of DNS”.

        1. 4

          It’s certainly an interesting response from Cloudflare’s CEO:


          https://news.ycombinator.com/item?id=19828702


          the integrity of DNS and the privacy and security promises

          the integrity of DNS and the privacy and security promises

          (yes, the above phrase is actually mentioned twice in separate sentences)

          This information leaks information about a requester’s IP and, in turn, sacrifices the privacy of users.

          motivation for the privacy and security policies of 1.1.1.1

          geolocation targeting without risking user privacy and security

          First, they mention privacy as a reason a lot; a total of 5 times in a 5-paragraph snippet; and “security” 4 times. It has now been debunked that their DoH (which they deem even more private and more secure) is less private, not more. https://lobste.rs/s/sno4wu/centralised_doh_is_bad_for_privacy_2019

          Whole thing doesn’t stand the most basic litmus test — after resolving a name, you still have to make HTTP/HTTPS requests, revealing not just the /24 subnet, but full 32 bits of the IPv4.

          nationstate actors have monitored EDNS subnet information to track individuals

          Yeah, this is just FUD, without even any attempt for a double-blind study. Which nationstate actors? Whom did they monitor? How instrumental was ECS, and how did it even came into the picture? It’s not like local regional providers have any reason to employ ECS. Did someone in China switch to an ECS-compliant third-party provider, without just going for a full VPN? Why? (Doesn’t it prove that it’s the third-party providers, like Cloudflare DNS, that facilitate this monitoring?) Whole thing just doesn’t make sense. You can always just monitor the HTTP/HTTPS traffic instead.

          If you need real security, you gotta use a real VPN, not a fake DNS bandaid.

          Lack of ECS on a global anycast resolver is not a security and privacy feature; it’s just a poor form to run a global public internet service.


          We publish the geolocation information of the IPs that we query from.

          Where? Not in DNS. Not in whois (there’s no rwhois referral, either).

          Every other provider that doesn’t provide ECS at least has very easy to understand rDNS on the source IP of their resolver; but not Cloudflare:

          % dig @a.resolvers.level3.net. o-o.myaddr.l.google.com -t txt +short | cut -f2 -d\" | xargs host
          4.14.0.8.in-addr.arpa domain name pointer dns-8-0-14-4.dallas1.level3.net.
          % dig @ordns.he.net. o-o.myaddr.l.google.com -t txt +short | cut -f2 -d\" | xargs host
          2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.7.0.0.0.0.0.0.0.7.4.0.1.0.0.2.ip6.arpa domain name pointer tserv1.dal1.he.net.
          % dig @one.one.one.one o-o.myaddr.l.google.com -t txt +short | cut -f2 -d\" | xargs host
          Host 33.220.162.108.in-addr.arpa not found: 2(SERVFAIL)
          %
          

          We are working with the small number of networks with a higher network/ISP density than Cloudflare (e.g., Netflix, Facebook, Google/YouTube) to come up with an EDNS IP Subnet alternative that gets them the information they need for geolocation targeting without risking user privacy and security. Those conversations have been productive and are ongoing. If archive.is has suggestions along these lines, we’d be happy to consider them.

          Here, contrary to some popular belief that big providers use anycast and don’t need ECS, he’s basically admitting that other providers actually do use ECS, and do benefit from resolvers with ECS. By not doing ECS, Cloudflare DNS makes all competing CDNs slower than their 3.5 billion USD own CDN. This is in case anyone has any doubts that ECS is actually used by the large providers who have the capacity to do anycast.

        2. 6

          EDNS is important. I don’t know how Cloudflare thinks you’ll get a good experience without it. It will cause odd behavior like an American being sent to an Australian CDN mirror.

          I was having this issue at work and then realized it was broken EDNS…

          1. 5

            I don’t know how Cloudflare thinks you’ll get a good experience without it.

            You get a good experience to Cloudflare hosted proxies. Which strenghtens their case when they say that your website gets faster if you let them MITM it.

            1. 2

              Don’t most global DNS/CDN provider rely on anycast + bgp for routing requests to closest PoP?

              1. 1

                That’s exactly the thing, isn’t it. CloudFlare, and many of their direct competitors like Fastly, do use anycast routing for their CDN. But that’s expensive, and making it work well is complicated, so Archive Today uses Geo DNS instead.

                1. 1

                  While anycast is complicated it can be done cheaply by those who know how, and for those who don’t, route53 is only marginally more expensive than a well-monitored “geo dns” setup.

                  “Geo dns” is so dumb, and so extremely error prone, and so very difficult to correctly handle many failure scenarios that are just simply automatic with the anycast approach, that I do not recommend it to anyone.

                  1. 3

                    can be done cheaply by those who know how

                    OK, I’m all ears here!

                    Anycast is completely inaccessible to the average advanced user. There are some providers which offer anycast service on their own IP space (vr.org/hostvirtual.com — only one provider I’m aware of, showing that there’s not even much competition), and you quickly realise just how inflexible the whole thing is, because IPv4 space has a /24 granularity, meaning, you gotta dedicate a whole /24 to a given anycast group, and every single member of the group is then bound to have anycast presence in every single location from which the anycast is announced.

                    Got a new PoP? Gotta have every customer onboard for purchasing the computing resources at the new location. Got a spiked load for one specific PoP? Better be using autoscaling! I.e., it’s an all-or-nothing kind of situation. There’s no individual scaling, individual selection of which PoPs you get to have your anycast in etc.

                    Otherwise, if you do want such control, you gotta have your own /24, and find providers willing to announce it for you. Where exactly can you get such custom services cheaply? I don’t see anything like that in the price lists of most hosting providers which are my go-to for the affordable dedicated servers.


                    This whole notion and all the arguments provided by Cloudflare and their fans resolve around the fact that they simply don’t care about the hobbyist and grass-roots internet services like archive.today. The writing on the wall being that archive.today is too small to need their own thing, should be a (paying) Cloudflare CDN customer, or just do things differently than is convenient for them in their free service without venture capital, as well as other bigotry justifying why Cloudflare, a 3.5 billion US dollar company playing dirty with their free Cloudflare DNS subsidised by their billion-dollar CDN operation to slow down all the other competing CDNs, is in the right. It’s sad that so many folks applaud these actions — they should not be celebrated. We need internet diversity. Some folks don’t even realise how much of a monopoly and a bully Cloudflare already is. It is hard but possible to do internet without Google. Not so easily without access to Cloudflare.

                    1. -1

                      Anycast is completely inaccessible to the average advanced user.

                      Be that as it may, I also think you’re probably less advanced than you think.

                      First issue: BGP has no security. Getting an IP and an ASN is just some paperwork, but you still need networks to let you announce.

                      I don’t know of any providers that would let you do this with a prepaid debit card you picked up in Walgreens, but once I’ve explained what I’m doing, I’ve found most virtual hosting providers will set things up appropriately. I’ve yet to have someone charge me for this.

                      Some of the providers I’ve used include Softlayer, Ghandi, HopOne, and OVH. Just one Linux instance, and I run gated (or zebra or whatever) on each site.

                      If you still can’t figure it out, you can suggest some use cases and maybe I would help you set up an experiment if you have an interesting idea.

                      There are a lot of things to get right if you don’t want to piss off other network admins on the Internet so it should not surprise you (or anyone else) that there’s a significant amount of KYC at the gates.

                      This whole notion and all the arguments provided by Cloudflare and their fans resolve around the fact that they simply don’t care about the hobbyist and grass-roots internet services like archive.today.

                      This is uncalled for.

                      I’m happy to engage on EDNS (client subnet extension) being a dumb/unnecessary privacy leak, and useless for load balancing or DDoS protection, and anycast being straightforward, but I can’t speak for Cloudflare or its fans. I’ve never used Cloudflare and my experience as a regular internet user would never have me recommend them or their patently silly “DDoS protection” product.

              2. -3

                It shouldn’t.

                If you have anycast dns servers who receive a request from a name server without EDNS support, you can basically assume the POPs nearest your nameserver should be returned.

                If you’re not anycast (eg because you’re the sad sort who thinks “Dns geoip load balancing” is a thing) then you should just fix that problem. Route53 is cheap.

                1. 3

                  Yeah, sorry, maybe with all your millions of dollars in venture capital you might think that DNS geoip is not a thing, but if you don’t have that kind of money, it still does the job just fine. (Extra 80ms in latency is nothing compared to the average page load times of several seconds on the modern web anyways.) And EDNS-Client-Subnet isn’t there for nothing, either. And what does AWS Route53 has to do with anycast in the first place? Also — AWS may be convenient, but it sure ain’t cheap, either.

                  1. 0

                    with all your millions of dollars in venture capital

                    Who the fuck do you think you’re talking to?

                    I don’t have millions of dollars in venture capital.

                    you might think that DNS geoip is not a thing

                    Not a thing? It’s something that inexperienced sysadmin do: It’s clearly a thing, just not a good thing.

                    And EDNS-Client-Subnet isn’t there for nothing, either.

                    Yes. It really is.

                    It’s a privacy leak engineered by Google, and there’s no point to it. You can’t trust it, so you must handle fallbacks with it missing instead of diverting traffic to 127/8 e.g. users who have a local resolver running on their laptop which might not know the IP address (or guess wrong). Just assume the web service closest to your nameserver and you’ll immediately be doing better.

                    The users who foolishly use 8.8.8.8 and etc because they believe it’s faster deserve what they get. They are great tools, and it’s touching how they’re used to subvert censorship and oppression but make no mistake: the only real reasons they serve is those to their masters.

                    AWS may be convenient, but it sure ain’t cheap, either.

                    Route53 is cheaper than trying to build your own reliable “geoip dns” crap: one of the biggest cost-savings is you don’t need a few dozen monitoring nodes second guessing the answers coming out of your servers. Split paths and partial network outages are extremely common and impossible for your dns servers to detect on their own.

                  2. 1

                    We use GeoDNS and anycast. It’s not rocket science and there are some very good reasons to leverage both.

                    1. 0

                      There aren’t. Really.

                      BGP is a much better/more-complete picture of the network shape, so there’s no reason to do an IP to arbitrary country or lat/lon table built every month by Ip2location (or your favourite vendor here), and given that database is obsolete before you even downloaded it, you might as well “just” use BGP and your health/monitoring network.

                2. 5

                  Cloudflare not passing EDNS on their end due to privacy concerns is silly. For the vast majority of users, their IP will get revealed to the origin server when they try to connect.

                  If you’re trying to conceal your IP, then you’re using a VPN (either private or Tor) and there is still no added value to blocking EDNS because you’d be obscuring your IP before even requesting from their resolver.

                  1. 1

                    There’s more to it: https://00f.net/2013/08/07/edns-client-subnet/

                    ECS part of EDNS can leak IP otherwise hidden by VPN or proxies. It also helps with cache timing attacks and cache pollution.

                    1. 1

                      Thanks for this article, good information present in there.

                      A DNS leak when using a VPN is still not a concern if your DNS is properly configured to also go through the VPN. This article raises the concern of unproxied DNS coupled with proxied IP access.

                      As far as newly exposed attack surface due to cache timing attacks in the case of not using a VPN, I still don’t think the threat is significant since the origin server will eventually get your IP anyway when you connect to their services.

                  2. 3

                    Can someone ELI5 this? It has 10 upvotes in just an hour but I have no idea what this is about.

                    1. 15

                      Difficult to explain DNS like you are five but I’ll try to boil down the idea assuming a certain level of knowledge:

                      1.1.1.1 is a DNS service from cloudflare. It’s like google’s 8.8.8.8, and other services.

                      If you use it, you will find you cannot resolve archive.is pages.

                      This is because they won’t resolve the EDNS subnet. Archive.is are using EDNS for DDOS protection.

                      Cloudflare provide DDOS protection too, so by not allowing EDNS on their service, they are blocking websites who otherwise wouldn’t be blocked if they were using cloudflare.

                      I don’t know if this really is their motive, but it’s easy to read between the lines…

                      1. 2

                        Difficult to explain DNS like you are five but I’ll try to boil down the idea assuming a certain level of knowledge

                        Heh, yeah, good assumption - I do know what DNS is :D

                        Thanks for the explanation, it helped me understand the issue

                        1. 1

                          Here’s resolution via 1.1.1.1 and via my default dns

                          jon~/i/d/d/h❯❯❯ dig archive.is
                          
                          ; <<>> DiG 9.10.6 <<>> archive.is
                          ;; global options: +cmd
                          ;; Got answer:
                          ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44289
                          ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
                          
                          ;; OPT PSEUDOSECTION:
                          ; EDNS: version: 0, flags:; udp: 4096
                          ;; QUESTION SECTION:
                          ;archive.is.			IN	A
                          
                          ;; ANSWER SECTION:
                          archive.is.		293	IN	A	212.80.216.76
                          
                          ;; Query time: 3 msec
                          ;; SERVER: 192.168.2.1#53(192.168.2.1)
                          ;; WHEN: Thu Oct 03 23:56:24 PDT 2019
                          ;; MSG SIZE  rcvd: 55
                          
                          jon~/i/d/d/h❯❯❯ dig archive.is @1.1.1.1
                          
                          ; <<>> DiG 9.10.6 <<>> archive.is @1.1.1.1
                          ;; global options: +cmd
                          ;; Got answer:
                          ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35264
                          ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
                          
                          ;; OPT PSEUDOSECTION:
                          ; EDNS: version: 0, flags:; udp: 1452
                          ;; QUESTION SECTION:
                          ;archive.is.			IN	A
                          
                          ;; ANSWER SECTION:
                          archive.is.		65936	IN	A	127.0.0.3
                          
                          ;; Query time: 252 msec
                          ;; SERVER: 1.1.1.1#53(1.1.1.1)
                          ;; WHEN: Thu Oct 03 23:56:28 PDT 2019
                          ;; MSG SIZE  rcvd: 55
                          
                        2. 14

                          Cloudflare makes all their money from running a CDN — Content Delivery Network — protecting web-site operators from DDoS attacks and speeding up website performance through multiple points-of-presence worldwide. They’ve had an IPO a few weeks ago, $NET, valued at 3.5 billion US dollars.

                          Last year, they’ve launched a free public recursive DNS resolver, called 1.1.1.1, similar to Google’s 8.8.8.8. There’s no statements about monetisation for it. Unlike 8.8.8.8, which supported EDNS-Client-Subnet, which is necessary for proper CDN functioning, and has been supported by Google since at least 2013, Cloudflare decided not to support it on 1.1.1.1, citing rather disingenuous privacy claims that don’t stand the most basic litmus test — your IP address is guaranteed to leak anyways once you make that HTTP/HTTPS request, so, hiding it from DNS is rather pointless, especially if it’s going to interfere with CDNs and hinder performance — for no added real privacy benefits, either.

                          archive.today had none of it. Folks noticed that archive.is doesn’t resolve through 1.1.1.1. The official statement on Twitter was that it’s due to Cloudflare not supporting EDNS-Client-Subnet.


                          To me, though, the conflict of interest, with Cloudflare making all their money from being the CDN, yet now running a public recursive resolver at 1.1.1.1 that makes it much more difficult for other CDNs to do their job properly, seems like a rather obvious CoI, yet noone seems to bothered to point this out up until now. ¯\_(ツ)_/¯

                        3. 3

                          Does anyone have any numbers on how effective ECS routing actually is?

                          I can’t imagine it working unless DNS resolvers shard their caches by client subnet, and send fresh requests for different subnets purely for the benefit of ECS routing? That seems like a useful thing to do, I just haven’t heard of anyone doing it.

                          In the same vein, individual DNS resolvers serving relatively small subnets obviously won’t share a cache. Their caches are effectively sharded by client subnet like above. But if that’s the case, ECS would only be necessary if these smaller resolvers had radically different IP addresses than their clients. I don’t see why that should ever be the case?

                          Does anyone have any first hand experience with this?

                          1. 1

                            Well, what does Google DNS do? I don’t see how you could cache it without considering ECS.

                            Given how much spam I get from Googlebot pretending to be all possible user-agent for every single page on my site, even though none of them 10k pages deviate based on User-Agent, it doesn’t seem like it’s an issue for Google to simply disregard such nuances, and err on the side of always assuming that ECS routing is in place.

                            1. 2

                              I have no idea what Google DNS does. I simply had never heard of ECS-aware caching; I’m no DNS expert.

                              However, I think have just answered my own question. Unbound supports optionally caching per EDNS client subnet. It looks like it will cache up to 100 subnets for a particular domain:

                              When an answer contains the ECS option the response and the option are placed in a specialized cache.

                              […] for each query only 100 different subnets are allowed to be stored for each address family. Exceeding that number, older entries will be purged from cache.

                              Now that I’ve heard of someone doing ECS-aware caching, I assume Google DNS does too. It’s interesting that Cloudflare doesn’t. Although Cloudflare’s public response points out that the geolocation of their DNS resolvers mitigate the issue:

                              EDNS IP subsets can be used to better geolocate responses for services that use DNS-based load balancing. However, 1.1.1.1 is delivered across Cloudflare’s entire network that today spans 180 cities. We publish the geolocation information of the IPs that we query from. That allows any network with less density than we have to properly return DNS-targeted results. For a relatively small operator like archive.is, there would be no loss in geo load balancing fidelity relying on the location of the Cloudflare PoP in lieu of EDNS IP subnets.

                              We are working with the small number of networks with a higher network/ISP density than Cloudflare (e.g., Netflix, Facebook, Google/YouTube) to come up with an EDNS IP Subnet alternative that gets them the information they need for geolocation targeting without risking user privacy and security. Those conversations have been productive and are ongoing. If archive.is has suggestions along these lines, we’d be happy to consider them.

                              I wonder what kind of solution they will deploy to share ECS information. They cite nationstates tracking individuals via unencrypted ECS as one motivation for their policy. Perhaps they will only forward ECS if the authoritative DNS server supports DoH?

                          2. 4

                            so, when i use firefox with default settings, DoH via cloudyflare, i can’t go to archive.is?

                            1. 3

                              That is correct, when DoH via Cloudflare is enabled, archive.is will not resolve to a correct IP address. Mozilla has announced that users from USA will be enrolled in this automatically.

                              1. 2

                                thanks, i’m on the ESR version and was too lazy to check this :)

                                maybe all the criticism about centralization is right, but at least they’ve promised to be nice.

                            2. 4

                              While we’re at it: can we please switch the cached link to archive.org?

                              • they are a well funded non-profit
                              • their archive is open and mirrored elsewhere

                              Meanwhile, archive.is

                              There is an old ticket on github (#331), but that was reverted without discussion a few months later.

                              @pushcx: you seem to have implemented that on barnacl.es, can this please also be considered for lobste.rs again? I’m happy to submit the PR.

                              1. 4

                                I’d be against such a change. Archive.today is faster, simpler, seems to require less JavaScript, has integration with all the other archives.

                                My view would be different if we were to integrate their database-driven non-semantic URL shortener links, which indeed would be a problem for long-term preservation; however, our cached button uses deterministic links that are better and more user-friendly at archive.today than archive.org, so, I see no good reason for a change.

                                1. 2

                                  Please start a meta thread about it. It sounds like there are at least four qualities (funding, caching speed, presentation, transparency) to trade off? Here’s the commit for that revert.