1. 64
  1. 5

    That’s nice and all, but ultimately gives me an even bigger impression that if I want to stay away from a service playing Man-In-The-Middle (cloudflare), I’m more or less a roadkill for those guys.

    1. 18

      Honestly, that’s how it’s always been. You have bigger pipes than the DDOSer or you pound sand and fall over. There’s no good way around it.

      Welcome to the modern internet, hoping you won’t get DDOSed is not a scalable strategy to prevent DDOS attacks. I personally only really use cloudflare defensively, because my site had been knocked over entirely with DDOS attacks and I don’t want to deal with that headache again.

      1. 4

        hoping you won’t get DDOSed is not a scalable strategy

        It worked so far, mostly, at least for years ;) But I also didn’t launch anything public really, and I’m at the point of changing that, which gives me nightmares. (And I really, really dread having one service taking the (TLS) keys to my kingdom..)

        P.S. Yes, you’re right..

        1. 2

          I personally only really use cloudflare defensively

          Could you elaborate on how you use them?

          We looked into them at my company, and I even had a call with their sales guys, but it was impossible to get useful information out of them. They basically wanted 6k / mo, or else just use the free tier…. and everything was business jargon.

          1. 9

            Sure. I am a person that likes to write about tech on their blog. I am also not a white man that is attracted to women, which unfortunately makes me a target for harassment, hate and apparently (as in this is something I have directly observed through my personal experience in this hellplane) getting biblical amounts of traffic sent at me with little warning. I use Cloudflare defensively because they are not likely going to take Cloudflare down with the amount of traffic that the kids that apparently decide that me being honest about observable facts about how I experience reality means that my site turns into a competition of how much traffic can booters and DDoS for hire services can throw at it.

            By using Cloudflare, which sits between the public internet and my blog server, the attackers will target cloudflare, not my blog server. This makes their attempted attacks do nothing. Combined with the ridiculous levels of engineering I have done to make my blog as fast as possible, I am fairly confident that my site can weather most DDoS attacks that will get thrown my way. The other part of this puzzle is to avoid writing things that are too inflammatory so that I am not an enticing target, but that is different and is more of a deterrence strategy than an outright technical thing.

            Cloudflare is part of my strategy to protect my website against random DDoS attacks. I really wish I didn’t have to use them, but overall I think that the net good outweighs the complications that I take doing this kind of stuff. Definitely worth the $20 a month I pay for the service, if only for the peace of mind.

            1. 1

              Thanks for the reply. That is a shitty situation. If I want to setup similar protection, are there specific technical options you use? Addons? Configuration? Etc…

              Or does just the standard service automatically give you the DDoS protection?

              1. 3

                I can check, but other than telling it “holy crap cache everything as much as you can” I’m pretty sure it’s fairly vanilla.

        2. 5

          Keep in mind that cloudflare is not the only game in town. Sure, it’s all effectively MitM, but there are other valid options from not-as-evil companies. Or as reezer mentioned, you can rely on your provider to some extent.

          1. 6

            Do you have any good sources talking about Cloudflare’s evilness? The amount of power they have is obviously absurd but I haven’t found much info about them actually abusing it.

            1. 7

              IIRC the loudest voices RE cloudflare object to their free speech absolutism & handling of abuse complaints - e.g. when they received complaints about far-right sites selling pro-insurrection merchandise, nazi memorabilia, etc they A) kept serving them at a profit, and B) took personal details including address from the complainant and passed it on to the site owners.

              While free speech absolutism has those for and against it, I’m sure you can imagine why someone might object to having their home address passed to a white nationalist group as “an activist against white nationalism with an asian name lives at this address”.

              1. 3

                For me this is where they reached the comic book villain level of evil https://mobile.twitter.com/stealthygeek/status/1485731083534667779 Since then I’m actively encouraging people to read it and use some other service instead.

            2. 3

              A lot can already be done with a haproxy<=>varnish sandwich¹ on a €30 dedicated machine.² But this not cool anymore, the new coolness is man-in-the-middle with vendor-lock-in-freemium and unreliability.

              What can I say, I’m an old man, now… :)

              ¹. https://www.haproxy.com/blog/haproxy-varnish-and-the-single-hostname-website/
              ². I dont want to advertise for a German provider which starts by “Hetz” and finishes by “ner” or a the semi-cheap branch of a french provider which name asks “so, you start?” There are many others, like one formerly known as “dedibox”, etc…

              1. 12

                You seem to dismiss the new MitM vendors, but anything you deploy to your dedicated machine has two basic issues:

                1. You can get DoS-ed by anyone with a similar pipe to yours. It makes no difference what you deploy on your machine - if your network interface is flooded with empty packets, that’s all you’ll see. So anyone with a €30 dedicated machine can kill your service.

                2. (mentioned in the post) - If you’re a popular DoS target, your service provider may not like to be constantly flooded by your traffic and will drop you. This has happened to many people already and there’s nothing you can do about it, unless you explicitly pay them for being your DoS shield.

                1. 4

                  You’re right. At the end if they want to DDoS, they will succeed unless you use these special companies (~= Cloudflare, Akamai, Fastly, …)

                  But in the case of the author, network was not the limiting factor, he got properly DoSed (not DDoS) because his web app had slow calls (= high CPU usage), and didn’t limit incoming connections. HAProxy (= limiting connection) and Varnish (= caching page) will go a long way before the network becomes a limit. And when the network starts becoming a limit these €30 dedicated machine providers (at least Hetzner, OVH/SoYouStart, Scaleway, …) have DDoS protections.

                  In the case of the author, HAProxy and Varnish would have done a lot already.

              2. 1

                Most bigger providers, be it Hetzner, OVH or others give you quite good DDOS protection for free.

                Also with the numbers shown in the article one really shouldn’t underestimate what resources are available for relatively cheap these days.

                And using Cloudflare can also mean that these “How X nearly caused the internet to go down” and similar turning downtime into marketing articles on the Cloudflare blog probably affect you, whereas you are fine otherwise. Don’t get me wrong, they are sometimes pretty informative, yet it’s something that one should consider.

                So it’s all a bit context dependent. Depending on what service you are hosting attacks might also look differently and individual solutions might or might not work. DOS can often be achieved in more effective ways than just hitting the landing page, that usually is quite static in first place many times. And if you have something that rapidly fills up databases or causes compute resources to be used recovering might be a lot harder. Of course dumb attack are the most frequent ones, but usually also the ones having the smallest effect.

                In short what I wanna say is that if you blindly use cloudflare (or any others) that form or that websocket thingy might still knock your service out, with a lot less resources on the attackers side.

                1. 4

                  Hetzner’s DDoS protection is not very effective. Sure, it’s best than other providers such as Contabo (another German provider) where their “DDoS protection” is a free nullroute, but nothing’s compared to OVH’s protection. And then, they (at least happened before, not sure nowadays) didn’t protect any traffic coming from inside their network - so have fun dealing with OVH boxes DDoSing other OVH boxes.

                  1. 1

                    I’d argue that dropping traffic early from your provider is a lot easier to deal than traffic from random places on the internet. Don’t they even have a firewall thingy, so you could block that even before touching any local packet filter.

                  2. 3

                    They don’t give you DDOS protection, they just don’t boot you for having higher traffic (as they did ~10 years ago, trust me). And they are so friendly to nullroute that traffic. It does not help with the load on your side.

                    1. 1

                      Again, depending on the kind of attack and the service the same is true for CloudFlare, etc. Sure, if your problem is someone GETing / then pretty much using a CDN will fix your problem. I just don’t think the sentiment that it’s CloudFlare or not is the right one. It’s not a 0 and 1 thing. Especially not for the particular case discussed here.

                2. 2

                  While the first attack seems to be properly distributed, the one in the update looks like it’s relying on hosting platforms. I’m often just dropping all traffic from those for my own services - there’s minimal collateral damage (homemade proxies and custom scrapers) but otherwise there’s no “real browser” in AWS or Hezner. Unless you have a good use case for those sources, you can consider disabling the housing providers before they start causing problems.

                  1. 3

                    Back in the pre-pandemic times when I traveled a lot, I ran an instance of Algo as a personal VPN for use when on strange wifi networks. And I know I’m not the only person who used it, or things like it, which means there most certainly are “real browsers” who appear to be in cloud hosting providers :)

                    1. 1

                      Yeah, that’s part of the “homemade proxy” collateral damage. But it’s such an edge case I don’t really mind.

                  2. 2

                    I really like the amount of stuff he shows that you can apply to a tower/hyper stack. I have been using actix-web for a very long time and some of his code really looks verbose (and pin-nightmarish) in comparison, but there is some real potential and libraries I could steal borrow.

                    Also, is there anything like honeycomb for selfhosting ? jaeger feels limited in contrast.

                    1. 2

                      Was just looking at that earlier today, perhaps https://signoz.io/ is worth a look? It looks very much as a work in progress though.

                      1. 1

                        that looks promising