1. 44
  1. 16

    The author says that Cloudflare is blocking “1500 hits an hour”, which is 25 hits a minute and about 0.42 hits a second… which isn’t a lot to my eyes. I mean that if a hit is equivalent to an HTTP request, a $5/month machine should be able to handle this load given how simple this website is. Some hosting providers even allow unlimited traffic at this price point. Am I missing something?

    1. 4

      Author here: I don’t think it is a deliberate attack and I wouldn’t have noticed it at all if they were just hitting the html file. But they appear to be mimicking real browser traffic and continually downloaded the rather large image I had on my front page, not to mention the js beacon I use for hit tracking.

      Neither of these really mattered either, there was no impact on the site, but the extra bandwidth bugged me. the fact that the attack kept doubling every few days forced me to take some action.

      1. 1

        was there a particular harm that forced you to take action? or were you projecting that it would eat up your bandwidth limits if it continued to grow exponentially?

        1. 3

          Good question. I actually didn’t anticipate any actual harm to my (almost entirely static) site. But the bandwidth used was continually growing and I didn’t want to hit any limits. Even here, I could reduce the bandwidth by reducing the size of the images on my front page.

          However, I decided to take more drastic action for a number of reasons:

          • Even after I temporary removed the images, the bandwidth usage kept growing.
          • Furthermore, why should I stop serving the images that I want to my readers? I shouldn’t have to self-censor because of somebody else’s actions.
          • Even if it is not affecting me, the botnet is obviously up to something I don’t want to be an unwilling participant in. Perhaps by blocking them I am disrupting in a small way whatever they are trying to do. This is a faint hope, since they don’t seem to have noticed yet.
          • Seriously, screw them.
          1. 1

            That’s fair, though there are always technical limits to what can be transmitted in a way that is accessible to a general audience.

            No shade for doing what everyone else does, but it should be acknowledged that we actually can have nice things if we are willing to sacrifice a little bit of stylistic freedom. We already can’t have every website be a 4k video.

      2. 3

        Maybe they’re hosting from a PI

      3. 12

        As someone who hosts websites, one of which is connected to my actual identity, on VPSes: the best DDoS protection for low- to mid-effort attacks is performance. On a VPS hosting a static site and a few lightweight daemons, I take the following measures:

        • I use TLS with 0-RTT enabled and ECDSA certs combined with OCSP Must-Staple
        • I enable TCP_FASTOPEN in both the kernel and my Nginx build
        • Since I’m not running any untrusted code (all the daemons running are either default system packages or programs I built myself), I disable Spectre/Meltdown mitigations for a perf boost.
        • (overkill) I channeled the learnings from my Gentoo days of yore to build a statically-linked Nginx from source along with with mimalloc, OpenSSL, PCRE, zlib-ng, and musl (with mimalloc it wasn’t noticeably slower) using gcc optimizations (-march, -flto if it worked, and several others).
        • Kept page weight extremely light (<40kb without next-gen image formats, home page <5kb).

        All of the above steps managed to nearly double the req/sec my server was able to handle to several thousand; at that point, I’m pretty sure the benchmark was benchmarking itself.

        If you notice a DDoS, try enabling Fail2Ban with a rule for rapidly-repeating requests. Enabling Fail2Ban with such a rule for a couple months combined with Nginx’s built-in DDoS mitigation has helped my servers weather a couple attacks so far.

        Recommended readings:

        Performance tuning outside application-level improvements is one hell of a rabbit hole.

        1. 2

          Just wanted to add that I’ve since revised my stance on disabling Spectre/Meltdown mitigations; if you’re using an affected processor, you should leave then enabled unless you’re benchmarking or using a disposable VM.

        2. 8

          Am I alone in feeling frustrated that botnets are ubiquitous in the modern Internet but very little seems to be being done to combat them? Are botnet takedowns not well publicised, or is it simply too much effort for it to be economical? Perhaps someone with experience in the area can enlighten me.

          1. 14

            Author here : you are not alone. This is the first time I have had to actually do anything but any server is continually being bombarded with obviously malicious traffic. In this case, I am not sure what the botnet is even trying to achieve but CloudFlare tells me that they are still out there averaging about 1000 hits per hour.

            I sometimes see hand-wringing articles on why the hobby website seems to be dying out. Constant maintenance in the face of persistent attacks is one reason.

            1. 4

              Big mood. My website (christine.website) gets like 150 GB of traffic per month and Cloudflare only really makes me send out about 50 GB of that. Most of it is poorly configured RSS readers and scraper bots that don’t respect robots.txt. Huge pain. My gitea instance had to have Russia and China blocked at the Cloudflare level to avoid it pegging a core constantly. It constantly oomed my Kubernetes cluster back when I hosted things on it.

              1. 1

                My gitea instance had to have Russia and China blocked at the Cloudflare

                Life already sucks for people stuck in Russia and China, and then people in the West ban them from their websites. From my experience, botnets are more or less evenly distributed in the big picture. I’d prefer people to not discriminate against millions of legitimate users just because at the moment the botnet distribution is (or seems) skewed.

                That’s especially bad for people in China who cannot setup a VPN due to the “great firewall”.

                In our project, we have a number of contributors from China. I can’t imagine just telling them: “your country is so full of botnets that it makes your participation not worth it, go f*ck yourself”.

                1. 2

                  Believe me, I didn’t do this as a first measure. I blocked user agent after user agent, throttled things with nginx rules but they kept scraping every single visible link on my git server. I just gave up and blocked the whole country until I could figure out a better way to do it. Maybe now that it’s been blocked for long enough the scraper bots will have given up trying to index my git server and I can re-enable it to Russia/China. The country of the IP address was the only common factor.

              2. 1

                Also don’t forget that cloudflare protection for your website is for free, try securing your minecraft/voIP/other realtime stuff/non-http speaking server without investing money..

              3. 3

                What’s being done to combat them is moving more of the Internet under control of centralized corporations like CloudFlare. There is understandable discontent with that, but it is also not surprising given our political-economic trajectory.

                Solving the problem in a satisfying or elegant way would not allow companies like CloudFlare to skim money off that top. And it’s not just CloudFlare: Big Tech in general benefits from the lack of a standardized distributed solution.

                1. 2

                  Takedowns tend to be publicized pretty well when they happen, so that probably supports your point that they don’t happen often enough. They are difficult to do, both technically and legally. There’s an understandably high bar for exploiting software running on computers within your borders, for example.

                  Of course, there’s also a many billion dollar AV industry that should prevent such malware in theory. Or network appliances that again help in theory. But these don’t seem to protect the little people all that well.

                  1. 9

                    The problem is humans.

                    It would not be difficult for CloudFlare, Akamai, Fastly, and all the various honeypots in the world to round up the IPs that they have, say, a 50+% confidence are involved in a botnet and send a report to the WHOIS-listed owner of that netblock.

                    Then what?

                    Some networks are well-run and will respond quickly. I think this is a minority.

                    Some networks won’t have anyone reading that email. Or they don’t read the language that it was sent in, and it looks just like more spam.

                    Some networks don’t have anyone who is willing to take the responsibility for disconnecting/deauthorizing a client – might not even want to warn the client.

                    It’s the spam problem all over again, but on a much larger scale.

                    1. 4

                      Some networks don’t have anyone who is willing to take the responsibility for disconnecting/deauthorizing a client – might not even want to warn the client

                      But apparently also no one wants to just block them for good until they fix their things. I mean, this is how the big four are doing it with email. They even go so far to just blackhole emails from IPs they don’t like. Try getting removed from microsofts suspicious list, fueled by AI, you won’t get far. There is also a law in germany that makes you personally liable for trash that comes from your home network, they may even disconnect your line.

                2. 5

                  You don’t need to go with Cloudflare, @AndrewStephens, as many hosters offer very reasonably-priced DDoS-protections. I can definitely recommend you Hetzner, it’s the best hoster I know. Don’t pay a company like Cloudflare that is opposing your ideals, as there is always an alternative.

                  Regarding botnets: We shouldn’t change the internet in a way that we prevent botnets from being created (because it would mean a large amount of surveillance and classism). Instead, we should look more at decentralized solutions where content is not hosted on a single server but instead spread peer-to-peer. One prime example for this is IPFS.

                  1. 4

                    Any idea what you did to tick off the owner (or renter, I suppose) of a botnet?

                    1. 4

                      No clue. Maybe they didn’t like my white-hot takes on the Star Wars films, or they scored badly in my Doctor Who quiz. More seriously, as attacks go it is fairly mild (30k hits a day) so I don’t think it is a deliberate DOS - I probably wouldn’t have even noticed except they were triggering my hit counting beacon.

                      I have speculated that perhaps it is an attempt to disguise more malicious traffic by including a bunch of traffic to legitimate sites but who knows?

                      1. 3

                        New Zealand-style apple dumplings, the recipe that launched 30 thousand hits.. every day (those look really good, btw).

                        This reminds me of a “marketing company” I ran into many (at least 15) years ago, where you could just purchase traffic. It was definitely not legit, but they would produce hits. I wonder if some group like that misconfigured their scripts or is using your site for testing.

                        1. 2

                          I think you may be right. It is odd that they appear to be using real browsers and faking traffic for analytics fraud would look very similar. They may be hitting innocent sites in an attempt to evade detection from google analytics by generating enough traffic to look like real humans.

                          If so, the joke is on them - I don’t use google analytics and no real humans read my site.

                    2. 3

                      I’ve been through the same situation. It was relatively worse because it increased the cost of my VM instance and there are currently limits to online spending in my country. So I had to act fast. I did multiple things but the two that worked the best were to add fail2ban rules and to actually find a pattern in the IPs contacting me, to whois them, and contact the abuse email (they actually replied and helped).

                      1. 3

                        I actually think that botnet and their existence is a beauty feature of the internet. The reason botnet existed is due to the amazing of the internet that enable them to operate. Think about it for a second, with all kind of lock down, rate limiting put in place, they existed.

                        The real problem to me is that bandwidth costs is hight nowaday. What if we turn thing around? bandwidth is charged to requester? Say I made a website, if people visit it, they should pay me for my bandwidth cost. no? If somehow the internet operate that way, we will be welcome the botnet :-)

                        1. 2

                          similarly, I’ve had a lot of ideas about making stuff like a collaborative online notepad or web hosting space but every time I think about bots abusing them and I give up on the idea.

                          1. 2

                            Redirect all known offending IPs to a porn site. If it’s a corporate or university network, it makes it more likely someone will notice. Another option is to redirect them to a page that generates as much CPU load as possible, but unfortunately, that will also disrupt users who’ve had their computers attacked. On a plus side, they might finally notice and the botnet would be destroyed, but on a negative side, some ordinary Joe will be freaking out that his computer doesn’t work properly when he has a deadline to meet.