1. 36
  1. 13

    This part

    Cache failure modes are also interesting because, when cache is used to serve a significant fraction of requests or fraction of data, cache outages or even degradation can easily cause a total outage because an architecture designed with cache performance in mind will not (and should not) have backing DB store performance that’s sufficient to keep the site up.

    reminds me of the recent paper about “metastable failures” which puts this failure mode in a more theoretical framework. (I’m not not sure “metastable failure” is a great name; it comes from physics apparently.)


    From the paper:

    Caching can also make architectures vulnerable to sustained outages, especially look-aside caching

    Assuming a 90% hit-rate and the same database as before, the web application can now handle 3,000 QPS because only 1 out of 10 user requests result in a database query. However, loads above 300 QPS are in the vulnerable state since every request may need to contact the database in the worst case. If cache contents are lost in the vulnerable state, the database will be pushed into an overloaded state with elevated latency. Unfortunately, the cache will remain empty since the web application is responsible for populating the cache, but its timeout will cause all queries to be considered as failed. Now the system is trapped in the metastable failure state: the low cache hit rate leads to slow database responses, which prevents filling the cache. In effect, losing a cache with a 90% hit-rate causes a 10× query amplification.

    More generally

    Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed.

    1. 4

      Fwiw I first heard “metastable” in the context of electronics when discussing flip-flops. :)

      Thank you for the second link.

      edit: I’m reminded of this old amusing / terrifying story about metastable system which was vulnerable to positive feedback because it couldn’t load-shed. https://rachelbythebay.com/w/2011/06/28/sp/

    2. 2

      Kinda timely considering Twitter is having some kind of outage right now.

      (Best of luck to you if you’re involved. #hugops)

      1. 2

        Very informative. I was surprised how often GC was implicated as a factor in services’ death spiral.

        1. 1

          meta: does a website really have to look like this in 2022? Luckily Firefox has “reader view”.

          1. 5

            It works well in mobile at least.

            1. 2

              Yeah, it seems just adding body { max-width: 36em } would be enough for the site to be readable on desktop as well.

            2. 4

              Heh. As soon as content from this author shows up on HN, there’s a gush of people fawning over the “clean” and “oldschool” look of their pages.

              Personally I’d be fine with it if the margins were a bit bigger.

              1. 6

                I like “brutalist” webdesign too, but readability should be the focus of published text IMO

                1. 2

                  As a counterweight, do you like or dislike this recent submission?


                  1. 6

                    Rachel’s site is unfashionable but fine. Dan’s site is an aggressive nose thumb to the reader. There’s no comparison.

                    1. 4

                      not beautiful but a lot more readable.

                2. 3

                  I think the author made his website like that to work for people with really bad connections. I think he also mentioned somewhere that wiping CSS off his site led to more engagement, but I could be wrong.

                  1. 6

                    Obligatory http://bettermotherfuckingwebsite.com/ I guess. 7 lines of CSS isn’t going to make a noticeable difference on load times even on dialup.

                    But yeah, how on earth do people survive without Reader View?

                  2. 1

                    +1, a column of text 70-90 characters wide is the most readable option personally, and it requires the least css. One could do it with <br> if one really wants.

                    1. 3

                      Don’t hard wrap! It will make it look horrible on harrow screens. max-width will do that in a sec.