Cache failure modes are also interesting because, when cache is used to serve a significant fraction of requests or fraction of data, cache outages or even degradation can easily cause a total outage because an architecture designed with cache performance in mind will not (and should not) have backing DB store performance that’s sufficient to keep the site up.
reminds me of the recent paper about “metastable failures” which puts this failure mode in a more theoretical framework. (I’m not not sure “metastable failure” is a great name; it comes from physics apparently.)
Caching can also make architectures vulnerable to sustained
outages, especially look-aside caching
Assuming a 90% hit-rate and the same database as before,
the web application can now handle 3,000 QPS because only
1 out of 10 user requests result in a database query. However,
loads above 300 QPS are in the vulnerable state since every
request may need to contact the database in the worst case.
If cache contents are lost in the vulnerable state, the database
will be pushed into an overloaded state with elevated latency.
Unfortunately, the cache will remain empty since the web
application is responsible for populating the cache, but its
timeout will cause all queries to be considered as failed. Now
the system is trapped in the metastable failure state: the
low cache hit rate leads to slow database responses, which
prevents filling the cache. In effect, losing a cache with a 90%
hit-rate causes a 10× query amplification.
More generally
Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed.
Fwiw I first heard “metastable” in the context of electronics when discussing flip-flops. :)
Thank you for the second link.
edit: I’m reminded of this old amusing / terrifying story about metastable system which was vulnerable to positive feedback because it couldn’t load-shed. https://rachelbythebay.com/w/2011/06/28/sp/
I think the author made his website like that to work for people with really bad connections. I think he also mentioned somewhere that wiping CSS off his site led to more engagement, but I could be wrong.
+1, a column of text 70-90 characters wide is the most readable option personally, and it requires the least css. One could do it with <br> if one really wants.
This part
Cache failure modes are also interesting because, when cache is used to serve a significant fraction of requests or fraction of data, cache outages or even degradation can easily cause a total outage because an architecture designed with cache performance in mind will not (and should not) have backing DB store performance that’s sufficient to keep the site up.
reminds me of the recent paper about “metastable failures” which puts this failure mode in a more theoretical framework. (I’m not not sure “metastable failure” is a great name; it comes from physics apparently.)
https://lobste.rs/s/2vujcd/metastable_failures_distributed
From the paper:
More generally
Fwiw I first heard “metastable” in the context of electronics when discussing flip-flops. :)
Thank you for the second link.
edit: I’m reminded of this old amusing / terrifying story about metastable system which was vulnerable to positive feedback because it couldn’t load-shed. https://rachelbythebay.com/w/2011/06/28/sp/
Kinda timely considering Twitter is having some kind of outage right now.
(Best of luck to you if you’re involved. #hugops)
Very informative. I was surprised how often GC was implicated as a factor in services’ death spiral.
meta: does a website really have to look like this in 2022? Luckily Firefox has “reader view”.
It works well in mobile at least.
Yeah, it seems just adding
body { max-width: 36em }would be enough for the site to be readable on desktop as well.Heh. As soon as content from this author shows up on HN, there’s a gush of people fawning over the “clean” and “oldschool” look of their pages.
Personally I’d be fine with it if the margins were a bit bigger.
I like “brutalist” webdesign too, but readability should be the focus of published text IMO
As a counterweight, do you like or dislike this recent submission?
http://rachelbythebay.com/w/2022/02/09/nice/
Rachel’s site is unfashionable but fine. Dan’s site is an aggressive nose thumb to the reader. There’s no comparison.
not beautiful but a lot more readable.
I think the author made his website like that to work for people with really bad connections. I think he also mentioned somewhere that wiping CSS off his site led to more engagement, but I could be wrong.
Obligatory http://bettermotherfuckingwebsite.com/ I guess. 7 lines of CSS isn’t going to make a noticeable difference on load times even on dialup.
But yeah, how on earth do people survive without Reader View?
+1, a column of text 70-90 characters wide is the most readable option personally, and it requires the least css. One could do it with <br> if one really wants.
Don’t hard wrap! It will make it look horrible on harrow screens.
max-widthwill do that in a sec.