It seems like people who get a lot of experience in scaling one day think, “It shouldn’t really be this hard, people shouldn’t have to make this many mistakes, people should be able to learn from my mistakes.” This seems reasonable, except that your blog is generally not a good place to put this information. I wonder if there is a better delivery system for something like this, maybe a wiki of best practices for scaling large systems, that delineates the different schools of thought, and what the existing tools and frameworks are for doing this kind of thing. As an example of a good starting resource is this link, posted on lobsters earlier, although it is not easily searchable, despite being organized by topic.
Any other thoughts on a good way to organize all of this shared knowledge? I hate to say “crowdsourcing” because I associate crowdsourcing with non-expert knowledge, and this is definitely expert knowledge, but this is crowdsourced in the same way that wikipedia is crowdsourced.
I think the right form of communication about this is a text book. But before we get there we also need to systemize our knowledge into a unified theory, “architecture patterns,” etc. While I think crowd sourcing is good for enumerating knowledge, I think we need a centralized effort to systemize it (see for example how IEEE S&P’s is soliciting “Systematization of Knowledge” papers for computer security).
The problem as I see it is that it usually Academia who develops such systematizations, but scalability tends to be a mostly industrial problem that academics are uninterested in (the only exception I can think of is “big data”).