This is really good - I appreciate the in-depth nature of it!
I remain skeptical about the practicality of event sourcing, but it helps a lot that rather than handwaving over the details of archiving and event schemas, you’ve actually gone into detail about how you do it.
What I’m still curious about (and most articles/books on event sourcing don’t discuss much) is:
Thanks for the kind feedback!
There are definitely tradeoffs to doing an event-sourced system I wrote this post up mostly about our experience overall, and trying to give enough detail about the system itself to make it understandable. From a business perspective it has enabled so much stuff to be so easy that we’ve found the tradeoffs to mostly be skewed in the direction of higher productivity and faster feature delivery. We’ll write more about specifics in future posts (not just by me). There is a whole analytics and BI component here that I didn’t even mention.
To answer your questions above:
Hope that helps answer some of the questions. We have been really practical about it and solve problems as they arrive, using an iterative approach rather than building a lot of support up front. We have a business to run first! :) Generally Protobuf, S3, Spark, and Athena mean we haven’t seen anything frightening yet. Quite the opposite: the tooling seems more than up to the job ahead.
Thanks so much for getting back to me, that’s really impressive :)