1. 13
  1. 5

    This is, as ever, great stuff.

    I just wanna nitpick the latency figure, which pictures 100ns as a square and 150ms as … 42 squares. When it ought to be 1,500,000 squares. Not sure what they were thinking.

    1. 1

      Thank you, and you are correct. I probably should ad units to the squares, but definitely an oversight in the illustration.

    2. 5

      I deal with teams who look after collections of services (microservices, data stores, functions, queues, etc.) and when they only have to deal with synchronous calls to and from http endpoints, all is well. Most of the team members, however, have no distributed systems experience. Some have vague recollections of what they were taught while in full time education, but that’s about it.

      This means that it’s hard to design anything where this kind of experience is required, and the system ends up being ‘distributed’ only in the sense that there are separately deployed components with boundaries between in the form of either requiring a JSON-structured call over http, or extremely simple event passing, for e.g. non-urgent email sending, where queues are added for reliability.

      Anyone have experience of moving beyond this to teams getting comfortable with more distributed systems concepts? I’m making progress but it’s slow!

      1. 5

        Anyone have experience of moving beyond this to teams getting comfortable with more distributed systems concepts?

        Honestly? I haven’t seen it happen.

        I think the big problem is the unlearning you have to do. Most programmers, when learning, build a mental model of successive steps of state transition. When you start doing distributed systems you have to unlearn this and find a different mental model, where you think in terms of final conditions you’re aiming for or invariants you’re maintaining, and then chain preconditions backwards from there. It’s almost like unification in Prolog. I will have the mental model of how the system will flow in time with the invariant or goal, and then I start trying to generate values for the holes in it the way Prolog would in its search.

        Most people, unless forced by suffering, will not unlearn their original model. I joke that most of the issues in database consistency are carved into my flesh as scars at this point, and I know that I arrived at the final condition/invariant point of view in dark moments when that final condition or invariant was my way out of urgent problems, which rather focused on the mind on them.

        1. 3

          I am also dealing with the same thing to some extent, its can be tricky finding the time for learning experiences in real systems.

          I think an internal talk with some notable examples could be a good way forward. Internal tech notes also are good.

          1. 1

            Thanks yes that does sound like the best route. For those who can’t get the hands-on experience yet, showing them some examples based on real-world experience with our own experience could be the closest to their mental models.

        2. 2

          Short little post about Fallacies of Distributed Systems. These are some thoughts I feel often get downplayed when engineers are designing microservices.

          1. 1

            These are spot on. Thanks for posting.