See also https://news.ycombinator.com/item?id=4164991.
See also Cap'n Proto RPC (http://kentonv.github.io/capnproto/rpc.html), which implements an elegant distributed object system. Cap'n Proto deals what this post calls “level 1” problems by providing promise pipelining; RPC actions are asynchronous and get reified as events and wrapped as promises, allowing the Cap'n Proto runtime to optimize away unnecessary network communication, and giving the programmer a simple, honest abstraction.
(Full disclosure: I’m a contributor on the Rust implementation of Cap'n Proto.)
Since I want better control over epoll, I’m interesting in packing more than RPC. I originally went down the path of trying to use protobufs, then msgpack, then cap'n proto for a C app I’m building (not C++). I ended up running into tons of issues with stable C library support with all of them. Currently I’m using XDR (using rpcgen as a base) and it seems to be the best so far out of what I’ve tried. Another one that recently caught my eye is MQTT, but I’m not familiar with it yet.
If you’re writing in C++ (or Rust perhaps?), you’ll probably have a different experience.
What level would it be to use CSP or temporal logic to model the distributed system? I know you can use model checkers to find flaws in the design, but they work by searching the program state space instead of doing proofs, and so can only randomly find errors.
One of the papers related to Chubby (Paxos made live I think?) has a section about testing their implementation. At one point, they mention that their test suite was running for days before they found a rare bug in their implementation.
Level 4 on this list should probably be something about testing your implementation. Level 5 should be the creation of new algorithms (and making liveness and safety proofs for them). Level 6 should be complete mastery (or some other level which I’m not aware of yet).
To be really complete, something about using logs and basic database design should be included somewhere (system r, aries, etc).