1. 36
  1.  

  2. 9

    Good article, although I didn’t quite follow why saying “exactly-once” is wrong.

    I do want to call out this one bit:

    Distributed systems typically handle large scale datasets (otherwise you would be running a single instance of PostgreSQL right?).

    This is true of centralized distributed systems, the stuff that people use in servers and data centers. Not necessarily true of decentralized, aka peer-to-peer, systems, where the size of the data is irrelevant, and being distributed is a desirable feature in its own right.

    For example if my family uses a hypothetical P2P app as our private social network it might have only a few megabytes of text (and a few hundred bigger JPEG and MPEG blobs) but being distributed means nobody else can get their hands on our data or serve us ads, and we don’t need to configure a server.

    I say this because distributed computing has over the past 20 years become more and more server-centric, and it’s good to keep in mind the bigger picture. Systems designed for servers tend to make assumptions that don’t work for P2P, since availability, latency, discoverability and trust are so much less of a problem.

    1. 3

      You know, now that you pointed it out, I don’t think you even have to go to decentralized systems. Maybe I only have a couple of 10s of gigabytes of data, but they’re being read very often, or maybe I just really need availability for that dataset, even if it’s small, or maybe it’s a compliance thing. In any case, I might want to run multiple replicas of my system even with a small dataset.

      1. 1

        I didn’t quite follow why saying “exactly-once” is wrong.

        For reference, the key difference is between exactly one processing and exactly one delivery. For example, you can’t guarantee that that a process is delivered exactly once (eg: if the process being delivered to fails; the message service can’t tell if that counts as delivered or not). But as the article says, you can process the message idempotently so multiple deliveries behave the same way as a single delivery.

      2. 4

        This is such a good piece. I would add the negative shibboleth of “trust is upstream” because I hear that pretty often right before an overly trusting component fails.

        1. 2
          1. 1

            Great question, it def could! I was referring more to buggy peers (clients, servers, etc.) sending either bad messages or messages badly. Your point is exceptional, however.

        2. 3

          Crash-tolerant software is indeed a virtue, but crash-only software often tends to encourage programming practices that make applications fiendishly difficult to model. If every error is a terminal error then you kind of opt out of deterministic control flow.

          1. 1

            crash-only software often tends to encourage programming practices that make applications fiendishly difficult to model.

            Can you expand on that a bit?

            If every error is a terminal error then you kind of opt out of deterministic control flow.

            Well, there’s a bit more depth to it than that. For example, within the lifecycle of an individual request, experiencing a transient error (such as a timeout) might be fatal to that request, but not to the web server as a whole. Or for example, if your message queue consumer loses it’s connection; then you’d usually only restart the consumer, rather than the process as a whole.

            1. 1

              Is that what crash only means? All errors are crashes? My understanding was more that there was no happy/sad path when you terminated, a normal exit is indistinguishable from an abrupt one due to a crash, so any recovery happens in the startup path (and is constantly exercised).

              1. 6

                Going by Wikipedia definition:

                Crash-only software refers to computer programs that handle failures by simply restarting, without attempting any sophisticated recovery.

                The argument usually is: sophisticated (precise) error recovery is hard, and if you’re attempting it, you already have a potentially-broken program in an inconsistent state. Throwing it all away and starting from a blank state again is easier, well-tested, and therefore more robust.

                Take for example an out-of-memory error: if an allocation fails, you can either carefully backtrack the current operation and report the problem to the caller, or just abort() the whole program.

                I generally agree with the approach, but it’s not a silver bullet. Crash-only systems are prone to DoS-ing themselves. A persistent bad input can put the whole system in a crash loop, instead of being carefully skipped over. Even when everything technically works (and makes incremental progress as it should), the restart may be expensive/disruptive (e.g. loses caches or performs extra work on startup) and the system can still fail due to insufficient throughput.

                1. 1

                  In a crash-only service, if an incoming request encounters a database invariant violation, does the entire process die? Or if the database connection fails, are all in-flight requests abandoned?

                  Designing software so that it can start up and recover from a wide variety of prior states is a good idea in general, but it’s structurally impossible to write a program that can crash at any point during its execution and reliably leave underlying resources in a recoverable state. Any nontrivial DB transaction commit, for example, is a multi-stage operation. Same with network operations.

                  More generally, it’s definitely a good idea to design the layer above the individual process to be resilient, but you can’t just assert that error handling isn’t a concern of the process. The process, the program, is what human beings need to mentally model and understand. That requires deterministic control flow.

                  1. 2

                    but you can’t just assert that error handling isn’t a concern of the process.

                    I agree that is a bad idea, and I would almost say objectively so. Which is why I don’t think it is actually what the idea of “crash-only” is trying to convey.

                    Again, my understanding was that crash-only software wasn’t “we crash on all errors/we don’t care about errors”, but rather “we don’t have shutdown code/the shutdown code isn’t where all the invariants are enforced”. It’s more about not having atexit handlers than not having catch blocks if you will. All program terminations are crashes, not all errors are crashes. If you have no shutdown code do you have to put that code somewhere else (periodic autosaves say, etc.), which means when you do crash you’re much likely to be closer to a recent good state.

                    1. 1

                      I may misunderstand what “crash-only” means. I take “crash” to mean “terminate the operating system process unceremoniously and without unwinding call stacks”, and I understand “only” not to mean all conceivable errors but definitely more than is strictly necessary.

                    2. 1

                      if an incoming request encounters a database invariant violation, does the entire process die? The original paper “Crash Only Software” talks about it in terms of individual components that can perform micro-reboots, and references Erlang heavily.

                      So for an example, you’d want to use tools such as transactions so you can pretend that a multi-step operation is a single one. Alternatively, make everything idempotent, and retry a lot.

                      structurally impossible to write a program that can crash at any point during its execution and reliably leave underlying resources in a recoverable state.

                      I’m reasonably sure that this is what database software should be designed to do. Unless I’m somehow misunderstanding you.

                      1. 1

                        There is no OS-level operation against e.g. disks, networks, etc. which can be interrupted at any arbitrary point, and can be reliably recovered-from. You can do your best to minimize the damage of that interruption – and maybe this is what crash-only is gunning for – but you can’t solve the problem. Every transition between layers of abstractions, between system models, represents an exchange of information over time. No matter how you design your protocol, no matter how tiny the “commit” or “sync” signal is, if your operation spans more than one layer of abstraction, and it’s possible for that operation to be interrupted, you can’t avoid the possibility of ending up in an invalid state. That’s fine! My point is only that systems at each layer of abstraction should not only do their best to recover from wonky initial state, but should also do their best to avoid making that wonky state in the first place. If you encounter an error, you should deal with it responsibly, to the best of your ability. That’s it.

                2. 1

                  Can someone ELI5 why it’s important to distinguish between “exactly once” and “at least once” plus “idempotent processing”?

                  Is there something functionally different about the two? Maybe “exactly once” implies a degree of efficiency that might matter in some applications?

                  Or is it just a way to signal that you have heard of Two Generals and aren’t doing all the easy, reasonable, wrong things that someone like me might try?

                  1. 1

                    Given a system that delivers messages, and a system that receives and processes those messages, “exactly once” means the receiving system doesn’t need to be idempotent. This is important because “idempotent processing” can be nontrivial to achieve in practice.

                    1. 2

                      Thanks, I hadn’t thought of it as a statement about an interface rather than a statement about an implementation detail inside a system.

                  2. 1

                    Does anyone know any good textbooks on this subject?

                    1. 5

                      Designing Data-Intensive Applications tends to be the go-to recommendation here

                      1. 2

                        Second that, plus the comparatively brief Distributed systems for fun and profit as an introduction.