1. 2

    Two things that struck me about this pattern after thinking about it for a while:

    1. How would you compare this to Redux? It seems there are a lot of touch points, except state in React/Redux being ephemeral
    2. Wouldn’t blocking I/O temporarily stop the processing of commands? This is something I’d worry about if I was making eg a bunch of calls to REST APIs. I guess async I/O is a way to handle this, but then you wouldn’t be able to maintain linearity across I/O boundaries. (I guess the part about reactors being able to emit commands sort of hinted at this?)

      Question 1:

      The architecture is very similar to Redux. The reasons for using Redux for a UI vs using MIP for a server-side application might be partly different since the constraints and trade-offs are different, but a lot of the advantages are also similar.

      I’ve actually found it advantageous to combine server-side MIP with a similar pattern client-side. Share the business logic code (the “reducers”) between client and server. When the UI initializes, get the latest state from the server. Send UI actions as commands to the server-side MIP application, and stream accepted commands to the client to update the UI state. Suddenly you have solved both persistence and realtime collaboration with one move.

      If at any point you are dissatisfied with the server round-trip delay for UI updates, you already have the perfect architecture for solving that:

      • Let the client calculate the latest state under the assumption that the commands sent to the server will be accepted, but also save the state implied by the latest command the server has accepted.
      • Backtrack as needed on timeouts or when the server contradicts the assumption.

        Question 2:

        If you make a bunch of calls to REST APIs, you never wait for the replies before processing the next command. Commands need to be deterministic. Emitting commands from reactors is a way to make something indeterministic look deterministic:

        Operations involving calling out to external services become less trivial. Since they cannot be considered deterministic, they can’t be implemented as a processor. Rather, they will have to become reactors, that may or may not produce more commands that are fed back to the application.

        Example: A command is sent to the application to perform a ping against some server. The result of the ping is not deterministic and therefore cannot be used to calculate the next application state, so doing the ping inside a projector is useless. You do it in a reactor, and when you have the result, you emit a command back into the application that basically says “the result of the ping was X”. This result-declaration-command can be used by processors. When the application is restarted, the actual ping is not performed since reactors are ignored by the reprocessor, but the perceived result of the ping, saved in a command, is the same. Doing this asynchronously is perfectly fine; if the external service call takes longer, it just means that the result-declaration-command might arrive later in the command sequence, but determinism is still preserved.

        I believe calling external services was the only reason you’d worry about blocking I/O. Still, let’s address that part:

        Decide what kind of durability guarantees you want, and make a fair comparison between MIP and another option. Processing commands in sequence does not mean that you need to do everything on a single thread. For example, each of the following tasks can run on its own thread:

        • Assign an index number to incoming commands
        • Serialize incoming commands into a log buffer
        • Write the log buffer to file
        • Run projectors/reducers to create the new state
        • Run reactors, possibly on several threads
        • Send replies to clients, possibly on several threads, optionally (depending on what durability guarantees you want) after making sure the corresponding command is persisted.

        Note that the only operation that is I/O-dependent here is writing to disk sequentially. Decades of database performance tuning has had “minimize disk seek operations” as a mantra, and here we have approximately no seeks at all. It’ll run fast. In practice you’re unlikely to need more than one thread. The above multi-thread model is just a way out if you ever find yourself needing more compute-heavy operations that take an amount of time within the same order of magnitude as disk writes.


          Thanks for elaborating! Good point about single-threading not being an explicit requirement, as long as commands are synchronous

      1. 3

        Would “Command sourcing” be a fitting name for it? It looks interesting. What kind of problem would this be a good solution for?

        1. 2

          I think “Command sourcing” is a great name, it brings “Event sourcing” to mind as both analogy and contrast. Perhaps it’s a better name than “Memory Image Pattern”.

          The kind of problems MIP is a good solution for include:

          • Complex business logic
          • Tight deadlines
          • Low latency or realtime requirements
          • Frequently changing requirements that traditionally would result in laborious database schema changes in production
          • Frequently changing requirements that makes it hard to know what kind of data is useful
          • A need to handle historical data and metadata
          • Any combination of the above

          But it’s probably not a good solution if you have any of the following:

          • A compute-intensive or very data-intensive application
          • Requirements for very high throughput
          • Requirements to purge/forget data
          • Very complex integration interfaces
        1. 5

          There’s an old Carmack .plan about this sort of pattern, too: https://github.com/ESWAT/john-carmack-plan-archive/blob/master/by_day/johnc_plan_19981014.txt

          1. 2

            Interesting! I like his point about the progress of time as another form of input. That way, your application becomes deterministic and very debugging-friendly.

            I’ve seen something similar mentioned in the context of either Event Sourcing (ES) or Domain-driven design (DDD): The clock is an external actor to your system. Can’t find a link right now though.

          1. 3

            This seems really naive. It is not trivial to implement low-latency and high-throughput write-ahead logging and checkpointing from scratch. The systems which have managed to do so efficiently are commonly known as “databases”.

            1. 2

              Databases also try to do so while processing several write requests concurrently and remaining scalable. Indeed, this is far from trivial, and traditional RDBMSs have been battle tested and refined over several decades to achieve this.

              If your application’s input rate is low enough that you can accept limited horizontal scalability and lack of concurrency - then write-ahead logging and checkpointing really do become trivial.

            1. 1

              This is a different name from what it used to have in the old Gang of Four book (I know, I know) but I can’t recall it because there were better things to expend neurons on (like the lyrics to Barbie girl for some reason)

              It’s mostly useful to make undo/redo much easier.

              As a state persistence mechanism though it’s really not the great because it results in excessive load and launch times, as well as unbound memory growth.

              The solution to these problems is coalescing commands in memory and flattening state on storage. Both of which get you to what this article is trying to avoid.

              1. 2

                Commands don’t need to be kept in memory after processing, so you get unbounded growth of disk data but not memory use. If disk data growth is unacceptable you can coalesce the command log but indeed then you lose many of the benefits of MIP, as detailed under Command log storage and system prevalence.

                The load/launch times can become an issue but this is more easily addressed without losing the benefits of MIP. Saving snapshots of application state to disk without deleting commands will speed up load/launch, see the heading Startup time.

              1. 1

                This only works if none of your commands have side effects

                1. 1

                  Commands can have side effects, but the logic that causes side effects needs to be separated from the logic that updates the application state. I call these pieces of logic “Reactors” and “Projectors”, respectively. (I should probably have clarified that Projectors must not have side effects.) With this separation it works, because the Reprocessor can rebuild the application state without causing side effects a second time.

                1. 3

                  Very interesting article!

                  Using a WAL log to store commands to the system is really clever. Also, doing serial command processing feels like an elegant way to sidestep a bunch of concurrency issues. Combine the two and you’ve got full ACID compliance!

                  However, one thing I haven’t figured out is how code deploys would work? Some runtimes support hot code upgrades out of the box (Erlang, a bunch of lisps), but what would the process look like for a runtime that needs to be restarted to reload code? Would you create a snapshot (maybe asynchronously?), restart and read the snapshot back?

                  In a traditional setup you’d typically have your application servers take turns to get upgraded so you always have some machine handling incoming traffic, but in this pattern you only have one machine, so while it’s upgrading the system is down I guess?

                  (For a runtime that can boot fast enough it might not cause significant downtime though?)

                  1. 3

                    Great question! You can upgrade the command (write) interface without downtime, using two machines A and B:

                    • A is serving the command interface with the old application logic.
                    • You start up a new machine B with the new application logic. The application data can have a wildly different structure, but the schema for the command log is the same.
                    • B reprocesses the log from the beginning while A continues to serve command requests and write to the log.
                    • You decide on some future command index N where control is to switch from A to B and communicate that to both machines.
                    • Starting at command index N, machine A stops processing the commands and instead proxies them to B.
                    • Approximately immediately thereafter, B will have processed command N-1 from the log and will begin processing incoming commands.

                    For the query (read) interface it’s even easier since you can actually have several machines processing queries and do a normal rolling upgrade.

                  1. 2

                    How does OCR work, AI or automation?

                    1. 2

                      If you believe the thesis of the article, OCR is AI made necessary only by our civilization’s inability to digitize everything, while automation would do away with both papers and the need for OCR.

                      You might be confused because AI is often seen as one kind of automation. I believe the article tries to distinguish them by correctness (without actually using that word). OCR and other “AI” can make mistakes and require human intervention, while “automation” can be proven correct.

                      1. 3

                        I think it’s self evident that a world without humans is easy to manage by shifting data to memory regions.

                        We all know printers are devilspawn.

                        Getting computers to work with us instead of us working with them, that’s the trick, and I’m not sure automation solves that problem. In fact I might be so bold as to say AI gets computers to work with us, and automation helps us work with computers.

                      2. 2

                        It’s a sliding scale. https://en.m.wikipedia.org/wiki/OCR-A

                        1. 1

                          I defined “AI” for the purpose of this as “using the term to mean some heuristic for learning equations from data or simulations rather than writing them out by hand”.

                          So I think OCR fits as AI, unless there’s a way to do OCR without training data.

                          Basically what axelsvensson said.

                          1. 1

                            A while ago I got this problem of the fuzziness of the term “AI” stuck in my head and thought about it a lot. I think “a computer program that takes on the world on its own terms (without the situation being arranged to suit the computer program)” is somewhat accurate. I think this matches your definition of “AI” and the parenthetical fits how you contrast it against “automation”.

                        1. 2

                          I like the concept. Another similarly helpful distinction would be to expose some functions/methods to testing code only, and leave them private otherwise.

                          1. 3

                            FWIW, previous discussions on the subject last year and two years ago.

                            1. 4

                              Replying to the previous comment “I’m curious what lobsters would come up with if given, say, only 5 sentences to describe a reasonable code of ethics”:

                              Most things I could think of can fall under one generalized rule: “Don’t enter into conflicts of interest with your users.” That covers data silos, monopoly abuse, dark patterns, proprietary formats as a way to lock out competition, and addictive gamification. In fact, I would rather use the other four sentences to carve out exceptions to that rule, such as commercialization and limited liability. If your interest is aligned, the remaining rules in the ACM post would rather be considered “good practice” rather than an ethical obligation. They only become relevant in an ethical sense as your interest diverges from the user’s interest.

                              1. 2

                                Interesting, but it is practical? If you carve out an exception for commercialization, then how do you prevent conflict of interest?

                                Maybe many of the ethical rules can be summarized as “Act as if you haven’t entered into conflicts of interest with your users”.

                                1. 1

                                  Interesting, but it is practical? If you carve out an exception for commercialization, then how do you prevent conflict of interest?

                                  I just mean there’s an inherent interest in that your users want a lower price, and you want a higher price. I think as long as you limit this to the general price of the software package, there’s nothing wrong with it. Once you start doing things like asking a high price to export their data, it begins to be skeevy.

                            1. 1

                              langhdk - It’s the fairly reliable result of me attempting to write something random. The overwrite confirmation is a reminder to never rely on mental entropy.

                              1. 4

                                Is there a reason you can’t use Cygwin? I use it for developing on Windows. The terminal is fine, and you can even build standalone windows applications without any Cygwin dependency.

                                Edit: Referring to the mintty terminal and the mingw64-i686-gcc-core package.

                                1. 3

                                  It’s slow though, especially for CPU intensive tasks

                                  1. 2

                                    Came here to ask basically this. I do some Windows development occasionally and just use MSYS2 or whatever with mingw64. Yes it is not fast, but it is usable. I also occasionally just use good old cmd.exe with a Makefile since that is how I started and it’s not like you can’t do that anymore… How does the poster think we wrote software in Windows 95/98 days??

                                    1. 1

                                      Mainly I am concerned about input lag–which was quite high when running gnome-terminal from wsl through vcxsrv–although, it may be better with cygwin since that’s running as a native app–except, maybe not: siblings suggest it’s slow.

                                    1. 6

                                      YAML has a variety of quoting mechanisms for strings. One of them, block scalar, maps quite nicely to the author’s indent quoting, e.g.

                                      helptext: >
                                          Usage: This is the first line
                                              This is the second line
                                          ^--- This whitespace is part of the string.
                                          <--- This whitespace is the current indent.
                                          We can include """, ''', " and ' just as they
                                          are, so we can quote any code this way,
                                          whether python or other.

                                      However, it gets complicated. A full description of the above might be written block scalar folded, with clip chomping. I regularly need to refer back to these, then use some trial and error

                                      1. 1

                                        Yes, this is similar. Maybe the only difference is the handling of line breaks at the end of the string. YAML seems to have three ways to do it, none of which can represent all strings:

                                        • Single newline at end (clip): exactly one newline
                                        • No newline at end (strip): exactly zero newlines
                                        • All newlines from end (keep): one or more newlines

                                        Indent quoting would be able to represent all strings using the same notation:

                                        The end of the string is represented by a newline, followed by an indentation decrease.

                                        In YAML terms, this would probably be expressed as All newlines from end except the last one.

                                      1. 2

                                        This multiple levels of escaping is nonsense, in my opinion, although this is nothing against the author.

                                        I’m disappointed that a better option, my preferred option, wasn’t mentioned: The beginning and ending character can be doubled to provide for a single instance. Follows is an example:


                                        In APL, this evaluates to a string with a single quote. Follows is another example:


                                        In Ada, this is a string containing a single double quote.

                                        Phrased differently, this is my preferred way to consider it, this is the simple idea that a string containing the string character itself can be represented by two strings juxtaposed, with the joining point becoming the string character in a new string that is the combination.

                                        It’s disappointing Common Lisp uses escaping instead of this much nicer way.

                                        1. 4

                                          I too prefer doubling the quote character over backslash + quote, since I personally find it more intuitive and aestetic.

                                          However, this is still escaping, at least in the way I used the term in the article—the quote character does not represent itself but instead is a dispatch character. It will still suffer some downsides of escaping, e.g. copying the text between the start and end quote characters will not put the actual string in your clipboard.

                                          1. 2

                                            The author appears to be aware of this method of escaping quotes:

                                            In PostgreSQL, apostrophe (’) is escaped as two apostrophes (’’).

                                            I first saw this kind of escaping in Plan 9’s rc(1):

                                            A quoted word is a sequence of characters surrounded by single quotes (’). A single quote is represented in a quoted word by a pair of quotes (’’).

                                          1. 4

                                            The primary example would be representing the length of the string followed by the string itself, which is very common in protocols and formats that are mostly read and written by machines. To use this technique in programming languages would mean making humans with no interest in counting symbols do just that, which is so painful that escaping is a better tradeoff.

                                            Someone’s never had to deal with ancient FORTRAN.

                                            1. 2

                                              You mean Hollerith constants in FORTRAN 66?

                                              4HCorr, 4Hect,, 4H I h, 4Had n, 4Hot s, 4Heen , 4Hthis, 4H bef, 4Hore.

                                            1. 1

                                              I think this vision can and should be divided into several sub-problems that can then be solved independently and in several ways.

                                              1. How to represent project data: The author suggests a new protocol using signature chains and point out how they are very similar in concept to git. I suggest they shouldn’t be just similar, but that the protocol to represent project data should be implemented in terms of actual git concepts. Such a protocol could for example make use of special rules for branch names, filenames, commit history structure etc to represent project data. Another thought is that it could be represented as a standardized directory structure with special merge rules, so that it becomes independent from the version control system.

                                              2. How to distribute project data: The author suggests SSB. Even if this could work out, trying to push a standard that is incompatible with existing workflows would probably not succeed very widely. Yes, web bridges are possible, but unless and until such bridges are implemented the use of the protocol would be proportionally hampered. I suggest using existing infrastructure and workflows albeit for new purposes, e.g. opening an issue using a pull request. If the project data is encoded in git concepts, then anything able to distribute git data can also distribute project data. This already includes git..b, http etc. With no change whatsoever, these protocols and platforms would already be compatible. Adding SSB to achieve some proposed higher level of decentralization would be a separate issue that could succeed or fail independently.

                                              3. How to authenticate project data: The author suggests developer signature chains, while I have suggested implementing something like signature chains using git concepts. A signature chain could be implemented as a git branch with signed commits, but authentication of project data could be provided in one of several ways. For example, github could be used as a web bridge if someone is authorized to authenticate data added by *@github users.

                                              What do you think, would this be viable and desirable?