1. 4

    Here’s a little context on Delta and the linked blog post:

    Delta is a format to describe documents’ contents and how it changes over time. This is a core piece of technology at Slab, that powers our real-time collaboration engine, thanks to the built-in support for Operational Transform (think multiple users working together in Google docs).

    Though we’ve been using it internally for almost 4 years now, we’re finally open-sourcing it to the wider Elixir community.

    Feel free to reach out if you have any questions or feedback!

    1. 1

      What kind of networking does the OT use underneath? e.g. are you using some form of epidemic broadcast?

      1. 1

        This library just implements the core OT format and algorithms, and not any networking component.

        Though at Slab, we use this with Phoenix Channels on top of a custom GenServer & Supervisor and it has worked out great for us.

      2. 1

        Do Google docs still use OTs? I thought they moved away from them about a decade ago because they hit the problem everyone hits just after they’ve invested a lot in OTs: that they hit combinatorial problems when you have a lot of operations and they all need to compose.

        1. 2

          AFAIK Google Docs still uses OT, though they have probably moved away from the extremely rich data model supported by Wave early on. A visible artifact of this is that for each doc, there is a server coordinating updates, and with a large number of clients it can get overloaded (“Editing is temporarily disabled”). OT requires a centralized server to handle the propagation of updates.

          CRDTs, conversely, allow a flatter topology in which the server is more like just another client. As you can imagine, having a centralized authority simplifies things, so CRDTs are generally more complex than OTs with equivalent operations, and were for a long time considered infeasible for real-time editing and similar use cases.

          1. 1

            Yes AFAWK they still use OT and never moved for realtime text editing.

            To implement transform for OT you do need to consider all interactions between the different operations so the complexity is n^2 where n is the number of operations (although many combinations may be similar). Google Wave ran into this issue and was a common criticism even amongst their own engineering team. That’s why our design only includes 3 operations: insert, delete, and retain. A tradeoff example is: there is no replace operation. To effectively implement replace you would use insert + delete. There are subtle differences between a replace and a insert+delete but we did not feel they were worth the complexity it would create.

        1. 2

          Working on a follow-up post to this one that dives into building a real-time collaborative editor with Elixir and Delta.

          Also preparing for my talk at ElixirConf ;)

          1. 5

            This obscure little feature in Ecto/Elixir is something we’ve been using at Slab.com for about a year. While the blog post only covers the feature itself, I wanted to give a better overview of WHY we’re using it.

            Slab is a multi-tenant Phoenix app where all tables in our database, including pivot tables, have an org_id foreign key that references the team the data belongs to. Using association defaults enforces that the correct org_id value is automatically set for all resources.

            We combine this with an OrgRepo module that wraps around our default Repo, which can only write or read other resources with the same org_id, ensuring that data for one team has no possibility of accidentally leaking to another team (using Repo is forbidden in our codebase).

            An added bonus is; when we eventually have to scale our databases via something like sharding and have to vertically partition the data based on org_id, we would already have this system in place to support us.

            1. 3

              This is a good blog post. Thanks for writing up how you do this. It’s important for the community to have some of these more advanced use cases documented!

            1. 1

              Just a heads up, the name Memento for a database might be confused with https://mementodatabase.com/ which has been around for several years.

              1. 1

                Oh I didn’t know about that. Memento in my case was just a pun on “amnesia”.

                1. 1

                  Great movie ;) I did think it was interesting you both thought of Memento when it doesn’t seem immediately related to a database.

              1. 4

                I’m using it in an upcoming project! Started with Postgres/Ecto, added an in-memory cache because I need fast access by key (like a hashtable, without socket overhead), decide to switch to an in-memory DB and remove an external server dependency. I’m using dirty reads directly from mnesia though :D

                1. 2

                  Glad to hear that! I also love your HTTPotion package!