1. 87
  1. 16

    Yes, yes, yes! SQLite is the choice to make for databases. Like especially with Litestream, you get to treat your database as one treats their filesystem (which is what you end up having to rely on anyway). And it becomes “infinitely” storable since you can use object storage with Litestream. So excited to see them embrace this and eager to learn from them.

    1. 9

      such as being able to run our entire test suite locally and hermetically quickly and easily, ideally without VMs or containers

      For postgres in python there’s https://pypi.org/project/testing.postgresql/ which sets up ephemeral test instances. It’s very convenient and I’m pretty sure they could implement that in any language they’re using. Startup time for me is 1-2s, so really nothing if you’re running integration tests.

      1. 1

        This is how we set up the test suites for Splunk’s SDKs. It’s a really great way to work.

      2. 8

        Why not just postgres?

        1. 7

          The claim is that they don’t want to do the operations work that requires. At this point it sounds like they’ve spent more time hacking random things together than they would have on just running PostgreSQL. But they also only have one giant table, apparently, so I think there may be deeper issues. I mean, if you only need a kv store, why not just start with a kv store like Tokyo Cabinet?

          1. 5

            If you read the article, it explains that they actually don’t want to keep using just a key-value store style, they want to migrate to more complex data and in fact are in the process of doing so.

            1. 3

              Long term support and viability, I’ve never heard of any projects using TC.

              1. 1

                Considering what they’ve been doing wrapping etcd and the like, I’m not sure it’s that big a concern to them. Plus TokyoCabinet (or KyotoCabinet now) have been around longer than etcd, I think. They’re just pretty stable.

                1. 1

                  TC is a building block, but using it directly you have to be careful. I’ve been bit by a corrupt TC more than once

            2. 3

              I consider this the most important question at the start of any system architecture discussion and it had better have an extremely detailed answer with considerable evidence. But I don’t work at an internet company

              1. 3

                I mean this in a pretty judge-y way but only for gossip, but this story and the previous one sound like a case of the CTO wanting to play with their Legos.

                Even if they want to move to a more complex data structure later, dismissing “just buy a managed SQL server with the money that companies give you every month” because of hand-wave-y ops work (it’s tailscale! They have money to hire someone who is competent at databases) feels really odd.

                1. 3

                  Except as they’ve said, they don’t want to deal with the operational overhead of Postgres or MySQL. They have people who could deal with it but they don’t want to. They’re a fairly small company so they aren’t going to hire someone just to deal with making sure a DB is replicating properly. They seem to only have one writer process and maybe some distributed readers? Litestream with the upcoming read replicas seems like the perfect fit for their use case.

                  1. 3

                    Sorry, you’re right that their current SQLite setup is very reasonable. This is more about their bespoke etcd model.

                    The operational efforts of a DB exist but if they have one writer process and some distributed readers, it’s a known quantity and is … mostly a solved problem? Importantly, it’s also a solved problem in the “pay another company some money to deal with it for you” variety in my opinion.

                    They wrote a custom etcd client because they didn’t want to set up a primary/secondary replica setup (and it’s not like those other solutions don’t also have the same operational concerns about backups and the like!)

                    There are operational concerns about any database, nice thing about most of the classic relational DBs is that there are documented, “boring” solutions.

            3. 7

              Hell yeah

              I think SQLite is very applicable to lots of use-cases and it seems to get less consideration than I think it deserves (although it does get plenty of attention!)

              1. 9

                SQLite JSON functions are also a fairly understated superpower!

                1. 2

                  Yes they are, I’ve managed to leverage them and implement a PostgREST compatible api over SQLite [1], i.e. SQLite itself is building the response payload.

                  [1] https://github.com/subzerocloud/blue-steel

                    1. 1

                      Oops, I meant the first one. Fixed!

                    2. 1

                      If you’re on 3.38 you also get the funny arrow operators :)

                  1. 4

                    I’ve had great success moving from one file to … multiple files. You can even use subdirectories!

                    1. 4

                      One database that seems to fit their requirements (except perhaps, for the familiarity part) is FoundationDB. Specifically, it has various modes of replication as part of the initial design (rather than a bolted-on afterthought). I’ve been reading the documentation and it looks very sensible (in particular, limitations are prominently and thoroughly documented). But I haven’t used it myself yet. Does anyone have any experience with FoundationDB that they can share?

                      1. 2

                        The features look nice, but the ops is supposedly very involved. They’re trying to completely avoid having DB ops in the traditional sense.

                      2. 1

                        Wow! I never knew sqllite and litestream was so amazing. Is there way to move Postgres SQL DB to SqlLite?

                        1. 7

                          Only if you’re using very few features of postgres. Any decent orm should be able to pipe from one store to the other. There’s also https://github.com/benbjohnson/postlite but it’s very alpha.

                          1. 4

                            If you’re low traffic enough for this to make sense, you can “just” dump your Postgres DB, and then import the dump into a SQLite DB after some light text processing. Add an audit log in Postgres if you have a bunch of data and will need to do incremental catch-ups for a low downtime cutover. How many writes/s does your Postgres instance handle?

                            1. 2

                              If you’re already on postgres I’m curious what you would gain by switching? You would lose a lot of features.

                              1. 7

                                a whole bunch of operational overhead

                                1. 1

                                  I’ve never found postgres to have overhead. Especially if you’re going to set up litestream anyway…

                                  1. 3

                                    simply running a postgres server is overhead

                                    yes, this is minimized a lot by using a database service from a cloud provider, but even then, assuming you make absolutely 0 tweaks to your postgres server… you still have much more to do than even just litestreaming to a bucket

                                    1. 1

                                      really? apt install postgres and then forget… ?

                                      1. 4

                                        At the very least you have to create some users and set permissions. You will also want to tweak the config a little (especially the memory settings and autovacuum). However, I would put this on the level of effort of litestreaming to a bucket. The real pain comes when you need to do some migrations…

                                        1. 3

                                          Backups, authentication management, and any form of replication all require work and upkeep.

                                          1. 1

                                            Tailscale is not a single backend server, it’s a tool that runs on users’ devices. It’s a really good idea to keep complexity low and not have to manage a separate Postgres process on every device, or even one single point of failure on a backend somewhere.

                                            SQLite is the smart choice here.

                                            1. 1

                                              If they’re talking about the client and not the service then that actually makes sense, but I doubt they’re running litestream on client devices…