Every 5-7 years we find a new place where SQLite can shine. It is a testament to the engineering and API that it powers our mobile apps (Core Data)/OSes, desktop apps (too many examples), and, eventually app servers, be they traditional monoliths (Litestream) or newer serverless variants, like what’s described here.
I also see a trend where we’re starting to question if all the ops-y stuff is really needed for every scale of app.
I’m in the same bubble, reading about LiteStream,
Fly.io and Tailscale. And I really love what they are doing in the SQLite ecosystem. But I don’t really understand how CloudFlare is using SQLite here. It’s not clear if SQLite is used as a library linked to the Worker runtime, which is the usual way to use it, or if is running in another server process, in which case it’s closer to the traditional client-server approach of PostgreSQL or MySQL.
Yeah this post is very low on technical detail, and I can’t seem to find any documentation about the platform yet - I guess once things open up in June we’ll know more.
Definitely keen to see if they are building something similar to Litestream, it seems like a model that makes sense for SQLite; a single writer with the WAL replicated to all readers in real time.
I’m trying to convince people at work that using a replicated SQLite database for our system instead of a read only PostgreSQL instance would make our lives a lot better, but sadly we don’t have the resources to make that change.
I guess CloudFlare D1 is based on CloudFlare Durable Objects, a kind of KV database accessible through a JavaScript API. They probably implemented a SQLite VFS driver mapping to Durable Objects (not sure how they mapped the file semantics to KV semantics though). If I understand correctly, Durable Objects is already replicated, which means they don’t need to replicate the WAL like Litestream.
I think there’s probably a marketing/tech trend right now for cloud vendors (fly.io, cloudflare) to push for this technology because it’s unfamiliar enough to most devs to be cool and, more importantly, it probably plays directly to the vendors’ strengths (maintaining these solutions is probably much easier than, say, running farms of Postgres or whatever at scale and competing against AWS or Azure).
If it’s any consolation, in another five or ten years people will probably rediscover features of bigger, more fuller-featured databases and sell them back to us as some new thing.
(FWIW, I’ve thought SQLite was cool back in the “SQLite is a replacement for fopen()” days. It’s great tech and a great codebase.)
Litestream author here. I think SQLite has trended recently because more folks are trying to push data to the edge and it can be difficult & expensive to do with traditional client/server databases.
There’s also a growing diversity of types of software and the trade-offs can change dramatically depending on what you’re trying to build. Postgres is great and there’s a large segment of the software industry where that is the right choice. But there’s also a growing segment of applications that can make better use of the trade-offs that SQLite makes.
Dr. Hipp has been referring to SQLite as “serverless” for a while. Maybe the cloud providers stole the branding. :) https://www.sqlite.org/serverless.html
D1 is built on SQLite. Not only is SQLite the most ubiquitous database in the world, used by billions of devices a day, it’s also the first ever serverless database. Surprised? SQLite was so ahead of its time, it dubbed itself “serverless” before the term gained connotation with cloud services, and originally meant literally “not involving a server”.
Is it my bubble or is sqlite everywhere lately?
Every 5-7 years we find a new place where SQLite can shine. It is a testament to the engineering and API that it powers our mobile apps (Core Data)/OSes, desktop apps (too many examples), and, eventually app servers, be they traditional monoliths (Litestream) or newer serverless variants, like what’s described here.
I also see a trend where we’re starting to question if all the ops-y stuff is really needed for every scale of app.
I’m in the same bubble, reading about LiteStream, Fly.io and Tailscale. And I really love what they are doing in the SQLite ecosystem. But I don’t really understand how CloudFlare is using SQLite here. It’s not clear if SQLite is used as a library linked to the Worker runtime, which is the usual way to use it, or if is running in another server process, in which case it’s closer to the traditional client-server approach of PostgreSQL or MySQL.
Yeah this post is very low on technical detail, and I can’t seem to find any documentation about the platform yet - I guess once things open up in June we’ll know more.
Definitely keen to see if they are building something similar to Litestream, it seems like a model that makes sense for SQLite; a single writer with the WAL replicated to all readers in real time.
I’m trying to convince people at work that using a replicated SQLite database for our system instead of a read only PostgreSQL instance would make our lives a lot better, but sadly we don’t have the resources to make that change.
I guess CloudFlare D1 is based on CloudFlare Durable Objects, a kind of KV database accessible through a JavaScript API. They probably implemented a SQLite VFS driver mapping to Durable Objects (not sure how they mapped the file semantics to KV semantics though). If I understand correctly, Durable Objects is already replicated, which means they don’t need to replicate the WAL like Litestream.
I think there’s probably a marketing/tech trend right now for cloud vendors (fly.io, cloudflare) to push for this technology because it’s unfamiliar enough to most devs to be cool and, more importantly, it probably plays directly to the vendors’ strengths (maintaining these solutions is probably much easier than, say, running farms of Postgres or whatever at scale and competing against AWS or Azure).
If it’s any consolation, in another five or ten years people will probably rediscover features of bigger, more fuller-featured databases and sell them back to us as some new thing.
(FWIW, I’ve thought SQLite was cool back in the “SQLite is a replacement for fopen()” days. It’s great tech and a great codebase.)
Litestream author here. I think SQLite has trended recently because more folks are trying to push data to the edge and it can be difficult & expensive to do with traditional client/server databases.
There’s also a growing diversity of types of software and the trade-offs can change dramatically depending on what you’re trying to build. Postgres is great and there’s a large segment of the software industry where that is the right choice. But there’s also a growing segment of applications that can make better use of the trade-offs that SQLite makes.
It’s funny how SQLite is described as serverless only to match the narative of cloud providers.
Dr. Hipp has been referring to SQLite as “serverless” for a while. Maybe the cloud providers stole the branding. :) https://www.sqlite.org/serverless.html
Cloudflare does address that in the announcement:
— https://blog.cloudflare.com/introducing-d1/#meet-d1-the-database-designed-for-cloudflare-workers