Is that actually a fix ? If I get this right you’re now crashing for people that try to reclaim disk space but don’t give it enough memory to perform vacuum. Kinda ironic.
If I get this right you’re now crashing for people that try to reclaim disk space but don’t give it enough memory to perform vacuum.
Yeah, I wish I could prevent the crashes entirely, but I think it might be out of my hands. I don’t think I can prevent SQLite from bloating the write-ahead-log on VACUUM when Litestream is attached.
That said, I suspect that in practice, this is unlikely to cause crashes. The users who want to do periodic VACUUM tend to run on full-blown server rather than a RAM-limited VM. If you have several GB of RAM, VACUUM should be fine.
I really enjoyed the article in hindsight to the debugging mechanisms available in go, but well, why not place the files in the regular filesystem as final solution?
why not place the files in the regular filesystem as final solution?
I did consider it because it was a bit of a mess to chase these issues down, but I value the ease of replication too highly. By putting everything into SQLite, I can just run Litestream, and Litestream handles replicatiing and restoring all application state. If I used the regular filesystem, then I have to roll my own solution for backup and restore.
yes,i understand litestream might be a nice solution to sync sqlite databases, but for my understanding, its just simpler to sync filesystem contents, isnt it? Plus for filesystems, its not like keeping snapshots is something like a new technoloy.
I’ve tried using cloud-backed FUSE filesystems in the past (e.g., s3fuse, gcsfuse), and I’ve found them to be much harder to work with than Litestream. They seem like they’re just a regular filesystem, but then I ran into lots of gotchas in places where they can’t fully behave like a local disk. And SQLite doesn’t support networked filesystems, so I’d still have a separate solution for replicating my database
I know there are tools like SyncThing that can sync a folder to cloud storage, but I think integrating SyncThing is more complex than Litestream.
Are there tools you’ve found that would work well here?
Is that actually a fix ? If I get this right you’re now crashing for people that try to reclaim disk space but don’t give it enough memory to perform vacuum. Kinda ironic.
Thanks for reading!
Yeah, I wish I could prevent the crashes entirely, but I think it might be out of my hands. I don’t think I can prevent SQLite from bloating the write-ahead-log on
VACUUMwhen Litestream is attached.That said, I suspect that in practice, this is unlikely to cause crashes. The users who want to do periodic
VACUUMtend to run on full-blown server rather than a RAM-limited VM. If you have several GB of RAM,VACUUMshould be fine.I really enjoyed the article in hindsight to the debugging mechanisms available in go, but well, why not place the files in the regular filesystem as final solution?
Thanks for reading!
I did consider it because it was a bit of a mess to chase these issues down, but I value the ease of replication too highly. By putting everything into SQLite, I can just run Litestream, and Litestream handles replicatiing and restoring all application state. If I used the regular filesystem, then I have to roll my own solution for backup and restore.
yes,i understand litestream might be a nice solution to sync sqlite databases, but for my understanding, its just simpler to sync filesystem contents, isnt it? Plus for filesystems, its not like keeping snapshots is something like a new technoloy.
I’ve tried using cloud-backed FUSE filesystems in the past (e.g., s3fuse, gcsfuse), and I’ve found them to be much harder to work with than Litestream. They seem like they’re just a regular filesystem, but then I ran into lots of gotchas in places where they can’t fully behave like a local disk. And SQLite doesn’t support networked filesystems, so I’d still have a separate solution for replicating my database
I know there are tools like SyncThing that can sync a folder to cloud storage, but I think integrating SyncThing is more complex than Litestream.
Are there tools you’ve found that would work well here?