Short version: compile SQLite using emscripten, turn the page size way down (4KB -> 1KB) and use the VFS layer to request only the necessary bits of the DB. Very clever, worth the read.
Not sure what’s the impact on latency of turning page size from 4k to 1k. The http and tls overhead would be much bigger and I am assuming few kilobytes doesn’t introduce significant load. I did something similar by accessing sqlite db over fuse/http. it’s less cooler than the author’s approach, but being more useful in non-web application.
The ultimate solution could be adding vfs-curl to sqlite itself, so we don’t need to hack our own copy.
Not sure what’s the impact on latency of turning page size from 4k to 1k.
Already addressed in the implementation and the article:
I implemented a pre-fetching system that tries to detect access patterns through three separate virtual read heads and exponentially increases the request size for sequential reads. This means that index scans or table scans reading more than a few KiB of data will only cause a number of requests that is logarithmic in the total byte length of the scan.
I’d love to have a website search solution that works in static sites, but also doesn’t use a lot of bandwidth. This is almost that, but it looks like the WASM file is still pretty hefty (400KB) which balances out the efficient retrieval of results.
The WASM file is pretty hefty, but it isn’t data dependent. That means you can probably avoid fetching it until the user needs it and can set its cache policy to something huge so that it’s a 400KB download that’s amortized over repeated visits in a year. It would be nice if this could be stored somewhere public and shared between multiple sites.
Once I have stumbled upon an old project by mozilla that integrated something similar natively into browser. It was called something like websqliteview, can’t find it anymore.
The first generation web local-database API literally just gave JS a SQLite db and let it run queries against it. Maybe that’s what you’re thinking of?
(It was soon deprecated and hopefully no longer exists. The first problem was that when the SQLite devs found out about this, they pointed out that SQLite had never been designed to handle malicious SQL, since it was assumed the queries were baked into the app. Thus, all sorts of vulnerabilities existed.)
Short version: compile SQLite using emscripten, turn the page size way down (4KB -> 1KB) and use the VFS layer to request only the necessary bits of the DB. Very clever, worth the read.
Not sure what’s the impact on latency of turning page size from 4k to 1k. The http and tls overhead would be much bigger and I am assuming few kilobytes doesn’t introduce significant load. I did something similar by accessing sqlite db over fuse/http. it’s less cooler than the author’s approach, but being more useful in non-web application.
The ultimate solution could be adding vfs-curl to sqlite itself, so we don’t need to hack our own copy.
Already addressed in the implementation and the article:
I’d love to have a website search solution that works in static sites, but also doesn’t use a lot of bandwidth. This is almost that, but it looks like the WASM file is still pretty hefty (400KB) which balances out the efficient retrieval of results.
There’s https://endler.dev/2019/tinysearch/, for instance.
Yeah, the search there is pretty limited, and (unlike this) it downloads whole index. And stork downloads whole index and has large WASM code.
The WASM file is pretty hefty, but it isn’t data dependent. That means you can probably avoid fetching it until the user needs it and can set its cache policy to something huge so that it’s a 400KB download that’s amortized over repeated visits in a year. It would be nice if this could be stored somewhere public and shared between multiple sites.
Once I have stumbled upon an old project by mozilla that integrated something similar natively into browser. It was called something like websqliteview, can’t find it anymore.
The first generation web local-database API literally just gave JS a SQLite db and let it run queries against it. Maybe that’s what you’re thinking of?
(It was soon deprecated and hopefully no longer exists. The first problem was that when the SQLite devs found out about this, they pointed out that SQLite had never been designed to handle malicious SQL, since it was assumed the queries were baked into the app. Thus, all sorts of vulnerabilities existed.)