This is such a clever extension of that brilliant hack to get WASM SQLite working against big static files using HTTP Range headers from a few months ago.
Really interesting performance numbers too, about how much better it performs than IndexedDB even though it wraps IndexedDB. Reminds me of https://www.sqlite.org/fasterthanfs.html
It would be really interesting to combine these and use the IndexedDB back end for offline editing and local storage and something that uses HTTP range requests for server-side persistence.
it once was but the problem is that there was no proper spec written - it was basically “whatever SQLite supports” which in turn was implemented by just linking against SQLite which in turn meant that there was only one actual implementation of the feature, but in order to become properly standardised, two independent implementations need to exist.
This issue came up in the middle of the NoSQL craze and so it was decided rather than to fully specify SQLite and rewrite it at least two times, it was easier and more with the times to just offer IndexedDB as its replacement.
Indeed. Nobody wanted to write an actual spec of all SQL query features and semantics of a database in a way that pretends to be a real independent standard.
“Just do whatever SQLite does” had a high risk of sites relying on every quirk and bug of a specific version of SQLite, so soon it would be impossible to upgrade it. At that time sites being “bug-compatible” with IE6 and IE6 only were still a thing. And SQLite has lots of quirky features (very lax type parser, dynamically typed storage, rowid, and so on).
My experience using sql.js was not awesome. It runs out of memory so quickly when you do naive queries like SELECT * FROM VALUES (...), ... where the VALUES are a bunch of JavaScript objects you want to inject in to be able to perform SQL on.
I just wanted to be able to do basic grouping, filtering, joining on data in memory using SQL syntax. But after a few hundred rows sql.js would always run out of memory.
I switched to alasql which can handle the 80MB of data I was trying to work with. It’s not as complete as SQLite but at least the basics work.
There must be more intelligent ways to use sql.js that work around the memory limits I had. They just weren’t obvious to me.
Yes that’s true. My point is that I wasn’t able to just use it on its own very easily. There are clearly other people using it correctly/more efficiently than I was able to figure out.
This is such a clever extension of that brilliant hack to get WASM SQLite working against big static files using HTTP Range headers from a few months ago.
Really interesting performance numbers too, about how much better it performs than IndexedDB even though it wraps IndexedDB. Reminds me of https://www.sqlite.org/fasterthanfs.html
It would be really interesting to combine these and use the IndexedDB back end for offline editing and local storage and something that uses HTTP range requests for server-side persistence.
This is so cool yet so sad. Why isn’t access to SQLite a web standard? It’s already included in every OS and browser under the sun.
it once was but the problem is that there was no proper spec written - it was basically “whatever SQLite supports” which in turn was implemented by just linking against SQLite which in turn meant that there was only one actual implementation of the feature, but in order to become properly standardised, two independent implementations need to exist.
This issue came up in the middle of the NoSQL craze and so it was decided rather than to fully specify SQLite and rewrite it at least two times, it was easier and more with the times to just offer IndexedDB as its replacement.
Indeed. Nobody wanted to write an actual spec of all SQL query features and semantics of a database in a way that pretends to be a real independent standard.
“Just do whatever SQLite does” had a high risk of sites relying on every quirk and bug of a specific version of SQLite, so soon it would be impossible to upgrade it. At that time sites being “bug-compatible” with IE6 and IE6 only were still a thing. And SQLite has lots of quirky features (very lax type parser, dynamically typed storage, rowid, and so on).
My experience using sql.js was not awesome. It runs out of memory so quickly when you do naive queries like
SELECT * FROM VALUES (...), ...where the VALUES are a bunch of JavaScript objects you want to inject in to be able to perform SQL on.I just wanted to be able to do basic grouping, filtering, joining on data in memory using SQL syntax. But after a few hundred rows sql.js would always run out of memory.
I switched to alasql which can handle the 80MB of data I was trying to work with. It’s not as complete as SQLite but at least the basics work.
There must be more intelligent ways to use sql.js that work around the memory limits I had. They just weren’t obvious to me.
The whole post is about not running sql.js with the memory backend thanks to James’ work 😅
Yes that’s true. My point is that I wasn’t able to just use it on its own very easily. There are clearly other people using it correctly/more efficiently than I was able to figure out.
Super happy someone made this! Really cool.
Not sure what I thought this would be, but it’s pretty cool, tbh. I’m a big SQLite fan, and can see this being helpful for a specific kind of project.