Perhaps update the URL to https://sr.ht/~amirouche/Babelia/ ? At least on mobile, the current link doesn’t show the project description - just a list of other repos owned by your user. Then I visited the deployed website, also didn’t understand what I was looking at, before finally finding the “summary” link.
I want people to dive into the code, and have glimpse of the conversational interface at https://babelia.one to inspire other project. The python codebase contains a couple of tricks ;)
SQLite LSM extension: why not vanilla SQLite with/without write-ahead log? Did you see performance wins using the LSM extension compared to vanilla SQLite? I looked for docs for the extension and found this page which seems to be draft quality from the now defunct SQLite4 project, rather than the usual high quality docs for the rest of the SQLite project.
FoundationDB: I’ve heard this is a complex system that’s a bear to operate. Are you running a cluster yet? What has your experience been like?
Re SQLite LSM extension, SQLite LSM extension is in ext directory of sqlite3 tree, the interface is very simple lsm.h. No I did no benchmarks comparing sqlite vs. lsm extension. I intend to move to an okvs interface ontop of sqlite or dqlite if I pursue a rpi4 cluster deployement route; fdb is too much memory hungry (default is 8GB of RAM per core), and still does not support musl.
FoundationDB: no I do not operate a cluster.
The python sample implementation rely on my asyncio bindings for foundationdb, I tried to optimize the best I could without re-implementing the wire protocol, and it is still slow, I believe it is because of the GIL but I have no proof yet. Async helps a lot. It is around 100x slower than sphinx search, but it is not a fair comparison, because it is easier to tweak the scoring, and scale the database size with foundationdb and python.
Perhaps update the URL to https://sr.ht/~amirouche/Babelia/ ? At least on mobile, the current link doesn’t show the project description - just a list of other repos owned by your user. Then I visited the deployed website, also didn’t understand what I was looking at, before finally finding the “summary” link.
I want people to dive into the code, and have glimpse of the conversational interface at https://babelia.one to inspire other project. The python codebase contains a couple of tricks ;)
I’m curious about the choice of databases:
SQLite LSM extension: why not vanilla SQLite with/without write-ahead log? Did you see performance wins using the LSM extension compared to vanilla SQLite? I looked for docs for the extension and found this page which seems to be draft quality from the now defunct SQLite4 project, rather than the usual high quality docs for the rest of the SQLite project.
FoundationDB: I’ve heard this is a complex system that’s a bear to operate. Are you running a cluster yet? What has your experience been like?
lsm.h. No I did no benchmarks comparing sqlite vs. lsm extension. I intend to move to an okvs interface ontop of sqlite or dqlite if I pursue a rpi4 cluster deployement route; fdb is too much memory hungry (default is 8GB of RAM per core), and still does not support musl.The python sample implementation rely on my asyncio bindings for foundationdb, I tried to optimize the best I could without re-implementing the wire protocol, and it is still slow, I believe it is because of the GIL but I have no proof yet. Async helps a lot. It is around 100x slower than sphinx search, but it is not a fair comparison, because it is easier to tweak the scoring, and scale the database size with foundationdb and python.