Architecture Notes is a brand new publication run by Mahdi Yusuf. This is the first interview published on the site, talking to me about the architectural design of my Datasette Python web application. It goes pretty deep into a bunch of topics: ASGI, plugin design, using SQLite with asyncio and bundling a Python web application up as both an Electron app and a WebAssembly client-side application.
Thanks for posting! I remember trying out Datasette Lite when it was announced and reading the project docs back then, but this interview offers a bunch more detail and tells a great story explaining the project’s rationale, too.
The Python global interpreter lock is released before calling any function exported by these libraries, and reacquired afterwards.
Cython also has a nogil keyword.
If we can generalize: any C foreign function call with release the CPython GIL, then one can scale SQLite better than the algorithm that is described in the linked article. A hint is that gunicorn used to recommend for synchronous workers 2 * nproc - 1 threads per CPython interpreter.
It seems to me Janus dependency is not necessary, async with app.database_write_mutex(): out = await app.transaction(func) may be enough.
Architecture Notes is a brand new publication run by Mahdi Yusuf. This is the first interview published on the site, talking to me about the architectural design of my Datasette Python web application. It goes pretty deep into a bunch of topics: ASGI, plugin design, using SQLite with asyncio and bundling a Python web application up as both an Electron app and a WebAssembly client-side application.
Thanks for posting! I remember trying out Datasette Lite when it was announced and reading the project docs back then, but this interview offers a bunch more detail and tells a great story explaining the project’s rationale, too.
At https://docs.python.org/3/library/ctypes.html?highlight=ctypes#loading-shared-libraries one can read:
Cython also has a
nogilkeyword.If we can generalize: any C foreign function call with release the CPython GIL, then one can scale SQLite better than the algorithm that is described in the linked article. A hint is that gunicorn used to recommend for synchronous workers
2 * nproc - 1threads per CPython interpreter.It seems to me Janus dependency is not necessary,
async with app.database_write_mutex(): out = await app.transaction(func)may be enough.Thanks for the write mutex tip, I’ll look into that.
I actually had some success with the nogil Python getting SQL queries to run in parallel recently - I wrote about that here: https://simonwillison.net/2022/May/6/weeknotes/