Threads for amirouche

  1. 1

    What is kernel mode setting and DRM?

    1. 2

      By the way of Wikipedia leading paragraphs

      DRM is Direct Rendering Manager:

      The Direct Rendering Manager (DRM) is a subsystem of the Linux kernel responsible for interfacing with GPUs of modern video cards. DRM exposes an API that user-space programs can use to send commands and data to the GPU and perform operations such as configuring the mode setting of the display. DRM was first developed as the kernel-space component of the X Server Direct Rendering Infrastructure, but since then it has been used by other graphic stack alternatives such as Wayland.

      User-space programs can use the DRM API to command the GPU to do hardware-accelerated 3D rendering and video decoding, as well as GPGPU computing.

      https://en.wikipedia.org/wiki/Direct_Rendering_Manager

      KMS is Kernel Mode Setting:

      Mode setting is a software operation that activates a display mode (screen resolution, color depth, and refresh rate) for a computer’s display controller by using VESA BIOS Extensions or UEFI Graphics extensions (on more modern computers).

      The display mode is set by the kernel. In user-space mode-setting (UMS), the display mode is set by a user-space process.

      Kernel mode-setting is more flexible and allows displaying of an error in the case of a fatal system error in the kernel, even when using a user-space display server.

      User-space mode setting would require superuser privileges for direct hardware access, so kernel-based mode setting shuns such requirement for the user-space graphics server.

      https://en.wikipedia.org/wiki/Mode_setting

    1. 2

      Very inspiring project. It shows how a limited set of available features can stir creativity, just in line with the other retro, and fantasy consoles.

        1. 1

          Custom protocol to communicate with DB. It’s same category as rqlite but worst because now you have to implement protocol as well.

        1. 1

          Working on my first take to solve the “micro-libraries” problem !

          1. 3

            Wandering aimlessly.

            1. 2

              Do you know a stone program that allows to compare other benchmark results?

              (Millions of writes per seconds looks like a lot)

              1. 1

                At https://docs.python.org/3/library/ctypes.html?highlight=ctypes#loading-shared-libraries one can read:

                The Python global interpreter lock is released before calling any function exported by these libraries, and reacquired afterwards.

                Cython also has a nogil keyword.

                If we can generalize: any C foreign function call with release the CPython GIL, then one can scale SQLite better than the algorithm that is described in the linked article. A hint is that gunicorn used to recommend for synchronous workers 2 * nproc - 1 threads per CPython interpreter.

                It seems to me Janus dependency is not necessary, async with app.database_write_mutex(): out = await app.transaction(func) may be enough.

                1. 1

                  Thanks for the write mutex tip, I’ll look into that.

                  I actually had some success with the nogil Python getting SQL queries to run in parallel recently - I wrote about that here: https://simonwillison.net/2022/May/6/weeknotes/

                1. 11

                  There’s another package called cdb.

                  1. 1

                    Came here to say this. Hopefully, the more people upvote this, the op will reconsider the name.

                    1. 4

                      I don’t mind changing the name at all! I didn’t expect this project to take off and get noticed by many. On reddit and here, name thing has been brought up already, so I am definitely considering to rename it.

                      1. 2

                        It ack’ed in the README:

                        Name

                        cdb is an acronym for Cask Database, and it is not related to D. J. Bernstein’s cdb. I don’t expect this to get popular or used in production environments, and naming things is hard pleading_face.

                        https://github.com/avinassh/cdb#name

                        1. 6

                          I will never understand the obsession with naming things that aren’t user logins or executable files three characters long. “Cask DB” is a wonderful name for the project.

                          1. 5

                            hey, thank you for the feedback. I wasn’t expecting this to be noticed by many, so I just kept it as cdb. It also refers to an internal joke between me and my partner.

                            I am considering to rename it

                            1. 2

                              I’d encourage you to rename it. cdb is a well-recognized name for djb’s project and since yours is fairly fresh, it’d be a good thing to not create an ambiguity, especially if you are aware of it. If nothing else, it will prevent any discussion of your work to veer immediately to this subject at the expense of the rest. Just like it did here.

                              1. 5

                                If nothing else, it will prevent any discussion of your work to veer immediately to this subject at the expense of the rest. Just like it did here.

                                This is a great point. I have renamed it to py-caskdb and I have also credited your message.

                    1. 1

                      Thanks for sharing. You might be interested in my project called LBST [0] to implement sorted kv.

                      [0] https://github.com/amirouche/lbst

                      1. 2

                        wow, this looks interesting, thanks for sharing! This is much closer to D. J. Bernstein’s constant db.

                      1. 4

                        Perhaps update the URL to https://sr.ht/~amirouche/Babelia/ ? At least on mobile, the current link doesn’t show the project description - just a list of other repos owned by your user. Then I visited the deployed website, also didn’t understand what I was looking at, before finally finding the “summary” link.

                        1. 1

                          I want people to dive into the code, and have glimpse of the conversational interface at https://babelia.one to inspire other project. The python codebase contains a couple of tricks ;)

                        1. 1

                          I’m curious about the choice of databases:

                          • SQLite LSM extension: why not vanilla SQLite with/without write-ahead log? Did you see performance wins using the LSM extension compared to vanilla SQLite? I looked for docs for the extension and found this page which seems to be draft quality from the now defunct SQLite4 project, rather than the usual high quality docs for the rest of the SQLite project.

                          • FoundationDB: I’ve heard this is a complex system that’s a bear to operate. Are you running a cluster yet? What has your experience been like?

                          1. 1
                            • Re SQLite LSM extension, SQLite LSM extension is in ext directory of sqlite3 tree, the interface is very simple lsm.h. No I did no benchmarks comparing sqlite vs. lsm extension. I intend to move to an okvs interface ontop of sqlite or dqlite if I pursue a rpi4 cluster deployement route; fdb is too much memory hungry (default is 8GB of RAM per core), and still does not support musl.
                            • FoundationDB: no I do not operate a cluster.

                            The python sample implementation rely on my asyncio bindings for foundationdb, I tried to optimize the best I could without re-implementing the wire protocol, and it is still slow, I believe it is because of the GIL but I have no proof yet. Async helps a lot. It is around 100x slower than sphinx search, but it is not a fair comparison, because it is easier to tweak the scoring, and scale the database size with foundationdb and python.

                          1. 1

                            1.1 Too Long, Don’t Read

                            We develop tiny SQL-like query engine from scratch in Common Lisp which performs on par with a cheating C++ version and handily outruns even more cheating Go version. […]

                            1. 2

                              Going back to basics: Writing down my project’s foundational goals, then the subgoals that are derived from them, then the strategies they imply. Really thinking hard about what’s truly foundational and what’s just circular reasoning or fancies. This will help me maintain focus and strip out some attractive but nonessential aspects from the protocol.

                              1. 1

                                What is the project?

                                1. 1

                                  A social journaling/blogging protocol, named “Cavern” for now: https://gitlab.com/timmc/cavern

                                  Local-first design, optimized for the simplest possible self-hosting. I’ve tried to design it so that one moderately tech-savvy person can host the journals of hundreds of acquaintances at minimal cost, and without those acquaintances having to trust them. (Hoster can’t read the data.) Hopefully that will thread the needle of self-hosting being the enemy of adoption.

                                  But I need to go back and think about exactly why local-first is important to me, and tease out the different elements of that, so that I can work to enhance those elements in other ways as well.

                              1. 1

                                I have some boilerplate code that allows to control the frontend from the backend without the need to write a line of frontend code. Here is an example REPL-like app, see the mockup to have an idea of how it is rendered. The kernel frontend side is in on.message, the other side of tube, the server is the websocket coroutine. It just works.

                                Did you consider emscripten?

                                1. 2

                                  Yesterday, I removed all css from my blog! This week-end I will continue my work on arew / ruse, a R7RS scheme on top of Chez Scheme.

                                  1. 2

                                    I plan to make a first release of elle-cli, command line transactional safety checker based on Jepsen, Elle and Knossos. I would like to hear any feedback from everyone who involved in testing of distributed systems or DBMS.

                                    1. 1

                                      I like the idea! I am not sure how to use it.

                                    1. 3

                                      /me investigate literate programming (related to reproducible research)

                                      1. 2

                                        I’m relearning React. It’s been two or three years since I last used it seriously, and I want to get comfortable with it once again especially when it comes to new features like Hooks.

                                        1. 1

                                          From a usuability point of view hooks look like a step-back to me. Also, while the previous version was not simple, I could imagine how it works under the hooks, and the upcoming scheduler are daunting. In any case, reactjs and react-query leads to code that is more difficult to read (they do not use async / await for performance reasons). WDYT?

                                          1. 2

                                            To be totally honest, I’m not a big fan of hooks (the little that I’ve used it). It feels like too much magic is going on, and it’s taking a while to wrap my head around it. But I want to keep an open mind and learn it properly since the community seems to largely accepted it as the new default way to write React.

                                            1. 1

                                              Did you change your mind regarding ReactJS and its hooks?

                                        1. 1

                                          I am not sure Lucene has only a vector-model-based approximate matching algorithm. Some one can point me to the relevant code? I was under the impression they also use BM5 / TF-IDF.

                                          Given it is sparse matrix, it seems unlikely to be computable tractable to compute the top-k-nearest neighbor over very large 10k x 10k matrix, where 10k is the count of words, lemme or stem in the corpus. And that without an online algorithm (that can be updated in-place while new documents possibly new words comes in)