1. 17

  2. 10

    Works on Arm & Packet have been generous enough to let me use a ThunderX machine with 96 threads for testing sled, and it has been a super interesting experience so far. When you start to get into these really high thread counts, you must really start to treat your programs like distributed systems in terms of how they communicate across sockets and NUMA nodes. If you can architect your system such that your software threads are pinned to sockets, and each socket maintains its own ring buffers for receiving messages from other sockets, and never sharing memory other than to send messages to other socket ring buffers, it can be a really nice approach.

    When a thread on a particular NUMA node accesses a memory page for the first time and causes it to be faulted in, it will be assigned based on a “first-touch” policy to that particular NUMA node, and if any other threads ever want to access it, they need to proxy access through that node. If an atomic operation happens on a thread running on a particular socket, all other sockets across the same and other NUMA nodes must coordinate to the extent that they share memory (and a little higher because hardware relies on all kinds of approximations for what it means to share memory so it can save space). This is the primary motivation behind partitioning threads across sockets and leaning on message passing as much as possible, where the message ends up being written to memory that was first touched and now owned by the receiving NUMA node.

    One nice thing about testing on ARM machines is that the memory model is weaker than x86, so bugs relating to incorrect memory orderings on atomic operations will pop out during testing more often.

    1. 2

      Woah, sled looks really cool! I’ve only had a few minutes to look at the page, but does it just do key/value, or does it have a SQL (or SQL-ish) query engine on top of it?

      1. 3

        It’s a KV store that can easily support SQL-like operations when combined with a 0-deserialization library like zerocopy. See this example for ways to work with things in a more structured way.

        I want it to be the storage engine behind different SQL (and other) databases and stateful applications, rather than cornering its use cases by imposing a specific query language etc…