This was a good technical talk about some cutting edge database stuff and a practical case study for Rust.
@jonhoo why did you do the MySQL protocol as opposed to Postgres - because MySQL and memcache are a common pairing or due to the protocol itself?
lobste.rs is mentioned at 42:17 in this video, and here is the source code for the presented benchmark.
Nice overview of the concurrent map implementation, and it’s cool to see Lobsters used in a benchmark like this!
But, does the final benchmark strike anyone else as a bit unfair? Comparing the author’s in-memory database to MySQL and “unnamed other database”, the former and presumably the latter of which will persist writes to disk, doesn’t seem like an apples-to-apples comparison IMHO. Also, the presenter’s comment about “we ran these exact queries” suggests that they didn’t opt in to having a materialized view on the legacy system tests. It might have been nice to compare against another in-memory system like Hekaton or a dataflow system like the aforementioned Naiad in order to better understand where the impressive speedup comes from, since there are a lot of variables in play here.
Basically, as a systems builder, I’m having trouble teasing out what the lessons are here: is it the contribution here most strongly the concurrent map data structure, the by-default view materialisation, or simply that it isn’t fsync()ing the writes to disk like the syscall’s going out of style?
 edit: per alynpost’s comment, it appears they did, so mea culpa.
Hi Nathan! I’m the presenter/author. The comparison is actually with the databases running entirely in-memory, with all persistence turned off, and with transactional guarantees turned as low as we can get them. We also compare against plain memcached, which approximates a cache system with no processing overheads. The paper also has a comparison with Naiad in a distributed setting.
This talk was intended to be a high-level overview, not a deep technical dive into the underlying research of Noria, which is why you feel like it’s light on details. If you’re looking for the research contributions, I think you’d find the paper the best place to look! The paper is about the parts of Noria that are actually novel, namely partially-stateful, dynamic data-flow, as opposed to the relatively high-level ideas of materialized views :)
Hi Jon! Thanks again for the talk and for the clarification; that seems pretty reasonable. (I didn’t mean for the post to be a criticism; sorry if it was read that way! I thought the talk was good and was curious about particulars, and didn’t feel it was light on details. Will add the paper to the reading queue!)
Haha, not at all, I understood what you meant :) If you’re interested in these kinds of systems, I think and hope you’ll find the paper a good read. Feel free to reach out if you have more questions as you read it!
Also on Hacker News and Reddit.