1. 1

    HTML5-based apps, as opposed to J2ME blobs or whatevertheheck the featurephone standard was, is cool. My reluctance to downgrade to an un-smartphone mostly stems from Maps and Signal* being crucial for me, and this design decision could be a nice way of getting the best of both worlds.

    • I mean, come on, who would pass up the choice to have E2E chat on the Matrix Phone? :)
    1. 3

      Nice overview of the concurrent map implementation, and it’s cool to see Lobsters used in a benchmark like this!

      But, does the final benchmark strike anyone else as a bit unfair? Comparing the author’s in-memory database to MySQL and “unnamed other database”, the former and presumably the latter of which will persist writes to disk, doesn’t seem like an apples-to-apples comparison IMHO. Also, the presenter’s comment about “we ran these exact queries” suggests that they didn’t opt in to having a materialized view on the legacy system tests[1]. It might have been nice to compare against another in-memory system like Hekaton or a dataflow system like the aforementioned Naiad in order to better understand where the impressive speedup comes from, since there are a lot of variables in play here.

      Basically, as a systems builder, I’m having trouble teasing out what the lessons are here: is it the contribution here most strongly the concurrent map data structure, the by-default view materialisation, or simply that it isn’t fsync()ing the writes to disk like the syscall’s going out of style?

      [1] edit: per alynpost’s comment, it appears they did, so mea culpa.

      1. 4

        Hi Nathan! I’m the presenter/author. The comparison is actually with the databases running entirely in-memory, with all persistence turned off, and with transactional guarantees turned as low as we can get them. We also compare against plain memcached, which approximates a cache system with no processing overheads. The paper also has a comparison with Naiad in a distributed setting.

        This talk was intended to be a high-level overview, not a deep technical dive into the underlying research of Noria, which is why you feel like it’s light on details. If you’re looking for the research contributions, I think you’d find the paper the best place to look! The paper is about the parts of Noria that are actually novel, namely partially-stateful, dynamic data-flow, as opposed to the relatively high-level ideas of materialized views :)

        1. 1

          Hi Jon! Thanks again for the talk and for the clarification; that seems pretty reasonable. (I didn’t mean for the post to be a criticism; sorry if it was read that way! I thought the talk was good and was curious about particulars, and didn’t feel it was light on details. Will add the paper to the reading queue!)

          1. 2

            Haha, not at all, I understood what you meant :) If you’re interested in these kinds of systems, I think and hope you’ll find the paper a good read. Feel free to reach out if you have more questions as you read it!

      1. 4

        I usually have one non-fiction and one fiction work going concurrently. At present:

        The Power Broker - Nominally an autobiography about the New York urban planner Robert Moses but really about how power accumulates and consolidates, and how it can curdle even the most idealistic. Even at over 1000 pages it’s a cracking read; it’s un-put-downable in a way that I imagine marathoning House Of Cards to be.

        A Fire Upon The Deep - found a copy in a neighbourhood lending library and figured it was time for a re-read. Bit of a slow start but once it gets going it’s great; if you’re into space opera and also pretty hilarious satires of ‘90s USENET then you’ll feel right at home with this one.

        1. 2

          I love The Power Broker. I read it my first year of university in a public policy seminar and it just blew me away. The LBJ books are also great, but nothing like as earth-shaking as the Moses one.

          1. 3

            It would have been awesome to read this with a group! Reading it solo, I feel as if the only outlet I have to talk about it is to subconsciously turn all conversations towards the book, e.g. “…oh, so that reminds me of something in The Power Broker, when…” ;)

        1. 2

          Always nice to see these sort of comprehensive overviews, since I always feel like the space is a bit overwhelming to know where to start learning.

          One question, not necessarily related to the material at hand, but something that stuck out at me:

          Soundness prevents false negatives, i.e., all possible unsafe inputs are guaranteed to be found, while completeness prevents false positives, i.e., input values deemed unsafe are actually unsafe.

          Did anyone else learn these definitions as switched from the above? In my education (and in informal usage of the terms), “sound” meant “if you’re given an answer, it is actually valid” whereas “complete” meant “if it’s valid, it’ll be guaranteed to be given as an answer” (e.g. so certain logic programmings might be sound but not complete), which is the opposite. Do different sub-disciplines use these terms in the other way that I learned it? (Or, did I learn it incorrectly?)

          1. 2

            Sorry for the late reply, this week has been trying!

            Wikipedia says it better than I will:

            In mathematical logic, a logical system has the soundness property if and only if every formula that can be proved in the system is logically valid with respect to the semantics of the system.

            “Complete” is a bit more complex, but basically something is complete when you can use it to derive every formula within the system. There are slight differences based one what completeness element you’re discussing, such as complete formal languages vs complete logical semantics.

            I don’t think you learnt it incorrectly, just probably focused on the area you were learning. Wrt the section outlined, the difference there is that you can either detect all possibly unsafe inputs (they are guaranteed to be logically valid for the domain and thus possibly unsafe) OR you can ensure that everything found is actually unsafe (i.e. it actually expresses the nature of “un-safety” to a particular program’s semantics).

            Does that make more sense? It’s quite early here and I’m still ingesting caffeine, so I apologize if not…

          1. 6

            Picked this up on Early Access and it’s lots of fun. Personally the learning curve seems less steep as compared to other Zachtronics games, so if you got frustrated early with others, or haven’t tried any before, this is a good entrypoint. Presentation-wise, it’s top-notch. The pixel art graphics are crisp, the music is great as always, and the varied mission types make the gameplay more varied than previous Zachtronics releases.

            The puzzling is compelling enough to keep me playing, but the plot is unfortunately pretty underwhelming on this one so far. Hopefully that’s something that’s still being worked on, or that it turns a corner towards the end of the game.

            1. 3

              Let’s be honest these games are not about plot, they’re programming games. I really enjoyed TIS-100, didn’t like SpaceChem or Opus Magnum at all though. This one looks good: more like the former than the latter - actually involving writing code.

              1. 5

                Sure, nobody plays puzzle games for the plot, but the designers put one in, and IMHO feels half-baked enough to detract from the overall experience. The [rot13]Bar unpx, bar qeht qbfr[/rot13] thing is set up as if it’s a core mechanic but never really mentioned after the beginning, characters float in and out, and nothing I’m doing (hacking, battling, conversation choices in cutscenes, etc) feels like it has any tension or consequence beyond [rot13]nzhfvat RZORE-2 naq cbffvoyl pnhfvat fbzr yvarf bs pung va na VEP punaary fbzrjurer[/rot13], leaving me feel weirdly isolated in the world. (That feeling of disquieting solitude played to TIS-100’s strengths, given where that game goes, but I don’t think it works here.)

                edit: to be clear the game is good and everyone should buy this game

            1. 1

              I would add that repeating yourself, in some cases, is also essential in performance critical software, not just a matter of avoiding wrong abstraction or avoiding an ugly architecture design. I’m writing a painting application (like Photoshop, Krita, GIMP), and I could use the DRY philosophy on the function to plot the brushes, but I would have a serious performance drop if I did that, because the if/else abstraction would happen in the middle of a rasterization:

              void plot(...)
              {
              	while (row < bottom) {
              		while (col < right) {
              			if (plot.hardness == 100) {
              				/* Use simple plot */
              			} else {
              				/* Use plot with smoothness */
              			}
              		}
              	}
              }
              

              Now, imagine this with multiple parameters (hardness, roughness, density, blending, …). Instead, I copy-paste the function and rewrite with the specific algorithm inside of the nested loop.

              1. 2

                You could also assign the drawing function to a function pointer (or lambda or whatever) outside of the loop and just call that within the loop. No branching in the loop, and no duplicated code.

                1. 3

                  But an indirect call (e.g. calling into a function pointer or a vtable, etc) is still a “branch” - it’s just one with an unknown number of known targets instead of two, which only adds more variables to the equation.

                  In order for this to be equivalent to inlining the duplicate-for-each-algorithm code, one would have to convince oneself that the indirect branch predictor on the processor is going to reliably guess the CALL and RET targets, that the calling convention doesn’t spill more registers than the inlined execution would (ideally it’s a leaf function so the compiler can elide call prologue/epilogues), and that the processor’s speculative execution system doesn’t have its memory dependency information invalidated by the presence of the call.

                  Caveat - the above might be less true if you’re programming in a managed runtime - if that function call can be inlined by the JIT compiler at runtime (many high-performance runtimes are very aggressive about function inliing, so it’s not an unrealistic thing to expect), then hopefully the above issues would be lessened.

                2. 2

                  If you get a chance, there’s a chapter in Beautiful Code that talks about runtime code generation for image processing that, IIRC, singles out stencling and plotting as a running example, that you might find relevant to your interests.

                  1. 2

                    Thanks, I will make sure to check it out!

                1. 3

                  I’ve often wondered about things like the connection between dependent typing and values in template declarations, so it’s nice to see an FP-oriented introduction to the topic.

                  1. 2

                    Would cubicles be better? That’s one cost efficient way to turn an office to a non-open office, I think 🤔

                    1. 2

                      Cubes combine the distracting background noise of an open plan office with having to sit by yourself :/

                      1. 2

                        Cubicles are far better than open offices in my opinion.

                        1. 3

                          Agreed - while headphones can at least block out audible distractions in an open-office plan, there’s nothing to be done for people shooting hoops and reenacting last night’s Warriors game, or people trying to circumvent my “do not disturb” notice on Slack by waving their hands in front of my monitor (both actual examples at $OLDJOB). Visual distractions, at least for me, are just as bad!

                          1. 1

                            Not really. Noise is way more disturbing if you can’t see it’s source, apparently.

                        1. 3

                          This is a great window into software jobs of the past. As much as I’m enjoying the 6502 assembly gore, what’s even more interesting are the quotidian posts, be it about bay area traffic (plus ca change), rooms for rent ($350/mo…cheap!), or the Great “What Do You Mean I Can’t Smoke In My Cubicle” Flamewar Of 1988.