1. 12

  2. 2

    The Q&A get’s abruptly cut at the end. :(

    1. 3

      It’s on the Mozilla livestream: https://mzl.la/Rust-Meet-Up-2018-11-13

      That’s a very clunky video link, though, requiring a livestream and an exception for it to create a popup.

      1. 1

        I think that was the last question :-)

      2. 2

        Pretty eager to try this as the storage format for a number of things in sled :]

        1. 3

          As someone who looked into flat buffers a few months ago and tried to determine its key advantages/disadvantages when compared to capnproto, could you perhaps share any insight that you might have? I found it difficult to tease out the differences. One way to make this more concrete, for example, is that I think Rust already has support for capnproto. What has prevented you (either technically or not) from using capnproto in sled?

          1. 4

            Here’s my understanding, please correct me if I’m off about something!

            At the core, capnproto does not use a vtable, but flatbuffers does. capnproto messages include all values, even if they are set to the default value (if the default is all zeroes on the wire, it can be packed down using capnproto’s simple compression scheme). Default values in flatbuffers are not included in individual messages. The lack of the vtable in capnproto makes schema evolution a bit clunky over time if you’re used to protobufs. You can add new numbered fields and change their human-readable names at will. No vtable = no potentially malicious pointer validation.

            Flatbuffers allows you to validate pointer integrity on the vtable, but this does not happen by default. Don’t accept potentially malicious flatbuffers without validating them. They assume friendly input.

            Flatbuffers is probably the better choice if you expect to have lots of fields, possibly many having the default value.

            Flatbuffers is a little more flexible, feels more comfortable coming from protobufs. Capnproto is simpler and you don’t have to be as mindful of security issues. Flatbuffers may have far better spatial performance if you have a lot of nonzero default values.

            Flatbuffers lets you use structs for simple types if you want to avoid vtable hops, but you lose a lot of flexibility. Flatbuffers (in other languages, not finished yet in rust AFAIK) lets you mutate fields, as they are backed by a Vec which can grow. Capnproto is immutable.

            sled: why flatbuffers and not capnproto? The system is changing very quickly still. I want forward compatible storage files. I am comforted by the flexibility of flatbuffers. To be honest there is not a specific case I can imagine needing flatbuffer’s flexibility for over capnproto’s ability to add numbered fields. But that’s the problem with forward compatibility, it’s hard to predict. I don’t anticipate the vtables in flatbuffers to be much of an issue, and where they are, I can migrate to flatbuffer structs. I mitigate the malicious vtable issue because I treat all data from the disk as malicious and checksum all segments and individual messages. But for me it boils down to flexibility and “good enough” performance.

            1. 2

              Thank you so much for that wonderfully thoughtful reply! I haven’t used capnproto or flat buffers in anger yet, but I might in the future, and want to be prepared to make the right decision. Thanks!

              1. 2

                Flatbuffers (in other languages, not finished yet in rust AFAIK) lets you mutate fields, as they are backed by a Vec which can grow.

                Quick clarification: in-place mutation is supported in the cases where the payload already has space for the value in question. So, if you want to change the value of a stored u32, that will be supported. But, if the field’s value data is missing (indicating a default or optional value), then you won’t be able to modify the field in-place.

              2. 1

                Ill add Cap n Proto is more secure than many other options.