1. 2
  1. 3

    What is the big cost that flat buffer is paying that Karmen is not? Just trying to understand where the performance gains are coming from, at what cost, and if they scale.

    1. 1

      Flatbuffers is known to have one more indirection to enable smooth schema evolution (i.e. you can mark fields as deprecated, these won’t show up in both generated code, and as encoded payload, doesn’t pay penalty for these fields). This seems just dumps fields into a memory buffer without that extra layer of indirection. For a well-tuned language / generator, it can perform well, but you lose:

      1. Cannot remove fields, only append new fields;
      2. Having a lot of empty optional fields doesn’t automatically translates to size saving (flatbuffers indirection allows default / optional fields).

      I am also not sure how they handle schema evolution for table contains table cases (table A contains table B, when table B appending more fields resulting in table A’ and table B’, how can you downgrade from A’ to A?).