Lobsters recently had posts on flatbuffers and SBE. Less recently, we also had a post on cap'n proto but it didn’t get any love.
On Kenton Varda’s point about SBE probably being a little faster by removing random access and variable-length integers, he pointed out that a lot of Martin Thompson’s focus with SBE was removing branch mispredicts on an HN thread, and that’s probably where the speed difference will come from.
It would be great if Cap'n Proto came out of beta and had support for a few more languages.
Are there any languages you’re particularly interested in?
Not the GP but I’m particularly interested in Scala. I’d want an idiomatic API and no JNI, which would probably be a lot of work.
I’m currently working on a pure Java implementation, and I’m in fact writing the tests in Scala, which seems to work quite well.
For Scala code that uses capnproto-java, the main non-idiomatic thing I’ve hit is that unions don’t get to take advantage of sum types and pattern matching. I’m by no means a Scala expert, though, so I’d be interested to hear about any other potential impedance mismatches.
How does building up messages work? For idiomatic Scala I’d expect an immutable COW style, which gives you an awkward choice; either instantiating an object instantiates the serialized version (which means serializing copies of all the partial instances) or you only create the serialized version once everything’s been built (which means an extra copy stage). Maybe some kind of effect style is the right way to go, building up a monad that knows how to create the serialized instance and then running it to actually serialize? I can’t imagine you could do that in Java though.
Reading is easier as I’d expect to read immutable messages in Scala.
For an example of building messages in Java, see the writeAddressBook() function here.
In Scala, type inference would allow this to be a bit less verbose.
You’re right that implementing copy-on-write would probably be tricky. Cap'n Proto works best if you’re writing to a message just once.
I do suspect that there are some clever things you could do in Scala to make things more ergonomic. It’d be fun to investigate the possibilities at some point.