This looks really neat but the whole “you have to have a bug-for-bug identical-with-node JSON serialization routine” for the signature stuff has always rubbed me the wrong way. Have there been any non-node implementations out in the wild yet? Last I looked the reference implementation was the only one you could actually use.
Personally, I’d use bencode. It was literally designed for use in P2P protocols, and it’s length-prefixed, so you can pass arbitrary stuff in it without having to escape anything. And there’s already good implementations of it in Node (because BitTorrent uses it).
The advantage over bencode is that there’s an attractive advanced representation, which is what I wrote in my post, rather than only the wire representation. The bencode version would, I think, look like this:
I wonder if the ease of reading problems couldn’t be fixed with a better text editor, making the debates between most of these essentially equivalent formats ‘moot’.
This could be kept in messagepack form, and just shown to the developer as json or s-exp or whatever you want in your favorite text editor. If anything the necessity for “human readable” files as plaintext or some specific format if anything an expression of the limitations of our tooling, as well as a lack of standards of encoding one format in another. If there were a standardized encoding between formats you could simply use the encoding that worked for you, and translate any implementation trivially.
It’s an interesting idea, but different formats have different styles. E.g. JSON tends to like objects, which are unordered maps; canonical S-expressions tend to be built on ordered lists, often of key-value pairs; other formats have their own styles.
I know, right? These folks are all like “boats do not have always-on internet connections” and I’m all like “but they have quad-core CPUs and double-digit megabits of memory?”
The other big thing I’ve noticed is that while it’s perfectly fine with ssb for your connection to be down most of the time, heaven forbid that your connection be metered. Initial sync upon joining a pub or following anyone can potentially be hundreds of megabytes – make sure it doesn’t happen on mobile data.
I hear you. I must point out that a gigabyte isn’t tens of gigabits and actually a quad-core phone would cost more like $50-70, right? I dunno, I haven’t bought a phone in a while.
It should be clear what I want though, right? Something that isn’t node.js or electron. Something that feels like weechat or mutt, or even keybase’s CLI. (Though, it is a performance hog, the background processes are.. maybe node.js? Something heavy.)
I love the overall concept, but Scuttlebutt seems to completely ignore all existing protocols for RPC, serialization, LAN discovery, cluster gossip, message encoding, etc. It reinvents a whole lot of wheels all at once. “But it’s simple! It’s just JSON!” they might say. But the difficulty isn’t in the syntax, it’s in the semantics.
Unless I missed something, local sync requires visibility of peers’ UDP broadcasts on a shared LAN. Nearly all public hotspots and home guest LANs disable peer visibility, so this seems like a nonstarter. BLE beacons could be an interesting alternative.
I’d rather use a version of ActivityPub or Solid (https://solid.mit.edu) that works truly offline, such as with BLE + Wi-Fi Direct.
Everyone’s (rightfully) talking about NodeJS, but I think this is an awesome walk through of a protocol. I feel like I’m often reverse engineering some crazy nasty stuff when I’ve had to write a protocol from scratch or learn it blackbox. I’ve always felt like protocol definitions are awesome, but don’t always help you understand the whole flow in a real setting. This does an extremely good job to help visualize the protocol in real function. I might start adopting a “protocol guide” alongside protocol definitions to help people learn.
Yep. I recently followed this guide and implemented the protocol up to the RPC level in Swift with the new Apple Networking framework, and it was quite straightforward.
(I then got a bit frustrated with the JSON serialization issue mentioned elsewhere in this discussion, so I haven’t proceeded with that project. I did fork a Swift JSON library to use an ordered dictionary, so it’s almost there…)
This looks really neat but the whole “you have to have a bug-for-bug identical-with-node JSON serialization routine” for the signature stuff has always rubbed me the wrong way. Have there been any non-node implementations out in the wild yet? Last I looked the reference implementation was the only one you could actually use.
This is exactly the use-case canonical S-expressions were designed for.
As an advantage, they’re more attractive than JSON, and encourage better practices. E.g.:
Might be:
Personally, I’d use bencode. It was literally designed for use in P2P protocols, and it’s length-prefixed, so you can pass arbitrary stuff in it without having to escape anything. And there’s already good implementations of it in Node (because BitTorrent uses it).
Note that canonical S-expressions are also length-encoded: the example I gave actually looks like this on the wire:
(21:create-history-stream(4:type6:source)(2:id(7:ed2551932:%<"8BAh={/f)))Assuming I didn’t mess up, that would be transport-encoded as:
{KDIxOmNyZWF0ZS1oaXN0b3J5LXN0cmVhbSg0OnR5cGU2OnNvdXJjZSkoMjppZCg3OmVkMjU1MTkzMjoUJf+2wMum5sI8op8ivDiBz5JCQdxoPXuzsYjqL/OJZikpKQ==}The advantage over bencode is that there’s an attractive advanced representation, which is what I wrote in my post, rather than only the wire representation. The bencode version would, I think, look like this:
l21:create-history-streaml4:type6:sourceel2:idl7:ed2551932:%<"8BAh={/feeeI think that the canonical S-expression’s wire format is much easier to read by hand, don’t you agree?
Huh, TIL. Then in that case, it probably would be pretty good.
I wonder if the ease of reading problems couldn’t be fixed with a better text editor, making the debates between most of these essentially equivalent formats ‘moot’.
aka
This could be kept in messagepack form, and just shown to the developer as json or s-exp or whatever you want in your favorite text editor. If anything the necessity for “human readable” files as plaintext or some specific format if anything an expression of the limitations of our tooling, as well as a lack of standards of encoding one format in another. If there were a standardized encoding between formats you could simply use the encoding that worked for you, and translate any implementation trivially.
It’s an interesting idea, but different formats have different styles. E.g. JSON tends to like objects, which are unordered maps; canonical S-expressions tend to be built on ordered lists, often of key-value pairs; other formats have their own styles.
There is currently an effort underway to rework the protocol to avoid that JSON serialization order misfeature.
See https://spec.scuttlebutt.nz/messages.html#legacy-json-encoding
This should also allow implementations in other languages.
Web standards are slowly migrating to using CBOR for deterministic order and raw byte friendlines. An example: https://www.w3.org/TR/webauthn/#conforming-all-classes
Thanks for the info; hope this lands soon.
I know, right? These folks are all like “boats do not have always-on internet connections” and I’m all like “but they have quad-core CPUs and double-digit megabits of memory?”
The other big thing I’ve noticed is that while it’s perfectly fine with ssb for your connection to be down most of the time, heaven forbid that your connection be metered. Initial sync upon joining a pub or following anyone can potentially be hundreds of megabytes – make sure it doesn’t happen on mobile data.
A $20-$30 secondhand smartphone can easily manage quad-CPU and a gigabyte of ram.
I hear you. I must point out that a gigabyte isn’t tens of gigabits and actually a quad-core phone would cost more like $50-70, right? I dunno, I haven’t bought a phone in a while.
It should be clear what I want though, right? Something that isn’t node.js or electron. Something that feels like weechat or mutt, or even keybase’s CLI. (Though, it is a performance hog, the background processes are.. maybe node.js? Something heavy.)
https://github.com/cryptoscope/ssb
I googled around for a bit and found this, which seems to be related. https://godoc.org/go.cryptoscope.co/ssb
How can somebody who doesn’t know much about go nor about ssb figure out what this repo is?
There are 14 branches in the repo. Is ‘master’ what most people would want?
It’s been a while, but I think it’s something like:
… open a browser and go to http://localhost:8008
Is that the reason there is no browser only client?
I love the overall concept, but Scuttlebutt seems to completely ignore all existing protocols for RPC, serialization, LAN discovery, cluster gossip, message encoding, etc. It reinvents a whole lot of wheels all at once. “But it’s simple! It’s just JSON!” they might say. But the difficulty isn’t in the syntax, it’s in the semantics.
Unless I missed something, local sync requires visibility of peers’ UDP broadcasts on a shared LAN. Nearly all public hotspots and home guest LANs disable peer visibility, so this seems like a nonstarter. BLE beacons could be an interesting alternative.
I’d rather use a version of ActivityPub or Solid (https://solid.mit.edu) that works truly offline, such as with BLE + Wi-Fi Direct.
Everyone’s (rightfully) talking about NodeJS, but I think this is an awesome walk through of a protocol. I feel like I’m often reverse engineering some crazy nasty stuff when I’ve had to write a protocol from scratch or learn it blackbox. I’ve always felt like protocol definitions are awesome, but don’t always help you understand the whole flow in a real setting. This does an extremely good job to help visualize the protocol in real function. I might start adopting a “protocol guide” alongside protocol definitions to help people learn.
Yep. I recently followed this guide and implemented the protocol up to the RPC level in Swift with the new Apple Networking framework, and it was quite straightforward.
(I then got a bit frustrated with the JSON serialization issue mentioned elsewhere in this discussion, so I haven’t proceeded with that project. I did fork a Swift JSON library to use an ordered dictionary, so it’s almost there…)