1. 33
  1. 9

    This looks really neat but the whole “you have to have a bug-for-bug identical-with-node JSON serialization routine” for the signature stuff has always rubbed me the wrong way. Have there been any non-node implementations out in the wild yet? Last I looked the reference implementation was the only one you could actually use.

    1. 13

      This is exactly the use-case canonical S-expressions were designed for.

      As an advantage, they’re more attractive than JSON, and encourage better practices. E.g.:

      {
        "name": ["createHistoryStream"],
        "type": "source",
        "args": [{"id": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519"}]
      }
      

      Might be:

      (create-history-stream
        (type source)
        (id (ed2559 |@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=|)))
      
      1. 4

        Personally, I’d use bencode. It was literally designed for use in P2P protocols, and it’s length-prefixed, so you can pass arbitrary stuff in it without having to escape anything. And there’s already good implementations of it in Node (because BitTorrent uses it).

        1. 4

          Note that canonical S-expressions are also length-encoded: the example I gave actually looks like this on the wire:

          (21:create-history-stream(4:type6:source)(2:id(7:ed2551932:%<"8BAh={/f)))

          Assuming I didn’t mess up, that would be transport-encoded as:

          {KDIxOmNyZWF0ZS1oaXN0b3J5LXN0cmVhbSg0OnR5cGU2OnNvdXJjZSkoMjppZCg3OmVkMjU1MTkzMjoUJf+2wMum5sI8op8ivDiBz5JCQdxoPXuzsYjqL/OJZikpKQ==}

          The advantage over bencode is that there’s an attractive advanced representation, which is what I wrote in my post, rather than only the wire representation. The bencode version would, I think, look like this:

          l21:create-history-streaml4:type6:sourceel2:idl7:ed2551932:%<"8BAh={/feee

          I think that the canonical S-expression’s wire format is much easier to read by hand, don’t you agree?

          1. 3

            Huh, TIL. Then in that case, it probably would be pretty good.

            1. 1

              I wonder if the ease of reading problems couldn’t be fixed with a better text editor, making the debates between most of these essentially equivalent formats ‘moot’.

              {  "create-history-stream" : {
                  "type": "source",
                  "args" : [{"id": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519"}]
                }
              }
              

              aka

              81 b5 63 72 65 61 74 65 2d 68 69 73 74 6f 72 79 2d 73 74 72 65 61 6d 82 a4 74 79 70 65 a6 73 6f 75 72 63 65 a4 61 72 67 73 91 81 a2 69 64 d9 35 40 46 43 58 2f 74 73 44 4c 70 75 62 43 50 4b 4b 66 49 72 77 34 67 63 2b 53 51 6b 48 63 61 44 31 37 73 37 47 49 36 69 2f 7a 69 57 59 3d 2e 65 64 32 35 35 31 39
              

              This could be kept in messagepack form, and just shown to the developer as json or s-exp or whatever you want in your favorite text editor. If anything the necessity for “human readable” files as plaintext or some specific format if anything an expression of the limitations of our tooling, as well as a lack of standards of encoding one format in another. If there were a standardized encoding between formats you could simply use the encoding that worked for you, and translate any implementation trivially.

              1. 1

                It’s an interesting idea, but different formats have different styles. E.g. JSON tends to like objects, which are unordered maps; canonical S-expressions tend to be built on ordered lists, often of key-value pairs; other formats have their own styles.

        2. 8

          There is currently an effort underway to rework the protocol to avoid that JSON serialization order misfeature.

          See https://spec.scuttlebutt.nz/messages.html#legacy-json-encoding

          This should also allow implementations in other languages.

          1. 5

            Web standards are slowly migrating to using CBOR for deterministic order and raw byte friendlines. An example: https://www.w3.org/TR/webauthn/#conforming-all-classes

            1. 1

              Thanks for the info; hope this lands soon.

            2. 7

              I know, right? These folks are all like “boats do not have always-on internet connections” and I’m all like “but they have quad-core CPUs and double-digit megabits of memory?”

              1. 9

                The other big thing I’ve noticed is that while it’s perfectly fine with ssb for your connection to be down most of the time, heaven forbid that your connection be metered. Initial sync upon joining a pub or following anyone can potentially be hundreds of megabytes – make sure it doesn’t happen on mobile data.

                1. 3

                  A $20-$30 secondhand smartphone can easily manage quad-CPU and a gigabyte of ram.

                  1. 2

                    I hear you. I must point out that a gigabyte isn’t tens of gigabits and actually a quad-core phone would cost more like $50-70, right? I dunno, I haven’t bought a phone in a while.

                    It should be clear what I want though, right? Something that isn’t node.js or electron. Something that feels like weechat or mutt, or even keybase’s CLI. (Though, it is a performance hog, the background processes are.. maybe node.js? Something heavy.)

                  1. 1

                    I googled around for a bit and found this, which seems to be related. https://godoc.org/go.cryptoscope.co/ssb

                    How can somebody who doesn’t know much about go nor about ssb figure out what this repo is?

                    There are 14 branches in the repo. Is ‘master’ what most people would want?

                    1. 2

                      It’s been a while, but I think it’s something like:

                      git clone git@github.com:cryptoscope/ssb.git
                      cd ssb/cmd/go-sbot
                      go build
                      ./go-sbot
                      

                      … open a browser and go to http://localhost:8008

                  2. 1

                    Is that the reason there is no browser only client?

                  3. 4

                    I love the overall concept, but Scuttlebutt seems to completely ignore all existing protocols for RPC, serialization, LAN discovery, cluster gossip, message encoding, etc. It reinvents a whole lot of wheels all at once. “But it’s simple! It’s just JSON!” they might say. But the difficulty isn’t in the syntax, it’s in the semantics.

                    Unless I missed something, local sync requires visibility of peers’ UDP broadcasts on a shared LAN. Nearly all public hotspots and home guest LANs disable peer visibility, so this seems like a nonstarter. BLE beacons could be an interesting alternative.

                    I’d rather use a version of ActivityPub or Solid (https://solid.mit.edu) that works truly offline, such as with BLE + Wi-Fi Direct.

                    1. 4

                      Everyone’s (rightfully) talking about NodeJS, but I think this is an awesome walk through of a protocol. I feel like I’m often reverse engineering some crazy nasty stuff when I’ve had to write a protocol from scratch or learn it blackbox. I’ve always felt like protocol definitions are awesome, but don’t always help you understand the whole flow in a real setting. This does an extremely good job to help visualize the protocol in real function. I might start adopting a “protocol guide” alongside protocol definitions to help people learn.

                      1. 2

                        Yep. I recently followed this guide and implemented the protocol up to the RPC level in Swift with the new Apple Networking framework, and it was quite straightforward.

                        (I then got a bit frustrated with the JSON serialization issue mentioned elsewhere in this discussion, so I haven’t proceeded with that project. I did fork a Swift JSON library to use an ordered dictionary, so it’s almost there…)