1. 25
  1.  

  2. 4

    If only there were a standardized binary format with JSON semantics and a canonical form.

    1. 5

      There’s CBOR (Concise Binary Object Representation), where the RFC does mention a canonical form.

      1. 2

        Unfortunately, like the rest of CBOR in my experience, the canonical form is neither as simple nor as rigorous as one might desire.

        1. 1

          Would you elaborate, please? I was under the impression that strict+canonical form following the recommendations in the RFC and normalizing unicode strings was safe.

          Do you have any counter example?

          1. 1

            The recommendations are recommendations, so a conforming implementation does not need to follow them. Following them is actually not trivial, given that it involves things like selecting sizes for integers that are far more grainular than I care about, ie, integers other than i32 or i64. So my implementation needs to be able to emit more complicated CBOR than it would otherwise find necessary. It says “integers must be as small as possible”, but is that as small as the value, or as small as the value’s possible domain? It’s unclear. It provides essentially no canonicalization for floats, just another suggestion or two. And normalizing Unicode strings, as the original post points out, is far from easy, though I suppose you’re going to have to do it somewhere no matter what.

            It’s not a canonical format, it’s a mixed bag of suggestions and directives for how to start designing a canonical format. It’s possible for two canonical parsers/emitters disagree on the same data, with details such as float signaling bits, whether or not duplicates are okay, etc.

            …actually, now that I look at it, CBOR in general is very weird. It describes very precisely what the representation of a data item should look like based off what the data is, and not what the the data may be. Maybe I’ve been programming strongly typed languages too long but that’s almost never what I actually want. Hm, I should try looking at it again from the opposite angle.

        2. 2

          Exactly. And it’s apparently still not widely known. It also supports chunked transmission if anyone needed to stream some JSON-like data and needed a framing protocol.

        3. 4

          ASN.1?

          1. 1

            ASN.1 is an interface description language not an encoding format.
            There are many different encoding rules BER, CER, DER, and PER.
            ASN.1 encoder/decoders are complex and have a long history of parsing bugs.
            I wouldn’t use ASN.1 for a new application unless it is required for interoperability.

            If ASN.1 isn’t required for interoperability then I would probably use Protocol Buffers instead.
            It’s simpler, well documented, fuzzed, and widely used.

            1. 1

              The problem with PB as far as I know is that there is nothing like CER, so there is no guarantee that 2 different implementations of PB will produce exactly the same output from the same data. What I would see instead is ASN.2 specification that would remove bloat while it would still have the nice property of having different encodings like BER, JER, XER, etc. Maybe one day I will find time to write such spec down.

          2. 2

            My vote would be for canonical S-expressions: human-readable and efficient and canonical too.

            No support for unordered sets like maps, but a) lack of ordering is part of the problem and b) they can be emulated with ordered pairs.

            1. 2

              You still need to canonicalize numbers (leading zeroes, for example) and Unicode strings.

            2. 1

              BSON! (I am 101% joking.)

              1. 1

                There’s bencode, which is almost compatible with JSON but only has a canonical form. It is also extremely simple to write encoders and decoders for; I’ve done a few in several different languages, and it’s rarely more than a day or two of work. (My most recent, bendy, took ~1 week because I wanted it to be zero-copy while still presenting a DOM-like interface) The main downside is that there is no boolean or floating point type.

              2. 4

                At work there was one client, that couldn’t implement webhook signing, because ExpressJS after decoding the json does not allow looking at the original body, so you cannot reliably verify the signature. In the end they decided that IP whitelisting would be enough ¯\_(ツ)_/¯

                1. 1

                  Patch Express?

                  1. 1

                    It is deep in Express architecture, so not quite an option. There are solutions around it, but they are a bit involved, and don’t always work.

                2. 3

                  If you needed a signed JSON object to retain JSON structure, why wouldn’t you add a valid JSON envelope with the token and the original payload as attributes like so:

                  {
                      "header": {
                          "alg": "HS256",
                          "typ": "JWT"
                      },
                      "token": "XXXXXXX",
                      "payload": {
                          "sub": "1234567890",
                          "name": "John Doe",
                          "iat": 1516239022
                      }
                  }
                  
                  1. 8

                    Imagine that the sender of the JSON document is Node and the ECMAScript JSON API, and the recipient of the document is using Rust and Serde.

                    Most cryptographic algorithms, including hashing functions, operate on bytes. So to take the hash of that payload, you need to decode the entire JSON document, pull the payload object out of memory, re-encode it as a JSON document, and perform the hashing algorithm on that. When you do this in Node, you’ll wind up hashing the ASCII bytes of {"sub":"1234567890","name":"John Doe","iat":1516239022}, getting 3032e801ce56c762a1485e5dc2971da67ffff81af5cc7dac49d13f5bfbe95ba6. Also, because of the way objects are represented in Node, seemingly innocuous changes to the code can result in the keys being in a different order when you initially build an object, but Node does preserve order in JSON documents when it decodes and reencodes it. (node also does not provide any good APIs for manipulating the order of keys in objects, as far as I know, because ECMAScript actually says that the order is unspecified)

                    Serde, on the other hand, does not preserve order when you decode a JSON document. There are basically two common ways to decode a JSON object: you can decode it into a HashMap, which literally randomizes the order, or you can decode it into a struct, which if you re-encode it, will encode it in the same order that the struct is written in. So, given this code:

                    #[derive(Deserialize,Serialize)]
                    struct FullMessage {
                        header: MessageHeader,
                        token: String,
                        payload: MessagePayload,
                    }
                    #[derive(Deserialize,Serialize)]
                    struct MessagePayload {
                        iat: u64,
                        name: String,
                        sub: String,
                    }
                    

                    If you decode into a FullMessage, and then re-encode the MessagePayload, you will wind up hashing {"iat":1516239022,"name":"John Doe","sub":"1234567890"}, which hashes to 907b71ecd7dbc6cb902905e053fe990ed5957aa5217150b2355c36583fcf9519. It will, thus, report that the payload was tampered with, even though both versions of the payload are equivalent for your purposes.

                    Because the JSON specifications say that order is not important in an object, both behaviors are spec-compliant.

                    1. 3

                      Gotcha. I don’t use Node or Rust, but I can understand how different JSON libraries could make this a problem. What if the payload was serialized?

                      {
                          "header": {
                              "alg": "HS256",
                              "typ": "JWT"
                          },
                          "token": "XXXXXXX",
                          "payload": "{\"iat\": 1516239022, \"sub\": \"1234567890\", \"name\": \"John Doe\"}"
                      }
                      

                      In this form, the token is computed and verified on the given bytes of the serialized payload, so differences in parsers should not matter.

                      1. 1

                        That would totally work. It’s basically the same as the OP’s recommendation (serialize your JSON, then concatenate it with the signature) except you’re using a much more complicated way of “concatenating” them.

                        1. 3

                          Right, but the result is still valid JSON, which was the problem they raised with “just concatenation.”

                          1. 3

                            Technically, yes. Unfortunately, whatever signature-unaware middleware you’re using won’t be able to get at the JSON keys and values within the payload part. Most people deploy such middleware specifically because they want to be able to filter or route based on the contents of the message, and you lose that.

                  2. 1

                    There is PASETO specification which in general uses public key cryptography, but in general follows similar suite.

                    1. 1

                      using JSON for something which must be signed is a terrible idea, as there is no canonical representation of a json message