JSON was initially described as a subset by Crockford (this page from 2006). I think it would be common to assume that if you use if you have two values from data of the subset that are unequal, they’d be unequal in the superset too. But with the JSON/JavaScript pair this isn’t the case. Not even after proposal-json-superset, which Wikipedia claims makes JSON a “strict subset”.
I think it would be best if we just explicitly pointed out that JSON isn’t a subset, but it is difficult when even TC-39 uses the terminology like this.
Or am I wrong, and we can’t talk about “equality” of JSON documents at all?
Or am I wrong, and we can’t talk about “equality” of JSON documents at all?
Correct.
JSON defines an encoding, and two specific JSON-encoded byte sequences are comparable for sure. But two abstract JSON objects are not directly comparable, for many reasons, among them (topically to this post) that JSON doesn’t define numbers as precise values, and allows implementations to assert arbitrary precision.
That is, given abstract JSON object obj1 {v: 9123372036854000123} and obj2 {v: 9123372036854000000}, then obj1 == obj2 is literally undefined. Of course the encoded JSON strings "{"v": 9123372036854000123}" and "{"v": 9123372036854000000}" are comparable, but encoding isn’t bijective, and so equality of encoded forms doesn’t imply equality of data, e.g. {v: 1} can be encoded to either "{"v":1}" or " { "v": 1}", which are not equivalent, even though they represent equivalent information.
The Twitter API is a great example of somewhere that you might run into this issue, since tweet IDs have been larger than the maximum regular JavaScript integer for quite a while.
The API offers a “id_str” property to help workaround this, but a lot of developers ignore that when they start using that API and run into trouble.
The parsed value [of {"count": 9123372036854000123}] is corrupted: the last three digits are reset to zero.
The JSON spec defines numbers without any specific precision guarantees, deferring that detail to implementations. A number which is encoded by one implementation with one precision, and then decoded by a different implementation with a different precision, can always result in language values which are not equivalent. JSON explicitly says this can happen and is acceptable. So it can’t be understood as corruption in any meaningful sense.
JSON numbers are always approximations. The spec actually makes this point pretty explicitly:
good interoperability can be achieved by implementations that expect no more precision or range than [IEEE 754-2008 binary64 numbers] provide, in the sense that implementations will approximate JSON numbers within the expected precision.
If your programs use numbers that are << typical precision limits of implementations, you can usually get away with encoding them as numeric types, and comparing their values directly. But if you have very large or very small numbers, and/or you need absolute precision, then it’s pretty well understood that you can’t use number literals, you have to encode your numbers as strings, which are always treated as raw and immutable bytes.
If you don’t have round-trip fidelity after loading and saving a document, that’s pretty unambiguously data corruption. Remember, this is a blog post about an editor.
If you don’t have round-trip fidelity after loading and saving a document, that’s pretty unambiguously data corruption.
JSON explicitly does not guarantee round-trip fidelity in the sense that you mean here. If an editor assumes that JSON objects, and specifically JSON numbers, are bijective in this way, that’s a problem with the editor, isn’t it?
I literally ran into this just last week. The number was of big/int in golang. I ended up using a regex replace to turn the int into a string in the json document and deal with the rest elsewhere. It is really annoying when JSON breaks b/c there are so many tools that have support for it.
side note: I wish golang had more powerful json libraries like java has with jackson.
Ran into this with Discord’s API a little while back. Everything was fine in the C# backend until I passed data to the frontend in JSON, and numbers silently corrupting. Had to convert them to strings, because JavaScript :-/
Because Javascript is broken by design and possibly the only language of its class with such limited numerics.
Javascript has actually had big integers for some time now, though it did not at the time that JSON was designed.
JSON was initially described as a subset by Crockford (this page from 2006). I think it would be common to assume that if you use if you have two values from data of the subset that are unequal, they’d be unequal in the superset too. But with the JSON/JavaScript pair this isn’t the case. Not even after proposal-json-superset, which Wikipedia claims makes JSON a “strict subset”.
I think it would be best if we just explicitly pointed out that JSON isn’t a subset, but it is difficult when even TC-39 uses the terminology like this.
Or am I wrong, and we can’t talk about “equality” of JSON documents at all?
Correct.
JSON defines an encoding, and two specific JSON-encoded byte sequences are comparable for sure. But two abstract JSON objects are not directly comparable, for many reasons, among them (topically to this post) that JSON doesn’t define numbers as precise values, and allows implementations to assert arbitrary precision.
That is, given abstract JSON object obj1
{v: 9123372036854000123}and obj2{v: 9123372036854000000}, thenobj1 == obj2is literally undefined. Of course the encoded JSON strings"{"v": 9123372036854000123}"and"{"v": 9123372036854000000}"are comparable, but encoding isn’t bijective, and so equality of encoded forms doesn’t imply equality of data, e.g.{v: 1}can be encoded to either"{"v":1}"or" { "v": 1}", which are not equivalent, even though they represent equivalent information.JSON was actually “designed” without any such limitations to numerics, hence the problem here.
Also, “quite some while” is not quite some while, with bigints being clumsy instead of the default.
Even tools that don’t use JavaScript have this limitation, for example
jq.Thinking of literals as something that can be stored in constant memory goes back further than JavaScript.
The Twitter API is a great example of somewhere that you might run into this issue, since tweet IDs have been larger than the maximum regular JavaScript integer for quite a while.
The API offers a “id_str” property to help workaround this, but a lot of developers ignore that when they start using that API and run into trouble.
There seems to be a misunderstanding here.
The JSON spec defines numbers without any specific precision guarantees, deferring that detail to implementations. A number which is encoded by one implementation with one precision, and then decoded by a different implementation with a different precision, can always result in language values which are not equivalent. JSON explicitly says this can happen and is acceptable. So it can’t be understood as corruption in any meaningful sense.
JSON numbers are always approximations. The spec actually makes this point pretty explicitly:
If your programs use numbers that are << typical precision limits of implementations, you can usually get away with encoding them as numeric types, and comparing their values directly. But if you have very large or very small numbers, and/or you need absolute precision, then it’s pretty well understood that you can’t use number literals, you have to encode your numbers as strings, which are always treated as raw and immutable bytes.
If you don’t have round-trip fidelity after loading and saving a document, that’s pretty unambiguously data corruption. Remember, this is a blog post about an editor.
JSON explicitly does not guarantee round-trip fidelity in the sense that you mean here. If an editor assumes that JSON objects, and specifically JSON numbers, are bijective in this way, that’s a problem with the editor, isn’t it?
It’s both. The editor shouldn’t assume it (because JSON) but that doesn’t excuse the fact that JSON has this.
I literally ran into this just last week. The number was of big/int in golang. I ended up using a regex replace to turn the int into a string in the json document and deal with the rest elsewhere. It is really annoying when JSON breaks b/c there are so many tools that have support for it.
side note: I wish golang had more powerful json libraries like java has with jackson.
Ran into this with Discord’s API a little while back. Everything was fine in the C# backend until I passed data to the frontend in JSON, and numbers silently corrupting. Had to convert them to strings, because JavaScript :-/
Sadly this also happens in golang
https://pkg.go.dev/encoding/json#Decoder.UseNumber
I am extremely grateful for this. Edit: I also have this problem on encoding which of course does not have the same method because why would it.