It’s not just theoretical. I skimmed the code – the linked parsers seem to have exploitable differences in how they handle line endings. In addition to \r and\n, Json5 supports \u2028 and \u2029 to end lines, dkjson.lua does not. Therefore:

{
   "user_id": 1,
   // maybe a 2-line comment \u2028  "user_id": 2
 }

will be interpreted differently by the two parsers. Swift Foundation implements a 3rd behavior; it assumes ‘\n’ and ‘\r\n’ are the only valid line endings, so:

{
   // maybe a 2-line comment \r "user_id": 2
 }

will be a 1 entry object when parsed by JSON5, dkjson.lua, and Sqlite, but empty when parsed by Swift.

I couldn’t be bothered to dig through the Jackson code.

I think there are 3 different use cases for JSON here:

• One for serialization. In this case, yes, comments are bad because they’re complicated and are not defined in spec. But also, why would you need comments in serialization?
• One for interop between different programs, that is a similar case as the above. In this case you may want comments, but it is best to avoid them (unless you’re using something like _comment: "my comment"), since you never know if the other implementations will have similar behavior
• A third use case is for something more interactive, e.g.: confiuration files. In this case you probably don’t care about other consumers, having comments can be really useful. If we are assuming those files are edited by humans trailing commas are also useful for similar reasons

But to a person who crafts security exploits, this looks like an opportunity to add “cloaked” JSON elements which are processed by some JSON parsers and ignored by other JSON parsers. That kind of difference in interpretation can be the beginnings of a security exploit.

This also depends on the context. For the third use case in many cases if you have the privilege of setting a configuration file you already lost.

Using comments in your JSON doesn’t mean including comments in spec-compliant JSON!

The gift the JSON spec is giving you is that by not allowing comments it’s telling you what to do with the comments: throw them in the garbage in some step before they get to the real JSON parser. In this way, many kinds of comments can be supported (properly, without vulnerability) because once they’re gone they really can’t possibly mean anything (solving the security problem).

JSON is a very forgiving language syntactically so % # // and /* */ <!-- --> are all equally valid ways to add unambiguous comments to your JSON. That is to say, since these syntaxes don’t conflict with JSON’s own syntaxes, there’s no chance of a real confusion where a comment is interpreted as valid data or visa versa. The fact that JSON absolutely rejects comments and other unexpected syntax is, paradoxically, what means that it works with every flavor of comments so long as you can define what it means to strip them out.

I agree.

Honestly, if you just want a more comfortable JSON and you control the parser, why not use toml or yaml or any other text data format that was designed from the ground up to be comfortable to write and read?

A standard’s usefulness is proportional to its goodness. I believe not having comments was a mistake, which reduces goodness.

I think you hit the nail on the head here.

You, along with several others in this comment section, have completely missed the point of the article. It is pretty clear (at least, imo) that the author is not arguing that adding comments to JSON is conformant to the spec. They are just saying what you do doesn’t have to conform to the spec (crucially) if it isn’t necessary.

You write your JSON with m4, have your comments with lines starting with dnl and pre-parse it with it m4

that is an arbitrary decision you would need to have a spec clarify.

It’s not arbitrary. There is only one defensible way to treat comments in Javascript Object Notation and that is the same way they are treated in Javascript.

Yeah exactly, this page would be more interesting if it actually said that

The hard part is the agreement, not coming up with something

I prefer the following if I need to put comments into a JSON that will be read by a relatively strict parser:

{
   "#comment#1": "val here is lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua",
   "key": "val",
   "#comment#2": "Acceptable values for foo: bar|baz",
   "foo": "bar"
}

This, of course, will not work if the system not only uses a strict parser but also does a strict schema checking, disallowing unrecognized comment keys. Though I have encountered more often systems that have a strict JSON parser than a strict schema checking.

the reason to have comments is for people reading and writing the format, rather than for machine-to-machine uses of JSON

I didn’t really want to get into in on the site because I wanted to keep it punchy, but the reason JSON is typically not used for config files is specifically that it doesn’t have comments, which pushes people towards using YAML, which leads to anger, which leads to the dark side, etc.

If this problem didn’t exist (if parser authors just made them tolerant of comments in defiance of the spec) then I think you’d see a lot less use for YAML, which would be a wonderful situation with Norway downsides.

(This doesn’t really have anything to do with machine-to-machine JSON.)

you’re going to have to modify a lot of parsers to make this broadly useful

Nope! JSON config files will almost always only be read by one parser. So each individual program can switch away from using YAML to using JSON, and each one that does is a step in the right direction regardless of how many others do it.

I see your reasoning, and while I’d love something better than JSON, I’ll challenge that this is an area where it’s easy to make the perfect the enemy of the good.

“JSON with one extra or two small changes standardized” feels like an accessible goal to see the industry adopt en masse. It’s forward compatible and requires the least code churn and retraining, and any actively-maintained project will eventually support receiving JSON++ files just by updating its dependencies or runtime.

Trying to anoint a JSON successor (or worse, having a war of succession) has a high risk of people deciding the successor isn’t better enough to be worth the effort of migrating. Especially in cases where that means getting third parties on board with your project’s change.

“Comments” can already be supported by making a key/value called “comment” or something, which works fine without having to support a convention for comments.

How would you comment individual array elements?

Even if you only need to comment key–value pairs, your suggestion relies on the convention that the consuming program will quietly ignore any keys it doesn’t recognize. If that isn’t true, then you can’t use this technique for comments. (And even if it is true, what happens when a new version of the application adds a new field to the configuration and starts assigning meaning to what was supposed to be a comment?)

The first point already isn’t really true - I’ve tried to maintain structure in a package.json file before, without comments, by ordering the data in a certain way, or by inserting newlines to make things easier to read. As soon as the NPM parser reads that in and writes it out again, that structure gets lost. So even with the current minimal tools that we have, naive JSON parsing causes things to get lost in translation.

Moreover, it’s not necessarily impossible to fix this. For most situations where you’re just passing data around, a naive parser is sufficient. But when you’re trying to parse a config file, you can switch to a parser that internally records the structure of the file, comments, indentations, whitespace and all, and makes minimal modifications to that structure when writing the data out again. This is, for example, what Rust’s Cargo does with TOML files. It requires a more complicated parser, but it works well.

The second point is theoretically possible, but I’ve never really seen it come to be, not even in YAML which has comments and has been abused to essentially become a full programming language in many situations. Most tools either use normal YAML syntax to essentially represent an AST, or they use a templating language to generate YAML, completely bypassing the comments. In neither case do comments get used as directives.

The third point, as others have pointed out, is only possible if the parser chooses to accept arbitrary key value pairs and ignore them, and even then can only comment certain nodes in single-line structures and incredibly uneconomically. It is surely the worst of all options - it has exactly the same theoretical downsides of the two previous points you made, but it’s uglier, more difficult to write, and more brittle.

I am not entirely convinced by the original article - as others have pointed out, there are already security risks associated with the different approaches taken by different parsers, and adding more variation feels like asking for trouble. But I am very convinced that the three arguments you’ve given (which come up every time this topic comes up) are brought up more out of rote habit than because they are good arguments.

Don’t forget to include a trailing newline. It’ll never cause problems.

As one of the nit-pickers, I’ll say I also agree with that point. What is more iffy is

If you maintain a parser, you can make it accept comments and nothing bad will happen!

For the security risks mentioned elsewhere.

Counterpoint, TOML has horrible support for nested objects, which makes it a poor fit for many json use cases.

This sounds overly complex. Why don’t we just write a FUSE module

I can’t tell if this is /j, if you’ve never written a FUSE filesystem for production use, or if Douglas Crockford is your dad.

[Comment removed by moderator pushcx: Pruning empty thread.]