UCL looks very interesting, if there’s no trap or ambiguity. The first thing I notice is that you seem to be able to omit the ugly braces around the whole document.
I might consider creating a Rust deserializer for it… I’ll have to look more at the language.
I guess my blog post will have to be renamed into “Why broot switched from TOML to Hjson then UCL for its configuration files”…
FreeBSD has adopted ucl from its core OS configuration files. The interesting thing is that it’s a silently supported format for a lot of files that still use either the old conf format or a subset of ucl that resembles the old conf format. For example, the pkg.conf files can be written in one of two different configuration languages and the default is not ucl, but a completely different ucl syntax is available and automatically detected/supported.
Correct me if I’m wrong, but it seems hjson suffers from some of the same bugs that yaml has around type ambiguity. Primarily, in formats where unquoted strings are allowed a human may write 1000 in a field which is supposed to be a string, and they’ll get a number.
The main problem with YAML used as a configuration format written by human isn’t that true isn’t interpreted as a string, because you expect it. The real problem in my opinion is those weird rules which makes for example off be interpreted as a boolean. It’s so random most people don’t know it (see the “Norway problem” ( https://hitchdev.com/strictyaml/why/implicit-typing-removed/ ) too). Hjson doesn’t have the same level of insanity (true is only true, not On nor YES or yes).
So I think this problem, for Hjson, isn’t really a problem. Or a small one.
Now, I said, perhaps in a cryptic way, that even the weird rules of YAML “may matter or not”. Because there’s absolutely no problem here if you use Hjson (or YAML) the way I do.
The reason is that the best way to deserialize a configuration, in my opinion, isn’t to use the automatically guessed type but to deserialize into the type required by the strongly typed structure.
And that’s the only public API I’ve put in my deserializer: you can’t use my Hjson deserializer ( https://github.com/Canop/deser-hjson) to read a free form file, you can only use it to deserialize into a Rust type, precisely to avoid the dangers related to type ambiguity. If the human typed 1000 in a field which is expected in Rust to be a string, it will be read as a string.
Agreed; deserializing into typed data structures really does make most of the issues with YAML go away in my experience. The other thing that helps when I’m writing YAML to configure code I didn’t write is to always wrap string values in quotes. Yes, it adds a little noise, but it is a pretty easy habit to pick up and then all the “what type is this value, anyway?” problems with YAML become irrelevant. The added noise of quoted strings isn’t too bothersome to me, though that’s obviously subjective.
Ah, yes, forcing deserialization to a datatype is a good way to deal with this, vs having some general HJson type and making an arbitrary decision about the types of fields. That said, I still find syntactic ambiguities like this one to be a major downside of these human readable formats.
It’s fascinating to me that there’s so much diversity in config formats. Maybe it’s just that everyone needs one and they’re easy to quibble with.
That said, I like that HJSON explicitly aims “to be used like a user interface for humans, to read and edit before passing the JSON data to the machine.” This seems similar to Strict YAML, which also focuses on giving humans a low-surprise way to define data structures that computers will eventually treat like JSON.
I would consider using HJSON over Strict YAML next time I need a config.
I don’t think there so much diversity. There’s less than a dozen common configuration formats, none of them really good (writing a deserializer for Hjson made me more aware of its shortcomings, especially for the quoteless strings and its indecisiveness regarding whether it’s a line oriented format or not).
I didn’t know about Strict YAML. It’s interesting and I’ll probably amend the document to mention it.
In my opinion, Json5, which I do use, is a better JSON, but while it’s suited for data exchange it’s behind Hjson for configuration (which wasn’t probably the main goal). It especially still asks for commas and doesn’t solve the indentation problem for multiline strings.
Rome-json (which I didn’t know) seems to be an intermediate between JSON and Hjson.
edit: I’ll edit my blog post to link to those alternatives
I don’t think it’s that many, when being compared to programming languages, image formats, etc. I also wouldn’t call all of these config format, in the sense that they were made to be mostly written by humans. JSON (and to a degree YAML) is a human readable data serialization format, and in many situations is generated vs. hand written. UCL/HCL and TOML are way more frequently touched manually.
Of course that doesn’t mean you cannot/should under no circumstances use other formats. I think they are tools to choose from and while I would agree there is a lot of “someone trying their own thing” projects, the same is again true for programming languages and many other formats.
It’s also a good thing that people experiment. In the majority projects you will still end up with a hand full of variation. So I think it’s more that people are interested in this topic and there is a lot of talk about it.
If there is one thing I wish configuration languages would adopt, its to drop commas all together. Clojure’s EDN format does this and its great. Whitespace is a natural separator for lists.
i have a config language that i wrote that does that. it converts to json, so for a while i was writing configs with this as a json preprocessor; the moon command converts moon files to json with moon to json; programs would just use the json file. i haven’t used it much recently because it drives other people nuts when you say “hey let’s use this config format that i invented that nobody else has ever seen”. https://github.com/jordanorelli/moon
What the user wanted here, when adding the last prop, was to set a top level property.
But what really happens is that the property is added to the second item of the "chars" array.
I have a hard time believing that anyone who has ever written or modified an INI file would be confused about this. Considering that this is the only illustrated criticism of TOML in this entire post, I’m puzzled as to why this person switched away from TOML.
Now, I understand what they mean when they call the language alien and confusing. But so are YAML, XML, etc. when you first encounter them, it’s just that people are accustomed to these decades old languages. Is it possible to create a configuration language that is intuitive to inexperienced newcomers and covers all the use-cases of TOML? Time will tell.
The problem isn’t with the software author. Being the author of broot, I wouldn’t have to change the format if I was the sole user. The problem is when people stop trying to configure your software because the changes they do have no effect.
Adding a line to a configuration file is something that linux users do all the time.
People who’ve been told that to achieve an effect they would have to add this line in their config.toml file won’t read the whole file to scan whether there’s a better location, they’re used to add it at the end as they do for example in their .bashrc file. Of course, coming back later they would probably try to organize everything but they won’t because their first reaction is that there’s a bug in the application since the added line did nothing.
I don’t pretend that TOML is bad for everything but that such confusion makes it unsuitable for many programs as it has proven to be for broot.
I see. I wish you went into more detail about this in the blog post because it’s a good point.
I think the solution here is not to use the global scope at all in an INI/TOML file, such that it’s always clear what section an attribute belongs to. In addition, your application could throw warnings or even errors if unknown keys are detected (careful with backwards compatibility, though), or it could print the configuration back to the user so that they can check if everything is configured properly. It could also offer a dry run of configuration parsing.
Interestingly, I noticed that many Linux utilities have this configuration problem, and many just “swallow” any nonsensical configuration and exhibit default behaviour, something that has often cost me a lot of time. It’s even worse if they use their own configuration language that is often fraught with parser ambiguities and problems. It’s good to see developers like you caring about how users interact with the configuration file.
I think the solution here is not to use the global scope at all in an INI/TOML file
That seems the obvious solution to me as well, if this single confusion that users have is the only issue with the format. Generally speaking, I also think this is better than this proliferation of almost-but-not-quite JSON formats, each with their own “conveniences” added that might cause more confusion. Especially the unquoted string values and possibly the implicit comma on newline seem like footguns in disguise.
Comments, optional trailing commas and unquoted string keys seem like much more obvious advantages to me. Especially considering unquoted keys don’t have the ambiguity of “should the key be evaluated?” (which JS actually never does, but for example Python does) and “is this key a reserved JavaScript keyword?” like in JavaScript. But then, if you drop that it’s technically not eval() able anymore in a JS interpreter, which was arguably JSON’s biggest “advantage” (with all the security problems that entailed).
hjson is buggy, the standard has a “bug” that results in an ambiguity in its representation of certain items (at least when converted from json). I could have sworn I filed a bug on github a few years back when I wrote an automated toml <–> hjson converter in rust (via serde), but I cannot find it now.
I noticed ambiguities too. I’ve handled them by 1) not allowing free form parsing but only type guided deserialization into Rust types (my crate is named “deser_hjson” and I won’t write a serializer), 2) not allowing quoteless strings in some positions (that’s why I says Hjson is line oriented even if it doesn’t say it). I don’t think Hjson converters are safe, I wouldn’t go that way in Rust even if I used Hjson -> JSON converters in the past in JavaScript and Java.
I’m glad to see other people are using HJSON, many of our internal tools at work were using it as the configuration format or as way to capture simple structured datasets. I’m surprised it’s not more popular, given its simplicity both in the design and use, and the article makes really good points about its advantages vs YAML or TOML.
I personally enjoy using libucl for config files, and find HCL “ok”.
hjson looks pretty interesting though! Going to take a deeper look at it.
UCL looks very interesting, if there’s no trap or ambiguity. The first thing I notice is that you seem to be able to omit the ugly braces around the whole document.
I might consider creating a Rust deserializer for it… I’ll have to look more at the language.
I guess my blog post will have to be renamed into “Why broot switched from TOML to Hjson then UCL for its configuration files”…
FreeBSD has adopted ucl from its core OS configuration files. The interesting thing is that it’s a silently supported format for a lot of files that still use either the old conf format or a subset of ucl that resembles the old conf format. For example, the pkg.conf files can be written in one of two different configuration languages and the default is not ucl, but a completely different ucl syntax is available and automatically detected/supported.
Correct me if I’m wrong, but it seems hjson suffers from some of the same bugs that yaml has around type ambiguity. Primarily, in formats where unquoted strings are allowed a human may write
1000in a field which is supposed to be a string, and they’ll get a number.The main problem with YAML used as a configuration format written by human isn’t that
trueisn’t interpreted as a string, because you expect it. The real problem in my opinion is those weird rules which makes for exampleoffbe interpreted as a boolean. It’s so random most people don’t know it (see the “Norway problem” ( https://hitchdev.com/strictyaml/why/implicit-typing-removed/ ) too). Hjson doesn’t have the same level of insanity (true is only true, not On nor YES or yes).So I think this problem, for Hjson, isn’t really a problem. Or a small one.
Now, I said, perhaps in a cryptic way, that even the weird rules of YAML “may matter or not”. Because there’s absolutely no problem here if you use Hjson (or YAML) the way I do.
The reason is that the best way to deserialize a configuration, in my opinion, isn’t to use the automatically guessed type but to deserialize into the type required by the strongly typed structure.
And that’s the only public API I’ve put in my deserializer: you can’t use my Hjson deserializer ( https://github.com/Canop/deser-hjson) to read a free form file, you can only use it to deserialize into a Rust type, precisely to avoid the dangers related to type ambiguity. If the human typed
1000in a field which is expected in Rust to be a string, it will be read as a string.Agreed; deserializing into typed data structures really does make most of the issues with YAML go away in my experience. The other thing that helps when I’m writing YAML to configure code I didn’t write is to always wrap string values in quotes. Yes, it adds a little noise, but it is a pretty easy habit to pick up and then all the “what type is this value, anyway?” problems with YAML become irrelevant. The added noise of quoted strings isn’t too bothersome to me, though that’s obviously subjective.
Ah, yes, forcing deserialization to a datatype is a good way to deal with this, vs having some general HJson type and making an arbitrary decision about the types of fields. That said, I still find syntactic ambiguities like this one to be a major downside of these human readable formats.
It’s fascinating to me that there’s so much diversity in config formats. Maybe it’s just that everyone needs one and they’re easy to quibble with.
That said, I like that HJSON explicitly aims “to be used like a user interface for humans, to read and edit before passing the JSON data to the machine.” This seems similar to Strict YAML, which also focuses on giving humans a low-surprise way to define data structures that computers will eventually treat like JSON.
I would consider using HJSON over Strict YAML next time I need a config.
I don’t think there so much diversity. There’s less than a dozen common configuration formats, none of them really good (writing a deserializer for Hjson made me more aware of its shortcomings, especially for the quoteless strings and its indecisiveness regarding whether it’s a line oriented format or not).
I didn’t know about Strict YAML. It’s interesting and I’ll probably amend the document to mention it.
I personally would also be interested in comparision between various human readable JSONs. There are at least:
In my opinion, Json5, which I do use, is a better JSON, but while it’s suited for data exchange it’s behind Hjson for configuration (which wasn’t probably the main goal). It especially still asks for commas and doesn’t solve the indentation problem for multiline strings.
Rome-json (which I didn’t know) seems to be an intermediate between JSON and Hjson.
edit: I’ll edit my blog post to link to those alternatives
I don’t think it’s that many, when being compared to programming languages, image formats, etc. I also wouldn’t call all of these config format, in the sense that they were made to be mostly written by humans. JSON (and to a degree YAML) is a human readable data serialization format, and in many situations is generated vs. hand written. UCL/HCL and TOML are way more frequently touched manually.
Of course that doesn’t mean you cannot/should under no circumstances use other formats. I think they are tools to choose from and while I would agree there is a lot of “someone trying their own thing” projects, the same is again true for programming languages and many other formats.
It’s also a good thing that people experiment. In the majority projects you will still end up with a hand full of variation. So I think it’s more that people are interested in this topic and there is a lot of talk about it.
If there is one thing I wish configuration languages would adopt, its to drop commas all together. Clojure’s EDN format does this and its great. Whitespace is a natural separator for lists.
i have a config language that i wrote that does that. it converts to json, so for a while i was writing configs with this as a json preprocessor; the
mooncommand converts moon files to json withmoon to json; programs would just use the json file. i haven’t used it much recently because it drives other people nuts when you say “hey let’s use this config format that i invented that nobody else has ever seen”. https://github.com/jordanorelli/moonA project with 30 stars isn’t really unknown. It looks like a good configuration language at first glance.
It would be interesting to have a precise comparison of the differences with what looks like the nearest one: Hjson.
Hjson lets you totally drop commas if you want. There’s none in my example.
I have a hard time believing that anyone who has ever written or modified an INI file would be confused about this. Considering that this is the only illustrated criticism of TOML in this entire post, I’m puzzled as to why this person switched away from TOML.
Now, I understand what they mean when they call the language alien and confusing. But so are YAML, XML, etc. when you first encounter them, it’s just that people are accustomed to these decades old languages. Is it possible to create a configuration language that is intuitive to inexperienced newcomers and covers all the use-cases of TOML? Time will tell.
The problem isn’t with the software author. Being the author of broot, I wouldn’t have to change the format if I was the sole user. The problem is when people stop trying to configure your software because the changes they do have no effect.
Adding a line to a configuration file is something that linux users do all the time.
People who’ve been told that to achieve an effect they would have to add this line in their config.toml file won’t read the whole file to scan whether there’s a better location, they’re used to add it at the end as they do for example in their .bashrc file. Of course, coming back later they would probably try to organize everything but they won’t because their first reaction is that there’s a bug in the application since the added line did nothing.
I don’t pretend that TOML is bad for everything but that such confusion makes it unsuitable for many programs as it has proven to be for broot.
I see. I wish you went into more detail about this in the blog post because it’s a good point.
I think the solution here is not to use the global scope at all in an INI/TOML file, such that it’s always clear what section an attribute belongs to. In addition, your application could throw warnings or even errors if unknown keys are detected (careful with backwards compatibility, though), or it could print the configuration back to the user so that they can check if everything is configured properly. It could also offer a dry run of configuration parsing.
Interestingly, I noticed that many Linux utilities have this configuration problem, and many just “swallow” any nonsensical configuration and exhibit default behaviour, something that has often cost me a lot of time. It’s even worse if they use their own configuration language that is often fraught with parser ambiguities and problems. It’s good to see developers like you caring about how users interact with the configuration file.
Thanks, I’m editing my blog post to take this into account, and try to make it clearer.
That seems the obvious solution to me as well, if this single confusion that users have is the only issue with the format. Generally speaking, I also think this is better than this proliferation of almost-but-not-quite JSON formats, each with their own “conveniences” added that might cause more confusion. Especially the unquoted string values and possibly the implicit comma on newline seem like footguns in disguise.
Comments, optional trailing commas and unquoted string keys seem like much more obvious advantages to me. Especially considering unquoted keys don’t have the ambiguity of “should the key be evaluated?” (which JS actually never does, but for example Python does) and “is this key a reserved JavaScript keyword?” like in JavaScript. But then, if you drop that it’s technically not
eval()able anymore in a JS interpreter, which was arguably JSON’s biggest “advantage” (with all the security problems that entailed).hjson is buggy, the standard has a “bug” that results in an ambiguity in its representation of certain items (at least when converted from json). I could have sworn I filed a bug on github a few years back when I wrote an automated toml <–> hjson converter in rust (via serde), but I cannot find it now.
I noticed ambiguities too. I’ve handled them by 1) not allowing free form parsing but only type guided deserialization into Rust types (my crate is named “deser_hjson” and I won’t write a serializer), 2) not allowing quoteless strings in some positions (that’s why I says Hjson is line oriented even if it doesn’t say it). I don’t think Hjson converters are safe, I wouldn’t go that way in Rust even if I used Hjson -> JSON converters in the past in JavaScript and Java.
I’m glad to see other people are using HJSON, many of our internal tools at work were using it as the configuration format or as way to capture simple structured datasets. I’m surprised it’s not more popular, given its simplicity both in the design and use, and the article makes really good points about its advantages vs YAML or TOML.