1. 1

    This is a great solution if you are already running prometheus, or if you are interested in doing so. I do like the simplicity of hchk.io for cases where I don’t want to run prometheus (and related services/tooling like grafana, and push-gateway).

    Great idea and writeup though! Next time I have to run prometheus at a job, I’ll definitely keep this in mind for tracking the errant cron jobs that always seems to sneak in there somewhere.

    1. 1

      As I mentioned in https://blog.bejarano.io/alertmanager-alerts-with-amazon-ses/#sup1, I do not run Grafana or any dashboarding because I consider it worthless and time-consuming to set up.

      Thanks for the feedback!

      1. 1

        At a small scale the expression browser is sufficient (I use it for most of my work), but once you get beyond that something like Grafana is essential.

    1. 4

      I run Prometheus at home, though I’m obviously a bit biased there.

      1. 2

        I’m not biased and I run Prometheus at home, and elsewhere. Blackbox Exporter running on RPis in various physical/network locations, with ZeroTier. Most of the Blackbox Exporter target configuration uses DNS service discovery. Alert Manager for alerting. I’ve used many different monitoring systems and recommend Prometheus+Grafana, with Netdata for some low-level monitoring.

      1. 1

        People is suggesting keeping your gmail account “alive” for a while, but in the case of that account being bound to something that you own, like your Git commits somewhere, it means that you’ll have to keep that account safe, forever.

        I have two questions:

        • Is there a way of changing your commit history to reflect to a new email address that does not belong to a centralized corporation but to you, in the form of a domain you own.
        • Is it possible to use another identification mechanism, a signature that is not bound to an email address? An email address requires infrastructure to work, and that eventually could belong to someone else, like your the domain your email is part of
        1. 2

          Is there a way of changing your commit history to reflect to a new email address that does not belong to a centralized corporation but to you, in the form of a domain you own.

          Yes in theory, however that changes all the hashes so no in practice.

          1. 2

            in my experience, just start committing with the new address and update any mailmap and authors files. can’t do anything about published history…

            1. 1

              You could use git filter-branch to rewrite the entire git repository to replace your old e-mail address with your new one, but that will change the hash of every commit so it will be a terrible experience for anyone who has an existing clone of your repository. I think it’s not worth it.

              1. 1

                Is it possible to use another identification mechanism, a signature that is not bound to an email address? An email address requires infrastructure to work, and that eventually could belong to someone else, like your the domain your email is part of

                In GitHub, you can choose to keep your email private and use something to the tune of username@users.noreply.github.com. See the details here

              1. 14

                I still can’t get over the fact that someone got an idea to refresh HTML document tree 60 times per second and make HTML document viewer render it over and over and call it as “application”.

                It’s so wrong on just too many levels that I don’t even know where to start, and people basically just don’t even notice that.

                1. 11

                  But it doesn’t actually work the way? In AJAX apps, DOM is only updated on events (e.g. click, user network data received). You would have to have a timer to actually update it regularly in the background.

                  Probably the place it gets close to that is when they hijack the scroll event, which is horrible. But even that’s not changing the DOM if you’re not scrolling.

                  FWIW I agree with the premise of the OP, but I don’t think your criticism is accurate.

                  1. 7

                    It’s not the first time that someone got an idea to build GUIs by just extending existing document-rendering technology…

                    1. 9

                      DPS is a little bit different, because postscript is a programming language (specifically, a forth dialect with logo/turtle-style pen control). It’s relatively sensible to do widget-drawing with a language optimized for drawing arbitrary line graphics. A web app is more like trying to use dot macros to modify an MS word document at 30fps.

                      1. 9

                        A web app is more like trying to use dot macros to modify an MS word document at 30fps.

                        That reminds me, years ago my dad, who was a chemical engineer in a large company, showed me a simulation he’d gotten as an email attachment from a colleague. It had a pretty decent graphical animation entirely within an Excel spreadsheet. Part of the sheet was a “normal” spreadsheet with the actual formulas, but another part had cells resized to be small and rectangular, and their colors were changed a few times a second by macros, producing a medium-resolution raster-graphics display basically. This was apparently relatively common, because it made the graphical output self-contained within the same spreadsheet that you could mail around.

                    2. 7

                      I am not actually that offended by this idea, because most GUI applications are enhanced document viewers. But I do think that when your application requires to be run at 60 fps, you should use something else.

                      For example: The interoperability problem has already been solved with Java and if you really need something with more performance than that, you’d basically have to resort to lower level code like C/C++.

                      But if “a glorified document viewer and/or editor” all your application is, then an web-application will more than suffice.

                      1. 3
                        1. 5

                          Web apps are a cool hack, and I absolutely love the perverse joy one gets from making something impressive using the absolute wrong tools for the job. But, the point of a cool hack is that the idea that somebody would use it seriously or for important tasks is absurd.

                        2. 3

                          A developer equivalent of https://xkcd.com/763/

                          1. 2
                          1. 1

                            I’m loving the “log stdout” part, everything else can basically be ignored.

                            1. 2

                              That’s definitely an improvement over the syslog situation, at least for our deployments. The native Prometheus export is neat as well; saves having to build an adapter to run alongside for metrics.

                              1. 2

                                I was partly joking. Not being able to log to stderr or stdout has caused so many problems because it’s basically impossible to debug haproxy without syslog being present (and HAProxy has the annoying tendence to stop logging if syslog hangs up such as happens when the network has a hiccup in an rsyslog sitaution)

                                1. 1

                                  That exporter is one of the oldest: https://github.com/prometheus/haproxy_exporter

                                  1. 2

                                    Nope, this is a new, exporter-less endpoint, built into HAProxy itself: https://www.haproxy.com/blog/haproxy-exposes-a-prometheus-metrics-endpoint/

                              1. 3

                                Speaking as a Prometheus developer, it’s very easy to run Prometheus locally and I’ve done this in the past to debug both Prometheus (https://www.robustperception.io/optimising-go-allocations-using-pprof) and other applications. Most of the time I’m debugging the sort of issue that metrics aren’t suitable for though, so I’ll be print lining.

                                1. 4

                                  https://www.robustperception.io/blog covers the Prometheus monitoring system, how to use it and why it is the way it is.

                                  1. 2

                                    Could one store the IP address of the initial request that causes you to generate a JWT in the token itself? Then you can validate that the current request comes from the same IP. If they’re different, then force them to log in again from their current IP.

                                    The user would need to re-login if they turn on a VPN or change locations, but that’s a small price to pay if that reduces the possibility for certain types of attacks. I’m definitely not a security expert, but working on a fairly sensitive app where a breach would be bad for a user. The fact that I haven’t seen this suggested next to more complex safeguards makes me think there’s a fundamental flaw in it that I’m just not thinking of.

                                    1. 5

                                      IPs aren’t a great factor to base stuff like this one, although that’s a good idea.

                                      I think what’s better is something like token binding (https://datatracker.ietf.org/wg/tokbind/documents/) which is a way to pin a certain token to a specific TLS session. This way you have some basic guarantees. But in the real world things are sorta messy =p

                                      1. 2

                                        Most home users would have to re log in every day. Services that tie my login to an IP address piss me off so much because they are constantly logging me out.

                                        1. 2

                                          The fact that I haven’t seen this suggested next to more complex safeguards makes me think there’s a fundamental flaw in it that I’m just not thinking of.

                                          It’s not a safe presumption that a users requests will always come from the same IP - even from request to request. Their internet access could be load balanced or otherwise change due to factors like roaming.

                                          1. 1

                                            Yeah that is also a common technique for cookies. If the remote IP changes you can invalidate the cookie.

                                          1. 4

                                            Nice article.

                                            Beware that InstrumentHandler is deprecated, and the functions in https://godoc.org/github.com/prometheus/client_golang/prometheus/promhttp are the recommend replacement.

                                            Splitting out latency with a success/failure label is also not recommended as a) if you have only successes or only failures, your queries break and b) users tend to create graphs of only success latency and miss all those slow failing requests. Separate success and failure metrics are better, and also easier to work with in PromQL.

                                            1. 3

                                              Thanks for the suggestions Brian! promhttp package contains even more nice things like in flight requests. Maybe we should explicitly say ok in docs that InstrumentHandler is deprecated in favor of promhttp types? I don’t mind making a PR in docs

                                            1. -1

                                              [Title] /proc/<pid>/stat is broken

                                              This sounds serious! Is the content of the pseudo-file associating incorrect PIDs or parent PIDs to processes?

                                              Let’s continue…

                                              Documentation (as in, man proc) tells us to parse this file using the scanf family, even providing the proper escape codes - which are subtly wrong.

                                              So it’s a documentation issue…

                                              When including a space character in the executable name, the %s escape will not read all of the executable name, breaking all subsequent reads

                                              I have literally never encountered an executable with a space in the name, although it’s perfectly legal from a file name perspective. (I’ve been a Linux user since 1998).

                                              The only reasonable way to do this with the current layout of the stats file would be to read all of the file and scan it from the end […]

                                              So… let’s do this instead?

                                              The proper fix (aside from introducing the above function) however should probably be to either sanitize the executable name before exposing it to /proc//stat […]

                                              Sounds reasonable to me.

                                              […], or move it to be the last parameter in the file.

                                              Thus breaking all existing implementations that rely on the documentation in man proc. But I guess it can be done in some backwardly compatible way?

                                              This problem could potentially be used to feed process-controlled data to all tools relying on reading /proc//stat

                                              I can’t really parse this. Do you mean “affect” instead of “used”?

                                              In conclusion: I can’t see any evidence of the functionality of this proc pseudo-file being “broken”. You have encountered an edge case (an executable name with a whitespace character in it). You’ve even suggested a workaround (scan from the end). If you had formulated this post as “here’s a workaround for this edge case” I believe you would have made a stronger case.

                                              1. 5

                                                I have literally never encountered an executable with a space in the name

                                                Well, tmux does this, for example. But my primary concern is not has it ever happened to me but, if it happens, what will my code do?. As this is a silent failure (as in, the recommended method fails in a non-obvious way without indicating failure), no action is taken by most implementations to guard against this. That, in my mind, counts as broken, and the least thing to do is to fix the documentation. Or expose single parameters in files instead of a huge conglomeration with parsing issues. Or… see above.

                                                So… let’s do this instead?

                                                I do, but only after I got sceptical while reading the documentation, ran some tests and had my hunch confirmed. Then I checked to see others making that very mistake.

                                                Thus breaking all existing implementations that rely on the documentation in man proc. But I guess it can be done in some backwardly compatible way?

                                                No, I don’t think so - except for introducing single-value files (and leaving /proc/<pid>/stats be as it is).

                                                This problem could potentially be used to feed process-controlled data to all tools relying on reading /proc//stat

                                                I can’t really parse this. Do you mean “affect” instead of “used”?

                                                Admittedly, English is not my first language, I do however think that sentence parses just fine. The discussed problem (which is present in several implementations based on the documentation), can potentially be used to inject data (controlled by the process, instead of the kernel) into third-party software.

                                                In conclusion: I can’t see any evidence of the functionality of this proc pseudo-file being “broken”.

                                                That depends on your view of broken - if erroneous documentation affecting close to all software relying on it with a silent failure does not sound broken to you, I guess it is not.

                                                You have encountered an edge case (an executable name with a whitespace character in it).

                                                I actually did not encounter it per se, I just noticed the possibility for it. But it is an undocumented edge case.

                                                You’ve even suggested a workaround (scan from the end).

                                                I believe that is good form.

                                                If you had formulated this post as “here’s a workaround for this edge case” I believe you would have made a stronger case.

                                                Maybe, but as we can see by the examples of recent vulnerabilities, you’ll need a catchy name and a logo to really get attention, so in my book I’m OK.

                                                1. 1

                                                  Thanks for taking the time to answer the questions I have raised.

                                                  The discussed problem (which is present in several implementations based on the documentation), can potentially be used to inject data (controlled by the process, instead of the kernel) into third-party software.

                                                  Much clearer, thanks.

                                                  On the use of “broken”

                                                  I’m maybe extra sensitive to this as I work in supporting a commercial software application. For both legal and SLA[1] we require our customers to be precise in their communication about the issues they face.

                                                  [1] Service level agreement

                                                  1. 1

                                                    Followup: can you give a specific example of how tmux does this? I checked the running instances of that application on my machine and only found the single word tmux in the output of stat files of the PIDs returned by pgrep.

                                                    1. 2

                                                      On my Debian 9 machine, when starting a tmux host session, the corresponding /proc/<pid>/stat file contains:

                                                      2972 (tmux: client) S 2964 2972 2964 […]

                                                2. 3

                                                  “Thus breaking all existing implementations that rely on the documentation in man proc. But I guess it can be done in some backwardly compatible way?”

                                                  I will never get the 100ms it took to read this sentence back….

                                                  1. 1

                                                    I dunno, maybe just duplicate the information at the end of the current format, in the author’s preferred format, and delimited by some character not otherwise part of the spec.

                                                    It’s not trivial, though.

                                                    That was my point.

                                                  2. 1

                                                    this was clearly overlooked when the api was designed, nobody is parsing that file from the end and nobody is supposed to

                                                    1. -1

                                                      What was overlooked? That executables can have whitespace in their names?

                                                      I can agree that this section of the manpage can be wrong (http://man7.org/linux/man-pages/man5/proc.5.html, search for stat):

                                                      (2) comm  %s
                                                          The filename of the executable, in parentheses.
                                                          This is visible whether or not the executable is
                                                          swapped out.
                                                      

                                                      From the manpage of scanf:

                                                      s: Matches a sequence of non-white-space characters; the next
                                                          pointer must be a pointer to the initial element of a
                                                          character array that is long enough to hold the input sequence
                                                          and the terminating null byte ('\0'), which is added
                                                          automatically.  The input string stops at white space or at
                                                          the maximum field width, whichever occurs first.
                                                      

                                                      So it’s clear no provision was made for executables having whitespace in them.

                                                      This issue can be simply avoided by not allowing whitespace in executable names, and by reporting such occurrences as a bug.

                                                      1. 8

                                                        This issue can be simply avoided by not allowing whitespace in executable names, and by reporting such occurrences as a bug

                                                        Ahhh, the Systemd approach to input validation!

                                                        Seriously, if the system allows running executables with whitespace in their names, and your program is meant to work with such a system, then it needs to work with executables with whitespace in their names.

                                                        I agree somewhat with the OP - the interface is badly thought out. But it’s a general problem: trying to pass structured data between kernel and userspace in plain-text format is, IMO, a bad idea. (I’d rather a binary format. You have the length of the string encoded in 4 bytes, then the string itself. Simple, easy to deal with. No weird corner cases).

                                                        1. 1

                                                          I agree it’s a bug.

                                                          However, there’s a strong convention that executables do not have whitespace in them, at least in Linux/Unix.[1]

                                                          If you don’t adhere to this convention, and you stumble across a consequence to this, does this mean that a format that’s been around as long as the proc system is literally broken? That’s where I reacted.

                                                          As far as I know, nothing crashes when you start an executable with whitespace in it. The proc filesystem isn’t corrupted.

                                                          One part of it is slightly harder to parse using C.

                                                          That’s my take, I’m happy to be enlightened further.

                                                          I also agree that exposing these kind of structures as plain text is arguably … optimistic, and prone to edge cases. (By the way, isn’t one of the criticisms of systemd that it has an internal binary format?).

                                                          [1] note I’m just going from personal observation here, it’s possible there’s a subset of Linux applications that are perfectly fine with whitespace in the executable name.

                                                          1. 3

                                                            I agree with most of what you just said, but I myself didn’t take “broken” to mean anything beyond “has a problem due to lack of forethought”. Maybe I’m just getting used to people exaggerating complaints (heck I’m surely guilty of it myself from time to time).

                                                            It’s true that we basically never see executables with a space (or various other characters) in their names, but it can be pretty frustrating when tools stop working or don’t work properly when something slightly unusual happens. I could easily see a new-to-linux person creating just such an executable because they “didn’t know better” and suffering as a result because other programs on their system don’t correctly handle it. In the worst case, this sort of problem (though not necessarily this exact problem) can lead to security issues.

                                                            Yes, it’s possible to correctly handle /proc/xxx/stat in the presence of executables with spaces in the name, but it’s almost certain that some programs are going to come into existence which don’t do so correctly. The format actually lends itself to this mistake - and that’s what’s “broken” about it. That’s my take, anyway.

                                                            1. 2

                                                              Thanks for this thoughtful response. I believe you and I are in agreement.

                                                              Looking at this from a slightly more usual perspective, how does the Linux system handle executables with (non-whitespace) Unicode characters?

                                                              1. 3

                                                                Well, I’m no expert on unicode, but I believe for the most part Linux (the kernel) treats filenames as strings of bytes, not strings of characters. The difference is subtle - unless you happen to be writing text in a language that uses characters not found in the ASCII range. However, UTF-8 encoding will (I think) never cause any bytes in the ASCII range (0-127) to appear as part of a multi-byte encoded character, so you can’t get spurious spaces or newlines or other control characters even if you treat UTF-8 encoded text as ASCII. For that reason, it poses less of a problem for things like /proc/xxx/stat and the like.

                                                                Of course filenames being byte sequences comes with its own set of problems, including that it’s hard to know encoding should be used to display filenames (I believe many command line tools use the locale’s default encoding, and that’s nearly always UTF-8 these days) and that a filename potentially contains an invalid encoding. Then of course there’s the fact that unicode has multiple ways of encoding the exact same text and so in theory you could get two “identical” filenames in one directory (different byte sequences, same character sequence, or at least same visible representation). Unicode seems like a big mess to me, but I guess the problem it’s trying to solve is not an easy one.

                                                                (minor edit: UTF-8 doesn’t allow 0-127 as part of a multi-byte encoded character. Of course they can appear as regular characters, equivalent to the ASCII).

                                                                1. 1
                                                                  ~ ❯ cd .local/bin
                                                                  ~/.l/bin ❯ cat > ą << EOF
                                                                  > #/usr/bin/env sh
                                                                  > echo ą
                                                                  > EOF
                                                                  ~/.l/bin ❯ chmod +x ą 
                                                                  ~/.l/bin ❯ ./ą
                                                                  ą
                                                                  
                                                              2. 2

                                                                If you don’t adhere to this convention, and you stumble across a consequence to this, does this mean that a format that’s been around as long as the proc system is literally broken?

                                                                Yes; the proc system’s format has been broken (well, misleadingly-documented) the whole time.

                                                                As you note, using pure text to represent this is a problem. I don’t recommend an internal, poorly-documented binary format either: canonical S-expressions have a textual representation but can still contain binary data:

                                                                (this is a canonical s-expression)
                                                                (so "is this")
                                                                (and so |aXMgdGhpcw==|)
                                                                

                                                                An example stat might be:

                                                                (stat
                                                                  (pid 123456)
                                                                  (command "evil\nls")
                                                                  (state running)
                                                                  (ppid 123455)
                                                                  (pgrp 6)
                                                                  (session 1)
                                                                  (tty 2 3)
                                                                  (flags 4567)
                                                                  (min-fault 16)
                                                                  …)
                                                                

                                                                Or, if you really cared about concision:

                                                                (12345 "evil\nls" R 123455 6 1 16361 4567 16 …)
                                                                
                                                            2. 3

                                                              nobody is parsing that file from the end

                                                              As an example the Python Prometheus client library uses this file, and allows for this.

                                                        1. 3

                                                          Nope. It’s 2017, it’s time to stop parsing strings with regular expressions. Use structured logging.

                                                          No thanks! I’ll stick to strings.

                                                          1. 2

                                                            Could you please explain why? Your comment, as it is, is not bringing any value.

                                                            1. 1

                                                              Not the OP but here’s why I don’t like structured logging

                                                              • logs will ultimately be read by humans and extra syntax gets in the way.
                                                              • structured logging tends to bulk the log with too much useless information.
                                                              • most of the use cases of structured logging could be better handled via instrumentation/metrics.
                                                              • string based logs can be emitted by any language without dependencies so every system you manage could have compatible logging.

                                                              Arguably a space separated line is a fixed-schema structured log with the least extraneous syntax possible.

                                                              1. 6

                                                                To me (in the same order):

                                                                • logs are ultimately read by human once correctly parsed/sorted. Which means that it should be machine readable first so that it can be processed easily to create a readable message.
                                                                • Too much informations is rarely a problem with logging, but not enough context is often an issue.
                                                                • Probably, but structured logging still offers some simpler ways for this.
                                                                • You just push the format problematic from the sender (that can use a simple format) to the receiver (that has to parse different formats according to what devs fancy)

                                                                To me the best recap on why I like structured logging is: https://kartar.net/2015/12/structured-logging/

                                                                1. 2

                                                                  most of the use cases of structured logging could be better handled via instrumentation/metrics.

                                                                  Speaking as a developer of Prometheus, you need both. Metrics are great for an overall view of the system and all its subsystems, but can’t tell you about every individual user request. Logs tell you about every request, but are limited in terms of understanding the broader system.

                                                                  I’ve wrote a longer article that touches on this at https://thenewstack.io/classes-container-monitoring/

                                                            1. 4

                                                              The public Prometheus/Grafana dashboard for the streaming: https://dashboard.congress.ccc.de/?refresh=5m&orgId=1

                                                              1. 3

                                                                I’d recommend reading My Philosophy on Alerting. Systems like Prometheus are designed to allow more sophisticated alerting, such as predicting when a disk will fill.

                                                                1. 1

                                                                   I had to spend time justifying the presence of that ‘0’ many times over the years as other developers questioned its purpose.

                                                                  I don’t get why he wouldn’t just put a code comment, or a regression test that checks memory usage. Then he doesn’t need to say anything.

                                                                  1. 10

                                                                    Per the article, he did both of those.

                                                                    1. 3

                                                                      That’s what I get for skimming and commenting. I deserve to look dumb there.

                                                                  1. 9

                                                                    tl;dr sudo parses /proc, fucks up

                                                                    1. 18

                                                                      I think that sells it a little short. There’s something to be said about a system design where parsing strings in /proc is a thing, how A leads to B leads to root, etc.

                                                                      1. 4

                                                                        It also illustrates why procfs in and of itself is bad for security.

                                                                        1. 4

                                                                          I’d take it more that hard to parse formats due to a poor choice of how to handle field separators that appears in your data lead to bugs. This particular parsing issue is one I’ve run into myself.

                                                                          1. 4

                                                                            Really it illustrates that plain text (byte sequences) as popularized by unix is a poor interface format. Unfortunately it continues to be popular because C’s retrograde type system and poor literal support discourage those who still write C from using anything better.

                                                                            1. 3

                                                                              This really has nothing to do with C, and the language blaming is unwarranted. There are perfectly good (C) APIs that do not involve parsing text, and such an API could have been used here. But some people think parsing text is still the way to go.

                                                                              1. 2

                                                                                For most things it’s very difficult to express a good API without sum types. In C you can’t even fake them with polymorphism and the visitor trick.

                                                                            2. 2

                                                                              Human readable formats vs machine readable formats, really…

                                                                          2. 2

                                                                            What does OpenBSD do there?

                                                                            1. 2

                                                                              In general, sysctl.

                                                                        1. 5

                                                                          Something I’ve noticed is docs that are reference have a tendency over time to expand also into user guides.

                                                                          I’ve never seen it work out, as the two use cases are very different.

                                                                          One requires quite technical and specific information. If you mix in guides, the reader is left to carefully read the guide to see if there’s a subtlety explained therein that is relevant.

                                                                          The other is more along the lines of a tutorial. Mentioning all the fine print only confuses a new user, as such details aren’t relevant to them at this stage.

                                                                          1. 5

                                                                            Strong agree. Long ago, I gave a talk about this: https://air.mozilla.org/rust-meetup-december-2013/

                                                                            TL;DR, API docs, guides, and reference materials have three different audiences, and so need to be three different things.

                                                                            1. 1

                                                                              Sorry, I’m too lazy to watch the talk. I can understand the difference between API docs and guides. What makes reference materials different to API docs?

                                                                          1. 2

                                                                            Processing of FOSDEM videos is ongoing, about 80% are processed. 544 are currently available.

                                                                            1. 0

                                                                              See, if we wanted to make the world a better place, everyone would pick a month to submit patch requests to all open source projects to replace their config files with JSON and to add a converter for legacy configs.

                                                                              And yes, I also want a pony. And world peace.

                                                                              1. 30

                                                                                Wait, making every configuration file with JSON would make the world a better place? I think it would turn it far worse! A lack of comments in my configuration file makes future me confused and frustrated at past me.

                                                                                1. 5

                                                                                  When I’m stuck with JSON-for-config, I’ll sometimes duplicate keys in the file - putting the comment in the first one. Every JSON decoder I’ve used so far ignores the first value and takes the second one.

                                                                                  I’m well aware of how insane that sounds, but I’d rather have comments than sensible files.

                                                                                  1. 3

                                                                                    I do something similar but make it an _{{key}}-comment (e.g. password has a sibling _password-comment before it). This works great in config files as it can be ignored by the application. It does fall apart in something like package dependencies though =(

                                                                                    1. 3

                                                                                      I do something similar, except I duplicate a specific key, so order is not important:

                                                                                      {
                                                                                        "//": "Here's a comment",
                                                                                        …
                                                                                        "//": "Another comment"
                                                                                      }
                                                                                      

                                                                                      But yeah, I’d rather just avoid JSON for such things.

                                                                                      1. 1

                                                                                        Some of these problems are addressed by JSON5: http://json5.org/ There is obviously the whole https://xkcd.com/927/ problem, but IMO it’s not as bad as it is with something dramatically different like TOML.

                                                                                        But in general I think it’s much better for a whole host of reasons to use Lua for configuring end-user applications.

                                                                                        1. 1

                                                                                          JSON5

                                                                                          Kind of reminds me of UCL.

                                                                                      2. 2

                                                                                        One problem with this, and with @alva’s solution below, is that the comments won’t survive deserialization/serialization in a meaningful way. @twolfson’s solution resolves that, though there’s still no guarantee the comment key is serialized anywhere near the key it is meant to be commenting.

                                                                                      3. 1

                                                                                        Having all configuration files in the same format would make the world a better place.

                                                                                        JSON does have some technical issues though, such as inability to represent all floating point values.

                                                                                        1. -1

                                                                                          Is the world really betterr off having different formats for grub, xorg, Apache, nginx, rust, npm, and all the other myriad ways of storing the same flavor of data?

                                                                                          1. 7

                                                                                            No, but there are enough standards for this that don’t suck as much for configuration data as json.

                                                                                            1. 1

                                                                                              yeah, all three of the systems suggested in the OP are better than json. i believe yaml has some parsing issues, but the other two look very clean and usable.

                                                                                              1. -1

                                                                                                Other than the common “muh comments” complaint, why do you think JSON sucks for configuration data?

                                                                                                1. 12

                                                                                                  Two reasons that come to mind for me are: you get a lot of syntactic ceremony that other formats don’t require (every file starting with ‘{’, string quoting, commas between list items, etc.) and the selection of types is odd. Your JSON parser is going to convert bare numbers to numbers but you have to quote your strings. What do you do if you have other types, like URLs or date/times or something particular to your application? You’ll have to quote them as strings and then do a second pass over your JSON configuration object to convert it to something else. What happens if your language has a more interesting suite of numeric types than JSON does? Do you have to tell people to quote their numbers so that you can parse them properly? The behavior is hidden from you by your JSON parser, so you aren’t likely to be able to detect when something like this has gone wrong.

                                                                                                  I like my code to require a certain amount of ceremony to catch problems before running, but I like my configuration files to be fairly lenient; if I can recover something from them, I can alert the user that the parse failed on these items or whatever and proceed. These options get narrowed when you conflate a programming language with a configuration language.

                                                                                                  In fact, even my JSON parser, I want it to be strict when I’m dealing with user input or form submissions or whatever, but if I’m reading a file like ~/.foorc, I want to be lenient. Does your JSON parser have options for that?

                                                                                              2. 5

                                                                                                Just because there is not one standard it doesn’t mean that JSON is a suitable one. In fact I would even prefer XML over JSON for this particular usecase, just because it has comments.

                                                                                            2. 7

                                                                                              JSON config files?! Do you have a shrine to Satan in your house as well?

                                                                                              1. 1

                                                                                                I might not mind if most json parsers sucked totally when it comes to error messages. rather i would prefer a good config parser which generates error messages and documentation WITH EXAMPLES.

                                                                                              1. 2

                                                                                                I think what the author of this article doesn’t understand is that Java and Go are very different languages and as such the characteristics of a good GC for Java are quite different from a good GC for Go. Java creates an order of magnitude more garbage than Go, often creating large numbers of objects with very short lifetimes. This means the GC is very stressed all the time in aggressively cleaning up short lived garbage. Go on the other hand allocates most small objects on the stack so the GC never has to worry about them at all. This frees up the GC to deal with mostly larger, mostly longer lived objects which is a much easier task than the Java GC has to deal with.

                                                                                                The other side of GC being an easier task for Go is that it’s much easier to optimise. When you think about it it’s not surprising that Go’s GC performs significantly better than Java’s. It’s just a much easier task.

                                                                                                1. 13

                                                                                                  This means the GC is very stressed all the time in aggressively cleaning up short lived garbage. Go on the other hand allocates most small objects on the stack so the GC never has to worry about them at all.

                                                                                                  I develop a Go application which with only moderate load generates 100MB/s of very short lived, small objects on the heap. This results in many gigabytes of RSS overhead which a different GC design may not have.

                                                                                                  Which GC is appropriate is more about allocation patterns than languages.

                                                                                                1. 7

                                                                                                  I’ve always trusted my ASUS laptops and I’ve heard good things about their ZenBook. It’s aluminum, seems to have decent specs for development, and reportedly works fine booting Linux distros.

                                                                                                  1. 7

                                                                                                    Can vouch for the ZenBook. Currently on the Asus ZenBook UX305. It’s a great laptop, and runs Linux perfectly (I’ve even run OpenBSD on it with very little trouble, apart from lacking trackpad support). Only issue with the hardware is the screen hinge is a little loose, so it flops around a bit (but I’m sure if I bothered to pop it open I could tighten the screw), but apart from that it’s great.

                                                                                                    1. 4

                                                                                                      Thirding the UX305. I bought a Broadwell-based one earlier this year to play with OpenBSD based on a @tedu post. Outside of the trackpad, it’s been great for on-the-go and taking to meetups.

                                                                                                      1. 1

                                                                                                        Forthing the UX305. Nice and light. SSD could be bigger and the aluminium scratches a bit easily but a nice performance/price point.

                                                                                                      2. 1

                                                                                                        My friend has a UX305 with Broadwell running Ubuntu he uses and he seems to really like it. It looks pretty nice!

                                                                                                        1. 1

                                                                                                          I have one for work and it’s pretty nice. The only caveat would be that if you live in Canada, avoid the one that has the French/English keyboard. It has a split the shift key on the left side and it is… not optimal.