Threads for 0x2ba22e11


    This sounds so broken. I kinda suspect the amount of code that uses Data.Ratio instead of Rational might be negligible across the whole Haskell ecosystem.

    1. 5

      As always, a nice and thorough report. Good work @aphyr!

      Is anybody using Redpanda instead of Kafka at work?

      1. 5

        No, but I’ve definitely been tempted, at least for low risk/ dev environments, but haven’t pulled the trigger. I’m awaiting to give it a good spin first. I’ve been looking for something like this post though.

        FWIW I’ve been reluctant mostly because having new folks working with Kafka is tricky enough without additional “Kafka compatibility “ issues popping up. Similar to the reasoning that had us decide doing kstreams in Java instead of scala or clojure. I’d file it as “nice but will break the innovation budget”.

        1. 3

          This may be the first time I hear the term: “innovation budget”. Astonishingly fitting.

          1. 6

            Sidenote if this topic is interesting to you: it was a principle in Rusts development under the name of “strangeness budget”.


            1. 6

              I believe it was popularized by this blog post from 2015: Choose Boring Technology. There it was conceptualized as innovation “tokens” like in friendlysock’s comment earlier.

              1. 3

                The concept of an “innovation budget” is something that is super important for all engineers, especially folks at a startup.

                Every team has some finite number of innovation tokens, which may or may not replenish over time. If you spend all of your innovation tokens on, say, building your webserver in Rust then you really can’t afford to do more exotic things like microservices or blue/green deploys or whatever else.

                Similarly, the model goes, if you pick a really boring webstack and ops stuff (say, modern-day Rails on Heroku), then you can spend those innovation tokens on more interesting problems like integer programming for logistics or something. The business version of this would be deciding to support multiple vendors or go with a new payment processor instead of just trusting Stripe or Braintree or whoever.

                My extension to the model is that, in a startup, you can exchange innovation tokens for efficiency tokens or cost coupons…if you build your stack in bespoke Julia it’s a lot harder to hire (and a lot harder to backfill departures!), whereas if you go with python or javascript then you can easily find interns or bodyshops to round out the workforce.

                One of the great open questions to me about the model is: how does an org deliberately replenish innovation tokens?


                  One of the great open questions to me about the model is: how does an org deliberately replenish innovation tokens

                  Apologies for necroing an old thread here but this just popped into my head: you can recycle an innovation token by using a given piece of originally-interesting technology until over time it becomes boring. This works if you’re huge or there arises a large community around it.

                  Rails is a prime example of this. It used to be an interesting choice and now it’s offered up as an example of a boring technology.

            2. 2

              Is anybody using Redpanda instead of Kafka at work?

              I’d run a PoC at work (and even found a minor bug). In my use-case the performance was exactly the same. In the end this did not move forward as I switched teams inside the organization and the old team went ahead with a managed service.

            1. 13

              The wire pulling example is exactly what I find to be the thing about coding in rust. If you stay within some default confines, you never get spooky action at a distance. Outside of OCaml I haven’t had that experience in other languages. It seems like Java or C++ should be close, but the write/compile/fix loop isn’t as tight in those and you end up with more chaotic trouble than the error guided refactoring I experience in Rust.

              1. 1


                There’s a saying “lean on the type system” in the haskell community. The advice there is that the more you encode your invariants in types, the better this kind of wire-pulling will work.

                I’ve recently had the experience that it’s really easy to see this effect in TypeScript, because in that language it is possible to write code with reasonably good or really bad, imprecise types. The difference is stark. With the former you can do big changes by changing one thing in one place then using compiler errors to find every other place that needs to match it. With the latter you find code still compiles after changes but crashes at runtime.

                It seems like Java or C++ should be close

                I think it’s not very surprising because they both have poor type systems.

              1. 10

                Generating C++ is an underrated strategy! C++ has great debugging and performance tools like uftrace, perf, CLion, etc. And you can link with anything.

                It can be annoying to write, e.g. writing all the redundant headers, but generating code from a different language solves that.

                Oil is doing something similar: Brief Descriptions of a Python to C++ Translator

                It gives us much better performance, as well as memory safety. There were several reactions like “Why C++? It’s not memory safe”. But the idea is to generate code with that property, as SerenityOS is doing. If you never write free() or delete in your source language, then there are no double-free / double-delete bugs.

                (The boundary between manually written C++ and generated C++ is still an issue .. I’ll be interested see their API. But it should be nicer and more direct than the interface between two totally separate languages.)

                There are also some interesting ways to generate fast code with C++ templates:

                i.e. you get a fairly nice language for writing a templatized “runtime”, and then you generate very high level typed code that targets it. Then the C++ compiler mushes it all together into fast code.

                Oil’s scheme is simpler, but it has this flavor. The garbage collector and templated types interact in an interesting way, e.g. consider tracing Dict<int, int> vs. Dict<int, Str*>. (I’m working on a blog post about this.)

                1. 5

                  There were several reactions like “Why C++? It’s not memory safe”.

                  As if assembly directly were any safer.

                  1. 3

                    D with the -betterC mode could also be a good target for transpilers. It’s got most of the good stuff from C++ and the imperative/inspection-based meta programming model could be easier to target, I think. Also has 3 compilers and all of them compile faster than C++ (no preprocessor etc). The only problem in Serenity’s case would be binding to existing C++ code seamlessly.

                    1. 2

                      Yeah D is nice because it’s garbage collected already! It is a shame that people seem to shy away for certain things because it is garbage collected, but I view that as a big benefit.

                      I’m guessing SerenityOS has a good reason for ARC though … So yeah that is a problem – GC’s are hard to tune and aren’t universal across applications. There was a thread here recently with Chris Lattner on GC vs ARC.

                      I think Nim lets you choose the allocator, but to me a big downside is that the generated code isn’t readable. It is a huge benefit to be able to read it, step through it in the debugger, pretty print data structures, etc.

                      1. 5

                        The -betterC subset is not GCed. It doesn’t have a runtime. However, the good stuff (RAII, meta-programming) is all there… it can be tedious to use by hand but as a transpiler target I think it is a good choice.

                    2. 1

                      Generating C++ is an underrated strategy

                      I have a suspicion that the reason it’s underrated is that common old wisdom says “since I’m going to all the effort to generate code at all, maybe I generate C instead to get faster compile times.”

                      I wonder to what extent that is no longer relevant? My belief is that there’s not an interesting difference now because most time goes into optimisers rather than parsing.

                      C++ totally makes sense for this context of course because they already have a C++ codebase which they want to link with. :)

                      1. 2

                        Well I definitely ran into some template bloat problems, so it’s a double-edged sword. I think it basically comes down to whether you want to use templates, classes, and exceptions, etc. My post mentions that:


                        Templates increase compile times but they can generate faster code, as the Souffle Datalog example shows. I would think of it as a very expressive and high level language for monomorphization.

                        Also, when you generate code you have more control over the translation units, so the nonlinear explosion in compile times can be mitigated. Right now Oil has 24 translation units, and I see the redundancy in build times. I even preprocessed each one and counted the 10x blowup in line count:

               (requires login)

                        So less than 100K lines of code blows up to about 1M once all the template headers are included. But I could probably cut it down to 6 or 8 translation units, and compile a linear amount of code. I think it would end up at 200K or less.

                        But yeah the point is that once you generate code you can do stuff like that, whereas manually writing C++ you often get “stuck”. Most big C++ projects have bad build systems and bad code structure …

                        1. 2

                          Thanks for weighing in. <3

                          Templates increase compile times but

                          I was kinda thinking you wouldn’t use them in a code generator, since you can generate exactly the needed monomorphic versions yourself? but that might not take advantage of the C++ compiler’s ability to coalesce identical implementations sometimes.

                          Also, when you generate code you have more control over the translation units…

                          I was thinking that perhaps it might be possible to generate code with absolutely minimal headers / forward declarations too, which could help some more?

                          1. 2

                            Yeah I think it’s mainly about the implementation effort … Trying to recreate what C++ is doing could be the difference between a 1 year project and a 5 year one. I should know :)

                            e.g. I liken it to one of the “wrong turns” I mentioned on the blog – to “just” write a Python bytecode interpreter. Well it turns out that is a huge project, and there’s no way to make it fast with a reasonable amount of effort.

                            You can get something slow working, but getting good performance is a whole different story!

                    1. 30

                      its release philosophy is supposed to avoid what I call “the problem with Python”: your code stops working if you don’t actively keep up with the latest version of the language.

                      This is a very real problem with the Python and Node ecosystems.

                      1. 16

                        I’m going to be grumpy and ask what this “problem” is supposed to mean, exactly. I can deploy, say, a Python application on a particular version of Python and a particular operating system today, and then walk away for years, and as long as I pay the hosting bill there is no technical reason why it would suddenly stop working. There’s no secret kill switch in Python that will say “ah-ha, it’s been too long since we made you upgrade, now the interpreter will refuse to start!”

                        So what people always actually mean when they say things like this is that they want to keep actively developing their code but have everybody else stand still forever, so that they can permanently avoid platform upgrades but also the language and ecosystem will never leave them behind. So the real “problem” is that one day nobody else will be willing to provide further bug and security fixes (free or perhaps even paid) for the specific combination of language/libraries/operating system I initially chose to build my app with, and on that day I have to choose between upgrading to a new combination, or doing that maintenance myself, or going without such maintenance entirely.

                        And although I’ve seen a lot of claims, I’ve seen no evidence as yet that Rust has actually solved this problem — the editions system is still young enough that the Rust community has not yet begun to feel the pain and cost of what it committed them to. “Stability without stagnation” is a nice slogan but historically has ranged from extremely difficult to downright impossible to achieve.

                        1. 16

                          “Stability without stagnation” is a nice slogan but historically has ranged from extremely difficult to downright impossible to achieve.

                          You just don’t make breaking changes. Add new APIs but don’t take away old ones. Go 1 has been stable for 10 years now. That means there are some ugly APIs in the standard library and some otherwise redundant bits, but it’s been fine. It hasn’t stopped them from adding major new features. Similarly, JavaScript in the browser only rolls forward and is great. Again, some APIs suck, but you can just ignore them for the most part and use the good versions. I’m not as familiar with the Linux kernel, but my impression there is that Linus’s law is you don’t break userland.

                          Most breaking changes are gratuitous. They make things a little nicer for the core developers and shove the work of repair out to everyone who uses their work. I understand why it happens, but I reserve the right to be grumpy about it, especially because I have the experience of working in ecosystems (Go, the browser) that don’t break, so I am very annoyed by ecosystems that do (NPM, Python).

                          1. 15

                            I’ll be up-front. My stance on breaking changes is that there are two types of projects:

                            1. Those which have decided to accept they are an inevitable fact of software development, and so have committed to the process of handling them, and
                            2. Those which haven’t yet.

                            And remember that we are talking not just about core language/standard library, but also ecosystems. Go has already effectively broken on this — the “rename your module to include /v2” hack is an admission of defeat, and allows the previous version to lapse into a non-maintained status. Which in turn means that sooner or later you will have to make the choice I talked about (upgrade, or do maintenance yourself, or go without maintenance).

                            And Rust has had breaking changes built into the ecosystem from the beginning. The Cargo/crates ecosystem isn’t built around every crate having to maintain eternal backwards-compatibility, it’s built on semantic versioning. If I publish version 1.0 of a crate today, I can publish 2.0 with breaking changes any time I want and stop maintaining the 1.x series, leaving users of it stranded.

                            So even if Rust editions succeed as a way of preserving compatibility of the core language with no ecosystem splits and no “stagnation”, which I strongly doubt in the long term, the ecosystem has already given up, and in fact gave up basically immediately. It has “the Python problem” already, and the only real option is to learn how to manage and adapt to change, not to insist that change is never permitted.

                            (and of course my own experience is that the “problem” is wildly exaggerated compared to how it tends to impact projects in actual practice, but that’s another debate)

                            1. 5

                              I think it’s hard to talk about this rigorously because there’s definitely some selection bias at play — we don’t have information about all the internal projects out there that might be bogged down by breaking changes between interpreter versions, and they’re likely motivated by very different incentives than the ones that govern open source projects — and there’s likely survivorship bias at play too, in that we don’t hear about the projects that got burnt out on the maintenance burden those breaking changes induce.

                              My anecdotal evidence is that I’ve worked at places with numerous Python projects bound to different, sometimes quite old, interpreter versions, and there just aren’t enough person-hours available to keep them all up to date, and updating them in the cases where it was truly necessary made for some real hassle. Even if you chalk that up to bad resource management, it’s still a pretty common situation for an organization to find itself in, and it’s reasonable to expect your tools to not punish you for having less than perfect operational discipline. In light of that, I think understanding it in this binary frame of either making breaking changes or not isn’t the most fruitful approach, because as you note it’s not realistic to expect that they never happen. But when they do happen, they cost, and I don’t think it’s unreasonable for an organization to weigh that total cost against their resources and decide against investing in the Python ecosystem. It’s not unreasonable to make the opposite choice either! I just don’t think that cost is trivial.

                              1. 7

                                Even if you chalk that up to bad resource management, it’s still a pretty common situation for an organization to find itself in and it’s reasonable to expect your tools to not punish you for having less than perfect operational discipline.

                                Imagine 20 years ago saying this about a project that suffered because they lost some crucial files and it turned out they weren’t using any kind of version control system.

                                Because that’s basically how I feel about it. Regularly keeping dependencies, including language tooling/platform, up-to-date, needs to become table stakes for software-producing entities the way that version control has. I’ve seen this as a theme now at four different companies across more than a decade, and the solution is never to switch and pray that the next platform won’t change. The solution is always to make change an expected part of the process. It can be done, it can be done in a way that minimizes the overhead, and it produces much better results. I know because I have done it.

                                1. 3

                                  Because that’s basically how I feel about it. Regularly keeping dependencies, including language tooling/platform, up-to-date, needs to become table stakes for software-producing entities the way that version control has.

                                  I don’t believe this is what’s being disputed: it’s that this being a fact is precisely why it’s important for the platform to facilitate ease of maintenance to the best of its ability in that regard. Doing so allows even under-resourced teams to stay on top of the upgrade treadmill, which is more of my point in the bit you quoted: tools that induce less overhead are more resilient to the practical exigencies that organizations face. I guess we’ll have to agree to disagree about where the Python ecosystem sits on that spectrum.

                                  1. 2

                                    This sounds like the “don’t write bugs” school of thought to me. Yes, ideally anything that’s an operational concern will get ongoing maintenance. In the real world… 

                                    More anecedata: my coworker built a small Python scraper that runs as a cron job to download some stuff from the web and upload it to an S3 bucket. My coworker left and I inherited the project. The cron job was no longer a high priority for the company, but we didn’t want to shut it off either. I couldn’t get it to run on my machine for a while because of the Python version problem. Eventually I got to the point where I could get it to run by using Python 3.6, IIRC, so that’s what it’s using to this day. Ideally, if I had time and resources I could have figured out why it was stuck and unstick it. (Something to do with Numpy, I think?) But things aren’t always ideal.

                                    If someone has to have discipline and try to stick closer to the “don’t write bugs” school, who should it be: language creators or end developers? It’s easy for me to say language creators, but there are also more of us (end developers) than them (language creators). :-) ISTM that being an upstream brings a lot of responsibility, and one of those should be the knowledge that your choices multiply out by all the people depending on you: if you impose a 1 hour upgrade burden on the 1 million teams who depend on you, that’s 1 million hours, etc.

                                    1. 1

                                      Generalizing from anecdata and my own experience, the killer is any amount of falling behind. If something is not being actively maintained with dependency updates on at least a monthly and ideally a weekly cadence, it is a time bomb. In any language, on any platform. Because the longer you go without updating things, the more the pending updates pile up and the more work there will be to do once you do finally sit down and update (which for many projects, unfortunately, tends to be only when they are absolutely forced to start doing updates and not a moment sooner).

                                      At my last employer I put in a lot of work on making the (Python) dependency management workflow as solid as I could manage with only the standard packaging tooling (which I believe you may have read about). But the other part of that was setting up dependabot to file PRs for all updates, not just security, to do so on a weekly basis, and to automate creation of Jira tickets every Monday to tell the team that owned a repository to go look at and apply their dependabot PRs. When you’re doing it on that kind of cadence it averages very little time to review and apply the updates, you find out immediately from CI on the dependabot PRs if something does have a breaking change so you can scope out the work to deal with it right then and there, and you never wind up in a situation where applying the one critical update you actually cared about takes weeks or months because of how much other stuff you let pile up in the meantime.

                                      Meanwhile I still don’t think Python or its ecosystem are uniquely bad in terms of breaking changes. I also don’t think Go or Rust are anywhere near as good as the claims made for them. And the fact that this thread went so quickly from absolutist “no breaking changes ever” claims to basically people’s personal opinions that one language’s or ecosystem’s breaking changes are justified and tolerable while another’s aren’t really shows that the initial framing was bad and was more or less flamebait, and probably should not be used again.

                                      1. 5

                                        No, I still think you’re wrong. :-)

                                        I agree that for a Python or Node project it is recommended to set up dependabot to keep up to date or else you have a ticking time bomb. However, a) that isn’t always practical and b) it doesn’t have to be like that. I routinely leave my Go projects unattended for years at a time, come back, upgrade the dependencies, and have zero problems with it.

                                        Here is a small project last touched in 2017 that uses Go and Node:

                                        Here is the full Terminal output of me getting it to build again with the most recent version of Go:

                                        (Fri, May 20  08:56:09 PM) (master|✔)
                                        $ go build .
                                        go: cannot find main module, but found Gopkg.lock in /var/folders/p7/jc4qc9n94r3f6ylg0ssh1rq00000gs/T/tmp.S6ZYg4FX/track-changes
                                                to create a module there, run:
                                                go mod init
                                        # status: 1 #
                                        (Fri, May 20  08:56:27 PM) (master|✔)
                                        $ go mod init
                                        go: creating new go.mod: module
                                        go: copying requirements from Gopkg.lock
                                        go: to add module requirements and sums:
                                                go mod tidy
                                        (Fri, May 20  08:57:00 PM) (master|…)
                                        $ go mod tidy -v
                                        go: finding module for package
                                        go: finding module for package
                                        go: finding module for package
                                        go: finding module for package
                                        go: finding module for package
                                        go: finding module for package
                                        go: downloading v0.0.0-20220513210516-0976fa681c29
                                        go: found in v1.7.1
                                        go: found in v1.7.1
                                        go: found in v0.3.7
                                        go: found in v0.3.7
                                        go: found in v0.3.7
                                        go: found in v0.0.0-20220513210516-0976fa681c29
                                        (Fri, May 20  08:57:10 PM) (master|…)
                                        $ go build .
                                        (Fri, May 20  08:57:18 PM) (master|…)

                                        As you can see, it took about a minute for me to get it building again. Note that this package predates the introduction of Go modules.

                                        Let’s upgrade some packages:

                                        (Fri, May 20  09:02:07 PM) (master|…)
                                        $ go get -v -u ./...
                                        go: downloading v0.0.0-20220520000938-2e3eb7b945c2
                                        go: downloading v0.101.1
                                        go: downloading v2.0.0-20220412131121-41eea1483964
                                        go: upgraded v0.16.0 => v0.100.2
                                        go: added v1.6.1
                                        go: upgraded v1.0.0 => v2.0.0+incompatible
                                        go: upgraded v0.0.0-20161224141413-349dd0209470 => v1.3.1
                                        go: upgraded v0.0.0-20171004151416-4cff9ef7b700 => v0.0.0-20180319155456-bca2e7f09a17
                                        go: upgraded v3.3.1+incompatible => v4.1.2+incompatible
                                        go: upgraded v0.0.0-20171113180720-1e59b77b52bf => v1.5.2
                                        go: upgraded v0.8.0 => v0.9.1
                                        go: upgraded v0.0.0-20171107184841-a337091b0525 => v0.0.0-20220520000938-2e3eb7b945c2
                                        go: upgraded v0.0.0-20171117235251-f95fa95eaa93 => v0.0.0-20220411215720-9780585627b5
                                        go: upgraded v1.0.0 => v1.6.7
                                        go: added v1.28.0
                                        go: upgraded v2.0.0-20171026120407-29680c88e31d => v2.0.0-20220412131121-41eea1483964
                                        (Fri, May 20  09:02:52 PM) (master|…)
                                        $ go build .
                                        ./handler.go:28:14: undefined: middleware.DefaultCompress
                                        # status: 2 #
                                        (Fri, May 20  09:02:58 PM) (master|…)
                                        $ go doc middleware
                                        package middleware // import ""
                                        // snip
                                        (Fri, May 20  09:03:12 PM) (master|…)
                                        $ go doc middleware.Compress
                                        package middleware // import ""
                                        func Compress(level int, types ...string) func(next http.Handler) http.Handler
                                            Compress is a middleware that compresses response body of a given content
                                            types to a data format based on Accept-Encoding request header. It uses a
                                            given compression level.
                                            NOTE: make sure to set the Content-Type header on your response otherwise
                                            this middleware will not compress the response body. For ex, in your handler
                                            you should set w.Header().Set("Content-Type",
                                            http.DetectContentType(yourBody)) or set it manually.
                                            Passing a compression level of 5 is sensible value
                                        (Fri, May 20  09:03:32 PM) (master|…)
                                        $ subl .
                                        (Fri, May 20  09:04:12 PM) (master|…)
                                        $ go build .
                                        (Fri, May 20  09:04:59 PM) (master|✚1…)

                                        Took about 3 minutes to upgrade the packages and fix the broken dependency (they renamed a middleware). Bear in mind that the upgrade I did deliberately did not try to upgrade past semantic version changes in its dependencies. Probably it would take another half hour or more if I wanted to chase down whatever breaking changes happened there.

                                        Suffice it to say, yarn cannot even install its packages, and the last time I tried this stunt a couple of years ago, I got past that and then ran into a problem with webpack that I couldn’t easily solve.

                                        Go is just a much more stable ecosystem than Node or Python. It’s not as stable as say browser JS, where one can reasonably expect working code to work until civilization collapses, but it’s fairly stable. And it’s not magic. If there were a communal expectation of this level of stability, it could exist everywhere. It’s a social value to keep things working in the Go ecosystem, and it’s not elsewhere.

                                        1. 2

                                          The other day I returned to a Python package I hadn’t touched in about a year. The actual Python dependencies portion of updating it was done in a few minutes: I updated the supported versions of Python to those currently supported by upstream, did the same for the supported versions of Django, and then had to change a whopping four lines of code, all in a unit-test file, to deal with a deprecation in Django that had finally been removed.

                                          The entire remainder of getting it ready for a new release was fighting with CI — updating from v1 to v3 of the GitHub Actions tasks for Python, which I mostly did by copy/pasting from a package by someone else who I trust.

                                          I mention this because while you have anecdotes about Go projects updating more or less seamlessly, I have just as many about Python projects, and other people in this thread have anecdotes about Go projects breaking in ways they found annoying.

                                          All of which is to say that you should stop trying to extrapolate from your anecdata to “Go is stable and values stability, while Python is not and does not”, because it just ends up looking silly when other people show up with their anecdata. Python is not uniquely “unstable” and neither Go nor Rust are uniquely “stable”. At best, some projects are sometimes lucky enough that they can give the false impression of stability in the language/ecosystem, despite the fact that the language/ecosystem is always moving on. And that’s why I have said, over and over, that the thing to do is embrace and accept change and build processes around it. Otherwise, you’re likely to wake up one day to find out that what you thought was stable and unchanging was neither, and that you are in for a lot of trouble.

                                          1. 3

                                            When I get a chance, I thought of an equivalently old Python package for me to try updating. I’ll try to do it this weekend or next week.

                                            But I just don’t buy this:

                                            Python is not uniquely “unstable” and neither Go nor Rust are uniquely “stable”.

                                            I don’t have experience with Rust, so I have no idea there. I do have years of working in Python, JavaScript, and Go and my experience is uniform: Python and JavaScript routinely have problems that make installing/updating take a workday, and Go does not. I’ve already given a lot of concrete examples, and I’m sure I could dig through my git history and find more. At a certain point, all I can say is this is my experience and if it’s not yours, great.

                                            1. 2

                                              You’re still missing the point I’m trying to make, though, because you keep diving for anecdotes to support your case, and I keep trying to remind you that if we allow generalizing from anecdotes then your own claims will get contradicted, because there are people who can bring similar anecdotes about languages like Go that you think don’t have this “problem”.

                                              My stance here is and always has been that no language or ecosystem is free of the need to keep up with dependencies and no language or ecosystem is free from breaking changes. People in this comment thread have posted experiences with Go breaking on them and you’ve mostly just ignored that or disallowed generalizations from them, while insisting that your own anecdotes support generalizations, about the objective state of particular languages/ecosystems.

                                              That is what I’m trying to get you to see, and trying to break you out of. No number of anecdotes one way or another will make your case for you, because the issue is not an insufficient number of presented anecdotes.

                                              At a certain point, all I can say is this is my experience and if it’s not yours, great.

                                              This is very close to where I’m trying to lead you, but

                                              • You never actually just say that and stop — you always feel a need to throw on another anecdote and then insist that your experience generalizes to objective statements about particular languages/ecosystems, and that’s where it goes off the rails.
                                              • You never really accept other people having experiences that don’t match yours, and thus disputing your generalizations. I don’t have the kinds of problems you do with Python. You don’t have the kinds of problems other people have had with Go. But I’m doing my best to acknowledge experiences other than my own when I say that no language/ecosystem is free of this issue: I expect that there will be people who do run into it in Python, and also in Go, and also in Rust, and also in every other language/ecosystem. You seem to be trying to erase the experiences of people who run into it in languages that you subjectively don’t believe have this problem, because it doesn’t fit the “this language has it, that language doesn’t” generalization you want to make.
                                              1. 2

                                                your own claims will get contradicted, because there are people who can bring similar anecdotes about languages like Go that you think don’t have this “problem”.

                                                I have given a ton of anecdotes and done two experiments in support of my view. On the other side, there is your a priori assertion that all ecosystems have the same problems (but is it of the same magnitude?) and an unreadably long article by a pro-Rust troll. If other people have anecdotes about Go being hard to upgrade, I’m happy to read them (assuming they can get to a point in less than 10,000 words) and theorize about why someone else might have that problem when I don’t. But that hasn’t happened in this thread.

                                                1. 3

                                                  Well, right here in this thread someone mentioned problems with upgrading in Go. But the way you and another user were trampling all criticism of Go with insults — “troll”, “clickbait”, and so on — literally drove that person out of the thread.

                                                  I honestly don’t know what else to tell you. It seems very clear that you’re not prepared to hear anything that contradicts the narrative you’ve got going, so I guess I’ll bow out too since it’s pointless to continue trying to get you to acknowledge stuff that was literally right in front of you.

                                                  1. 1

                                                    I could never tell what the actual problems were based on the links posted by m0th were. They linked to long pieces with no summaries, so while I would love to comment, I cannot. Maybe it’s because they felt targeted by ad hominem. I have a different theory for why, which is that the criticisms were scattershot and not serious. Again, if they want to come back and provide a useable summary, great.

                                                    1. 1

                                                      And yet I’d bet anything that if you’d been given short concise statements of problems you’d dismiss them as being trivial or minor on those grounds. Like someone else already said, I simply cannot assume that you’re actually trying in good faith to engage with criticism of a thing you like; instead all I’ve seen you do is smear and deflect and dismiss without argument.

                                                      1. 1

                                                        If you call this “without argument” I don’t know what to say. I really feel that the “without argument” side is yours, and it is very frustrating to me, because you’re a writer whose work I respect which makes it very hard for me to let this go. But when you write things like “anecdotes don’t count”… What is a personal dev blog but anecdotes? I agree that supporters of language X can be vociferous and annoying, and Go is one such X. In that case of Go, the vociferous attitude comes from having been attacked as using a “bad language” by people like fasterthanlime and our own experience of initially thinking things like the v2 are dumb and then slowly seeing the benefits and defending them. I agree that my initial claim was a bit too glib in its characterization, but I stand by a less inflammatory version of the same idea. I don’t agree that anecdotes aren’t the data of software engineering, and I don’t agree it’s somehow impossible to characterize software ecosystems with the understanding that any generalization will always be at best partial and incomplete but nevertheless it’s better to have the generalization than not.

                                                        1. 2

                                                          People brought up criticism of Go. You didn’t engage with it or argue with the points made — you ignored them, or dismissed them as too long to be worth your time to read, or just outright insulted the author of one piece of Go criticism.

                                                          It’s hard for me to see a way to take this as good faith. It appears that you like Go, which is fine! Same with Python: if you don’t like it, then don’t like it — that’s fine!

                                                          What’s not fine is the double-standard “my anecdotes are generalizable to broad claims about the language/ecosystem as a whole, but other people’s anecdotes are not”. Which is basically what you’re doing. If your anecdotes about Python generalize, then so do mine and so do other people’s anecdotes about Go. Which would then contradict the claims you want to make, which is probably why you’re being so cagey about not allowing others’ anecdotes to generalize while insisting that yours do. But again it’s going to be, at least, extremely difficult to take that as a good-faith argument.

                                                          1. 1

                                                            When I engage with the criticism, I’m just dismissing it. If I don’t engage because I don’t have enough detail (or drowning in irrelevant details so I can’t tell why monotonic time is supposedly bad), I’m ignoring it. It’s no win.

                                                            1. 2

                                                              As far as I can tell the only “engagement” you’ve given to criticism of Go was the second paragraph in this comment, and I think that’s being charitable since it’s also possible to read as a “well, this breaking change didn’t count, and it just means Go is great instead of the greatest”. If I said Python’s stability guarantees are good but just “don’t go far enough”, I doubt you’d view it charitably.

                                                              You have, on the other hand, repeatedly dismissed critiques both by users in this thread and linked to on other sites as being too long or too “scattershot” or “fair to handwave … away”. Again: if I tried pulling that sort of thing in defense of Python you would, I hope, call me on it. You’re doing it here in defense of Go. I’m calling you on it.


                                                                I wrote a long reply, decided to sleep on it, and I give up. You haven’t conceded any points. You’ve repeatedly accused me of bad faith. You’ve decided Python is as good as it gets and by definition nothing else can be better in some ways but not others. All I can say is I didn’t invent the phrase “the problem with Python”. Other people feel this way too. Maybe we’re wrong, and everything has the same problem to the same degree. I don’t think so though.

                                                2. 1

                                                  You seem to be simultaneously saying there is nothing especially broken with python while defending the culture of constantly breaking changes and saying it is inevitable. I think your attitude sums up the problem, there are a lot of enablers.

                                                  1. 3

                                                    This thread started out with an accusation that there is a problem that uniquely exists in certain languages, and does not exist at all in others. People were very confidently stating not that the breaking changes in Go are less frequent, or less annoying, or more justified — people were very confidently stating that breaking changes simply do not exist at all in Go.

                                                    Don’t believe me? Here, and a lot of the rest of this sub-thread is people trying to quibble about “oh well that was just a change to (thing that shouldn’t count)” or explain “well these changes were justified and tolerable while those other languages’ changes aren’t” in order to somehow cling to the notion that Go is a bastion of perfect stable compatibility, even in the face of solid evidence that it isn’t.

                                                    And I’m sorry if people don’t like to hear it, but I’ve just gotta call bullshit on that. Every language and ecosystem has breaking changes. There is no magic in Go that makes it somehow unchanging and perfectly compatible forever, nor is there any magic in the ecosystem that somehow makes all maintainers good stewards who never ever make unjustified changes.

                                                    Still, people seem to be wildly exaggerating both the frequency of breaking changes in the languages they want to criticize, while minimizing for the languages they want to praise. It’s very human and very subjective, and very not a good way to have a reasoned discussion about this.

                                                    Especially when it’s coupled with very obvious bad-faith tactics like the way critics of Go keep being dismissed or even smeared in this thread.

                                                    If you dislike Python, then by all means dislike it. There are things I dislike about it. But there’s way too much in this thread, and in our industry, of people being unable to handle anything other than extremes — something is either 100% good or 0%, either completely perfectly backwards-compatible always or never — and of very clearly contradicting themselves and making really bad arguments to try to justify their subjective dislikes. Just come out and say you don’t like a thing and move on. You don’t also have to prove that it’s the worst in order to justify your dislike.

                                                    1. 2

                                                      I don’t know if python is especially bad. I personally found npm projects I have tried break when updating dependencies a few years later, and I have not had that experience in Go. Maybe I am just lucky.

                                                      1. 1

                                                        It seems to me that the disconnect here is about the scope of what constitutes “breaking changes” in a language. There certainly isn’t an objective definition! Some people consider only the core language and it’s standard library, and others include everything out to its tooling.

                                                  2. 1

                                                    Okay, I tried this with a Datasette project from around the same time. (It was actually deployed again in early 2020, so it’s a bit fresher than the Go project, but whatever, close enough.) Again, Node didn’t work. I think the issue with that is that libsass is dead and doesn’t compile anymore, so you need to switch to dart-sass instead. In all likelihood, the fastest solution to the Node issues is to drop all of the dependencies and just start over from scratch with only my user code, since the dependencies were just there to build a Vue project.

                                                    On the Python side, it wouldn’t work with Python 3.9, but when I used Python 3.7, I got it to run again. Terminal output is below. It only took 15 minutes to get it going, but compare this to Go, which works with the current version of Go even though the package predates modules (which caused a lot of breakage not covered by the Go 1 guarantee) and I got the dependencies upgraded in a total of 5 minutes. By contrast, the Python installations all took long enough that they break flow: since the installation is going to take a while, I switch away from my Terminal, which is chance for me to get distracted and lose my place. I think this project did pretty well because it used your recommended pattern of having a requirements-freeze.txt file and it had a Bash script to automate the actual install commands. But the error when UVLoop was broken was pretty demoralizing: I have no idea how I would fix it, so getting up to Python 3.9 or 3.10 would probably involve a lot more Googling than I’m willing to do for an internet comments example. Again, the simplest fix might be to just blow away what I have now and start from scratch. I think Datasette has been relatively stable in spite of being <1.0, so I suspect that it wouldn’t be that hard to get it working, but again, it’s more than I want to do for an example. A nice thing about Go is that most dependencies don’t use C, so when something does go wrong, like that middleware that was broken in the other project, you aren’t confronted with errors in a language you don’t know using a build system you don’t understand. In general, it’s just much less intimidating to get a Go project back up to speed.

                                                    So this is fairly reflective of my lived experience: Node projects, especially those that use Webpack, break in ways that are more or less unfixable and need to be restarted from scratch; Python projects can be kept running if you are willing to pin old versions of Python but give a lot of scary compiler errors and don’t have clear paths forward; Go projects can typically be upgraded by typing go get -u ./... and maybe reading some release notes somewhere. Go isn’t perfect and there are still problems, but the quantity of problems is so much less than it creates a qualitative difference in feeling.

                                                    $ gh repo clone baltimore-sun-data/salaries-datasette
                                                    Cloning into 'salaries-datasette'...
                                                    remote: Enumerating objects: 1017, done.
                                                    remote: Counting objects: 100% (94/94), done.
                                                    remote: Compressing objects: 100% (72/72), done.
                                                    remote: Total 1017 (delta 63), reused 23 (delta 20), pack-reused 923
                                                    Receiving objects: 100% (1017/1017), 53.78 MiB | 4.25 MiB/s, done.
                                                    Resolving deltas: 100% (568/568), done.
                                                    Updating files: 100% (74/74), done.
                                                    (Sat, May 21  12:45:40 PM)
                                                    $ cd salaries-datasette/
                                                    (Sat, May 21  12:45:54 PM) (master|✔)
                                                    $ more
                                                    # salaries-datasette
                                                    Public salary data acquired by the Baltimore Sun. Currently, we just have data from the state of Maryland for 2017.
                                                    ## Usage
                                                    Run `./ setup` to install locally. The script assumes you have either Python 3 or Homebrew for Mac installed. Run `./ setup-frontend` to install front end dependencies.
                                                    Run `./ create-db` to create a SQLite database out of the provided CSVs.
                                                    Run `./` or `./ serve` to run server at http://localhost:9001.
                                                    Run the JS/CSS frontend server in another tab with `./ frontend`.
                                                    `./ format` will format Python and Javascript code according to the coding standards of the project.
                                                    `Dockerfile` is also provided for running/deploying with Docker. The image can be built with `./ docker-build` and tested with `./ docker`. The server only responds to correct hostnames (not localhost), so edit `/etc/hosts` to add `` and then test in the browser.
                                                    (Sat, May 21  12:46:06 PM) (master|✔)
                                                    $ ./ setup
                                                    snip a ton of output from installing things including a lot of scary errors
                                                    × Encountered error while trying to install package.
                                                    ╰─> uvloop
                                                    note: This is an issue with the package mentioned above, not pip.
                                                    hint: See above for output from the failure.
                                                    # status: 1 #
                                                    (Sat, May 21  12:49:40 PM) (master|✔)
                                                    $ # by reading the DOCKERFILE, I learn that this used Python 3.7 when it was made
                                                    (Sat, May 21  12:51:16 PM) (master|✔)
                                                    $ pyenv install 3.7.13
                                                    python-build: use openssl@1.1 from homebrew
                                                    python-build: use readline from homebrew
                                                    Downloading Python-3.7.13.tar.xz...
                                                    Installing Python-3.7.13...
                                                    (Stripping trailing CRs from patch.)
                                                    patching file Doc/library/ctypes.rst
                                                    (Stripping trailing CRs from patch.)
                                                    patching file Lib/test/
                                                    (Stripping trailing CRs from patch.)
                                                    patching file Modules/_ctypes/_ctypes.c
                                                    (Stripping trailing CRs from patch.)
                                                    patching file Modules/_ctypes/callproc.c
                                                    (Stripping trailing CRs from patch.)
                                                    patching file Modules/_ctypes/ctypes.h
                                                    (Stripping trailing CRs from patch.)
                                                    patching file
                                                    (Stripping trailing CRs from patch.)
                                                    patching file 'Misc/NEWS.d/next/Core and Builtins/2020-06-30-04-44-29.bpo-41100.PJwA6F.rst'
                                                    (Stripping trailing CRs from patch.)
                                                    patching file Modules/_decimal/libmpdec/mpdecimal.h
                                                    (Stripping trailing CRs from patch.)
                                                    patching file
                                                    python-build: use tcl-tk from homebrew
                                                    python-build: use readline from homebrew
                                                    python-build: use zlib from xcode sdk
                                                    Installed Python-3.7.13 to /Users/adhoc/.pyenv/versions/3.7.13
                                                    (Sat, May 21  12:55:00 PM) (master|✔)
                                                    $ ./ setup
                                                    × Encountered error while trying to install package.
                                                    ╰─> uvloop
                                                    note: This is an issue with the package mentioned above, not pip.
                                                    hint: See above for output from the failure.
                                                    # status: 1 #
                                                    (Sat, May 21  12:57:08 PM) (master|✔)
                                                    $ # I become worried that maybe Pyenv didn't activate for some reason, so I try explicitly adding it to my PATH
                                                    (Sat, May 21  12:59:30 PM) (master|✔)
                                                    $ PATH=$HOME/.pyenv/versions/3.7.13/bin/:$PATH ./ setup
                                                    (Sat, May 21  01:01:58 PM) (master|✔)
                                      2. 3

                                        This thread has grown a lot overnight while I was sleeping! I still haven’t caught up, but here is my top of head takeaway:

                                        I think it’s fair to complain that the Go 1 stability guarantee doesn’t go far enough, but it seems like some people elsewhere in the thread are trying to claim that adding monotonic time somehow breaks the guarantee, which makes no sense to me. At this point, someone citing Fasterthanlime basically means they don’t know anything about Go, IMO because he has been so egregious in his misrepresentation of the language. Just recently, I had to pin a specific subversion of Go 1.18 in my tests because I needed a particular behavior for the Go tool. I don’t consider that to have been the Go team breaking the stability guarantee, just an example of how you can run into the limits of it when you’re talking about the tooling.

                                        Obviously, it’s going to be impossible to never break anything. (Which is what I take from your points 1 and 2.) If nothing else, security problems can force breaking changes. Certainly, you need to get to a point where you’re happy enough to commit to a design before you stabilize it: Go was not stable at all before version 1. JavaScript wasn’t stable before ECMA etc.

                                        But the question to me is whether the upstream developers take breaking changes seriously or commit them wantonly. To rephrase your points:

                                        1. Projects which accept that breaking changes are fact of software development, so they do it whenever it’s convenient.

                                        2. Projects which accept that breaking changes are fact of software development, so they do it as little as possible.

                                        I was being a bit too flip when I wrote “You ‘just’ don’t make breaking changes.” That’s like saying, you just don’t write bugs. It is unavoidable sometimes. But there is a qualitative difference as an end developer in relying on a project which attempts to avoid breaking changes and one which does not.

                                        In Node, I routinely run into software that will not run on the latest versions of Node. When a new version of Node comes out, I don’t think “Oh great, more speed, more features!” I think, “Ugh, crap, what’s going to break this time?” I had a hard to diagnose break in Babel (a very popular library!) that was caused not by a major version jump in Node, but a minor version bump. It was a waste of a day of development time for me for no reason. You wrote “as long as I pay the hosting bill there is no technical reason why it would suddenly stop working.” But in this case, there wasn’t anything I did intentionally which broke my setup. I just hadn’t pinned my version of Node, Homebrew moved up by a minor version because of some other thing, and suddenly I lost a workday. It also broke production because prod was only pinned to the major version not the minor or patch.

                                        A similar example was that I had an Amazon ECS server using a Docker image to serve some Python app, and one day it just stopped working in prod. The problem was that when the Dockerfile was written, it didn’t specify the version of Python, and then the version where async went from a semi-keyword to a full keyword came out and broke the dependencies I was using. It wouldn’t have mattered if the Docker image had been already baked, but the way this project was setup, the Docker image would be periodically rebuilt from the Dockerfile whenever things would restart or something. That at least was relatively easy to track down and fix because the change of Python versions was noticeable once I started debugging it.

                                        You call the /v2 thing a “hack” and it is ugly and inelegant, but it does solve this problem: when you make a breaking change, let users upgrade at their own pace, instead of having them accidentally get moved onto the new thing without being aware of it. (In browser JS, "strict mode" and type="module" also have the effect of being opt-in upgrades!) That’s really the core of the so-called “Python problem” to me. Just let me decide when to work through the breaking changes. I’m not expecting eternal support or whatever. I just want something that works today to work tomorrow unless I press the big red “attempt upgrade” button.

                                        1. 5

                                          At this point, someone citing Fasterthanlime basically means they don’t know anything about Go, IMO because he has been so egregious in his misrepresentation of the language.

                                          I’m the one who cited the monotonic time issue. I’m having trouble seeing good faith in this statement, so I’ll bow out of this thread after this. But just so you understand that this is adamantly not the case: I’ve written Go since before 1.0, contributing to some of the largest projects in the ecosystem, and as a manager moved several teams over to using it (from Python, no less.) I also don’t think those credentials should be necessary for a critique.

                                          I had a hard to diagnose break in Babel (a very popular library!) that was caused not by a major version jump in Node, but a minor version bump.

                                          To reiterate, this has happened to me with minor version bumps in Go, due to both modules and stdlib changes. This is even worse because it was Go itself, not a third-party library.

                                          You call the /v2 thing a “hack” and it is ugly and inelegant, but it does solve this problem: when you make a breaking change, let users upgrade at their own pace

                                          This actually touches on a major reason why Go’s ecosystem could be so unstable before modules: you could either vendor which had its own can of worms, or rely on authors using this hack. Both patterns emerged as common practice only after years of Go devs experiencing breaking changes in libraries. You could argue that those instabilities don’t apply to Go itself, which I suppose is fair, but that applies to your argument about Babel as well.

                                      3. 13

                                        Go makes breaking changes all of the time. The last one that bit me was when modules were enabled by default. Suddenly packages starting behaving differently depending on a variety of factors, including the version number of the package:

                                        Python at least had the sense to bundle up all the breakages in a major version change, which was released 15 years ago.

                                        1. 8

                                          Eh no. Python does not follow semantic versioning and makes intentionally breaking changes between minor versions. In fact Python 3.x releases come with documentation titled “Porting to Python 3.x” which lists all breaking changes made intentionally.

                                          1. 6

                                            Python makes breaking stdlib changes in minor version releases. Not even remotely the same as a tooling change.

                                            1. 3

                                              I’ve never been affected by stdlib changes in python in a minor release. I have by go, where it suddenly stopped respecting system DNS settings. Or there were the backwards-incompatible changes to time, maybe best summarized here.

                                              1. 4

                                                I believe you, but you are lucky. With search I found things like which matches my experience.

                                                1. 2

                                                  I know this isn’t likely to change anyone’s mind, but the thing you linked is an example of something that had been deprecated and raising deprecation warnings for years. Here’s an example warning I get from a Python 3.7 install I had lying around (oldest I could find, I tend not to keep EOL’d Pythons on my personal laptop):

                                                  DeprecationWarning: Using or importing the ABCs from ‘collections’ instead of from ‘’ is deprecated since Python 3.3,and in 3.9 it will stop working

                                                  And it appears they actually held off and didn’t finally remove that until 3.10. That’s at least four years after Python 3.7 (which issued that warning). It’s nearly ten years after Python 3.3.

                                                  Python does use a rolling deprecation cycle, yes. So do some major Python projects like Django. This means you should ensure you’re bubbling up deprecation warnings in your CI, and probably read release notes when new versions come out, yes.

                                                  But these things don’t just come surprise out of nowhere; they’re documented, they raise warnings, they’re incorporated into the release cycles. Knowing how Python’s deprecation cycles work is, for a Python user, the same kind of table stakes as knowing how semantic versioning works is for a Rust/cargo/crates user – if you updated a crate across a major version bump and it broke your code, everyone would tell you that you could have seen that coming.

                                                2. 2

                                                  It happens, no doubt, but IME Python’s changes are much more egregious than Go’s few changes over the years. Where Go does make changes, they are typically as solutions to bugs, not gratuitous API changes.

                                                  Also, speaking personally, I try to be as objective as possible and is not an objective source in any sense. It’s clickbait. There’s no pretence of objectivity.

                                                  1. 13

                                                    I’d like to point out that this thread has gone from a claim that Go has maintained backwards compatibility to, now, a claim that Go’s backwards-incompatible changes just aren’t as “egregious” as Python’s, and to what’s basically an ad hominem attack on someone who criticized Go.

                                                    Which is kind of a microcosm of this whole framing of “Python problem” or “Node problem”. What it’s really about is not some magical language and ecosystem that never has backwards-incompatible changes, what it’s about is subjective taste. You think Go’s breaking changes are not as “egregious” and are “solutions to bugs” while Python’s are “gratuitous”. But that’s a subjective judgment based on your tastes and opinions. Someone else can have an opposite and equally-valid subjective opinion.

                                                    Or, more bluntly: “No breaking changes” nearly always turns out to actually mean “Breaking changes, but only ones I personally think are justified/tolerable, and I don’t think yours are”. Which is where this thread predictably went within the space of just a few replies.

                                                    Getting back to my original claim: change is inevitable. Entities which produce software can adapt to it and make it a normal and expected part of their processes, or they can suffer the consequences of not doing so. There is no third option for “all external change stops”. Nothing that lives does so changelessly.

                                                    1. 3

                                                      Or, more bluntly: “No breaking changes” nearly always turns out to actually mean “Breaking changes, but only ones I personally think are justified/tolerable, and I don’t think yours are”. Which is where this thread predictably went within the space of just a few replies.

                                                      No I don’t think so. It has nothing to do with opinion and everything to do with experience. What someone sees directly is how they perceive reality. I’m expressing my perception of reality as someone who has experience of both Python and Go, and m0th is expressing their experience.

                                                      I’m not minimising m0th’s view, which is why I phrased it as “in my experience”.

                                                      Getting back to my original claim: change is inevitable. Entities which produce software can adapt to it and make it a normal and expected part of their processes, or they can suffer the consequences of not doing so. There is no third option for “all external change stops”. Nothing that lives does so changelessly.

                                                      Change is inevitable, I agree, but I do think the degree matters. Python makes breaking changes often, removing APIs, etc. Go does not and only with very good reason.

                                                      1. 3

                                                        Change is inevitable, I agree, but I do think the degree matters. Python makes breaking changes often, removing APIs, etc. Go does not and only with very good reason.

                                                        Again, the original claim at the start was that there are languages which don’t have breaking changes, and Go was specifically named as an example. That has now been walked back to the kinds of statements you are making. And your statements are, effectively, just that in your opinion one language’s (Go) breaking changes are justified and not too frequent, while another language’s (Python) breaking changes are not justified and too frequent. Which is just you stating your own personal tastes and opinions. And it’s fine as long as you are willing to admit that. It is not so fine to present one’s personal tastes and opinions as if they are objective facts. It also is not so fine to engage in the kind of ad hominem you did about criticism of Go.

                                                        1. 3

                                                          Again, the original claim at the start was that there are languages which don’t have breaking changes, and Go was specifically named as an example.

                                                          I didn’t make that claim, so I’m not sure why you’re arguing with me about it.

                                                          Which is just you stating your own personal tastes and opinions. And it’s fine as long as you are willing to admit that.

                                                          I did? I explicitly said “In My Experience”. I’m not sure how I can be any clearer.

                                                          It is not so fine to present one’s personal tastes and opinions as if they are objective facts. It also is not so fine to engage in the kind of ad hominem you did about criticism of Go.

                                                          What are you even talking about? I legitimately can’t even understand what you’re referring to. Where’s the “ad hominem” comment I made? I think the most controversial thing I said was

                                                          “Python makes breaking stdlib changes in minor version releases.”

                                                          Which is objectively true as mentioned by other commenters. I made no judgement about whether it was justified or not. It’s also not “ad hominem”, which would require Python to have a position on this.

                                                          Anyway, I think you’re taking this way too personally for some reason. I like Python, I have used it for many years. But I’m out.

                                                          1. 3

                                                            Where’s the “ad hominem” comment I made?

                                                            IIUC, ubernostrum is referring to you explaining that is not a reliable source of information. That’s not an ad hominem though. His attacks are so scattershot that if someone cites them, it’s totally fair to just handwave it away.

                                                            1. 3

                                                              Where’s the “ad hominem” comment I made?

                                                              It’s this:

                                                              Also, speaking personally, I try to be as objective as possible and is not an objective source in any sense. It’s clickbait. There’s no pretence of objectivity.

                                                              You don’t bother to engage with the content, just brush off the author with a quick smear and then move on. Which is a problem, because the content is reasonably well-argued, from my perspective as someone who has a bit of Go experience and initially couldn’t quite describe why it rubbed me the wrong way. Putting into words the way that Go (which is not unique in this, but is a prominent example of it) likes to pretend complex things are simple, rather than just admitting they’re complex and dealing with it, was extremely useful for me, both for giving me a way to talk about what bugged me in Go and because it relates to things I’ve ranted about in the Python world previously (most notably why Python 3’s string changes were the right thing to do).

                                                            2. 2

                                                              Go the language doesn’t have breaking changes, in the sense that code written against a 1.0 language spec will — generally— continue to compile and run the same way against any 1.x language spec compiler. This isn’t an absolute statement of fact, but it’s a design principle that’s taken very seriously and is violated very rarely. The tooling doesn’t necessarily abide the same rules.

                                                              1. 1

                                                                Your comment is a bit of a roller coaster.

                                                                • Starts with “doesn’t have breaking changes”
                                                                • Immediately qualifies with “in the sense that…”
                                                                • Then walks it back with “This isn’t an absolute statement of fact”
                                                                • Then walks it back even further with “is violated very rarely”
                                                                • Then throws it all out the window with “The tooling doesn’t necessarily abide the same rules”

                                                                I’m not even sure what that’s trying to say.

                                                3. 9

                                                  Go 1 has been stable for 10 years now.

                                                  If you just ignore all those breaking changes they made..

                                                  1. 1

                                                    I only remember Modules. After Modules landed I never experienced any breaking build when updating to a new Go version. A whole different story was updating Java versions and the dozens of subtle ways they can break your service, especially at run-time (caused by some Spring DI magic).

                                                    1. 1

                                                      What breaking language changes have been made?

                                                      1. 3

                                                        Modules seems like the biggest change.

                                                        1. 3

                                                          I mean, Go module is not part of Go 1 stability guarantee. In my opinion, this shows how limited Go’s stability guarantee is. Go 1 is stable, but Go is not, at least if you are using “go build” to build your Go code.

                                                          1. 2

                                                            I agree but that’s not a language change, and the impact is not as significant as random language changes. I can still use and build a package from 8 years ago with no issue. And whatever tool they happened to use to manage dependencies 8 years ago will still work fine (probably).

                                                            1. 3

                                                              I saw some breaking changes in one of our old projects. Written in the Go v1.4 era, IIRC. I checked it with a Go v1.14 release, and boom, it doesn’t compile due to the module changes.

                                                              Yes, it wasn’t that hard to fix (it only took a few minutes of Internet searching), but I still count that as a breaking change.

                                                  2. 2

                                                    When was there ever a breaking change in a new version of ecmascript? Node is not a language.

                                                    1. 4

                                                      There’s a bunch of things that vary in stability.

                                                      • V8 the interpreter breaks working JS code very seldom
                                                      • The libraries that come with Node break JS code a little more often, but still IME not very often.
                                                      • V8 breaks native extensions very often. The NodeJS ecosystem discourages writing them because of this.
                                                      • Some add-on packages from npm break their consumers all the time.

                                                      Many packages are extremely careful not to break consumers. Others are less so. The experience you have with backwards compatibility tends to track the backwards compatibility stance of the worst thing in your entire dependency tree. When you have tens of thousands of transitive dependencies, you usually end up seeing 99.99%-ile bad behaviour somewhere in there at least once.

                                                      1. 1

                                                        The core problem is that a handful of “core” packages break, and nodes ecosystem is way too much about plugins, so many things that you use have 3 layers to them (all maintained by different people)

                                                        The ecosystem would be a lot more stable if we were vendoring in packages more

                                                      2. 2

                                                        Node is a development target. The pure abstract notion of a “language” doesn’t really matter here, because people write code for Node specifically.

                                                        And Node does make breaking changes. It literally makes semver-major releases, with backwards-incompatible changes that can break existing npm packages (mainly due to changes in the native API, but there also have been changes to Node’s standard library).

                                                        For example, I can’t build any projects that used Gulp v3. Node broke packages that it depends on, and I either have to run deprecated/unsupported/outdated version of Node, or rewrite my build scripts. OTOH a Rust/Cargo project I’ve developed at the same time still works with the latest version of Rust/Cargo released yesterday.

                                                        1. 2

                                                          Yes, that’s why I said “Node ecosystem” which breaks all the damn time and not “browser ECMAScript” which has not broken since standardization.

                                                          1. 2

                                                            The quote you posted says language, not ecosystem. Your comparison was a false equivalency.

                                                          2. 1

                                                            ecmascript itself is a language but not a language implementation, so while you can write a lot of code in ecmascript, at the end of the day the ecmascript specification can’t run it, so you’ll need to run it using an implementation, that’s why people are talking about the Node ecosystem, the language didn’t break the ecosystem did

                                                        1. 1

                                                          Distributed locks can’t work, even in theory.

                                                          There are two ways to handle dead machines. Either the lock can be held indefinitely, or it can expire.

                                                          If it’s held indefinitely, every time hardware breaks requires manual intervention, and can cause deadlock. This is obviously inoperable at scale, and painful even in the small.

                                                          If a timeout is added, then a stalled machine or a split network can cause the lock to expire, which means your code is run outside of a critical section. Your code needs to handle this, so the lock becomes unnecessary for correct code.

                                                          In a distributed system, the only option is wait-free code ( )

                                                          1. 1

                                                            Aye, if folks expect them to behave the same way as an in-process mutex, they’ll be disappointed. But for reference, Martin Kleppmann’s article How to do distributed locking seems to be a good explainer here, though.

                                                            1. 2

                                                              That entire article is about how redlock is broken, and how you can’t do a safe lock with it.

                                                              You can use it as a hint for efficiency, but the code still needs to be wait-free in order to be correct. Distributed locks don’t work for locking.

                                                              The example usage in the package doc is the exact pattern that is mentioned in Kleppman’s article. However, the usage example is missing the comment Kleppmann’s article contains:

                                                              // THIS CODE IS BROKEN
                                                            2. 1

                                                              I think aphyr explained this as, roughly, “distributed locks aren’t possible, the best you can do is distributed leases”.

                                                              In a distributed system, the only option is wait-free code

                                                              Wait freedom is a strong property. I think many systems get by with just non-blocking semantics and an acceptable low probability of livelock?

                                                              1. 2

                                                                Yeah, you might be right. The essential property is that all other processes make progress if one of the participants in a protocol stalls out for an arbitrary amount of time (or permanently). Locks prevent this, and timing out locks means that in pessimal cases, things act as though you never got the lock.

                                                                1. 1

                                                                  Well, I’m asking because it’s plausible to me that I might be wrong. Maybe at some very large scale you end up running into some effect along the lines of “with enough machines, there’s almost always at least one group of them that have gotten wedged into a livelock at any given time, so your overall availability is low without at least lock freedom”. I don’t know either way?

                                                              2. 1

                                                                Yes, we are well aware of the latter issue, this is often taken care off through either fencing token or atomic lock data update, depending on the constraints your application runs under. In delta-rs, we leverage atomic lock data update to implement PUT if absence logic on top of S3. Our algo is formally verified at

                                                              1. 3

                                                                I assume yes but I didn’t see this mentioned: is there still backpressure? I assume you don’t want the background thread to buffer data forever and make emacs OOM if the command runs too much faster than eshell can consume its output for too long.

                                                                1. 4

                                                                  Yes, this is still back pressure at the background output thread level. The nice thing about this though is it’s back pressure at a larger size instead of 1024 byte back pressure.

                                                                1. 2

                                                                  There’s a 2014 blog post from Dan Luu to roughly the same effect. It sounds like the issues we much worse at that point but the root cultural problems may be unchanged.

                                                                  That post was and it was discussed here at

                                                                  1. 5

                                                                    Delightful. Alternative solution: Instead of doing current time + 100 ms do last fire time + 100 ms. I THINK that would make it so that if your math accidentally calculates + 101 ms you would catch up on the next frame. You might have occasional single-frame glitches in firing rate, but it should correct itself. If I’m phrasing it right; I haven’t had coffee yet. You can also change the timing code from if(last_time + dt > current_time) { do_stuff } to while(last_time < current_time) { last_time += dt; do_stuff } and with some interpolation it catches up on its own.

                                                                    Using floating point numbers for times is dangerous! This is why modern timekeeping API’s tend to do integer seconds + integer nanoseconds. (Rust’s Duration is the one I’m most familiar with, but there are others.)

                                                                    1. 1

                                                                      Yeah this is ideal. It is similar to Bresenham’s line algorithm. It works really well for keeping the average rate of events exactly on track. Using this also lets you make things happen at the right rate with a non integer period too.

                                                                      Using floating point numbers for times is dangerous

                                                                      Yes. Perhaps they could have used simulation ticks (or milli-ticks) as their time unit? Then everything would be exact integers that seldom do anything very strange.

                                                                    1. 0

                                                                      Does that count as a rickroll btw?

                                                                      1. 1

                                                                        Nah; it wasn’t audible.

                                                                      1. 2

                                                                        Nice! If I might be so bold, I’d like to suggest that you could try varying which TLS cipher suite gets used. curl has a --ciphers option for example. Configuring the client to only allow one cipher suite should do it. In theory I think the newer ones like AES_GCM might be the fastest?

                                                                        1. 2

                                                                          curl is using “SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384”, so that should be fastest. I also tried generating an ECDSA key instead of an RSA key, but it made no performance difference.

                                                                          1. 4

                                                                            I dare you to poke it into a slower mode to see. ;) I would also be super interested to hear about ChaCha+Poly modes (which are apparently new and shiny but not hardware accelerated), CBC (which sucks) and anything with CTR in the name (which I think should be slower than GCM because the to use a separate MAC but IDK for certain).

                                                                            It does make sense that you don’t see throughput change when you switch between RSA and ECDSA. The asymmetric crypto is only used very briefly at the start of the connection. The two sides negotiate a new randomly generated key for symmetric encryption. Once that’s done, the bulk of the data is encrypted with the symmetric cryptosystem.

                                                                            1. 1

                                                                              What HTTP version? Can you force it to 1.1?

                                                                              1. 1

                                                                                Everything in my tests is on HTTP 1.1, yes.

                                                                                HTTP 2 interfered with KTLS sendfile in nginx for some reason.

                                                                          1. 3

                                                                            This sounds like a thing which might be more convenient with some tooling support. Like you have a (partial) ordering over all .h files, the IDE knows about it, and if you type var foo = std::make_unique<Bar>() then the IDE automatically inserts #imports for <bar.h> and also all the headers that <bar.h> depends on, in the right order so that everything works out.

                                                                            …at which point you’ve invented like half of a proper import system, but oh well.

                                                                            1. 4

                                                                              …at which point you’ve invented like half of a proper import system, but oh well.

                                                                              Maybe? Proper import system I think is an unsolved problem for languages with C++/Rust style of monomorphisation. Semantics-wise, Rust crate/module system is great (that’s my favorite feature apart from unsafe). But in terms of physical architecture (what the article talks about) it’s not so great.

                                                                              • There’s nothing analogous to pimpl/forward declaration, which significantly hamstrings separate compilation. C++ is better here.
                                                                              • Although parsing and typechecking of templates happens once, monomorphisation is repeated for every compilation unit, which bloats compile time and binary size in a big way.
                                                                              1. 1

                                                                                analogous to pimpl/forward declaration

                                                                                Box<>‘d opaque types? I’ve seen multiple blog posts mentioning using this for mitigating dependency chains.

                                                                                Although parsing and…

                                                                                I miss the SPECIALIZE pragma from GHC Haskell. Your generic functions get a slow fully polymorphic version generated (with an implicitly passed around dictionary object holding typeclass method pointers) and then you could easily write out a list of SPECIALIZE pragmas and to generate monomorphic copies for specific types you really care about the performance on.

                                                                                This feels like it ought to be possible in principle to deduplicate monomorphisations happening in different compilation units with a mutex and a big hash table.

                                                                                1. 1

                                                                                  Box<>‘d opaque types? I’ve seen multiple blog posts mentioning using this for mitigating dependency chains.

                                                                                  I don’t believe there’s a functional analogue to pimpl in Rust, but I need to see a specific example to argue why it isn’t.

                                                                                  What you could do in Rust is introducing dynamic dispatch, but it has significantly different semantics, is rather heavy weight syntactically (requires introducing single-implementation interfaces and a separate crate), and only marginally improves compilation time (the CU which “ties the knot” would still needs to be recompiled. And you generally want to tie the knot for tests).

                                                                              2. 2

                                                                                Tooling increasingly supports modules, which require you to do the opposite thing: have a single header for each library, parse it once, serialise the AST, and lazily load the small subset that you need. This composes with additional tooling such as Sony’s ‘compilation database’ work that caches template instantiations and even IR for individual snippets.

                                                                                The approach advocated in this article imposes a much larger burden on the programmer and makes it very hard for tooling to improve the situation.

                                                                                1. 2

                                                                                  This reminds me a lot of Robert Dewar’s paper on the GNAT compilation model,

                                                                                  He ditched the traditional Ada library database, and instead implemented Ada’s with dependency clauses in a similar manner to C #include, which made the compiler both simpler and faster.

                                                                                  1. 1

                                                                                    Interesting, thanks. I am vastly out of touch with what’s happened in C++ since 1998.

                                                                                    1. 1

                                                                                      In 2004 the approach advocated in the article paid off. And the larger burden was not quite enough of an ongoing thing to really hurt.

                                                                                      Modules would be much nicer if the ecosystem support is there. (I’m kind of thankful not to need to know whether it is… I spend a lot less time with my C++ tooling in 2022 than I did in 2004.)

                                                                                      And this:

                                                                                      additional tooling such as Sony’s ‘compilation database’ work that caches template instantiations

                                                                                      sounds like the stuff dreams are made of.

                                                                                  1. 10

                                                                                    I hope the author gets the help they need, but I don’t really see how the blame for their psychological issues should be laid at the feet of their most-recent employer.

                                                                                    1. 50

                                                                                      In my career I’ve seen managers cry multiple times, and this is one of the places that happened. A manager should never have to ask whether they’re a coward, but that happened here.

                                                                                      I dunno, doesn’t sound like they were the only person damaged by the experience.

                                                                                      Eventually my physicians put me on forced medical leave, and they strongly encouraged me to quit…

                                                                                      Seems pretty significant when medical professionals are telling you the cure for your issues is “quit this job”?

                                                                                      1. 16

                                                                                        Seems pretty significant when medical professionals are telling you the cure for your issues is “quit this job”?

                                                                                        A number of years ago I developed some neurological problems, and stress made it worse. I was told by two different doctors to change or quit my job. I eventually did, and it helped, but the job itself was not the root cause, nor was leaving the sole cure.

                                                                                        I absolutely cannot speak for OP’s situation, but I just want to point out that a doctor informing you to rethink your career doesn’t necessarily imply that the career is at fault. Though, in this case, it seems like it is.

                                                                                        1. 4

                                                                                          It doesn’t seem like the OP’s doctors told them to change careers though, just quit that job.

                                                                                          1. 3

                                                                                            To clarify, I’m using “career change” in a general sense. I would include quitting a job as a career change, as well as leaving one job for another in the same industry/domain. I’m not using it in the “leave software altogether” sense.

                                                                                      2. 24

                                                                                        I’m trusting the author’s causal assessment here, but employers (especially large businesses with the resources required) can be huge sources of stress and prevent employees from having the time or energy needed to seek treatment for their own needs, so they can both cause issues and worsen existing ones.

                                                                                        It’s not uncommon, for example, for businesses to encourage unpaid out-of-hours work for salaried employees by building a culture that emphasizes personal accountability for project success; this not only increases stress and reduces free time that could otherwise be used to relieve work-related stress, it teaches employees to blame themselves for what could just as easily be systemic failures. Even if an employee resists the social pressure to put in extra hours in such an environment, they’ll still be penalized with (real or imagined) blame from their peers, blame from themselves for “not trying hard enough”, and likely less job safety or fewer benefits.

                                                                                        In particular, there’s relevance from the business’ failure to support effective project management, manage workloads, or generally address problems repeatedly and clearly brought up to them. These kinds of things typically fuel burnout. The author doesn’t go into details enough for an outside observer to make a judgment call one way or the other, but if you trust the author’s account of reality then it seems reasonable to blame the employer for, at the least, negligently fueling these problems through gross mismanagement.

                                                                                        Arguably off-topic, but I think it might squeak by on the grounds that it briefly ties the psychological harm to the quality of a technical standard resulting from the mismanaged business process.

                                                                                        1. 3

                                                                                          a culture that emphasizes personal accountability for project success; this not only increases stress and reduces free time that could otherwise be used to relieve work-related stress, it teaches employees to blame themselves for what could just as easily be systemic failures.

                                                                                          This is such a common thing. An executive or manager punts on actually organizing the work, whether from incompetence or laziness, and then tries to make the individuals in the system responsible for the failures that occur. It’s hardly new. Deming describes basically this in ‘The New Economics’ (look up the ‘red bead game’).

                                                                                          More cynically, is WebAssembly actuall in Google’s interests? It doesn’t add revenue to Google Cloud. It’s going to make their data collection harder (provide Google analytics libraries for how many languages?). It was clearly a thing that was gaining momentum, so if they were to damage it, they would need to make sure they had a seat at the table and then make sure that the seat was used as ineffectually and disruptively as possible.

                                                                                          1. 9

                                                                                            More cynically, is WebAssembly actually in Google’s interests?

                                                                                            I think historically the answer would have been yes. Google has at various points been somewhat hamstrung by shipping projects with slow front end JS in them and responded by trying to make browsers themselves faster. e.g. creating V8 and financially contributing to Mozilla.

                                                                                            I couldn’t say if Google now has any incentive to not make JS go fast. I’m not aware of one. I suspect still the opposite. I think they’re also pushing mobile web apps as a way to inconvenience Apple; I think Google currently want people to write portable software using web tech instead of being tempted to write native apps for iOS only.

                                                                                            That said, what’s good for the company is not the principle factor motivating policy decisions. What’s good for specific senior managers inside Google is. Otherwise you wouldn’t see all these damn self combusting promo cycle driven chat apps from Google. A company is not a monolith.

                                                                                            ‘The New Economics’

                                                                                            I have this book and will have to re-read at least this bit tomorrow. I have slightly mixed feelings about it, mostly about the writing style.

                                                                                            1. 1

                                                                                              Making JS fast is one thing. Making a target for many other languages, as opposed to maintaining analytics libraries and other ways of gathering data for one languages?

                                                                                              Your point about the senior managers’ interests driving what’s done is on point, though. Google and Facebook especially are weird because ads fund the company, and the rest is all some kind of loss leader floating around divorced from revenue.

                                                                                              The only thing I’ll comment about Deming is that the chapter on intrinsic vs extrinsic motivation should be ignored, as that’s entirely an artifact despite its popularity. The rest of the book has held up pretty well.

                                                                                              1. 10

                                                                                                Making JS fast is one thing. Making a target for many other languages, as opposed to maintaining analytics libraries and other ways of gathering data for one languages?

                                                                                                Google doesn’t need to maintain their analytics libraries in many other languages, only to expose APIs callable from those languages. All WebAssembly languages can call / be called by JavaScript.

                                                                                                More generally, Google has been the biggest proponent of web apps instead of web services. Tim Berners-Lee’s vision for the web was that you’d have services that provided data with rich semantic markup. These could be rendered as web pages but could equally plug into other clients. The problem with this approach is that a client that can parse the structure of the data can choose to render it in a way that simply ignores adverts. If all of your adds are in an <advert provider="google"> block then an ad blocker is a trivial browser extension, as is something that displays ads but restricts them to plain text. Google’s web app push has been a massive effort to convince everyone to obfuscate the contents of their web pages. This has two key advantages for Google:

                                                                                                • Writing an ad blocker is hard if ads and contents are both generated from a Turing-complete language using the same output mechanisms.
                                                                                                • Parsing such pages for indexing requires more resources (you can’t just parse the semantic markup, you must run the interpreter / JIT in your crawler, which requires orders of magnitude more hardware than simply parsing some semantic mark-up. This significantly increases the barrier to entry for new search engines, protecting Google’s core user-data-harvesting tool.

                                                                                                WebAssembly fits very well into Google’s vision for the web.

                                                                                                1. 2

                                                                                                  I used to work for a price-comparison site, back when those were actual startups. We had one legacy price information page that was Java applet (remember those?) Supposedly the founders were worried about screen scrapers so wanted the entire site rendered with applets to deter them.

                                                                                                  1. 1

                                                                                                    This makes more sense than my initial thoughts. Thanks.

                                                                                                  2. 2

                                                                                                    Making a target for many other languages, as opposed to maintaining analytics libraries and other ways of gathering data for one languages?

                                                                                                    This is something I should have stated explicitly but didn’t think to: I don’t think wasm is actually going to be the future of non-JS languages in the browser. I think they for the next couple decades at least, wasm is going to be used for compute kernels (written in other langs like C++ and Rust) that get called from JS.

                                                                                                    I’m taking a bet here that targeting wasm from langs with substantial runtimes will remain unattractive indefinitely due to download weight and parsing time.

                                                                                                    about Deming

                                                                                                    I honestly think many of the points in that book are great but hoo boy the writing style.

                                                                                            2. 1

                                                                                              That is exactly what I thought while reading this. I understand that to a lot of people, WebAssembly is very important, and they have a lot of emotions vested into the success. But to the author’s employer, it might not be as important, as it might not directly generate revenue. The author forgets that to the vast, vast majority of people on this earth, having the opportunity to work on such a technology at a company like Google is an unparalleled privilege. Most people on this earth do not have the opportunity to quit their job just because a project is difficult, or because meetings run long or it is hard to find consensus. Managing projects well is incredibly hard. But I am sure that the author was not living on minimum wage, so there surely was compensation for the efforts.

                                                                                              It is sad to hear that the author has medical issues, and I hope those get sorted out. And those kinds of issues do exacerbate stressful jobs. But that is not a good reason for finger pointing. Maybe the position just was not right for the author, maybe there are more exciting projects that are waiting in the future. I certainly hope so. But it is important not to blame one’s issues on others, that is not a good attitude in life.

                                                                                              1. 25

                                                                                                Using the excuse that because there exist others less fortunate, it’s not worth fighting to make something better is also not a good attitude in life.

                                                                                                Reading between the lines, it feels to me like there was a lot that the author left unsaid, and that’s fine. It takes courage to share a personal story about mental wellbeing, and an itemized list of all the wrongs that took place is not necessary to get the point the author was trying to make across.

                                                                                                My point is that I’d be cautious about making assumptions about the author’s experiences as they didn’t exactly give a lot of detail here.

                                                                                                1. 3

                                                                                                  Using the excuse that because there exist others less fortunate, it’s not worth fighting to make something better is also not a good attitude in life.

                                                                                                  This is true. It is worth fighting to make things better

                                                                                                  Reading between the lines, it feels to me like there was a lot that the author left unsaid, and that’s fine. It takes courage to share a personal story about mental wellbeing, and an itemized list of all the wrongs that took place is not necessary to get the point the author was trying to make across.

                                                                                                  There is a lot of things that go into mental wellbeing. Some things you can control, some things are genetic. I don’t know what the author left out, but I have not yet seen a study showing that stressful office jobs give people brain damage. There might be things the author has not explained, but at the same time that is a very extreme claim. In fact, if that were true, I am sure that the author should receive a lot in compensation.

                                                                                                  My point is that I’d be cautious about making assumptions about the author’s experiences as they didn’t exactly give a lot of detail here.

                                                                                                  I agree with you, but I also think that if someone makes a very bold claim about an employer, especially about personal injury, that these claims should be substantiated. There is a very big difference between “working there was hard, I quit” and “the employer acted recklessly and caused me personal injury”. And I don’t really know which one the author is saying, because from the description could be interpreted as it just being a difficult project to see through.

                                                                                                  1. 8

                                                                                                    In fact, if that were true, I am sure that the author should receive a lot in compensation.

                                                                                                    By thinking about it for a few seconds you can realize that this can easily not happen. The OP itself says that they don’t have documented evidence from the time because of all the issues they were going through. And it’s easy to see why: if your mental health is damaged, your brain is not working right, would you be mindful enough to take detailed notes of every incident and keep a trail of evidence for later use in compensation claims? Or are you saying that compensation would be given out no questions asked?

                                                                                                    1. 3

                                                                                                      All I’m saying is, there is a very large difference between saying this job was very stressful, I had trouble sleeping and it negatively affected my concentration and memory and saying this job gave me brain damage. Brain damage is relatively well-defined:

                                                                                                      The basic definition of brain damage is an injury to the brain caused by various conditions such as head trauma, inadequate oxygen supply, infections, or intracranial hemorrhage. This damage may be associated with a behavioral or functional abnormality.

                                                                                                      Additionally, there are ways to test for this, a neurologist can make that determination. I’m not a neurologist. But it would be the first time I heard that brain damage be caused by psychosomatic issues. I believe that the author may have used this term in error. That’s why I said what I said — if you, or anyone, has brain damage as a result of your occupation, that is definitely grounds for compensation. And not a small compensation either, as brain damage is no joke. This is a very different category from mere psychological stress from working for an apparently mismanaged project.

                                                                                                      1. 5


                                                                                                        Brain damage is an injury that causes the destruction or deterioration of brain cells.

                                                                                                        Anxiety, stress, lack of sleep, and other factors can potentially do that. So I don’t see any incorrect use of the phrase ‘brain damage’ here. And anyway, you missed the point. Saying ‘This patient has brain damage’ is different from saying ‘Working in the WebAssembly team at Google caused this patient’s brain damage’. When you talk about causation and claims of damage and compensation, people tend to demand documentary evidence.

                                                                                                        I agree brain damage is no joke, but if you look at society it’s very common for certain types of relatively-invisible mental illnesses to be downplayed and treated very lightly, almost as a joke. Especially by people and corporations who would suddenly have to answer for causing these injuries.

                                                                                                        1. 4

                                                                                                          Anxiety, stress, lack of sleep and other factors cannot, ever, possibly, cause brain damage. I think you have not completely read that article. It states – as does the definition that I linked:

                                                                                                          All traumatic brain injuries are head injuries. But head injury is not necessarily brain injury. There are two types of brain injury: traumatic brain injury and acquired brain injury. Both disrupt the brain’s normal functioning.

                                                                                                          • Traumatic Brain Injury(TBI) is caused by an external force – such as a blow to the head – that causes the brain to move inside the skull or damages the skull. This in turn damages the brain.
                                                                                                          • Acquired Brain Injury (ABI) occurs at the cellular level. It is most often associated with pressure on the brain. This could come from a tumor. Or it could result from neurological illness, as in the case of a stroke.

                                                                                                          There is no kind of brain injury that is caused by lack of sleep or stress. That is not to say that these things are not also damaging to one’s body and well-being.

                                                                                                          Mental illnesses can be very devastating and stressful on the body. But you will not get a brain injury from a mental illness, unless it makes you physically impact your brain (causing traumatic brain injury), ingest something toxic, or have a stroke. It is important to be very careful with language and not confuse terms. The term “brain damage” is colloquially often used to describe things that are most definitely not brain damage, like “reading this gave me brain damage”. I hope you understand what I’m trying to state here. Again, the author has possibly misused the term “brain damage”, or there is some physical trauma that happened that the author has not mentioned in the article.

                                                                                                          I hope you understand what I am trying to say here!

                                                                                                          1. 9

                                                                                                            Anxiety and stress raise adrenaline levels, which in turn cause short- and long-term changes in brain chemistry. It sounds like you’ve never been burnt out; don’t judge others so harshly.

                                                                                                            1. 3

                                                                                                              Anxiety and stress are definitely not healthy for a brain. They accelerate aging processes, which is damaging. But brain damage in a medical context refers to large-scale cell death caused by genetics, trauma, stroke or tumors.

                                                                                                            2. 8

                                                                                                              There seems to be a weird definitional slide here from “brain damage” to “traumatic brain injury.” I think we are all agreed that her job did not give her traumatic brain injury, and this is not claimed. But your claim that stress and sleep deprivation cannot cause (acquired) brain injury is wrong. In fact, you will find counterexamples by just googling “sleep deprivation brain damage”.

                                                                                                              “Mental illnesses can be … stressful on the body.” The brain is part of the body!

                                                                                                              1. 1

                                                                                                                I think you – and most of the other people that have responded to my comment – have not quite understood what I’m saying. The argument here is about the terms being used.

                                                                                                                Brain Damage

                                                                                                                Brain damage, as defined here, is damage caused to the brain by trauma, tumors, genetics or oxygen loss, such as during a stroke. This leads to potentially large chunks of your brain to die off. This means you can lose entire brain regions, potentially permanently lose some abilities (facial recognition, speech, etc).

                                                                                                                Sleep Deprivation

                                                                                                                See Fundamental Neuroscience, page 961:

                                                                                                                The crucial role of sleep is illustrated by studies showing that prolonged sleep deprivation results in the distruption of metabolic processes and eventually death.

                                                                                                                When you are forcibly sleep deprived for a long time, such as when you are being tortured, your body can lose the ability to use nutrients and finally you can die. You need to not sleep at all for weeks for this to happen, generally this is not something that happens to people voluntarily, especially not in western countries.


                                                                                                                The cells in your brain only have a finite lifespan. At some point, they die and new ones take their place (apoptosis). Chronic stress and sleep deprivation can speed up this process, accelerating aging.

                                                                                                                Crucially, this is not the same as an entire chunk of your brain to die off because of a stroke. This is a very different process. It is not localized, and it doesn’t cause massive cell death. It is more of a slow, gradual process.


                                                                                                                Mental illnesses can be … stressful on the body.” The brain is part of the body!

                                                                                                                Yes, for sure. It is just that the term “brain damage” is usually used for a very specific kind of pattern, and not for the kind of chronlc, low-level damage done by stress and such. A doctor will not diagnose you with brain damage after you’ve had a stressful interaction with your coworker. You will be diagnosed with brain damage in the ICU after someone dropped a hammer on your head. Do you get what I’m trying to say?

                                                                                                                1. 4

                                                                                                                  I get what you are trying to say, I think you are simply mistaken. If your job impairs your cognitive abilities, then it has given you brain damage. Your brain, is damaged. You have been damaged in your brain. The cells and structures in your brain have taken damage. You keep trying to construct this exhaustive list of “things that are brain damage”, and then (in another comment) saying that this is about them not feeling appreciated and valued or sort of vaguely feeling bad, when what they are saying is that working at this job impaired their ability to form thoughts. That is a brain damage thing! The brain is an organ for forming thoughts. If the brain can’t thoughts so good no more, then it has been damaged.

                                                                                                                  The big picture here is that a stressful job damaged this person’s health. Specifically, their brain’s.

                                                                                                                  1. 3

                                                                                                                    I understand what you are trying to say, but I think you are simply mistaken. We (as a society) have definitions for the terms we use. See

                                                                                                                    Neurotrauma, brain damage or brain injury (BI) is the destruction or degeneration of brain cells. Brain injuries occur due to a wide range of internal and external factors. In general, brain damage refers to significant, undiscriminating trauma-induced damage.

                                                                                                                    This is not “significant, undiscriminating trauma-induced damage” (for context, trauma here refers to physical trauma, such as an impact to the head, not psychological trauma). What the author describes does not line up with any of the Causes of Brain Damage. It is simply not the right term.

                                                                                                                    Yes, the author has a brain, and there is self-reported “damage” to it. But just because someone is a man and feels like he polices the neighborhood, does not make me a “police man”. Just because I feel like my brain doesn’t work right after a traumatic job experience does not mean I have brain damage™.

                                                                                                                    1. 1

                                                                                                                      The Wikipedia header is kind of odd. The next sentence after “in general, brain damage is trauma induced” lists non-trauma-induced categories of brain damage. So I don’t know how strong that “in general” is meant to be. At any rate, “in general” is not at odds with the use of the term for non-trauma induced stress/sleep depriv damage.

                                                                                                                      At any rate, if you click through to Acquired Brain Injury, it says “These impairments result from either traumatic brain injury (e.g. …) or nontraumatic injury … (e.g. listing a bunch of things that are not traumatic.)”

                                                                                                                      Anyway, the Causes of Brain Damage list is clearly not written to be exhaustive. “any number of conditions, including” etc.

                                                                                                              2. 2

                                                                                                                There is some evidence that lack of sleep may kill brain cells:

                                                                                                                It’s also possible to suffer from mini-strokes due to the factors discussed above.

                                                                                                                In any case, I feel like you’re missing the forest for the trees. Sure, it’s important to be correct with wording. But is that more important than the bigger picture here, that a stressful job damaged this person’s health?

                                                                                                                1. 2

                                                                                                                  the bigger picture here, that a stressful job damaged this person’s health

                                                                                                                  Yes, that is true, and it is a shame. I really wish that the process around WASM be less hostile, and that this person not be impacted negatively, even if stressful and hard projects are an unfortunate reality for many people.

                                                                                                                  I feel like you’re missing the forest for the trees.

                                                                                                                  I think that you might be missing the forest for the trees – I’m not saying that this person was not negatively impacted, I am merely stating that it is (probably, unless there is evidence otherwise) to characterize this impact as “brain damage”, because from a medical standpoint, that term has a more narrow definition that damage due to stress does not fulfill.

                                                                                                        2. 4

                                                                                                          Hello, you might enjoy this study.


                                                                                                          I looked through a lot of studies to try and find a review that was both broad and to the point.

                                                                                                          Now, you are definitely mixing a lot of terms here… but I hope that if you read the research, you can be convinced, at the very least, that stress hurts brains (and I hope that reading the article and getting caught in this comment storm doesn’t hurt yours).

                                                                                                          1. 2

                                                                                                            Sleep Deprivation and Oxidative Stress in Animal Models: A Systematic Review tells us that sleep deprivation can be shown to increase oxidative stress:

                                                                                                            Current experimental evidence suggests that sleep deprivation promotes oxidative stress. Furthermore, most of this experimental evidence was obtained from different animal species, mainly rats and mice, using diverse sleep deprivation methods.

                                                                                                            Although, disagrees with this. Furthermore, it is known that oxidative stress promotes apoptosis, see Oxidative stress and apoptosis :

                                                                                                            Recent studies have demonstrated that reactive oxygen species (ROS) and the resulting oxidative stress play a pivotal role in apoptosis. Antioxidants and thiol reductants, such as N-acetylcysteine, and overexpression of manganese superoxide (MnSOD) can block or delay apoptosis.

                                                                                                            The article that you linked Stress effects on the hippocampus: a critical review mentions that stress has an impact on the development of the brain and on it’s workings:

                                                                                                            Uncontrollable stress has been recognized to influence the hippocampus at various levels of analysis. Behaviorally, human and animal studies have found that stress generally impairs various hippocampal-dependent memory tasks. Neurally, animal studies have revealed that stress alters ensuing synaptic plasticity and firing properties of hippocampal neurons. Structurally, human and animal studies have shown that stress changes neuronal morphology, suppresses neuronal proliferation, and reduces hippocampal volume

                                                                                                            I do not disagree with this. I think that anyone would be able to agree that stress is bad for the brain, possibly by increasing apoptosis (accelerating ageing), decreasing the availability of nutrients. My only argument is that the term brain damage is quite narrowly defined (for example here) as (large-scale) damage to the brain caused by genetics, trauma, oxygen starvation or a tumor, and it can fall into one of two categories: traumatic brain injuries and acquired brain injuries. If you search for “brain damage” on pubmed, you will find the term being used like this:

                                                                                                            You will not find studies or medical diagnoses of “brain damage due to stress”. I hope that you can agree that using the term brain damage in a context such as the author’s, without evidence of traumatic injury or a stroke, is wrong. This does not take away the fact that the author has allegedly experienced a lot of stress at their previous employer, one of the largest and high-paying tech companies, and that this experience has caused the author personal issues.

                                                                                                            On an unrelated note: what is extremely fascinating to me is that some chemicals such as methamphetamine (at low concentrations) or minocycline are neuroprotective being able to lessen brain damage for example due to stroke. But obviously, at larger concentrations the opposite is the case.

                                                                                                            1. 1

                                                                                                              How about this one then?

                                                                                                              We can keep going, it is not difficult to find these… Your’re splitting a hair which should not be split.

                                                                                                              What’s so wrong about saying a bad work environment can cause brain damage?

                                                                                                              1. 1

                                                                                                                Your’re splitting a hair which should not be split.

                                                                                                                There is nothing more fun than a civil debate. I would argue that any hair deserves being split. Worst case, you learn something new, or form a new opinion.

                                                                                                                What’s so wrong about saying a bad work environment can cause brain damage?

                                                                                                                Nothing is wrong with that, if the work environment involves heavy things, poisonous things, or the like. This is why OSHA compliance is so essential in protecting people’s livelihoods. I just firmly believe, and I think that the literature agrees with me on this, that “brain damage” as a medical definition refers to large-scale cell death due to trauma or stroke, and not chronic low-level damage caused by stress. The language we choose to use is extremely important, it is the only facility we have to exchange information. Language is not useful if it is imprecise or even wrong.

                                                                                                                How about this one then?

                                                                                                                Let’s take a look what we got here. I’m only taking a look at the abstract, for now.

                                                                                                                Stress is a risk factor for a variety of illnesses, involving the same hormones that ensure survival during a period of stress. Although there is a considerable ambiguity in the definition of stress, a useful operational definition is: “anything that induces increased secretion of glucocorticoids”.

                                                                                                                Right, stress causes elevated levels of glucocorticoids, such as cortisol.

                                                                                                                The brain is a major target for glucocorticoids. Whereas the precise mechanism of glucocorticoid-induced brain damage is not yet understood, treatment strategies aimed at regulating abnormal levels of glucocorticoids, are worth examining.

                                                                                                                Glucocorticoids are useful in regulating processes in the body, but they can also do damage. I had never heard of the term glucocorticoid-induced brain damage, and searching for it in the literature only yields this exact article, so I considered this a dead end. However, in doing some more research, I did find two articles that somewhat support your hypothesis:

                                                                                                                In Effects of brain activity, morning salivary cortisol, and emotion regulation on cognitive impairment in elderly people, it is mentioned that high cortisol levels are associated with hippocampus damage, supporting your hypothesis, but it only refers to elderly patients with Mild Cognitive Impairment (MCI):

                                                                                                                Cognitive impairment is a normal process of aging. The most common type of cognitive impairment among the elderly population is mild cognitive impairment (MCI), which is the intermediate stage between normal brain function and full dementia.[1] MCI and dementia are related to the hippocampus region of the brain and have been associated with elevated cortisol levels.[2]

                                                                                                                Cortisol regulates metabolism, blood glucose levels, immune responses, anti-inflammatory actions, blood pressure, and emotion regulation. Cortisol is a glucocorticoid hormone that is synthesized and secreted by the cortex of adrenal glands. The hypothalamus releases a corticotrophin-releasing hormone and arginine vasopressin into hypothalamic-pituitary portal capillaries, which stimulates adrenocorticotropic hormone secretion, thus regulating the production of cortisol. Basal cortisol elevation causes damage to the hippocampus and impairs hippocampus-dependent learning and memory. Chronic high cortisol causes functional atrophy of the hypothalamic-pituitary-adrenal axis (HPA), the hippocampus, the amygdala, and the frontal lobe in the brain.

                                                                                                                Additionally, Effects of stress hormones on the brain and cognition: Evidence from normal to pathological aging mentions that chronic stress is a contributor to memory performance decline.

                                                                                                                We might be able to find a few mentions of brain damage outside of the typical context (as caused by traumatic injury, stroke, etc) in the literature, but at least we can agree that the term brain damage is quite unusual in the context of stress, can we not? Out of the 188,764 articles known by pubmed, only 18,981 mention “stress”, and of those the almost all are referring to “oxidative stress” (such as that experienced by cells during a stroke). I have yet to find a single study or article that directly states brain damage as being a result of chronic stress, in the same way that there are hundreds of thousands of studies showing brain damage from traumatic injuries to the brain.

                                                                                                                1. 2

                                                                                                                  Well, if anybody asks me I will tell them that too much stress at work causes brain damage… and now I can even point to some exact papers!

                                                                                                                  I agree that it’s a little hyperbolic, but it’s not that hyperbolic. If we were talking about drug use everyone would kind of nod and say, ‘yeah, brain damage’ even if the effects were tertiary and the drug use was infrequent.

                                                                                                                  But stress at work! Ohohoho, that’s just life my friend! Which really does not need to be the way of the world… OP was right to get out, especially once they started exhibiting symptoms suspiciously like the ones cited in that last paper (you know, the sorts of symptoms you get when your brain is suffering from some damage).

                                                                                                                  1. 2

                                                                                                                    If someone tells me that they got brain damage from stress at work, I will laugh, tell them to read the Wikipedia article article and then move on. But that is okay, we can agree to disagree. I understand that there are multiple possible definitions for the term brain damage.

                                                                                                                    If we were talking about drug use everyone would kind of nod and say, ‘yeah, brain damage’ even if the effects were tertiary and the drug use was infrequent.

                                                                                                                    In my defense, people often use terms incorrectly.

                                                                                                                    OP was right to get out

                                                                                                                    I agree. Brain damage or not, Google employee or not, if you are suffering at work you should not stay there. We all have very basic needs, and one of them is being valued and being happy to work.

                                                                                                                    Anyways, I hope you have a good weekend!

                                                                                                          2. 6

                                                                                                            I have not yet seen a study showing that stressful office jobs give people brain damage.

                                                                                                            This is a bizarre and somewhat awful thread. Please could you not post things like this in future?

                                                                                                            1. 8

                                                                                                              I disagree. The post seemed polite, constructive, and led to (IMO) a good conversation (including some corrections to the claims in the post).

                                                                                                              1. 4

                                                                                                                Parent left a clear method for you to disprove them by providing a counter-example.

                                                                                                                If you can point to some peer-reviewed research on the topic, by all means do so.

                                                                                                                1. 5

                                                                                                                  Yea but this is an obnoxious, disrespectful, and disingenuous way to conduct an argument. I haven’t read any studies proving anything about this subject one way or another. Because I am not a mental health researcher. So it’s easy for me to make that claim, and present the claim as something that matters, when really it’s a pointless claim that truly does not matter at all.

                                                                                                                  Arguing from an anecdotal position based on your own experience, yet demanding the opposing side provide peer-reviewed studies to contradict your anecdotal experience, places a disproportionate burden on them to conduct their argument. And whether intentional or not, it strongly implies that you have little to no respect for their experiences or judgement. That you will only care about their words if someone else says them.

                                                                                                      1. 4

                                                                                                        I’m not in favour. I like the fact that lobsters doesn’t have much humour on it so I would rather it not be encouraged too much. The amount of satire here suffices for me.

                                                                                                        1. 11

                                                                                                          I’d go with “funny” just to avoid a “humor vs humour” debate.

                                                                                                          1. 2

                                                                                                            And to avoid cultivating a subcommunity of humorism proponents? :)

                                                                                                            1. 1

                                                                                                              The obvious compromise is “humur” to annoy everyone. ;)

                                                                                                              1. 1

                                                                                                                It’s humorous to think that people waste time on debating UK vs US spelling when having a localised interface should be the answer both sides strive for.

                                                                                                              1. 1

                                                                                                                I suppose, if there is one core influence on me, it’s Envelop Basic: I never understood the language very well when I was writing it, but I definitely picked up an attitude of “however I can get something to work” from abusing UI controls to make a video game, even though that was not their intended purpose.

                                                                                                                Which means that I don’t a lot of misgivings about doing non-standard things, if it feels called for by the constraints of the project I’m working on. This attitude has lead me to do things like adding string-join to SQL Server via .NET code, or writing a project in Ruby in Sequel and Sinatra instead of Rails, or building a .NET server/client on web sockets, where the server was intended to make the client easier to test/validate against another service.

                                                                                                                It also means that I read widely, so as to add to my bag of tricks that I can reference when I hit a tricky situation.

                                                                                                                The counter-balance to this is that because I like reading code, I also want to write code that other people like to read, which has also been a long-running influence in how I write code, and it helps me try to avoid treating any particular bit of code as Sacred or Precious, and means that I don’t do non-standard things in shared projects without a reason.

                                                                                                                1. 3

                                                                                                                  … I don’t do non-standard things in shared projects without a reason.

                                                                                                                  This is the key, for me. Do whatever you want in side projects or toys, or even “real” code that only you will ever need to maintain. But if there’s a good chance other people will need to be involved, make “boring” choices. This was a hard lesson for me to learn, personally, so I understand why some people recoil a bit, but it’s important.

                                                                                                                  1. 2

                                                                                                                    I will say, I do think it’s useful to be able to go off the beaten path when it’s called for. It can be a bit of strong leverage, or help you sort out messes that other folks have gotten themselves into.

                                                                                                                    1. 2

                                                                                                                      I think it’s useful to have confidence that you could, if you had to, go and do something complicated and weird. The confidence permits you to go ahead with a straightforward implementation without any hedging in case it’s not “fast enough”. That often runs plenty fast on the first try anyway.

                                                                                                                      1. 2

                                                                                                                        I agree. Granted, to do that, you have to try and do complicated/weird things. I’ve done done both as part of work and as part of hobby stuff. For work, I try to only do it where there seems to be much to gain by doing so.

                                                                                                                        For hobby stuff, I work almost exclusively in niche tech, mostly to keep stuff interesting.

                                                                                                                        I saw a suggestion that developers should be given some room to play outside of production, because if you don’t, they’ll play inside production. And, for me, I took that as a mandate to try strange/different things on my own time, and to generally not do them in prod without a good reason.

                                                                                                                        And, now that I’ve been that for many years, a lot of stuff I’d have had questions about some years ago seems relatively standard. When you’ve written in non-standard programming paradigm (stack-based was this for me), more standard paradigms seem tame and easy to follow by comparison.

                                                                                                                1. 8

                                                                                                                  Cute benchmark pic, but:

                                                                                                                  • No decompression runtime / memory.
                                                                                                                  • No lz4 nor zstd comparison points.
                                                                                                                  1. 12

                                                                                                                    I agree, the benchmark is weird. Besides the fact that some important competitors are missing, I understand that compression implementations make trade-offs between time/memory and compression efficiency, so it’s hard to compare them fairly by just testing one of their configuration. Instead you should plot the graph, over varying configurations, of time/memory consumption and compression ratio. (If the line drawn by one algorithm is always “above” another, then one can say that it is always better, but often they will cross in some places, telling us interesting facts about the strengths and weaknesses of both.)

                                                                                                                    1. 2

                                                                                                                      Good point. And there should be a tool for running benchmarks and plotting the charts automatically.

                                                                                                                      1. 2

                                                                                                                        First commit on May 1st this year. Hopefully they’ll get to it!

                                                                                                                    2. 2

                                                                                                                      They do compare to lz4 and zstd for some of the test workloads, not sure why not for everything. They’re not the comparison I’d like though, I don’t see an xz comparison anywhere. For me, xz has completely replaced bzip2, for the things where compression ratio is the most important factor. lz4 and zstd are replacing gzip for things where I need more of a balance between compression ratio and throughput / memory consumption.

                                                                                                                      1. 2

                                                                                                                        They do compare to lz4 and zstd for some of the test workloads, not sure why not for everything.

                                                                                                                        They added that after my comment… and yeah, it’s odd they didn’t do it on all workloads.

                                                                                                                        xz is based on lzma, but not exactly the same. Maybe they thought including lzma was enough.

                                                                                                                        1. 1

                                                                                                                          Tangentially, I tried using brotli for something at work recently and at compression level 4 it beats deflate wl hands down, at about the same compression speed. I was impressed.

                                                                                                                      1. 17

                                                                                                                        On the one hand, I totally get the value of a lack of a build step. Build steps are annoying. On the other hand, authoring directly in HTML is something I am perfectly happy to do as little of as possible. It’s just not a pleasant language to write in for any extended amount of time!

                                                                                                                        1. 20

                                                                                                                          I’m pretty convinced that Markdown is the local maxima for the “low effort, nice looking content” market.

                                                                                                                          1. 10

                                                                                                                            Agreed. ASCIIDoc, reStructuredText, LaTeX, and other more-robust-than-Markdown syntaxes all have significantly more power but also require a great deal more from you as a result. For just put words out, Markdown is impressively “good enough”.

                                                                                                                            1. 4

                                                                                                                              I can never remember Markdown syntax (or any other wiki syntax for that matter), while I’m fairly fluent in HTML, and I’m not even a frontend dev. HTML also has the advantage that if some sort of exotic markup is necessary, you know it’s expressble, given time and effort.

                                                                                                                              1. 7

                                                                                                                                That’s fine, because Markdown allows embedded HTML [1]

                                                                                                                                About the only thing that’s a bit obtuse is the link syntax, and I’ve gladly learned that to not have to manually enclose every damn list with or tags.

                                                                                                                                [1] at least Gruber’s OG Markdown allowed it by default, and I recently learned CommonMark has an “unsafe” mode to allow it too.

                                                                                                                                1. 11

                                                                                                                                  The trick to remember how to do links in Markdown is to remember that there are brackets and parentheses involved, then think what syntax would make sense, then do the opposite.

                                                                                                                                  1. 4

                                                                                                                                    For reference: a Markdown [link](

                                                                                                                                    Elaboration on the mnemonic you describe

                                                                                                                                    I thought like you when I first started learning Markdown:

                                                                                                                                    • Parentheses () are normal English punctuation, so you would intuitively expect them to surround the text, but they don’t.
                                                                                                                                    • Square brackets [] are technical symbols, so you would intuitively expect them to surround the URL, but they don’t.

                                                                                                                                    However, I find “don’t do this” mnemonics easy to accidentally negate, so I don’t recommend trying to remember the order that way.

                                                                                                                                    Another mnemonic

                                                                                                                                    I think Markdown’s order of brackets and parentheses is easier to remember once one recognizes the following benefit:

                                                                                                                                    When you read the first character in […](…), it’s clear that you’re reading a link. ‘[’ is a technical symbol, so you know you’re not reading a parenthetical, which would start with ‘(’. Demonstration:

                                                                                                                                    In this Markdown, parentheticals (which are everywhere) and
                                                                                                                                    [links like these]( can quickly be told
                                                                                                                                    apart when reading from left to right.
                                                                                                                                    Why not URL first?

                                                                                                                                    Since you wrote that Markdown does “the opposite”, I wonder if you also intuitively expect the syntax to put the URL before the text, like in [ MediaWiki’s syntax] (actual link: MediaWiki’s syntax). I never found that order intuitive, but I can explain why I prefer text first:

                                                                                                                                    When trying to read only the text and skip over the URLs, it’s easier to skip URLs if they come between grammatical phrases of the text (here), rather than interrupting a (here) phrase. And links are usually written at the end of phrases, rather than at the beginning.

                                                                                                                                    1. 2

                                                                                                                                      Well I’ll be dammed. That completely makes sense.

                                                                                                                                      I do, however, wonder whether this is a post-hoc rationalization and the real reason for the syntax is much dumber.

                                                                                                                                    2. 3

                                                                                                                                      Hah. The mnemonic I use is everyone gets the ) on the end of their wiki URLs fucked up by markdown… because the () goes around the URL. therefore it is s []().

                                                                                                                                      1. 2

                                                                                                                                        This is exactly what I do. Parens are for humans, square brackets are for computers, so obviously it’s the other way around in markdown.

                                                                                                                                      2. 3

                                                                                                                                        A wiki also implies a social contract about editability. If my fellow editors have expressed that they’re uncomfortable with HTML, it’s not very polite of me to use it whenever I find Markdown inconvenient.

                                                                                                                                        1. 1

                                                                                                                                          Of course. I was replying in context of someone writing for themselves.

                                                                                                                                      3. 3

                                                                                                                                        This is interesting: I’ve heard that same experience report from a number of people over the years so I believe it’s a real phenomenon (the sibling comment about links especially being the most common) but Markdown clicked instantly for me so I always find it a little surprising!

                                                                                                                                        I have hypothesized that it’s a case of (a) not doing it in a sustained way, which of course is the baseline, and (b) something like syntactical cross-talk from having multiple markup languages floating around; I took longer to learn Confluence’s wiki markup both because it’s worse than Markdown but also because I already had Markdown, rST, and Textile floating around in my head.

                                                                                                                                        I’m curious if either or both of those ring true, or if you think there are other reasons those kinds of markup languages don’t stick for you while HTML has?

                                                                                                                                        1. 2

                                                                                                                                          I’m not Michiel, but for me, it’s because HTML is consistent (even if it’s tedious). In my opinion, Gruber developed Markdown to make it easier for him to write HTML, and to use conventions that made sense to him for some shortcuts (the fact that you could include HTML in his Markdown says to me that he wasn’t looking to replace HTML). Markdown was to avoid having to type common tags like <P> or <EM>.

                                                                                                                                          For years I hand wrote the HTML for my blog (and for the record, I still have to click the “Markdown formatting available” link to see how to make links here). A few years ago I implemented my own markup language [1] that suits me. [2] My entries are still stored as HTML. That is a deliberate decision so I don’t get stuck with a subpar markup syntax I late come to hate. I can change the markup language (I’ve done it a few times already) and if I need to edit past entries, I can deal with the HTML.

                                                                                                                                          [1] Sample input file

                                                                                                                                          [2] For instance, a section for quoting email, which I do quite often. Or to include pictures in my own particular way. Or tabular data with a very light syntax and some smarts to generate the right class on <TD> elements consisting of numeric data (so they’re flush right). Stuff like that.

                                                                                                                                          1. 2

                                                                                                                                            Yeah, with markdown, I often accidentally trigger some of its weird syntax. It needs a bunch of arbitrary escapes, whereas HTML you can get away with just using &lt;. Otherwise, it is primarily just those <p> tags that get you; the rest are simple or infrequent enough to not worry about.

                                                                                                                                            whereas again, with the markdown, it is too easy to accidentally write something it thinks is syntax and break your whole thing.

                                                                                                                                            1. 1

                                                                                                                                              Yes, I’ve found that with mine as well.

                                                                                                                                            2. 1

                                                                                                                                              I don’t mean this as an offense, but I did a quick look at your custom markup sample and I hated pretty much everything about it.

                                                                                                                                              Since we’re all commenting under a post from someone that is handwriting HTML, I think it goes without saying that personal preferences can vary enormously.

                                                                                                                                              Updated: I don’t hate the tables syntax, and, although I don’t particularly like que quote syntax, having a specific syntax for it is cool and a good idea.

                                                                                                                                              1. 1

                                                                                                                                                Don’t worry about hating it—even I hate parts of it. It started out as a mash up of Markdown or Org mode. The current version I’m using replaces the #+BEGIN_blah #+END_blah with #+blah #-blah. I’m still working on the quote syntax. But that’s the thing—I can change the syntax of the markup, because I don’t store the posts in said markup format.

                                                                                                                                            3. 2

                                                                                                                                              You’re absolutely right, and so is spc476; HTML has a regular syntax. Even if I’ve never seen the <aside> tag, I can reason about what it does. Escaping rules are known and well-defined. If you want to read the text, you know you can just ignore anything inside the angular brackets.

                                                                                                                                              Quick: in Markdown, if I want to use a backtick in a fixed-width span, do I have to escape it? How about an italic block?

                                                                                                                                              This would all be excusable if Markdown was a WYSIWYG plain-text format (as per Gruber’s later rationalisation in the CommonMark debate). Then I could mix Markdown, Mediawiki, rST and email syntax freely, because it’s intended for humans to read, and humans tend to be very flexible.

                                                                                                                                              But people do expect to render it to HTML, and then the ambiguities and flexibility become weaknesses, rather than strengths.

                                                                                                                                          2. 2


                                                                                                                                            While I agree about the others, I fairly strongly disagree about AsciiDoc (in asciidoctor dialect). When I converted my blog from md to adoc, the only frequent change was the syntax of links (in adoc, URL goes first). Otherwise, markdown is pretty much valid asciidoc.

                                                                                                                                            Going in the opposite direction would be hard though — adoc has a bunch of stuff inexpressible in markdown.

                                                                                                                                            I am fairly certain in my opinion that, purely as a language, adoc is far superior for authoring html-shaped documents. But it does have some quality of implementation issues. I am hopeful that, after it gets a standard, things in that front would improve.

                                                                                                                                            1. 1

                                                                                                                                              That’s helpful feedback! It’s limped with the others in my head because I had such an unhappy time trying to use it when working with a publisher[1] a few years back; it’s possible the problem was the clumsiness of the tools more than the syntax. I’ll have to give it another look at some point!

                                                                                                                                              [1] on a contract they ultimately dropped after an editor change, alas

                                                                                                                                          3. 4

                                                                                                                                            Agree, I’ve been using it a ton since 2016 and it has served me well. I think it’s very “Huffman coded” by people who have written a lot. In other words, the common constructs are short, and the rare constructs are possible with embedded HTML.

                                                                                                                                            However I have to add that I started with the original (written ~2004) and it had some serious bugs.

                                                                                                                                            Now I’m using the CommonMark reference implementation and it is a lot better.

                                                                                                                                            CommonMark is a Useful, High-Quality Project (2018)

                                                                                                                                            It has additionally standardized markdown with HTML within markdown, which is useful, e.g.

                                                                                                                                            <div class="">
                                                                                                                                            this is *markdown*

                                                                                                                                            I’ve used both ASCIIDoc and reStructuredText and prefer markdown + embedded HTML.

                                                                                                                                            1. 3

                                                                                                                                              I tend to agree, but there’s a very sharp usability cliff in Markdown if you go beyond the core syntax. With GitHub-flavoured Markdown, I can specify the language for a code block, but if I write virtual then there’s no consistent syntax to specify that it’s a C++ code snippet and not something else where the word ‘virtual’ is an identifier and not a keyword. I end up falling back to things like liquid or plain HTML. In contrast, in LaTeX I’d write \cxx{virtual} and define a macro elsewhere.

                                                                                                                                              I wish Markdown had some kind of generic macro definition syntax like this, which I could use to provide inline domain-specific semantic markup that was easier to type (and use) than <cxx>virtual</cxx> and an XSLT to convert it into <code style="cxx">virtual</code> or whatever.

                                                                                                                                              1. 3

                                                                                                                                                I agree. What sometimes makes me a bit sad is that markdown had a feature compared to others that you can write it to make a nice looking text document as well that you might just output on the terminal for example.

                                                                                                                                                It kind of has that nicely formated plain text email style. Also with the alternative syntax for headings.

                                                                                                                                                Yet when looking at READMEs in many projects it is really ugly and hard to read for various reasons.

                                                                                                                                                1. 4

                                                                                                                                                  The biggest contributor there in my experience (and I’m certainly “guilty” here!) is unwrapped lines. That has other upsides in that editing it doesn’t produce horrible diffs when rewrapping, but that in turn highlights how poor most of our code-oriented tools are at working with text. Some people work around the poor diff experience by doing a hard break after every sentence so that diffs are constrained and that makes reading as plain text even worse.

                                                                                                                                                  A place I do wrap carefully while using Markdown is git commit messages, which are basically a perfect use case for the plain text syntax of Markdown.

                                                                                                                                                  1. 1

                                                                                                                                                    I honestly don’t care that much about the diffs? I always wrap at around 88/90 (Python’s black’s default max line length), and diffs be dammed.

                                                                                                                                                    I also pretty much NEVER have auto wrap enabled, specially for code. I’d rather suffer the horizontal scroll than have the editor lie about where the new lines are

                                                                                                                                              2. 4

                                                                                                                                                It’s not just that they’re annoying, computing has largely been about coping with annoyances ever since the Amiga became a vintage computer :-). But in the context of maintaining a support site, which is what the article is about, you also have to deal with keeping up with whatever’s building the static websites, the kind of website that easily sits around for like 10-15 years. The technology that powers many popular static site generators today is… remarkably fluid. Unless you want to write your own static site generator using tools you trust to stay sane, there’s a good chance that you’re signing up for a bunch of tech churn that you really don’t want to deal with for a support site.

                                                                                                                                                Support sites tend to be built by migrating a bunch of old pages in the first two weeks, writing a bunch of new ones for the first two months, and then infrequently editing existing pages and writing maybe two new pages each year for another fifteen years. With most tools today, after those first two or three honeymoon years, you end up spending more time just keeping the stupid thing buildable than actually writing the support pages.

                                                                                                                                                Not that writing HTML is fun, mind you :(.

                                                                                                                                                (Please don’t take this as a “back in my day” lament. A static site generator that lasts 10 years is doable today and really not bad at all – how many tools written in 1992 could you still use in 2002, with good results, not as an exercise in retrocomputing? It’s not really a case of “kids these days ruined it” – it’s just time scales are like that ¯\(ツ)/¯ )

                                                                                                                                                1. 1

                                                                                                                                                  Heh. I was using an editor written in 1981 in 2002! [1] But more seriously, I wrote a static site generator in 2002 that I’m still using (I had to update it once in 2009 due to a language change). On the down side, the 22 year old codebase requires the site to be stored in XML, and uses XSLT (via xsltproc) to convert it to HTML. On the plus side, it generates all the cross-site links automatically.

                                                                                                                                                  [1] Okay, it was to edit text files on MS-DOS/Windows.

                                                                                                                                                2. 2

                                                                                                                                                  I find that writing and edititing XML or HTML isn’t so much of a pain if you use some kind of structural editor. I use tagedit in Emacs along with a few snippets / templates and imo it’s pretty nice once you get used to it.

                                                                                                                                                1. 3

                                                                                                                                                  I’m curious about what the graphics layer in Haiku (and BeOS, since I assume they’re real similar) looks like.

                                                                                                                                                  1. 2

                                                                                                                                                    Do you mean API, or internals?

                                                                                                                                                    You can read the old Be Book ( for a basic overview of the API, which indeed Haiku has all of (and then extended with more controls and drawing modes, some of which are documented in the newer API docs:

                                                                                                                                                    Internals are another matter, we don’t have a lot of documentation for that unfortunately (we try to follow “the code is its own documentation” but of course that doesn’t always work out.) I can try to answer whatever questions you might have about that or point you to the source for more details.

                                                                                                                                                    1. 3

                                                                                                                                                      Do you mean API, or internals?

                                                                                                                                                      Yes. ;)

                                                                                                                                                      Thanks! I was really only after a very rough overview because I know literally nothing about it. Like, does the interface or implementationit resemble Wayland or X11 or Win32 GDI or a raw framebuffer or anything else a reader might be familiar with? :)

                                                                                                                                                      1. 5

                                                                                                                                                        Like, does the interface or implementationit resemble Wayland or X11 or Win32 GDI or a raw framebuffer or anything else a reader might be familiar with?

                                                                                                                                                        The interface, being a standard “widget-based” user interface toolkit written in C++ has some similarities to Qt or maybe gtkmm, but of course it is heavily tied to one platform and thus makes use of OS features (the Be/Haiku IPC ports and messages system, area-based shared memory, etc. etc.)

                                                                                                                                                        It also has some significant differences with other “high level” toolkits in that it’s simultaneously a “high level” toolkit (like Qt or GTK) and a “low level” toolkit (like Xlib or Wayland) at once, because there’s nothing else below it like Wayland or X11 below Qt or GTK even on Linux, so there are options and flags that can be used which most other toolkits do not bother exposing or don’t support at all.

                                                                                                                                                        The other thing is multithreading – some toolkits allow it loosely, some disallow it altogether, but on BeOS and Haiku, multithreading is mandatory; each window gets is own thread, and its own communication ports to the display server. All rendering occurs server-side, which makes it a bit like “classic” X11, but only rendering (control layouts, resizing, etc. are computed on the client side.)

                                                                                                                                                        You can, of course, just request a shared-memory bitmap buffer and copy pixels into that and make a window’s contents be just the bitmap, which ported toolkits like Qt that want to draw things themselves do.

                                                                                                                                                        1. 3

                                                                                                                                                          Thanks, this is awesome to read, and I’m not even the person that asked the question(thanks to the question asker too!).

                                                                                                                                                          1. 2

                                                                                                                                                            Thank you

                                                                                                                                                    1. 1

                                                                                                                                                      I was looking around for state of the art in binary patches for software updates/installs. I found aging references to courgette and bsdiff but I can’t tell if they’re still a thing. Or maybe there is other software that has superseded them. Someone pointed me at this project which looks well kept.

                                                                                                                                                      But I’m curious: what’s the fastest/best out there today and what works on all of win/mac/linux?

                                                                                                                                                      1. 1

                                                                                                                                                        My understanding is that bsdiff isn’t really specific to any one platform? because it creates and applies (concise) patches for diffs between arbitrary files and it’s just that it uses heuristics that often work well for object code.

                                                                                                                                                        There’s a fork of a Go implementation of bsdiff in the wharf project here, vendored in and mentioned at the end of the readme.

                                                                                                                                                        1. 1

                                                                                                                                                          There’s a fork of a Go implementation of bsdiff in the wharf project here, vendored in and mentioned at the end of the readme.

                                                                                                                                                          Yup saw that. Cool project!

                                                                                                                                                          My understanding is that bsdiff isn’t really specific to any one platform?

                                                                                                                                                          I have no clue just wanted to make sure.