1.  

    I did the shift from datacenter to AWS a long time ago, but things were written with simple server type equipment in mind. So, the biggest shocks were simply that it was going to take much larger instances to replicate the hardware we had. It was a sticker shock. After that the work was identifying the biggest wins for adapting what we had to what AWS offered to cut costs. The big wins came early, and over time we were able to add more and push costs down to close to what they had been before. It is still probably more expensive, but it is also much more agile when I need more or less resources in a way that physical hardware would never offer. It was a mature application, and stopping to rewrite would have not made sense for the company with its resources. So, I disagree with the first point in at least some cases.

    I think the case for not running your own kubernetes cluster doesn’t go far enough. I think EKS is for many users just an option given to them to give them the sense that they aren’t giving up control and knowledge. Most people just don’t need the extended configurability and the technical debt is significant. If you seriously consider things, I think ECS actually makes the most sense for the vast majority of situations. It feels like FM, but that is kind of the point. You’re already buying into AWS to some degree, so why not just give in and use the services they offer. It’s my understanding Amazon runs very large ECS services internally, and efficient use of your time makes the most business sense.

    For alerting, you have to make sure your alerts make sense and that you diligently handle them. The whole point is to automate things so your time goes farther, so if something alerts it should be analyzed to see if it can be prevented in the future. If you start ignoring alerts they are either not necessary or you aren’t fixing them or they actually are rare cases. Also, you should pay attention to the problems you do have that aren’t covered by an alert and look at how that might be a missed metric across other parts of your project. Pager burnout is a serious problem that you cannot afford to have. It signals a failure in at least one place in your organization. This is often where operations has to push back on development to fix root causes. If the error message is indecipherable and you haven’t briefed ops on how things work enough to diagnose the issue, I am going to look at the git log for who is responsible and call them. AWS offers a horrible service (the name I forget), which is ostensibly for automating responses and writing playbooks for errors, but I think is probably most used to automate reboots and encourages bad habits for all but the largest systems.

    Python, Ruby, Rust, and Go all suffer from the dependency problems. I can’t deploy NixOS into production, but one thing I love is being able to write a derivation for some random python thing and fence it in with its dependencies. I think you just have to use shell scripting when it is simple, and standardize on a language for actual tools and stick with it. Configuration management should really handle dependencies, and having everyone just use the same language means that everyone is comfortable with the same style and language. Every language essentially sucks in its own special way anyway. Compiled languages for admin/ops tools have always seemed like the wrong way to go due to the additional complexity of rolling them out in traditional linux environments. Also, your tool may end up being replaced by an industry standard one down the line and you should just migrate to that. I have often had my own tool before someone else’s took off and became better at what I was doing.

    1. 6

      Python, Ruby, Rust, and Go all suffer from the dependency problems

      How do Rust and Go suffer from dependency problems? They both ship statically-linked binaries, right?

      1.  

        Security issues, large numbers of dependencies for simple tasks if you aren’t careful.

        1. 10

          I don’t really understand. All languages that support importing third-party packages are subject to dependency concerns, but Rust and Go discourage dependencies to the maximum feasible extent… what language/ecosystem has a better stance in this dimension?

    1. 11

      Don’t write internal cli tools in python

      I think this is shortsighted, bad advice. Binaries are simpler to distribute, this is true. But I would never, ever, ever write infra scripts in Go or Rust. At least not the kind that I end up writing. Simple scripts that perform a few commands, a little bit of argument parsing, a few HTTP requests where concurrency doesn’t matter. Nothing crazy.

      In this case, a scripting language is so much more productive. I currently use Python for this, and it’s just not a problem with pipenv.

      1.  

        Can you define what you mean by “infra scripts”? If your scripts need to be run by anyone other than yourself, then it seems to be pretty well-established that the productivity granted by Python is outweighed by the efficacy of distribution granted by Go.

        1. 6

          Infra scripts meaning scripts that power a CI / CD pipeline, dev tooling for everyday operations like creating hotfixes / finding which environments commits live in, ops scripts like rolling back to previous versions of containers in ECS.

          Probably important background is that I generally work at product companies with less than 100 engineers. These scripts aren’t open source tools that are distributed to random people. These are just internal tools that are automating parts of our workflow for building and shipping code.

          For that use case, I would never in a million years use Go.

          1.  

            Gotcha! If you can rely on your consumers having a minimum version of Python installed on their machines then this seems reasonable. Go certainly isn’t a great choice if you’re just doing bash++ 👍

            1.  

              Yes in this case I can message my “consumers” because they are my coworkers. We also provide a script which keeps everyones dependencies in sync, but all that does is install pipenv in the case of Python. We also use pipenv-shebang so a script can be invoked with no knowledge of python dependencies.

              Yes, distributing a binary is simpler. I would consider that if these were tools that were used by people that I didn’t work with, but our dev environment is already automated so we keep the team in sync that way.

              Actually, the bash++ effect was the main reason to switching to Python. We got relatively far with just bash, but after a while it becomes unmanageable. You can’t compare curl and jq to something like using boto3 to interact with AWS.

            2.  

              Interesting, would love to hear more how you managed to get all that to work. In one of the previous companies we tried to have python in our CI stack, but it was always a hassle with dependencies, different python versions (as different people used different python locally), and generally pipenv being so slow to download and install dependencies. True, we could have done many things better, but we had elastic workers, so many had to be started from scratch and tools installed on the first run of any pipeline (with many different pipelines).

              1.  

                We just use pipenv. Like I said, these are relatively simple scripts. Many times the only dependency is something like click, a CLI library. Right now when a new CI worker is started, it does have to install dependencies like you said, but that’s taking ~20 seconds, not minutes. One thing we’re considering is creating a Docker image with all of the dependencies installed and just using that. But it’s honestly not even noticeable at this point.

        1. 3

          One of my big hopes with generics in 1.18 is JSON encoding / decoding gets faster because we can use concrete types and not use reflection (obviously this will take time as the stdlib will be mostly generics-free for some time).

          The encoding/json performance problems in the go standard library are a major issue, not insurmountable though.

          My suggestion to the author is to try and non-stdlib / optimized JSON unmarshaler.

          1. 3

            That’s not going to happen. The way generics work doesn’t allow for compile time execution (or even specialization, although that will probably happen eventually), so there’s no way serialization will ever work with Go generics. For the foreseeable future, if you want to use a concrete type when serializing, it will need to be go:generated.

            1. 3

              Yes, it’s likely that JSON unmarshaling is a performance bottleneck here. For “big data” JSON filtering, this is a real concern. I saw https://github.com/bytedance/sonic just the other day, and I know there are other performance-focused JSON libs.

              That said, I think it’s a stretch to say the “the encoding/json performance problems in the go standard library are a major issue”. For many common use cases like web applications it just doesn’t matter much. I’ve used Go’s encoding/json extensively for small servers as well as reasonably high-throughput servers (10s or 100s of requests per second, not thousands or millions) and it works fine – database query performance was usually the bottleneck.

              1. 2

                The share of issue depends on how much your program is doing in each area; if your program is just processing json, and the json library is slow, it’s an issue (as in, this case).

                The other piece of context here is how the performance is “relatively” to other runtimes; here is where the standard library implementation suffers the most, as compared to other common runtimes it’s significantly slower.

                1. 2

                  What other common runtimes are significantly faster than Go, and in which dimensions? As far as I’m aware, Go’s runtime, which is optimized for latency, is as fast or faster than anything on the market, but I’d be happy to see data suggesting otherwise!

                  1. 2

                    Sorry should have been clearer, *runtime standard library json packages. The terminology (“runtime”) there was used poorly.

                    1. 1

                      Gotcha! Thanks.

            1. 10

              It doesn’t recognise that software breaks are common and not all equal.

              I disagree with both parts.

              Not all breakage is equal in the sense that library developers should be even more reluctant to introduce a change that breaks say 50% of existing projects than a change that only break 0.5%.

              However, for downstream users, all breaking changes are equal. If it’s my code that breaks after updating from libfoo 1.0.0 to 2.0.0, I don’t care how many other projects have the same problem since it doesn’t reduce the need to adapt my code.

              If you define breaking change severity as the number of affected API functions, the picture doesn’t really change. A small number of affected functions doesn’t always mean a simple fix.

              I agree that software breakage is common, but the real solution is to stop making breaking changes without a really good reason, not to invent versioning schemes that make breaking changes less visible in the major version number.

              1. 8

                As a consumer of many libraries in dynamically typed languages “this easily grappable thing has a different call signature, same results” is qualitatively different from “this API disappeared/is now semantically entirely different”.

                Sure don’t break things without good reasons. But that’s true in any respect!

                1. 5

                  However, for downstream users, all breaking changes are equal.

                  Disagree on that. If a breaking change fixes a typo in a function name, renames something, or even just deprecates something with a clear and easy replacement path, I don’t care as a user. It’s just grooming.

                  The distinction between major and minor breaking changes fits my expectations both as a user and as a developer.

                  1. 3

                    However, for downstream users, all breaking changes are equal.

                    A breaking change that doesn’t break my code is facially less bad than one that does.

                    This emerging notion that breaking changes are an Ur-failure of package authors which must be avoided at all costs biases so strongly toward consumers that it’s actually harmful for software development in the large. Software needs to have breaking changes over time, to evolve, in order to be healthy. Nobody gets it right the first time, and setting that as the baseline expectation is unrealistic and makes everyone’s experience net worse.

                    1. 4

                      That notion is biased towards the ecosystem as a whole. We all are both producers and consumers.

                      If there are two libraries with comparable functionality and one makes breaking changes often while the other doesn’t, it’s the latter that brings more benefit to the ecosystem by saving developer time. In essence, compatibility is a feature.

                      I’m not saying that people should never make breaking changes, only that it requires a really good justification. Fixing unfortunate function names, for example, doesn’t require a breaking change—keeping the old name as an alias and marking it deprecated is all that’s needed, and it adds too few bytes to the library to consider that “bloat”.

                      1. 2

                        I’m not saying that people should never make breaking changes, only that it requires a really good justification. Fixing unfortunate function names, for example, doesn’t require a breaking change—keeping the old name as an alias and marking it deprecated is all that’s needed, and it adds too few bytes to the library to consider that “bloat”.

                        The amount of justification required to make a breaking change isn’t constant, it’s a function of many project-specific variables. It can easily be that the cost of keeping that deprecated function around — not just in terms of size but also in API coherence, future maintainability, etc. — outweighs the benefit of avoiding a breaking change.

                        1. 1

                          A lot of time the “maintaining API coherence” argument is a euphemism for making it look as if the design mistake was never made. Except it was, and now it’s my responsibility as the maintainer to ensure minimal impact on the people who chose to trust my code.

                          I completely agree that those arguments can be valid, but breaking changes I see in the wild tend to be of the avoidable variety where a small bit of effort on the maintainer’s side was all that needed to fix it for future users without affecting existing ones.

                          Breaking changes unannounced and perhaps even unnoticed by the authors are even worse.

                          1. 2

                            design mistake

                            I just can’t get on board with this framing. “Design mistakes” are an essential and intractable part of healthy software projects. In fact they’re not mistakes at all, they’re simply stages in the evolution of an API. Our tools and practices have to reflect this truth.

                            1. 2

                              Both in this thread and every other time I’ve seen you talk about this topic on this website you’re always on the rather extreme side of “it’s impossible not to have completely unstable software”.

                              There are many examples of very stable software out there, and there’s plenty of people who are very appreciative of very stable software. There are very stable software distributions out there too (debian) and there are plenty of people who are very appreciative of the fact that if they leave auto-updates on for 4 years and leave things alone that there’s a very small chance that they will ever have to fix any breakages, and when they eventually have to update, that things will also be very well documented and they likely won’t have to deal with a subtle and unnoticed breakage.

                              Yes, it is a reality that new and bleeding edge concepts need to go through a maturation stage where they experiment with ideas before they can be solidified into a design, but pretending like this is true for every software project in existence and that therefore instability must just be accepted is just flat out wrong.

                              We have had pretty decent stability for quite some time and plenty of people have been very happy with the tradeoff between stability and the bleeding edge.

                              What I’m saying here is really that nobody is asking you to take your extremely immature project for which you don’t have the experience or foresight (yet) to stabilise and stabilise it prematurely. Of course there will always be projects like that and like anything they’re going to be bleeding-edge avant-garde ordeals which may become extremely successful and useful projects that everyone loves or end up in a dumpster. What I am saying here is that it’s okay to have an unstable project, but it’s not okay to mislead people about its stability and it is not okay to pretend like nobody should ever expect stability from any project. If your project makes no effort to stop breaking changes and is nowhere near being at a point where the design mistakes have been figured out then it should make that clear in its documentation. If your project looks like it uses a 3 part versioning scheme then it should make it clear that it’s not semver. And most importantly, just because your project can’t stabilise yet doesn’t mean that nobody’s project can stabilise.

                              1. 1

                                there are plenty of people who are very appreciative of the fact that if they leave auto-updates on for 4 years and leave things alone that there’s a very small chance that they will ever have to fix any breakages

                                Let me distinguish software delivered to system end users via package managers like apt, from software delivered to programmers via language tooling like cargo. I’m not concerned with the former, I’m only speaking about the latter.

                                With that said…

                                you’re always on the rather extreme side of “it’s impossible not to have completely unstable software”

                                I don’t think that’s a fair summary of my position. I completely agree that it’s possible to have stable software.

                                Let’s first define stable. Assuming the authors follow semver, I guess the definition you’re advancing is that it [almost] never increments the major version, because all changes over time don’t break API compatibility. (If that’s not your working definition, please correct me!)

                                I’m going to make some claims now which I hope are noncontroversial. First, software that exhibits this property of stability is net beneficial to existing consumers of that software, because they can continue to use it, and automatically upgrade it, without fear of breakage in their own applications. Second, it is a net cost to the authors of that software, because maintaining API compatibility is, generally, more work than making breaking changes when the need arises. Third, it is, over time, a net cost to new consumers of that software, because avoiding breaking changes necessarily produces an API surface area which is less coherent as a unit whole than otherwise. Fourth, that existing consumers, potential/new consumers, and authors/maintainers are each stakeholders in a software project, and further that their relative needs are not always the same but can change depending on the scope and usage and reach of the project.

                                If you buy these claims then I hope it is not much of a leap to get to the notion that the cost of a breaking change is not constant. And, further, that it is at least possible that a breaking change delivers, net, more benefit than cost, versus avoiding that change to maintain API compatibility. One common way this can be true is if the software is not consumed by very many people. Another common way is if the consumers of that software don’t have the expectation that they should be able to make blind upgrades, especially across major versions, without concomitant code changes on their side.

                                If you buy that, then what we’re down to is figuring out where the line is. And my claim is that the overwhelming majority of software which is actively worked-on by human beings is in this category where breaking changes are not a big deal.

                                First and foremost, because the overwhelming majority of software is private, written in market-driven organizations, and must respond to business requirements which are always changing by their very nature. If you bind the authors of that software to stability guarantees as we’ve defined them, you make it unreasonably difficult for them to respond to the needs of their business stakeholders. It’s non-viable. And it’s unnecessary! Software in this category rarely has a high consumer-to-producer ratio. The cost of a breaking change is facially less than, say, the AWS SDK.

                                By my experience this describes something like 80-90% of software produced and maintained in the world. The GNU greps and the AWS SDKs and those class of widely-consumed things are a superminority of software overall. Important! But non-representative. You can’t define protocols and practices and ecosystem expectations for general-purpose programming languages using them as exemplars.

                                Second, because consumers shouldn’t have the expectation that they can make blind upgrades across major versions without code changes on their side. It implies a responsibility that authors have over their consumers which is at best difficult in closed software ecosystems like I described above, and actually literally impossible in open software ecosystems like the OSS space. As an OSS author I simply don’t have any way to know how many people are using my software, it’s potentially infinite; and I simply can’t own any of the risk they incur by using it, it doesn’t scale. I should of course make good faith effort toward making their lives easier, but that can’t be a mandate, or even an expectation, of the ecosystem or it’s tooling.

                                And, thirdly, because stability as we’ve defined it is simply an unreasonable standard for anything produced by humans, especially humans who aren’t being supported by an organization and remunerated for their efforts. Let me give you an example. Let’s say I come up with a flag parsing library that provides real value versus the status quo, by cleanly supporting flag input from multiple sources: commandline flags, environment variables, and config files. Upon initial release I support a single config file format, and allow users to specify it with an option called ConfigParser. It’s a hit! It delivers real value to the community. Shortly, a PR comes in to support JSON and YAML parsers. This expands the cardinality of my set of parsers from 1 to N, which means that ConfigParser is now insufficiently precise to describe what it’s controlling. If I had included support for multiple config files from the start, I would have qualified the options names: PlainConfigParser, JSONConfigParser, YAMLConfigParser, and so on. But now I’m faced with the question: do I avoid a breaking change, leave ConfigParser as it is, and just add JSONConfigParser and YAMLConfigParser in addition? Or do I rename ConfigParser to PlainConfigParser when I add the other two, in order to keep the options symmetric?

                                My initial decision to call the option ConfigParser was not a design mistake. The software and its capabilities evolved over time, and it’s not reasonable to expect me to predict all possible future capabilities on day one. And the non-breaking change option is not necessarily the right one! If I have 100 users now, but will have 100,000 users in a year’s time, then breaking my current users — while bumping the major version, to be clear! — is the better choice, in order to provide a more coherent API to everyone in the future.

                                Some people would say that this class of software should stay at major version 0 until it reaches a point of stability. This, too, isn’t reasonable. First, because stability is undefinable, and any judgment I make will be subjective and wrong in someone’s eyes. Second, because it strips me of the power to signal breaking vs. non-breaking changes to my users. Semver stipulates the semantics of each of its integers, but nothing beyond that — it doesn’t stipulate a mandatory rate of change, or anything like that. It leaves stability to be defined by the authors of the software. To be clear, more stable software is easier to consume and should be preferred when possible. But it can’t be the be-all end-all metric that software must optimize for.

                                1. 2

                                  So let’s get some things out of the way.

                                  I don’t care about private spaces, and I don’t care about what people do in them.

                                  I also don’t care about de jure enforcement of stablity through limitations or tooling or whatever. My interest is specifically in de facto stability expectations.

                                  For ease of my writing, let’s define the term interface to mean something which you could reasonably write a standard to describe. This means things like protocols, APIs or programming languages. Let’s define producer to be someone who produces an interface. Let’s define consumer to be someone who uses an interface.

                                  I am of the opinion that in the programming world (encompassing both consumers and producers of interfaces) the following expectations should be de jure:

                                  • If you are the author of an interface which you publish for users, it is your responsibility to be honest about your stability promises with regards to that interface.

                                  • People should expect stability from commonly used interfaces.

                                  Just like people enjoy the benefits of the stability of debian, there are plenty of people who enjoy the benefits of the stability of a language like C. C programs I wrote 5 years ago still compile and run today, C programs I will write today will still compile and run in 5 years. I obviously have to follow the standard correctly and ensure my program is written correctly, but if for some reason my program cannot be compiled and cannot be ran in 5 years, I have nobody to blame but myself. This is an incredibly useful quality.

                                  The fact that I can’t expect the same of rust completely discounts rust as a viable choice of software for me.

                                  There is no reason why I should have to be excluded from new features of an interface just to avoid breaking changes.

                                  Moreover, in the modern world where security of software is an ever-growing concern, telling consumers to chose between interface stability and security is an awful idea.

                                  Now with that out of the way, let’s go over your points (I have summarised them, if you think my summary is wrong then we may be talking past each other, feel free to try to address my misunderstanding).

                                  Interface stability is beneficial for existing consumers.

                                  Yes.

                                  Interface stability is a cost to interface producers.

                                  Yes.

                                  Interface stability is a cost for new consumers.

                                  No.

                                  Keeping your interface stable does not prevent you from expanding it in a non-breaking way to improve it. It does not prevent you from then changing the documentation of the interface (or expanding on its standard) to guide new consumers towards taking the improved view of the interface. It does not prevent you from stopping further non-essential development on the old portions of the interface. It does not stop you from creating a new major version of the interface with only the new parts of the interface and only maintaining the old interface insofar as it requires essential fixes.

                                  Interface consumers and producers are both interface stakeholders but have different needs.

                                  Sure.

                                  The cost of a breaking change is not constant.

                                  Sure.

                                  It is possible that a breaking change delivers more benefit than cost versus avoiding the breaking change.

                                  I think this is disputable because of what I said earlier about being able to improve an interface without breaking it.

                                  But I can see this being the case when a critical security makes consumers of your interface vulnerable through normal use of the interface. This is a rare corner case though and doesn’t really make the whole idea or goal of interface stability any less worthwhile.

                                  This can happen your interface is not used by many consumers.

                                  Yes, you obviously should feel less obliged to keep things stable in this case, but you should still be obliged to make consumers well aware of just how stable your interface is.

                                  This can happen if consumers do not have high stability expectations.

                                  Which you can ensure by making them well aware of what stability expectations they should have.

                                  But I would argue that in general, consumers should be allowed and encouraged to have high stability expectations.

                                  especially across major versions, without concomitant code changes on their side

                                  I don’t see how this is relevant given that you already defined stability a certain way.

                                  Of course if your interface follows semver and you increment the major then you’ve made your stability expectations clear and if the consumers do not understand this then they can only blame themselves.

                                  The majority of interfaces belong in a category which excludes them from high stability expectations.

                                  Yes, as interfaces follow a pareto distribution, very few of them have a significant number of consumers where breaking changes will have a significant impact.

                                  The majority of interfaces are private

                                  Actually this is irrelevant, you can just categorise this as “tiny userbase”.

                                  That being said, there’s two sides to this coin.

                                  Google mail’s internal APIs may have a “tiny userbase” (although google is a big company so maybe not so), on the other hand, google mail’s user facing interface has a massive userbase.

                                  If you force producers of those interfaces to guarantee stability as defined above then you make it unreasonably difficult to respond to the needs of their business stakeholders.

                                  Nobody is forcing or suggesting forcing anyone to do anything.

                                  You can’t use interfaces with enormous amounts of consumers as baselines for how people should handle interface stability of their interfaces with fewer users.

                                  I’d say this is irrelevant.

                                  Your hello world script doesn’t need interface stability. That is one extreme. The C standard needs extreme stability, that is the other extreme. If you consider the majority of the things I am actually thinking of when we talk about “things being horribly unstable” then they definitely don’t fall 1% point away from hello world. I talk about things like rust, nim, major libraries within those languages, etc.

                                  Those things have significant numbers of cosumers yet their interfaces seem to come with the same stability expectations as the hello world program.

                                  Yes, stability is a scale, it’s not binary, but the current situation is that a lot of people are treating it as completely optional in situations where it’s certainly shouldn’t be. I don’t think it’s an unreasonable expectation to expect more stability from some of these projects. It is also not an unreasonable expectation to

                                  Because consumers shouldn’t high interface stability expectations across major versions this implies producers have a responsibility which is at best difficult in closed software ecosystems described above.

                                  I’m not actually sure how to interpret this but it seems to talk about things irrelevant to the point. Feel free to clarify.

                                  As the producer of an open source interface I don’t have any way to know how many consumers I have and therefore I cannot own any risk incurred by the consumers consuming my interface.

                                  Nobody is asking anyone to own any risk (especially in the open source ABSOLUTELY NO WARRANTY case). I am simply saying that if your project is popular and you break stability expectations when you have made it clear that people should have them, you should expect people to distrust your interface, stop using it and potentially distrust future interfaces you produce. I think this is only fair, and really the most I can ask of people.

                                  Moreover, you do have many ways of estimating how many people are using your software. If enough people are using your software for interface stability to matter then you will know about it (or you’re some expert hermit who manages to successfully maintain key project while never knowing anything about the outside world).

                                  Stability as defined above is an unreasonable and unreachable standard for human producers of interfaces.

                                  No, I don’t think it is, or rather, I thing you have produced a false dichotomy. Correct me if I’m wrong but you seem to think that because perfect stability is impossible, imperfect stability is no longer worthwhile. I think this is just a poor way of looking at things, imperfect stability is achievable to a high level and is worthwhile in many cases.

                                  … especially humans who aren’t being supported by an organization and remunerated for their efforts.

                                  Let’s put it this way, just because you decided to volunteer to do something doesn’t mean you have no responsibility. You take on responsibility by making the risk of creating something which people might come to rely on and use. If you do not wish to take on this responsibility, then it is at the very least your responsibility to make THAT clear.

                                  I would say it is childish to suggest that just because you are doing something for free that therefore you have no responsibility for the consequences of what happens when people come to rely on that free activity. There are a million and one real world examples of how this does not pan out.

                                  Let me give you an example. Let’s say I come up with a flag parsing library that provides real value versus the status quo, by cleanly supporting flag input from multiple sources: commandline flags, environment variables, and config files. Upon initial release I support a single config file format, and allow users to specify it with an option called ConfigParser. It’s a hit! It delivers real value to the community. Shortly, a PR comes in to support JSON and YAML parsers. This expands the cardinality of my set of parsers from 1 to N, which means that ConfigParser is now insufficiently precise to describe what it’s controlling. If I had included support for multiple config files from the start, I would have qualified the options names: PlainConfigParser, JSONConfigParser, YAMLConfigParser, and so on. But now I’m faced with the question: do I avoid a breaking change, leave ConfigParser as it is, and just add JSONConfigParser and YAMLConfigParser in addition? Or do I rename ConfigParser to PlainConfigParser when I add the other two, in order to keep the options symmetric?

                                  I would say that it’s actually worthwhile to avoid breaking API here. If your bar for breaking API is this low then you’re going to be breaking API every day of the week. Moreover, you won’t learn anything about API design from your project.

                                  There is real value in living with your mistakes for a while and letting things like this accumulate for a while. Simply accepting every breaking change as it comes is going to teach you less about which parts of your API are actually good or not than waiting for a good point to create config_parser_of_the_snake_case_variety a few years down the road with all the things you’ve learned incorporated. The end result will be a far better design than whatever you could cobble together with a weekly breaking change.

                                  You’re also ignoring the possibility of just making a new API (also, surely the correct option is not a name-per-format but rather an option) and wiring the old API to just forward to the new one. You can begin to think about future proofing your API while keeping the old one around (to everyone’s great benefit).

                                  Finally, if you’ve just written a library, why is it 1.0.0 already? Why would you do that to yourself? Spend a few months at least with it in a pre-1.0.0 state. Stabilize it once you iron out all the issues.

                                  My initial decision to call the option ConfigParser was not a design mistake.

                                  Of course not, the mistake was pretending that JSON and YAML are config formats. (This is a joke.)

                                  The software and its capabilities evolved over time, and it’s not reasonable to expect me to predict all possible future capabilities on day one.

                                  Of course not. But I think you’re being unreasonably narrow-minded with regards to possible options for how to evolve a project while keeping the interface stable.

                                  And the non-breaking change option is not necessarily the right one! If I have 100 users now, but will have 100,000 users in a year’s time, then breaking my current users — while bumping the major version, to be clear! — is the better choice, in order to provide a more coherent API to everyone in the future.

                                  It’s also extremely unlikely that you will be able to predict the requirements of your library one year in the future. Which is possibly why your library shouldn’t be a library yet and should wait a few years of usecases through before it solidifies into one.

                                  There’s too many tiny libraries which do half a thing poorly out there, at least part of this stability discussion should be around whether some interfaces should exist to begin with.

                                  Some say this class of interface should stay at major version 0.

                                  Yes, I think this is synonymous with my position that it shouldn’t be a library to begin with.

                                  This is unreasonable because stability is undefinable.

                                  No, you already defined it a couple of times in a way I would agree with. A stable interface is an interface where a consumer should not expect things to break if they update the software to the latest version (of that particular major version).

                                  Any stability judgement I make will be questioned by someone.

                                  Why leave the house then? You might die. This comes back again to the all-or-nothing mentality I mentioned earlier where you seem to suggest that just because perfect stability is impossible, imperfect stability is not worthwhile.

                                  Second, because it strips me of the power to signal breaking vs. non-breaking changes to my users.

                                  It doesn’t do that. You have literally all the tools at your disposal including your changelog and documentation. Moreover, if your project is at 0.x.y then you are completely free to tell people that if x increases then the interface will have changed significantly and if y increases then the interface probably hasn’t changed. Semver does not specify how x and y are to be interpreted for 0.x.y, only for M.x.y where M > 0.

                                  Semver explains what the numbers mean but nothing else. It does not specify the rate of change. It leaves stability to be defined by the authors of the software.

                                  Yes, these are true statements.

                                  Stability should not be the the most important goal for all software.

                                  Again, a true statement. My point is and always has been that the issue is not that all interfaces aren’t perfectly stable, it’s that there’s a lot of interfaces which are far less stable than they reasonably should be and that normalising this means that software is going to get a lot more confusing, a lot more insecure and a lot more broken as time goes on.

                                  Overall I think we’re in agreement on most parts except for how easy stability and how important it is relatively speaking.

                                  1. 1

                                    There is real value in living with your mistakes for a while and letting things like [ConfigParserX] accumulate for a while. Simply accepting every breaking change as it comes is going to teach you less about which parts of your API are actually good or not than waiting for a good point to create config_parser_of_the_snake_case_variety a few years down the road with all the things you’ve learned incorporated. The end result will be a far better design than whatever you could cobble together with a weekly breaking change.

                                    I don’t know how better to say this: the decisions in this example are not mistakes. They are made by a human being at a particular point in the evolution of a software project.

                                    Finally, if you’ve just written a library, why is it 1.0.0 already? Why would you do that to yourself? Spend a few months at least with it in a pre-1.0.0 state. Stabilize it once you iron out all the issues.

                                    1.x.y in the semver nomenclature represents a notion of stability which I as the author get to define. It isn’t a universal notion of stability, it’s nothing more than what I define it to be. There is no single or objective denomination of “when I iron out all of the issues” which I can meet in order to qualify a 1.x.y version identifier. It’s all subjective!

                                    just because you decided to volunteer to do something doesn’t mean you have no responsibility. You take on responsibility by making the risk of creating something which people might come to rely on and use. If you do not wish to take on this responsibility, then it is at the very least your responsibility to make THAT clear.

                                    Semver provides all of the mechanics I need to signal what you demand. If I make a breaking change, I increment the major version number. That’s it! There’s nothing more to it.

                                    1. 2

                                      I don’t know how better to say this: the decisions in this example are not mistakes. They are made by a human being at a particular point in the evolution of a software project.

                                      If it was reasonable to assume that the API would need that feature eventually then yes it’s a mistake. If it was not reasonable to assume that the API would need that feature (I would err on this side since neither YAML nor JSON are actually meant for configuration) then there’s two possibilities: a, that the design change is unnecessary and belongs in an unrelated project or b, that the design change should be made with no need for breaking the existing interface.

                                      1.x.y in the semver nomenclature represents a notion of stability which I as the author get to define. It isn’t a universal notion of stability, it’s nothing more than what I define it to be. There is no single or objective denomination of “when I iron out all of the issues” which I can meet in order to qualify a 1.x.y version identifier. It’s all subjective!

                                      The fact that it’s subjective and you can’t with perfect accuracy determine when it’s appropriate to make a 1.x.y release does not mean that lots of projects haven’t already made good-enough guesses on when this is appropriate and does not mean that a good enough guess is not an adequate substitute for perfect. Once again, you seem to suggest that just because it’s impossible to do something perfectly that it’s not worth doing it at all.

                                      Semver provides all of the mechanics I need to signal what you demand. If I make a breaking change, I increment the major version number. That’s it! There’s nothing more to it.

                                      Yeah, so do that. I don’t get where the disagreement lies here.

                                      *As a completely unrelated side note, the fact that firefox binds ^W to close tab is a hideous design choice.

                                      1. 1

                                        you seem to suggest that just because it’s impossible to do something perfectly that it’s not worth doing it at all.

                                        I am 100% for releasing 1.x.y versions of software projects. What I’m saying is merely that a subsequent release of 2.z.a, and afterwards 3.b.c, and then 4.d.e, is normal and good and not indicative of failure in any sense.

                                        If it was reasonable to assume that the API would need that feature eventually then yes it’s a mistake.

                                        It is incoherent to claim that this represents a mistake. If you can’t agree to that then there’s no possibility of progress in this conversation; if this is a mistake then you’re mandating perfect omniscient prescience in API design, which is plainly impossible.

                                        1. 1

                                          I am 100% for releasing 1.x.y versions of software projects. What I’m saying is merely that a subsequent release of 2.z.a, and afterwards 3.b.c, and then 4.d.e, is normal and good and not indicative of failure in any sense.

                                          I would say that any interface which has need to change that often is too vaguely defined to be a real interface. Fundamentally rooted in the idea of an interface is the idea of being able to re-use it in multiple places, and the fundamental use in re-using something in multiple places is if adjustments to that thing benefit all users (otherwise why not just copy paste the code). As such, if your interface changes constantly, it does not provide any benefit as an interface and should not be one. Put another way, if every time I try to update your interface I have to change my code, what benefit is there at all in me even calling it a proper interface and not just maintaining my own version (either by periodically merging in new changes to the upstream version or just cherry-picking new features I like)?

                                          It is incoherent to claim that this represents a mistake.

                                          How so? Are you suggesting it is impossible to have design oversights?

                                          If you can’t agree to that then there’s no possibility of progress in this conversation; if this is a mistake then you’re mandating perfect omniscient prescience in API design, which is plainly impossible.

                                          Just because foresight is impossible, doesn’t mean you’ve not made a mistake. If foresight is impossible, maybe the mistake is attempting to foresee in the first place? In either case, I don’t understand why you think my stance on whether design mistakes are a real thing or not matters in a discussion about whether it’s appropriate for interfaces to be extremely unstable.

                                          Moreover, given the vagueness of your example config parser case, I never actually said if the scenario you presented constituted a design mistake, I merely outlined a set of possibilities, one of which was that there was a design mistake.

                                          So far (when it comes to your config parser example), I think that you’re being quite narrow-minded in terms of possible approaches to non-breaking interface changes which would avoid the problem entirely without sacrifices. I also think that the example you provided is extremely contrived and the initial design sounds like a bad idea to begin with (although once again, I don’t think this is the reason why you think there needs to be a breaking change, I think you think there needs to be a breaking change solely because I don’t think you have explored all the possibilities of how a non-breaking change may be implemented, but the example lacks a lot of detail so it’s difficult to understand your reasoning).

                                          Maybe if you picked a more realistic design which didn’t have obvious problems to begin with and demonstrated a sensible design change you think would necessitate a breaking change (to avoid compromising on API quality) we could discuss how I would solve that and you can point out the disadvantages, according to you, of my approach to solving the breakage?

                                          1. 1

                                            if every time I try to update your interface I have to change my code, what benefit is there at all in me even calling it a proper interface

                                            The benefit that my library provides to you as a consumer is mostly about the capabilities it offers for you, and only a little bit about the stability of its API over time. Updating the version of my library which you use is also an opt-in decision that you as a consumer make for yourself.

                                            I also think that the example you provided is extremely contrived

                                            It is adapted directly from a decision I had to make in my actual flag parsing library and chosen because I feel it is particularly exemplary of the kinds of design evolution I experience every day as a software author.

                                            you can point out the disadvantages, according to you, of my approach to solving the breakage?

                                            I don’t think we need an example. My position is that as long as you abide semver and bump the major version there is almost no net disadvantage to breaking changes for the vast majority of software produced today. And that avoiding breaking changes is a cost to coherence that rarely makes sense, except for a superminority of software modules. I have explained this as well as I think I can in a previous comment so if that’s not convincing then I think we can agree to disagree.

                                            1. 1

                                              The benefit that my library provides to you as a consumer is mostly about the capabilities it offers for you, and only a little bit about the stability of its API over time. Updating the version of my library which you use is also an opt-in decision that you as a consumer make for yourself.

                                              You can call it a library or an interface all day long, I dispute you calling it that more than I dispute its usefulness. It might function and be useful but it does not function as an interface. It is really not an interface if it does not end up shared among components, if it is not shared among components then there is obviously no real problem (at least in terms of stability, since there’s an enormous number of problems in terms of general code complexity, over-abstraction, security issues and auditability issues).

                                              It is adapted directly from a decision I had to make in my actual flag parsing library and chosen because I feel it is particularly exemplary of the kinds of design evolution I experience every day as a software author.

                                              Can you point me to the breaking change?

                                              My position is that as long as you abide semver and bump the major version there is almost no net disadvantage to breaking changes for the vast majority of software produced today.

                                              And like I already explained, talking about “most software” is pointless since almost nobody uses it.

                                              The discussion is about the small fraction of software which people actually use, where there is a clear benefit to avoiding random breaking changes.

                                              And that avoiding breaking changes is a cost to coherence that rarely makes sense, except for a superminority of software modules.

                                              A superminority which anything worth talking about is already part of.

                                              I have explained this as well as I think I can in a previous comment so if that’s not convincing then I think we can agree to disagree.

                                              There is nothing left to explain, it is not an issue of explanation, you are simply making a bunch of assertions including the assertion that, given an interface which people actually use, most normal design changes which change the interface are better* than design changes which preserve the interface. You have yet to provide any evidence of this (although hopefully if you send me the commit for your project we will finally have something concrete to discuss).

                                              *better being defined as “the disadvantages of breaking the interface do not outweigh the disadvantages of lost clarity by expanding keeping the interface”

                                              1. 1

                                                I don’t care about “interfaces” — that’s terminology which you introduced. We’re talking about software modules, or packages, or libraries, with versioned APIs that are subject to the rules of semver.

                                                And like I already explained, talking about “most software” is pointless since almost nobody uses it . . . The discussion is about the small fraction of software which people actually use

                                                The discussion I’m having is about what software authors should be doing. So I’m concerned with the body of software produced irrespective of how many consumers a given piece of software has. If you’re only concerned with software that has enormously more consumers than producers than we are having entirely different discussions. I acknowledge this body of software exists but I call it almost irrelevant in the context of what programming language ecosystems (and their tools) should be concerned with.

                                                1. 1

                                                  I don’t care about “interfaces” — that’s terminology which you introduced. We’re talking about software modules, or packages, or libraries, with versioned APIs that are subject to the rules of semver.

                                                  If you had an issue with my definition of interface then you should have raised it a bit earlier. That being said, I don’t think that you have a problem with the definition (since it seems completely compatible) but maybe the issue lies somewhere else.

                                                  The discussion I’m having is about what software authors should be doing. So I’m concerned with the body of software produced irrespective of how many consumers a given piece of software has. If you’re only concerned with software that has enormously more consumers than producers than we are having entirely different discussions. I acknowledge this body of software exists but I call it almost irrelevant in the context of what programming language ecosystems (and their tools) should be concerned with.

                                                  Programming ecosystems are interfaces with large numbers of users, their package handling tools exist to solely serve the packages which have a significant number of users. Making package handling tools work for the insignificant packages is a complete waste of time and benefits basically nobody.

                                                  Your package, like it or not, belongs in the group of packages I’m talking about.

                                                  Discussions about semver and interfaces and stability and versioning are obviously going to be completely irrelevant if nobody or almost nobody uses your package. More importantly, it’s very unproductive to use this majority of unused or barely used packages to inform design decisions of software tooling or to inform people how they should go about maintaining packages people actually use.

                                                  1. 1

                                                    [ecosystem] package handling tools exist to solely serve the packages which have a significant number of users

                                                    No, they exist to serve the needs of the entire ecosystem.

                                                    Your package, like it or not, [is insignificant]

                                                    I guess we’re done here.

                                                    1. 1

                                                      No, they exist to serve the needs of the entire ecosystem.

                                                      Focusing on the needs of wannabe libraries which nobody uses is nonsensical. Ecosystems focus on the needs of packages which people use and the people who use those packages.

                                                      Your package, like it or not, [is insignificant] I guess we’re done here.

                                                      Did you intentionally misrepresent me because you were bored of the discussion? Because I literally said the opposite.

                                                      1. 1

                                                        I apologize if you were saying that my package represents something with a large number of users. I parsed your sentence as the opposite meaning.

                                                        Nevertheless, we’re clearly at an impasse. Your position is that

                                                        package handling tools exist to solely serve the packages which have a significant number of users

                                                        which is basically the antipode of my considered belief, and it doesn’t appear like you’re open to reconsideration. So I’m not sure there’s much point to continuing this already very long thread :)

                                                        edit: I can maybe speculate at the underlying issue here, which is that you seem to be measuring relevant software by the number of consumers it has, whereas I’m measuring relevant software by it’s mere existence. So for you maybe an ecosystem with 1000 packages with 2 consumers each and 2 packages with 1000 consumers each is actually just 2 packages big, so to speak, and those 2 packages’ needs dictate the requirements of the tool; and for me it’s 1002 packages big and weighted accordingly. Does this make sense?

                                                        1. 1

                                                          I get your point, I don’t think there’s any misunderstanding there. I just can’t understand why when it comes to tools which are designed to facilitate interoperation you care about the packages for which … because almost nobody uses them … interoperation is not as important as the packages which lots of people use. I think the other problem is maybe that you’ve got a skewed idea of the relationship between package producers and users. I would say that it’s extremely likely (and I don’t have numbers to back this up but there’s no real reason why this doesn’t follow the Pareto distribution) that 20% of packages constitute 80% of package usage. These tools are designed to facilitate package use above package creation (since far fewer people will be creating packages rather than using them) therefore focusing on the 20% of packages which which constitute 80% of the usage would surely make sense?

                                                          1. 1

                                                            far fewer people will be creating packages rather than using them

                                                            Just to make this explicit, I’m speaking from a context which includes open-source software (maybe 10-20% of all software produced in the world) and closed-source software written and maintained at market-driven organizations (maybe 80% of all software).

                                                            In this context it’s my experience that essentially every programmer is both a consumer and producer of packages. Even if a package has only a single consumer it is still a member of the package ecosystem, and the needs of it’s (singular) author and consumer are relevant! So I don’t agree that far fewer people are creating packages than consuming them. In fact I struggle to summon a single example of someone who consumes but doesn’t produce packages.

                                                            20% of packages constitute 80% of package usage

                                                            I agree that the packages-vs-consumers curve is pretty exponential. I agree that tooling can and should support the needs of those relatively few packages with vastly more consumers than producers. But I don’t think it’s as extreme as you’re describing here. I think the “long tail” of packages consumed by a single-digit number of consumers represents the 80% of overall package consumption, the 80% of the area under the curve.

                                    2. 1

                                      I want to revisit two things here…

                                      I don’t care about private spaces.

                                      That’s fine! If you’re not interested in this category of software personally then that’s totally groovy. But “private spaces” hold the overwhelming majority of software produced and consumed in the world. Package management tooling for a general-purpose programming language which doesn’t prioritize the needs of these users doesn’t serve it’s purpose.

                                      if your project is popular and you break stability expectations

                                      When you say “stability expectations” do you mean a consumer’s expectation that I won’t violate the rules of semver? Or an expectation that I won’t hardly ever increment my major version number? Or something else?

                                      As an example, go-github is currently on major version 41 — is this a problem?

                      2. 2

                        Bollocks. A minor breaking change is “re reversed the order of parameters on this method to match the other methods”. A major breaking change is “switched from callbacks to promises”

                        1. 2

                          A switch from callbacks to promises that comes without a compatibility interface should be reflected in the library name, not just the version. It’s not even the same library anymore if literally every line of code that is using the old version must be rewritten to use it again.

                          1. 1

                            I agree, but we’re in the minority, despite Rich Hickey’s efforts.

                      1. 4

                        Surprised @stapelberg didn’t mention the lack of tiling window manager (he develops i3). That’s what keeps me off of Macs, manual window management drives me nuts.

                        1. 3

                          I use this computer little enough that I’m usually in a browser in fullscreen, and at most a terminal or two side by side. If I wanted to use it for any real work, I’d definitely prefer Linux+i3, but as I wrote in the article, it’ll be a while before Linux will be a reality on this machine…

                          1. 1

                            There are a variety of tiling window managers for macOS, e.g. https://github.com/ianyh/Amethyst

                            1. 2

                              Yeah, I tried Amethyst but it’s a real band aid solution. Its about 70% of the way, but not enough to make it a daily driver.

                              1. 2

                                Not free, but I recently saw some people recommending https://hookshot.app

                                1. 1

                                  This one is really great, thanks for sharing!

                                2. 1

                                  I’ve used Amethyst, SizeUp, Divvy, Spectacle, and now Moom — I’ve found Moom to be the best of the bunch.

                                  1. 1

                                    I’ve been using yabai for a while, I’m probably far from the extremes of Xmonad usage but it works well for my purposes.

                              1. 7

                                As usual, the touchpad (which Apple calls “trackpad”) is great, much better than any touchpad I have ever used on a PC laptop

                                Here we go again…

                                There is nothing special about the hardware (well, Apple is an early adopter of force-sensitive pads, but current Synaptics devices are force-sensitive too). Nothing magic about macOS either. It’s just usually compared against bad software. Windows 10 with any HID-multitouch touchpad feels pretty much the same as macOS. On the unix side, other than ensuring you’re not using anything legacy stuff (xf86-input-synaptics lol), you might need this patch to make scrolling look totally smooth in GTK apps (“might” because it already looks fine if the touchpad’s event rate is high enough, e.g. I never noticed any jitter with a 125Hz touchpad + 60Hz display).

                                1. 29

                                  Honestly, I think you’re wrong. There’s nothing magical, but macOS is just so much more reliable than any trackpad I’ve used, and I’ve used trackpads across a relatively wide range of high-end non-Mac laptops. The mac is, for example, the only laptop line I’ve used which always, with 100% accuracy, detects a two finger click as a right click, regardless of how sloppy I am. All other laptops will sometimes just emit a left click rather than a right click if, say, my fingers aren’t horizontal enough or are too close. Also, Wayland currently has a horrible bug where touchpad scrolling is 1.5x faster than it should’ve been (https://gitlab.gnome.org/GNOME/mutter/-/issues/1731). Scrolling in Linux is also universally much more jittery than in macOS (yes, even with libinput drivers), where the screen will jump around a bit when I hold my fingers still. Macs are also the only systems I’ve used where a one-finger tap is registered instantly; there’s always at least what feels like a few hundred milliseconds of delay in Linux with libinput.

                                  There’s a lot to like about non-Mac hardware. But I still haven’t found any non-Mac laptops with a decent touch pad, even in the Mac’s price range. I don’t know if it’s a hardware thing or a software thing; I’m guessing it’s a combination.

                                  And maybe my experience isn’t representative. Maybe every non-Mac laptop I have ever laid my hands on in my life have just been especially bad. I don’t think so though.

                                  1. 9

                                    a few hundred milliseconds

                                    Simply no way. I’ve used Linux on the most random laptops since 2013 and have never noticed input latency this high and I used to play rhythm games with Synaptics touchpads.

                                    1. 5

                                      I might be exaggerating, I don’t know. All I know is that it doesn’t feel instant, and the number I hear people throw around as the delay that’s required for something to feel “instant” is 100ms, which puts my best guess somewhere above 100ms. Though you do adjust to it, and I didn’t really understand that there’s a delay before I used a Mac trackpad again and it felt unnaturally fast.

                                      Do note that I’m talking about tapping, not clicking. Clicking is always fast in my experience. And again, maybe there are differences in hardware here. But this one really feels like a software thing, and everyone’s using libinput these days.

                                      1. 3

                                        The latency is there, you just got used to it. Linux is pretty bad, unless you’re using the PREEMPT_RT patches.

                                        You can easily measure how bad it is by running cyclictest from rt-test, with SMP enabled and SCHED_FIFO policy. It will set alarms and then measure the difference between the requested time and the time it runs at.

                                        Minimum is irrelevant, average has some limited value, max is the column you want to look at. Unit is µs.

                                        With mainline kernel, you’ll see max cross into ms range in a matter of minutes, if not seconds. Leaving it running (its cpu usage is low) for a day or two, you’ll see max get into tens of milliseconds.

                                        linux-rt patchset does help, but only for scheduler latency. The userspace Linux input stack is another layer of bad.

                                        Latency is something that needs to be a core design target. Like security, it cannot be tacked in.

                                        To put things in perspective, AmigaOS (1985) input.device task ran with priority 20, making it the highest priority task in the system, which is realtime with hard priorities; if a higher priority task becomes runnable, it will run immediately.

                                    2. 2

                                      (“might” because it already looks fine if the touchpad’s event rate is high enough, e.g. I never noticed any jitter with a 125Hz touchpad + 60Hz display)

                                      So it sounds like there ARE hardware differences? I don’t know whether it’s hardware or software, but every time I use a Windows machine (and I’m talking about higher end Dell XPS laptops, mostly) I end up frustrated with the janky touchpad behavior. For example, the scrolling speed is always wonky and the device doesn’t pick up my two-finger scroll gesture until I’ve moved my fingers almost a centimeter.

                                      1. 3

                                        Agreed. I’ve used a lot of PCs with both Windows and Linux, and I’ve tried everything I could find to get the touchpads calibrated optimally and they never came close to a Mac, much to my frustration. That said, even Linux running on a Mac doesn’t have good trackpad options, so I don’t doubt that much of the problem is in software–but that’s not much of a consolation so long as “the right software” doesn’t exist for Linux.

                                        1. 1

                                          Funnily enough the author of the gtk patch was using an external Apple Magic Trackpad 2 which is 90Hz. Basically no one has noticed any jitter in gtk’s naive processing with internal trackpads :)

                                          doesn’t pick up my two-finger scroll gesture

                                          Windows probably has the highest thresholds for scroll recognition, though they shouldn’t be that big. Also, what applications are you testing with? I’ve heard that Windows does some weird delay with legacy apps that only support basic mouse scrolling. Something like modern browsers, which do use touchpad native panning APIs, shouldn’t be affected by that.

                                          1. 2

                                            When I find myself on a Windows device, I’m almost always using Chrome. Now that I’m thinking about it, though, I wonder if part of my frustration with trackpads on Windows is the (lack of) inertial scrolling. I’m accustomed to being able to kind of “flick” the trackpad lightly to scroll down a bit while I’m reading something, etc. Windows seems to stubbornly assume that I’m using a wheel mouse.

                                            1. 2

                                              Even with drivers and software that attempt to make scrolling ‘smooth’, it’s never anything like on MacOS, where it’s not that it’s just ‘smooth’, it’s that you scroll to where you expected. It’s almost like it’s been tuned to perfectly match expectations - and scroll there smoothly and so fast it never feels like it’s lagging behind. Connected to your fingers.

                                              I’d liken it to riding a lightweight road bike with higher end components. You move your body and the bike moves with you. You don’t feel like you’re dragging the bike along.

                                              1. 1

                                                Chrome was developing the proper support for all this like 4 years ago so it’s definitely in the production release by now. Are you actually testing “precision touchpad” (HID-multitouch) devices?

                                                1. 2

                                                  I have no idea what the hardware is, like I said, the machines I’ve used are new and new-ish Dell XPS laptops. My point isn’t that you can’t get a good experience on Windows, it’s that I haven’t gotten a good experience on Windows, even with $2500 laptops (equivalent in price to a Mac).

                                          2. 1

                                            There is nothing special about . . .

                                            I’ve always likened Mac vs. non-Mac trackpads to driving an e.g. BMW 335i vs. a Chevrolet truck. It’s just a more precise and overall nicer user experience.

                                            1. 1

                                              It could also be familiarity. Different vendors and OSs have different acceleration mappings. People who are used to macs use a not Mac and it feels wrong, and they tell everyone it’s bad. People who are used to windows or linear have the same experience. It’s like how my grandma’s cookies are just better than yours’s. I can’t explain it, and I don’t need to, it’s just true, and my whole family thinks so too. If you disagree you must be confused or misguided.

                                              Also, are the pads literally force sensitive? I’ve found putting my palm on it registers as more “force” than pushing hard on it, but my laptop is 7 years old and not apple.

                                              1. 1

                                                putting my palm on it registers as more “force” than pushing hard on it

                                                That means it’s not force sensitive, which makes sense for a 7-year-old not-Apple. On the non-Apple side, as I said, only the very latest Synaptics generation is force sensitive.

                                                1. 1

                                                  Ok that makes sense, cause the one I have was billed as I think “pressure sensitive” or something weaseley like that. I’m not exactly a close follower of new laptop tech, historically I’ve only gotten new ones when the current one breaks. TBF I’m pretty impressed with how long this one has lasted.

                                            1. 25

                                              Go doesn’t need async/await, it has goroutines and channels.

                                              1. 12

                                                +1

                                                c := make(chan int)      // future
                                                go func() { c <- f() }() // async
                                                v := <-c                 // await
                                                
                                                1. 1

                                                  I had a negative knee-jerk reaction when I saw the async/await naming. But if you rename/rejig the api a bit, it seems a lot less bad. See: https://gotipplay.golang.org/p/IoHS5HME1bm w/context https://gotipplay.golang.org/p/Uwmn1uq5vdU

                                                2. 2

                                                  this uses goroutines and channels under the hood …

                                                  1. 8

                                                    why do you need to abstract-away goroutines and channels?

                                                    1. 4

                                                      I have no idea. It’s like seeing someone recite a phone book from memory: I can appreciate the enthusiasm without understanding the why

                                                  2. 1

                                                    They said Go didn’t need generics either :)

                                                    I get your point though. Hence why almost every bit of this repo screams “experimental.” I have been just playing around with the pattern in some work/personal projects and seeing how it works ergonomically and seeing if it improves areas with lots of asynchronous operations.

                                                    But, only a matter of time until more folks begin trying to abstract away the “nitty-gritty” of goroutines/channels with generics. I personally point goroutines/channels out as Go’s greatest features, but I have seen others really want to abstract them away.

                                                    1. 4

                                                      Goroutines and channels are there to abstract away asynchronous code.

                                                      1. 5

                                                        Goroutines and channels are abstractions that are a marked improvement on the state of the art prior to Go, but I find that they tend to be too low-level for many of the problems that programmers are using them to solve. Structured concurrency (or something like it) and patterns like errgroup seem to be what folks actually need,

                                                        1. 5

                                                          Yeah, I also long time ago thought, that one area where generics in Go could hopefully help, would be in abstracting away channel patterns - things like fan-out, fan-in, debouncing, etc.

                                                          1. 2

                                                            honestly I just want to be able to call select on N channels where N is not known at compile time. A cool thing about promises is being able to create collections of promises. You can’t meaningfully create collections of channels. I mean sure, you can make a slice of channels, but you can’t call select on a slice of channels. select on a slice of channels is probably not the answer but is a hint at the right direction . Maybe all := join(c, c2) where all three of those values are of the same type chan T. I dunno, just spitballing I haven’t given that much thought, but the ability to compose promises and the relative inability to compose channels with the same expressive power is worth facing honestly.

                                                            I actually fully hate using async and await in JS but every time I need to fan-out, fan-in channels manually I get a little feeling that maybe there’s a piece missing here.

                                                            1. 3

                                                              I just want to be able to call select on N channels where N is not known at compile time.

                                                              You can.

                                                              https://golang.org/pkg/reflect#Select.

                                                              1. 2

                                                                the argument that I’m making is that promises have ergonomics that channels lack, and that although I don’t think Go needs promises, that the project in question is reflective of how promise ecosystems have invested heavily in ergonomics in many scenarios that Go leaves for every developer to solve on their own. Calling reflect.Select is not a solution to a problem of ergonomics, because reflect.Select is terribly cumbersome to use.

                                                              2. 1

                                                                honestly I just want to be able to call select on N channels where N is not known at compile time

                                                                That’s still too low-level, in my experience. And being able to do this doesn’t, like, unlock any exciting new capabilities or anything. It makes some niche use cases easier to implement but that’s about it. If you want to do this you just create a single receiver goroutine that loops over a set of owned channels and does a nonblocking recv on each of them and you’re done.

                                                                every time I need to fan-out, fan-in channels manually I get a little feeling that maybe there’s a piece missing here.

                                                                A channel is a primitive that must be owned and written to and maybe eventually closed by a single goroutine. It can be received from by multiple goroutines. This is just what they are and how they work. Internalize these rules and the usage patterns flow naturally from them.

                                                                1. 2

                                                                  loops over a set of owned channels and does a nonblocking recv on each of them and you’re done.

                                                                  How do you wait with this method? Surely it’s inefficient to do this in a busy/polling loop. Or maybe I’m missing something obvious.

                                                                  Other approaches are one goroutine per channel sending to a common channel, or reflect. Select().

                                                                  1. 1

                                                                    Ah, true, if you need select’s blocking behavior over a dynamic number of channels then you’re down to the two options you list. But I’ve never personally hit this use case… the closest I’ve come is the subscriber pattern, where a single component broadcasts updates to an arbitrary number of receivers, which can come and go. That’s effectively solved with the method I suggested originally.

                                                                  2. 1

                                                                    I’ve been programming Go for ten years. I know how channels work.

                                                                    Promises can be composed, and that is a useful feature of promises. Channels cannot be composed meaningfully, and that is rather disappointing. The composition of channels has much to give us. Incidentally, the existence of errgroup and broadly most uses of sync.WaitGroup are the direct result of not having an ability to compose channels, and channel composition would obviate their necessity entirely.

                                                                    What is it that sync.WaitGroup and errgroup are solving when people generally use them? Generally, these constructs are used in the situation that you have N concurrent producers. A common pattern would be to create a channel for output, spawn N producers, give every producer that channel, and then have all producers write to one channel. The problem being solved is that once a channel has multiple writers, it cannot be closed. sync.WaitGroup is often used to signal that all producers have finished.

                                                                    This means that practically speaking, producer functions very often have a signature that looks like this:

                                                                    func work(c chan T) { ... }
                                                                    

                                                                    Instead of this:

                                                                    func work() <-chan T { ... }
                                                                    

                                                                    This is in practice very bothersome. In the situation that you have exactly one producer that returns a channel and closes it, you could do this:

                                                                    for v := range work() {
                                                                    }
                                                                    

                                                                    This is great and wonderfully ergonomic. The producer simply closes the channel when it’s done. But when you have N producers, where N is not known until runtime, what can you do? That signature is no longer useful, so instead you do this:

                                                                    func work(wg *sync.WaitGroup, c chan T) {
                                                                        defer wg.Done()
                                                                        // do whatever, write to c but don't close c
                                                                    }
                                                                    
                                                                    var wg sync.WaitGroup
                                                                    c := make(chan T)
                                                                    for i := 0; i < n; i++ {
                                                                        wg.Add(1)
                                                                        go work(&wg, c)
                                                                    }
                                                                    
                                                                    done := make(chan struct{})
                                                                    go func() {
                                                                        wg.Wait()
                                                                        close(done)
                                                                    }()
                                                                    
                                                                    for {
                                                                        select {
                                                                        case <-c:
                                                                            // use the result of some work
                                                                        case <-done:
                                                                            // break out of two loops
                                                                        }
                                                                    }
                                                                    

                                                                    That’s pretty long-winded. The producer written for the case of being 1 of 1 producer and the producer written for the case of being 1 of N producers have to be different. Maybe you dispense with the extra done channel and close c, maybe you use errgroup to automatically wrap things up for you, it’s all very similar.

                                                                    But what if instead of N workers writing to 1 channel, every worker had their own channel and we had the ability to compose those channels? In this case, composing channels would mean that given the channels X and Y, we compose those those channels to form the channel Z. A read on Z would be the same as reading from both X and Y together in a select statement. Closing X would remove its branch from the select statement. Once X and Y are both closed, Z would close automatically. Given this function, we could simply have the worker definition return its own channel and close it when its done, then compose all of those, and then read off that one channel. No errgroup or sync.WaitGroup necessary. Here is an example of what that would look like:

                                                                    func work() <-chan T {}
                                                                    
                                                                    var c <-chan T
                                                                    for i := 0; i < n; i++ {
                                                                        c = join(c, work())
                                                                    }
                                                                    
                                                                    for v := range c {
                                                                        // use the result of some work
                                                                    }
                                                                    

                                                                    Here is a working program that implements this concept at the library level: https://gist.github.com/jordanorelli/5debfbf8dfa0e8c7fa4dfcb3b08f9478

                                                                    Tada. No errgroup necessary, no sync.WaitGroup, none of that. The producer is completely unaware that it is in a group and the consumer is completely unaware that there are multiple producers. You could use that producer and read its results as if it’s just one, or one of many in the exact same way.

                                                                    It makes consuming the result of N workers much easier, it makes it so that a worker may be defined in the event that it is 1 of 1 and 1 of N in exactly the same way, and it makes it so that consumers can consume the work from a channel without any knowledge of how many producers that channel has or any coordination outside of seeing the channel closed. Of course, implementing this at the library level and not at the language level means adding an overhead of additional goroutines to facilitate the joining. If it could be implemented at the language level so that joining N channels into 1 does not require N-1 additional goroutines, that would be neat.

                                                                    This implementation is also subtly broken in that composing X and Y to form Z makes it so that you can’t read off of X and Y on their own correctly now; this is not a full implementation, and there’s certainly a question of implementation feasibility here.

                                                                    1. 1

                                                                      Channels cannot be composed

                                                                      I don’t think I agree. It’s straightforward to build higher-order constructs from goroutines and channels as long as you understand that a channel must be owned by a single producer.

                                                                      The problem being solved is that once a channel has multiple writers, it cannot be closed.

                                                                      It doesn’t need to be closed. If you have 1 channel receiving N sends, then you just do

                                                                      c := make(chan int, n)
                                                                      for i := 0; i < cap(c); i++ {
                                                                          go func() { c <- 123 }()
                                                                      }
                                                                      for i := 0; i < cap(c); i++ {
                                                                          log.Println(<-c)
                                                                      }
                                                                      

                                                                      This means that practically speaking, producer functions very often have a signature that looks like func work(c chan T) { ... }

                                                                      Hopefully not! Your worker function signature should be synchronous, i.e.

                                                                      func work() T
                                                                      

                                                                      and you would call it like

                                                                      go func() { c <-work() }()
                                                                      

                                                                      Or, said another way,

                                                                      go work(&wg, c)

                                                                      As a rule, it’s a red flag if concurrency primitives like WaitGroups and channels appear in function signatures. Functions should by default do their work synchronously, and leave concurrency as something the caller can opt-in to.

                                                                      But what if . . .

                                                                      If you internalize the notion that workers (functions) should be synchronous, then you can do whatever you want in terms of concurrency at the call site. I totally agree that goroutines and channels are, in hindsight, too low-level for the things that people actually want to do with them. But if you bite that bullet, and understand that patterns like the one you’re describing should be expressed by consumers rather than mandated by producers, then everything kind of falls into place.

                                                                      1. 1

                                                                        It’s clear that you didn’t read the gist. Your example falls apart immediately when the workers need to produce more than one value.

                                                                        Your worker function signature should be synchronous, i.e. func work() T

                                                                        That’s not a worker. It’s just a function; a unit of work. That’s not at all the problem at hand and has never been the problem at hand. Maybe try reading the gist.

                                                                        The workers in the gist aren’t producing exactly 1 output. They’re producing between 0 and 2 outputs. The case of “run a function N times concurrently and collect the output” is trivial and is not the problem at hand.

                                                                        The workers are producing an arbitrary number of values that is not known in advance to the consumer. The workers are not aware that they’re in the pool. The consumer is not aware that they’re reading from the pool. There is nothing shared between producers to make them coordinate, and nothing shared with consumers to make them coordinate. There is no coordination between producers and consumers at all. The consumer is not aware of how many workers there are or how many values are produced by each worker, they are only interested in the sum of all work. The workers simply write to a channel and close it when they’re done. The consumer simply reads a channel until the end. That’s it. No errgroup requiring closures to implement the other half of the pattern, no sync.WaitGroup required to manually setup the synchronization. Just summing channels. The case of 1 worker and 1 consumer is handled by a worker having signature func f() <-chan T. The case of 1 worker and N consumers, N workers and 1 consumer, and N workers and M consumers are all handled with the same worker signature, with no additional coordination required.

                                                                        1. 1

                                                                          It’s clear that you didn’t read the gist.

                                                                          I mean, I did, I just reject the premise :)

                                                                          That’s not a worker. It’s just a function; a unit of work

                                                                          Given work is e.g. func work() T then my claim is that a “worker” should be an anonymous function defined by the code which invokes the work func, rather than a first-order function provided by the author of the work func itself.

                                                                          The workers are producing an arbitrary number of values that is not known in advance to the consumer . . . the consumer simply reads a channel until the end.

                                                                          Channels simply don’t support the access pattern of N produers + 1 consumer without a bit of additional code. It’s fair to criticize them for that! But it’s not like the thing is impossible, you just have to add a bit of extra scaffolding on top of the primitives provided by the language.

                                                            2. 2

                                                              I think generics will make channels much easier to use correctly. The shenanigans required to handle cancellation, error reporting, fan-out with limits, etc etc etc means that very few programs handle the edge cases around goroutines. Certainly when I wrote go, I wouldn’t follow the patterns needed to prevent infinite go routine leaks, and often I’d decide to panic on error instead of figuring out how to add error channels or result structs with a null error pointer, etc.

                                                              What I like about Promise is that it’s a Result[T] - but ideally I’d be able to get the composition boilerplate of structured CSP stuff out of the way with generics instead of adopting the Promise model wholesale.

                                                              (My history: I loved writing go for many years but eventually burned out from all the boilerplate and decided to wait for generics)

                                                          1. 21

                                                            Strong disagreement.

                                                            First, “premature optimization” has been twisted very far from its original meaning. Most people only know the bit “premature optimization is the root of all evil in computer science”, but they don’t know the part that comes before. In particular:

                                                            The improvement in speed from Example 2 to Example 2a is only about 12%, and many people would pronounce that insignificant. The conventional wisdom shared by many of today’s software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by penny-wise-and-pound-foolish programmers, who can’t debug or maintain their “optimized” programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal; and I believe the same viewpoint should prevail in software engineering

                                                            (NB: I didn’t know about this part until very recently, unfortunately.)

                                                            Knuth argues that we should not overlooked the easily obtained 12% extra performance. By selecting a different language, for example Go, instead of Python, we can have gains of 10x, even 100x! What does it say about our profession that we would turn up our noses at such possible gains? (Fun exercise: write a program that does rot13 translation from stdin to stdout. I don’t know Go very well, but I was able to write a program that processes a 1 GB file in ~0.5s; my best Python version takes 82s, 160x slower.)

                                                            Second, this advice makes an assumption—one that I used to subscribe to—that is not borne by practice: that you can wring out performance out of a program that was not at all designed with performance in mind. There’s this idea that we can change a couple of algorithms, run a profiler and address the worst outliers and that we’ll have a fast program. But that’s rarely so simple: we often have a uniform mud ball of slowness and nothing in the profile really stands out. Making one part of the program faster has little effect, because all the rest is slow. There is a very good quote at the beginning of the io_uring paper:

                                                            the initial design was centered around efficiency. Efficiency isn’t something that can be an afterthought, it has to be designed in from the start - you can’t wring it out of something later on once the interface is fixed.

                                                            Python does have a place, I use it pretty much every week, but I don’t think it’s fair to say that it doesn’t have performance issues and it’s really disingenuous to suggest that taking those issues in consideration in our initial design is a form of malpractice.

                                                            1. 2

                                                              In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal

                                                              He never specifies about what kind of improvement it is. It might be a manufacturing improvement, a maintainability improvement, a performance improvement, etc. The same with computer programs. Many improvements are actually tradeoffs. One way of writing code may result in more performant code, but it often will result in slower time to completion and/or harder maintainability. It’s almost always a trade-off, and I feel like Knuth was ignoring that.

                                                              1. 2

                                                                his advice makes an assumption—one that I used to subscribe to—that is not borne by practice: that you can wring out performance out of a program that was not at all designed with performance in mind . . . Efficiency isn’t something that can be an afterthought, it has to be designed in from the start

                                                                Ah, but what do you mean by efficiency? :) If it’s akin to “non-pathological” — e.g. use well-defined components, or a connection pool for your DB rather than connecting with each request — then I agree totally. But I frequently see penny-wise-and-pound-foolish premature optimizations made in the name of “efficiency” — or, more precisely, presumed yet unvalidated assumptions about efficiency. One common example is that service-to-service communication should be over gRPC rather than e.g. JSON-over-HTTP because it’s faster. But gRPC carries enormous costs to complexity, often isn’t actually faster than alternatives in any significant way, and it’s rare that service-to-service communication costs are a performance bottleneck in the first place!

                                                                1. 4

                                                                  Ah, but what do you mean by efficiency? :)

                                                                  In the past, I’ve defined it a bit nebulously as making “reasonable” use of the computer’s resources. Now, we could argue about what constitutes “reasonable”: if a program is processing data at 10% the maximum theoretical speed of your memory, is that reasonable? Would 5% still be reasonable? Would 1%? I get that different people and different problems will have different thresholds, but basically it’s about not wasting too much machine time when solving the problem.

                                                                  One common problem of not thinking about performance upfront—and one that still is present in a lot of my code—is to start with “individual-level thinking” (to quote Casey Muratori): building classes that represent a single object and implement methods that work on a single object. If, for example, a method foo does a dynamic memory allocation and we call foo on 100,000 objects, then our program (and its users!) will pay the cost of 100,000 allocation syscalls. If instead we design our system around groups of objects (and your DB connection pool is an example of that), then foo can allocate once and use that memory to process a large number of objects, and thus amortize the cost of the allocations.

                                                                  If a program is not initially designed around groups of objects and the public API works in terms of single objects, it’s going to be hard, long, and difficult to change the design to use batches instead.

                                                                  As to your comparison of gRPC vs JSON-over-HTTP, I don’t know enough about gRPC to argue in the favour of either, but I think the point of batches applies: it doesn’t matter which one we pick if we’re going to be doing 100,000 service-to-service calls instead of, say, 1000.

                                                                  1. 1

                                                                    I agree that an API which deals in high-cardinality entities but doesn’t offer batching capabilities is a problem that’s always worth solving. I’d file this under the “non-pathological” umbrella, though admittedly knowing the difference between pathological architectural issues and deferrable optimizations is more art than science.

                                                              1. 1

                                                                the trouble with synchronous interfaces is that since they have to stay in lock-step, the system can only run at the speed of the slowest link. It produces an effectively single-threaded implementation that can only do one thing at a time

                                                                This is true, but typically that single-threaded execution path is isolated to a single unit of work: a request, a job, etc. which are typically numerous in a given system. Is that not the case for the replicator?

                                                                1. 2

                                                                  No; it’s not good for performance to treat every unit (a revision of a document) separately.

                                                                  • At the database level you get big economies of scale by querying for and updating many documents at once.
                                                                  • At the network level you need to multiplex the stream or else you introduce round-trip delays during every request-response cycle.
                                                                  • By processing a document synchronously, each one ties up a CPU thread (this is C++ not Go). That adds a bunch of overhead, esp. memory, to running them in parallel, which is a big deal on mobile devices.
                                                                1. 2

                                                                  What I’ve never understood is, who cares? Shared memory concurrency is more abstract, more complex, and less performant (has worse locality) than message passing. So what if these memory models are so complex? You shouldn’t be doing shared memory concurrency anyway, you should be asking CPU vendors for explicit inter-core message passing support that bypasses cache coherency. Then you can build whatever memory model you want in “userspace”.

                                                                  1. 13

                                                                    You shouldn’t be doing shared memory concurrency anyway, you should be asking CPU vendors for explicit inter-core message passing support that bypasses cache coherency. Then you can build whatever memory model you want in “userspace”.

                                                                    How is your message passing implemented? Most high-performance message-passing systems use shared memory to build zero-copy lock-free data structures for the message passing. Implementing these correctly requires a memory model that you can reason about. So, to answer your ‘who cares?’ question: it’s those of us who are implementing the message-passing abstractions that you want to use.

                                                                    You shouldn’t be doing shared memory concurrency anyway, you should be asking CPU vendors for explicit inter-core message passing support that bypasses cache coherency. Then you can build whatever memory model you want in “userspace”.

                                                                    I agree in principle but it’s not that simple. The hypervisor and kernel are both virtualising the CPU for you. The hypervisor gives the OS VCPUs that map N:M to cores over time. The OS provides threads, which map N:M to VCPUs. A language runtime for a message-passing language provides actors or similar that map N:M to threads. To do efficient message passing, you need a namespace for cores that supports this kind of virtualisation, such that I can say ‘send a message to actor {foo}’ and the hardware knows where to route that message even if the actor is not currently scheduled (or if it is scheduled on a thread but that thread isn’t scheduled, or if the thread is scheduled but on a VCPU that is not scheduled).

                                                                    At the moment, the way we identify programmable endpoints is via virtual memory. My actor’s message queue is identified by a virtual memory address, which is bound to a thread purely by software, the thread is bound to a VCPU by the page-table base register and the VCPU a PCPU by the second-level address-translation page-table base register. The cache coherency bus is basically a message-passing interface, so when you do a store to that message queue you’re really doing a buffered store to something that the other actor can pick up when it’s scheduled. It’s not at all clear (in spite of researchers looking at this problem for at least 40 years) that there’s a better abstraction.

                                                                    1. 2

                                                                      How is your message passing implemented?

                                                                      By using the message-passing support that the CPU already supports internally, and which it uses to implement the abstraction of shared memory.

                                                                      The other approach - of implementing message passing on top of shared memory on top of message passing - is a ridiculous abstraction inversion. The fact that it requires us to develop a rich memory model for the entirely superfluous step of shared memory in the middle, is an indication of how ridiculous it is.

                                                                      The cache coherency bus is basically a message-passing interface, so when you do a store to that message queue you’re really doing a buffered store to something that the other actor can pick up when it’s scheduled. It’s not at all clear (in spite of researchers looking at this problem for at least 40 years) that there’s a better abstraction.

                                                                      Wait, I think this is confusing the issues. If the cache coherence bus was literally only an interface to buffered, virtualizable message passing, with programmable endpoints implemented through virtual memory - then everything would be fine. It’s because we also implement shared memory with that bus that we run into problems and have to come up with a memory model.

                                                                      At least, that’s how it seems to me? Like I said, I don’t understand why everyone cares about memory models when this seems like a much simpler alternative direction.

                                                                      1. 7

                                                                        In reverse order:

                                                                        At least, that’s how it seems to me? Like I said, I don’t understand why everyone cares about memory models when this seems like a much simpler alternative direction.

                                                                        This was part of my C is not a low-level language article and the reason people care about memory models really boils down to the fact that the hardware and ‘low-level’ languages are trying to maintain the illusion that you have a fast PDP-11. If you want to scale up a language whose abstract machine is a virtualised PDP-11 to multiple cores, you need shared memory. If you want to program shared memory, you need a memory model. If you want to interoperate with any code written in this style, you need to care about that memory model.

                                                                        Wait, I think this is confusing the issues. If the cache coherence bus was literally only an interface to buffered, virtualizable message passing, with programmable endpoints implemented through virtual memory - then everything would be fine. It’s because we also implement shared memory with that bus that we run into problems and have to come up with a memory model.

                                                                        The cache coherency bus is a message-passing protocol. On top of this is a cache-coherency protocol that works with physical addresses as the identifying tokens defining what data needs to be sent where. The virtual memory abstraction is built on top of this. It’s not clear (in spite of many researchers trying to build systems that expose message-passing primitives over the last few decades) what a virtualised message-queue mechanism should look like. In particular, you also want it to support zero copy, so you actually do want something like shared memory, just with explicit transfer of ownership. The transfer of ownership part becomes a bit tricky if your messages contain complex data structures because you need an O(n) walk of the structure to identify which cache lines (or other memory granules) to include in the message, which is almost as bad as a copy.

                                                                        By using the message-passing support that the CPU already supports internally, and which it uses to implement the abstraction of shared memory.

                                                                        And this is fine if your messages are 64 bytes or smaller, or can be streamed as chunks of 64 bytes. If you want to be able to prepare a message and then send it to another core and have that core mutate it and pass it somewhere else, then just sending a NoC message for the language’s message send is not sufficient, you also need to serialise the message. Or you can define a memory model that guarantees that the receiving core will see the message if it looks after it receives the pointer to the message.

                                                                        The other approach - of implementing message passing on top of shared memory on top of message passing - is a ridiculous abstraction inversion. The fact that it requires us to develop a rich memory model for the entirely superfluous step of shared memory in the middle, is an indication of how ridiculous it is.

                                                                        You’re not wrong, but I think you’re underestimating the difficulty of building a sensible model that supports sending messages that are actually useful to software, directly on top of the SoC’s network. I’d suggest that you grab one of the open-source RISC-V cores (I like Toooba for this kind of thing), or even gem5, and try building something. If you can make something that scales well to 1024 cores and has a programmer model that is easy to use, it would easily win the best paper award at ASPLOS.

                                                                        1. 2

                                                                          The transfer of ownership part becomes a bit tricky if your messages contain complex data structures because you need an O(n) walk of the structure to identify which cache lines (or other memory granules) to include in the message, which is almost as bad as a copy.

                                                                          If you want to be able to prepare a message and then send it to another core and have that core mutate it and pass it somewhere else, then just sending a NoC message for the language’s message send is not sufficient, you also need to serialise the message.

                                                                          Yeah, but not doing this is exactly why message passing is more efficient - sure you can do shared memory over big datastructures that are fragmented across your address space, but that’s slow. Sending large messages (through the same mechanism as small messages, anyway) is not desirable, because transferring large amounts of data between cores/main memory is not desirable, whether that’s through shared memory or message passing.

                                                                          But I suppose that’s not immediately obvious. So thanks to your comment I think I can see where memory models/shared memory are coming from: Wanting a single mechanism that scales up to transferring large data structures and down to small messages.

                                                                          So in this sense, a memory model is a serialization mechanism - in both meanings of “serialization”! :)

                                                                          the reason people care about memory models really boils down to the fact that the hardware and ‘low-level’ languages are trying to maintain the illusion that you have a fast PDP-11

                                                                          I certainly deeply agree with your core idea here. But I do have one nitpick: In retrospect, it’s not clear to me that shared memory was the natural generalization of C-on-Unix-on-PDP-11 to multiple cores. Unix didn’t start with shared memory, it started with pipes and a filesystem and processes. The shared memory idea was added later - of course, we’re stuck with it now, and everyone is trying to speed it up as much as possible.

                                                                          1. 2

                                                                            Yeah, but not doing this is exactly why message passing is more efficient - sure you can do shared memory over big datastructures that are fragmented across your address space, but that’s slow.

                                                                            It’s not very slow on modern processors. In the worst case, you miss in all the caches, but a load from a remote cache in the same node is fairly cheap and you can hide the latency with speculative execution or SMT. Importantly, you do it only for the parts of the data structure that are actually used by the recipient. This lets you have a model of moving the compute to the data, rather than the other way around, which tends to be more ergonomic and more efficient.

                                                                            Sending large messages (through the same mechanism as small messages, anyway) is not desirable, because transferring large amounts of data between cores/main memory is not desirable, whether that’s through shared memory or message passing.

                                                                            As you point out later, it’s really eliminating the serialisation requirement that’s a problem. With message sending, you have an implicit copy to implement a move. With shared memory, you can do lazy serialisation: you don’t need to copy the data until just before the data is accessed. You can treat the non-cache memory as a multi-producer, multi-consumer queue with arbitrary reordering, so things that the consumer doesn’t need immediately go there to be picked out later.

                                                                            You might be interested in what we’re doing with Verona. The programmer model fits well with the low-level bits of a cache coherency protocol: all data is either mutable and guaranteed not to alias between concurrent units of execution, or immutable and therefore safe to copy to every core / node in a distributed system. I’m very interested in how we can evolve datacenter memory designs to allow us to scale this kind of design up to entire racks or even entire datacenters.

                                                                            1. 1

                                                                              This lets you have a model of moving the compute to the data, rather than the other way around, which tends to be more ergonomic and more efficient.

                                                                              For sure - this is very much how I think about it - I was writing out something about this in my previous comment (as a reply to “the difficulty of building a sensible model”) but ended up deleting it.

                                                                              One of my big interests is in making large-scale distributed programming easier. With being able to deal with the locations of distributed resources, and solving the code mobility problem, all at the language level using a sophisticated type system.

                                                                              It may be over-applying my one idea, but I think this is also a good way to deal with SoC-scale programming. Having explicit low-level models in the type system of where resources are located, and therefore what code can access them. And make it straightforward to talk about moving code and resources between cores, and orchestrating the network of communicating cores. (And virtualize at the language-level, with object-capability-security and a trusted compiler.) Of course the big difference is that distributed programming is more complicated, because it has multiple failure domains, but I’m more interested in the SoC-scale single-failure-domain case anyway.

                                                                              So I think that would be a good approach to make a model for message-passing-networks-of-cores that really works. But I haven’t really got the chance to work on the SoC-scale, and don’t really know anyone working on this.

                                                                              You might be interested in what we’re doing with Verona. The programmer model fits well with the low-level bits of a cache coherency protocol: all data is either mutable and guaranteed not to alias between concurrent units of execution, or immutable and therefore safe to copy to every core / node in a distributed system. I’m very interested in how we can evolve datacenter memory designs to allow us to scale this kind of design up to entire racks or even entire datacenters.

                                                                              Yes, that is interesting. From reading through https://microsoft.github.io/verona/explore.html, Verona has a potent combination of features. It would be great if memory was designed around being programmed with a Verona-like model, rather than a totally-uncontrolled-arbitrary-sharing C model. That may not be the message passing I want, but it would surely be a vast improvement.

                                                                            2. 1

                                                                              You are the first person I have ever encountered who claims that message passing is more efficient than shared memory. The claim also contravenes my experience. The fundamental difference between the two paradigms is surely that message passing necessarily performs a copy, is that not true? If yes, then how can it be more efficient than the alternative? If no, then is it not by definition shared memory?

                                                                              1. 1

                                                                                It’s all about locality. Read, for example, https://dl.acm.org/doi/pdf/10.1145/3132747.3132771

                                                                                1. 1

                                                                                  Well, I can’t read that, unfortunately… does “ffwd” perform a copy?

                                                                                  edit: I found the morning paper coverage – as I understand it, the approach is similar to the actor model, where data is owned by a single thread, which acts as a natural synchronization point. That’s I guess a hybrid approach, neither message passing nor shared memory, right? I’ve used it quite successfully in a number of projects, but it doesn’t fully generalize, in my experience…

                                                                    1. 1
                                                                      .PHONY: install
                                                                      install: $(alib)
                                                                          mkdir -p $(DESTDIR)$(PREFIX)/lib
                                                                          mkdir -p $(DESTDIR)$(PREFIX)/include
                                                                          cp $(alib) $(DESTDIR)$(PREFIX)/lib/$(alib)
                                                                          cp foo.h $(DESTDIR)$(PREFIX)/include/
                                                                      

                                                                      Should $(DESTDIR)$(PREFIX)/{lib,include} not themselves be targets?

                                                                      1. 1

                                                                        No. You don’t care about them and caring about too much gets you in trouble when running a makefile somewhere else.

                                                                        1. 2

                                                                          I don’t quite understand. Why don’t I care about them? Aren’t they prerequisites for the files that are put there?

                                                                          1. 1

                                                                            They are not products of your build. At best you want a ‘make -p’ or do an ‘install’ that makes dirs for you.

                                                                            The files put inside those dirs are where you should focus.

                                                                            You don’t specify a filesystem or OS when you use make, even though you need them. Need to draw the line somewhere, and I draw them at “do I actually care about them”

                                                                            1. 2

                                                                              They are the products of my build. i.e. they don’t exist in the checkout of the repo. They’re intermediate artifacts. Right? That’s categorically different than the FS or OS, which are already present and implicit. AFAIU the line is way more concrete than “do I need them” — it’s “are they checked in”. Or do I misunderstand what a Makefile is for?

                                                                              1. 1

                                                                                A makefile is for turning existing artifacts into paths that don’t yet exist. You specify as little as necessary to get from point a to b so you don’t write an overly brittle makefile. If it’s not necessary, you don’t care.

                                                                                1. 1

                                                                                  overly brittle

                                                                                  Honestly curious: what makes specifying intermediate steps of a build as explicit targets brittle?

                                                                                  1. 1

                                                                                    Default build targets are amazing and capable.

                                                                                    And generally portable.

                                                                                    1. 1

                                                                                      I also use default build targets when I explicitly specify intermediate steps of a build?

                                                                                      1. 1

                                                                                        Which default target makes sure a directory exists?

                                                                                        1. 1

                                                                                          No default target makes sure a directory exists. I think we’re talking past each other. I’m saying instead of

                                                                                          PREFIX = /usr/local
                                                                                          
                                                                                          .PHONY: install
                                                                                          install: mygame
                                                                                              mkdir -p $(DESTDIR)$(PREFIX)/bin
                                                                                              cp $< $(DESTDIR)$(PREFIX)/bin/mygame
                                                                                          

                                                                                          you should maybe do

                                                                                          PREFIX = /usr/local
                                                                                          DEST = ${DESTDIR}${PREFIX}/bin
                                                                                          
                                                                                          .PHONY: install
                                                                                          install: ${DEST}/mygame
                                                                                          
                                                                                          ${DEST}/mygame: ${DEST} mygame
                                                                                              cp mygame $@
                                                                                          
                                                                                          ${DEST}:
                                                                                              mkdir -p ${DEST}
                                                                                          
                                                                        2. 1

                                                                          Yes. You can do something like

                                                                          DIRS := $(addprefix $(DESTDIR)$(PREFIX),lib include)
                                                                          $(DIRS):
                                                                          	mkdir -p $@
                                                                          
                                                                          .PHONY: install
                                                                          install: $(alib) | $(DIRS)
                                                                          	cp $(alib) $(DESTDIR)$(PREFIX)/lib/alib
                                                                          	cp foo.h $(DESTDIR)$(PREFIX)/include/
                                                                          

                                                                          But of course, you can also just use install.

                                                                          If all you are doing is installing configuration files, and can arrange them beforehand, I particularly like

                                                                          $(DESTDIR)$(PREFIX)%: %
                                                                          	install -Dm644 $< $@
                                                                          
                                                                          .PHONY: install
                                                                          install: $(addprefix $(DESTDIR)$(PREFIX),$(FILES))
                                                                          
                                                                        1. 4

                                                                          Great article, but IMO http.ServeMux has a couple of other gotchas in addition to the ones mentioned, that make it more painful to use than it should be:

                                                                          1. No way to match “/foo” and “/foo/” with a single Handle() call - handler on “/foo/” will not match “/foo”, and vice versa.
                                                                          2. “/foo/” matches the entire sub-tree “/foo/…” - there is no way to match a single path component under a sub-tree.

                                                                          Yes, these can be worked around, but it’s usually just more straightforward to use a better router.

                                                                          1. 5

                                                                            it’s usually just more straightforward to use a better router.

                                                                            You can duplicate Handle calls to solve those problems. I dunno. Most of my projects these days have like zero to five dependencies? I think that’s something worth gunning for; if there’s a way to stay within the stdlib, even with a bit of toil, seems like it’s worth doing.

                                                                          1. 8

                                                                            A local font stack is all well and good, but Helvetica and Arial, et al are just…well…boring.

                                                                            This is my only disagreement with the article. I prefer to just use the user’s default font. Sure, that is boring but I personally don’t find much value in every site having their own “exciting” font. I just want to see my nice, readable default font almost everywhere. (Some things like games have a reasonable excuse to use their own.)

                                                                            So drop the font stack altogether and just leave the default. Or if you really must font: sans-serif (or serif). Or the newfangled ui-sans-serif (if I remember that correctly).

                                                                            Of course the downside is that a lot of browsers use garbage default fonts by default…

                                                                            1. 8

                                                                              Of course the downside is that a lot of browsers use garbage default fonts by default…

                                                                              This is the real underlying problem. The overwhelming vast majority of users don’t even know they can change the default font much less know how to.

                                                                              1. 6

                                                                                100%. Which is super disappointing because it would take browsers very little effort to fix.

                                                                                Firefox doesn’t even default to my system font by default (which is quite nice) so I need to teach it to follow my system font which ~0% of users will do.

                                                                              2. 7

                                                                                Helvetica and Arial, et al are just…well…boring.

                                                                                Boring is good! Most of the time, typography’s job is to get out of the way. Browser-default sans-serif is usually what you want, with a slight bump to the size.

                                                                                1. 4

                                                                                  If everyone needs to bump the size maybe we should petition browsers to make the default bigger…

                                                                                  1. 2

                                                                                    Agree! font-size: 16pt; is a much better default.

                                                                                2. 4

                                                                                  I just want to see my nice, readable default font almost everywhere.

                                                                                  This is why I disable custom fonts entirely i my browser. Makes me laugh when a bad website (looking at anything created by Google) uses idiotic icon fonts, but whatever, those sites aren’t worth using anyway.

                                                                                  1. 2

                                                                                    I’ve done this on-and-off but I’m not completely sold. Luckily icon fonts are somewhat falling out of style but I think there are legitimate uses for custom fonts, especially for things that are more wacky and fun. I think it is just a shame that they are abused (IMHO) on just about every website.

                                                                                    1. 4

                                                                                      Not somewhat. Fonts for icons have not been a recommended practice in since 2014 when the people shifted to SVG icons via symbols. Prevalence is likely devs that learned about icon fonts at one point, especially in the past when IE compatibility took away many options, and those people do not do enough front-end to bother keeping up with most trends.

                                                                                  2. 3

                                                                                    The people who think to change their browser’s default font are probably the same people who can figure out how to force websites to use the default font.

                                                                                    Shipping a font with a site is the same as any other styling: helpful for the average user, overridable by the power user.

                                                                                    1. 2

                                                                                      I’m kind of interested right now in both improving my website’s typography and getting rid of things like background images and web fonts. The page is already pretty light, but it can be lighter, and I’m questioning the value of the fonts I’m using. I do want to exercise some choice in fonts, though. The ui-sans-serif (etc) look like good options, but so far they’re only supported on Safari. I’m probably going to do some research on a local font stack that doesn’t include awful fonts like Helvetica and Times New Roman, has a reasonable chance of working cross-platform, and falls back to serif, sans-serif, and monospace.

                                                                                      1. 4

                                                                                        I don’t know that many people think Helvetica is “awful”.

                                                                                        Just cut things. Cut the background images entirely; replace with a nice single color. Start by cutting all the web fonts: specify serif and sans-serif and no more than four or five sizes of each. Make sure your margins are reasonable and text column widths are enjoyable. How many colors are you using? Is there enough contrast?

                                                                                        Once you’ve stripped everything back to a readable, comfortable page, see what you actually need to distinguish yourself rather that trying a bunch of trends. What image are you trying to project? Classic and authoritative? Modern? Comedic? Simple? Fashionable? Pick one approach and try to align things to that feel.

                                                                                        1. 1

                                                                                          Helvetica and Times New Roman are de facto standards for a reason. They’re solid, utilitarian typefaces. I wish more sites used them.

                                                                                      1. 3

                                                                                        It seems to me that proponents of dynamic linking kind of miss the forest for the trees. The system is incidental, the only thing that matters are the applications. The value of a binary that just works is enormously higher than the value of system-wide security patches or etc.

                                                                                        1. 2

                                                                                          I agree.

                                                                                          I think following your comment would be a great way to design a user-friendly OS: you want to make things “just work” as you say, while having security.

                                                                                        1. 8

                                                                                          User beware. The hand-rolled recopies showcased here are better than your average hand-rolled recopies I see on many projects, but they still leave a lot to be desired. For example mkdir dir; cp foo dir is not nearly as robust as install -D -t dir foo. But that’s trivial compared to the bash specific syntax in an sh file that should be plain POSIX, or the fact that the simulated argument handling doesn’t work quite the way as the real ./configure script generated by autotools does.

                                                                                          As ugly as it can be sometimes, using autotools (autoconf, automake) is far more robust than hand-rolled scripts will ever be precisely because it has such a long history and wide usage all the edge cases not considered by any one person are already taken care of. And in the end the amount of code you actually have to write and maintain would be less for the examples given.

                                                                                          1. 7

                                                                                            IME many autotools-using projects have failed to build for me, and writing a 12 line makefile by hand could build the same project fine.

                                                                                            1. 6

                                                                                              2 points.

                                                                                              1. That is almost certainly not autotools fault, but its so confusing to get started using that it doesn’t always get setup right, and that can certainly make it break for users.

                                                                                              2. Your 12 line makefile may work for you, but it likely doesn’t work for systems that are not like yours. It probably leaves distro packages writing their own build routines and different OS layouts hacking around the over-simplified model shown in 12 lines. And I say this as someone who wrote THOUSANDS of lines of hand-brew makefiles before finally turning to autotools and finding it solved so many of the edge cases I was working around it really was useful.

                                                                                              1. 7

                                                                                                That is almost certainly not autotools fault, but its so confusing to get started using that it doesn’t always get setup right, and that can certainly make it break for users.

                                                                                                I’d argue that if it is confusing/hard to use, it is the fault of the tool.

                                                                                                1. 3

                                                                                                  Your 12 line makefile may work for you, but it likely doesn’t work for systems that are not like yours.

                                                                                                  How important is it that a given piece of software be buildable on the maximum number of architectures?

                                                                                                  Where is the line beyond which complexity to achieve portability is more cost than benefit? Is there such a line?

                                                                                                  1. 5

                                                                                                    How important is it that a given piece of software be buildable on the maximum number of architectures?

                                                                                                    Just as important as it is to have access to more systems that x86_64, and run more operating systems than Windows (or, whatever, Aarch64 and macOS). Portability isn’t just about being able to build libpng for OpenVMS in 2021, it’s also about being able to build whatever is going to be important, under whatever operating system is going to be important, in 2025.

                                                                                                    You could presumably drop support for the former without causing much of a fuss, but autotools & friends allow us to support any number of architectures, whether maximum or not. If we start to package everything with 12-line makefiles we might as well all buy x86_64 laptops running whatever Linux distro op is running and give it up. Then in 2031 we can all gather here and whine how the next best operating system supports a direct-to-brain interface but is held back by the fact that most software only works with Ubuntu and Docker (or, realistically, Windows and macOS…)

                                                                                                    1. 1

                                                                                                      Just as important as it is to have access to more systems that x86_64, and run more operating systems than Windows (or, whatever, Aarch64 and macOS).

                                                                                                      I don’t think I understand your point. I agree that it’s important to support more than a single platform. But my claim is that there’s diminishing returns beyond a certain point. Do you dispute that? If not, where is that point? I guess I’m claiming it’s somewhere around the top 5-10 in terms of userbase. Do you think it’s more than that?

                                                                                                      1. 2

                                                                                                        My point is two-fold (sorry, it may have been useful to be a little more clear).

                                                                                                        First, “the maximum number of architectures” is just another way of saying “any architectures one needs”. Any system, no matter how useful, including Linux, started by not being even close to the top 5-10 in terms of userbase. If you restrict your build system’s capability to supporting only the leading two or three platforms of the day (realistically, a short Makefile will at most get you Linux, BSDs, and maybe Cygwin, mingw & friends need quite a few hacks for non-trivial programs – 5-10 is already way more than you can realistically manage for non-trivial software), you’re dooming it to obsolescence as soon as the next different enough thing hits the market.

                                                                                                        I bet that most of the programs that use autotools don’t actually run (at all, let alone well) on all platforms that autotools can build them on. That’s only as big a problem as the users of said software make it to be though, presumably nobody is losing too much sleep over not being able to run qemu on 4.4 BSD. However, if and when you need, you’re at least gonna be able to build it and you can go right ahead and do the substantial porting work you need. Clunky though they may be, programs built using autotools are gonna compile on Fuchsia 3.1 for AR Workgroups or whatever pretty much out of the box (barring any platform-specific things that need code, not build system incantations, of course). Beautiful, minimalistic, hand-rolled, organic Makefiles are gonna be a curse as soon as Ubuntu 20.04 LTS will be a distant memory instead of something that was pretty cool five years ago,

                                                                                                        It’s not like we haven’t seen this already, autotools became a thing because, terrible though it may have been even then (like, m4, seriously?) it was still better than managing Makefiles for even two or three commercial Unices (which meant way more than just the OS back then – also different libcs, different C compilers and so on). It didn’t come up in a vacuum, either, it was based on (or rather tried to improve) a bunch of prior art (Metaconfig, imake).

                                                                                                        Second, and strictly regarding the number of platforms: in my experience, while the tooling required to build on 1-2 platforms vs. 5-10 platforms is massively different, past that point, the complexity of the tooling itself is pretty much the same, whether you support 5-10 platforms or 50-100. The number of platform-specific quirks obviously increases with the number of platforms, but extending a tool that already supports 10 platforms so as to allow it to support 100 is largely an exercise in implementing platform-specific hacks, whereas extending the tooling (“12-line makefile”) to support 10 platforms instead of 2 basically amounts to writing a new build system.

                                                                                                        So while it takes a lot of effort to go from supporting 1 platforms to 5-10, there’s not that much effort in ensuring your software can be built on more than that. If you’ve done that, you might as well go all the way and use a system that has the traction and popularity required to work everywhere for the foreseeable future. autotools isn’t the only one. CMake is another, for example. It’s terrible but lots of popular software is built with it. Anyone who wants their platform to be relevant will have to ensure CMake-built software runs on it, otherwise they’re gonna be stuck without some leading DBMSs, web browsers and so on.

                                                                                                        1. 1

                                                                                                          I think I understand your point here. But I think the difference between our positions is that you’re judging autotools by its asserted capabilities, and I’m judging it by it’s practical drawbacks. In my experience autotools fails more often than it succeeds, and it delivers net-negative value to projects that use it. I’m sure your experience is different!

                                                                                                          If you restrict your build system’s capability to supporting only the leading two or three platforms of the day (realistically, a short Makefile will at most get you Linux, BSDs, and maybe Cygwin, mingw & friends need quite a few hacks for non-trivial programs – 5-10 is already way more than you can realistically manage for non-trivial software), you’re dooming it to obsolescence as soon as the next different enough thing hits the market.

                                                                                                          I don’t think I buy this. Software is a living thing, and if my project is important enough to stand the test of time, it will grow build capabilities for new architectures as they become en vogue.

                                                                                                          1. 2

                                                                                                            I’m sure your experience is different!

                                                                                                            I think so. While it’s definitely far from my favourite build system, I can’t remember the last time I’ve seen it fail. However, it’s so widespread that it’s very easy to be lucky and only encounter it in scenarios where it works fine.

                                                                                                  2. 3

                                                                                                    Your 12 line makefile may work for you, but it likely doesn’t work for systems that are not like yours.

                                                                                                    When someone else’s 12 line makefile doesn’t work, it’s clear how to fix it. The same isn’t true for autotools, cmake, or many of the other more complicated alternatives.

                                                                                                    1. 1

                                                                                                      tbh I don’t actually agree this. make unfortunately is quite difficult to debug without turning to external tools. Honestly, I find both cmake and autotools a lot easier to debug than just a raw Makefile, since they have debugging facilities.

                                                                                                      That said, I don’t turn to cmake or autotools until I hit the level of complexity or reach where that matters, and most of the time just use simple Makefiles.

                                                                                                    2. 2

                                                                                                      If it is really 12 lines, it is probably less annoying for distros to maintain their own version than to maintain a command line of ./configure .... If something breaks, it’s much easier to fix a Makefile with 12 lines than the whole auto* setup.

                                                                                                      1. 1

                                                                                                        Not to mention default rules which are good and can change to fit your platform without you trying or caring.

                                                                                                      2. 1

                                                                                                        Your 12 line makefile may work for you, but it likely doesn’t work for systems that are not like yours.

                                                                                                        Absolutely, I always assume pkg-config and other of these sorts of things of course that autotools will try to paper over for you.

                                                                                                        1. 2

                                                                                                          Can you build a proper shared library on AIX without using libtool? I’d like to see your makefile try.

                                                                                                          1. 4

                                                                                                            No idea, and also no interest from me :) If someone wants to build something on AIX that’s cool I guess, and they probably should be using autotools.

                                                                                                            1. 1

                                                                                                              You seem to be implying that using -G and ar aren’t enough. Care to elaborate on why not?

                                                                                                              1. 4

                                                                                                                This is worthy of a blog post, but while technically you CAN just go gcc -shared libhober.so libhober.c and call it a day, there’s a lot of subtleties. Specifically:

                                                                                                                • Your binary will export everything by default because symbol visibility sucks.

                                                                                                                • Your binary is one architecture only - AIX actually has fat libraries, but not binaries. It works by making libraries .a archives, and you can import based on member.

                                                                                                                • To circle back a point, you can also put a file in the archive (shr.imp/shr_64.imp - it is very tempting to call them shrimp files) indicating what should be imported, and its name to link against (we’ll come back to that later.)

                                                                                                                • Want versioning? GNU fucked it up by coming up with two schemas: libhober.a(libhober.so.69) and libhober.so.69 (with an implicit shr.o/shr_64.o depending on arch). The former scheme is insane to manage via a package manager, so everyone sane went for the latter.

                                                                                                                • Problem: if you do cc -lhober, it’ll link against libhober.so, not libhober.so.69. This works, but is subtly wrong (don’t have the devel package installed, what if soname bumps for a reason?), but it’s easy not to catch because it works in development.

                                                                                                                • To work around that, your shrimp file can declare the real name of the library to link against.

                                                                                                                This kinda makes sense but is unnecessarily brittle. It does make for a great smoke test of how committed to portability you are though. libtool is like violence, it’s the cause and solution to all of life’s problems.

                                                                                                                1. 1

                                                                                                                  Your binary is one architecture only - AIX actually has fat libraries, but not binaries. It works by making libraries .a archives, and you can import based on member.

                                                                                                                  I mentioned using “-G” and “ar” to specifically create AIX-like shared libraries, which doesn’t suffer from this issue.

                                                                                                                  Some of the others may be issues if you need some of those features, but I don’t see how that really precludes you from building a “proper shared library” that doesn’t use those features with just the compiler and archive tools and a really simple makefile.

                                                                                                                  1. 1

                                                                                                                    I think most people using the really simple makefiles probably aren’t using -G and ar to make a shared library (do you think Makefile curators give a shit about AIX?), and now you have to curate your shrimp file for exports (careful, it’s REALLY easy to accidentally end up exporting strcpy if you do all, because strcpy is only a static object with AIX libc :) ). Hell, stuff like CMake barely does it.

                                                                                                                    1. 1

                                                                                                                      The original position was that shared libraries on AIX are hard or impossible without libtool - but my position is, a simple Makefile can handle that just fine (if the author wants to support AIX explicitly, of course). Now, using libtool may give you different platforms like AIX automatically, but that doesn’t support the claim that a simple Makefile couldn’t build “proper shared libraries” on AIX, if desired.

                                                                                                                      strcpy is only a static object with AIX libc

                                                                                                                      Are you sure?

                                                                                                                      $ dump -Tv test.a
                                                                                                                      
                                                                                                                      test.a[test.o]:
                                                                                                                      
                                                                                                                                              ***Loader Section***
                                                                                                                      
                                                                                                                                              ***Loader Symbol Table Information***
                                                                                                                      [Index]      Value      Scn     IMEX Sclass   Type           IMPid Name
                                                                                                                      
                                                                                                                      [0]     0x0000fc00    undef      IMP     XO EXTref   libc.a(shr.o) ___strcpy
                                                                                                                      [1]     0x20000244    .data      EXP     DS SECdef        [noIMid] f
                                                                                                                      
                                                                                                                      1. 1

                                                                                                                        Perhaps it’s changed in 7.2; I remember having plenty of versions like strcpy end up exported in shared libraries naively exporting all, instead of a curated export list or using nm on objects to generate one.

                                                                                                    1. 22

                                                                                                      (context: I’ve used Go in production for about a year, and am neither a lover nor hater of the language, though I began as a hater.)

                                                                                                      With that said, my take on the article is:

                                                                                                      1. The “order dependence” problem is a non-problem. It doesn’t come up that often, dealing with it easy – this is simply low-priority stuff. If I wanted to mention it, it would be as an ergonomic nitpick.
                                                                                                      2. The infamous Go error handling bloat, while annoying to look at, has the great benefit of honesty and explicitness: You have to document your errors as part of your interface, and you have to explicitly deal with any error-producing code you use. Despite personally really caring about aesthetics and hygiene – and disliking the bloat like the author – I’ll still take this tradeoff. I also work in ruby, and while raising errors allows you to avoid this boilerplate, it also introduces a hidden, implicit part of your interface, which is worse.

                                                                                                      It’s also worth pointing out Rob Pike’s Errors are Values which offers advice for mitigating this kind of boilerplate in some situations.

                                                                                                      1. 22

                                                                                                        There’s a difference between explicitness and pointless tediousness.

                                                                                                        Go’s error handling is more structured compared to error handling in C, and more explicit and controllable compared to unchecked exceptions in C++ and similar languages. But that’s a very low bar now.

                                                                                                        Once you get a taste of error handling via sum types (generic enums with values), you can see you can have the cake and eat it too. You can have very explicit error documentation (via types), errors as values, and locally explicit control flow without burdensome syntax (via the ? syntax sugar).

                                                                                                        1. 4

                                                                                                          I agree.

                                                                                                          But Go, eg, is not Haskell, and that’s an explicit language design decision. I think Haskell is a more beautiful language than Go, but Go has its reasons for not wanting to go that direction – Go values simple verbosity over more abstract elegance.

                                                                                                          1. 15

                                                                                                            If it’s Go’s decision then ¯\_(ツ)_/¯

                                                                                                            but I’ve struggled with its error handling in many ways. From annoyances where commenting out one line requires changing = to := on another, silly errors due to juggling err and err2, to app leaking temp files badly due to lack of some robust “never forget to clean up after error” feature (defer needs to be repeated in every function, there isn’t errdefer even, and there’s no RAII or deterministic destruction).

                                                                                                            1. 6

                                                                                                              Sounds like you’re fighting the language 🤷

                                                                                                              1. 5

                                                                                                                there isn’t errdefer even

                                                                                                                I mean, it’s a pretty trivial helper func if you want it:

                                                                                                                func Errdefer(errp *error, f func()) {
                                                                                                                    if (*err) != nil {
                                                                                                                        f()
                                                                                                                    }
                                                                                                                }
                                                                                                                
                                                                                                                func whatever() (err error) {
                                                                                                                    defer Errdefer(&err, func() {
                                                                                                                       // cleanup
                                                                                                                    })
                                                                                                                    // ...
                                                                                                                }
                                                                                                                

                                                                                                                In general, to have fun in Go, you have to have a high tolerance for figuring out what 3 line helper funcs would make your life easier and then just writing them. If you get into it, it’s the fun part of writing Go, but if you’re not into it, you’re going to be “why do I have to write my own flatmap!!” every fourth function.

                                                                                                                1. 3

                                                                                                                  commenting out one line requires changing = to := on another

                                                                                                                  I do not agree that this is a problem. := is an explicit and clear declaration that helps the programmer to see in which scope the variable is defined and to highlight clear boundaries between old and new declarations for a given variable name. Being forced to think about this during refactoring is a good thing.

                                                                                                                  1. 1

                                                                                                                    Explicit binding definition by itself is good, but when it’s involved in error propagation it becomes a pointless chore.

                                                                                                                    That’s because variable (re)definition is not the point of error handling, it’s only self-inflicted requirement go made for itself.

                                                                                                                    1. 3

                                                                                                                      Go takes the stance that error propagation is not different than any other value propagation. You don’t have to agree that it’s a good decision, but if you internalize the notion that errors are not special and don’t get special consideration, things fall into place.

                                                                                                                  2. 1

                                                                                                                    commenting out one line requires changing = to := on another

                                                                                                                    IMHO := (outside of if, for & switch) was a mistake; I prefer a C-style var block at the top of my function.

                                                                                                                    silly errors due to juggling err and err2

                                                                                                                    I think that this is mostly avoidable.

                                                                                                                  3. 6

                                                                                                                    Yup, Go (well, presumably Rob Pike) made a lot of explicit design decisions like this, which drove me away from the language after a year or two and many thousands of LOC written.

                                                                                                                    Beside the awfulness of error handling, other big ones were the inane way you have to rename a variable/function just to change its visibility, the lack of real inheritance, and the NIH attitude to platform ABIs that makes Go a mess to integrate with other languages. The condescending attitude of the Go team on mailing lists didn’t help either.

                                                                                                                    1. 3

                                                                                                                      There is no value in verbosity, though. It’s a waste of characters. The entire attitude is an apology for the bare fact that Go doesn’t have error-handling syntax.

                                                                                                                      1. 11

                                                                                                                        What you label “verbosity” I see as “explicitness”. What to you is a lack of error-handling syntax is to me a simplification that normalizes execution paths.

                                                                                                                        It’s very clear to me that the people who dislike Go’s approach to error handling see errors as a first-class concept in language design, which deserves special accommodation at the language level. I get that. I understand the position and perspective. But this isn’t an objective truth, or something that is strictly correct. It’s just a perspective, a model, which has both costs and benefits. This much at least is hopefully noncontroversial. And Go makes the claim that, for the contexts which it targets, this model of error handling has more costs than benefits. If you want to object to that position then that’s fine. But it’s — bluntly — incorrect to claim that this is some kind of apologia, or that Go is missing things that it should objectively have.

                                                                                                                        1. 5

                                                                                                                          It often feels to me that people who complain about error handling in Go have never suffered dealing with throwing and catching exceptions in a huge codebase. At least in Go, you can be very explicit on how to handle errors (in particular, non-fatal ones) without the program trying to catapult you out of an escape hatch. Error handing is tedious in general, in any language. I don’t think Go’s approach is really any more tedious than anywhere else.

                                                                                                                          1. 5

                                                                                                                            Error handing is tedious in general, in any language. I don’t think Go’s approach is really any more tedious than anywhere else.

                                                                                                                            Yep, and a bit more — it brings the “tedium” forward, which is for sure a short term cost. But that cost creates disproportionate long-term benefits, as the explicitness reduces risk otherwise created by conveniences.

                                                                                                                        2. 5

                                                                                                                          The argument isn’t that verbosity has a value in itself – it doesn’t.

                                                                                                                          The argument is that if you have to choose between “simple, but concrete and verbose” and “more complex and abstract, but elegant”, it’s better to choose the former. It’s a statement about relative values. And you see it everywhere in Go. Think about the generics arguments:

                                                                                                                          People: “WTF! I have to rewrite my function for every fucking datatype!”.
                                                                                                                          Go: “What’s the big deal? It’s just some repeated code. Better than us bloating the language and making Go syntax more complex”

                                                                                                                          They caved on that one eventually, but the argument is still germane.

                                                                                                                          As I said, I don’t personally like all the decisions, and it’s not my favorite language, but once I got where they were coming from, I stopped hating it. The ethos has value.

                                                                                                                          It all stems from taking a hard line against over-engineering. The whole language is geared toward that. No inheritance. You don’t even get map! “Just use a for loop.” You only see the payoff of the philosophy in a large team setting, where you have many devs of varying experience levels working over years on something. The “Go way” isn’t so crazy there.

                                                                                                                    2. 3

                                                                                                                      Java included exceptions in the function signature and everyone hated those, even Kotlin made them optional. Just like how this Python developer has grown to enjoy types, I also enjoy the explicit throws declarations.

                                                                                                                      1. 3

                                                                                                                        you have to explicitly deal with any error-producing code you use.

                                                                                                                        Except if you forget to deal with it, forget to check for the error, or just drop it.

                                                                                                                      1. 6

                                                                                                                        A slightly related Go nit, the case of structure members determines whether they’re exported or not. It’s crazy, why not explicitly add a private keyword or something?

                                                                                                                        1. 19

                                                                                                                          why not explicitly add a private keyword or something?

                                                                                                                          Because capitalization does the same thing with less ceremony. It’s not crazy. It’s just a design decision.

                                                                                                                          1. 4

                                                                                                                            And limiting variable names to just “x”, “y” and “z” are also simpler and much less ceremony than typing out full variable names

                                                                                                                            1. 1

                                                                                                                              I’m not sure how this relates. Is your claim that the loss of semantic information that comes with terse identifiers is comparable to the difference between type Foo struct and e.g. type public foo struct?

                                                                                                                              1. 1

                                                                                                                                That is actually a Go convention, too. Two-letter or three-letter variable names like cs instead of customerService.

                                                                                                                            2. 6

                                                                                                                              This would be a more substantive comment chain if you can express why it’s crazy, not just calling it crazy. Why is it important that it should be a private keyword “or something”? In Go, the “or something” is literally the case sensitive member name…which is an explicit way of expressing whether it’s exported or not. How much more explicit can you get than a phenotypical designation? You can look at the member name and know then and there whether it’s exported. An implicit export would require the reader to look at the member name and at least one other source to figure out if it’s exported.

                                                                                                                              1. 7

                                                                                                                                It’s bad because changing the visibility of a member requires renaming it, which requires finding and updating every caller. This is an annoying manual task if your editor doesn’t do automatic refactoring, and it pollutes patches with many tiny one-character diffs.

                                                                                                                                It reminds me of old versions of Fortran where variables that started with I, J, K L or M were automatically integers and the rest were real. 🙄

                                                                                                                                1. 5

                                                                                                                                  M-x lsp-rename

                                                                                                                                  I don’t think of those changes as patch pollution — I think of them as opportunities to see where something formerly private is now exposed. E.g. when a var was unexported I knew that my package controlled it, but if I export it now it is mutable outside my control — it is good to see that in the diff.

                                                                                                                                  1. 2

                                                                                                                                    I guess I don’t consider changing the capitalization of a letter as renaming the variable

                                                                                                                                    1. 2

                                                                                                                                      That’s not the point. The point is you have to edit every place that variable/function appears in the source.

                                                                                                                                      1. 3

                                                                                                                                        I was going to suggest that gofmt‘s pattern rewriting would help here but it seems you can’t limit it to a type (although gofmt -r 'oldname -> Oldname' works if the fieldname is unique enough.) Then I was going to suggest gorename which can limit to struct fields but apparently hasn’t been updated to work with modules. Apparently gopls is the new hotness but testing that, despite the “it’ll rename throughout a package”, when I tested it, specifying main.go:9:9 Oldname only fixed it (correctly!) in main.go, not the other files in the main package.

                                                                                                                                        In summary, this is all a bit of a mess from the Go camp.

                                                                                                                                        1. 1

                                                                                                                                          It looks like rsc’s experimental “refactor” can do this - successfully renamed a field in multiple files for me with rf 'mv Fish.name Fish.Name'.

                                                                                                                                  2. 5

                                                                                                                                    The author of the submitted article wrote a sequel article, Go’ing Insane Part Two: Partial Privacy. It includes a section Privacy via Capitalisation that details what they find frustrating about the feature.

                                                                                                                                  3. 4

                                                                                                                                    A slightly related not-Go nit, the private keyword determines whether struct fields are exported or not. It’s crazy, why not just use the case of the field names saving everyone some keypresses?

                                                                                                                                    1. 2

                                                                                                                                      I really appreciate it, and find myself missing it on every other language. To be honest, I have difficulty understanding why folding would want anything else.

                                                                                                                                      1. 2

                                                                                                                                        On the contrary, I rather like that it’s obvious in all cases whether something is exported or not without having to find the actual definition.

                                                                                                                                      1. 35

                                                                                                                                        return err is almost always the wrong thing to do. Instead of:

                                                                                                                                        if err := foo(); err != nil {
                                                                                                                                        	return err
                                                                                                                                        }
                                                                                                                                        

                                                                                                                                        Write:

                                                                                                                                        if err := foo(); err != nil {
                                                                                                                                        	return fmt.Errorf("fooing: %w", err)
                                                                                                                                        }
                                                                                                                                        

                                                                                                                                        Yes, this is even more verbose, but doing this is what makes error messages actually useful. Deciding what to put in the error message requires meaningful thought and cannot be adequately automated. Furthermore, stack traces are not adequate context for user-facing, non-programming errors. They are verbose, leak implementation details, are disrupted by any form of indirection or concurrency, etc.

                                                                                                                                        Even with proper context, lots of error paths like this is potentially a code smell. It means you probably have broader error strategy problems. I’d try to give some advice on how to improve the code the author provided, but it is too abstract in order to provide any useful insights.

                                                                                                                                        1. 18

                                                                                                                                          I disagree on a higher level. What we really want is a stacktrace so we know where the error originated, not manually dispensed breadcrumbs…

                                                                                                                                          1. 32

                                                                                                                                            maybe you do, but I prefer an error chain that was designed. A Go program rarely has just one stack, because every goroutine is its own stack. Having the trace of just that one stack isn’t really a statement of the program as a whole since there’s many stacks, not one. Additionally, stack traces omit the parameters to the functions at each frame, which means that understanding the error means starting with your stack trace, and then bouncing all over your code and reading the code and running it in your head in order to understand your stack trace. This is even more annoying if you’re looking at an error several days later in a heterogeneous environment where you may need the additional complication of having to figure out which version of the code was running when that trace originated. Or you could just have an error like “failed to create a room: unable to reserve room in database ‘database-name’: request timed out” or something similar. Additionally, hand-crafted error chains have the effect that they are often much easier to understand for people who operate but don’t author something; they may have never seen the code before, so understanding what a stack trace means exactly may be difficult for them, especially if they’re not familiar with the language.

                                                                                                                                            1. 6

                                                                                                                                              I dunno. Erlang and related languages give you back a stack trace (with parameters) in concurrently running processes no problem

                                                                                                                                              1. 5

                                                                                                                                                It’s been ages since I wrote Erlang, but I remember that back then I rarely wanted a stack trace. My stack were typically 1-2 levels deep: each process had a single function that dispatched messages and did a small amount of work in each one. The thing that I wanted was the state of the process that had sent the unexpected message. I ended up with some debugging modes that attached the PID of the sending process and some other information so that I could reconstruct the state at the point where the problem occurred. This is almost the same situation as Go, where you don’t want the stack trace of the goroutine, you want to capture a stack trace of the program at the point where a goroutine was created and inspect that at the point where the goroutine failed.

                                                                                                                                                This isn’t specific to concurrent programs, though it is more common there, it’s similar for anything written in a dataflow / pipeline style. For example, when I’m debugging something in clang’s IR generation I often wish I could go back and see what had caused that particular AST node to be constructed during parsing or semantic analysis. I can’t because all of the state associated with that stack is long gone.

                                                                                                                                            2. 10

                                                                                                                                              FWIW, I wrote a helper that adds tracing information.

                                                                                                                                              I sort of have two minds about this. On the one hand, yeah, computers are good at tracking stack traces, why are we adding them manually and sporadically? OTOH, it’s nice that you can decide if you want the traces or not and it gives you the ability to do higher level things like using errors as response codes and whatnot.

                                                                                                                                              The thing that I have read about in Zig that I wish Go had is an error trace which is different from the stack trace, which shows how the error was created, not the how the error propagates back to the execution error boundary which is not very interesting in most scenarios.

                                                                                                                                              1. 7

                                                                                                                                                The nice thing about those error traces is that they end where the stack trace begins, so it’s seamless to the point that you don’t even need to know that they are a thing, you just get exactly the information that otherwise you would be manually looking for.

                                                                                                                                              2. 8

                                                                                                                                                In a multiprocess system that’s exchanging messages: which stack?

                                                                                                                                                1. 2

                                                                                                                                                  see: erlang

                                                                                                                                                2. 5

                                                                                                                                                  You don’t want stack traces; you want to know what went wrong.

                                                                                                                                                  A stack trace can suggest what may have gone wrong, but an error message that declares exactly what went wrong is far more valuable, no?

                                                                                                                                                  1. 8

                                                                                                                                                    An error message is easy, we already have that: “i/o timeout”. A stack trace tells me the exact code path that lead to that error. Building up a string of breadcrumbs that led to that timeout is just a poorly implemented, ad-hoc stack trace.

                                                                                                                                                    1. 5

                                                                                                                                                      Indeed and I wouldn’t argue with that. I love a good stack trace, but I find they’re often relied upon in lieu of useful error messages and I think that’s a problem.

                                                                                                                                                      1. 2

                                                                                                                                                        Building up a string of breadcrumbs that led to that timeout is just a poorly implemented, ad-hoc stack trace.

                                                                                                                                                        That’s a bit of an over-generalization. A stack trace is inherently a story about the construction of the program that originated the error, while an error chain is a story about the events that led to an error. A stack trace can’t tell you what went wrong if you don’t have access to the program’s source code in the way that a hand crafted error chain can. A stack trace is more about where an error occurred, while an error chain is more about why an error occurred. I think they’re much more distinct than you are suggesting.

                                                                                                                                                        and of course, if people are just bubbling up errors without wrapping them, yeah you’re going to have a bad time, but I think attacking that case is like suggesting that every language that has exceptions encourages Pokémon exception handling. That’s a bad exception-handling pattern, but I don’t think that the possibility of this pattern is a fair indictment of exceptions generally. Meanwhile you’re using examples of bad error handling practices that are not usually employed by Go programmers with more than a few weeks experience to indict the entire paradigm.

                                                                                                                                                    2. 4

                                                                                                                                                      Stack traces are expensive to compute and inappropriate to display to most users. Also, errors aren’t exceptions.

                                                                                                                                                      1. 1

                                                                                                                                                        That’s why Swift throws errors instead. Exceptions immediately abort the program.

                                                                                                                                                      2. 3

                                                                                                                                                        What really is the “origin” of an error? Isn’t that somewhat arbitrary? If the error comes from a system call, isn’t the origin deeper in the kernel somewhere? What if you call in to a remote, 3rd party service. Do you want the client to get the stack trace with references to the service’s private code? If you’re using an interface, presumably the purpose is to abstract over the specific implementation. Maybe the stack trace should be truncated at the boundary like a kernel call or API call?

                                                                                                                                                        Stack traces are inherently an encapsulation violation. They can be useful for debugging your internals, but they are an anti-feature for your users debugging their own system. If your user sees a stack trace, that means your program is bugged, not theirs.

                                                                                                                                                        1. 5

                                                                                                                                                          I get a line of logging output: error: i/o timeout. What do I do with that? With Ruby, I get a stack trace which tells me exactly where the timeout came from, giving me a huge lead on debugging the issue.

                                                                                                                                                          1. 6

                                                                                                                                                            I get a line of logging output: error: i/o timeout. What do I do with that?

                                                                                                                                                            Well, that’s a problem you fix by annotating your errors properly. You don’t need stack traces.

                                                                                                                                                            1. 3

                                                                                                                                                              When your Ruby service returns an HTTP 500, do you send me the stack trace in the response body? What do I do with that?

                                                                                                                                                              Go will produce stack traces on panics as well, but that’s precisely the point here: these are two different things. Panics capture stack traces as a “better than nothing” breadcrumb trail for when the programmer has failed to account for a possibility. They are for producers of code, not consumers of it.

                                                                                                                                                            2. 2

                                                                                                                                                              There’s definitely competing needs between different audiences and environments here.

                                                                                                                                                              A non-technical end user doesn’t want to see anything past “something went wrong on our end, but we’re aware of it”. Well, they don’t even want to see that.

                                                                                                                                                              A developer wants to see the entire stack trace, or at least have it available. They probably only care about frames in their own code at first, and maybe will want to delve into library code if the error truly doesn’t seem to come from their code or is hard to understand in the first place.

                                                                                                                                                              A technical end user might want to see something in-between: they don’t want to see “something was wrong”. They might not even want to see solely the outer error of “something went wrong while persisting data” if the root cause was “I couldn’t reach this host”, because the latter is something they could actually debug within their environment.

                                                                                                                                                          2. 9

                                                                                                                                                            This is one reason I haven’t gone back to Go since university - There’s no right way to do anything. I think I’ve seen a thousand different right ways to return errors.

                                                                                                                                                            1. 10

                                                                                                                                                              Lots of pundits say lots of stuff. One good way to learn good patterns (I won’t call them “right”), is to look at real code by experienced Go developers. For instance, if you look at https://github.com/tailscale/tailscale you’ll find pervasive use of fmt.Errorf. One thing you might not see – at least not without careful study – is how to handle code with lots of error paths. That is by it’s very nature harder to see because you have to read and understand what the code is trying to do and what has to happen when something goes wrong in that specific situation.

                                                                                                                                                              1. 6

                                                                                                                                                                there is a right way to do most things; but it takes some context and understanding for why.

                                                                                                                                                                the mistake is thinking go is approachable for beginners; it’s not.

                                                                                                                                                                go is an ergonomic joy for people that spend a lot of time investing in it, or bring a ton of context from other languages.

                                                                                                                                                                for beginners with little context, it is definitely a mess.

                                                                                                                                                                1. 9

                                                                                                                                                                  I thought Go was for beginners, because Rob Pike doesn’t trust programmers to be good.

                                                                                                                                                                  1. 19

                                                                                                                                                                    I’d assume that Rob Pike, an industry veteran, probably has excellent insight into precisely how good the average programmer at Google is, and what kind of language will enable them to be productive at the stuff Google makes. If this makes programming languages connaisseurs sad, that’s not his problem.

                                                                                                                                                                    1. 9

                                                                                                                                                                      Here’s the actual quote:

                                                                                                                                                                      The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

                                                                                                                                                                      So I have to wonder who is capable of understanding a “brilliant language” …

                                                                                                                                                                      1. 8

                                                                                                                                                                        So I have to wonder who is capable of understanding a “brilliant language” …

                                                                                                                                                                        Many people. They don’t work at Google at an entry-level capacity, that’s all.

                                                                                                                                                                        There’s a subtle fallacy at work here - Google makes a lot of money, so Google can afford to employ smart people (like Rob Pike!) It does not follow that everyone who works at Google is, on average, smarter than anyone else.

                                                                                                                                                                        (edited to include quote)

                                                                                                                                                                        1. 8

                                                                                                                                                                          Let’s say concretely we are talking about OCaml. Surely entry-level Googlers are capable of understanding OCaml. Jane Street teaches it to all new hires (devs or not) in a two-week bootcamp. I’ve heard stories of people quickly becoming productive in Elm too.

                                                                                                                                                                          The real meaning of that quote is not ‘entry-level Googlers are not capable of it’, it’s ‘We don’t trust them with it’ and ‘We’re not willing to invest in training them in it’. They want people to start banging out code almost instantly, not take some time to ramp up.

                                                                                                                                                                          1. 8

                                                                                                                                                                            Let’s say concretely we are talking about OCaml. Surely entry-level Googlers are capable of understanding OCaml. Jane Street teaches it to all new hires (devs or not) in a two-week bootcamp.

                                                                                                                                                                            I suspect that Jane Street’s hiring selects for people who are capable of understanding OCaml; I guarantee that the inverse happens and applicants interested in OCaml self select for careers at Jane Street, just like Erlang-ers used to flock towards Ericsson.

                                                                                                                                                                            Google has two orders of magnitude more employees than Jane Street. It needs a much bigger funnel and is likely far less selective in hiring. Go is “the law of large numbers” manifest as a programming language. That’s not necessarily bad, just something that is important for a massive software company and far less important for small boutiques.

                                                                                                                                                                            1. 3

                                                                                                                                                                              And I remember when Google would require at minimum a Masters Degree before hiring.

                                                                                                                                                                              1. 1

                                                                                                                                                                                I had a master’s degree in engineering (though not in CS) and I couldn’t code my way out of a paper bag when I graduated. Thankfully no-one cared in Dot Com Bubble 1.0!

                                                                                                                                                                              2. 2

                                                                                                                                                                                applicants interested in OCaml self select for careers at Jane Street,

                                                                                                                                                                                As I said, they teach it to all hires, including non-devs.

                                                                                                                                                                                Google has two orders of magnitude more employees than Jane Street. It needs a much bigger funnel and is likely far less selective in hiring

                                                                                                                                                                                Surely though, they are not so loose that they hire Tom Dick and Harry off the street. Why don’t we actually look at an actual listing and check? E.g. https://careers.google.com/jobs/results/115367821606560454-software-developer-intern-bachelors-summer-2022/

                                                                                                                                                                                Job title: Software Developer Intern, Bachelors, Summer 2022 (not exactly senior level)

                                                                                                                                                                                Minimum qualifications:

                                                                                                                                                                                Pursuing a Bachelor’s degree program or post secondary or training experience with a focus on subjects in software development or other technical related field. Experience in Software Development and coding in a general purpose programming language. Experience coding in two of C, C++, Java, JavaScript, Python or similar.

                                                                                                                                                                                I’m sorry but there’s no way I’m believing that these candidates would be capable of learning Go but not OCaml (e.g.). It’s not about their capability, it’s about what Google wants to invest in them. Another reply even openly admits this! https://lobste.rs/s/yjvmlh/go_ing_insane_part_one_endless_error#c_s3peh9

                                                                                                                                                                              3. 3

                                                                                                                                                                                They want people to start banging out code almost instantly, not take some time to ramp up.

                                                                                                                                                                                Yes, and? The commodification of software developers is a well-known trend (and goal) of most companies. When your assets are basically servers, intangible assets like software and patents, and the people required to keep the stuff running, you naturally try to lower the costs of hiring and paying salary, just like you try to have faster servers and more efficient code.

                                                                                                                                                                                People are mad at Rob Pike, but he just made a language for Google. It’s not his fault the rest of the industry thought “OMG this is the bee’s knees, let’s GO!” and adopted it widely.

                                                                                                                                                                                1. 1

                                                                                                                                                                                  Yes, I agree that the commodification of software developers is prevalent today. And we can all see the result, the profession is in dire straits–hard to hire because of bonkers interview practices, hard to keep people because management refuses to compensate them properly, and cranking out bugs like no tomorrow.

                                                                                                                                                                                2. 3

                                                                                                                                                                                  on the contrary, google provides a ton of ramp up time for new hires because getting to grips with all the internal infrastructure takes a while (the language is the least part of it). indeed, when I joined a standard part of the orientation lecture was that whatever our experience level was, we should not expect to be productive any time soon.

                                                                                                                                                                                  what go (which I do not use very much) might be optimising for is a certain straightforwardness and uniformity in the code base, so that engineers can move between projects without having to learn essentially a new DSL every time they do.

                                                                                                                                                                                  1. 1

                                                                                                                                                                                    You may have a misconception that good programming languages force people to ‘essentially learn a new DSL’ in every project. In any case, as you yourself said, the language is the least part of the ramp-up of a new project, so even if that bit were true, it’s still optimizing for the wrong thing.

                                                                                                                                                                                    1. 1

                                                                                                                                                                                      no, you misunderstood what i was getting at. i was saying that go was optimisng for straightforwardness and uniformity so that there would be less chance of complex projects evolving their own way of doing things, not that better languages would force people to invent their own DSLs per project.

                                                                                                                                                                                      also the ramp-up time i was referring to was for new hires; a lot of google’s internal libraries and services are pretty consistently used across projects (and even languages via bindings and RPC) so changing teams requires a lot less ramp up than joining google in the first place.

                                                                                                                                                                                      1. 1

                                                                                                                                                                                        i was saying that go was optimisng for straightforwardness and uniformity so that there would be less chance of complex projects evolving their own way of doing things,

                                                                                                                                                                                        Again, the chances of that happening are not really as great as the Go people seem to be afraid it is, provided we are talking about a reasonable, good language. So let’s say we leave out Haskell or Clojure. The fear of language-enabled complexity seems pretty overblown to me. Especially considering the effort put into the response, creating an entirely new language and surrounding ecosystem.

                                                                                                                                                                          2. 9

                                                                                                                                                                            No, Rob observed, correctly, that in an organization of 10,000 programmers, the skill level trends towards the mean. And so if you’re designing a language for this environment, you have to keep that in mind.

                                                                                                                                                                            1. 4

                                                                                                                                                                              it’s not just that. It’s a language that has to reconcile the reality that skill level trends toward the mean, with the fact that the way that google interviews incurs a selection/survival bias towards very junior programmers who think they are the shit, and thus are very dangerous with the wrong type of power.

                                                                                                                                                                              1. 4

                                                                                                                                                                                As I get older and become, presumably, a better programmer, it really does occur to me just how bad I was for how long. I think because I learned how to program as a second grader, I didn’t get how much of a factor “it’s neat he can do it all” was in my self-assessment. I was pretty bad, but since I was being compared to the other kids who did zero programming, it didn’t matter that objectively I was quite awful, and I thought I was hot shit.

                                                                                                                                                                              2. 4

                                                                                                                                                                                Right! But the cargo-cult mentality of the industry meant that a language designed to facilitate the commodification of software development for a huge, singular organization escaped and was inflicted on the rest of us.

                                                                                                                                                                                1. 4

                                                                                                                                                                                  But let’s be real for a moment:

                                                                                                                                                                                  a language designed to facilitate the commodification of software development

                                                                                                                                                                                  This is what matters.

                                                                                                                                                                                  It doesn’t matter if you work for a company of 12 or 120,000: if you are paid to program – that is, you are not a founder – the people who sign your paychecks are absolutely doing everything within their power to make you and your coworkers just cogs in the machine.

                                                                                                                                                                                  So I don’t think this is a case of “the little fish copying what big bad Google does” as much as it is an essential quality of being a software developer.

                                                                                                                                                                                  1. 1

                                                                                                                                                                                    Thank you, yes. But also, the cargo cult mentality is real.

                                                                                                                                                                              3. 3

                                                                                                                                                                                Go is for compilers, because Google builds a billion lines a day.

                                                                                                                                                                          3. 2

                                                                                                                                                                            return errors.Wrapf(err, "fooing %s", bar) is a bit nicer.

                                                                                                                                                                            1. 13

                                                                                                                                                                              That uses the non-standard errors package and has been obsolete since 1.13: https://stackoverflow.com/questions/61933650/whats-the-difference-between-errors-wrapf-errors-errorf-and-fmt-errorf

                                                                                                                                                                              1. 1

                                                                                                                                                                                Thanks, that’s good to know.

                                                                                                                                                                              2. 8

                                                                                                                                                                                return fmt.Errorf("fooing %s %w", bar, err) is idiomatic.

                                                                                                                                                                                1. 9

                                                                                                                                                                                  Very small tweak: normally you’d include a colon between the current message and the %w, to separate error messages in the chain, like so:

                                                                                                                                                                                  return fmt.Errorf("fooing %s: %w", bar, err)
                                                                                                                                                                                  
                                                                                                                                                                              3. 1

                                                                                                                                                                                It makes error messages useful but if it returns a modified err then I can’t catch it further up with if err == someErr, correct?

                                                                                                                                                                                1. 2

                                                                                                                                                                                  You can use errors.Is to check wrapped errors - https://pkg.go.dev/errors#Is

                                                                                                                                                                                  Is unwraps its first argument sequentially looking for an error that matches the second. It reports whether it finds a match. It should be used in preference to simple equality checks

                                                                                                                                                                                  1. 2

                                                                                                                                                                                    Thanks! I actually didn’t know about that.

                                                                                                                                                                                  2. 2

                                                                                                                                                                                    Yes, but you can use errors.Is and errors.As to solve that problem. These use errors.Unwrap under the hood. This error chaining mechanism was introduced in Go 1.13 after being incubated in the “errors” package for a long while before that. See https://go.dev/blog/go1.13-errors for details.

                                                                                                                                                                                1. 20

                                                                                                                                                                                  It’d be nice to have some actual background on hashing in here instead of just broad generalizations and links to various hash functions. Examples:

                                                                                                                                                                                  • There’s no mention of cyclic redundancy checks and why they are not valid as crypto functions (a mistake some programmers have made).
                                                                                                                                                                                  • There’s no mention of avalanche effects, which is a good way of seeing how “random” a digest scheme is (with some implications for how well the output can be predicted/controlled by an attacker).
                                                                                                                                                                                  • The mentioned attack on JSON hash tables in PHP (if you dig into it) would’ve been a great place to talk about trivial hashes (e.g., f(x) =0 or f(x)=x) and why they cause problems even in non-hostile environments, but that would’ve required more of an introduction to how hashing works…)
                                                                                                                                                                                  • Lots of usage of jargon like “non-invertible”, “collision-resistance”, “preimage attack resistance”, etc. which is probably inaccessible if your audience is programmers who “don’t understand hash functions”.
                                                                                                                                                                                  • There’s not really an explanation about the differences/similarities of crypto-strong hash functions, password hash functions, and key derivation functions, other than a mention that there is some relation but which isn’t elaborated on at all.
                                                                                                                                                                                  • There’s not really any useful information at all about perceptual hashing vs other forms of multimedia digest approaches–there’s just some Apple hate.
                                                                                                                                                                                  • etc.

                                                                                                                                                                                  Programmers might not understand hash functions, but infosec furries may also not understand pedagogy.

                                                                                                                                                                                  (also, can you please cool it with the inflammatory article headlines?)

                                                                                                                                                                                  1. 24

                                                                                                                                                                                    Programmers might not understand hash functions, but infosec furries may also not understand pedagogy.

                                                                                                                                                                                    Please don’t pick a fight. It seems more angry than friendly.

                                                                                                                                                                                    1. 22

                                                                                                                                                                                      Honestly I think it’s a valid concern. One of the biggest problems with the computer security world, as stated repeatedly by leading experts in the field, is communication and teaching.

                                                                                                                                                                                      1. 23

                                                                                                                                                                                        A valid concern would be “infosec experts may not understand pedagogy” but why call out “infosec furries” specifically? Unless we should be concerned about infosec furries in particular vs other infosec experts?

                                                                                                                                                                                        Are these acceptable?

                                                                                                                                                                                        • but infosec gays may also not understand pedagogy
                                                                                                                                                                                        • but infosec women may also not understand pedagogy
                                                                                                                                                                                        • but infosec people of color may also not understand pedagogy

                                                                                                                                                                                        No. So why furries? People need to get over it and quit furry bashing. This isn’t acceptable behavior on Lobste.rs, and I’m tired of it.

                                                                                                                                                                                        1. 3

                                                                                                                                                                                          See elsewhere for the explanation; furry bashing doesn’t enter into it, though I see why you might have read it that way. Furries are internet denizens like the rest of us, with all that entails.

                                                                                                                                                                                          1. 12

                                                                                                                                                                                            I agree with you that it’s a bad title.

                                                                                                                                                                                            I also think that you wouldn’t have reacted nearly this strongly to the title if it wasn’t a furry blog.

                                                                                                                                                                                            1. 11

                                                                                                                                                                                              I read your other comments. But you said what you said, and that undermines all your pontificating about the harm of “insulting/demeaning a group” and “the sort of microaggression/toxicity that everybody talks so much about.” Take your own advice.

                                                                                                                                                                                            2. 2

                                                                                                                                                                                              “Furry” is a kink, not an identity or protected class. And normally you have to get people’s consent before you bring them into your kink.

                                                                                                                                                                                              1. 7

                                                                                                                                                                                                I don’t see any sexual imagery in this blog post.

                                                                                                                                                                                                1. 2

                                                                                                                                                                                                  The OP’s site has some pretty well reasoned and presented articles on precisely why “furry” cannot reasonably be summarized as “a kink”.

                                                                                                                                                                                                  And, no, you do not “normally” have to get someone’s consent to introduce them to the idea of your kink, unless said introduction involves you engaging them in the practice of your kink.

                                                                                                                                                                                                2. 1

                                                                                                                                                                                                  Sorry, I didn’t realize the “furry” part was what you were opposed to. It sounded like you were upset with the implication that the infosec world is bad at teaching.

                                                                                                                                                                                            3. 6

                                                                                                                                                                                              Programmers might not understand hash functions, but infosec furries may also not understand pedagogy.

                                                                                                                                                                                              (also, can you please cool it with the inflammatory article headlines?)

                                                                                                                                                                                              https://www.youtube.com/watch?v=S2xHZPH5Sng

                                                                                                                                                                                              1. 10

                                                                                                                                                                                                One of the things he talks about there is testing the hypothesis and seeing which title actually worked. I only clicked this link because I recognized your domain name and knew you had written interesting articles in the past and might legitimately explain something I didn’t know. If not for that, I probably would have bypassed it since the title alone was not interesting at all.

                                                                                                                                                                                                1. 9

                                                                                                                                                                                                  Even so, it is still possible to write clickbait titles that aren’t predicated on insulting/demeaning a group.

                                                                                                                                                                                                  • “Hash functions: hard or just misunderstood?”
                                                                                                                                                                                                  • “Things I wish more programmers knew about hashes”
                                                                                                                                                                                                  • “Programmer hashes are not infosec hashes”
                                                                                                                                                                                                  • “Are you hashing wrong? It’s more common than you might think”
                                                                                                                                                                                                  • “uwu whats this notices ur hash function

                                                                                                                                                                                                  How would you feel if I wrote “Gay furries don’t understand blog posting”? Even if I raise good points, and even if more people would click on it (out of outrage, presumably), it would still probably annoy a gay furry who wrote blogs and they’d go in with their hackles raised.

                                                                                                                                                                                                  1. 8

                                                                                                                                                                                                    The important difference between what I wrote and your hypothetical is the difference between punching up and punching down.

                                                                                                                                                                                                    My original title was along the same lines as “Falsehoods Programmers Believe About _____” but I’ve grown a distaste for the cliche.

                                                                                                                                                                                                    1. 7

                                                                                                                                                                                                      The difference between “Programmers don’t understand hash functions” and “Gay furries don’t understand blog posting” is quite obvious to me and I definitely don’t want to engage in whatever Internet flame is going on here. Especially since, uh, I have a preeetty good idea about what the problem here is, and I tend to think it’s about gay furries, not article titles, which is definitely not a problem that I have. (This should probably be obvious but since I’m posting in this particular thread, I wanted to make sure :P).

                                                                                                                                                                                                      But I also think this title really is needlessly nasty, independent of how it might be titled if it were about other audiences. It’s a bad generalisation – there are, in fact, plenty of programmers who understand hash functions – and it’s not exactly encouraging to those programmers who want to get into security, or who think their understanding of these matters is insufficient.

                                                                                                                                                                                                      I am (or was?) one of them – this was an interest of mine many, many years ago, at a time when I was way too young to understand the advanced math. My career took me elsewhere, and not always where I wanted to go, and I tried to keep an eye on these things in the hope that maybe one day it’ll take me there. Needless to say, there’s only so much you can learn about these topics by spending a couple of evenings once in a blue moon studying them, so I never really got to be any good at it. So I think the explanation is amazing, but it would definitely benefit from not reminding me of my inadequacy.

                                                                                                                                                                                                      And I’m in a happy boat, actually, this is only an interest of mine – but there are plenty of people who have to do it as part of their jobs, are not provided with adequate training of any kind, have no time to figure it out on their own, and regularly get yelled at when they get it wrong.

                                                                                                                                                                                                      Now, I realise the title is tongue-in-cheek to some degree, the playful furries and the clever humour scattered throughout the post sort of gives it away. If you think about it for a moment it’s pretty clear that this is meant to grab attention, not remind people how much they suck. But it’s worth remembering that, in an age where web syndication is taken for granted to the point where it sounds like a Middle English term, this context isn’t carried everywhere. Case in point, this lobste.rs page includes only the title. Some people might react to it by clicking because you grabbed their attention, but others might just say yeah, thanks for reminding me, I’ll go cry in a corner.

                                                                                                                                                                                                      Even if I didn’t realise it was tongue-in-cheek, it probably wouldn’t bother me, partly because I understand how writing “competitively” works (ironically, from around the same time), partly because I’ve developed a thick skin, and partly because, honestly, I’ve kindda given up on it, so I don’t care about it as much as I once did. But I can see why others would not feel the same way at all. You shouldn’t count on your audience having a thick skin or being old enough to have given up on most of their dreams anyway.

                                                                                                                                                                                                      I know this is a real struggle because that’s just how blogs and blogging work today. You have to compete for attention to some degree, and this is particularly important when a large part of the technical audience is “confined” to places like HN and lobste.rs, where you have to grab attention through the title because there’s nothing else to grab attention through. But maybe you can find a kinder way to grab it, I dunno, maybe a clever pun? That never hurt anyone. These radical, blunt (supposedly “bluntly honest” but that’s just wishful thinking) headlines are all the rage in “big” Internet media because, just like Internet trolls, they thrive on controversy, us vs. them and a feeling of smugness, but is that really the kind of thing you want to borrow?

                                                                                                                                                                                                      (Edit: just to make sure I get the other part of my message across, because I think it’s even more important: title aside, which could be nicer, the article was super bloody amazing: the explanation’s great, and I like the additional pointers, and the humour, and yes, the drawings! Please don’t take any of all that stuff above as a criticism of some sort: I wanted to present a different viewpoint from which the title might read differently than you intended, not that the article is bad. It’s not!)

                                                                                                                                                                                                      1. 15

                                                                                                                                                                                                        How do you know that you’re punching up?

                                                                                                                                                                                                        What if the person encountering your blog is a programmer from an underrepresented background, just barely overcoming imposter syndrome, and now here’s this scary suggestion that they don’t understand hash functions? What if they actually made one of the mistakes in the article, and feel like they’re a complete fraud, and should leave the industry? This is the sort of microaggression/toxicity that everybody talks so much about, if I’m not mistaken.

                                                                                                                                                                                                        The point is: you don’t know. You can’t know.

                                                                                                                                                                                                        So, err on the side of not adding more negative shit to the world accidentally in the name of pageviews–especially when there are many, many other more positive options in easy reach.

                                                                                                                                                                                                        EDIT:

                                                                                                                                                                                                        I wouldn’t care if it weren’t for the fact that you’re a smart dude and clearly passionate about your work and that you have good knowledge to share, and that it pains me to see somebody making mistakes I’ve made in the past.

                                                                                                                                                                                                        1. 8

                                                                                                                                                                                                          I wouldn’t care if it weren’t for the fact that you’re a smart dude and clearly passionate about your work

                                                                                                                                                                                                          I’m neither of those things :P

                                                                                                                                                                                                          and that you have good knowledge to share, and that it pains me to see somebody making mistakes I’ve made in the past.

                                                                                                                                                                                                          I appreciate your compassion on this subject. It’s definitely new territory for me (since forever I’ve been in the “boring headline out of clickbait adversion” territory).

                                                                                                                                                                                                          1. 9

                                                                                                                                                                                                            Do you actually not see a difference between saying a slightly negative thing about people of a certain profession and how they engage in that profession, and an ad-hominem using sexual orientation? What a weird and bad analogy?

                                                                                                                                                                                                            I’m trying to assume good intent here but all your comments make it sound like you’re annoyed at the furry pics and awkwardly trying to use cancel culture to lash out the author.

                                                                                                                                                                                                            1. 7

                                                                                                                                                                                                              Neither the label of programmers (with which I identify) nor of gay furries (with which the author identifies, according to their writing) is being misapplied. I’m sorry you feel that a plain statement of fact is somehow derogatory–there is nothing wrong with being a proud programmer or a proud gay furry.

                                                                                                                                                                                                              My point in giving that example was to critique the used construction of “ is ”. I picked that label because the author identified with it, and I picked the “bad at blogging” because it’s pretty obviously incorrect in its bluntness. If I had picked “lobsters” or “internet randos” the conjured association for the person I was in discussion with may not have had the same impact it that “programmers” had on me, so I went with what seemed reasonable.

                                                                                                                                                                                                              1. 4

                                                                                                                                                                                                                What do you gain by emphasizing soatok’s sexual identity, other than this morass of objections?

                                                                                                                                                                                                              2. 5

                                                                                                                                                                                                                I’m trying to assume good intent here

                                                                                                                                                                                                                that’s exactly what friendlysock is hoping for

                                                                                                                                                                                                                1. 5

                                                                                                                                                                                                                  you’re right but it’s best not to feed them

                                                                                                                                                                                                                2. 8

                                                                                                                                                                                                                  What if the person encountering your blog is a programmer from an underrepresented background, just barely overcoming imposter syndrome, and now here’s this scary suggestion that they don’t understand hash functions?

                                                                                                                                                                                                                  Or they may read this and think ‘I’m glad it’s not just me!’. As a programmer who probably has a better than average understanding of hash functions, I don’t feel demeaned by this generalisation, if I were worried about my level of understanding I’d feel comforted by the idea that I wasn’t in a minority in my lack of understanding.

                                                                                                                                                                                                                  What if they actually made one of the mistakes in the article, and feel like they’re a complete fraud, and should leave the industry?

                                                                                                                                                                                                                  Or they may feel better that this mistake is so common that someone writes about it on a list of mistakes programmers make.

                                                                                                                                                                                                                  1. 1

                                                                                                                                                                                                                    What if the person encountering your blog is a programmer from an underrepresented background….

                                                                                                                                                                                                                    While I said you’re picking a fight (and would add: “look at the thread, it’s a fight”), I see what you’re saying in this paragraph. I also value non-judgmental explanations.

                                                                                                                                                                                                                3. 6

                                                                                                                                                                                                                  My problem with the title isn’t that it’s insulting, but that it’s inaccurate. Clearly some programmers do understand hash functions, even if other programmers do not. If nothing else, @soatok, a programmer, presumably understands hash functions, or why else would he write a blog post purporting to explain the right way to use them?

                                                                                                                                                                                                                  Programmers don’t understand hash functions, and I can demonstrate this to most of the people that will read this with a single observation:

                                                                                                                                                                                                                  When you saw the words “hash function” in the title, you might have assumed this was going to be a blog post about password storage.

                                                                                                                                                                                                                  Specifically is wrong, at least about me, and almost certainly among other programmers as well. I don’t claim to have deep knowledge about cryptography, and I do expect that there’s probably something I could learn from this blog post, which I will read more carefully when I have a chance. But I am aware that the computer science concept of hash functions is useful for a variety of programming problems, and not just storing password-related data.

                                                                                                                                                                                                            1. 24

                                                                                                                                                                                                              Upgrading @golang versions is actually a pleasurable task for me:

                                                                                                                                                                                                              1. I’m 99% sure nothing will break.
                                                                                                                                                                                                              2. Speedups of 5-10% are common.
                                                                                                                                                                                                              3. New compiler or vet warnings tell me how to improve my code.
                                                                                                                                                                                                              4. Excellent release notes.

                                                                                                                                                                                                              Does any other language get this as right?

                                                                                                                                                                                                              1. 7

                                                                                                                                                                                                                Go’s secret sauce is that they never† break BC. There’s nothing else where you can just throw it into production like that because you don’t need to check for deprecations and warnings first.

                                                                                                                                                                                                                † That said, 1.17 actually did break BC for security reasons. If you were interpreting URL query parameters so that ?a=1&b=2 and ?a=1;b=2 are the same, that’s broken now because they removed support for semicolons for security reasons. Seems like the right call, but definitely one of the few times where you could get bitten by Go.

                                                                                                                                                                                                                Another issue is that the language and standard library has a compatibility guarantee, but the build tool does not, so e.g. if you didn’t move to modules, that can bite you. Still, compared to Python and Node, it’s a breath of fresh air.

                                                                                                                                                                                                                1. 2

                                                                                                                                                                                                                  I’ve been upgrading since 1.8 or so. There have been (rarely) upgrades that broke my code, but it was always for a good reason and easy to fix. None in recent memory.

                                                                                                                                                                                                                  1. 1

                                                                                                                                                                                                                    Are semicolons between query params a common practice? I’ve never heard of this before.

                                                                                                                                                                                                                    1. 2

                                                                                                                                                                                                                      No, which is why they removed it. It was in an RFC which was why it was added in the first place.

                                                                                                                                                                                                                    2. 1

                                                                                                                                                                                                                      1.16 or 1.15 also broke backwards compatibility with the TLS ServerName thing.

                                                                                                                                                                                                                    3. 4

                                                                                                                                                                                                                      Java is damn good about backward compatibility.

                                                                                                                                                                                                                      From what I recall, their release notes are pretty good as well.

                                                                                                                                                                                                                      1. 3

                                                                                                                                                                                                                        I had a different experience, going from Java 8 to Java 11 broke countless libraries for me. Especially bad is that they often break at run- and not at compile time.

                                                                                                                                                                                                                        1. 2

                                                                                                                                                                                                                          As someone with just a little experience with Go, what’s the situation with dependencies? In Java and maven, it becomes a nightmare with exclusions when one wants to upgrade a dependency, as transitive dependencies might then clash.

                                                                                                                                                                                                                          1. 3

                                                                                                                                                                                                                            It’s a bit complicated, but the TL;DR is that Go 1.11 (this is 1.17, recall) introduced “modules” which is the blessed package management system. It’s based on URLs (although weirdly, it’s github.com, not com.github, hmm…) that tell the system where to download external modules. The modules are versioned by git tags (or equivalent for non-git SCMs). Your package can list the minimum versions of external packages it wants and also hardcode replacement versions if you need to fork something. The expectation is that if you need to break BC as a library author, you will publish your package with a new URL, typically by adding v2 or whatever to the end of your existing URL. Package users can import both github.com/user/pkg/v1 and github.com/user/pkg/v2 into the same program and it will run both, but if you want e.g. both v1 and v1.5 in the same application, you’re SOL. It’s extremely opinionated in that regard, but I haven’t run into any problems with it.

                                                                                                                                                                                                                            Part of the backstory is that before Go modules, you were just expected to never break BC as a library author because there was no way to signal it downstream. When they switched to modules, Russ Cox basically tried to preserve that property by requiring URL changes for new versions.

                                                                                                                                                                                                                            1. 2

                                                                                                                                                                                                                              The module name and package ImportPath are not required to be URLs. Them being a URL is overloading done by go get. Nothing in the language spec requires them to be URLs.

                                                                                                                                                                                                                              1. 2

                                                                                                                                                                                                                                Yes, but I said “TL;DR” so I had to simplify.

                                                                                                                                                                                                                            2. 2

                                                                                                                                                                                                                              I also have only a little experience with Go. I have not yet run into frustrations with dependencies via Go modules.

                                                                                                                                                                                                                              Russ Cox gave a number of great articles talking about how Go’s dependency management solves problems with transitive dependencies. I recall this one being very good (https://research.swtch.com/vgo-import). It also calls out a constraint that programmers must follow:

                                                                                                                                                                                                                              In Go, if an old package and a new package have the same import path, the new package must be backwards compatible with the old package.

                                                                                                                                                                                                                              Is this constraint realistic and followed by library authors? If not, you’re going to run into problems with Go modules.

                                                                                                                                                                                                                              I’ve run into dependency hell in: Java, JavaScript, Python, and PHP – In every programming language I’ve had to do major development in. It’s a hard problem to solve!

                                                                                                                                                                                                                                1. 1

                                                                                                                                                                                                                                  Is this constraint realistic and followed by library authors? If not, you’re going to run into problems with Go modules.

                                                                                                                                                                                                                                  It is (obviously) not realistic for most software produced in the world.

                                                                                                                                                                                                                              1. 1

                                                                                                                                                                                                                                I strongly agree. The first time major stuff broke was Java 9, which is exceedingly recent, and wasn’t an LTS. And that movement has more in common with the Go 2 work than anything else, especially as Java 8 continues to be fully supported.