Threads for peterbourgon

  1. 5

    What’s wrong with dicts?

    I think the points author made are acceptable trade offs for faster development speed.

    It does not matter that refactoring a function is hard when each iteration takes a lot less time vs trying to create a concrete OOP structure around things.

    So pick the tight tradeoffs that meet your business needs. Dont refactor for the fear of ‘things will be hard in the future’, because YAGNI

    1. 5

      It does not matter that refactoring a function is hard when each iteration takes a lot less time vs trying to create a concrete OOP structure around things.

      I’ve never once seen a codebase — edit: maintained by more than one person — where untyped kwargs were net beneficial to product velocity vs. typed arguments.

      1. 3

        And I’ve seen (and worked on) several like that. Anecdote vs. anecdote?

        P.S. Those code bases also had 100% test coverage (or close to it), which I’m also hearing is nigh impossible and is too much effort.

        1. 3

          Yeah, tests are a tool to reduce risk, and it’s rare that 100% test coverage delivers benefits commensurate with its costs.

        2. 1

          I have seen millions/billion dollars companies being built with little to no typing: perl, python, ruby etc…

          Yes adding types helps making the code base more maintainable, but thats not my point. My point is that you would want to tradeoff that maintainability against velocity depending on your business needs:

          • Big enterprises code base should have type as their business needs might aim toward reliability and maintenance cost started to add up.
          • But in a startup environment, where you have to build the airplane while flying it, time-to-market is key. Loose typing enables quicker iterations for these small companies to achieve their focus goals.

          This is why we see many companies opt into a gradual typing approach as a transition bridge once their code base hit a certain size. I.e. Facebook’s Pyre or Stripe’s Sorbet are some good examples. But as always, it’s a tradeoffs that should be decided based on the business context.

          1. 1

            I have seen millions/billion dollars companies being built with little to no typing: perl, python, ruby etc…

            Me too!

            Yes adding types helps making the code base more maintainable, but thats not my point. My point is that you would want to tradeoff that maintainability against velocity depending on your business needs:

            My claim — which is definitely arguable — is that the inflection point where lack-of-typing switches from helping velocity to hurting velocity, is (a) when the codebase has more than 1 developer, or (b) when the codebase grows beyond O(1k) SLoC, whichever comes first. By which I mean: a lot sooner than you might think.

            1. 1

              I think that claim is definitely arguable and varies depending on context.

        3. 3

          You don’t have to do everything all OOP to use dataclasses or pydantic or whatever. You just write out the types for each field.

        1. 4

          If I want to define the API via language independent IDL . . .

          Do you? Why?

          IDLs are schemas that define a formal protocol between communicating parties. They represent a dependency, typically a build-time dependency, coupling producers and consumers. That can bring benefits, definitely. But if I can’t call your API without fetching a specific file from you, somehow, and incorporating it into my client, that’s also a significant cost! Open networks are supposed to work explicitly without this layer of specificity. HTTP is all I need to talk to any website, right?

          HTTP+JSON is pretty language-agnostic. I can even do it at the shell. JSON-RPC is nowhere near as mature, or as supported. What does this extra layer get you? What risks does it actually reduce? And are those risks relevant for your use cases? Why do you think e.g. Stripe doesn’t require JSON-RPC for its API?

          IMO the only place that IDLs make sense are in closed software ecosystems — i.e. corporate environments — that have both experienced specific pains from loosely-defined JSON APIs, and can effectively mandate IDL usage across team boundaries. Anywhere else, I find it’s far more pain than pleasure.

          1. 3

            Heh, I actually was thinking about your’s similar comment in another thread when asking, thanks for elaborating!

            I think I agree with your reasoning, but somehow still come to the opposite conclusion.

            First, agree that the JSON-RPC is a useless layer of abstraction. I’ve had to use it twice, and both times the value was negative. In this question, I am asking not about the jsonrpc 2.0, but about an RPC which works over HTTP, encoding payloads in JSON. I do have an RPC, rather than REST, style in mind though.

            I also couldn’t agree more about the issue with build-time dependencies is worth solving. Build time deps is exactly the problem I see at my $dayjob. We have an JSON-over-HTTP RPC interface and the interface is defined “in code” – there’s a bunch of structs with #[derive(Serialize)] structs. And this leads people to thinking along the lines of “I need to use the API. The API is defined by this structs. Therefore, I must depend on the code implementing the API”. This wasn’t explicitly designed for, just a path of least resistance if you don’t define API explicitly, and your language has derives.

            That being said, I think there has to be some dependency between producer and consumers? Unless you go full HATEOAS, you somehow need to know which method to call and (in a typed language, for ergonomics) which shape the resulting JSON would have. For stripe, I need to fetch https://stripe.com/docs/api/pagination/search to figure out what’s available. And, again, there is https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json.

            And, if we need at least an informal doc for the API, I don’t see a lot of drawbacks in making it more formal and writing, say, literal typescript rather than free-form text, provided that the formalism is lightweight. The biggest practical obstacle there seems to be absence of such lightweight formalisms.

            So, the specific “why”s for me wanting an IDL in an open decentralized ecosystem are:

            • reifying “this is the boundary exposed to outside world” in some sort of the specific file, so that it is abundantly clear that, if you are changing this file, you might break API and external clients. You could do that in the “API is informally specified in code” scenario, but that requires discipline, and discipline is finite resource, running out especially quickly in larger teams.
            • providing the low-friction way to document and publish promises regarding the API to the outside world. Again, with some discipline, documenting API in api.md file would work, but it seems to me that doc-comments require less effort for upkeep than separate guides.
            • making sure that details about the language the implementation is written in don’t accidentally leak. Eg, APIs probably shouldn’t use integers larger than 32 bits because they won’t work in JavaScript, but, if my impl language is Rust, I might not realize that as native serialization libraries would make u64 just work. More generally, in Rust just slapping derive(Serialize) always works, and often little thought is given to the fact that the resulting JSON might be quite ugly to work with in any non-rust language (or just to look at).
            • Maaaaybe generating stub clients and servers from the spec.
            1. 3

              That being said, I think there has to be some dependency between producer and consumers?

              Yeah! When a client calls a server there is definitely a dependency there. And as you note, there has to be some kind of information shared between server in client in order for that call to succeed. I suppose my point is about the… nature? of that dependency. Or maybe the efficacy of how it’s modeled?

              At the end of the day, you’re just sending some bytes over a socket and getting some bytes back. An IDL acts as a sort of filter, that prevents a subset of those byte-sets from leaving the client, if they don’t satisfy some criteria which is assumed will be rejected by the server. Cool! That reduces a class of risk that would otherwise result in runtime errors.

              1. What is impact of that risk?
              2. What benefits do I get from preventing that risk?
              3. What costs do I incur by preventing that risk?

              I suppose I’m claiming that, outside of a narrow set of use cases, and certainly for general-purpose public APIs, the answers to these questions are: (1) quite low, (2) relatively few, and (3) very many. (Assuming you’re careful enough to not break existing consumers by modifying the behavior of existing endpoints, and etc. etc.)

              reifying “this is the boundary exposed to outside world” in some sort of the specific file, so that it is abundantly clear that, if you are changing this file, you might break API and external clients. You could do that in the “API is informally specified in code” scenario, but that requires discipline, and discipline is finite resource, running out especially quickly in larger teams.

              I get that desire! The problem is that the IDL is a model of reality, not the truth, and, like all models, it’s a fiction :) Which can be useful! But even if you publish an IDL, some dingus can still call your API directly, without satisfying the IDL’s requirements. That’s the nature of open ecosystems. And there’s no way for you to effectively mandate IDL consumption with plain ol’ HTTP APIs, because (among other reasons) HTTP is by construction an IDL-free protocol. So the IDL is in some sense an optimistic, er, optimization. It helps people who use it — but those people could just as easily read your API docs and make the requests correctly without the IDL, eh? Discipline is required to read the API docs, but also to use the IDL.

              . . . document and publish … API [details] . . . making sure that details about the language [ / ] implementation don’t accidentally leak . . .

              IDLs convert these nice-to-haves into requirements, true.

              generating stub clients and servers

              Of course there is value in this! But this means that client and server are not independent actors communicating over a semantically-agnostic transport layer, they are two entities coupled at the source layer. Does this model of your distributed system reflect reality?

              I dunno really, it’s all subjective. Do what you like :)

              1. 1

                Convinced! My actual problem was that for the thing I have in mind the client and the server are implemented in the same code base. I test them against each other and I know they are compatible, but I don’t actually know how the JSON on the wire looks. They might silently exchange xml for all I know :-)

                I thought to solve this problem with an IDL which would be close to on-the-wire format, but it’s probably easier to just write some “this string can deserialize” tests instead.

                I’d still prefer to use IDL here if there were an IDL geared towards “this are docs to describe what’s on the wire” rather than “these are types to restrict and validate what’s on the wire”, but it does seem there isn’t such descriptive thing at the moment.

                1. 1

                  docs to describe what’s on the wire [vs] types to restrict and validate what’s on the wire

                  Is there a difference here? I suppose, literally, “docs” wouldn’t involve any executable code at all, would just be for humans to read; but otherwise, the formalisms necessary for description and for parsing seem almost identical to me.

                  1. 1

                    an IDL which would be close to on-the-wire format

                    (Just in case you missed it, this is what Preserves Schema does. You give a host-language-neutral grammar for JSON values (actually Preserves, but JSON is a subset, so one can stick to that). The tooling generates parsers, unparsers, and type definitions, in the various host languages.)

                    1. 1

                      If your client and server exist in the same source tree, then you don’t have the problems that IDLs solve :)

                      edit: For the most part. I guess if you don’t control deployments, IDLs could help, but certainly aren’t required.

                      1. 1

                        They are the sever and a client — I need something to test with, but I don’t want that specific client to be the only way to drive thing.

                        1. 1

                          They are the sever and a client — I need something to test with, but I don’t want that specific client to be the only way to drive thing.

                          I test them against each other and I know they are compatible, but I don’t actually know how the JSON on the wire looks. They might silently exchange xml for all I know :-)

                          It seems you have a problem :)

                          If the wire protocol isn’t specified or at least knowable independent of the client and server implementations, then it’s an encapsulated implementation detail of that specific client and server pair. If they both live in the same repo, and are reasonably tested, and you will never have other clients or servers speaking to them, then no problem! It’s a closed system. The wire data doesn’t matter, the only thing that matters is that client-server interactions succeed.

                          If all you want to do is test interactions without creating a full client or server component, which I agree is a good idea, you for sure don’t need an IDL to do it. You just tease apart the code which speaks this custom un-specified protocol from the business logic of the client and server.

                          package protocol
                          package server // imports protocol
                          package client // imports protocol
                          

                          Now you can write a mock server and/or mock client which also import protocol.

                          1. 2

                            This (extracted protocol with opaque wire format) is exactly the situation I am in. I kinda do want to move to the world where the protocol is knowable indepedently of the specfic client and server, hence I am seeking an IDL. And your comment helped me realize that just writing tests against specific wire format is a nice intermediate step towards making the protocol specification – this is an IDL-less way to actually see what the protocol looks like, and fix any weirdness. As in, after writing such tests, I changed JSON representations of some things because they didn’t make sense for an external protocol (it’s not stable/public yet, so it’s changable so far).

              1. 9

                As far as I understand, this thing stores secrets as encrypted files on disk, a daemon sits on top of those files, and clients call the daemon to read and write. The daemon has a notion of being locked or unlocked, and must be unlocked before secrets can be accessed. But it seems like that state isn’t per client or per session but rather global, meaning client A can unlock the daemon and disconnect, and then client B can read secrets without needing to unlock?

                If that’s the right reading, I don’t see how it makes sense. But maybe it’s not the right reading.

                1. 2

                  https://git.sr.ht/~sircmpwn/himitsu/tree/master/item/cmd/himitsud/cmd.ha#L76 seems like the right reading… though honestly it seems like you could “quickfix” this by having a client pass in a random UUID as their ID (though you’d want to share that around for certain use cases…)

                  This kinda reminds me of going in real deep into sudo and thinking about how that ends up working (well at least the standard plugin setup where it stores processes that successfully authenticated).

                  1. 1

                    client A can unlock the daemon and disconnect, and then client B can read secrets without needing to unlock?

                    Not exactly. From what I understand, the daemon spawn a predifined “granting” program, that the user must accept in order to provide the secret.

                    So client A asks for a a secret, and user must first unlock the store with their passphrase. Then the passphrase (or rather, the key derived from it) is stored in memory by the daemon. When client B asks for a secret, the daemon spawns a dialog asking the user to grant access to a secret from client B. If the user accepts, the daemon replies with the secret.

                    This is IMO the same thing as ssh-agent or gpg-agent, which hold the key in memory and simply provide it to whoever can read the socket file.

                    1. 1

                      Not exactly. From what I understand, the daemon spawn a predifined “granting” program, that the user must accept in order to provide the secret.

                      Interesting. I’m not familiar with this architecture. Can you point me to the bit in Himitsu which does this? What does it mean for a user to “accept” a granting program?

                      So client A asks for a a secret, and user must first unlock the store with their passphrase.

                      Is a client the same as a user?

                      When client B asks for a secret, the daemon spawns a dialog asking the user to grant access to a secret from client B. If the user accepts, the daemon replies with the secret.

                      What does it mean to say “the daemon spawns a dialog”? Does the daemon assume that all clients are on the same host?

                      This is IMO the same thing as ssh-agent or gpg-agent, which hold the key in memory and simply provide it to whoever can read the socket file.

                      Ah, okay, key bit is here, I guess:

                      const sockpath = path::string(&buf);
                      const sock = match (unix::listen(sockpath, net::sockflags::NOCLOEXEC)) {
                      

                      It binds only to a Unix domain socket, which by definition is only accessible from localhost. Then I guess we’re at the second branch of my confusion block ;) in a sibling comment, namely

                      If connections can only come from localhost, then I can’t quite see why you’d use a client-server architecture in the first place — AFACT it would be simpler and safer to operate on the filesystem directly.

                      Or: what makes the ssh-agent architectural model for secret management the best one?

                      1. 1

                        Check out the himitsu-prompter(5) manpage for info on the granting program. I didn’t use it at all, and I just read all the documentation because I find it quite interresting. To answer all your questions, the daemon is configured to spawn a program, the prompter, whenever a client (external program) request access to a secret. This program must ask the user for permission to provide the secret (basically a grant/deny dialog window). Once access is granted, the requesting client gets a reply from the server.

                        A client is any application that can write to the Unix socket created by the daemon (permission is hereby granted by the user spawning the himitsu daemon.

                        Regarding the Unix socket, my guess is that the project is aiming toward a possibility to use the program over the network using TCP, given that it’s heavily inspired by the factotum server on plan9 (which is basically the same thing, but over the net via the 9p protocol).

                        1. 1

                          the daemon is configured to spawn a program, the prompter, whenever a client (external program) request access to a secret. This program must ask the user for permission to provide the secret (basically a grant/deny dialog window)

                          If the daemon accepts requests for secrets on a Unix domain socket, then there is no way for it to know that a dialog window is observable by the client. The client can be on a different host, and its requests shuttled thru a proxy that listens on TCP and sends to the local Unix socket.

                          If “locked/unlocked” is a state that’s per-session, or per-connection, then no problem! But I don’t see anything which indicates that’s the case. It seems to be a state that’s per-daemon, which I understand as a singleton per host. Is that true?

                          1. 2

                            The client (say, a mail client than need access to your IMAP credentials) performs the request via the Unix socket. Upon receiving that request, the daemon itself spawns a runtime configured program (say, /usr/local/bin/himitsu-prompter) and talks to it over stdin/stdout (not a Unix socket this time). Which means that the dialog must run on the same host as the daemon. Then this program can either spawn a dialog window, send a push request to a phone or even ask for a fingerprint check to confirm acceptance by the user. If the program (which is a child of the daemon) returns 0, then the request is accepted, and the daemon delivers the secret to the requesting program (the mail client here). Otherwise, the request is denied.

                            You’re right about the store state though, it is locked per-daemon. There can be multiple daemon running on the same host though (like, one per user). There is no knowledge of “session” here, which could be a problem to spawn the prompter program for example (eg, to retrieve the $DISPLAY variable of the current user). But I’m confidemt that improvements will be made over time to improve this situation, like a session-bound prompter for example.

                            1. 1

                              Thanks a bunch, this was a great explanation and I learned some things I didn’t know before.

                              If I understand correctly, the daemon receives requests for secrets from clients that connect over a Unix domain socket. Requests cause the daemon to spawn a child process, which it communicates with over stdin/stdout. The daemon will issue commands, basically requests, to the child process, based on the details of the client request, as well as its own internal “locked” or “unlocked” state. The child process is assumed to communicate with “the user” to get authentication details like passwords, and a successful return code is treated as consent by “the user” to authorize the client request.

                              If that’s basically right, then I get it now. It’s a secrets storage manager — for desktop users, managing secrets on the local filesystem. That’s fine! But when I read “secrets storage manager” I think HashiCorp Vault, or maybe Bitwarden; definitely not ssh-agent ;) which was the root cause of my confusion.

                              edit: I guess I’m still confused by the design, though. Interacting with this system requires clients to speak the IPC protocol, which requires code changes in the client. If the receiving daemon is only ever accessible on a Unix domain socket, and therefore that clients will only ever connect to servers on localhost, then, by definition, the clients should have access to the same filesystem as the daemon, right? In that case, it’s not clear to me what value the intermediating daemon is providing. Couldn’t the clients just as easily import some package that did the work of the daemon, spawning the prompter child process and reading the secret data directly from the filesystem, themselves? I guess I see some value in that the daemon architecture avoids the need for programming-language-specific client libraries… is there something more?

                              1. 1

                                The process you describe is correct. This is indeed very different from Vault. It’s more of a password manager that clients can interact with to fill in login credentials.

                                Regarding your idea about using a client library, this would cause a major issue: the client would then have the ability to unlock and read the full keystore, as you’d provide the master password to every single application, and trust them to only read the secrets they need. This would also require to provide the master password to each new application, as there wouldn’t be a « master process » running to keep the keystore unlocked (or rather, in « softlock » state as the documentation puts it. And as I stated earlier, I can see this program move toward using TCP as well for client requests, given its similarities to factotum(4). The latter is an authentication daemon that can be queried over the network to authenticate users over a network, like you’d query an LDAP server for example.
                                I think this would require a bunch of changes to the daemon though to run as standalone mode, like TLS encryption over TCP, and possibly the ability to « switch » from one keystore to another to be able to provide secrets from multiple users. This is risky though so I think that for now, usage in local mode only is more than enough for an initial release.

                    2. 1

                      My current setup with pass and GPG behaves in the same way. Why is this a problem, in your opinion?

                      1. 4

                        Say I have two sandboxed applications, one of which I want to grant keyring access to and the other of which I don’t. Perhaps the second doesn’t have any legitimate reason for keyring access and it asking for keyring access is going to make me suspect it’s compromised.

                        It would be useful to be able to gate access for these apps differently, without the legitimate app unlocking the keyring “for” the illegitimate app.

                        1. 1

                          This is exactly how it works. Applications to not get granted keyring access directly, they query the daemon which then asks permission to the user to provide access to a specific key, for a specific application. This is referred to as « softlock » mode in the documentation. So if am illegitimate application requests access to a secret, they won’t get anything without the user’s approval.

                        2. 3

                          What prevents an arbitrary third party from connecting to the relevant server, issuing a request (without any sort of authentication or authorization) and then receiving sensitive information?

                          If connections can come from multiple hosts, then I can’t quite see how a single shared security context for all of those users isn’t a fatal security risk. If connections can only come from localhost, then I can’t quite see why you’d use a client-server architecture in the first place — AFACT it would be simpler and safer to operate on the filesystem directly.

                      1. 2

                        Excuse my newbieness, but what’s IDL? There’s just so many acronyms being thrown around here, it’s barely readable.

                        1. 3

                          An IDL is, as @fstamour says, and interface definition language. These are used to define interfaces in a way that is agnostic to the implementation language. Most ’80s object broker systems (COM, CORBA, and so on) had quite rich ones that allowed you to automatically map the abstractions that they provided into different languages. Many lightweight RPC mechanisms that evolved in the ‘90s in response to the perceived complexity of things like DCOM and CORBA also had them, though not all. As I recall, the W3C maintains one as well, for all of the DOM APIs, to provide the option for non-JavaScript languages in the browser to be able to take advantage of the same APIs (though since IE died, taking VBScript with it, and WebAssembly avoided direct access to JavaScript APIs, this hasn’t been very important).

                          The idea behind an IDL is to provide a language-agnostic description of an interface. If your underlying RPC mechanism uses JSON then you implicitly have two languages that you need to support: JSON for the wire transport and whatever language you use to implement the endpoints and you want to be able to translate from foo(42) to something that generates some JSON that looks a bit like { method: "foo", args: [ 42 ] } (or, hopefully, something more sensible) and have the foo that’s exposed to the implementation language’s type checker (does foo always take one argument and does it have to be an integer?).

                          Part of the problem with JSON here is that it doesn’t have a particularly rich set of primitive types. Everything is encoded as text, which has representations problems for binary floating point numbers, and JSON doesn’t have any mechanism for defining things like 32-bit integers other than as numbers. Worse, it can’t represent the entire range fo 64-bit integers at all.

                          JSON Schema lets you represent integers by defining that they must be a multiple of 1 and fixed-width integers by expressing the minimum and maximum values, but that’s a validation constraint rather than something that’s expressed in the wire format. This means that, if you’re using JSON, you actually want to get three things out of your IDL:

                          • A mapping into your implementation language(s).
                          • A projection into JSON for your wire format.
                          • A validator for the JSON that ensure that the values are all within the required ranges.

                          OpenAPI is an IDL for web services. AutoRest can generate Java, C#, Python, and TypeScript definitions from OpenAPI definitions. There are other tools that will generate the IDL from annotations in the server-side implementation.

                          1. 1

                            Worse, it can’t represent the entire range fo 64-bit integers at all.

                            Do not confuse JavaScript with JSON. There is no such limitation in JSON since it’s numbers are just strings.

                            1. 2

                              The JSON spec doesn’t define limits on the number type, but it does advise that

                              Note that when such software is used, numbers that are integers and
                              are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
                              sense that implementations will agree exactly on their numeric
                              values.
                              
                          2. 1

                            Interface Definition Language

                          1. 6

                            There is no way to modify an existing rope: all operations on a rope return a new rope.

                            I wrote a bunch of stuff that worked this way. Some stuff that even got acquired! Thought it was nice and elegant. But each and every one of those systems failed once they got beyond trivial scale. You just can’t do Go like this. It thrashes the GC and can’t be made performant.

                            1. 3

                              The expected maximum scale here is “human typing speed on reasonably-sized text files” and thus far seems to be okay.

                              1. 1

                                Incidentally do you have any recommendations for things to learn how to write fast go? My recent attempts have had very mixed success - it’s easy to write code that is much slower than naive code.

                                  1. 1

                                    You, sir, are a gentleman and a scholar

                              1. 2

                                I’d say it’s also important to try to avoid breaking changes in the first place. Deprecate all you want, print warnings, all of that… but if you make a breaking change in a library, you’re always risking dependency hell for your dependents (and their dependents).

                                Rich Hickey had a bold suggestion: If you want to change what do_foo does, just add a do_foo2 next to it and deprecate the old one. Not always applicable, but an approach I always keep in mind as an option.

                                1. 1

                                  if you make a breaking change in a library, you’re always risking dependency hell for your dependents (and their dependents).

                                  As long as you bump the major version with each breaking change, there is no consequent hell for any consumer. Anyone depending on major version N remains entirely unaffected. When consumers upgrade from major version N to major version N+1, they must, of course, verify that their code continues to work as expected, which always requires explicit attention.

                                  If you want to change what do_foo does, just add a do_foo2 next to it and deprecate the old one. Not always applicable, but an approach I always keep in mind as an option.

                                  This approach optimizes for minimizing breaking changes, and therefore minimizing major version bumps. But it also results in APIs which are strictly worse than the alternative. If foolib v1 has 100 consumers, and barlib v1 has 100k consumers, is the cost of a breaking change equivalent for them both? If you are a new consumer, how much better is a foolib with only a single, “correct” do_foo function, versus one with a deprecated do_foo and a “correct” do_foo2”?

                                  The cost or benefit a breaking change is a function of a lot of variables. It’s not strictly negative.

                                  1. 1

                                    As long as you bump the major version with each breaking change, there is no consequent hell for any consumer.

                                    This is incorrect for many large codebases with complicated dependency trees. Take the situation where you have two dependencies, A and B, each of which depends on C. A starts depending on a newer major version of C due to a newer feature, but B still requires the old major version due to breaking changes. Now you’re stuck on this version of A, and getting farther out of date. Or worse, you’re trying to set up your dependencies in the first place and can’t find a compatible combination.

                                    This is a simplified situation. It gets worse in larger, more complicated dependency trees.

                                    (And what’s even worse is when you have pinned some dependency somewhere because of breakage, and then other things get pinned as a result, and then three years later you have to start unwinding the trail of pins and going through multiple version upgrades just to get everything back to latest.)

                                    So yeah, there are reasons to not make breaking changes in the first place, regardless of versioning schemes.

                                    (Maybe if you use Nix or something you can get out of this, since A and B can each have their own C…)

                                    1. 1

                                      I understand this scenario intimately 😉 but this is (a) an uncommon hell, and (b) of the consumer’s making. IMO it’s something that should influence the major-version-bump cost calculus, absolutely, but not dominate it.

                                      1. 1

                                        I’m not sure how you can say it’s of the consumer’s making. If I need to depend on A and B, it’s not my fault that they at some point each depend on incompatible versions of C.

                                        Anyway, there’s a middle ground: Announce the deprecation well in advance, emit warnings, allow a good amount of calendar time (and a few versions) to pass, and finally make the breaking change. This reduces the chances of problems.

                                        1. 1

                                          I’m not sure how you can say it’s of the consumer’s making. If I need to depend on A and B, it’s not my fault that they at some point each depend on incompatible versions of C.

                                          That’s true. But, in practice, when this happens, there are usually a lot of mitigating factors. There’s almost always some (prior) versions of A and B which have compatible requirements against C, which you can fall back to. If you’re prevented from using those versions for whatever reason — security vulnerabilities, etc. — then it’s usually just a short period of time before the maintainers of the dependencies update them to a compatible state. And even in the situation where this dep graph is truly unsolvable, you always have the fall-back option of forking and patching.

                                          My point isn’t to diminish the real pain introduced by the points you raise, it’s real! But I think these conditions are basically always pathological, usually quite rare, generally ephemeral, and can often be solved in a variety of ways.

                                          I think it’s important that software authors can write and release software that delivers value for other people, without necessarily binding them to support each released version into perpetuity. Software is a living thing, no API is perfect, and no package can be expected to be perfect. The cost of a breaking change isn’t determined solely by the impact on consumers, it’s a complex equation that involves consumers, producers, scope, impact, domain, and a bunch of other variables.

                                          1. 1

                                            True, at work we’ve had to fork and patch a few times. It usually does work as a final escape valve.

                                1. 5

                                  gRPC is an IDL-based protocol, and like all IDL-based protocols, it relies on communicating parties sharing knowledge of a common schema a priori. That shared schema provides benefits: it reduces a category of runtime risks related to incompatibilities, and it — can, sometimes —improve wire performance. That schema also carries costs, chief among them that it requires producers and consumers to share a dependency graph, and usually one that’s enforced at build-time. That represents a coupling between services. But isn’t one of the main goals of a service-oriented architecture to decouple services?

                                  Over many years, and across many different domains, I’ve consistently found that, for service-to-service communication, informally-specified HTTP/JSON APIs let teams work at very high velocity, carry negligible runtime risk over time, and basically never represent a performance bottleneck in the overall system. Amusingly, I’ve found many counter-factuals — where gzipped HTTP+JSON APIs significantly outperformed gRPC and/or custom binary protocols.

                                  I’m sure there are situations where gRPC is right tool for the job! But all my experience suggests it’s a far narrower set of use-cases than is commonly understood, almost all in closed software ecosystems. But maybe I’m missing some angle?

                                  1. 2

                                    I’m sure there are situations where gRPC is right tool for the job! But all my experience suggests it’s a far narrower set of use-cases than is commonly understood, almost all in closed software ecosystems. But maybe I’m missing some angle?

                                    I’ve found cases where gRPC works better than gzipped HTTP+JSON, but I have to agree that it’s in very limited cases. Specifically I’ve worked with a rate limiter which has a “frontend” service to talk to the backend which actually keeps track of counts, and here the requests were both very repetitive (increment in-flight request, decrement in-flight request) and the fields that changed very specific. The service would receive very high request throughput and its repetitive requests made a gRPC implementation a lot faster (and less compute heavy to avoid any de/compression) than an HTTP+JSON implementation. We ran rigorous tests and found anywhere from 3-20x speedups depending on the type of load we were receiving.

                                    I think gRPC matters more for services that see high scale, but if you’re working at a shop that has a few high scale services, it may still make more sense to standardize around gRPC just so that the high-scale services don’t have to work completely differently than the rest of the shop. gRPC has a lot less (how many channels to create, how will interceptors work, etc) tooling around it than HTTP+JSON so it pays to develop that expertise in-house. When we decided to use gRPC for a few services, it was painful having to learn the ecosystem of debugging and monitoring tools especially when there were so many easily available tools and well documented RFCs for HTTP+JSON on the general net.

                                    EDIT: If low latency is important to your service though, gRPC staves off a lot of the overhead inherent in setting up and transmitting/receiving an HTTPS stream. If latency is of the utmost importance (say you’re building an SDP/VoIP signaling layer) though, gRPC may be the way to go.

                                    1. 1

                                      If low latency is important to your service though, gRPC staves off a lot of the overhead inherent in setting up and transmitting/receiving an HTTPS stream.

                                      Is this still true with HTTP/2?

                                  1. 4

                                    aspects of code formatting that contribute to write amplification – how big a change becomes in the resulting diff

                                    I love code formatters, but this problem is my pet peeve. When I choose a coding standard for a new project, my top priority is choosing a flavor that avoids write amplification. A standardized format is great, but not if it interferes with code review. I’ve encountered too many bugs hiding inside a diff hunk that was 90% auto-formatting noise. (I’ve had no luck with diffs that hide white space changes.)

                                    in some ways better than any autoformatter could ever come up with, because the human knows best

                                    I agree that a conscientious developer can format code better than a machine, at the margin. However, I’ve not found the marginal improvement to be worth the marginal cost in developer time/attention. A machine can format code 90% as good as a human in 1% of the time. Something like reformatter for emacs automatically reformats the buffer when I save it, so I spend almost no time on layout at all. I enjoy the way it lets me stay in flow.

                                    It’s also the case that not all programmers are human. Maybe it’s a fancy refactoring tool or maybe it’s a lazy Perl script to munge away a recently discovered anti-pattern. Either way, they’re terrible at formatting. Having a human reformat this code manually gets expensive. Of course, non-conscientious developers who format at random also border on non-human :-)

                                    1. 3

                                      I’ve encountered too many bugs hiding inside a diff hunk that was 90% auto-formatting noise. (I’ve had no luck with diffs that hide white space changes.)

                                      Have you tried difftastic? Seems like it’s designed to address exactly this problem.

                                      1. 1

                                        A standardized format is great, but not if it interferes with code review. I’ve encountered too many bugs hiding inside a diff hunk that was 90% auto-formatting noise.

                                        I think the real mistake here is that it sounds like you’re reviewing diffs? That’s always the wrong move – you want to review the resulting file, not the minimal diff going into it.

                                        All a diff can tell you is that someone inserted a new method into a file. If you just review the diff you might walk away thinking it’s a well-written change. If you review the resulting file after applying the diff, you might catch the fact that the added method is duplicative of other methods, that this has become a real issue in the file, and recommend the submitter abstract out the common logic and leave things better than they found them rather than trying to do the bare minimum and making the underlying mess worse.

                                        Etc. The diff just does not contain enough information to do a proper code review. Worrying about whitespace in the diff is getting hung up on the wrong details.

                                        1. 3

                                          A typical LLVM PR changes a hundred lines of code in 3-4 files, each of which is thousands of lines long. Telling people that they should review the entire file when reviewing a PR rather than the changes is the same as telling them that they should not bother doing code review: you’re advocating for something that is completely infeasible.

                                          1. 1

                                            A typical LLVM PR changes a hundred lines of code in 3-4 files, each of which is thousands of lines long. Telling people that they should review the entire file when reviewing a PR rather than the changes is the same as telling them that they should not bother doing code review: you’re advocating for something that is completely infeasible.

                                            a) Most things aren’t LLVM (indeed, nothing in the person I was replying to’s bio or GitHub seems to indicate that they’re an LLVM dev – and believe it or not, I do not frame everything I write always in terms of you personally and the unique challenges you face, stranger). “This doesn’t work in the most extreme case, so it’s not a good idea in any case” is just letting the perfect be the enemy of the good.

                                            b) In most projects, the solution to “I can’t review 4 ten thousand line-long files” is “don’t let your files become 10s of thousands of lines long in the first place”. That’s really a gigantic, phenomenally industrial strength C++ codebase problem, which isn’t most codebases. It shouldn’t be surprising that LLVM is a pretty extreme outlier with fairly atypical challenges! We’ve got linters that scream bloody murder when files hit 500 lines, and something in the 200-1000 line range is typical. “That this doesn’t work in the extremes means this is always a bad idea, to me” is, again, just letting the perfect be the enemy of the good.

                                            c) There is such a thing as using your head. Don’t review just the diff, because it does not have enough context; also don’t exhaustively review 14,000 lines that aren’t changing in the 30 line diff. Skim it. Get the gist of the file or at least the things that are near it and which it touches. Try your best to find a reasonable balance. Code reviews are only as valuable as the effort you put into them.

                                        2. 0

                                          When I choose a coding standard for a new project, my top priority is choosing a flavor that avoids write amplification.

                                          I guess the argument is that “the coding standard” in the code formatting sense shouldn’t be something that’s different from project to project, it should be (more or less) a universal property of the language.

                                          1. 1

                                            Then the “formatting” should be enforced at the compiler level. Why leave it up to another tool?

                                            1. 1

                                              I agree!

                                        1. 5

                                          This feels very backwards to me. Breaking changes are bad. If a library has a breaking change that takes 10 minutes to fix and is used by 60,000 developers, that’s 10,000 hours down the drain. The version numbers are important only insofar as they signal “here comes a waste of time.” If for marketing reasons, you need to promote a release, that’s fine, but that’s a different issue from when and how you should make a breaking change.

                                          1. 11

                                            Breaking changes are bad.

                                            Breaking changes are by themselves value neutral. It can easily be the case that a breaking change yields more benefit to the software project than cost to its consumers. And it’s not like upgrading dependencies is obligatory, right? It’s always an opt-in decision.

                                            1. 9

                                              It’s always an opt-in decision.

                                              Only until there’s a security issue in the package and the fix is not released for the older major versions. Then it’s upgrade or potentially do lots of custom fork work.

                                              (I don’t disagree with your post though - breaking doesn’t mean bad)

                                              1. 4

                                                But surely that’s true no matter what version numbering scheme the package uses: old releases of whatever numbering structure will eventually not get security patches.

                                                1. 3

                                                  Compatible versions don’t take significant effort to update.

                                              2. 2

                                                It can easily be the case that a breaking change yields more benefit to the software project than cost to its consumers

                                                One might argue that very frequent major version bumps mean that the authors are bad at getting their design straight. Of course like all such things that’s not true in every case, but it can be an indication for something going wrong in a project. What’s frequent itself depends on the context of course, and of course what the breaking changes are like is another topic. If it’s about removing hacks for things deprecated a decade ago that’s different from a v2 (v3, v4, …) that can’t hold up with what v1 already brought on the table, yet requires rewriting whole systems on the user’s side.

                                                As others pointed out security issues or other types of bugs might mean that you are forced to upgrade (or fork).

                                                I’d say breaking changes are not neutral, but but bad, which might however be completely overshadowed by any kind of benefit it brings alone. In some cases it’s actually worsened by a worse design, lack of feature completeness of v2 or some company deciding they wanna have a stronger grip to maximize profits, artificially creating incompatibilities (Mapbox, etc.).

                                                1. 1

                                                  I’d say breaking changes are not neutral, but but bad, which might however be completely overshadowed by any kind of benefit it brings alone.

                                                  Breaking changes are bad for consumers, but generally beneficial for authors.

                                              3. 7

                                                This is entirely the reason why SemVer gates them behind major versions - to communicate them. The rest comes down to management and upgrade pressure. As a positive example, let’s look at the “nom” library in Rust. Each major version is a different take on the same concept and api - also sometimes because new language features lead to better ergonomics or tie better into new features of the Rust language. Still, it’s intended use is that you pick a version of nom and then don’t upgrade.

                                                The other option would be what was the norm in the Ruby community for a while: you had take X on a concept, let’s call the library “apples”. It becomes 1.0 and never changes. Suddenly, someone forks it, rips half of it out and calls it slightly different, say “oranges”. So suddenly, you ended up in a game of community telephone, which library was currently the most recent take, considered modern.

                                                1. 2

                                                  The other end of the spectrum is bindgen that bumps major incompatible version whenever their output or clang changes slightly.

                                                2. 4

                                                  I recently released a breaking change to a library. The breaking change was to change behavior so that people would stop having their Unicode characters silently stripped from data, and being confused at weird slugs

                                                  The change will mean that new users will stop having this issue by default. Existing users, when upgrading, might also realize they had a problem and will have a fixed bug. And of course I have a setting in place to maintain the old behavior (but it’s opt-in because the behavior is bad)

                                                  I feel like basically every well managed project does breaking changes to make stuff better, include on-ramps and off-ramps, and just is doing stuff practically. There’s definitely pain! But good projects all seem to handle this well.

                                                1. 7

                                                  I have the same issue with rustfmt. It does what people imagine gofmt doing, but not what it actually does.

                                                  gofmt is an error-fixing formatter. It fixes things about formatting it knows are wrong (like braces, indentation), but leaves everything else more or less unchanged (like 1-line vs multi-line choice). There are many ways to format the same construct, and gofmt respects humans’ high-level choice about the “layout” of the code.

                                                  Most other formatters are destructive canonicalizers, which completely replace formatting of the input with their own heuristics. The difference between these approaches is very significant, because reformatting based on heuristics doesn’t leave room for common sense, and bulldozes over formatting exceptions.

                                                  Unfortunately, these unforgiving blunt tools are still used due to the fallacy of “we must do something; this is something; therefore we must do this”.

                                                  1. 3

                                                    I thought gofmt was a destructive canonicalizer. So you are telling me that gofmt will not change:

                                                    foo
                                                    {
                                                      bar;
                                                      baz;
                                                    }
                                                    

                                                    into

                                                    foo {
                                                            bar;
                                                            baz;
                                                    }
                                                    

                                                    The former isn’t an error, just a difference in opinion.

                                                    1. 5

                                                      gofmt has an opinion on braces when they’re on multiple lines, but it will preserve 1-liner foo {bar, baz} version as-is. This is unlike rustfmt that will splat 1-liners as high as it wants, or re-wrap multi-line constructs into long spaghetti, depending on which heuristic they hit.

                                                      1. 3

                                                        So unless spacing causes a compilation error, then I wouldn’t call gofmt an error-fixing formatter.

                                                        1. 4

                                                          I mean an error in a general sense from perspective of formatting, like using 3 spaces to indent is an erroneous formatting.

                                                          1. 2

                                                            No. If 3 spaces for indenting is an “error” then it should be enforced at the language level. Rob Pike wimped out in this regard.

                                                            1. 5

                                                              I think you’re just arguing about the meaning of the word “error”, rather than what I’ve said about formatters?

                                                              I’ve used the word “error” in its non-technical English meaning of “deviation from what is correct”, and not the other meaning of “what Rob believes must stop compilation”. If that ambiguity bothers you, please read my original post with “error” replaced with “imprudent deviation”:

                                                              gofmt is an imprudent-deviation-fixing formatter.

                                                              1. 1

                                                                If the language allows white space between tokens, why is X spaces correct, while X-n or X+n spaces incorrect? X is an arbitrary value; it’s just an opinion being enforced. If you won’t want people to have opinions on the “correct formatting” for a language, make it impossible to have said opinions at the compiler level, not with some external tool.

                                                      2. 1

                                                        It will change the former to the latter. And you’re right, that doesn’t represent a fixing of an error. But the whole point of the tool is to remove style choices like this from the set of things that programmers can have different opinions on! :) As the proverb goes, gofmt‘s style is nobody’s favorite, but gofmt is everybody’s favorite.

                                                      3. 1

                                                        Amen.

                                                        I mean that rustfmt has sane heuristics. But yes, it is unfortunately a destructive bulldozer too. I was unaware that gofmt was not. Good tip!

                                                      1. 1

                                                        It’s weird that the first thing you criticize in a Critical Retrospective is something syntax-related that you yourself call superficial. It makes it hard to take the rest of the post seriously

                                                        1. 29

                                                          If syntax impacts understandability, is it actually superficial?

                                                          1. 19

                                                            Because I don’t think that’s the fault of the syntax. Huge part of criticism is expectations/preferences and lack of understanding of the trade-offs that made it the way it is. When Rust is different than whatever other language someone is used to, they compare familiar with unfamiliar (see Stroustrup’s Rule). But it’s like saying the Korean alphabet is unreadable, because you can’t read any of it.

                                                            People who don’t like Rust’s syntax usually can’t propose anything better than a bikeshed-level tweak that has other downsides that someone else would equally strongly dislike.

                                                            For example, <> for generics is an eyesore. But if Rust used [] for generics, it’d make array syntax either ambiguous (objectively a big problem) or seem pointlessly weird to anyone used to C-family languages. Whatever else you pick is either ambiguous, clashes with meaning in other languages, or isn’t available in all keyboard layouts.

                                                            The closure syntax || expr may seem like line noise, but in practice it’s important for closures to be easy to write and make it easy to focus on their body. JS went from function { return expr } to () => expr. Double arrow closures aren’t objectively better, and JS users criticize them too. A real serious failure of Rust regarding closures is that they have lifetime elision rules surprisingly different than standalone functions, and that is a problem deeper than the syntax.

                                                            Rust initially didn’t have the ? shortcut for if err != nil { return nil, err } pattern, and it had a problem of a low signal-to-noise ratio. Rust then tried removing boilerplate with a try!() macro, but it worked poorly with chains of fallible function calls (you’d have a line starting with try!(try!(try!(… and then have to figure out where each of them have the other paren). Syntax has lots of trade-offs, and even if the current one isn’t ideal in all aspects, it doesn’t mean alternatives would be better.

                                                            And there are lots of things that Rust got right about the syntax. if doesn’t have a “goto fail” problem. Function definitions are greppable. Syntax of nested types is easy to follow, especially compared to C’s “spiral rule” types.

                                                            1. 14

                                                              I think a lot of criticism about syntax is oblique. People complain about “syntax” because it’s just… the most convenient way to express “I find it hard to learn how to write correct programs, and I find it hard to interpret written programs, even after substantial practice”.

                                                              Lots of people complain that Common Lisp syntax is hard. Lisp syntax is so easy that you can write a parser in a few dozen lines. Common Lisp has a few extra things but, realistically, the syntax is absolutely trivial. But reading programs written in it is not, even after substantial practice, and I get that (as in, I like Common Lisp, and I have the practice, and I get that).

                                                              Same thing here. A lot of thought went into Rust’s syntax, probably more than in, say, C’s syntax, if only because there was a lot more prior art for Rust to consider. So there’s probably not much that can be done to improve Rust’s syntax while not basically inventing another language. That doesn’t take away from the fact that the language is huge, so it has a syntax that’s unambiguous and efficient but also huge, so it’s just a whole lot of it to learn and keep in your head at once. I get it, I’ve been writing Rust on and off but pretty much weekly for more than an year now and I still regularly need to go back to the book when reading existing code. Hell, I still need it when reading existing code that I wrote. You pay a cognitive price for that.

                                                              1. 3

                                                                “I find it hard to learn how to write correct programs . . .

                                                                Do you believe “correctness” is a boolean property of a program?

                                                                1. 1

                                                                  I do, as in, I think you can always procure a “correctness oracle” that will tell you if a program’s output is the correct one and which, given a log of the program’s operations, can even tell you if the method through which it achieved the result is the correct one (so it can distinguish between correct code and buggy code that happens to produce correct output). That oracle can be the person writing the program or, in commercial settings, a product manager or even a collective – a focus group, for example. However:

                                                                  • That oracle works by decree. Not everyone may agree with its edicts, especially with user-facing software. IMHO that’s inherent to producing things according to man-made specs. There’s always an “objective” test to the correctness of physics simulation programs, for example, but the correctness of a billing program is obviously tied to whatever the person in charge of billings thinks is correct.
                                                                  • The oracle’s answer may not be immediately comprehensible, and they are not necessarily repeatable (like the Oracle in Delphi, it’s probably best to consider the fact that its answers do come from someone who’s high as a kite). IMHO that’s because not all the factors that determine a program’s correctness are inherent to the program’s source code, and presumably, some of them may even escape our quantitative grasp (e.g. “that warm fuzzy feeling” in games). Consequently, not all the knowledge that determines if a program is correct may reside with the programmer at the time of writing the code.

                                                                  More to the point, I think it’s always possible to say if something is a bug or a feature, yes :-D.

                                                                  1. 1

                                                                    Wow! I guess I can just say that I wish I worked in your domain! 😉 I can’t think of more than a handful of programs I’ve written in my entire life which have a well-defined notion of correct, even in part. Almost all of my programs have been approximate models of under-specified concepts that can change at the whims of their stakeholders. Or, as you say it,

                                                                    the correctness of a billing program is obviously tied to whatever the person in charge of billings thinks is correct.

                                                                    Exactly!

                                                                    not all the knowledge that determines if a program is correct may reside with the programmer at the time of writing the code.

                                                                    In my experience it rarely exists anywhere! Not in one person, or many, or even conceptually.

                                                                    1. 1

                                                                      I can’t think of more than a handful of programs I’ve written in my entire life which have a well-defined notion of correct, even in part.

                                                                      Oh, don’t get me wrong – that describes most of the code I wrote, too, even some of the code for embedded systems :-D. It may well be the case that, for many programs, the “correct” way to do it currently escapes everyone (heh, web browsers, for example…) But I am content with a more restricted definition of correctness that embraces all this arbitrariness.

                                                                2. 2

                                                                  Well, there was a lot of prior art even when C was created, and they actively chose to disregard it. They also chose to disregard discoveries in C itself in the 70s and 80s, freezing the language far too early considering the impact it would have in the following decades.

                                                                3. 4

                                                                  it’s like saying the Korean alphabet is unreadable, because you can’t read any of it.

                                                                  But like there is a reasonably objective language difficulty ranking index (from the perspective of English-native speakers) and Korean is squarely in the most-difficult tranche, I guess in no small part due to its complex symbology, at least in comparison to Roman alphabets. Are you saying that this dimension of complexity is, net, irrelevant?

                                                                  1. 11

                                                                    Korean uses the Hangul alphabet, which is very easy to learn. It’s much simpler than our alphabet. You can learn Hangul in a day or two. You’re thinking of Japanese, which is a nightmare based on people writing Chinese characters in cursive and italics while drinking a bottle of sake.

                                                                    1. 1

                                                                      some simplified hanzi does look like kanji, but I would appreciate an example of a japanese character looking like a cursive or italic version of a chinese glyph before I go on to tell your analogy to everyone at parties.

                                                                      1. 1

                                                                        It’s not an analogy. It’s the historical truth of kana: https://en.wikipedia.org/wiki/Kana. Japanese kanji and hanzi are mostly the same modulo some font changes and simplification in the 20th c.

                                                                        1. 1

                                                                          I meant the drunken japanese people part.

                                                                          1. 3

                                                                            We can’t prove they weren’t drunk. :-)

                                                                    2. 11

                                                                      from the perspective of English-native speakers

                                                                      I think that’s what they were getting at; there’s nothing inherently difficult about it but your background as an English speaker makes it look hard to read when objectively speaking it’s dramatically simpler than English due to its regularity and internal logic.

                                                                      1. 2

                                                                        I guess I would say that there is no “objectively speaking” in this domain? Like, there is no superhuman who can look at things invariant of a language background.

                                                                        1. 3

                                                                          If you’re talking about “easy to learn” then I agree.

                                                                          If you’re talking about simplicity, then I disagree. The number of rules, consistency, and prevalence of exceptions can be measured without reference to your background.

                                                                      2. 7

                                                                        I’ve specifically mentioned the Hangul alphabet (a syllabary, strictly speaking), not the language. The Korean language (vocabulary, grammar, spoken communication) may be hard to learn, but the alphabet itself is actually very simple and logical. It’s modern, and it has been specifically designed to be easy to learn and a good fit for the Korean language, rather than being a millennia-old historical borrowed mash-up like many other writing systems.

                                                                        I think it’s a very fitting analogy to having an excellent simple syntax for a complex programming language. You may not understand the syntax/alphabet at all, but it doesn’t mean it’s bad. And the syntax/alphabet may be great, but the language it expresses may still be difficult to learn for other reasons.

                                                                        With Rust I think people complaining about the syntax are shooting the messenger. For example, T: for<'a> Fn(&'a) makes lifetime subtyping contravariant for the loan in the argument of a function item trait in a generic trait bound. Is it really hard because of the syntax? No. Even when it’s expressed in plain English (with no computer language syntax at all) it’s an unintelligible techno-babble you wouldn’t know how to use unless you understand several language features it touches. That for<'a> syntax is obscure even by Rust’s standards, but syntactically it’s not hard. What’s hard is knowing when it needs to be used.

                                                                      3. 4

                                                                        People who don’t like Rust’s syntax usually can’t propose anything better than a bikeshed-level tweak that has other downsides that someone else would equally strongly dislike.

                                                                        The problem with Rust’s syntax isn’t that they made this or that wrong choice for expressing certain features; it’s that there’s simply far too much of it. “Too many notes,” as Joseph II supposedly said.

                                                                        1. 4

                                                                          I agree with this, which is why I object to blaming the syntax for it. For a language that needs to express so many features, Rust’s syntax is doing well.

                                                                          Rust chose to be a language that aims to have strong compile-time safety, low-level control, and nearly zero run-time overhead, while still having higher-level abstractions. Rust could drop a ton of features if it offered less control and/or moved checks to run-time or relaxed safety guarantees, but there are already plenty of languages that do that. Novelty of Rust is in not compromising in any of these, and this came at a cost of having lots of features to control all of these aspects.

                                                                          1. 4

                                                                            You can have many features without a lot of syntax. See Lisp.

                                                                            1. 3

                                                                              I think Lisp gets away here only on a technicality. It can still have plenty of obscure constructs to remember, like the CL’s (loop).

                                                                              The example from the article isn’t really any simpler or more readable if you lispify it:

                                                                              (try (map-result (static-call (def-lifetime a-heavy (:Trying :to-read a-heavy)) 
                                                                                  syntax (lambda (like) (can_be this maddening))) ()))
                                                                              

                                                                              It could be made nicer if it was formatted in multiple lines, but so would the Rust example.

                                                                              1. 2

                                                                                If you pick the feature set for simplicity. Rust had other goals.

                                                                                1. 4

                                                                                  I literally just said that simple syntax doesn’t necessitate simple features.

                                                                              2. 1

                                                                                I don’t know. I strongly suspect that in the coming years, we will see new languages that offer the same safety guarantees as Rust, also with no runtime, but with syntax that is simpler than Rust. Lately I’ve seen both Vale and Koko exploring this space.

                                                                          2. 12

                                                                            The syntax complexity of Rust is actually a big factor in why I abandoned my effort to learn it. I was only learning on my own time, and came to the realization I had a long way to go before I’d be able to pick apart a line like the author’s example.

                                                                            So for me, it wasn’t just superficial.

                                                                            1. 5

                                                                              The syntax complexity of Rust is actually a big factor in why I abandoned my effort to learn it.

                                                                              Same.

                                                                            2. 4

                                                                              If syntax impacts understandability, is it actually superficial?

                                                                              I’d say so.

                                                                              The problem is that “this syntax is ugly” is a completely subjective judgement largely influenced by the peculiarities of ones’ own background. Coming from Perl and Ruby, I happen to find Rust pleasant to look at and easy to read, whereas I find both Python and Go (which many other people prefer) unreasonably frustrating to read and just generally odd-looking. It’s not that Python and Go are doing anything objectively less understandable, per-se, but they’re certainly have an unfamiliar look, and people react to unfamiliarity as if it were objectively incorrect rather than just, well, making unfamiliar choices with unfamiliar tradeoffs.

                                                                              It’s pure personal preference, and framing ones’ personal preferences as something that has objective reality outside oneself and which some other party is doing “wrong” is, to me, the definition of a superficial complaint.

                                                                              1. 8

                                                                                It’s pure personal preference

                                                                                Is it pure personal preference? I dunno. Personal preference is a part of it, but I don’t think it’s controversial to say that Python is in general easier to understand than the q language, for example. Human cognition and coherence actually abides pretty well-defined rules, at the macro scale. Sigils are harder to grok than words. And so on.

                                                                                1. 12

                                                                                  Personal preference is a part of it, but I don’t think it’s controversial to say that Python is in general easier to understand than the q language, for example.

                                                                                  Maybe, maybe not. What I do think is that if we’re going to try to make objective claims, we need some real objective measures and measurements. These conversations tend to be nothing but pseudoscience-y assertions and anecdata masquerading as irrefutable facts.

                                                                                  Human cognition and coherence actually abides pretty well-defined rules, at the macro scale. Sigils are harder to grok than words.

                                                                                  (In no way am I trying to pick on you, but) Case in point: “Sigils are harder to grok than words” feels like a strong objective claim but… is this actually in any way true? 馬 is a much more complicated symbol than $ or @ or ->, but we have something like 1.5 billion people in the world happily reading and writing in languages that require a knowledge of thousands of such symbols to achieve literacy, and they turn out to somehow show lower rates of dyslexia than in alphabet based languages while doing so!

                                                                                  Sigil-y writing systems are indeed actually quite common throughout history, so again we have this thing where what feels like a perfectly simple fact actually looks a heck of a lot like a simple case of familiarity when you scratch it just a little bit. The dominance of a few alphabetic writing systems outside of Asia could simply be a historical accident for all we know – there are no strong results from cognitive science supporting any claim that it’s objectively more fit to “human cognition”. We really don’t have any idea whether words are simpler or more efficient than symbols, or whether python is a global maxima of readability, a local minima, or anything in between. There are almost no good studies proving out any of this, just a lot of handwaving and poorly supported claims based on what people happen to like or be most familiar with.

                                                                                  1. 2

                                                                                    馬 is a word. It happens to be written as a single character, but that doesn’t make it punctuation.

                                                                                    1. 2

                                                                                      I’m aware. I speak Japanese.

                                                                                      “Sigil” does not mean “punctuation”. It actually means something like “symbol with occult powers”, but in a programming context I think we can understand it as “symbol that conveys an important functional meaning”, like -> being the symbol meaning “returns a value of the following type”. The point being that OP was being pretty silly when they wrote that it’s a “rule of the human mind” that it’s easier to understand not written out as “not” rather than ! when the existence of a billion plus people using languages with things like “不” at least weakly implies that a single symbol for not is no more mentally taxing to understand.

                                                                                      (that in many programming languages most sigils are punctuation is mostly just an artifact of what’s easy to type on a western keyboard, but it’s by no means the rule. See: APL, which can be chockfull of non-punctuation sigils)

                                                                                      1. 1

                                                                                        The point is that the symbol has a natural pronunciation, which makes it easy to read for a Japanese spsaker. In contrast, when I see !foo or &foo or $foo, my mind just makes an unintelligible noise followed by “foo”, so I have to concentrate on what the symbol means.

                                                                                        1. 1

                                                                                          But these symbols all do have actual pronunciations that are generally specified in the language or are established conventionally, eg) !foo is read “not foo”, &foo is “addressof foo” (at least in C) or “ref foo” in Rust, etc. Good learning resources almost always provide a reading when they introduce the symbol (Blandy et al’s Programming Rust is very good about this, for instance).

                                                                                          Also fwiw not everyone “vocalizes” what they’re reading in their head, that’s actually not a universal thing.

                                                                                    2. 1

                                                                                      When I speak about “understandability” or whatever I’m not making a claim against an abstract Ur-human raised in a vacuum, I’m speaking about humans as they exist today, including cultural and historical influences, and measured on a demographic (macro) scale, rather than an individual (micro) scale. That is, I’m making a descriptive argument, not a normative one. In this context, “familiarity” is I guess a totally reasonable thing to account for! People understand better the things they are familiar with. Right?

                                                                                      1. 3

                                                                                        That is, I’m making a descriptive argument, not a normative one.

                                                                                        It’s not a very good descriptive argument, though, insofar as you’re really failing to describe a lot of things in order to make your argument fit the conclusion that “sigils are harder to grok than words”.

                                                                                        Even if we confine ourselves to Western English speakers… what about mathematics? Why does almost everyone prefer y = x+1 to Cobol’s ADD 1 TO X GIVING Y? It’s more familiar, right? There doesn’t seem to be any long-term push to make Mathematics more wordy over time (most of the established symbols have hung around for hundreds of years and had ample opportunity to get out-competed by more grokkable approaches, if word-based approaches were found by people to be any more grokkable), so if we’re describing the long-term pressures on artificial languages I don’t think “sigils are harder to grok than words” is an accurate descriptive statement.

                                                                                        In this context, “familiarity” is I guess a totally reasonable thing to account for! People understand better the things they are familiar with. Right?

                                                                                        Well, sure. But “in some contexts words are more familiar than sigils to western audiences” is a much different claim than “sigils are harder to grok than words” in any sense, and it leaves a lot more room to talk about sigils in programming languages in a rational way. Things like “dereferencing pointers” aren’t really familiar to anyone in words or sigils, so it’s not obvious to me that x = valueat y is any more or less “correct”/“intuitive”/“grokable” than x = *y.

                                                                                        If anything, given the relative unpopularity of the Pascal/Ada & Cobol language families, a certain amount of “unfamiliar concepts compressed into sigils” seems to be appreciated by programmers at large. But other people disagree, which seems to point at this mostly being a superficial argument over tastes and perhaps how much maths background one has, rather than some kind of concrete and objective variation in any measurable metric of “understandability”.

                                                                                        1. 2

                                                                                          what about mathematics?

                                                                                          Well, I think this substantiates my point? In the sense that way more people can read prose than can understand nontrivial math. Right?

                                                                                          “in some contexts words are more familiar than sigils to western audiences” is a much different claim than “sigils are harder to grok than words”

                                                                                          Not some but most or even almost all, depending on just how many sigils we’re talking about.

                                                                                          Authors generally don’t invent new languages in order to express their literary works; they take the language(s) they already know, with all their capabilities and constraints, and work within those rules. They do this because their goal is generally not to produce the most precise representation of their vision, but instead to produce something which can be effectively consumed by other humans. The same is true of programming.

                                                                                          1. 2

                                                                                            Well, I think this substantiates my point? In the sense that way more people can read prose than can understand nontrivial math. Right?

                                                                                            More people can read prose (in general) than the prose portions of an advanced Mathematics text (in specific). It’s not the orthography of mathematics that’s the limiting factor here.

                                                                                            Authors generally don’t invent new languages in order to express their literary works; they take the language(s) they already know, with all their capabilities and constraints, and work within those rules. They do this because their goal is generally not to produce the most precise representation of their vision, but instead to produce something which can be effectively consumed by other humans. The same is true of programming.

                                                                                            Which speaks to my point. Programming uses “sigils” because in many cases these sigils are already familiar to the audience, or are at least no less familiar to the audience for the concepts involved than anything else would be, and audiences seem to show some marked preference for sigils like { … } vs begin … end, y = x + 1 seems pretty definitely preferred for effective consumption by audiences over ADD 1 TO X GIVING Y, etc.

                                                                                            At any rate, we seem to have wandered totally away from “sigils are objectively less readable” and fully into “it’s all about familiarity”, which was my original point.

                                                                                            1. 2

                                                                                              I’m not claiming that sigils are objectively less readable than prose. I’m objecting to the notion that syntax is a superficial aspect of comprehension.

                                                                                              1. 1

                                                                                                You’ve made claims that terse syntax impedes comprehension (“Sigils are harder to grok than words”), where the reality is in the “it depends” territory.

                                                                                                For novices, mathematical notation is cryptic, so they understand prose better. But experts often prefer mathematical notation over prose, because its precision and terseness makes it easier for them to process and manipulate it. This is despite the fact that the notation is objectively terrible in some cases due to its ad-hoc evolution — even where the direction is right, we tend to get details wrong.

                                                                                                Forms of “compression” for common concepts keep appearing everywhere in human communication (e.g. in spoken languages we have contractions & abbreviations, and keep inventing new words for things instead of describing them using whole phrases), so I don’t think it’s an easy case of “terse bad verbose good”, but a trade-off between unfamiliarity and efficiency of communication.

                                                                                                1. 1

                                                                                                  I agree with all of your claims here.

                                                                                                2. 0

                                                                                                  . I’m objecting to the notion that syntax is a superficial aspect of comprehension.

                                                                                                  It’s not fully, but “the * operator should be spelled valueat/ {} should be spelled begin end” stuff is a superficial complaint unless and until we have objective, measurable reasons to favor one syntactical presentation over the other. Otherwise it’s just bikeshedding preferences.

                                                                                                  But I’m sorry, let’s not continue this. I’m not buying the goalpost move here. You wrote that human cognition obeys “well-defined rules. Sigils are harder to grok than words”. That’s pretty obviously a claim that “sigils are objectively less readable than prose” due to these “well defined rules of cognition”. That’s the kind of handwavey, pseudoscience-as-fact discourse I was objecting to and pointing out these discussions are always full of.

                                                                                                  I’ve pointed out that this is, in several ways, basically just a load of hot air inconsistent with any number of things true of humans in general (symbol based writing systems) and western readers in specific.

                                                                                                  Now your “well-defined rules of human cognition which include that sigils are less readable than words” weren’t trying to be an objective claim about readability?

                                                                                                  Sure. I’m done. Have a good one.

                                                                                2. 24

                                                                                  I would warmly suggest making an effort to hit Page Down twice to get past the syntax bit and read the rest of the post though, because it’s a pretty good and pragmatic take, based on the author’s experience writing and maintaining a kernel. Xous is a pretty cool microkernel which runs on actual hardware, it’s actually a pretty good test of Rust’s promises in terms of safety and security.

                                                                                  1. 10

                                                                                    It’s interesting but also has the weird dichotomy that the only two choices for systems programming are C or Rust. C++ also has a lot of the strengths that the author likes about Rust (easy to write rich generic data structures, for example), and has a bunch of other things that are useful in a kernel, such as support in the standard library for pluggable memory allocators, mechanisms for handling allocation failure, a stable standard library API, and so on.

                                                                                    1. 5

                                                                                      I had exactly the same thought. C++ burnt through a lot of good will in the C++98 era where it was admittedly a hot mess (and all the compilers where buggy dumpster fires). Now on one hand we have people who publicly and loudly swore off touching C++ ever again based on this experience (and even more people parroting the “C++ is a mess” statement without any experience) and on the other the excitement of Rust with all the hype making people invest a large amount of effort into learning it. But the result, as this article shows, is often not all roses. I believe oftentimes the result would have been better if people invested the same amount of time into learning modern C++. Oh well.

                                                                                      1. 5

                                                                                        Writing C++ is like writing Rust but with your whole program wrapped in unsafe{}. You have to manage your memory and hope you did it right.

                                                                                        1. 4

                                                                                          As I hope this article clearly demonstrates, there is a lot more to a language chice than memory safety. Also, FWIW, I write fairly large programs and I don’t find memory management particularly challenging in modern C++. At the same time, I highly doubt that these programs can be rewritten in Rust with the result having comparable performance, compilation times, and portability properties.

                                                                                          1. 1

                                                                                            What would hinder Rust from having comparable performance, compilation times, and portability properties, in your opinion?

                                                                                            1. 1

                                                                                              In summary:

                                                                                              Performance: having to resort to dynamic memory allocations to satisfy borrow checker.

                                                                                              Compilation: in Rust almost everything is a template (parameterized over lifetimes).

                                                                                              Portability: C/C++ toolchain is available out of the box. I also always have an alternative compiler for each platform.

                                                                                        2. 4

                                                                                          string_view of temporaries makes dangling pointers instead of compilation errors. optional allows unchecked dereferencing without warnings, adding more UB to the modern C++. I haven’t met a C++ user who agrees these are fatal design errors. Sorry, but this is not up to safe Rust’s standards. From Rust perspective modern C++ continues to add footguns that Rust was designed to prevent.

                                                                                          1. 1

                                                                                            I haven’t met a C++ user who agrees these are fatal design errors.

                                                                                            I haven’t used string_view much so can’t categorically say it’s not a design error (it very well may be). But for optional I can certainly say it is a trade-off: you have the choice of checked access (optional::value()) or unchecked and you decide what to use. I personally always use unchecked and never had any problems. Probably because I pay attention to what I am writing.

                                                                                            1. 5

                                                                                              This is the difference in approaches of the two languages. In C++ if the code is vulnerable, the blame is on the programmer. In Rust if the code is vulnerable, Rust considers it a failure of the language, and takes responsibility to stop even “bad” programmers from writing vulnerable code. I can’t stress enough how awesome it is that I can be a careless fool, and still write perfectly robust highly multi-threaded code that never crashes.

                                                                                              In terms of capabilities, Rust’s Option is identical, but the the default behavior is safe, and there’s a lot of syntax sugar (match, if let, tons of helper methods) to make the safe usage the preferred option even for “lazy” programmers. The UB-causing version is written unsafe { o.unwrap_unchecked() }, which is deliberately verbose and clunky, so that the dangerous version stands out in code reviews, unlike subtle * or -> that are commonly used everywhere.

                                                                                              Rust’s equivalent of string_view is &str, and it’s practically impossible to use the language without embracing it, and it’s borrow-checked, so it won’t compile if you misuse it.

                                                                                        3. 2

                                                                                          Eh, maybe the author just didn’t write that much low-level/kernel code in C++. I try not to read too much into these things. If I were to start learning F# tomorrow, then tried to write a similar piece two years from now, I’d probably end up with something that would have the weird dichotomy that the only two choices for functional programming are Scheme and F#.

                                                                                          1. 2

                                                                                            Scheme is honestly so hard to do functional in. It’s shockingly imperitive by nature given the reputation.

                                                                                        4. 3

                                                                                          I did read the entire post, but I wanted to voice that focusing on the wrong thing first makes people not take you seriously, especially when the author expresses it doesn’t matter, but they still decided to make it first?

                                                                                          1. 3

                                                                                            I may not be interpreting this correctly but I didn’t take the author qualifying it as a superficial complaint to mean that it doesn’t matter. Based on the issues he mentions regarding the readability of Rust macros, for example, I think it’s superficial as in “superficial velocity”, i.e. occurring or characterising something that occurs at the surface.

                                                                                            (But note that I may be reading too much into it because reviewing and auditing Rust code that uses macros is really not fun so maybe I’m projecting here…)

                                                                                        5. 20

                                                                                          The final sentence of that section said, in summary, “Rust just has a steep learning curve in terms of syntax”. A critical retrospective that does not mention the horrendous syntax or its learning curve would lack credibility.

                                                                                          1. 5

                                                                                            I find Rust’s syntax perfectly clear and sensible. I am not the only one.

                                                                                          2. 9

                                                                                            I liked that it starts with that TBH. Rust’s dense syntax is probably the first impression of the language for many people—it was for me at least. And putting the author’s first impression first in the article makes it read more like a person telling a story, rather then a list of technical observations sorted by importance.

                                                                                            I like to read stories by humans; i feel it easier to connect with the author and therefore to retain some of what they say. YMMV of course.

                                                                                            1. 2

                                                                                              And if they think rust is hard to read, wait until they discover lisp!

                                                                                              (I know this author probably is already familiar with lisp and many other things, but the comparison stands.)

                                                                                              1. 6

                                                                                                I find it the other way around. If you temporarily put aside the issues of special forms and macros, the syntax of Lisp is extremely minimal and regular (it’s almost all lists and atoms). So Lisp stands at kind of an opposite extreme from Rust, with more familiar languages somewhere in between.

                                                                                                1. 5

                                                                                                  Nim still has a dumpLisp routine to show you the shape of an AST you may want to manipulate.

                                                                                                  Syntax can be very personal, but I strongly prefer Nim’s to Rust’s and see no compelling language feature of Rust to tempt me away, though Nim is not without its own issues.

                                                                                                  1. 2

                                                                                                    Nim isn’t really comparable is it? More like Go with a GC etc?

                                                                                                    1. 2

                                                                                                      “How comparable” mostly depends upon what you mean by “a GC etc”. Nim’s (AutomaticRC/OptimizedRC) memory management seems fairly similar to Rust, but I am no Rust expert and most PLs have quite a few choices either directly or ecosystem-wide. (Even C has Boehm.) There is no “GC thread” like Java/Go. The ORC part is for cycle collection. You can statically specify {.acyclic.}, sink, lent, etc. in Nim to help run-time perf. Some links that go into more detail are: https://nim-lang.org/blog/2020/10/15/introduction-to-arc-orc-in-nim.html https://nim-lang.org/blog/2020/12/08/introducing-orc.html

                                                                                                      1. 0

                                                                                                        “Go with a GC” is Go.

                                                                                                        1. 1

                                                                                                          Yes, that’s why I said it

                                                                                                    2. 2

                                                                                                      The complaint in the article is about noisy hard to read though, and lisp is definitely that, even if it is simple and regular that simplicity leads everything to look the same.

                                                                                                      1. 3

                                                                                                        I always wondered why indentation-based reader macros (SRFI-49 is a simple one) never became popular. I can see “whys” for big macro writer types since they often want to pick apart parse trees and this adds friction there. Most programmers are not that (in any language). My best guess is a kind of community dynamic where tastes of village elders make beginners adapt more. Or wishful promises/hopes for beginners to become elders? Or a bit of both/etc.?

                                                                                                        Of course, part of the regularity is “prefix notation” which can remain a complaint.

                                                                                                  2. 1

                                                                                                    It makes it hard to take the rest of the post seriously

                                                                                                    As x64k said the post is pretty well made I think and some honest criticism. If anything you can criticize the bad blog layout, which has big white bars on mobile and desktop, giving you a hard time reading it from any device.

                                                                                                  1. 1

                                                                                                    The lack of fish is conspicuous here.

                                                                                                    1. 1

                                                                                                      It probably should be mentioned with all the other *nix shells, but it doesn’t really have any special built-in support beyond what the others have, unless I’ve forgotten something…

                                                                                                      1. 1

                                                                                                        it doesn’t really have any special built-in support beyond what the others have, unless I’ve forgotten something…

                                                                                                        Special? Maybe not. But different? Aye! To crib your example, what is in Bash

                                                                                                        $ myvar=$(dig www.google.com | jc --dig | jq -r '.[0].answer[0].data')
                                                                                                        $ echo $myvar
                                                                                                        64.233.185.104
                                                                                                        

                                                                                                        is in Fish

                                                                                                        $ set myvar (dig www.google.com | jc --dig | jq -r '.[0].answer[0].data')
                                                                                                        $ echo $myvar
                                                                                                        64.233.185.104
                                                                                                        
                                                                                                        1. 1

                                                                                                          But that’s not special support for JSON, and is just the fact that fish’s syntax is slightly different.

                                                                                                    1. 32

                                                                                                      its release philosophy is supposed to avoid what I call “the problem with Python”: your code stops working if you don’t actively keep up with the latest version of the language.

                                                                                                      This is a very real problem with the Python and Node ecosystems.

                                                                                                      1. 16

                                                                                                        I’m going to be grumpy and ask what this “problem” is supposed to mean, exactly. I can deploy, say, a Python application on a particular version of Python and a particular operating system today, and then walk away for years, and as long as I pay the hosting bill there is no technical reason why it would suddenly stop working. There’s no secret kill switch in Python that will say “ah-ha, it’s been too long since we made you upgrade, now the interpreter will refuse to start!”

                                                                                                        So what people always actually mean when they say things like this is that they want to keep actively developing their code but have everybody else stand still forever, so that they can permanently avoid platform upgrades but also the language and ecosystem will never leave them behind. So the real “problem” is that one day nobody else will be willing to provide further bug and security fixes (free or perhaps even paid) for the specific combination of language/libraries/operating system I initially chose to build my app with, and on that day I have to choose between upgrading to a new combination, or doing that maintenance myself, or going without such maintenance entirely.

                                                                                                        And although I’ve seen a lot of claims, I’ve seen no evidence as yet that Rust has actually solved this problem — the editions system is still young enough that the Rust community has not yet begun to feel the pain and cost of what it committed them to. “Stability without stagnation” is a nice slogan but historically has ranged from extremely difficult to downright impossible to achieve.

                                                                                                        1. 16

                                                                                                          “Stability without stagnation” is a nice slogan but historically has ranged from extremely difficult to downright impossible to achieve.

                                                                                                          You just don’t make breaking changes. Add new APIs but don’t take away old ones. Go 1 has been stable for 10 years now. That means there are some ugly APIs in the standard library and some otherwise redundant bits, but it’s been fine. It hasn’t stopped them from adding major new features. Similarly, JavaScript in the browser only rolls forward and is great. Again, some APIs suck, but you can just ignore them for the most part and use the good versions. I’m not as familiar with the Linux kernel, but my impression there is that Linus’s law is you don’t break userland.

                                                                                                          Most breaking changes are gratuitous. They make things a little nicer for the core developers and shove the work of repair out to everyone who uses their work. I understand why it happens, but I reserve the right to be grumpy about it, especially because I have the experience of working in ecosystems (Go, the browser) that don’t break, so I am very annoyed by ecosystems that do (NPM, Python).

                                                                                                          1. 15

                                                                                                            I’ll be up-front. My stance on breaking changes is that there are two types of projects:

                                                                                                            1. Those which have decided to accept they are an inevitable fact of software development, and so have committed to the process of handling them, and
                                                                                                            2. Those which haven’t yet.

                                                                                                            And remember that we are talking not just about core language/standard library, but also ecosystems. Go has already effectively broken on this — the “rename your module to include /v2” hack is an admission of defeat, and allows the previous version to lapse into a non-maintained status. Which in turn means that sooner or later you will have to make the choice I talked about (upgrade, or do maintenance yourself, or go without maintenance).

                                                                                                            And Rust has had breaking changes built into the ecosystem from the beginning. The Cargo/crates ecosystem isn’t built around every crate having to maintain eternal backwards-compatibility, it’s built on semantic versioning. If I publish version 1.0 of a crate today, I can publish 2.0 with breaking changes any time I want and stop maintaining the 1.x series, leaving users of it stranded.

                                                                                                            So even if Rust editions succeed as a way of preserving compatibility of the core language with no ecosystem splits and no “stagnation”, which I strongly doubt in the long term, the ecosystem has already given up, and in fact gave up basically immediately. It has “the Python problem” already, and the only real option is to learn how to manage and adapt to change, not to insist that change is never permitted.

                                                                                                            (and of course my own experience is that the “problem” is wildly exaggerated compared to how it tends to impact projects in actual practice, but that’s another debate)

                                                                                                            1. 5

                                                                                                              I think it’s hard to talk about this rigorously because there’s definitely some selection bias at play — we don’t have information about all the internal projects out there that might be bogged down by breaking changes between interpreter versions, and they’re likely motivated by very different incentives than the ones that govern open source projects — and there’s likely survivorship bias at play too, in that we don’t hear about the projects that got burnt out on the maintenance burden those breaking changes induce.

                                                                                                              My anecdotal evidence is that I’ve worked at places with numerous Python projects bound to different, sometimes quite old, interpreter versions, and there just aren’t enough person-hours available to keep them all up to date, and updating them in the cases where it was truly necessary made for some real hassle. Even if you chalk that up to bad resource management, it’s still a pretty common situation for an organization to find itself in, and it’s reasonable to expect your tools to not punish you for having less than perfect operational discipline. In light of that, I think understanding it in this binary frame of either making breaking changes or not isn’t the most fruitful approach, because as you note it’s not realistic to expect that they never happen. But when they do happen, they cost, and I don’t think it’s unreasonable for an organization to weigh that total cost against their resources and decide against investing in the Python ecosystem. It’s not unreasonable to make the opposite choice either! I just don’t think that cost is trivial.

                                                                                                              1. 7

                                                                                                                Even if you chalk that up to bad resource management, it’s still a pretty common situation for an organization to find itself in and it’s reasonable to expect your tools to not punish you for having less than perfect operational discipline.

                                                                                                                Imagine 20 years ago saying this about a project that suffered because they lost some crucial files and it turned out they weren’t using any kind of version control system.

                                                                                                                Because that’s basically how I feel about it. Regularly keeping dependencies, including language tooling/platform, up-to-date, needs to become table stakes for software-producing entities the way that version control has. I’ve seen this as a theme now at four different companies across more than a decade, and the solution is never to switch and pray that the next platform won’t change. The solution is always to make change an expected part of the process. It can be done, it can be done in a way that minimizes the overhead, and it produces much better results. I know because I have done it.

                                                                                                                1. 4

                                                                                                                  Because that’s basically how I feel about it. Regularly keeping dependencies, including language tooling/platform, up-to-date, needs to become table stakes for software-producing entities the way that version control has.

                                                                                                                  I don’t believe this is what’s being disputed: it’s that this being a fact is precisely why it’s important for the platform to facilitate ease of maintenance to the best of its ability in that regard. Doing so allows even under-resourced teams to stay on top of the upgrade treadmill, which is more of my point in the bit you quoted: tools that induce less overhead are more resilient to the practical exigencies that organizations face. I guess we’ll have to agree to disagree about where the Python ecosystem sits on that spectrum.

                                                                                                                  1. 2

                                                                                                                    This sounds like the “don’t write bugs” school of thought to me. Yes, ideally anything that’s an operational concern will get ongoing maintenance. In the real world… 

                                                                                                                    More anecedata: my coworker built a small Python scraper that runs as a cron job to download some stuff from the web and upload it to an S3 bucket. My coworker left and I inherited the project. The cron job was no longer a high priority for the company, but we didn’t want to shut it off either. I couldn’t get it to run on my machine for a while because of the Python version problem. Eventually I got to the point where I could get it to run by using Python 3.6, IIRC, so that’s what it’s using to this day. Ideally, if I had time and resources I could have figured out why it was stuck and unstick it. (Something to do with Numpy, I think?) But things aren’t always ideal.

                                                                                                                    If someone has to have discipline and try to stick closer to the “don’t write bugs” school, who should it be: language creators or end developers? It’s easy for me to say language creators, but there are also more of us (end developers) than them (language creators). :-) ISTM that being an upstream brings a lot of responsibility, and one of those should be the knowledge that your choices multiply out by all the people depending on you: if you impose a 1 hour upgrade burden on the 1 million teams who depend on you, that’s 1 million hours, etc.

                                                                                                                    1. 1

                                                                                                                      Generalizing from anecdata and my own experience, the killer is any amount of falling behind. If something is not being actively maintained with dependency updates on at least a monthly and ideally a weekly cadence, it is a time bomb. In any language, on any platform. Because the longer you go without updating things, the more the pending updates pile up and the more work there will be to do once you do finally sit down and update (which for many projects, unfortunately, tends to be only when they are absolutely forced to start doing updates and not a moment sooner).

                                                                                                                      At my last employer I put in a lot of work on making the (Python) dependency management workflow as solid as I could manage with only the standard packaging tooling (which I believe you may have read about). But the other part of that was setting up dependabot to file PRs for all updates, not just security, to do so on a weekly basis, and to automate creation of Jira tickets every Monday to tell the team that owned a repository to go look at and apply their dependabot PRs. When you’re doing it on that kind of cadence it averages very little time to review and apply the updates, you find out immediately from CI on the dependabot PRs if something does have a breaking change so you can scope out the work to deal with it right then and there, and you never wind up in a situation where applying the one critical update you actually cared about takes weeks or months because of how much other stuff you let pile up in the meantime.

                                                                                                                      Meanwhile I still don’t think Python or its ecosystem are uniquely bad in terms of breaking changes. I also don’t think Go or Rust are anywhere near as good as the claims made for them. And the fact that this thread went so quickly from absolutist “no breaking changes ever” claims to basically people’s personal opinions that one language’s or ecosystem’s breaking changes are justified and tolerable while another’s aren’t really shows that the initial framing was bad and was more or less flamebait, and probably should not be used again.

                                                                                                                      1. 7

                                                                                                                        No, I still think you’re wrong. :-)

                                                                                                                        I agree that for a Python or Node project it is recommended to set up dependabot to keep up to date or else you have a ticking time bomb. However, a) that isn’t always practical and b) it doesn’t have to be like that. I routinely leave my Go projects unattended for years at a time, come back, upgrade the dependencies, and have zero problems with it.

                                                                                                                        Here is a small project last touched in 2017 that uses Go and Node: https://github.com/baltimore-sun-data/track-changes

                                                                                                                        Here is the full Terminal output of me getting it to build again with the most recent version of Go:

                                                                                                                        (Fri, May 20  08:56:09 PM) (master|✔)
                                                                                                                        $ go build .
                                                                                                                        go: cannot find main module, but found Gopkg.lock in /var/folders/p7/jc4qc9n94r3f6ylg0ssh1rq00000gs/T/tmp.S6ZYg4FX/track-changes
                                                                                                                                to create a module there, run:
                                                                                                                                go mod init
                                                                                                                        # status: 1 #
                                                                                                                        (Fri, May 20  08:56:27 PM) (master|✔)
                                                                                                                        $ go mod init github.com/baltimore-sun-data/track-changes
                                                                                                                        go: creating new go.mod: module github.com/baltimore-sun-data/track-changes
                                                                                                                        go: copying requirements from Gopkg.lock
                                                                                                                        go: to add module requirements and sums:
                                                                                                                                go mod tidy
                                                                                                                        (Fri, May 20  08:57:00 PM) (master|…)
                                                                                                                        $ go mod tidy -v
                                                                                                                        go: finding module for package github.com/stretchr/testify/assert
                                                                                                                        go: finding module for package golang.org/x/text/unicode/norm
                                                                                                                        go: finding module for package golang.org/x/text/secure/bidirule
                                                                                                                        go: finding module for package golang.org/x/text/unicode/bidi
                                                                                                                        go: finding module for package golang.org/x/sync/errgroup
                                                                                                                        go: finding module for package github.com/stretchr/testify/suite
                                                                                                                        go: downloading golang.org/x/sync v0.0.0-20220513210516-0976fa681c29
                                                                                                                        go: found github.com/stretchr/testify/assert in github.com/stretchr/testify v1.7.1
                                                                                                                        go: found github.com/stretchr/testify/suite in github.com/stretchr/testify v1.7.1
                                                                                                                        go: found golang.org/x/text/secure/bidirule in golang.org/x/text v0.3.7
                                                                                                                        go: found golang.org/x/text/unicode/bidi in golang.org/x/text v0.3.7
                                                                                                                        go: found golang.org/x/text/unicode/norm in golang.org/x/text v0.3.7
                                                                                                                        go: found golang.org/x/sync/errgroup in golang.org/x/sync v0.0.0-20220513210516-0976fa681c29
                                                                                                                        (Fri, May 20  08:57:10 PM) (master|…)
                                                                                                                        $ go build .
                                                                                                                        (Fri, May 20  08:57:18 PM) (master|…)
                                                                                                                        $ 
                                                                                                                        

                                                                                                                        As you can see, it took about a minute for me to get it building again. Note that this package predates the introduction of Go modules.

                                                                                                                        Let’s upgrade some packages:

                                                                                                                        (Fri, May 20  09:02:07 PM) (master|…)
                                                                                                                        $ go get -v -u ./...
                                                                                                                        go: downloading golang.org/x/net v0.0.0-20220520000938-2e3eb7b945c2
                                                                                                                        go: downloading cloud.google.com/go v0.101.1
                                                                                                                        go: downloading gopkg.in/Iwark/spreadsheet.v2 v2.0.0-20220412131121-41eea1483964
                                                                                                                        go: upgraded cloud.google.com/go v0.16.0 => v0.100.2
                                                                                                                        go: added cloud.google.com/go/compute v1.6.1
                                                                                                                        go: upgraded github.com/ChimeraCoder/anaconda v1.0.0 => v2.0.0+incompatible
                                                                                                                        go: upgraded github.com/andybalholm/cascadia v0.0.0-20161224141413-349dd0209470 => v1.3.1
                                                                                                                        go: upgraded github.com/garyburd/go-oauth v0.0.0-20171004151416-4cff9ef7b700 => v0.0.0-20180319155456-bca2e7f09a17
                                                                                                                        go: upgraded github.com/go-chi/chi v3.3.1+incompatible => v4.1.2+incompatible
                                                                                                                        go: upgraded github.com/golang/protobuf v0.0.0-20171113180720-1e59b77b52bf => v1.5.2
                                                                                                                        go: upgraded github.com/pkg/errors v0.8.0 => v0.9.1
                                                                                                                        go: upgraded golang.org/x/net v0.0.0-20171107184841-a337091b0525 => v0.0.0-20220520000938-2e3eb7b945c2
                                                                                                                        go: upgraded golang.org/x/oauth2 v0.0.0-20171117235251-f95fa95eaa93 => v0.0.0-20220411215720-9780585627b5
                                                                                                                        go: upgraded google.golang.org/appengine v1.0.0 => v1.6.7
                                                                                                                        go: added google.golang.org/protobuf v1.28.0
                                                                                                                        go: upgraded gopkg.in/Iwark/spreadsheet.v2 v2.0.0-20171026120407-29680c88e31d => v2.0.0-20220412131121-41eea1483964
                                                                                                                        (Fri, May 20  09:02:52 PM) (master|…)
                                                                                                                        $ go build .
                                                                                                                        # github.com/baltimore-sun-data/track-changes
                                                                                                                        ./handler.go:28:14: undefined: middleware.DefaultCompress
                                                                                                                        # status: 2 #
                                                                                                                        (Fri, May 20  09:02:58 PM) (master|…)
                                                                                                                        $ go doc middleware
                                                                                                                        package middleware // import "github.com/go-chi/chi/middleware"
                                                                                                                        
                                                                                                                        // snip
                                                                                                                        
                                                                                                                        (Fri, May 20  09:03:12 PM) (master|…)
                                                                                                                        $ go doc middleware.Compress
                                                                                                                        package middleware // import "github.com/go-chi/chi/middleware"
                                                                                                                        
                                                                                                                        func Compress(level int, types ...string) func(next http.Handler) http.Handler
                                                                                                                            Compress is a middleware that compresses response body of a given content
                                                                                                                            types to a data format based on Accept-Encoding request header. It uses a
                                                                                                                            given compression level.
                                                                                                                        
                                                                                                                            NOTE: make sure to set the Content-Type header on your response otherwise
                                                                                                                            this middleware will not compress the response body. For ex, in your handler
                                                                                                                            you should set w.Header().Set("Content-Type",
                                                                                                                            http.DetectContentType(yourBody)) or set it manually.
                                                                                                                        
                                                                                                                            Passing a compression level of 5 is sensible value
                                                                                                                        
                                                                                                                        (Fri, May 20  09:03:32 PM) (master|…)
                                                                                                                        $ subl .
                                                                                                                        (Fri, May 20  09:04:12 PM) (master|…)
                                                                                                                        $ go build .
                                                                                                                        (Fri, May 20  09:04:59 PM) (master|✚1…)
                                                                                                                        $
                                                                                                                        

                                                                                                                        Took about 3 minutes to upgrade the packages and fix the broken dependency (they renamed a middleware). Bear in mind that the upgrade I did deliberately did not try to upgrade past semantic version changes in its dependencies. Probably it would take another half hour or more if I wanted to chase down whatever breaking changes happened there.

                                                                                                                        Suffice it to say, yarn cannot even install its packages, and the last time I tried this stunt a couple of years ago, I got past that and then ran into a problem with webpack that I couldn’t easily solve.

                                                                                                                        Go is just a much more stable ecosystem than Node or Python. It’s not as stable as say browser JS, where one can reasonably expect working code to work until civilization collapses, but it’s fairly stable. And it’s not magic. If there were a communal expectation of this level of stability, it could exist everywhere. It’s a social value to keep things working in the Go ecosystem, and it’s not elsewhere.

                                                                                                                        1. 2

                                                                                                                          The other day I returned to a Python package I hadn’t touched in about a year. The actual Python dependencies portion of updating it was done in a few minutes: I updated the supported versions of Python to those currently supported by upstream, did the same for the supported versions of Django, and then had to change a whopping four lines of code, all in a unit-test file, to deal with a deprecation in Django that had finally been removed.

                                                                                                                          The entire remainder of getting it ready for a new release was fighting with CI — updating from v1 to v3 of the GitHub Actions tasks for Python, which I mostly did by copy/pasting from a package by someone else who I trust.

                                                                                                                          I mention this because while you have anecdotes about Go projects updating more or less seamlessly, I have just as many about Python projects, and other people in this thread have anecdotes about Go projects breaking in ways they found annoying.

                                                                                                                          All of which is to say that you should stop trying to extrapolate from your anecdata to “Go is stable and values stability, while Python is not and does not”, because it just ends up looking silly when other people show up with their anecdata. Python is not uniquely “unstable” and neither Go nor Rust are uniquely “stable”. At best, some projects are sometimes lucky enough that they can give the false impression of stability in the language/ecosystem, despite the fact that the language/ecosystem is always moving on. And that’s why I have said, over and over, that the thing to do is embrace and accept change and build processes around it. Otherwise, you’re likely to wake up one day to find out that what you thought was stable and unchanging was neither, and that you are in for a lot of trouble.

                                                                                                                          1. 4

                                                                                                                            When I get a chance, I thought of an equivalently old Python package for me to try updating. I’ll try to do it this weekend or next week.

                                                                                                                            But I just don’t buy this:

                                                                                                                            Python is not uniquely “unstable” and neither Go nor Rust are uniquely “stable”.

                                                                                                                            I don’t have experience with Rust, so I have no idea there. I do have years of working in Python, JavaScript, and Go and my experience is uniform: Python and JavaScript routinely have problems that make installing/updating take a workday, and Go does not. I’ve already given a lot of concrete examples, and I’m sure I could dig through my git history and find more. At a certain point, all I can say is this is my experience and if it’s not yours, great.

                                                                                                                            1. 2

                                                                                                                              You’re still missing the point I’m trying to make, though, because you keep diving for anecdotes to support your case, and I keep trying to remind you that if we allow generalizing from anecdotes then your own claims will get contradicted, because there are people who can bring similar anecdotes about languages like Go that you think don’t have this “problem”.

                                                                                                                              My stance here is and always has been that no language or ecosystem is free of the need to keep up with dependencies and no language or ecosystem is free from breaking changes. People in this comment thread have posted experiences with Go breaking on them and you’ve mostly just ignored that or disallowed generalizations from them, while insisting that your own anecdotes support generalizations, about the objective state of particular languages/ecosystems.

                                                                                                                              That is what I’m trying to get you to see, and trying to break you out of. No number of anecdotes one way or another will make your case for you, because the issue is not an insufficient number of presented anecdotes.

                                                                                                                              At a certain point, all I can say is this is my experience and if it’s not yours, great.

                                                                                                                              This is very close to where I’m trying to lead you, but

                                                                                                                              • You never actually just say that and stop — you always feel a need to throw on another anecdote and then insist that your experience generalizes to objective statements about particular languages/ecosystems, and that’s where it goes off the rails.
                                                                                                                              • You never really accept other people having experiences that don’t match yours, and thus disputing your generalizations. I don’t have the kinds of problems you do with Python. You don’t have the kinds of problems other people have had with Go. But I’m doing my best to acknowledge experiences other than my own when I say that no language/ecosystem is free of this issue: I expect that there will be people who do run into it in Python, and also in Go, and also in Rust, and also in every other language/ecosystem. You seem to be trying to erase the experiences of people who run into it in languages that you subjectively don’t believe have this problem, because it doesn’t fit the “this language has it, that language doesn’t” generalization you want to make.
                                                                                                                              1. 3

                                                                                                                                your own claims will get contradicted, because there are people who can bring similar anecdotes about languages like Go that you think don’t have this “problem”.

                                                                                                                                I have given a ton of anecdotes and done two experiments in support of my view. On the other side, there is your a priori assertion that all ecosystems have the same problems (but is it of the same magnitude?) and an unreadably long article by a pro-Rust troll. If other people have anecdotes about Go being hard to upgrade, I’m happy to read them (assuming they can get to a point in less than 10,000 words) and theorize about why someone else might have that problem when I don’t. But that hasn’t happened in this thread.

                                                                                                                                1. 3

                                                                                                                                  Well, right here in this thread someone mentioned problems with upgrading in Go. But the way you and another user were trampling all criticism of Go with insults — “troll”, “clickbait”, and so on — literally drove that person out of the thread.

                                                                                                                                  I honestly don’t know what else to tell you. It seems very clear that you’re not prepared to hear anything that contradicts the narrative you’ve got going, so I guess I’ll bow out too since it’s pointless to continue trying to get you to acknowledge stuff that was literally right in front of you.

                                                                                                                                  1. 2

                                                                                                                                    I could never tell what the actual problems were based on the links posted by m0th were. They linked to long pieces with no summaries, so while I would love to comment, I cannot. Maybe it’s because they felt targeted by ad hominem. I have a different theory for why, which is that the criticisms were scattershot and not serious. Again, if they want to come back and provide a useable summary, great.

                                                                                                                                    1. 1

                                                                                                                                      And yet I’d bet anything that if you’d been given short concise statements of problems you’d dismiss them as being trivial or minor on those grounds. Like someone else already said, I simply cannot assume that you’re actually trying in good faith to engage with criticism of a thing you like; instead all I’ve seen you do is smear and deflect and dismiss without argument.

                                                                                                                                      1. 2

                                                                                                                                        If you call this “without argument” I don’t know what to say. I really feel that the “without argument” side is yours, and it is very frustrating to me, because you’re a writer whose work I respect which makes it very hard for me to let this go. But when you write things like “anecdotes don’t count”… What is a personal dev blog but anecdotes? I agree that supporters of language X can be vociferous and annoying, and Go is one such X. In that case of Go, the vociferous attitude comes from having been attacked as using a “bad language” by people like fasterthanlime and our own experience of initially thinking things like the v2 are dumb and then slowly seeing the benefits and defending them. I agree that my initial claim was a bit too glib in its characterization, but I stand by a less inflammatory version of the same idea. I don’t agree that anecdotes aren’t the data of software engineering, and I don’t agree it’s somehow impossible to characterize software ecosystems with the understanding that any generalization will always be at best partial and incomplete but nevertheless it’s better to have the generalization than not.

                                                                                                                                        1. 2

                                                                                                                                          People brought up criticism of Go. You didn’t engage with it or argue with the points made — you ignored them, or dismissed them as too long to be worth your time to read, or just outright insulted the author of one piece of Go criticism.

                                                                                                                                          It’s hard for me to see a way to take this as good faith. It appears that you like Go, which is fine! Same with Python: if you don’t like it, then don’t like it — that’s fine!

                                                                                                                                          What’s not fine is the double-standard “my anecdotes are generalizable to broad claims about the language/ecosystem as a whole, but other people’s anecdotes are not”. Which is basically what you’re doing. If your anecdotes about Python generalize, then so do mine and so do other people’s anecdotes about Go. Which would then contradict the claims you want to make, which is probably why you’re being so cagey about not allowing others’ anecdotes to generalize while insisting that yours do. But again it’s going to be, at least, extremely difficult to take that as a good-faith argument.

                                                                                                                                          1. 2

                                                                                                                                            When I engage with the criticism, I’m just dismissing it. If I don’t engage because I don’t have enough detail (or drowning in irrelevant details so I can’t tell why monotonic time is supposedly bad), I’m ignoring it. It’s no win.

                                                                                                                                            1. 2

                                                                                                                                              As far as I can tell the only “engagement” you’ve given to criticism of Go was the second paragraph in this comment, and I think that’s being charitable since it’s also possible to read as a “well, this breaking change didn’t count, and it just means Go is great instead of the greatest”. If I said Python’s stability guarantees are good but just “don’t go far enough”, I doubt you’d view it charitably.

                                                                                                                                              You have, on the other hand, repeatedly dismissed critiques both by users in this thread and linked to on other sites as being too long or too “scattershot” or “fair to handwave … away”. Again: if I tried pulling that sort of thing in defense of Python you would, I hope, call me on it. You’re doing it here in defense of Go. I’m calling you on it.

                                                                                                                                              1. 2

                                                                                                                                                I wrote a long reply, decided to sleep on it, and I give up. You haven’t conceded any points. You’ve repeatedly accused me of bad faith. You’ve decided Python is as good as it gets and by definition nothing else can be better in some ways but not others. All I can say is I didn’t invent the phrase “the problem with Python”. Other people feel this way too. Maybe we’re wrong, and everything has the same problem to the same degree. I don’t think so though.

                                                                                                                                2. 2

                                                                                                                                  You seem to be simultaneously saying there is nothing especially broken with python while defending the culture of constantly breaking changes and saying it is inevitable. I think your attitude sums up the problem, there are a lot of enablers.

                                                                                                                                  1. 3

                                                                                                                                    This thread started out with an accusation that there is a problem that uniquely exists in certain languages, and does not exist at all in others. People were very confidently stating not that the breaking changes in Go are less frequent, or less annoying, or more justified — people were very confidently stating that breaking changes simply do not exist at all in Go.

                                                                                                                                    Don’t believe me? Here, and a lot of the rest of this sub-thread is people trying to quibble about “oh well that was just a change to (thing that shouldn’t count)” or explain “well these changes were justified and tolerable while those other languages’ changes aren’t” in order to somehow cling to the notion that Go is a bastion of perfect stable compatibility, even in the face of solid evidence that it isn’t.

                                                                                                                                    And I’m sorry if people don’t like to hear it, but I’ve just gotta call bullshit on that. Every language and ecosystem has breaking changes. There is no magic in Go that makes it somehow unchanging and perfectly compatible forever, nor is there any magic in the ecosystem that somehow makes all maintainers good stewards who never ever make unjustified changes.

                                                                                                                                    Still, people seem to be wildly exaggerating both the frequency of breaking changes in the languages they want to criticize, while minimizing for the languages they want to praise. It’s very human and very subjective, and very not a good way to have a reasoned discussion about this.

                                                                                                                                    Especially when it’s coupled with very obvious bad-faith tactics like the way critics of Go keep being dismissed or even smeared in this thread.

                                                                                                                                    If you dislike Python, then by all means dislike it. There are things I dislike about it. But there’s way too much in this thread, and in our industry, of people being unable to handle anything other than extremes — something is either 100% good or 0%, either completely perfectly backwards-compatible always or never — and of very clearly contradicting themselves and making really bad arguments to try to justify their subjective dislikes. Just come out and say you don’t like a thing and move on. You don’t also have to prove that it’s the worst in order to justify your dislike.

                                                                                                                                    1. 2

                                                                                                                                      I don’t know if python is especially bad. I personally found npm projects I have tried break when updating dependencies a few years later, and I have not had that experience in Go. Maybe I am just lucky.

                                                                                                                                      1. 1

                                                                                                                                        It seems to me that the disconnect here is about the scope of what constitutes “breaking changes” in a language. There certainly isn’t an objective definition! Some people consider only the core language and it’s standard library, and others include everything out to its tooling.

                                                                                                                                  2. 1

                                                                                                                                    Okay, I tried this with a Datasette project from around the same time. (It was actually deployed again in early 2020, so it’s a bit fresher than the Go project, but whatever, close enough.) Again, Node didn’t work. I think the issue with that is that libsass is dead and doesn’t compile anymore, so you need to switch to dart-sass instead. In all likelihood, the fastest solution to the Node issues is to drop all of the dependencies and just start over from scratch with only my user code, since the dependencies were just there to build a Vue project.

                                                                                                                                    On the Python side, it wouldn’t work with Python 3.9, but when I used Python 3.7, I got it to run again. Terminal output is below. It only took 15 minutes to get it going, but compare this to Go, which works with the current version of Go even though the package predates modules (which caused a lot of breakage not covered by the Go 1 guarantee) and I got the dependencies upgraded in a total of 5 minutes. By contrast, the Python installations all took long enough that they break flow: since the installation is going to take a while, I switch away from my Terminal, which is chance for me to get distracted and lose my place. I think this project did pretty well because it used your recommended pattern of having a requirements-freeze.txt file and it had a Bash script to automate the actual install commands. But the error when UVLoop was broken was pretty demoralizing: I have no idea how I would fix it, so getting up to Python 3.9 or 3.10 would probably involve a lot more Googling than I’m willing to do for an internet comments example. Again, the simplest fix might be to just blow away what I have now and start from scratch. I think Datasette has been relatively stable in spite of being <1.0, so I suspect that it wouldn’t be that hard to get it working, but again, it’s more than I want to do for an example. A nice thing about Go is that most dependencies don’t use C, so when something does go wrong, like that middleware that was broken in the other project, you aren’t confronted with errors in a language you don’t know using a build system you don’t understand. In general, it’s just much less intimidating to get a Go project back up to speed.

                                                                                                                                    So this is fairly reflective of my lived experience: Node projects, especially those that use Webpack, break in ways that are more or less unfixable and need to be restarted from scratch; Python projects can be kept running if you are willing to pin old versions of Python but give a lot of scary compiler errors and don’t have clear paths forward; Go projects can typically be upgraded by typing go get -u ./... and maybe reading some release notes somewhere. Go isn’t perfect and there are still problems, but the quantity of problems is so much less than it creates a qualitative difference in feeling.

                                                                                                                                    $ gh repo clone baltimore-sun-data/salaries-datasette
                                                                                                                                    Cloning into 'salaries-datasette'...
                                                                                                                                    remote: Enumerating objects: 1017, done.
                                                                                                                                    remote: Counting objects: 100% (94/94), done.
                                                                                                                                    remote: Compressing objects: 100% (72/72), done.
                                                                                                                                    remote: Total 1017 (delta 63), reused 23 (delta 20), pack-reused 923
                                                                                                                                    Receiving objects: 100% (1017/1017), 53.78 MiB | 4.25 MiB/s, done.
                                                                                                                                    Resolving deltas: 100% (568/568), done.
                                                                                                                                    Updating files: 100% (74/74), done.
                                                                                                                                    (Sat, May 21  12:45:40 PM)
                                                                                                                                    $ cd salaries-datasette/
                                                                                                                                    (Sat, May 21  12:45:54 PM) (master|✔)
                                                                                                                                    $ more README.md
                                                                                                                                    # salaries-datasette
                                                                                                                                    Public salary data acquired by the Baltimore Sun. Currently, we just have data from the state of Maryland for 2017.
                                                                                                                                    
                                                                                                                                    ## Usage
                                                                                                                                    
                                                                                                                                    Run `./run.sh setup` to install locally. The script assumes you have either Python 3 or Homebrew for Mac installed. Run `./run.sh setup-frontend` to install front end dependencies.
                                                                                                                                    
                                                                                                                                    Run `./run.sh create-db` to create a SQLite database out of the provided CSVs.
                                                                                                                                    
                                                                                                                                    Run `./run.sh` or `./run.sh serve` to run server at http://localhost:9001.
                                                                                                                                    
                                                                                                                                    Run the JS/CSS frontend server in another tab with `./run.sh frontend`.
                                                                                                                                    
                                                                                                                                    `./run.sh format` will format Python and Javascript code according to the coding standards of the project.
                                                                                                                                    
                                                                                                                                    `Dockerfile` is also provided for running/deploying with Docker. The image can be built with `./run.sh docker-build` and tested with `./run.sh docker`. The server only responds to correct hostnames (not localhost), so edit `/etc/hosts` to add `127.0.0.1   local.salaries.news.baltimoresun.com` and then test http://local.salaries.news.baltimoresun.com in the browser.
                                                                                                                                    (Sat, May 21  12:46:06 PM) (master|✔)
                                                                                                                                    $ ./run.sh setup
                                                                                                                                    
                                                                                                                                    snip a ton of output from installing things including a lot of scary errors
                                                                                                                                    
                                                                                                                                    × Encountered error while trying to install package.
                                                                                                                                    ╰─> uvloop
                                                                                                                                    
                                                                                                                                    note: This is an issue with the package mentioned above, not pip.
                                                                                                                                    hint: See above for output from the failure.
                                                                                                                                    # status: 1 #
                                                                                                                                    (Sat, May 21  12:49:40 PM) (master|✔)
                                                                                                                                    $ # by reading the DOCKERFILE, I learn that this used Python 3.7 when it was made
                                                                                                                                    (Sat, May 21  12:51:16 PM) (master|✔)
                                                                                                                                    $ pyenv install 3.7.13
                                                                                                                                    python-build: use openssl@1.1 from homebrew
                                                                                                                                    python-build: use readline from homebrew
                                                                                                                                    Downloading Python-3.7.13.tar.xz...
                                                                                                                                    -> https://www.python.org/ftp/python/3.7.13/Python-3.7.13.tar.xz
                                                                                                                                    Installing Python-3.7.13...
                                                                                                                                    (Stripping trailing CRs from patch.)
                                                                                                                                    patching file Doc/library/ctypes.rst
                                                                                                                                    (Stripping trailing CRs from patch.)
                                                                                                                                    patching file Lib/test/test_unicode.py
                                                                                                                                    (Stripping trailing CRs from patch.)
                                                                                                                                    patching file Modules/_ctypes/_ctypes.c
                                                                                                                                    (Stripping trailing CRs from patch.)
                                                                                                                                    patching file Modules/_ctypes/callproc.c
                                                                                                                                    (Stripping trailing CRs from patch.)
                                                                                                                                    patching file Modules/_ctypes/ctypes.h
                                                                                                                                    (Stripping trailing CRs from patch.)
                                                                                                                                    patching file setup.py
                                                                                                                                    (Stripping trailing CRs from patch.)
                                                                                                                                    patching file 'Misc/NEWS.d/next/Core and Builtins/2020-06-30-04-44-29.bpo-41100.PJwA6F.rst'
                                                                                                                                    (Stripping trailing CRs from patch.)
                                                                                                                                    patching file Modules/_decimal/libmpdec/mpdecimal.h
                                                                                                                                    (Stripping trailing CRs from patch.)
                                                                                                                                    patching file setup.py
                                                                                                                                    python-build: use tcl-tk from homebrew
                                                                                                                                    python-build: use readline from homebrew
                                                                                                                                    python-build: use zlib from xcode sdk
                                                                                                                                    Installed Python-3.7.13 to /Users/adhoc/.pyenv/versions/3.7.13
                                                                                                                                    
                                                                                                                                    (Sat, May 21  12:55:00 PM) (master|✔)
                                                                                                                                    $ ./run.sh setup
                                                                                                                                    
                                                                                                                                    snip
                                                                                                                                    
                                                                                                                                    × Encountered error while trying to install package.
                                                                                                                                    ╰─> uvloop
                                                                                                                                    
                                                                                                                                    note: This is an issue with the package mentioned above, not pip.
                                                                                                                                    hint: See above for output from the failure.
                                                                                                                                    # status: 1 #
                                                                                                                                    (Sat, May 21  12:57:08 PM) (master|✔)
                                                                                                                                    $ # I become worried that maybe Pyenv didn't activate for some reason, so I try explicitly adding it to my PATH
                                                                                                                                    (Sat, May 21  12:59:30 PM) (master|✔)
                                                                                                                                    $ PATH=$HOME/.pyenv/versions/3.7.13/bin/:$PATH ./run.sh setup
                                                                                                                                    
                                                                                                                                    snip
                                                                                                                                    
                                                                                                                                    success!
                                                                                                                                    
                                                                                                                                    (Sat, May 21  01:01:58 PM) (master|✔)
                                                                                                                                    $ 
                                                                                                                                    
                                                                                                                      2. 3

                                                                                                                        This thread has grown a lot overnight while I was sleeping! I still haven’t caught up, but here is my top of head takeaway:

                                                                                                                        I think it’s fair to complain that the Go 1 stability guarantee doesn’t go far enough, but it seems like some people elsewhere in the thread are trying to claim that adding monotonic time somehow breaks the guarantee, which makes no sense to me. At this point, someone citing Fasterthanlime basically means they don’t know anything about Go, IMO because he has been so egregious in his misrepresentation of the language. Just recently, I had to pin a specific subversion of Go 1.18 in my tests because I needed a particular behavior for the Go tool. I don’t consider that to have been the Go team breaking the stability guarantee, just an example of how you can run into the limits of it when you’re talking about the tooling.

                                                                                                                        Obviously, it’s going to be impossible to never break anything. (Which is what I take from your points 1 and 2.) If nothing else, security problems can force breaking changes. Certainly, you need to get to a point where you’re happy enough to commit to a design before you stabilize it: Go was not stable at all before version 1. JavaScript wasn’t stable before ECMA etc.

                                                                                                                        But the question to me is whether the upstream developers take breaking changes seriously or commit them wantonly. To rephrase your points:

                                                                                                                        1. Projects which accept that breaking changes are fact of software development, so they do it whenever it’s convenient.

                                                                                                                        2. Projects which accept that breaking changes are fact of software development, so they do it as little as possible.

                                                                                                                        I was being a bit too flip when I wrote “You ‘just’ don’t make breaking changes.” That’s like saying, you just don’t write bugs. It is unavoidable sometimes. But there is a qualitative difference as an end developer in relying on a project which attempts to avoid breaking changes and one which does not.

                                                                                                                        In Node, I routinely run into software that will not run on the latest versions of Node. When a new version of Node comes out, I don’t think “Oh great, more speed, more features!” I think, “Ugh, crap, what’s going to break this time?” I had a hard to diagnose break in Babel (a very popular library!) that was caused not by a major version jump in Node, but a minor version bump. It was a waste of a day of development time for me for no reason. You wrote “as long as I pay the hosting bill there is no technical reason why it would suddenly stop working.” But in this case, there wasn’t anything I did intentionally which broke my setup. I just hadn’t pinned my version of Node, Homebrew moved up by a minor version because of some other thing, and suddenly I lost a workday. It also broke production because prod was only pinned to the major version not the minor or patch.

                                                                                                                        A similar example was that I had an Amazon ECS server using a Docker image to serve some Python app, and one day it just stopped working in prod. The problem was that when the Dockerfile was written, it didn’t specify the version of Python, and then the version where async went from a semi-keyword to a full keyword came out and broke the dependencies I was using. It wouldn’t have mattered if the Docker image had been already baked, but the way this project was setup, the Docker image would be periodically rebuilt from the Dockerfile whenever things would restart or something. That at least was relatively easy to track down and fix because the change of Python versions was noticeable once I started debugging it.

                                                                                                                        You call the /v2 thing a “hack” and it is ugly and inelegant, but it does solve this problem: when you make a breaking change, let users upgrade at their own pace, instead of having them accidentally get moved onto the new thing without being aware of it. (In browser JS, "strict mode" and type="module" also have the effect of being opt-in upgrades!) That’s really the core of the so-called “Python problem” to me. Just let me decide when to work through the breaking changes. I’m not expecting eternal support or whatever. I just want something that works today to work tomorrow unless I press the big red “attempt upgrade” button.

                                                                                                                        1. 5

                                                                                                                          At this point, someone citing Fasterthanlime basically means they don’t know anything about Go, IMO because he has been so egregious in his misrepresentation of the language.

                                                                                                                          I’m the one who cited the monotonic time issue. I’m having trouble seeing good faith in this statement, so I’ll bow out of this thread after this. But just so you understand that this is adamantly not the case: I’ve written Go since before 1.0, contributing to some of the largest projects in the ecosystem, and as a manager moved several teams over to using it (from Python, no less.) I also don’t think those credentials should be necessary for a critique.

                                                                                                                          I had a hard to diagnose break in Babel (a very popular library!) that was caused not by a major version jump in Node, but a minor version bump.

                                                                                                                          To reiterate, this has happened to me with minor version bumps in Go, due to both modules and stdlib changes. This is even worse because it was Go itself, not a third-party library.

                                                                                                                          You call the /v2 thing a “hack” and it is ugly and inelegant, but it does solve this problem: when you make a breaking change, let users upgrade at their own pace

                                                                                                                          This actually touches on a major reason why Go’s ecosystem could be so unstable before modules: you could either vendor which had its own can of worms, or rely on authors using this hack. Both patterns emerged as common practice only after years of Go devs experiencing breaking changes in libraries. You could argue that those instabilities don’t apply to Go itself, which I suppose is fair, but that applies to your argument about Babel as well.

                                                                                                                      3. 13

                                                                                                                        Go makes breaking changes all of the time. The last one that bit me was when modules were enabled by default. Suddenly packages starting behaving differently depending on a variety of factors, including the version number of the package: https://stackoverflow.com/questions/57355929/what-does-incompatible-in-go-mod-mean-will-it-cause-harm/57372286#57372286

                                                                                                                        Python at least had the sense to bundle up all the breakages in a major version change, which was released 15 years ago.

                                                                                                                        1. 8

                                                                                                                          Eh no. Python does not follow semantic versioning and makes intentionally breaking changes between minor versions. In fact Python 3.x releases come with documentation titled “Porting to Python 3.x” which lists all breaking changes made intentionally.

                                                                                                                          1. 6

                                                                                                                            Python makes breaking stdlib changes in minor version releases. Not even remotely the same as a tooling change.

                                                                                                                            1. 3

                                                                                                                              I’ve never been affected by stdlib changes in python in a minor release. I have by go, where it suddenly stopped respecting system DNS settings. Or there were the backwards-incompatible changes to time, maybe best summarized here.

                                                                                                                              1. 4

                                                                                                                                I believe you, but you are lucky. With search I found things like https://github.com/wazuh/wazuh/issues/13365 which matches my experience.

                                                                                                                                1. 3

                                                                                                                                  I know this isn’t likely to change anyone’s mind, but the thing you linked is an example of something that had been deprecated and raising deprecation warnings for years. Here’s an example warning I get from a Python 3.7 install I had lying around (oldest I could find, I tend not to keep EOL’d Pythons on my personal laptop):

                                                                                                                                  DeprecationWarning: Using or importing the ABCs from ‘collections’ instead of from ‘collections.abc’ is deprecated since Python 3.3,and in 3.9 it will stop working

                                                                                                                                  And it appears they actually held off and didn’t finally remove that until 3.10. That’s at least four years after Python 3.7 (which issued that warning). It’s nearly ten years after Python 3.3.

                                                                                                                                  Python does use a rolling deprecation cycle, yes. So do some major Python projects like Django. This means you should ensure you’re bubbling up deprecation warnings in your CI, and probably read release notes when new versions come out, yes.

                                                                                                                                  But these things don’t just come surprise out of nowhere; they’re documented, they raise warnings, they’re incorporated into the release cycles. Knowing how Python’s deprecation cycles work is, for a Python user, the same kind of table stakes as knowing how semantic versioning works is for a Rust/cargo/crates user – if you updated a crate across a major version bump and it broke your code, everyone would tell you that you could have seen that coming.

                                                                                                                                2. 2

                                                                                                                                  It happens, no doubt, but IME Python’s changes are much more egregious than Go’s few changes over the years. Where Go does make changes, they are typically as solutions to bugs, not gratuitous API changes.

                                                                                                                                  Also, speaking personally, I try to be as objective as possible and fasterthanli.me is not an objective source in any sense. It’s clickbait. There’s no pretence of objectivity.

                                                                                                                                  1. 13

                                                                                                                                    I’d like to point out that this thread has gone from a claim that Go has maintained backwards compatibility to, now, a claim that Go’s backwards-incompatible changes just aren’t as “egregious” as Python’s, and to what’s basically an ad hominem attack on someone who criticized Go.

                                                                                                                                    Which is kind of a microcosm of this whole framing of “Python problem” or “Node problem”. What it’s really about is not some magical language and ecosystem that never has backwards-incompatible changes, what it’s about is subjective taste. You think Go’s breaking changes are not as “egregious” and are “solutions to bugs” while Python’s are “gratuitous”. But that’s a subjective judgment based on your tastes and opinions. Someone else can have an opposite and equally-valid subjective opinion.

                                                                                                                                    Or, more bluntly: “No breaking changes” nearly always turns out to actually mean “Breaking changes, but only ones I personally think are justified/tolerable, and I don’t think yours are”. Which is where this thread predictably went within the space of just a few replies.

                                                                                                                                    Getting back to my original claim: change is inevitable. Entities which produce software can adapt to it and make it a normal and expected part of their processes, or they can suffer the consequences of not doing so. There is no third option for “all external change stops”. Nothing that lives does so changelessly.

                                                                                                                                    1. 3

                                                                                                                                      Or, more bluntly: “No breaking changes” nearly always turns out to actually mean “Breaking changes, but only ones I personally think are justified/tolerable, and I don’t think yours are”. Which is where this thread predictably went within the space of just a few replies.

                                                                                                                                      No I don’t think so. It has nothing to do with opinion and everything to do with experience. What someone sees directly is how they perceive reality. I’m expressing my perception of reality as someone who has experience of both Python and Go, and m0th is expressing their experience.

                                                                                                                                      I’m not minimising m0th’s view, which is why I phrased it as “in my experience”.

                                                                                                                                      Getting back to my original claim: change is inevitable. Entities which produce software can adapt to it and make it a normal and expected part of their processes, or they can suffer the consequences of not doing so. There is no third option for “all external change stops”. Nothing that lives does so changelessly.

                                                                                                                                      Change is inevitable, I agree, but I do think the degree matters. Python makes breaking changes often, removing APIs, etc. Go does not and only with very good reason.

                                                                                                                                      1. 3

                                                                                                                                        Change is inevitable, I agree, but I do think the degree matters. Python makes breaking changes often, removing APIs, etc. Go does not and only with very good reason.

                                                                                                                                        Again, the original claim at the start was that there are languages which don’t have breaking changes, and Go was specifically named as an example. That has now been walked back to the kinds of statements you are making. And your statements are, effectively, just that in your opinion one language’s (Go) breaking changes are justified and not too frequent, while another language’s (Python) breaking changes are not justified and too frequent. Which is just you stating your own personal tastes and opinions. And it’s fine as long as you are willing to admit that. It is not so fine to present one’s personal tastes and opinions as if they are objective facts. It also is not so fine to engage in the kind of ad hominem you did about criticism of Go.

                                                                                                                                        1. 3

                                                                                                                                          Again, the original claim at the start was that there are languages which don’t have breaking changes, and Go was specifically named as an example.

                                                                                                                                          I didn’t make that claim, so I’m not sure why you’re arguing with me about it.

                                                                                                                                          Which is just you stating your own personal tastes and opinions. And it’s fine as long as you are willing to admit that.

                                                                                                                                          I did? I explicitly said “In My Experience”. I’m not sure how I can be any clearer.

                                                                                                                                          It is not so fine to present one’s personal tastes and opinions as if they are objective facts. It also is not so fine to engage in the kind of ad hominem you did about criticism of Go.

                                                                                                                                          What are you even talking about? I legitimately can’t even understand what you’re referring to. Where’s the “ad hominem” comment I made? I think the most controversial thing I said was

                                                                                                                                          “Python makes breaking stdlib changes in minor version releases.”

                                                                                                                                          Which is objectively true as mentioned by other commenters. I made no judgement about whether it was justified or not. It’s also not “ad hominem”, which would require Python to have a position on this.

                                                                                                                                          Anyway, I think you’re taking this way too personally for some reason. I like Python, I have used it for many years. But I’m out.

                                                                                                                                          1. 3

                                                                                                                                            Where’s the “ad hominem” comment I made?

                                                                                                                                            IIUC, ubernostrum is referring to you explaining that fasterthanli.me is not a reliable source of information. That’s not an ad hominem though. His attacks are so scattershot that if someone cites them, it’s totally fair to just handwave it away.

                                                                                                                                            1. 3

                                                                                                                                              Where’s the “ad hominem” comment I made?

                                                                                                                                              It’s this:

                                                                                                                                              Also, speaking personally, I try to be as objective as possible and fasterthanli.me is not an objective source in any sense. It’s clickbait. There’s no pretence of objectivity.

                                                                                                                                              You don’t bother to engage with the content, just brush off the author with a quick smear and then move on. Which is a problem, because the content is reasonably well-argued, from my perspective as someone who has a bit of Go experience and initially couldn’t quite describe why it rubbed me the wrong way. Putting into words the way that Go (which is not unique in this, but is a prominent example of it) likes to pretend complex things are simple, rather than just admitting they’re complex and dealing with it, was extremely useful for me, both for giving me a way to talk about what bugged me in Go and because it relates to things I’ve ranted about in the Python world previously (most notably why Python 3’s string changes were the right thing to do).

                                                                                                                                            2. 2

                                                                                                                                              Go the language doesn’t have breaking changes, in the sense that code written against a 1.0 language spec will — generally— continue to compile and run the same way against any 1.x language spec compiler. This isn’t an absolute statement of fact, but it’s a design principle that’s taken very seriously and is violated very rarely. The tooling doesn’t necessarily abide the same rules.

                                                                                                                                              1. 1

                                                                                                                                                Your comment is a bit of a roller coaster.

                                                                                                                                                • Starts with “doesn’t have breaking changes”
                                                                                                                                                • Immediately qualifies with “in the sense that…”
                                                                                                                                                • Then walks it back with “This isn’t an absolute statement of fact”
                                                                                                                                                • Then walks it back even further with “is violated very rarely”
                                                                                                                                                • Then throws it all out the window with “The tooling doesn’t necessarily abide the same rules”

                                                                                                                                                I’m not even sure what that’s trying to say.

                                                                                                                                3. 9

                                                                                                                                  Go 1 has been stable for 10 years now.

                                                                                                                                  If you just ignore all those breaking changes they made..

                                                                                                                                  1. 1

                                                                                                                                    I only remember Modules. After Modules landed I never experienced any breaking build when updating to a new Go version. A whole different story was updating Java versions and the dozens of subtle ways they can break your service, especially at run-time (caused by some Spring DI magic).

                                                                                                                                    1. 1

                                                                                                                                      What breaking language changes have been made?

                                                                                                                                      1. 3

                                                                                                                                        Modules seems like the biggest change.

                                                                                                                                        1. 3

                                                                                                                                          I mean, Go module is not part of Go 1 stability guarantee. In my opinion, this shows how limited Go’s stability guarantee is. Go 1 is stable, but Go is not, at least if you are using “go build” to build your Go code.

                                                                                                                                          1. 2

                                                                                                                                            I agree but that’s not a language change, and the impact is not as significant as random language changes. I can still use and build a package from 8 years ago with no issue. And whatever tool they happened to use to manage dependencies 8 years ago will still work fine (probably).

                                                                                                                                            1. 3

                                                                                                                                              I saw some breaking changes in one of our old projects. Written in the Go v1.4 era, IIRC. I checked it with a Go v1.14 release, and boom, it doesn’t compile due to the module changes.

                                                                                                                                              Yes, it wasn’t that hard to fix (it only took a few minutes of Internet searching), but I still count that as a breaking change.

                                                                                                                                  2. 2

                                                                                                                                    When was there ever a breaking change in a new version of ecmascript? Node is not a language.

                                                                                                                                    1. 4

                                                                                                                                      There’s a bunch of things that vary in stability.

                                                                                                                                      • V8 the interpreter breaks working JS code very seldom
                                                                                                                                      • The libraries that come with Node break JS code a little more often, but still IME not very often.
                                                                                                                                      • V8 breaks native extensions very often. The NodeJS ecosystem discourages writing them because of this.
                                                                                                                                      • Some add-on packages from npm break their consumers all the time.

                                                                                                                                      Many packages are extremely careful not to break consumers. Others are less so. The experience you have with backwards compatibility tends to track the backwards compatibility stance of the worst thing in your entire dependency tree. When you have tens of thousands of transitive dependencies, you usually end up seeing 99.99%-ile bad behaviour somewhere in there at least once.

                                                                                                                                      1. 1

                                                                                                                                        The core problem is that a handful of “core” packages break, and nodes ecosystem is way too much about plugins, so many things that you use have 3 layers to them (all maintained by different people)

                                                                                                                                        The ecosystem would be a lot more stable if we were vendoring in packages more

                                                                                                                                      2. 2

                                                                                                                                        Node is a development target. The pure abstract notion of a “language” doesn’t really matter here, because people write code for Node specifically.

                                                                                                                                        And Node does make breaking changes. It literally makes semver-major releases, with backwards-incompatible changes that can break existing npm packages (mainly due to changes in the native API, but there also have been changes to Node’s standard library).

                                                                                                                                        For example, I can’t build any projects that used Gulp v3. Node broke packages that it depends on, and I either have to run deprecated/unsupported/outdated version of Node, or rewrite my build scripts. OTOH a Rust/Cargo project I’ve developed at the same time still works with the latest version of Rust/Cargo released yesterday.

                                                                                                                                        1. 2

                                                                                                                                          Yes, that’s why I said “Node ecosystem” which breaks all the damn time and not “browser ECMAScript” which has not broken since standardization.

                                                                                                                                          1. 2

                                                                                                                                            The quote you posted says language, not ecosystem. Your comparison was a false equivalency.

                                                                                                                                          2. 1

                                                                                                                                            ecmascript itself is a language but not a language implementation, so while you can write a lot of code in ecmascript, at the end of the day the ecmascript specification can’t run it, so you’ll need to run it using an implementation, that’s why people are talking about the Node ecosystem, the language didn’t break the ecosystem did

                                                                                                                                        1. 2

                                                                                                                                          Rust is hardly a panacea. It expresses a cleaner set of domain concepts than C++, sure, but that’s not hard to do 😉

                                                                                                                                          A programming language is a system, defined by its composite parts i.e. its features, and judged by the consequences of the interactions of those features in the compositional whole. A language feature by itself is value-neutral; its value can only be measured as it exists in relation to other features.

                                                                                                                                          Yes, you can absolutely have too many features. The value of a feature is a function of the benefit it delivers, minus the cost it demands, as applied against all other features in the language in which it exists. If a new feature interacts non-deterministically with other pre-existing features, then its cost almost certainly dominates its value, and it shouldn’t be added.

                                                                                                                                          Rust in particular asserts that specific properties of memory and ownership management should be lifted into the application domain. That’s useful in some circumstances, but it’s absolutely not a given that it’s useful in all, or IMO even most, circumstances.

                                                                                                                                          1. 4

                                                                                                                                            Another thing to consider is that a cryptographic hash can reliably give you 128 bits that will uniquely and deterministically identify any piece of data. Very handy if you need to make UUIDs for something that doesn’t have them.

                                                                                                                                            1. 1

                                                                                                                                              But even a cryptographic hash has collisions, however unlikely. So there is always a chance that two distinct pieces of data will end up with the same id. But probably the same can happen, for example, with random UUID (except here you rely on the quality of your randomness source rather than the quality of your hash function and the shape of your data). Somehow using a crypto hash as a long-term id always feels iffy to me.

                                                                                                                                              1. 3

                                                                                                                                                Given the size of hashes, using the hash of the content as its id is totally safe. The hashing algorithms are designed such that collisions are so unlikely that they are impossible to happen.

                                                                                                                                                If this wasn’t the case, all systems based on content adressing would be in serious trouble. Systems like git or ipfs

                                                                                                                                                1. 1

                                                                                                                                                  If this wasn’t the case, all systems based on content adressing would be in serious trouble. Systems like git or ipfs

                                                                                                                                                  Any system that assumes a hash of some content uniquely identifies that content is in fact in serious trouble! They work most of the time, but IPFS is absolutely unsound in this regard. So is ~every blockchain.

                                                                                                                                                  1. 2

                                                                                                                                                    I’ve seen several cryptographic systems rely on exactly this fact for their security. So while it’s probabilistic, you’re relying on the Birthday Paradox to ensure it’s highly unlikely.

                                                                                                                                                    From that table, for a 128-bit hash function, for a 0.1% chance of a collision, you’d need to hash 8.3e17 items. In practical terms, a machine that can hash 1,000,000 items per second would need to run for just over 26 millennia to have 0.1% chance of a collision.

                                                                                                                                                    For systems that use 256-bit digests (like IPFS), it would take many orders of magnitude longer.

                                                                                                                                                    1. 1

                                                                                                                                                      I’ve seen several cryptographic systems rely on exactly this fact for their security. So while it’s probabilistic, you’re relying on the Birthday Paradox to ensure it’s highly unlikely.

                                                                                                                                                      If collisions are an acceptable risk as long as their frequency is low enough, then sure, no problem! Engineering is all about tradeoffs like this one. You just can’t assert that something which is very improbable is impossible.

                                                                                                                                                      1. 1

                                                                                                                                                        I can still pretend and almost certainly get away with it. If the chances of getting bitten by this pretense is ten orders of magnitude lower than the chance of a cosmic ray hitting the right transistor in a server’s RAM and cause something really bad to happen, then for all practical purposes, I’m safe to live in blissful ignorance. And what a bliss it is; Assuming a SHA-256 hash uniquely identifies a given string can immensely simplify your system architecture.

                                                                                                                                                      2. 1

                                                                                                                                                        You’ve misread the table. 1.5e37 is for 256 bit hashes. For 128 bits it’s 8.3e17, which is obviously a lot smaller.

                                                                                                                                                        For context with IPFS, the Google search index is estimated to contain between 10 and 50 billion pages.

                                                                                                                                                        1. 1

                                                                                                                                                          You’ve misread the table

                                                                                                                                                          Thanks. Although it’s what happens when you start with an 256bit example, then remember everyone’s talking about UUIDs, and hastily re-calculate everything. :/

                                                                                                                                                  2. 3

                                                                                                                                                    No, this is absolutely not a consideration. Your CPU and RAM have lower reliability than a 128-bit cryptographic hash. If you ever find a collision by chance it’ll be more likely to be false positive due to hardware failure (we’re talking 100-year timespan at a constant rate of billions of uuids per second).

                                                                                                                                                    And before you mention potential cryptographic weaknesses, consider that useful attacks need a preimage attack, and the “collision” attacks known currently are useless for making such uuids collide.

                                                                                                                                                    1. 2

                                                                                                                                                      Whenever you map from a set of cardinality N (content) to a set of cardinality less than N (hashes of that content) by definition you will have collisions. A hash of something is just definitionally not equivalent to that thing, and doesn’t uniquely identify it.

                                                                                                                                                      As an example, if I operate a multi-tenant hosting service, and a customer uploads a file F, I simply can’t assert that any hash of the content of F can be used as a globally unique reference to F. Another customer can upload a different file F’ which hashes identically.

                                                                                                                                                      “Highly unlikely” isn’t equivalent to “impossible”.

                                                                                                                                                      1. 4

                                                                                                                                                        It’s absolutely impossible for all practical purposes. It’s a useless pedantry to consider otherwise.

                                                                                                                                                        Remember we’re talking about v4 UUID here which already assumes a “risk” of collisions. Cryptographic hashes are indistinguishable from random data, and are probably more robust than your prng.

                                                                                                                                                        The risk of an accidental collision is so small, you can question whether there’s enough energy available to our entire civilisation to compute enough data to ever collide in the 128-bit space in it’s lifetime.

                                                                                                                                                        1. 1

                                                                                                                                                          It’s absolutely impossible for all practical purposes. It’s a useless pedantry to consider otherwise.

                                                                                                                                                          I mean it literally isn’t, right? “Absolutely impossible” is just factually not equivalent to “highly improbable” — or am I in the wrong timeline again? 😉 Going back to the hosting example, if you want to use UUIDs to identify customer documents that’s fine, but you can’t depend on the low risk of collisions to establish tenant isolation, you still have to namespace them.

                                                                                                                                                          1. 1

                                                                                                                                                            By your definition of impossible, there is literally no possible solution since it would require infinite memory. At that point you should question why using a computer at all.

                                                                                                                                                            The fact is that people don’t quite understand uuid and use it everywhere in meme fashion. Most uuid usages as database keys i’ve seen, it is even stored as a string. Which creates much more serious problems than those in discussion here.

                                                                                                                                                            1. 1

                                                                                                                                                              You don’t need infinite memory to uniquely identify documents among customers. You just need a coördination point to assign namespaces.

                                                                                                                                                              I agree that UUID using in the wild is…. wild. I don’t think most developers even really understand that the dashed-hex form of a UUID is actually just one of many possible encodings of what is ultimately just 16 bytes in memory.

                                                                                                                                                        2. 3

                                                                                                                                                          The only system behaviour that the universe guarantees is that all systems will eventually decay.

                                                                                                                                                          For any other behaviour you have to accept some probability that it won’t happe (hardware failure, bugs, operator error, attacks, business failure, death, and, yes, hash collisions).

                                                                                                                                                          Hash collisions with a good algorithm will often be a risk much lower than other factors that you can’t control. When they are, what sense does it make to worry about them?

                                                                                                                                                          1. 1

                                                                                                                                                            There is a categorical difference between hash collisions and the kind of unavoidable risk you’re describing, e.g. solar flares flipping a bit in my main memory.

                                                                                                                                                        3. 1

                                                                                                                                                          After doing some more thinking/reading, I would agree with you for something like SHA-256. But using a 128-bit hash still seems like a bad idea. I found this paragraph from a reply on HN summarizes it quite well:

                                                                                                                                                          In cases where you can demonstrate that you only care about preimage resistance and not collision resistance, then a 128-bit hash would be sufficient. However often collision attacks crop in in unexpected places or when your protocol is used in ways you didn’t design for. Better to just double the hash size and not worry about it.

                                                                                                                                                          I think Git’s messy migration from SHA-1 is a good cautionary tale. Or do you believe it’s completely unnecessary?

                                                                                                                                                          1. 1

                                                                                                                                                            Git’s migration is due to a known weakness in SHA-1, not due to the hash size being too small. I believe git would be perfectly fine if it used a different 128-bit cryptographic hash.

                                                                                                                                                            The first sentence you’ve quoted is important. There are uses of git where this SHA-1 weakness could matter. For UUID generation it’s harder to imagine scenarios where it could be relevant. But remember you don’t need to use SHA-1 — you can use 128 bits of any not-yet-broken cryptographic hash algorithm, and you can even pepper it if you’re super paranoid about that algorithm getting cracked too.

                                                                                                                                                            1. 1

                                                                                                                                                              The first sentence you’ve quoted is important. There are uses of git where this SHA-1 weakness could matter.

                                                                                                                                                              Yes, but isn’t git’s situation exactly what the rest of that paragraph warns about: SHA-1 is succeptible to a collision attack, not a preimage attack. And now everyone is trying to figure out whether this could be exploited in some way even though on the surface git is just a simple conten-addressable system where collision attacks shouldn’t matter. And as far as I can tell there is still no consensus either way.

                                                                                                                                                              And as the rest of that reply explains, 128-bit is not enough to guarantee collision resistance.

                                                                                                                                                              1. 1

                                                                                                                                                                If the 128-bit space is not enough for you, then it means you can’t use UUID v4 at all.

                                                                                                                                                                Their whole principle of these UUIDs is based on the fact that random collisions in the 128-bit space are so massively improbable that they can be safely assumed to never ever happen. I need to reiterate that outputs of a not-broken cryptographic hash are entirely indistinguishable from random.

                                                                                                                                                                Resistance of a hash algorithm to cryptographic (analytic) attacks is only slightly related to the hash size. There are other much more important factors like the number of rounds that the hash uses, and that factor is independent of the output size, so it’s inaccurate to say that 128-bit hashes are inherently weaker than hashes with a larger output.

                                                                                                                                                                Please note that you don’t need to use SHA-1. SHA-1’s weakness is unique its specific algorithm, not to 128-bit hashes in general. You can pick any other algorithm. You can use SHA-2, SHA-3, bcrypt/scrypt, or whatever else you like, maybe even a XOR of all of them together.

                                                                                                                                                    1. 3

                                                                                                                                                      Crash-tolerant software is indeed a virtue, but crash-only software often tends to encourage programming practices that make applications fiendishly difficult to model. If every error is a terminal error then you kind of opt out of deterministic control flow.

                                                                                                                                                      1. 1

                                                                                                                                                        crash-only software often tends to encourage programming practices that make applications fiendishly difficult to model.

                                                                                                                                                        Can you expand on that a bit?

                                                                                                                                                        If every error is a terminal error then you kind of opt out of deterministic control flow.

                                                                                                                                                        Well, there’s a bit more depth to it than that. For example, within the lifecycle of an individual request, experiencing a transient error (such as a timeout) might be fatal to that request, but not to the web server as a whole. Or for example, if your message queue consumer loses it’s connection; then you’d usually only restart the consumer, rather than the process as a whole.

                                                                                                                                                        1. 1

                                                                                                                                                          Is that what crash only means? All errors are crashes? My understanding was more that there was no happy/sad path when you terminated, a normal exit is indistinguishable from an abrupt one due to a crash, so any recovery happens in the startup path (and is constantly exercised).

                                                                                                                                                          1. 6

                                                                                                                                                            Going by Wikipedia definition:

                                                                                                                                                            Crash-only software refers to computer programs that handle failures by simply restarting, without attempting any sophisticated recovery.

                                                                                                                                                            The argument usually is: sophisticated (precise) error recovery is hard, and if you’re attempting it, you already have a potentially-broken program in an inconsistent state. Throwing it all away and starting from a blank state again is easier, well-tested, and therefore more robust.

                                                                                                                                                            Take for example an out-of-memory error: if an allocation fails, you can either carefully backtrack the current operation and report the problem to the caller, or just abort() the whole program.

                                                                                                                                                            I generally agree with the approach, but it’s not a silver bullet. Crash-only systems are prone to DoS-ing themselves. A persistent bad input can put the whole system in a crash loop, instead of being carefully skipped over. Even when everything technically works (and makes incremental progress as it should), the restart may be expensive/disruptive (e.g. loses caches or performs extra work on startup) and the system can still fail due to insufficient throughput.

                                                                                                                                                            1. 1

                                                                                                                                                              In a crash-only service, if an incoming request encounters a database invariant violation, does the entire process die? Or if the database connection fails, are all in-flight requests abandoned?

                                                                                                                                                              Designing software so that it can start up and recover from a wide variety of prior states is a good idea in general, but it’s structurally impossible to write a program that can crash at any point during its execution and reliably leave underlying resources in a recoverable state. Any nontrivial DB transaction commit, for example, is a multi-stage operation. Same with network operations.

                                                                                                                                                              More generally, it’s definitely a good idea to design the layer above the individual process to be resilient, but you can’t just assert that error handling isn’t a concern of the process. The process, the program, is what human beings need to mentally model and understand. That requires deterministic control flow.

                                                                                                                                                              1. 2

                                                                                                                                                                but you can’t just assert that error handling isn’t a concern of the process.

                                                                                                                                                                I agree that is a bad idea, and I would almost say objectively so. Which is why I don’t think it is actually what the idea of “crash-only” is trying to convey.

                                                                                                                                                                Again, my understanding was that crash-only software wasn’t “we crash on all errors/we don’t care about errors”, but rather “we don’t have shutdown code/the shutdown code isn’t where all the invariants are enforced”. It’s more about not having atexit handlers than not having catch blocks if you will. All program terminations are crashes, not all errors are crashes. If you have no shutdown code do you have to put that code somewhere else (periodic autosaves say, etc.), which means when you do crash you’re much likely to be closer to a recent good state.

                                                                                                                                                                1. 1

                                                                                                                                                                  I may misunderstand what “crash-only” means. I take “crash” to mean “terminate the operating system process unceremoniously and without unwinding call stacks”, and I understand “only” not to mean all conceivable errors but definitely more than is strictly necessary.

                                                                                                                                                                2. 1

                                                                                                                                                                  if an incoming request encounters a database invariant violation, does the entire process die? The original paper “Crash Only Software” talks about it in terms of individual components that can perform micro-reboots, and references Erlang heavily.

                                                                                                                                                                  So for an example, you’d want to use tools such as transactions so you can pretend that a multi-step operation is a single one. Alternatively, make everything idempotent, and retry a lot.

                                                                                                                                                                  structurally impossible to write a program that can crash at any point during its execution and reliably leave underlying resources in a recoverable state.

                                                                                                                                                                  I’m reasonably sure that this is what database software should be designed to do. Unless I’m somehow misunderstanding you.

                                                                                                                                                                  1. 1

                                                                                                                                                                    There is no OS-level operation against e.g. disks, networks, etc. which can be interrupted at any arbitrary point, and can be reliably recovered-from. You can do your best to minimize the damage of that interruption – and maybe this is what crash-only is gunning for – but you can’t solve the problem. Every transition between layers of abstractions, between system models, represents an exchange of information over time. No matter how you design your protocol, no matter how tiny the “commit” or “sync” signal is, if your operation spans more than one layer of abstraction, and it’s possible for that operation to be interrupted, you can’t avoid the possibility of ending up in an invalid state. That’s fine! My point is only that systems at each layer of abstraction should not only do their best to recover from wonky initial state, but should also do their best to avoid making that wonky state in the first place. If you encounter an error, you should deal with it responsibly, to the best of your ability. That’s it.

                                                                                                                                                            1. 8

                                                                                                                                                              C does not provide maps, and when I really need one I can implement it in less than 200 lines (won’t be generic, though). Were I to design a language like Hare or Zig, providing an actual (hash) map implementation would be way down my list of priorities. Even if it belongs in the standard library, my first order of business would be to make sure we can implement that kind of things.

                                                                                                                                                              In fact, Go made a mistake when it provided maps directly without providing general purpose generics. That alone hinted at a severe lack of orthogonality. If maps have to be part of the core language, that means users can’t write one themselves. Which means they probably can’t write many other useful data structures. As Go authors originally did, you could fail to see that if the most common ones (arrays, hash tables…) are already part of the core language.

                                                                                                                                                              The most important question is not whether your language has maps. It’s whether we can add maps that really matters. Because if we can’t, there’s almost certainly a much more serious root cause, such as the lack of generics.

                                                                                                                                                              1. 11

                                                                                                                                                                I think this is too a one-sided debate. Generics have benefits and drawbacks (to argument from authority, see https://nitter.net/graydon_pub/status/1036279571341967360).

                                                                                                                                                                Go’s original approach of providing just three fundamental generic data structures (vec, map, and chan) definitely was a worthwhile experiment in language design, and I have the feeling that it almost worked.

                                                                                                                                                                1. 10

                                                                                                                                                                  At this point I’d argue that the benefits of even the simplest version of generics (not bounded, template-style or ML-functor-style, whatever) are so huge compared to the downsides, that it’s just poor design to create a new statically typed language without them. It’s almost like creating a language without function calls.

                                                                                                                                                                  Go finally fixed that — which doesn’t fix all the other design issues like zero values or lack of sum types — but their initial set of baked-in generic structures was necessary to make the language not unbearable to use. If they hadn’t baked these in, who would use Go at all?

                                                                                                                                                                  1. 3

                                                                                                                                                                    other design issues like zero values

                                                                                                                                                                    Could you share more here? I agree about Go generics, but its zero values are one thing I miss when using other imperative languages. They’re less helpful in functional languages, but I even miss zero values when using OCaml in an imperative style.

                                                                                                                                                                    1. 5

                                                                                                                                                                      Zero values are:

                                                                                                                                                                      • not always something that makes sense (what’s the 0 value for a file descriptor? an invalid file descriptor, is what. For a mutex? same thing.) The criticism in a recent fasterthanlime article points this out well: Go makes up some weird rules about nil channels because it has to, instead of just… preventing channels from being nil ever.
                                                                                                                                                                      • error prone: you add a field to a struct type, and suddenly you need to remember to update all the places you create this struct
                                                                                                                                                                      • encouraging bad programming by not forcing definition to go with declaration. This is particularly true in OCaml, say: there are no 0 values, so you always have to initialize your variables. Good imperative languages might allow var x = undefined; (or something like that) but should still warn you if a path tries to read before writing to the field.
                                                                                                                                                                      1. 3

                                                                                                                                                                        nitpick: Go’s sync.Mutex has a perfectly valid and actually useful zero value: an unlocked mutex.

                                                                                                                                                                        That said, I broadly agree with you; some types simply do not have a good default, and the best solution is not to fudge it and require explicit initialization.

                                                                                                                                                                        @mndrix, note that there is a middle ground that gives you the best of both worlds: Haskell and Rust both have a Default type class/trait, that can be defined for types for which it does makes sense. Then you can just write

                                                                                                                                                                        (in haskell):

                                                                                                                                                                        let foo = def
                                                                                                                                                                          in ...
                                                                                                                                                                        

                                                                                                                                                                        (or rust):

                                                                                                                                                                        let foo = Default::default();
                                                                                                                                                                        

                                                                                                                                                                        Note you can even write this in Go, it just applies to more types than it should:

                                                                                                                                                                        func Zero[T any]() T {
                                                                                                                                                                           var ret T
                                                                                                                                                                           return ret
                                                                                                                                                                        }
                                                                                                                                                                        
                                                                                                                                                                        // Use:
                                                                                                                                                                        foo := Zero[T]()
                                                                                                                                                                        

                                                                                                                                                                        You could well define some mechanism for restricting this to certain types, rather than just any. Unfortunately, it’s hard for me to see how you could retrofit this.

                                                                                                                                                                        1. 1

                                                                                                                                                                          Thank you for the correction!

                                                                                                                                                                        2. 3

                                                                                                                                                                          not always something that makes sense (what’s the 0 value for a file descriptor? an invalid file descriptor, is what. For a mutex? same thing.)

                                                                                                                                                                          Partially agreed on a mutex (though on at least some platforms, a 0 value for a pthread mutex is an uninitialised, unlocked, mutex and will be lazily initialised on the first lock operation). If you bias your fd numbers by one then a 0 value corresponds to -1, which is always invalid and is a useful placeholder, but your example highlights something very important: the not-present value may be defined externally.

                                                                                                                                                                          I saw a vulnerability last year that was a direct result of zero initialisation, of a UID field. A zero value on *NIX means root. If you hit the code path that accidentally skipped initialising the field properly, then the untrusted thing would run as root. Similarly, on most *NIX systems (all that I know of, though POSIX doesn’t actually mandate this), fd 0 is stdin, which is (as you point out) a terrible default.

                                                                                                                                                                          Any time you’re dealing with an externally defined interface, there’s a chance that either there is no placeholder value or there is a placeholder value and it isn’t 0.

                                                                                                                                                                          1. 2

                                                                                                                                                                            not always something that makes sense

                                                                                                                                                                            Agreed. However, my experience is that zero values are sensible for roughly 90% of types, and Go’s designers made the right Huffman coding decision here.

                                                                                                                                                                            The criticism in a recent fasterthanlime article points this out well: Go makes up some weird rules about nil channels because it has to, instead of just… preventing channels from being nil ever.

                                                                                                                                                                            For anyone who comes along later, I think this is the relevant fasterthanlime article. Anyway, the behavior of nil and closed channels is well-grounded in the semantics of select with message passing, and quite powerful in practice. For me, this argument ends up favoring zero values, for channels at least.

                                                                                                                                                                            you add a field to a struct type, and suddenly you need to remember to update all the places you create this struct

                                                                                                                                                                            My experience has been that they’d all be foo: 0 anyway. Although in practice I rarely use struct literals outside of a constructor function in Go (same with records in OCaml) because I inevitably want to enforce invariants and centralize how my values are created. In both languages, I only have to change one place after adding a field.

                                                                                                                                                                            by not forcing definition to go with declaration

                                                                                                                                                                            The definition there but it’s implicit. I guess I don’t see much gained by having repetitive = 0 on each declaration, like I often encounter in C.

                                                                                                                                                                            1. 1

                                                                                                                                                                              what’s the 0 value for a file descriptor?

                                                                                                                                                                              standard input

                                                                                                                                                                              1. 2

                                                                                                                                                                                I hope you’re not serious. I mean, sure, but it makes absolutely no sense whatsoever that leaving a variable uninitialized just means “use stdin” (if it’s still open).

                                                                                                                                                                                1. 1

                                                                                                                                                                                  File descriptor 0 is standard input on unix systems. (Unless you close it and it gets reused, of course, leading to fun bugs when code expects it to be standard input.)

                                                                                                                                                                                  1. 1

                                                                                                                                                                                    As ludicrous as it would be, it would be a natural default value to have for careless language implementers, and before you know it users come to expect it. Even in C, static variables are all zero initialised and using one on read(2) would indeed read from standard input.

                                                                                                                                                                                    I’m sure we can point out various language quirks or weird idioms that started out that way.

                                                                                                                                                                                2. 1

                                                                                                                                                                                  A zero-value file descriptor is invalid, sure, but a zero-value mutex is just an unlocked mutex. Why would that be invalid?

                                                                                                                                                                                3. 1

                                                                                                                                                                                  There is no “zero” postal code, telephone number or user input.

                                                                                                                                                                              2. 7

                                                                                                                                                                                Thing is, Go not only is statically typed, it is garbage collected.

                                                                                                                                                                                As such, it is quite natural for it to use heap allocation for (almost) everything, and compensate for this with a generational GC. Now things get a little more complicated if they want to support natively sized integers (OCaml uses 31/62-bit integers to have one bit to distinguish them from pointers, so the GC isn’t confused), but the crux of the issue is that when you do it this way, generics become dead simple: everything is a pointer, and that’s it. The size of objects is often just irrelevant. It may sometime be a problem when you want to copy mutable values (so one might want to have an implicit size field), but for mere access, since everything are pointers size does not affect the layout of your containing objects.

                                                                                                                                                                                This is quite different from C++ and Rust, whose manual memory management and performance goals kinda force them to favour the stack and avoid pointers. So any kind of generic mechanism there will have to take into account the fact that every type might have a different size, forcing them to go to a specialization based template, which may be more complex to implement (especially if they want to be clever and specialise by size instead of by type).

                                                                                                                                                                                What’s clear to me is that Go’s designers didn’t read Pierce’s Type and Programming Languages, and the resulting ignorance caused them to fool themselves into thinking generics were complicated. No they aren’t. It would have taken a couple additional weeks to implement them at most, and that would have saved time elsewhere (for instance they wouldn’t have needed to make maps a built in type, and pushed that out to the standard library).

                                                                                                                                                                                I have personally implemented a small scripting language to pilot a test environment. Plus it had to handle all C numeric types, because the things it tests were low level. I went for static typing for better error reporting, local type inference to make things easier on the user, and added an x.f() syntax and a simple type based static dispatch over the first argument to get an OO feel. I quickly realised that some of the functions I needed required generics, so I added generics. It wasn’t perfect, but it took me like 1 week. I know that generics are simple.

                                                                                                                                                                                The reason the debate there is so one sided is because Go should have had generics for the start. The benefits are enormous, the drawbacks very few. It’s not more complex for users who don’t use generics, generic data structures can still be used as if they were built in, it hardly complicates the implementation, and it improves orthogonality across the board.

                                                                                                                                                                                “LoL no generics” was the correct way to react, really.

                                                                                                                                                                                1. 8

                                                                                                                                                                                  It still seems to me that you are overconfident in this position. That’s fanboyism from my side, but Graydon certainly read TAPL, and if Graydon says “there’s a tradeoff between expressiveness and cognitive load” in the context of Go’s generics, it does seem likely that there’s some kind of tradeoff there. Which still might mean “LoL no generics” is the truth, but not via a one-sided debate.

                                                                                                                                                                                  Having covered meta issues, let me respond to specific points, which are all reasonable, but also are debatable :)

                                                                                                                                                                                  First, I don’t think the GC/no-GC line of argument holds for Go, at least in a simple form. Go deliberately distinguishes value types and pointer types (up to having a dedicated syntax for pointers), so “generics are easy ‘cause everything is a pointer” argument doesn’t work. You might have said that in Go everything should have been a pointer, but that’s a more complex argument (especially in the light of Java trying to move away from that).

                                                                                                                                                                                  Second, “It’s not more complex for users who don’t use generics,” – this I think is just in general an invalid line of argumentation. It holds in specific context: when you own the transitive closure of the code you are working with (handmade-style projects, or working on specific things at the base of the stack, like crypto libraries, alone or in a very small and tightly-knit teams). For “industrial” projects (and that’s the niche for Go) user’s simply don’t have the luxury of ignoring parts of the language. If you work on an average codebase with >10 programmers and >100k lines of code, the codebase will use everything which is accepted by the compiler without warnings.

                                                                                                                                                                                  Third, I personally am not aware of languages which solve generics problem in a low cognitive-load way. Survey:

                                                                                                                                                                                  C++ is obviously pretty bad in terms of complexity – instantiation-time errors, up-until-recently a separate “weird machine” for compile-time computations, etc.

                                                                                                                                                                                  Rust – it does solve inscrutable instantiation-time errors, but at the cost of far-more complex system, which is incomplete (waiting for GATod), doesn’t compose with other language features (async traits, const traits), and still includes “weird machine” for compile-time evaluation.

                                                                                                                                                                                  Zig. Zig is exciting – it fully and satisfactory solves the “weird machine” problem by using the same language for compile-time (including parametric polymorphism) and run-time computation. It’s also curious in that, as far as I understand, it also essentially “ignores TAPL” – there are no generics in the type system. It, however, hits “instantiation time errors” on the full speed. It seems to be that building tooling for such a language would be pretty hard (it would be anti-go in this sense). But yeah, Zig so far for me is one of the languages which might have solved generics.

                                                                                                                                                                                  Haskell – it’s a bit hard to discuss what even is the generics impl in Haskell, as it’s unclear which subset of pragmas are we talking about, but I think any subset generally leads to galactic-brain types.

                                                                                                                                                                                  OCaml – with questionable equality semantics, and modular implicit which are coming soon, I think it’s clear that generics are not solved yet. It also has a separate language for functors, which seems pretty cognitively complex.

                                                                                                                                                                                  Scala – complexity-wise, I think it’s a super-set of both Haskell and Java? I don’t have a working memory of Scala to suggest specific criticisms, but I tend to believe “Scala is complex” meme. Although maybe it’s much better in Scala 3?

                                                                                                                                                                                  Java – Java’s type system is broken in a trivial way (covariant arrays) and a (couple of?) interesting way (they extended type-inference when they added lambdas, and that inference allowed materialization of some un-denotable types which break the type system). I am also not sure that its “LoL covariant arrays”, as some more modern typed languages also make this decision, citing reduction of cognitive load. And variance indeed seems to be quite complex topic – Effective Java (I think?) spends quite some pages explaining “producer extends, consumer super”.

                                                                                                                                                                                  C# – I know very little about C#, but it probably can serve as a counter example for “generics in GC languages are simple”. Like Go, C# has value types, and, IIRC, it implements generics by just-in-time monomorphisation, which seems to be quite a machinery.


                                                                                                                                                                                  Now, what I think makes these systems complicated is not just parametric polymorphism, but bounded quantitication. The desire to express not only <T>, but <T: Ord>. Indeed, to quote TAPL,

                                                                                                                                                                                  This chapter introduces bounded quantification, which arises when polymorphism and subtyping are combined, substantially increasing both the expressive power of the system and its metatheoretic complexity.

                                                                                                                                                                                  I do think that there’s an under-explored design space of non-bounded generics, and very much agree with @c-cube . I am not quite convince that it would work and that Go should have been SML without functors and with channels, but that also doesn’t seem obviously worse than just three generic types! The main doubt for me is that having both interfaces and unbounded generics feels weird. But yeah, once I have spare time for implementing a reasonable complete language, unbounded generics is what I’d go for!

                                                                                                                                                                                  EDIT: forgot Swift, which absolutely tops my personal chart of reasons why adding generics to a language is not a simple matter: https://forums.swift.org/t/swift-type-checking-is-undecidable/39024.

                                                                                                                                                                                  1. 1

                                                                                                                                                                                    Graydon certainly read TAPL

                                                                                                                                                                                    Ah. that’s one hypothesis down then. Thanks for the correction.

                                                                                                                                                                                    It holds in specific context: when you own the transitive closure of the code you are working with (handmade-style projects, or working on specific things at the base of the stack, like crypto libraries, alone or in a very small and tightly-knit teams). For “industrial” projects (and that’s the niche for Go) user’s simply don’t have the luxury of ignoring parts of the language. If you work on an average codebase with >10 programmers and >100k lines of code, the codebase will use everything which is accepted by the compiler without warnings.

                                                                                                                                                                                    OK, while I do have some experience with big projects, my best work by far was in smaller ones (including my crypto library, which by the way did not even need any generics to implement). What experience I do have with bigger projects however have shown me that most of the time (that is, as long as I don’t have to debug something), the only pieces of language I have to care about are those used for the API of whatever I’m using. And those tend to be much more reasonable than whatever was needed to implement them. Take the C++ STL for an extreme example: when was the last time you actually specified the allocator of a container? Personally I’ve never done it in over 10 years being paid to work with C++.

                                                                                                                                                                                    I personally am not aware of languages which solve generics problem in a low cognitive-load way

                                                                                                                                                                                    I have written one. Not public, but that language I’ve written for test environments? It had generics (unbounded, no subtyping), and I didn’t even tell my users. I personally needed them to write some of the functions of the standard library, but once that was done, I thought users would not really need them. (Yeah, it was easier to add full blown generics than having a couple ad-hoc generic primitives.)

                                                                                                                                                                                    Now, what I think makes these systems complicated is not just parametric polymorphism, but bounded quantitication. The desire to express not only <T>, but <T: Ord>.

                                                                                                                                                                                    Yeah, about subtyping…

                                                                                                                                                                                    In the code I write, which I reckon has been heavily influenced by an early exposure to OCaml (without the object part), I almost never use subtyping. Like, maybe 3 or 4 times in my entire career, two of which in a language that didn’t have closures (C++98 and C). If I design a language for myself, subtyping will be way down my list of priorities. Generics and closures will come first, and with closures I’ll have my poor man’s classes in the rare cases I actually need them. Even in C I was able to add virtual tables by hand that one time I had to have subtype polymorphism (It’s in my crypto library, it’s the only way I found to support several EdDSA hashes without resorting to a compilation flag).

                                                                                                                                                                                    I’ve heard of subtyping being successfully used elsewhere. Niklaus Witrth took tremendous advantage of it with its Oberon programming language and operating system. But I believe he didn’t have parametric polymorphism or closures either, and in this case I reckon subtyping & class based polymorphism are an adequate substitute.

                                                                                                                                                                                    Type classes (or traits) however are really enticing. I need more practice to have a definite opinion on them, though.

                                                                                                                                                                                    1. 2

                                                                                                                                                                                      Ah. that’s one hypothesis down then. Thanks for the correction.

                                                                                                                                                                                      To clarify, graydon, as far as I know, didn’t participate in Go’s design at all (he designed Rust), so this has no bearing on your assumption about designers of Go.

                                                                                                                                                                                      1. 2

                                                                                                                                                                                        Crap! Well, at least his argument has value.

                                                                                                                                                                                  2. 2

                                                                                                                                                                                    If generics are simple, can you read this issue thread and tell everyone else how to fix comparable to be consistent? TIA.

                                                                                                                                                                                    1. 2

                                                                                                                                                                                      Generics are simple under a critical condition: Design them from the ground up

                                                                                                                                                                                      Done after the fact in a system not designed for them, of course they’re going to be difficult. Then again, why waste a couple weeks of up front design when you can afford years of waiting and months of pain?

                                                                                                                                                                                      1. 3

                                                                                                                                                                                        I don’t agree that generics are simple, but I agree that designing them in from the start is vastly easier than trying to retrofit them to an existing language. There are a lot of choices in how generics interact with your type system (especially in the case of a structural type system, which Go has, and even more so in an algebraic type system). If you build generics in as part of your type system from the start then you can explore the space of things allowed by generics and the other features that you want. If you don’t, then you may find that the point that you picked in the space of other things that you want is not in the intersection of that and generics.

                                                                                                                                                                                        1. 0

                                                                                                                                                                                          I don’t agree that generics are simple

                                                                                                                                                                                          I kinda had to change my mind on that one. Generics can be very simple in some contexts (like my own little language), but I see now that Go wasn’t one of them.

                                                                                                                                                                                          If you build generics in as part of your type system from the start then you can explore the space of things allowed by generics and the other features that you want. If you don’t, then you may find that the point that you picked in the space of other things that you want is not in the intersection of that and generics.

                                                                                                                                                                                          One thing I take for granted since I went out of college in 2007, is that we want generics. If your language is even slightly general purpose, it will need generics. Even my little specialised language, I didn’t plan to add generics, but some functions in my standard library required it. So this idea of even attempting to design a language without generics feels like an obvious waste of time to me. Likewise for closures and sum types by the way.

                                                                                                                                                                                          There is one thing that can make me change my mind: systematic experimentation for my niche of choice. I design my language, and I write a real program with it, trying to use as few features as possible. For instance, my experience in writing a cryptographic library in C convinced me they don’t need generics at all. Surprisingly though, I did see a case for subtype polymorphism, in some occasions, but that happens rarely enough that an escape hatch like writing your vtable by hand is good enough. I believe Jonathan Blow is doing something similar for his gaming language, and Niklaus Wirth definitely did the same for Pascal (when he devised Modula and Oberon).

                                                                                                                                                                                          1. 2

                                                                                                                                                                                            One thing I take for granted since I went out of college in 2007, is that we want generics. If your language is even slightly general purpose, it will need generics

                                                                                                                                                                                            I completely agree here. I wrote a book about Go that was finished just after Go reached 1.0 and, even then, I thought it was completely obvious that once you’ve realised that you need generic maps you should realise that you will need other generic types. Having maps (and arrays and slices) as special-case generics felt like a very bad decision.

                                                                                                                                                                                            Mind you, I also thought that having a language that encouraged concurrency and made data races undefined behaviour, but didn’t provide anything in the type system to allow you to define immutable types or to limit aliasing was also a terrible idea. It turns out that most of the folks I’ve spoken to who use Go use it as a statically compiled Python replacement and don’t use the concurrency at all. There was a fantastic paper at ASPLOS a couple of years back that looked at concurrency bugs in Go and found that they are very common.

                                                                                                                                                                                            1. 1

                                                                                                                                                                                              I think Oberon and Go have a lot in common. Both were designed by an experienced language constructor late in his career, emphasising “simplicity” over everything else, leaving out even basic things such as enums, even though he had created more expressive languages earlier on.

                                                                                                                                                                                              1. 2

                                                                                                                                                                                                I tend to be more sympathetic to Wirth’s decisions, because he was working in a closed ecosystem he completely controlled and understood. I mean, they were like less than 5 people working on the entire OS + compiler + main applications, and in the case of the Lilith computer, and FPGA Oberon, even the hardware!

                                                                                                                                                                                                He could have criteria such as “does this optimisation makes the entire compiler bootstrap faster? Here it’s for speed, but I can see the idea that every single piece of complexity has to pay for itself. It was not enough for a feature to be beneficial, the benefits had to outweigh the costs. And he could readily see when they did, because he could fit the entire system in his head.

                                                                                                                                                                                                Go is in a different situation, where from the start it was intended for a rather large audience. Thus, the slightest benefit to the language, even if it comes at a significant up-front cost, is liable to pay huge dividends as it becomes popular. So while I can believe even generics may not be worth the trouble for a 10K LOC system (the size of the Oberon system), that’s a different story entirely when people collectively write hundreds of millions of lines of code.

                                                                                                                                                                                                1. 2

                                                                                                                                                                                                  The best characterisation of Go and Oberon I can come up with is »stubborn«.

                                                                                                                                                                                      2. 2

                                                                                                                                                                                        While Go is garbage collected, it is still very much “value oriented” in that it gives control to the programmer for the layout of your memory, much like C++ and Rust. Just making everything a pointer and plugging your ears to the issues that brings isn’t solving the problem.

                                                                                                                                                                                        I’m glad that you added generics to your small scripting language for a test environment in 1 week. I don’t think that speaks much to the difficulty in adding it to a different language with a different type system and different goals. When they started the language, things like “very fast compile times” where very high priority, with the template system of C++ generics heavily inspiring that goal. It would be inane to start a project to avoid a problem in a language and then cause the exact same problem in it.

                                                                                                                                                                                        So, they didn’t want to implicitly box everything based on their experience with Java, and they didn’t want to template everything based on their experience with C++. Finding and implementing a middle ground is, in fact, difficult. Can you name any languages that avoid the problems described in https://research.swtch.com/generic?

                                                                                                                                                                                        The problem with “lol no generics” is that the word “generics” sweeps the whole elephant under the rug. There are a fractal of decisions to make when designing them with consequences in the type system, runtime, compiler implementation, programmer usability, and more. I can’t think of any languages that have the exact same generics system. Someone who prefers Rust generics can look at any other language and say “lol no traits”, and someone who prefers avoiding generics entirely (which, I promise, is a reasonable position to hold) may look around and say “lol 2 hour compile times”. None of those statements advance any conversation or are a useful way to react.

                                                                                                                                                                                        1. 1

                                                                                                                                                                                          I can’t think of any languages that have the exact same generics system. Someone who prefers Rust generics can look at any other language and say “lol no traits”,

                                                                                                                                                                                          Aren’t Rust traits analogous to Swift protocols?

                                                                                                                                                                                          1. -1

                                                                                                                                                                                            While Go is garbage collected, it is still very much “value oriented” in that it gives control to the programmer for the layout of your memory, much like C++ and Rust.

                                                                                                                                                                                            That kind of changes everything… Oops. (Really, I mean it.)

                                                                                                                                                                                            When they started the language, things like “very fast compile times” where very high priority, with the template system of C++ generics heavily inspiring that goal.

                                                                                                                                                                                            Several things conspire to make C++ compile times slow. The undecidable grammar, the complex templates, and the header files. Sure we have pre-compiled headers, but in general, those are just copy pasta that are being parsed and analysed over and over and over again. I’ve seen bloated header-only libraries add a full second of compilation time per .cpp file. And it was a logging library, so it was included everywhere. Properly isolating it solved the problem, and the overhead was reduce to one second for the whole project.

                                                                                                                                                                                            A simpler grammar is parsed basically instantly. Analysis may be slower depending on how advanced static checks are, but at least in a reasonable language it only has to happen once. Finally there’s code generation for the various instantiations, and that may take some time if you have many types to instantiate. But at least you don’t have to repeat the analysis, and the most efficient optimisations don’t take that much compilation time anyway.

                                                                                                                                                                                            1. 3

                                                                                                                                                                                              The undecidable grammar, the complex templates, and the header files. Sure we have pre-compiled headers, but in general, those are just copy pasta that are being parsed and analysed over and over and over again.

                                                                                                                                                                                              The parsing and analysis isn’t the whole problem. The C++ compilation model is an incremental evolution of Mary Allen Wilkes’ design from the ’70s, which was specifically designed to allow compiling complex problems on machines with around 2 KiB of memory. Each file is compiled separately and then pasted together in a completely separate link phase. In C++, inline and templated functions (including methods on templated classes) are emitted in every compilation unit that uses them. If 100 files use std::vector<int>::push_back then 100 instances of the compiler will create that template instantiation (including semantic analysis), generate IR for it, optimise it, and (if they still have calls to it left after inlining) spit out a copy of it in a COMDAT in the final binary. It will then be discarded at the end.

                                                                                                                                                                                              Sony has done some great work on a thing that they call a ‘compilation database’ to address this. In their model, when clang sees a request for std::vector<int>::push_back, it queries a central service to see if it’s already been generated. It can skip the AST generation and pull the IR straight from the service. Optimisers can then ignore this function except for inlining (and can provide partially optimised versions to the database). A single instance is emitted in the back end. This gives big compile time speedups, without redesigning the language.

                                                                                                                                                                                              It’s a shame that the Rust developers didn’t build on this model. Rust has a compilation model that’s more amenable to this kind of (potentially distributed) caching than C++.

                                                                                                                                                                                          2. 1

                                                                                                                                                                                            What’s clear to me is that Go’s designers didn’t read Pierce’s Type and Programming Languages,

                                                                                                                                                                                            I’ll just leave this here https://www.research.ed.ac.uk/en/publications/featherweight-go

                                                                                                                                                                                            1. 0

                                                                                                                                                                                              I meant back when Go first came out. That was over 12 years ago, in 2009. This paper is from 2020.

                                                                                                                                                                                              Nevertheless, the present thread significantly lowered my confidence in that claim. I am no longer certain Go designers failed to read Pierce’s work or equivalent, I now merely find it quite plausible.

                                                                                                                                                                                            2. 1

                                                                                                                                                                                              What’s clear to me is that Go’s designers didn’t read Pierce’s Type and Programming Languages . . .

                                                                                                                                                                                              Do you really think that Ken Thompson, Rob Pike, and Robert Greisemer were ignorant to this degree? That they made the design decisions they did based on a lack of theoretical knowledge?

                                                                                                                                                                                              1. 2

                                                                                                                                                                                                None of them is known for statically typed functional languages, and I know for a fact there is little cross talk between that community and the rest of the world. See Java, designed in 1995, twenty years after ML showed the world not only how to do generics, but how neat sum types are. Yet Java’s designers chose to have null instead, and generics came only years later. Now I kinda forgive them for not including generics at a time they likely believed class based polymorphism would replace parametric polymorphism (a.k.a. generics), but come on, adding null when we have a 20 year old better alternative out there?

                                                                                                                                                                                                So yeah, ignorance is not such an outlandish hypotheses, even with such people. (Edit: apparently one of them did read TAPL, so that should falsify the ignorance hypothesis after all.)

                                                                                                                                                                                                But that’s not my only hypothesis. Another possibility is contempt for their users. In their quest for simplicity, they may have thought the brains of Go programmers would be too weak to behold the frightening glory of generics. That instead they’d stay in the shelter of more familiar languages like Python or C. I’m not sure how right they may have been on that one to be honest. There are so many people that don’t see the obvious truth that programming is a form of applied maths (some of them explicitly fled maths), that I can understand they may panic at the first sight of an unspecified type. But come on, we don’t have to use generics just because they’re there. There’s no observable difference between a built in map and one that uses generics. Users can use the language now, and learn generics later. See how many people use C++’s STL without knowing the first thing about templates.

                                                                                                                                                                                                Yet another hypothesis is that they were in a real hurry, JavaScript style, and instead of admitting they were rushed, they rationalised the lack of generics like it was a conscious decision. Perhaps they were even told by management to not admit to any mistake or unfinished job.

                                                                                                                                                                                                1. 6

                                                                                                                                                                                                  But that’s not my only hypothesis. Another possibility is contempt for their users. … Yet another hypothesis is that they were in a real hurry, JavaScript style, and instead of admitting they were rushed, they rationalised the lack of generics like it was a conscious decision. Perhaps they were even told by management to not admit to any mistake or unfinished job.

                                                                                                                                                                                                  This is exhausting and frustrating. It’s my own fault for reading this far, but you should really aim for more charity when you interpret others.

                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                    In the words of Rob Pike himself:

                                                                                                                                                                                                    The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

                                                                                                                                                                                                    I’d say that makes it quite clear.

                                                                                                                                                                                                    1. 0

                                                                                                                                                                                                      Ah, I didn’t remember that quote, thank you. That makes the contempt hypothesis much more plausible.

                                                                                                                                                                                                      That being said, there’s a simple question of fact that is very difficult to ascertain: what the “average programmer” is capable of understanding and using, and at what cost? I personally have a strong intuition that generics don’t introduce unavoidable complexity significant enough to make people lives harder, but I’m hardly aware of any scientific evidence to that effect.

                                                                                                                                                                                                      We need psychologists and sociologists to study is.

                                                                                                                                                                                                    2. 1

                                                                                                                                                                                                      I’ve ran out of charitable interpretations to be honest. Go designers made a mistake, plain and simple. And now that generics have been added, that mistake has mostly been fixed.

                                                                                                                                                                                                      1. 1

                                                                                                                                                                                                        I’m surprised you’re saying this after noticing that you didn’t know basic things about Go’s type system and implementation and learning those details “changes everything” (which, in my opinion, is commendable). Indeed, you’ve also apparently learned many facts about the authors and their history in this thread. Perhaps this is a good moment to be reflective about the arguments you’re presenting, with how much certainty you’re presenting them, and why.

                                                                                                                                                                                                        1. 2

                                                                                                                                                                                                          A couple things:

                                                                                                                                                                                                          • I still think that omitting generics from a somewhat general purpose language past 2005 or so is a mistake. The benefits are just too large.
                                                                                                                                                                                                          • I’ve seen comments about how the standard library itself had to jump through some hoops that wouldn’t be there if Go had generics from the start. So Go authors did have some warning.
                                                                                                                                                                                                          • Go now has generics, even though adding them after the fact is much harder. There can be lots of reasons for this change, but one of them remains an admission of guilt: “oops we should have added generics, here you are now”.

                                                                                                                                                                                                          So yeah, I still believe beyond reasonable doubt that omitting generics back then was a mistake.

                                                                                                                                                                                                    3. 3

                                                                                                                                                                                                      All of your “analyses” are rooted in a presumption of ignorance, or malice, or haughty superiority, or some other bad-faith foundation. Do you really thing that’s the truth of the matter?

                                                                                                                                                                                                      There are so many people that don’t see the obvious truth that programming is a form of applied maths

                                                                                                                                                                                                      Some programming is a form of applied math. Most programming, as measured by the quantity of code which exists and is maintained by human beings, is not. Most programming is the application of computational resources to business problems. It’s imperative, it’s least-common-denominator, and it’s boring.

                                                                                                                                                                                                      1. 1

                                                                                                                                                                                                        All of your “analyses” are rooted in a presumption of ignorance, or malice, or haughty superiority, or some other bad-faith foundation. Do you really thing that’s the truth of the matter?

                                                                                                                                                                                                        The only way it’s false is if omitting generics was the right thing to do. I don’t believe that for a second. It was a mistake, plain and simple. And what could possibly cause mistakes, if not some form of incompetence or malice?

                                                                                                                                                                                                        Most programming is the application of computational resources to business problems. It’s imperative, it’s least-common-denominator, and it’s boring.

                                                                                                                                                                                                        It’s also maths. It’s also the absolutely precise usage of a formal notation that ends up being transformed into precise instructions for an (admittedly huge) finite state machine. Programs are still dependency graphs, whose density is very important for maintainability — even the boring ones.

                                                                                                                                                                                                        It’s not the specific kind of maths you’ve learned in high school, but it remains just as precise. More precise in fact, given how unforgiving computers are.

                                                                                                                                                                                                        1. 2

                                                                                                                                                                                                          The only way it’s false is if omitting generics was the right thing to do. I don’t believe that for a second.

                                                                                                                                                                                                          “The right thing to do” is a boolean outcome of some function. That function doesn’t have a a single objective definition, it’s variadic over context. Can you not conceive of a context in which omitting generics was the right thing to do?

                                                                                                                                                                                                          1. 2

                                                                                                                                                                                                            I see some:

                                                                                                                                                                                                            1. Designing a language before Y2K. Past 2005, it is too easy to know about them to ignore them.
                                                                                                                                                                                                            2. Addressing a specific niche for which generics don’t buy us much.
                                                                                                                                                                                                            3. Generics are too difficult to implement.
                                                                                                                                                                                                            4. Users would be too confused by generics.
                                                                                                                                                                                                            5. Other features incompatible with generics are more important.

                                                                                                                                                                                                            Go was designed too late for (1) to fly with me, and it is too general purpose for (2). I even recall seeing evidence that its standard library would have significantly benefited from generics. I believe Rust and C++ have disproved (3) despite Go using value types extensively. And there’s no way I believe (4), given my experience in OCaml and C++. And dammit, Go did add generics after the fact, which disavows (4), mostly disproves (3), and utterly destroys (5). (And even back then I would have a hard time believing (5), generics are too important in my opinion.)

                                                                                                                                                                                                            So yeah, I can come up with various contexts where omitting generics is the right think to do. What I cannot do is find one that is plausible. If you can, I’m interested.

                                                                                                                                                                                                            1. 1

                                                                                                                                                                                                              [Go] is too general purpose for (2). I even recall seeing evidence that its standard library would have significantly benefited from generics.

                                                                                                                                                                                                              You don’t need to speculate about this stuff, the rationale is well-defined and recorded in the historical record. Generics were omitted from the initial release because they didn’t provide value which outweighed the cost of implementation, factoring in overall language design goals, availability of implementors, etc. You can weight those inputs differently than the authors did, and that’s fine. But what you can’t do is claim they were ignorant of the relevant facts.

                                                                                                                                                                                                              I believe Rust and C++ have disproved (3)

                                                                                                                                                                                                              A language is designed as a whole system, and its features define a vector-space that’s unique to those features. Details about language L1 don’t prove or disprove anything about language L1. The complexity of a given feature F1 in language L1 is completely unrelated to any property of that feature in L2. So any subjective judgment of Rust has no impact on Go.

                                                                                                                                                                                                              Go did add generics after the fact, which disavows (4), mostly disproves (3), and utterly destroys (5).

                                                                                                                                                                                                              Do you just not consider the cost of implementation and impact on the unit whole as part of your analysis? Or do you weight these things so minimally as to render them practically irrelevant?

                                                                                                                                                                                                              Generics did not materially impact the success of the goals which Go set out to solve initially. Those goals did not include any element of programming language theory, language features, etc., they were explicitly expressed at the level of business objectives.

                                                                                                                                                                                                              1. 1

                                                                                                                                                                                                                You don’t need to speculate about this stuff, the rationale is well-defined and recorded in the historical record.

                                                                                                                                                                                                                What record I have read did not convince me. If you know of a convincing article or discussion thread, I’d like to read it. A video would work too.

                                                                                                                                                                                                                I believe Rust and C++ have disproved (3)

                                                                                                                                                                                                                A language is designed as a whole system […]

                                                                                                                                                                                                                I picked Rust for a specific reason: manual memory management, which means value types everywhere, and the difficulties they imply for generics. That said, I reckon that Go had the additional difficulty of having suptyping. But here’s the thing: in a battle between generics and subtyping, if implementing both is too costly, I personally tend to sacrifice subtyping. In a world of closures, subtyping and suptype polymorphism simply are not needed.

                                                                                                                                                                                                                Do you just not consider the cost of implementation and impact on the unit whole as part of your analysis?

                                                                                                                                                                                                                I’m not sure what you mean there… I think pretty much everyone agrees that designing and implementing generics up front is much easier than doing so after the fact, in a system not designed for them. If the Go team/community were able to shoulder the much higher cost of after-the-fact generics, then they almost certainly could have shouldered the cost of up-front generics back then —even though the team was much smaller.

                                                                                                                                                                                                                Generics did not materially impact the success of the goals which Go set out to solve initially.

                                                                                                                                                                                                                Well if they just wanted to have a big user base, I agree. The Google brand and the reputation of its designers did most of that work. As for real goals, they’re the same as any language: help the target audience write better programs for cheaper in the target niche. And for this, I have serious doubts about the design of Go.

                                                                                                                                                                                                                Now as @xigoi pointed out, Go authors targetted noobs. That meant making the language approachable by people who don’t know the relevant theory. That didn’t mean making the language itself dumb. Because users can’t understand your brilliant language doesn’t mean they won’t be able to use it. See every C++ tutorial ever, where you’re introduced to its features bit by bit. For instance when we learn I/O in C++ we don’t get taught about operator overloading (<< and >> magically work on streams, and we don’t need to know why just yet). Likewise we don’t learn template meta programming when we first encounter std::vector.

                                                                                                                                                                                                                People can work with generics before understanding them. They won’t write generic code just yet, but they absolutely can take advantage of already written code. People can work with algebraic data types. They won’t write those types right away, but they can absolutely take advantage of the option type for the return value of functions that may fail.

                                                                                                                                                                                                                A language can be brilliant and approachable. Yet Go explicitly chose to be dumb, as if it was the only way to be easy to work with. Here’s the thing though: stuff like the lack of generics and sum types tends to make Go harder to work with. Every time someone needed a generic data structure, they had to sacrifice type safety and resort to various conversion to and from the empty interface. Every time someone needs to report failures to the caller, they ended up returning multiple values, making things not only cumbersome, but also fairly easy to miss —with sum types at least the compiler warns you when you forget a case.

                                                                                                                                                                                                                It’s all well and good to design a language for other people to use, but did they study the impact of their various decision on their target audience? If I’m writing a language for myself I can at least test it on myself, see what feels good, what errors I make, how fast I program… and most of my arguments will be qualitative. But if I’m writing for someone else, I can’t help but start out with preconceived notions about my users. Maybe even a caricature. At some point we need to test our assumptions.

                                                                                                                                                                                                                Now I say that, such studies are bloody expensive, so I’m not sure what’s the solution there. When I made my little language, I relied on preconceived notions too. We had the requirements of course, but all I knew wast that my users weren’t programmers by trade. So I made something that tries its best to get out of their way (almost no explicit typing by default), and reports errors early (static typing rules). I guess I got lucky, because I was told later that they were happy with my language (and home grown languages have this reputation for being epically unusable).

                                                                                                                                                                                                                1. 1

                                                                                                                                                                                                                  But here’s the thing: in a battle between generics and subtyping, if implementing both is too costly, I personally tend to sacrifice subtyping. In a world of closures, subtyping and suptype polymorphism simply are not needed.

                                                                                                                                                                                                                  Do you consider generics and subtyping and polymorphism and other programming language properties means to an end, or ends in themselves?

                                                                                                                                                                                                                  1. 0

                                                                                                                                                                                                                    Of course they’re a means to an end, why do you even ask?

                                                                                                                                                                                                                    In the programs I write, I need inheritance or subtyping maybe once a year. Rarely enough that using closures as a poor’s man classes is adequate. Heck, even writing the odd virtual table in C is enough in practice.

                                                                                                                                                                                                                    Generics however were much more useful to me, for two purposes: first, whenever I write a new data structure or container, it’s nice to have it work on arbitrary data types. For standard libraries it is critical: you’ll need what, arrays & slices, hash tables, maybe a few kind of trees (red/black, AVL…).

                                                                                                                                                                                                                    I don’t write those on a day to day basis, though. I’ve also grown susceptible to Mike Acton’s arguments that being generic often causes more problems than it solves, at least when speed matters. One gotta shape one’s program to one’s data, and that makes generic data structures much less useful.

                                                                                                                                                                                                                    My second purpose is less visible, but even more useful: the right kind of generics help me enforce separation of concerns to prevent bugs. That significantly speeds up my development. See, when a type is generic you can’t assume anything about values of that type. At best you can copy values around. Which is exactly what I’m looking for when I want to isolate myself from that type. That new data structure I’m devising just for a particular type? I’m still going to use generics if I can, because they make sure my data structure code cannot mess with the objects it contains. This drastically reduces the space of possible programs, which is nice when the correct programs are precisely in that space. You can think of it as defining bugs out of existence, like Ousterhout recommends in A Philosophy of Software Design.

                                                                                                                                                                                                                2. 1

                                                                                                                                                                                                                  A language is designed as a whole system, and its features define a vector-space that’s unique to those features.

                                                                                                                                                                                                                  How do you add two languages, or multiply a language by an element of a field?

                                                                                                                                                                                              1. 5

                                                                                                                                                                                                I don’t really agree that maps are a basic data structure, and I think the amount of C++ implementations and comparisons linked in the author’s post proves that. They are widely used and providing a “good enough in most cases” implementation like Go does would probably be useful, but I wouldn’t say it’s necessary in a language as barebones as Hare. And it would require the language to have some form of generics (which should’ve been in from the start, imo) or some compiler magic like Go did with map[].

                                                                                                                                                                                                1. 5

                                                                                                                                                                                                  I don’t really agree that maps are a basic data structure

                                                                                                                                                                                                  I’ve personally never written a nontrivial program which didn’t make use of a map-like data structure.

                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                    Many C programs often use a singly linked list for n < ~10 - that usually performs very well, a lookup is often only 10 comparisons which is probably pretty close to the cost of running a hash function anyway… I need to run a benchmark.

                                                                                                                                                                                                    I have seen compilers do this - its quite common to have fewer than 10 variables in a scope. It’s still a map, but for many human scale problems a hash map doesn’t matter.

                                                                                                                                                                                                1. 1

                                                                                                                                                                                                  Can someone ELI5 why it’s important to distinguish between “exactly once” and “at least once” plus “idempotent processing”?

                                                                                                                                                                                                  Is there something functionally different about the two? Maybe “exactly once” implies a degree of efficiency that might matter in some applications?

                                                                                                                                                                                                  Or is it just a way to signal that you have heard of Two Generals and aren’t doing all the easy, reasonable, wrong things that someone like me might try?

                                                                                                                                                                                                  1. 1

                                                                                                                                                                                                    Given a system that delivers messages, and a system that receives and processes those messages, “exactly once” means the receiving system doesn’t need to be idempotent. This is important because “idempotent processing” can be nontrivial to achieve in practice.

                                                                                                                                                                                                    1. 2

                                                                                                                                                                                                      Thanks, I hadn’t thought of it as a statement about an interface rather than a statement about an implementation detail inside a system.

                                                                                                                                                                                                  1. 31

                                                                                                                                                                                                    This post can be summed up by “I prefer promise syntax.” The argument that try / catch sucks but somehow Promise.then().then().then().catch() is better or easier to understand is subjective at best. I’ve never met someone who has written a lot of JS who prefers promise syntax to async/await, but I guess there’s a lid for every pot.

                                                                                                                                                                                                    1. 9

                                                                                                                                                                                                      I prefer promises to async/await. Async/await is a bad, leaky abstraction. For anything non-trivial you have to fall back to promises (eg Promise.all). If I’m going to have promises anyway, I might as well just use them everywhere. The other thing is that async/await doesn’t compose at all while promises do.

                                                                                                                                                                                                      async/await also directs the programmer towards inefficient code by default by encouraging sequential async operations (I see people writing code like that often).

                                                                                                                                                                                                      1. 15

                                                                                                                                                                                                        Promises themselves are a bad abstraction though.

                                                                                                                                                                                                        • They’re invoked immediately so you can’t build up a pipeline are pass them around easily and this causes a lot of devs to thunk their promises
                                                                                                                                                                                                        • Catching errors was never required so errors are often swallowed or are rethrown in a way that it’s hard to debug
                                                                                                                                                                                                        • Cancelation had to be added way later than it should have been (and many folk had warned during the proposal phase) and as a result we have this bad API for AbortControllers
                                                                                                                                                                                                        • all vs allSettled should have been there from the get-go instead added of years later
                                                                                                                                                                                                        • Folks realized it’d be nice to have a do-like notation so async & await were rushed out without considering the ergonomics for the parallel APIs

                                                                                                                                                                                                        What JavaScript wanted was an asynchronous Task and could have been copied from established languages, but what we got was Promise. We’d be better off normalizing picking for our projects one of the Fluture library, fp-ts’s Task, Aff from PureScript, or whatever the API is called in effect-ts (they are all close to the same thing).

                                                                                                                                                                                                        1. 3

                                                                                                                                                                                                          Totally agree! It’s just that async/await is even worse, and all I have to fall back on is promises.

                                                                                                                                                                                                        2. 3

                                                                                                                                                                                                          Async/await are promises. It doesn’t have to be an either/or choice.

                                                                                                                                                                                                          1. 2

                                                                                                                                                                                                            That’s precisely the problem. Is there any value in a tiny bit of syntactic sugar that also encourages inefficient code? In my view, the value is negative.

                                                                                                                                                                                                            1. 1

                                                                                                                                                                                                              A lot of the code where this would be used is going to have branching logic and overall not be terribly long anyways, so the emphasis on inefficiency doesn’t really ring true to me.

                                                                                                                                                                                                              I will admit to not knowing the intricacies of javascript optimization, but I fail to see how using a syntactic sugar would impact it at all.

                                                                                                                                                                                                              This whole discussion sounds like like arguing between the do syntax and the bind operator in Haskell. I for one prefer my monads easier to read. As code should be for reading first and compilers second.

                                                                                                                                                                                                              The only thing I will concede is that I have seen developers try to break out of the asynchronous context by assigning variables in the synchronous context. This might be worse in rxjs though. The use of Promise directly does force one to read it as asynchronous compared to missing the async keyword.

                                                                                                                                                                                                          2. 2

                                                                                                                                                                                                            They are both silly and limited. They both rely on. Funny looking code. They both introduce disturbance the code execution thredline rather then logically allow the programmer to control when an arbitrary code block is executed. We have to really on somewhat funny syntax.

                                                                                                                                                                                                            Both of them are limited to “execute this on the side and when you are ready execute that. But there are no means of controling concurrency. If you want to do thousands of instances of something, both APIs become essentially fork bombs.

                                                                                                                                                                                                            Dijkstra wrote down a blueprint for intuitive concurrency (P() and V()) which has been used in many languages for decades. I don’t understand why we have those funny looking things instead of what is tried and true.

                                                                                                                                                                                                            Even python went down the path of async wait. The result is a gigantic divider on any python codebase that uses. Any and every little IO operation becomes subject to this great divide. Feels like each time a line if code with IO on it comes, there’s a gun pointed to the head asking: sync or async?

                                                                                                                                                                                                            1. 2

                                                                                                                                                                                                              Sure, I agree that both mechanisms are bad, it’s just that async/await is worse. I much prefer the approach to concurrency taken by Elixir/Erlang. I don’t think, however, that semaphores by themselves are the only mechanism we need.

                                                                                                                                                                                                              1. 1

                                                                                                                                                                                                                What do you need more than a semaphore and a method to fork execution in two? You could trivialy implement a pool with it.

                                                                                                                                                                                                              2. 1

                                                                                                                                                                                                                Dijkstra wrote down a blueprint for intuitive concurrency (P() and V()) which has been used in many languages for decades. I don’t understand why we have those funny looking things instead of what is tried and true.

                                                                                                                                                                                                                P/V models a specific concurrency primitive usually known as a semaphore, right? As far as I understand them, semaphores are a useful tool, but don’t represent a solution to concurrent programming in general — am I mistaken?

                                                                                                                                                                                                                1. 4

                                                                                                                                                                                                                  You can model any concurrency problem using semaphores. That doesn’t mean you want to.

                                                                                                                                                                                                                  1. 2

                                                                                                                                                                                                                    It has been very long since I have red the paper. I don’t remember if it explicitly mentioned that some means of parelelize execution is needed at some point or that was assumed. Fork() spawn(), thread.new(). Whatever it looks like. The problem with old school multi process or multi threading was not the API, but rather the overhead. There is no reason why such an API could not be used with say, green threads. Eventlet does just that.

                                                                                                                                                                                                              3. 8

                                                                                                                                                                                                                Yeah this whole article reads like the author wanted to say “I find async/await new and unsettling” without saying those words. And the parallel example feels like a strawman: why would async/await make you forget that you want to wrap parallel work in Promise.all?

                                                                                                                                                                                                                I think it’s fine to say “I prefer the old ways” without needing some objective justification.

                                                                                                                                                                                                                1. 1

                                                                                                                                                                                                                  I think it’s fine to say “I prefer the old ways” without needing some objective justification.

                                                                                                                                                                                                                  In a professional setting, you need objective decisions to convince people. It’s very difficult to use your own preferences to change the style of other authors.

                                                                                                                                                                                                                  1. 4

                                                                                                                                                                                                                    Totally. This article is not that, tho.

                                                                                                                                                                                                                2. 1

                                                                                                                                                                                                                  The post cannot be summed up that way.