Threads for atk

  1. 4

    GCC and Clang have an available warning for when you combine some operators that are common causes of precedence bugs — I think one example is == and &. It’ll nag you to add parens to make your intentions clear. I’m pretty sure this is one of the warnings included with -Wall.

    Yet another good reason to turn on lots of warnings plus -Werror! In fact I’ve taken to opting out of warnings, not in — I include a canned file in my build system (XcodeWarnings.xcconfig) that turns on nearly every available compiler warning, then I follow it with some lines to turn off specific warnings I don’t want.

    1. 4

      For a long time, I have used a style rule in my own code that says ‘any sequence containing two operators must be parenthesised’. This means that A op1 B op1 C op1 D is fine but A op1 B op2 C must be written as either (A op1 B) op2 C or A op1 (B op2 C), whichever is intended. Since I started following this rule, I have never had to think about operator precedence. It doesn’t matter if the sequence is unambiguous to the compiler, it must also be unambiguous to the reader. If the reader has some cognitive load from remembering precedence, then that is mental effort that they are not able to spend finding real bugs in my code.

      1. 2

        Parsing adjacent (( parentheses and matching them across expressions is also a cognitive load (yes, even if you colour the parentheses). So is having to figure out WHY someone put parentheses in their expression as usually that’s an indication that normal operator precedence wasn’t applicable. Adding parentheses is not zero cost for everyone and I only do it explicitly (in situations where default precedence would otherwise have conveyed the same meaning) when mixing && and || or where & or | are used within a larger expression.

        The same can be said about useless comments in code, when people comment something it draws the eye and makes the reader pay special attention, when the comments are literally just a translation of the code into English this puts unnecessary cognitive strain on the reader.

        1. 0

          Parsing adjacent (( parentheses and matching them across expressions is also a cognitive load (yes, even if you colour the parentheses)

          Fortunately, most people reading my code use a visual cortex that evolved to recognise predators that have bilateral symmetry. As such, they can offload matching parentheses to some dedicated processing before it reaches their conscious mind.

      2. 2

        I’m not a fan of -Werror. I like compiling with -std=c99 -pedantic and there are a few POSIX features that run afoul of that. For example, “ISO C forbids assignment between function pointer and `void *’”, but POSIX requires that.

        1. 1

          What’s an example of POSIX requiring it? AFAIK that assignment is undefined behaviour.

          1. 2

            Since POSIX specifies the functions, such as dlopen, commonly found in dlfcn.h, and dlopen returns a void * it is therefore required to work on a POSIX compliant system. I think the standard calls it out explicitly somewhere but you’ll have to do your own legwork there.

            1. 1

              After a sleep I noticed the mistake, it’s dlsym that matters not dlopen but the point still stands.

          2. 1

            Better IMHO to leave -Werror on but turn off the specific warning(s) you don’t like. The warning message includes the name of the flag that controls it.

        1. 2

          Let’s hope that a: there’s a fallback, b: there’s an option to keep using passwords and c: the protocol used is something open and standard so that if I want to I can write an appropriate “authenticator” application for any platform and don’t have to rely on software blessed by microsoft, apple and google (and d: the protocol isn’t some abomination which is basically impossible to implement by a single human, and e: the implementation doesn’t have to be blessed by an authority who will just deny your application because you’re a lone developer writing something for yourself).

          1. 2

            The protocol I believe is FIDO U2F. But who knows if they add some extra to it to make it unique to their platform, like MS did with Active Directory. I hope not, but history would say I shouldn’t keep my hopes very high.

            I also hope they allow hardware U2F keys like Yubikey, NitroKey, etc.

            1. 1

              It’s an extension to WebAuthn

              1. 3

                I’m not an expert but this article from Mozilla suggests the opposite, that WebAuthentication is an extension of FIDO. https://blog.mozilla.org/security/2019/04/04/shipping-fido-u2f-api-support-in-firefox/

            1. 3

              This has some weird bugs. The precedence of a(b) >> c &= d is definitely not (a(b)) >> (c &= d), it is (((a(b)) >> c) &= d) (which is invalid in C as you can’t assign to an expression). The precedence of a *= b != c ? d : e is not ((a *= (b != c)) ? d : e), it is (a *= ((b != c) ? d : e)). There seems to be some issues with this game’s understanding of the precedence of assignment operators and also the associativity of comparison operators.

              1. 1

                Perhaps this is a bug in Tree-sitter? You can see the AST it makes here.

                https://tree-sitter.github.io/tree-sitter/playground

                Edit: Reported: https://github.com/tree-sitter/tree-sitter-c/issues/101

              1. 2

                I put the backup codes in the password manager, along with TOTP secrets and passwords. Yes, I also see the weird irony.

                1. 29

                  It’s still too easy to click a link with this. There needs to be a mode where no matter how long the page has been loaded, it tracks the mouse cursor, and when it looks like the user is about to click something, it loads an advertisement in a way which pushes the link out of the way and makes the user click on the advertisement.

                  1. 7

                    onclick="load_popup(); click_popup();"

                    1. 9

                      For a bit more realism, put the popup under the mouse on mousedown instead of on click. That way, the user feels the horror of what’s about to happen for a fraction of a second as they continue their motion and release the mouse button. It makes the user blame themselves for the click: if they had better reaction times they could have dragged the mouse off the popup.

                  1. 35

                    a poor intern that had a difficult to describe kind of flabbergasted expression on his face once the call connected.

                    … is it really that surprising?

                    I mean this project is interesting and all that, but it seems kind of unusual to use it for social interaction with coworkers. It seems even more distancing than just not having video. Is it really necessary to distract your coworkers with things like these?

                    1. 22

                      Agreed. This kind of pushes the bounds of “professional” behavior well beyond what I’d expect people to accept. I’m 100% in favor of people being able to express themselves at work, when it’s not a distraction to actually doing work. However these avatars fall so deeply into the depths of the uncanny valley that they can be incredibly painfully distracting to look at, and I find myself unable to actually pay attention to the content. I’d immediately ask a coworker who used one to turn it off… no webcam at all would be vastly preferable.

                      1. 10

                        At work, I never turn on my webcam. On my desktop, I don’t even have one plugged in. No one cares.

                      2. 16

                        I admit that in hindsight that was a mistake. However a lot of the reason I use it sparingly is because I hate how I look and would much rather have the ability to present myself in a way that is not the body I was cursed into when I was born into this plane.

                        1. 1

                          I think this is fine as long as it is opt-in. The UI would advise everyone involved that there is a ridiculous distraction and only show it to those who are OK with it.

                        2. 14

                          “Anime is real?!” –Intern

                          Honestly I’m personally glad that people are putting in the social capital to make this acceptable. Morphological freedom should be a human right; this is a small step towards that.

                        1. 10

                          It doesn’t recognise that software breaks are common and not all equal.

                          I disagree with both parts.

                          Not all breakage is equal in the sense that library developers should be even more reluctant to introduce a change that breaks say 50% of existing projects than a change that only break 0.5%.

                          However, for downstream users, all breaking changes are equal. If it’s my code that breaks after updating from libfoo 1.0.0 to 2.0.0, I don’t care how many other projects have the same problem since it doesn’t reduce the need to adapt my code.

                          If you define breaking change severity as the number of affected API functions, the picture doesn’t really change. A small number of affected functions doesn’t always mean a simple fix.

                          I agree that software breakage is common, but the real solution is to stop making breaking changes without a really good reason, not to invent versioning schemes that make breaking changes less visible in the major version number.

                          1. 8

                            As a consumer of many libraries in dynamically typed languages “this easily grappable thing has a different call signature, same results” is qualitatively different from “this API disappeared/is now semantically entirely different”.

                            Sure don’t break things without good reasons. But that’s true in any respect!

                            1. 5

                              However, for downstream users, all breaking changes are equal.

                              Disagree on that. If a breaking change fixes a typo in a function name, renames something, or even just deprecates something with a clear and easy replacement path, I don’t care as a user. It’s just grooming.

                              The distinction between major and minor breaking changes fits my expectations both as a user and as a developer.

                              1. 3

                                However, for downstream users, all breaking changes are equal.

                                A breaking change that doesn’t break my code is facially less bad than one that does.

                                This emerging notion that breaking changes are an Ur-failure of package authors which must be avoided at all costs biases so strongly toward consumers that it’s actually harmful for software development in the large. Software needs to have breaking changes over time, to evolve, in order to be healthy. Nobody gets it right the first time, and setting that as the baseline expectation is unrealistic and makes everyone’s experience net worse.

                                1. 4

                                  That notion is biased towards the ecosystem as a whole. We all are both producers and consumers.

                                  If there are two libraries with comparable functionality and one makes breaking changes often while the other doesn’t, it’s the latter that brings more benefit to the ecosystem by saving developer time. In essence, compatibility is a feature.

                                  I’m not saying that people should never make breaking changes, only that it requires a really good justification. Fixing unfortunate function names, for example, doesn’t require a breaking change—keeping the old name as an alias and marking it deprecated is all that’s needed, and it adds too few bytes to the library to consider that “bloat”.

                                  1. 2

                                    I’m not saying that people should never make breaking changes, only that it requires a really good justification. Fixing unfortunate function names, for example, doesn’t require a breaking change—keeping the old name as an alias and marking it deprecated is all that’s needed, and it adds too few bytes to the library to consider that “bloat”.

                                    The amount of justification required to make a breaking change isn’t constant, it’s a function of many project-specific variables. It can easily be that the cost of keeping that deprecated function around — not just in terms of size but also in API coherence, future maintainability, etc. — outweighs the benefit of avoiding a breaking change.

                                    1. 1

                                      A lot of time the “maintaining API coherence” argument is a euphemism for making it look as if the design mistake was never made. Except it was, and now it’s my responsibility as the maintainer to ensure minimal impact on the people who chose to trust my code.

                                      I completely agree that those arguments can be valid, but breaking changes I see in the wild tend to be of the avoidable variety where a small bit of effort on the maintainer’s side was all that needed to fix it for future users without affecting existing ones.

                                      Breaking changes unannounced and perhaps even unnoticed by the authors are even worse.

                                      1. 2

                                        design mistake

                                        I just can’t get on board with this framing. “Design mistakes” are an essential and intractable part of healthy software projects. In fact they’re not mistakes at all, they’re simply stages in the evolution of an API. Our tools and practices have to reflect this truth.

                                        1. 2

                                          Both in this thread and every other time I’ve seen you talk about this topic on this website you’re always on the rather extreme side of “it’s impossible not to have completely unstable software”.

                                          There are many examples of very stable software out there, and there’s plenty of people who are very appreciative of very stable software. There are very stable software distributions out there too (debian) and there are plenty of people who are very appreciative of the fact that if they leave auto-updates on for 4 years and leave things alone that there’s a very small chance that they will ever have to fix any breakages, and when they eventually have to update, that things will also be very well documented and they likely won’t have to deal with a subtle and unnoticed breakage.

                                          Yes, it is a reality that new and bleeding edge concepts need to go through a maturation stage where they experiment with ideas before they can be solidified into a design, but pretending like this is true for every software project in existence and that therefore instability must just be accepted is just flat out wrong.

                                          We have had pretty decent stability for quite some time and plenty of people have been very happy with the tradeoff between stability and the bleeding edge.

                                          What I’m saying here is really that nobody is asking you to take your extremely immature project for which you don’t have the experience or foresight (yet) to stabilise and stabilise it prematurely. Of course there will always be projects like that and like anything they’re going to be bleeding-edge avant-garde ordeals which may become extremely successful and useful projects that everyone loves or end up in a dumpster. What I am saying here is that it’s okay to have an unstable project, but it’s not okay to mislead people about its stability and it is not okay to pretend like nobody should ever expect stability from any project. If your project makes no effort to stop breaking changes and is nowhere near being at a point where the design mistakes have been figured out then it should make that clear in its documentation. If your project looks like it uses a 3 part versioning scheme then it should make it clear that it’s not semver. And most importantly, just because your project can’t stabilise yet doesn’t mean that nobody’s project can stabilise.

                                          1. 2

                                            there are plenty of people who are very appreciative of the fact that if they leave auto-updates on for 4 years and leave things alone that there’s a very small chance that they will ever have to fix any breakages

                                            Let me distinguish software delivered to system end users via package managers like apt, from software delivered to programmers via language tooling like cargo. I’m not concerned with the former, I’m only speaking about the latter.

                                            With that said…

                                            you’re always on the rather extreme side of “it’s impossible not to have completely unstable software”

                                            I don’t think that’s a fair summary of my position. I completely agree that it’s possible to have stable software.

                                            Let’s first define stable. Assuming the authors follow semver, I guess the definition you’re advancing is that it [almost] never increments the major version, because all changes over time don’t break API compatibility. (If that’s not your working definition, please correct me!)

                                            I’m going to make some claims now which I hope are noncontroversial. First, software that exhibits this property of stability is net beneficial to existing consumers of that software, because they can continue to use it, and automatically upgrade it, without fear of breakage in their own applications. Second, it is a net cost to the authors of that software, because maintaining API compatibility is, generally, more work than making breaking changes when the need arises. Third, it is, over time, a net cost to new consumers of that software, because avoiding breaking changes necessarily produces an API surface area which is less coherent as a unit whole than otherwise. Fourth, that existing consumers, potential/new consumers, and authors/maintainers are each stakeholders in a software project, and further that their relative needs are not always the same but can change depending on the scope and usage and reach of the project.

                                            If you buy these claims then I hope it is not much of a leap to get to the notion that the cost of a breaking change is not constant. And, further, that it is at least possible that a breaking change delivers, net, more benefit than cost, versus avoiding that change to maintain API compatibility. One common way this can be true is if the software is not consumed by very many people. Another common way is if the consumers of that software don’t have the expectation that they should be able to make blind upgrades, especially across major versions, without concomitant code changes on their side.

                                            If you buy that, then what we’re down to is figuring out where the line is. And my claim is that the overwhelming majority of software which is actively worked-on by human beings is in this category where breaking changes are not a big deal.

                                            First and foremost, because the overwhelming majority of software is private, written in market-driven organizations, and must respond to business requirements which are always changing by their very nature. If you bind the authors of that software to stability guarantees as we’ve defined them, you make it unreasonably difficult for them to respond to the needs of their business stakeholders. It’s non-viable. And it’s unnecessary! Software in this category rarely has a high consumer-to-producer ratio. The cost of a breaking change is facially less than, say, the AWS SDK.

                                            By my experience this describes something like 80-90% of software produced and maintained in the world. The GNU greps and the AWS SDKs and those class of widely-consumed things are a superminority of software overall. Important! But non-representative. You can’t define protocols and practices and ecosystem expectations for general-purpose programming languages using them as exemplars.

                                            Second, because consumers shouldn’t have the expectation that they can make blind upgrades across major versions without code changes on their side. It implies a responsibility that authors have over their consumers which is at best difficult in closed software ecosystems like I described above, and actually literally impossible in open software ecosystems like the OSS space. As an OSS author I simply don’t have any way to know how many people are using my software, it’s potentially infinite; and I simply can’t own any of the risk they incur by using it, it doesn’t scale. I should of course make good faith effort toward making their lives easier, but that can’t be a mandate, or even an expectation, of the ecosystem or it’s tooling.

                                            And, thirdly, because stability as we’ve defined it is simply an unreasonable standard for anything produced by humans, especially humans who aren’t being supported by an organization and remunerated for their efforts. Let me give you an example. Let’s say I come up with a flag parsing library that provides real value versus the status quo, by cleanly supporting flag input from multiple sources: commandline flags, environment variables, and config files. Upon initial release I support a single config file format, and allow users to specify it with an option called ConfigParser. It’s a hit! It delivers real value to the community. Shortly, a PR comes in to support JSON and YAML parsers. This expands the cardinality of my set of parsers from 1 to N, which means that ConfigParser is now insufficiently precise to describe what it’s controlling. If I had included support for multiple config files from the start, I would have qualified the options names: PlainConfigParser, JSONConfigParser, YAMLConfigParser, and so on. But now I’m faced with the question: do I avoid a breaking change, leave ConfigParser as it is, and just add JSONConfigParser and YAMLConfigParser in addition? Or do I rename ConfigParser to PlainConfigParser when I add the other two, in order to keep the options symmetric?

                                            My initial decision to call the option ConfigParser was not a design mistake. The software and its capabilities evolved over time, and it’s not reasonable to expect me to predict all possible future capabilities on day one. And the non-breaking change option is not necessarily the right one! If I have 100 users now, but will have 100,000 users in a year’s time, then breaking my current users — while bumping the major version, to be clear! — is the better choice, in order to provide a more coherent API to everyone in the future.

                                            Some people would say that this class of software should stay at major version 0 until it reaches a point of stability. This, too, isn’t reasonable. First, because stability is undefinable, and any judgment I make will be subjective and wrong in someone’s eyes. Second, because it strips me of the power to signal breaking vs. non-breaking changes to my users. Semver stipulates the semantics of each of its integers, but nothing beyond that — it doesn’t stipulate a mandatory rate of change, or anything like that. It leaves stability to be defined by the authors of the software. To be clear, more stable software is easier to consume and should be preferred when possible. But it can’t be the be-all end-all metric that software must optimize for.

                                            1. 2

                                              So let’s get some things out of the way.

                                              I don’t care about private spaces, and I don’t care about what people do in them.

                                              I also don’t care about de jure enforcement of stablity through limitations or tooling or whatever. My interest is specifically in de facto stability expectations.

                                              For ease of my writing, let’s define the term interface to mean something which you could reasonably write a standard to describe. This means things like protocols, APIs or programming languages. Let’s define producer to be someone who produces an interface. Let’s define consumer to be someone who uses an interface.

                                              I am of the opinion that in the programming world (encompassing both consumers and producers of interfaces) the following expectations should be de jure:

                                              • If you are the author of an interface which you publish for users, it is your responsibility to be honest about your stability promises with regards to that interface.

                                              • People should expect stability from commonly used interfaces.

                                              Just like people enjoy the benefits of the stability of debian, there are plenty of people who enjoy the benefits of the stability of a language like C. C programs I wrote 5 years ago still compile and run today, C programs I will write today will still compile and run in 5 years. I obviously have to follow the standard correctly and ensure my program is written correctly, but if for some reason my program cannot be compiled and cannot be ran in 5 years, I have nobody to blame but myself. This is an incredibly useful quality.

                                              The fact that I can’t expect the same of rust completely discounts rust as a viable choice of software for me.

                                              There is no reason why I should have to be excluded from new features of an interface just to avoid breaking changes.

                                              Moreover, in the modern world where security of software is an ever-growing concern, telling consumers to chose between interface stability and security is an awful idea.

                                              Now with that out of the way, let’s go over your points (I have summarised them, if you think my summary is wrong then we may be talking past each other, feel free to try to address my misunderstanding).

                                              Interface stability is beneficial for existing consumers.

                                              Yes.

                                              Interface stability is a cost to interface producers.

                                              Yes.

                                              Interface stability is a cost for new consumers.

                                              No.

                                              Keeping your interface stable does not prevent you from expanding it in a non-breaking way to improve it. It does not prevent you from then changing the documentation of the interface (or expanding on its standard) to guide new consumers towards taking the improved view of the interface. It does not prevent you from stopping further non-essential development on the old portions of the interface. It does not stop you from creating a new major version of the interface with only the new parts of the interface and only maintaining the old interface insofar as it requires essential fixes.

                                              Interface consumers and producers are both interface stakeholders but have different needs.

                                              Sure.

                                              The cost of a breaking change is not constant.

                                              Sure.

                                              It is possible that a breaking change delivers more benefit than cost versus avoiding the breaking change.

                                              I think this is disputable because of what I said earlier about being able to improve an interface without breaking it.

                                              But I can see this being the case when a critical security makes consumers of your interface vulnerable through normal use of the interface. This is a rare corner case though and doesn’t really make the whole idea or goal of interface stability any less worthwhile.

                                              This can happen your interface is not used by many consumers.

                                              Yes, you obviously should feel less obliged to keep things stable in this case, but you should still be obliged to make consumers well aware of just how stable your interface is.

                                              This can happen if consumers do not have high stability expectations.

                                              Which you can ensure by making them well aware of what stability expectations they should have.

                                              But I would argue that in general, consumers should be allowed and encouraged to have high stability expectations.

                                              especially across major versions, without concomitant code changes on their side

                                              I don’t see how this is relevant given that you already defined stability a certain way.

                                              Of course if your interface follows semver and you increment the major then you’ve made your stability expectations clear and if the consumers do not understand this then they can only blame themselves.

                                              The majority of interfaces belong in a category which excludes them from high stability expectations.

                                              Yes, as interfaces follow a pareto distribution, very few of them have a significant number of consumers where breaking changes will have a significant impact.

                                              The majority of interfaces are private

                                              Actually this is irrelevant, you can just categorise this as “tiny userbase”.

                                              That being said, there’s two sides to this coin.

                                              Google mail’s internal APIs may have a “tiny userbase” (although google is a big company so maybe not so), on the other hand, google mail’s user facing interface has a massive userbase.

                                              If you force producers of those interfaces to guarantee stability as defined above then you make it unreasonably difficult to respond to the needs of their business stakeholders.

                                              Nobody is forcing or suggesting forcing anyone to do anything.

                                              You can’t use interfaces with enormous amounts of consumers as baselines for how people should handle interface stability of their interfaces with fewer users.

                                              I’d say this is irrelevant.

                                              Your hello world script doesn’t need interface stability. That is one extreme. The C standard needs extreme stability, that is the other extreme. If you consider the majority of the things I am actually thinking of when we talk about “things being horribly unstable” then they definitely don’t fall 1% point away from hello world. I talk about things like rust, nim, major libraries within those languages, etc.

                                              Those things have significant numbers of cosumers yet their interfaces seem to come with the same stability expectations as the hello world program.

                                              Yes, stability is a scale, it’s not binary, but the current situation is that a lot of people are treating it as completely optional in situations where it’s certainly shouldn’t be. I don’t think it’s an unreasonable expectation to expect more stability from some of these projects. It is also not an unreasonable expectation to

                                              Because consumers shouldn’t high interface stability expectations across major versions this implies producers have a responsibility which is at best difficult in closed software ecosystems described above.

                                              I’m not actually sure how to interpret this but it seems to talk about things irrelevant to the point. Feel free to clarify.

                                              As the producer of an open source interface I don’t have any way to know how many consumers I have and therefore I cannot own any risk incurred by the consumers consuming my interface.

                                              Nobody is asking anyone to own any risk (especially in the open source ABSOLUTELY NO WARRANTY case). I am simply saying that if your project is popular and you break stability expectations when you have made it clear that people should have them, you should expect people to distrust your interface, stop using it and potentially distrust future interfaces you produce. I think this is only fair, and really the most I can ask of people.

                                              Moreover, you do have many ways of estimating how many people are using your software. If enough people are using your software for interface stability to matter then you will know about it (or you’re some expert hermit who manages to successfully maintain key project while never knowing anything about the outside world).

                                              Stability as defined above is an unreasonable and unreachable standard for human producers of interfaces.

                                              No, I don’t think it is, or rather, I thing you have produced a false dichotomy. Correct me if I’m wrong but you seem to think that because perfect stability is impossible, imperfect stability is no longer worthwhile. I think this is just a poor way of looking at things, imperfect stability is achievable to a high level and is worthwhile in many cases.

                                              … especially humans who aren’t being supported by an organization and remunerated for their efforts.

                                              Let’s put it this way, just because you decided to volunteer to do something doesn’t mean you have no responsibility. You take on responsibility by making the risk of creating something which people might come to rely on and use. If you do not wish to take on this responsibility, then it is at the very least your responsibility to make THAT clear.

                                              I would say it is childish to suggest that just because you are doing something for free that therefore you have no responsibility for the consequences of what happens when people come to rely on that free activity. There are a million and one real world examples of how this does not pan out.

                                              Let me give you an example. Let’s say I come up with a flag parsing library that provides real value versus the status quo, by cleanly supporting flag input from multiple sources: commandline flags, environment variables, and config files. Upon initial release I support a single config file format, and allow users to specify it with an option called ConfigParser. It’s a hit! It delivers real value to the community. Shortly, a PR comes in to support JSON and YAML parsers. This expands the cardinality of my set of parsers from 1 to N, which means that ConfigParser is now insufficiently precise to describe what it’s controlling. If I had included support for multiple config files from the start, I would have qualified the options names: PlainConfigParser, JSONConfigParser, YAMLConfigParser, and so on. But now I’m faced with the question: do I avoid a breaking change, leave ConfigParser as it is, and just add JSONConfigParser and YAMLConfigParser in addition? Or do I rename ConfigParser to PlainConfigParser when I add the other two, in order to keep the options symmetric?

                                              I would say that it’s actually worthwhile to avoid breaking API here. If your bar for breaking API is this low then you’re going to be breaking API every day of the week. Moreover, you won’t learn anything about API design from your project.

                                              There is real value in living with your mistakes for a while and letting things like this accumulate for a while. Simply accepting every breaking change as it comes is going to teach you less about which parts of your API are actually good or not than waiting for a good point to create config_parser_of_the_snake_case_variety a few years down the road with all the things you’ve learned incorporated. The end result will be a far better design than whatever you could cobble together with a weekly breaking change.

                                              You’re also ignoring the possibility of just making a new API (also, surely the correct option is not a name-per-format but rather an option) and wiring the old API to just forward to the new one. You can begin to think about future proofing your API while keeping the old one around (to everyone’s great benefit).

                                              Finally, if you’ve just written a library, why is it 1.0.0 already? Why would you do that to yourself? Spend a few months at least with it in a pre-1.0.0 state. Stabilize it once you iron out all the issues.

                                              My initial decision to call the option ConfigParser was not a design mistake.

                                              Of course not, the mistake was pretending that JSON and YAML are config formats. (This is a joke.)

                                              The software and its capabilities evolved over time, and it’s not reasonable to expect me to predict all possible future capabilities on day one.

                                              Of course not. But I think you’re being unreasonably narrow-minded with regards to possible options for how to evolve a project while keeping the interface stable.

                                              And the non-breaking change option is not necessarily the right one! If I have 100 users now, but will have 100,000 users in a year’s time, then breaking my current users — while bumping the major version, to be clear! — is the better choice, in order to provide a more coherent API to everyone in the future.

                                              It’s also extremely unlikely that you will be able to predict the requirements of your library one year in the future. Which is possibly why your library shouldn’t be a library yet and should wait a few years of usecases through before it solidifies into one.

                                              There’s too many tiny libraries which do half a thing poorly out there, at least part of this stability discussion should be around whether some interfaces should exist to begin with.

                                              Some say this class of interface should stay at major version 0.

                                              Yes, I think this is synonymous with my position that it shouldn’t be a library to begin with.

                                              This is unreasonable because stability is undefinable.

                                              No, you already defined it a couple of times in a way I would agree with. A stable interface is an interface where a consumer should not expect things to break if they update the software to the latest version (of that particular major version).

                                              Any stability judgement I make will be questioned by someone.

                                              Why leave the house then? You might die. This comes back again to the all-or-nothing mentality I mentioned earlier where you seem to suggest that just because perfect stability is impossible, imperfect stability is not worthwhile.

                                              Second, because it strips me of the power to signal breaking vs. non-breaking changes to my users.

                                              It doesn’t do that. You have literally all the tools at your disposal including your changelog and documentation. Moreover, if your project is at 0.x.y then you are completely free to tell people that if x increases then the interface will have changed significantly and if y increases then the interface probably hasn’t changed. Semver does not specify how x and y are to be interpreted for 0.x.y, only for M.x.y where M > 0.

                                              Semver explains what the numbers mean but nothing else. It does not specify the rate of change. It leaves stability to be defined by the authors of the software.

                                              Yes, these are true statements.

                                              Stability should not be the the most important goal for all software.

                                              Again, a true statement. My point is and always has been that the issue is not that all interfaces aren’t perfectly stable, it’s that there’s a lot of interfaces which are far less stable than they reasonably should be and that normalising this means that software is going to get a lot more confusing, a lot more insecure and a lot more broken as time goes on.

                                              Overall I think we’re in agreement on most parts except for how easy stability and how important it is relatively speaking.

                                              1. 2

                                                There is real value in living with your mistakes for a while and letting things like [ConfigParserX] accumulate for a while. Simply accepting every breaking change as it comes is going to teach you less about which parts of your API are actually good or not than waiting for a good point to create config_parser_of_the_snake_case_variety a few years down the road with all the things you’ve learned incorporated. The end result will be a far better design than whatever you could cobble together with a weekly breaking change.

                                                I don’t know how better to say this: the decisions in this example are not mistakes. They are made by a human being at a particular point in the evolution of a software project.

                                                Finally, if you’ve just written a library, why is it 1.0.0 already? Why would you do that to yourself? Spend a few months at least with it in a pre-1.0.0 state. Stabilize it once you iron out all the issues.

                                                1.x.y in the semver nomenclature represents a notion of stability which I as the author get to define. It isn’t a universal notion of stability, it’s nothing more than what I define it to be. There is no single or objective denomination of “when I iron out all of the issues” which I can meet in order to qualify a 1.x.y version identifier. It’s all subjective!

                                                just because you decided to volunteer to do something doesn’t mean you have no responsibility. You take on responsibility by making the risk of creating something which people might come to rely on and use. If you do not wish to take on this responsibility, then it is at the very least your responsibility to make THAT clear.

                                                Semver provides all of the mechanics I need to signal what you demand. If I make a breaking change, I increment the major version number. That’s it! There’s nothing more to it.

                                                1. 2

                                                  I don’t know how better to say this: the decisions in this example are not mistakes. They are made by a human being at a particular point in the evolution of a software project.

                                                  If it was reasonable to assume that the API would need that feature eventually then yes it’s a mistake. If it was not reasonable to assume that the API would need that feature (I would err on this side since neither YAML nor JSON are actually meant for configuration) then there’s two possibilities: a, that the design change is unnecessary and belongs in an unrelated project or b, that the design change should be made with no need for breaking the existing interface.

                                                  1.x.y in the semver nomenclature represents a notion of stability which I as the author get to define. It isn’t a universal notion of stability, it’s nothing more than what I define it to be. There is no single or objective denomination of “when I iron out all of the issues” which I can meet in order to qualify a 1.x.y version identifier. It’s all subjective!

                                                  The fact that it’s subjective and you can’t with perfect accuracy determine when it’s appropriate to make a 1.x.y release does not mean that lots of projects haven’t already made good-enough guesses on when this is appropriate and does not mean that a good enough guess is not an adequate substitute for perfect. Once again, you seem to suggest that just because it’s impossible to do something perfectly that it’s not worth doing it at all.

                                                  Semver provides all of the mechanics I need to signal what you demand. If I make a breaking change, I increment the major version number. That’s it! There’s nothing more to it.

                                                  Yeah, so do that. I don’t get where the disagreement lies here.

                                                  *As a completely unrelated side note, the fact that firefox binds ^W to close tab is a hideous design choice.

                                                  1. 2

                                                    you seem to suggest that just because it’s impossible to do something perfectly that it’s not worth doing it at all.

                                                    I am 100% for releasing 1.x.y versions of software projects. What I’m saying is merely that a subsequent release of 2.z.a, and afterwards 3.b.c, and then 4.d.e, is normal and good and not indicative of failure in any sense.

                                                    If it was reasonable to assume that the API would need that feature eventually then yes it’s a mistake.

                                                    It is incoherent to claim that this represents a mistake. If you can’t agree to that then there’s no possibility of progress in this conversation; if this is a mistake then you’re mandating perfect omniscient prescience in API design, which is plainly impossible.

                                                    1. 1

                                                      I am 100% for releasing 1.x.y versions of software projects. What I’m saying is merely that a subsequent release of 2.z.a, and afterwards 3.b.c, and then 4.d.e, is normal and good and not indicative of failure in any sense.

                                                      I would say that any interface which has need to change that often is too vaguely defined to be a real interface. Fundamentally rooted in the idea of an interface is the idea of being able to re-use it in multiple places, and the fundamental use in re-using something in multiple places is if adjustments to that thing benefit all users (otherwise why not just copy paste the code). As such, if your interface changes constantly, it does not provide any benefit as an interface and should not be one. Put another way, if every time I try to update your interface I have to change my code, what benefit is there at all in me even calling it a proper interface and not just maintaining my own version (either by periodically merging in new changes to the upstream version or just cherry-picking new features I like)?

                                                      It is incoherent to claim that this represents a mistake.

                                                      How so? Are you suggesting it is impossible to have design oversights?

                                                      If you can’t agree to that then there’s no possibility of progress in this conversation; if this is a mistake then you’re mandating perfect omniscient prescience in API design, which is plainly impossible.

                                                      Just because foresight is impossible, doesn’t mean you’ve not made a mistake. If foresight is impossible, maybe the mistake is attempting to foresee in the first place? In either case, I don’t understand why you think my stance on whether design mistakes are a real thing or not matters in a discussion about whether it’s appropriate for interfaces to be extremely unstable.

                                                      Moreover, given the vagueness of your example config parser case, I never actually said if the scenario you presented constituted a design mistake, I merely outlined a set of possibilities, one of which was that there was a design mistake.

                                                      So far (when it comes to your config parser example), I think that you’re being quite narrow-minded in terms of possible approaches to non-breaking interface changes which would avoid the problem entirely without sacrifices. I also think that the example you provided is extremely contrived and the initial design sounds like a bad idea to begin with (although once again, I don’t think this is the reason why you think there needs to be a breaking change, I think you think there needs to be a breaking change solely because I don’t think you have explored all the possibilities of how a non-breaking change may be implemented, but the example lacks a lot of detail so it’s difficult to understand your reasoning).

                                                      Maybe if you picked a more realistic design which didn’t have obvious problems to begin with and demonstrated a sensible design change you think would necessitate a breaking change (to avoid compromising on API quality) we could discuss how I would solve that and you can point out the disadvantages, according to you, of my approach to solving the breakage?

                                                      1. 2

                                                        if every time I try to update your interface I have to change my code, what benefit is there at all in me even calling it a proper interface

                                                        The benefit that my library provides to you as a consumer is mostly about the capabilities it offers for you, and only a little bit about the stability of its API over time. Updating the version of my library which you use is also an opt-in decision that you as a consumer make for yourself.

                                                        I also think that the example you provided is extremely contrived

                                                        It is adapted directly from a decision I had to make in my actual flag parsing library and chosen because I feel it is particularly exemplary of the kinds of design evolution I experience every day as a software author.

                                                        you can point out the disadvantages, according to you, of my approach to solving the breakage?

                                                        I don’t think we need an example. My position is that as long as you abide semver and bump the major version there is almost no net disadvantage to breaking changes for the vast majority of software produced today. And that avoiding breaking changes is a cost to coherence that rarely makes sense, except for a superminority of software modules. I have explained this as well as I think I can in a previous comment so if that’s not convincing then I think we can agree to disagree.

                                                        1. 1

                                                          The benefit that my library provides to you as a consumer is mostly about the capabilities it offers for you, and only a little bit about the stability of its API over time. Updating the version of my library which you use is also an opt-in decision that you as a consumer make for yourself.

                                                          You can call it a library or an interface all day long, I dispute you calling it that more than I dispute its usefulness. It might function and be useful but it does not function as an interface. It is really not an interface if it does not end up shared among components, if it is not shared among components then there is obviously no real problem (at least in terms of stability, since there’s an enormous number of problems in terms of general code complexity, over-abstraction, security issues and auditability issues).

                                                          It is adapted directly from a decision I had to make in my actual flag parsing library and chosen because I feel it is particularly exemplary of the kinds of design evolution I experience every day as a software author.

                                                          Can you point me to the breaking change?

                                                          My position is that as long as you abide semver and bump the major version there is almost no net disadvantage to breaking changes for the vast majority of software produced today.

                                                          And like I already explained, talking about “most software” is pointless since almost nobody uses it.

                                                          The discussion is about the small fraction of software which people actually use, where there is a clear benefit to avoiding random breaking changes.

                                                          And that avoiding breaking changes is a cost to coherence that rarely makes sense, except for a superminority of software modules.

                                                          A superminority which anything worth talking about is already part of.

                                                          I have explained this as well as I think I can in a previous comment so if that’s not convincing then I think we can agree to disagree.

                                                          There is nothing left to explain, it is not an issue of explanation, you are simply making a bunch of assertions including the assertion that, given an interface which people actually use, most normal design changes which change the interface are better* than design changes which preserve the interface. You have yet to provide any evidence of this (although hopefully if you send me the commit for your project we will finally have something concrete to discuss).

                                                          *better being defined as “the disadvantages of breaking the interface do not outweigh the disadvantages of lost clarity by expanding keeping the interface”

                                                          1. 2

                                                            I don’t care about “interfaces” — that’s terminology which you introduced. We’re talking about software modules, or packages, or libraries, with versioned APIs that are subject to the rules of semver.

                                                            And like I already explained, talking about “most software” is pointless since almost nobody uses it . . . The discussion is about the small fraction of software which people actually use

                                                            The discussion I’m having is about what software authors should be doing. So I’m concerned with the body of software produced irrespective of how many consumers a given piece of software has. If you’re only concerned with software that has enormously more consumers than producers than we are having entirely different discussions. I acknowledge this body of software exists but I call it almost irrelevant in the context of what programming language ecosystems (and their tools) should be concerned with.

                                                            1. 1

                                                              I don’t care about “interfaces” — that’s terminology which you introduced. We’re talking about software modules, or packages, or libraries, with versioned APIs that are subject to the rules of semver.

                                                              If you had an issue with my definition of interface then you should have raised it a bit earlier. That being said, I don’t think that you have a problem with the definition (since it seems completely compatible) but maybe the issue lies somewhere else.

                                                              The discussion I’m having is about what software authors should be doing. So I’m concerned with the body of software produced irrespective of how many consumers a given piece of software has. If you’re only concerned with software that has enormously more consumers than producers than we are having entirely different discussions. I acknowledge this body of software exists but I call it almost irrelevant in the context of what programming language ecosystems (and their tools) should be concerned with.

                                                              Programming ecosystems are interfaces with large numbers of users, their package handling tools exist to solely serve the packages which have a significant number of users. Making package handling tools work for the insignificant packages is a complete waste of time and benefits basically nobody.

                                                              Your package, like it or not, belongs in the group of packages I’m talking about.

                                                              Discussions about semver and interfaces and stability and versioning are obviously going to be completely irrelevant if nobody or almost nobody uses your package. More importantly, it’s very unproductive to use this majority of unused or barely used packages to inform design decisions of software tooling or to inform people how they should go about maintaining packages people actually use.

                                                              1. 2

                                                                [ecosystem] package handling tools exist to solely serve the packages which have a significant number of users

                                                                No, they exist to serve the needs of the entire ecosystem.

                                                                Your package, like it or not, [is insignificant]

                                                                I guess we’re done here.

                                                                1. 1

                                                                  No, they exist to serve the needs of the entire ecosystem.

                                                                  Focusing on the needs of wannabe libraries which nobody uses is nonsensical. Ecosystems focus on the needs of packages which people use and the people who use those packages.

                                                                  Your package, like it or not, [is insignificant] I guess we’re done here.

                                                                  Did you intentionally misrepresent me because you were bored of the discussion? Because I literally said the opposite.

                                                                  1. 2

                                                                    I apologize if you were saying that my package represents something with a large number of users. I parsed your sentence as the opposite meaning.

                                                                    Nevertheless, we’re clearly at an impasse. Your position is that

                                                                    package handling tools exist to solely serve the packages which have a significant number of users

                                                                    which is basically the antipode of my considered belief, and it doesn’t appear like you’re open to reconsideration. So I’m not sure there’s much point to continuing this already very long thread :)

                                                                    edit: I can maybe speculate at the underlying issue here, which is that you seem to be measuring relevant software by the number of consumers it has, whereas I’m measuring relevant software by it’s mere existence. So for you maybe an ecosystem with 1000 packages with 2 consumers each and 2 packages with 1000 consumers each is actually just 2 packages big, so to speak, and those 2 packages’ needs dictate the requirements of the tool; and for me it’s 1002 packages big and weighted accordingly. Does this make sense?

                                                                    1. 1

                                                                      I get your point, I don’t think there’s any misunderstanding there. I just can’t understand why when it comes to tools which are designed to facilitate interoperation you care about the packages for which … because almost nobody uses them … interoperation is not as important as the packages which lots of people use. I think the other problem is maybe that you’ve got a skewed idea of the relationship between package producers and users. I would say that it’s extremely likely (and I don’t have numbers to back this up but there’s no real reason why this doesn’t follow the Pareto distribution) that 20% of packages constitute 80% of package usage. These tools are designed to facilitate package use above package creation (since far fewer people will be creating packages rather than using them) therefore focusing on the 20% of packages which which constitute 80% of the usage would surely make sense?

                                                                      1. 2

                                                                        far fewer people will be creating packages rather than using them

                                                                        Just to make this explicit, I’m speaking from a context which includes open-source software (maybe 10-20% of all software produced in the world) and closed-source software written and maintained at market-driven organizations (maybe 80% of all software).

                                                                        In this context it’s my experience that essentially every programmer is both a consumer and producer of packages. Even if a package has only a single consumer it is still a member of the package ecosystem, and the needs of it’s (singular) author and consumer are relevant! So I don’t agree that far fewer people are creating packages than consuming them. In fact I struggle to summon a single example of someone who consumes but doesn’t produce packages.

                                                                        20% of packages constitute 80% of package usage

                                                                        I agree that the packages-vs-consumers curve is pretty exponential. I agree that tooling can and should support the needs of those relatively few packages with vastly more consumers than producers. But I don’t think it’s as extreme as you’re describing here. I think the “long tail” of packages consumed by a single-digit number of consumers represents the 80% of overall package consumption, the 80% of the area under the curve.

                                                                        1. 1

                                                                          In this context it’s my experience that essentially every programmer is both a consumer and producer of packages. Even if a package has only a single consumer it is still a member of the package ecosystem, and the needs of it’s (singular) author and consumer are relevant! So I don’t agree that far fewer people are creating packages than consuming them. In fact I struggle to summon a single example of someone who consumes but doesn’t produce packages.

                                                                          I think the term might be a bit loose here. I struggle to see a scenario where people produce as many packages as they consume. A lot of projects out there consume a lot of packages but only produce one.

                                                                          I agree that the packages-vs-consumers curve is pretty exponential. I agree that tooling can and should support the needs of those relatively few packages with vastly more consumers than producers. But I don’t think it’s as extreme as you’re describing here. I think the “long tail” of packages consumed by a single-digit number of consumers represents the 80% of overall package consumption, the 80% of the area under the curve.

                                                                          Okay, let’s just ignore the smaller packages for a moment, because as I said in my other comment, I don’t see how what I am trying to advocate for here affects those packages negatively in any way.

                                                                          Why is it remotely acceptable for the rust programming language to be both so completely unstable to the point that if you install it from the repositories of a non-rolling-release distro, there’s a good chance you won’t be able to build anything of even insignificant size?

                                                                          Why is it remotely acceptable for an average popular rust package to itself include up to hundreds of other tiny dependencies?

                                                                          Why are these not considered the horrible software practices that they are?

                                                                          1. 2

                                                                            Why are these not considered the horrible software practices that they are?

                                                                            Because “horrible” is a subjective judgment, not an objective metric.

                                                                            1. 1

                                                                              … and?

                                                                              Having the language be unstable makes it effectively unusable, you can’t write software against a moving target unless you want to live with the fact that if you stop maintaining it it will stop working. How is this beneficial?

                                                                              Having packages which include hundreds of packages makes things impossible to audit, makes security vulnerabilities more difficult to fix and makes maintenance more difficult.

                                                                              Are you happier with the above elaborations? I really wasn’t expecting to have to explain why these are horrible software practices.

                                                                              1. 1

                                                                                I understand and agree that a high rate of breaking changes is horrible for consumers like you, no need to elaborate there :)

                                                                                It’s just not true to say it’s horrible period, that it’s a horrible software practice in general. Consumers like you aren’t the only members of the software ecosystem; for many stakeholders, breaking changes provide large benefits. They count, too.

                                                                                I’m guessing you’re going to disagree with that, and say that what I’m describing as beneficial is actually just laziness or something. But I don’t think this is true, and I’m confident it’s not a productive line of reasoning. Regardless if it’s due to rational actions or character flaws or whatever else, breaking changes are the reality of software development in the large. Human beings can’t write C without memory bugs, and we can’t do software over time without breaking changes. Just the reality :)

                                                                                1. 1

                                                                                  for many stakeholders, breaking changes provide large benefits

                                                                                  We never really finished addressing this.

                                                                                  Like I said, I am of the opinion that breaking changes are not a given even when you’re trying to make things better. There are many ways to make an API cleaner for example without deprecating the old API immediately.

                                                                                  Human beings can’t write C without memory bugs, and we can’t do software over time without breaking changes. Just the reality :)

                                                                                  We have lots of enormous software projects which maintain backwards compatibility for decades.

                                                2. 2

                                                  I want to revisit two things here…

                                                  I don’t care about private spaces.

                                                  That’s fine! If you’re not interested in this category of software personally then that’s totally groovy. But “private spaces” hold the overwhelming majority of software produced and consumed in the world. Package management tooling for a general-purpose programming language which doesn’t prioritize the needs of these users doesn’t serve it’s purpose.

                                                  if your project is popular and you break stability expectations

                                                  When you say “stability expectations” do you mean a consumer’s expectation that I won’t violate the rules of semver? Or an expectation that I won’t hardly ever increment my major version number? Or something else?

                                                  As an example, go-github is currently on major version 41 — is this a problem?

                                                  1. 1

                                                    Okay, sorry for the long gap in replies, life became a bit busy before and around Christmas.

                                                    That’s fine! If you’re not interested in this category of software personally then that’s totally groovy. But “private spaces” hold the overwhelming majority of software produced and consumed in the world. Package management tooling for a general-purpose programming language which doesn’t prioritize the needs of these users doesn’t serve it’s purpose.

                                                    Let me rephrase. I don’t see how a open source software ecosystem in which my concept of stability expectations is de-facto and in which software tooling makes accommodations for these de-facto expectations (i.e., by having project metadata which allows specifying semver based version requirements, having dependency location tools which can make semver based choices and package managers which can make semver based choices) would in any way negatively impact or prevent private organizations from performing whatever they want to do (including version pinning things) in any way they want to.

                                                    When you say “stability expectations” do you mean a consumer’s expectation that I won’t violate the rules of semver? Or an expectation that I won’t hardly ever increment my major version number? Or something else?

                                                    No, I mean whatever expectations you set out in your project. I won’t depend on a project unless I can see just from reputation that the author: follows semver and doesn’t break the expectations of semver and doesn’t bump the major version very regularly for changes which could have quite easily been made in a non-breaking way. Also, for software at v2 and up I look to see if the prior versions have some explicitly documented information about long term support, at least in the form “security patches and major bug fixes will be backported for X time”.

                                                    As an example, go-github is currently on major version 41 — is this a problem?

                                                    Yes, because in the long term, if there’s security or general bugs in the library, when updating I would likely have to make changes to the rest of my software and keep re-learning random things about the library. Although I welcome improvements and will happily change my software to make use of new improved APIs, I don’t feel like it’s something I should have to worry about doing every time if I don’t want to. At the end of the day it’s also NOT a problem, as this project is clearly advertising that people like me should NOT use it.

                                                    1. 1

                                                      I don’t see how . . .

                                                      I think we’re in broad agreement, actually: I agree that all packages should strictly follow semver, and that tooling can and should leverage those versions as appropriate. The point of contention is around the rate-of-change of major versions.

                                                      Two facts: (1) a breaking change isn’t well-defined, and could include any change to the package whatsoever if you define API compatibility in terms of Hyrum’s Law; (2) the cost of a breaking change is different from project to project.

                                                      As a consumer you can of course decide what rate-of-change you’re comfortable with. But that’s the point: what constitutes “good” or “bad” rates of change is a subjective decision, not an objective truth. Tooling cannot make that decision for you.

                                                      In short,

                                                      go-github is currently on major version 41 — is this a problem?

                                                      Yes

                                                      For you, okay! Sure! But not for everyone. It’s not a problem for me.

                                                      1. 1

                                                        For you, okay! Sure! But not for everyone. It’s not a problem for me.

                                                        Why not? Genuinely, why is it not a problem for you? How do you justify the wasted time dealing with breaking changes to yourself?

                                                        1. 1

                                                          I don’t waste time with breaking changes. I don’t upgrade dependencies unless there is a specific need, and there is almost never a specific need. I think in the last 10 years, I could count on one hand the number of times I’ve upgraded my project deps for any reason other than I needed a new feature. And when I do upgrade my dependencies, I fully expect that it will require changes in my code, even on patch updates.

                                                          I don’t use dependencies that are subject to security vulnerabilities. I don’t use deps that require constant maintenance to remain functional. (shrug)

                                                          1. 1

                                                            I don’t use dependencies that are subject to security vulnerabilities.

                                                            That’s great but unless you’re only writing single player games then it seems incredibly difficult to be in the ideal situation you describe.

                                                            1. 1

                                                              For context, I’ve been writing Go for the last ~forever years, which has a comprehensive stdlib and discourages imports kind of philosophically. As a result, in that ecosystem, it’s actually super easy. My projects tend to have on the order of 10 dependencies, and they’re all pretty narrow in scope: something for ULIDs, something to handle flags better, etc.

                                                              But I’ve just started writing Rust in anger, and it’s definitely made me more sympathetic to the problem. Rust has a small stdlib as an explicit design goal, and that philosophy seems to be transitive throughout the ecosystem. It seems like it’s basically not possible to accomplish anything without a whole slurry third-party crates, which import their own huge set of crates, and so on. (Most of which are written by single individuals in their spare time, and many of which are abandoned — but that’s a separate discussion!)

                                  2. 2

                                    Bollocks. A minor breaking change is “re reversed the order of parameters on this method to match the other methods”. A major breaking change is “switched from callbacks to promises”

                                    1. 2

                                      A switch from callbacks to promises that comes without a compatibility interface should be reflected in the library name, not just the version. It’s not even the same library anymore if literally every line of code that is using the old version must be rewritten to use it again.

                                      1. 1

                                        I agree, but we’re in the minority, despite Rich Hickey’s efforts.

                                  1. 35

                                    Our sbase project, implementing the POSIX coreutils, makes cut -c behave as expected because it handles code points instead of bytes (and offers the flag -b for explicit byte-ranges).

                                    Still left to do is support for grapheme clusters. For that purpose I developed a super-easy-to-use and simple C99-library called libgrapheme (see grapheme.h and the manuals) that automatically parses the Unicode standard files to offer extended grapheme break point detection with all bells and whistles (full test-coverage, automatic test generation from Unicode, emoji support, etc.).

                                    Statically linked it only adds around 25K to a binary. Maybe it is useful to somebody, though I didn’t get around to write a README and make a release yet. Let me know if you use it, then I’ll tag a version 1 and set up a README and website sooner than planned! :)

                                    1. 2

                                      Wow, when I read “simple” I thought “impossible” but then you said it parses the unicode standard files I was impressed. This is an amazing achievement (if it works, which from a quick glance at the code it seems like it should).

                                      I think I would like to use it in a couple of projects (but please first focus on making sure the API will be stable, don’t rush to a 1.0 release just because some people want to use it).

                                    1. 4

                                      As it stands, Nim is well on its way to becoming as complex as Ada or C++. I guess the price for “one programming language for everything” is that you then have to satisfy everyone, which is only possible if the language becomes continually more extensive.

                                      1. 3

                                        That’s a fair critism, but I came to Nim from C++ because I thought most of the C++ complexity was ad-hoc and unjustified.

                                        1. 2

                                          It’s worth taking a look at how C++ came to be. Stroutrup added concepts to the existing C language that he knew from Simula and found useful. In this sense, “C with classes” was minimal and complete. Ada, on the other hand, started with the claim to support all applications of the DoD at that time with only one language (i.e. “one programming language for everything”); already the first version of Ada was accordingly large and complex. In the meantime, C++ has also reached an almost incomprehensible size and complexity. With C++ 11, a lot of “syntactic sugar” was introduced, i.e. things that could already be done, but perhaps somewhat less elegantly; this trend continued and the result we see in C++17 and 20; the price is an ever larger language scope. How much is enough (i.e., optimal) is a difficult question. At the moment I am trying to find an answer to this with http://oberon-lang.ch.

                                          1. 1

                                            I believe that Stroustroup planned for C++ to be multi-paradigm from the beginning, at least that’s the take I got from his book. C++ just happened to luck into the OO craze and I guess that paradigm became dominant.

                                            1. 2

                                              “multi-paradigm” is not the same as “one programming language for everything”. C++ was “multi-paradigm” by construction in that OO features were added to a procedural language without removing anything. But the new features were not just useful for OO, but also e.g. for better modularization and resource management.

                                            2. 1

                                              oberon+ looks neat (and reminds me of ada but presumably a lot less complex). But I can’t find any resources for learning it or any information about any standard libraries it has. Do the standard libraries use camelCase (as shown in the examples) this would also be a blocker for me.

                                              1. 2

                                                Do the standard libraries use camelCase (as shown in the examples) this would also be a blocker for me.

                                                honest question, just curious: why is casing so important for you that you basically ignore all other technical merits of a language?

                                                1. 1

                                                  I find it hard and aesthetically displeasing to read. (Also worth noting that languages which like to rely on camel case often pick the wrong side of XMLHttpRequest.) I find it hard to type on a keyboard. Given the choice of learning a language or not dealing with camelCase I simply pick not to learn the language. There are in fact so many languages out there that it is difficult to “miss out” on much by making such an arbitrary choice. There is likely at least one more language out there with mostly overlapping technical merits which does not force camelCase upon me.

                                                  That being said, I have recently thought about investigating using something like treesitter to basically place a thin layer over the top of a language which can translate snake_case to camelCase using a variety of rules (or even by also communicating with an LSP server) so that I can learn a language like Haskell comfortably.

                                                2. 1

                                                  Oberon+ is a union and extension of the existing Oberon 90, Oberon-2 and Oberon-07 dialects (see https://en.wikipedia.org/wiki/Oberon_(programming_language)). Historically there is no “standard library” for Oberon, but there is the Oberon System, which is a full operating system with a lot of modules also available for custom applications. There was an initiative to define Oberon System independent standard libraries (see http://www.edm2.com/index.php/The_Oakwood_Guidelines_for_Oberon-2_Compiler_Developers) which my compiler supports. But I will eventually implement my own standard libraries; up to then you can use any C library by the foreign function interface built into Oberon+; at http://oberon-lang.ch there is a language specification and some other articles; see also https://github.com/rochus-keller/Oberon.

                                            3. 2

                                              From what I can tell, the feeping creaturism is a hell of a lot more integrated than in other languages I’ve seen, and all of the language grammar in the above article is pretty well thought-out.

                                            1. 2

                                              The machine is the land-rover defender of the coffee-machine world.

                                              As someone with a La Pavoni Europiccola, I disagree :)

                                              It’s robust, and spare parts are available for everything inside. It’s repairable, which is an unfortunately rare thing in 2019.

                                              This is also a reason I went with a Pavoni. I’ve self-serviced mine multiple times, and the tools you need are basically a screwdriver and a soft mallet. The design of the machine hasn’t fundamentally changed in 70 years.

                                              The learning curve for producing great shots from it is pretty steep, but immensely rewarding when you nail it and can then pull amazing shots from, effectively, muscle memory.

                                              1. 3

                                                And the gaggia can be serviced with some allen keys and a screwdriver. I don’t get why you disagree. Yes, the La Pavoni has fewer moving parts (no pump, etc) but as you say, it’s much harder to master. I like to tinker with espresso but I wouldn’t want to spend a year learning how to use a machine, at some point you just get sick and tired of having bad shots and go back to immersion brewing or other percolation methods.

                                                1. 0

                                                  I believe you misunderstood, and also took what was (I thought) a pretty light-hearted jab at Gaggia from a pretty well-known rivalry from lever machine enthusiasts vs. well, non-lever machine enthusiasts.

                                                  I’m also very familiar with the Gaggia and what it can do, having personally refurbished one myself.

                                                  I cheekily disagreed that the Gaggia is the “land-rover defender” of the coffee world, in that it’s “robust, and spare parts are available for everything inside. It’s repairable, which is an unfortunately rare thing[…]”. The Gaggia is all of those things, but the LPE is more robust, and easier to repair.

                                                  I like to tinker with espresso but I wouldn’t want to spend a year learning how to use a machine

                                                  The irony of this comment in light of what the OP did to their Gaggia (which, kudos to OP, is awesome - love the post!) to get, and I quote, “much more consistently good coffee” is almost too much for me to take :p

                                                  1. 2

                                                    Installing a PID is relatively little effort compared to learning how to temp surf a La Pavonia or even how to use it properly.

                                              1. 2

                                                So I would say that far more helpful than any refactoring or renaming (especially of lsb to least_significant_bit or 1 to FIRST_BIT) would be to split out the implementation of this Galois LFSR into a struct called galois_lfsr and a function called galois_lfsr_step. Maybe even providing some macro for maximum period length N bit tap configurations. A function comment (as suggested by the book “The Practice of Programming” by Brian Kernighan and Rob Pike) stating: // galois_lfsr_step: Step a Galois Linear Feedback Shift Register would be the final cherry on top.

                                                Something like this:

                                                struct galois_lfsr {
                                                	unsigned long state;
                                                	unsigned long taps;
                                                };
                                                
                                                #define GALOIS_LFSR_MAX_PERIOD_TAPS_17_BITS 0x12000ul
                                                
                                                // galois_lfsr_step: Step a Galois Linear Feedback Shift Register
                                                unsigned long galois_lfsr_step(struct galois_lfsr *l)
                                                {
                                                	bool lsb;
                                                
                                                	assert(l != NULL);
                                                
                                                	lsb = l->state & 1;
                                                	l->state >>= 1;
                                                	if (lsb) l->state ^= l->taps;
                                                
                                                	return l->state;
                                                }
                                                

                                                Now you have a very specific function, which implements a very specific algorithm, you don’t have to worry about the comment becoming out of sync with the code as it would be quite an impressive level of ineptitude for someone to change this function’s logic rather than replacing it. The name of the function and the contents of the single comment would allow potential readers to quickly find a resource such as https://en.wikipedia.org/wiki/Linear-feedback_shift_register#Galois_LFSRs and would mean that rather than having to worry about explaining that if (lsb) rndval ^= TAPS; is actually xoring the taps with the output bit with zero comments, you can instead allow the reader to find a much more verbose and detailed resource on the topic themselves.

                                                Now you can also use this code in another function which can focus on conveying the fact that 2^8 is 256 (or the smallest power of 2 which covers all the Y coordinates) and that 2^9 is 512 (or the smallest power of 2 which covers all the X coordinates). e.g.:

                                                enum {
                                                	SCREEN_WIDTH = 320,
                                                	SCREEN_HEIGHT = 200,
                                                	SCREEN_WIDTH_BITS = 9,
                                                	SCREEN_HEIGHT_BITS = 8,
                                                };
                                                
                                                #define ULONG_BITS (sizeof (unsigned long) * CHAR_BIT)
                                                #define BITMASK(n) (ULONG_MAX >> (ULONG_BITS - (n)))
                                                
                                                #define SCREEN_WIDTH_MASK BITMASK(SCREEN_WIDTH_BITS)
                                                #define SCREEN_HEIGHT_MASK BITMASK(SCREEN_HEIGHT_BITS)
                                                
                                                // As a sidenote: typedefing the non-pointer type allows you to typecheck implementations of your callback like:
                                                // fizzle_pixel_func your_impl;
                                                // void your_impl(unsigned long x, unsigned long y, something /* whoops */ *context) { ... }
                                                // This typechecking will happen BEFORE you ever write the call to the function.
                                                // Not the most useful, but not totally worthless either.
                                                typedef void fizzle_pixel_func(unsigned long x, unsigned long y, void *context);
                                                void fizzlefade(fizzle_pixel_func *fizzle_pixel, void *context) {
                                                	static const unsigned long lfsr_start_state = 1;
                                                	struct galois_lfsr lfsr = {
                                                		lfsr_start_state,
                                                		GALOIS_LFSR_MAX_PERIOD_TAPS_17_BITS,
                                                	};
                                                	do {
                                                		unsigned long x, y;
                                                
                                                		x = (lfsr.state & SCREEN_WIDTH_MASK) - 1;
                                                		y = (lfsr.state >> SCREEN_WIDTH_BITS) & SCREEN_HEIGHT_MASK;
                                                
                                                		if (x < SCREEN_WIDTH && y < SCREEN_HEIGHT)
                                                			fizzle_pixel(x, y, context);
                                                
                                                		galois_lfsr_step(&lfsr);
                                                	} while(lfsr.state != lfsr_start_state);
                                                }
                                                

                                                With the above code the only real things requiring explanation / fixing are:

                                                1. SCREEN_{WIDTH,HEIGHT}_BITS are really log2(SCREEN_{WIDTH,HEIGHT}) except…

                                                2. SCREEN_WIDTH_BITS does not convey the fact that you actually need the log2 of SCREEN_WIDTH + 1 because…

                                                3. 17 bit maximum period taps cover all 2^17 values except for 0.

                                                4. I would avoid the callback approach, this should really be done in a similar style to the galois_lfsr struct and function, that you can actually use this code in a modern game to implement a fizzle effect rather than just having the entire screen fizzle in one frame.

                                                5. By doing it the non-callback way, you can create an initialiser function which solves problem #1 for you and avoids having to maintain the values manually. Alternatively you could calculate those values at compile time which is not a bad idea since generating a header file should be trivial.

                                                6. Since division is no longer the most expensive operation in the world, using % and / you can make the code a lot simpler. e.g.

                                                  do {
                                                  	unsigned long x, y, pixel;
                                                  
                                                  	pixel = lfsr.state - 1;
                                                  
                                                  	x = pixel % SCREEN_WIDTH;
                                                  	y = pixel / SCREEN_WIDTH;
                                                  
                                                  	if (y < SCREEN_HEIGHT)
                                                  		fizzle_pixel(x, y, context);
                                                  
                                                  	galois_lfsr_step(&lfsr);
                                                  } while(lfsr.state != lfsr_start_state);
                                                  
                                                1. 1

                                                  How does this sort of thing compare with formal verification? I’m not entirely well versed in formal verification. Is this the same thing?

                                                  1. 1

                                                    Well, it is a form of lightweight formal verification. Its not full static verification because, in most cases, it only checks a program is correct up to some bound (e.g. arrays of most size 3 in my example). Therefore, its possible that there are bugs exposed only for larger arrays. In contrast, full formal verification checks for all possible sizes (but is actually quite a lot harder to do).

                                                  1. 4

                                                    The copious use of interactive examples was excellent. The only think I would like to know now is how you would go about doing this in a GLSL shader.

                                                    1. 4

                                                      It might be because I spend a lot of time dealing with serialisation and deserialisation in C but I spotted this bug immediately. It’s surprising nobody else noticed this in 6 years. It’s also possible that this wasn’t noticed because most people who use GLFW will be using it for games which run full screen (and therefore setting an icon seems moot) but I’m not sure.

                                                      1. 1

                                                        Nobody experienced the bug though. The generated code without ubsan seems to do what you’d expect.

                                                        1. 2

                                                          What I mean is that for a seasoned C language lawyer, any instance of (foo << n) | (bar << m) is immediately alarming. I was surprised no language lawyer (and I don’t even think I would call myself a C language lawyer yet) had read the code until now since this jumped out at me as soon as I saw the first code snippet.

                                                      1. 3

                                                        FYI casting the return value of malloc in C is not necessary and not advised. It can easily hide bugs if you happen to be compiling with a compiler which allows implicit function declarations. Really this kind of issue only affects beginners so I would especially avoid writing such code in any instructive material.

                                                        1. 2

                                                          Wow, I went on a long read of the thread and found this piece of gold:

                                                          On Fri, Aug 26, 2005 at 11:21:06AM -0400, Keith Moore wrote:

                                                          I am perhaps just being slow and dim-witted after minor surgery, but why should a protocol that no-one will use be standards track ?

                                                          Why should we accept a few (mostly axe-grinding) peoples’ assertions that no-one will use it?

                                                          Keith

                                                          we shouldn’t. LLMNR has waded through the lengthy IETF standardization process to get to where it is. That Microsoft has been patient and spent the money needed to keep people on this task long enough to get it here should be rewarded with the IETF imprinture. Of course even Microsoft has hedged its bets (even they are aware of the need to ship products) wrt LLMNR. But that is no reason for the IETF to not sanction this work.

                                                          –bill

                                                          Unsurprisingly this got called out. But it was clear some people were motivated by strong emotions during this discussion.

                                                          1. 5

                                                            Complaining about ABI stability causing language stagnation and design issues and then complaining about a lack of ABI stability in BSDs (because they don’t want to be held back) forcing the use of dynamic linking is a little interesting.

                                                            And default statically-linked binaries is one of the reasons the Go language became popular. I don’t think that’s a coincidence.

                                                            An appeal to popularity.

                                                            Other than that, the possible solutions section at the end FINALLY addresses actual problems I have always had with statically linking everything with very practical and simple solutions. I applaud this. I’ve read all three related articles now (this one, Dynamic Linking Needs To Die and Static Linking Considered Harmful) and this really is the best of all three.

                                                            I would note though that this doesn’t propose a solution for my most annoying problem with static linking in the C world: namespace pollution. My current solution is to give all functions, that are not part of the API, macros which suffix/prefix their name with something random to minimize the possibility of collisions.

                                                            1. 1

                                                              Complaining about ABI stability causing language stagnation and design issues and then complaining about a lack of ABI stability in BSDs (because they don’t want to be held back) forcing the use of dynamic linking is a little interesting.

                                                              They’re different types of ABI’s. But then again, I also present ideas to get around the problems with dynamic linking, allowing BSD’s to have the lack of ABI stability. At that point, I’d be okay with them not having ABI stability too.

                                                              An appeal to popularity.

                                                              I understand your sentiment, but I used it because some people want their software to be popular. I’m just saying that this might be something that helps. I want my programming language to be popular, so this appeal to popularity works on me, at least.

                                                              But I also had an implied argument there: that static linking is easiest for downstream developers. It’s implied because people usually gravitate to what’s easiest.

                                                              Other than that, the possible solutions section at the end FINALLY addresses actual problems I have always had with statically linking everything with very practical and simple solutions.

                                                              I apologize that it took so long to get there; that’s how it flowed in my mind. Should I have put them first?

                                                              I applaud this. I’ve read all three related articles now (this one, Dynamic Linking Needs To Die and Static Linking Considered Harmful) and this really is the best of all three.

                                                              Thank you. :) And I apologize for “Dynamic Linking Needs to Die”.

                                                              I would note though that this doesn’t propose a solution for my most annoying problem with static linking in the C world: namespace pollution.

                                                              I would like to talk to you more about this because my programming language uses C-like names and could suffer from namespace pollution. It’s helped somewhat because prefixes are automatically added for items in packages (i.e., for the function foo() in the package bar, its C name is bar_foo()), but I have not been able to ensure it avoids all of the problems.

                                                              My current solution is to give all functions, that are not part of the API, macros which suffix/prefix their name with something random to minimize the possibility of collisions.

                                                              Is this automatic? Should it be automatic?

                                                              I kind of think I know what you’re going at here, and I think it’s about the same thing that gcc and other C compilers do for functions with static visibility. So in my language, any functions and types private to a package would be treated as though they have static visibility.

                                                              I think that would be sufficient because I think the problem C has is that something does not have static visibility unless you mark it so, leading to a lot of functions that should be static polluting the namespace.

                                                              I’m not entirely sure, so what do you think?

                                                              1. 4

                                                                But I also had an implied argument there: that static linking is easiest for downstream developers. It’s implied because people usually gravitate to what’s easiest.

                                                                There was an article on here or the other website recently about people’s focus on what is easiest for developers. I personally think that we should be looking at what’s best for users not what’s easiest for developers.

                                                                I apologize that it took so long to get there; that’s how it flowed in my mind. Should I have put them first?

                                                                No, I just meant I read dozens of these “dynamic linking bad” articles which complain about dynamic linking and tout the benefits of static linking as if there were literally no benefits to dynamic linking (usually after claiming that because 90% of libraries aren’t shared as much as 10% that means we should ditch 100% of dynamic linking). This was also the vibe of your original article (and a similar vibe of the “static linking considered harmful” article except in the opposite direction).

                                                                The problem is that while yes, on the face of it you could say that static linking as things currently stand is less of a shit show, nobody seemed to acknowledge that it might be worth trying to solve the problem in a way which keeps some of the key benefits of dynamic linking. Your article is the first I’ve read so far which actually took this into consideration. That’s why I said I finally have read an article which covers these things.

                                                                Is this automatic? Should it be automatic?

                                                                It’s not so easy to automate these things portably. I have not looked into automating it but at the same time I almost never write libraries (my opinion is on the other extreme from npm proponents, don’t write a library unless you’ve already written similar code in a number of places and know what the API should look like from real experience).

                                                                I kind of think I know what you’re going at here, and I think it’s about the same thing that gcc and other C compilers do for functions with static visibility. So in my language, any functions and types private to a package would be treated as though they have static visibility.

                                                                I think that would be sufficient because I think the problem C has is that something does not have static visibility unless you mark it so, leading to a lot of functions that should be static polluting the namespace.

                                                                I’m not entirely sure, so what do you think?

                                                                Fundamentally the problem is that in C and C++ to get a static library on linux you compile your code into object files and then you put them into an weird crusty old archive format. At the end of the day, unlike with a shared object, you can’t mark symbols as being local to the library since when it comes to linking, it’s as if you had directly added the list of object files to the linking command.

                                                                Let me give you an example:

                                                                ==> build <==
                                                                #!/bin/sh -ex
                                                                cc -std=c11 -c lib.c
                                                                cc -std=c11 -c internal.c
                                                                ar rc lib.a lib.o internal.o
                                                                ranlib lib.a
                                                                cc -shared lib.o internal.o -o liblib.so
                                                                cc -std=c11 -I. -c prog.c
                                                                cc prog.o lib.a -o prog_static
                                                                cc prog.o -L. -llib -Wl,-rpath=. -o prog_dynamic
                                                                
                                                                ==> clean <==
                                                                #!/bin/sh
                                                                rm -f *.o *.a *.so prog_*
                                                                
                                                                ==> internal.c <==
                                                                #include "internal.h"
                                                                #include <stdio.h>
                                                                void print(const char *message)
                                                                {
                                                                	puts(message);
                                                                }
                                                                
                                                                ==> internal.h <==
                                                                #ifndef INTERNAL_H
                                                                #define INTERNAL_H
                                                                __attribute__ ((visibility ("hidden")))
                                                                void print(const char *message);
                                                                #endif
                                                                
                                                                ==> lib.c <==
                                                                #include "lib.h"
                                                                #include "internal.h"
                                                                __attribute__ ((visibility ("default")))
                                                                void api(void)
                                                                {
                                                                	print("api");
                                                                }
                                                                
                                                                ==> lib.h <==
                                                                #ifndef LIB_H
                                                                #define LIB_H
                                                                __attribute__ ((visibility ("default")))
                                                                void api(void);
                                                                #endif
                                                                
                                                                ==> prog.c <==
                                                                #include <lib.h>
                                                                int main(void)
                                                                {
                                                                	api();
                                                                	extern void print(const char *message);
                                                                	print("main");
                                                                }
                                                                

                                                                In the above example, lib.h exposes an API, this consists of the function api. internal.h is internal to the library and exposes a function print. The visibility has been marked but from the point of view of the linker, linking prog_static from prog.o and lib.a is equivalent to linking prog.o, lib.o and internal.o. This means the linking succeeds and print can be called from main. In the library case, the linking step happens first, the visibilities are honored and once you try to link to the library the print function is no longer exposed.

                                                                There is no way to hide the print function in this case. And given how internal functions usually don’t have namespaced names, they have a higher chance of colliding. The solution would be to replace internal.h with the following:

                                                                #ifndef INTERNAL_H
                                                                #define INTERNAL_H
                                                                #define print lib_internal_print
                                                                __attribute__ ((visibility ("hidden")))
                                                                void print(const char *message);
                                                                #endif
                                                                

                                                                (Or something similar like #define print print_3b48214e)

                                                                This would namespace the function without having the entire codebase end up having to call the function by the long name.

                                                                But this is all hacks, and doesn’t solve the visibility problem.

                                                                I hope you see what I mean now.

                                                                1. 1

                                                                  There was an article on here or the other website recently about people’s focus on what is easiest for developers. I personally think that we should be looking at what’s best for users not what’s easiest for developers.

                                                                  I agree, but I think making certain things easier on developers would reduce mistakes, which would make things easier on users. So whenever I can reduce mistakes by developers by making things easier, I do so.

                                                                  No, I just meant I read dozens of these “dynamic linking bad” articles which complain about dynamic linking and tout the benefits of static linking as if there were literally no benefits to dynamic linking (usually after claiming that because 90% of libraries aren’t shared as much as 10% that means we should ditch 100% of dynamic linking). This was also the vibe of your original article (and a similar vibe of the “static linking considered harmful” article except in the opposite direction).

                                                                  Oh, I see. Yeah, sorry again about the first one. It was unprofessional.

                                                                  It’s not so easy to automate these things portably. I have not looked into automating it but at the same time I almost never write libraries (my opinion is on the other extreme from npm proponents, don’t write a library unless you’ve already written similar code in a number of places and know what the API should look like from real experience).

                                                                  I think this is a great guideline.

                                                                  Regarding the code, yes, I think I see what you mean.

                                                                  As much as it pains me, I think this means that I am going to have to “break” ABI with C enough to make such private functions not visible to outside code. (Maybe I’ll add a suffix to them, and the suffix will be the compiler-internal ID of their scope?) But that means that I can make it automatic.

                                                                  However, I can probably only do this in the compiler for my programming language, not for C itself.

                                                                  1. 2

                                                                    My recommendation is: make your own static linking format. But if you want static linking interop with C then there’s nothing stopping you from exposing that as a single object file (the issue comes from having multiple object files in the archive).

                                                            1. 5

                                                              This is an interesting article, but some things that stood out to me:

                                                              • I’d have liked to see the article engage more with the libc case. The article points out several arguments against statically linking libc, specifically - it’s the only supported interface to the OS on most *nix-ish systems that are not Linux, it’s a very widely used library (so disk/memory savings can be substantial), and as part of the system it should be quite stable. Also, Go eventually went back to (dynamically) linking against libc rather than directly making syscalls [EDIT: on many non-Linux systems]
                                                              • LLVM IR is platform-specific and not portable. And existing C code using e.g. #ifdef __BIG_ENDIAN__ (or using any system header using such a construction!) is not trivial to compile into platform-independent code. Yes, you can imagine compiling with a whole host of cross-compilers (and system headers!), but at some point just shipping the source code begins looking rather attractive…
                                                              • exploiting memory bugs is a deep topic, but the article is a bit too simplistic in its treatment of stack smashing. There’s a lot to dislike about ASLR, but ASLR is at least somewhat effective against - to think up a quick example - a dangling pointer in a heap-allocated object being use(-after-free)d to overwrite a function / vtable pointer on the stack.
                                                              • in general, there’s a lot of prior art that could be discussed; e.g. I believe that Windows randomizes (randomized?) a library’s address system-wide, rather than per-process.
                                                              1. 3

                                                                And existing C code using e.g. #ifdef __BIG_ENDIAN__ (or using any system header using such a construction!) is not trivial to compile into platform-independent code.

                                                                The article addresses this.

                                                                I would also note as an aside that any code utilising #ifdef __BIG_ENDIAN__ is just plain wrong. Yes, even /usr/include/linux/tcp.h. Just don’t do that. Write the code properly.

                                                                1. 2

                                                                  I’ll bite. I have written a MC6809 emulator that makes the following assumptions of the host system:

                                                                  • A machine with 8-bit chars (in that it does support uint8_t)

                                                                  • A 2’s complement architecture

                                                                  I also do the whole #ifdef __BIG_ENDIAN__ (effectively), check out the header file. How would you modify the code? It’s written that way to a) make it easier on me to understand the code and b) make it a bit more performant.

                                                                  1. 3

                                                                    I would write inline functions which would look like:

                                                                    static inline mc6809byte__t msb(mc6809word__t word) { return (word >> 8) & 0xff; }
                                                                    static inline mc6809byte__t lsb(mc6809word__t word) { return word & 0xff; }
                                                                    

                                                                    And I would use those in place of the current macro trick of replacing something->A with something->d.b[MSB] etc.

                                                                    I don’t think this would significantly impact readability. Clang seems to produce identical code for both cases, although from a benchmark there’s still some minor (10% if your entire workload is just getting data out of the MSB and LSB) performance impact although this may be some issue with my benchmark. gcc seems to struggle to realize that they are equivalent and keeps the shr but the performance impact is only 13%.

                                                                    If you need to support writing to the MSB and LSB you would need a couple more inline functions:

                                                                    static inline set_msb(mc6809word__t *word, mc6809byte__t msb) { *word = lsb(*word) | ((msb & 0xff) << 8); }
                                                                    static inline set_lsb(mc6809word__t *word, mc6809byte__t lsb) { *word = (msb(*word) << 8) | (lsb & 0xff); }
                                                                    

                                                                    I haven’t benchmarked these.

                                                                    I think the point I would make is that you should benchmark your code against these and see whether there is a real noticeable performance impact moving from your version to this version. Through this simple change you can make steps to drop your assumption of CHAR_BIT == 8 and your code no longer relies on type punning which may or may not produce the results you expect depending on what machine you end up on. Even though your current code is not doing any in-place byte swapping, you still risk trap representations.

                                                                    P.S. *_t is reserved for type names by POSIX.

                                                                    1. 2

                                                                      I would definitely have to benchmark the code, but I can’t see it being better than what I have unless there exists a really magical C compiler that can see through the shifting/masking and replace them with just byte read/writes (which is what I have now). Your set_lsb() function is effectively:

                                                                      *word = ((*word >> 8) & 0xff) << 8 | lsb & 0xff;
                                                                      

                                                                      A 10-13% reduction in performance seems a bit steep to me.

                                                                      you still risk trap representations.

                                                                      I have to ask—do you know of any computer sold new today that isn’t byte oriented and 2’s complement? Or hell, any machine sold today that actually has a trap representation? Because I’ve been programming for over 35 years now, and I have yet to come across one machine that a) isn’t byte oriented; b) 2’s complement; c) has trap representations. Not once. So I would love to know of any actual, physical machines sold new that breaks one of these assumptions. I know they exist, but I don’t know of any that have been produced since the late 60s.

                                                                      1. 2

                                                                        a really magical C compiler that can see through the shifting/masking and replace them with just byte read/writes

                                                                        Yes, that’s what clang did when I tested it on godbolt. In fact I can get it to do it in all situations by swapping the order of the masking and the shifting.

                                                                        Here’s the result:

                                                                                                output[i].lower = in & 0xff;
                                                                          4011c1:       88 54 4d 00             mov    %dl,0x0(%rbp,%rcx,2)
                                                                                                output[i].upper = (in & 0xff00) >> 8;
                                                                          4011c5:       88 74 4d 01             mov    %dh,0x1(%rbp,%rcx,2)
                                                                        

                                                                        You underestimate the power of compilers, although I’m not sure why gcc can’t do it, it’s really a trivial optimisation all things considered.

                                                                        I just checked further and it seems the only reason that the clang compiled mask&shift variant performs differently is because of different amounts of loop unrolling and also because the mask&shift code uses the high and low registers instead of multiple movbs. The godbolt code didn’t use movbs, it was identical for both cases in clang.

                                                                        My point being that in reality you may get 10% (absolute worst case) difference in performance just because the compiler felt like it that day.

                                                                        I have to ask—do you know of any computer sold new today that isn’t byte oriented and 2’s complement? Or hell, any machine sold today that actually has a trap representation?

                                                                        I don’t personally keep track of the existence of such machines.

                                                                        For me it’s not about “any machine” questions, it’s about sticking to the C abstract machine until there is a genuine need to stray outside of that, a maybe 13% at absolute unrealistic best performance improvement is not worth straying outside the definitions of the C abstract machine.

                                                                        In general I have found this to produce code with fewer subtle bugs. To write code which conforms to the C abstract machine you just have to know exactly what is well defined. To write code which goes past the C abstract machine you have to know with absolute certainty about all the things which are not well defined.

                                                                        edit: It gets worse. I just did some benchmarking and I can get swings of +-30% performance by disabling loop unrolling. I can get both benchmarks to perform the same by enabling and disabling optimization options.

                                                                        This is a tight loop doing millions of the same operation. Your codebase a lot more variation than that. It seems more likely you’ll get a 10% performance hit/improvement by screwing around with optimisation than you will by making the code simply more correct.

                                                                2. 2

                                                                  Wait, Go dynamically links to libc now? Do you have more details? I thought Go binaries have zero dependencies and only the use of something like CGO would change that.

                                                                  1. 15

                                                                    As the person responsible for making Go able to link to system libraries in the first places (on illumos/Solaris, others later used this technology for OpenBSD, AIX and other systems), I am baffled why people have trouble understanding this.

                                                                    Go binaries, just like any other userspace binary depend at least on the operating system. “Zero dependency” means binaries don’t require other dependencies other than the system itself. It doesn’t mean that the dependency to the system cannot use dynamic linking.

                                                                    On systems where “the system” is defined by libc or its equivalent shared library, like Solaris, Windows, OpenBSD, possibly others, the fact that Go binares are dynamically linked with the system libraries doesn’t make them not “zero dependency”. The system libraries are provided by the system!

                                                                    Also note that on systems that use a shared library interface, Go doesn’t require the presence of the target shared library at build time, only compiled binaries require it at run time. Cross-compiling works without having to have access to target system libraries. In other words, all this is an implementation detail with no visible effect to Go users, but somehow many Go users think this is some kind of a problem. It’s not.

                                                                    1. 3

                                                                      I (clarified) was thinking about e.g. OpenBSD, not Linux; see e.g. cks’ article.

                                                                      1. 2

                                                                        Under some circumstances Go will still link to libc. The net and os packages both use libc for some calls, but have fallbacks that are less functional when CGO is disabled.

                                                                        1. 6

                                                                          The way this is usually explained is a little bit backwards. On Linux, things like non-DNS name resolution (LDAP, etc), are under the purview of glibc (not any other libc!) with its NSCD protocol and glibc-specific NSS shared libraries.

                                                                          Of course that if you want to use glibc-specific NSS, you have to to link to glibc, and of course that if you elect not to link with glibc you don’t get NSS support.

                                                                          Most explanations of Go’s behavior are of the kind “Go is doing something weird”, while the real weirdness is that in Linux name resolution is not something under of the purview of the system, but of a 3rd party component and people accept this sad state of affairs.

                                                                          1. 3

                                                                            How is glibc a 3rd party component on, say, Debian? Or is every core component 3rd party since Debian does not develop any of them?

                                                                            1. 4

                                                                              Glibc is a 3rd party component because it is not developed by the first party, which is the Linux developers.

                                                                              Glibc sure likes to pretend it’s first party though. It’s not, a fact simply attested by the fact other Linux libc libraries exists, like musl.

                                                                              Contrast that with the BSDs, or Solaris, or Windows, where libc (or its equivalent) is a first party component developed by BSDs, Solaris, or Windows developers.

                                                                              I would hope that Debian Linux would be a particular instance of a Linux system, rather than an abstract system itself, and I could use “Linux software” on it but glibc’s portability quirks and ambitions of pretending to be a 1st party system prevent one from doing exactly that.

                                                                              Even if you think that Linux+glibc should be an abstract system in itself, distrinct from, say, pure Linux, or Linux+musl, irrespective of the pain that would instil on me, the language developer, glibc is unfeasible as an abstract interface because it is not abstract.

                                                                              1. 3

                                                                                Wait, how are the Linux developers “first party” to anything but the kernel?

                                                                                I would hope that Debian Linux would be a particular instance of a Linux system

                                                                                There’s no such thing as “a Linux system” only “a system that uses Linux as a component”. Debian is a system comparable to FreeBSD, so is RHEL. Now some OS are specifically derived from others, so you might call Ubuntu “a Debian system” and then complain that snap is an incompatible bolt on or something (just an example, not trying to start an argument about snap).

                                                                                1. 5

                                                                                  Of course there is such a thing as a Linux system, you can download it from kernel.org, it comes with a stable API and ABI, and usually, in fact with the exception of NSS above, absolutely always does exactly what you want from it.

                                                                                  Distributions might provide some kind of value for users, and because they provide value they overestimate their technical importance with silly statements like “there is no Linux system, just distributions”, no doubt this kind of statement comes from GNU itself, with its GNU+Linux stance, but from a language designer none of this matters at all. All that matter are APIs and ABIs and who provides them. On every normal system, the developer of the system dictates and provides its API and ABI, and in the case of Linux that’s not different, Linux comes with its stable API and ABI and as a user of Linux I can use it, thank you very much. The fact that on Linux this ABI comes though system calls, while on say, Solaris, comes from a shared library is an implementation detail. Glibc, a 3rd party component comes with an alternative API and ABI, and for whatever reason some people think that is more canonical than the first party API and ABI provided by the kernel itself. The audacity of glibc developers claiming authority over such a thing is unbelievable.

                                                                                  As a language developer, and in more general as an engineer, I work with defined systems. A system is whatever has an API and an ABI, not some fuzzy notion defined by some social organization like a distribution, or a hostile organization like GNU.

                                                                                  As an API, glibc is valuable (but so is musl), as an ABI glibc has negative value both for the language developer and for its users. The fact that in Go we can ignore glibc, means not only freedom from distributions’ and glibc’s ABI quirks, but it also means I can have systems with absolutely no libc at. Just a Linux kernel and Go binaries, a fact that plenty of embedded people make use of.

                                                                                  1. 2

                                                                                    Of course there is such a thing as a Linux system, you can download it from kernel.org,

                                                                                    So is libgpiod part of the system too? You can download that from kernel.org as well. You can even download glibc there.

                                                                                    it comes with a stable API and ABI, and usually, in fact with the exception of NSS above, absolutely always does exactly what you want from it.

                                                                                    Unless you want to do something other than boot :)

                                                                      2. 2

                                                                        I’d have liked to see the article engage more with the libc case.

                                                                        Fair, though I feel like I engaged with it a lot. I even came up with ideas that would make it so OS authors can keep their dynamically-linked libc while not causing problems with ABI and API breaks.

                                                                        What would you have liked me to add? I’m not really sure.

                                                                        LLVM IR is platform-specific and not portable.

                                                                        Agreed. I am actually working on an LLVM IR-like alternative that is portable.

                                                                        And existing C code using e.g. #ifdef BIG_ENDIAN (or using any system header using such a construction!) is not trivial to compile into platform-independent code.

                                                                        This is a good point, but I think it can be done. I am going to do my best to get it done in my LLVM alternative.

                                                                        the article is a bit too simplistic in its treatment of stack smashing.

                                                                        Fair; it wasn’t the big point of the post.

                                                                        There’s a lot to dislike about ASLR, but ASLR is at least somewhat effective against - to think up a quick example - a dangling pointer in a heap-allocated object being use(-after-free)d to overwrite a function / vtable pointer on the stack.

                                                                        I am not sure what your example is. Could you walk me through it?

                                                                        1. 3

                                                                          Agreed. I am actually working on an LLVM IR-like alternative that is portable.

                                                                          This is not possible for C/C++ without either:

                                                                          • Significantly rearchitecting the front end, or
                                                                          • Effectively defining a new ABI that is distinct from the target platform’s ABI.

                                                                          The second of these is possible with LLVM today. This is what pNaCl did, for example. If you want to target the platform ABI then the first is required because C code is not portable after the preprocessor has run. The article mentions __BIG_ENDIAN__ but that’s actually a pretty unusual corner case. It’s far more common to see things that conditionally compile based on pointer size - UEFI bytecode tried to abstract over this and attempts to make Clang and GCC target it have been made several times and failed and that was with a set of headers written to be portable. C has a notion of an integer constant expression that, for various reasons, must be evaluated in the front end. You can make this symbolic in your IR, but existing front ends don’t.

                                                                          The same is true of C++ templates, where it’s easy to instantiate a template with T = long as the template parameter and then define other types based on sizeof(T) and things like if constexpr (sizeof(T) < sizeof(int)), at which point you need to preserve the entire AST. SFINAE introduces even more corner cases where you actually need to preserve the entire AST and redo template instantiation for each target (which may fail on some platforms).

                                                                          For languages that are designed to hide ABI details, it’s easy (see: Java or CLR bytecode).

                                                                          1. 2

                                                                            I believe you are correct, but I’ll point out that I am attempting to accomplish both of the points you said would need to happen, mostly with a C preprocessor with some tricks up its sleeve.

                                                                            Regarding C++ templates, I’m not even going to try.

                                                                            I would love to disregard C and C++ entirely while building my programming language, but I am not disregarding C at least because my compiler will generate C. This will make my language usable in the embedded space (there will be a -nostd compiler flag equivalent), and it will allow the compiler to generate its own C source code, making bootstrap easy and fast because unlike Rust or Haskell, where you have to follow the bootstrap chain from the beginning, my compiler will ship with its own C source, making bootstrap as simple as:

                                                                            1. Compile C source.
                                                                            2. Compile Yao source. (Yao is the name of the language.)
                                                                            3. Recompile Yao source.
                                                                            4. Ensure the output of 2 and 3 match.

                                                                            With it that easy, I hope that packagers will find it easy enough to do in their packages.

                                                                            1. 2

                                                                              If your input is a language that doesn’t expose any ABI-specific details and your output is C (or C++) code that includes platform-specific things, then this is eminently tractable.

                                                                              This is basically what Squeak does. The core VM is written in a subset of Smalltalk that can be statically compiled to C. The C code can then be compiled with your platform’s favourite C compiler. The rest of the code is all bytecode that is executed by the interpreter (or JIT with Pharo).

                                                                          2. 3

                                                                            Agreed. I am actually working on an LLVM IR-like alternative that is portable.

                                                                            If you’re looking for prior art, Tendra Distribution Format was an earlier attempt at a UNCOL for C.

                                                                            1. 1

                                                                              Thank you for the reference!

                                                                            2. 3

                                                                              With respect to libc - indeed, the article engaged quite a bit with libc! That’s why I was waiting for a clear conclusion.

                                                                              E.g. cosmopolitan clearly picks “all the world’s a x86_64, but may be running different OSes”, musl clearly picks “all the world’s Linux, but may be running on different architectures”. You flirt with both “all the world’s Linux” and something like Go-on-OpenBSD’s static-except-libc. Which are both fine enough.

                                                                              With respect to ASLR: I agree that this isn’t the main point of your article, and I don’t think I explained what I meant very well. Here’s some example code, where the data from main() is meant to represent hostile input; I mean to point out that merely segregating arrays and other data / data and code doesn’t fix e.g. lifetime issues, and that ASLR at least makes the resulting bug a bit harder to exploit (because the adversary has to guess the address of system()). A cleaned-up example can be found below, (actually-)Works-For-Me code here.

                                                                              static void buggy(void *user_input) {
                                                                                  uintptr_t on_stack_for_now;
                                                                                  /* Bug here: on_stack_for_now doesn't live long enough! */
                                                                                  scheduler_enqueue(write_what_where, user_input, &on_stack_for_now);
                                                                              }
                                                                              
                                                                              static void victim(const char *user_input) {
                                                                                  void (*function_pointer)() = print_args;
                                                                              
                                                                                  if (scheduler_run() != 0)
                                                                                      abort();
                                                                              
                                                                                  function_pointer(user_input);
                                                                              }
                                                                              
                                                                              int main(void) {
                                                                                  buggy((void *)system);
                                                                                  victim("/bin/sh");
                                                                              }
                                                                              
                                                                              1. 2

                                                                                With respect to libc - indeed, the article engaged quite a bit with libc! That’s why I was waiting for a clear conclusion.

                                                                                I see. Do you mean a conclusion as to whether to statically link libc or dynamically link it?

                                                                                I don’t think there is a conclusion, just a reality. Platforms require programmers to dynamically link libc, and there’s not much we can do to get around that, though I do like the fact that glibc and musl together give us the choice on Linux.

                                                                                However, I think the conclusion you are looking for might be that it does not matter because both suck with regards to libc!

                                                                                If you statically link on platforms without a stable syscall ABI, good luck! You can probably only make it work on machines with the same OS version.

                                                                                If you dynamically link, you’re probably going to face ABI breaks eventually.

                                                                                So to me, the conclusion is that the new ideas I gave are necessary to make working with libc easier on programmers. Right now, it sucks; with my ideas, it wouldn’t (I hope).

                                                                                Does that help? Sorry that I didn’t make it more clear in the post.

                                                                                Regarding your code, I think I get it now. You have a point. I tried to be nuanced, but I should not have been.

                                                                                I am actually developing a language where lifetimes are taken into account while also having separated stacks. I hope that doing both will either eliminate the possibility of such attacks or make them infeasible.

                                                                          1. 23

                                                                            My practice routine used to consist of idling on ##c on freenode, looking at questions people were having with regards to C and solving any code problems they come up with. I usually made sure that if I were to send the person the code it was either instructive (in the case of outright rewriting what they wrote) or didn’t completely solve the problem (in the case of questions on how to do something). This meant I could solve problems I would not normally spend my time solving, keep my understanding of C very sharp and provide a high quality of help to people asking in the channel.

                                                                            1. 12

                                                                              This is both brilliant and obvious. Obvious in the sense that helping others helps yourself; it’s a tried and tested method. Brilliant in the way that skill sharing is not something ingrained in the culture.

                                                                              Don’t get me wrong - there are a lot of places to get help on the internet and that’s a great thing. But you’ll know it’s part of the culture when “productivity” is measured - in part - by the amount that you help other people.

                                                                              1. 2

                                                                                Ex #linux (IRCNet) and #csharp (Freenode) hanger-outer here, learning in the same way. Free time? See an interesting question? Try to solve it. If it seems like it’ll help, post it. The original requestor or others provide more info and you end up with a full picture. A fantastic way to learn.

                                                                                1. 2

                                                                                  Did you do this on your own time or as psrt if your job? For the discussion on the industry culture that would make a big difference.

                                                                                  1. 2

                                                                                    I did this entirely on my own time.

                                                                                1. 3

                                                                                  Everyone trots out “DLL hell”, but I don’t think I’ve ever seen this on Windows and it’s purely rhetoric. If anything, dynamic linking is how Windows has maintained herculean backwards compatibility for so long.

                                                                                  1. 8

                                                                                    In 16-bit versions of Windows, a DLL is identified by its filename (for example, C:\WINDOWS\SYSTEM\MCI.DLL is identified as “MCI”), and is reference-counted. So if one application starts up and loads the MCI library, then another application starts up and loads the same library, it’s given a handle to the same library that the first application is using… even if that’s a different file on disk than the application would have loaded if it had started first.

                                                                                    Thus, even if your application shipped the exact version of the DLL you wanted to use, and installed it beside the executable so it wouldn’t be clobbered by other applications, when your application loaded the library you could still wind up with any version of the library, with any ABI, without any bugs or bug-fixes your application relied on… and because every library in the universe had to share the 8-character identifier space, collisions with entirely unrelated libraries weren’t unheard of either.

                                                                                    This is truly DLL Hell.

                                                                                    Modern versions of Windows are less prone to these problems, since Windows 98 allowed different applications to load different DLLs with the same name, since the convention to ship DLLs beside the executable rather than install them to C:\WINDOWS, since modern PCs have enough RAM to occasionally load duplicate copies of a library.

                                                                                    1. 3

                                                                                      On 16- and 32-bit x86, position-independent code was fairly expensive: there’s no PC-relative addressing mode and so the typical way of faking it is to do a PC-relative call (which does exist) to the next instruction and then pop the return address (which is now your PC) from the stack into a register and use that as the base for loads and stores. This burns a general-purpose register (of which you have 6 on 32-bit x86).

                                                                                      Windows avoided this entirely by statically relocating DLLs. I think on 16-bit Windows each DLL was assigned to a segment selector[1] on 286s, on real-mode or 32-bit Windows there was a constraint solver that looked at every .EXE on the system and the set of .DLLs that it linked and tried to find a base address for each DLL that didn’t conflict with other DLLs in any EXE that linked that DLL. If the solver couldn’t find a solution then you’d end up with some DLLs that would need to be resident twice with different relocations applied. I think in some versions of Windows it would just fail to load an EXE.

                                                                                      [1] 16-bit Windows is really 24-bit Windows. The exciting 8086 addressing mode is a 16-bit segment base right shifted by 8 and added to the address, giving a 24-bit address space. As I recall, .COM files were true 16-bit programs in a 64KiB segment, Windows provided an ABI where pointers were 16-bit segment-relative things but far pointers were stored as 32-bit integers that expanded to either a 24- or 32-bit address depending on whether you were on real-mode, standard-mode (286) or 386-enhanced mode (protected mode) Windows.

                                                                                    2. 1

                                                                                      Even if it’s not a thing on Windows (anymore), and it probably isn’t, problems with dynamic linking are a problem on Linux.

                                                                                      1. 3

                                                                                        I’ve seen DLL hell on windows, never on linux. Package managers are awfully effective at avoiding it in my experience. If you go behind your distro package manager’s back, that’s on you.

                                                                                        1. 1

                                                                                          I agree with you, but I’ve seen it with package managers too.

                                                                                          1. 2

                                                                                            “I’ve seen it” is quite annecdotal. I think the point is, like u/atk said, it’s very rare and not a problem on linux, that most people writing code for linux would have to deal with.

                                                                                            1. 1

                                                                                              That’s fair.