Yeah, 30kloc really isn’t very big… I regularly work with a ~14 kloc C++ code base which builds in 15 seconds on my 3.5 year old laptop, and a ~20 kloc code base (+ vendored catch2, + auto generated protobuf code) which builds in just under a minute on the same laptop. And C++ isn’t exactly known for its fast compile times either. 10 minutes is an eternity, and I imagine Apple gives its developer significantly more powerful machines than my 2021 M1 Pro.
I want to add: the auto generated protobuf code is 39kloc across 24 source files. I didn’t know it was that much code. So in total it’s 59 kloc, twice as much as the Swift code base in the article.
I can’t find build time benchmarks but both communities are of the opinion compile times are not good enough. In this project’s case I suspect the ten minutes includes pulling their dependencies and building them too, which is the norm with SwiftPM.
As Ralf responded to that post, it takes those optimizations as a given because that is how existing compilers work, and the “obvious” alternative is far away from the one we live in today.
Discussions of pointer provenance always seem to have someone show up to make this argument that the whole system is just wrong and should be remade from scratch. But the people who make this argument never seem to have anything to say about the reasons compilers and language specs for C-like languages are this way, or how their preferred approach might tackle those problems.
It would be much more interesting to discuss what this alternative approach would actually look like. Presumably very few people would be satisfied with merely falling back to LLVM or GCC’s -O0 output- so which optimizations should compilers perform, and how can they be justified?
Nowhere does it discuss the obvious alternative: those “seemingly” correct optimizations are not, in fact, correct.
The entire post is a discussion of which of the three optimizations should be deemed incorrect.
I think that the issue should be resolved by saying that pointers have provenance but integers do not, which means that it is the second optimization that is wrong … But ultimately, it will be up to the LLVM community to make this decision. All I can say for sure is that they have to make a choice, because the status quo of doing all three of these optimizations leads to incorrect compilation results.
Do you feel the same way about the strict aliasing rules? My understanding is that those were added back when C was competing for performance with Fortran. The performance penalty for “just doing what the programmer wrote” has only gotten larger since then.
I think the case for this with regards strict aliasing is actually stronger. That is, we know which optimizations need to assume strict aliasing and how much performance they win you - not a terrible amount, it turns out. That’s why GCC and Clang have -fno-strict-aliasing and why Linux uses it but is still plenty fast.
Provenance is much more central to how modern compilers work, its assumptions are so baked in that you can’t even tell these compilers “-fno-provenance” and see how much speed you lose.
That is, we have usable production compilers that don’t need strict aliasing, we don’t have any that don’t need provenance. So saying “strict aliasing is wrong” is a tenable position, and indeed Rust doesn’t have strict aliasing.
I spoke to Apple folks when their compiler team switched the default to strict aliasing. They reported that it made key workloads 5-10% faster and the fixes were much easier to do and upstream than I would have expected. My view of -fstrict-aliasing at the time was that it was a flag that let you generate incorrect code that ran slightly faster. They had actual data that convinced me otherwise.
In support of your argument, we do have wasm which has no UB. In some half-assed experiments I’ve found that no amount of optimizing wasm backend will make up for generating wasm with llvm O0, which seems like weak evidence that something in the c language semantics was necessary for better-than-O0 performance.
Poking through the generated code a lot of it is pointless stack loads/stores, which are impossible to eliminate unless you are allowed to make some assumptions about which pointers might alias local variables.
As a curmudgeonly old C guy, I hear you, but on the other hand, if I were to try to write that “everything else” down as a precise specification, I bet I couldn’t do it.
It feels like C is trying to be too many things at once. Maybe there should be a #pragma do_what_I_said for code that needs to relate to the actual hardware. I mean, when I was doing low-level C seriously, even that optimization of hoisting i+j out of the loop would have been annoying. If I wanted it hoisted, I woud have hoisted it myself!
Last time I thought about this, I concluded that without provenance, a huge proportion of optimizations would be invalid, so to me it seems like much more than just a nice to have.
I haven’t followed Swift much recently, but the original Swift was simple because it punted on all of the hard problems. It had no mechanism for error handling (there were some small mods towards NSError). It had no support for concurrency and the everything-is-shared-mutable-state model made that really hard to fix (the structured concurrency and actor extensions are trying). It had no support for shared libraries (Swift’s shared library support now is really nicely designed and it’s the only modern language that I think has addressed this problem well, but that inevitably came with complexity).
For macOS development, Objective-C++ is actually a very nice language. C++ for local code that is specialised over strong types, Objective-C for late-bound interfaces that are exposed across module boundaries,
I have never used Objective C++, but from afar it sounds horrifying. Take a monstrously, ineffably complex programming language and another quite complex programming language and mash them together into an amalgamation. Can anybody hope to make sense of it?
Speaking as someone who used it for years, it actually works quite well. Obj-C is not very complex, and its OO stuff is pretty separable from the C part. Obj-C++ extends the C part.
I’ll also point out that modern Objective-C’s ARC (automatic reference counting) composes very well with C++’s object lifetime rules. In pure Objective-C with ARC,
is not allowed, because NSString* is an ARC pointer and the compiler needs to be able to reason about its initialisation and invalidation. The semantics of C structs don’t allow for that, which can make implementing custom data structures tricky: you either have to do it all in the context of Objective-C classes (which has certain overheads), use indirection, or you turn off ARC in the relevant source files and do manual reference counting.
The same code quoted above will compile on Objective-C++ however, because the ARC pointer is treated as having the usual set of C++ constructors, destructor, assignment operator, and so on - it’s not a POD (plain old data) type. This means the struct also gets implicit constructor/destructor/operator implementations when not explicitly implemented.
You can therefore shove pointers to Objective-C objects into all sorts of C++ data structures, that have been implemented without special Objective-C support, including the STL. It all composes rather nicely.
(A significant proportion of my work in recent years has been working on macOS device drivers and related apps, daemons/agents, etc.; I’ve mostly been using Objective-C++ on that, although I’ve recently introduced Rust in one part of such a project. My limited contact with Swift has been exceedingly frustrating, so I’ve avoided it where possible; it never appealed to me in the first place due to the reasons David mentioned, and the practicalities around extremely poor forward- and backwards-compatibility were a nightmare to deal with on a project where it was forced upon me.)
ARC certainly makes this easy, though prior to ARC I implemented a C++ smart pointer class for Objective-C references that did the relevant retain and release operation, which meant that this was already possible in Objective-C++ without ARC, it just became nicer because you didn’t need to use the smart pointer.
Definitely, you weren’t the only one to implement such a smart pointer class. I guess my point was more that modern Objective-C actually composes better with C++ libraries than it does with C libraries and data structures.
I also used it for years and we made Mac apps that spoke to hardware devices through the IOKit kernel driver API (which is in C++). It was indeed quite nice.
There is a non-trivial amount of ObjC++ in Firefox, too, or at least there was last time I checked. For TenFourFox I used ObjC++ as glue code to connect up libraries.
To add to what others have said: a lot of the pain in C++ comes from trying to do things that are easy in Objective-C, and vice versa. With C++, it’s easy to create rich types with no run-time overhead, but that create tight coupling. With Objective-C, you have late-bound interfaces everywhere, but avoiding dynamic dispatch is very hard / hacky. The combination means that you can completely avoid things like raw C pointers. You can used C++ collections inside a module, Objective-C ones across the boundaries.
You should give Obj-C a try, I think! It’s a surprisingly thin layer on top of C, giving a lot of bang for the buck for writing dynamic (but fast!) programs. It’s quite unique in that you have two extremes: C on the one hand, and a fully dynamic OO runtime (you can determine implementation at runtime). Overall syntax is outdated and weird (smalltalk influence), but it is still unmatched in that niche.
It’s not bad in practice. The use case is, you need Objective-C system frameworks but you can’t do without particular C++ libraries. You still spend the bulk of application code in one language.
I had a game that used Bullet physics in this way. I migrated most of the code to Swift after it was introduced, but I kept some Objective-C++ in order to keep Bullet. These days Swift has direct C++ interop, both for library use and for gradual migration of C++ projects to Swift.
I worked with making an iOS app at one point and I found while thinking in Lisp like patterns it seemed to get out my way if I wanted it to. But that is a beginner and greenfield perspective for sure.
I don’t think it sounds too bad, but I haven’t used it myself.
My understanding is it’s just the extra OOP bits from Objective-C overlayed on C++, similar to how it was overlayed on C in the first place. Basically just a second, independent object system. I understand why people wouldn’t like that, but it doesn’t sound too different than C++/CLI or using C++/JNI.
With the caveat that I’ve read their design docs, but not actually used it in anger:
They make a clear distinction between ABI-stable and ABI-unstable shapes of structures. Within a library, there are no ABI guarantees. At a library boundary, you have a choice whether you want to sacrifice some performance for the ability to change a layout later, or sacrifice flexibility for performance. This is a per-structure choice. Depending on the choice that you make, the compiler either lowers to something similar to Objective-C non-fragile ivars, or C struct fields.
They actually have a language notion of a library boundary. This means that generics can be statically reified within a library, but fall back to dynamic dispatch across library boundaries. Contrast this with C++ where templates either have to live in headers (and then end up copied in every compilation unit, including the implementation, and it’s a violation of the one-definition rule to link two libraries that use different versions of the same template) or they are private to a library. The Swift model gracefully falls back. Things may be faster inside a library, but they still work from outside, and the ABI doesn’t leak implementation details of the generics, only their interfaces.
but the original Swift was simple because it punted on all of the hard problems
Hmm…I’d say it was already incredibly complicated despite punting on a lot of hard problems and largely because it tried to codify “solutions” to non-problems into the language. Which never works, because reality, even code reality, is way too messy for that.
As an example, I wrote about the mess that is initialization back in 2014, so right after Swift apepared. There was some pushback from a prominent member of the Swift team saying that my goal of simplicity just wasn’t compatible with some of the goals they had. Hmm….
There was also a great rant by a prominent member of the community about Swift being just a mess of special cases upon special cases. I think he left the Apple ecosystem, and he’s not the only one. Alas I can’t find it and I don’t remember the name.
Anyway, I predicted back then that because they had so many language features for initialization it would never actually work out and there would be a renaissance of builder and factory method patterns and there would be even more language features for initialization. Five years later: hello SwiftUI! :-)
So the failure of Swift now isn’t surprising, the trajectory was already set in stone the day it launched and there wasn’t really much one could have done about it afterward…much less so since the same faulty reasoning that led to the initial problems was still present and guided later evolution.
Anyway, I predicted back then that because they had so many language features for initialization it would never actually work out and there would be a renaissance of builder and factory method patterns and there would be even more language features for initialization. Five years later: hello SwiftUI! :-)
I think this is an instance of correlation not being causation? My understanding is that the actual cause of SwiftUI is the successful design of Flutter (which gave raise to both SwiftUI and Kotlin Compose), and it is relatively orthogonal to language machinery.
Case in point, Kotlin’s initialization story is much more tame than Swit’s one (as it doesn’t try/doesn’t need to prove initialization safety statically), but it also converged on the essentially same design (or rather, vice-verse, IIRC Kotlin work in the area predate’s Swift’s).
Not to disagree with your wider point on constructors, which I agree with, just to point out that SwiftUI is not I think a particularly strong argument here.
I think you might want to have a look at the actual article. Swift introduced yet more special syntax for the part of SwiftUI that creates the view tree. So yet more language features for yet another way of constructing views^Wobjects^Wstructs.
I have read the articles! If I understand your argument correctly, it says that the fact that they needed to add new stuff to support Swift UI means that the original rules were inadequate. My counter-argument is even languages that don’t have Swift-style maze of initialization rule add special cases to support SwiftUI patterns. Ergo, adding stuff for SwiftUI is orthogonal to your normal way to initialize objects. In other words I claim that, in counter-factual where Swift doesn’t have complicated initialization rules and uses Java/Go style “everything is null to start with” or Rust/ML style “everything starts with all the parts specified”, it would have added more or less the same features still for SwiftUI.
The story is even more illustrative with Kotlin — it was specifically designed for DSLs like SwiftUI/Compose. The whole language, with its second-class-lambdas, extensions, and coming out-of-fashion implicit this, works towards that goal. And yet, when the actual UIs started to be implemented, it was quickly apparent that no one wants to write +button(), and a bit more of compiler special sauce is needed for nice surface syntax.
I must be a lousy communicator, because you seem to have misunderstood the article almost completely.
The point was not that Swift has the wrong initialization rules or too many of them. The point is, as it says in the title: “Remove features for greater power”. The many initialization rules are not the source of the problem, they are a symptom of the problem.
First rule of baking programming conventions into the language: Don’t do it!
The second rule of baking programming conventions into the language (experts only): Don’t do it yet!
The problem is trying to bake this stuff into the language. As a consequence, you get 30 pages of initialization rules. As a further consequence, those 30 pages will be forever insufficient.
So for me, the supposed counter-point you bring with Kotlin actually supports my point. You write:
it was specifically designed for DSLs like SwiftUI/Compose. The whole language, with its second-class-lambdas, extensions, and coming out-of-fashion implicit this, works towards that goal
So they baked a whole bunch of features into the language to support the DSL use case. What was the title of the blog post again?
“Remove features for greater power”
So they added a lot of features into the language especially for this use-case and it didn’t even work out for this particular use-case. Surprise surprise!
First rule of baking programming conventions into the language: Don’t do it!
The second rule of baking programming conventions into the language (experts only): Don’t do it yet!
I simply don’t think the static/compiler-oriented mindset is compatible with the sorts of things these languages are trying to do. You put way too much into the language/compiler, and you do it way too early.
Ruby has had a bunch of these kinds of frameworks, and as far as I know they did not require any changes to the language. Because Ruby had fewer but more flexible features to start with.
With Objective-S I seem to be violating that rule, because it certainly does put things into the language. Or at least seems to do so. What I am doing, however, is following the second rule: “don’t do it yet”. (With quite a bit of trepidation, because it is “experts only”).
And I am not actually baking all that much into the language. I am baking a bit of useful surface syntax and the associated metaobject-protocol into the language. What lies behind those metaobject protocols is quite flexible.
So far this appears to strike a good balance between providing some syntactic convenience and compiler support while not making the mistake of baking way too much into the language.
Indeed! I misunderstood your original comment as meaning that SwiftUI is a downstream consequence of initialization rules. I agree that both are rather the result of the lack of expressiveness, which doesn’t allow the user to “do it yourself” in userland code. The Kotlin example was exactly to illustrate that point.
There was also a great rant by a prominent member of the community about Swift being just a mess of special cases upon special cases. I think he left the Apple ecosystem, and he’s not the only one. Alas I can’t find it and I don’t remember the name.
Crucially, the vast majority of this is incidental complexity, not essential complexity. Swift is a crescendo of special cases stopping just short of the general; the result is complexity in the semantics, complexity in the behaviour (i.e. bugs), and complexity in use (i.e. workarounds).
How to fix:
What, then, should be removed? It’s probably too late to remove any of that, …
So I certainly wasn’t the only one who accurately predicted the current sad state of affairs. It was extremely obvious to a lot of people.
I am if the opinion that the only avenue for making a well rounded language that lasts is to put all of the skill points towards a small handful of skills (“fast”, “fast to compile”, “ease of dynamic linking”, “easy to use”, “easy to learn”, whatever) where each “skill” can be thought as a different axis in a graph. Go all in on a few niches. This helps with clarity of purpose for the project itself, a clear pitch for prospective users, and makes it possible to show the benefits for that niche. Without doing this the evolution of the project ends up rudderless, and adoption can stagnate either slowing down or killing the project. Once you’ve reached critical mass, fight for dear life not to abandon those hard earned early design decisions that allowed you to carve a niche, but go all-in on a handful of other axis as well. Rinse and repeat. A language can start as easy to use and once it has a clear base experience it can start trying to produce faster binary. Or it can start as fast to compile and afterwards try to produce optimized code. Or it can start producing super optimized code and later try to become faster to compile or easier to use. All the early design decisions will hamper the later pivot, and that’s ok! No language will ever be all things for everyone (barring some unrealistic investment in monstrous amounts of research, development and maintenance).
Swift had a clear initial view of what it wanted to accomplish, and did so successfully (I believe, don’t have personal experience with using Swift, nor do I keep up with its development). It is now on its expansion era, to allow for niches that previously were either difficult or impossible to cater to. It is a delicate process that can easily go haywire because of subtle interactions between completely unrelated features can make the result worse than the sum of its parts. But I can’t blame them for trying.
I could maybe argue that sync Rust had excellent “early design decisions that allowed you to carve a niche”, but the async related problems are now occupying a vast majority of the brain time of the lang teams. While I don’t think we’re heading to 217 keywords, I certainly feel uneasy seeing “keyword generics” and similar efforts.
I still find this argument as unconvincing as it is intriguing. As far as I can tell, the first part is about hardware being far more complex than what “C” gives you access to. Yes, computers are not fast PDP-11s any more.
However…most of this hardware is just as hidden from assembly language and the machine instruction set. Is machine language “not a low-level language”? Seriously?
Of course, the title is true in one sense, but not in the sense intended by the author: when C came out and until at least the mid to late 80s, C was considered a high level language, not a low-level language. Because the low level language was assembly.
Wrapping file i/o in actors is exactly what we do in the Goblins distributed programming system. It ensures that only one operation is happening against the file descriptor at any given time while allowing other actors to use it asynchronously.
then you have lost all static typing. It is too low level, so the analogy doesn’t really hold up IMO.
I think this is generally true: the Actor model is very low-level. Synchronous messaging with automatic dispatch is a convenience that is layered on top.
Is it just me, or are these two seemingly unrelated schools of computing converging on the exact same idea? You send messages to some target
No it’s not just you. We (as in “the computing world”) is finally figuring out OO, that is the original message-oriented OO, rather than the scoped-procedural computing that we called OO.
I was actually thinking about Alan Kay when I wrote it. There’s somewhere where he talks about objects as “mini computers”, that it’s mini computers all the way down, and he says something like “why would we want to go to lower than that?”. It’s been in my head rent free.
In computer terms, Smalltalk is a recursion on the notion of computer itself. Instead of dividing “computer stuff” into things each less strong than the whole—like data structures, procedures, and functions which are the usual paraphernalia of programming languages—each Smalltalk object is a recursion on the entire possibilities of the computer. Thus its semantics are a bit like having thousands and thousands of computers all hooked together by a very fast network. Questions of concrete representation can thus be postponed almost indefinitely because we are mainly concerned that the computers behave appropriately, and are interested in particular strategies only if the results are off or come back too slowly.
If it’s separate computers, the messaging is pretty obviously asynchronous. Having that mechanism simulated by dispatching synchronous procedure calls is a simplified and optimized implementation, but it’s not the model.
I’m not sure about that. The ur-OO was Simula, a language for implementing simulations of real world systems like hospitals. That would have benefited from asynchronous messaging (but I don’t know Simula so I’m just guessing.)
Joe Armstrong has gone on record saying that he thinks that Erlang is “possibly the only object-oriented language” (the context adds “OO in the Alan Kay meaning of the word”). Johnson, in the same interview, points out the same things that Sean Copenhaver says in his answer: in the small, Erlang’s a purely functional language; in the large, with processes, it looks exactly like Kay-style object orientation.
The early implementations were synchronous. This was seen as a problem.
As a German software developer, I find that I need to use a US QWERTY keyboard to be reasonably productive (also lived in the US for a bit, so used to it), because the German QWERTZ layout puts all the braces and stuff in weird, hard to access places.
However, on macOS that leaves all the Umlauts behind dead keys for the ¨ diacritic, which is very inconvenient. My layout puts all the Umlauts at Alt-, with other accented characters accessed via dead keys.
The second thing is that for Objective-S, I put the left and right arrows ← and → on Alt-[ and Alt-]. Alt-{ is ≝. In addition ≠ is at Alt-=. π is at Alt-p. Objective-S uses ← for assignment and → for architectural connection, though you can also use ASCII := and -> if you are so inclined.
So nothing earth-shattering, but a few tweaks that make things much, much nicer.
Oh, and of course the Ctrl Key mapped where DEC had intended it to be, where people without a clue put the entirely useless Caps-Lock key. Almost forgot about that one…
Not just that, I’d venture to say that Objective-C is vastly simpler than C++.
I’d strongly suspect that if you try to understand Objective-C by translating it to C++, you will almost completely fail to understand it, even though you will see how to make it work, because this fact that it is so much simpler will get lost in the translation. By definition.
What you will see is an utterly weird contraption that does not make any sense whatsoever, and shake your head in dismay. Looking at Swift, I can’t help but think that this is exactly what happened.
In my experience in industry, I saw very little to no object-oriented programming. I saw a lot of what I would call “scoped procedural” programming with some abstract data types thrown in.
That is, for most of the call sites, it was 100% clear what method/procedure would be called. object.operation(x,y) was completely equivalent to operation(object,x,y). (And I once did some analysis to quantify “most”. For the project I was working on it was >98%).
And no this isn’t “you are all too stupid to get true dynamic messaging-oriented OO”. While that’s not entirely wrong, it is also not entirely and probably not even mostly correct. This was in a highly dynamic language, so you certainly could go more true dynamic OO if you wanted to.
For me, one of the primary reasons is that our languages are not, in fact, object-oriented. They can at best be described as “class oriented”, although that doesn’t really show the problem, IMHO, which doesn’t go away if you go to a prototype language. The problem is that all those languages are basically scoped-procedural. The procedures that you write are now called methods, but they differ very little from other procedures.
And so what you can do easily doesn’t differ that much from what you can do easily with procedural programming.
People can do it. But it takes quite a bit of skill to express those object-oriented systems indirectly via scoped-procedural programming languages. Only a little less than with plain procedural.
And it gets worse, potentially: the indirection that is required for “good” object-orientation is often misinterpreted as being the actual good part, not the bad part you need to endure to get to the good part. And so you can get quite horrible architecture-astronautics. Which gave OO a bad reputation it only very partially deserves.
So we need to go beyond scoped-procedural programming. Not abandon it completely, because it is absolutely useful. But insufficient,
As an example of the insufficiency: we are told to use composition over inheritance, because it is better. What linguistic support is there for object composition? <crickets> There is linguistic support for inheritance, which partly explains its continued popularity. Not only can you express (write) inheritance in the code, you can also read it. It’s stated explicitly in the code. Not so with composition. Composition is implemented procedurally, so you have to infer the actual composition of the objects from the procedural code that performs the compositing. And once a composite is created, it is also mediated procedurally, with objects calling procedures on each other.
At some point you can be forgiven if you say “let’s just cut out the middleman and go full hog procedure”.
Not quite crickets, there are procedural languages with support for object composition. Starting with Plan9 C, continued in Alef, and present in Go. There may be others.
That is the ability to embed an anonymous struct by type within another, and then call the functions for the embedded struct against the outer struct.
Yeah, I was going to mention that procedural languages, including plain old C, actually tend to have better support for “object composition” than “OO” languages.
I found this really ironic.
Although this is really support for complex (nested) object literals, not really for object composition. Which is also very useful and goes in the right direction, but not the same thing.
What would it take for a “scoped procedural” language to become actually object oriented? What would it take for a program to make the same transition?
What linguistic support is there for object composition?
Not sure what support you have in mind, right now the only thing I can think of is importing the public fields (including methods) of the contained class inside the scope of the containing one. Basically inheritance, save for two crucial details:
Unlike most implementations of inheritance, it doesn’t have to imply sub-typing as well (though I’m betting it could).
Methods of the contained class would never rely on code from the containing class, so we’re safe from the fragile base class problem.
And I can see a number of patterns would be made more convenient that way. How often and how much though? The way I see it, it would only cut a little boilerplate. If I’m correct, it’s not surprising language designers didn’t feel the need to add such a feature.
On the other hand, the end state is often: Dear friend, you have built a LAMP server, using k8s (and Helm, and operators, and cert-manager, and Prometheus, and…).
I hope that this doesn’t come across as too snarky but I’ve just suffered the pain of getting a single-node Ansible AWX server running. Their only supported architecture is k8s and their operator. It took days! I remember back in the old days we used to say “simple should be easy and complicated should be possible.” Now it’s “everyone needs Google scale so that’s all we’re going to do.”
As an example of doing it the right way, the Immich photo management app is comparable to AWX in run-time complexity but it comes with a Compose file that makes it trivial to run and manage on any machine with docker already installed.
FWIW you don’t need Helm for Kubernetes, and you’re going to want something like cert-manager regardless (and if you don’t want cert-manager, just don’t do anything—I wish k8s would include it by default, but it doesn’t).
Similarly, there’s a lot of middle ground between single node and Google scale, and you should have at least one replica running in production in most cases, which i means you’re going to need some sort of load balancer and some deployment automation. At this point you may as well start considering Kubernetes.
Not only does Kubernetes provide load balancing and deployment automation, but it also allows you to easily run cloudnativepg, an operator that makes it easy to run a Postgres cluster with replication, backups, and failover. Maybe you don’t need that for your production application, but if you do, it’s a lot easier to run k8s and that operator than it is to use other tools to manage Postgres IMHO.
which means you’re going to need some sort of load balancer and some deployment automation. At this point you may as well start considering Kubernetes.
That’s an astonishing jump. For a handful of machines, a few scripts with ssh and some pretty standard networking tricks would be what I think of long before k8s.
I’m not sure exactly what you’re thinking with your “few scripts and networking tricks”, but it sounds like you’re trying to refute the need for a load balancer by arguing that you can implement your own load balancer.
Moreover, while any given feature Kubernetes provides out of the box could be implemented easily enough by a sufficiently competent, experienced engineer, the point of the article is that the total effort is much greater and you still end up with a platform for which no documentation, or training, or experienced hiring pool exists and your expensive competent, experienced professional has spent his time building Kubernetes instead of doing more valuable tasks (or more likely, your professional only thinks he’s sufficiently competent and experienced, and you wind up with a very broken Kubernetes that causes endless frustration).
it sounds like you’re trying to refute the need for a load balancer by arguing that you can implement your own load balancer.
If you have two machines you don’t need a load balancer. You just need a way to fail over to the second machine. That’s just network configuration, which you have to do anyway, Kubernetes or no Kubernetes.
That’s the problem with this argument. You still have to deploy machines, configure operating systems and networking, install services (whether your services or Kubernetes’s services), etc. unless you’re using a 3rd party managed Kubernetes, which renders the comparison irrelevant anyway. So you can expose Kubernetes as a uniform target to a bunch of teams, but you still have a lot of stuff that is local deployment. If you don’t have a platform team providing a uniform target to lots of other teams, and you’re already doing all this configuration anyway, why do the configuration of Kubernetes as well?
If you have two machines you don’t need a load balancer. You just need a way to fail over to the second machine. That’s just network configuration, which you have to do anyway, Kubernetes or no Kubernetes.
Kubernetes generally takes care of the failover for you. You don’t really need to think about it.
That’s the problem with this argument. You still have to deploy machines, configure operating systems and networking, install services (whether your services or Kubernetes’s services), etc.
There’s much less operating system configuration if you’re using Kubernetes than if you’re rolling your own hosts, and the networking setup is “install MetalLB”. It’s extremely minimal. Similarly, certificate management is “install cert-manager”. DNS is “install external-dns”. Similarly there are standard, high-quality packages for logging, monitoring, alerting, volume management, etc. While I’m surprised there are Kubernetes distros that include this stuff out of the box, it’s still less work than doing it yourself and you get the benefits of standardization: abundant, high quality documentation, training resources, and an experienced hiring pool. And moreover, as your requirements scale, you can incrementally opt into additional automation (Kubernetes is a platform for infrastructure automation).
unless you’re using a 3rd party managed Kubernetes, which renders the comparison irrelevant anyway.
I’m not sure what you mean here. Why would managed Kubernetes render the comparison obsolete?
So you can expose Kubernetes as a uniform target to a bunch of teams, but you still have a lot of stuff that is local deployment. If you don’t have a platform team providing a uniform target to lots of other teams, and you’re already doing all this configuration anyway, why do the configuration of Kubernetes as well?
I’m not following you here. What do you mean by “uniform target” vs “local deployment”.
No, it’s not. You still have to set up the underlying network that the physical machines are on. That’s the key point. You can install all this stuff within Kubernetes…but it’s all dependent on nodes that are fully configured outside of Kubernetes.
I see what you mean. Yes, you need to have nodes that are on a network, but “fully configured” for running Kubernetes basically means “SSH is configured”, which is dramatically easier than any configuration I’ve seen for running production-grade applications directly on nodes (no need to haul out ansible or anything like it).
Kubernetes generally takes care of the failover for you. You don’t really need to think about it.
By design, sure. In practice though the biggest failures I’ve had with kubernetes have been nginx ingress pods just randomly losing their ability to connect to upstream and silently causing downtime that wouldn’t haven’t happened if the app was exposed directly, using an AWS ALB, etc. I’m not saying this is everyone’s experience. It might be due to noob misconfiguration mistakes. But that’s sorta the point the article misses. Kubernetes might offer repeatable solutions for these problems (and I appreciate it for that) but it really isn’t a solution on its own. Every team running a kubernetes cluster has their own collection of weird scripts with bespoke workarounds for kubermetes quirks.
Thank you! In this case I was told to stand up an AWX instance, and I didn’t know what I didn’t know, but I’ll take a look at your suggestion. If there’s a “next time” then I’ll be able to advocate for an alternative.
OMG this. At work I tried to fix a broken internal app that I had no role in getting set up. Everyone who had worked on it had departed. So started the slog of figuring out how it worked, where it was deployed, and getting access. Oh cool, this runs on the “internal dev platform”. What is that? Oh it’s a kunernetes cluster. Neat that they tried to think ahead spread some infra costa across projects. I wonder what else is running on this thing… nothing. It is a 2 node kubernetes deployment that runs one application pod. Not per node. Total.
It is a 2 node kubernetes deployment that runs one application pod. Not per node. Total.
I was working in an Oracle shop where devs would spin up an Oracle instance for a web application that had 4 data items. Not databases. Not tables. Items.
My production app ran completely without databases and was several orders of magnitude faster, smaller, and more reliable than its predecessor.
But think. How the devs could otherwise move on in their careers without “designed, built, deployed and maintained a scalable K8 solution” on their CVs?
Ding ding ding! You got it. On a lark I searched Google for large sections of the docs they “wrote” for it and sure enough it was all basically copy/pasted from guides on how to set up RBAC.
The current standards of evaluation in the PL community are set from a masculine perspective as well, valuing formalism and formal methods over user studies, quantitative over qualitative work, and the examining of technical aspects over context and people.
I just can’t connect the dots between masculine and valuing formalism, formal methods, quantitative over qualitative work, and the examining of technical aspects. Can someone eli5.
It reminds me a bit of the “punctuality, standardized testing, etc are white supremacy” stuff from the previous decade. Honestly this stuff always seems a lot more like harmful stereotyping and I would appreciate a clear explanation for why it’s not as toxic as it seems.
Except the stereotype is true: women are raised to care more about people than men are, at least in the Western countries I’m aware of. This whole thing about formalism being more masculine than studies, does match my intuition about why we find so more men in “hard” sciences than we do in the humanities. (Note: I hear that Asia has different stereotypes, and many more women in hard sciences. In Japanese animation for instance, a typical support character is a female scientist or technician — yeah, the main/support gender bias is still there I think.)
Now the famous “Programs must be written for people to read, and only incidentally for machines to execute.” have indeed been coined by men (Harold Abelson and Gerald Jay Sussman), and on that same page I learned that another man, Martin Fowler, said “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” Clearly human stuff is important to men too. And there are some crazy impressive low-level woman programmers out there too.
And yet, I can’t shake the feeling that indeed, we dudes tend to ignore the human element more than women do. One striking example would be the Linux kernel, and the recent Rust debacle. It did reek of not trying particularly hard to take care of humans. Or take my own readability article: the whole point is to cater to humans (increase readability), but then I reduce humans to a single limitation (working memory), and evaluate various advice from that alone. Which, when I think about it, is only possible because I have such a mechanical outlook of the human mind — and I don’t even have any expertise there!
Now that paragraph in the paper could probably have benefited from a couple references showing the prevalence, or at least existence, of the stereotypes it uses in its exposition. Without it feels like the author has internalised those stereotypes more than she would have liked to admit.
While an inaccurate stereotype may be more harmful than an accurate one, an accurate stereotype is still pretty bad. Even if it happened to be true that men are better drivers (whether by nature or nurture), it seems pretty harmful to refer to good driving as a masculine trait and also a thing can be true for a group of people and not be true for all individuals in that group, so applying stereotypes (even accurate stereotypes) to individuals is usually unjust. That’s probably not what the paper is advocating, but it’s an inevitable consequence of trading in stereotypes and moreover I don’t see how stereotyping improves the paper.
For a specific idea of a harmful true stereotype see also “black people don’t swim”. There’s lots of stuff in society which comes from the environment and not conscious choice. It doesn’t mean we should assume that’s just how things are and expect them to continue. Sometimes things are more universal if we allow it / enable everyone to do better in whatever way they choose to approach things.
There was a fascinating experiment on stereotype threat where they got a bunch of Asian and Black Americans to try golf. They told half of them that golf was very mathematical and the other half that it was very athletic. There was a statistically significant difference: the black folks who were told it was athletic did better and the Asians who were told it was mathematical did better. Both groups did better at the thing that they’d been stereotyped at being better at even when it was the same task. So reason one that even accurate stereotypes are bad is that they impact the people being stereotyped.
Reason two is that they are still about probabilities. If a stereotype is 90% accurate, it will not apply to all of the members of a group that you can fit in a fairly small room. If 90% of green people are bad at programming, you’ll see fewer green people in software development, but all of the ones you see will be statistical outliers and the stereotype tells you nothing about that subset, but people interpret as if it does. And even the most accurate stereotypes apply to far fewer than 90%. So reason two that they’re bad.
Reason three is that a lot of accurate stereotyped behaviour is due to how people interact with members of a group at an early age. People subconsciously reward behaviours in children that conform to expectations. Boys and girls are praised for different things by every adult around them. This makes stereotypes self perpetuating.
Couldn’t you just as easily argue that formal methods (conscientious, detail-oriented, theoretical) are feminine, whereas user-studies (pragmatic, empirical, utilitarian) are masculine?
…it was once, not even so long ago, seen as women’s work. People used to believe that women are good at programming because they are fastidious, “they worry how all the details fit together while still keeping the big picture in mind” [22] and “programming requires patience and persistence and these are traits that many girls (sic.) have” [47].
One striking example would be the Linux kernel, and the recent Rust debacle.
But the C advocates were arguing against formalism and formal methods, qualities that the quote describes as “masculine”. In fact, I’ve seen people argue that the “C is fine, just write good code” mindset is an example of masculine arrogance (or what have you). So we’re in a situation where emphasis on formal methods is masculine and disdain for them is also masculine.
Except the stereotype is true: women are raised to care more about people than men are,
I’d say the stereotype was true: we’ve observed much change in society over the last ~15 years, particularly in gender roles. I don’t think the stereotype is true anymore. If it is, its decreasingly so year after year, and will soon be definitively false.
The ground of the feminist critique is what we could call a critical history. Traditionally, we think of philosophy or science arising from a neutral form of “reason” – detached from social or historical context. A critical history challenges this claim, and views thinking as necessarily embedded in a place and time. Thought is not considered to be a universal means at finding objective truth but rather the thought arising out of a specific place and time, under regimes of thinking that ground our understanding of what is true or false. This is not subjectivism (“everyone’s perspective is valid”) but rather a sort of uncovering of the origins of ways of thinking which we take for granted.
Given that the historical context of Western civilization, and more specifically, the 20th century development of computers, feminists can critique the development of computing from the lens of a specifically feminist history – uncovering the manner in which thinking about computers came out of a specifically patriarchal context and mode of thinking. The point is not that early computing pioneers were men, but rather than they were embedded in a society which prioritized ‘masculinist’ forms of thinking and potentially reproduced those structures: structures that exist independent of any individual’s identity, but rather have a sort of independent existence
It’s important to understand that it’s not a universalizing critique, ie we can’t understand everything about computing through feminist critique, but it is one lens through which we can look at the history of computing
The point is not that early computing pioneers were men, but rather than they were embedded in a society which prioritized ‘masculinist’ forms of thinking and potentially reproduced those structures: structures that exist independent of any individual’s identity, but rather have a sort of independent existence
Said that way makes more sense, at least to me, in which “masculine” refers to the patriarchy-way-of-thinking. Would have used other term, but that’s nitpicking.
The original text in the paper sounds too much like a stereotype applied to men as a gender, maybe because I’m mixing in my head my own perspective with the past and everything in between. “men molded the field like men do, because they all behave the same way”.
Based on your description, it seems like critical theory and feminist critique is principally about ideological framing? As someone who is interested in programming language design, I probably don’t have a lot of interest as to whether we should believe that computing originated out of something called “patriarchy” or whether it originated out of some neutral reasoning, I just want tools that help me accomplish my goals more easily. These ideological frames feel unhelpful at best and divisive or even toxic at worst (I made another comment about how claims like “formalism is masculine” and “objectivity is white supremacy culture” seem like really harmful stereotypes and yet they seem to closely orbit critical theory).
I just want tools that help me accomplish my goals more easily
This is not a statement free of ideology. First off, you are conveying a form of instrumentalization: ie, you view programming languages as a tool to be utilized. I’m not saying this is incorrect or bad (obviously, there are many cases in which we do use PLs this way), but it is not the only way of conceiving of programming language design. We could think of programming languages as toys to play with, forms of expression, or places to try out new ideas. Consider the whole field of esoteric programming languages – almost all of them totally fail as “tools to accomplish goals more easily”.
I probably don’t have a lot of interest as to whether we should believe that computing originated out of something called “patriarchy” or whether it originated out of some neutral reasoning
The point is, there is no “neutral reasoning”. Everyone is always thinking within a culture that has a certain ideological frame. This not a moralistic point (although sometimes this sort of analysis does end up distorted into popular culture that way), it just means that you can’t separate computing from its historical, cultural, and ideological history if you want to understand it fully. I mean, consider the basic point (referenced in the paper) that most PLs are monolingual in English – this isn’t because English is the “objectively best” language for programming, it’s necessarily tied to a long history of British and American cultural imperialism. And again, I want to emphasize, the point isn’t that it is “morally wrong” that programming languages are in English. It’s the academic point that this development is inseparable from the history of colonialism. And this critical history can inform experiments, development, and research into non-English based programming languages.
I made another comment about how claims like “formalism is masculine” and “objectivity is white supremacy culture” seem like really harmful stereotypes and yet they seem to closely orbit critical theory
Your issue appears to be with specific applications of critical theory, yet you expand this to include the entire field itself. One could certainly make a bad feminist critique of PLs, but that doesn’t mean that feminist critique is a useless enterprise, which is what you seem to be claiming.
We could think of programming languages as toys to play with, forms of expression, or places to try out new ideas. Consider the whole field of esoteric programming languages – almost all of them totally fail as “tools to accomplish goals more easily”.
Nit: the “goal” in those cases is to have something to play with, something to express oneself with, or some place to try new ideas. You aren’t really disagreeing.
I mean, consider the basic point (referenced in the paper) that most PLs are monolingual in English – this isn’t because English is the “objectively best” language for programming, it’s necessarily tied to a long history of British and American cultural imperialism.
I’ve seen a good chunk of listings for game code written in German that would disagree with this premise. I’d imagine that the Soviet code listings in Cyrillic probably also disagree–Drakon, for example.
Re: imperialism, it’s well-known that CS is rife with accomplished Englishmen like Alain Colmerauer and Americans like Edsger Dijkstra. Similarly, we all run on silicon manufactured in the United States by red-blooded American companies like TSMC and using phones made by god-fearing businesses in Western countries like Huawei.
The thing is, that lens means a particular thing in academia, and outside of academia–where we are here–it’s easy to “disprove” using common sense and rudimentary historical knowledge by the casual practitioner. That’s not to say it’s a bad or even incorrect lens, but merely that it requires more nuance in application and context than is usually found here–people spend their entire doctorates haggling over narrow aspects of imperialism and feminism, after all.
EDIT: After talking with a colleague, I’ll concede that if by “forms of expression” you mean a programming language itself as a sort of work of art then my lumping it in as a tool is a bit unfair. Paint vs paintbrush.
I’m not sure if I understand your point here or if we really disagree. I said most programming languages are English monolingual, and that this fact is tied to the history of British and American cultural imperialism. It is not that we can understand technology solely through that lens, just that a purely neutral, technical, ahistorical viewpoint is not the correct one. The example of Huawei is particularly interesting here – it’s been sanctioned by the US since 2020, and for geopolitical, not technical reasons.
But isn’t it the case that certain fields are dominated by a particular language? Musical terms are Italian; cooking terms and techniques are French; large portions of philosophy are French and German, not to mention the endless Latin found in medicine and law. Could it not be that computers are largely English because English-speaking countries dominated the space early on?
Another thing I’ve heard (or read, sorry, no citations available) is that non-English speaking programmers tend to prefer computer languages not in their native language, for whatever reason. Maybe just a simple “translation of keywords into target language” just doesn’t work linguistically? Not enough non-English documentation/learning materials?
The ability to share code and libraries with the rest of the world. Imagine if you had to learn French to use sqlalchemy, German for numpy, Japanese for pandas, and Russian for requests.
consider that constructed languages are (with few exceptions) designed to be much simpler to learn than natural languages. it’s a lot easier to learn 200 sitelen pona than 2000 kanji. (and it’s a lot easier to learn a phonetic spelling than whatever english is doing)
After we’ve finally finished migrating everything to Esperanto, it will be found to be problematic because of certain reasons. Then we will need to migrate to a new created human language base for programming languages. I vote for Tolkien’s Quenya.
TBH it’s a great case for programming languages in English, which is the most popular spoken language in the world thanks to its strong popularity as a second language globally.
https://www.ethnologue.com/insights/ethnologue200/
He spent a lot of time at the University of Texas in Austin, hence I think it makes sense that the quotes are in English (as that is the actual quote). And yes, I guess most Dutch readers are also familiar enough with English that they do not need a translation.
I just want tools that help me accomplish my goals more easily
This is not a statement free of ideology.
Yeah, it pretty much is. It is a statement of personal preference, not a set of beliefs.
but it is not the only way of conceiving of programming language design.
And it does not claim that it is “the only way of conceiving of programming language design.” It is an expression of a personal preference.
this isn’t because English is the “objectively best” language for programming
And nobody claims that that is the case, so this is just a straw man.
inseparable from the history of colonialism
It has almost nothing whatsoever to do with “colonialism”.
critical theory, yet you expand this to include the entire field itself.
Critical “theory” is not really a “field”, and certainly not a scientific one. It (accurately) self-identifies as ideology, and then goes on to project its own failings onto everything else. With “critical theory” the old saying applies: every accusation is a confession.
what goals do you want to accomplish? why do you want to accomplish them? in what socio-politico-technical context are the tools that are more or less fit for that purpose developed, and how does that context influence them?
What goals or socio-political contexts lend themselves more to more or less formal programming methods? Like even if I want to build some “smash the patriarchy” app rather than a “i ❤️patriarchy” app, would I benefit more from languages and tools that are less “formal” (or other qualities that the poet ascribes to masculinity)? I can’t think of any programming goals that would benefit from an ideological reframing.
the current state of the art in PL research and design prioritizes machines over people, there is a focus on programming languages that are easy to read for the machine, rather than easy for people
I’ve not finished the paper yet but it seems like the authors are using “masculine” as a shorthand for “thing oriented”. Which isn’t a perfect analogy, but in my experience “thing oriented” typically skews male and “person oriented” typically skews female.
Yes, and this is a stereotype that has been semi-deliberately produced and spread. You can read lots of stuff about women in technical fields in the late 19th through mid 20th century, going to universities, applying for jobs, etc and a lot of it involves people explicitly or implicitly saying “women can’t do X because men think logically/rationally/analytically, while women think emotionally/impulsively/socially”. Like most stereotypes there’s a grain of truth in it from the right angle – I recall talking to my sister when she was pregnant with her son and she often said things like “I get so emotional for no reason” and “it’s so draining, I feel like my IQ has dropped 20 points”… now take this phenomenon and project it to a time when families commonly had 8 kids or whatever, your average woman would spend a decade or two of her life either pregnant or recovering from pregnancy.
But the real reason that stereotype exists is because it is used as a proxy for “women shouldn’t be able to do X because I do X and I’m obviously better than a woman”.
Women make up the majority of college graduates. Have you considered the crazy notion that women are capable of making their own decisions as to what fields they wish to study and later work in?
98% of all people choose to not go into computer science. What’s wrong with the world that with males that percentage is only 97%?
As a white boy, no one ever told me that I shouldn’t program because it’s not a white thing. No one ever told me boys can’t code. No one ever told me that computers are not for people like me. No one ever sent a less-competent girl to a maths or programming extracurricular event because programming is a girls thing and so they didn’t consider me. No one ever told me not to bother with programming because it doesn’t lead to appropriate careers.
As Director of Studies for Computer Science at an all-women Cambridge College, every single one of the students that I admitted had stories like that from people in positions of authority. Young women who went on to get first-class honours degrees in computer science at one of the top departments in the world were told girls can’t program by their computing teachers. Girls who scored Gold in the Mathematics Olympiad were encouraged not to enter by their teachers. These aren’t isolated things, these are small social pressures that have been relentlessly applied to them from birth. And the ones I see are the ones who succeeded in spite of that. A much larger number are pushed out along the way.
Now maybe social pressure like this doesn’t apply to you and so this is not relatable for you. There’s an easy test: do you wear a skirt in the summer when it’s too warm for trousers to be comfortable?
Now why do I care? Because I’m selfish. I want to work with the best people. Having more than half of the best people self-deselect out of the profession as a result of unintentional social pressure means I don’t get to work with them.
Thank you for saying what I wanted to far better than I ever could.
I’ll follow up on the research on sex-preferences in newborns or other primates, ‘cause it’s really interesting: built-in differences in preferences are totally real, sure! The differences exist. But everyone is an individual. The way I think of it is that looking at whatever human behavior you want to measure, you’ll probably get a big bell-curve, and if you measure it by sex you’ll probably get two different bell-curves. The interesting question for science is how much those bell curves overlap, and how much the differences are environmental vs. intrinsic. The studies on newborns and such are interesting ‘cause cause there’s barely any cultural or environmental impact at all, so that’s relatively easy to control for.
But when people do studies on “vaguely what do these bell curves look like”, such as https://doi.org/10.3389/fpsyg.2011.00178, they tend to find that the difference within each bell-curve are far larger than the difference between the bell curves. That one is neat ‘cause they look at (fuzzy, self-reported) cultural differences as well and find it has a big effect! They look at generic-white-canadian men and women and get one pair of bell curves, then look at south/east-asian men and women and get a quite distinct pair. Sometimes the male/female bell-curves in each culture overlap more with each other than they do with the bell-curves of the other culture! (figure 4 of that paper.) Sometimes the cultural differences outweigh the intrinsic/genetic ones! Sometimes they don’t! It’s wild, and in that paper deliberately quite fuzzy, but the broad strokes make me really wonder about the details.
I am far, far, far, far, far from an expert on psychology, but the conclusions are pretty compelling to me. If you could magically make a perfect test of “technical ability, whatever that means” that corrected for environment/upbringing/etc and applied it to a big random sample of humans, it seems like you’d expect to get two different bell curves separated by gender, but with far more overlap than difference. So then the question is, why has every computer-tech job, class or social venue I’ve ever been in 80-100% male?
So then the question is, why has every computer-tech job, class or social venue I’ve ever been in 80-100% male?
It’s not just computers. It’s that “thing” oriented jobs are overwhelmingly male. “people” oriented jobs are overwhelmingly female. it’s not that women are bad at “things”, it’s that they often have other interests which are more important to them. Or, they have other skills which means that they’re better off leveraging those skills instead of “thing” oriented skills.
For people who suggest that men and women should have similar skills and interests, I would point to physical differences between men and women. Evolution has clearly worked on male and female bodies to make them different. But we’re asked to believe that those physical differences have no affect on the brain? And that despite the physical differences, men and women have exactly the same kind of interests and abilities?
For people who suggest that men and women should have similar skills and interests, I would point to physical differences between men and women. Evolution has clearly worked on male and female bodies to make them different. But we’re asked to believe that those physical differences have no affect on the brain? And that despite the physical differences, men and women have exactly the same kind of interests and abilities?
Obviously there is going to be an effect. We’ve measured that effect. We would probably expect something like a 55:45 spit, based on biological factors. but instead we see something much closer to a 90:10 split, which is way way way more than we would expect. “maybe it has something to do with culture and society” is a pretty reasonable hypothesis.
See “non-linear effects”. The average woman is only a bit less strong than the average man. But the world record for male “clean and jerk” is essentially “pick up the woman doing the womans record, and add 100 pounds”.
Or look at male high school athletes, who compete pretty evenly against female olympians.
Are we really going to say that similar effects don’t exist for hobbies / interests / intellectual skills?
And why does the comparison always include male oriented jobs? Why not point out that 90% of nurses and dental technicians are female? Why not address the clear societal expectations / discrimination / etc. against men there?
The arguments are generally only one way. Why is that?
‘Cause the female-dominated jobs are uniformly the less prestigious and less well-paid ones ofc~ But it’s still a fun question to ask ’cause, do females get forced into less-prestigious jobs, or do the jobs that females tend to prefer become marked as less prestigious? Probably lots of both!
Makes you realize how much we’re still on the tail-end of 15th century social structures, where males got to run things entirely because they could usually swing a sword harder, hold a bigger pike, and pull a stronger crossbow. And how much of that is built in, I wonder? Testosterone definitely makes a person more eager to go out and do something stupid and dangerous for a potentially big payoff.
it seems like you’d expect to get two different bell curves separated by gender, but with far more overlap than difference. So then the question is, why has every computer-tech job, class or social venue I’ve ever been in 80-100% male?
Because the professional technical ability is not about averages, but extreme tail ends. Professional developers/hackers are not average Joe or Jane, but the extreme .1% percent of population that has both ability and interest, to dedicate their life to it. At that point two otherwise very close bell curves are largely disjoint. Example: at point r there are no more red samples, even though the curves themselves are not that far away. In 2024 I thought this phenomenon was largely know. Almost everything in society is not about averages, but extremes for similar reasons.
Yeah but why is there any gendered difference in the bell curve? Natural aptitude seems like a real cop out unless it’s been established and there are no societal mechanisms that reinforce it.
I started doing computers in the mid 1980s, when it wasn’t socially acceptable. My recollection is getting laughed at and mocked by the girls my age, for being interested in such a stupid thing as “computers”. 25 years later, the same people were sneering at me, telling me that “the only reason you’re a success is because you’re a white male”.
I’d contrast my experience with Bill Gates, for example. He had in-depth experience with computers 10-15 years before me. He had connected parents who helped his business. While it wasn’t inevitable that he was a success, he had enormous amounts of opportunity and support. In contrast, I spent my summers sitting on a tractor. My coding was alone, with no help, with computers I bought with my own money.
I get told I’m similar in “privilege” to Bill Gates, simply because of shared physical attributes. I can’t help but conclude that such conclusions are based on ideology, and are untouched by reality or logic.
I started doing computers in the mid 1980s, when it wasn’t socially acceptable. My recollection is getting laughed at and mocked by the girls my age, for being interested in such a stupid thing as “computers”.
And do you remember how much worse that mocking was for girls who were interested in computers? Do you even remember any girls being willing to express an interest in computers when the consensus of their peer group was that it was a thing for no-life boys?
I get told I’m similar in “privilege” to Bill Gates, simply because of shared physical attributes. I can’t help but conclude that such conclusions are based on ideology, and are untouched by reality or logic.
This feels like a reductio ad absurdum. There is a large spectrum of privilege between Bill Gates and starving refugee. Just because you have less privilege than Gates, doesn’t mean that you don’t have more than other people.
What I find offensive, racist, and sexist, is people who lump me into the same “privilege” category as Bill Gates, simply because of shared physical attributes That’s what I said, and that’s what I meant. There’s no reason to conclude that I disbelieve in a spectrum of privilege. Especially when I explicitly point out that Bill Gates has more privilege than me.
This is one of my main issues with people making those arguments. They are based almost entirely in ideology, in reducto ad absurdum, in racism (“you can’t be racist against white people”), and in sexism (“men can’t be raped”).
Even here, I’m apparently not allowed to point out the hypocrisy of those people, based on my own experiences. Instead, my experiences are invalidated, because women have it worse.
So I’m not arguing here that different people have different opportunities. I’m not disagreeing that bad things happen to women / black people / etc. I’m pointing out that the most of the arguments I’ve seen about this subject are blatantly dishonest.
Now maybe social pressure like this doesn’t apply to you and so this is not relatable for you. There’s an easy test: do you wear a skirt in the summer when it’s too warm for trousers to be comfortable?
Without intending to criticize your preceding remarks about social pressure, I find this “easy” test to seem more culturally-specific than you present it. “it’s too warm for [full-length] trousers to be comfortable” itself is not relatable for me, and the men and women I know for whom it is both wear short trousers in that case, and of course in some cultures both might wear skirts.
Now maybe social pressure like this doesn’t apply to you and so this is not relatable for you. There’s an easy test: do you wear a skirt in the summer when it’s too warm for trousers to be comfortable?
No one ever sent a less-competent girl to a maths or programming extracurricular event because programming is a girls thing and so they didn’t consider me. No one ever told me not to bother with programming because it doesn’t lead to appropriate careers.
I frequently hear women talking about how they were encouraged to pursue STEM fields they weren’t very intrinsically interested in, because of the presence of Women in STEM programs in their educational or social environment and the social expectation that it is laudable for women to participate in things that have relatively low female participation rates on feminist grounds. I’ve even heard women talk about wanting to quit STEM programs and do something else, but feeling social pressure - that they would be a bad feminist - not to do this.
This happens to everyone though, male and female. My uncle became a dentist because his father pressured him into it. He hated it, his whole life. There, now we have two matching anecdotes.
There is of course rather a lot of actual science about this stuff. For example, the lightest, most effortless google offers “Gender stereotypes about interests start early and cause gender disparities in computer science and engineering”, Master, Meltzoff and Cheryan, PNAS 2021, https://www.pnas.org/doi/10.1073/pnas.2100030118.
You can’t take the assumption that “there exist programming tools that work well for me” and replace it with “everyone has tools that make it as easy for them to program as it is for me.” You speak English.
You may not even notice the advantage you have because everyone around you has the exact same advantage! Since a competitive advantage over your peers is not conferred on you it is quite easy to forget that there are some people who are still at a serious disadvantage – they’re just mostly shut out of the system where they can’t be seen or heard from.
The connection to feminism is a bit tangential then I think: it’s a lens through which it is possible to see the distortions the in-group effect has caused, even as the effect is completely invisible to most people who have accepted the status quo.
I think there are some valuable ideas in this paper. On the other hand… do we really need to get gender into programming languages? Are we going to have toxic masculinity of language design? Is everything in life about oppression, or do people just build systems in a way that they are useful to them, and maybe a different set of people builds systems in a different way, and in a pluralistic world we can have both and learn from each other?
If I had written this paper, I would not have brought gender/sex into play. You can easily substitute feminism for accessibility or other terms for part of their reasoning, and make this paper useful to programming language designers without evoking a political agenda.
Section 2, titled “Setting the Scene: Why Feminism and PL?” spends 2.5 pages answering your question, and sections 5.1 and 5.2 have more.
To expand on the last paragraph of section 2.4, using feminism allows the authors to build on existing work. There’s dozens of citations from outside of programming, bringing in 50 years of material to hybridize with the familiar.
To your suggested accessibility framing, there’s a recent talk How to Accessibility If You’re Mostly Back-End that hits some similar points, but it’s much more about industry practice than language design. (I saw an unrecorded version at Madison Ruby a few months ago but at a skim this presentation is at least very close.)
To expand on the last paragraph of section 2.4, using feminism allows the authors to build on existing work. There’s dozens of citations from outside of programming, bringing in 50 years of material to hybridize with the familiar.
Yes, yes, this is an essay “escaped from the lab” to justify an academic’s take on the field and to publish a paper.
To expand on the last paragraph of section 2.4, using feminism allows the authors to build on existing work.
The existing work being built upon should arguably be, you know, programming languages and programming, instead of feminist theory. I imagine that’s what many folks will bristle at.
I’ve skimmed this once, and I’ll give it a thorough reading later, but sections like 5.1 and 5.2 emphasize–to me at least–that the target audience of this sort of thing is fellow academics and people already sold on the feminist lens. This is an academic talking to other academics, and we do tend to skew a bit more towards professionals creating artifacts and systems.
I don’t really have high hopes of useful discussion on Lobsters here, since the major reactions I expect to this are either “Oh, neat, a feminist look at programming, okay whatever”, “Hell yes, a paper talking about how unfair we are to non-white non-men”, or “Hell no, why do we have to inject gender into everything?”. To the degree to which any of those are correct, our community’s ability and capacity to discuss them civilly while remaining on topic for the site is suspect.
The idea that feminism is a novel (and somehow intrusive/invasive) political agenda, rather than a lens through which you can view and critique the political agenda we’ve all been raised within, seems to be part of the paper’s basic point. Gender is already implicitly part of programming languages (and all fields of human endeavour), the idea is to recognize it and question if and how (and to what degree) it’s influenced the field. The act of doing so isn’t advancing (or advocating for) a new political agenda, it’s critiquing one that already exists.
BTW, a non-author of this paper swapping “accessibility” for “feminism” here, when the author chose the latter precisely because it is not equivalent to the former, would actually be pretty spot-on example of why adopting a feminist perspective is necessary. Accessibility is about making systems more adoptable to humans with other-than-default access needs irrespective of gender, feminism is about making systems more adoptable to humans with other-than-default gender irrespective of their access needs… they’re literally two topics that don’t overlap except in causing us to critique our “default” way of designing systems; if you think looking at the accessibility bias built into systems is important and/or valuable you probably should think of looking at the gender (and other) bias of those systems is important and/and valuable too.
I have only read the linked article and paper intro (for now), so there might be more, but what seems to be taken from feminism here is the analysis of power structures, social dynamics, shared values and how that all holds back the field by narrowing what research is done.
Reading the intro will provide concrete examples from the authors’ personal experiences.
If the paper was about, say, applying an economic approach to PLT, would you have engaged more deeply to get answers?
I ask this not as a gotcha, but to create an opportunity to reflect on bias.
I personally acknowledge my initial reaction was “why feminism?” but am happy that I also had the reflex of going past that.
I am considerably more willing to believe that feminist critiques are written in good faith than economic ones, and it behooves the reader to understand that neither will represent an “apolitical” perspective.
If the paper was about, say, applying an economic approach to PLT, would you have engaged more deeply to get answers?
Conversely, if it applied a flat earth theory approach, would you engage less? I probably would. Is it wrong to use our past experiences (our biases) with particular fields to determine which lengthy papers we do and don’t read?
Horkheimer described a theory as critical insofar as it seeks “to liberate human beings from the circumstances that enslave them”. – Critical Theory
So the “theory” in the name is already a lie. This is not “theory”, it is politics and ideology.
There is nothing wrong with politics, but please don’t pass off politics as science. And in particular, don’t try to make this non-science the arbiter of all the other sciences. Yeah, I know that this “theory” claims that all the other sciences are actually not scientific because they are just power grabs. Well, that’s just projection.
You, coming to these comments to tear down the work of others published in the same proceedings as your own, by calling it “non-science” and “not far away” from flat earth-ism is demonstrative of the bare misogyny that this paper is asking the audience to start taking notice and accept less of. Stop it.
On the other hand… do we really need to get gender into programming languages?
When we exclude spreadsheet from programming languages, I think we already have to some extent: one big reason it’s excluded is because spreadsheets are perceived as not as prestigious as “actual” programming. And I bet my hat one reason it’s not is because spreadsheets are generally a secretary’s tool. Female secretary most of the time.
There used to be a time where computers were women (or “girls” as the men around them often called them). With the advent of the automatic computer, a good deal of those women turned to programming. And for a time, this role was not that prestigious. Over time it did became so, though. And over time we did see a smaller and smaller share of women going into programming. Coincidence? I think not.
Anyway, giving “programming language” status to spreadsheets would elevate the status of secretaries to programmers, and “real” programmers can’t have that. Hmm “real programmer”. Why do this always conjure an image of a man in my head? You’d have to admit, the XCKD über hacker mom isn’t your stereotypical hacker.
I think the simpler and more correct explanation is that spreadsheets dominate other sectors and industries (engineering, medicine, hospitality) so thoroughly that it simply never occurs to most programmers that they’re a valid programming environment.
This is also why I’ve seen an MBA beat a bunch of programmers’ asses using only pivot tables and sheer stubbornness.
I bet my hat one reason it’s not is because spreadsheets are generally a secretary’s tool.
Spreadsheets, in the context of programming (generally excel) are coded as management tools or more generally “business people”. Not by gender (these are, more often than not, men as well, although probably not quite as male dominated as programming).
When we exclude spreadsheet from programming languages, I think we already have to some extent: one big reason it’s excluded is because spreadsheets are perceived as not as prestigious as “actual” programming. And I bet my hat one reason it’s not is because spreadsheets are generally a secretary’s tool. Female secretary most of the time.
Do we exclude spreadsheets? Microsoft claimed excel was the most popular programming language in the world in 2020.
Excel isn’t present at all in the technology section of the 2024 Stack Overflow survey results. And that isn’t even specifically a list of programming languages; the page has several categories of technologies, including an “other” category. So while Microsoft may rate Excel—and to point out the obvious, they have a financial interest in doing so!—I don’t think that’s necessarily a widespread view.
The original question was whether Excel is a PL in the context of PL research. And in that context, I think it’s especially obvious that it is. It has variables (called cells), and loops. It’s Turing complete, and not in a gotcha-kinda way. Excel only has user-defined functions defined in another language, but Google Sheets has user-defined functions defined in the same language. It has control flow, made interesting by cells referencing each other in a DAG that needs to be toposorted before being evaluated. It has compound data: 1D and 2D arrays.
You could absolutely write a small step semantics for it, or a type system for it, and neither would be trivial. In fact I’d like to read such a paper for Google Sheets to understand what ARRAYFORMULA is doing, there’s some quantifier hiding in there but I’m not sure where.
Oh, I do think that Excel is a programming language! I realize that my comment didn’t make that clear at all. I was trying to push back on the claim that spreadsheets are commonly considered to be programming languages. I think Excel is a PL, but my impression is that there aren’t a lot of other people who think that.
I think I disagree on the widespread view comment. Anecdotally most people I talk to agree that excel (or spreadsheets more generally) meet the definition of a programming language/environment. I would argue that the community of users on stack overflow is not representative of the broader population and the population choice is the crux of the issue here.
Maybe it’s just because I’m young, but the first introduction to “historical” programmers I had was Ada Lovelace, and later then Grace Hopper.
Honestly, I can’t say I even necessarily have a generally positive perspective on the “typical hacker” persona - maybe partially because there are some like RMS who come with a lot of baggage.
That is the fun bit. This totally eclipse the reality of the history of our field.
I recommend a tremendous book, “Programmed Inequality” by Mar Hicks, for a historian work on some of this. It is fascinating and may help shed some light on the topic and the lens.
spreadsheets are generally a secretary’s tool. Female secretary most of the time.
Maybe this is true in your culture, but do you have evidence for this? I have no evidence, only my perceptions.
My perception, in my culture, is that spreadsheets are stereotyped as being for “business people” in general, or for very organized people, or for “numbers-oriented people”.
My perception, in my culture, is that “business people in general” and “numbers-oriented people”, and maybe “very organized people”, are stereotypically male, i.e., that women are stereotyped as less likely than men to be in those groups.
Although secretaries exist in my culture, I perceive their numerousness and cultural prominence as being low now, much decreased since their peak in the 20th Century, I guess because computers and economic downturns made businesses decide that they could do with many fewer secretaries.
When we exclude spreadsheet from programming languages, I think we already have to some extent: one big reason it’s excluded is because spreadsheets are perceived as not as prestigious as “actual” programming. And I bet my hat one reason it’s not is because spreadsheets are generally a secretary’s tool. Female secretary most of the time.
TBH, I figured it was excluded because there is one hyper dominant spreadsheet implementation (Excel) that was created by a much maligned company (Microsoft).
Though I suppose that might be why there is one hyper dominant implementation. If people were more interested in spreadsheets we might have a lot more options and genomic studies would be safe from coercion errors.
With header files don’t you have to write all your type signatures and type definitions twice? I hated them when I worked with OCaml.
It makes more sense to just implement the described feature in IDEs/editors, instead of making every programmer copy/paste their types/signatures to a separate file.
Dunno about OCaml, but in C public types go in the header, private types in the implementation, so all that gets duplicated is the function prototypes.
It might be nice to allow function definitions to be abbreviated, so the types don’t have to be repeated, but on the other hand it might be bad for the full prototype to be so far from the body. On the gripping hand, I usually have both files open so that I can make sure the implementation and documentation match.
But my prehensile tail points out that having the documentation, type signature, and body right next to each other in one file is a lot more convenient, and rustdoc does a fine job of pulling out and presenting just the public interface.
Yes, header files can totally serve this purpose. They provide a nice overview of what’s the public interface of a module.
And, in the spirit of improving UX (which is what this post is about), i think it would be crucial to have seamless integration between header files and implementation files. E.g., it should be trivial to toggle between them; if i “go to definition”, the definition on the implementation file should be open; if i edit the signature of a function in the implementation file, the change should be made in the header file automatically; etc.
But, if we had all this, then the header file would become almost a “projection” of the implementation file. In practical terms, it’d be almost a modified view of the same source code. Something that could be entirely derived, but for some reason still exists as a separate source file.
I.e., header files with good UX would be something very very similar to the feature this post is proposing; minus the need for duplicated information in two different files :)
As @fanf mentions, the article of which we’re talking touches on this. But, if this is a genuine question(*), i also commented on why the article’s proposal is IMO superior to an “outline” view elsewhere on this discussion :)
*: I initially read it as a kind of slight, “don’t you know this has been a common thing for decades?”, but you probably didn’t mean it that way. Or at least i hope so! hehe
I don’t understand, the post doesn’t mention at all the IDE’s existing “outline” feature which already does everything mentioned here. Like when I open a code file, I have a separate panel which tells me all the methods in a file with their parameters, etc. and this has worked pretty much in every IDE I’ve ever worked in. e.g. this : https://ibb.co/48nXwn4 ; this was already a thing in dev-c++ in 2005.
Header files have the documentation and the complete public types. That outline looks fine for navigating a source file, but it isn’t a substitute for a header or for Matklad’s folded view.
But, if we had all this, then the header file would become almost a “projection” of the implementation file. In practical terms, it’d be almost a modified view of the same source code. Something that could be entirely derived, but for some reason still exists as a separate source file.
Ideally, (part of) the implementation would be a projection of the header, because you really want the interface to be primary. And no, I have no idea how to implement that, at least not one that’s also in any way reasonable.
(One company that I worked at did automatic header generation as a matter of course. It worked. It was…hmm…let’s just say I adopted many of the habits and practices from there, but not that one)
Yes. It’s good that languages are less redundant nowadays, but we definitely lost some nice ergonomic and information-hiding aspects of headers.
When I was working in C# back in the 00s, I invented a “header file” format for C# (basically the public/protected class and function declarations with no bodies) and made a little tool that generated it from a DLL.
I think we did have Intellisense in Visual Studio back then, but that just shows one method at a time, whereas this let you see the full picture of a library (or your own code).
My main problem with header files is that any refactor that involves changing a function’s signature requires changing it in two places, which is annoying.
Objective-C has this: Smalltalk and C. And the overhead between the two parts was essentially nil. Pretty awesome when you manage to take advantage of it.
I always struggle with the YAGNI concept because, in my experience, writing generic code is not more complex than writing over-specialised code and can often lead to simplifications for the specific use case. I then often end up being able to either copy code from one project to another, or factor it out into a library that both can depend on.
The one constant in software engineering is that your requirements will have changed by the time you’ve implemented them. Writing code that’s flexible in the presence of changing requirements is software engineering.
I like to distinguish “generic” from “general”, whereby “generic” means “independent of irrelevant details” and “general” means “handling many cases”. I believe in YAGN generality, but like you, I value genericity.
I tend to break YAGNI defensively. If, early in the implementation work, my gut tells me that the current “thin” specs are the result of an inability to formulate use cases in sufficient depth on the product management side, then there’s a very high chance that I am, in fact, GNI. In that case, I will totally write the more generic version, because there are two things that can happen one year down the line:
We actually NI, and most of it is already done.
We don’t actually NI and, worst-case scenario, I spend an hour on a refactoring that drops features, rather than several hours implementing and testing new features, in a module with high potential for regressions, on a tight deadline.
What if I don’t have time for that refactoring? Haha. If I can’t find an hour for a simple feature-dropping refactoring, I definitely won’t find the way more refactoring hours required to morph the code that implemented what we thought we needed a year ago into the code that implements what we actually understand we need today.
The undelying assumption in the “but what if you don’t have time to refactor it?” is that, as more requirements are formulated, code only needs to be “extended” to cover them, you just “add” things to it. That’s one of the worst sermons of the engineering management cargo cult.
Also, code written by successive generations of YAGNI people is some of the worst there is. It never evolves into a generic solution, it just snowballs into umpteen particular solutions held together by some inconsequential generics.
For me, YAGNI is not necessarily saying you shouldn’t write generic code. I’ll try to explain with an example from my team’s current project. We’ve been writing a program to manage hardware and the software running on it because none of the infrastructure management software we evaluated fit our needs well enough. So we have a daemon on our edge hardware and a service running in the cloud that coordinates that hardware and hands out configuration. That service connects to a database, which in my team’s case is Postgres. The engineer who wrote the initial draft of this system, however, added in a constant and some code to be able to work with any SQL database. However, it is highly unlikely that we would ever use a database besides Postgres since we’re already using it. This is the sort of thing YAGNI is about - you’re working on a team that has infrastructure in place, you know what you’re writing is internal and not meant to be consumed/built on by others. Why would you put in the work to support different types of infrastructure than you need to?
So yeah, requirements will change, but what you’re building on top of usually will not. So don’t try to support more than what your team uses unless you know with certainty that you need more.
The deciding factor for me is usually indirection. If the more specific approach lets me avoid a layer or two of indirection, I’ll usually pick that. If it doesn’t, I’ll go with the broader version.
At my previous job, I saw a feature added that wasn’t once used in the 11 years I was there. It stared out as a way to segment SS7 traffic to different backends, and a later addition of SIP had to support the same concept, yet it was never used. It was discussed from time to time to use it, but never came to fruition. And it would take more work, and coordination with another development team and ops, to fully remove the feature.
writing generic code is not more complex than writing over-specialised code
I think YAGNI is against adding feature or making code more complex in anticipation of future value. I have never seen anything against just making the code just as simple (or even simpler) by using a more general version.
I’ve heard this called “implement for today, design for tomorrow” and I think it is fundamentally at odds with YAGNI. The problem is it is (by definition?) more complex to read, and therefore maintain, tomorrow. There’s a quote about it:
[YAGNI] is making a bet. It is betting that it is better to do a simple thing today and pay a little more tomorrow to change it if needs it, than to do a more complicated thing today that may never be used anyway.
The quote is from the XP book, but I think in the context of this thread we can safely replace the word with YAGNI. Have you made the bet in the past and it hasn’t paid off, or have you just not tried it yet?
The bet is also that when you do need to make that change, you will have more information, so you will be able to do a better job then than you will be by anticipating those needs now.
I was turning “micro services” back into a well-structured monolith back in 2003, turned a set of ~100 processes into a single Java JAR by 2005.
Improved performance and reliability by roughly 100x. Number of services provided increased, code decreased. Development velocity dramatically increased as the previous tangle was unmaintainable.
Used REST principles and a few well-chosen interfaces to clearly separate the modules within the monolith. Wouldn’t use an IPC mechanism, though interesting that they thought this was necessary.
So far, I would say it has raised the floor rather than raised the ceiling. It’s much easier to generate code that has already been written by 10,000 other people 10,000 times by just describing it. Getting it to write something no one has ever written before or even very few people have written is still quite difficult.
One thing I think it helps with is turning specialists into “T shaped” coders. Many projects involve some creative and/or intricate core surrounded by a whole bunch of common yet multi disciplinary stuff (infra, deployment, website, app front end/backend, database, etc). This helps you turn a proof of concept into a full-blown project.
I think this gets closer to the mark. I’m by some accounts a good programmer. The floor hasn’t been raised for me, but it has made me a lot more productive. LLMs are taking a lot of the effort out of working out ideas, writing initial concepts etc. and while I’m still crossing the finish line with code that I wrote myself LLMs are making the easy parts faster.
So the problem I have with this is that we shouldn’t really be writing code that has been written before by 10000 other people 10000 times, we should be reusing it, right? Other than for educational purposes in which case having it generated for us is not going to exactly be the right thing either.
To me, the fact that GenAI does appear to be useful in these case in software development seems like a huge failure, and one that should be fixed. And GenAI doesn’t help fix the failure but rather helps entrench it. We really need to look at why we have failed in this way.
There are other use cases for which it can be quite helpful, I think, but this ain’t it.
The context of io_uring is a process that has lots of concurrent IO in flight per thread. The kind of thing that traditionally was built around select() or poll(), or more recently kqueue or epoll. Event-based systems like node.js.
The core loop of this kind of program has a lot of back-and-forth between the kernel and userland because the kernel doesn’t remember much about what the process is doing from one system call to the next. It’s like,
u: can I do IO?
k: yes!
u: ok, do the IO
k: done
u: can I do IO?
k: yes!
u: ok, do the IO
k: done
The basic idea of io_uring is to make it more like
u: here’s a list of IO to do
k: I have done some of it, what’s next?
u: here’s some more IO to do
k: done more of the 1st list and some of the 2nd, what’s next?
That is, the program can submit a batch of IO for multiple file descriptors (network connections, listening sockets, disk files, etc.) and get pending results from previous batches, in one go. Instead of an IO syscall per file descriptor per loop, there’s just one syscall per loop. In effect, the program has multiple read()s and write()s in flight concurrently.
The “ring” refers to how the list of requests and results are laid out in memory. In more advanced uses of io_uring, the program can set up the ring buffers so that it can add requests and read results without explicitly handing off to the kernel, so its main loop can have zero system calls.
As ~david_chisnall said this setup is like the way a paravirtualized kernel talks to its hypervisor. It’s also like the way a driver talks to a device that does DMA. Which is why ~matklad complains about blocking APIs being wrong: io_uring is much closer in design to the way the hardware works.
At the core, it’s a ring buffer that’s mapped into userspace. Rather than issuing system calls to initiate operations (typically I/O, though it’s grown to support some other things, so the io_ prefix is somewhat misleading now), you put them into the ring buffer. If you’ve used a system such as QNX with asynchronous system calls, this is probably quite familiar. It’s also likely to be familiar if you’ve ever programmed against a hypervisor interface: domain transitions to a hypervisor are (or, at least, were) much more expensive than system calls and hypervisors were typically designed to support multiple cores, so they used ring buffers with asynchronous communication for most device interfaces.
There’s also a lot of stuff layered on top. The most interesting bit, from my perspective, is that the io_uring equivalents of most system calls that would create a new file descriptor can (optionally) not register that file descriptor in the process’ descriptor table. This is useful because it means that you can work around one of the annoyances of POSIX. POSIX mandates that each new file descriptor must be created at the lowest free file descriptor number. This was originally a clever hack in early UNIX to allow the shell to redirect standard in/out/error (the first three file descriptors) by simply closing them and then opening something else. It was obsolete when dup2 was invented (though necessary for legacy compatibility). It was painful when threading came along because it made every open (or socket, accept, pipe, or whatever) and every close global synchronisation points within a process. With io_uring, you can avoid this and make the operations local to a ring (on the kernel side, you can have a single thread that processes the ring state and dispatches to kernel work queues for the lon-running things). This means that you can batch up an open, read, close sequence and have them operate with no synchronisation with the rest of the process state.
The ability to batch commands is what made io_uring interesting for other things as well, such as process creation.
QNX syscalls aren’t async. The main messaging primitive, MsgSend, is tightly coupled to the scheduler in a manner that makes it appear to the caller almost as a vanilla function call. Maybe pulses are, I can’t tell from what @BenjaminRi brought up the last time this discussion came up: https://lobste.rs/s/a7hvcs/qnx_demo_disk_extending_possibilities#c_haq7v1
Ooops, I’ll try to remember that next time! I never used QNX, but I remember their marketing saying that they were faster because they did async system calls. I wonder what they were talking about.
Well, it’s a tool for low-level programmers, so if that’s not you then it’s fine to ignore it. That said, I’ll try a higher-level explanation:
Normally, when you want the kernel to do something, you issue a system call. On modern systems this is a dedicated instruction, but traditionally it was a software interrupt. When this happens, the CPU jumps to a specific address and switches to executing in kernel mode. These happen together because you don’t want to switch to a more privileged mode and give unprivileged code control over what runs.
The speed of this switch has changed a bit over time. It used to be very slow. Then CPU vendors made it faster. Then they discovered that a bunch of the things that made it fast led to side channels that let userspace processes leak kernel data, so now it’s slower again.
When the kernel enters the system-call handler, it has to save some state for the userspace process, work out which system call was invoked, and do a bunch of other book-keeping work, then do the thing userspace asked for.
There are two sources of overhead here. The first is that you need all of this book keeping for a single operation. The second is that it’s a synchronous transition and so the kernel also has to synchronise what it’s doing.
Xen tried to avoid this for all guest VM I/O and exposed devices as ring buffers. A ring buffer is just a block of memory that you pretend is arranged in a circle. When you get to the end of the block, you go back to the start. Each entry was filled in as a request in the guest and then replaced with a response from the hypervisor. This meant that you could put a load of requests into the ring without explicitly invoking the hypervisor at all. The only time you needed to do anything explicit was when the ring transitioned from empty to not-empty. This let the thread that was handling the I/O outside the guest[1] sleep when the ring is empty and wake when there’s something to do, and then keep running as long as requests are coming in.
If you have one core, the guest would batch up a bunch of things, then the threat that implemented the device would run for a bit and process some or all of the batch, then the guest would run again. That’s more efficient than a transition for each operation, but it’s an even bigger win when you have multiple cores because both sides of the ring can be running at the same time and the guest can do other work at the same time as the device thread runs.
This is closely tied to the other advantage. Most I/O is intrinsically asynchronous. When you read from disk, the kernel might have a copy of the data in the buffer cache and be able to just copy it to you, but it may not. If not, it doesn’t (at least since the kid ‘90s) sit there reading data, it sends some commands to the disk controller, which then uses direct memory access (DMA) to write the requested blocks into memory somewhere. Similarly, if you’re writing to disk, the network, or some other device, you,just set up the operation and then it runs while you do other things. It’s useful to be able to expose this higher up the stack. In Xen, a guest would write a bunch of blocks to disk by queueing them up in a device ring, then the device emulator thread would run and tell the disk controller to write them to disk, then the CPUs are free to do other things. When the disk controller is finished, the completion is threaded through the other layers and they can reuse the slots in the ring buffer for new requests.
The io_uring model is similar to this, except it deals with the kinds of things userspace processes operate on, rather than the things VMs deal with. It has sockets, files, and so on, rather than block devices and Ethernet devices. The advantages are the same though. You can queue up a bunch of work and the kernel can handle it asynchronously while you do other things.
[1] In the case of Xen, this is actually in another VM, but that’s not relevant for the purpose of this explanation.
Most syscalls consist of three separate steps, which as far as userspace is concerned, happen atomically (because the calling thread is blocked for the duration of the syscall):
Ask the kernel to start doing the thing.
Wait while the kernel does the thing.
Get the result back from the kernel.
io_uring allows applications to do the steps individually, and to do other things between the steps. In particular, if an application wants to do syscalls A and B , it can do the steps in the order: A1, B1, A2, A3, B2, B3. Applications can also wait on any of their submitted requests and deal with the results in whatever order they happen to finish.
Separating the request from the response also allows you to batch multiple requests into a single syscall. Hardware is so fast nowadays that syscall overhead can often be the limiting factor, as I discovered a while ago:
Communicate with kernel threads using atomics instead of switching into kernel mode yourself? In the limit, only switch into kernel mode when you’ve run out of other stuff to do, and you want to sleep until stuff arrives.
At some point I need to write an actual article about this, but I’ve recently been thinking that “blocking” APIs in general are a design mistake: everything is fundamentally concurrent, but OSes use blocking APIs to hide this concurrency, which tends to inevitably break, and mixes concurrency with parallelism. Even something as simple as “spawn a process, collecting both its stdout and stderr” requires proper concurrency. See how Rust’s Command::outputcan’t be expressed using std-exposed APIs.
Which is the smaller problem! The bigger problem is that we don’t actually know how to write concurrent programs. This is an unsolved area in language design. And the systems reason for why the problem is unsolved is, thanks to the existence of blocking APIs, languages are allowed to shirk here, moving what is fundamentally a language-design issue to the OS-design.
IO-uring (and previously, hilariously, JavaScript) models the problem of concurrency more directly, without introducing a midlayer. Hopefully it will create enough of evolutionary pressure for language design to get the problem solved finally.
There was an interesting period about 25 years ago, when the web was growing fast and really stressing the capabilities of the hardware and operating systems of the time. This is when the c10k problem was identified. The performance pressure led to a bunch of interesting operating systems research.
I’m pretty sure I read a paper around that time which described a kernel/userland API with a design roughly along the lines of io_uring — though its point of comparison was hardware interfaces with command queues and DMA buffers. I had a very vague memory of Vivek Pai’s IO-Lite but that has a different shape so there must be another paper I’m failing to remember.
I wondered if a fully asynchronous kernel API could solve the M:N threading problems that were happening around the same time. But that whole area of research dried up, as the problems were dealt with by sendfile(), 1:1 threading, and computers getting faster. Boring!
It was really hard to handle 10,000 concurrent connections from one server in 1999. The challenge was to make it easier, to make the kind of performance engineering that went into cdrom.com available to a lot more systems.
IIRC, the fundamental challenge is handling 10K long-running connections that are not very active. If each of those connections is actually producing significant load on the hardware, either CPU load or I/O load, then the hardware can’t handle 10K connections anyway, because we have neither 10K cores nor 10K independent I/O channels.
I found a copy of the SEDA SOSP paper which is a bit shorter than Welsh’s PhD thesis :-) It seems to be about getting more concurrency out of Java, not about changing the userland/kernel API.
It’s a compelling argument, but I think in some ways async APIs end up constraining the design space of the framework/OS/whatever offering it. For example, QNX’s main message passing primitives are blocking so that, on the fast path, context is switched to the callee without a pass through the scheduler.
You absolutely hit the nail on the head. Given how absolutely fundamental asynchronous computation is to how computers work, it’s surprisingly underexplored.
I think this is partly due to a decline of groundbreaking OS research, and partly because there are barely - if any - modern languages purpose built for doing deep, systems research. The problems become apparent almost instantly. At the latest when you’re a few hundred lines into your first kernel, you’ll be presented with DMA interfaces for all kinds of peripherals. What now?
This is not the kind of problem that most language designers are facing. Most PLs are made to solve application-level problems, instead of having a principled, hardware-first approach to language design. I think Rust with embassy-rs has been solving some of these challenges in a pretty nice way, but it shouldn’t end there.
I agree on the lack of OS research. Mothy gave a keynote at SOSP a few years ago pointing out that twenty years ago there were multiple new kernel papers each year at OSDI / SOSP whereas now we average less than one. I’m slightly biased, because the last paper I submitted with a new OS design was rejected on the ground that we didn’t do a performance comparison to an OS that required ten times as much hardware and that we were possibly vulnerable to an attack that, by construction, cannot happen on our system. Ho hum.
I disagree on language research though. There’s a lot of interesting work on asynchrony. Most PL research takes at least ten years to make it into mainstream programming languages. Verona’s behaviour-oriented concurrency model is designed to work directly with things like DMA and we’ve ported it to C++, Rust, and Python. It works best with a memory management model that puts object graphs in regions and allows them to be atomically transferred, which depends on a type system that can do viewpoint adaptation. We’re not the only people working on things like this and I expect mainstream languages in the 2030s to have a lot of these concepts.
Mothy gave a keynote at SOSP a few years ago pointing out that twenty years ago there were multiple new kernel papers each year at OSDI / SOSP whereas now we average less than one.
That’s brutal. Too brutal in my opinion to only come from spurious rejections. From the outside, I would suspect the main drive is that nobody makes kernels any more: apart from specific niches like real time or formal verification, the world has basically been taken over by NT, Linux, and XNU. And a major reason behind such concentration is that every kernel needs to have dedicated drivers for the insane diversity of hardware we use — except NT, hardware vendors being business savvy enough to write the driver for us. (See The 30 Million Lines Problem.)
Those three kernels are big. Making any addition or change to them is likely a significant endeavour. We likely need the ability to quickly write practical kernels from scratch if research is to take off again. But no one can possibly write a practical kernel if they have to, let’s be kind, port all the driver code from Linux. So the first step has for hardware vendors to design interfaces for humans, and then give us the manual. But given how suicidal this would be business wise (who would ever buy hardware that doesn’t come with a Windows driver out of the box?), we probably need regulators to step in: say the US, or the entirety of the EU, ban the sale of any hardware for which the same company distributes software — or at the very least, proprietary software. With such a setup, I would expect hardware vendors to quickly converge on relatively uniform hardware interfaces that aren’t stupidly complex.
But then there’s the other side of the fence: user space. Since the ossification of OSes we came do depend on a huge stack of software that would take significant effort to port anywhere not POSIX-ish. And that’s for the stuff we have the source code of…
I don’t really think that’s true. For a research project, you need a few drivers and most people just pick up the NetBSD ones (the RUMP kernel work makes this trivial). There’s a big unexplored space for both big and small operating systems. With a very small team, we were able to write an RTOS that has a solid FreeRTOS compat layer and can run most existing embedded software (source compatible) and we were designing and building the core that ran it as well. No one has really built an OS for cloud computing, but it could remove a lot of stuff that desktop operating systems need (no IPC or storage layers, since those will both sit on top of a network).
fwiw I think existing unikernels fit that bill (e.g. unikraft, nanos). They mostly target virtualized hardware like KVM, and seem pretty focussed on the cloud use-case (fast boot, small images, and efficiency). Being in KVM-land certainly removes a lot of implementation complexity, since bringup and platform init is mostly done.
I disagree. I think unikernels are applications for cloud computing. They’re hampered by the fact that the OS that they run on (Xen, Hyper-V, Linux) is not designed for the cloud. They have network devices, for example, but a good cloud OS would make it easy to take advantage of hardware TLS offload and terminate connections directly in a unikernel.
Unikernels have had almost no commercial adoption precisely because there isn’t a good OS to run them on. The benefits of unikernels are incompatible with the billing models of cloud systems. A unikernel invocation may consume a few MiBs of RAM and a few CPU seconds (or hundreds of CPU milliseconds). The operating systems that run them are not set up to handle this at all. They are intended to account for CPU usage in minutes and RAM in GiBs.
If you’re running a unikernel in KVM, there’s little benefit in it over just running a Linux process. You then get to share a TCP/IP stack and so on with other processes and amortise more costs.
Ah then I misunderstood your original comment. So it sounds like you’re unhappy about the currently dominant hypervisor abstraction, that I would agree with. The interfaces for efficient resource sharing are pretty borked.
A unikernel invocation may consume a few MiBs of RAM and a few CPU seconds
Are you talking about serverless here? Unikernels can be long-running services, at least that’s how I’ve run them in the past.
For me, the big benefit of unikernels is their ability to scale down. If you’re handling sustained throughputs of thousands of network messages per second or more, there are some benefits, but if you’re implementing services that have long idle periods of no activity and bursts of very high usage, unikernels’ abilities to either exit completely and then restart in one network round trip time or scale back to almost no resource usage is a huge win. Cloud billing systems are not set up for that at all.
I don’t really like the term serverless, because you don’t get rid of servers. The way most FaaS services are implemented today comes with ludicrous amounts of overhead. Each service starts in a lightweight Linux VM, that runs an OCI container, that runs a language runtime and some support code, which then runs the customer code. We estimated an efficient system with good hardware-software co-design to do build an attestable base with TLS flow isolation to individual services would allow at least an order of magnitude denser hosting and provide better security properties.
Thanks for the link. In your view, would the BoC as a concurrency paradigm allow for a competitive implementation for e.g. performance-critical embedded systems? Are there places where you’d say “this place needs an escape hatch to close the last few percent perf gap” or so?
I’m genuinely interested, because I was looking for things beside stackless async/await that would work well for highly constrained environments.
Possibly. BoC is really equivalent to stackless coroutines, but with a coordination layer on top and an ownership model that makes it safe. Each when captures the things it need to complete and runs without the parent’s stack.
The problem for BoC on resource-constrained systems is that any behaviour can spawn arbitrary numbers of other behaviours, which can exhaust memory, so you’d need to be a bit careful. It’s probably no different to avoiding deep recursion though.
You definitely wouldn’t want to use the Verona runtime for small embedded systems (it’s nice in a kernel though. I ported it to run in the FreeBSD kernel a few years ago) but I have been pondering what an embedded BoC runtime would look like.
I think this is partly due to a decline of groundbreaking OS research
I mean, utah2k was written nearly a quarter-century ago. (There is a certain irony of someone from Bell Labs writing this, considering the output of Bell Labs doing a lot to slow systems research…)
In the 1970s, this was addressed.
Quite a few popular OSs back then treated all base I/O calls as async.
Blocking was a unix thing.
Out of necessity, the Berkeley unix apis on unix tried to walk that back. (They also hacked a few other unix design choices)
Programmers on those older OSs successfully built multiuser database and timesharing systems.
For simple programs, many liked the simplicity of the unix style,
IMNSHO, the underlying problem is that we decided a long time ago that the procedure call (subroutine call, function, method, etc.) is our fundamental model of abstraction. Not just for interacting with the operating system, but pretty much for everything. (“Decided” might be overstating it…it just sort of happened for various reasons, many good at the time).
I’m not 100% convinced that concurrent-only is the way to go (and I say this as a JS dev who also uses Javascript as a good example of what concurrent-only could look like). But I agree that most languages need to choose one or the other. You either go all in on concurrency (like in Javascript, where synchronous APIs are rare and usually only exist a special-case exception alongside an equivalent async API), or you accept that concurrency is always going to be an opt-in feature with less support that will require its own way of doing things.
I’ve seen this a lot in Python, and I think we’re seeing it now again in Rust, where async support gets added, and creates a split ecosystem. You can’t mix and match, so you need to decide: am I using the sync APIs or the async ones? I wonder if there are other concurrency ideas that could mitigate this somewhat, but I think it’s just a fundamental divide. Reading files, interacting with sockets, executing processes etc are fundamental OS-level tasks, and deciding how you’re going to do those cuts to the core of a language’s standard library. In Rust, I know that some of the big proponents of the Async Way are aware of this problem, but I think their goal is to push async as the predominant way, and I’m not sure Rust is set up for that yet.
(As an aside: I know C# also was synchronous and then added the async keyword - are there similar issues there? Is there a divide between the sync and async worlds?)
The thing is, you can’t choose only one! Here’s a mundane, boring example:
You spawn a process. You want to collect its stdout and stderr. You can’t do just
let stdout_bytes = child.stdout.read_to_end();
let stderr_bytes = child.stderr.read_to_end();
because this might deadlock. The child process might interleave printing to stderr&stdout, so you have to interleave reading them. What you want to do here is to issue both read syscalls concurrently and wait for one of them to finish. So you end up writing platform specific code like this:
Another case you always hit when programming mundane things is that something does a read call, and you really want to “cancel” that read (even if it currently blocks). This again necessitates pretty horrific work-arounds with installing a process-global signal handler: https://github.com/rust-lang/jobserver-rs/commit/4c8a5985e32fa193350f988f6a09804e565f0448
In the opposite direction, DNS (and, until io_uring, even file IO) — there’s just no non-blocking APIs for these!
So, I don’t think it’s up to the languages to fix this, really. It’s the OS which first needs to provide reasonable API. Only then can language designers of production languages figure out how to express the concurrency in the source code (to echo David’s comment, I suppose we might have figured how to deal with concurrency in academia, but this certainly haven’t quite percolated to day-to-day languages. Go & Rust seems like they are improvements, but also both feel quite far from optimal).
There are plenty of nonblocking DNS APIs. The gap is in the higher-level name service APIs that bundle up lots of non-DNS name databases, such as the hosts file, mDNS, NetBIOS, NIS, etc.
You can’t perfectly choose one, but I think you can abstract surprisingly well by choosing one route or the other. As you point out yourself, we’ve done pretty well pretending that things like file IO (and even memory access) are synchronous, idealised things. Yes, there are weird edge cases where the abstraction breaks, but most of the time it holds surprisingly well.
Going the fully async route brings its own problems, but it’s still in many ways just a different abstraction. Many async libraries handling file IO just run synchronous APIs on a different thread, because it’s simple, it works, and we still get the concurrency we were expecting.
So I think a lot of this is about the abstractions we chose to use on top of these sorts of APIs, and I think language designers have a lot of leeway there. Yes, the OS APIs will affect those abstractions - what works well, what works efficiently, what leaks constantly into the abstraction layer etc - but there is no shortage of interesting approaches to concurrency on top of largely synchronous APIs.
Many async libraries handling file IO just run synchronous APIs on a different thread, because it’s simple, it works, and we still get the concurrency we were expecting.
Until recently, Linux didn’t have usable async file IO, so you had to do that.
10 minutes of just build time would be too much for me…
Yeah, 30kloc really isn’t very big… I regularly work with a ~14 kloc C++ code base which builds in 15 seconds on my 3.5 year old laptop, and a ~20 kloc code base (+ vendored catch2, + auto generated protobuf code) which builds in just under a minute on the same laptop. And C++ isn’t exactly known for its fast compile times either. 10 minutes is an eternity, and I imagine Apple gives its developer significantly more powerful machines than my 2021 M1 Pro.
Yeah, 50 lines per second.
Swift has ridiculous compile times.
60 MB binary also seems excessive for 30KLOC.
Only 4x faster than (interpreted) Python 2 also seems a bit on the slow side for the a compiled language. Unless it’s all DB access or other I/O.
I want to add: the auto generated protobuf code is 39kloc across 24 source files. I didn’t know it was that much code. So in total it’s 59 kloc, twice as much as the Swift code base in the article.
That is kind of long! But I assume that’s a clean build; hopefully incremental builds are a lot quicker.
Compile times are my main complaint. If not for that it would be my clear favorite language.
Is this worse than Rust?
I’m compiling 12,000 lines in five seconds on an ancient Mac Mini. No dependencies outside the standard library though.
I can’t find build time benchmarks but both communities are of the opinion compile times are not good enough. In this project’s case I suspect the ten minutes includes pulling their dependencies and building them too, which is the norm with SwiftPM.
This post makes the same mistake as part 1: it takes “nice to have” optimizations as a given, as something that has priority over everything else.
Nowhere does it discuss the obvious alternative: those “seemingly” correct optimizations are not, in fact, correct. And thus need to be abandoned.
As Ralf responded to that post, it takes those optimizations as a given because that is how existing compilers work, and the “obvious” alternative is far away from the one we live in today.
Discussions of pointer provenance always seem to have someone show up to make this argument that the whole system is just wrong and should be remade from scratch. But the people who make this argument never seem to have anything to say about the reasons compilers and language specs for C-like languages are this way, or how their preferred approach might tackle those problems.
It would be much more interesting to discuss what this alternative approach would actually look like. Presumably very few people would be satisfied with merely falling back to LLVM or GCC’s -O0 output- so which optimizations should compilers perform, and how can they be justified?
The entire post is a discussion of which of the three optimizations should be deemed incorrect.
Do you feel the same way about the strict aliasing rules? My understanding is that those were added back when C was competing for performance with Fortran. The performance penalty for “just doing what the programmer wrote” has only gotten larger since then.
I think the case for this with regards strict aliasing is actually stronger. That is, we know which optimizations need to assume strict aliasing and how much performance they win you - not a terrible amount, it turns out. That’s why GCC and Clang have -fno-strict-aliasing and why Linux uses it but is still plenty fast.
Provenance is much more central to how modern compilers work, its assumptions are so baked in that you can’t even tell these compilers “-fno-provenance” and see how much speed you lose.
That is, we have usable production compilers that don’t need strict aliasing, we don’t have any that don’t need provenance. So saying “strict aliasing is wrong” is a tenable position, and indeed Rust doesn’t have strict aliasing.
I spoke to Apple folks when their compiler team switched the default to strict aliasing. They reported that it made key workloads 5-10% faster and the fixes were much easier to do and upstream than I would have expected. My view of -fstrict-aliasing at the time was that it was a flag that let you generate incorrect code that ran slightly faster. They had actual data that convinced me otherwise.
In support of your argument, we do have wasm which has no UB. In some half-assed experiments I’ve found that no amount of optimizing wasm backend will make up for generating wasm with llvm O0, which seems like weak evidence that something in the c language semantics was necessary for better-than-O0 performance.
Poking through the generated code a lot of it is pointless stack loads/stores, which are impossible to eliminate unless you are allowed to make some assumptions about which pointers might alias local variables.
As a curmudgeonly old C guy, I hear you, but on the other hand, if I were to try to write that “everything else” down as a precise specification, I bet I couldn’t do it.
It feels like C is trying to be too many things at once. Maybe there should be a
#pragma do_what_I_saidfor code that needs to relate to the actual hardware. I mean, when I was doing low-level C seriously, even that optimization of hoistingi+jout of the loop would have been annoying. If I wanted it hoisted, I woud have hoisted it myself!Volatile is pretty close to that, no?
For the specific case of dereferencing individual pointer variables, yes. Though it’s still just an indirect hint and pretty easy to miss.
Assembly, inline or otherwise, is that pragma.
Which is tedious and annoying, which is why C was invented in the first place!
Last time I thought about this, I concluded that without provenance, a huge proportion of optimizations would be invalid, so to me it seems like much more than just a nice to have.
I haven’t followed Swift much recently, but the original Swift was simple because it punted on all of the hard problems. It had no mechanism for error handling (there were some small mods towards NSError). It had no support for concurrency and the everything-is-shared-mutable-state model made that really hard to fix (the structured concurrency and actor extensions are trying). It had no support for shared libraries (Swift’s shared library support now is really nicely designed and it’s the only modern language that I think has addressed this problem well, but that inevitably came with complexity).
For macOS development, Objective-C++ is actually a very nice language. C++ for local code that is specialised over strong types, Objective-C for late-bound interfaces that are exposed across module boundaries,
I have never used Objective C++, but from afar it sounds horrifying. Take a monstrously, ineffably complex programming language and another quite complex programming language and mash them together into an amalgamation. Can anybody hope to make sense of it?
Speaking as someone who used it for years, it actually works quite well. Obj-C is not very complex, and its OO stuff is pretty separable from the C part. Obj-C++ extends the C part.
I’ll also point out that modern Objective-C’s ARC (automatic reference counting) composes very well with C++’s object lifetime rules. In pure Objective-C with ARC,
is not allowed, because
NSString*is an ARC pointer and the compiler needs to be able to reason about its initialisation and invalidation. The semantics of C structs don’t allow for that, which can make implementing custom data structures tricky: you either have to do it all in the context of Objective-C classes (which has certain overheads), use indirection, or you turn off ARC in the relevant source files and do manual reference counting.The same code quoted above will compile on Objective-C++ however, because the ARC pointer is treated as having the usual set of C++ constructors, destructor, assignment operator, and so on - it’s not a POD (plain old data) type. This means the
structalso gets implicit constructor/destructor/operator implementations when not explicitly implemented.You can therefore shove pointers to Objective-C objects into all sorts of C++ data structures, that have been implemented without special Objective-C support, including the STL. It all composes rather nicely.
(A significant proportion of my work in recent years has been working on macOS device drivers and related apps, daemons/agents, etc.; I’ve mostly been using Objective-C++ on that, although I’ve recently introduced Rust in one part of such a project. My limited contact with Swift has been exceedingly frustrating, so I’ve avoided it where possible; it never appealed to me in the first place due to the reasons David mentioned, and the practicalities around extremely poor forward- and backwards-compatibility were a nightmare to deal with on a project where it was forced upon me.)
ARC certainly makes this easy, though prior to ARC I implemented a C++ smart pointer class for Objective-C references that did the relevant retain and release operation, which meant that this was already possible in Objective-C++ without ARC, it just became nicer because you didn’t need to use the smart pointer.
Definitely, you weren’t the only one to implement such a smart pointer class. I guess my point was more that modern Objective-C actually composes better with C++ libraries than it does with C libraries and data structures.
This made me curious. What was your use-case? iOS apps? Mac apps? Something else?
I also used it for years and we made Mac apps that spoke to hardware devices through the IOKit kernel driver API (which is in C++). It was indeed quite nice.
Mac apps, libraries for use in Mac/iOS apps.
There is a non-trivial amount of ObjC++ in Firefox, too, or at least there was last time I checked. For TenFourFox I used ObjC++ as glue code to connect up libraries.
To add to what others have said: a lot of the pain in C++ comes from trying to do things that are easy in Objective-C, and vice versa. With C++, it’s easy to create rich types with no run-time overhead, but that create tight coupling. With Objective-C, you have late-bound interfaces everywhere, but avoiding dynamic dispatch is very hard / hacky. The combination means that you can completely avoid things like raw C pointers. You can used C++ collections inside a module, Objective-C ones across the boundaries.
You should give Obj-C a try, I think! It’s a surprisingly thin layer on top of C, giving a lot of bang for the buck for writing dynamic (but fast!) programs. It’s quite unique in that you have two extremes: C on the one hand, and a fully dynamic OO runtime (you can determine implementation at runtime). Overall syntax is outdated and weird (smalltalk influence), but it is still unmatched in that niche.
It’s not bad in practice. The use case is, you need Objective-C system frameworks but you can’t do without particular C++ libraries. You still spend the bulk of application code in one language.
I had a game that used Bullet physics in this way. I migrated most of the code to Swift after it was introduced, but I kept some Objective-C++ in order to keep Bullet. These days Swift has direct C++ interop, both for library use and for gradual migration of C++ projects to Swift.
Probably people who have actually used it.
I worked with making an iOS app at one point and I found while thinking in Lisp like patterns it seemed to get out my way if I wanted it to. But that is a beginner and greenfield perspective for sure.
I don’t think it sounds too bad, but I haven’t used it myself.
My understanding is it’s just the extra OOP bits from Objective-C overlayed on C++, similar to how it was overlayed on C in the first place. Basically just a second, independent object system. I understand why people wouldn’t like that, but it doesn’t sound too different than C++/CLI or using C++/JNI.
Can you tell us more about why Swift’s shared library support is well-designed?
With the caveat that I’ve read their design docs, but not actually used it in anger:
They make a clear distinction between ABI-stable and ABI-unstable shapes of structures. Within a library, there are no ABI guarantees. At a library boundary, you have a choice whether you want to sacrifice some performance for the ability to change a layout later, or sacrifice flexibility for performance. This is a per-structure choice. Depending on the choice that you make, the compiler either lowers to something similar to Objective-C non-fragile ivars, or C struct fields.
They actually have a language notion of a library boundary. This means that generics can be statically reified within a library, but fall back to dynamic dispatch across library boundaries. Contrast this with C++ where templates either have to live in headers (and then end up copied in every compilation unit, including the implementation, and it’s a violation of the one-definition rule to link two libraries that use different versions of the same template) or they are private to a library. The Swift model gracefully falls back. Things may be faster inside a library, but they still work from outside, and the ABI doesn’t leak implementation details of the generics, only their interfaces.
Wonderful explanation, thank you!
An overview from Rust folks: https://faultlore.com/blah/swift-abi/
Hmm…I’d say it was already incredibly complicated despite punting on a lot of hard problems and largely because it tried to codify “solutions” to non-problems into the language. Which never works, because reality, even code reality, is way too messy for that.
As an example, I wrote about the mess that is initialization back in 2014, so right after Swift apepared. There was some pushback from a prominent member of the Swift team saying that my goal of simplicity just wasn’t compatible with some of the goals they had. Hmm….
There was also a great rant by a prominent member of the community about Swift being just a mess of special cases upon special cases. I think he left the Apple ecosystem, and he’s not the only one. Alas I can’t find it and I don’t remember the name.
Anyway, I predicted back then that because they had so many language features for initialization it would never actually work out and there would be a renaissance of builder and factory method patterns and there would be even more language features for initialization. Five years later: hello SwiftUI! :-)
So the failure of Swift now isn’t surprising, the trajectory was already set in stone the day it launched and there wasn’t really much one could have done about it afterward…much less so since the same faulty reasoning that led to the initial problems was still present and guided later evolution.
I think this is an instance of correlation not being causation? My understanding is that the actual cause of SwiftUI is the successful design of Flutter (which gave raise to both SwiftUI and Kotlin Compose), and it is relatively orthogonal to language machinery.
Case in point, Kotlin’s initialization story is much more tame than Swit’s one (as it doesn’t try/doesn’t need to prove initialization safety statically), but it also converged on the essentially same design (or rather, vice-verse, IIRC Kotlin work in the area predate’s Swift’s).
Not to disagree with your wider point on constructors, which I agree with, just to point out that SwiftUI is not I think a particularly strong argument here.
I think you might want to have a look at the actual article. Swift introduced yet more special syntax for the part of SwiftUI that creates the view tree. So yet more language features for yet another way of constructing views^Wobjects^Wstructs.
The more general problems with SwiftUI (and related) are another issue, which I talk about a little bit here: UIs Are Not Pure Functions of the Model - React.js and Cocoa Side by Side
Last I checked the inspiration for Flutter and SwiftUI etc. was React.
I have read the articles! If I understand your argument correctly, it says that the fact that they needed to add new stuff to support Swift UI means that the original rules were inadequate. My counter-argument is even languages that don’t have Swift-style maze of initialization rule add special cases to support SwiftUI patterns. Ergo, adding stuff for SwiftUI is orthogonal to your normal way to initialize objects. In other words I claim that, in counter-factual where Swift doesn’t have complicated initialization rules and uses Java/Go style “everything is null to start with” or Rust/ML style “everything starts with all the parts specified”, it would have added more or less the same features still for SwiftUI.
The story is even more illustrative with Kotlin — it was specifically designed for DSLs like SwiftUI/Compose. The whole language, with its second-class-lambdas, extensions, and coming out-of-fashion implicit
this, works towards that goal. And yet, when the actual UIs started to be implemented, it was quickly apparent that no one wants to write+button(), and a bit more of compiler special sauce is needed for nice surface syntax.I must be a lousy communicator, because you seem to have misunderstood the article almost completely.
The point was not that Swift has the wrong initialization rules or too many of them. The point is, as it says in the title: “Remove features for greater power”. The many initialization rules are not the source of the problem, they are a symptom of the problem.
The problem is trying to bake this stuff into the language. As a consequence, you get 30 pages of initialization rules. As a further consequence, those 30 pages will be forever insufficient.
So for me, the supposed counter-point you bring with Kotlin actually supports my point. You write:
So they baked a whole bunch of features into the language to support the DSL use case. What was the title of the blog post again?
So they added a lot of features into the language especially for this use-case and it didn’t even work out for this particular use-case. Surprise surprise!
I simply don’t think the static/compiler-oriented mindset is compatible with the sorts of things these languages are trying to do. You put way too much into the language/compiler, and you do it way too early.
Ruby has had a bunch of these kinds of frameworks, and as far as I know they did not require any changes to the language. Because Ruby had fewer but more flexible features to start with.
https://github.com/AndyObtiva/glimmer
With Objective-S I seem to be violating that rule, because it certainly does put things into the language. Or at least seems to do so. What I am doing, however, is following the second rule: “don’t do it yet”. (With quite a bit of trepidation, because it is “experts only”).
And I am not actually baking all that much into the language. I am baking a bit of useful surface syntax and the associated metaobject-protocol into the language. What lies behind those metaobject protocols is quite flexible.
So far this appears to strike a good balance between providing some syntactic convenience and compiler support while not making the mistake of baking way too much into the language.
Indeed! I misunderstood your original comment as meaning that SwiftUI is a downstream consequence of initialization rules. I agree that both are rather the result of the lack of expressiveness, which doesn’t allow the user to “do it yourself” in userland code. The Kotlin example was exactly to illustrate that point.
Found it: Which features overcomplicate Swift? What should be removed?, by Rob Rix.
How to fix:
So I certainly wasn’t the only one who accurately predicted the current sad state of affairs. It was extremely obvious to a lot of people.
And silly me: of course I had referenced it an another one of my posts: The Curious Case of Swift’s Adoption of Smalltalk Keyword Syntax
It’s another example of complexity begetting more complexity, special cases begetting more special cases.
I am if the opinion that the only avenue for making a well rounded language that lasts is to put all of the skill points towards a small handful of skills (“fast”, “fast to compile”, “ease of dynamic linking”, “easy to use”, “easy to learn”, whatever) where each “skill” can be thought as a different axis in a graph. Go all in on a few niches. This helps with clarity of purpose for the project itself, a clear pitch for prospective users, and makes it possible to show the benefits for that niche. Without doing this the evolution of the project ends up rudderless, and adoption can stagnate either slowing down or killing the project. Once you’ve reached critical mass, fight for dear life not to abandon those hard earned early design decisions that allowed you to carve a niche, but go all-in on a handful of other axis as well. Rinse and repeat. A language can start as easy to use and once it has a clear base experience it can start trying to produce faster binary. Or it can start as fast to compile and afterwards try to produce optimized code. Or it can start producing super optimized code and later try to become faster to compile or easier to use. All the early design decisions will hamper the later pivot, and that’s ok! No language will ever be all things for everyone (barring some unrealistic investment in monstrous amounts of research, development and maintenance).
Swift had a clear initial view of what it wanted to accomplish, and did so successfully (I believe, don’t have personal experience with using Swift, nor do I keep up with its development). It is now on its expansion era, to allow for niches that previously were either difficult or impossible to cater to. It is a delicate process that can easily go haywire because of subtle interactions between completely unrelated features can make the result worse than the sum of its parts. But I can’t blame them for trying.
How do you see that wrt to Rust and async?
I could maybe argue that sync Rust had excellent “early design decisions that allowed you to carve a niche”, but the async related problems are now occupying a vast majority of the brain time of the lang teams. While I don’t think we’re heading to 217 keywords, I certainly feel uneasy seeing “keyword generics” and similar efforts.
I still find this argument as unconvincing as it is intriguing. As far as I can tell, the first part is about hardware being far more complex than what “C” gives you access to. Yes, computers are not fast PDP-11s any more.
However…most of this hardware is just as hidden from assembly language and the machine instruction set. Is machine language “not a low-level language”? Seriously?
The second part is about optimizations. It vastly overstates the effectiveness and importance of fairly fringe optimizations and mistakenly characterizes these optimizations that go against the spirit of C as essential to the language. See Proebsting’s Law, Fran Allen got all the good ones, What every compiler writer should know about programmers or “Optimization” based on undefined behaviour hurts performance and The death of optimizing compilers.
Of course, the title is true in one sense, but not in the sense intended by the author: when C came out and until at least the mid to late 80s, C was considered a high level language, not a low-level language. Because the low level language was assembly.
Wrapping file i/o in actors is exactly what we do in the Goblins distributed programming system. It ensures that only one operation is happening against the file descriptor at any given time while allowing other actors to use it asynchronously.
I think this is generally true: the Actor model is very low-level. Synchronous messaging with automatic dispatch is a convenience that is layered on top.
No it’s not just you. We (as in “the computing world”) is finally figuring out OO, that is the original message-oriented OO, rather than the scoped-procedural computing that we called OO.
I was actually thinking about Alan Kay when I wrote it. There’s somewhere where he talks about objects as “mini computers”, that it’s mini computers all the way down, and he says something like “why would we want to go to lower than that?”. It’s been in my head rent free.
You probably meant The Early History of Smalltalk where he writes:
If it’s separate computers, the messaging is pretty obviously asynchronous. Having that mechanism simulated by dispatching synchronous procedure calls is a simplified and optimized implementation, but it’s not the model.
Message passing in original OO is not asynchronous.
I’m not sure about that. The ur-OO was Simula, a language for implementing simulations of real world systems like hospitals. That would have benefited from asynchronous messaging (but I don’t know Simula so I’m just guessing.)
Some people beg to differ:
The early implementations were synchronous. This was seen as a problem.
As a German software developer, I find that I need to use a US QWERTY keyboard to be reasonably productive (also lived in the US for a bit, so used to it), because the German QWERTZ layout puts all the braces and stuff in weird, hard to access places.
However, on macOS that leaves all the Umlauts behind dead keys for the ¨ diacritic, which is very inconvenient. My layout puts all the Umlauts at Alt-, with other accented characters accessed via dead keys.
The second thing is that for Objective-S, I put the left and right arrows ← and → on Alt-[ and Alt-]. Alt-{ is ≝. In addition ≠ is at Alt-=. π is at Alt-p. Objective-S uses ← for assignment and → for architectural connection, though you can also use ASCII := and -> if you are so inclined.
So nothing earth-shattering, but a few tweaks that make things much, much nicer.
Oh, and of course the Ctrl Key mapped where DEC had intended it to be, where people without a clue put the entirely useless Caps-Lock key. Almost forgot about that one…
I’m sure it’s been said, but I understand Objective-C way better than I understand C++ ;)
Not just that, I’d venture to say that Objective-C is vastly simpler than C++.
I’d strongly suspect that if you try to understand Objective-C by translating it to C++, you will almost completely fail to understand it, even though you will see how to make it work, because this fact that it is so much simpler will get lost in the translation. By definition.
What you will see is an utterly weird contraption that does not make any sense whatsoever, and shake your head in dismay. Looking at Swift, I can’t help but think that this is exactly what happened.
Procedural programming never went away.
In my experience in industry, I saw very little to no object-oriented programming. I saw a lot of what I would call “scoped procedural” programming with some abstract data types thrown in.
That is, for most of the call sites, it was 100% clear what method/procedure would be called. object.operation(x,y) was completely equivalent to operation(object,x,y). (And I once did some analysis to quantify “most”. For the project I was working on it was >98%).
And no this isn’t “you are all too stupid to get true dynamic messaging-oriented OO”. While that’s not entirely wrong, it is also not entirely and probably not even mostly correct. This was in a highly dynamic language, so you certainly could go more true dynamic OO if you wanted to.
For me, one of the primary reasons is that our languages are not, in fact, object-oriented. They can at best be described as “class oriented”, although that doesn’t really show the problem, IMHO, which doesn’t go away if you go to a prototype language. The problem is that all those languages are basically scoped-procedural. The procedures that you write are now called methods, but they differ very little from other procedures.
And so what you can do easily doesn’t differ that much from what you can do easily with procedural programming.
People can do it. But it takes quite a bit of skill to express those object-oriented systems indirectly via scoped-procedural programming languages. Only a little less than with plain procedural.
Andrew Black put it well: “The program’s text is a meta-description of the program behavior, and it is not always easy to infer the behavior from the meta-description.” Object-oriented programming: Some history, and challenges for the next fifty years (pdf)
And it gets worse, potentially: the indirection that is required for “good” object-orientation is often misinterpreted as being the actual good part, not the bad part you need to endure to get to the good part. And so you can get quite horrible architecture-astronautics. Which gave OO a bad reputation it only very partially deserves.
So we need to go beyond scoped-procedural programming. Not abandon it completely, because it is absolutely useful. But insufficient,
As an example of the insufficiency: we are told to use composition over inheritance, because it is better. What linguistic support is there for object composition? <crickets> There is linguistic support for inheritance, which partly explains its continued popularity. Not only can you express (write) inheritance in the code, you can also read it. It’s stated explicitly in the code. Not so with composition. Composition is implemented procedurally, so you have to infer the actual composition of the objects from the procedural code that performs the compositing. And once a composite is created, it is also mediated procedurally, with objects calling procedures on each other.
At some point you can be forgiven if you say “let’s just cut out the middleman and go full hog procedure”.
Not quite crickets, there are procedural languages with support for object composition. Starting with Plan9 C, continued in Alef, and present in Go. There may be others.
That is the ability to embed an anonymous struct by type within another, and then call the functions for the embedded struct against the outer struct.
For Plan 9 C, see section 3.3 of this: https://plan9.io/sys/doc/compiler.pdf
Having that form in the struct definition strongly hints that such composition is being used, otherwise the version with a named field would be used.
The same is specified here for Go: https://go.dev/ref/spec#Struct_types and demonstrated thus:
I would agree that most so called OO languages are actually “Classful” languages.
Yeah, I was going to mention that procedural languages, including plain old C, actually tend to have better support for “object composition” than “OO” languages.
I found this really ironic.
Although this is really support for complex (nested) object literals, not really for object composition. Which is also very useful and goes in the right direction, but not the same thing.
Maybe I’m missing it, but I don’t see the object literal there - other than the outer one. To me, the thing embedded in it is doing composition.
So if you don’t view that as composition, how would you wish to see things structured in order to count?
(Is it merely the fact that I can directly reference the internals of Lck? If so then for Go, that is addressed by moving Lck in to its own package.)
What would it take for a “scoped procedural” language to become actually object oriented? What would it take for a program to make the same transition?
Not sure what support you have in mind, right now the only thing I can think of is importing the public fields (including methods) of the contained class inside the scope of the containing one. Basically inheritance, save for two crucial details:
And I can see a number of patterns would be made more convenient that way. How often and how much though? The way I see it, it would only cut a little boilerplate. If I’m correct, it’s not surprising language designers didn’t feel the need to add such a feature.
On the other hand, the end state is often: Dear friend, you have built a LAMP server, using k8s (and Helm, and operators, and cert-manager, and Prometheus, and…).
I hope that this doesn’t come across as too snarky but I’ve just suffered the pain of getting a single-node Ansible AWX server running. Their only supported architecture is k8s and their operator. It took days! I remember back in the old days we used to say “simple should be easy and complicated should be possible.” Now it’s “everyone needs Google scale so that’s all we’re going to do.”
As an example of doing it the right way, the Immich photo management app is comparable to AWX in run-time complexity but it comes with a Compose file that makes it trivial to run and manage on any machine with docker already installed.
FWIW you don’t need Helm for Kubernetes, and you’re going to want something like cert-manager regardless (and if you don’t want cert-manager, just don’t do anything—I wish k8s would include it by default, but it doesn’t).
Similarly, there’s a lot of middle ground between single node and Google scale, and you should have at least one replica running in production in most cases, which i means you’re going to need some sort of load balancer and some deployment automation. At this point you may as well start considering Kubernetes.
Not only does Kubernetes provide load balancing and deployment automation, but it also allows you to easily run cloudnativepg, an operator that makes it easy to run a Postgres cluster with replication, backups, and failover. Maybe you don’t need that for your production application, but if you do, it’s a lot easier to run k8s and that operator than it is to use other tools to manage Postgres IMHO.
That’s an astonishing jump. For a handful of machines, a few scripts with ssh and some pretty standard networking tricks would be what I think of long before k8s.
I’m not sure exactly what you’re thinking with your “few scripts and networking tricks”, but it sounds like you’re trying to refute the need for a load balancer by arguing that you can implement your own load balancer.
Moreover, while any given feature Kubernetes provides out of the box could be implemented easily enough by a sufficiently competent, experienced engineer, the point of the article is that the total effort is much greater and you still end up with a platform for which no documentation, or training, or experienced hiring pool exists and your expensive competent, experienced professional has spent his time building Kubernetes instead of doing more valuable tasks (or more likely, your professional only thinks he’s sufficiently competent and experienced, and you wind up with a very broken Kubernetes that causes endless frustration).
If you have two machines you don’t need a load balancer. You just need a way to fail over to the second machine. That’s just network configuration, which you have to do anyway, Kubernetes or no Kubernetes.
That’s the problem with this argument. You still have to deploy machines, configure operating systems and networking, install services (whether your services or Kubernetes’s services), etc. unless you’re using a 3rd party managed Kubernetes, which renders the comparison irrelevant anyway. So you can expose Kubernetes as a uniform target to a bunch of teams, but you still have a lot of stuff that is local deployment. If you don’t have a platform team providing a uniform target to lots of other teams, and you’re already doing all this configuration anyway, why do the configuration of Kubernetes as well?
Kubernetes generally takes care of the failover for you. You don’t really need to think about it.
There’s much less operating system configuration if you’re using Kubernetes than if you’re rolling your own hosts, and the networking setup is “install MetalLB”. It’s extremely minimal. Similarly, certificate management is “install cert-manager”. DNS is “install external-dns”. Similarly there are standard, high-quality packages for logging, monitoring, alerting, volume management, etc. While I’m surprised there are Kubernetes distros that include this stuff out of the box, it’s still less work than doing it yourself and you get the benefits of standardization: abundant, high quality documentation, training resources, and an experienced hiring pool. And moreover, as your requirements scale, you can incrementally opt into additional automation (Kubernetes is a platform for infrastructure automation).
I’m not sure what you mean here. Why would managed Kubernetes render the comparison obsolete?
I’m not following you here. What do you mean by “uniform target” vs “local deployment”.
No, it’s not. You still have to set up the underlying network that the physical machines are on. That’s the key point. You can install all this stuff within Kubernetes…but it’s all dependent on nodes that are fully configured outside of Kubernetes.
I see what you mean. Yes, you need to have nodes that are on a network, but “fully configured” for running Kubernetes basically means “SSH is configured”, which is dramatically easier than any configuration I’ve seen for running production-grade applications directly on nodes (no need to haul out ansible or anything like it).
By design, sure. In practice though the biggest failures I’ve had with kubernetes have been nginx ingress pods just randomly losing their ability to connect to upstream and silently causing downtime that wouldn’t haven’t happened if the app was exposed directly, using an AWS ALB, etc. I’m not saying this is everyone’s experience. It might be due to noob misconfiguration mistakes. But that’s sorta the point the article misses. Kubernetes might offer repeatable solutions for these problems (and I appreciate it for that) but it really isn’t a solution on its own. Every team running a kubernetes cluster has their own collection of weird scripts with bespoke workarounds for kubermetes quirks.
/sympathize with random nginx Ingress failures leading to widespread 503 outages.
not sure if this is welcome here, but try https://semaphoreui.com/ instead
Thank you! In this case I was told to stand up an AWX instance, and I didn’t know what I didn’t know, but I’ll take a look at your suggestion. If there’s a “next time” then I’ll be able to advocate for an alternative.
OMG this. At work I tried to fix a broken internal app that I had no role in getting set up. Everyone who had worked on it had departed. So started the slog of figuring out how it worked, where it was deployed, and getting access. Oh cool, this runs on the “internal dev platform”. What is that? Oh it’s a kunernetes cluster. Neat that they tried to think ahead spread some infra costa across projects. I wonder what else is running on this thing… nothing. It is a 2 node kubernetes deployment that runs one application pod. Not per node. Total.
I was working in an Oracle shop where devs would spin up an Oracle instance for a web application that had 4 data items. Not databases. Not tables. Items.
My production app ran completely without databases and was several orders of magnitude faster, smaller, and more reliable than its predecessor.
But think. How the devs could otherwise move on in their careers without “designed, built, deployed and maintained a scalable K8 solution” on their CVs?
Ding ding ding! You got it. On a lark I searched Google for large sections of the docs they “wrote” for it and sure enough it was all basically copy/pasted from guides on how to set up RBAC.
I just can’t connect the dots between masculine and valuing formalism, formal methods, quantitative over qualitative work, and the examining of technical aspects. Can someone eli5.
It reminds me a bit of the “punctuality, standardized testing, etc are white supremacy” stuff from the previous decade. Honestly this stuff always seems a lot more like harmful stereotyping and I would appreciate a clear explanation for why it’s not as toxic as it seems.
Except the stereotype is true: women are raised to care more about people than men are, at least in the Western countries I’m aware of. This whole thing about formalism being more masculine than studies, does match my intuition about why we find so more men in “hard” sciences than we do in the humanities. (Note: I hear that Asia has different stereotypes, and many more women in hard sciences. In Japanese animation for instance, a typical support character is a female scientist or technician — yeah, the main/support gender bias is still there I think.)
Now the famous “Programs must be written for people to read, and only incidentally for machines to execute.” have indeed been coined by men (Harold Abelson and Gerald Jay Sussman), and on that same page I learned that another man, Martin Fowler, said “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” Clearly human stuff is important to men too. And there are some crazy impressive low-level woman programmers out there too.
And yet, I can’t shake the feeling that indeed, we dudes tend to ignore the human element more than women do. One striking example would be the Linux kernel, and the recent Rust debacle. It did reek of not trying particularly hard to take care of humans. Or take my own readability article: the whole point is to cater to humans (increase readability), but then I reduce humans to a single limitation (working memory), and evaluate various advice from that alone. Which, when I think about it, is only possible because I have such a mechanical outlook of the human mind — and I don’t even have any expertise there!
Now that paragraph in the paper could probably have benefited from a couple references showing the prevalence, or at least existence, of the stereotypes it uses in its exposition. Without it feels like the author has internalised those stereotypes more than she would have liked to admit.
While an inaccurate stereotype may be more harmful than an accurate one, an accurate stereotype is still pretty bad. Even if it happened to be true that men are better drivers (whether by nature or nurture), it seems pretty harmful to refer to good driving as a masculine trait and also a thing can be true for a group of people and not be true for all individuals in that group, so applying stereotypes (even accurate stereotypes) to individuals is usually unjust. That’s probably not what the paper is advocating, but it’s an inevitable consequence of trading in stereotypes and moreover I don’t see how stereotyping improves the paper.
For a specific idea of a harmful true stereotype see also “black people don’t swim”. There’s lots of stuff in society which comes from the environment and not conscious choice. It doesn’t mean we should assume that’s just how things are and expect them to continue. Sometimes things are more universal if we allow it / enable everyone to do better in whatever way they choose to approach things.
There was a fascinating experiment on stereotype threat where they got a bunch of Asian and Black Americans to try golf. They told half of them that golf was very mathematical and the other half that it was very athletic. There was a statistically significant difference: the black folks who were told it was athletic did better and the Asians who were told it was mathematical did better. Both groups did better at the thing that they’d been stereotyped at being better at even when it was the same task. So reason one that even accurate stereotypes are bad is that they impact the people being stereotyped.
Reason two is that they are still about probabilities. If a stereotype is 90% accurate, it will not apply to all of the members of a group that you can fit in a fairly small room. If 90% of green people are bad at programming, you’ll see fewer green people in software development, but all of the ones you see will be statistical outliers and the stereotype tells you nothing about that subset, but people interpret as if it does. And even the most accurate stereotypes apply to far fewer than 90%. So reason two that they’re bad.
Reason three is that a lot of accurate stereotyped behaviour is due to how people interact with members of a group at an early age. People subconsciously reward behaviours in children that conform to expectations. Boys and girls are praised for different things by every adult around them. This makes stereotypes self perpetuating.
Found the 1999 study at https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=1061f5d1d9d35a13e8deaf2f32f6f386b19c489f
Couldn’t you just as easily argue that formal methods (conscientious, detail-oriented, theoretical) are feminine, whereas user-studies (pragmatic, empirical, utilitarian) are masculine?
In fact the paper does cite that argument:
But the C advocates were arguing against formalism and formal methods, qualities that the quote describes as “masculine”. In fact, I’ve seen people argue that the “C is fine, just write good code” mindset is an example of masculine arrogance (or what have you). So we’re in a situation where emphasis on formal methods is masculine and disdain for them is also masculine.
I’d say the stereotype was true: we’ve observed much change in society over the last ~15 years, particularly in gender roles. I don’t think the stereotype is true anymore. If it is, its decreasingly so year after year, and will soon be definitively false.
The ground of the feminist critique is what we could call a critical history. Traditionally, we think of philosophy or science arising from a neutral form of “reason” – detached from social or historical context. A critical history challenges this claim, and views thinking as necessarily embedded in a place and time. Thought is not considered to be a universal means at finding objective truth but rather the thought arising out of a specific place and time, under regimes of thinking that ground our understanding of what is true or false. This is not subjectivism (“everyone’s perspective is valid”) but rather a sort of uncovering of the origins of ways of thinking which we take for granted.
Given that the historical context of Western civilization, and more specifically, the 20th century development of computers, feminists can critique the development of computing from the lens of a specifically feminist history – uncovering the manner in which thinking about computers came out of a specifically patriarchal context and mode of thinking. The point is not that early computing pioneers were men, but rather than they were embedded in a society which prioritized ‘masculinist’ forms of thinking and potentially reproduced those structures: structures that exist independent of any individual’s identity, but rather have a sort of independent existence
It’s important to understand that it’s not a universalizing critique, ie we can’t understand everything about computing through feminist critique, but it is one lens through which we can look at the history of computing
Said that way makes more sense, at least to me, in which “masculine” refers to the patriarchy-way-of-thinking. Would have used other term, but that’s nitpicking.
The original text in the paper sounds too much like a stereotype applied to men as a gender, maybe because I’m mixing in my head my own perspective with the past and everything in between. “men molded the field like men do, because they all behave the same way”.
Based on your description, it seems like critical theory and feminist critique is principally about ideological framing? As someone who is interested in programming language design, I probably don’t have a lot of interest as to whether we should believe that computing originated out of something called “patriarchy” or whether it originated out of some neutral reasoning, I just want tools that help me accomplish my goals more easily. These ideological frames feel unhelpful at best and divisive or even toxic at worst (I made another comment about how claims like “formalism is masculine” and “objectivity is white supremacy culture” seem like really harmful stereotypes and yet they seem to closely orbit critical theory).
This is not a statement free of ideology. First off, you are conveying a form of instrumentalization: ie, you view programming languages as a tool to be utilized. I’m not saying this is incorrect or bad (obviously, there are many cases in which we do use PLs this way), but it is not the only way of conceiving of programming language design. We could think of programming languages as toys to play with, forms of expression, or places to try out new ideas. Consider the whole field of esoteric programming languages – almost all of them totally fail as “tools to accomplish goals more easily”.
The point is, there is no “neutral reasoning”. Everyone is always thinking within a culture that has a certain ideological frame. This not a moralistic point (although sometimes this sort of analysis does end up distorted into popular culture that way), it just means that you can’t separate computing from its historical, cultural, and ideological history if you want to understand it fully. I mean, consider the basic point (referenced in the paper) that most PLs are monolingual in English – this isn’t because English is the “objectively best” language for programming, it’s necessarily tied to a long history of British and American cultural imperialism. And again, I want to emphasize, the point isn’t that it is “morally wrong” that programming languages are in English. It’s the academic point that this development is inseparable from the history of colonialism. And this critical history can inform experiments, development, and research into non-English based programming languages.
Your issue appears to be with specific applications of critical theory, yet you expand this to include the entire field itself. One could certainly make a bad feminist critique of PLs, but that doesn’t mean that feminist critique is a useless enterprise, which is what you seem to be claiming.
Nit: the “goal” in those cases is to have something to play with, something to express oneself with, or some place to try new ideas. You aren’t really disagreeing.
I’ve seen a good chunk of listings for game code written in German that would disagree with this premise. I’d imagine that the Soviet code listings in Cyrillic probably also disagree–Drakon, for example.
Re: imperialism, it’s well-known that CS is rife with accomplished Englishmen like Alain Colmerauer and Americans like Edsger Dijkstra. Similarly, we all run on silicon manufactured in the United States by red-blooded American companies like TSMC and using phones made by god-fearing businesses in Western countries like Huawei.
The thing is, that lens means a particular thing in academia, and outside of academia–where we are here–it’s easy to “disprove” using common sense and rudimentary historical knowledge by the casual practitioner. That’s not to say it’s a bad or even incorrect lens, but merely that it requires more nuance in application and context than is usually found here–people spend their entire doctorates haggling over narrow aspects of imperialism and feminism, after all.
EDIT: After talking with a colleague, I’ll concede that if by “forms of expression” you mean a programming language itself as a sort of work of art then my lumping it in as a tool is a bit unfair. Paint vs paintbrush.
I’m not sure if I understand your point here or if we really disagree. I said most programming languages are English monolingual, and that this fact is tied to the history of British and American cultural imperialism. It is not that we can understand technology solely through that lens, just that a purely neutral, technical, ahistorical viewpoint is not the correct one. The example of Huawei is particularly interesting here – it’s been sanctioned by the US since 2020, and for geopolitical, not technical reasons.
But isn’t it the case that certain fields are dominated by a particular language? Musical terms are Italian; cooking terms and techniques are French; large portions of philosophy are French and German, not to mention the endless Latin found in medicine and law. Could it not be that computers are largely English because English-speaking countries dominated the space early on?
Another thing I’ve heard (or read, sorry, no citations available) is that non-English speaking programmers tend to prefer computer languages not in their native language, for whatever reason. Maybe just a simple “translation of keywords into target language” just doesn’t work linguistically? Not enough non-English documentation/learning materials?
The ability to share code and libraries with the rest of the world. Imagine if you had to learn French to use sqlalchemy, German for numpy, Japanese for pandas, and Russian for requests.
This is a great case for programming languages in Esperanto!
Maybe Lojban or Interlingua (or toki pona) :) Esperanto is very Eurocentric.
I knew this reply would appear when I posted my comment 😁
:D Thanks for the dance!
In more seriousness, though, Lojban’s “logical” bent does make it appealing for PL experiments, in an anti-Perl kind of way ..
I’ll do a different one for variety: Native English speakers who learn a constructed language instead of a normal one are xenophobic.
consider that constructed languages are (with few exceptions) designed to be much simpler to learn than natural languages. it’s a lot easier to learn 200 sitelen pona than 2000 kanji. (and it’s a lot easier to learn a phonetic spelling than whatever english is doing)
After we’ve finally finished migrating everything to Esperanto, it will be found to be problematic because of certain reasons. Then we will need to migrate to a new created human language base for programming languages. I vote for Tolkien’s Quenya.
TBH it’s a great case for programming languages in English, which is the most popular spoken language in the world thanks to its strong popularity as a second language globally. https://www.ethnologue.com/insights/ethnologue200/
I find it funny that, even on the NL wiki they only have English quotes of Dijkstra , and sources to English papers of Dijkstra
https://nl.wikipedia.org/wiki/Edsger_Dijkstra
Does this not itself show how English-centric some of this stuff is?
I think that’s more of peculiarity of how multi-lingual the Dutch society is, and how good they are at using English.
He spent a lot of time at the University of Texas in Austin, hence I think it makes sense that the quotes are in English (as that is the actual quote). And yes, I guess most Dutch readers are also familiar enough with English that they do not need a translation.
Yeah, it pretty much is. It is a statement of personal preference, not a set of beliefs.
And it does not claim that it is “the only way of conceiving of programming language design.” It is an expression of a personal preference.
And nobody claims that that is the case, so this is just a straw man.
It has almost nothing whatsoever to do with “colonialism”.
Critical “theory” is not really a “field”, and certainly not a scientific one. It (accurately) self-identifies as ideology, and then goes on to project its own failings onto everything else. With “critical theory” the old saying applies: every accusation is a confession.
Well, noone was claiming that anyone was claiming that, so ironically, someone could say that your strawman claim is itself a strawman.
This saying actually isn’t that old, the earliest example seems to be from around 2020.
what goals do you want to accomplish? why do you want to accomplish them? in what socio-politico-technical context are the tools that are more or less fit for that purpose developed, and how does that context influence them?
What goals or socio-political contexts lend themselves more to more or less formal programming methods? Like even if I want to build some “smash the patriarchy” app rather than a “i ❤️patriarchy” app, would I benefit more from languages and tools that are less “formal” (or other qualities that the poet ascribes to masculinity)? I can’t think of any programming goals that would benefit from an ideological reframing.
I’ve not finished the paper yet but it seems like the authors are using “masculine” as a shorthand for “thing oriented”. Which isn’t a perfect analogy, but in my experience “thing oriented” typically skews male and “person oriented” typically skews female.
Yes, and this is a stereotype that has been semi-deliberately produced and spread. You can read lots of stuff about women in technical fields in the late 19th through mid 20th century, going to universities, applying for jobs, etc and a lot of it involves people explicitly or implicitly saying “women can’t do X because men think logically/rationally/analytically, while women think emotionally/impulsively/socially”. Like most stereotypes there’s a grain of truth in it from the right angle – I recall talking to my sister when she was pregnant with her son and she often said things like “I get so emotional for no reason” and “it’s so draining, I feel like my IQ has dropped 20 points”… now take this phenomenon and project it to a time when families commonly had 8 kids or whatever, your average woman would spend a decade or two of her life either pregnant or recovering from pregnancy.
But the real reason that stereotype exists is because it is used as a proxy for “women shouldn’t be able to do X because I do X and I’m obviously better than a woman”.
As a white boy, no one ever told me that I shouldn’t program because it’s not a white thing. No one ever told me boys can’t code. No one ever told me that computers are not for people like me. No one ever sent a less-competent girl to a maths or programming extracurricular event because programming is a girls thing and so they didn’t consider me. No one ever told me not to bother with programming because it doesn’t lead to appropriate careers.
As Director of Studies for Computer Science at an all-women Cambridge College, every single one of the students that I admitted had stories like that from people in positions of authority. Young women who went on to get first-class honours degrees in computer science at one of the top departments in the world were told girls can’t program by their computing teachers. Girls who scored Gold in the Mathematics Olympiad were encouraged not to enter by their teachers. These aren’t isolated things, these are small social pressures that have been relentlessly applied to them from birth. And the ones I see are the ones who succeeded in spite of that. A much larger number are pushed out along the way.
Now maybe social pressure like this doesn’t apply to you and so this is not relatable for you. There’s an easy test: do you wear a skirt in the summer when it’s too warm for trousers to be comfortable?
Now why do I care? Because I’m selfish. I want to work with the best people. Having more than half of the best people self-deselect out of the profession as a result of unintentional social pressure means I don’t get to work with them.
Thank you for saying what I wanted to far better than I ever could.
I’ll follow up on the research on sex-preferences in newborns or other primates, ‘cause it’s really interesting: built-in differences in preferences are totally real, sure! The differences exist. But everyone is an individual. The way I think of it is that looking at whatever human behavior you want to measure, you’ll probably get a big bell-curve, and if you measure it by sex you’ll probably get two different bell-curves. The interesting question for science is how much those bell curves overlap, and how much the differences are environmental vs. intrinsic. The studies on newborns and such are interesting ‘cause cause there’s barely any cultural or environmental impact at all, so that’s relatively easy to control for.
But when people do studies on “vaguely what do these bell curves look like”, such as https://doi.org/10.3389/fpsyg.2011.00178, they tend to find that the difference within each bell-curve are far larger than the difference between the bell curves. That one is neat ‘cause they look at (fuzzy, self-reported) cultural differences as well and find it has a big effect! They look at generic-white-canadian men and women and get one pair of bell curves, then look at south/east-asian men and women and get a quite distinct pair. Sometimes the male/female bell-curves in each culture overlap more with each other than they do with the bell-curves of the other culture! (figure 4 of that paper.) Sometimes the cultural differences outweigh the intrinsic/genetic ones! Sometimes they don’t! It’s wild, and in that paper deliberately quite fuzzy, but the broad strokes make me really wonder about the details.
I am far, far, far, far, far from an expert on psychology, but the conclusions are pretty compelling to me. If you could magically make a perfect test of “technical ability, whatever that means” that corrected for environment/upbringing/etc and applied it to a big random sample of humans, it seems like you’d expect to get two different bell curves separated by gender, but with far more overlap than difference. So then the question is, why has every computer-tech job, class or social venue I’ve ever been in 80-100% male?
It’s not just computers. It’s that “thing” oriented jobs are overwhelmingly male. “people” oriented jobs are overwhelmingly female. it’s not that women are bad at “things”, it’s that they often have other interests which are more important to them. Or, they have other skills which means that they’re better off leveraging those skills instead of “thing” oriented skills.
For people who suggest that men and women should have similar skills and interests, I would point to physical differences between men and women. Evolution has clearly worked on male and female bodies to make them different. But we’re asked to believe that those physical differences have no affect on the brain? And that despite the physical differences, men and women have exactly the same kind of interests and abilities?
Just… no.
Obviously there is going to be an effect. We’ve measured that effect. We would probably expect something like a 55:45 spit, based on biological factors. but instead we see something much closer to a 90:10 split, which is way way way more than we would expect. “maybe it has something to do with culture and society” is a pretty reasonable hypothesis.
See “non-linear effects”. The average woman is only a bit less strong than the average man. But the world record for male “clean and jerk” is essentially “pick up the woman doing the womans record, and add 100 pounds”.
Or look at male high school athletes, who compete pretty evenly against female olympians.
Are we really going to say that similar effects don’t exist for hobbies / interests / intellectual skills?
And why does the comparison always include male oriented jobs? Why not point out that 90% of nurses and dental technicians are female? Why not address the clear societal expectations / discrimination / etc. against men there?
The arguments are generally only one way. Why is that?
‘Cause the female-dominated jobs are uniformly the less prestigious and less well-paid ones ofc~ But it’s still a fun question to ask ’cause, do females get forced into less-prestigious jobs, or do the jobs that females tend to prefer become marked as less prestigious? Probably lots of both!
Makes you realize how much we’re still on the tail-end of 15th century social structures, where males got to run things entirely because they could usually swing a sword harder, hold a bigger pike, and pull a stronger crossbow. And how much of that is built in, I wonder? Testosterone definitely makes a person more eager to go out and do something stupid and dangerous for a potentially big payoff.
Anyway, I’m rambling.
Because the professional technical ability is not about averages, but extreme tail ends. Professional developers/hackers are not average Joe or Jane, but the extreme .1% percent of population that has both ability and interest, to dedicate their life to it. At that point two otherwise very close bell curves are largely disjoint. Example: at point r there are no more red samples, even though the curves themselves are not that far away. In 2024 I thought this phenomenon was largely know. Almost everything in society is not about averages, but extremes for similar reasons.
Yeah but why is there any gendered difference in the bell curve? Natural aptitude seems like a real cop out unless it’s been established and there are no societal mechanisms that reinforce it.
The reasons for it have been obvious to humanity for thousands of years.
That’s not a reason, that’s the phenomenon in question.
I started doing computers in the mid 1980s, when it wasn’t socially acceptable. My recollection is getting laughed at and mocked by the girls my age, for being interested in such a stupid thing as “computers”. 25 years later, the same people were sneering at me, telling me that “the only reason you’re a success is because you’re a white male”.
I’d contrast my experience with Bill Gates, for example. He had in-depth experience with computers 10-15 years before me. He had connected parents who helped his business. While it wasn’t inevitable that he was a success, he had enormous amounts of opportunity and support. In contrast, I spent my summers sitting on a tractor. My coding was alone, with no help, with computers I bought with my own money.
I get told I’m similar in “privilege” to Bill Gates, simply because of shared physical attributes. I can’t help but conclude that such conclusions are based on ideology, and are untouched by reality or logic.
And do you remember how much worse that mocking was for girls who were interested in computers? Do you even remember any girls being willing to express an interest in computers when the consensus of their peer group was that it was a thing for no-life boys?
This feels like a reductio ad absurdum. There is a large spectrum of privilege between Bill Gates and starving refugee. Just because you have less privilege than Gates, doesn’t mean that you don’t have more than other people.
I think you’re largely missing my point.
What I find offensive, racist, and sexist, is people who lump me into the same “privilege” category as Bill Gates, simply because of shared physical attributes That’s what I said, and that’s what I meant. There’s no reason to conclude that I disbelieve in a spectrum of privilege. Especially when I explicitly point out that Bill Gates has more privilege than me.
This is one of my main issues with people making those arguments. They are based almost entirely in ideology, in reducto ad absurdum, in racism (“you can’t be racist against white people”), and in sexism (“men can’t be raped”).
Even here, I’m apparently not allowed to point out the hypocrisy of those people, based on my own experiences. Instead, my experiences are invalidated, because women have it worse.
So I’m not arguing here that different people have different opportunities. I’m not disagreeing that bad things happen to women / black people / etc. I’m pointing out that the most of the arguments I’ve seen about this subject are blatantly dishonest.
Without intending to criticize your preceding remarks about social pressure, I find this “easy” test to seem more culturally-specific than you present it. “it’s too warm for [full-length] trousers to be comfortable” itself is not relatable for me, and the men and women I know for whom it is both wear short trousers in that case, and of course in some cultures both might wear skirts.
What does this test determine and how?
Whether a man’s experiences and behaviors have been shaped by societal defaults of gender.
I frequently hear women talking about how they were encouraged to pursue STEM fields they weren’t very intrinsically interested in, because of the presence of Women in STEM programs in their educational or social environment and the social expectation that it is laudable for women to participate in things that have relatively low female participation rates on feminist grounds. I’ve even heard women talk about wanting to quit STEM programs and do something else, but feeling social pressure - that they would be a bad feminist - not to do this.
This happens to everyone though, male and female. My uncle became a dentist because his father pressured him into it. He hated it, his whole life. There, now we have two matching anecdotes.
There is of course rather a lot of actual science about this stuff. For example, the lightest, most effortless google offers “Gender stereotypes about interests start early and cause gender disparities in computer science and engineering”, Master, Meltzoff and Cheryan, PNAS 2021, https://www.pnas.org/doi/10.1073/pnas.2100030118.
You can’t take the assumption that “there exist programming tools that work well for me” and replace it with “everyone has tools that make it as easy for them to program as it is for me.” You speak English.
You may not even notice the advantage you have because everyone around you has the exact same advantage! Since a competitive advantage over your peers is not conferred on you it is quite easy to forget that there are some people who are still at a serious disadvantage – they’re just mostly shut out of the system where they can’t be seen or heard from.
The connection to feminism is a bit tangential then I think: it’s a lens through which it is possible to see the distortions the in-group effect has caused, even as the effect is completely invisible to most people who have accepted the status quo.
So how else do you explain this prevalent tendency?
I think there are some valuable ideas in this paper. On the other hand… do we really need to get gender into programming languages? Are we going to have toxic masculinity of language design? Is everything in life about oppression, or do people just build systems in a way that they are useful to them, and maybe a different set of people builds systems in a different way, and in a pluralistic world we can have both and learn from each other?
If I had written this paper, I would not have brought gender/sex into play. You can easily substitute feminism for accessibility or other terms for part of their reasoning, and make this paper useful to programming language designers without evoking a political agenda.
Section 2, titled “Setting the Scene: Why Feminism and PL?” spends 2.5 pages answering your question, and sections 5.1 and 5.2 have more.
To expand on the last paragraph of section 2.4, using feminism allows the authors to build on existing work. There’s dozens of citations from outside of programming, bringing in 50 years of material to hybridize with the familiar.
To your suggested accessibility framing, there’s a recent talk How to Accessibility If You’re Mostly Back-End that hits some similar points, but it’s much more about industry practice than language design. (I saw an unrecorded version at Madison Ruby a few months ago but at a skim this presentation is at least very close.)
Yes, yes, this is an essay “escaped from the lab” to justify an academic’s take on the field and to publish a paper.
The existing work being built upon should arguably be, you know, programming languages and programming, instead of feminist theory. I imagine that’s what many folks will bristle at.
I’ve skimmed this once, and I’ll give it a thorough reading later, but sections like 5.1 and 5.2 emphasize–to me at least–that the target audience of this sort of thing is fellow academics and people already sold on the feminist lens. This is an academic talking to other academics, and we do tend to skew a bit more towards professionals creating artifacts and systems.
I don’t really have high hopes of useful discussion on Lobsters here, since the major reactions I expect to this are either “Oh, neat, a feminist look at programming, okay whatever”, “Hell yes, a paper talking about how unfair we are to non-white non-men”, or “Hell no, why do we have to inject gender into everything?”. To the degree to which any of those are correct, our community’s ability and capacity to discuss them civilly while remaining on topic for the site is suspect.
So, flag and move on.
Accessibility, and the lack thereof, is also political.
The idea that feminism is a novel (and somehow intrusive/invasive) political agenda, rather than a lens through which you can view and critique the political agenda we’ve all been raised within, seems to be part of the paper’s basic point. Gender is already implicitly part of programming languages (and all fields of human endeavour), the idea is to recognize it and question if and how (and to what degree) it’s influenced the field. The act of doing so isn’t advancing (or advocating for) a new political agenda, it’s critiquing one that already exists.
BTW, a non-author of this paper swapping “accessibility” for “feminism” here, when the author chose the latter precisely because it is not equivalent to the former, would actually be pretty spot-on example of why adopting a feminist perspective is necessary. Accessibility is about making systems more adoptable to humans with other-than-default access needs irrespective of gender, feminism is about making systems more adoptable to humans with other-than-default gender irrespective of their access needs… they’re literally two topics that don’t overlap except in causing us to critique our “default” way of designing systems; if you think looking at the accessibility bias built into systems is important and/or valuable you probably should think of looking at the gender (and other) bias of those systems is important and/and valuable too.
I have only read the linked article and paper intro (for now), so there might be more, but what seems to be taken from feminism here is the analysis of power structures, social dynamics, shared values and how that all holds back the field by narrowing what research is done.
Reading the intro will provide concrete examples from the authors’ personal experiences.
If the paper was about, say, applying an economic approach to PLT, would you have engaged more deeply to get answers?
I ask this not as a gotcha, but to create an opportunity to reflect on bias.
I personally acknowledge my initial reaction was “why feminism?” but am happy that I also had the reflex of going past that.
I am considerably more willing to believe that feminist critiques are written in good faith than economic ones, and it behooves the reader to understand that neither will represent an “apolitical” perspective.
I agree, FWIW I chose economics with the mindset of “it’s also politics but more accepted.”
Conversely, if it applied a flat earth theory approach, would you engage less? I probably would. Is it wrong to use our past experiences (our biases) with particular fields to determine which lengthy papers we do and don’t read?
So you put feminism and critical studies on the same level as flat earthism?
Not exactly, but not far away.
So the “theory” in the name is already a lie. This is not “theory”, it is politics and ideology.
There is nothing wrong with politics, but please don’t pass off politics as science. And in particular, don’t try to make this non-science the arbiter of all the other sciences. Yeah, I know that this “theory” claims that all the other sciences are actually not scientific because they are just power grabs. Well, that’s just projection.
You, coming to these comments to tear down the work of others published in the same proceedings as your own, by calling it “non-science” and “not far away” from flat earth-ism is demonstrative of the bare misogyny that this paper is asking the audience to start taking notice and accept less of. Stop it.
When we exclude spreadsheet from programming languages, I think we already have to some extent: one big reason it’s excluded is because spreadsheets are perceived as not as prestigious as “actual” programming. And I bet my hat one reason it’s not is because spreadsheets are generally a secretary’s tool. Female secretary most of the time.
There used to be a time where computers were women (or “girls” as the men around them often called them). With the advent of the automatic computer, a good deal of those women turned to programming. And for a time, this role was not that prestigious. Over time it did became so, though. And over time we did see a smaller and smaller share of women going into programming. Coincidence? I think not.
Anyway, giving “programming language” status to spreadsheets would elevate the status of secretaries to programmers, and “real” programmers can’t have that. Hmm “real programmer”. Why do this always conjure an image of a man in my head? You’d have to admit, the XCKD über hacker mom isn’t your stereotypical hacker.
I think the simpler and more correct explanation is that spreadsheets dominate other sectors and industries (engineering, medicine, hospitality) so thoroughly that it simply never occurs to most programmers that they’re a valid programming environment.
This is also why I’ve seen an MBA beat a bunch of programmers’ asses using only pivot tables and sheer stubbornness.
Spreadsheets, in the context of programming (generally excel) are coded as management tools or more generally “business people”. Not by gender (these are, more often than not, men as well, although probably not quite as male dominated as programming).
Do we exclude spreadsheets? Microsoft claimed excel was the most popular programming language in the world in 2020.
https://www.theregister.com/AMP/2020/12/04/microsoft_excel_lambda/
I can’t find the references for them but I’ve listened to a number of talks dating back to at least the 2010s that said more or less the same thing.
Excel isn’t present at all in the technology section of the 2024 Stack Overflow survey results. And that isn’t even specifically a list of programming languages; the page has several categories of technologies, including an “other” category. So while Microsoft may rate Excel—and to point out the obvious, they have a financial interest in doing so!—I don’t think that’s necessarily a widespread view.
The original question was whether Excel is a PL in the context of PL research. And in that context, I think it’s especially obvious that it is. It has variables (called cells), and loops. It’s Turing complete, and not in a gotcha-kinda way. Excel only has user-defined functions defined in another language, but Google Sheets has user-defined functions defined in the same language. It has control flow, made interesting by cells referencing each other in a DAG that needs to be toposorted before being evaluated. It has compound data: 1D and 2D arrays.
You could absolutely write a small step semantics for it, or a type system for it, and neither would be trivial. In fact I’d like to read such a paper for Google Sheets to understand what ARRAYFORMULA is doing, there’s some quantifier hiding in there but I’m not sure where.
EDIT: Clarity.
Oh, I do think that Excel is a programming language! I realize that my comment didn’t make that clear at all. I was trying to push back on the claim that spreadsheets are commonly considered to be programming languages. I think Excel is a PL, but my impression is that there aren’t a lot of other people who think that.
I think I disagree on the widespread view comment. Anecdotally most people I talk to agree that excel (or spreadsheets more generally) meet the definition of a programming language/environment. I would argue that the community of users on stack overflow is not representative of the broader population and the population choice is the crux of the issue here.
A few references for more widespread acceptance:
https://ieeexplore.ieee.org/abstract/document/7476773
https://dl.acm.org/doi/pdf/10.1145/2814189.2814201
https://onlinelibrary.wiley.com/doi/abs/10.1002/9780470050118.ecse415
Maybe it’s just because I’m young, but the first introduction to “historical” programmers I had was Ada Lovelace, and later then Grace Hopper.
Honestly, I can’t say I even necessarily have a generally positive perspective on the “typical hacker” persona - maybe partially because there are some like RMS who come with a lot of baggage.
That is the fun bit. This totally eclipse the reality of the history of our field.
I recommend a tremendous book, “Programmed Inequality” by Mar Hicks, for a historian work on some of this. It is fascinating and may help shed some light on the topic and the lens.
Oh I’ve heard of Mar Hicks before! Thanks for the recommendation. I’ll add it to my reading list. :)
Maybe this is true in your culture, but do you have evidence for this? I have no evidence, only my perceptions.
My perception, in my culture, is that spreadsheets are stereotyped as being for “business people” in general, or for very organized people, or for “numbers-oriented people”.
My perception, in my culture, is that “business people in general” and “numbers-oriented people”, and maybe “very organized people”, are stereotypically male, i.e., that women are stereotyped as less likely than men to be in those groups.
Although secretaries exist in my culture, I perceive their numerousness and cultural prominence as being low now, much decreased since their peak in the 20th Century, I guess because computers and economic downturns made businesses decide that they could do with many fewer secretaries.
TBH, I figured it was excluded because there is one hyper dominant spreadsheet implementation (Excel) that was created by a much maligned company (Microsoft).
Though I suppose that might be why there is one hyper dominant implementation. If people were more interested in spreadsheets we might have a lot more options and genomic studies would be safe from coercion errors.
Xcode used to do code folding, and I think you could choose what the default state was.
It seemed like it would be useful, but it turned out to be meh at best. In fact, it might still be available somewhere, I wouldn’t know.
The folding features are still there. I didn’t see a fold by default setting though.
So header files aren’t a bad idea after all?
Header files are the best thing ever! I wish so hard that Rust kept .crate files (the old ones, like .mli, not the Cargo tarballs).
With header files don’t you have to write all your type signatures and type definitions twice? I hated them when I worked with OCaml.
It makes more sense to just implement the described feature in IDEs/editors, instead of making every programmer copy/paste their types/signatures to a separate file.
Dunno about OCaml, but in C public types go in the header, private types in the implementation, so all that gets duplicated is the function prototypes.
It might be nice to allow function definitions to be abbreviated, so the types don’t have to be repeated, but on the other hand it might be bad for the full prototype to be so far from the body. On the gripping hand, I usually have both files open so that I can make sure the implementation and documentation match.
But my prehensile tail points out that having the documentation, type signature, and body right next to each other in one file is a lot more convenient, and rustdoc does a fine job of pulling out and presenting just the public interface.
I just went full-circle thinking about this.
Yes, header files can totally serve this purpose. They provide a nice overview of what’s the public interface of a module.
And, in the spirit of improving UX (which is what this post is about), i think it would be crucial to have seamless integration between header files and implementation files. E.g., it should be trivial to toggle between them; if i “go to definition”, the definition on the implementation file should be open; if i edit the signature of a function in the implementation file, the change should be made in the header file automatically; etc.
But, if we had all this, then the header file would become almost a “projection” of the implementation file. In practical terms, it’d be almost a modified view of the same source code. Something that could be entirely derived, but for some reason still exists as a separate source file.
I.e., header files with good UX would be something very very similar to the feature this post is proposing; minus the need for duplicated information in two different files :)
Why would you need header files when all the IDEs since 30 years have an “outline” feature?
As @fanf mentions, the article of which we’re talking touches on this. But, if this is a genuine question(*), i also commented on why the article’s proposal is IMO superior to an “outline” view elsewhere on this discussion :)
*: I initially read it as a kind of slight, “don’t you know this has been a common thing for decades?”, but you probably didn’t mean it that way. Or at least i hope so! hehe
I read a great blog post earlier which explains that IDEs lack the right kind of outlining.
I don’t understand, the post doesn’t mention at all the IDE’s existing “outline” feature which already does everything mentioned here. Like when I open a code file, I have a separate panel which tells me all the methods in a file with their parameters, etc. and this has worked pretty much in every IDE I’ve ever worked in. e.g. this : https://ibb.co/48nXwn4 ; this was already a thing in dev-c++ in 2005.
Header files have the documentation and the complete public types. That outline looks fine for navigating a source file, but it isn’t a substitute for a header or for Matklad’s folded view.
Ideally, (part of) the implementation would be a projection of the header, because you really want the interface to be primary. And no, I have no idea how to implement that, at least not one that’s also in any way reasonable.
(One company that I worked at did automatic header generation as a matter of course. It worked. It was…hmm…let’s just say I adopted many of the habits and practices from there, but not that one)
Yeah, I think that every modern language server/IDE has supported seamless navigation between header and source files like this for 10+ years 😅
Header files should be auto-generated. The painful thing about headers is having to make changes in two places.
For example, Xcode displays auto-generated “headers” of Swift system frameworks.
We used to generate ObjC headers all the time. It has upsides and downsides.
The upside is obviously less duplication.
However, the downsides are also significant, because you have the implementation driving the interface.
Yes. It’s good that languages are less redundant nowadays, but we definitely lost some nice ergonomic and information-hiding aspects of headers.
When I was working in C# back in the 00s, I invented a “header file” format for C# (basically the public/protected class and function declarations with no bodies) and made a little tool that generated it from a DLL.
I think we did have Intellisense in Visual Studio back then, but that just shows one method at a time, whereas this let you see the full picture of a library (or your own code).
My main problem with header files is that any refactor that involves changing a function’s signature requires changing it in two places, which is annoying.
Even better - Pascal’s interfaces ;)
Though it’s still tangential to what the author is asking for, i.e. overally easier file navigation, which I 100% agree with.
At least in VSCode I can fold only specific level, which works most of the time.
I always liked header files for this reason. It provides an overview of what you can do with a module.
Objective-C has this: Smalltalk and C. And the overhead between the two parts was essentially nil. Pretty awesome when you manage to take advantage of it.
Of course it has the disadvantage of being a car crash of two distinct languages. Objective-S tries to fix that: The Four Stages of Objective-Smalltalk.
I always struggle with the YAGNI concept because, in my experience, writing generic code is not more complex than writing over-specialised code and can often lead to simplifications for the specific use case. I then often end up being able to either copy code from one project to another, or factor it out into a library that both can depend on.
The one constant in software engineering is that your requirements will have changed by the time you’ve implemented them. Writing code that’s flexible in the presence of changing requirements is software engineering.
I like to distinguish “generic” from “general”, whereby “generic” means “independent of irrelevant details” and “general” means “handling many cases”. I believe in YAGN generality, but like you, I value genericity.
I tend to break YAGNI defensively. If, early in the implementation work, my gut tells me that the current “thin” specs are the result of an inability to formulate use cases in sufficient depth on the product management side, then there’s a very high chance that I am, in fact, GNI. In that case, I will totally write the more generic version, because there are two things that can happen one year down the line:
What if I don’t have time for that refactoring? Haha. If I can’t find an hour for a simple feature-dropping refactoring, I definitely won’t find the way more refactoring hours required to morph the code that implemented what we thought we needed a year ago into the code that implements what we actually understand we need today.
The undelying assumption in the “but what if you don’t have time to refactor it?” is that, as more requirements are formulated, code only needs to be “extended” to cover them, you just “add” things to it. That’s one of the worst sermons of the engineering management cargo cult.
Also, code written by successive generations of YAGNI people is some of the worst there is. It never evolves into a generic solution, it just snowballs into umpteen particular solutions held together by some inconsequential generics.
For me, YAGNI is not necessarily saying you shouldn’t write generic code. I’ll try to explain with an example from my team’s current project. We’ve been writing a program to manage hardware and the software running on it because none of the infrastructure management software we evaluated fit our needs well enough. So we have a daemon on our edge hardware and a service running in the cloud that coordinates that hardware and hands out configuration. That service connects to a database, which in my team’s case is Postgres. The engineer who wrote the initial draft of this system, however, added in a constant and some code to be able to work with any SQL database. However, it is highly unlikely that we would ever use a database besides Postgres since we’re already using it. This is the sort of thing YAGNI is about - you’re working on a team that has infrastructure in place, you know what you’re writing is internal and not meant to be consumed/built on by others. Why would you put in the work to support different types of infrastructure than you need to?
So yeah, requirements will change, but what you’re building on top of usually will not. So don’t try to support more than what your team uses unless you know with certainty that you need more.
The deciding factor for me is usually indirection. If the more specific approach lets me avoid a layer or two of indirection, I’ll usually pick that. If it doesn’t, I’ll go with the broader version.
At my previous job, I saw a feature added that wasn’t once used in the 11 years I was there. It stared out as a way to segment SS7 traffic to different backends, and a later addition of SIP had to support the same concept, yet it was never used. It was discussed from time to time to use it, but never came to fruition. And it would take more work, and coordination with another development team and ops, to fully remove the feature.
I think YAGNI is against adding feature or making code more complex in anticipation of future value. I have never seen anything against just making the code just as simple (or even simpler) by using a more general version.
I’ve heard this called “implement for today, design for tomorrow” and I think it is fundamentally at odds with YAGNI. The problem is it is (by definition?) more complex to read, and therefore maintain, tomorrow. There’s a quote about it:
The quote is from the XP book, but I think in the context of this thread we can safely replace the word with YAGNI. Have you made the bet in the past and it hasn’t paid off, or have you just not tried it yet?
The bet is also that when you do need to make that change, you will have more information, so you will be able to do a better job then than you will be by anticipating those needs now.
I was turning “micro services” back into a well-structured monolith back in 2003, turned a set of ~100 processes into a single Java JAR by 2005.
Improved performance and reliability by roughly 100x. Number of services provided increased, code decreased. Development velocity dramatically increased as the previous tangle was unmaintainable.
In-Process REST at the BBC
Used REST principles and a few well-chosen interfaces to clearly separate the modules within the monolith. Wouldn’t use an IPC mechanism, though interesting that they thought this was necessary.
So far, I would say it has raised the floor rather than raised the ceiling. It’s much easier to generate code that has already been written by 10,000 other people 10,000 times by just describing it. Getting it to write something no one has ever written before or even very few people have written is still quite difficult.
One thing I think it helps with is turning specialists into “T shaped” coders. Many projects involve some creative and/or intricate core surrounded by a whole bunch of common yet multi disciplinary stuff (infra, deployment, website, app front end/backend, database, etc). This helps you turn a proof of concept into a full-blown project.
I think this gets closer to the mark. I’m by some accounts a good programmer. The floor hasn’t been raised for me, but it has made me a lot more productive. LLMs are taking a lot of the effort out of working out ideas, writing initial concepts etc. and while I’m still crossing the finish line with code that I wrote myself LLMs are making the easy parts faster.
So the problem I have with this is that we shouldn’t really be writing code that has been written before by 10000 other people 10000 times, we should be reusing it, right? Other than for educational purposes in which case having it generated for us is not going to exactly be the right thing either.
To me, the fact that GenAI does appear to be useful in these case in software development seems like a huge failure, and one that should be fixed. And GenAI doesn’t help fix the failure but rather helps entrench it. We really need to look at why we have failed in this way.
There are other use cases for which it can be quite helpful, I think, but this ain’t it.
Yes, I worry about this. It raises the floor by making the floor into a tarpit. :-(
I’ve been waiting for a clear, simple explanation of what
io_uringis, or does, or is for.This, sadly, is not it.
I guess I’ll have to keep waiting.
The context of io_uring is a process that has lots of concurrent IO in flight per thread. The kind of thing that traditionally was built around select() or poll(), or more recently kqueue or epoll. Event-based systems like node.js.
The core loop of this kind of program has a lot of back-and-forth between the kernel and userland because the kernel doesn’t remember much about what the process is doing from one system call to the next. It’s like,
The basic idea of io_uring is to make it more like
That is, the program can submit a batch of IO for multiple file descriptors (network connections, listening sockets, disk files, etc.) and get pending results from previous batches, in one go. Instead of an IO syscall per file descriptor per loop, there’s just one syscall per loop. In effect, the program has multiple read()s and write()s in flight concurrently.
The “ring” refers to how the list of requests and results are laid out in memory. In more advanced uses of io_uring, the program can set up the ring buffers so that it can add requests and read results without explicitly handing off to the kernel, so its main loop can have zero system calls.
As ~david_chisnall said this setup is like the way a paravirtualized kernel talks to its hypervisor. It’s also like the way a driver talks to a device that does DMA. Which is why ~matklad complains about blocking APIs being wrong: io_uring is much closer in design to the way the hardware works.
At the core, it’s a ring buffer that’s mapped into userspace. Rather than issuing system calls to initiate operations (typically I/O, though it’s grown to support some other things, so the
io_prefix is somewhat misleading now), you put them into the ring buffer. If you’ve used a system such as QNX with asynchronous system calls, this is probably quite familiar. It’s also likely to be familiar if you’ve ever programmed against a hypervisor interface: domain transitions to a hypervisor are (or, at least, were) much more expensive than system calls and hypervisors were typically designed to support multiple cores, so they used ring buffers with asynchronous communication for most device interfaces.There’s also a lot of stuff layered on top. The most interesting bit, from my perspective, is that the
io_uringequivalents of most system calls that would create a new file descriptor can (optionally) not register that file descriptor in the process’ descriptor table. This is useful because it means that you can work around one of the annoyances of POSIX. POSIX mandates that each new file descriptor must be created at the lowest free file descriptor number. This was originally a clever hack in early UNIX to allow the shell to redirect standard in/out/error (the first three file descriptors) by simply closing them and then opening something else. It was obsolete whendup2was invented (though necessary for legacy compatibility). It was painful when threading came along because it made everyopen(orsocket,accept,pipe, or whatever) and everycloseglobal synchronisation points within a process. Withio_uring, you can avoid this and make the operations local to a ring (on the kernel side, you can have a single thread that processes the ring state and dispatches to kernel work queues for the lon-running things). This means that you can batch up an open, read, close sequence and have them operate with no synchronisation with the rest of the process state.The ability to batch commands is what made
io_uringinteresting for other things as well, such as process creation.QNX syscalls aren’t async. The main messaging primitive, MsgSend, is tightly coupled to the scheduler in a manner that makes it appear to the caller almost as a vanilla function call. Maybe pulses are, I can’t tell from what @BenjaminRi brought up the last time this discussion came up: https://lobste.rs/s/a7hvcs/qnx_demo_disk_extending_possibilities#c_haq7v1
Ooops, I’ll try to remember that next time! I never used QNX, but I remember their marketing saying that they were faster because they did async system calls. I wonder what they were talking about.
[Apologetically] I am already lost.
I’m not a programmer. I have experimented with QNX, yes, but not programming it.
Well, it’s a tool for low-level programmers, so if that’s not you then it’s fine to ignore it. That said, I’ll try a higher-level explanation:
Normally, when you want the kernel to do something, you issue a system call. On modern systems this is a dedicated instruction, but traditionally it was a software interrupt. When this happens, the CPU jumps to a specific address and switches to executing in kernel mode. These happen together because you don’t want to switch to a more privileged mode and give unprivileged code control over what runs.
The speed of this switch has changed a bit over time. It used to be very slow. Then CPU vendors made it faster. Then they discovered that a bunch of the things that made it fast led to side channels that let userspace processes leak kernel data, so now it’s slower again.
When the kernel enters the system-call handler, it has to save some state for the userspace process, work out which system call was invoked, and do a bunch of other book-keeping work, then do the thing userspace asked for.
There are two sources of overhead here. The first is that you need all of this book keeping for a single operation. The second is that it’s a synchronous transition and so the kernel also has to synchronise what it’s doing.
Xen tried to avoid this for all guest VM I/O and exposed devices as ring buffers. A ring buffer is just a block of memory that you pretend is arranged in a circle. When you get to the end of the block, you go back to the start. Each entry was filled in as a request in the guest and then replaced with a response from the hypervisor. This meant that you could put a load of requests into the ring without explicitly invoking the hypervisor at all. The only time you needed to do anything explicit was when the ring transitioned from empty to not-empty. This let the thread that was handling the I/O outside the guest[1] sleep when the ring is empty and wake when there’s something to do, and then keep running as long as requests are coming in.
If you have one core, the guest would batch up a bunch of things, then the threat that implemented the device would run for a bit and process some or all of the batch, then the guest would run again. That’s more efficient than a transition for each operation, but it’s an even bigger win when you have multiple cores because both sides of the ring can be running at the same time and the guest can do other work at the same time as the device thread runs.
This is closely tied to the other advantage. Most I/O is intrinsically asynchronous. When you read from disk, the kernel might have a copy of the data in the buffer cache and be able to just copy it to you, but it may not. If not, it doesn’t (at least since the kid ‘90s) sit there reading data, it sends some commands to the disk controller, which then uses direct memory access (DMA) to write the requested blocks into memory somewhere. Similarly, if you’re writing to disk, the network, or some other device, you,just set up the operation and then it runs while you do other things. It’s useful to be able to expose this higher up the stack. In Xen, a guest would write a bunch of blocks to disk by queueing them up in a device ring, then the device emulator thread would run and tell the disk controller to write them to disk, then the CPUs are free to do other things. When the disk controller is finished, the completion is threaded through the other layers and they can reuse the slots in the ring buffer for new requests.
The io_uring model is similar to this, except it deals with the kinds of things userspace processes operate on, rather than the things VMs deal with. It has sockets, files, and so on, rather than block devices and Ethernet devices. The advantages are the same though. You can queue up a bunch of work and the kernel can handle it asynchronously while you do other things.
[1] In the case of Xen, this is actually in another VM, but that’s not relevant for the purpose of this explanation.
Fascinating. Thank you!
I am now wondering if there’s an article in this… If I try, might I bounce it off you for tech review?
There might be something interesting in the evolution of interfaces for this kind of thing.
Absolutely!
Most syscalls consist of three separate steps, which as far as userspace is concerned, happen atomically (because the calling thread is blocked for the duration of the syscall):
io_uring allows applications to do the steps individually, and to do other things between the steps. In particular, if an application wants to do syscalls A and B , it can do the steps in the order: A1, B1, A2, A3, B2, B3. Applications can also wait on any of their submitted requests and deal with the results in whatever order they happen to finish.
Separating the request from the response also allows you to batch multiple requests into a single syscall. Hardware is so fast nowadays that syscall overhead can often be the limiting factor, as I discovered a while ago:
mkfile(8) is severely syscall limited on OS X
Thanks. That is slightly more comprehensible.
Only slightly, though. It sheds a little light but not a lot.
It sounds a little like queuing or batching calls to the kernel, in a similar manner to DMA-driven disk access as opposed to PIO-based access. Fair?
That is exactly what it’s like, and in fact what it’s inspired by. The application-kernel boundary is analogous to the CPU-device boundary.
Oh good! I am glad that I got it right. :-) Thanks.
These are pretty good introductions to the why and what of the basic mechanism:
https://unixism.net/loti/
https://kernel.dk/io_uring.pdf
I started to read both.
Both seem to be aimed at an audience of programmers, which sadly for this context, I am not.
Communicate with kernel threads using atomics instead of switching into kernel mode yourself? In the limit, only switch into kernel mode when you’ve run out of other stuff to do, and you want to sleep until stuff arrives.
At some point I need to write an actual article about this, but I’ve recently been thinking that “blocking” APIs in general are a design mistake: everything is fundamentally concurrent, but OSes use blocking APIs to hide this concurrency, which tends to inevitably break, and mixes concurrency with parallelism. Even something as simple as “spawn a process, collecting both its stdout and stderr” requires proper concurrency. See how Rust’s
Command::outputcan’t be expressed using std-exposed APIs.Which is the smaller problem! The bigger problem is that we don’t actually know how to write concurrent programs. This is an unsolved area in language design. And the systems reason for why the problem is unsolved is, thanks to the existence of blocking APIs, languages are allowed to shirk here, moving what is fundamentally a language-design issue to the OS-design.
IO-uring (and previously, hilariously, JavaScript) models the problem of concurrency more directly, without introducing a midlayer. Hopefully it will create enough of evolutionary pressure for language design to get the problem solved finally.
There was an interesting period about 25 years ago, when the web was growing fast and really stressing the capabilities of the hardware and operating systems of the time. This is when the c10k problem was identified. The performance pressure led to a bunch of interesting operating systems research.
I’m pretty sure I read a paper around that time which described a kernel/userland API with a design roughly along the lines of io_uring — though its point of comparison was hardware interfaces with command queues and DMA buffers. I had a very vague memory of Vivek Pai’s IO-Lite but that has a different shape so there must be another paper I’m failing to remember.
I wondered if a fully asynchronous kernel API could solve the M:N threading problems that were happening around the same time. But that whole area of research dried up, as the problems were dealt with by
sendfile(), 1:1 threading, and computers getting faster. Boring!I do not actually see a clear specification of what the problem is in that article, though.
It was really hard to handle 10,000 concurrent connections from one server in 1999. The challenge was to make it easier, to make the kind of performance engineering that went into cdrom.com available to a lot more systems.
IIRC, the fundamental challenge is handling 10K long-running connections that are not very active. If each of those connections is actually producing significant load on the hardware, either CPU load or I/O load, then the hardware can’t handle 10K connections anyway, because we have neither 10K cores nor 10K independent I/O channels.
You’re probably thinking of Matt Welsh’s SEDA paper.
https://search.worldcat.org/title/892830129
I found a copy of the SEDA SOSP paper which is a bit shorter than Welsh’s PhD thesis :-) It seems to be about getting more concurrency out of Java, not about changing the userland/kernel API.
It’s a compelling argument, but I think in some ways async APIs end up constraining the design space of the framework/OS/whatever offering it. For example, QNX’s main message passing primitives are blocking so that, on the fast path, context is switched to the callee without a pass through the scheduler.
More broadly, this paper argues that it’s not possible to make a fast and DoS-resistant async message passing system: https://www.researchgate.net/publication/4015956_Vulnerabilities_in_synchronous_IPC_designs
You absolutely hit the nail on the head. Given how absolutely fundamental asynchronous computation is to how computers work, it’s surprisingly underexplored.
I think this is partly due to a decline of groundbreaking OS research, and partly because there are barely - if any - modern languages purpose built for doing deep, systems research. The problems become apparent almost instantly. At the latest when you’re a few hundred lines into your first kernel, you’ll be presented with DMA interfaces for all kinds of peripherals. What now?
This is not the kind of problem that most language designers are facing. Most PLs are made to solve application-level problems, instead of having a principled, hardware-first approach to language design. I think Rust with embassy-rs has been solving some of these challenges in a pretty nice way, but it shouldn’t end there.
I agree on the lack of OS research. Mothy gave a keynote at SOSP a few years ago pointing out that twenty years ago there were multiple new kernel papers each year at OSDI / SOSP whereas now we average less than one. I’m slightly biased, because the last paper I submitted with a new OS design was rejected on the ground that we didn’t do a performance comparison to an OS that required ten times as much hardware and that we were possibly vulnerable to an attack that, by construction, cannot happen on our system. Ho hum.
I disagree on language research though. There’s a lot of interesting work on asynchrony. Most PL research takes at least ten years to make it into mainstream programming languages. Verona’s behaviour-oriented concurrency model is designed to work directly with things like DMA and we’ve ported it to C++, Rust, and Python. It works best with a memory management model that puts object graphs in regions and allows them to be atomically transferred, which depends on a type system that can do viewpoint adaptation. We’re not the only people working on things like this and I expect mainstream languages in the 2030s to have a lot of these concepts.
Timothy Roscoe, “it’s time for operating systems to rediscover hardware”
Which kicks off with the observation that only 6% of the papers at the USENIX OSDI conference were about operating system design and implementation.
That’s brutal. Too brutal in my opinion to only come from spurious rejections. From the outside, I would suspect the main drive is that nobody makes kernels any more: apart from specific niches like real time or formal verification, the world has basically been taken over by NT, Linux, and XNU. And a major reason behind such concentration is that every kernel needs to have dedicated drivers for the insane diversity of hardware we use — except NT, hardware vendors being business savvy enough to write the driver for us. (See The 30 Million Lines Problem.)
Those three kernels are big. Making any addition or change to them is likely a significant endeavour. We likely need the ability to quickly write practical kernels from scratch if research is to take off again. But no one can possibly write a practical kernel if they have to, let’s be kind, port all the driver code from Linux. So the first step has for hardware vendors to design interfaces for humans, and then give us the manual. But given how suicidal this would be business wise (who would ever buy hardware that doesn’t come with a Windows driver out of the box?), we probably need regulators to step in: say the US, or the entirety of the EU, ban the sale of any hardware for which the same company distributes software — or at the very least, proprietary software. With such a setup, I would expect hardware vendors to quickly converge on relatively uniform hardware interfaces that aren’t stupidly complex.
But then there’s the other side of the fence: user space. Since the ossification of OSes we came do depend on a huge stack of software that would take significant effort to port anywhere not POSIX-ish. And that’s for the stuff we have the source code of…
I don’t really think that’s true. For a research project, you need a few drivers and most people just pick up the NetBSD ones (the RUMP kernel work makes this trivial). There’s a big unexplored space for both big and small operating systems. With a very small team, we were able to write an RTOS that has a solid FreeRTOS compat layer and can run most existing embedded software (source compatible) and we were designing and building the core that ran it as well. No one has really built an OS for cloud computing, but it could remove a lot of stuff that desktop operating systems need (no IPC or storage layers, since those will both sit on top of a network).
fwiw I think existing unikernels fit that bill (e.g. unikraft, nanos). They mostly target virtualized hardware like KVM, and seem pretty focussed on the cloud use-case (fast boot, small images, and efficiency). Being in KVM-land certainly removes a lot of implementation complexity, since bringup and platform init is mostly done.
I disagree. I think unikernels are applications for cloud computing. They’re hampered by the fact that the OS that they run on (Xen, Hyper-V, Linux) is not designed for the cloud. They have network devices, for example, but a good cloud OS would make it easy to take advantage of hardware TLS offload and terminate connections directly in a unikernel.
Unikernels have had almost no commercial adoption precisely because there isn’t a good OS to run them on. The benefits of unikernels are incompatible with the billing models of cloud systems. A unikernel invocation may consume a few MiBs of RAM and a few CPU seconds (or hundreds of CPU milliseconds). The operating systems that run them are not set up to handle this at all. They are intended to account for CPU usage in minutes and RAM in GiBs.
If you’re running a unikernel in KVM, there’s little benefit in it over just running a Linux process. You then get to share a TCP/IP stack and so on with other processes and amortise more costs.
Ah then I misunderstood your original comment. So it sounds like you’re unhappy about the currently dominant hypervisor abstraction, that I would agree with. The interfaces for efficient resource sharing are pretty borked.
Are you talking about serverless here? Unikernels can be long-running services, at least that’s how I’ve run them in the past.
For me, the big benefit of unikernels is their ability to scale down. If you’re handling sustained throughputs of thousands of network messages per second or more, there are some benefits, but if you’re implementing services that have long idle periods of no activity and bursts of very high usage, unikernels’ abilities to either exit completely and then restart in one network round trip time or scale back to almost no resource usage is a huge win. Cloud billing systems are not set up for that at all.
I don’t really like the term serverless, because you don’t get rid of servers. The way most FaaS services are implemented today comes with ludicrous amounts of overhead. Each service starts in a lightweight Linux VM, that runs an OCI container, that runs a language runtime and some support code, which then runs the customer code. We estimated an efficient system with good hardware-software co-design to do build an attestable base with TLS flow isolation to individual services would allow at least an order of magnitude denser hosting and provide better security properties.
Thanks for the link. In your view, would the BoC as a concurrency paradigm allow for a competitive implementation for e.g. performance-critical embedded systems? Are there places where you’d say “this place needs an escape hatch to close the last few percent perf gap” or so?
I’m genuinely interested, because I was looking for things beside stackless async/await that would work well for highly constrained environments.
Possibly. BoC is really equivalent to stackless coroutines, but with a coordination layer on top and an ownership model that makes it safe. Each
whencaptures the things it need to complete and runs without the parent’s stack.The problem for BoC on resource-constrained systems is that any behaviour can spawn arbitrary numbers of other behaviours, which can exhaust memory, so you’d need to be a bit careful. It’s probably no different to avoiding deep recursion though.
You definitely wouldn’t want to use the Verona runtime for small embedded systems (it’s nice in a kernel though. I ported it to run in the FreeBSD kernel a few years ago) but I have been pondering what an embedded BoC runtime would look like.
I mean, utah2k was written nearly a quarter-century ago. (There is a certain irony of someone from Bell Labs writing this, considering the output of Bell Labs doing a lot to slow systems research…)
In the 1970s, this was addressed. Quite a few popular OSs back then treated all base I/O calls as async. Blocking was a unix thing. Out of necessity, the Berkeley unix apis on unix tried to walk that back. (They also hacked a few other unix design choices)
Programmers on those older OSs successfully built multiuser database and timesharing systems.
For simple programs, many liked the simplicity of the unix style,
IMNSHO, the underlying problem is that we decided a long time ago that the procedure call (subroutine call, function, method, etc.) is our fundamental model of abstraction. Not just for interacting with the operating system, but pretty much for everything. (“Decided” might be overstating it…it just sort of happened for various reasons, many good at the time).
I talked about this, somewhat haphazardly here: Can Programmers Escape the Gentle Tyranny of call/return?].
You can also find some of that in Mary Shaw’s Myths and mythconceptions: what does it mean to be a programming language, anyhow?, and of course many, many others have made observations about the limitations of procedure calls and proposed alternatives as the basic building blocks.
Lobsters thread: https://lobste.rs/s/vypmkr/myths_mythconceptions_what_does_it_mean
I don’t think this has been submitted here, but a piece commenting on it was submitted: https://lobste.rs/s/alzaim/thoughts_on_gentle_tyranny_call_return.
I’m not 100% convinced that concurrent-only is the way to go (and I say this as a JS dev who also uses Javascript as a good example of what concurrent-only could look like). But I agree that most languages need to choose one or the other. You either go all in on concurrency (like in Javascript, where synchronous APIs are rare and usually only exist a special-case exception alongside an equivalent async API), or you accept that concurrency is always going to be an opt-in feature with less support that will require its own way of doing things.
I’ve seen this a lot in Python, and I think we’re seeing it now again in Rust, where async support gets added, and creates a split ecosystem. You can’t mix and match, so you need to decide: am I using the sync APIs or the async ones? I wonder if there are other concurrency ideas that could mitigate this somewhat, but I think it’s just a fundamental divide. Reading files, interacting with sockets, executing processes etc are fundamental OS-level tasks, and deciding how you’re going to do those cuts to the core of a language’s standard library. In Rust, I know that some of the big proponents of the Async Way are aware of this problem, but I think their goal is to push async as the predominant way, and I’m not sure Rust is set up for that yet.
(As an aside: I know C# also was synchronous and then added the async keyword - are there similar issues there? Is there a divide between the sync and async worlds?)
The thing is, you can’t choose only one! Here’s a mundane, boring example:
You spawn a process. You want to collect its stdout and stderr. You can’t do just
because this might deadlock. The child process might interleave printing to stderr&stdout, so you have to interleave reading them. What you want to do here is to issue both read syscalls concurrently and wait for one of them to finish. So you end up writing platform specific code like this:
https://github.com/rust-lang/cargo/blob/master/crates/cargo-util/src/read2.rs
Another case you always hit when programming mundane things is that something does a
readcall, and you really want to “cancel” that read (even if it currently blocks). This again necessitates pretty horrific work-arounds with installing a process-global signal handler: https://github.com/rust-lang/jobserver-rs/commit/4c8a5985e32fa193350f988f6a09804e565f0448In the opposite direction, DNS (and, until io_uring, even file IO) — there’s just no non-blocking APIs for these!
So, I don’t think it’s up to the languages to fix this, really. It’s the OS which first needs to provide reasonable API. Only then can language designers of production languages figure out how to express the concurrency in the source code (to echo David’s comment, I suppose we might have figured how to deal with concurrency in academia, but this certainly haven’t quite percolated to day-to-day languages. Go & Rust seems like they are improvements, but also both feel quite far from optimal).
There are plenty of nonblocking DNS APIs. The gap is in the higher-level name service APIs that bundle up lots of non-DNS name databases, such as the hosts file, mDNS, NetBIOS, NIS, etc.
You can’t perfectly choose one, but I think you can abstract surprisingly well by choosing one route or the other. As you point out yourself, we’ve done pretty well pretending that things like file IO (and even memory access) are synchronous, idealised things. Yes, there are weird edge cases where the abstraction breaks, but most of the time it holds surprisingly well.
Going the fully async route brings its own problems, but it’s still in many ways just a different abstraction. Many async libraries handling file IO just run synchronous APIs on a different thread, because it’s simple, it works, and we still get the concurrency we were expecting.
So I think a lot of this is about the abstractions we chose to use on top of these sorts of APIs, and I think language designers have a lot of leeway there. Yes, the OS APIs will affect those abstractions - what works well, what works efficiently, what leaks constantly into the abstraction layer etc - but there is no shortage of interesting approaches to concurrency on top of largely synchronous APIs.
Until recently, Linux didn’t have usable async file IO, so you had to do that.