The TypeScript dev lead posted this response about the language choice on Reddit, for anyone who’s curious:
(dev lead of TypeScript here, hi!)
We definitely knew when choosing Go that there were going to be people questioning why we didn’t choose Rust. It’s a good question because Rust is an excellent language, and barring other constraints, is a strong first choice when writing new native code.
Portability (i.e. the ability to make a new codebase that is algorithmically similar to the current one) was always a key constraint here as we thought about how to do this. We tried tons of approaches to get to a representation that would have made that port approach tractable in Rust, but all of them either had unacceptable trade-offs (perf, ergonomics, etc.) or devolved in to “write your own GC”-style strategies. Some of them came close, but often required dropping into lots of unsafe code, and there just didn’t seem to be many combinations of primitives in Rust that allow for an ergonomic port of JavaScript code (which is pretty unsurprising when phrased that way - most languages don’t prioritize making it easy to port from JavaScript/TypeScript!).
In the end we had two options - do a complete from-scrach rewrite in Rust, which could take years and yield an incompatible version of TypeScript that no one could actually use, or just do a port in Go and get something usable in a year or so and have something that’s extremely compatible in terms of semantics and extremely competitive in terms of performance.
And it’s not even super clear what the upside of doing that would be (apart from not having to deal with so many “Why didn’t you choose Rust?” questions). We still want a highly-separated API surface to keep our implementation options open, so Go’s interop shortcomings aren’t particularly relevant. Go has excellent code generation and excellent data representation, just like Rust. Go has excellent concurrency primitives, just like Rust. Single-core performance is within the margin of error. And while there might be a few performance wins to be had by using unsafe code in Go, we have gotten excellent performance and memory usage without using any unsafe primitives.
In our opinion, Rust succeeds wildly at its design goals, but “is straightforward to port to Rust from this particular JavaScript codebase” is very rationally not one of its design goals. It’s not one of Go’s either, but in our case given the way we’ve written the code so far, it does turn out to be pretty good at it.
And it’s not even super clear what the upside of doing that would be (apart from not having to deal with so many “Why didn’t you choose Rust?” questions)
People really miss the forest for the trees.
I looked at the repo and the story seems clear to me: 12 people rewrote the TypeScript compiler in 5 months, getting a 10x speed improvement, with immediate portability to many different platforms, while not having written much Go before in their lives (although they are excellent programmers).
This is precisely the reason why Go was invented in the first place. “Why not Rust?” should not be the first thing that comes to mind.
I honestly do think the “Why not Rust?” question is a valid question to pop into someone’s head before reading the explanation for their choice.
First of all, if you’re the kind of nerd who happens to follow the JavaScript/TypeScript dev ecosystem, you will have seen a fair number of projects either written, or rewritten, in Rust recently. Granted, some tools are also being written/rewritten in other languages like Go and Zig. But, the point is that there’s enough mindshare around Rust in the JS/TS world that it’s fair to be curious why they didn’t choose Rust while other projects did. I don’t think we should assume the question is always antagonistic or from the “Rust Evangelism Strike Force”.
Also, it’s a popular opinion that languages with algebraic data types (among other things) are good candidates for parsers and compilers, so languages like OCaml and Rust might naturally rank highly in languages for consideration.
So, I honestly had the same question, initially. However, upon reading Anders’ explanation, I can absolutely see why Go was a good choice. And your analysis of the development metrics is also very relevant and solid support for their choice!
I guess I’m just saying, the Rust fanboys (myself, included) can be obnoxious, but I hope we don’t swing the pendulum too far the other way and assume that it’s never appropriate to bring Rust into a dev conversation (e.g., there really may be projects that should be rewritten in Rust, even if people might start cringing whenever they hear that now).
While tweaking a parser / interpreter a few years ago written in Go, I specifically replaced a struct with an ‘interface {}’ in order to exercise its pseudo-tagged-union mechanisms. Together with using type-switch form.
These day’s I’d actually make it a closed interface such that it is more akin to a tagged-union. Which I did for another project which was passing around instances of variant-structs (i.e. a tagged union), rather than building an AST.
So it is quite possible to use that pattern in Go as a form of sum-type, if for some reason one is inclined to use Go as the implementation language.
A relevant quote is that C# has “some ahead-of-time compilation options available, but they’re not on all platforms and don’t really have a decade or more of hardening.”
Yeah Hjelsberg also talks about value types being necessary, or at least useful, in making language implementations fast
If you want value types and automatically managed memory, I think your only choices are Go, D, Swift, and C# (and very recently OCaml, though I’m not sure if that is fully done).
I guess Hjelsberg is conceding that value types are a bit “second class” in C#? I think I was surprised by the “class” and “struct” split, which seemed limiting, but I’ve never used it. [1]
And that is a lesson learned from the Oils Python -> C++ translation. We don’t have value types, because statically typed Python doesn’t, and that puts a cap on speed. (But we’re faster than bash in many cases, though slower in some too)
Now that I’ve worked on a garbage collector, I see a sweet spot in languages like Go and C# – they have both value types deallocated on the stack and GC. Both Java and Python lack this semantic, so the GCs have to do more work, and the programmer has less control.
There was also a talk that hinted at some GC-like patterns in Zig, and I proposed that TinyGo get “compressed pointers” like Hotspot and v8, and then you would basically have that:
Yeah, I’m kind of curious about whether OCaml was considered at some point (I asked about this in the Reddit thread, haven’t gotten a reply yet).
OCaml seems much more similar to TS than Go, and has a proven track record when it comes to compilers. Maybe portability issues? (Good portability was mentioned as a must-have IIRC)
Not sure what more TypeScript would have needed. In fact, Flow’s JavaScript parser is available as a separate library, so they would have shaved off at least a month from the proof of concept…
Although thinking about it a bit more, I think Nim, Julia, (and maybe Crystal) are like C#, in that they are not as general as Go / D / Swift.
You don’t have a Foo* type as well as a Foo type, i.e. the layout is orthogonal to whether it’s a value or reference. Instead, Nim apparently has value objects and reference objects. I believe C# has “structs” for values and classes for references.
I think Hjelsberg was hinting at this category when saying Go wins a bit on expressiveness, and it’s also “as close to native as you can get with GC”.
I think the reason this Go’s model is uncommon is because it forces the GC to support interior pointers, which is a significant complication (e.g. it is not supported by WASM GC). Go basically has the C memory model, with garbage collection.
I think C#, Julia, and maybe Nim/Crystal do not support interior pointers (interested in corrections)
Someone should write a survey of how GC tracing works with each language :) (Nim’s default is reference counting without cycle collection.)
Yeah that’s interesting. Julia has a distinction between struct (value) and mutable struct (reference). You can use raw pointers but safe interior references (to an element of an array for example) include a normal reference to the (start of the) backing array, and the index.
I can understand how in Rust you can safely have an interior pointer as the borrow checker ensures a reference to an array element is valid for its lifetime (the array can’t be dropped or resized before the reference is dropped). I’m very curious - I would like to understand how Go’s tracing GC works with interior pointers now! (I would read such a survey).
Ok - Go’s GC seems to track a memory span for each object (struct or array), stored in kind of a span tree (interval tree) for easy lookup given some pointer to chase. Makes sense. I wonder if it smart enough to deallocate anything dangling from non-referenced elements of an array / fields of a struct, or just chooses to be conservative (and if so do users end up accidentally creating memory leaks very often)? What’s the performance impact of all of this compared to runtimes requiring non-interior references? The interior pointers themselves will be a performance win, at the expense of using an interval tree during the mark phase.
It’s been a few years since I’ve written any Go, but I have a vague recollection that the difference between something being heap or stack allocated was (sometimes? always?) implicit based on compiler analysis of how you use the value. Is that right? How easy it, generally, to accidentally make something heap-allocated and GC’d?
That’s the only thing that makes me nervous about that as a selling point for performance. I feel like if I’m worried about stack vs heap or scoped vs memory-managed or whatever, I’d probably prefer something like Swift, Rust, or C# (I’m not familiar at all with how D’s optional GC stuff works).
$ go build -gcflags "-m" main.go
.\main.go:8:14: *y escapes to heap
.\main.go:11:13: x does not escape
So the toolchain is pretty transparent. This is actually something I would like for the Oils Python->C++ compiler, since we have many things that are “obviously” locals that end up being heap allocated. And some not so obvious cases. But I think having some simple escape analysis would be great.
Hejlsberg said they got about 3x performance from native compilation and value types, which also halved the memory usage of the compiler. They got a further 3x from shared-memory multithreading. He talked a lot about how neither of those are possible with the JavaScript runtime, which is why it wasn’t possible to make tsc 10x faster while keeping it written in TypeScript.
Yeah but I can get bigger memory wins while staying inside JS by sharing the data structures between many tools that currently hold copies of the same data: the linter, the pretty-formatter, the syntax highlighter, and the type checker
I can do this because I make my syntax tree nodes immutable! TS cannot make their syntax tree nodes immutable (even in JS where it’s possible) because they rely on the node.parent reference. Because their nodes are mutable-but-typed-as-immutable, these nodes can never safely be passed as arguments outside the bounds of the TS ecosystem, a limitation that precludes the kind of cross-tool syntax tree reuse that I see as being the way forward
Hejlsberg said that the TypeScript syntax tree nodes are, in fact, immutable. This was crucial for parallelizing tsgo: it parses all the source files in parallel in the first phase, then typechecks in parallel in the second phase. The parse trees from the first phase are shared by all threads in the second phase. The two phases spread the work across threads differently. He talks about that kind of sharing and threading being impractical in JavaScript.
In fact he talks about tsc being designed around immutable and incrementally updatable data structures right from the start. It was one of the early non-batch compilers, hot on the heels of Roslyn, both being designed to support IDEs.
It’s true that I haven’t watched the interview yet, but I have confirmed with the team that the nodes are not immutable. My context is different than Hejlsberg’s context. For Hejlsberg if something is immutable within the boundaries of TS, it’s immutable. Since I work on JS APIs if something isn’t actually locked down with Object.freeze it isn’t immutable and can’t safely be treated as such. They can’t actually lock their objects down because they don’t actually completely follow the rules of immutability, and the biggest thing they do that you just can’t do with (real, proper) immutable structures is have a node.parent reference.
So they have this kinda-immutable tech, but those guarantees only hold if all the code that ever holds a reference to the node is TS code. That is why all this other infrastructure that could stand to benefit from a shared standard format for frozen nodes can’t: it’s outside the walls of the TS fiefdom, so the nodes are meant to be used as immutable but any JS code (or any-typed code) the trees are ever exposed to would have the potential to ruin them by mutating the supposedly-immutable data
To be more specific about the node.parent reference, if your tree is really truly immutable you need to replace a leaf node you must replace all the nodes on the direct path from the root to that leaf. TS does this, which is good.
The bad part is that then all the nodes you didn’t replace have chains of node.parent references that lead to the old root instead of the new one. Fixing this with immutable nodes would mean replacing every node in the tree, so the only alternative is to mutate node.parent, which means that 1) you can’t actually Object.freeze(node) and 2) you don’t get all the wins of immutability since the old data structure is corrupted by the creation of the new one.
See https://ericlippert.com/2012/06/08/red-green-trees/ for why Roslyn’s key innovation in incremental syntax trees was actually breaking the node.parent reference by splitting into the red and green trees, or as I call them paths and nodes. Nodes are deeply immutable trees and have no parents. Paths are like an address in a particular tree, tracking a node and its parents.
Hm yeah it was a very good talk. My summary of the type checking part is
The input to the type checker is immutable ASTs
That is, parsing is “embarassingly parallel”, and done per file
They currently divide the program into 4 parts (e.g. 100 files turns into 4 groups of 25 files), and they do what I’d call “soft sharding”.
That is, the translation units aren’t completely independent. Type checking isn’t embarassingly parallel. But you can still parallelize it and still get enough speedup – he says ~3x from parallelism, and ~3x from Go’s better single core perf, which gives you ~10x overall.
What wasn’t said:
I guess you have to de-duplicate the type errors? Because some type errors might come twice, since you are duplicating some work
Why the sharding is in 4 parts, and not # CPUs. Even dev machines have 8-16 cores these days, and servers can have 64-128 cores.
I guess this is just because, empirically, you don’t get more than 3x speedup.
That is interesting, but now I think it shows that TypeScript is not designed for parallel type checking. I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library. Though it’s hard to compare, since it also has to generate code
A separate thing I found kinda disappointing from the talk is that TypeScript is literally what the JavaScript code was. There was never a spec and will never be one. They have to do a line-for-line port.
There was somebody who made a lot of noise on the Github issue tracker about this, and it was basically closed “Won’t Fix” because “nobody who understands TypeScript well enough has enough time to work on a spec”. (Don’t have a link right now, but I saw it a few months ago)
Why the sharding is in 4 parts, and not # CPUs. Even dev machines have 8-16 cores these days, and servers can have 64-128 cores.
Pretty sure he said it was an arbitrary choice and they’d explore changing it. The ~10x optimization they’ve gotten so far is enough by itself to keep the project moving. Further optimization is bound to happen later.
I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library.
Work has been going on for years to parallelize rust’s frontend, but it apparently still has some issues, and so isn’t quite ready for prime time just yet, though it’s expected to be ready in the near term.
Under 8 cores and 8 threads, the parallel front end can reduce the clean build (cargo build with -Z threads=8 option) time by about 30% on average. (These crates are from compiler-benchmarks of rustc-perf)
I guess this is just because, empirically, you don’t get more than 3x speedup.
In my experience, once you start to do things “per core” and want to actually get performance out of it, you end up having to pay attention to caches, and get a bit into the weeds. Given just arbitrarily splitting up the work as part of the port has given a 10x speed increase, it’s likely they just didn’t feel like putting in the effort.
But check the chapters, they’re really split into good details. The video is interesting anyway, technically focused, no marketing spam. I can also highly recommend watching it.
Another point on “why Go and not C#” is that, he said, their current (typescript) compiler is highly functional, they use no classes at all. And Go is “just functions and data structures”, where C# has “a lot of classes”. Paraphrasing a little, but that’s roughly what he said.
Acknowledging some weak spots, Go’s in-proc JS interop story is not as good as some of its alternatives. We have upcoming plans to mitigate this, and are committed to offering a performant and ergonomic JS API.
This was interesting and nice to see - I realize that I’ve been thinking about what I like about each approach without being able to state it quite in these terms.
Could we have a type-system for data-oriented programs?
My thoughts are that structural type systems (like TypeScript) can work well - we don’t need to “represent everything with generic untyped dicts and lists” as a sibling comment states; we can jot down the shape upon construction and let the compiler ensure we didn’t make a typo somewhere else. The same type constraints might even help with compilation / execution speed.
When I pondered this a while ago, after first trying out Clojure, I came to a similar conclusion: you can do this with static typing but you need structural types, which OCaml and TypeScript have. However, as the author of this piece points out, one of the things that makes this work so well in Clojure is the way that all the relevant container types work with all the standard library functions:
I think the core distinction here is that Clojure has been designed with a small set of specific data structures in mind, and all of its core abstractions are interchangable over them. Filter operates just as well over maps as it does lists and so on and so forth.
Its not that you couldn’t make this work in other languages, its just much more natural and easier in Clojure, which makes it simple to experiment or react to changing requirements.
It also makes Lisp-style development with a REPL integrated into your editor possible, which is perhaps the real distinguishing feature of this style of development. The author hinted at this when they noted that the Clojure version didn’t bother to write a visualization function until the very end, whereas that was one of the first things they had to do in OCaml.
When I pondered this a while ago, after first trying out Clojure, I came to a similar conclusion: you can do this with static typing but you need structural types, which OCaml and TypeScript have. However, as the author of this piece points out, one of the things that makes this work so well in Clojure is the way that all the relevant container types work with all the standard library functions:
I think the core distinction here is that Clojure has been designed with a small set of specific data structures in mind, and all of its core abstractions are interchangable over them. Filter operates just as well over maps as it does lists and so on and so forth.
Its not that you couldn’t make this work in other languages, its just much more natural and easier in Clojure, which makes it simple to experiment or react to changing requirements.
I’m not convinced that Clojure is special at all in this regard. Pretty much every language has a bunch of standard functionality around lists and maps/dicts. In the quote you include, the author says “Filter operates just as well over maps as it does lists […]”. So what? What languages don’t have the concept of filtering over a map/dict’s entries? Even in Clojure, it’s not like you can use the same predicate for filtering a dict as for filtering a list (the dict filter predicate has to work on a list argument of 2 elements, whereas the list filter predicate works on an argument of a single element of the list). So, that’s the same as Java, JavaScript, Kotlin, Swift, Rust, etc, etc.
I’ve used Clojure before. It’s probably the only non-static-typed language that I’ve actually enjoyed working with, and it’s clearly very well designed. But, the cynic in me feels like the Clojure community really wants to force “data-driven development” to be a thing and for that thing to be a special, unique, aspect of Clojure. I feel like this idea of Clojure having a standard library that works primarily over dicts and lists as being special is just a truism. It all does work pretty well in practice, but, IMO, there’s no such thing as “data-driven development”. It’s just functional programming, and I don’t feel like the article showcases anything except that developing in different languages is… different. (It’s still a well-written article and I enjoyed walking through the examples)
You really can’t include JavaScript on your list. There is no filter method on Map, and even the name of Map.size differs from Array.length for completely no reason.
In all languages, there is a very real tension between handling map values or map key-value pairs - this is a good point.
Rust’s iterators are a nice “narrow waist” for these kinds of transformations though! Perhaps JavaScript’s upcoming iterator “helper” methods will improve the situation somewhat?
Rust’s Iterator trait is the biggest counter-example I was thinking of to the Clojurist rhetoric about map() and filter() and the implication that only Clojure or other dynamically typed languages can pull off such wonderfully flexible and extensible APIs. (I’m only slightly sorry for the sarcasm)
The truth is that there is nothing special about Clojure’s map() and filter() functions. If a data type is to work with them, then either the function definitions need to change to accommodate it, or the data type has to be written with the intention of working with map() and filter(). That’s literally the same thing in every programming language. Just because the interface required by map() and filter() is implicit doesn’t mean it doesn’t exist. In Rust, you’d have to make sure your data type impls the Iterator (or IntoIterator, etc) trait, just like in Clojure you’d have to implicitly make something iterable in whatever sense map() and filter() require.
This is again why I don’t buy that “data-driven development” is a new concept that needs a name. It’s just functional programming with a dynamically typed language.
There is something cool about working in Clojure, but you can’t really isolate it to a subset of specific features/choices. It’s one of those “the whole is greater than the sum of its parts” things. Instead of trying to coin cute terms, we can replace “data-driven development” with “programming in Clojure”.
It drives me bonkers how array/set/dictionary methods unnecessary differ in so many languages. Javascript’s length vs size is a really obvious example.
Filter operates just as well over maps as it does lists and so on and so forth.
I have a somewhat well-known package in Julia to make this stuff “just work” the same for dicts and sets as it does for arrays, somewhat inspired by Rich Hickey’s talks about clojure. (And another for dataframes with the same interface). To be fair to all the language and stdlib implementers out there, it takes a surprising amount of thought and work to make this work.
No kidding! It reminds me of a meme from a clip of the Simpsons cartoon where one of the characters is shown to be living in a rundown apartment with no furniture or food and he says “Don’t tell anyone how I live.” I guess that’s gotta be me based on my desk space…
Your use of Firefox must follow Mozilla’s Acceptable Use Policy, and you agree that you will not use Firefox to infringe anyone’s rights or violate any applicable laws or regulations.
The acceptable use policy, bullet points replaced with a numbered list for discussion purposes
You may not use any of Mozilla’s services to:
Do anything illegal or otherwise violate applicable law,
Threaten, harass, or violate the privacy rights of others; send unsolicited communications; or intercept, monitor, or modify communications not intended for you,
Harm users such as by using viruses, spyware or malware, worms, trojan horses, time bombs or any other such malicious codes or instructions,
Deceive, mislead, defraud, phish, or commit or attempt to commit identity theft,
Engage in or promote illegal gambling,
Degrade, intimidate, incite violence against, or encourage prejudicial action against someone or a group based on age, gender, race, ethnicity, national origin, religion, sexual orientation, disability, geographic location or other protected category,
Exploit or harm children,
Sell, purchase, or advertise illegal or controlled products or services,
Upload, download, transmit, display, or grant access to content that includes graphic depictions of sexuality or violence,
Collect or harvest personally identifiable information without permission. This includes, but is not limited to, account names and email addresses,
Engage in any activity that interferes with or disrupts Mozilla’s services or products (or the servers and networks which are connected to Mozilla’s services),
Violate the copyright, trademark, patent, or other intellectual property rights of others,
Violate any person’s rights of privacy or publicity,
You may not use any Mozilla service in a way that violates this Acceptable Use Policy or any other binding terms, including any license or terms of service, that apply to the particular service. You also may not sell, resell, or duplicate any Mozilla product or service without written permission from Mozilla.
These are only examples. You should not consider this a complete list, and we may update the list from time to time. Mozilla reserves the right to remove any content or suspend any users that it deems in violation of these conditions.
Please also be aware of Mozilla’s Community Participation Guidelines, which address participation in Mozilla communities.
Since the only thing the acceptable use policy does is restrict services, and it explicitly applies to your use of Firefox, I conclude that Firefox must be contained in the word “service”. I also note that the terms include
Some features in Firefox require you to opt in to them specifically. In order to use them, you will need to agree to the specific Terms and Privacy Notice for each service you use.
Which would tend to reject any interpretation of the above callout of the acceptable use policy which has it only apply to certain services you use within it.
That is messed up. I feel strongly about ‘freedom zero’ of free software – that is The freedom to run the program as you wish, for any purpose. It seems we’re backsliding on the very fundamentals.
In that same vein, I’m absolutely baffled and exasperated by software licenses feeling the need to try and make me agree to not use their software to break the law.
First of all, I’m with you: software freedom zero is that I can use the program however I wish. Fuck off with telling me how I may or may not use the program.
Second: Breaking the law is already… against the law. Who thinks that someone would fire up a piece of software with the intent of breaking the law, but then decide against it because of a license agreement? I’m willing to break the law and have actual consequences, but I’m not willing to break a license agreement where the worst case is that the vendor might somehow disable access to their stupid program? I’m so tired of living in a world that is so unserious.
Third: I know this is a “cover your ass” thing, but that’s not good enough for me. Here in the U.S., people shoot and kill each other, yet gun manufacturers aren’t on the hook for the crimes. And I’m pretty sure you don’t have to sign a license agreement to purchase a gun or ammunition wherein you promise not to break the law with it. Alcohol companies aren’t liable for someone getting drunk and killing someone with their car. Neither is the car manufacturer. The idea that a fucking web browser feels the need to “protect itself” from being liable when a user breaks the law is a joke.
Separating out discussion of the exact terms into a self reply since it feels like a comment that should be voted on separately.
This is a shockingly restrictive policy. For some examples:
Bullet point 8 prohibits using firefox to sell, purchase, or advertise controlled products or services. You cannot use firefox to legally acquire life saving medication. Entirely law abiding drug manufacturers cannot use firefox to sell or advertise their products.
Bullet point 9 forbids graphic depictions of violence in all manners - uploading, downloading, transmiting, displaying, or granting access. You can’t use firefox for making, publishing, or even just consuming journalism on police brutality, war, etc.
Bullet point’s 9 restriction on graphic violence and sexuality also forbids using firefox to watch most PG or higher movies, browse many social media sites, etc.
Bullet point 2 forbids sending unsolicited communications, you can’t do cold outreach to people using firefox.
Bullet point 2 forbids intercepting or monitoring communications not intended for you, you can’t do security research using firefox, something as simple as investigating how your bed is spying on you would require intercepting communications not intended for you.
Bullet point 4 prohibits deception. You can’t play many common games using firefox.
Bullet point 10 is a super-charged version of the GDPR. If you collect personal information without consent, even if you have a legitimate use for it that would satisfy every privacy law in the world, you may not use firefox to do that.
Every bullet point here except 1, 5, 12, and 13 is basically Mozilla trying to step in and define their own version of what is acceptable to say, do, and promote that is highly restrictive, offensive to the principles of free speech, and contains none of the nuance that even the most authoritarian dictatorships would recognize is necessary in regulating conduct.
I take no significant issue with bullet point 1 because it seems unlikely that breaking this contract is in any way worse than breaking the actual law, and bullet points 5, 12, and 13 because they appear to be strict subsets of what is already forbidden under number 1.
I take no significant issue with bullet point 1 because it seems unlikely that breaking this contract is in any way worse than breaking the actual law, and bullet points 5, 12, and 13 because they appear to be strict subsets of what is already forbidden under number 1.
It does contain some jurisdiction smuggling, even if it is unlikely to be a practical issue: if «applicable law» is on the books, but is either in the process of being inevitably struck down in court, or is explicitly condemned by your current place of residence, are you still violating the contract?
Update: They’ve removed the portion of the terms that incorporated this acceptable use policy :)
Though it’s frustrating to see them pretend this is because people misunderstood the terms and not admit it’s because they messed up the drafting of them.
I remember John Carmack describing in one of his Doom 3 talks how he was shocked to discover that he made a mistake in the game loop that caused one needless frame of input latency. To his great relief, he discovered it just in time to fix it before the game shipped. He cares about every single millisecond. Meanwhile, the display server and compositing window manager introduce latency left and right. It’s painful to see how the computing world is devolving in many areas, particularly in usability and performance.
He cares about every single millisecond. Meanwhile, the display server and compositing window manager introduce latency left and right.
I will say the unpopular-but-true thing here: Carmack probably was wrong to do that, and you would be just as wrong to adopt that philosophy today. The bookkeeping counting-bytes-and-cycles side of programming is, in the truest Brooksian sense, accidental complexity which we ought to try to vanquish in order to better attack the essential complexity of the problems we work on.
There are still, occasionally, times and places when being a Scrooge, sitting in his counting-house and begrudging every last ha’penny of expenditure, is forced on a programmer, but they are not as common as is commonly thought. Even in game programming – always brought up as the last bastion of Performance-Carers who Care About Performance™ – the overwhelming majority very obviously don’t actually care about performance the way Carmack or Muratori do, and don’t have to care and haven’t had to for years. “Yeah, but will it run Crysis?” reached meme status nearly 20 years ago!
The point of advances in hardware has not been to cause us to become ever more Scrooge-like, but to free us from having to be Scrooges in the first place. Much as Scrooge himself became a kindly and generous man after the visitation of the spirits, we too can become kinder and have more generous performance budgets after being visited by even moderately modern hardware,
(and the examples of old software so often held up as paragons of Caring About Performance are basically just survivorship bias anyway – the average piece of software always had average performance for and in its era, and we forget how many mediocre stuff was out there while holding up only one or two extreme outliers which were in no way representative of programming practice at the time of their creation)
There is certainly a version of performance optimization where the juice is not worth the squeeze, but is there any indication that Carmack’s approach fell into that category? The given example of “a mistake in the game loop that caused one needless frame of input latency” seems like a bug that definitely should have been fixed.
I’m having a hard time following your reasons for saying Carmack was “wrong” to care so much about performance. Is there some way in which the world would be better if he didn’t? Are you saying he should have cared about something else more?
There are different kinds of complexity. Everything in engineering is about compromises. If you decide to trade some latency for some other benefit, that’s fine. If you introduce latency because you weren’t modelling it in your trade-off space, that’s quite another.
the overwhelming majority very obviously don’t actually care about performance the way Carmack or Muratori do, and don’t have to care and haven’t had to for years. “Yeah, but will it run Crysis?” reached meme status nearly 20 years ago!
the amount of people complaining about game performance in literally any game forum, steam reviews / comments / whatnot obviously shows that wrong. Businesses don’t care about performance but actual humans being do care ; the problem is the constantly increasing disconnect between business and people.
Minecraft – the best-selling video game of all time – is known for both its horrid performance and for being almost universally beloved by players.
The idea that “business” is somehow forcing this onto people (especially when Minecraft started out and initially exploded in popularity as an indie game with even worse performance than it has today) is just not supported by empirical reality, sorry.
But the success is despite the game’s terrible performance, not thanks to it. Or do you think if you asked people if they would prefer minecraft to be faster they would say no ?
If it was not a problem then a mod that does a marginal performance improvement certainly would not have 10M downloads: https://modrinth.com/mod/moreculling . So people definitely do care ; they just don’t have a choice because if you want to play “minecraft” with your friends this is your only option. Just like for instance Slack, Gitlab or Jira are absolutely terrible but you don’t have a choice to use it because that’s where your coworkers are.
I don’t know of any game that succeeded because of their great performance, but I know of plenty that have succeeded despite their horrible performance. While performance can improve player satisfaction, for games, it’s a secondary measure of success, and it’s foolish to focus on it without having the rest of the game being good to play. It’s the case for most other software as well - most of the time, it’s “do the job well, in a convenient to use way, and preferably fast”. There’s fairly few problems where the main factor for software solving it is their speed first.
Bad performance can kill a decent game. Good performance cannot bring success to an otherwise mediocre game. If it worked that way, my simple games that run at ~1000FPS would have taken over the world already.
Or do you think if you asked people if they would prefer minecraft to be faster they would say no ?
Even if a game was written by an entire army of Carmacks and Muratoris squeezing every last bit of performance they could get, people would almost certainly answer “yes” to “would you prefer it to be faster”. It’s a meaningless question, because nobody says no to it even when the performance is already very good.
And the fact that Minecraft succeeded as an indie game based on people loving its gameplay even though it had terrible performance really and truly does put the lie to the notion that game dev is somehow a unique performance-carer industry or that people who play games are somehow super uniquely sensitive to performance. Gamers routinely accept things that are way worse than the sins of your least favorite Electron app or React SPA.
I think a more generous interpretation of the hypothetical would be to phrase the question as: “Do you think the performance of Minecraft is a problem?”
In that scenario, I would imagine that even people who love the game would likely say yes. At the same time, if you asked that question about some Carmack-ified game, you might get mostly “no” responses.
Can you clarify the claim that you are making, and why the chosen example has any bearing on it? Obviously gaming is different from other industries in some ways and the same in other ways.
I think the Scrooge analogy only works in some cases. Scrooge was free to become more generous because he was dealing with his own money. In the same way, when writing programs that run on our own servers, we should feel free to trade efficiency for other things if we wish. But when writing programs that run on our users’ machines, the resources, whether RAM or battery life, aren’t ours to take, so we should be as sparing with them as possible while still doing what we need to do.
Unfortunately, that last phrase, “while still doing what we need to do”, is doing a lot of work there. I have myself shipped a desktop app that uses Electron, because there was a need to get it out quickly, both to make money for my (small, bootstrapped) company and to solve a problem which no other product has solved. But I’ve still put in some small efforts here and there to make the app frugal for an Electron app, while not nearly as frugal as it would be if it were fully native.
I used to be passionate about this too, but I really think villianizing accidental complexity is a false idol. Accidental complexity is the domain of the programmer. We will always have to translate some idealized functionality into a physically executable system. And that system should be fast. And that will always mean reorganizing the data structures and algorithms to be more performant.
My point of view today is that implementation details should be completely embraced, and we should build software that takes advantage of its environment to the fullest. The best way to do this while also retaining the essential complexity of the domain is by completely separating specification from implementation. I believe we should be writing executable specifications and using them in model-based tests on the real implementation. The specifications disregard implementation details, making them much smaller and more comprehensible.
I have working examples of doing this if this sounds interesting, or even farfetched.
I agree with this view. I used to be enamored by the ideas of Domain Driven Design (referring to the code implementation aspects here and not the human aspects) and Clean/Hexagonal Architecture and whatever other similar design philosophies where the shape of your actual code is supposed to mirror the shape of the domain concepts.
One of the easiest ways to break that spell is to try to work on a system with a SQL database where there are a lot of tables with a lot of relations, where ACID matters (e.g., you actually understand and leverage your transaction isolation settings), and where performance matters (e.g., many records, can’t just SELECT * from every JOINed table, etc).
I don’t know where I first heard the term, but I really like to refer to “mechanical sympathy”. Don’t write code that exactly mirrors your business logic; your job as a programmer is to translate the business logic into machine instructions, not to translate business logic into business logic. So, write instructions that will run well on the machine.
Everything is a tradeoff. For example, in C++, when you create a vector and grow it, it is automatically zeroed. You could improve performance by using a plain array that you allocate yourself. I usually forgo this optimization because it costs time and often makes the code more unpleasant to work with. I also don’t go and optimize the assembly by hand, unless there is no other way to achieve what I want. With that being said, performance is a killer feature and lack of performance can kill a product. We absolutely need developers who are more educated in performance matters. Performance problems don’t just cripple our own industry, they cripple the whole world which relies on software. I think the mindset you described here is defeatist and, if it proliferates, will lead to worse software.
You could improve performance by using a plain array that you allocate yourself.
This one isn’t actually clear cut. Most modern CPUs do store allocate in L1. If you write an entire L1 line in the window of the store buffer, it will materialise the line in L1 without fetching from memory or a remote cache (just sending out some broadcast invalidates if the line is in someone else’s cache). If you zero, this will, definitely happen. If you don’t and initialise piecemeal, you may hit the same optimisation, but you may end up pulling in data from memory and then overwriting it.
If the array is big and you do this, you may find that it’s triggering some page faults eagerly to allocate the underlying storage. If you were going to use only a small amount of the total space, this will increase memory usage and hurt your cache. If you use all of it, then the kernel may see that you’ve rapidly faulted on two adjacent pages and handle a bit more in the page faults eagerly handler. This pre-faulting may also move page faults off some later path and reduce jitter.
Ah, you must be one of those “Performance-Carers who Care About Performance™” ;)
Both approaches will be faster in some settings.
This is so often the case, and it always worries me that attitudes like the GP lead to people not even knowing about how to properly benchmark and performance analyse anymore. Not too long ago I showed somebody who was an L4 SWE-SRE at Google a flamegraph - and he had never seen one before!
Ah, you must be one of those “Performance-Carers who Care About Performance™” ;)
Sometimes, and that’s the important bit. Performance is one of the things that I can optimise for, sometimes it’s not the right thing. I recently wrote a document processing framework for my next book. It runs all of its passes in Lua. It simplifies memory management by doing a load of copies of std::string. For a 200+ page book, well under one second of execution time is spent in all of that code, the vast majority is spent in libclang parsing all of the C++ examples and building semantic markup from them. The code is optimised for me to be able to easily add lowerings from new kinds of semantic markup to semantic HTML or typeset PDF, not for performance.
Similarly, a lot of what I work on now is an embedded platform. Microcontrollers are insanely fast relative to memory sizes these days. The computers I learned to program on had a bit less memory, but CPUs that were two orders of magnitude slower. So the main thing I care about is code and data size. If an O(n) algorithm is smaller than an O(log(n)) one, I may still prefer it because I know n is probably 1, and never more than 8 in a lot of cases.
But when I do want to optimise for performance, I want to understand why things are slow and how to fix it. I learned this lesson as a PhD student, where my PhD supervisor gave me some code that avoided passing things in parameters down deep function calls and stored them in globals instead. On the old machine he’d written it for, that was a speedup. Parameters were all passed on the stack and globals were fast to access (no PIC, load a global was just load from a hard-coded address). On the newer machines, it meant things had to go via a slower sequence for PC-relative loads and the accesses to globals impeded SSA construction and so inhibited a load of optimisation. Passing the state down as parameters kept it in registers and enabled local reasoning in the compiler. Undoing his optimisation gave me a 20% speedup. Introducing his optimisation gave him a greater speedup on the hardware that he originally used.
This is so often the case, and it always worries me that attitudes like the GP lead to people not even knowing about how to properly benchmark and performance analyse anymore.
I know how to and I teach it to people I work with. Just recently at work I rebuilt a major service, cut the DB queries it was doing by a factor of about 4 in the process, and it went from multi-second to single-digit-millisecond p95 response times.
But I also don’t pull constant all-nighters worrying that there might be some tiny bit of performance I left on the table, or switching from “slow” to “faster” programming languages, or really any of the stuff people always allege I ought to be doing if I really “care about performance”. I approach a project with a reasonable baseline performance budget, and if I’m within that then I leave it alone and move on to the next thing. I’m not going to wake up in a cold sweat wondering if maybe I could have shaved another picosecond somewhere.
And the fact that you can’t really respond to or engage with criticism of hyper-obsession with performance (or, you can but only through sneering strawmen) isn’t really helpful, y’know?
And the fact that you can’t really respond to or engage with criticism of hyper-obsession with performance (or, you can but only through sneering strawmen) isn’t really helpful, y’know?
How were we supposed to know that you were criticizing “hyper-obsession” that leads to all-nighters, worry, and loss of sleep over shaving off picoseconds? From your other post it sounded like you were criticizing Carmack’s approach, and I haven’t seen any indication that it corresponds to the “hyper-obsession” you describe.
I did a consulting gig a few years ago where just switching from zeroing with std::vector to pre-zeroed with calloc was a double-digit % improvement on Linux.
I think answer is somewhere in the middle: should game programmers in general care? Maybe too broad of a statement. Does ID Software, producers of top-of-the-class, extremely fast shooters benefit from someone who cares so deeply to make sure their games are super snappy? Probably yes.
You think thats bad, consider the advent of “web-apps” for everything.
On anything other than an M-series Apple computer they feel sluggish, even with absurd computer specifications. The largest improvement I felt going from a i9-9900K to an M1 was that Slack suddenly felt like a native app, going back to my old PC felt like going back to the 90’s.
There is a wide spectrum of performance in Electron apps. Although it’s mostly VS Code versus everyone else. VS Code is not particularly snappy, but it’s decent. Discord also feels faster than other messengers. The rest of highly interactive webapps are used are unbearably sluggish.
So I think these measured 6ms are irrelevant. I’m on Wayland Gnome and everything feels snappy except highly interactive webapps. Even my 10-year-old laptop felt great, but I retired it because some webapps were too painful (while compiling Rust felt… OK? Also browsing non-JS content sites was great).
Heck, my favorite comparison is to run Q2 on WASM. How can that feel so much snappier than a chat application like Slack?
I got so accustomed to the latency, when I use something with nearly zero latency (e.g. an 80’s computer with CRT), I get the surreal impression that the character appeared before I pressed the button.
I had the same feeling recently with a commadore64.
It really was striking how a computer with less power than the microcontroller in my keyboard could feel so fast, but obviously when you actually give it an instruction to think about, the limitations of the computer are clear.
As a user on smaller platforms without native apps, I will gladly take a web app or PWA over no access.
In the ’90s almost everything was running Microsoft Windows with on x86 for personal computers with almost everyone running at the 5 different screen resolunions so it was more reasonable to make a singular app for a singular CPU architecture & call it a day. Also security was an afterthought. To support all of these newer platforms, architectures, device types, & have the code in a sandbox, going the HTML + CSS + JavaScript route is a tradeoff many are willing to take for portability since browsers are ubiquitous. The weird thing is that a web app doesn’t have to be slow, & not every application has the same demands to warrant a native release.
Having been around the BSD, and the Linux block 20+ years ago, I share the sentiment. Quirky and/or slow apps are annoying, but still more efficient than no apps.
Besides, as far as UIs go, “native” is just… a moderately useful description at this point. macOS is the only one that’s sort of there, but that wasn’t always the case in all this time, either (remember when it shipped with two toolkits and three themes?). Windows has like three generations of UI toolkits, and one of the two on which the *nix world has mostly converged is frequently used along with things like Kirigami, making it native in the sense that it all eventually goes through some low-level Qt drawing code and color schemes kind of work, but that’s about it.
Don’t get me wrong, I definitely prefer a unified “native” experience; even several native options were tolerable, like back when you could tell a Windows 3.x-era application from other Windows 98 applications because the Open file… dialog looked different and whatnot, but keybindings were generally the same, widgets mostly worked the same etc.
But that’s a lost cause, this is not how applications are developed anymore – both because developers have lost interest in it and because most platform developers (in the wide sense – e.g. Microsoft) have lost interest in it. A rich, native framework is one of the most complex types of software to maintain, with some of the highest validation and maintenance costs. Building one, only to find almost everyone avoids it due to portability or vendor lock-in concerns unless they literally don’t have a choice, and that even then they try to use as little of it as humanly possible, is not a very good use of already scant resources in an age where most of the profit is in mobile and services, not desktop.
You can focus on the bad and point out that the vast majority of Electron applications out there are slow, inconsistent, and their UIs suck. Which is true, but you can also focus on the good and point out that the corpus of Electron applications we have now is a lot wider and more capable than their Xaw/Motif/Wx/Xforms/GTK/Qt/a million others – such consistency, much wow! – equivalents from 25 years ago, whose UIs also sucked.
Perhaps one day in the future we will all live in a post-scarcity utopia where effectively-unlimited compute power and bandwidth are free of charge to all.
Today, we do not live in that world. Today, compute power and bandwidth cost money. So, either you have an internet where every person’s presence must be paid for by that person out of their own pocket and those who can’t afford that are barred from it, or you have an internet where some people can have a presence without paying. Thus far the only we we’ve figured out of actually pulling off the latter at sufficient scale to let the world participate is via advertising; there are not enough donors with deep enough pockets to do it via charity.
Thus far the only we we’ve figured out of actually pulling off the latter at sufficient scale to let the world participate is via advertising; there are not enough donors with deep enough pockets to do it via charity.
Are you sure that’s the only way? Can you genuinely think of no routes to an alternative?
Fediverse is for hobbyists, it does not address the monetization issue.
From the perspective of the content provider / publisher: a publishing / tooling web business (even if small) has as possible income sources mainly the options provided above (paid accounts and ads). Nobody figured out another better way to build a business on the web yet, which I think is @ubernostrum’s point.
Even just a non-business personal web presence is a problem. Either you pay for your web presence or someone else pays for it, and the only successful way we’ve found to do “someone else pays for it” is to have the someone else be an advertiser who gets ads on the page in exchange for their money. And even then it hasn’t always been enough – Geocities, which so many people nostalgically yearn for, had ads and still struggled financially.
Remains to be seen if it can actually hold up. The cost in both money and time to run instances is non-trivial, and it’s a cliché to point out instances whose admins burn out/flame out.
Also, yeah, I will complain about the “sufficient scale”. Big, huge, gigantic instances tend to range from ten thousand or so up to perhaps a couple hundred thousand active accounts. Even if you have a bunch of instances in that size group, you’re looking at roughly the population of a good-sized city. We live in a world of billions of people.
Obviously a larger Fediverse user base would come with a larger donor base. What makes you so sure the increased revenue would not keep up with with the increased costs?
I would still be curious to understand what you meant in your reply to the linked comment, but perhaps you changed your position. You haven’t completely abandoned it though so there should still be plenty to discuss!
The demand to “indicate your preference” rings hollow when the Internet is already both things, as @aspensmonster pointed out. There is no realistic proposal to avoid either one; it will always be a spectrum.
I would like to ask once again if you really think the web would be poorer if websites were less like Facebook and Twitter and more like Wikipedia and Lobsters?
My home internet connection is right now $83 / month. (It used to be $15 :( but the cable company keeps jacking it up… even though it delivers very little more practical benefit to me than it used to…)
That internet connection would be absolutely worthless if there was nobody on the other side to connect to. Maybe some fraction of that $83 / month could go to the people who make the internet valuable as well as the people who make the internet possible? I’m not saying the cable company is worthless, just that maybe we could find some way to spread the money without advertising middle men. (The money to pay for ads also ultimately comes from the users who buy the products…)
Does this make poor people second class or forbidden? Well, if I didn’t pay my cable bill, they wouldn’t let me connect anymore sooooooooo already reality? And a lot of the best on line content is behind additional paywalls now too - even with the advertising. So eh, maybe this isn’t an ideal situation but it at least doesn’t seem much worse than it is now.
I won’t fall for the false dichotomy. The web was not as hostile in the 90s and 00s. Yes, there were flashing ad banners in places, but it was, overall, much less invasive (spying/tracking).
Communities grew out of passions and common interests. Not every single fucking thing needed to be a profitable endeavor. (Look at Reddit in the early days)
I feel like we’ve become more and more like the Ferengi species from Star Trek.
The Ferengi were always a metaphor for western capitalist societies in general, and the US in particular. Check out the Lower Decks episode where Boimler gets trapped in an hotel on Ferenginar for a weekend because he has no resistance to the addictive nature of Ferengi daytime TV.
The problem is that such articles have to claim superiority by denigrating another paradigm with examples and code that shows poor practice.
People like FP because it is well suited to the transformation of data, and for many programmers that is all there is. Try modeling a complex payroll system into code using such a transformation of data approach. Enterprises tried that for over a decade during the “structured analysis” era, but as “engineers” we all know that don’t we?
I have no idea what you are alluding to but I am honestly very intrigued. As someone who likes functional programming and complex enterprise systems I would definitely like to hear your thoughts/experiences regarding why FP might not be a good match for enterprise software development. (I do know about structured design, if you’re referring to the Yourdon/Constantine sort of thing. But not sure how it’s connected here.)
As for the reasons why people like FP, I agree transformation of data is one factor, but I would say it’s “pick 3-5 out of:”
referential transparency
immutability
higher-order functions
pattern matching
“wholemeal programming”
comprehensive type checking
type-level programming
freedom through constraints
Most of those can be found in non-FP languages, but when you have a critical number of them come together in a cohesive way, that certain magic starts to happen. There is no single definition that fits all languages that are commonly categorized under the FP label. If I were to pick one, I would say “high-order functions as the default and most convenient way of doing everything”.
Most of those can be found in non-FP languages, but when you have a critical number of them come together in a cohesive way, that certain magic starts to happen.
Indeed. You can do FP in most languages, and you can do OOP in most languages (even C!), but if the language is designed and optimized for certain styles/patterns of programming, then you’re usually giving up both convenience/ergonomics and performance by going against the grain. Case in point: JavaScript devs trying to lean into FP patterns and making copious, unnecessary, copies of local arrays and objects just for the “immutable-only cred” or writing a “curried” functions when that means actually creating tons of temporary Function objects that need to be garbage collected at runtime (as opposed to languages like OCaml that are able to optimize these things so well that curried functions mostly don’t have any runtime overhead compared to uncurried functions).
But, I will say this: I really don’t love FP (as in the “use mostly pure functions to just transform data” definition) for IO-heavy applications (really, almost anything with a DBMS underneath). Doing “proper” FP style is often the antithesis of “mechanical sympathy”, meaning we have to choose between writing elegant data transformation code and writing code with reasonable performance characteristics.
For example, it’s extremely common when working with a SQL database to open a transaction, read some data, make a decision based on that data, and then conditionally read or write more data. To implement this in conventional FP style, you have basically two options:
Break up this unit of logic into multiple pure functions and have your top-level handler piece them together like IO -> pure function 1 -> IO -> pure function 2 -> IO. That’s fine if your pure functions actually make sense to be separate named operations. Otherwise, you’re breaking something that is semantically a single business operation into multiple pieces that don’t actually mean anything in your domain.
Read ALL of the data you could possibly need upfront and then feed it into the pure function before writing the final result. This is obviously a bad idea in any system that is supposed to be able to scale. Doing extra database queries when you might not even use the results should send you straight to jail.
Because there just isn’t much point to doing that for a web server application. This isn’t a typical, interactive, user application where you’d want to present an error message to whomever issued the command. This is more likely to be something that’s just supposed to run automatically at boot or something, so just blowing up and showing a stack trace with exactly whatever line number failed pretty much is the friendly error message for the “user” (the guy managing the server).
IMO, of course. But, this is exactly the thought process I use with my Rust web service that is running in a docker container. Failing to start is failing to start and I’m not going to waste time to hide information from myself by writing a more “product-y” error message or whatever.
Because there just isn’t much point to doing that for a web server application.
Lots of great reasons for reporting errors like that, especially if you have a bunch of copies of that application running in a bunch of different places. Either in-house logging or something like Sentry are fantastic error sinks. Basically, I’d want to know when the application is not able to bind to a port and something like systemd is trying to restart it, causing a boot loop. That is a problem I’d want to know about.
Fair enough. And those specifics will depend on your infrastructure and deployment process. If your code fails to deploy for some reason even before your app starts running, you’re not going to get your Sentry errors, anyway (or if it just hard crashes, AFAIK). What if your Sentry logger doesn’t initialize properly, or the network is misconfigured, etc? So, you still need something that’s going to report bad health that’s outside of your application’s code, IMO. If that’s the case, then I still don’t see much value in writing and logging a pretty error message when your app can’t even finish starting up. Let it crash or bootloop or whatever, and the external health checks should alert us to something going wrong just as well as a remote logging service.
I’ve never used any BSDs in earnest (not counting macOS because I don’t want to count it…). I’ve played with a few in virtual machines and read documentation and forums for several of them back in my distro-hopping days.
But, any progress at all in running free software operating systems on our machines gets a huge cheer from me. I fear that the future is only going to get more and more locked down to the point that we won’t even be allowed to install “unapproved” software on our hardware, or that we won’t be able to disable various tracking and spying mechanisms on our machines. But, I continue to hope that was can avoid that future, and I’ll continue to fight it every step of the way. The only computer in my house that runs a non-Linux OS is my son’s school computer that, unfortunately, requires Windows to run certain things. But, I have to work constantly to disabled all of the analytics bullshit and there’s no linked Microsoft account on it despite them making it harder and harder to use it that way. How about you stop trying to spy on children you evil pieces of… well, never mind… /rant
I’ve always been intrigued by the hexagonal architecture, probably because it’s similar to the „functional core, imperative shell“ idea. But by now I’ve become wary of the abstractions this post and others use to implement it. All the complexity that comes with implementing e.g. a repository trait doesn’t seem worth it to me, especially when a database imposes so many implicit assumptions on the whole codebase that switching it would require a lot more than just adding a trait implementation.
I wonder if there’s a hexagonal „layout“ that’s more focused on just modules and structural conventions to get some of the benefits with less overhead.
My team implemented a hexagonal architecture for a certain project, but development was very slow. I argued basically that for ordinary business applications, we’re not switching from Postgres + SQL Alchemy to anything else anytime soon, and that if the product owners want faster development this is the low-hanging fruit. The proliferation of repositories and repository hiding in general caused me to conclude that, at least for the persistence element, it isn’t worth the potential benefit.
Architecture is good, IMO, if it forces you to think hard about your priorities, but I think hexagonal architecture in particular looks like it solves all your problems, but you have to bear in mind that like every other architecture, it comes with tradeoffs, and the con with hexagonal is that it tends to make the entire application very abstract and slow to build.
I can’t say I disagree with that, it’s not the first time I’ve read an essay on hexagonal architecture and come away with the conclusion that it’s something you refactor your application into if and when you reach the need for the flexibility and substitutability.
To that, this specific essay has the significant ick factor of calling a perfectly serviceable and maintainable application “very bad application”, then proceeding to convert 260 lines of rust in three simple files (including a simple and completely linear main setup) into 750 lines spread over nearly 20 files.
And for all its harping on testability, it has one test. Which needs 80 lines to test that a two line handler function calls into the (mocked) handler service and forward its result (which is a success because that’s the only thing it tests). It takes about 3 lines to create an in-memory sqlite, wrap that in the original application’s AppState, and start calling the same handler and testing its output.
While it’s of course silly to build a 260 line application like this, I have experienced at least some of the concepts translating well into anything bigger.
It’s nice to look at a module/folder of code and know “hey this is for saving/getting things to/from the DB” and this other code is just about wrapping a remote API. That other code is just for taking things from/to HTTP or whatever and passing it somewhere that doesn’t need to know.
I’m not sure if this is very hexagonal, but the most workable codebases I’ve got to work in where things where you just generated the API layer with something like OpenAPI or GRPC, and the DB interactions with something like https://sqlc.dev/. Then you just focus on the actual logic of the task at hand between those.
Meanwhile, some of the worst codebases I’ve had to work in were places where side effects could happen at any moment in a 5 layer deep call chain with db transactions and remote IO seemingly happening in every helper function.
Maybe the important essence really is more “functional core, imperative shell” once things get big.
One day I should write this up, but a lot of what I do at work currently is web services built in Litestar, using svcs as a backing “service locator” implementation and Litestar’s own SQLAlchemy repository implementation (which they also package for independent use as advanced-alchemy). I do this not because I believe in the need to maintain flexible “hexagonal” or whatever architecture, but because it hits the right balance of architectural patterns and practical payoffs for me.
Basically:
The repository implementation they provide is useful not as an abstraction to introduce layers between my code and the underlying data store, but as something that just implements a bunch of common generic CRUD-ish query methods in a way that’s easy to drop in and use.
The service-locator implementation can nicely serve as a backing registry for Litestar’s and pytest’s dependency injection and provides a clear place to define and override things. It’s not about switching out the persistence layer for something else, it’s about being able to say things like “in testing use this ephemeral local DB container”.
In a Django-specific context I’ve made similar arguments against introducing “service layer” abstractions, largely because A) for web applications, at least, I don’t fully believe in trying to produce isolated pure “unit” tests that can run independently of a database and B) you’re never going to meaningfully swap out how you do persistence without significant rewriting anyway. TO quote myself there:
But what about the data modeling and access component (which in Django’s case is its ORM)? In theory, hiding it behind a service layer means you can change it out for something else without rewriting other code. But how often does that really happen in practice? My experience — and I’ve been working in this industry for over 15 years at places both big and small, and been through some pretty major platform migrations — is that it happens almost never. Swapping something as major as how you model and persist/retrieve your core data is not something you do lightly, and typically only happens at a time when you’re making other massive changes — completely switching platform/language/framework, or otherwise doing a ground-up rewrite of the entire system. And, ironically, it’s in the kinds of organizations most likely to insist on “enterprise” architecture patterns that the payoff is least likely ever to be realized, because such organizations are almost fanatically averse to the kind of change this architecture is meant to enable. See, for example, all the places for which we know a decade (or longer) is not even close to enough time to do platform upgrades.
At the developer scale I pretty much agree with you completely, although I haven’t used the tools you mention (thanks for sharing links!). I made a similar argument here, along the lines of, it seems unlikely to me that on the timescale of 10-20 years, SQL Alchemy and PostgreSQL are likely to still be obtaining active support, because they’ve been around for ~20 and ~40 years respectively, and they are the dominant paradigms in their niche. Preemptively building abstraction to support the somewhat far-fetched scenario that they fall out of popularity before this project gets retired seems like a waste of effort. Moreover, sticking entirely to the common SQL subset of what you can do with SQL Alchemy is also a bit of a waste. There isn’t going to be a cost-based motive for switching to something besides Postgres (unless what, they give you money to use it?) and a feature-based motive would give the lie to using that common SQL subset anyway. So it’s a way of throwing money away, basically. Use the tools that expedite development.
At the architectural scale, it isn’t a foregone conclusion. If I were building something to last 100 years or longer, I would start to wonder about things like that. Cost or speed of development may not be the most important thing. It again comes down to the prioritization of quality attributes (performance, reliability, flexibility, etc. etc.) and the determination of architecturally significant requirements. Within your daily life and mine, these scenarios may be far-fetched, but for other people maybe they aren’t and the whole kit & caboodle of hexagonal architecture may be the best solution. Although it seems unlikely to me that hexagonal architecture should be used as often as it gets talked about.
Then again, I kind of flunked out of architecture, so my opinions may not be very valuable.
I dove into one of our hexagonal services yesterday to see if we could replace DynamoDB with Postgres and in theory I’d “just” have to replace the repository implementation but in reality it’s just an endless jumping around between files and the way they implemented half the AWS API was bleeding into the business layer anyway.
I’m unimpressed just as I am with say VIPER on mobile. Maybe instead get good and program the thing in a straight line.
The core idea of hexagonal architecture is that your business logic is not coupled to implementation choices made. But if you’re okay giving up some of the benefits, you don’t need to abstract things.
I like starting with services and using integration tests with devcontainers, for instance, instead of a repositories/wrappers around 3rd party APIs. Services get me enough of a benefit to use them from the start and think more about what my code is doing, rather than gunking them up with HTTP details.
I used to strive for doing things similar to hexagonal architecture and/or domain driven design (DDD) and/or clean architecture.
The weak point of all of these techniques and abstractions is data persistence if the data store is a SQL database. There is a very large space of applications that live in between “TODO List App For My Blog” and “eventual consistency with multiple CQRS data stores, caching layers, background syncing, etc”.
At the “TODO List” end of the spectrum, your domain models pretty much map 1-1 with your database tables, or might have some one-to-many, parent-children, relations. These trivial demo apps almost never need transactions (and certainly never seem to acknowledge or handle database constraints) or row locks, etc.
On the other end, you also don’t typically utilize the SQL database’s transaction and locking functionality, but for different reasons.
But, in the middle range, all of these “Repository” abstractions are really inefficient and/or leaky and/or just plain incapable of handling concepts like database transactions.
Sometimes the way this is handled is that you write your Repository in a way that pretends it has no concept of transactions and then you begin a transaction at some higher level (i.e., in a Service class or an HTTP handler, etc). But that is a (very) leaky abstraction! If the whole point of a Repository was to abstract away the persistence details, then how or why should my higher level code know that I need to start a database transaction?
Others will say that the above approach is improper (especially in DDD). They’ll say that the Repository should handle any transactions and that if you want atomicity between two Repositories, that just means you need to define a new domain entity and write a new Repository for that entity. But, I still don’t ever see anyone writing Repository classes that would allow us to accomplish something like a SELECT ... FOR UPDATE followed by an UPDATE ... and/or DELETE ... inside a transaction. Rather, the best they seem able to do is have each individual method run inside a transaction.
And that’s before even getting to Rust, specifically, which adds even a little more friction to using this kind of pattern. Boxing up a bunch of trait objects to be passed around or held by other trait objects, etc, is pretty ugly and not ergonomic.
I enjoyed this post and think it’s a great walkthrough of one way to factor your code and why/when that’s helpful!
I agree with your comment about the repository trait. I think it’s usually good to avoid making traits which only have a single implementor – having an struct AuthorRepository { sqlite: sqlx::SqlitePool } is just as abstracting from the perspective of someone using the AuthorRepository without needing to make a separate trait AuthorRepository and then a struct Sqlite which implements it. Traits are great but you can still run into limitations (eg around async) relative to just using a struct. (If the trait exist just to support mocking in tests, you can do things like mockall::double
I’ve been using a hexagonal-ish layout for Go for the last few years, I like what it has evolved to. There’s a reason for the “-ish”, I don’t really care about hot-swapping implementations except for a few specifics (vector DB, file storage/S3, email sender) and found a lot of abstractions useless. What I settled on was flexibility and sticking as close to idiomatic Go as possible, same would go for Rust, when in Rome and all that.
Ultimately, the “rules” of architectures (be it 3-layer, hexagonal or whatever’s trending) is that they are best treated as a rough map to navigate growing complexity and every codebase/product/team will have different needs, ideas and tradeoffs. Sometimes it’s fine to just call a repo from a HTTP handler, the rules say that’s bad but you gotta spend time wisely. The rules say everything should be an interface/trait, but if you know you’re never going to suddenly switch database one day (usually very unlikely) it’s not worth the hassle.
cue the pirates of the carribean “they’re more guidelines than actual rules” scene
I wonder if there’s a hexagonal „layout“ that’s more focused on just modules and structural conventions to get some of the benefits with less overhead.
This is what I try to do. My shell has two roles
inbound - transform data into a type/model that I can use internally.
outbound - transform an internal type/model into some data so it can be sent externally.
I do not make unnecessary abstractions or create traits only for testing. If I need to test a route that saves data to postgres, then I have postgres running and wired up to the app. I use random values for all test data so I can run tests in parallel.
If a test depends on some other data, then I set up that data (again with random values) before running the test. A nice byproduct of this pattern that you slowly create an internal SDK that calls your API which forces you to act as the “client” during testing and actually use your API.
I’m not sure about perfect since you fail to mention the whole anyerror (and the other crate whose name I forget here) sadness. If it were perfect, then there wouldn’t be a need for a crate.
I would say though that yes, it’s much better than any of the other programming languages I commonly have to write.
There appears to be anyhow, thiserror, eyre, snafu, error-stack, and a number of others that are no longer the hotness. (anyerror is a Zig keyword, incidentally!)
Those aren’t strictly necessary, though; you can just use Box<dyn Error> in almost all cases where you’d use anyhow::Error. (Sure, it’s not as good, and it’s still a bit of a pain.)
I’ve heard this is fine in applications but bad practice in libraries (i kinda get why). I’m not sure how the other error crates work though, I’ve only written applications and I just use this anyway and feel a bit dirty doing it.
Right. And the anyhow crate that the person above you mentioned is specifically intended/encouraged to be used for applications rather than libraries. The anyhow error type is essentially a not-quite-as-flexible Box<dyn Error>.
Java is still the de facto king of backwards compatibility that actually has the years to show for it. Just based on the title, that ‘title’ would go to Java in my opinion.
I would argue that it is at most as bad as with any other language, but arguably better as the ecosystem is significantly more stable than any other. (What is in the same ballpark in size are JS, and Python, neither is famous for their stability).
But of course third-party libs may decide to break their public APIs at any point, that’s irrelevant to language choice, besides “culture” — and Java’s is definitely not “going fast and breaking stuff”.
Sadly, starting with Java 9, while the core language is backwards compatible, the library situation is a nightmare. And when you’re doing an upgrade, you don’t care if it’s the core language or the library that’s changed, either way, it’s work to upgrade and ensure you don’t have regressions.
Lots of things have changed the package that they live in. In principle, this sounds like it’s not a very difficult change to accomodate, just find and replace, but it wreaks havoc when you have libraries that have to run on multiple JVM versions.
If you’re unlucky enough to be using Spring, it’s even worse. That framework has no concept of backwards compatibility, and is like a virus that spreads throughout your entire codebase.
I can’t share your experience at all. Even large jumps pre-8 to post-11 (though after like 11 they are basically trivial) are reasonably easy, and the few libraries that made use of JVM internals that got increasingly locked down (to prevent further breaking changes in the future) have largely been updated to a post-module version, so very often it’s just bumping the version numbers.
I don’t understand some of what you’re describing.
And when you’re doing an upgrade, you don’t care if it’s the core language or the library that’s changed, either way, it’s work to upgrade and ensure you don’t have regressions.
Are you saying that other languages can somehow prevent the problem of third-party libraries breaking backwards compatibility? Because, if you aren’t saying that, then the core language being stable is going to make the situation objectively better to deal with than if you have to worry about breaking changes in libraries and in the core language…
Lots of things have changed the package that they live in. In principle, this sounds like it’s not a very difficult change to accomodate, just find and replace, but it wreaks havoc when you have libraries that have to run on multiple JVM versions.
Yes, it can be tricky to write code to target multiple versions of Java at the same time, but in my experience, it’s about 100 times less tricky to do this with Java than almost any other language. JavaScript might be the only one that’s even better about running old code on newer runtimes without much problem. Are there others you consider better than Java at this? Specifically for running the SAME code on multiple versions of the language? I remember back in the day when I worked on a lot of C++, it was a nightmare to figure out what features I could use from C++11 to allow the project to build on various versions of GCC that shipped with different Linux distros (Ugh, RedHat’s ancient versions!).
Are you saying that other languages can somehow prevent the problem of third-party libraries breaking backwards compatibility? Because, if you aren’t saying that, then the core language being stable is going to make the situation objectively better to deal with than if you have to worry about breaking changes in libraries and in the core language…
I think that depending on the community, some languages have less BC issues because of libraries than others. Case in point for me was Clojure: both the core language and community have a stance of avoiding breaking compatibility, even if the language itself doesn’t offer any special mechanism for that. Quite the opposite actually, since the language is really dynamic and you can even access private functions without much difficulty.
Are you saying that other languages can somehow prevent the problem of third-party libraries breaking backwards compatibility?
Of course they can’t, and that’s not my point.
My point is that communities develop norms, and those norms include more or less careful treatment of backwards compatibility in libraries. My sense is that Haskell and JavaScript are probably the worst. Java is actually mixed, as there are a lot of libraries that do great work. Some even have a single version that runs on Java 5 through Java 21. But at my day job, we’re using Spring, and the situation is bad.
Though with regard to languages, I will say dynamic class-loading can make the situation worse. I’ve dealt with issues where the order of requests to a web server determined which version of a service provider was loaded. So 9 times out of 10, the code ran on the newer version of Java, but failed 1 time in 10.
Because, if you aren’t saying that, then the core language being stable is going to make the situation objectively better to deal with than if you have to worry about breaking changes in libraries and in the core language…
It sounds like your argument is “having two problems is worse than having one”. But two little problems are better than one big problem.
I greatly appreciate the backwards compatibility of the Java language. But I would be happier with a slightly less backwards compatible core language, if I could trade it for an ecosystem that’s much better at backwards compatibility.
My point is that communities develop norms, and those norms include more or less careful treatment of backwards compatibility in libraries. My sense is that Haskell and JavaScript are probably the worst. Java is actually mixed, as there are a lot of libraries that do great work. Some even have a single version that runs on Java 5 through Java 21. But at my day job, we’re using Spring, and the situation is bad.
Right. I understand and agree about ecosystem “culture”. But, at the end of the day, someone said Java was good with backwards compatibility because it makes it easy to write code that will continue to work “forever”.
I guess maybe your point is that the language prioritizing backwards compatibility is only necessary, but not sufficient, for developers actually getting to experience the benefit of the backwards compatibility. Would you say that’s a decent interpretation/restating? I do agree with that. And if the language itself doesn’t care about backwards compatibility, then it’s impossible for the ecosystem to have stability.
Though with regard to languages, I will say dynamic class-loading can make the situation worse. I’ve dealt with issues where the order of requests to a web server determined which version of a service provider was loaded. So 9 times out of 10, the code ran on the newer version of Java, but failed 1 time in 10.
Definitely true! Luckily, I only remember one time where I had a nightmare of a time figuring out some class or service loading bug.
It sounds like your argument is “having two problems is worse than having one”. But two little problems are better than one big problem.
Yeah, that’s exactly what I was saying. But there’s no reason to think the “one problem” (community doesn’t value backwards compat.) would be bigger or smaller for either case. So, it’s like comparing x to x + 1.
I greatly appreciate the backwards compatibility of the Java language. But I would be happier with a slightly less backwards compatible core language, if I could trade it for an ecosystem that’s much better at backwards compatibility.
I really feel like this is mostly just Spring. I hate Spring, too, so I have been fortunate enough to not have to use it in several years now. But, as for the Java libraries I am using, I honestly couldn’t tell you about their JVM version compatibility over time. But, just judging by API stability, it seems that most “general purpose” or “utility” style libraries stay pretty backward compatible (thinking of the Apache and Guava libs, etc). It’s mostly the frameworky ones (like Spring, Spark, etc) that like to rewrite big chunks of their APIs all the time.
Can you give an example of what you’re referring to?
I’m semi-active on the PHP internals mailing list (i.e. the list where development of php itself is discussed) and BC breaks are a near constant discussion point with any proposed change, so I’m kind of curious what breaking change you’re referring to here?
Well, since I’m not the person you originally asked, I obviously can’t speculate on which breaking changes they’ve actually bumped into. But, I’ve works on several long-lived PHP projects going all the way back to version 5.3 and all the way up to 8.1 or 8.2 (I don’t quite remember which), and I’ve definitely had to fix broken code from minor version bumps. Luckily, we do have those migration lists, so I learned pretty early on to read them carefully and grep through any code base to fix things on the list before even running unit tests.
But, I’m not sure what the point is. The person said that PHP introduces breaking changes in minor version bumps, and that it frustrates them. Maybe they’re misguided for being frustrated by that, but it’s objectively true.
Personally, I’m perfectly fine with breaking bug fixes in minor versions. It’s not like you’re going to accidentally or unknowingly version bump the programming language of your projects. On the other hand, many of these changes are NOT bug fixes, but just general “improvements”, like the range() changes or the “static properties in traits” change in the 8.3 list. I can see how this would be frustrating for someone who wants the bug fix changes, but doesn’t want to pay the extra “tax” of dealing with the non-essential breaking changes.
But, again, I personally don’t care. I treat PHP minor versions as a big deal. PHP 8.2 is not the same language as PHP 8.1, which is fine. But, if you care about backwards compatibility and painless upgrades, Java is pretty much the king.
It’s mostly the ripple effects into libraries and “big” frameworks. For instance, I remember a few years ago a client was on still on PHP 5.3 (I think) and his hosting provider was about to shut down this version, so he had to get his site working on newer versions. This site was built on an in-house CMS of questionable code quality, using CakePHP.
The problem I ran into was that I had to step-by-step upgrade CakePHP through some major versions because the older versions wouldn’t run in the newer PHP due to BC breakage. Eventually, I completely hit a wall when I got to one major version where I discovered in the changelog that they completely rewrote their ORM. Because of the shitty code quality of the CMS (with no tests), I had to admit defeat. The client ended up having to commission the building of a new site.
Java and Go are really great in this regard.
Every other language, including Python (which makes breaking changes from 3.12.3 -> 3.12.4!) is horrible on that front.
I know ragging on JS is cool, and yes there are a few misfeatures, but you can easily avoid more or less all of them. The same isn’t true of many other languages (C and C++ for instance have myriad foot guns that cannot be trivially avoided, python has various bizarre behaviors around basic functionality, perl is basically all weirdities).
All languages have things that were in retrospect not fantastic, JS is no different in that regard, but acting like it is somehow an especially “bad” language is not an accurate representation.
You might be able to avoid the misfeatures as a user, but you can’t as a language implementor.
I worked in the Mozilla JS team in 2009 and it was pretty much a daily thing to hear someone mutter “Who invented this shit?”. ::all look at Brendan::
Brendan is a good guy, but wasn’t given much time and I’m sure could never have anticipated how JS is used today – or the performance of current implementations, which is better than C compiled with GCC or Clang without a ‘-O’ option (so implicitly -O0).
It would have been better if they’d just used Scheme or elisp. At least they didn’t use TCL.
I worked on JSC for a decade or so, so I understand all of this. |with| is not a challenging feature to support, it’s just a developer hazard. It could even be made much faster - but the messaging was so strongly “don’t use with()” that |with| perf simply does not matter. Which is good, because of the myriad foot guns that with has :D
I feel the adoption of scheme or lisp would have fairly aggressively held back adoption - think about just how much devs still push back against prototypes today, to the extent that we added classes to the language, that are literally just sugar on prototypes :-/ Certainly it would have opened potential for MS to not have had to copy JS and have that IE developed language win.
think about just how much devs still push back against prototypes today, to the extent that we added classes to the language, that are literally just sugar on prototypes :-/
I don’t think that’s in any way surprising: JS using prototypal inheritance was always a convenient expedient more than anything else, working with them was always a painful verbose chore compared to languages actually designed for it (in contrast playing with Self was a pleasure, not to mention the realisation that a significant part of the confusion is JavaScript mislabelling things entirely).
The class syntax added what i very much believe had been intended from the start by Bram, to the extent that anything was ever intended. It is also able to work correctly with native object, something which is absolutely painful using prototypes (e.g. creating bespoke error types).
In early 1995, Brendan Eich was recruited to Netscape with the bait of “come and do Scheme in the browser.” But when Eich joined Netscape on April 3, 1995, he found a complex product marketing and programming language situation.
The candidates for a Web page scripting language included research languages such as Scheme; practical Unix-based languages such as Perl, Python, and Tcl; and proprietary languages such as Microsoft’s Visual Basic. Brendan Eich was expecting to implement Scheme in the browser. But in early 1995 Sun Microsystems had started a guerrilla marketing campaign [Byous 1998] for its still unreleased Java language. Sun and Netscape quickly engaged with each other to strike a deal whereby Java would be integrated into Netscape 2.
Rapid strategizing inside Netscape to choose a scripting language severely handicapped Scheme, Perl, Python, Tcl, and Visual Basic as not viable due to business interests and/or time to market considerations. The only approach considered viable by senior managers at Netscape and Sun, notably Marc Andreessen and Sun’s Bill Joy, was to design and implement a “little language” to complement Java.
Marc Andreessen proposed the code-name “Mocha” for the browser scripting language with, according to Eich, the hope that the language would be renamed “JavaScript” in due course. This companion language to Java would have to “look like Java” while remaining easy to use and “object-based” rather than class-based, like Java.
That’s completely orthogonal to your point though, and to @olliej’s point. If you mean “like other languages, JavaScript has lots of misfeatures, but the blast radius is larger because JS is hard to avoid,” then say that.
I know ragging on JS is cool, and yes there are a few misfeatures, but you can easily avoid more or less all of them.
On the contrary, I feel like there has been significant backlash lately against anyone who still criticizes JavaScript or PHP. Say something critical of the languages and you’re in for a lot of accusations about the last time you must have actually used said language or whether you’re just mindlessly parroting criticisms to be “edgy”.
I think with JS, specifically, we just spend so much time with it that we get used to all the nonsense. You say we can easily avoid the pitfalls/misfeatures of JS, but here’s where the fallacy is, IMO: each individual misfeature might be easy to avoid, but when you have 1,000 of them, it becomes very hard to consistently avoid all of them.
Not only that, but us “seniors” are probably taking for granted how long it took us to develop the habits to avoid those misfeatures. Sure, maybe after 5+ years of writing JS, you’ll finally be able to write with the same confidence as a beginner in almost any other language, but that’s not good enough to stop me from criticizing it–whether or not I think I’m being “cool” by doing so.
It’s hard to juggle null and undefined. Especially because a lot of existing JS code uses “truthiness” checks rather than === checks. We have to remember that if foo: { x: number } | null that const x = foo?.x will have the type number | undefined (NOT null!). We also have to remember that writing y == null will check that y is either nullorundefined! WHAT?
We also have to remember that typeof undefined === 'undefined' while typeof null === 'object'.
We have to be careful how we pass callbacks into functions. If you want to call an object method as a callback, you can’t just pass the method; you have to either create a closure that captures the object and calls the method or you have to “bind” it: o.method.bind(o). If you want to pass a function with an optional second argument to something like Array.map, you’ll get incorrect results if you just pass the function directly; instead you have to wrap it in a closure that takes a single argument and calls the function with just the first argument.
There’s just an endless list I could go through of bizarre, unintuitive, and frankly silly stuff in the language. It’s really not trivial to avoid them. It just takes a lot of experience.
Right on. I can only work in the ecosystem using TS on strict mode. Call it a skill issue, I don’t care, I just need tooling that watches my back as much as possible to feel like I can ship things I’m confident in. I’ve shipped some stuff to production in TS and used it for 2 years on and off. There’s a certain lack of solidness to it I still perceive, and it keeps me on guard. Probably comes down to the fact that TS is ultimately just a suggestion and you can do whatever you want. I think part of my brain gets deeply annoyed that the primitives of JS are just…wonky. Hard to trust everything downstream of that.
Lately I’ve been writing Kotlin for web stuff, favoring as little JS as possible, and it’s a joy. Kotlin’s a great balance of expressiveness and predictability, and JetBrains IDEs are top tier.
I recently emailed Richard Stallman and asked about the ethics of this practice. This was his reply:
It is my understanding that as the copyright holders they have the right to do it without any problems. They leverage the AGPLv3 to make it harder for their competitors to use the code to compete against them.
I see what you mean. The original developer can engage in a practice that blocks coopertation.
By contrast, using some other license, such as the ordinary GPL, would permitt ANY user of the program to engage in that practice. In a perverse sense that could seem more fair, but I think it is also more harmful.
yeah, I dunno. the original introduction of the concept of copyleft had all this rhetoric about how it’s a license that gives users rights rather than takes them away and reserves them for the company.
I was pretty big on the ethical source movement for a couple years; it needed to be its own thing because by its nature, it imposes restrictions on how software can be used, which not only violates the founding principles of free software, but wasn’t even okay with the corporate lawyers who decide what things OSI will sign off on calling “open source”.
I eventually decided that, yeah, imposing those restrictions isn’t really part of a viable theory of change, because the bad actors it’s intended to constrain are not going to obey the license. if the intention is to make a clear and firm statement condemning certain things, it’s just as effective to do that in a form that isn’t a legal document. with that said, I don’t see any great harm in licenses which attempt to impose usage restrictions, I just think there are better ways to focus whatever attention programmers want to spend on promoting ethics in software.
I’ll spare you the explanation of where I went after moving on from that, because it’s irrelevant here. at any rate, the AGPL is exactly such a license that attempts to impose usage restrictions. I think I may have new feelings about the AGPL as a result of your comment, @mattheusmoreia, and I very much thank you for it. it’s going to take time for me to coalesce those feelings into coherent thoughts, so I’ll leave it there.
(yes, it was pretty hypocritical of the people on OSI’s legal discussion list to object to usage restrictions only when they’re rhetorically tied to ethics, and honestly, from how the discussion went, they knew it. oh well. there’s no reason to worry about it now.)
Minor nit, the AGPL doesn’t restrict usage but rather modification. I understand it restricts usage of the modification, but that’s not quite the same as usage in general.
certainly it’s hard to see a coherent argument that AGPL passes a proper formulation of the desert island test
(if you’re trapped on a desert island with no communication with the outside world, can you still legally use the software for everything you could do with it outside? you can probably word the test in a legalistic way to allow AGPL to pass, but you shouldn’t)
and it’s in that spirit that I was calling it a usage restriction. but you’re correct, of course.
(if you’re trapped on a desert island with no communication with the outside world, can you still legally use the software for everything you could do with it outside? you can probably word the test in a legalistic way to allow AGPL to pass, but you shouldn’t)
No one can connect to your software over the network, so you can’t possibly violate the AGPL.
No one can connect to your software over the network, so you can’t possibly violate the AGPL.
But the AGPL doesn’t place restrictions on connecting software over the network, it puts restrictions on modifying the code. This is the biggest misconception about the AGPL.
Making that code accessible over a network (which must include a source offer)
If you never make your modified software accessible over a network, it behaves just like the GPLv3. So I think characterizing it as “restricting modification” is not really the best way to think about it.
I’m not characterizing anything, that’s literally what the license says. It has not a single clause putting any restrictions on how you run the program, including network connections. It does have a clause that says that when you modify the code, you must modify it in such a way that whenever anyone later runs the code, it will direct network users to a copy of the code. It’s framed entirely in terms of modification and what your duties as someone who is modifying the code are.
As I recall, AGPL mostly just requires that you make your source available under the AGPL to anyone who runs your code or accesses it over the network.
That’s not a problem in any situation I can think of unless you want to keep some part of the source secret from users.
the point in the linked article, and the top-level comment I was replying to, is precisely that the original corporation that owns the copyright gets to develop secret features they don’t release, while no competitor would get to do that. that’s a usage restriction in my view.
But is this actually true? In the case in question (Database Product offered by the Vendor as PaaS), the actors you’re primarily concerned about are not outlaw “software pirates” but - as stated in the post - big cloud companies taking the product and offering a better PaaS on their own infrastructure. This is exactly what ended up happening to Elasticsearch for example.
Are there any cases where AWS, GCP, Azure etc. have taken an AGPL product and just ran with it? I don’t think there have.
AGPL doesn’t stop AWS from selling a hosted offering of the unmodified code. It doesn’t prevent them from cross-subsidising it to undercut you.
AGPL doesn’t stop AWS from adding code that integrates with other (paid, proprietary) back-end services, releasing the code that integrates with those services, but then having a version that works better on AWS than elsewhere and lets AWS charge more.
Most importantly, changing to a restrictive license doesn’t stop AWS from doing a quick internal rewrite and providing equivalent service but now closed source just weeks later. As we’ve seen AWS do multiple times.
In my opinion, keeping AWS “happy” and contributing upstream to a proper OSS codebase with a generous license is a better outcome for everyone than creating an adversarial environment where they prefer to compete with a closed source alternative. At least in this scenario, some of AWS’s resources flow back into the OSS ecosystem.
First of all, the outlaw “pirates” we’re talking about are MediaTek, VMWare, UMD Global (branded as “Nokia”), and realistically every major tech firm.
Second, yes, several companies have famously relicensed from AGPL to a nonfree license because AWS was happy to run the AGPL version. IIRC this includes your elastic search example, including that AWS then forked the AGPL version and became the steward of the freedomware fork.
That is not quite the whole story. All business features/plugins in Elasticsearch were proprietary licensed since the 1.0 version. That includes their access control product.
It’s not like “AWS took the product”, AWS always operated on open core and I think that’s very fine.
yeah, or people don’t want to believe it, which is fair enough but copyleft has been around long enough that there’s empirical evidence now
or people believe it but still see value in taking a stand - but in that case, stand on the high ground, not on the repurposed tool of oppression (copyright law)
To give an anecdote from my personal experience, as one of the things that pushed me towards permissive licenses:
I was a contributor to a GNU project. We discovered that there was a company selling a modified version of our GPL/LGPL code, with the copyright notices stripped and not providing source code to their customers. They had made some improvements but were not contributing them back.
The maintainer told the FSF and they decided that the cost of legal action was too great and so didn’t enforce the license. They required copyright assignment and so were the only ones with standing to sue, but that didn’t matter because none of us had the money for a lawsuit that would get us, maybe, a few tens of thousands of lines of code if we won (paying developers to rewrite the code would have been cheaper than paying lawyers to get the proprietary code and had a better chance of actually working).
At the same time, we heard from other companies that did want to be good citizens of the F/OSS ecosystem that they were avoiding our code because their lawyers deemed [L]GPLv3 too high risk. They were picking up a much less mature MIT-licensed alternative and contributing to it, rather than to our codebase.
In exchange for contributing to a copyleft project, I ended up with a project that (because it wasn’t all my own work), placed a bunch of restrictions on what I could do with it that didn’t apply to people who were willing to just ignore the license, and those people were making a bunch of money. At the same time, I was losing out on contributions from people who did want to follow the rules. People who intend to follow the rules are far more likely to be concerned about exactly what the rules allow, people who plan on ignoring them don’t care what they say.
I’ve never had the experience of someone violating the license of my work because I always use MIT for the reasons you stated, but I’ve seen exactly what you’re describing:
paying developers to rewrite the code would have been cheaper than paying lawyers to get the proprietary code and had a better chance of actually working
On multiple occasions I’ve worked within a company that saw some copyleft-licensed code they would have used, but the license was too high risk so they just did an internal clean room rewrite. This is so much easier than people think:
You don’t need to rewrite all the functionality, often only a tiny subset that is relevant to what you need. Let’s say 10-50% of effort.
All of the decisions and mistakes have already been made, even in a clean room rewrite. The surface area of the API is extremely informative of how things should work. Not having to make these decisions and mistakes saves 80%+ of development time.
Combined, a company who sees a copyleft codebase that took 50 weeks to build can often get an internal rewrite in just a few weeks. Best case scenario they release this code under a more permissive license, but more likely they just keep it private/internal because maintaining OSS is additional effort.
On the other hand, when a company sees a permissive license, they would much rather contribute upstream for the same reason: Maintaining code is additional effort, so an internal fork is counterproductive. Nobody enjoys rebasing and maintaining internal forks!
I’ve seen this again and again: Copyleft licenses result in more internal private rewrites, permissive licenses result in more upstream collaboration. It may feel counterintuitive against how the licenses are marketed, but this is reality.
I’m a copyright abolitionist. I want copyright gone. I want licenses to not exist at all.
However, until the day copyright is abolished, and that day may never come, it’s not really in our best interests to give away software with zero strings attached. That’s just wealth transfer, directly into other people’s pockets.
Permissive makes sense in a world without copyright. That world is not yet reality. The reality is they can copy open source software all they want but we can’t copy their proprietary software. They’ve managed to make it illegal to even reverse engineer their proprietary things because that often involves breaking protections.
I won’t claim to know what “most people”–or any people–do or don’t realize/understand, but for me, personally, it’s not that I don’t realize that actually enforcing copyright licenses is expensive and impractical. It’s just that I don’t find that to be a great argument against using licenses that attempt to do more things.
You can apply the same argument to many things. Just enforcing the “default” copyright restrictions on a published written work is expensive. If you’re an amateur author who self-publishes a book and some publishing giant sells a book that’s a total rip-off of yours, what are the odds that you can afford to actually beat them in court? Should small-time authors just publish everything they write under some Creative Commons license just because enforcing anything more restrictive might not be feasible? On the flip side, what if we become super-successful from our great book and we’re more able to enforce the copyright in the future? Or vice versa: we think we can afford to enforce the license now, but maybe our financial fortunes ebb and flow over time in ways that make it harder for us to enforce the license later on.
It would be great if enforcement of these things were free, but just because they aren’t doesn’t mean we should give people permission to do something we don’t want them to do.
I realize that analogies such as above are sometimes used dishonestly to set up a “straw man” to attack, but I’m not intending anything like that. It sounds like you and the parent comment are more-or-less arguing that we should only put things into our licenses that we are confident that we can actually afford to enforce. And that’s the idea that rubs me the wrong way.
Should small-time authors just publish everything they write under some Creative Commons license
Creative Commons have widely argued that their license is not fit for source code and software projects. So the answer would probably be “No”. You might want one of the BSD licenses?
This is the grand irony of AGPL when people complain about the recent trend of BSL/etc - regardless of whether they use AGPL or BSL or whatever other license, the original copyright owners always have the right to do whatever they want with it, while anyone else is subservient to the licence requirements; The only difference is that the AGPL licensed one has a bunch of mini Stallman’s chanting about how it is the one true way.
For all that people complain about MIT/BSD/Apache License as allowing forkers to do what they want - IMO it’s a huge red flag if a project isn’t permissively licensed.
For all that people complain about MIT/BSD/Apache License as allowing forkers to do what they want - IMO it’s a huge red flag if a project isn’t permissively licensed.
It’s only a red flag if you want to steal and make money with foreign work. This kind of rhetoric is exactly what big companies want you to parrot. Good thing the people who you call mini stallmans (this sounds derogatory 🙄) exist and think ahead.
It’s only a red flag if you want to steal and make money with foreign work.
Not at all. It’s a red flag because it puts the initial project creators in a privileged position, where they can create money through non-public enhancements/etc, but others cannot. It’s specifically why I mentioned the irony of AGPL fans complaining about BSL etc.
Good thing the people … exist and think ahead.
Good for who exactly? As described in this very thread there are numerous downsides to *GPL licenses, particularly the way it’s often used when backed by a company (with copyright assignment).
who you call mini stallmans (this sounds derogatory 🙄)
All I’m suggesting is that they have similar, hard-line views on software licensing, but aren’t as well known. If you think that’s derogatory, that says a lot about your own views on Stallman.
It’s a red flag because it puts the initial project creators in a privileged position, where they can create money through non-public enhancements/etc, but others cannot.
You are complaining about copyright assignment (CLA), not about AGPL.
If you run a project that is MIT licensed and ask everyone to assign copyright to you, it’s irrelevant as the LICENSE still allows them to do whatever they wish.
The *GPL is the limiting factor here, because it denies others the freedom to use the code the same way the copyright holders can.
No. Any license limits non-owners only, there’s nothing special about AGPL/GPL here.
If you contribute under CLA to a MIT-licensed project, tomorrow the owner can change the license of “your” changes to GPL or to any other. If you are looking around for a “red flag”, the CLA fits the bill best.
If I contribute to an MIT project with a CLA the owner can indeed do whatever the fuck they want with the code, but they also can’t limit what I can do with the code either.
The GPL, and specifically the AGPL does limit what I can do, forever, regardless of what the copyright owner does.
You can blame a CLA all you want, a CLA only has practical effects with a license like GPL. A CLA on a permissive licensed project is about as useful as a fart in an elevator.
These creators have released free software to the world. Free as in freedom software. I want them to have every privilege imaginable.
There is no irony whatsoever. BSL is not a free software license while AGPLv3 is. When something is released as AGPLv3, our freedom is ensured. When it’s released as BSL, it is not.
The drawbacks you describe are not drawbacks of the license itself, they are drawbacks of the justice system, especially the american one. It does not follow that AGPLv3 should not be used just because people can violate it whenever they want.
The copyright owner always has that privilege regardless of license. That will always be the case no matter which license is picked. It doesn’t make any sense to pick anything other than AGPLv3. I don’t want anyone else to have those freedoms.
GPL is about user freedom, not developer freedom. It’s about me, as a user, being able to have absolute control over the software running on my machine. It’s not about developer freedom to make money by creating proprietary software and services.
It’s not an AGPLv3 problem. The original copyright owner always has the right to do whatever they want to do. The code is literally their property. The license merely governs how others are allowed to use it.
Also there is no such thing as a “red flag”. People wrote the code, they can license it however they want. Others are lucky they can even see it.
I don’t believe copyright should exist but that’s the reality we find ourselves in right now. Permissive licensing is just wealth transfer. The value of your labor goes straight into the pockets of millionaires and corporations. It’s quite simply irrational in a world where copyright exists. It’s either AGPLv3 or all rights reserved.
If they want free code, they absolutely should have to abide by the AGPLv3’s terms. If they can’t, then there’s a business solution for that too: they can pay for permission to use it.
And Stallman is right. It’s better when only the copyright owner has that right. Software freedom is maximized this way. Stallman also advocates selling exceptions to the GPL, he has suggested this to companies, Trolltech and Qt being the most notable example. This made it possible for software to be freed.
It’s always amusing when RMS reaches a point where he concludes that GPL’d software has less value than proprietary software but then doesn’t reach beyond that to realise that, just maybe, the GPL is not the right tool if you want to maximise the amount of Free Software in the world.
I don’t think that’s an entirely fair characterisation, I’d put it somewhat differently:
GPL proponents want to minimise the amount of proprietary software in the world and are willing to accept less Free Software as a compromise.
Permissive-license proponents want to maximise the amount of Free Software in the world and are willing to accept more proprietary software as a compromise.
It is absolutely the case that a GPL licence artificially limits the number of entities willing to contribute to a given open source project. Yes some projects are popular enough and high profile enough that they get plenty of contributions anyway. But it’s naive to believe that the GPL doesn’t prevent people or organisations from contributing to projects, because of its limitations.
So it’s not just the case of “less” vs “more” - you have to consider what effect the license has on quality as well.
It’s not about developer’s freedoms, it’s about user’s freedoms. As a user, I absolutely want to have control over software. I don’t want developers to have the freedom to take control away from me at all, such as by making proprietary software or services.
Permissive licenses are the opposite. You can do whatever you want, including proprietary software and services. It enables exploitation of both users and fellow open source developers. People labor over code, corporations take it, make millions, don’t pay them a cent and don’t even give source code back. That’s not the freedom I want to support. It’s exploitation.
As a user, I absolutely want to have control over software. I don’t want developers to have the freedom to take control away from me at all, such as by making proprietary software or services.
This does not compute. Some company taking some permissively-licensed e.g. MIT or Apache software does not “take control away from the user”. It doesn’t make that permissive license disappear, or the software itself that was permissively licensed. So as far as “user freedoms” go, a permissive license is perfectly fine.
You want something else entirely: You want the power to prevent companies from using existing software, unless they adopt your system of morals. You may choose to argue and bicker about whether “adopting your system of morals” is the question here, but copyleft is unabashedly a system of morals. And there’s nothing wrong with that choice for software that you have created: As the copyright holder, you have that right to demand that companies adopt your system of morals if they want to use software that you wrote. And as is evident from AGPLv3, companies generally seem to refuse to adopt your system of morals.
That’s not the freedom I want to support. It’s exploitation.
It’s an interesting definition of freedom that you have, that you argue against someone’s freedom to choose to allow their work to be exploited. Whether or not I agree with the position you are taking, there does exist a logical contradiction in your position.
Some company taking some permissively-licensed e.g. MIT or Apache software does not “take control away from the user”.
It enables the development of software that does. I don’t really want to contribute to the creation of software that’s not part of the commons. Not for free, at least.
You want the power to prevent companies from using existing software, unless they adopt your system of morals.
I do. And I want to go even further, and prevent a lot more things than that.
And as is evident from AGPLv3, companies generally seem to refuse to adopt your system of morals.
Then they should pay for the software or hire someone to do it themselves.
you argue against someone’s freedom to choose to allow their work to be exploited
I’m arguing the opposite. Copyright owners always have that freedom, no matter which license they choose. Licenses only apply to others.
They can release software to the world AGPLv3 licensed and strike business deals with corporations that let them use the software under different terms – for a price.
Those who aren’t willing to pay will have to comply with the AGPLv3 and that’s the ideal outcome. They won’t be able to exploit the work, but the truth is it’s not really their work to exploit to begin with.
Permissive licensing reduces the leverage of all software developers, it just gives it all away at zero cost. They don’t even have to pay the cost of sharing source code. AGPLv3 gives you leverage, and with leverage comes negotiation, and hopefully business.
I want people to appreciate the work I’ve done and the value of what I’ve made.
Not pass on by waving “sucker” as they drive their fancy cars.
If you use permissive licenses, you will be taken advantage of. You’re laboring all over your code and someone will come and take it, use it to make billions and not even give you the source code back which is literally the bare minimum. So it’s either AGPLv3 or all rights reserved, no free software at all. Those are the only choices which make any sense whatsoever as far as I’m concerned.
He’s also written a great post on relations between corporations and open source:
If you use permissive licenses, you will be taken advantage of. You’re laboring all over your code and someone will come and take it, use it to make billions and not even give you the source code back which is literally the bare minimum.
If making billions from my code were easy, I would be a billionaire. Sony ships some of my code on the newer PlayStations, for example, and if they didn’t then they’d have to spend a few thousand dollars rewriting it in house. There is alternative option that means I get money from every PlayStation sold.
In my experience, companies are more likely to give code back to permissively licensed projects than GPL’d ones. With GPL, they have several options if they’re using the project in house:
Make changes and give them back.
Make changes and keep them proprietary (totally fine unless they’re distributing the code).
Reimplement the project and keep their version proprietary.
Reimplement the project and release it under a permissive license to get other contributors and defray some of their costs.
Of these, the first option is the one that the lawyers like the least. As soon as you start contributing changes upstream, you’re announcing that you’re using a project that has a complex license. It is much lower risk to keep the changes private. Unless your project is (at least) hundreds of thousands of lines of code, the third and fourth options are often affordable. And may be lower risk than touching something GPL’d.
In contrast, if it’s permissively licensed, their choices are:
Make changes and keep them proprietary.
Make changes and contribute them upstream.
Most importantly, they can change their mind easily. And this happens quite often. I’ve worked with several companies that have forked permissively licensed projects and then realised that this is expensive. They implement features that are then implemented differently upstream and then have painful merges. Unless something is really critical to their commercial differentiators, it’s usually cheaper to upstream.
In contrast, if they’ve looked at a GPL’d project and decide to build something proprietary, there’s no path to ever contributing it to a totally different GPL’d codebase. This is really important because companies that aren’t use to working with an open-source upstream (remember, most software is developed for in-house use, not as off-the-shelf software).
I release things as F/OSS for three principle reasons:
It costs me nothing to do.
If it’s useful to someone else, that’s nice.
If someone sends contributions, I get free work done on the project.
None of these is made better by a non-permissive license. It costs me the same amount. It will restrict the people who can use it. It may prevent someone from taking my work and making a pile of money, but I don’t lose anything because someone else makes money. It reduces the chance that someone else will send contributions and, unless I make contributors sign a CLA (which pushes even more away), it then restricts what I can do with the combined project. If I release an MIT-licensed library and someone sends me a patch, I can still use the library in whatever project I want, even something proprietary. If I release a GPL’d component and someone sends me a GPL’d patch, I can now do less: if it’s just my code, I can give myself a different license, but if it’s my code + someone else’s, then I need their consent to give myself that license.
And it’s not just reusing my code under a proprietary license, it’s also reusing it in a project that wants to pull in a dependency under a different but incompatible OSI and FSF-approved license. I’ve been bitten by that in the past with libPoppler, which was extracted from xpdf, which was GPLv2-only and so incompatible with a bunch of other things (including Apache 2). With more convoluted licenses such as AGPL, the probability of incompatibility goes up.
I’m unconvinced. There are generally 2 incompatible worlds in competition: permissive and GPL.
Your corporation and its lawyers want GPL world gone, and in the name of risk mitigation they want to be legally able to buy off all maintainers for peanuts and kill permissive world on any given day. Historically, this nearly happened once with Unices. No, thank you.
On the other hand, if GPL world wins, nothing bad happens.
Your corporation and its lawyers want GPL world gone
Most corporations don’t care about GPL world at all. It’s just a load of code that they don’t want to touch, as is most proprietary software.
and in the name of risk mitigation they want to be legally able to buy off all maintainers for peanuts
I don’t really understand what this means.
and kill permissive world on any given day.
How does that work? Most permissive licenses and the GPL are irrevocable. You can’t kill either. And, given that most corporations benefit from a thriving ecosystem of permissively licensed things that they can use and modify, why do you think they want to kill it?
Historically, this nearly happened once with Unices. No, thank you.
Not really. AT&T sued Berkeley (for largely unrelated and stupid reasons), but most of the proprietary UNIX vendors coexisted happily with the BSD ecosystem, contributed some things to it, and kept their own differentiating bits. The differentiating bits were non-portable and so most software avoided them. Things like NFS came out of proprietary UNIX and were released under a BSD license to encourage adoption.
On the other hand, if GPL world wins, nothing bad happens.
The problem is the lack of path from here to there. For the ‘GPL world wins’ outcome, you have to get people to contribute more to GPL’d code than to permissive and proprietary software. And, in 30 years of the GPL, this hasn’t happened.
As I posted elsewhere in this thread, I have personally seen companies take GPL’d code, violate the license, and the FSF (the copyright holder) do nothing to enforce it, meaning that GPL’d code is proprietary for anyone with enough money. I have seen companies decide to do a from-scratch proprietary implementation of a GPL’d thing rather than use the GPL’d thing, meaning that the use of the GPL increases the amount of proprietary software in the world. I have seen companies pay for a license for a proprietary thing rather than use a GPL’d thing (thereby funding the development of more proprietary software, meaning that the choice of the GPL increases the amount of proprietary software in the world)). I have seen companies do from-scratch permissively licensed implementations of GPL’d things rather than use the GPL’d thing (meaning that the choice of the GPL increased the amount of permissively licensed software, though by less than starting with a permissive license would).
When was the last time you saw a GPL’d implementation of something displace a permissively licensed one? When was the last time you saw anyone fund development of a GPL’d replacement for an existing permissively licensed project?
There is alternative option that means I get money from every PlayStation sold.
They pay you royalties on every console they sell? I’m not sure I believe that.
Every once in a while we have some sort of public meltdown because someone has been thanklessly and profitlessly maintaining code that even billionaire corporations use and depend on. Microsoft employees will show up to the ffmpeg issue tracker, self assign a high priority issue and then offer them a few thousand dollars when they propose a support contract.
So why should anyone believe that corporations will just happen to choose to pay people royalties for no reason?
I’ve worked with several companies that have forked permissively licensed projects and then realised that this is expensive. They implement features that are then implemented differently upstream and then have painful merges. Unless something is really critical to their commercial differentiators, it’s usually cheaper to upstream.
That’s by design. It should be as expensive as possible for them to not develop free and open source software. It’s the only leverage we have.
That’s also true of GPL software, by the way. Linux is famous for weaponizing it by deliberately keeping the internal kernel ABI unstable. Linux gets multiple tens of patches every single hour. They get so many patches, these corporations will never be able to keep up unless they upstream their code. If they don’t, they get left behind, their products don’t run mainline kernels, they don’t work out of the box, they don’t benefit from the continuous improvements, they are buggier and just generally worse as a result.
I’ve worked with corporations too. About a week ago, one of them sent me GPLv2 Linux drivers for a power supply. I’m trying to clean it up so I can upstream it. The drivers target Linux 3.10 which was released over a decade ago. It would have been much better if they had simply upstreamed the code: Linux would have had builtin support for the device and all products which used it would have worked out of the box. Why they don’t do that is beyond me.
They pay you royalties on every console they sell? I’m not sure I believe that.
Sorry, typo, ‘There is alternative’ was meant to read ‘There is no alternative’.
No matter what license I choose, Sony will not pay me money. They will either:
Use my code and not contribute anything.
Use my code and contribute improvements.
Use something else.
These are the three choices that exist. I prefer to incentivise the second option.
Every once in a while we have some sort of public meltdown because someone has been thanklessly and profitlessly maintaining code that even billionaire corporations use and depend on.
And that’s fine. If they want support, they can pay for it. I’m not going to prioritise their bugs just because they’re big. If something is a show-stopper for them, that’s their problem, not mine. If they fix it, I’ll review their PR when I get around to it.
Perhaps my experience is unusual, but I’ve had far more positive interactions with folks at big companies than negative. I’ve had important bug fixes and new features contributed by companies because they want the feature and have worked with upstream. I’ve never had a big company drive by the issue tracker and demand fixes (I have had individuals do this, on a lot of occasions). I did once have Microsoft create a fork of one of my projects because they had some code that was a bit ugly to make things work now for a product release, but it was a friendly fork and they worked with me to clean up the code and upstream it, so everyone won in the end.
That’s by design. It should be as expensive as possible for them to not develop free and open source software. It’s the only leverage we have.
But that isn’t what happens. It incentivises developing proprietary alternatives to Free Software projects.
They get so many patches, these corporations will never be able to keep up unless they upstream their code
Tell that to Google, who kept entire filesystems out of tree for their internal infrastructure. A lot of companies keep in-house forks of Linux.
I’ve worked with corporations too. About a week ago, one of them sent me GPLv2 Linux drivers for a power supply. I’m trying to clean it up so I can upstream it. The drivers target Linux 3.10 which was released over a decade ago. It would have been much better if they had simply upstreamed the code: Linux would have had builtin support for the device and all products which used it would have worked out of the box. Why they don’t do that is beyond me.
Because the GPL sets up an adversarial relationship. They are not sending you the drivers because they feel they get any value from it, they are sending you drivers because they are legally obliged to. They’ve talked to lawyers and those lawyers have worked out the minimal set of things that they need to do to be compliant. And they won’t change because the GPL has locked them into a mindset that engaging with the community is a source of liability.
No matter what license I choose, Sony will not pay me money.
So why should they get free code with no strings attached at all?
Sony is using open source code to make locked down game consoles, digital fiefdoms where we are their serfs. The only reason the playstations ever ran Linux is because they’d get tax benefits in some regions. That’s how they look at us, free code they can use to make billions and dodge taxes.
I’d prefer that they not use my code at all than watch them use it to make a killing off of things like DRM which I think should be illegal to begin with. I feel like publishing code which enables a corporation like that is actively working against my own interests which is just irrational. I’d probably think differently if they were paying me fat stacks, but that’s just the thing, they aren’t doing that.
Tell that to Google, who kept entire filesystems out of tree for their internal infrastructure. A lot of companies keep in-house forks of Linux.
That can’t be easy. Nor should it be made easy.
This is reflected in the effort they’re making to run mainline kernels on Androids and especially their own hardware, no doubt because the firmware situation is too painful to cope with otherwise. Google has contributed a lot towards making smartphones and laptops more free by improving mainline Linux support.
they are sending you drivers because they are legally obliged to
As they should be. And they should be legally obliged to allow me, the user and owner of the machine, to customize that code and run it on the device I bought, too. It should be literally my right, the GPL should not even be necessary to make it happen.
So why should they get free code with no strings attached at all?
Okay, so your starting point is that everything has to be a zero-sum game and no one can ever benefit from things without contributing? I guess that explains why we disagree. I would hate to live in a society that was built according to your ideals.
As they should be. And they should be legally obliged
So you build a society that is not enforced by people agreeing to act in together for shared interests, but by the threat of legal action. Again, I don’t want to live in the kind of society that you build. Fortunately, neither do most other people and so there’s little danger of it.
You’re laboring all over your code and someone will come and take it, use it to make billions and not even give you the source code back which is literally the bare minimum.
Wow. I thought things got hyperbolic before. I was going to engage on a few points but I don’t think it’s really worthwhile when you clearly don’t understand what words like “literal” mean, and then assume everyone else must have the same values and view point as you do.
Edit to add:
I just want to add a very fucking relevant quote from that first article you posted, given how insistent you are that my choice is “wrong”:
I really don’t care what license people use on their software, it’s their software and bitching at them for the license they choose is offensive. They wrote it, put their blood and sweat into it, and you should be glad that you get the privilege of even seeing it.
I respect your right to license your work however you want. I don’t really understand the reasons for using permissive licensing because I just don’t see the advantage, but that’s okay. I can’t make the choice for others.
I’m aware of that quote. I even paraphrased it in another reply. As far as this discussion is concerned, I’m just trying to address the points made in each comment I replied to.
That quote is especially relevant here given the claim that using AGPLv3 is a “red flag”. Personally I think it’s the only reasonable choice, and I made it a point to explain why I think that. Not to convince you personally to use the license but because I don’t want my own projects to have “red flags” in them.
There’s also another part in that text that wasn’t cited but is very relevant:
At a minimum you’re insulting someone’s belief in open source and free software, and how they feel it should progress in our culture.
I believe free software is about maximizing our own advantages as developers and above all as computer users, and that it’s not at all about just giving everything away for free. I don’t really appreciate it when that belief is reduced to a “red flag”.
I don’t want my own projects to have “red flags” in them
That’s literally impossible. Red flags are inherently an individual thing. For you a red flag might be that it’s not GPL licensed. You literally cannot please everyone.
I don’t really appreciate it when that belief is reduced to a “red flag”.
You can believe whatever you want. I believe different things. It’s that simple.
It’s exactly because of this issue that my company has considered assigning our copyrights to a charity, so that we couldn’t do that even if we wanted to and would be bound by the license ourselves.
The meme-loving functional programmers would giggle that you unwrapped your burrito.
So I’ve been thinking about why the “monads are just like X!” analogies fall flat, and I realized that the core of it is that monads are purely an abstraction. Monads are just a set of operations, and that happens to be useful when applied to real things that follow certain laws.
It’s almost a copout, except that’s actually the default for mathematics: numbers are purely an abstraction too. Consider:
Jack has four apples and Jill gives him three apples. How many apples does Jack have?
Jack is four years old, Jill is three years older. How old is Jill?
Jack got third place in a race, Jill came in four places behind him. What place did Jill get?
Quantities, timespans, and ordinals are “real things”, totally incomparable with each other, and yet all are modelable as 4 + 3 = 7. It’s only that numbers feel natural because we use them all the time, so we don’t think about how they’re actually a weird abstraction over reality.
I’ve always felt a little weird when I read about how hard it is to understand monads, or that they’re “like boxes” or “like burritos”. I even feel weird when someone mockingly cites a technical definition that has “scary” sounding words like “endofunctor”.
I feel weird because I don’t find it complicated and I’m not impressed/scared by technical terms like “endofunctor” even though I never use such terms enough to even remember what it means most of the time. But I feel like if I say that, it either means I’m missing something so obviously “complicated” to everyone else, or that I’ll come across as condescending or full of myself.
To my mind, a monad is just a monad. It’s just a word with a definition. It’s a “thing” that holds values from a specific set and, by definition, upholds certain invariants and allows certain operations over those values. This doesn’t feel any different to me than any other mathy definition. “What’s a rational number? It’s an element of the set of numbers that can be represented by a ratio of P/Q where P is any integer and Q is any integer that’s not 0.” (I don’t know if that’s actually a good definition, but you get my point).
I was halfway through my thesis when I realized this. I’m not special for understanding this, I just care enough. Then again, maybe all of science works this way.
…I’m not impressed/scared by technical terms like “endofunctor”…a monad is just a monad. It’s just a word with a definition…
Totally agree. It’s sad because I sometimes read on lobster people keeps scary words to threaten others. But when going just a little further, ask them about interesting examples of “functors”, they cannot give any but the functor between categories of sets.
And here I thought a monad was a type with a functional application that is associative, a value that acts like an identity, and (most important but almost never mentioned) require lazy evaluation to actually work mathematically.
Why does it need lazy evaluation? There’s enough lambda abstraction in the monad laws that I can’t see where eagerness could affect them. Or, to put it another way, the algebraic structure of a monad imposes an order of evaluation, which is why monads are useful for imperative functional programming.
Aha, thanks, that (after a bit of thought) makes sense. (Though the issue is really purity rather than laziness.)
Eventually I worked out where my mistake came from: in monads like State and IO there’s another layer of lambda abstraction wrapping the action represented by the monad. The action doesn’t occur until it is applied to an initial state; what the monad algebra does is construct the action so that the state is threaded through linearly.
It’s also much easier to appreciate that abstraction when you already care about the examples! For me, “functor is anything you can map over” and “monoid is anything you can append” feel much more obvious as patterns that pop up a lot, that I’d want an interface for. And no more abstract than an Iterator or Collection interface (besides the choice of names).
I think people may learn monads backwards, because they hear about “the IO monad” and think you need monads to do IO–but that’s like saying you need functors to do lists, because of “the List Functor”. I much prefer the Learn you a Haskell approach, where you learn about “IO actions”, and how to construct them and chain them together.
I mostly agree, but in another light, I’d make the analogy that monads are to lists as groups are to (types of) numbers.
You have the familiar operations (addition, multiplication), but shorn of guarantees that we typically take for granted (commutativity). In the case of a list, it’s trivial to get a value out of a list. In the case of a monad, not so much.
Of course, monads also muck up the matter by not overloading the common operations. It would probably be confusing in a different way if the monad operations were “push” and “append” or whatnot.
I think this is generally true. People get familiar with specific abstractions through practice.
As another instance, computation is “just” an abstraction. We’re dimly aware that processors do “something” much more complex, but it’s nice to think about sequenced orders of steps and mutable memory and the like.
When you were first learning it, you probably had to work really hard to build familiarity with that abstraction. And then over time it becomes second nature and you can think of it occurring all over the place.
Working with specific abstractions after a lot of practice is relatively common. Working with the act of abstraction and generating new ones is less common. Mathematicians definitely do it a lot. We’d think programmers would too, but I think, often, programmers prefer to imagine things concretely.
When talking to folks new to the Rust language, a common misconception I encounter is that ? is some kind of special syntax that only works for Result.
…
To be able to use the Try trait, we need to use the nightly compiler at the moment, as it is not yet stabilized. We then also need to activate the feature for the trait, and import a few dependencies:
No wonder there is a “misconception” that it’s special syntax when it actually is special syntax in non-experimental versions of the language.
The ? operator was originally stabilized in 1.13. Before then, there was the try! macro (which now requires special syntax to call as of the 2018 edition). The try! syntax was originally syntactic sugar for this:
// let x = try!(foo);
// expands to:
let x = match foo {
Ok(value) => value,
Err(error) => {
return Err(error.into());
},
};
But the rest is spot on, ? originally desugared to the same thing as try!, and only added an unstable trait behind-the-scenes sometime later.
When talking to folks new to the Rust language, a common misconception I encounter is that ? is some kind of special syntax that only works for Result. […]
No wonder there is a “misconception” that it’s special syntax when it actually is special syntax in non-experimental versions of the language.
Even in stable Rust, the standard library can use unstable features and implements Try for more types, including the common Option<T>.
To be fair, one reason one would assume it’s special syntax is simply because you never need to implement Try. I have only ever once in my life had the desire to implement a custom Try implementation, and in the end I realized I really should just use the standard Result and that I was tryharding way too much, akin to operator overload abuse. Outside of that, people probably never even take a moment to think if they can overload ?.
Aside: I’ve always found the timers APIs to be a little tricky, anyway. They aren’t “hard”, per se, but it’s a lot like implementing a binary search: easy, but also easy to mess up.
I feel similarly to the whole EventEmitter approach. Every EventEmitter has its own semantics about what events are emitted and when, and it can be very easy to accidentally assume certain behaviors will or won’t happen, or that certain event orders are guaranteed, etc. And then it’s also easy to cause memory leaks by not removing event listeners when you’re done with them. I’ve seen a lot of slightly-incorrect StackOverflow answers when it comes to dealing with Node.js’s various EventEmitters.
Nicely written, and I do think that’s a big part of the issue that I had never considered before. I’ve complained about the speed of software/computers today, but I simultaneously don’t delude myself into remembering with rose-tinted glasses. It always used to take a long time, on any OS, to load a large directory from a spinny-HDD in whatever GUI file manager. But, like the article says: that’s expected to be slow- humans all understand that more items to process means more time is needed to process them. But, spending a bunch of time to just switch between documents that are, allegedly, already “open” is a surprising point of friction to encounter and that’s why it’s so frustrating. Or, when key presses or clicks are “lost” because the UI was too busy doing something else and just didn’t even queue up the inputs.
It’s definitely the “stupid slow” that we’re the most frustrated about. Thank you for this new vocabulary! :)
Spent a lot of time on firefox perf engineering, been writing simple uis since windows 3.11
Current UI trend of implemented on top of dom in functional-reactive manner has a really good reason: UI state is freaking hard. Prior to React UIs were much more buggy. Even simple things like resizing windows was a minefield. UI components are difficult to theme, introspect. Supporting multiple platforms and mobile in same codebase was merely a dream. Displaying tables of data was incredibly frustrating because every platforms had different limits in what one could do in tables.
So yeah DOM-based UI sucks for perf due to not having explicit control over every pixel, but I think it’s pretty cool that modern devs take functional multi platform ui rendering for granted. We would be much worse off if web never took off and we were all stuck with a mix of Borland, Microsoft, Sun swing, etc. Imagine if the hated slack ui looked like a windows app on every platform.
It’s about wanting to spend on dev. A company like Slack is big enough to handle 3 native implementations. They don’t do it because their customers are okay with the bloat. Look at 1Password – they went from great native apps, to bloated electron apps. Once you get enterprise customers on your side, you don’t have to be lean.
So yeah DOM-based UI sucks for perf due to not having explicit control over every pixel
I’m not at all an expert in this domain, but it’s hard for me to imagine that the reason these UIs suck at performance/latency has anything to do with having “explicit control over every pixel”. I doubt any of the GUI apps that I use that feel “snappy” are actually working at such a low level that they’re dealing with individual pixels or doing their own refresh rate calculations, etc.
But, I honestly don’t know what the issue is. Like the article describes, there’s really no reason that MS Teams should be slow to switch to a different chat. There’s obviously just some lackluster engineering of what’s being done on the main thread vs what should be offloaded to a background task (maybe loading too much history upfront, or applying too much styling to elements that wouldn’t be visible yet anyway, etc).
I imagine it’s because developers are more likely to add a network call to any given UI action these days. If the expectation is that everyone has a fast reliable internet connection, then why not put a network call? Only issue is that large tech companies have big systems so that one network call from a user might be making a large number of calls to other internal services.
Plus, companies want to own your data so they are not necessarily incentivized to keep it device local. Attitudes towards collecting data probably subtlety influence app design in a way that hurts latency.
DOM-based browser control is relatively easy to use correctly, but extremely hard to reason when it’s gonna do stuff. It’s one of the least intuitive architectures when it comes to performance. It’s a gigantic stack with all kinds of hacks, much harder to reason about that the single-threaded event loops of old school desktop apps.
For example modern browsers offload renders to graphics cards, which results in really smooth animations and scrolling(atleast sometimes), but a lot of the [implicit] communication between layout engine and gpu causes UI freezes and your app has 0 control over that. This is a case where using “higher-performance” apis sometimes results in speedups and often in slowdowns. Same case with multi-process rendering..shoveling tons of data over Interprocess-Comms is also very likely to cause your app to stall and you have little visibility into that.
One of the reasons some people like the web canvas api for high-perf stuff is because it is much closer to the old simpler apis.
IMO a major factor to perceived slow UI on modern systems is that modern UI frameworks are designed with simplified APIs compared to what people used in the ’90s and early ’00s. React-ish web frameworks running in an Electron shell are the prototypical example (and several Electron-based programs are mentioned in the article), but the same is true of SwiftUI vs Cocoa, or the various .NET UI frameworks vs Win32.
For example, last year I wrote a small macOS program that needed to display a big list of values in a table. I used SwiftUI for it and it was slow, to the point where my laptop was dropping frames when trying to scroll. I switched the table widget to NSTableView and it was much faster. The difference was that NSTableView exposed additional API that allowed me to reuse allocated rows, whereas the SwiftUI table just doesn’t provide that functionality at all (as far as I can tell).
However, the SwiftUI version took about 30 minutes to write, starting from googling “swiftui table tutorial”. The NSTableView version took me about two days, and even after finishing it I have maybe 30% confidence in the code not being buggy somehow. This isn’t a slight against Cocoa, it’s just a much lower-level API and I have zero experience in macOS programming.
From the perspective of a fairly junior programmer employed in a feature factory, working to a strict deadline to produce software that just needs to be “good enough”, an API that can take an absolute beginner from an empty file to a completed ticket in 30 minutes is great.
I have to support some pretty old iOS devices, so I still haven’t been able to move to SwiftUI even if I wanted to, but it’s really disappointing to hear that the performance is still garbage (or was a year ago, anyway) after four or five years of being pushed as the de facto way to do MacOS/iOS UI code.
The TypeScript dev lead posted this response about the language choice on Reddit, for anyone who’s curious:
Source: https://www.reddit.com/r/typescript/comments/1j8s467/comment/mh7ni9g
People really miss the forest for the trees.
I looked at the repo and the story seems clear to me: 12 people rewrote the TypeScript compiler in 5 months, getting a 10x speed improvement, with immediate portability to many different platforms, while not having written much Go before in their lives (although they are excellent programmers).
This is precisely the reason why Go was invented in the first place. “Why not Rust?” should not be the first thing that comes to mind.
I honestly do think the “Why not Rust?” question is a valid question to pop into someone’s head before reading the explanation for their choice.
First of all, if you’re the kind of nerd who happens to follow the JavaScript/TypeScript dev ecosystem, you will have seen a fair number of projects either written, or rewritten, in Rust recently. Granted, some tools are also being written/rewritten in other languages like Go and Zig. But, the point is that there’s enough mindshare around Rust in the JS/TS world that it’s fair to be curious why they didn’t choose Rust while other projects did. I don’t think we should assume the question is always antagonistic or from the “Rust Evangelism Strike Force”.
Also, it’s a popular opinion that languages with algebraic data types (among other things) are good candidates for parsers and compilers, so languages like OCaml and Rust might naturally rank highly in languages for consideration.
So, I honestly had the same question, initially. However, upon reading Anders’ explanation, I can absolutely see why Go was a good choice. And your analysis of the development metrics is also very relevant and solid support for their choice!
I guess I’m just saying, the Rust fanboys (myself, included) can be obnoxious, but I hope we don’t swing the pendulum too far the other way and assume that it’s never appropriate to bring Rust into a dev conversation (e.g., there really may be projects that should be rewritten in Rust, even if people might start cringing whenever they hear that now).
While tweaking a parser / interpreter a few years ago written in Go, I specifically replaced a struct with an ‘interface {}’ in order to exercise its pseudo-tagged-union mechanisms. Together with using type-switch form.
https://github.com/danos/yang/commit/c98b220f6a1da7eaffbefe464fd9e734da553af0
These day’s I’d actually make it a closed interface such that it is more akin to a tagged-union. Which I did for another project which was passing around instances of variant-structs (i.e. a tagged union), rather than building an AST.
So it is quite possible to use that pattern in Go as a form of sum-type, if for some reason one is inclined to use Go as the implementation language.
That is great explanation of “Why Go and not Rust?”
If you’re looking for “Why Go and not AOT-compiled C#?” see here: https://youtu.be/10qowKUW82U?t=1154s
A relevant quote is that C# has “some ahead-of-time compilation options available, but they’re not on all platforms and don’t really have a decade or more of hardening.”
That interview is really interesting, worth watching the whole thing.
Yeah Hjelsberg also talks about value types being necessary, or at least useful, in making language implementations fast
If you want value types and automatically managed memory, I think your only choices are Go, D, Swift, and C# (and very recently OCaml, though I’m not sure if that is fully done).
I guess Hjelsberg is conceding that value types are a bit “second class” in C#? I think I was surprised by the “class” and “struct” split, which seemed limiting, but I’ve never used it. [1]
And that is a lesson learned from the Oils Python -> C++ translation. We don’t have value types, because statically typed Python doesn’t, and that puts a cap on speed. (But we’re faster than bash in many cases, though slower in some too)
Related comment about GC and systems languages (e.g. once you have a million lines of C++, you probably want GC): https://lobste.rs/s/gpb0qh/garbage_collection_for_systems#c_rrypks
There was also a talk that hinted at some GC-like patterns in Zig, and I proposed that TinyGo get “compressed pointers” like Hotspot and v8, and then you would basically have that:
https://lobste.rs/s/2ah6bi/programming_without_pointers#c_5g2nat
[1] BTW Guy Steele’s famous 1998 “growing a language” actually advocated value types in Java. AFAIK as of 2025, “Project Valhalla” has not landed yet
Compilers written in OCaml are famous for being super-fast. See eg OCaml itself, Flow, Haxe, BuckleScript (now ReScript).
Yeah, I’m kind of curious about whether OCaml was considered at some point (I asked about this in the Reddit thread, haven’t gotten a reply yet).
OCaml seems much more similar to TS than Go, and has a proven track record when it comes to compilers. Maybe portability issues? (Good portability was mentioned as a must-have IIRC)
Maybe, but given that Flow, its main competitor, distributes binaries for all major platforms: https://github.com/facebook/flow/releases/tag/v0.264.0
Not sure what more TypeScript would have needed. In fact, Flow’s JavaScript parser is available as a separate library, so they would have shaved off at least a month from the proof of concept…
Also Nim.
Also Julia.
There surely are others.
Yes good points, I left out Nim and Julia. And apparently Crystal - https://colinsblog.net/2023-03-09-values-and-references/
Although thinking about it a bit more, I think Nim, Julia, (and maybe Crystal) are like C#, in that they are not as general as Go / D / Swift.
You don’t have a
Foo*type as well as aFootype, i.e. the layout is orthogonal to whether it’s a value or reference. Instead, Nim apparently has value objects and reference objects. I believe C# has “structs” for values and classes for references.I think Hjelsberg was hinting at this category when saying Go wins a bit on expressiveness, and it’s also “as close to native as you can get with GC”.
I think the reason this Go’s model is uncommon is because it forces the GC to support interior pointers, which is a significant complication (e.g. it is not supported by WASM GC). Go basically has the C memory model, with garbage collection.
I think C#, Julia, and maybe Nim/Crystal do not support interior pointers (interested in corrections)
Someone should write a survey of how GC tracing works with each language :) (Nim’s default is reference counting without cycle collection.)
Yeah that’s interesting. Julia has a distinction between
struct(value) andmutable struct(reference). You can use raw pointers but safe interior references (to an element of an array for example) include a normal reference to the (start of the) backing array, and the index.I can understand how in Rust you can safely have an interior pointer as the borrow checker ensures a reference to an array element is valid for its lifetime (the array can’t be dropped or resized before the reference is dropped). I’m very curious - I would like to understand how Go’s tracing GC works with interior pointers now! (I would read such a survey).
Ok - Go’s GC seems to track a memory span for each object (struct or array), stored in kind of a span tree (interval tree) for easy lookup given some pointer to chase. Makes sense. I wonder if it smart enough to deallocate anything dangling from non-referenced elements of an array / fields of a struct, or just chooses to be conservative (and if so do users end up accidentally creating memory leaks very often)? What’s the performance impact of all of this compared to runtimes requiring non-interior references? The interior pointers themselves will be a performance win, at the expense of using an interval tree during the mark phase.
https://forum.golangbridge.org/t/how-gc-handles-interior-pointer/36195/5
It’s been a few years since I’ve written any Go, but I have a vague recollection that the difference between something being heap or stack allocated was (sometimes? always?) implicit based on compiler analysis of how you use the value. Is that right? How easy it, generally, to accidentally make something heap-allocated and GC’d?
That’s the only thing that makes me nervous about that as a selling point for performance. I feel like if I’m worried about stack vs heap or scoped vs memory-managed or whatever, I’d probably prefer something like Swift, Rust, or C# (I’m not familiar at all with how D’s optional GC stuff works).
Yes, that is a bit of control you give up with Go. Searching for “golang escape analysis”, this article is helpful:
https://medium.com/@trinad536/escape-analysis-in-golang-fc81b78f3550
So the toolchain is pretty transparent. This is actually something I would like for the Oils Python->C++ compiler, since we have many things that are “obviously” locals that end up being heap allocated. And some not so obvious cases. But I think having some simple escape analysis would be great.
Yes, the stack/heap distinction is made by the compiler, not the programmer, in Go.
Why did you leave JS/TS off the list? They seem to have left it off too and that confuses me deeply because it also has everything they need
Hejlsberg said they got about 3x performance from native compilation and value types, which also halved the memory usage of the compiler. They got a further 3x from shared-memory multithreading. He talked a lot about how neither of those are possible with the JavaScript runtime, which is why it wasn’t possible to make tsc 10x faster while keeping it written in TypeScript.
Yeah but I can get bigger memory wins while staying inside JS by sharing the data structures between many tools that currently hold copies of the same data: the linter, the pretty-formatter, the syntax highlighter, and the type checker
I can do this because I make my syntax tree nodes immutable! TS cannot make their syntax tree nodes immutable (even in JS where it’s possible) because they rely on the
node.parentreference. Because their nodes are mutable-but-typed-as-immutable, these nodes can never safely be passed as arguments outside the bounds of the TS ecosystem, a limitation that precludes the kind of cross-tool syntax tree reuse that I see as being the way forwardHejlsberg said that the TypeScript syntax tree nodes are, in fact, immutable. This was crucial for parallelizing tsgo: it parses all the source files in parallel in the first phase, then typechecks in parallel in the second phase. The parse trees from the first phase are shared by all threads in the second phase. The two phases spread the work across threads differently. He talks about that kind of sharing and threading being impractical in JavaScript.
In fact he talks about tsc being designed around immutable and incrementally updatable data structures right from the start. It was one of the early non-batch compilers, hot on the heels of Roslyn, both being designed to support IDEs.
Really, you should watch the interview https://youtu.be/10qowKUW82U
AIUI a typical LSP implementation integrates all the tools you listed so they are sharing a syntax tree already.
It’s true that I haven’t watched the interview yet, but I have confirmed with the team that the nodes are not immutable. My context is different than Hejlsberg’s context. For Hejlsberg if something is immutable within the boundaries of TS, it’s immutable. Since I work on JS APIs if something isn’t actually locked down with
Object.freezeit isn’t immutable and can’t safely be treated as such. They can’t actually lock their objects down because they don’t actually completely follow the rules of immutability, and the biggest thing they do that you just can’t do with (real, proper) immutable structures is have anode.parentreference.So they have this kinda-immutable tech, but those guarantees only hold if all the code that ever holds a reference to the node is TS code. That is why all this other infrastructure that could stand to benefit from a shared standard format for frozen nodes can’t: it’s outside the walls of the TS fiefdom, so the nodes are meant to be used as immutable but any JS code (or any-typed code) the trees are ever exposed to would have the potential to ruin them by mutating the supposedly-immutable data
To be more specific about the
node.parentreference, if your tree is really truly immutable you need to replace a leaf node you must replace all the nodes on the direct path from the root to that leaf. TS does this, which is good.The bad part is that then all the nodes you didn’t replace have chains of
node.parentreferences that lead to the old root instead of the new one. Fixing this with immutable nodes would mean replacing every node in the tree, so the only alternative is to mutatenode.parent, which means that 1) you can’t actuallyObject.freeze(node)and 2) you don’t get all the wins of immutability since the old data structure is corrupted by the creation of the new one.See https://ericlippert.com/2012/06/08/red-green-trees/ for why Roslyn’s key innovation in incremental syntax trees was actually breaking the
node.parentreference by splitting into the red and green trees, or as I call them paths and nodes. Nodes are deeply immutable trees and have no parents. Paths are like an address in a particular tree, tracking a node and its parents.You are not joking, just the hack to make type checking itself parallel is well worth an entire hour!
Hm yeah it was a very good talk. My summary of the type checking part is
That is, the translation units aren’t completely independent. Type checking isn’t embarassingly parallel. But you can still parallelize it and still get enough speedup – he says ~3x from parallelism, and ~3x from Go’s better single core perf, which gives you ~10x overall.
What wasn’t said:
I guess this is just because, empirically, you don’t get more than 3x speedup.
That is interesting, but now I think it shows that TypeScript is not designed for parallel type checking. I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library. Though it’s hard to compare, since it also has to generate code
A separate thing I found kinda disappointing from the talk is that TypeScript is literally what the JavaScript code was. There was never a spec and will never be one. They have to do a line-for-line port.
There was somebody who made a lot of noise on the Github issue tracker about this, and it was basically closed “Won’t Fix” because “nobody who understands TypeScript well enough has enough time to work on a spec”. (Don’t have a link right now, but I saw it a few months ago)
Pretty sure he said it was an arbitrary choice and they’d explore changing it. The ~10x optimization they’ve gotten so far is enough by itself to keep the project moving. Further optimization is bound to happen later.
Work has been going on for years to parallelize rust’s frontend, but it apparently still has some issues, and so isn’t quite ready for prime time just yet, though it’s expected to be ready in the near term.
In my experience, once you start to do things “per core” and want to actually get performance out of it, you end up having to pay attention to caches, and get a bit into the weeds. Given just arbitrarily splitting up the work as part of the port has given a 10x speed increase, it’s likely they just didn’t feel like putting in the effort.
Can you share the timestamp to the discussion of this hack, for those who don’t have one hour?
I think this one: https://www.youtube.com/watch?v=10qowKUW82U&t=2522s
But check the chapters, they’re really split into good details. The video is interesting anyway, technically focused, no marketing spam. I can also highly recommend watching it.
Another point on “why Go and not C#” is that, he said, their current (typescript) compiler is highly functional, they use no classes at all. And Go is “just functions and data structures”, where C# has “a lot of classes”. Paraphrasing a little, but that’s roughly what he said.
They also posted a (slightly?) different response on GitHub: https://github.com/microsoft/typescript-go/discussions/411
Yes please!
This was interesting and nice to see - I realize that I’ve been thinking about what I like about each approach without being able to state it quite in these terms.
My thoughts are that structural type systems (like TypeScript) can work well - we don’t need to “represent everything with generic untyped dicts and lists” as a sibling comment states; we can jot down the shape upon construction and let the compiler ensure we didn’t make a typo somewhere else. The same type constraints might even help with compilation / execution speed.
When I pondered this a while ago, after first trying out Clojure, I came to a similar conclusion: you can do this with static typing but you need structural types, which OCaml and TypeScript have. However, as the author of this piece points out, one of the things that makes this work so well in Clojure is the way that all the relevant container types work with all the standard library functions:
Its not that you couldn’t make this work in other languages, its just much more natural and easier in Clojure, which makes it simple to experiment or react to changing requirements.
It also makes Lisp-style development with a REPL integrated into your editor possible, which is perhaps the real distinguishing feature of this style of development. The author hinted at this when they noted that the Clojure version didn’t bother to write a visualization function until the very end, whereas that was one of the first things they had to do in OCaml.
I’m not convinced that Clojure is special at all in this regard. Pretty much every language has a bunch of standard functionality around lists and maps/dicts. In the quote you include, the author says “Filter operates just as well over maps as it does lists […]”. So what? What languages don’t have the concept of filtering over a map/dict’s entries? Even in Clojure, it’s not like you can use the same predicate for filtering a dict as for filtering a list (the dict filter predicate has to work on a list argument of 2 elements, whereas the list filter predicate works on an argument of a single element of the list). So, that’s the same as Java, JavaScript, Kotlin, Swift, Rust, etc, etc.
I’ve used Clojure before. It’s probably the only non-static-typed language that I’ve actually enjoyed working with, and it’s clearly very well designed. But, the cynic in me feels like the Clojure community really wants to force “data-driven development” to be a thing and for that thing to be a special, unique, aspect of Clojure. I feel like this idea of Clojure having a standard library that works primarily over dicts and lists as being special is just a truism. It all does work pretty well in practice, but, IMO, there’s no such thing as “data-driven development”. It’s just functional programming, and I don’t feel like the article showcases anything except that developing in different languages is… different. (It’s still a well-written article and I enjoyed walking through the examples)
You really can’t include JavaScript on your list. There is no
filtermethod onMap, and even the name ofMap.sizediffers fromArray.lengthfor completely no reason.In all languages, there is a very real tension between handling map values or map key-value pairs - this is a good point.
Rust’s iterators are a nice “narrow waist” for these kinds of transformations though! Perhaps JavaScript’s upcoming iterator “helper” methods will improve the situation somewhat?
For JavaScript, I was indeed thinking of the “iterator helper methods”, which I thought were widely available, but it seems like maybe Safari doesn’t have stable/default support for them (?). https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Iterator#iterator_helper_methods
Rust’s Iterator trait is the biggest counter-example I was thinking of to the Clojurist rhetoric about map() and filter() and the implication that only Clojure or other dynamically typed languages can pull off such wonderfully flexible and extensible APIs. (I’m only slightly sorry for the sarcasm)
The truth is that there is nothing special about Clojure’s map() and filter() functions. If a data type is to work with them, then either the function definitions need to change to accommodate it, or the data type has to be written with the intention of working with map() and filter(). That’s literally the same thing in every programming language. Just because the interface required by map() and filter() is implicit doesn’t mean it doesn’t exist. In Rust, you’d have to make sure your data type impls the Iterator (or IntoIterator, etc) trait, just like in Clojure you’d have to implicitly make something iterable in whatever sense map() and filter() require.
This is again why I don’t buy that “data-driven development” is a new concept that needs a name. It’s just functional programming with a dynamically typed language.
There is something cool about working in Clojure, but you can’t really isolate it to a subset of specific features/choices. It’s one of those “the whole is greater than the sum of its parts” things. Instead of trying to coin cute terms, we can replace “data-driven development” with “programming in Clojure”.
Yes - I totally agree with all of those points!
It drives me bonkers how array/set/dictionary methods unnecessary differ in so many languages. Javascript’s
lengthvssizeis a really obvious example.I have a somewhat well-known package in Julia to make this stuff “just work” the same for dicts and sets as it does for arrays, somewhat inspired by Rich Hickey’s talks about clojure. (And another for dataframes with the same interface). To be fair to all the language and stdlib implementers out there, it takes a surprising amount of thought and work to make this work.
…is this a joke that went over my head, or are my standards just really low? Now I’m questioning how much clutter is “normal”.
No kidding! It reminds me of a meme from a clip of the Simpsons cartoon where one of the characters is shown to be living in a rundown apartment with no furniture or food and he says “Don’t tell anyone how I live.” I guess that’s gotta be me based on my desk space…
Credit to winkelmann on HN for calling this out first.
The acceptable use policy, bullet points replaced with a numbered list for discussion purposes
Since the only thing the acceptable use policy does is restrict services, and it explicitly applies to your use of Firefox, I conclude that Firefox must be contained in the word “service”. I also note that the terms include
Which would tend to reject any interpretation of the above callout of the acceptable use policy which has it only apply to certain services you use within it.
That is messed up. I feel strongly about ‘freedom zero’ of free software – that is The freedom to run the program as you wish, for any purpose. It seems we’re backsliding on the very fundamentals.
In that same vein, I’m absolutely baffled and exasperated by software licenses feeling the need to try and make me agree to not use their software to break the law.
First of all, I’m with you: software freedom zero is that I can use the program however I wish. Fuck off with telling me how I may or may not use the program.
Second: Breaking the law is already… against the law. Who thinks that someone would fire up a piece of software with the intent of breaking the law, but then decide against it because of a license agreement? I’m willing to break the law and have actual consequences, but I’m not willing to break a license agreement where the worst case is that the vendor might somehow disable access to their stupid program? I’m so tired of living in a world that is so unserious.
Third: I know this is a “cover your ass” thing, but that’s not good enough for me. Here in the U.S., people shoot and kill each other, yet gun manufacturers aren’t on the hook for the crimes. And I’m pretty sure you don’t have to sign a license agreement to purchase a gun or ammunition wherein you promise not to break the law with it. Alcohol companies aren’t liable for someone getting drunk and killing someone with their car. Neither is the car manufacturer. The idea that a fucking web browser feels the need to “protect itself” from being liable when a user breaks the law is a joke.
I feel like I’m taking crazy pills.
Separating out discussion of the exact terms into a self reply since it feels like a comment that should be voted on separately.
This is a shockingly restrictive policy. For some examples:
Bullet point 8 prohibits using firefox to sell, purchase, or advertise controlled products or services. You cannot use firefox to legally acquire life saving medication. Entirely law abiding drug manufacturers cannot use firefox to sell or advertise their products.
Bullet point 9 forbids graphic depictions of violence in all manners - uploading, downloading, transmiting, displaying, or granting access. You can’t use firefox for making, publishing, or even just consuming journalism on police brutality, war, etc.
Bullet point’s 9 restriction on graphic violence and sexuality also forbids using firefox to watch most PG or higher movies, browse many social media sites, etc.
Bullet point 2 forbids sending unsolicited communications, you can’t do cold outreach to people using firefox.
Bullet point 2 forbids intercepting or monitoring communications not intended for you, you can’t do security research using firefox, something as simple as investigating how your bed is spying on you would require intercepting communications not intended for you.
Bullet point 4 prohibits deception. You can’t play many common games using firefox.
Bullet point 10 is a super-charged version of the GDPR. If you collect personal information without consent, even if you have a legitimate use for it that would satisfy every privacy law in the world, you may not use firefox to do that.
Every bullet point here except 1, 5, 12, and 13 is basically Mozilla trying to step in and define their own version of what is acceptable to say, do, and promote that is highly restrictive, offensive to the principles of free speech, and contains none of the nuance that even the most authoritarian dictatorships would recognize is necessary in regulating conduct.
I take no significant issue with bullet point 1 because it seems unlikely that breaking this contract is in any way worse than breaking the actual law, and bullet points 5, 12, and 13 because they appear to be strict subsets of what is already forbidden under number 1.
It does contain some jurisdiction smuggling, even if it is unlikely to be a practical issue: if «applicable law» is on the books, but is either in the process of being inevitably struck down in court, or is explicitly condemned by your current place of residence, are you still violating the contract?
Update: They’ve removed the portion of the terms that incorporated this acceptable use policy :)
Though it’s frustrating to see them pretend this is because people misunderstood the terms and not admit it’s because they messed up the drafting of them.
I remember John Carmack describing in one of his Doom 3 talks how he was shocked to discover that he made a mistake in the game loop that caused one needless frame of input latency. To his great relief, he discovered it just in time to fix it before the game shipped. He cares about every single millisecond. Meanwhile, the display server and compositing window manager introduce latency left and right. It’s painful to see how the computing world is devolving in many areas, particularly in usability and performance.
I will say the unpopular-but-true thing here: Carmack probably was wrong to do that, and you would be just as wrong to adopt that philosophy today. The bookkeeping counting-bytes-and-cycles side of programming is, in the truest Brooksian sense, accidental complexity which we ought to try to vanquish in order to better attack the essential complexity of the problems we work on.
There are still, occasionally, times and places when being a Scrooge, sitting in his counting-house and begrudging every last ha’penny of expenditure, is forced on a programmer, but they are not as common as is commonly thought. Even in game programming – always brought up as the last bastion of Performance-Carers who Care About Performance™ – the overwhelming majority very obviously don’t actually care about performance the way Carmack or Muratori do, and don’t have to care and haven’t had to for years. “Yeah, but will it run Crysis?” reached meme status nearly 20 years ago!
The point of advances in hardware has not been to cause us to become ever more Scrooge-like, but to free us from having to be Scrooges in the first place. Much as Scrooge himself became a kindly and generous man after the visitation of the spirits, we too can become kinder and have more generous performance budgets after being visited by even moderately modern hardware,
(and the examples of old software so often held up as paragons of Caring About Performance are basically just survivorship bias anyway – the average piece of software always had average performance for and in its era, and we forget how many mediocre stuff was out there while holding up only one or two extreme outliers which were in no way representative of programming practice at the time of their creation)
There is certainly a version of performance optimization where the juice is not worth the squeeze, but is there any indication that Carmack’s approach fell into that category? The given example of “a mistake in the game loop that caused one needless frame of input latency” seems like a bug that definitely should have been fixed.
I’m having a hard time following your reasons for saying Carmack was “wrong” to care so much about performance. Is there some way in which the world would be better if he didn’t? Are you saying he should have cared about something else more?
16ms of input latency is enormous for a fast-faced mouse driven game; definitely something the player can notice.
There are different kinds of complexity. Everything in engineering is about compromises. If you decide to trade some latency for some other benefit, that’s fine. If you introduce latency because you weren’t modelling it in your trade-off space, that’s quite another.
the amount of people complaining about game performance in literally any game forum, steam reviews / comments / whatnot obviously shows that wrong. Businesses don’t care about performance but actual humans being do care ; the problem is the constantly increasing disconnect between business and people.
Minecraft – the best-selling video game of all time – is known for both its horrid performance and for being almost universally beloved by players.
The idea that “business” is somehow forcing this onto people (especially when Minecraft started out and initially exploded in popularity as an indie game with even worse performance than it has today) is just not supported by empirical reality, sorry.
But the success is despite the game’s terrible performance, not thanks to it. Or do you think if you asked people if they would prefer minecraft to be faster they would say no ? If it was not a problem then a mod that does a marginal performance improvement certainly would not have 10M downloads: https://modrinth.com/mod/moreculling . So people definitely do care ; they just don’t have a choice because if you want to play “minecraft” with your friends this is your only option. Just like for instance Slack, Gitlab or Jira are absolutely terrible but you don’t have a choice to use it because that’s where your coworkers are.
I don’t know of any game that succeeded because of their great performance, but I know of plenty that have succeeded despite their horrible performance. While performance can improve player satisfaction, for games, it’s a secondary measure of success, and it’s foolish to focus on it without having the rest of the game being good to play. It’s the case for most other software as well - most of the time, it’s “do the job well, in a convenient to use way, and preferably fast”. There’s fairly few problems where the main factor for software solving it is their speed first.
… every competitive shooter ? you think counter-strike would have succeeded if it had the performance of, say, neverwinter nights 2 ?
Bad performance can kill a decent game. Good performance cannot bring success to an otherwise mediocre game. If it worked that way, my simple games that run at ~1000FPS would have taken over the world already.
Even if a game was written by an entire army of Carmacks and Muratoris squeezing every last bit of performance they could get, people would almost certainly answer “yes” to “would you prefer it to be faster”. It’s a meaningless question, because nobody says no to it even when the performance is already very good.
And the fact that Minecraft succeeded as an indie game based on people loving its gameplay even though it had terrible performance really and truly does put the lie to the notion that game dev is somehow a unique performance-carer industry or that people who play games are somehow super uniquely sensitive to performance. Gamers routinely accept things that are way worse than the sins of your least favorite Electron app or React SPA.
I think a more generous interpretation of the hypothetical would be to phrase the question as: “Do you think the performance of Minecraft is a problem?”
In that scenario, I would imagine that even people who love the game would likely say yes. At the same time, if you asked that question about some Carmack-ified game, you might get mostly “no” responses.
how is accepting things an argument for anything ? we are better than this as a species
Can you clarify the claim that you are making, and why the chosen example has any bearing on it? Obviously gaming is different from other industries in some ways and the same in other ways.
I think the Scrooge analogy only works in some cases. Scrooge was free to become more generous because he was dealing with his own money. In the same way, when writing programs that run on our own servers, we should feel free to trade efficiency for other things if we wish. But when writing programs that run on our users’ machines, the resources, whether RAM or battery life, aren’t ours to take, so we should be as sparing with them as possible while still doing what we need to do.
Unfortunately, that last phrase, “while still doing what we need to do”, is doing a lot of work there. I have myself shipped a desktop app that uses Electron, because there was a need to get it out quickly, both to make money for my (small, bootstrapped) company and to solve a problem which no other product has solved. But I’ve still put in some small efforts here and there to make the app frugal for an Electron app, while not nearly as frugal as it would be if it were fully native.
I used to be passionate about this too, but I really think villianizing accidental complexity is a false idol. Accidental complexity is the domain of the programmer. We will always have to translate some idealized functionality into a physically executable system. And that system should be fast. And that will always mean reorganizing the data structures and algorithms to be more performant.
My point of view today is that implementation details should be completely embraced, and we should build software that takes advantage of its environment to the fullest. The best way to do this while also retaining the essential complexity of the domain is by completely separating specification from implementation. I believe we should be writing executable specifications and using them in model-based tests on the real implementation. The specifications disregard implementation details, making them much smaller and more comprehensible.
I have working examples of doing this if this sounds interesting, or even farfetched.
I agree with this view. I used to be enamored by the ideas of Domain Driven Design (referring to the code implementation aspects here and not the human aspects) and Clean/Hexagonal Architecture and whatever other similar design philosophies where the shape of your actual code is supposed to mirror the shape of the domain concepts.
One of the easiest ways to break that spell is to try to work on a system with a SQL database where there are a lot of tables with a lot of relations, where ACID matters (e.g., you actually understand and leverage your transaction isolation settings), and where performance matters (e.g., many records, can’t just SELECT * from every JOINed table, etc).
I don’t know where I first heard the term, but I really like to refer to “mechanical sympathy”. Don’t write code that exactly mirrors your business logic; your job as a programmer is to translate the business logic into machine instructions, not to translate business logic into business logic. So, write instructions that will run well on the machine.
Everything is a tradeoff. For example, in C++, when you create a vector and grow it, it is automatically zeroed. You could improve performance by using a plain array that you allocate yourself. I usually forgo this optimization because it costs time and often makes the code more unpleasant to work with. I also don’t go and optimize the assembly by hand, unless there is no other way to achieve what I want. With that being said, performance is a killer feature and lack of performance can kill a product. We absolutely need developers who are more educated in performance matters. Performance problems don’t just cripple our own industry, they cripple the whole world which relies on software. I think the mindset you described here is defeatist and, if it proliferates, will lead to worse software.
This one isn’t actually clear cut. Most modern CPUs do store allocate in L1. If you write an entire L1 line in the window of the store buffer, it will materialise the line in L1 without fetching from memory or a remote cache (just sending out some broadcast invalidates if the line is in someone else’s cache). If you zero, this will, definitely happen. If you don’t and initialise piecemeal, you may hit the same optimisation, but you may end up pulling in data from memory and then overwriting it.
If the array is big and you do this, you may find that it’s triggering some page faults eagerly to allocate the underlying storage. If you were going to use only a small amount of the total space, this will increase memory usage and hurt your cache. If you use all of it, then the kernel may see that you’ve rapidly faulted on two adjacent pages and handle a bit more in the page faults eagerly handler. This pre-faulting may also move page faults off some later path and reduce jitter.
Both approaches will be faster in some settings.
Ah, you must be one of those “Performance-Carers who Care About Performance™” ;)
This is so often the case, and it always worries me that attitudes like the GP lead to people not even knowing about how to properly benchmark and performance analyse anymore. Not too long ago I showed somebody who was an L4 SWE-SRE at Google a flamegraph - and he had never seen one before!
Sometimes, and that’s the important bit. Performance is one of the things that I can optimise for, sometimes it’s not the right thing. I recently wrote a document processing framework for my next book. It runs all of its passes in Lua. It simplifies memory management by doing a load of copies of
std::string. For a 200+ page book, well under one second of execution time is spent in all of that code, the vast majority is spent in libclang parsing all of the C++ examples and building semantic markup from them. The code is optimised for me to be able to easily add lowerings from new kinds of semantic markup to semantic HTML or typeset PDF, not for performance.Similarly, a lot of what I work on now is an embedded platform. Microcontrollers are insanely fast relative to memory sizes these days. The computers I learned to program on had a bit less memory, but CPUs that were two orders of magnitude slower. So the main thing I care about is code and data size. If an O(n) algorithm is smaller than an O(log(n)) one, I may still prefer it because I know n is probably 1, and never more than 8 in a lot of cases.
But when I do want to optimise for performance, I want to understand why things are slow and how to fix it. I learned this lesson as a PhD student, where my PhD supervisor gave me some code that avoided passing things in parameters down deep function calls and stored them in globals instead. On the old machine he’d written it for, that was a speedup. Parameters were all passed on the stack and globals were fast to access (no PIC, load a global was just load from a hard-coded address). On the newer machines, it meant things had to go via a slower sequence for PC-relative loads and the accesses to globals impeded SSA construction and so inhibited a load of optimisation. Passing the state down as parameters kept it in registers and enabled local reasoning in the compiler. Undoing his optimisation gave me a 20% speedup. Introducing his optimisation gave him a greater speedup on the hardware that he originally used.
I know how to and I teach it to people I work with. Just recently at work I rebuilt a major service, cut the DB queries it was doing by a factor of about 4 in the process, and it went from multi-second to single-digit-millisecond p95 response times.
But I also don’t pull constant all-nighters worrying that there might be some tiny bit of performance I left on the table, or switching from “slow” to “faster” programming languages, or really any of the stuff people always allege I ought to be doing if I really “care about performance”. I approach a project with a reasonable baseline performance budget, and if I’m within that then I leave it alone and move on to the next thing. I’m not going to wake up in a cold sweat wondering if maybe I could have shaved another picosecond somewhere.
And the fact that you can’t really respond to or engage with criticism of hyper-obsession with performance (or, you can but only through sneering strawmen) isn’t really helpful, y’know?
How were we supposed to know that you were criticizing “hyper-obsession” that leads to all-nighters, worry, and loss of sleep over shaving off picoseconds? From your other post it sounded like you were criticizing Carmack’s approach, and I haven’t seen any indication that it corresponds to the “hyper-obsession” you describe.
Where’s the strawman really?
I did a consulting gig a few years ago where just switching from zeroing with std::vector to pre-zeroed with calloc was a double-digit % improvement on Linux.
I think answer is somewhere in the middle: should game programmers in general care? Maybe too broad of a statement. Does ID Software, producers of top-of-the-class, extremely fast shooters benefit from someone who cares so deeply to make sure their games are super snappy? Probably yes.
You think thats bad, consider the advent of “web-apps” for everything.
On anything other than an M-series Apple computer they feel sluggish, even with absurd computer specifications. The largest improvement I felt going from a i9-9900K to an M1 was that Slack suddenly felt like a native app, going back to my old PC felt like going back to the 90’s.
I would love to dig into why.
The bit that was really shocking to me was how ctrl-1 and ctrl-2 (switching Slack workspaces) took around a second on a powerful AMD laptop on Linux.
At work we use Matrix/Element. It has its share of issues but the performance isn’t nearly as bad.
I don’t really see how switching tabs inside a program is really related to the DRM subsystem, or to Kernel Mode Setting.
I thought they were mentioning ctrl-alt-F1/F2 switching (virtual terminals), which used to be indeed slow.
My bad.
There is a wide spectrum of performance in Electron apps. Although it’s mostly VS Code versus everyone else. VS Code is not particularly snappy, but it’s decent. Discord also feels faster than other messengers. The rest of highly interactive webapps are used are unbearably sluggish.
So I think these measured 6ms are irrelevant. I’m on Wayland Gnome and everything feels snappy except highly interactive webapps. Even my 10-year-old laptop felt great, but I retired it because some webapps were too painful (while compiling Rust felt… OK? Also browsing non-JS content sites was great).
Heck, my favorite comparison is to run Q2 on WASM. How can that feel so much snappier than a chat application like Slack?
I got so accustomed to the latency, when I use something with nearly zero latency (e.g. an 80’s computer with CRT), I get the surreal impression that the character appeared before I pressed the button.
I had the same feeling recently with a commadore64.
It really was striking how a computer with less power than the microcontroller in my keyboard could feel so fast, but obviously when you actually give it an instruction to think about, the limitations of the computer are clear.
EDIT: Oh hey, I wasn’t kidding.
The CPU in my keyboard is 16MHz: ControllerBoard Microcontroller PDF Datasheet
The CPU in the commadore64 I was using was 0.9-1MHz: https://en.wikipedia.org/wiki/MOS_Technology_6510
As a user on smaller platforms without native apps, I will gladly take a web app or PWA over no access.
In the ’90s almost everything was running Microsoft Windows with on x86 for personal computers with almost everyone running at the 5 different screen resolunions so it was more reasonable to make a singular app for a singular CPU architecture & call it a day. Also security was an afterthought. To support all of these newer platforms, architectures, device types, & have the code in a sandbox, going the HTML + CSS + JavaScript route is a tradeoff many are willing to take for portability since browsers are ubiquitous. The weird thing is that a web app doesn’t have to be slow, & not every application has the same demands to warrant a native release.
Having been around the BSD, and the Linux block 20+ years ago, I share the sentiment. Quirky and/or slow apps are annoying, but still more efficient than no apps.
Besides, as far as UIs go, “native” is just… a moderately useful description at this point. macOS is the only one that’s sort of there, but that wasn’t always the case in all this time, either (remember when it shipped with two toolkits and three themes?). Windows has like three generations of UI toolkits, and one of the two on which the *nix world has mostly converged is frequently used along with things like Kirigami, making it native in the sense that it all eventually goes through some low-level Qt drawing code and color schemes kind of work, but that’s about it.
Don’t get me wrong, I definitely prefer a unified “native” experience; even several native options were tolerable, like back when you could tell a Windows 3.x-era application from other Windows 98 applications because the Open file… dialog looked different and whatnot, but keybindings were generally the same, widgets mostly worked the same etc.
But that’s a lost cause, this is not how applications are developed anymore – both because developers have lost interest in it and because most platform developers (in the wide sense – e.g. Microsoft) have lost interest in it. A rich, native framework is one of the most complex types of software to maintain, with some of the highest validation and maintenance costs. Building one, only to find almost everyone avoids it due to portability or vendor lock-in concerns unless they literally don’t have a choice, and that even then they try to use as little of it as humanly possible, is not a very good use of already scant resources in an age where most of the profit is in mobile and services, not desktop.
You can focus on the bad and point out that the vast majority of Electron applications out there are slow, inconsistent, and their UIs suck. Which is true, but you can also focus on the good and point out that the corpus of Electron applications we have now is a lot wider and more capable than their Xaw/Motif/Wx/Xforms/GTK/Qt/a million others – such consistency, much wow! – equivalents from 25 years ago, whose UIs also sucked.
So long as Firefox views its purpose as “building an alternative infrastructure for the advertising industry,” it cannot ever be superior [1].
[1] https://blog.mozilla.org/en/mozilla/improving-online-advertising/
Your choices when it comes to the internet are:
There is no secret third thing. Please indicate your preference from the above choices.
The Internet is already Thing 1 and Thing 2. Take a dose of revolutionary optimism and think just a little bit bigger than capitalist realism.
Perhaps one day in the future we will all live in a post-scarcity utopia where effectively-unlimited compute power and bandwidth are free of charge to all.
Today, we do not live in that world. Today, compute power and bandwidth cost money. So, either you have an internet where every person’s presence must be paid for by that person out of their own pocket and those who can’t afford that are barred from it, or you have an internet where some people can have a presence without paying. Thus far the only we we’ve figured out of actually pulling off the latter at sufficient scale to let the world participate is via advertising; there are not enough donors with deep enough pockets to do it via charity.
Once again: please indicate your preference.
Are you sure that’s the only way? Can you genuinely think of no routes to an alternative?
If you have another way which already exists and already works at sufficient scale, right here, right now, today, feel free to point to it.
Otherwise, please indicate your preference from the options available today.
You and I are already on it. It’s the Fediverse. Feel free to complain about “sufficient scale” though.
We cannot change the world by conforming to its every constraint. Instead, we build the alternatives of tomorrow, today.
Fediverse is for hobbyists, it does not address the monetization issue.
From the perspective of the content provider / publisher: a publishing / tooling web business (even if small) has as possible income sources mainly the options provided above (paid accounts and ads). Nobody figured out another better way to build a business on the web yet, which I think is @ubernostrum’s point.
Even just a non-business personal web presence is a problem. Either you pay for your web presence or someone else pays for it, and the only successful way we’ve found to do “someone else pays for it” is to have the someone else be an advertiser who gets ads on the page in exchange for their money. And even then it hasn’t always been enough – Geocities, which so many people nostalgically yearn for, had ads and still struggled financially.
The only “successful” way? Well it wasn’t successful in the case of Geocities was it? Neocities on the other hand still exists.
Remains to be seen if it can actually hold up. The cost in both money and time to run instances is non-trivial, and it’s a cliché to point out instances whose admins burn out/flame out.
Also, yeah, I will complain about the “sufficient scale”. Big, huge, gigantic instances tend to range from ten thousand or so up to perhaps a couple hundred thousand active accounts. Even if you have a bunch of instances in that size group, you’re looking at roughly the population of a good-sized city. We live in a world of billions of people.
Obviously a larger Fediverse user base would come with a larger donor base. What makes you so sure the increased revenue would not keep up with with the increased costs?
Wikipedia, which I mentioned the last time we talked about this.
I would still be curious to understand what you meant in your reply to the linked comment, but perhaps you changed your position. You haven’t completely abandoned it though so there should still be plenty to discuss!
The demand to “indicate your preference” rings hollow when the Internet is already both things, as @aspensmonster pointed out. There is no realistic proposal to avoid either one; it will always be a spectrum.
I would like to ask once again if you really think the web would be poorer if websites were less like Facebook and Twitter and more like Wikipedia and Lobsters?
My home internet connection is right now $83 / month. (It used to be $15 :( but the cable company keeps jacking it up… even though it delivers very little more practical benefit to me than it used to…)
That internet connection would be absolutely worthless if there was nobody on the other side to connect to. Maybe some fraction of that $83 / month could go to the people who make the internet valuable as well as the people who make the internet possible? I’m not saying the cable company is worthless, just that maybe we could find some way to spread the money without advertising middle men. (The money to pay for ads also ultimately comes from the users who buy the products…)
Does this make poor people second class or forbidden? Well, if I didn’t pay my cable bill, they wouldn’t let me connect anymore sooooooooo already reality? And a lot of the best on line content is behind additional paywalls now too - even with the advertising. So eh, maybe this isn’t an ideal situation but it at least doesn’t seem much worse than it is now.
I won’t fall for the false dichotomy. The web was not as hostile in the 90s and 00s. Yes, there were flashing ad banners in places, but it was, overall, much less invasive (spying/tracking).
Communities grew out of passions and common interests. Not every single fucking thing needed to be a profitable endeavor. (Look at Reddit in the early days)
I feel like we’ve become more and more like the Ferengi species from Star Trek.
The Ferengi were always a metaphor for western capitalist societies in general, and the US in particular. Check out the Lower Decks episode where Boimler gets trapped in an hotel on Ferenginar for a weekend because he has no resistance to the addictive nature of Ferengi daytime TV.
The problem is that such articles have to claim superiority by denigrating another paradigm with examples and code that shows poor practice.
People like FP because it is well suited to the transformation of data, and for many programmers that is all there is. Try modeling a complex payroll system into code using such a transformation of data approach. Enterprises tried that for over a decade during the “structured analysis” era, but as “engineers” we all know that don’t we?
I have no idea what you are alluding to but I am honestly very intrigued. As someone who likes functional programming and complex enterprise systems I would definitely like to hear your thoughts/experiences regarding why FP might not be a good match for enterprise software development. (I do know about structured design, if you’re referring to the Yourdon/Constantine sort of thing. But not sure how it’s connected here.)
As for the reasons why people like FP, I agree transformation of data is one factor, but I would say it’s “pick 3-5 out of:”
Most of those can be found in non-FP languages, but when you have a critical number of them come together in a cohesive way, that certain magic starts to happen. There is no single definition that fits all languages that are commonly categorized under the FP label. If I were to pick one, I would say “high-order functions as the default and most convenient way of doing everything”.
Indeed. You can do FP in most languages, and you can do OOP in most languages (even C!), but if the language is designed and optimized for certain styles/patterns of programming, then you’re usually giving up both convenience/ergonomics and performance by going against the grain. Case in point: JavaScript devs trying to lean into FP patterns and making copious, unnecessary, copies of local arrays and objects just for the “immutable-only cred” or writing a “curried” functions when that means actually creating tons of temporary Function objects that need to be garbage collected at runtime (as opposed to languages like OCaml that are able to optimize these things so well that curried functions mostly don’t have any runtime overhead compared to uncurried functions).
But, I will say this: I really don’t love FP (as in the “use mostly pure functions to just transform data” definition) for IO-heavy applications (really, almost anything with a DBMS underneath). Doing “proper” FP style is often the antithesis of “mechanical sympathy”, meaning we have to choose between writing elegant data transformation code and writing code with reasonable performance characteristics.
For example, it’s extremely common when working with a SQL database to open a transaction, read some data, make a decision based on that data, and then conditionally read or write more data. To implement this in conventional FP style, you have basically two options:
Break up this unit of logic into multiple pure functions and have your top-level handler piece them together like IO -> pure function 1 -> IO -> pure function 2 -> IO. That’s fine if your pure functions actually make sense to be separate named operations. Otherwise, you’re breaking something that is semantically a single business operation into multiple pieces that don’t actually mean anything in your domain.
Read ALL of the data you could possibly need upfront and then feed it into the pure function before writing the final result. This is obviously a bad idea in any system that is supposed to be able to scale. Doing extra database queries when you might not even use the results should send you straight to jail.
Why not
matchthe result and handle the error even if it’s just logging it somewhere and terminating withstd::process::exitor something.Because there just isn’t much point to doing that for a web server application. This isn’t a typical, interactive, user application where you’d want to present an error message to whomever issued the command. This is more likely to be something that’s just supposed to run automatically at boot or something, so just blowing up and showing a stack trace with exactly whatever line number failed pretty much is the friendly error message for the “user” (the guy managing the server).
IMO, of course. But, this is exactly the thought process I use with my Rust web service that is running in a docker container. Failing to start is failing to start and I’m not going to waste time to hide information from myself by writing a more “product-y” error message or whatever.
Lots of great reasons for reporting errors like that, especially if you have a bunch of copies of that application running in a bunch of different places. Either in-house logging or something like Sentry are fantastic error sinks. Basically, I’d want to know when the application is not able to bind to a port and something like systemd is trying to restart it, causing a boot loop. That is a problem I’d want to know about.
Fair enough. And those specifics will depend on your infrastructure and deployment process. If your code fails to deploy for some reason even before your app starts running, you’re not going to get your Sentry errors, anyway (or if it just hard crashes, AFAIK). What if your Sentry logger doesn’t initialize properly, or the network is misconfigured, etc? So, you still need something that’s going to report bad health that’s outside of your application’s code, IMO. If that’s the case, then I still don’t see much value in writing and logging a pretty error message when your app can’t even finish starting up. Let it crash or bootloop or whatever, and the external health checks should alert us to something going wrong just as well as a remote logging service.
I’ve never used any BSDs in earnest (not counting macOS because I don’t want to count it…). I’ve played with a few in virtual machines and read documentation and forums for several of them back in my distro-hopping days.
But, any progress at all in running free software operating systems on our machines gets a huge cheer from me. I fear that the future is only going to get more and more locked down to the point that we won’t even be allowed to install “unapproved” software on our hardware, or that we won’t be able to disable various tracking and spying mechanisms on our machines. But, I continue to hope that was can avoid that future, and I’ll continue to fight it every step of the way. The only computer in my house that runs a non-Linux OS is my son’s school computer that, unfortunately, requires Windows to run certain things. But, I have to work constantly to disabled all of the analytics bullshit and there’s no linked Microsoft account on it despite them making it harder and harder to use it that way. How about you stop trying to spy on children you evil pieces of… well, never mind… /rant
I’ve always been intrigued by the hexagonal architecture, probably because it’s similar to the „functional core, imperative shell“ idea. But by now I’ve become wary of the abstractions this post and others use to implement it. All the complexity that comes with implementing e.g. a repository trait doesn’t seem worth it to me, especially when a database imposes so many implicit assumptions on the whole codebase that switching it would require a lot more than just adding a trait implementation.
I wonder if there’s a hexagonal „layout“ that’s more focused on just modules and structural conventions to get some of the benefits with less overhead.
My team implemented a hexagonal architecture for a certain project, but development was very slow. I argued basically that for ordinary business applications, we’re not switching from Postgres + SQL Alchemy to anything else anytime soon, and that if the product owners want faster development this is the low-hanging fruit. The proliferation of repositories and repository hiding in general caused me to conclude that, at least for the persistence element, it isn’t worth the potential benefit.
Architecture is good, IMO, if it forces you to think hard about your priorities, but I think hexagonal architecture in particular looks like it solves all your problems, but you have to bear in mind that like every other architecture, it comes with tradeoffs, and the con with hexagonal is that it tends to make the entire application very abstract and slow to build.
I can’t say I disagree with that, it’s not the first time I’ve read an essay on hexagonal architecture and come away with the conclusion that it’s something you refactor your application into if and when you reach the need for the flexibility and substitutability.
To that, this specific essay has the significant ick factor of calling a perfectly serviceable and maintainable application “very bad application”, then proceeding to convert 260 lines of rust in three simple files (including a simple and completely linear
mainsetup) into 750 lines spread over nearly 20 files.And for all its harping on testability, it has one test. Which needs 80 lines to test that a two line handler function calls into the (mocked) handler service and forward its result (which is a success because that’s the only thing it tests). It takes about 3 lines to create an in-memory sqlite, wrap that in the original application’s
AppState, and start calling the same handler and testing its output.While it’s of course silly to build a 260 line application like this, I have experienced at least some of the concepts translating well into anything bigger.
It’s nice to look at a module/folder of code and know “hey this is for saving/getting things to/from the DB” and this other code is just about wrapping a remote API. That other code is just for taking things from/to HTTP or whatever and passing it somewhere that doesn’t need to know.
I’m not sure if this is very hexagonal, but the most workable codebases I’ve got to work in where things where you just generated the API layer with something like OpenAPI or GRPC, and the DB interactions with something like https://sqlc.dev/. Then you just focus on the actual logic of the task at hand between those.
Meanwhile, some of the worst codebases I’ve had to work in were places where side effects could happen at any moment in a 5 layer deep call chain with db transactions and remote IO seemingly happening in every helper function.
Maybe the important essence really is more “functional core, imperative shell” once things get big.
One day I should write this up, but a lot of what I do at work currently is web services built in Litestar, using svcs as a backing “service locator” implementation and Litestar’s own SQLAlchemy repository implementation (which they also package for independent use as
advanced-alchemy). I do this not because I believe in the need to maintain flexible “hexagonal” or whatever architecture, but because it hits the right balance of architectural patterns and practical payoffs for me.Basically:
In a Django-specific context I’ve made similar arguments against introducing “service layer” abstractions, largely because A) for web applications, at least, I don’t fully believe in trying to produce isolated pure “unit” tests that can run independently of a database and B) you’re never going to meaningfully swap out how you do persistence without significant rewriting anyway. TO quote myself there:
At the developer scale I pretty much agree with you completely, although I haven’t used the tools you mention (thanks for sharing links!). I made a similar argument here, along the lines of, it seems unlikely to me that on the timescale of 10-20 years, SQL Alchemy and PostgreSQL are likely to still be obtaining active support, because they’ve been around for ~20 and ~40 years respectively, and they are the dominant paradigms in their niche. Preemptively building abstraction to support the somewhat far-fetched scenario that they fall out of popularity before this project gets retired seems like a waste of effort. Moreover, sticking entirely to the common SQL subset of what you can do with SQL Alchemy is also a bit of a waste. There isn’t going to be a cost-based motive for switching to something besides Postgres (unless what, they give you money to use it?) and a feature-based motive would give the lie to using that common SQL subset anyway. So it’s a way of throwing money away, basically. Use the tools that expedite development.
At the architectural scale, it isn’t a foregone conclusion. If I were building something to last 100 years or longer, I would start to wonder about things like that. Cost or speed of development may not be the most important thing. It again comes down to the prioritization of quality attributes (performance, reliability, flexibility, etc. etc.) and the determination of architecturally significant requirements. Within your daily life and mine, these scenarios may be far-fetched, but for other people maybe they aren’t and the whole kit & caboodle of hexagonal architecture may be the best solution. Although it seems unlikely to me that hexagonal architecture should be used as often as it gets talked about.
Then again, I kind of flunked out of architecture, so my opinions may not be very valuable.
I dove into one of our hexagonal services yesterday to see if we could replace DynamoDB with Postgres and in theory I’d “just” have to replace the repository implementation but in reality it’s just an endless jumping around between files and the way they implemented half the AWS API was bleeding into the business layer anyway.
I’m unimpressed just as I am with say VIPER on mobile. Maybe instead get good and program the thing in a straight line.
The core idea of hexagonal architecture is that your business logic is not coupled to implementation choices made. But if you’re okay giving up some of the benefits, you don’t need to abstract things.
I like starting with services and using integration tests with devcontainers, for instance, instead of a repositories/wrappers around 3rd party APIs. Services get me enough of a benefit to use them from the start and think more about what my code is doing, rather than gunking them up with HTTP details.
I used to strive for doing things similar to hexagonal architecture and/or domain driven design (DDD) and/or clean architecture.
The weak point of all of these techniques and abstractions is data persistence if the data store is a SQL database. There is a very large space of applications that live in between “TODO List App For My Blog” and “eventual consistency with multiple CQRS data stores, caching layers, background syncing, etc”.
At the “TODO List” end of the spectrum, your domain models pretty much map 1-1 with your database tables, or might have some one-to-many, parent-children, relations. These trivial demo apps almost never need transactions (and certainly never seem to acknowledge or handle database constraints) or row locks, etc.
On the other end, you also don’t typically utilize the SQL database’s transaction and locking functionality, but for different reasons.
But, in the middle range, all of these “Repository” abstractions are really inefficient and/or leaky and/or just plain incapable of handling concepts like database transactions.
Sometimes the way this is handled is that you write your Repository in a way that pretends it has no concept of transactions and then you begin a transaction at some higher level (i.e., in a Service class or an HTTP handler, etc). But that is a (very) leaky abstraction! If the whole point of a Repository was to abstract away the persistence details, then how or why should my higher level code know that I need to start a database transaction?
Others will say that the above approach is improper (especially in DDD). They’ll say that the Repository should handle any transactions and that if you want atomicity between two Repositories, that just means you need to define a new domain entity and write a new Repository for that entity. But, I still don’t ever see anyone writing Repository classes that would allow us to accomplish something like a
SELECT ... FOR UPDATEfollowed by anUPDATE ...and/orDELETE ...inside a transaction. Rather, the best they seem able to do is have each individual method run inside a transaction.And that’s before even getting to Rust, specifically, which adds even a little more friction to using this kind of pattern. Boxing up a bunch of trait objects to be passed around or held by other trait objects, etc, is pretty ugly and not ergonomic.
I enjoyed this post and think it’s a great walkthrough of one way to factor your code and why/when that’s helpful!
I agree with your comment about the repository trait. I think it’s usually good to avoid making traits which only have a single implementor – having an
struct AuthorRepository { sqlite: sqlx::SqlitePool }is just as abstracting from the perspective of someone using the AuthorRepository without needing to make a separatetrait AuthorRepositoryand then astruct Sqlitewhich implements it. Traits are great but you can still run into limitations (eg around async) relative to just using a struct. (If the trait exist just to support mocking in tests, you can do things like mockall::doubleI’ve been using a hexagonal-ish layout for Go for the last few years, I like what it has evolved to. There’s a reason for the “-ish”, I don’t really care about hot-swapping implementations except for a few specifics (vector DB, file storage/S3, email sender) and found a lot of abstractions useless. What I settled on was flexibility and sticking as close to idiomatic Go as possible, same would go for Rust, when in Rome and all that.
Ultimately, the “rules” of architectures (be it 3-layer, hexagonal or whatever’s trending) is that they are best treated as a rough map to navigate growing complexity and every codebase/product/team will have different needs, ideas and tradeoffs. Sometimes it’s fine to just call a repo from a HTTP handler, the rules say that’s bad but you gotta spend time wisely. The rules say everything should be an interface/trait, but if you know you’re never going to suddenly switch database one day (usually very unlikely) it’s not worth the hassle.
cue the pirates of the carribean “they’re more guidelines than actual rules” scene
This is what I try to do. My shell has two roles
I do not make unnecessary abstractions or create traits only for testing. If I need to test a route that saves data to postgres, then I have postgres running and wired up to the app. I use random values for all test data so I can run tests in parallel.
If a test depends on some other data, then I set up that data (again with random values) before running the test. A nice byproduct of this pattern that you slowly create an internal SDK that calls your API which forces you to act as the “client” during testing and actually use your API.
I’m not sure about perfect since you fail to mention the whole
anyerror(and the other crate whose name I forget here) sadness. If it were perfect, then there wouldn’t be a need for a crate.I would say though that yes, it’s much better than any of the other programming languages I commonly have to write.
I believe the title is specifically a riff on this article from a few months back (lobste.rs discussion).
There appears to be
anyhow,thiserror,eyre,snafu,error-stack, and a number of others that are no longer the hotness. (anyerroris a Zig keyword, incidentally!)Those aren’t strictly necessary, though; you can just use
Box<dyn Error>in almost all cases where you’d useanyhow::Error. (Sure, it’s not as good, and it’s still a bit of a pain.)I’ve heard this is fine in applications but bad practice in libraries (i kinda get why). I’m not sure how the other error crates work though, I’ve only written applications and I just use this anyway and feel a bit dirty doing it.
Right. And the
anyhowcrate that the person above you mentioned is specifically intended/encouraged to be used for applications rather than libraries. Theanyhowerror type is essentially a not-quite-as-flexibleBox<dyn Error>.PHP makes breaking changes between minor versions (when activating all errors), which is certainly great to keep developers working, but a major PITA.
After 10 years of PHP, I value the Go 1.0 compatibility promise very much…
Java is still the de facto king of backwards compatibility that actually has the years to show for it. Just based on the title, that ‘title’ would go to Java in my opinion.
Until you try out the nightmare that is upgrading the maven, ant or gradle dependencies of an old project, then sure.
I would argue that it is at most as bad as with any other language, but arguably better as the ecosystem is significantly more stable than any other. (What is in the same ballpark in size are JS, and Python, neither is famous for their stability).
But of course third-party libs may decide to break their public APIs at any point, that’s irrelevant to language choice, besides “culture” — and Java’s is definitely not “going fast and breaking stuff”.
Sadly, starting with Java 9, while the core language is backwards compatible, the library situation is a nightmare. And when you’re doing an upgrade, you don’t care if it’s the core language or the library that’s changed, either way, it’s work to upgrade and ensure you don’t have regressions.
Lots of things have changed the package that they live in. In principle, this sounds like it’s not a very difficult change to accomodate, just find and replace, but it wreaks havoc when you have libraries that have to run on multiple JVM versions.
If you’re unlucky enough to be using Spring, it’s even worse. That framework has no concept of backwards compatibility, and is like a virus that spreads throughout your entire codebase.
I can’t share your experience at all. Even large jumps pre-8 to post-11 (though after like 11 they are basically trivial) are reasonably easy, and the few libraries that made use of JVM internals that got increasingly locked down (to prevent further breaking changes in the future) have largely been updated to a post-module version, so very often it’s just bumping the version numbers.
I don’t understand some of what you’re describing.
Are you saying that other languages can somehow prevent the problem of third-party libraries breaking backwards compatibility? Because, if you aren’t saying that, then the core language being stable is going to make the situation objectively better to deal with than if you have to worry about breaking changes in libraries and in the core language…
Yes, it can be tricky to write code to target multiple versions of Java at the same time, but in my experience, it’s about 100 times less tricky to do this with Java than almost any other language. JavaScript might be the only one that’s even better about running old code on newer runtimes without much problem. Are there others you consider better than Java at this? Specifically for running the SAME code on multiple versions of the language? I remember back in the day when I worked on a lot of C++, it was a nightmare to figure out what features I could use from C++11 to allow the project to build on various versions of GCC that shipped with different Linux distros (Ugh, RedHat’s ancient versions!).
I think that depending on the community, some languages have less BC issues because of libraries than others. Case in point for me was Clojure: both the core language and community have a stance of avoiding breaking compatibility, even if the language itself doesn’t offer any special mechanism for that. Quite the opposite actually, since the language is really dynamic and you can even access private functions without much difficulty.
Of course they can’t, and that’s not my point.
My point is that communities develop norms, and those norms include more or less careful treatment of backwards compatibility in libraries. My sense is that Haskell and JavaScript are probably the worst. Java is actually mixed, as there are a lot of libraries that do great work. Some even have a single version that runs on Java 5 through Java 21. But at my day job, we’re using Spring, and the situation is bad.
Though with regard to languages, I will say dynamic class-loading can make the situation worse. I’ve dealt with issues where the order of requests to a web server determined which version of a service provider was loaded. So 9 times out of 10, the code ran on the newer version of Java, but failed 1 time in 10.
It sounds like your argument is “having two problems is worse than having one”. But two little problems are better than one big problem.
I greatly appreciate the backwards compatibility of the Java language. But I would be happier with a slightly less backwards compatible core language, if I could trade it for an ecosystem that’s much better at backwards compatibility.
Right. I understand and agree about ecosystem “culture”. But, at the end of the day, someone said Java was good with backwards compatibility because it makes it easy to write code that will continue to work “forever”.
I guess maybe your point is that the language prioritizing backwards compatibility is only necessary, but not sufficient, for developers actually getting to experience the benefit of the backwards compatibility. Would you say that’s a decent interpretation/restating? I do agree with that. And if the language itself doesn’t care about backwards compatibility, then it’s impossible for the ecosystem to have stability.
Definitely true! Luckily, I only remember one time where I had a nightmare of a time figuring out some class or service loading bug.
Yeah, that’s exactly what I was saying. But there’s no reason to think the “one problem” (community doesn’t value backwards compat.) would be bigger or smaller for either case. So, it’s like comparing x to x + 1.
I really feel like this is mostly just Spring. I hate Spring, too, so I have been fortunate enough to not have to use it in several years now. But, as for the Java libraries I am using, I honestly couldn’t tell you about their JVM version compatibility over time. But, just judging by API stability, it seems that most “general purpose” or “utility” style libraries stay pretty backward compatible (thinking of the Apache and Guava libs, etc). It’s mostly the frameworky ones (like Spring, Spark, etc) that like to rewrite big chunks of their APIs all the time.
Happy to agree with your last paragraph. It’s very close to what I think.
Hey, have you heard of Perl?
Can you give an example of what you’re referring to?
I’m semi-active on the PHP internals mailing list (i.e. the list where development of php itself is discussed) and BC breaks are a near constant discussion point with any proposed change, so I’m kind of curious what breaking change you’re referring to here?
PHP maintains change logs for every minor version. Here’s the most recent list for migrating from version 8.2 to 8.3 (a minor version bump): https://www.php.net/manual/en/migration83.incompatible.php
Yes, I’m aware of the migration lists. I was asking to try and get an example of a real world issue that’s likely to affect users.
In that migration list specifically, the vast majority of the “breaks” are fixing inconsistencies.
Yes they are technically a BC break. But part of the discussion on each RFC for PHP is the expected real-world impact if there is a BC break.
Well, since I’m not the person you originally asked, I obviously can’t speculate on which breaking changes they’ve actually bumped into. But, I’ve works on several long-lived PHP projects going all the way back to version 5.3 and all the way up to 8.1 or 8.2 (I don’t quite remember which), and I’ve definitely had to fix broken code from minor version bumps. Luckily, we do have those migration lists, so I learned pretty early on to read them carefully and grep through any code base to fix things on the list before even running unit tests.
But, I’m not sure what the point is. The person said that PHP introduces breaking changes in minor version bumps, and that it frustrates them. Maybe they’re misguided for being frustrated by that, but it’s objectively true.
Personally, I’m perfectly fine with breaking bug fixes in minor versions. It’s not like you’re going to accidentally or unknowingly version bump the programming language of your projects. On the other hand, many of these changes are NOT bug fixes, but just general “improvements”, like the
range()changes or the “static properties in traits” change in the 8.3 list. I can see how this would be frustrating for someone who wants the bug fix changes, but doesn’t want to pay the extra “tax” of dealing with the non-essential breaking changes.But, again, I personally don’t care. I treat PHP minor versions as a big deal. PHP 8.2 is not the same language as PHP 8.1, which is fine. But, if you care about backwards compatibility and painless upgrades, Java is pretty much the king.
It’s mostly the ripple effects into libraries and “big” frameworks. For instance, I remember a few years ago a client was on still on PHP 5.3 (I think) and his hosting provider was about to shut down this version, so he had to get his site working on newer versions. This site was built on an in-house CMS of questionable code quality, using CakePHP.
The problem I ran into was that I had to step-by-step upgrade CakePHP through some major versions because the older versions wouldn’t run in the newer PHP due to BC breakage. Eventually, I completely hit a wall when I got to one major version where I discovered in the changelog that they completely rewrote their ORM. Because of the shitty code quality of the CMS (with no tests), I had to admit defeat. The client ended up having to commission the building of a new site.
Java and Go are really great in this regard. Every other language, including Python (which makes breaking changes from 3.12.3 -> 3.12.4!) is horrible on that front.
If they deprecated all of JavaScript’s features that can easily lead to bugs that are hard to track down, there won’t be much left, I presume.
I know ragging on JS is cool, and yes there are a few misfeatures, but you can easily avoid more or less all of them. The same isn’t true of many other languages (C and C++ for instance have myriad foot guns that cannot be trivially avoided, python has various bizarre behaviors around basic functionality, perl is basically all weirdities).
All languages have things that were in retrospect not fantastic, JS is no different in that regard, but acting like it is somehow an especially “bad” language is not an accurate representation.
You might be able to avoid the misfeatures as a user, but you can’t as a language implementor.
I worked in the Mozilla JS team in 2009 and it was pretty much a daily thing to hear someone mutter “Who invented this shit?”. ::all look at Brendan::
Brendan is a good guy, but wasn’t given much time and I’m sure could never have anticipated how JS is used today – or the performance of current implementations, which is better than C compiled with GCC or Clang without a ‘-O’ option (so implicitly -O0).
It would have been better if they’d just used Scheme or elisp. At least they didn’t use TCL.
I worked on JSC for a decade or so, so I understand all of this. |with| is not a challenging feature to support, it’s just a developer hazard. It could even be made much faster - but the messaging was so strongly “don’t use with()” that |with| perf simply does not matter. Which is good, because of the myriad foot guns that with has :D
I feel the adoption of scheme or lisp would have fairly aggressively held back adoption - think about just how much devs still push back against prototypes today, to the extent that we added classes to the language, that are literally just sugar on prototypes :-/ Certainly it would have opened potential for MS to not have had to copy JS and have that IE developed language win.
I don’t think that’s in any way surprising: JS using prototypal inheritance was always a convenient expedient more than anything else, working with them was always a painful verbose chore compared to languages actually designed for it (in contrast playing with Self was a pleasure, not to mention the realisation that a significant part of the confusion is JavaScript mislabelling things entirely).
The class syntax added what i very much believe had been intended from the start by Bram, to the extent that anything was ever intended. It is also able to work correctly with native object, something which is absolutely painful using prototypes (e.g. creating bespoke error types).
And we’d all be programming Visual BasicScript…
Wasn’t JS originally planned to be Lisp-like?
The JavaScript HOPL paper says on p. 77:7
The problem with your reasoning is that it is easier to avoid most other languages than JavaScript today.
That’s completely orthogonal to your point though, and to @olliej’s point. If you mean “like other languages, JavaScript has lots of misfeatures, but the blast radius is larger because JS is hard to avoid,” then say that.
On the contrary, I feel like there has been significant backlash lately against anyone who still criticizes JavaScript or PHP. Say something critical of the languages and you’re in for a lot of accusations about the last time you must have actually used said language or whether you’re just mindlessly parroting criticisms to be “edgy”.
I think with JS, specifically, we just spend so much time with it that we get used to all the nonsense. You say we can easily avoid the pitfalls/misfeatures of JS, but here’s where the fallacy is, IMO: each individual misfeature might be easy to avoid, but when you have 1,000 of them, it becomes very hard to consistently avoid all of them.
Not only that, but us “seniors” are probably taking for granted how long it took us to develop the habits to avoid those misfeatures. Sure, maybe after 5+ years of writing JS, you’ll finally be able to write with the same confidence as a beginner in almost any other language, but that’s not good enough to stop me from criticizing it–whether or not I think I’m being “cool” by doing so.
It’s hard to juggle
nullandundefined. Especially because a lot of existing JS code uses “truthiness” checks rather than===checks. We have to remember that iffoo: { x: number } | nullthatconst x = foo?.xwill have the typenumber | undefined(NOTnull!). We also have to remember that writingy == nullwill check thatyis eithernullorundefined! WHAT?We also have to remember that
typeof undefined === 'undefined'whiletypeof null === 'object'.We have to be careful how we pass callbacks into functions. If you want to call an object method as a callback, you can’t just pass the method; you have to either create a closure that captures the object and calls the method or you have to “bind” it:
o.method.bind(o). If you want to pass a function with an optional second argument to something likeArray.map, you’ll get incorrect results if you just pass the function directly; instead you have to wrap it in a closure that takes a single argument and calls the function with just the first argument.There’s just an endless list I could go through of bizarre, unintuitive, and frankly silly stuff in the language. It’s really not trivial to avoid them. It just takes a lot of experience.
Right on. I can only work in the ecosystem using TS on strict mode. Call it a skill issue, I don’t care, I just need tooling that watches my back as much as possible to feel like I can ship things I’m confident in. I’ve shipped some stuff to production in TS and used it for 2 years on and off. There’s a certain lack of solidness to it I still perceive, and it keeps me on guard. Probably comes down to the fact that TS is ultimately just a suggestion and you can do whatever you want. I think part of my brain gets deeply annoyed that the primitives of JS are just…wonky. Hard to trust everything downstream of that.
Lately I’ve been writing Kotlin for web stuff, favoring as little JS as possible, and it’s a joy. Kotlin’s a great balance of expressiveness and predictability, and JetBrains IDEs are top tier.
You can avoid them by using the band-aid language Typescript that they had to put over Javascript.
And Typescript itself is pretty meh. It’s the subversion of programming languages.
Honestly, they should deprecate the entire damn language and we should use
Swift or RustKotlin to write web apps or something.If they would finally agree on a way to use the DOM with WASM and without one single line of JS glue, that would already be doable.
I recently emailed Richard Stallman and asked about the ethics of this practice. This was his reply:
huh.
yeah, I dunno. the original introduction of the concept of copyleft had all this rhetoric about how it’s a license that gives users rights rather than takes them away and reserves them for the company.
I was pretty big on the ethical source movement for a couple years; it needed to be its own thing because by its nature, it imposes restrictions on how software can be used, which not only violates the founding principles of free software, but wasn’t even okay with the corporate lawyers who decide what things OSI will sign off on calling “open source”.
I eventually decided that, yeah, imposing those restrictions isn’t really part of a viable theory of change, because the bad actors it’s intended to constrain are not going to obey the license. if the intention is to make a clear and firm statement condemning certain things, it’s just as effective to do that in a form that isn’t a legal document. with that said, I don’t see any great harm in licenses which attempt to impose usage restrictions, I just think there are better ways to focus whatever attention programmers want to spend on promoting ethics in software.
I’ll spare you the explanation of where I went after moving on from that, because it’s irrelevant here. at any rate, the AGPL is exactly such a license that attempts to impose usage restrictions. I think I may have new feelings about the AGPL as a result of your comment, @mattheusmoreia, and I very much thank you for it. it’s going to take time for me to coalesce those feelings into coherent thoughts, so I’ll leave it there.
(yes, it was pretty hypocritical of the people on OSI’s legal discussion list to object to usage restrictions only when they’re rhetorically tied to ethics, and honestly, from how the discussion went, they knew it. oh well. there’s no reason to worry about it now.)
Minor nit, the AGPL doesn’t restrict usage but rather modification. I understand it restricts usage of the modification, but that’s not quite the same as usage in general.
oh - yes, that’s fair and I very much appreciate you pointing it out.
certainly it’s hard to see a coherent argument that AGPL passes a proper formulation of the desert island test
(if you’re trapped on a desert island with no communication with the outside world, can you still legally use the software for everything you could do with it outside? you can probably word the test in a legalistic way to allow AGPL to pass, but you shouldn’t)
and it’s in that spirit that I was calling it a usage restriction. but you’re correct, of course.
No one can connect to your software over the network, so you can’t possibly violate the AGPL.
oh.
hm yeah the test as-written has that problem, doesn’t it.
But the AGPL doesn’t place restrictions on connecting software over the network, it puts restrictions on modifying the code. This is the biggest misconception about the AGPL.
There are two steps involved:
If you never make your modified software accessible over a network, it behaves just like the GPLv3. So I think characterizing it as “restricting modification” is not really the best way to think about it.
I’m not characterizing anything, that’s literally what the license says. It has not a single clause putting any restrictions on how you run the program, including network connections. It does have a clause that says that when you modify the code, you must modify it in such a way that whenever anyone later runs the code, it will direct network users to a copy of the code. It’s framed entirely in terms of modification and what your duties as someone who is modifying the code are.
As I recall, AGPL mostly just requires that you make your source available under the AGPL to anyone who runs your code or accesses it over the network.
That’s not a problem in any situation I can think of unless you want to keep some part of the source secret from users.
the point in the linked article, and the top-level comment I was replying to, is precisely that the original corporation that owns the copyright gets to develop secret features they don’t release, while no competitor would get to do that. that’s a usage restriction in my view.
Assuming they take no contributions or have a CLA or assignment agreement
yes, which seems like it must be the case since otherwise their business model falls apart. right?
This is the key insight most people are missing. Copyright enforcement is expensive and impractical at scale
But is this actually true? In the case in question (Database Product offered by the Vendor as PaaS), the actors you’re primarily concerned about are not outlaw “software pirates” but - as stated in the post - big cloud companies taking the product and offering a better PaaS on their own infrastructure. This is exactly what ended up happening to Elasticsearch for example.
Are there any cases where AWS, GCP, Azure etc. have taken an AGPL product and just ran with it? I don’t think there have.
AGPL doesn’t stop AWS from selling a hosted offering of the unmodified code. It doesn’t prevent them from cross-subsidising it to undercut you.
AGPL doesn’t stop AWS from adding code that integrates with other (paid, proprietary) back-end services, releasing the code that integrates with those services, but then having a version that works better on AWS than elsewhere and lets AWS charge more.
Most importantly, changing to a restrictive license doesn’t stop AWS from doing a quick internal rewrite and providing equivalent service but now closed source just weeks later. As we’ve seen AWS do multiple times.
In my opinion, keeping AWS “happy” and contributing upstream to a proper OSS codebase with a generous license is a better outcome for everyone than creating an adversarial environment where they prefer to compete with a closed source alternative. At least in this scenario, some of AWS’s resources flow back into the OSS ecosystem.
First of all, the outlaw “pirates” we’re talking about are MediaTek, VMWare, UMD Global (branded as “Nokia”), and realistically every major tech firm.
Second, yes, several companies have famously relicensed from AGPL to a nonfree license because AWS was happy to run the AGPL version. IIRC this includes your elastic search example, including that AWS then forked the AGPL version and became the steward of the freedomware fork.
No, Elasticsearch was Apache licensed when AWS took it. That’s why they were able to add their own proprietary features and never give out the source.
That is not quite the whole story. All business features/plugins in Elasticsearch were proprietary licensed since the 1.0 version. That includes their access control product.
It’s not like “AWS took the product”, AWS always operated on open core and I think that’s very fine.
I’d argue OpenAI and ai companies have taken source code without providing theirs for ai training.
yeah, or people don’t want to believe it, which is fair enough but copyleft has been around long enough that there’s empirical evidence now
or people believe it but still see value in taking a stand - but in that case, stand on the high ground, not on the repurposed tool of oppression (copyright law)
To give an anecdote from my personal experience, as one of the things that pushed me towards permissive licenses:
I was a contributor to a GNU project. We discovered that there was a company selling a modified version of our GPL/LGPL code, with the copyright notices stripped and not providing source code to their customers. They had made some improvements but were not contributing them back.
The maintainer told the FSF and they decided that the cost of legal action was too great and so didn’t enforce the license. They required copyright assignment and so were the only ones with standing to sue, but that didn’t matter because none of us had the money for a lawsuit that would get us, maybe, a few tens of thousands of lines of code if we won (paying developers to rewrite the code would have been cheaper than paying lawyers to get the proprietary code and had a better chance of actually working).
At the same time, we heard from other companies that did want to be good citizens of the F/OSS ecosystem that they were avoiding our code because their lawyers deemed [L]GPLv3 too high risk. They were picking up a much less mature MIT-licensed alternative and contributing to it, rather than to our codebase.
In exchange for contributing to a copyleft project, I ended up with a project that (because it wasn’t all my own work), placed a bunch of restrictions on what I could do with it that didn’t apply to people who were willing to just ignore the license, and those people were making a bunch of money. At the same time, I was losing out on contributions from people who did want to follow the rules. People who intend to follow the rules are far more likely to be concerned about exactly what the rules allow, people who plan on ignoring them don’t care what they say.
Huge +1000.
I’ve never had the experience of someone violating the license of my work because I always use MIT for the reasons you stated, but I’ve seen exactly what you’re describing:
On multiple occasions I’ve worked within a company that saw some copyleft-licensed code they would have used, but the license was too high risk so they just did an internal clean room rewrite. This is so much easier than people think:
Combined, a company who sees a copyleft codebase that took 50 weeks to build can often get an internal rewrite in just a few weeks. Best case scenario they release this code under a more permissive license, but more likely they just keep it private/internal because maintaining OSS is additional effort.
On the other hand, when a company sees a permissive license, they would much rather contribute upstream for the same reason: Maintaining code is additional effort, so an internal fork is counterproductive. Nobody enjoys rebasing and maintaining internal forks!
I’ve seen this again and again: Copyleft licenses result in more internal private rewrites, permissive licenses result in more upstream collaboration. It may feel counterintuitive against how the licenses are marketed, but this is reality.
I’m a copyright abolitionist. I want copyright gone. I want licenses to not exist at all.
However, until the day copyright is abolished, and that day may never come, it’s not really in our best interests to give away software with zero strings attached. That’s just wealth transfer, directly into other people’s pockets.
Permissive makes sense in a world without copyright. That world is not yet reality. The reality is they can copy open source software all they want but we can’t copy their proprietary software. They’ve managed to make it illegal to even reverse engineer their proprietary things because that often involves breaking protections.
I won’t claim to know what “most people”–or any people–do or don’t realize/understand, but for me, personally, it’s not that I don’t realize that actually enforcing copyright licenses is expensive and impractical. It’s just that I don’t find that to be a great argument against using licenses that attempt to do more things.
You can apply the same argument to many things. Just enforcing the “default” copyright restrictions on a published written work is expensive. If you’re an amateur author who self-publishes a book and some publishing giant sells a book that’s a total rip-off of yours, what are the odds that you can afford to actually beat them in court? Should small-time authors just publish everything they write under some Creative Commons license just because enforcing anything more restrictive might not be feasible? On the flip side, what if we become super-successful from our great book and we’re more able to enforce the copyright in the future? Or vice versa: we think we can afford to enforce the license now, but maybe our financial fortunes ebb and flow over time in ways that make it harder for us to enforce the license later on.
It would be great if enforcement of these things were free, but just because they aren’t doesn’t mean we should give people permission to do something we don’t want them to do.
I realize that analogies such as above are sometimes used dishonestly to set up a “straw man” to attack, but I’m not intending anything like that. It sounds like you and the parent comment are more-or-less arguing that we should only put things into our licenses that we are confident that we can actually afford to enforce. And that’s the idea that rubs me the wrong way.
Yes?
Creative Commons have widely argued that their license is not fit for source code and software projects. So the answer would probably be “No”. You might want one of the BSD licenses?
This is the grand irony of AGPL when people complain about the recent trend of BSL/etc - regardless of whether they use AGPL or BSL or whatever other license, the original copyright owners always have the right to do whatever they want with it, while anyone else is subservient to the licence requirements; The only difference is that the AGPL licensed one has a bunch of mini Stallman’s chanting about how it is the one true way.
For all that people complain about MIT/BSD/Apache License as allowing forkers to do what they want - IMO it’s a huge red flag if a project isn’t permissively licensed.
It’s only a red flag if you want to steal and make money with foreign work. This kind of rhetoric is exactly what big companies want you to parrot. Good thing the people who you call mini stallmans (this sounds derogatory 🙄) exist and think ahead.
Not at all. It’s a red flag because it puts the initial project creators in a privileged position, where they can create money through non-public enhancements/etc, but others cannot. It’s specifically why I mentioned the irony of AGPL fans complaining about BSL etc.
Good for who exactly? As described in this very thread there are numerous downsides to *GPL licenses, particularly the way it’s often used when backed by a company (with copyright assignment).
All I’m suggesting is that they have similar, hard-line views on software licensing, but aren’t as well known. If you think that’s derogatory, that says a lot about your own views on Stallman.
You are complaining about copyright assignment (CLA), not about AGPL.
If you run a project that is MIT licensed and ask everyone to assign copyright to you, it’s irrelevant as the LICENSE still allows them to do whatever they wish.
The *GPL is the limiting factor here, because it denies others the freedom to use the code the same way the copyright holders can.
No. Any license limits non-owners only, there’s nothing special about AGPL/GPL here.
If you contribute under CLA to a MIT-licensed project, tomorrow the owner can change the license of “your” changes to GPL or to any other. If you are looking around for a “red flag”, the CLA fits the bill best.
If I contribute to an MIT project with a CLA the owner can indeed do whatever the fuck they want with the code, but they also can’t limit what I can do with the code either.
The GPL, and specifically the AGPL does limit what I can do, forever, regardless of what the copyright owner does.
You can blame a CLA all you want, a CLA only has practical effects with a license like GPL. A CLA on a permissive licensed project is about as useful as a fart in an elevator.
These creators have released free software to the world. Free as in freedom software. I want them to have every privilege imaginable.
There is no irony whatsoever. BSL is not a free software license while AGPLv3 is. When something is released as AGPLv3, our freedom is ensured. When it’s released as BSL, it is not.
The drawbacks you describe are not drawbacks of the license itself, they are drawbacks of the justice system, especially the american one. It does not follow that AGPLv3 should not be used just because people can violate it whenever they want.
So as I said. Freedom for me but not for thee.
The copyright owner always has that privilege regardless of license. That will always be the case no matter which license is picked. It doesn’t make any sense to pick anything other than AGPLv3. I don’t want anyone else to have those freedoms.
GPL is about user freedom, not developer freedom. It’s about me, as a user, being able to have absolute control over the software running on my machine. It’s not about developer freedom to make money by creating proprietary software and services.
Yes, that much is abundantly clear, believe me.
It’s not an AGPLv3 problem. The original copyright owner always has the right to do whatever they want to do. The code is literally their property. The license merely governs how others are allowed to use it.
Also there is no such thing as a “red flag”. People wrote the code, they can license it however they want. Others are lucky they can even see it.
I don’t believe copyright should exist but that’s the reality we find ourselves in right now. Permissive licensing is just wealth transfer. The value of your labor goes straight into the pockets of millionaires and corporations. It’s quite simply irrational in a world where copyright exists. It’s either AGPLv3 or all rights reserved.
If they want free code, they absolutely should have to abide by the AGPLv3’s terms. If they can’t, then there’s a business solution for that too: they can pay for permission to use it.
And Stallman is right. It’s better when only the copyright owner has that right. Software freedom is maximized this way. Stallman also advocates selling exceptions to the GPL, he has suggested this to companies, Trolltech and Qt being the most notable example. This made it possible for software to be freed.
Freedom for me but not for thee. Got it.
Don’t hold back, tell me what you really think mate. 🙄
The hyperbole has reached ridiculous levels.
It’s always amusing when RMS reaches a point where he concludes that GPL’d software has less value than proprietary software but then doesn’t reach beyond that to realise that, just maybe, the GPL is not the right tool if you want to maximise the amount of Free Software in the world.
I remember reading something years ago, I don’t remember who wrote it, or the exact words but to paraphrase it from memory:
GPL proponents care very much that their software remains “free” (for some definitions of free, obviously), above all else.
Permissive license proponents care very much that their software is as good as it can be, and available for use by anyone.
I don’t think that’s an entirely fair characterisation, I’d put it somewhat differently:
GPL proponents want to minimise the amount of proprietary software in the world and are willing to accept less Free Software as a compromise.
Permissive-license proponents want to maximise the amount of Free Software in the world and are willing to accept more proprietary software as a compromise.
I think that simplifies it too much though.
It is absolutely the case that a GPL licence artificially limits the number of entities willing to contribute to a given open source project. Yes some projects are popular enough and high profile enough that they get plenty of contributions anyway. But it’s naive to believe that the GPL doesn’t prevent people or organisations from contributing to projects, because of its limitations.
So it’s not just the case of “less” vs “more” - you have to consider what effect the license has on quality as well.
It’s not about developer’s freedoms, it’s about user’s freedoms. As a user, I absolutely want to have control over software. I don’t want developers to have the freedom to take control away from me at all, such as by making proprietary software or services.
Permissive licenses are the opposite. You can do whatever you want, including proprietary software and services. It enables exploitation of both users and fellow open source developers. People labor over code, corporations take it, make millions, don’t pay them a cent and don’t even give source code back. That’s not the freedom I want to support. It’s exploitation.
This does not compute. Some company taking some permissively-licensed e.g. MIT or Apache software does not “take control away from the user”. It doesn’t make that permissive license disappear, or the software itself that was permissively licensed. So as far as “user freedoms” go, a permissive license is perfectly fine.
You want something else entirely: You want the power to prevent companies from using existing software, unless they adopt your system of morals. You may choose to argue and bicker about whether “adopting your system of morals” is the question here, but copyleft is unabashedly a system of morals. And there’s nothing wrong with that choice for software that you have created: As the copyright holder, you have that right to demand that companies adopt your system of morals if they want to use software that you wrote. And as is evident from AGPLv3, companies generally seem to refuse to adopt your system of morals.
It’s an interesting definition of freedom that you have, that you argue against someone’s freedom to choose to allow their work to be exploited. Whether or not I agree with the position you are taking, there does exist a logical contradiction in your position.
It enables the development of software that does. I don’t really want to contribute to the creation of software that’s not part of the commons. Not for free, at least.
I do. And I want to go even further, and prevent a lot more things than that.
Then they should pay for the software or hire someone to do it themselves.
I’m arguing the opposite. Copyright owners always have that freedom, no matter which license they choose. Licenses only apply to others.
They can release software to the world AGPLv3 licensed and strike business deals with corporations that let them use the software under different terms – for a price.
Those who aren’t willing to pay will have to comply with the AGPLv3 and that’s the ideal outcome. They won’t be able to exploit the work, but the truth is it’s not really their work to exploit to begin with.
Permissive licensing reduces the leverage of all software developers, it just gives it all away at zero cost. They don’t even have to pay the cost of sharing source code. AGPLv3 gives you leverage, and with leverage comes negotiation, and hopefully business.
Fine. Don’t use anything that isn’t GPL, and you’re all set.
If that means you can’t use a piece of software that isn’t GPL, that sounds like a you problem.
This guy explains it better than I ever could:
Why I (A/L)GPL - Zed A. Shaw
If you use permissive licenses, you will be taken advantage of. You’re laboring all over your code and someone will come and take it, use it to make billions and not even give you the source code back which is literally the bare minimum. So it’s either AGPLv3 or all rights reserved, no free software at all. Those are the only choices which make any sense whatsoever as far as I’m concerned.
He’s also written a great post on relations between corporations and open source:
The Beggar Barons - Zed A. Shaw
Remember that episode where Microsoft people showed up in the ffmpeg issue tracker and self-assigned a high priority issue? Yeah.
If making billions from my code were easy, I would be a billionaire. Sony ships some of my code on the newer PlayStations, for example, and if they didn’t then they’d have to spend a few thousand dollars rewriting it in house. There is alternative option that means I get money from every PlayStation sold.
In my experience, companies are more likely to give code back to permissively licensed projects than GPL’d ones. With GPL, they have several options if they’re using the project in house:
Of these, the first option is the one that the lawyers like the least. As soon as you start contributing changes upstream, you’re announcing that you’re using a project that has a complex license. It is much lower risk to keep the changes private. Unless your project is (at least) hundreds of thousands of lines of code, the third and fourth options are often affordable. And may be lower risk than touching something GPL’d.
In contrast, if it’s permissively licensed, their choices are:
Most importantly, they can change their mind easily. And this happens quite often. I’ve worked with several companies that have forked permissively licensed projects and then realised that this is expensive. They implement features that are then implemented differently upstream and then have painful merges. Unless something is really critical to their commercial differentiators, it’s usually cheaper to upstream.
In contrast, if they’ve looked at a GPL’d project and decide to build something proprietary, there’s no path to ever contributing it to a totally different GPL’d codebase. This is really important because companies that aren’t use to working with an open-source upstream (remember, most software is developed for in-house use, not as off-the-shelf software).
I release things as F/OSS for three principle reasons:
None of these is made better by a non-permissive license. It costs me the same amount. It will restrict the people who can use it. It may prevent someone from taking my work and making a pile of money, but I don’t lose anything because someone else makes money. It reduces the chance that someone else will send contributions and, unless I make contributors sign a CLA (which pushes even more away), it then restricts what I can do with the combined project. If I release an MIT-licensed library and someone sends me a patch, I can still use the library in whatever project I want, even something proprietary. If I release a GPL’d component and someone sends me a GPL’d patch, I can now do less: if it’s just my code, I can give myself a different license, but if it’s my code + someone else’s, then I need their consent to give myself that license.
And it’s not just reusing my code under a proprietary license, it’s also reusing it in a project that wants to pull in a dependency under a different but incompatible OSI and FSF-approved license. I’ve been bitten by that in the past with libPoppler, which was extracted from xpdf, which was GPLv2-only and so incompatible with a bunch of other things (including Apache 2). With more convoluted licenses such as AGPL, the probability of incompatibility goes up.
I’m unconvinced. There are generally 2 incompatible worlds in competition: permissive and GPL.
Your corporation and its lawyers want GPL world gone, and in the name of risk mitigation they want to be legally able to buy off all maintainers for peanuts and kill permissive world on any given day. Historically, this nearly happened once with Unices. No, thank you.
On the other hand, if GPL world wins, nothing bad happens.
Most corporations don’t care about GPL world at all. It’s just a load of code that they don’t want to touch, as is most proprietary software.
I don’t really understand what this means.
How does that work? Most permissive licenses and the GPL are irrevocable. You can’t kill either. And, given that most corporations benefit from a thriving ecosystem of permissively licensed things that they can use and modify, why do you think they want to kill it?
Not really. AT&T sued Berkeley (for largely unrelated and stupid reasons), but most of the proprietary UNIX vendors coexisted happily with the BSD ecosystem, contributed some things to it, and kept their own differentiating bits. The differentiating bits were non-portable and so most software avoided them. Things like NFS came out of proprietary UNIX and were released under a BSD license to encourage adoption.
The problem is the lack of path from here to there. For the ‘GPL world wins’ outcome, you have to get people to contribute more to GPL’d code than to permissive and proprietary software. And, in 30 years of the GPL, this hasn’t happened.
As I posted elsewhere in this thread, I have personally seen companies take GPL’d code, violate the license, and the FSF (the copyright holder) do nothing to enforce it, meaning that GPL’d code is proprietary for anyone with enough money. I have seen companies decide to do a from-scratch proprietary implementation of a GPL’d thing rather than use the GPL’d thing, meaning that the use of the GPL increases the amount of proprietary software in the world. I have seen companies pay for a license for a proprietary thing rather than use a GPL’d thing (thereby funding the development of more proprietary software, meaning that the choice of the GPL increases the amount of proprietary software in the world)). I have seen companies do from-scratch permissively licensed implementations of GPL’d things rather than use the GPL’d thing (meaning that the choice of the GPL increased the amount of permissively licensed software, though by less than starting with a permissive license would).
When was the last time you saw a GPL’d implementation of something displace a permissively licensed one? When was the last time you saw anyone fund development of a GPL’d replacement for an existing permissively licensed project?
They pay you royalties on every console they sell? I’m not sure I believe that.
Every once in a while we have some sort of public meltdown because someone has been thanklessly and profitlessly maintaining code that even billionaire corporations use and depend on. Microsoft employees will show up to the ffmpeg issue tracker, self assign a high priority issue and then offer them a few thousand dollars when they propose a support contract.
So why should anyone believe that corporations will just happen to choose to pay people royalties for no reason?
That’s by design. It should be as expensive as possible for them to not develop free and open source software. It’s the only leverage we have.
That’s also true of GPL software, by the way. Linux is famous for weaponizing it by deliberately keeping the internal kernel ABI unstable. Linux gets multiple tens of patches every single hour. They get so many patches, these corporations will never be able to keep up unless they upstream their code. If they don’t, they get left behind, their products don’t run mainline kernels, they don’t work out of the box, they don’t benefit from the continuous improvements, they are buggier and just generally worse as a result.
I’ve worked with corporations too. About a week ago, one of them sent me GPLv2 Linux drivers for a power supply. I’m trying to clean it up so I can upstream it. The drivers target Linux 3.10 which was released over a decade ago. It would have been much better if they had simply upstreamed the code: Linux would have had builtin support for the device and all products which used it would have worked out of the box. Why they don’t do that is beyond me.
Sorry, typo, ‘There is alternative’ was meant to read ‘There is no alternative’.
No matter what license I choose, Sony will not pay me money. They will either:
These are the three choices that exist. I prefer to incentivise the second option.
And that’s fine. If they want support, they can pay for it. I’m not going to prioritise their bugs just because they’re big. If something is a show-stopper for them, that’s their problem, not mine. If they fix it, I’ll review their PR when I get around to it.
Perhaps my experience is unusual, but I’ve had far more positive interactions with folks at big companies than negative. I’ve had important bug fixes and new features contributed by companies because they want the feature and have worked with upstream. I’ve never had a big company drive by the issue tracker and demand fixes (I have had individuals do this, on a lot of occasions). I did once have Microsoft create a fork of one of my projects because they had some code that was a bit ugly to make things work now for a product release, but it was a friendly fork and they worked with me to clean up the code and upstream it, so everyone won in the end.
But that isn’t what happens. It incentivises developing proprietary alternatives to Free Software projects.
Tell that to Google, who kept entire filesystems out of tree for their internal infrastructure. A lot of companies keep in-house forks of Linux.
Because the GPL sets up an adversarial relationship. They are not sending you the drivers because they feel they get any value from it, they are sending you drivers because they are legally obliged to. They’ve talked to lawyers and those lawyers have worked out the minimal set of things that they need to do to be compliant. And they won’t change because the GPL has locked them into a mindset that engaging with the community is a source of liability.
So why should they get free code with no strings attached at all?
Sony is using open source code to make locked down game consoles, digital fiefdoms where we are their serfs. The only reason the playstations ever ran Linux is because they’d get tax benefits in some regions. That’s how they look at us, free code they can use to make billions and dodge taxes.
I’d prefer that they not use my code at all than watch them use it to make a killing off of things like DRM which I think should be illegal to begin with. I feel like publishing code which enables a corporation like that is actively working against my own interests which is just irrational. I’d probably think differently if they were paying me fat stacks, but that’s just the thing, they aren’t doing that.
That can’t be easy. Nor should it be made easy.
This is reflected in the effort they’re making to run mainline kernels on Androids and especially their own hardware, no doubt because the firmware situation is too painful to cope with otherwise. Google has contributed a lot towards making smartphones and laptops more free by improving mainline Linux support.
As they should be. And they should be legally obliged to allow me, the user and owner of the machine, to customize that code and run it on the device I bought, too. It should be literally my right, the GPL should not even be necessary to make it happen.
Okay, so your starting point is that everything has to be a zero-sum game and no one can ever benefit from things without contributing? I guess that explains why we disagree. I would hate to live in a society that was built according to your ideals.
So you build a society that is not enforced by people agreeing to act in together for shared interests, but by the threat of legal action. Again, I don’t want to live in the kind of society that you build. Fortunately, neither do most other people and so there’s little danger of it.
Wow. I thought things got hyperbolic before. I was going to engage on a few points but I don’t think it’s really worthwhile when you clearly don’t understand what words like “literal” mean, and then assume everyone else must have the same values and view point as you do.
Edit to add:
I just want to add a very fucking relevant quote from that first article you posted, given how insistent you are that my choice is “wrong”:
I respect your right to license your work however you want. I don’t really understand the reasons for using permissive licensing because I just don’t see the advantage, but that’s okay. I can’t make the choice for others.
I’m aware of that quote. I even paraphrased it in another reply. As far as this discussion is concerned, I’m just trying to address the points made in each comment I replied to.
That quote is especially relevant here given the claim that using AGPLv3 is a “red flag”. Personally I think it’s the only reasonable choice, and I made it a point to explain why I think that. Not to convince you personally to use the license but because I don’t want my own projects to have “red flags” in them.
There’s also another part in that text that wasn’t cited but is very relevant:
I believe free software is about maximizing our own advantages as developers and above all as computer users, and that it’s not at all about just giving everything away for free. I don’t really appreciate it when that belief is reduced to a “red flag”.
That’s literally impossible. Red flags are inherently an individual thing. For you a red flag might be that it’s not GPL licensed. You literally cannot please everyone.
You can believe whatever you want. I believe different things. It’s that simple.
It’s exactly because of this issue that my company has considered assigning our copyrights to a charity, so that we couldn’t do that even if we wanted to and would be bound by the license ourselves.
So I’ve been thinking about why the “monads are just like X!” analogies fall flat, and I realized that the core of it is that monads are purely an abstraction. Monads are just a set of operations, and that happens to be useful when applied to real things that follow certain laws.
It’s almost a copout, except that’s actually the default for mathematics: numbers are purely an abstraction too. Consider:
Quantities, timespans, and ordinals are “real things”, totally incomparable with each other, and yet all are modelable as
4 + 3 = 7. It’s only that numbers feel natural because we use them all the time, so we don’t think about how they’re actually a weird abstraction over reality.I think you’re hitting on the right vein.
I’ve always felt a little weird when I read about how hard it is to understand monads, or that they’re “like boxes” or “like burritos”. I even feel weird when someone mockingly cites a technical definition that has “scary” sounding words like “endofunctor”.
I feel weird because I don’t find it complicated and I’m not impressed/scared by technical terms like “endofunctor” even though I never use such terms enough to even remember what it means most of the time. But I feel like if I say that, it either means I’m missing something so obviously “complicated” to everyone else, or that I’ll come across as condescending or full of myself.
To my mind, a monad is just a monad. It’s just a word with a definition. It’s a “thing” that holds values from a specific set and, by definition, upholds certain invariants and allows certain operations over those values. This doesn’t feel any different to me than any other mathy definition. “What’s a rational number? It’s an element of the set of numbers that can be represented by a ratio of P/Q where P is any integer and Q is any integer that’s not 0.” (I don’t know if that’s actually a good definition, but you get my point).
The whole thing gets a shrug from me…
I was halfway through my thesis when I realized this. I’m not special for understanding this, I just care enough. Then again, maybe all of science works this way.
Totally agree. It’s sad because I sometimes read on lobster people keeps scary words to threaten others. But when going just a little further, ask them about interesting examples of “functors”, they cannot give any but the functor between categories of sets.
And here I thought a monad was a type with a functional application that is associative, a value that acts like an identity, and (most important but almost never mentioned) require lazy evaluation to actually work mathematically.
Why does it need lazy evaluation? There’s enough lambda abstraction in the monad laws that I can’t see where eagerness could affect them. Or, to put it another way, the algebraic structure of a monad imposes an order of evaluation, which is why monads are useful for imperative functional programming.
I’m specifically thinking of side effects like I/O. With eager evaluation, associativity doesn’t quite hold. In C:
Yes, I know, this isn’t monadic at all, but pretend that the integer parameters are monads. When I compile and run this, I get:
If C did have lazy evaluation, this should print
And while output can be buffered, input really suffers here with eager evaluation.
Aha, thanks, that (after a bit of thought) makes sense. (Though the issue is really purity rather than laziness.)
Eventually I worked out where my mistake came from: in monads like State and IO there’s another layer of lambda abstraction wrapping the action represented by the monad. The action doesn’t occur until it is applied to an initial state; what the monad algebra does is construct the action so that the state is threaded through linearly.
Nooooo, a monad is a burrito.
I know it’s a meme, but when taken seriously you can use burrito allegory. It may not explain what a monad is but it explains properties of a monad.
This blog post by Mark Dominus is really quite good. https://blog.plover.com/prog/burritos.html
Yes, I think that’s right. People don’t get hung up on “understanding numbers” but they do get hung up on “understanding monads”, for some reason.
It’s also much easier to appreciate that abstraction when you already care about the examples! For me, “functor is anything you can map over” and “monoid is anything you can append” feel much more obvious as patterns that pop up a lot, that I’d want an interface for. And no more abstract than an Iterator or Collection interface (besides the choice of names).
I think people may learn monads backwards, because they hear about “the IO monad” and think you need monads to do IO–but that’s like saying you need functors to do lists, because of “the List Functor”. I much prefer the Learn you a Haskell approach, where you learn about “IO actions”, and how to construct them and chain them together.
Yeah, this is the point jerf makes in his Monad tutorial. He compares them to Iterators: it’s an interface.
https://www.jerf.org/iri/post/2958/
I mostly agree, but in another light, I’d make the analogy that monads are to lists as groups are to (types of) numbers.
You have the familiar operations (addition, multiplication), but shorn of guarantees that we typically take for granted (commutativity). In the case of a list, it’s trivial to get a value out of a list. In the case of a monad, not so much.
Of course, monads also muck up the matter by not overloading the common operations. It would probably be confusing in a different way if the monad operations were “push” and “append” or whatnot.
I think this is generally true. People get familiar with specific abstractions through practice.
As another instance, computation is “just” an abstraction. We’re dimly aware that processors do “something” much more complex, but it’s nice to think about sequenced orders of steps and mutable memory and the like.
When you were first learning it, you probably had to work really hard to build familiarity with that abstraction. And then over time it becomes second nature and you can think of it occurring all over the place.
Working with specific abstractions after a lot of practice is relatively common. Working with the act of abstraction and generating new ones is less common. Mathematicians definitely do it a lot. We’d think programmers would too, but I think, often, programmers prefer to imagine things concretely.
This is the right mindset.
No wonder there is a “misconception” that it’s special syntax when it actually is special syntax in non-experimental versions of the language.
Furthermore, I’m pretty sure that the operator really was just special syntax at first and the Try trait was added retroactively some time after 1.0.
The
?operator was originally stabilized in 1.13. Before then, there was thetry!macro (which now requires special syntax to call as of the 2018 edition). Thetry!syntax was originally syntactic sugar for this:But the rest is spot on,
?originally desugared to the same thing astry!, and only added an unstable trait behind-the-scenes sometime later.Ah, yes! I totally forgot about the try! macro. Which is funny, because I was actually writing Rust code before the ? operator landed.
Even in stable Rust, the standard library can use unstable features and implements
Tryfor more types, including the commonOption<T>.To be fair, one reason one would assume it’s special syntax is simply because you never need to implement
Try. I have only ever once in my life had the desire to implement a customTryimplementation, and in the end I realized I really should just use the standardResultand that I was tryharding way too much, akin to operator overload abuse. Outside of that, people probably never even take a moment to think if they can overload?.Aside: I’ve always found the timers APIs to be a little tricky, anyway. They aren’t “hard”, per se, but it’s a lot like implementing a binary search: easy, but also easy to mess up.
I feel similarly to the whole EventEmitter approach. Every EventEmitter has its own semantics about what events are emitted and when, and it can be very easy to accidentally assume certain behaviors will or won’t happen, or that certain event orders are guaranteed, etc. And then it’s also easy to cause memory leaks by not removing event listeners when you’re done with them. I’ve seen a lot of slightly-incorrect StackOverflow answers when it comes to dealing with Node.js’s various EventEmitters.
Nicely written, and I do think that’s a big part of the issue that I had never considered before. I’ve complained about the speed of software/computers today, but I simultaneously don’t delude myself into remembering with rose-tinted glasses. It always used to take a long time, on any OS, to load a large directory from a spinny-HDD in whatever GUI file manager. But, like the article says: that’s expected to be slow- humans all understand that more items to process means more time is needed to process them. But, spending a bunch of time to just switch between documents that are, allegedly, already “open” is a surprising point of friction to encounter and that’s why it’s so frustrating. Or, when key presses or clicks are “lost” because the UI was too busy doing something else and just didn’t even queue up the inputs.
It’s definitely the “stupid slow” that we’re the most frustrated about. Thank you for this new vocabulary! :)
Spent a lot of time on firefox perf engineering, been writing simple uis since windows 3.11
Current UI trend of implemented on top of dom in functional-reactive manner has a really good reason: UI state is freaking hard. Prior to React UIs were much more buggy. Even simple things like resizing windows was a minefield. UI components are difficult to theme, introspect. Supporting multiple platforms and mobile in same codebase was merely a dream. Displaying tables of data was incredibly frustrating because every platforms had different limits in what one could do in tables.
So yeah DOM-based UI sucks for perf due to not having explicit control over every pixel, but I think it’s pretty cool that modern devs take functional multi platform ui rendering for granted. We would be much worse off if web never took off and we were all stuck with a mix of Borland, Microsoft, Sun swing, etc. Imagine if the hated slack ui looked like a windows app on every platform.
It’s about wanting to spend on dev. A company like Slack is big enough to handle 3 native implementations. They don’t do it because their customers are okay with the bloat. Look at 1Password – they went from great native apps, to bloated electron apps. Once you get enterprise customers on your side, you don’t have to be lean.
I’m not at all an expert in this domain, but it’s hard for me to imagine that the reason these UIs suck at performance/latency has anything to do with having “explicit control over every pixel”. I doubt any of the GUI apps that I use that feel “snappy” are actually working at such a low level that they’re dealing with individual pixels or doing their own refresh rate calculations, etc.
But, I honestly don’t know what the issue is. Like the article describes, there’s really no reason that MS Teams should be slow to switch to a different chat. There’s obviously just some lackluster engineering of what’s being done on the main thread vs what should be offloaded to a background task (maybe loading too much history upfront, or applying too much styling to elements that wouldn’t be visible yet anyway, etc).
I imagine it’s because developers are more likely to add a network call to any given UI action these days. If the expectation is that everyone has a fast reliable internet connection, then why not put a network call? Only issue is that large tech companies have big systems so that one network call from a user might be making a large number of calls to other internal services.
Plus, companies want to own your data so they are not necessarily incentivized to keep it device local. Attitudes towards collecting data probably subtlety influence app design in a way that hurts latency.
DOM-based browser control is relatively easy to use correctly, but extremely hard to reason when it’s gonna do stuff. It’s one of the least intuitive architectures when it comes to performance. It’s a gigantic stack with all kinds of hacks, much harder to reason about that the single-threaded event loops of old school desktop apps. For example modern browsers offload renders to graphics cards, which results in really smooth animations and scrolling(atleast sometimes), but a lot of the [implicit] communication between layout engine and gpu causes UI freezes and your app has 0 control over that. This is a case where using “higher-performance” apis sometimes results in speedups and often in slowdowns. Same case with multi-process rendering..shoveling tons of data over Interprocess-Comms is also very likely to cause your app to stall and you have little visibility into that.
One of the reasons some people like the web canvas api for high-perf stuff is because it is much closer to the old simpler apis.
IMO a major factor to perceived slow UI on modern systems is that modern UI frameworks are designed with simplified APIs compared to what people used in the ’90s and early ’00s. React-ish web frameworks running in an Electron shell are the prototypical example (and several Electron-based programs are mentioned in the article), but the same is true of SwiftUI vs Cocoa, or the various .NET UI frameworks vs Win32.
For example, last year I wrote a small macOS program that needed to display a big list of values in a table. I used SwiftUI for it and it was slow, to the point where my laptop was dropping frames when trying to scroll. I switched the table widget to
NSTableViewand it was much faster. The difference was thatNSTableViewexposed additional API that allowed me to reuse allocated rows, whereas the SwiftUI table just doesn’t provide that functionality at all (as far as I can tell).However, the SwiftUI version took about 30 minutes to write, starting from googling “swiftui table tutorial”. The
NSTableViewversion took me about two days, and even after finishing it I have maybe 30% confidence in the code not being buggy somehow. This isn’t a slight against Cocoa, it’s just a much lower-level API and I have zero experience in macOS programming.From the perspective of a fairly junior programmer employed in a feature factory, working to a strict deadline to produce software that just needs to be “good enough”, an API that can take an absolute beginner from an empty file to a completed ticket in 30 minutes is great.
I have to support some pretty old iOS devices, so I still haven’t been able to move to SwiftUI even if I wanted to, but it’s really disappointing to hear that the performance is still garbage (or was a year ago, anyway) after four or five years of being pushed as the de facto way to do MacOS/iOS UI code.