The TypeScript dev lead posted this response about the language choice on Reddit, for anyone who’s curious:
(dev lead of TypeScript here, hi!)
We definitely knew when choosing Go that there were going to be people questioning why we didn’t choose Rust. It’s a good question because Rust is an excellent language, and barring other constraints, is a strong first choice when writing new native code.
Portability (i.e. the ability to make a new codebase that is algorithmically similar to the current one) was always a key constraint here as we thought about how to do this. We tried tons of approaches to get to a representation that would have made that port approach tractable in Rust, but all of them either had unacceptable trade-offs (perf, ergonomics, etc.) or devolved in to “write your own GC”-style strategies. Some of them came close, but often required dropping into lots of unsafe code, and there just didn’t seem to be many combinations of primitives in Rust that allow for an ergonomic port of JavaScript code (which is pretty unsurprising when phrased that way - most languages don’t prioritize making it easy to port from JavaScript/TypeScript!).
In the end we had two options - do a complete from-scrach rewrite in Rust, which could take years and yield an incompatible version of TypeScript that no one could actually use, or just do a port in Go and get something usable in a year or so and have something that’s extremely compatible in terms of semantics and extremely competitive in terms of performance.
And it’s not even super clear what the upside of doing that would be (apart from not having to deal with so many “Why didn’t you choose Rust?” questions). We still want a highly-separated API surface to keep our implementation options open, so Go’s interop shortcomings aren’t particularly relevant. Go has excellent code generation and excellent data representation, just like Rust. Go has excellent concurrency primitives, just like Rust. Single-core performance is within the margin of error. And while there might be a few performance wins to be had by using unsafe code in Go, we have gotten excellent performance and memory usage without using any unsafe primitives.
In our opinion, Rust succeeds wildly at its design goals, but “is straightforward to port to Rust from this particular JavaScript codebase” is very rationally not one of its design goals. It’s not one of Go’s either, but in our case given the way we’ve written the code so far, it does turn out to be pretty good at it.
And it’s not even super clear what the upside of doing that would be (apart from not having to deal with so many “Why didn’t you choose Rust?” questions)
People really miss the forest for the trees.
I looked at the repo and the story seems clear to me: 12 people rewrote the TypeScript compiler in 5 months, getting a 10x speed improvement, with immediate portability to many different platforms, while not having written much Go before in their lives (although they are excellent programmers).
This is precisely the reason why Go was invented in the first place. “Why not Rust?” should not be the first thing that comes to mind.
I honestly do think the “Why not Rust?” question is a valid question to pop into someone’s head before reading the explanation for their choice.
First of all, if you’re the kind of nerd who happens to follow the JavaScript/TypeScript dev ecosystem, you will have seen a fair number of projects either written, or rewritten, in Rust recently. Granted, some tools are also being written/rewritten in other languages like Go and Zig. But, the point is that there’s enough mindshare around Rust in the JS/TS world that it’s fair to be curious why they didn’t choose Rust while other projects did. I don’t think we should assume the question is always antagonistic or from the “Rust Evangelism Strike Force”.
Also, it’s a popular opinion that languages with algebraic data types (among other things) are good candidates for parsers and compilers, so languages like OCaml and Rust might naturally rank highly in languages for consideration.
So, I honestly had the same question, initially. However, upon reading Anders’ explanation, I can absolutely see why Go was a good choice. And your analysis of the development metrics is also very relevant and solid support for their choice!
I guess I’m just saying, the Rust fanboys (myself, included) can be obnoxious, but I hope we don’t swing the pendulum too far the other way and assume that it’s never appropriate to bring Rust into a dev conversation (e.g., there really may be projects that should be rewritten in Rust, even if people might start cringing whenever they hear that now).
While tweaking a parser / interpreter a few years ago written in Go, I specifically replaced a struct with an ‘interface {}’ in order to exercise its pseudo-tagged-union mechanisms. Together with using type-switch form.
These day’s I’d actually make it a closed interface such that it is more akin to a tagged-union. Which I did for another project which was passing around instances of variant-structs (i.e. a tagged union), rather than building an AST.
So it is quite possible to use that pattern in Go as a form of sum-type, if for some reason one is inclined to use Go as the implementation language.
A relevant quote is that C# has “some ahead-of-time compilation options available, but they’re not on all platforms and don’t really have a decade or more of hardening.”
Yeah Hjelsberg also talks about value types being necessary, or at least useful, in making language implementations fast
If you want value types and automatically managed memory, I think your only choices are Go, D, Swift, and C# (and very recently OCaml, though I’m not sure if that is fully done).
I guess Hjelsberg is conceding that value types are a bit “second class” in C#? I think I was surprised by the “class” and “struct” split, which seemed limiting, but I’ve never used it. [1]
And that is a lesson learned from the Oils Python -> C++ translation. We don’t have value types, because statically typed Python doesn’t, and that puts a cap on speed. (But we’re faster than bash in many cases, though slower in some too)
Now that I’ve worked on a garbage collector, I see a sweet spot in languages like Go and C# – they have both value types deallocated on the stack and GC. Both Java and Python lack this semantic, so the GCs have to do more work, and the programmer has less control.
There was also a talk that hinted at some GC-like patterns in Zig, and I proposed that TinyGo get “compressed pointers” like Hotspot and v8, and then you would basically have that:
Yeah, I’m kind of curious about whether OCaml was considered at some point (I asked about this in the Reddit thread, haven’t gotten a reply yet).
OCaml seems much more similar to TS than Go, and has a proven track record when it comes to compilers. Maybe portability issues? (Good portability was mentioned as a must-have IIRC)
Not sure what more TypeScript would have needed. In fact, Flow’s JavaScript parser is available as a separate library, so they would have shaved off at least a month from the proof of concept…
Although thinking about it a bit more, I think Nim, Julia, (and maybe Crystal) are like C#, in that they are not as general as Go / D / Swift.
You don’t have a Foo* type as well as a Foo type, i.e. the layout is orthogonal to whether it’s a value or reference. Instead, Nim apparently has value objects and reference objects. I believe C# has “structs” for values and classes for references.
I think Hjelsberg was hinting at this category when saying Go wins a bit on expressiveness, and it’s also “as close to native as you can get with GC”.
I think the reason this Go’s model is uncommon is because it forces the GC to support interior pointers, which is a significant complication (e.g. it is not supported by WASM GC). Go basically has the C memory model, with garbage collection.
I think C#, Julia, and maybe Nim/Crystal do not support interior pointers (interested in corrections)
Someone should write a survey of how GC tracing works with each language :) (Nim’s default is reference counting without cycle collection.)
Yeah that’s interesting. Julia has a distinction between struct (value) and mutable struct (reference). You can use raw pointers but safe interior references (to an element of an array for example) include a normal reference to the (start of the) backing array, and the index.
I can understand how in Rust you can safely have an interior pointer as the borrow checker ensures a reference to an array element is valid for its lifetime (the array can’t be dropped or resized before the reference is dropped). I’m very curious - I would like to understand how Go’s tracing GC works with interior pointers now! (I would read such a survey).
Ok - Go’s GC seems to track a memory span for each object (struct or array), stored in kind of a span tree (interval tree) for easy lookup given some pointer to chase. Makes sense. I wonder if it smart enough to deallocate anything dangling from non-referenced elements of an array / fields of a struct, or just chooses to be conservative (and if so do users end up accidentally creating memory leaks very often)? What’s the performance impact of all of this compared to runtimes requiring non-interior references? The interior pointers themselves will be a performance win, at the expense of using an interval tree during the mark phase.
It’s been a few years since I’ve written any Go, but I have a vague recollection that the difference between something being heap or stack allocated was (sometimes? always?) implicit based on compiler analysis of how you use the value. Is that right? How easy it, generally, to accidentally make something heap-allocated and GC’d?
That’s the only thing that makes me nervous about that as a selling point for performance. I feel like if I’m worried about stack vs heap or scoped vs memory-managed or whatever, I’d probably prefer something like Swift, Rust, or C# (I’m not familiar at all with how D’s optional GC stuff works).
$ go build -gcflags "-m" main.go
.\main.go:8:14: *y escapes to heap
.\main.go:11:13: x does not escape
So the toolchain is pretty transparent. This is actually something I would like for the Oils Python->C++ compiler, since we have many things that are “obviously” locals that end up being heap allocated. And some not so obvious cases. But I think having some simple escape analysis would be great.
Hejlsberg said they got about 3x performance from native compilation and value types, which also halved the memory usage of the compiler. They got a further 3x from shared-memory multithreading. He talked a lot about how neither of those are possible with the JavaScript runtime, which is why it wasn’t possible to make tsc 10x faster while keeping it written in TypeScript.
Yeah but I can get bigger memory wins while staying inside JS by sharing the data structures between many tools that currently hold copies of the same data: the linter, the pretty-formatter, the syntax highlighter, and the type checker
I can do this because I make my syntax tree nodes immutable! TS cannot make their syntax tree nodes immutable (even in JS where it’s possible) because they rely on the node.parent reference. Because their nodes are mutable-but-typed-as-immutable, these nodes can never safely be passed as arguments outside the bounds of the TS ecosystem, a limitation that precludes the kind of cross-tool syntax tree reuse that I see as being the way forward
Hejlsberg said that the TypeScript syntax tree nodes are, in fact, immutable. This was crucial for parallelizing tsgo: it parses all the source files in parallel in the first phase, then typechecks in parallel in the second phase. The parse trees from the first phase are shared by all threads in the second phase. The two phases spread the work across threads differently. He talks about that kind of sharing and threading being impractical in JavaScript.
In fact he talks about tsc being designed around immutable and incrementally updatable data structures right from the start. It was one of the early non-batch compilers, hot on the heels of Roslyn, both being designed to support IDEs.
It’s true that I haven’t watched the interview yet, but I have confirmed with the team that the nodes are not immutable. My context is different than Hejlsberg’s context. For Hejlsberg if something is immutable within the boundaries of TS, it’s immutable. Since I work on JS APIs if something isn’t actually locked down with Object.freeze it isn’t immutable and can’t safely be treated as such. They can’t actually lock their objects down because they don’t actually completely follow the rules of immutability, and the biggest thing they do that you just can’t do with (real, proper) immutable structures is have a node.parent reference.
So they have this kinda-immutable tech, but those guarantees only hold if all the code that ever holds a reference to the node is TS code. That is why all this other infrastructure that could stand to benefit from a shared standard format for frozen nodes can’t: it’s outside the walls of the TS fiefdom, so the nodes are meant to be used as immutable but any JS code (or any-typed code) the trees are ever exposed to would have the potential to ruin them by mutating the supposedly-immutable data
To be more specific about the node.parent reference, if your tree is really truly immutable you need to replace a leaf node you must replace all the nodes on the direct path from the root to that leaf. TS does this, which is good.
The bad part is that then all the nodes you didn’t replace have chains of node.parent references that lead to the old root instead of the new one. Fixing this with immutable nodes would mean replacing every node in the tree, so the only alternative is to mutate node.parent, which means that 1) you can’t actually Object.freeze(node) and 2) you don’t get all the wins of immutability since the old data structure is corrupted by the creation of the new one.
See https://ericlippert.com/2012/06/08/red-green-trees/ for why Roslyn’s key innovation in incremental syntax trees was actually breaking the node.parent reference by splitting into the red and green trees, or as I call them paths and nodes. Nodes are deeply immutable trees and have no parents. Paths are like an address in a particular tree, tracking a node and its parents.
Hm yeah it was a very good talk. My summary of the type checking part is
The input to the type checker is immutable ASTs
That is, parsing is “embarassingly parallel”, and done per file
They currently divide the program into 4 parts (e.g. 100 files turns into 4 groups of 25 files), and they do what I’d call “soft sharding”.
That is, the translation units aren’t completely independent. Type checking isn’t embarassingly parallel. But you can still parallelize it and still get enough speedup – he says ~3x from parallelism, and ~3x from Go’s better single core perf, which gives you ~10x overall.
What wasn’t said:
I guess you have to de-duplicate the type errors? Because some type errors might come twice, since you are duplicating some work
Why the sharding is in 4 parts, and not # CPUs. Even dev machines have 8-16 cores these days, and servers can have 64-128 cores.
I guess this is just because, empirically, you don’t get more than 3x speedup.
That is interesting, but now I think it shows that TypeScript is not designed for parallel type checking. I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library. Though it’s hard to compare, since it also has to generate code
A separate thing I found kinda disappointing from the talk is that TypeScript is literally what the JavaScript code was. There was never a spec and will never be one. They have to do a line-for-line port.
There was somebody who made a lot of noise on the Github issue tracker about this, and it was basically closed “Won’t Fix” because “nobody who understands TypeScript well enough has enough time to work on a spec”. (Don’t have a link right now, but I saw it a few months ago)
Why the sharding is in 4 parts, and not # CPUs. Even dev machines have 8-16 cores these days, and servers can have 64-128 cores.
Pretty sure he said it was an arbitrary choice and they’d explore changing it. The ~10x optimization they’ve gotten so far is enough by itself to keep the project moving. Further optimization is bound to happen later.
I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library.
Work has been going on for years to parallelize rust’s frontend, but it apparently still has some issues, and so isn’t quite ready for prime time just yet, though it’s expected to be ready in the near term.
Under 8 cores and 8 threads, the parallel front end can reduce the clean build (cargo build with -Z threads=8 option) time by about 30% on average. (These crates are from compiler-benchmarks of rustc-perf)
I guess this is just because, empirically, you don’t get more than 3x speedup.
In my experience, once you start to do things “per core” and want to actually get performance out of it, you end up having to pay attention to caches, and get a bit into the weeds. Given just arbitrarily splitting up the work as part of the port has given a 10x speed increase, it’s likely they just didn’t feel like putting in the effort.
But check the chapters, they’re really split into good details. The video is interesting anyway, technically focused, no marketing spam. I can also highly recommend watching it.
Another point on “why Go and not C#” is that, he said, their current (typescript) compiler is highly functional, they use no classes at all. And Go is “just functions and data structures”, where C# has “a lot of classes”. Paraphrasing a little, but that’s roughly what he said.
Acknowledging some weak spots, Go’s in-proc JS interop story is not as good as some of its alternatives. We have upcoming plans to mitigate this, and are committed to offering a performant and ergonomic JS API.
Once again I am baffled by how many people dismiss this with the usual “this is just legal boilerplate” or “the distro will patch it out”, just because it is Mozilla. I dislike Google as much as the next guy, but if this was Google, people would not have given the benefit of the doubt as much as they are doing now.
I am tired of companies like Mozilla, Duckduckgo et al. getting a free pass in the FOSS community, even when the last few years clearly have shown that they use privacy as a publicity stunt.
It is legal boilerplate. I would make the same comment if EvilCorp had the same clause in their terms.
And I am sick and tired of people trying to find pointless things to be outraged about. There are enough actual bad things in the world. You don’t need to go inventing fake ones.
Do you think this is a “signal” of a “direction” the Python Package Index and/or the Python Software Foundation is moving?
It seems to me that, contrary to the claim above, Mozilla isn’t getting a “free pass” — Mozilla is getting far more scrutiny and far less benefit of the doubt than other entities doing the same thing. And for no rational reason I can discern.
Well there are differences and similarities. Compare:
If you upload Content other than under an Included License, then you grant the PSF, its legal successors and assigns, and all other users of the Service an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, and publish the Content, including in digital form.
vs
When you upload or input information through Firefox, you hereby grant us a nonexclusive, royalty-free, worldwide license to use that information to help you navigate, experience, and interact with online content as you indicate with your use of Firefox.
The PyPI text is clearly about presenting the content in a way that could be construed as copyright infringement. The Firefox text has the much broader notion of use, which could mean anything including using the data to build a profile and selling it to partners.
Another comparison:
We reserve the right, at our sole discretion, to amend these Terms of Service at any time and will update these Terms of Service in the event of any such amendments. We will notify our Users of material changes to this Agreement, such as price increases, at least 30 days prior to the change taking effect by posting a notice on our Website or sending email to the primary email address specified in your PyPI account.
vs
Every once in a while, Mozilla may decide to update these Terms. We will post the updated Terms online. We will take your continued use of Firefox as acceptance of such changes. We will post an effective date at the top of this page to make it clear when we made our most recent update.
These have more in common. Both agreements grant the service the right to quietly update the agreement on their website without notification or obtaining any further consent. PyPI didn’t have this provision in their previous terms either, and should not be given the benefit of the doubt for it IMO.
I’m not the person you asked, but I think it’s a signal that the organization is considering diversifying their revenue streams at the expense of user rights. It’s also an indication that they’re spending money on lawyers that could be spent on the service. Does that help?
It has a very similar clause stating you grant them a license to things you send to them.
But this new Firefox TOS isn’t about data I might send to Mozilla, it covers “input information through Firefox”. And the supposedly lawful bases of data gathering they have in their privacy notice are both vague and broad.
If my comment about patching it out came across as dismissive, I assure you it was not intentional. (In fact, I’m concerned that Debian will fail to do so.) I count myself lucky to be insulated from the worst of this shit by my distro, but the fact is that most people aren’t, and even if it doesn’t affect me personally I see it as a serious problem (but not a surprise, given Mozilla’s recent track record).
Thanks for clearing that up - your comment was indeed a bad example to pick. Clearly it would have been better to pick the one sarcastically leaning towards “It’s not like Mozilla will feed your input to an AI”, completely oblivious to the fact that five of the articles on the frontpage of the Mozilla blog are about AI.
If you use a distro-packaged build of Firefox that comes from competent packagers, this will certainly be stripped out. Unfortunately it’s not clear to me that said competent packagers exist. Debian in most cases is very good at this kind of thing, but the Debian packager for Firefox is employed by Mozilla and has a terrible track record for removing even the most egregiously unpopular shit from Firefox.
Debian in most cases is very good at this kind of thing, but the Debian packager for Firefox is employed by Mozilla and has a terrible track record for removing even the most egregiously unpopular shit from Firefox.
I would suspect that “having a terrible track record for removing even the most egregiously unpopular shit” is likely a direct prerequisite of Debian’s permission to use the Firefox branding.
IOWs, if stripping-out ever happens, in any distro, the result will have to be called Iceweasel.
These Terms only apply to the Executable Code version of Firefox, not the Firefox source code.
…the “Executable Code” version of the Firefox web browser, which is the ready-to-run version of Firefox from an authorized source that you can open and use right away.
It sounds like this wouldn’t apply to distro packages, depending on what the undefined “authorized sources” are.
It’s the terms of service. You don’t have to agree to terms of service to use (read, modify, etc.) the source code, because the source code in itself does not offer any services.
You only have to agree to the terms of service to use a binary that presumably has the API keys (that would enable the binary to communicate with the provider of the service) compiled in. Whether you need to agree to these terms to use a given binary probably depends on whether it has Mozilla’s API keys compiled in.
Yes, and the would-be benevolent packager would take these there sources, which provide no services, and build his own executable code.
The question though is, did they need to accept the terms to download the sources? If the packager never needed to see the terms and didn’t agree to them, do the terms apply to this other binary?
The question though is, did they need to accept the terms to download the sources? If the packager never needed to see the terms and didn’t agree to them, do the terms apply to this other binary?
I’d expect that the packager would have to agree to the terms of service (and, transitively, agree to bind their users to those terms) in order to get the Mozilla API keys, or something like that.
Debian in most cases is very good at this kind of thing, but the Debian packager for Firefox is employed by Mozilla and has a terrible track record for removing even the most egregiously unpopular shit from Firefox.
Can you elaborate and/or provide a reference? The only identifier I could find for the packager was “Maintainers of Mozilla-related packages”:
It would also be good to get some clarity on the channels that could be used to ensure Debian is following its principles. I know they pride themselves on some form of internal democracy.
If you know of git-annex or any of his other work, you’ll know Joey Hess has been deeply involved in Debian in the past, but admittedly it’s not a super thorough source.
Of course they didn’t really answer the question “why now?” What is it about the “much different technology landscape” that makes the rights that users have by default under the law untenable?
Why now? I suppose Google, after overbuilding AI capacity, has them at financial gunpoint about feeding user data to their slop machines.
There is nothing stopping Mozilla Corporation, or even the employees themselves, from moving away from a for-profit model and towards a non-profit model like the Wikimedia Foundation. The only people holding a financial gun to Mozilla’s head are Mozillans.
300 million of which is dedicated to all Program expenses
(260 million of which is dedicated to “software development,” as per page four; presumably some combination of the line-item expenses from page five adds up to that number).
Alas, these numbers reflect the Mozilla Foundation as a whole, and not Mozilla Corp specifically, but it’s likely that the lion’s share of “software development” expenses in the Foundation fall under the Mozilla Corp bucket (if that isn’t the case, then the situation would look a bit rosier, I suppose). Administration expenses come out to 192 million. Administration percentage of expenses is 38.7%.
Wikimedia Foundation’s 2023-2024 revenue shows total revenue of 177 million, with 145.5 million coming from:
“online emails” (38 million)
“online recurring” (33 million)
“banner and other” (74.5 million).
Their expenses bucket 137.8 million for Programmatic (100 million of which is personnel), 17.9 million for Fundraising, and 21.3 million for General and Admin. Administration percentage of expenses is 12%.
Being pessimistic, and assuming Firefox could only ever match Wikimedia Foundation’s 145 million in revenue from individual users (which excludes large grants or other enterprise contracts), that leaves a large gap no matter how you slice it. Mozilla Foundation’s admin expenses are absurd, but even excluding them entirely and only focusing exclusively on software development Program expenses (260 million) leaves a 115 million dollar gap. Mozillans would need to cut their annual compensation in half to even come close to closing that gap in such a scenario (they’d still be about 15 million in the hole, and that’s still optimistic as it excludes any admin or other expense).
So, can it be done? That depends. If Mozillans didn’t want to take any pay cuts at all, then they’d need to get serious about monthly recurring revenue to make that happen. Firefox averages 150 million MAU (monthly active users). Taking the Wikimedia Foundation’s “if every user donated X this year” approach, that comes out to at least two dollars per user each year to get revenue of 300 million, which would cover Program expenses at least. Aiming for three dollars would give them 450 million annual revenue. A few different ways to hit that 450M target:
3 USD/year (0.25 USD/month) for 150M users (100% of users)
30 USD/year (2.5 USD/month) for 15M users (10% of users)
300 USD/year (25 USD/month) for 1.5M users (1% of users)
3000 USD/year (250 USD/month) for 0.15M users (0.1% of users)
I suspect 25 USD/month is on the upper end of what significant numbers of folks would be willing to pay for. And I’m not convinced they could find 1.5 million monthly customers to do it. Meaning that whatever the shortfall was, they’d have to find elsewhere. I think they could find that shortfall in grant funding from governments, especially if said governments were genuinely interested in breaking up Alphabet’s monopoly.
Basically, I don’t think it’d be impossible. I don’t think it’d be easy either. But I definitely think it is necessary for the future of the web.
At that point, I would seriously root for open democratic governments to bank-roll that non-profit model.
It would be a win-win, protecting those democracies and the common-goods internet that democracy needs to function.
There is an occasional comment or article suggesting that the EU should just fork Firefox and take over financing completely. I guess they could fine there Mozilla Foundation for gdpr breaches (e.g. with MDN) to bootstrap the first year.
That’s what the idea essentially conveys. Take over Firefox and the devs and run it via some actual non-profit, fully funded by the EU and not by the businesses.
Not saying it’s a good idea necessarily, but I often remember it.
It could also have something to do with the chaos being wrought by the new administration, causing many other worse things for people to pay attention to and creating opportunities to ratchet up unpopular things like the erosion of user rights with less blowback.
When you upload or input information through Firefox, you hereby grant us a nonexclusive, royalty-free, worldwide license to use that information to help you navigate, experience, and interact with online content as you indicate with your use of Firefox.
That’s… wow. Thank you for highlighting that. I am seriously considering using something other than Firefox for the first time in… ever. Regardless of how one might choose to interpret that statement, it’s frightening that they would even write it. This is not the Mozilla I knew or want. I’d love to know what alternatives people might suggest that are more community focused and completely FOSS, ideally still non-Chromium.
Isn’t it? Most GDPR consent screens have an easy “accept to everything” button and requires going through multiple steps to “not accept”, and many many more steps to “object” to their “legitimate interest” in tracking for the purposes of advertising. As long as these screens remain allowed and aren’t cracked down on (which I don’t foresee happening, ever), that’s the de facto meaning of “consent” in GDPR as far as I’m concerned: something that’s assumed given unless you actively go out of your way to revoke it.
It’s not what the text of the GDPR defines it as, but the text isn’t relevant; only its effect on the real world is.
Yes, definitely. Consent in GDPR is opt-in not opt-out. If it’s opt-out, that’s not consensual. And the law is the law.
Furthermore, for interstitials, to reject everything should be at least as easy as it is to accept everything, without dark patterns. Interstitials (e.g., from IAB and co.) first tried to make it hard to reject everything, but now you usually get a clear button for rejecting everything on most websites.
As I mentioned in another comment, the DPAs are understaffed and overworked. But they do move. A real-world example of a company affected by the GDPR, and that tries testing its limits, is Meta with Facebook. For user profiling, first they tried the Terms of Service, then they tried claiming a legitimate interest, then they introduced expensive subscriptions for those that tried to decline, now they introduced a UI degradation, delaying the user scrolling, which is illegal as well.
Many complain, on one hand, that the EU is too regulated, suffocating inovation, and with US’s tech oligarhs now sucking up to Trump to force the EU into allowing US companies to break the law. On the other hand, there are people who believe that the GDPR isn’t enforced enough. I wish people would make up their mind.
Many complain, on one hand, that the EU is too regulated, suffocating inovation, and with US’s tech oligarhs now sucking up to Trump to force the EU into allowing US companies to break the law. On the other hand, there are people who believe that the GDPR isn’t enforced enough. I wish people would make up their mind.
Those are different people, all who have made up their mind.
I thought I made it reasonably clear that I don’t care that much about what the text of the law is, I care about what material impact it has on the world.
To be fair, @mort’s feeling may come from non-actually-GDPR-compliant cookie consent forms. I have certainly seen where I couldn’t find the “reject all” button, and felt obligated to manually click up to 15 “legitimate interest” boxes. (And dammit could they please stop with their sliding buttons and use actual square check boxes instead?)
The facts you provided aren’t relevant. I’m talking about the de facto situation as it applies to 99% of companies, you’re talking about the text of the law and enforcement against one particular company. These are different things which don’t have much to do with each other.
You even acknowledge that DPAs are understaffed and overworked, which results in the lacking enforcement which is exactly what I’m complaining about. For what I can tell, we don’t disagree about any facts here.
I’m talking about GDPR as well, focusing about what impact it has in practice. I have been 100% consistent on that, since my first message in this sub-thread (https://lobste.rs/s/de2ab1/firefox_adds_terms_use#c_3sxqe1) which explicitly talks about what it means de facto. I don’t know where you got the impression that I’m talking about something else.
But there is enforcement, it’s just slower than we’d like. For example, screens making it harder to not opt in rather than opt in have gotten much rarer than they used to be. IME now they mostly come from American companies that don’t have much of a presence in the EU. So enforcement is causing things to move in the right direction, even if it is at a slow pace.
There is a website tracking fines against companies for GDPR violations [1] and as you can see, there are lots of fines against companies big and small every single month. “Insufficient legal basis for data processing” isn’t close to being the most common violation, but it’s pretty common, and has also been lobbed against companies big and small. It is not the case that there is only enforcement against a few high profile companies.
it’s the other way around - most of the time you have to actively revoke “legitimate interest”, consent should be off by default. Unfortunately, oftentimes “legitimate interest” is just “consent, but on by default” and they take exactly the same data for the same purpose (IIRC there are NGOs (such as NOYB, Panoptykon) fighting against IAB and other companies in those terms)
“Legitimate interest” is the GDPR loophole that ad tech companies use to spy on us without an easy opt-out option, right? I don’t know what this means in this context but I don’t trust it.
It is not, ad tech has been considered not a legitimate interest for… Ever… By the Europeans DPAs. Report to your DPA the one that abuse this. There have been enforcement.
Every website with a consent screen has a ton of ad stuff under “legitimate interest”, most ask you to “object” to each individually. The continued existence of this patterns means it’s de facto legal under the GDPR in my book. “Legitimate interest” is a tool to continue forced ad tracking.
I don’t think you’re disagreeing with me. It’s de jure illegal but de facto legal. I don’t care much what the text of the GDPR says, I care about its material effect on the real world; and the material effect is one where websites put up consent screens where the user has to “object” individually to every ad tech company’s “legitimate interest” in tracking the user for ad targeting purposes.
I used to be optimistic about the GDPR because there’s a lot of good stuff in the text of the law, but it has been long enough that we can clearly see that most of its actual effect is pretty underwhelming. Good law without enforcement is worthless.
De facto illegal for entities at Facebook’s scale? Maybe. But it’s certainly de facto legal for everyone else. It has been 7 years since it was implemented; if it was going to have a positive effect we’d have seen it by now. My patience has run out. GDPR failed.
I just gave you a concrete example of a powerful Big Tech company, with infinite resources for political lobbying, that was blasted for their practices. They first tried hiding behind their Terms of Use, then they tried claiming a legitimate interest, then they offered the choice of a paid subscription, and now they’ve introduced delays in scrolling for people that don’t consent to being profiled, which will be deemed illegal as well.
Your patience isn’t important. This is the legal system in action. Just because, for example, tax evasion happens, that doesn’t mean that anti tax evasion laws don’t work. Similarly with data protection laws. I used to work in the adtech industry. I know for a fact that there have been companies leaving the EU because of GDPR. I also know some of the legwork that IAB tried pulling off, but it won’t last.
Just the fact that you’re getting those interstitials is a win. Microsoft’s Edge browser, for example, gives EU citizens that IAB dialog on the first run, thus informing them that they are going to share their data with the entire advertising industry. That is in itself valuable for me, because it informs me that Edge is spyware.
I agree that the “we’re spying on you” pop-ups is a win in itself. I’m just complaining that it’s so toothless as to in practice allow websites to put up modals where each ad tech company’s “legitimate interest” in tracking me has to be individually disabled. If the goal of the GDPR was to in any way make it reasonably easy for users to opt out of tracking, it failed.
I agree that the “we’re spying on you” pop-ups is a win in itself.
I’m not so sure. I’ve even seen this used as an argument against the GDPR: The spin they give it is “this is the law that forces us to put up annoying cookie popups”. See for example this article on the Dutch public broadcasting agency (which is typically more left-leaning and not prone to give a platform to liberals).
“Alle AI-innovaties werken hier slechter dan in de VS. En waarom moet je op elke website op cookies klikken?”, zegt Van der Voort.
Roughly translated “all innovations in AI don’t work as well here as in the US. And why do you have to click on cookies (sic) on every single website?”
I’ve even seen this used as an argument against the GDPR: The spin they give it is “this is the law that forces us to put up annoying cookie popups”.
I have seen that as well, and I think it’s bullshit. The GDPR doesn’t force anyone to make any form of pop-up, nobody is forced to track users in a way which requires consent. The GDPR only requires disclosure and an opt-out mechanism if you do decide to spy on your users, which I consider good..
The GDPR only requires disclosure and an opt-out mechanism if you do decide to spy on your users, which I consider good..
I agree, but at the same time I think the average user just sees it as a nuisance, especially because in most cases there’s no other place to go where they don’t have a cookie popup. The web development/advertising industry knowingly and willfully “complied” in the most malicious and obnoxious way possible, resulting in this shitty situation. That’s 1 for the industry, 0 for the lawgivers.
I agree that it didn’t have the desired effect (which, incidentally, I have spent a lot of this thread complaining about, hehe). I think everyone was surprised about just how far everyone is willing to go in destroying their website’s user experience in order to keep tracking people.
has to “object” individually to every ad tech company’s “legitimate interest” in tracking the user
I’m not sure if you’re deep in grumpy posting or didn’t understand the idea here, but for legitimate interest you don’t need to agree and companies normally don’t give you the option. If you’re talking about the extra options you unset manually, they’re a different thing. The “legitimate interest” part is for example validating your identity through a third party before paying out money. Things you typically can’t opt out of without also refusing to use the service.
If you get a switch for “tracking” or “ads” that you can turn off, that’s not a part of the “legitimate interest” group of data.
I’m sorry but this isn’t true. I have encountered plenty consent screens with two tabs, “consent” and “legitimate interest”, and where the stuff under “consent” are default off while the stuff under “legitimate interest” is on by default and must be “objected to” individually. Some have an “object to all” button to “object” to all ad tracking in the “legitimate interest” category.
Here’s one example: https://i.imgur.com/J4dnptX.png, the Financial Times is clearly of the opinion that tracking for the purpose of advertising counts as “legitimate interest”.
I’m not saying that there’s any relationship between this pattern and what’s actually required by the GDPR, my understanding of the actual text of the law reflects yours. I’m saying that this is how it works in practice.
Mozilla updated the article with a clarifying statement:
UPDATE: We’ve seen a little confusion about the language regarding licenses, so we want to clear that up. We need a license to allow us to make some of the basic functionality of Firefox possible. Without it, we couldn’t use information typed into Firefox, for example. It does NOT give us ownership of your data or a right to use it for anything other than what is described in the Privacy Notice.
the problem is it doesn’t clarify anything. “basic functionality” is not defined. my guess is they want to be able to feed anything we type or upload to a site, to also be able to feed that into an LLM. “anything other than what is described” doesnt help because what is described is so vague as to mean anything “help you experience and interact with online content”
Mozilla updated the article with a clarifying statement:
UPDATE: We’ve seen a little confusion about the language regarding licenses, so we want to clear that up. We need a license to allow us to make some of the basic functionality of Firefox possible. Without it, we couldn’t use information typed into Firefox, for example. It does NOT give us ownership of your data or a right to use it for anything other than what is described in the Privacy Notice.
That is… not clarifying. And not comforting. “What is described” in the ToS is “to help you navigate, experience, and interact with online content.” That’s absurdly vague. And what is described in the Privacy Notice is absurdly broad:
To provide you with the Firefox browser
To adapt Firefox to your needs
To provide and improve search functionality
To serve relevant content and advertising on Firefox New Tab
To provide Mozilla Accounts
To provide AI Chatbots
To provide Review Checker, including serving sponsored content
To enable add-ons (addons.mozilla.org, “AMO”), including offering personalized suggestions
To maintain and improve features, performance and stability
To improve security
To understand usage of Firefox
To market our services.
To pseudonymize, de-identify, aggregate or anonymize data.
To communicate with you.
To comply with applicable laws, and identify and prevent harmful, unauthorized or illegal activity.
I’m glad we have this contextless legalese to clarify things. I wonder if there’s some kind of opt-in data collection in Firefox that Mozilla might have legal obligations to clarify their rights to? Couldn’t be that… No, let’s put a pause on critical thinking and post stupid TOS excerpts as if Mozilla are going to steal our Deviantart uploads and sell them as AI training data.
I’m glad we have this contextless legalese to clarify things. I wonder if there’s some kind of opt-in data collection in Firefox that Mozilla might have legal obligations to clarify their rights to? Couldn’t be that… No, let’s put a pause on critical thinking and post stupid TOS excerpts as if Mozilla are going to steal our Deviantart uploads and sell them as AI training data.
If they need a ToS for a particular feature, then that “contextless legalese” should be scoped to that feature, not to Firefox as a whole.
This is precisely why the same organization should not do all of these things. If they want to do non-tool stuff to continue funding their mission they should start up independently managed companies that can establish these consents for a narrow band of services. They can give the existing organization control as a majority shareholder, with dividends flowing back to the main organization. That is the way to ensure that incentives don’t become misaligned with the mission.
I don’t know where the author was during the late twenty-teens, but I recall the HTTPS revolution being quite noisy with the whole “ENCRYPT ALL THE THINGS!” being screamed from the roof tops. Also, does this mean I need HTTPS for connecting to 127/8 and 192.168/16 networks? I either need to expose the internal network or run my own CA (and if I’m correctly listening to “security experts” is something I should not do)?
I know but since this month I both got a new job - with crappy WiFi in office - and a Mac to go with it - so I’m affected. I don’t use these networks at home usually :)
Even it’s the only thing, it’s still open. Whereas now nothing is open. It may be low risk, but compared to 0, it’s a lot.
Another thing though is that someone needs to set that up. I did configure my own bind instance 20 years ago, personally and professionally, but most people don’t have a clue how to set up WiFi semi-securely.
A third thing here is that someone now needs to teach the end users. I’m a professional software engineer and I’ve seen many developers that have no clue how any of this works. They couldn’t configure an nginx without training and time.
Note that I’m not judging the idea of forced HTTPS either way, I’m just pointing out some of the obvious costs.
The DNS server used by the challenge doesn’t have to be on your LAN.
Basically to host a LAN accessible service on thing.home.example.com, I do:
Get an ACME cert by proving I control the domain with my usual hosted DNS provider
Have my LAN DNS return the LAN IP hosting thing.home.example.com
This is of course all automated in my setup. And I actually use a wildcard cert to not leak the list of services I host in the cert transparency logs. The security downside of a wildcard doesn’t matter to me in this case.
With plenty of other Western governments hungry to attack digital privacy under their current administration, this is going to prove to be letting camel’s nose into the tent. They will be lining up to be the next customer of this feature.
Personally I worry more about the cost (and not just energy cost) of training models than the cost of inference, especially for models where inference can be run from a laptop – a single inference is not going to consume much energy, because laptops are reasonably low-power tools. Fine-tuning an existing model looks to be feasible in reasonable amount of compute (and thus energy), but training a new foundational model from scratch is counted in TWh range these days, has been increasing rapidly in the last few years, as is something that all large tech companies (and all technologically advanced states, all militaries, etc.) are going to want to learn to do to remain technologically independent and avoid poisoning issues that nobody knows how to detect.
Said otherwise: using an LLM is not that expensive energy-wise. But collectively getting used to depending on LLMs more an more gives money and power to entities that can train new ones, which incentivizes a lot more training, and that is expensive.
I’m not sure how to estimate this total training energy cost – as previously discussed, the easiest estimate I could think of is to look at data-center GPU sales to estimate the global (training + inference) power usage – and we are looking at millions of new units sold each year, which translate into several GW (several nuclear plants worth) of instantaneous consumption – and the current trends is that this is going up, fast.
If you read through the cited article by Husom et al. you’ll see that a laptop actually uses more energy than a better specced workstation or server. I would guess the main reason is that it takes longer to process and thus stays at an increased power draw for longer.
Also consider the fact that training a model is a one-off energy cost, that gets less severe the more the model is used. For inference, the opposite is true.
About laptops, I think we agree. I pointed out that if we can reasonably run inference from a laptop, then that means a single inference is reasonably cheap – even cheaper on specialized hardware.
Training may be a one-off energy cost for a given model, but that does not mean that it is small. From the post that we are commenting on, it looks like training one model is of the order of 10^12 times more energy-consuming that running one user query (ChatGPT: 1.3TWh of training, 2.7Wh per query). Do most models actually get a thousand billion queries over their lifetime, before they are dropped to use a better model? ChatGPT appears to serve one billion queries per day these days; at this rate they would need to run for 500 days on the same model to break even. From a distance it appears that they are not breaking even, they change the foundational models more often than every 500 days – but maybe they avoid retraining them from scratch each time? And this is going to be much worse the more entities start training their own foundational model (there are already more companies doing this than just OpenAI, including those we don’t know about), as they will each get a fraction of the userbase.
Okay, then it seems like we mostly agree. All in all, it is very difficult when the only information one has are estimations like I made in this post, and not any actual information. “Open” AI really is a misnomer.
Doesn’t this analysis show that training cost is still not that important? Even if they only use a model for 100 days, that means that training cost is 5x inference cost, but inference cost is tiny, and 6x tiny is still tiny. If, say, drinking an extra cup of coffee costs a comparable amount of energy as typical LLM use including training, that seems entirely reasonable.
I suspect that future growth is fairly well limited by the payoff it is bringing us. Investors will bet a couple of billion dollars on something that may not pay off, but it seems unlikely that they’d invest several trillion without solid evidence that it will pay off in terms of value generated.
I suspect that future growth is fairly well limited by the payoff it is bringing us.
The problem is that we don’t understand yet what the payoff is, and the natural response in this case is to experiment like crazy. The LLM boom came from the fact that we got substantial quality improvements by making models much larger and training them a lot more. The next thing to try of course is to give them a lot more data again (scraped all the textual web? maybe we could train them on all of Netflix this time!), and to train them a lot more again. There is a very real risk that, three years from now, half the R&D budget in the world is being spent powering GPUs to do matrix multiplications. (It would also be bad for the environment to spend the same amount roasting and grinding coffee beans, but that does not make me feel particularly better about it.) It is of course possible that this risk-taking pays off and we are ushered in a new world of… (what? robot-assisted comfortable living for all?), but it is also possible that current techniques reach a plateau and we only notice after burning way too much fuel on them.
The payoff is also not an objective metric, it depends in a large part on how people view these things. As long as people are convinced that LLMs are godlike powers of infinite truth, they are going to pay for them, and the payoff will be estimated as fairly large. If we try to have a more balanced evaluation of the benefits and the costs of those tools, we might take the rozy glasses off (low confidence here, some people still believe in the blockchain), and the payoff – and incentive to spend energy on it – will be smaller.
Sometimes it looks like the only thing that can force people to reduce pollution is not data, information or basic common sense, but just energy becoming fucking expensive. I suspect that this would definitely make people behave more rationally about LLMs.
Blockchain was never meaningfully useful; it was (and is) a speculative bubble. LLMs already are useful, and millions of people use them. Spending a lot of R&D money on it is entirely justified, and this R&D is mostly privately funded anyway. The glasses aren’t at all too rosy, in my view, rather the opposite. Sure, there are AI hypers that have highly inflated expectations, but most people underestimate what LLMs can do.
Even the basic ability to have a conversation with your computer would have been considered absolute sci-fi 5 years ago. These things type out hundreds of lines of code in a few seconds, often without making any mistakes, without ever even running that code. When I gave o3-mini-high some math puzzle I came up with, it solved it in a minute. Out of colleagues and students, only one was able to solve it. Maybe it existed on the internet somewhere, and maybe it was just a glorified fuzzy lookup table, but fuzzy lookup tables are very useful too (e.g., Google).
Yesterday I “wrote” 98% of the code of a UI for some decision procedure that I was working on by literally talking to my computer out loud for an hour. It compiled the Rust decision procedure to WASM, set up a JS api for it, created the interface, wrote a parser, generated examples based on the Rust unit tests, etc. I wrote maybe 20 lines total myself. Heck, I haven’t even read 80% of the code it wrote.
This is all for some infinitesimally small fraction of the energy use of various frivolous things people regularly do. This seems to me like getting a functional magic wand from the Harry Potter world and then complaining that a branch of a tree had to be cut off to get the wood. Yes, sure, we don’t want to cut down the entire Amazon rainforest to make magic wands, nor may the magic wand solve all of our problems. But the more pertinent fact here is that it does magic, not that a branch was cut off from a tree.
I don’t disagree. I should clarify that my worry is not, per se, that people invest too much effort in LLMs – I think of them as the alchemy of the modern times, a pre-scientific realm of misunderstandings and wonders, and I find it perfectly understand that people would be fascinated and decide to study them or experiment with them. My worry is that this interest translates into very large amount of energy consumption (again: half the R&D money could be spent powering machine learning hardware), as long as the dominant way to play with neural networks is to try make them bigger. Most sectors of human activity have been trying to reduce energy consumption in the face of expanded usage, but the current AI boon is a sector that has been increasing its energy needs sharply, and shows no signs of stopping. The availability of clean-ish energy has not scaled up for that demand.
I do also think that there are unreasonable expectations about AI in general – people turning to neural networks for tasks that could reasonably be done without them, especially in industry. For instance a friend working in a garbage-collecting plants was telling me about futile attempts by the company to train neural networks to classify garbage from a video feed, which were vastly underperforming simple heuristics that were already in place.
The next thing to try of course is to give them a lot more data again (scraped all the textual web?
I think this isn’t the case.
My layman understanding of the consensus is that people agree that there are limits for scaling based on input data. We maybe aren’t there yet (new data sets such as video are probably still ripe) but I don’t think we’re far, personally.
DeepSeek’s whole thing is that it’s a much smaller, faster model, with competitive performance to vastly larger ones - all because they changed the training process.
We’ve already observed optimization on model sizes, training costs, etc, which is unsurprising since companies are incentivized to do this (faster training == faster iteration == money).
energy becoming fucking expensive.
This is tricky from a policy perspective (raising the cost of all energy consumption seems sort of obviously bad, but we’ve seen things like “off-grid hours” work, etc) but regardless energy costs are, of course, already passed on to consumers. Perhaps VC money is making the difference right now, but ultimately consumers eat the costs of electricity with a margin, otherwise companies wouldn’t be able to stay up and running.
It is one-off, but I think it’s a bit misleading - the commenter above mentions this exactly: more and more entities are training more and more (and more powerful) models. So while it is a one-off cost, it’s also constant.
using an LLM is not that expensive energy-wise. But collectively getting used to depending on LLMs more an more gives money and power to entities that can train new ones, which incentivizes a lot more training, and that is expensive.
You’re not wrong but that’s how pretty much everything works. Isn’t it disingenuous to focus on LLMs in this regard?
I understand that it’s easier as it’s the new thing but we should do environmental impact analysis on everything else, too. How much do we spend making video games? How much more do we spend playing them? How about the whole of TV? Movies? Books? Fast fashion? Cars? Housing? Food? I haven’t seen much of the backlash on environmental impact grounds to anything else along with LLM.
And that makes a valid argument in isolation feel like another attack blown out of proportion. Sure, training data set is dubiously sourced. Sure, there are all sorts of moral questions about the use of LLM output. Let’s tack on environmental issues as well, why not? And let’s not bring it up for anything else to make it look like LLMs alone will drain all our water and use all the energy. Rings a bit hollow.
You’re not wrong but that’s how pretty much everything works. Isn’t it disingenuous to focus on LLMs in this regard?
The specificity of LLMs is that they are a new technology that is exploding in popularity and is highly energy-consuming. This is not unique but it is still fairly rare. People where also very concerned about the energy cost of blockchains (and still are), which were in a similar situation, at least when proof-of-work was the only scheme widely deployed. Of course computing-focused websites like Lobsters are going to discuss the energy impact (and environmental impact) of computer-adjacent technologies much more than non-computer-related things like cars and housing – this is the area that we have some expertise in and some agency over. Digital technology is also one of the main drivers of increased energy consumption and pollution these days – right now it is comparable to transportation, but growing fast every year, while the impact of most other areas of human activity are increasing less fast, stable or even decreasing. (This could of course become much worse with a few bad policy decisions.)
How about the whole of TV? Movies? Books? Fast fashion? Cars? Housing? Food? I haven’t seen much of the backlash on environmental impact grounds to anything else along with LLM.
This reads like whataboutism. I don’t know what your cultural and political context is, but around me there has been a lot of discussions of the impact of cars (why are people pushing for electric cars?), housing (entire laws have been passed, for example to force landlord to improve thermal insulation), food (why do you think is the proportion of flexitarian / vegetarian / vegan people growing fast in western countries?), planes (why are people talking about reducing flights to limit their carbon impact?), etc. You may live in a different environment where these discussions have been occulted, or have personally decided not to care about this and not get informed on the matter.
And that makes a valid argument in isolation feel like another attack blown out of proportion.
More facts, less feelings, please. If you can demonstrate that the energy/pollution costs of LLM is neglectible compared to other reasonable causes of concerns, that would be a very valid way to say that this is out of proportion. We could have more, better data about this, but there is already enough available to make ballpark estimates and make an informed decision. (For example, see this december-2024 article reporting that US data-centers energy consumption tripled over the last decade, consumed 4.4% of total US electricity in 2023, and that this proportion is expected to raise to 7%-12% by 2028.)
If you can demonstrate that the energy/pollution costs of LLM is neglectible compared to other reasonable causes of concerns, that would be a very valid way to say that this is out of proportion.
Well, that’s exactly the point I’m trying to make. Environmental impact concerns around LLMs never given in any context relating to other things. Sure, data is out there. I can bust out a spreadsheet and put in all the data and crunch the numbers and find the cost of my LLM query in burgers, ICE miles, books, TV minutes, or—like in the OP—shower minutes.
The issue in the discourse quality I’m trying to point out is that I have to go out of my way and do my own research. A valid critique of LLMs is rarely if ever put into an understandable context. And this is why I think it largely has no impact. What you call whataboutism is better reframed as putting things in context. Is 1.5 l of water for may LLM query a lot? An average person doesn’t know what to do with that number. They have nothing to relate it to. Maybe an hour of TV in the background worse? Maybe an hour of playing a game on their maxed out gaming PC is better? How would they know?
I thought this article was going to be in dialog with the referred article, maybe expanding or critiquing it, but it literally copied many of the same subheadings verbatim, and a lot of the content is similar, just reworded.
Is this some SEO thing for backlinks and reputation?
I hate to be cynical, but the posts are extremely similar, and I’m not sure why bitfield submitted this one, instead of his own.
One of my students writes a blog post about software testing, citing an article by me. I share it here, and that’s somehow… suspicious?
I promise you Jakub is a real person (you can find many videos of him giving talks, for example) and has his own views on this stuff: some of them aligned with my own, some not. Many of my other students write blog posts, too, and I don’t apologise for sharing and boosting them where I think their work deserves a larger audience. If you don’t agree, don’t upvote them, but let’s give the ‘just asking questions’ routine a rest, shall we?
The suspicious part was how incredibly similar the articles were, down to structure, order, and including “literally copied many of the same subheadings verbatim”. It doesn’t look naturally written at all. It looks like if your student sat down with your post and retyped it, changing small bits as they went along.
If not for the unusually high similarity, I wouldn’t have batted an eyelash.
Most of Asahi’s linux work is in Rust, such as their new GPU drivers for Apple Silicon. Part of that requires access to the DMA subsystem. An asahi maintainer other than Marcan wanted to upstream Rust bindings to DMA. These would have been part of the “Rust for Linux” subsystem, which explicitly is exempted from the usual kernel stability guarantees. The maintainer of DMA in Linux objected to them upstreaming those bindings, as he felt he would then be obligated to maintain the Rust code, and he did not want to maintain Rust. This was combined with some uncharitable comments on the Rust for Linux project’s goals. There has been a lot of debate on other internet forums about whether these comments were uncharitable about Rust, Rust for Linux, or some other distinction that personally I think doesn’t really matter.
Marcan and that maintainer then exchanged words on the LKML. Unsatisfied, Marcan appealed to his Mastodon for support. Linus then stepped in with a reply heavily condemning the appeal to social media and not really addressing the complaints that led to that point. Marcan later resigned from his kernel maintainerships, and now also from the Asahi project.
I think this understates the position of the maintainer. He sent a message to LKML that states it pretty clearly: he thinks a multi-language kernel is a terrible mistake. And I can understand his position — which to me demonstrates that there is a leadership failure happening, because this kind of fundamental conflict shouldn’t be allowed to go on this long. “Disagree and Commit”, as they say at Amazon.
Every additional bit that the another language creeps in drastically reduces the maintainability of the kernel as an integrated project. The only reason Linux managed to survive so long is by not having internal boundaries, and adding another language complely breaks this. You might not like my answer, but I will do everything I can do to stop this. This is NOT because I hate Rust.
…
The common ground is that I have absolutely no interest in helping to spread a multi-language code base. I absolutely support using Rust in new codebase, but I do not at all in Linux.
Since there hasn’t been a reply so far, I assume that we’re good with
maintaining the DMA Rust abstractions separately.
Hence, the next version of this patch series will have the corresponding
maintainer entry.
Danilo
I.e. after the explicit statement that the rust people, not Hellwig, would be maintaining it separately.
(And then we reach the post you quoted, where he makes it clear that his reasons are that he doesn’t want Rust in Linux at all, not that he doesn’t want to maintain it)
Yes, I understated the understatement. :) His stated goal is to use his influence and control over the DMA subsystem to prevent Rust (edit: or any other non-C language), from being in the kernel at all. As his NAK says:
If you want to make Linux
impossible to maintain due to a cross-language codebase do that in
your driver so that you have to do it instead of spreading this cancer
to core subsystems. (where this cancer explicitly is a cross-language
codebase and not rust itself […])
Classic LKML rhetorical style here, to use a loaded word like “cancer” to make what is actually a reasonable technical point.
It is not a reasonable technical point! It demonstrates a fundamental incuriosity about technical matters. Categorical technical statements are rarely reasonable.
I disagree in general, but not in specifics. I’ve spoken to Chris Hellman at lengths (literally 6 hours) about the topic of Rust and it was not combative. He was curious. It’s hard to sum up at length, but his disagreements are rooted in specifics and he’s a hardliner about the single source language thing. But not for the sake of being a hardliner. His hard-line stance comes out of different weighting of the issues at hand, not an outright dismissal.
Those people existing is to be expected and it doesn’t invalidate their opinions.
Obviously not a fan of LKML rhetoric, but yep, you get the tone you built over years.
You can disagree, but for the rest of us who haven’t had these super private, intimate exchanges, where we get to “really know” someone, what he puts into the public is our perception of him. To that end, he’s called the project a cancer, and declared he’s going to do whatever he can to stop it which is not what a curios person does. He comes across instead as a religious zealot or “hardliner” in your terms.
That’s fair, but it’s also a categorical mistake to read a (brusque and inappropriate) email and judge on a whole person for days and weeks on the internet. I’m not saying you “really need to know someone”, but you also need to be aware that hundreds of hours are spent on reading what those statements may mean.
I’ve personally had to make the decision whether to allow an additional language into a large codebase, and there are some major negatives to consider that are unrelated to the technical aspects of the language. The maintainer said three times in the above excerpts that he doesn’t have a negative opinion of Rust itself.
I’ve always heard that Google allows only four languages in their codebase, and one of those is only because they invented it. :)
By “categorical”, you mean saying “no second language could ever have upsides that overcome the downsides of having two languages”? I agree that’s a pretty conservative position. It’s hard to even imagine another candidate for second language, though (I dunno, Zig maybe?), so I didn’t take it as literally categorical.
And I also do not want another maintainer. If you want to make Linux
impossible to maintain due to a cross-language codebase do that in
your driver so that you have to do it instead of spreading this cancer
to core subsystems. (where this cancer explicitly is a cross-language
codebase and not rust itself, just to escape the flameware [sic] brigade).
I think Zig sees itself as a candidate second language for C projects, but that’s probably beside the point; nobody’s trying to bring it into the kernel.
Indeed, but that call was already decided when Rust support was first merged.
The goal of the experiment is to explore those tradeoffs. You can’t do that if you don’t merge things!
I’ve always heard that Google allows only four languages in their codebase
And yet they are one of the companies pushing Rust for Linux… They understand the tradeoff. The rust for linux people understand the tradeoff. They just think it is worth it.
Totally. But that does seem like a valid engineering question people could disagree about.
However, this is exactly the kind of huge impactful question that should get escalated to the chief architect. And if there’s a sanctioned scouting effort to gather experience to figure out the right answer, somebody found to be actively subverting that effort should be…corrected…by the chief architect.
It’s interesting that they won’t allow it in their internal codebase but they’re happy to let the Linux maintainers take on the burden of adding it there.
R4L is officially an “experiment” with goal to explore tradeoffs. I.E. satisfy curiosity.
This whole debate is so tired. The “including Rust” ship has, at least temporarily, sailed, and maintainers blocking it based on not wanting Rust/a second language doesn’t make sense when the whole point is to explore that option.
Maybe all the code will get deleted at some point, but the code needs to be merged first to then decide that based on how well it works in practice.
To have a thorough experiment, it must be merged into the official tree. One must be able to see how this interacts with a diverse set of hardware and software configurations, and also how this affects the development process inside the kernel community. Some maintainers are afraid of how the introduction of Rust would affect them in the future and the maintainability of the code base. Without it being in the official tree, there wouldn’t be any conclusion on those points.
It sounds like some of the kernel maintainers at least don’t want to have to pay the cost of this large-scale experiment? Like they don’t feel it’s worth the downsides of massively complicating the kernel into a multi-language codebase? Like they understand the potential benefits of memory safety but they feel they are outweighed by the risks?
These individuals believe that they know the result of the experimental before it’s been run. But they do not, because they cannot, by definition. Of course they don’t want to pay the cost, because change is hard, and deep down they know that there’s a chance that the experiment is successful, and thus their responsibilities will either change or they will have to cede responsibility to someone more capable.
Whether there is benefit to Rust in theory, there is little question. It provably solves problems that the kernel has. Whether it can solve those problems without causing larger ones is why the experiment is required. Blocking the experiment for essentially any reason is more or less indefenisble, because the entire point is that they don’t know.
That sounds way too confident and assured of a statement to me. You are discounting the opinion of subject matter experts as if they have no idea what they’re talking about. If someone came to you at your workplace and said that they want to run an experiment to convert your codebase to a new language, and they want you to help maintain and support that experiment while also delivering on all your existing obligations, would you be so quick to accept that? I don’t know about you, but personally I would and have pushed back on rewrite suggestions as a bad idea, despite the perceived benefits (for example, Management thinks that rewriting in JavaScript will make it easier to hire people for the project in the future).
Would rewriting in JavaScript have possibly made me redundant? Maybe. But it would also be massively expensive, cause huge churn, and have a real possibility of failing as a project. We can’t just ignore the very real possibility of risks because of the expected benefits.
There is no interpretation in which it’s too assured. My position is that the result of the experiment is unknown, and that the potential benefits are known, both of which are hard facts.
I have run such experiments in my workplace before. Some have succeeded and some have failed. The successes have massively outweighed the impact of the failures.
Assuming that you know the outcome before you begin is the pinnacle of hubris, which we see time and again with these entrenched maintainers. They may be domain experts, but they deserve every ounce of criticism that they receive. Closed mindedness is not how progress is achieved.
My point is that there must be some threshold that you use to decide whether some experiment is going to be useful to run or not even worth the effort because your experience and expertise tells you otherwise? Or would you accept every experiment that anyone proposed? Eg, suppose someone wanted to model your entire technical stack with TLA+ for high assurance purposes? On paper it sounds like a great idea–formally verify your concurrency properties–but you don’t see how a reasonable project lead might say ‘While this project could certainly bring great benefits, it’s not worth the tradeoff with our team size and capabilities right now. That may change in the future’?
Some threshold must exist, yes. Presumably, that threshold should be somewhere below “the progenitor of the system has decide that the experiment should be run”, which is currently the case of RFL.
Individual system maintainers should not and must not be able to usurp the direction of the project as a whole.
Your core misunderstanding is that we are coming from a tabula rasa. We are not, and if your position is that Linus’s previous endorsements shouldn’t stand, then we’re having an entirely different (and much less tenable) conversation.
“the progenitor of the system has decide that the experiment should be run”, which is currently the case of RFL.
Is it though? To my understanding the decision was only that they will try it and see if it works on both the technical and the social levels. There was never any guarantee given that Rust will be shipped in Linux at all costs. To my understanding Linus’s approach is to see if Rust is overall viable for the kernel, and that includes the opinions of the other subsystem maintainers. Because if they are overall against it, the experiment is dead in the water.
Again, that decision has already been made. There’s no point questioning it over and over. Doing so is distracting from the debates that are important now: how can leadership do better at enforcing these project wide decisions to not let the situation fester like R4L, and reviewing the Rust code without raising the same solved questions again and again (concern trolling).
If you’re genuinely curious about those questions, there’s plenty of text written about it. LWN is a good place to start.
I hear you, but it is a bit in the nature of project that are very distributed in decision making with high individual authority that indeed, decisions will be questioned again and again and again at every point. Those projects need a long breath and the willingness to talk about points again and again and again.
It comes with its strength, particularly, those projects are extremely resilient and fast in other regards. You can’t have both. Pushing top-down decisions in such an environment can have other unintended effects.
You’re right that it’s in rust/, tho unclear to me if it’s properly in the rust subsystem or not. You and others may be right that this patch may still go thru, that seems to depend on Greg KH and Linus.
so he has the authority to block it, but not to reject it? him being the guy referred to above as “the guy who blocked the patches that the Asahi Linux folks wanted.”
There has been a lot of debate on other internet forums about whether these comments were uncharitable about Rust, Rust for Linux, or some other distinction that personally I think doesn’t really matter.
Personally I think it does matter when a core maintainer calls a project that’s part of the kernel “cancer” and the guy who’s in charge doesn’t seem to care.
And I also do not want another maintainer. If you want to make Linux
impossible to maintain due to a cross-language codebase do that in
your driver so that you have to do it instead of spreading this cancer
to core subsystems. (where this cancer explicitly is a cross-language
codebase and not rust itself, just to escape the flameware brigade).
The cancer is not Marcan, Asahi, R4L, or Rust. The cancer is anything-but-C. The cancer is a two-language codebase.
Still a very discouraging comment from a maintainer, and it bucks Linus’ decision that trying out Rust in the kernel is OK, but the distortion of this comment has been bugging me for the last week.
I don’t have an opinion, but I think a fair and less charged way than “cancer” of stating the position is just this: using Rust would have positive benefits, but they are far outweighed by the negatives of allowing a second language into the kernel.
The pros and cons of RfL are somewhat subjective, and the funal balance is hotly debated.
But I find it quite telling that Hellwig’s core argument (that a multi-lamguage codebase requires more work) is held by people who didn’t try doing any of that work. Whereas the kernel devs who started using Rust are explicitly saying “the API compat churn isn’t that big a deal, we can do that work for you if you want”.
We mostly hear about the drama, but it seems that the overall feeling toward Rust among kernel devs (not just RfL devs) is generally positive.
I don’t think the metaphor conveys any technical meaning beyond what I said. I don’t know what “kills the organism” is supposed to relate to. Is the “organism” the kernel? The community of kernel developers? And what does “kill” equate to in reality? It would be better to have the discussion in concrete terms rather than metaphors.
It does convey the personal outrage of the maintainer better, certainly.
Well, notice that you aren’t absorbing my actual point too well, because you’re very focused on making your own.
yes, I’m ignoring the less absurd things you’ve said because I don’t know what to take seriously when you haven’t retracted the most absurd thing.
To give him the benefit of the doubt, he could have been trying to convey the technical meaning of “growth” in the sense of “if you let Rust in, we’ll just have more Rust”. However, that’s an utterly vacuous thing to say, because obviously the whole point of the exercise was to let Rust in and see what happens, and why would you do that unless you want to have more of it if the experiment succeeds?
one characteristic of cancerous growth is that it happens whether you want it to or not.
It conveys no useful information beyond “I don’t think Rust is a good idea at all, for reasons”; in other words, “the benefits don’t outweigh the negatives”. In fact it confuses matters because it implies he thinks Rust would be OK if only it didn’t grow.
it doesn’t imply that, but I do think he would be much more OK with a little bit of Rust if the scope were somehow guaranteed not to expand. like I don’t know, maybe there’s a carve-out for testing a particular class of rust drivers in linux to help find bugs, but eventually the drivers are ported to C for use in linux and the rust version is used in a separate kernel written from scratch in rust?
I’m actually trying to be polite to the guy here, because I truly believe it was an emotional outburst, not an attempt at a technical argument. His actual technical argument (stated much more clearly elsewhere) is that there should be no Rust at all in the kernel, ever, not that once there is some, there will be more. So if “cancer” was supposed to mean “growth” as a technical argument, he misstated his own argument.
so you wouldn’t say that you want to have no cancer at all in your body, ever? if it’s a non-cancerous tumor, I have much less of a problem with it being in my body.
I don’t want to dig into the minutiae of why this metaphor is inaccurate — my point is that using this metaphor at all is unprofessional and counterproductive because of the heavy emotional baggage it comes with. Just say “the introduction of Rust anywhere in the codebase will inevitably result in a growing amount of Rust in the codebase”.
okay, so do you retract your previous statement that “I don’t think the metaphor conveys any technical meaning beyond what I said”?
my intention was not to get into minutia either, but we’ve gone back and forth three times and you still haven’t retracted that statement so I don’t know where you stand.
It’s especially odd in this context, given the history of this metaphor regarding Linux.
so our enemies allow themselves to use the metaphor to efficiently communicate with each other, but we have to forswear it.
If you don’t want it there at all, it’s redundant to also complain that it’s growing. Cancer also has many other attributes besides “growing” that are irrelevant or contradictory to the position taken.
The reason he used that particular metaphor was to express disgust and horror, not to be technically accurate. If he just wanted to evoke “growth” he could have said it was like ivy. Disgust and horror are not an appropriate device to call on in this context, IMO.
Re Mr. Ballmer, the enemies were wrong that time, so don’t you think evoking that argument by using the same metaphor muddies the rhetorical water a bit?
if I can’t tell whether you stand by your statements, it feels futile to interpret them as if you believe them. that’s my issue with continuing in light of the fact that you are just leaving “I don’t think the metaphor conveys any technical meaning beyond what I said” without clarifying whether you believe it.
“complaining that it’s growing” is completely different from identifying the growth as a factor in the technical drawbacks.
Cancer has a multitude of characteristics, of which growth is only one. If anything, it has many characteristics contradictory to the point being made: it’s an undesirable outcome of natural processes, it is typically self-generated from within rather than deliberately introduced, it is hard to eliminate because it’s nourished by the same system that sustains healthy cells…
If you have to tell me exactly which part of the metaphor I’m supposed to pay attention to in order to get the technical meaning, then you aren’t conveying technical meaning, you’re muddying the water. Just say what you mean.
The additional meaning being conveyed here is not technical, it’s emotional, and I do agree emotional meaning was conveyed.
The additional meaning being conveyed here is not technical
I take this to mean that you still stand by the statement that “I don’t think the metaphor conveys any technical meaning beyond what I said.” if you treat all technical metaphors like this, I’m glad you’re not my coworker.
If you have to tell me exactly which part of the metaphor I’m supposed to pay attention to in order to get the technical meaning, then you aren’t conveying technical meaning, you’re muddying the water.
or it could mean I am conveying technical meaning but you’re being willfully obtuse. just maybe…
Well, notice that you aren’t absorbing my actual point too well, because you’re very focused on making your own.
To give him the benefit of the doubt, he could have been trying to convey the technical meaning of “growth” in the sense of “if you let Rust in, we’ll just have more Rust”. However, that’s an utterly vacuous thing to say, because obviously the whole point of the exercise was to let Rust in and see what happens, and why would you do that unless you want to have more of it if the experiment succeeds? It conveys no useful information beyond “I don’t think Rust is a good idea at all, for reasons”; in other words, “the benefits don’t outweigh the negatives”. In fact it confuses matters because it implies he thinks Rust would be OK if only it didn’t grow.
I’m actually trying to be polite to the guy here, because I truly believe it was an emotional outburst, not an attempt at a technical argument. His actual technical argument (stated much more clearly elsewhere) is that there should be no Rust at all in the kernel, ever, not that once there is some, there will be more. So if “cancer” was supposed to mean “growth” as a technical argument, he misstated his own argument.
I could be wrong, but I get the feeling that people didn’t quite get why Hellwig used the word “cancer”. I could be wrong about this in a lot of ways, maybe I too misunderstood what Hellwid meant, maybe I misjudged how others are interpreting what he said. But allow me to add my 2 cents.
People seem to be interpreting what Hellwig said (i.e. calling R4L “cancer”) as him saying R4L is bad, evil, a terrible thing, etc. And I totally understand why many would think that, and would agree this is a terrible choice of words. But I think that Hellwig’s focus is actually on the fact that cancer spreads. A better word to use would probably be like “infectious”, or “viral”. Although I disagree with him, I think what he was saying is that, despite what Rust developers promised, Rust will spread to more corners of the kernel and increase the maintenance burden and will have a negative impact on the kernel as a whole.
I think you’re right, but that doesn’t exculpate Hellwig in the least. One of the defining characteristics of cancer is that it is malignant. It seems pretty clear that he used this metaphor because he thought that Rust would spread and that it is dangerous and harmful.
“infectious” or “viral” have different connotations. viral would be if they thought the BSDs might catch it. infectious is not as specific as cancer; cancer specifically grows from one area and consumes more and more of the things next to it. infections can do that but not necessarily.
Unrelated to the rust issue, my understanding is that such toxic language actually isn’t all that strange in the Linux kernel maintainer community. It feels like they try somewhat not to overdo it, but they aren’t that much concerned about the toxicity. More like, “I’ll try but I don’t actually care”.
It’s useful in some cases, but when the effect of it is raising the temperature of the discussion, then I would say it isn’t useful in that instance. There are other ways to convey the same meaning that wouldn’t have attracted the same amount of strong emotions. We’ve seen examples of such alternate arguments in this thread that would have been more suitable.
sure. but then don’t fixate on the word, and acknowledge that their choice to use strong language can also be an effect of the raising temperature of the discussion, for which responsibility is shared among all participants.
There’s this idea that multi-language codebases are categorically a disaster, right? What really bothers me is how small-minded it is. There are many C and C++ codebases that have introduced Rust. It’s not easy but it is possible, and it often is worth it. Some attempts have been successful and others haven’t. You can learn so much from other people’s experiences here!
Simply calling multi-language codebases a cancer closes off discussion rather then opening it. That goes against the basic curiosity that is a hallmark of engineering excellence.
Honestly everything around R4L that has been going has just cemented in me the belief that the average long time Linux maintainer is fundamentally incompetent. They’re reasonably good at One Thing, but totally incurious and scared of anything outside of that One Thing they’re good at.
Is it possible that we could take an experienced kernel maintainer’s opinion seriously instead of dismissing it immediately as incurious and ‘against’ engineering excellence? If a co-worker who specializes in a different domain came to you and kept insisting that something you know is very difficult and not worth the tradeoff is actually quite easy, would you be incurious and ‘against’ engineering excellence if you dismissed their wild suggestions without a deep discussion of the tradeoffs?
Evaluating who to take seriously and who not to is a matter of judgment cultivated over a lifetime.
If a co-worker who specializes in a different domain came to you and kept insisting that something you know is very difficult and not worth the tradeoff is actually quite easy, would you be incurious and ‘against’ engineering excellence if you dismissed their wild suggestions without a deep discussion of the tradeoffs?
Depends on many things, but in general, yes, of course, especially if they were someone I took seriously on other matters. If nothing else, it’s a chance to sharpen my arguments. If I keep having the same arguments over and over I might even step back and write an essay or two explaining what I believe and why I believe it. Here’s one that’s public that was a result of the sort of back-and-forth you’re talking about.
Reading the discussions on this elsewhere, there’s no way to describe this that wasn’t going to get accused of bias by people holding strong opinions on either marcan or on the rust for linux project.
I’m not accusing of bias (which is more or less unavoidable), but of constructing a narrative. The facts are: he exchanged words on the LKML and then wrote Mastodon posts about the exchange. The narrative you’re constructing is: he exchanged words on LKML, then because he was unsatisfied with how that discussion was going, he wrote Mastodon posts about the exchange in an attempt to gather support.
I think it would be better to leave out the guesswork about intentions, or at least clearly separate it from the factual retelling of events.
Correction: Marcan never “exchanged words” with Christoph Hellwig on the patches, which had not even been posted by him. He just replied to the thread with a tirade.
I have a different recollection, it seems:
Junior: let’s reboot the computer
Senior: let’s troubleshoot, learn what’s wrong and fix this
Staff: let’s do magic incantations and summon dark powers
Principal: let’s reboot the computer
Ah, an instance of one of my favourite memes! There’s something very pleasing when the “smart” solution looks externally identical to the dumb solution.
Couldn’t agree more! I think I shared this on lobsters years ago, but my favorite thing in my ~/.zshrc is this little guy:
function t {
pushd $(mktemp -d /tmp/$1.XXXX)
}
If I run ‘t blah’, I’ll drop into a temporary directory with a name like /tmp/blah.1tyC. I can goof around without worrying about cleaning up the mess later. When I’m done I can popd back to where ever I was. On the off chance I like what I did, I can just move the folder somewhere permanent. I use this every day; my $HOME would be unnavigable without it.
I like to automate the “popd && rm” part in the alias directly. This bash function enters a subshell inside the tmpdir and when I exit or ^D, it pops back to the previous directory and deletes the tmpdir – with a little safeguard when you mounted something inside it! Had a bad time when experimenting with mount namespaces and accidentally deleted my home directory because it was mounted inside this tmpdir …
tmp() {
history -w || true
t=$(mktemp --tmpdir -d tmpdir-XXXXXX) \
&& { $SHELL -c \
"cd '$t' \
&& printf '\033[31m%s\033[0m\n' 'this directory will be removed upon exit' \
&& pwd \
&& exec $SHELL" \
|| true; \
} \
&& if awk '{ print $2 }' /etc/mtab | grep "$t"; then
echo -e "\033[31maborting removal due to mounts\033[0m" >&2
else
echo -e "\033[31mremoving temporary directory ...\033[0m" >&2
rm -rf "$t"
fi
}
I have nearly the same function, but I’m using ~/tmp has the base, precisely because /tmp is often emptied on boot and I know that I sometimes want to go back to these experiments.
Using ~/tmp helps keep my home directory clean and makes it obvious that the stuff is easily removable, but if, on an off-chance, I might need one of those experiments again, it’s there, waiting for me in ~/tmp even though I might have rebooted in the mean time.
But in general, yes, this is the way to go. I’ve learned it here on lobsters years ago and I’m using it daily.
I like this idea, I came to the same conclusion as you, but instead of doing ~/tmp, I added “tmp” to my personal monorepo’s .gitignore. But sometimes I would put random downloads in that folder, lol. Thanks for the idea, I feel like having two of these makes sense, one for coding stuff and one for everything else
This is absolutely fantastic. I’ve been independently using this trick for years (through ^R instead of an alias) and I love it too. I didn’t know mktemp took an argument though, thank you!
I like to do programming-language-specific versions of this. So like “newrust” creates a temp dir, puts a Cargo hello world project in there, cd’s in there, and opens my editor. Similarly “newgo”, “newcpp”, etc. Great for playing around.
I tend to have two use-cases for these, one that’s very very temporary (single shell instance), and one that should stick around a little longer (i.e. duration of the boot).
This works out to something along the lines of
t() {
mkdir -p /tmp/t
cd /tmp/t
}
tt() {
local dir=$(mktemp -d)
[ -d "$dir" ] || return 1
cd "$dir"
trap "rm -rf '$dir'" EXIT
}
Notably, I use cd because I rarely popd at the end, I usually just close the shell once I’m done with it. I probably should do it anyway though :)
Personally I often try out small experiments with different programming languages which need a project structure to be set up before they can work properly. For this a new folder is often needed.
For me, I just do a lot of stuff directly in /tmp, but this seems nice to keep things organized in case I want to move a directory to somewhere more permanent.
Scenario A: I’m trying to test something in C/Zig and don’t want to pollute my actual project directory. I just make a /tmp/test.{c, zig} and compile in /tmp. I think putting it in a temp dir would be nice, if unnecessary.
Scenario B: I, semi-frequently, will clone git repos into /tmp if I’m just trying to quickly look at something. Occasionally, I clone multiple related repos at the same time. Having a temp dir to keep them together would be nice if I ended up wanting to move them out.
For me it’s poking around in exploded jar files, or tarballs, or other archive formats mostly.
Sometimes you don’t just want to list the contents, or maybe there are container formats in container formats, a zip inside a tar, inside a cpio for example.
I want to unpack all these, look around, examine files, etc. without worrying about needing to clean up the mess after.
When people complain about there being too many dependencies and dependencies being too large, remember these fundamental tensions. You get to pick big packages or many dependencies, but you don’t get to complain about both in the same breath.
There is a third way, which is my own personal preference: as few dependencies as possible and make them as small as possible.
In JavaScript this can mean Vanilla JS - trying to build as much functionality as possible using native browser APIs without pulling in dependencies. I’m finding this approach increasingly productive, browsers are really good these days.
In languages like Python or Go this can mean sticking to stuff in the standard library, which tends to be a whole lot more stable (aka boring) than the wider dependency world.
One of the things I like most about Django’s “batteries included” philosophy is that it means one package - Django - gets you almost everything you need to build a web app - routing and templates and an ORM and suchlike. This compares to the “best tool for the job” alternative where you put together your own framework from a bunch of parts, but since each part is on a separate release cycle with separate maintainers you lose the guarantee that they’ll upgrade in lock-step with each other.
Django counts as a “big package” in this case, which the article proposes makes it “likelier to change more often”. Thankfully Django has a well-won reputation for smooth upgrades, so it’s one of the larger dependencies that I have the most confidence in.
Django has also been around for decades, which partially explains why it’s so stable. And in your case, it’s probably close to what you’d call “standard library”. But there are good reasons for “many small packages” approach as well. Again, the question is only which packages you take.
Even though I don’t know the tar flags, I’m pretty certain that an alias I wrote 20 years ago would probably still work. It’s not as stable in, say, JavaScript world mentioned in the article, but good old tools and utils - say, express, mocha, request, lodash - still work mostly the same. Sure, you can update them to newer versions, but you’ll probably not have any changes to your code.
I think this is often a good approach! I also don’t think it’s actually really arguing against what I articulated here, which is about what happens when you do use a third-party package. In particular, the thing I was mostly trying to get at is that it would be pretty lame to then complain about Django being a large package—because: well, no, you’re getting something for that! I should also add that by “change” I didn’t mean “breakage”, only that they are likelier to have ongoing updates in the form of bug fixes, security releases, and indeed new features, than a very small library that does one thing.
I also don’t have a horse in the big-package-small-package race; I think there are substantial tradeoffs between the two.
All the popular open source media servers have clunky UI or were unreliable; mostly both
Without specifically mentioning the ones you’re referring to it’s hard to know what this relates to.
I’ve recently adopted Jellyfin, and found it to be generally fine - a few quirks here and there but generally fine. The biggest downside I’ve seen so far is that there isn’t (to my knowledge) a good Mac open source client for it - but this article is about an AppleTV which doesn’t have that issue. I’m aware some of the iOS/iPadOS clients can run on macOS, but none of the ones I’ve tried work very well with a large screen.
Can someone expand on what specifically makes Infuse better?
Infuse has native support for jellyfin too. It’s quite a good app and has support for more codecs than just native iOS player (which jellyfin uses).
But, I don’t mind the jellyfin client on iOS? It seems to do what I want it to.
My guess is the author doesn’t want multi-device streaming /status syncing so jellyfin isn’t required. In which case infuse on appletv is a decent choice.
There’s a few mentions in there of having a phone and tablet; I do appreciate that Infuse makes it very easy to start watching a show on one screen and continue on another.
I didn’t name the specific open source apps because I didn’t want to get into an inevitably pretty nitpicky arguments about each. As a programmer running a linux desktop I’m pretty tolerant of clunky UI, but the apps I tried felt significantly worse and were soundly rejected by the nontechnical TV users. There’s probably a lot of familiarity at work in the personal taste; Infuse looks a lot like the popular streaming services.
I get that, and I have infuse on ios devices and very much like it. The cool thing is, if you ever want to expand and do plex/jellyfin style stuff, infuse has support for it baked right in.
The other thing I like is offline downloads, for when I hit the road but still wanna watch curb on repeat
This HN thread on Jellyfin hits all of the issues I had with the open source options and also hits all of the “you’re doing it wrong” nitpicking I was not interested in participating in.
LOL My needs are pretty modest, and I’m just impressed by all the free engineering hours that have gone into it and the features it has. What should I look out for? The fact that they have a Roku App was pretty amazing for me.
IMO the best way to use jellyfin on a personal computer is with jellyfin-mpv-shim and mpv. It’s what I use on Windows and Linux. jellyfin-mpv-shim has instructions for macOS so I am assuming that both that and mpv are available on macOS. mpv the media player is very minimalistic so it takes getting used to because the keybinds and such are not apparent but it works really well if you know how to handle it.
I switched over to Jellyfin Media Player because it uses MPV internally, and includes the web interface so it feels less clunky. The shim would sometimes break, and when I’m trying to watch something, I definitely don’t want to be debugging things.
JMP uses Qt’s Web engine, which is not extra lightweight but at least shared with other apps, so it’s worth the tradeoff for me.
There are some codecs that the browser can’t play. If you host jellyfin on a weak machine, you have to disable transcoding. So dedicated clients who are able to handle a larger variety of codecs are preferable to browsers.
I’ve noticed significantly better UI responsiveness and playback from the native app running on an AppleTV (which connects over average quality WiFi) than I do from a M2 Ultra Mac Studio with 128GB of RAM connecting to the same Jellyfin server over wired gigabit ethernet.
I assume it’s because the native app is better able to cache and preload content than the browser based client.
I mean a native client app for playback, rather than accessing it through a browser - the browser client is fine for admin tasks but for playback it’s less ideal IME: https://lobste.rs/s/sn8buz/tv_setup#c_altp8l
Sorry if this is a bit unrelated, but this question is bugging me for quite some time now: People who say “I like LLMs for boilerplate”, don’t they have templates or snippets?
I have a large collection of those collected over the past 20 years of programming professionally and I don’t think anything can be easier than for example cp ~/templates/symfony/abstract-auth-controller.tpl.php src/Controller/NewFeatureController.php && nvim src/Controller/NewFeatureController.php?
My experiments with LLMs were not that smooth at least.
*nod* I used UltiSnips for a while and I’m currently using coc.nvim’s support for UltiSnips snippets. Python script? boiler<Tab>.
Sure, I’ve played around with so-called A.I.… using a self-hosted copy of Stable Diffusion. I just don’t understand the appeal of rushing to become dependent on an external service provider who will eventually need a “monetization strategy”.
No, and this is anecdotal, but most these people did not have these workflows defined and just slugged it out in the days before LLMs, with the help of a search engine.
The author used nvim but most folks use VS Code, where the community at large leans towards vanilla config.
A few things. One, like you said, people starting now would need 20 years to get to the same amount of snippets.
Yes, I know, you can probably find decent collections already collected, from a plugin, then update them for different projects which potentially use different versions of whatever it is that you’re templating.
That said, I still think it’s largely unneeded. It’s just not the only way.
Yeah, it took 20 years to become a lot. But it was useful from day 1, when I noticed that I was doing the same things in my IDE (Delphi, back then) every time. (I haven’t kept everything - don’t have the Delphi templates anymore, for example.)
I support @crmsnbleyd on this. It may well have been overly attending to encoding / space even for the era.
While the blog post does say this, it doesn’t say “how slow” or even if it was seek or throughput bound. So, I looked at the original McIlroy paper which mentions “scanning the whole dictionary” being slow in the prototype (so throughput), but not how long that took. So, then I looked up a bit about historical disk IO rates and found this wikipage indicating 885 kB/s is not crazy for IBM3340 Winchester in 1973. Let’s say a mini-computer in 1970 was about 5..10x slower. That’s still 90..180 kB/s or 250 kB dictionary in about 1.4 .. 2.8 seconds.
It would have been plenty fast to: sort all the document/chapter/whatever words (presumably much smaller than a dictionary and fits in RAM), saving (line, column) locations, then do the equivalent of an “ordered merge” of those words against the also sorted on-disk dictionary in a single pass - just checking for existence/membership rather than actually merging. The whole procedure including loading the document (presumably much smaller than a whole dictionary) should have been at most a few seconds. No real-time perf numbers are discussed in McIlroy’s 1982 paper other than “100 program uses per week” to indicate popularity, but I’m guessing it wasn’t much faster than that.
Another point is that back then, computers were mostly multi-user beasts, in the true sense of the word. I don’t know what kind of computers these people worked on, but the article mentions Unix, so probably also a multi-user beast. Plus, there were a lot less hardware bits that worked with DMA - the CPU would have to be managing the disk I/O in addition to everything else. So I would probably divide that number by at least one more order of magnitude again.
So, waiting ~2 seconds is perhaps not bad, but waiting 20 - when the alternative was the “instant” thing that this in-memory thing enabled - is probably not as comfortable. Especially if you had to do that on every damn word of your document.
I don’t know if that’s the case, but I would assume something like this was a reason for even trying to do this.
Which brings up an interesting tangent, I think. We barely have any true multi-user computers left. Sure, we ahve servers, that serve millions of requests, but they’re mostly optimized for one a few applications. I think there’s only the occasional Windows Server with specialized software, where people RDP their way in for one specific application, or niche things like tildeverse. Probably a few other things that I’m thinking, but majority of the machines out there are single-user, even though probably even CPU cache buffers for many of them are around the 64kb that the article is revolving about.
We barely have any true multi-user computers left.
I think compute clusters (supercomputers and setups of the kind run by @cks) typically have login servers attached, for compiling and managing jobs. Not full-spectrum timesharing but close.
Especially if you had to do that on every damn word of your document. I don’t know if that’s the case, but I would assume something like this was a reason for even trying to do this.
The idea is to not do for word in doc: Lookup, but instead to realize “I am checking all the words in doc” and sort -u those to-be-checked words against a pre-arranged sort-u dictionary.” (Yes, you could prefix-compress the dictionary to save on that IO time, too, and you are right - someone should find a 1975 benchmark timing “cat 1MiBfile” for a better idea of real-time perf). I tried to explain this in words elsethread both here and there, but let’s try Python code this time:
def notIn(a, d): # elts of a not in d
print("in ", a, "but not in", d)
i, j = 0, 0 # a,d both "sort -u"
while i < len(a) and j < len(d):
if a[i] < d[j]:
print(a[i]) # not in d
i += 1
elif a[i] == d[j]: # advance both
i += 1; j += 1
else:
j += 1
while i < len(a):
print(a[i])
i += 1
unabridged = [1,3,5,7] # dictionary
notIn([3,5], unabridged)
notIn([0,1], unabridged) # 0 not in dict
notIn([1,8], unabridged) # 8 not in dict
notIn([0,8], unabridged) # 0,8 not in
This code tests out ok on the odd-even inside/left/right/outside edge cases, but I really just wrote it. So, it’s possible there are bugs.
In fact, the 1982 McIlroy article says they did do a sort-u on document words and that this took about 20% of the run-time for some unspecified sized documents (though he notes vocabularies are pretty small). My point was more that you don’t need any very fancy encoding/hashing/Bloom filters or any of that. Just a (maybe prefix-compressed) form of the unabridged dictionary on disk and linear sweeps.
I tried to find Steve Johnson’s original spell.c from 1975 online that was “too slow”, but could not. I did find some later code doing binary search lookups and the language of McIlroy 1982 suggests the algo might well have been for word in doc: binarySearch which would have indeed been very slow (as you fear). Also, the V7 /usr/dict/words was only 196_513 bytes, but is also only 24_001 words and so not very “unabridged”. A SOWPODS scrabble dictionary I have lying around is 2_707_014 bytes and 267_751 words. So, even with a 2X slower than 885 kB/s it would still only take 6 seconds to scan through it, even without prefix-compression (which would probably compensate back in IO costs, cutting off about 66% of that run-time, say, since less abridged dictionaries compress much better than abridged ones). Frankly, the “standard of care” for research papers back then was kind of frustratingly low, not that it’s great today. I could not even find any “n-fold speed-up guestimate” from McIlroy 1982. So, we’re left to make a bunch of guesses.
In fact, in the ensuing half century, bandwidth/throughput has generally gotten much better than latency. This “batch” idea might well be faster than point lookups today, though if you want to assess that you should use a fast language like C or Nim rather than Python and it should be noted CPUs play a lot of tricks hiding latency to measure this and few care how many nanoseconds per word a user-facing thing like spell check takes. Spell check has moved on to fancy suggestions or LLM statistical models anyway.
The 885 kB/s is for sequential access, but I imagine accessing a spell checking dictionary is highly non-sequential, so it’d be the seek time that matters more, especially if you want to also read a document at the same time.. With the dictionary entirely stored in RAM, you get much cheaper random access. I don’t have any numbers/benchmarks to verify this though.
I think you misunderstood the idea. Only sequential access should matter if you can fully buffer the sorted document words (plausible for small documents). Both are sorted. So, you just sequentially go through the dictionary from A -> Z matching or passing the current doc word. Then you advance a doc word and advance probably just advance a little in the dictionary and so on.
The TypeScript dev lead posted this response about the language choice on Reddit, for anyone who’s curious:
Source: https://www.reddit.com/r/typescript/comments/1j8s467/comment/mh7ni9g
People really miss the forest for the trees.
I looked at the repo and the story seems clear to me: 12 people rewrote the TypeScript compiler in 5 months, getting a 10x speed improvement, with immediate portability to many different platforms, while not having written much Go before in their lives (although they are excellent programmers).
This is precisely the reason why Go was invented in the first place. “Why not Rust?” should not be the first thing that comes to mind.
I honestly do think the “Why not Rust?” question is a valid question to pop into someone’s head before reading the explanation for their choice.
First of all, if you’re the kind of nerd who happens to follow the JavaScript/TypeScript dev ecosystem, you will have seen a fair number of projects either written, or rewritten, in Rust recently. Granted, some tools are also being written/rewritten in other languages like Go and Zig. But, the point is that there’s enough mindshare around Rust in the JS/TS world that it’s fair to be curious why they didn’t choose Rust while other projects did. I don’t think we should assume the question is always antagonistic or from the “Rust Evangelism Strike Force”.
Also, it’s a popular opinion that languages with algebraic data types (among other things) are good candidates for parsers and compilers, so languages like OCaml and Rust might naturally rank highly in languages for consideration.
So, I honestly had the same question, initially. However, upon reading Anders’ explanation, I can absolutely see why Go was a good choice. And your analysis of the development metrics is also very relevant and solid support for their choice!
I guess I’m just saying, the Rust fanboys (myself, included) can be obnoxious, but I hope we don’t swing the pendulum too far the other way and assume that it’s never appropriate to bring Rust into a dev conversation (e.g., there really may be projects that should be rewritten in Rust, even if people might start cringing whenever they hear that now).
While tweaking a parser / interpreter a few years ago written in Go, I specifically replaced a struct with an ‘interface {}’ in order to exercise its pseudo-tagged-union mechanisms. Together with using type-switch form.
https://github.com/danos/yang/commit/c98b220f6a1da7eaffbefe464fd9e734da553af0
These day’s I’d actually make it a closed interface such that it is more akin to a tagged-union. Which I did for another project which was passing around instances of variant-structs (i.e. a tagged union), rather than building an AST.
So it is quite possible to use that pattern in Go as a form of sum-type, if for some reason one is inclined to use Go as the implementation language.
That is great explanation of “Why Go and not Rust?”
If you’re looking for “Why Go and not AOT-compiled C#?” see here: https://youtu.be/10qowKUW82U?t=1154s
A relevant quote is that C# has “some ahead-of-time compilation options available, but they’re not on all platforms and don’t really have a decade or more of hardening.”
That interview is really interesting, worth watching the whole thing.
Yeah Hjelsberg also talks about value types being necessary, or at least useful, in making language implementations fast
If you want value types and automatically managed memory, I think your only choices are Go, D, Swift, and C# (and very recently OCaml, though I’m not sure if that is fully done).
I guess Hjelsberg is conceding that value types are a bit “second class” in C#? I think I was surprised by the “class” and “struct” split, which seemed limiting, but I’ve never used it. [1]
And that is a lesson learned from the Oils Python -> C++ translation. We don’t have value types, because statically typed Python doesn’t, and that puts a cap on speed. (But we’re faster than bash in many cases, though slower in some too)
Related comment about GC and systems languages (e.g. once you have a million lines of C++, you probably want GC): https://lobste.rs/s/gpb0qh/garbage_collection_for_systems#c_rrypks
There was also a talk that hinted at some GC-like patterns in Zig, and I proposed that TinyGo get “compressed pointers” like Hotspot and v8, and then you would basically have that:
https://lobste.rs/s/2ah6bi/programming_without_pointers#c_5g2nat
[1] BTW Guy Steele’s famous 1998 “growing a language” actually advocated value types in Java. AFAIK as of 2025, “Project Valhalla” has not landed yet
Compilers written in OCaml are famous for being super-fast. See eg OCaml itself, Flow, Haxe, BuckleScript (now ReScript).
Yeah, I’m kind of curious about whether OCaml was considered at some point (I asked about this in the Reddit thread, haven’t gotten a reply yet).
OCaml seems much more similar to TS than Go, and has a proven track record when it comes to compilers. Maybe portability issues? (Good portability was mentioned as a must-have IIRC)
Maybe, but given that Flow, its main competitor, distributes binaries for all major platforms: https://github.com/facebook/flow/releases/tag/v0.264.0
Not sure what more TypeScript would have needed. In fact, Flow’s JavaScript parser is available as a separate library, so they would have shaved off at least a month from the proof of concept…
Also Nim.
Also Julia.
There surely are others.
Yes good points, I left out Nim and Julia. And apparently Crystal - https://colinsblog.net/2023-03-09-values-and-references/
Although thinking about it a bit more, I think Nim, Julia, (and maybe Crystal) are like C#, in that they are not as general as Go / D / Swift.
You don’t have a
Foo*type as well as aFootype, i.e. the layout is orthogonal to whether it’s a value or reference. Instead, Nim apparently has value objects and reference objects. I believe C# has “structs” for values and classes for references.I think Hjelsberg was hinting at this category when saying Go wins a bit on expressiveness, and it’s also “as close to native as you can get with GC”.
I think the reason this Go’s model is uncommon is because it forces the GC to support interior pointers, which is a significant complication (e.g. it is not supported by WASM GC). Go basically has the C memory model, with garbage collection.
I think C#, Julia, and maybe Nim/Crystal do not support interior pointers (interested in corrections)
Someone should write a survey of how GC tracing works with each language :) (Nim’s default is reference counting without cycle collection.)
Yeah that’s interesting. Julia has a distinction between
struct(value) andmutable struct(reference). You can use raw pointers but safe interior references (to an element of an array for example) include a normal reference to the (start of the) backing array, and the index.I can understand how in Rust you can safely have an interior pointer as the borrow checker ensures a reference to an array element is valid for its lifetime (the array can’t be dropped or resized before the reference is dropped). I’m very curious - I would like to understand how Go’s tracing GC works with interior pointers now! (I would read such a survey).
Ok - Go’s GC seems to track a memory span for each object (struct or array), stored in kind of a span tree (interval tree) for easy lookup given some pointer to chase. Makes sense. I wonder if it smart enough to deallocate anything dangling from non-referenced elements of an array / fields of a struct, or just chooses to be conservative (and if so do users end up accidentally creating memory leaks very often)? What’s the performance impact of all of this compared to runtimes requiring non-interior references? The interior pointers themselves will be a performance win, at the expense of using an interval tree during the mark phase.
https://forum.golangbridge.org/t/how-gc-handles-interior-pointer/36195/5
It’s been a few years since I’ve written any Go, but I have a vague recollection that the difference between something being heap or stack allocated was (sometimes? always?) implicit based on compiler analysis of how you use the value. Is that right? How easy it, generally, to accidentally make something heap-allocated and GC’d?
That’s the only thing that makes me nervous about that as a selling point for performance. I feel like if I’m worried about stack vs heap or scoped vs memory-managed or whatever, I’d probably prefer something like Swift, Rust, or C# (I’m not familiar at all with how D’s optional GC stuff works).
Yes, that is a bit of control you give up with Go. Searching for “golang escape analysis”, this article is helpful:
https://medium.com/@trinad536/escape-analysis-in-golang-fc81b78f3550
So the toolchain is pretty transparent. This is actually something I would like for the Oils Python->C++ compiler, since we have many things that are “obviously” locals that end up being heap allocated. And some not so obvious cases. But I think having some simple escape analysis would be great.
Yes, the stack/heap distinction is made by the compiler, not the programmer, in Go.
Why did you leave JS/TS off the list? They seem to have left it off too and that confuses me deeply because it also has everything they need
Hejlsberg said they got about 3x performance from native compilation and value types, which also halved the memory usage of the compiler. They got a further 3x from shared-memory multithreading. He talked a lot about how neither of those are possible with the JavaScript runtime, which is why it wasn’t possible to make tsc 10x faster while keeping it written in TypeScript.
Yeah but I can get bigger memory wins while staying inside JS by sharing the data structures between many tools that currently hold copies of the same data: the linter, the pretty-formatter, the syntax highlighter, and the type checker
I can do this because I make my syntax tree nodes immutable! TS cannot make their syntax tree nodes immutable (even in JS where it’s possible) because they rely on the
node.parentreference. Because their nodes are mutable-but-typed-as-immutable, these nodes can never safely be passed as arguments outside the bounds of the TS ecosystem, a limitation that precludes the kind of cross-tool syntax tree reuse that I see as being the way forwardHejlsberg said that the TypeScript syntax tree nodes are, in fact, immutable. This was crucial for parallelizing tsgo: it parses all the source files in parallel in the first phase, then typechecks in parallel in the second phase. The parse trees from the first phase are shared by all threads in the second phase. The two phases spread the work across threads differently. He talks about that kind of sharing and threading being impractical in JavaScript.
In fact he talks about tsc being designed around immutable and incrementally updatable data structures right from the start. It was one of the early non-batch compilers, hot on the heels of Roslyn, both being designed to support IDEs.
Really, you should watch the interview https://youtu.be/10qowKUW82U
AIUI a typical LSP implementation integrates all the tools you listed so they are sharing a syntax tree already.
It’s true that I haven’t watched the interview yet, but I have confirmed with the team that the nodes are not immutable. My context is different than Hejlsberg’s context. For Hejlsberg if something is immutable within the boundaries of TS, it’s immutable. Since I work on JS APIs if something isn’t actually locked down with
Object.freezeit isn’t immutable and can’t safely be treated as such. They can’t actually lock their objects down because they don’t actually completely follow the rules of immutability, and the biggest thing they do that you just can’t do with (real, proper) immutable structures is have anode.parentreference.So they have this kinda-immutable tech, but those guarantees only hold if all the code that ever holds a reference to the node is TS code. That is why all this other infrastructure that could stand to benefit from a shared standard format for frozen nodes can’t: it’s outside the walls of the TS fiefdom, so the nodes are meant to be used as immutable but any JS code (or any-typed code) the trees are ever exposed to would have the potential to ruin them by mutating the supposedly-immutable data
To be more specific about the
node.parentreference, if your tree is really truly immutable you need to replace a leaf node you must replace all the nodes on the direct path from the root to that leaf. TS does this, which is good.The bad part is that then all the nodes you didn’t replace have chains of
node.parentreferences that lead to the old root instead of the new one. Fixing this with immutable nodes would mean replacing every node in the tree, so the only alternative is to mutatenode.parent, which means that 1) you can’t actuallyObject.freeze(node)and 2) you don’t get all the wins of immutability since the old data structure is corrupted by the creation of the new one.See https://ericlippert.com/2012/06/08/red-green-trees/ for why Roslyn’s key innovation in incremental syntax trees was actually breaking the
node.parentreference by splitting into the red and green trees, or as I call them paths and nodes. Nodes are deeply immutable trees and have no parents. Paths are like an address in a particular tree, tracking a node and its parents.You are not joking, just the hack to make type checking itself parallel is well worth an entire hour!
Hm yeah it was a very good talk. My summary of the type checking part is
That is, the translation units aren’t completely independent. Type checking isn’t embarassingly parallel. But you can still parallelize it and still get enough speedup – he says ~3x from parallelism, and ~3x from Go’s better single core perf, which gives you ~10x overall.
What wasn’t said:
I guess this is just because, empirically, you don’t get more than 3x speedup.
That is interesting, but now I think it shows that TypeScript is not designed for parallel type checking. I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library. Though it’s hard to compare, since it also has to generate code
A separate thing I found kinda disappointing from the talk is that TypeScript is literally what the JavaScript code was. There was never a spec and will never be one. They have to do a line-for-line port.
There was somebody who made a lot of noise on the Github issue tracker about this, and it was basically closed “Won’t Fix” because “nobody who understands TypeScript well enough has enough time to work on a spec”. (Don’t have a link right now, but I saw it a few months ago)
Pretty sure he said it was an arbitrary choice and they’d explore changing it. The ~10x optimization they’ve gotten so far is enough by itself to keep the project moving. Further optimization is bound to happen later.
Work has been going on for years to parallelize rust’s frontend, but it apparently still has some issues, and so isn’t quite ready for prime time just yet, though it’s expected to be ready in the near term.
In my experience, once you start to do things “per core” and want to actually get performance out of it, you end up having to pay attention to caches, and get a bit into the weeds. Given just arbitrarily splitting up the work as part of the port has given a 10x speed increase, it’s likely they just didn’t feel like putting in the effort.
Can you share the timestamp to the discussion of this hack, for those who don’t have one hour?
I think this one: https://www.youtube.com/watch?v=10qowKUW82U&t=2522s
But check the chapters, they’re really split into good details. The video is interesting anyway, technically focused, no marketing spam. I can also highly recommend watching it.
Another point on “why Go and not C#” is that, he said, their current (typescript) compiler is highly functional, they use no classes at all. And Go is “just functions and data structures”, where C# has “a lot of classes”. Paraphrasing a little, but that’s roughly what he said.
They also posted a (slightly?) different response on GitHub: https://github.com/microsoft/typescript-go/discussions/411
Yes please!
Once again I am baffled by how many people dismiss this with the usual “this is just legal boilerplate” or “the distro will patch it out”, just because it is Mozilla. I dislike Google as much as the next guy, but if this was Google, people would not have given the benefit of the doubt as much as they are doing now.
Even before this terms-of-use change, Mozilla has reportedly been acting evil for a long time and privacy-based Firefox forks have been necessary for just as long as Chromium has been deemed the monopoly.
I am tired of companies like Mozilla, Duckduckgo et al. getting a free pass in the FOSS community, even when the last few years clearly have shown that they use privacy as a publicity stunt.
It is legal boilerplate. I would make the same comment if EvilCorp had the same clause in their terms.
And I am sick and tired of people trying to find pointless things to be outraged about. There are enough actual bad things in the world. You don’t need to go inventing fake ones.
It is a legal boilerplate, but it is also (another) signal of the direction in which the Mozilla Foundation is moving.
I do agree that people should worry more about more impactful bad things, but this is also impactful and “actual”. Just maybe not for all.
The Python Package Index announced updated TOS yesterday. It has a very similar clause stating you grant them a license to things you send to them.
https://blog.pypi.org/posts/2025-02-25-terms-of-service/
Do you think this is a “signal” of a “direction” the Python Package Index and/or the Python Software Foundation is moving?
It seems to me that, contrary to the claim above, Mozilla isn’t getting a “free pass” — Mozilla is getting far more scrutiny and far less benefit of the doubt than other entities doing the same thing. And for no rational reason I can discern.
Well there are differences and similarities. Compare:
vs
The PyPI text is clearly about presenting the content in a way that could be construed as copyright infringement. The Firefox text has the much broader notion of use, which could mean anything including using the data to build a profile and selling it to partners.
Another comparison:
vs
These have more in common. Both agreements grant the service the right to quietly update the agreement on their website without notification or obtaining any further consent. PyPI didn’t have this provision in their previous terms either, and should not be given the benefit of the doubt for it IMO.
I’m not the person you asked, but I think it’s a signal that the organization is considering diversifying their revenue streams at the expense of user rights. It’s also an indication that they’re spending money on lawyers that could be spent on the service. Does that help?
But this new Firefox TOS isn’t about data I might send to Mozilla, it covers “input information through Firefox”. And the supposedly lawful bases of data gathering they have in their privacy notice are both vague and broad.
You underestimate my capacity for outrage! ;-)
Is the outrage you are outraged about not something pointless to be outraged about?
If my comment about patching it out came across as dismissive, I assure you it was not intentional. (In fact, I’m concerned that Debian will fail to do so.) I count myself lucky to be insulated from the worst of this shit by my distro, but the fact is that most people aren’t, and even if it doesn’t affect me personally I see it as a serious problem (but not a surprise, given Mozilla’s recent track record).
Thanks for clearing that up - your comment was indeed a bad example to pick. Clearly it would have been better to pick the one sarcastically leaning towards “It’s not like Mozilla will feed your input to an AI”, completely oblivious to the fact that five of the articles on the frontpage of the Mozilla blog are about AI.
If you use a distro-packaged build of Firefox that comes from competent packagers, this will certainly be stripped out. Unfortunately it’s not clear to me that said competent packagers exist. Debian in most cases is very good at this kind of thing, but the Debian packager for Firefox is employed by Mozilla and has a terrible track record for removing even the most egregiously unpopular shit from Firefox.
I’m curious to see what Fedora will do.
I would suspect that “having a terrible track record for removing even the most egregiously unpopular shit” is likely a direct prerequisite of Debian’s permission to use the Firefox branding.
IOWs, if stripping-out ever happens, in any distro, the result will have to be called Iceweasel.
That or LibreWolf. Either one would be a big improvement IMO.
From the new Terms of Service:
It sounds like this wouldn’t apply to distro packages, depending on what the undefined “authorized sources” are.
I think you’re parsing it the wrong way around.
It’s the terms of service. You don’t have to agree to terms of service to use (read, modify, etc.) the source code, because the source code in itself does not offer any services.
You only have to agree to the terms of service to use a binary that presumably has the API keys (that would enable the binary to communicate with the provider of the service) compiled in. Whether you need to agree to these terms to use a given binary probably depends on whether it has Mozilla’s API keys compiled in.
Yes, and the would-be benevolent packager would take these there sources, which provide no services, and build his own executable code.
The question though is, did they need to accept the terms to download the sources? If the packager never needed to see the terms and didn’t agree to them, do the terms apply to this other binary?
What you’re describing would be a flagrant DFSG violation, so any packager would hopefully not package it in the first place if so.
I’d expect that the packager would have to agree to the terms of service (and, transitively, agree to bind their users to those terms) in order to get the Mozilla API keys, or something like that.
Does this new EULA conflict with the DFSG?
Edit: this doesnt apply to the source code anyway
Can you elaborate and/or provide a reference? The only identifier I could find for the packager was “Maintainers of Mozilla-related packages”:
https://qa.debian.org/developer.php?login=team%2Bpkg-mozilla%40tracker.debian.org
It would also be good to get some clarity on the channels that could be used to ensure Debian is following its principles. I know they pride themselves on some form of internal democracy.
I wish I had a better reference but all I have is https://hachyderm.io/@joeyh/113082688008129977 and recollections of other discussions I’ve had with him.
If you know of git-annex or any of his other work, you’ll know Joey Hess has been deeply involved in Debian in the past, but admittedly it’s not a super thorough source.
Of course they didn’t really answer the question “why now?” What is it about the “much different technology landscape” that makes the rights that users have by default under the law untenable?
Why now? I suppose Google, after overbuilding AI capacity, has them at financial gunpoint about feeding user data to their slop machines.
There is nothing stopping Mozilla Corporation, or even the employees themselves, from moving away from a for-profit model and towards a non-profit model like the Wikimedia Foundation. The only people holding a financial gun to Mozilla’s head are Mozillans.
Is that economically viable for Mozilla? Has anyone done the projections?
I would not be surprised at all if Mozilla couldn’t support Firefox development solely with donations.
Mozilla Foundation’s Dec 31 2023 expenses (page five):
Alas, these numbers reflect the Mozilla Foundation as a whole, and not Mozilla Corp specifically, but it’s likely that the lion’s share of “software development” expenses in the Foundation fall under the Mozilla Corp bucket (if that isn’t the case, then the situation would look a bit rosier, I suppose). Administration expenses come out to 192 million. Administration percentage of expenses is 38.7%.
Wikimedia Foundation’s 2023-2024 revenue shows total revenue of 177 million, with 145.5 million coming from:
Their expenses bucket 137.8 million for Programmatic (100 million of which is personnel), 17.9 million for Fundraising, and 21.3 million for General and Admin. Administration percentage of expenses is 12%.
Being pessimistic, and assuming Firefox could only ever match Wikimedia Foundation’s 145 million in revenue from individual users (which excludes large grants or other enterprise contracts), that leaves a large gap no matter how you slice it. Mozilla Foundation’s admin expenses are absurd, but even excluding them entirely and only focusing exclusively on software development Program expenses (260 million) leaves a 115 million dollar gap. Mozillans would need to cut their annual compensation in half to even come close to closing that gap in such a scenario (they’d still be about 15 million in the hole, and that’s still optimistic as it excludes any admin or other expense).
So, can it be done? That depends. If Mozillans didn’t want to take any pay cuts at all, then they’d need to get serious about monthly recurring revenue to make that happen. Firefox averages 150 million MAU (monthly active users). Taking the Wikimedia Foundation’s “if every user donated X this year” approach, that comes out to at least two dollars per user each year to get revenue of 300 million, which would cover Program expenses at least. Aiming for three dollars would give them 450 million annual revenue. A few different ways to hit that 450M target:
I suspect 25 USD/month is on the upper end of what significant numbers of folks would be willing to pay for. And I’m not convinced they could find 1.5 million monthly customers to do it. Meaning that whatever the shortfall was, they’d have to find elsewhere. I think they could find that shortfall in grant funding from governments, especially if said governments were genuinely interested in breaking up Alphabet’s monopoly.
Basically, I don’t think it’d be impossible. I don’t think it’d be easy either. But I definitely think it is necessary for the future of the web.
At that point, I would seriously root for open democratic governments to bank-roll that non-profit model. It would be a win-win, protecting those democracies and the common-goods internet that democracy needs to function.
There is an occasional comment or article suggesting that the EU should just fork Firefox and take over financing completely. I guess they could fine there Mozilla Foundation for gdpr breaches (e.g. with MDN) to bootstrap the first year.
Forking without their devs instead of steering its development community sounds to me like a sure disaster.
That’s what the idea essentially conveys. Take over Firefox and the devs and run it via some actual non-profit, fully funded by the EU and not by the businesses. Not saying it’s a good idea necessarily, but I often remember it.
It could also have something to do with the chaos being wrought by the new administration, causing many other worse things for people to pay attention to and creating opportunities to ratchet up unpopular things like the erosion of user rights with less blowback.
https://www.mozilla.org/en-US/about/legal/terms/firefox/
:)
That’s… wow. Thank you for highlighting that. I am seriously considering using something other than Firefox for the first time in… ever. Regardless of how one might choose to interpret that statement, it’s frightening that they would even write it. This is not the Mozilla I knew or want. I’d love to know what alternatives people might suggest that are more community focused and completely FOSS, ideally still non-Chromium.
Thankfully, the lawful base for data use is spelled out in their privacy policy:
https://www.mozilla.org/en-US/privacy/firefox/#lawful-bases
e.g. Browsing, Interaction and Search data are “Legitimate interest” and “Consent”-based.
Consent being the kind that I haven’t given, but I’m supposed to actively revoke? Until the next update?
That unfortunately seems to be the current usage of the term “consent” in the tech industry.
Fortunately, that’s not consent as the GDPR defines it
Isn’t it? Most GDPR consent screens have an easy “accept to everything” button and requires going through multiple steps to “not accept”, and many many more steps to “object” to their “legitimate interest” in tracking for the purposes of advertising. As long as these screens remain allowed and aren’t cracked down on (which I don’t foresee happening, ever), that’s the de facto meaning of “consent” in GDPR as far as I’m concerned: something that’s assumed given unless you actively go out of your way to revoke it.
It’s not what the text of the GDPR defines it as, but the text isn’t relevant; only its effect on the real world is.
Yes, definitely. Consent in GDPR is opt-in not opt-out. If it’s opt-out, that’s not consensual. And the law is the law.
Furthermore, for interstitials, to reject everything should be at least as easy as it is to accept everything, without dark patterns. Interstitials (e.g., from IAB and co.) first tried to make it hard to reject everything, but now you usually get a clear button for rejecting everything on most websites.
As I mentioned in another comment, the DPAs are understaffed and overworked. But they do move. A real-world example of a company affected by the GDPR, and that tries testing its limits, is Meta with Facebook. For user profiling, first they tried the Terms of Service, then they tried claiming a legitimate interest, then they introduced expensive subscriptions for those that tried to decline, now they introduced a UI degradation, delaying the user scrolling, which is illegal as well.
Many complain, on one hand, that the EU is too regulated, suffocating inovation, and with US’s tech oligarhs now sucking up to Trump to force the EU into allowing US companies to break the law. On the other hand, there are people who believe that the GDPR isn’t enforced enough. I wish people would make up their mind.
Those are different people, all who have made up their mind.
I thought I made it reasonably clear that I don’t care that much about what the text of the law is, I care about what material impact it has on the world.
I corrected you with facts, and you’re replying with your feelings. Fair enough.
To be fair, @mort’s feeling may come from non-actually-GDPR-compliant cookie consent forms. I have certainly seen where I couldn’t find the “reject all” button, and felt obligated to manually click up to 15 “legitimate interest” boxes. (And dammit could they please stop with their sliding buttons and use actual square check boxes instead?)
I think the worse case is you click “reject all”, but you don’t actually reject all, and the legitimate interests are still checked.
The facts you provided aren’t relevant. I’m talking about the de facto situation as it applies to 99% of companies, you’re talking about the text of the law and enforcement against one particular company. These are different things which don’t have much to do with each other.
You even acknowledge that DPAs are understaffed and overworked, which results in the lacking enforcement which is exactly what I’m complaining about. For what I can tell, we don’t disagree about any facts here.
Well, other people in this sub-thread are talking about GDPR. You might have switched the topic, but that isn’t alexelcu’s fault.
I’m talking about GDPR as well, focusing about what impact it has in practice. I have been 100% consistent on that, since my first message in this sub-thread (https://lobste.rs/s/de2ab1/firefox_adds_terms_use#c_3sxqe1) which explicitly talks about what it means de facto. I don’t know where you got the impression that I’m talking about something else.
But there is enforcement, it’s just slower than we’d like. For example, screens making it harder to not opt in rather than opt in have gotten much rarer than they used to be. IME now they mostly come from American companies that don’t have much of a presence in the EU. So enforcement is causing things to move in the right direction, even if it is at a slow pace.
There is a website tracking fines against companies for GDPR violations [1] and as you can see, there are lots of fines against companies big and small every single month. “Insufficient legal basis for data processing” isn’t close to being the most common violation, but it’s pretty common, and has also been lobbed against companies big and small. It is not the case that there is only enforcement against a few high profile companies.
[1] https://www.enforcementtracker.com/
Why do you lay this at the feet of GDPR?
it’s the other way around - most of the time you have to actively revoke “legitimate interest”, consent should be off by default. Unfortunately, oftentimes “legitimate interest” is just “consent, but on by default” and they take exactly the same data for the same purpose (IIRC there are NGOs (such as NOYB, Panoptykon) fighting against IAB and other companies in those terms)
“Legitimate interest” is the GDPR loophole that ad tech companies use to spy on us without an easy opt-out option, right? I don’t know what this means in this context but I don’t trust it.
It is not, ad tech has been considered not a legitimate interest for… Ever… By the Europeans DPAs. Report to your DPA the one that abuse this. There have been enforcement.
Every website with a consent screen has a ton of ad stuff under “legitimate interest”, most ask you to “object” to each individually. The continued existence of this patterns means it’s de facto legal under the GDPR in my book. “Legitimate interest” is a tool to continue forced ad tracking.
Yes, all of that is illegal under GDPR.
The problem has been that DPAs are understaffed and overworked.
I don’t think you’re disagreeing with me. It’s de jure illegal but de facto legal. I don’t care much what the text of the GDPR says, I care about its material effect on the real world; and the material effect is one where websites put up consent screens where the user has to “object” individually to every ad tech company’s “legitimate interest” in tracking the user for ad targeting purposes.
I used to be optimistic about the GDPR because there’s a lot of good stuff in the text of the law, but it has been long enough that we can clearly see that most of its actual effect is pretty underwhelming. Good law without enforcement is worthless.
No, it’s de facto illegal a well, law enforcement is just slower that we’d like. Ask, for example, Facebook.
De facto illegal for entities at Facebook’s scale? Maybe. But it’s certainly de facto legal for everyone else. It has been 7 years since it was implemented; if it was going to have a positive effect we’d have seen it by now. My patience has run out. GDPR failed.
I just gave you a concrete example of a powerful Big Tech company, with infinite resources for political lobbying, that was blasted for their practices. They first tried hiding behind their Terms of Use, then they tried claiming a legitimate interest, then they offered the choice of a paid subscription, and now they’ve introduced delays in scrolling for people that don’t consent to being profiled, which will be deemed illegal as well.
Your patience isn’t important. This is the legal system in action. Just because, for example, tax evasion happens, that doesn’t mean that anti tax evasion laws don’t work. Similarly with data protection laws. I used to work in the adtech industry. I know for a fact that there have been companies leaving the EU because of GDPR. I also know some of the legwork that IAB tried pulling off, but it won’t last.
Just the fact that you’re getting those interstitials is a win. Microsoft’s Edge browser, for example, gives EU citizens that IAB dialog on the first run, thus informing them that they are going to share their data with the entire advertising industry. That is in itself valuable for me, because it informs me that Edge is spyware.
I agree that the “we’re spying on you” pop-ups is a win in itself. I’m just complaining that it’s so toothless as to in practice allow websites to put up modals where each ad tech company’s “legitimate interest” in tracking me has to be individually disabled. If the goal of the GDPR was to in any way make it reasonably easy for users to opt out of tracking, it failed.
I’m not so sure. I’ve even seen this used as an argument against the GDPR: The spin they give it is “this is the law that forces us to put up annoying cookie popups”. See for example this article on the Dutch public broadcasting agency (which is typically more left-leaning and not prone to give a platform to liberals).
Roughly translated “all innovations in AI don’t work as well here as in the US. And why do you have to click on cookies (sic) on every single website?”
I have seen that as well, and I think it’s bullshit. The GDPR doesn’t force anyone to make any form of pop-up, nobody is forced to track users in a way which requires consent. The GDPR only requires disclosure and an opt-out mechanism if you do decide to spy on your users, which I consider good..
I agree, but at the same time I think the average user just sees it as a nuisance, especially because in most cases there’s no other place to go where they don’t have a cookie popup. The web development/advertising industry knowingly and willfully “complied” in the most malicious and obnoxious way possible, resulting in this shitty situation. That’s 1 for the industry, 0 for the lawgivers.
I agree that it didn’t have the desired effect (which, incidentally, I have spent a lot of this thread complaining about, hehe). I think everyone was surprised about just how far everyone is willing to go in destroying their website’s user experience in order to keep tracking people.
I’m not sure if you’re deep in grumpy posting or didn’t understand the idea here, but for legitimate interest you don’t need to agree and companies normally don’t give you the option. If you’re talking about the extra options you unset manually, they’re a different thing. The “legitimate interest” part is for example validating your identity through a third party before paying out money. Things you typically can’t opt out of without also refusing to use the service.
If you get a switch for “tracking” or “ads” that you can turn off, that’s not a part of the “legitimate interest” group of data.
I’m sorry but this isn’t true. I have encountered plenty consent screens with two tabs, “consent” and “legitimate interest”, and where the stuff under “consent” are default off while the stuff under “legitimate interest” is on by default and must be “objected to” individually. Some have an “object to all” button to “object” to all ad tracking in the “legitimate interest” category.
Here’s one example: https://i.imgur.com/J4dnptX.png, the Financial Times is clearly of the opinion that tracking for the purpose of advertising counts as “legitimate interest”.
I’m not saying that there’s any relationship between this pattern and what’s actually required by the GDPR, my understanding of the actual text of the law reflects yours. I’m saying that this is how it works in practice.
So when I login to lobste.rs (or any other important website) do I grant them the permission to use my credentials? ;-)
Pretty much
this comment remains property of the Mozilla Foundation and is presented here with their kind permission
Mozilla updated the article with a clarifying statement:
the problem is it doesn’t clarify anything. “basic functionality” is not defined. my guess is they want to be able to feed anything we type or upload to a site, to also be able to feed that into an LLM. “anything other than what is described” doesnt help because what is described is so vague as to mean anything “help you experience and interact with online content”
That is… not clarifying. And not comforting. “What is described” in the ToS is “to help you navigate, experience, and interact with online content.” That’s absurdly vague. And what is described in the Privacy Notice is absurdly broad:
Yes. That’s the fucking point.
I’m glad we have this contextless legalese to clarify things. I wonder if there’s some kind of opt-in data collection in Firefox that Mozilla might have legal obligations to clarify their rights to? Couldn’t be that… No, let’s put a pause on critical thinking and post stupid TOS excerpts as if Mozilla are going to steal our Deviantart uploads and sell them as AI training data.
If they need a ToS for a particular feature, then that “contextless legalese” should be scoped to that feature, not to Firefox as a whole.
This is precisely why the same organization should not do all of these things. If they want to do non-tool stuff to continue funding their mission they should start up independently managed companies that can establish these consents for a narrow band of services. They can give the existing organization control as a majority shareholder, with dividends flowing back to the main organization. That is the way to ensure that incentives don’t become misaligned with the mission.
They’re future-proofing their terms of service. That’s even worse than future-proofing one’s code, Though for different reasons.
That language comes off a bit … onerous
But what does it mean? To “navigate”.
That’s it I guess. Thanks for the find! Firefox is dead to me now. What’s the non-evil browser to go to nowadays?
librewolf seems to be the rage now: https://librewolf.net/
On MacOS/iOS there is the Kagi browser Orion: https://kagi.com/orion/
I don’t know where the author was during the late twenty-teens, but I recall the HTTPS revolution being quite noisy with the whole “ENCRYPT ALL THE THINGS!” being screamed from the roof tops. Also, does this mean I need HTTPS for connecting to 127/8 and 192.168/16 networks? I either need to expose the internal network or run my own CA (and if I’m correctly listening to “security experts” is something I should not do)?
I just hope that Apple doesn’t remove http support from Safari until they fix guest network sign in. Thanks to whoever runs neverssl.
That’s an issue on every OS. I used NeverSSL several times this week.
I know but since this month I both got a new job - with crappy WiFi in office - and a Mac to go with it - so I’m affected. I don’t use these networks at home usually :)
You can use ACME DNS validation (or expose only what’s needed for HTTP validation, only when needed, but that’s less easy).
That is still exposing my internal network.
How so? With ACME DNS-01 the only thing you need to expose is a DNS server. Which you can easily put outside your internal network.
Even it’s the only thing, it’s still open. Whereas now nothing is open. It may be low risk, but compared to 0, it’s a lot.
Another thing though is that someone needs to set that up. I did configure my own
bindinstance 20 years ago, personally and professionally, but most people don’t have a clue how to set up WiFi semi-securely.A third thing here is that someone now needs to teach the end users. I’m a professional software engineer and I’ve seen many developers that have no clue how any of this works. They couldn’t configure an nginx without training and time.
Note that I’m not judging the idea of forced HTTPS either way, I’m just pointing out some of the obvious costs.
The DNS server used by the challenge doesn’t have to be on your LAN.
Basically to host a LAN accessible service on
thing.home.example.com, I do:thing.home.example.comThis is of course all automated in my setup. And I actually use a wildcard cert to not leak the list of services I host in the cert transparency logs. The security downside of a wildcard doesn’t matter to me in this case.
With plenty of other Western governments hungry to attack digital privacy under their current administration, this is going to prove to be letting camel’s nose into the tent. They will be lining up to be the next customer of this feature.
I thought us and uk secret services potentially share this type of data freely? Something five eyes or whatever, for five agencies?
Five countries: US, UK, Canada, Australia, New Zealand
Here’s the LKML link.
Personally I worry more about the cost (and not just energy cost) of training models than the cost of inference, especially for models where inference can be run from a laptop – a single inference is not going to consume much energy, because laptops are reasonably low-power tools. Fine-tuning an existing model looks to be feasible in reasonable amount of compute (and thus energy), but training a new foundational model from scratch is counted in TWh range these days, has been increasing rapidly in the last few years, as is something that all large tech companies (and all technologically advanced states, all militaries, etc.) are going to want to learn to do to remain technologically independent and avoid poisoning issues that nobody knows how to detect.
Said otherwise: using an LLM is not that expensive energy-wise. But collectively getting used to depending on LLMs more an more gives money and power to entities that can train new ones, which incentivizes a lot more training, and that is expensive.
I’m not sure how to estimate this total training energy cost – as previously discussed, the easiest estimate I could think of is to look at data-center GPU sales to estimate the global (training + inference) power usage – and we are looking at millions of new units sold each year, which translate into several GW (several nuclear plants worth) of instantaneous consumption – and the current trends is that this is going up, fast.
If you read through the cited article by Husom et al. you’ll see that a laptop actually uses more energy than a better specced workstation or server. I would guess the main reason is that it takes longer to process and thus stays at an increased power draw for longer.
Also consider the fact that training a model is a one-off energy cost, that gets less severe the more the model is used. For inference, the opposite is true.
About laptops, I think we agree. I pointed out that if we can reasonably run inference from a laptop, then that means a single inference is reasonably cheap – even cheaper on specialized hardware.
Training may be a one-off energy cost for a given model, but that does not mean that it is small. From the post that we are commenting on, it looks like training one model is of the order of 10^12 times more energy-consuming that running one user query (ChatGPT: 1.3TWh of training, 2.7Wh per query). Do most models actually get a thousand billion queries over their lifetime, before they are dropped to use a better model? ChatGPT appears to serve one billion queries per day these days; at this rate they would need to run for 500 days on the same model to break even. From a distance it appears that they are not breaking even, they change the foundational models more often than every 500 days – but maybe they avoid retraining them from scratch each time? And this is going to be much worse the more entities start training their own foundational model (there are already more companies doing this than just OpenAI, including those we don’t know about), as they will each get a fraction of the userbase.
Okay, then it seems like we mostly agree. All in all, it is very difficult when the only information one has are estimations like I made in this post, and not any actual information. “Open” AI really is a misnomer.
Doesn’t this analysis show that training cost is still not that important? Even if they only use a model for 100 days, that means that training cost is 5x inference cost, but inference cost is tiny, and 6x tiny is still tiny. If, say, drinking an extra cup of coffee costs a comparable amount of energy as typical LLM use including training, that seems entirely reasonable.
I suspect that future growth is fairly well limited by the payoff it is bringing us. Investors will bet a couple of billion dollars on something that may not pay off, but it seems unlikely that they’d invest several trillion without solid evidence that it will pay off in terms of value generated.
The problem is that we don’t understand yet what the payoff is, and the natural response in this case is to experiment like crazy. The LLM boom came from the fact that we got substantial quality improvements by making models much larger and training them a lot more. The next thing to try of course is to give them a lot more data again (scraped all the textual web? maybe we could train them on all of Netflix this time!), and to train them a lot more again. There is a very real risk that, three years from now, half the R&D budget in the world is being spent powering GPUs to do matrix multiplications. (It would also be bad for the environment to spend the same amount roasting and grinding coffee beans, but that does not make me feel particularly better about it.) It is of course possible that this risk-taking pays off and we are ushered in a new world of… (what? robot-assisted comfortable living for all?), but it is also possible that current techniques reach a plateau and we only notice after burning way too much fuel on them.
The payoff is also not an objective metric, it depends in a large part on how people view these things. As long as people are convinced that LLMs are godlike powers of infinite truth, they are going to pay for them, and the payoff will be estimated as fairly large. If we try to have a more balanced evaluation of the benefits and the costs of those tools, we might take the rozy glasses off (low confidence here, some people still believe in the blockchain), and the payoff – and incentive to spend energy on it – will be smaller.
Sometimes it looks like the only thing that can force people to reduce pollution is not data, information or basic common sense, but just energy becoming fucking expensive. I suspect that this would definitely make people behave more rationally about LLMs.
Blockchain was never meaningfully useful; it was (and is) a speculative bubble. LLMs already are useful, and millions of people use them. Spending a lot of R&D money on it is entirely justified, and this R&D is mostly privately funded anyway. The glasses aren’t at all too rosy, in my view, rather the opposite. Sure, there are AI hypers that have highly inflated expectations, but most people underestimate what LLMs can do.
Even the basic ability to have a conversation with your computer would have been considered absolute sci-fi 5 years ago. These things type out hundreds of lines of code in a few seconds, often without making any mistakes, without ever even running that code. When I gave o3-mini-high some math puzzle I came up with, it solved it in a minute. Out of colleagues and students, only one was able to solve it. Maybe it existed on the internet somewhere, and maybe it was just a glorified fuzzy lookup table, but fuzzy lookup tables are very useful too (e.g., Google).
Yesterday I “wrote” 98% of the code of a UI for some decision procedure that I was working on by literally talking to my computer out loud for an hour. It compiled the Rust decision procedure to WASM, set up a JS api for it, created the interface, wrote a parser, generated examples based on the Rust unit tests, etc. I wrote maybe 20 lines total myself. Heck, I haven’t even read 80% of the code it wrote.
This is all for some infinitesimally small fraction of the energy use of various frivolous things people regularly do. This seems to me like getting a functional magic wand from the Harry Potter world and then complaining that a branch of a tree had to be cut off to get the wood. Yes, sure, we don’t want to cut down the entire Amazon rainforest to make magic wands, nor may the magic wand solve all of our problems. But the more pertinent fact here is that it does magic, not that a branch was cut off from a tree.
I don’t disagree. I should clarify that my worry is not, per se, that people invest too much effort in LLMs – I think of them as the alchemy of the modern times, a pre-scientific realm of misunderstandings and wonders, and I find it perfectly understand that people would be fascinated and decide to study them or experiment with them. My worry is that this interest translates into very large amount of energy consumption (again: half the R&D money could be spent powering machine learning hardware), as long as the dominant way to play with neural networks is to try make them bigger. Most sectors of human activity have been trying to reduce energy consumption in the face of expanded usage, but the current AI boon is a sector that has been increasing its energy needs sharply, and shows no signs of stopping. The availability of clean-ish energy has not scaled up for that demand.
I do also think that there are unreasonable expectations about AI in general – people turning to neural networks for tasks that could reasonably be done without them, especially in industry. For instance a friend working in a garbage-collecting plants was telling me about futile attempts by the company to train neural networks to classify garbage from a video feed, which were vastly underperforming simple heuristics that were already in place.
I think this isn’t the case.
My layman understanding of the consensus is that people agree that there are limits for scaling based on input data. We maybe aren’t there yet (new data sets such as video are probably still ripe) but I don’t think we’re far, personally.
DeepSeek’s whole thing is that it’s a much smaller, faster model, with competitive performance to vastly larger ones - all because they changed the training process.
We’ve already observed optimization on model sizes, training costs, etc, which is unsurprising since companies are incentivized to do this (faster training == faster iteration == money).
This is tricky from a policy perspective (raising the cost of all energy consumption seems sort of obviously bad, but we’ve seen things like “off-grid hours” work, etc) but regardless energy costs are, of course, already passed on to consumers. Perhaps VC money is making the difference right now, but ultimately consumers eat the costs of electricity with a margin, otherwise companies wouldn’t be able to stay up and running.
It is one-off, but I think it’s a bit misleading - the commenter above mentions this exactly: more and more entities are training more and more (and more powerful) models. So while it is a one-off cost, it’s also constant.
You’re not wrong but that’s how pretty much everything works. Isn’t it disingenuous to focus on LLMs in this regard?
I understand that it’s easier as it’s the new thing but we should do environmental impact analysis on everything else, too. How much do we spend making video games? How much more do we spend playing them? How about the whole of TV? Movies? Books? Fast fashion? Cars? Housing? Food? I haven’t seen much of the backlash on environmental impact grounds to anything else along with LLM.
And that makes a valid argument in isolation feel like another attack blown out of proportion. Sure, training data set is dubiously sourced. Sure, there are all sorts of moral questions about the use of LLM output. Let’s tack on environmental issues as well, why not? And let’s not bring it up for anything else to make it look like LLMs alone will drain all our water and use all the energy. Rings a bit hollow.
The specificity of LLMs is that they are a new technology that is exploding in popularity and is highly energy-consuming. This is not unique but it is still fairly rare. People where also very concerned about the energy cost of blockchains (and still are), which were in a similar situation, at least when proof-of-work was the only scheme widely deployed. Of course computing-focused websites like Lobsters are going to discuss the energy impact (and environmental impact) of computer-adjacent technologies much more than non-computer-related things like cars and housing – this is the area that we have some expertise in and some agency over. Digital technology is also one of the main drivers of increased energy consumption and pollution these days – right now it is comparable to transportation, but growing fast every year, while the impact of most other areas of human activity are increasing less fast, stable or even decreasing. (This could of course become much worse with a few bad policy decisions.)
This reads like whataboutism. I don’t know what your cultural and political context is, but around me there has been a lot of discussions of the impact of cars (why are people pushing for electric cars?), housing (entire laws have been passed, for example to force landlord to improve thermal insulation), food (why do you think is the proportion of flexitarian / vegetarian / vegan people growing fast in western countries?), planes (why are people talking about reducing flights to limit their carbon impact?), etc. You may live in a different environment where these discussions have been occulted, or have personally decided not to care about this and not get informed on the matter.
More facts, less feelings, please. If you can demonstrate that the energy/pollution costs of LLM is neglectible compared to other reasonable causes of concerns, that would be a very valid way to say that this is out of proportion. We could have more, better data about this, but there is already enough available to make ballpark estimates and make an informed decision. (For example, see this december-2024 article reporting that US data-centers energy consumption tripled over the last decade, consumed 4.4% of total US electricity in 2023, and that this proportion is expected to raise to 7%-12% by 2028.)
Well, that’s exactly the point I’m trying to make. Environmental impact concerns around LLMs never given in any context relating to other things. Sure, data is out there. I can bust out a spreadsheet and put in all the data and crunch the numbers and find the cost of my LLM query in burgers, ICE miles, books, TV minutes, or—like in the OP—shower minutes.
The issue in the discourse quality I’m trying to point out is that I have to go out of my way and do my own research. A valid critique of LLMs is rarely if ever put into an understandable context. And this is why I think it largely has no impact. What you call whataboutism is better reframed as putting things in context. Is 1.5 l of water for may LLM query a lot? An average person doesn’t know what to do with that number. They have nothing to relate it to. Maybe an hour of TV in the background worse? Maybe an hour of playing a game on their maxed out gaming PC is better? How would they know?
I’m a little confused about the purpose of this post.
It refers to another article by bitfield, who submitted it to Lobsters, and calls him his mentor.
I thought this article was going to be in dialog with the referred article, maybe expanding or critiquing it, but it literally copied many of the same subheadings verbatim, and a lot of the content is similar, just reworded.
Is this some SEO thing for backlinks and reputation?
I hate to be cynical, but the posts are extremely similar, and I’m not sure why bitfield submitted this one, instead of his own.
I think you need to connect some dots for us here.
One of my students writes a blog post about software testing, citing an article by me. I share it here, and that’s somehow… suspicious?
I promise you Jakub is a real person (you can find many videos of him giving talks, for example) and has his own views on this stuff: some of them aligned with my own, some not. Many of my other students write blog posts, too, and I don’t apologise for sharing and boosting them where I think their work deserves a larger audience. If you don’t agree, don’t upvote them, but let’s give the ‘just asking questions’ routine a rest, shall we?
The suspicious part was how incredibly similar the articles were, down to structure, order, and including “literally copied many of the same subheadings verbatim”. It doesn’t look naturally written at all. It looks like if your student sat down with your post and retyped it, changing small bits as they went along.
If not for the unusually high similarity, I wouldn’t have batted an eyelash.
To me it seemed a bit satirical, maybe, but I can’t be sure.
Can anyone explain what the kernel maintainer drama they’re talking about is?
Most of Asahi’s linux work is in Rust, such as their new GPU drivers for Apple Silicon. Part of that requires access to the DMA subsystem. An asahi maintainer other than Marcan wanted to upstream Rust bindings to DMA. These would have been part of the “Rust for Linux” subsystem, which explicitly is exempted from the usual kernel stability guarantees. The maintainer of DMA in Linux objected to them upstreaming those bindings, as he felt he would then be obligated to maintain the Rust code, and he did not want to maintain Rust. This was combined with some uncharitable comments on the Rust for Linux project’s goals. There has been a lot of debate on other internet forums about whether these comments were uncharitable about Rust, Rust for Linux, or some other distinction that personally I think doesn’t really matter.
Marcan and that maintainer then exchanged words on the LKML. Unsatisfied, Marcan appealed to his Mastodon for support. Linus then stepped in with a reply heavily condemning the appeal to social media and not really addressing the complaints that led to that point. Marcan later resigned from his kernel maintainerships, and now also from the Asahi project.
I think this understates the position of the maintainer. He sent a message to LKML that states it pretty clearly: he thinks a multi-language kernel is a terrible mistake. And I can understand his position — which to me demonstrates that there is a leadership failure happening, because this kind of fundamental conflict shouldn’t be allowed to go on this long. “Disagree and Commit”, as they say at Amazon.
https://lwn.net/ml/all/20250131075751.GA16720@lst.de/
I’ll go further, it misstates the maintainers position.
The maintainer never indicated “he felt he would then be obligated to maintain the Rust code”.
The maintainers “NAK” occurred in response to the following post
I.e. after the explicit statement that the rust people, not Hellwig, would be maintaining it separately.
(And then we reach the post you quoted, where he makes it clear that his reasons are that he doesn’t want Rust in Linux at all, not that he doesn’t want to maintain it)
Yes, I understated the understatement. :) His stated goal is to use his influence and control over the DMA subsystem to prevent Rust (edit: or any other non-C language), from being in the kernel at all. As his NAK says:
Classic LKML rhetorical style here, to use a loaded word like “cancer” to make what is actually a reasonable technical point.
It is not a reasonable technical point! It demonstrates a fundamental incuriosity about technical matters. Categorical technical statements are rarely reasonable.
I disagree in general, but not in specifics. I’ve spoken to Chris Hellman at lengths (literally 6 hours) about the topic of Rust and it was not combative. He was curious. It’s hard to sum up at length, but his disagreements are rooted in specifics and he’s a hardliner about the single source language thing. But not for the sake of being a hardliner. His hard-line stance comes out of different weighting of the issues at hand, not an outright dismissal.
Those people existing is to be expected and it doesn’t invalidate their opinions.
Obviously not a fan of LKML rhetoric, but yep, you get the tone you built over years.
You can disagree, but for the rest of us who haven’t had these super private, intimate exchanges, where we get to “really know” someone, what he puts into the public is our perception of him. To that end, he’s called the project a cancer, and declared he’s going to do whatever he can to stop it which is not what a curios person does. He comes across instead as a religious zealot or “hardliner” in your terms.
That’s fair, but it’s also a categorical mistake to read a (brusque and inappropriate) email and judge on a whole person for days and weeks on the internet. I’m not saying you “really need to know someone”, but you also need to be aware that hundreds of hours are spent on reading what those statements may mean.
Wanting to keep a large, complex, codebase in a single language is a very reasonable technical point.
Using the term “cancer” to describe it the adoption of another language, though, is inflammatory and probably didn’t help matters.
A general inclination towards this is reasonable (and one I agree with). A categorical statement is not.
Again, many other C and/or C++ projects have introduced Rust into them. Some have succeeded, while others have not. It is worth learning from them.
I’ve personally had to make the decision whether to allow an additional language into a large codebase, and there are some major negatives to consider that are unrelated to the technical aspects of the language. The maintainer said three times in the above excerpts that he doesn’t have a negative opinion of Rust itself.
I’ve always heard that Google allows only four languages in their codebase, and one of those is only because they invented it. :)
Yes, I was part of the first Rust team at Meta so I’m quite aware of the upsides and downsides! That’s why I said “categorical”.
By “categorical”, you mean saying “no second language could ever have upsides that overcome the downsides of having two languages”? I agree that’s a pretty conservative position. It’s hard to even imagine another candidate for second language, though (I dunno, Zig maybe?), so I didn’t take it as literally categorical.
This is the quote:
This is quite categorical!
I think Zig sees itself as a candidate second language for C projects, but that’s probably beside the point; nobody’s trying to bring it into the kernel.
Indeed, but that call was already decided when Rust support was first merged.
The goal of the experiment is to explore those tradeoffs. You can’t do that if you don’t merge things!
And yet they are one of the companies pushing Rust for Linux… They understand the tradeoff. The rust for linux people understand the tradeoff. They just think it is worth it.
Totally. But that does seem like a valid engineering question people could disagree about.
However, this is exactly the kind of huge impactful question that should get escalated to the chief architect. And if there’s a sanctioned scouting effort to gather experience to figure out the right answer, somebody found to be actively subverting that effort should be…corrected…by the chief architect.
It’s interesting that they won’t allow it in their internal codebase but they’re happy to let the Linux maintainers take on the burden of adding it there.
What’s your source for Google not allowing Rust in their internal code base? Google publishes crate audits for google3: https://github.com/google/rust-crate-audits/blob/main/manual-sources/google3-audits.toml
I’m not sure where you get that from - I worked at Google and literally all the code I wrote was in rust
In the google3 main repo?
Is the Linux kernel a place for curiosity or for consistency?
R4L is officially an “experiment” with goal to explore tradeoffs. I.E. satisfy curiosity.
This whole debate is so tired. The “including Rust” ship has, at least temporarily, sailed, and maintainers blocking it based on not wanting Rust/a second language doesn’t make sense when the whole point is to explore that option.
Maybe all the code will get deleted at some point, but the code needs to be merged first to then decide that based on how well it works in practice.
Why does it need to be merged though? If it’s officially an experiment, maintain it in a separate R4L tree, and everybody’s happy?
To have a thorough experiment, it must be merged into the official tree. One must be able to see how this interacts with a diverse set of hardware and software configurations, and also how this affects the development process inside the kernel community. Some maintainers are afraid of how the introduction of Rust would affect them in the future and the maintainability of the code base. Without it being in the official tree, there wouldn’t be any conclusion on those points.
It sounds like some of the kernel maintainers at least don’t want to have to pay the cost of this large-scale experiment? Like they don’t feel it’s worth the downsides of massively complicating the kernel into a multi-language codebase? Like they understand the potential benefits of memory safety but they feel they are outweighed by the risks?
These individuals believe that they know the result of the experimental before it’s been run. But they do not, because they cannot, by definition. Of course they don’t want to pay the cost, because change is hard, and deep down they know that there’s a chance that the experiment is successful, and thus their responsibilities will either change or they will have to cede responsibility to someone more capable.
Whether there is benefit to Rust in theory, there is little question. It provably solves problems that the kernel has. Whether it can solve those problems without causing larger ones is why the experiment is required. Blocking the experiment for essentially any reason is more or less indefenisble, because the entire point is that they don’t know.
That sounds way too confident and assured of a statement to me. You are discounting the opinion of subject matter experts as if they have no idea what they’re talking about. If someone came to you at your workplace and said that they want to run an experiment to convert your codebase to a new language, and they want you to help maintain and support that experiment while also delivering on all your existing obligations, would you be so quick to accept that? I don’t know about you, but personally I would and have pushed back on rewrite suggestions as a bad idea, despite the perceived benefits (for example, Management thinks that rewriting in JavaScript will make it easier to hire people for the project in the future).
Would rewriting in JavaScript have possibly made me redundant? Maybe. But it would also be massively expensive, cause huge churn, and have a real possibility of failing as a project. We can’t just ignore the very real possibility of risks because of the expected benefits.
There is no interpretation in which it’s too assured. My position is that the result of the experiment is unknown, and that the potential benefits are known, both of which are hard facts.
I have run such experiments in my workplace before. Some have succeeded and some have failed. The successes have massively outweighed the impact of the failures.
Assuming that you know the outcome before you begin is the pinnacle of hubris, which we see time and again with these entrenched maintainers. They may be domain experts, but they deserve every ounce of criticism that they receive. Closed mindedness is not how progress is achieved.
My point is that there must be some threshold that you use to decide whether some experiment is going to be useful to run or not even worth the effort because your experience and expertise tells you otherwise? Or would you accept every experiment that anyone proposed? Eg, suppose someone wanted to model your entire technical stack with TLA+ for high assurance purposes? On paper it sounds like a great idea–formally verify your concurrency properties–but you don’t see how a reasonable project lead might say ‘While this project could certainly bring great benefits, it’s not worth the tradeoff with our team size and capabilities right now. That may change in the future’?
Some threshold must exist, yes. Presumably, that threshold should be somewhere below “the progenitor of the system has decide that the experiment should be run”, which is currently the case of RFL.
Individual system maintainers should not and must not be able to usurp the direction of the project as a whole.
Your core misunderstanding is that we are coming from a tabula rasa. We are not, and if your position is that Linus’s previous endorsements shouldn’t stand, then we’re having an entirely different (and much less tenable) conversation.
Is it though? To my understanding the decision was only that they will try it and see if it works on both the technical and the social levels. There was never any guarantee given that Rust will be shipped in Linux at all costs. To my understanding Linus’s approach is to see if Rust is overall viable for the kernel, and that includes the opinions of the other subsystem maintainers. Because if they are overall against it, the experiment is dead in the water.
where do the interests of actual linux users factor into this?
I like it when drivers work correctly and don’t break every other time I update the kernel due to weird concurrency issues.
then you should update your kernel twice as often, then it will be every fourth time instead of every other time.
Again, that decision has already been made. There’s no point questioning it over and over. Doing so is distracting from the debates that are important now: how can leadership do better at enforcing these project wide decisions to not let the situation fester like R4L, and reviewing the Rust code without raising the same solved questions again and again (concern trolling).
If you’re genuinely curious about those questions, there’s plenty of text written about it. LWN is a good place to start.
I hear you, but it is a bit in the nature of project that are very distributed in decision making with high individual authority that indeed, decisions will be questioned again and again and again at every point. Those projects need a long breath and the willingness to talk about points again and again and again.
It comes with its strength, particularly, those projects are extremely resilient and fast in other regards. You can’t have both. Pushing top-down decisions in such an environment can have other unintended effects.
It’s not something to take up (period, but even more so) with the R4L devs.
Doing that is like berating a cashier for the price of an item.
It sounds like the decision has actually not been made, just dictated by ‘top management’ and now getting pushback from the rank-and-file?
The guy who blocked it is ‘top management’, inasmuch as that exists for Linux.
sorry, the guy who blocked what? has the decision been made?
The guy who blocked the patches that the Asahi Linux folks wanted, which I think were to allow writing some rust drivers.
The guy who blocked has the authority to reject patches in their subsystem.
It wasn’t a patch in their subsystem though, it was in
rust/and used the DMA subsystem.You’re right that it’s in
rust/, tho unclear to me if it’s properly in the rust subsystem or not. You and others may be right that this patch may still go thru, that seems to depend on Greg KH and Linus.but did they have the authority to reject the patch?
I don’t believe so.
so he has the authority to block it, but not to reject it? him being the guy referred to above as “the guy who blocked the patches that the Asahi Linux folks wanted.”
So the original statement, “that decision has already been made,” is supposed to mean that the decision not to allow that patch has already been made?
ThinkChaos was talking about the decision to do the Rust for Linux (R4L) experiment.
I was talking about the decision to block the patch.
Sorry for the confusion.
how could the decision have been made, if the guy who blocked the patches is ‘top management’, inasmuch as that exists for Linux?
how was the decision made?
I sure hope it’s for both! If we cannot find a balance between the two, then we have failed as engineers.
Personally I think it does matter when a core maintainer calls a project that’s part of the kernel “cancer” and the guy who’s in charge doesn’t seem to care.
Marcan got a lot of mileage out of theatrically misinterpreting the “cancer” comment, and you’re carrying it here.
The actual comment:
The cancer is not Marcan, Asahi, R4L, or Rust. The cancer is anything-but-C. The cancer is a two-language codebase.
Still a very discouraging comment from a maintainer, and it bucks Linus’ decision that trying out Rust in the kernel is OK, but the distortion of this comment has been bugging me for the last week.
R4L is a project to make Linux a two-language code base. So he quite explicitly called the goal of the project cancer.
Yep, that’s what the commenter above is referring to. He thinks that project goal is unworthy. Or, well, in his words, cancer.
I don’t have an opinion, but I think a fair and less charged way than “cancer” of stating the position is just this: using Rust would have positive benefits, but they are far outweighed by the negatives of allowing a second language into the kernel.
The pros and cons of RfL are somewhat subjective, and the funal balance is hotly debated.
But I find it quite telling that Hellwig’s core argument (that a multi-lamguage codebase requires more work) is held by people who didn’t try doing any of that work. Whereas the kernel devs who started using Rust are explicitly saying “the API compat churn isn’t that big a deal, we can do that work for you if you want”.
We mostly hear about the drama, but it seems that the overall feeling toward Rust among kernel devs (not just RfL devs) is generally positive.
but then you lose the meaning conveyed by the metaphor, of something that grows beyond its intended scope until it kills the organism.
I don’t think the metaphor conveys any technical meaning beyond what I said. I don’t know what “kills the organism” is supposed to relate to. Is the “organism” the kernel? The community of kernel developers? And what does “kill” equate to in reality? It would be better to have the discussion in concrete terms rather than metaphors.
It does convey the personal outrage of the maintainer better, certainly.
thread depth limit reached; continuing here.
yes, I’m ignoring the less absurd things you’ve said because I don’t know what to take seriously when you haven’t retracted the most absurd thing.
one characteristic of cancerous growth is that it happens whether you want it to or not.
it doesn’t imply that, but I do think he would be much more OK with a little bit of Rust if the scope were somehow guaranteed not to expand. like I don’t know, maybe there’s a carve-out for testing a particular class of rust drivers in linux to help find bugs, but eventually the drivers are ported to C for use in linux and the rust version is used in a separate kernel written from scratch in rust?
so you wouldn’t say that you want to have no cancer at all in your body, ever? if it’s a non-cancerous tumor, I have much less of a problem with it being in my body.
I agree that “kill” could have multiple meanings.
I think “growing beyond its intended scope” is technical meaning that is not included in “the benefits outweigh the negatives.” you disagree?
Cancer has no intended scope, right? “Allowing into the kernel” conveys that there shouldn’t be any in the kernel.
sure. so do you think “growing” is a technically meaningful word that is not implied in “the benefits outweigh the negatives”?
I don’t want to dig into the minutiae of why this metaphor is inaccurate — my point is that using this metaphor at all is unprofessional and counterproductive because of the heavy emotional baggage it comes with. Just say “the introduction of Rust anywhere in the codebase will inevitably result in a growing amount of Rust in the codebase”.
It’s especially odd in this context, given the history of this metaphor regarding Linux.
okay, so do you retract your previous statement that “I don’t think the metaphor conveys any technical meaning beyond what I said”?
my intention was not to get into minutia either, but we’ve gone back and forth three times and you still haven’t retracted that statement so I don’t know where you stand.
so our enemies allow themselves to use the metaphor to efficiently communicate with each other, but we have to forswear it.
If you don’t want it there at all, it’s redundant to also complain that it’s growing. Cancer also has many other attributes besides “growing” that are irrelevant or contradictory to the position taken.
The reason he used that particular metaphor was to express disgust and horror, not to be technically accurate. If he just wanted to evoke “growth” he could have said it was like ivy. Disgust and horror are not an appropriate device to call on in this context, IMO.
Re Mr. Ballmer, the enemies were wrong that time, so don’t you think evoking that argument by using the same metaphor muddies the rhetorical water a bit?
if I can’t tell whether you stand by your statements, it feels futile to interpret them as if you believe them. that’s my issue with continuing in light of the fact that you are just leaving “I don’t think the metaphor conveys any technical meaning beyond what I said” without clarifying whether you believe it.
“complaining that it’s growing” is completely different from identifying the growth as a factor in the technical drawbacks.
Cancer has a multitude of characteristics, of which growth is only one. If anything, it has many characteristics contradictory to the point being made: it’s an undesirable outcome of natural processes, it is typically self-generated from within rather than deliberately introduced, it is hard to eliminate because it’s nourished by the same system that sustains healthy cells…
If you have to tell me exactly which part of the metaphor I’m supposed to pay attention to in order to get the technical meaning, then you aren’t conveying technical meaning, you’re muddying the water. Just say what you mean.
The additional meaning being conveyed here is not technical, it’s emotional, and I do agree emotional meaning was conveyed.
I take this to mean that you still stand by the statement that “I don’t think the metaphor conveys any technical meaning beyond what I said.” if you treat all technical metaphors like this, I’m glad you’re not my coworker.
or it could mean I am conveying technical meaning but you’re being willfully obtuse. just maybe…
Well, notice that you aren’t absorbing my actual point too well, because you’re very focused on making your own.
To give him the benefit of the doubt, he could have been trying to convey the technical meaning of “growth” in the sense of “if you let Rust in, we’ll just have more Rust”. However, that’s an utterly vacuous thing to say, because obviously the whole point of the exercise was to let Rust in and see what happens, and why would you do that unless you want to have more of it if the experiment succeeds? It conveys no useful information beyond “I don’t think Rust is a good idea at all, for reasons”; in other words, “the benefits don’t outweigh the negatives”. In fact it confuses matters because it implies he thinks Rust would be OK if only it didn’t grow.
I’m actually trying to be polite to the guy here, because I truly believe it was an emotional outburst, not an attempt at a technical argument. His actual technical argument (stated much more clearly elsewhere) is that there should be no Rust at all in the kernel, ever, not that once there is some, there will be more. So if “cancer” was supposed to mean “growth” as a technical argument, he misstated his own argument.
I don’t see the misinterpretation, lonjil said
Not
Or whatever phrase you would need to make your objection stand.
I could be wrong, but I get the feeling that people didn’t quite get why Hellwig used the word “cancer”. I could be wrong about this in a lot of ways, maybe I too misunderstood what Hellwid meant, maybe I misjudged how others are interpreting what he said. But allow me to add my 2 cents.
People seem to be interpreting what Hellwig said (i.e. calling R4L “cancer”) as him saying R4L is bad, evil, a terrible thing, etc. And I totally understand why many would think that, and would agree this is a terrible choice of words. But I think that Hellwig’s focus is actually on the fact that cancer spreads. A better word to use would probably be like “infectious”, or “viral”. Although I disagree with him, I think what he was saying is that, despite what Rust developers promised, Rust will spread to more corners of the kernel and increase the maintenance burden and will have a negative impact on the kernel as a whole.
I think technical leaders should focus on clarity rather than everyone else trying to do exegesis on what they write.
That’s very fair, I was not trying to defend Hellwig’s communication skills in the least here.
I think you’re right, but that doesn’t exculpate Hellwig in the least. One of the defining characteristics of cancer is that it is malignant. It seems pretty clear that he used this metaphor because he thought that Rust would spread and that it is dangerous and harmful.
yes, which is a perfectly valid technical position and to disallow a maintainer from expressing that position is completely insane.
“infectious” or “viral” have different connotations. viral would be if they thought the BSDs might catch it. infectious is not as specific as cancer; cancer specifically grows from one area and consumes more and more of the things next to it. infections can do that but not necessarily.
actively expelling the cancer probably would have drawn even more ire.
Unrelated to the rust issue, my understanding is that such toxic language actually isn’t all that strange in the Linux kernel maintainer community. It feels like they try somewhat not to overdo it, but they aren’t that much concerned about the toxicity. More like, “I’ll try but I don’t actually care”.
The only reason they’ve toned it down is due to financial pressure from major funders of LF.
The fact that many of those LF funders are also the employers of major Linux maintainers is also relevant, I feel.
I think it’s kind of crazy to label a vivid and extremely useful metaphor as “toxic.”
It’s useful in some cases, but when the effect of it is raising the temperature of the discussion, then I would say it isn’t useful in that instance. There are other ways to convey the same meaning that wouldn’t have attracted the same amount of strong emotions. We’ve seen examples of such alternate arguments in this thread that would have been more suitable.
sure. but then don’t fixate on the word, and acknowledge that their choice to use strong language can also be an effect of the raising temperature of the discussion, for which responsibility is shared among all participants.
There’s this idea that multi-language codebases are categorically a disaster, right? What really bothers me is how small-minded it is. There are many C and C++ codebases that have introduced Rust. It’s not easy but it is possible, and it often is worth it. Some attempts have been successful and others haven’t. You can learn so much from other people’s experiences here!
Simply calling multi-language codebases a cancer closes off discussion rather then opening it. That goes against the basic curiosity that is a hallmark of engineering excellence.
Honestly everything around R4L that has been going has just cemented in me the belief that the average long time Linux maintainer is fundamentally incompetent. They’re reasonably good at One Thing, but totally incurious and scared of anything outside of that One Thing they’re good at.
Do you suppose there might be some kind of incentive structure behind this pattern?
That seems likely, though I don’t think I can usefully speculate on the details.
Is it possible that we could take an experienced kernel maintainer’s opinion seriously instead of dismissing it immediately as incurious and ‘against’ engineering excellence? If a co-worker who specializes in a different domain came to you and kept insisting that something you know is very difficult and not worth the tradeoff is actually quite easy, would you be incurious and ‘against’ engineering excellence if you dismissed their wild suggestions without a deep discussion of the tradeoffs?
Evaluating who to take seriously and who not to is a matter of judgment cultivated over a lifetime.
Depends on many things, but in general, yes, of course, especially if they were someone I took seriously on other matters. If nothing else, it’s a chance to sharpen my arguments. If I keep having the same arguments over and over I might even step back and write an essay or two explaining what I believe and why I believe it. Here’s one that’s public that was a result of the sort of back-and-forth you’re talking about.
Your post strayed away from a factual re-telling of facts and towards narrativization here.
Reading the discussions on this elsewhere, there’s no way to describe this that wasn’t going to get accused of bias by people holding strong opinions on either marcan or on the rust for linux project.
I’m not accusing of bias (which is more or less unavoidable), but of constructing a narrative. The facts are: he exchanged words on the LKML and then wrote Mastodon posts about the exchange. The narrative you’re constructing is: he exchanged words on LKML, then because he was unsatisfied with how that discussion was going, he wrote Mastodon posts about the exchange in an attempt to gather support.
I think it would be better to leave out the guesswork about intentions, or at least clearly separate it from the factual retelling of events.
marcan has posted about this, so it’s hardly guesswork: https://lore.kernel.org/rust-for-linux/208e1fc3-cfc3-4a26-98c3-a48ab35bb9db@marcan.st/
Correction: Marcan never “exchanged words” with Christoph Hellwig on the patches, which had not even been posted by him. He just replied to the thread with a tirade.
Amen:
I have a different recollection, it seems: Junior: let’s reboot the computer Senior: let’s troubleshoot, learn what’s wrong and fix this Staff: let’s do magic incantations and summon dark powers Principal: let’s reboot the computer
Ah, an instance of one of my favourite memes! There’s something very pleasing when the “smart” solution looks externally identical to the dumb solution.
I enjoyed the post, but found it funny that the code examples are all in Rust, but the linked encoder/decoder is implemented in TypeScript :)
Rust doesn’t work in a browser, I guess :)
Except wasm
Couldn’t agree more! I think I shared this on lobsters years ago, but my favorite thing in my
~/.zshrcis this little guy:If I run ‘
t blah’, I’ll drop into a temporary directory with a name like/tmp/blah.1tyC. I can goof around without worrying about cleaning up the mess later. When I’m done I canpopdback to where ever I was. On the off chance I like what I did, I can just move the folder somewhere permanent. I use this every day; my$HOMEwould be unnavigable without it.I like to automate the “
popd && rm” part in the alias directly. This bash function enters a subshell inside the tmpdir and when Iexitor^D, it pops back to the previous directory and deletes the tmpdir – with a little safeguard when you mounted something inside it! Had a bad time when experimenting with mount namespaces and accidentally deleted my home directory because it was mounted inside this tmpdir …Here is a more recent one for fish as well.
RAII, shell-style
I have nearly the same function, but I’m using
~/tmphas the base, precisely because/tmpis often emptied on boot and I know that I sometimes want to go back to these experiments.Using
~/tmphelps keep my home directory clean and makes it obvious that the stuff is easily removable, but if, on an off-chance, I might need one of those experiments again, it’s there, waiting for me in~/tmpeven though I might have rebooted in the mean time.But in general, yes, this is the way to go. I’ve learned it here on lobsters years ago and I’m using it daily.
I like this idea, I came to the same conclusion as you, but instead of doing ~/tmp, I added “tmp” to my personal monorepo’s .gitignore. But sometimes I would put random downloads in that folder, lol. Thanks for the idea, I feel like having two of these makes sense, one for coding stuff and one for everything else
Heh, i do both:)
This is absolutely fantastic. I’ve been independently using this trick for years (through ^R instead of an alias) and I love it too. I didn’t know
mktemptook an argument though, thank you!I like to do programming-language-specific versions of this. So like “newrust” creates a temp dir, puts a Cargo hello world project in there, cd’s in there, and opens my editor. Similarly “newgo”, “newcpp”, etc. Great for playing around.
I tend to have two use-cases for these, one that’s very very temporary (single shell instance), and one that should stick around a little longer (i.e. duration of the boot). This works out to something along the lines of
Notably, I use
cdbecause I rarelypopdat the end, I usually just close the shell once I’m done with it. I probably should do it anyway though :)Oh man, thank you so much, this is the kind of “small purpose, great impact” tools that I love! Here’s my fish version where the argument is optional.
What kind of experiments are you doing with folders exactly? I’m curious about the overall workflow where you would like a folder’s contents or not.
Personally I often try out small experiments with different programming languages which need a project structure to be set up before they can work properly. For this a new folder is often needed.
For me, I just do a lot of stuff directly in /tmp, but this seems nice to keep things organized in case I want to move a directory to somewhere more permanent.
Scenario A: I’m trying to test something in C/Zig and don’t want to pollute my actual project directory. I just make a /tmp/test.{c, zig} and compile in /tmp. I think putting it in a temp dir would be nice, if unnecessary.
Scenario B: I, semi-frequently, will clone git repos into /tmp if I’m just trying to quickly look at something. Occasionally, I clone multiple related repos at the same time. Having a temp dir to keep them together would be nice if I ended up wanting to move them out.
For me it’s poking around in exploded jar files, or tarballs, or other archive formats mostly.
Sometimes you don’t just want to list the contents, or maybe there are container formats in container formats, a zip inside a tar, inside a cpio for example.
I want to unpack all these, look around, examine files, etc. without worrying about needing to clean up the mess after.
I have the exact same function. It’s so freakin’ useful.
I am curious, how long does it take to scan 9k photos like this?
There is a third way, which is my own personal preference: as few dependencies as possible and make them as small as possible.
In JavaScript this can mean Vanilla JS - trying to build as much functionality as possible using native browser APIs without pulling in dependencies. I’m finding this approach increasingly productive, browsers are really good these days.
In languages like Python or Go this can mean sticking to stuff in the standard library, which tends to be a whole lot more stable (aka boring) than the wider dependency world.
One of the things I like most about Django’s “batteries included” philosophy is that it means one package - Django - gets you almost everything you need to build a web app - routing and templates and an ORM and suchlike. This compares to the “best tool for the job” alternative where you put together your own framework from a bunch of parts, but since each part is on a separate release cycle with separate maintainers you lose the guarantee that they’ll upgrade in lock-step with each other.
Django counts as a “big package” in this case, which the article proposes makes it “likelier to change more often”. Thankfully Django has a well-won reputation for smooth upgrades, so it’s one of the larger dependencies that I have the most confidence in.
Django has also been around for decades, which partially explains why it’s so stable. And in your case, it’s probably close to what you’d call “standard library”. But there are good reasons for “many small packages” approach as well. Again, the question is only which packages you take.
Even though I don’t know the tar flags, I’m pretty certain that an alias I wrote 20 years ago would probably still work. It’s not as stable in, say, JavaScript world mentioned in the article, but good old tools and utils - say, express, mocha, request, lodash - still work mostly the same. Sure, you can update them to newer versions, but you’ll probably not have any changes to your code.
I think this is often a good approach! I also don’t think it’s actually really arguing against what I articulated here, which is about what happens when you do use a third-party package. In particular, the thing I was mostly trying to get at is that it would be pretty lame to then complain about Django being a large package—because: well, no, you’re getting something for that! I should also add that by “change” I didn’t mean “breakage”, only that they are likelier to have ongoing updates in the form of bug fixes, security releases, and indeed new features, than a very small library that does one thing.
I also don’t have a horse in the big-package-small-package race; I think there are substantial tradeoffs between the two.
Yeah, I think your article is a really good discussion of that key trade-off!
Without specifically mentioning the ones you’re referring to it’s hard to know what this relates to.
I’ve recently adopted Jellyfin, and found it to be generally fine - a few quirks here and there but generally fine. The biggest downside I’ve seen so far is that there isn’t (to my knowledge) a good Mac open source client for it - but this article is about an AppleTV which doesn’t have that issue. I’m aware some of the iOS/iPadOS clients can run on macOS, but none of the ones I’ve tried work very well with a large screen.
Can someone expand on what specifically makes Infuse better?
Infuse has native support for jellyfin too. It’s quite a good app and has support for more codecs than just native iOS player (which jellyfin uses).
But, I don’t mind the jellyfin client on iOS? It seems to do what I want it to.
My guess is the author doesn’t want multi-device streaming /status syncing so jellyfin isn’t required. In which case infuse on appletv is a decent choice.
There’s a few mentions in there of having a phone and tablet; I do appreciate that Infuse makes it very easy to start watching a show on one screen and continue on another.
I didn’t name the specific open source apps because I didn’t want to get into an inevitably pretty nitpicky arguments about each. As a programmer running a linux desktop I’m pretty tolerant of clunky UI, but the apps I tried felt significantly worse and were soundly rejected by the nontechnical TV users. There’s probably a lot of familiarity at work in the personal taste; Infuse looks a lot like the popular streaming services.
I get that, and I have infuse on ios devices and very much like it. The cool thing is, if you ever want to expand and do plex/jellyfin style stuff, infuse has support for it baked right in.
The other thing I like is offline downloads, for when I hit the road but still wanna watch curb on repeat
This HN thread on Jellyfin hits all of the issues I had with the open source options and also hits all of the “you’re doing it wrong” nitpicking I was not interested in participating in.
Jellyfin rocks: https://kaushikghose.wordpress.com/2025/01/27/the-jellyfin-media-server-is-awesome/
Give it a bit of time.
I’ve been 2 years problem free now, really surprised and impressed with it
LOL My needs are pretty modest, and I’m just impressed by all the free engineering hours that have gone into it and the features it has. What should I look out for? The fact that they have a Roku App was pretty amazing for me.
PS. Thanks for reading!
IMO the best way to use jellyfin on a personal computer is with jellyfin-mpv-shim and mpv. It’s what I use on Windows and Linux. jellyfin-mpv-shim has instructions for macOS so I am assuming that both that and mpv are available on macOS. mpv the media player is very minimalistic so it takes getting used to because the keybinds and such are not apparent but it works really well if you know how to handle it.
https://github.com/jellyfin/jellyfin-mpv-shim/blob/master/README.md#osx-installation
I switched over to Jellyfin Media Player because it uses MPV internally, and includes the web interface so it feels less clunky. The shim would sometimes break, and when I’m trying to watch something, I definitely don’t want to be debugging things.
JMP uses Qt’s Web engine, which is not extra lightweight but at least shared with other apps, so it’s worth the tradeoff for me.
What do you get with the jellyfish client? I set up myself a nad and jellyfin on top, but watching via a browser is okay for our needs.
There are some codecs that the browser can’t play. If you host jellyfin on a weak machine, you have to disable transcoding. So dedicated clients who are able to handle a larger variety of codecs are preferable to browsers.
I’ve noticed significantly better UI responsiveness and playback from the native app running on an AppleTV (which connects over average quality WiFi) than I do from a M2 Ultra Mac Studio with 128GB of RAM connecting to the same Jellyfin server over wired gigabit ethernet.
I assume it’s because the native app is better able to cache and preload content than the browser based client.
Ps. What do you mean open source client for Mac? I use Firefox.
I mean a native client app for playback, rather than accessing it through a browser - the browser client is fine for admin tasks but for playback it’s less ideal IME: https://lobste.rs/s/sn8buz/tv_setup#c_altp8l
Sorry if this is a bit unrelated, but this question is bugging me for quite some time now: People who say “I like LLMs for boilerplate”, don’t they have templates or snippets?
I have a large collection of those collected over the past 20 years of programming professionally and I don’t think anything can be easier than for example
cp ~/templates/symfony/abstract-auth-controller.tpl.php src/Controller/NewFeatureController.php && nvim src/Controller/NewFeatureController.php?My experiments with LLMs were not that smooth at least.
Am I missing something here?
*nod* I used UltiSnips for a while and I’m currently using coc.nvim’s support for UltiSnips snippets. Python script?
boiler<Tab>.Sure, I’ve played around with so-called A.I.… using a self-hosted copy of Stable Diffusion. I just don’t understand the appeal of rushing to become dependent on an external service provider who will eventually need a “monetization strategy”.
No, and this is anecdotal, but most these people did not have these workflows defined and just slugged it out in the days before LLMs, with the help of a search engine.
The author used nvim but most folks use VS Code, where the community at large leans towards vanilla config.
A few things. One, like you said, people starting now would need 20 years to get to the same amount of snippets.
Yes, I know, you can probably find decent collections already collected, from a plugin, then update them for different projects which potentially use different versions of whatever it is that you’re templating.
That said, I still think it’s largely unneeded. It’s just not the only way.
Yeah, it took 20 years to become a lot. But it was useful from day 1, when I noticed that I was doing the same things in my IDE (Delphi, back then) every time. (I haven’t kept everything - don’t have the Delphi templates anymore, for example.)
Instead of inventing a new method of compression I would’ve just not loaded all of the dictionary into RAM.
The article mentions that disk I/O was too slow.
I support @crmsnbleyd on this. It may well have been overly attending to encoding / space even for the era.
While the blog post does say this, it doesn’t say “how slow” or even if it was seek or throughput bound. So, I looked at the original McIlroy paper which mentions “scanning the whole dictionary” being slow in the prototype (so throughput), but not how long that took. So, then I looked up a bit about historical disk IO rates and found this wikipage indicating 885 kB/s is not crazy for IBM3340 Winchester in 1973. Let’s say a mini-computer in 1970 was about 5..10x slower. That’s still 90..180 kB/s or 250 kB dictionary in about 1.4 .. 2.8 seconds.
It would have been plenty fast to: sort all the document/chapter/whatever words (presumably much smaller than a dictionary and fits in RAM), saving (line, column) locations, then do the equivalent of an “ordered merge” of those words against the also sorted on-disk dictionary in a single pass - just checking for existence/membership rather than actually merging. The whole procedure including loading the document (presumably much smaller than a whole dictionary) should have been at most a few seconds. No real-time perf numbers are discussed in McIlroy’s 1982 paper other than “100 program uses per week” to indicate popularity, but I’m guessing it wasn’t much faster than that.
Another point is that back then, computers were mostly multi-user beasts, in the true sense of the word. I don’t know what kind of computers these people worked on, but the article mentions Unix, so probably also a multi-user beast. Plus, there were a lot less hardware bits that worked with DMA - the CPU would have to be managing the disk I/O in addition to everything else. So I would probably divide that number by at least one more order of magnitude again.
So, waiting ~2 seconds is perhaps not bad, but waiting 20 - when the alternative was the “instant” thing that this in-memory thing enabled - is probably not as comfortable. Especially if you had to do that on every damn word of your document.
I don’t know if that’s the case, but I would assume something like this was a reason for even trying to do this.
Which brings up an interesting tangent, I think. We barely have any true multi-user computers left. Sure, we ahve servers, that serve millions of requests, but they’re mostly optimized for one a few applications. I think there’s only the occasional Windows Server with specialized software, where people RDP their way in for one specific application, or niche things like tildeverse. Probably a few other things that I’m thinking, but majority of the machines out there are single-user, even though probably even CPU cache buffers for many of them are around the 64kb that the article is revolving about.
I think compute clusters (supercomputers and setups of the kind run by @cks) typically have login servers attached, for compiling and managing jobs. Not full-spectrum timesharing but close.
The idea is to not do
for word in doc: Lookup, but instead to realize “I am checking all the words in doc” and sort -u those to-be-checked words against a pre-arranged sort-u dictionary.” (Yes, you could prefix-compress the dictionary to save on that IO time, too, and you are right - someone should find a 1975 benchmark timing “cat 1MiBfile” for a better idea of real-time perf). I tried to explain this in words elsethread both here and there, but let’s try Python code this time:This code tests out ok on the odd-even inside/left/right/outside edge cases, but I really just wrote it. So, it’s possible there are bugs.
In fact, the 1982 McIlroy article says they did do a sort-u on document words and that this took about 20% of the run-time for some unspecified sized documents (though he notes vocabularies are pretty small). My point was more that you don’t need any very fancy encoding/hashing/Bloom filters or any of that. Just a (maybe prefix-compressed) form of the unabridged dictionary on disk and linear sweeps.
I tried to find Steve Johnson’s original
spell.cfrom 1975 online that was “too slow”, but could not. I did find some later code doing binary search lookups and the language of McIlroy 1982 suggests the algo might well have beenfor word in doc: binarySearchwhich would have indeed been very slow (as you fear). Also, the V7 /usr/dict/words was only 196_513 bytes, but is also only 24_001 words and so not very “unabridged”. A SOWPODS scrabble dictionary I have lying around is 2_707_014 bytes and 267_751 words. So, even with a 2X slower than 885 kB/s it would still only take 6 seconds to scan through it, even without prefix-compression (which would probably compensate back in IO costs, cutting off about 66% of that run-time, say, since less abridged dictionaries compress much better than abridged ones). Frankly, the “standard of care” for research papers back then was kind of frustratingly low, not that it’s great today. I could not even find any “n-fold speed-up guestimate” from McIlroy 1982. So, we’re left to make a bunch of guesses.In fact, in the ensuing half century, bandwidth/throughput has generally gotten much better than latency. This “batch” idea might well be faster than point lookups today, though if you want to assess that you should use a fast language like
Cor Nim rather than Python and it should be noted CPUs play a lot of tricks hiding latency to measure this and few care how many nanoseconds per word a user-facing thing like spell check takes. Spell check has moved on to fancy suggestions or LLM statistical models anyway.The 885 kB/s is for sequential access, but I imagine accessing a spell checking dictionary is highly non-sequential, so it’d be the seek time that matters more, especially if you want to also read a document at the same time.. With the dictionary entirely stored in RAM, you get much cheaper random access. I don’t have any numbers/benchmarks to verify this though.
I think you misunderstood the idea. Only sequential access should matter if you can fully buffer the sorted document words (plausible for small documents). Both are sorted. So, you just sequentially go through the dictionary from A -> Z matching or passing the current doc word. Then you advance a doc word and advance probably just advance a little in the dictionary and so on.
Distraction-free DOS is a great idea… until you remember the wonderful games from that era :-)
Ha! True enough. I am not a gamer and this didn’t even cross my mind.
Yes, you could…
If that is what someone wants, I’m all for it. I was thinking of uploading a “bare image” to make just that easier, in fact.
Lot of work, though. DosBox or something is probably quicker and easier.
DosBox probably does make it quicker and easier, but has the same problem that made you make your project - distractions.
Exactly. :-)