No, and this also isn’t important in Rama’s model since PStates are not your source of truth. Depots are the source of truth and the definition of the data can and should be completely normalized. Any consistency issue in your PStates can be fixed with a recompute from depot data.
In RDBMS’s, referential integrity only protects against a subset of possible data integrity issues. When you need to denormalize your schemas in an RDBMS (due to being forced to for performance reasons), maintaining data integrity becomes an application problem. And this should never be the case for a supposed source of truth.
I don’t see how normalisation or denormalization affects whether I want transactions or not.
Can a user write into the depot with a transaction? Concretely: can they observe some state in the depot or a pstate at time T1 and then write something to the depot at time T2 only if that observed state is still valid?
Or do you get around it by having the depot be an atomic message log that you write higher-level messages to, and then each pstate can get access to that log?
Rama has two kinds of topologies, streaming and microbatching.
Microbatch topologies are cross-partition transactions for every PState in every case. Our bank transfer example relies on this.
Stream topologies are transactional for all PState changes on one partition of a module done in the same event, no matter how many.
You don’t need transactions for depots, since all data that must be atomic should just be in the same depot record. Like for bank transfer, you append one depot record that says “Transfer $10 from Alice to Bob” and not two depot records saying “Deduct $10 from Alice” and “Give $10 to Bob”.
The ACID page in the docs goes into this in more detail.
There’s latency and extra network traffic, though with a static site those should be pretty low.
The UX might get a little weird in some ways — in the case of the site menu, the menu-open state becomes part of the history, so if you open the menu and click a link and go back, the menu’s still open. Then you go back again and it’s the same page but with the menu shut.
I expected the history to fill up as you described, but it seems the implementation avoids that by making the menu close button go back in history if possible. It does that with this HTML (shown unminified):
The code I quoted does work as intended in Safari on iPadOS 17: it prevents history from being filled when I repeatedly tap the menu button and the menu close button.
Perhaps rather than replying to me, you meant to comment on the article as a whole. I see that on iPadOS, the CSS transition animations don’t work. The page just blinks between states like any other page load.
I like this idea! Do you think it’s extreme to try and implement dark/light mode using static HTML? I can’t seem to find a good workaround for a javascript-less solution to give people the option to choose to deviate from their system preference.
But it sure feels like overkill to generate a copy of each page just to avoid making someone enable JS to change the colors on their screen… which I don’t even do because I prefer everything in dark mode anyway.
I don’t understand why so many web sites implement a dark mode toggle anyway. If your page uses CSS conditionally on prefers-color-scheme to apply a light theme or dark theme depending on the user’s system preference, why isn’t that enough?
For example, if the user is looking at your page in light theme and suddenly they think their bright screen is hurting their eyes, wouldn’t they change their systempreference or their browser’spreference to dark? (If they don’t solve the problem by just lowering their screen brightness.) After they do so, not only your page but all their other apps would look dark, fixing their problem more thoroughly.
For apps (native or web) the user hangs around in for a long time, I can see some reasons to allow customizing the app’s theme to differ from the system’s. A user of an image editing app might want a light or dark mode depending on the brightness of the images they edit, or a user might want to theme an app’s windows so it’s easily recognizable in their window switcher. But for the average blog website, these reasons don’t apply.
I am curious about how many people use it as well. But it certainly is easier to change by clicking a button in your window than going into your system or browser settings, which makes me think that it would be nice to add. Again, for the imagined person who decides to deviate from their system preference.
Although you’ve made me realize that even thinking about this without putting work into other, known-to-be-used accessibility features is kind of ridiculous. There is lower hanging fruit.
Here’s a concrete example. I generally keep my browser set to dark mode. However, when using dark mode, the online training portal at work switches from black text on a white background to white text on a white background. If I wanted to read the training material, I would need to go into my browser settings and switch to light mode, which then ruins any other tab I would switch to.
If there was a toggle button at the training portal, I could switch off dark mode for that specific site, making the text readable but not breaking my other tabs. Or, if the training portal at work won’t add the button, I could at least re-enable dark mode in every tab whose site had added such a toggle.
Sure, but only on sites that provide a button. It seems a little silly that one bad site should mean that you change your settings on every other site / don’t have your preferred theme on those sites.
Given how widely different colour schemes can vary, even just within the broad realms of “light” and “dark”, I can imagine some users would prefer to see some sites in light mode, even if they want to see everything else in dark mode. It’s the same reason I’ve set my browser to increase the font size for certain websites, despite mostly liking the defaults.
It would be nicer if this could be done at the browser level, rather than individually for each site (i.e. if there was a toggle somewhere in the browser UI to switch toggle between light/dark mode, and if the browser could remember this preference). As it is, a lot of sites that do have this toggle need to either handle the preference server-side (not possible with static sites, unnecessary cookies), handle the preference client-side (FOUC, also unnecessary cookies), or don’t save the preference at all and have the user manually toggle with every visit. None of these options are really ideal.
That said, I still have a theme switcher on my own site, mostly because I wanted to show off that I made two different colour schemes for my website, and that I’m proud of both of them… ;)
I remember the days when you could do <link rel="alternate stylesheet" title="thing" href="..."> and the browser would provide its own nice little ui for switching. Actually, Firefox still does if you look down its menu (view -> page style), but it doesn’t remember your preference across loads or refreshes, so meh, not a good user experience. But hey, page transitions are an IE6 feature coming back again, so maybe alternate stylesheets will too someday.
The prefers dark mode css thing really also ought to be a trivial button on the browser UI too. I’m pretty sure it is somewhere in the F12 things but I can’t even find it so woe on the users lol.
But on the topic in general too, like I think static html is overrated. Remember you can always generate html on demand with a trivial program on the server with these changes and still use all the same browser features…
I’ve been preparing something like this. You can do it with css pseudo selectors and a checkbox: :root:has(#checkbox-id:checked) or so; then you use this to either ‘respect’ the system theme, or invert it.
The problems I’m having with this approach:
navigating away resets the checkbox state
svg and picture elements have support for dark/light system theme, but not for this solution
Yeah, I think I saw the checkbox trick before, but the problems you outline make the site/page/dark and site/page/light solution seem more enticing, since they can avoid especially the state reset issue. I like the idea of respecting/inverting the system theme as a way of preserving a good default, though!
Yeah, as an alternative, for the state issue I was thinking of using a cookie + choose the styles based on it, but that brings a whole host of other “issues”
I love this point about avoiding enshittification.
The most infuriating experience for me was the sudden pricing change of replit. I used to default to creating a repl when testing out an idea. However they did go through an identity crisis, and it feels like they are not interested in having people like me in their audience anymore.
I ended up cooking building my own platform for home-brewed apps: https://smallweb.run. Essentially, it allows me to have a new webapp available by just creating a new folder on my disk (with an automatic https domain).
A standard “worst case” for allocators is producer-consumer: in a loop, one thread allocates, sends the pointer to the other thread, which then deallocates. I think for this scenario the presented allocator will give infinite growth of memory usage?
If I am reading the code correctly (which is not given!), in the producer-consumer scenario, producer will fill its slab, while consumer will fill its free list. Then, producer will rotate to the next Thread, and fill its slab. Eventually, producer will “steal” consumer’s thread (once both have the same thread index, they’ll contend for mutex and one will be edited to the next id), and re-use all entities from the free list.
But then this repeats! Once stolen free list is exhausted, we allocate one more slab, and do a merry-go-round again!
Yup, that’s the case that snmalloc and mimalloc tune for because improving it can double the throughput in a lot of transaction-processing workloads. Thread caching approaches are the worst here because the allocating thread always misses in its cache (no freed objects) and the freeing thread ends up having to walk its cache list periodically and properly free everything.
It’s also worth noting that benchmarking allocators is incredibly hard. For example, we found that snmalloc was much worse than everything else on one microbenchmark. We eventually tracked the issue down to cache contention. Our allocation pattern meant that the metadata ended up in a small handful of cache associativity sets and so we were using a small fraction of the total cache. We added some extra logic to add some jitter to the displacements to avoid this. The benchmark got faster, but everything else got slower. What made the benchmark special? It wasn’t ever touching any of the memory that it allocated and so almost 100% of the cache was available for the allocator to use. Making the benchmark touch even 20% of each allocation eliminated the gap between snmalloc and the other fast allocators. In practice, if you’re allocating a load of memory and then not touching it, the right fix is not to make the allocator faster for that workload, it’s to not allocate the memory (outside a few niche use cases such as kernel memory that you need to preallocate so an interrupt handler can use it in a situation where it can’t acquire locks).
It’s a great example because I didn’t think of it until someone pointed it out to me.
There is a performance data point of that example program in the readme file in that repository.
Your analysis is almost spot-on, but consider that it first tries to rotate before allocating another slab. So it is more likely that the rotation finds another freelist, and continues reusing memory from there.
At first I thought that adding rotation to the free function would improve the situation. When I tried it, I found that in the linked example it indeed reduced peak memory usage in exchange for performance, but in real world scenarios, it actually performed worse than the simple free implementation on all metrics.
Hm, thinking about this for another five minutes, I think this still breaks if you have, say, 4 producer threads and one consumer thread? In this situation, producer will be more likely to rotate over another producer. While sometimes we will steal consumer’s free list, we should still fall back to slab alllocation infinitely often? I’d run a benchmark, but I have a rule against laptops in bed :P
Great find. That prompts the question, is the next step to try to mitigate such problems in SmpAllocator, or keep it how it is, document its performance characteristics under different workloads, and create another allocator that is designed for mpsc? 🤔
I’m leaning towards the latter, since choosing an allocator tailored to the application is something Zig is uniquely positioned to offer.
My gut feeling answer is to just wait for some community member to write a cool pure-zig allocator that can become the default recommendation, but, at this point, I’ve been waiting for seven years for that to happen in Rust, and that didn’t quite materialize (to be fair, for a long time writing allocators in rust was hamstrung by slow threadlocals, but that has been fixed years ago as well).
With my TigerBeetle hat on: never call free, problem solved :P
I think the default allocator should perform tolerably well in all situations. SmpAllocator performs very badly in the benchmark that matklad presented, so I don’t think it should be the default. I think it would be fine to document the performance characteristics of SmpAllocator and leave it as an option.
I guess this can be said more simply: in a producer-consumer scenario, if producer and consumer have different thread_index, then slab allocation happens eventually. If they have the same index, contention happens eventually and the indexes become different.
This idea reminds me of my statistical genetics days. There’s an (in)famous piece of software, PLINK, which is just gobsmackingly fast. It achieves this by “compressing” genotypes into two-bits. The high school understanding of genetics as AA, Aa, aA, aa, can roughly be made to work even with modern sequencing datasets. Usually you can’t tell Aa from aA so you really just have AA, aA, aa (hom-ref, het, hom-alt). You can clearly stuff that into two bits: 00, 01, and 10. We can repurpose 11 for “no data” or N/A.
Now that you’ve got 32 genotypes per u64, you can start implementing all sorts of operations very quickly. Mean genotype? You can compute that with a couple masks and pop counts, followed by one division. Count of homozygous alternates? Masks and pop counts. Correlation between two genomic positions? I bet PLINK even has a linear least squares implementation written in terms of bitfiddles.
The trouble is PLINK is a special purpose binary tool. What I actually want is a library of tools for working with compressed arrays so I can use these ideas to build new tools. I hope Vortex becomes such a toolkit.
There is the BitArray in Julia, which is supposed to work directly with the underlying bits as bools, but abstracting that away so you can write code as if it was a normal array. I can’t vouch for how well optimized the code is in practice though.
As a warning, in C++ there’s a specialization for std::vector<bool> that does something similar but breaks many of the assumptions that you expect to be true from a generic std::vector<T>. The major one: It isn’t necessarily stored in a contiguous array, whereas the generic version is. This means that you can’t really take the address of a start and end element to a function that expects them to be contiguous, among other things. It was more than a decade ago that I ran into these kinds of issues (before I’d learned to very carefully read cppreference.com all the time) but it was definitely surprising to learn about, even just to learn that there was a specialization for it.
The addressing problem is not from a non-contiguous array, it’s from being a bitset and the obvious impossibility of addressing individual bits. As a result indexing into a vector<bool> returns a proxy object which breaks all kinds of assumptions about the interaction between vector and its T.
Having a separate dedicated bitset causes none of the issues, because people don’t expect it to behave exactly like an array, and more importantly can not use it in generic contexts.
Yeah, I wish they’d made it a separate type in C++ to make it clear that you can’t expect it to behave the same as the generic version. It’s a tricky design issue because lots of stuff does work the same, like iterating over it or accessing elements by index but then other parts… don’t.
For genetic data, the work for other kinds of packed arrays is already done by the BioJulia team here.
I think their code is flexible enough that you could define a new alphabet for a different domain and still use the optimised implementations for LongSequence.
I was planning to make the move from zsh to nushell as a Rust-based shell because of its structured data capabilities, but fish might be a more mature option to try out something new. Does anyone know how the Rust rewrite of fish compares to nushell in terms of performance?
Nushell was noticeably slower than fish last I used it (maybe 6+ months ago?), but unless Nushell got job control, I don’t consider them direct competitors. Nushell is something I keep checking to use in place of things like PowerShell for my use cases. fish is awesome just being my interactive shell.
Ah I see :( I was mainly wondering if the performance benefits of not having to pass data as text between commands may outweigh the relative immaturity compared to fish
It is the command-line tools that you use, not the shell, which impose passing text around. This has always been true. For example, you can build up database-like operations (with database-like performance or possibly even better with properly PGO SIMD compilation optimizations..) using streams where there is no parsing on a per input byte basis or formatting on a per output byte basis in a couple thousand lines of Nim.
It can be helpful, of course, for very small data (where the time to fork & exec seem like large costs) to have things built in to the shell, and then the shell can matter. For such tiny data performance only really matters if there is an “outer loop” over a great many situations, though.
I couldn’t find any published research showing that Hedy is particularly good versus other approaches.
There is some limited evidence that kids can learn with it, but that would also be true of many other approaches.
In one of their research articles they claim that Hedy is the first gradual programming language, which is a bit annoying cos racket has had teaching languages that gradually introduce concepts for much longer, and they probably weren’t the first.
I feel like a lot of Meyer’s stuff follows a pattern: he denigrates other people’s approaches to problems because they’re not his approach, and his approach is best by the virtue of being his approach. One example from this paper:
Consider first Gilles Kahn’s classic “Natural Semantics” paper [19]. It specifies a simple programming lan-
guage and has a list of definitions for programming constructs, assuming a considerable mathematical baggage.
The semantics of the conditional expression (“if … then … else …”), for example, is specified through two
inference rules: [cut]
One can only shake one’s head: the notion of conditional expression can be explained in full to a 6-year-old: wherever E1 holds use E2 , otherwise use E3 . You may summon Fig. 4 for help.
Meyer mocks natural semantics without ever explaining the actual point of it: as a basis for tooling. Sure, conditional expression is intuitive to a 6-year-old, but I can’t force a 6-year-old inside my type checker!
This is especially ironic, because here’s Meyer’s later definition of the “conditional expression” if C then p else q:
if C then p else q end
->
if C: p, C’: q end
->
(C: p) ∪ (C': q)
->
<post_p / C, Pre_p ∩ C> ∪ <post_q / C', Pre_q ∩ C'>
->
<⌊post_p / C⌋ ∪ ⌊post_q / C'⌋ , (Pre_p ∩ C) ∪ (Pre_q ∩ C')>
Where C’ = “the complement of C” (not defined, but later indicated to be “¬C”), ⌊p⌋ = “Rounded version of p” ⟨post_p / Pre_p, Pre_p⟩, r / X is the set {[x, y] ∈ R | x ∈ X]. I’m not fully clear on what it means to “round” a postcondition; he only defines “rounding” a whole program.
Now compare this to what was so bad one could only “shake one’s head”:
but yeah, rounding is defined in the article only for programs, not for postconditions.
I’m skeptical about what this article contributes to the field (has this really not been done before?), but I could believe that building a theory of computation directly on set theory would be convenient for reusing set theory ideas and tools in computation theory.
With a mighty Pearson’s correlation of 0.091, the data indicates that this could be true! If you ignore the fact that the correlation is so weak that calling it ‘statistically insignificant’ would be quite generous.
This silly post about kebab shop reviews is not very good statistics.
If the hypothesis is that “the closer to the train station, the worse the kebab”, then google reviews are clearly a sample of all reviews and it is the p-value of the correlation test that the experimenter should be looking at (again, comparing against a threshold identified before calculation), but also the google reviews are likely a biased sample, etc, etc.
If the hypothesis is that “average google review of kebab shops in the study area is correlated with distance from the nearest train station” vs “they’re not”, and we assume that data collection has gathered essentially all of the reviews (so it’s not a sample), then a threshold for the correlation coefficient should really be identified before calculation.
But then, without built-in understanding of utf8, is this supposed to only cover us-ascii? The same string might be encoded in multiple ways in uncode + utf8.
Example: in unicode, å can be either the direct U+00E5 or a composition of U+0061 + U+030A, both render the same. That means different lengths.
I didn’t know of any languages canonicalising unicode strings before comparing them, but it turns out that Swift does this.
Two String values (or two Character values) are considered equal if their extended grapheme clusters are canonically equivalent. Extended grapheme clusters are canonically equivalent if they have the same linguistic meaning and appearance, even if they’re composed from different Unicode scalars behind the scenes.
I think this gets into the discussion at the top of the article, which is that switch (color) should obviously be semantically equivalent to if color == ... else if color == ..., but the definition of == is ambiguous. Pointer and string equality are two answers, what you’re talking about is whether two byte strings of valid utf encoding represent the same sequence of glyphs. (*)
That’s useful in some contexts but it’s more specialized than just “match on this set of byte strings”.
(*) to be super pedantic, { U+00E5 } and { U+0061, U+030A } are different strings, but encode the same glyph. Text is hard.
I had been imagining something like the ability to extend switch to operate on any aggregate type that has a hash-function definition available, but having a comparator generalizes that, and even supports some interesting use-cases like canonicalization-in-comparison (as @andrewrk asked about) as well as even fancier things like edit distance matching!
Most of Asahi’s linux work is in Rust, such as their new GPU drivers for Apple Silicon. Part of that requires access to the DMA subsystem. An asahi maintainer other than Marcan wanted to upstream Rust bindings to DMA. These would have been part of the “Rust for Linux” subsystem, which explicitly is exempted from the usual kernel stability guarantees. The maintainer of DMA in Linux objected to them upstreaming those bindings, as he felt he would then be obligated to maintain the Rust code, and he did not want to maintain Rust. This was combined with some uncharitable comments on the Rust for Linux project’s goals. There has been a lot of debate on other internet forums about whether these comments were uncharitable about Rust, Rust for Linux, or some other distinction that personally I think doesn’t really matter.
Marcan and that maintainer then exchanged words on the LKML. Unsatisfied, Marcan appealed to his Mastodon for support. Linus then stepped in with a reply heavily condemning the appeal to social media and not really addressing the complaints that led to that point. Marcan later resigned from his kernel maintainerships, and now also from the Asahi project.
I think this understates the position of the maintainer. He sent a message to LKML that states it pretty clearly: he thinks a multi-language kernel is a terrible mistake. And I can understand his position — which to me demonstrates that there is a leadership failure happening, because this kind of fundamental conflict shouldn’t be allowed to go on this long. “Disagree and Commit”, as they say at Amazon.
Every additional bit that the another language creeps in drastically reduces the maintainability of the kernel as an integrated project. The only reason Linux managed to survive so long is by not having internal boundaries, and adding another language complely breaks this. You might not like my answer, but I will do everything I can do to stop this. This is NOT because I hate Rust.
…
The common ground is that I have absolutely no interest in helping to spread a multi-language code base. I absolutely support using Rust in new codebase, but I do not at all in Linux.
Since there hasn’t been a reply so far, I assume that we’re good with
maintaining the DMA Rust abstractions separately.
Hence, the next version of this patch series will have the corresponding
maintainer entry.
Danilo
I.e. after the explicit statement that the rust people, not Hellwig, would be maintaining it separately.
(And then we reach the post you quoted, where he makes it clear that his reasons are that he doesn’t want Rust in Linux at all, not that he doesn’t want to maintain it)
Yes, I understated the understatement. :) His stated goal is to use his influence and control over the DMA subsystem to prevent Rust (edit: or any other non-C language), from being in the kernel at all. As his NAK says:
If you want to make Linux
impossible to maintain due to a cross-language codebase do that in
your driver so that you have to do it instead of spreading this cancer
to core subsystems. (where this cancer explicitly is a cross-language
codebase and not rust itself […])
Classic LKML rhetorical style here, to use a loaded word like “cancer” to make what is actually a reasonable technical point.
It is not a reasonable technical point! It demonstrates a fundamental incuriosity about technical matters. Categorical technical statements are rarely reasonable.
I disagree in general, but not in specifics. I’ve spoken to Chris Hellman at lengths (literally 6 hours) about the topic of Rust and it was not combative. He was curious. It’s hard to sum up at length, but his disagreements are rooted in specifics and he’s a hardliner about the single source language thing. But not for the sake of being a hardliner. His hard-line stance comes out of different weighting of the issues at hand, not an outright dismissal.
Those people existing is to be expected and it doesn’t invalidate their opinions.
Obviously not a fan of LKML rhetoric, but yep, you get the tone you built over years.
You can disagree, but for the rest of us who haven’t had these super private, intimate exchanges, where we get to “really know” someone, what he puts into the public is our perception of him. To that end, he’s called the project a cancer, and declared he’s going to do whatever he can to stop it which is not what a curios person does. He comes across instead as a religious zealot or “hardliner” in your terms.
That’s fair, but it’s also a categorical mistake to read a (brusque and inappropriate) email and judge on a whole person for days and weeks on the internet. I’m not saying you “really need to know someone”, but you also need to be aware that hundreds of hours are spent on reading what those statements may mean.
I’ve personally had to make the decision whether to allow an additional language into a large codebase, and there are some major negatives to consider that are unrelated to the technical aspects of the language. The maintainer said three times in the above excerpts that he doesn’t have a negative opinion of Rust itself.
I’ve always heard that Google allows only four languages in their codebase, and one of those is only because they invented it. :)
By “categorical”, you mean saying “no second language could ever have upsides that overcome the downsides of having two languages”? I agree that’s a pretty conservative position. It’s hard to even imagine another candidate for second language, though (I dunno, Zig maybe?), so I didn’t take it as literally categorical.
And I also do not want another maintainer. If you want to make Linux
impossible to maintain due to a cross-language codebase do that in
your driver so that you have to do it instead of spreading this cancer
to core subsystems. (where this cancer explicitly is a cross-language
codebase and not rust itself, just to escape the flameware [sic] brigade).
I think Zig sees itself as a candidate second language for C projects, but that’s probably beside the point; nobody’s trying to bring it into the kernel.
Indeed, but that call was already decided when Rust support was first merged.
The goal of the experiment is to explore those tradeoffs. You can’t do that if you don’t merge things!
I’ve always heard that Google allows only four languages in their codebase
And yet they are one of the companies pushing Rust for Linux… They understand the tradeoff. The rust for linux people understand the tradeoff. They just think it is worth it.
Totally. But that does seem like a valid engineering question people could disagree about.
However, this is exactly the kind of huge impactful question that should get escalated to the chief architect. And if there’s a sanctioned scouting effort to gather experience to figure out the right answer, somebody found to be actively subverting that effort should be…corrected…by the chief architect.
It’s interesting that they won’t allow it in their internal codebase but they’re happy to let the Linux maintainers take on the burden of adding it there.
R4L is officially an “experiment” with goal to explore tradeoffs. I.E. satisfy curiosity.
This whole debate is so tired. The “including Rust” ship has, at least temporarily, sailed, and maintainers blocking it based on not wanting Rust/a second language doesn’t make sense when the whole point is to explore that option.
Maybe all the code will get deleted at some point, but the code needs to be merged first to then decide that based on how well it works in practice.
To have a thorough experiment, it must be merged into the official tree. One must be able to see how this interacts with a diverse set of hardware and software configurations, and also how this affects the development process inside the kernel community. Some maintainers are afraid of how the introduction of Rust would affect them in the future and the maintainability of the code base. Without it being in the official tree, there wouldn’t be any conclusion on those points.
It sounds like some of the kernel maintainers at least don’t want to have to pay the cost of this large-scale experiment? Like they don’t feel it’s worth the downsides of massively complicating the kernel into a multi-language codebase? Like they understand the potential benefits of memory safety but they feel they are outweighed by the risks?
These individuals believe that they know the result of the experimental before it’s been run. But they do not, because they cannot, by definition. Of course they don’t want to pay the cost, because change is hard, and deep down they know that there’s a chance that the experiment is successful, and thus their responsibilities will either change or they will have to cede responsibility to someone more capable.
Whether there is benefit to Rust in theory, there is little question. It provably solves problems that the kernel has. Whether it can solve those problems without causing larger ones is why the experiment is required. Blocking the experiment for essentially any reason is more or less indefenisble, because the entire point is that they don’t know.
That sounds way too confident and assured of a statement to me. You are discounting the opinion of subject matter experts as if they have no idea what they’re talking about. If someone came to you at your workplace and said that they want to run an experiment to convert your codebase to a new language, and they want you to help maintain and support that experiment while also delivering on all your existing obligations, would you be so quick to accept that? I don’t know about you, but personally I would and have pushed back on rewrite suggestions as a bad idea, despite the perceived benefits (for example, Management thinks that rewriting in JavaScript will make it easier to hire people for the project in the future).
Would rewriting in JavaScript have possibly made me redundant? Maybe. But it would also be massively expensive, cause huge churn, and have a real possibility of failing as a project. We can’t just ignore the very real possibility of risks because of the expected benefits.
There is no interpretation in which it’s too assured. My position is that the result of the experiment is unknown, and that the potential benefits are known, both of which are hard facts.
I have run such experiments in my workplace before. Some have succeeded and some have failed. The successes have massively outweighed the impact of the failures.
Assuming that you know the outcome before you begin is the pinnacle of hubris, which we see time and again with these entrenched maintainers. They may be domain experts, but they deserve every ounce of criticism that they receive. Closed mindedness is not how progress is achieved.
My point is that there must be some threshold that you use to decide whether some experiment is going to be useful to run or not even worth the effort because your experience and expertise tells you otherwise? Or would you accept every experiment that anyone proposed? Eg, suppose someone wanted to model your entire technical stack with TLA+ for high assurance purposes? On paper it sounds like a great idea–formally verify your concurrency properties–but you don’t see how a reasonable project lead might say ‘While this project could certainly bring great benefits, it’s not worth the tradeoff with our team size and capabilities right now. That may change in the future’?
Some threshold must exist, yes. Presumably, that threshold should be somewhere below “the progenitor of the system has decide that the experiment should be run”, which is currently the case of RFL.
Individual system maintainers should not and must not be able to usurp the direction of the project as a whole.
Your core misunderstanding is that we are coming from a tabula rasa. We are not, and if your position is that Linus’s previous endorsements shouldn’t stand, then we’re having an entirely different (and much less tenable) conversation.
“the progenitor of the system has decide that the experiment should be run”, which is currently the case of RFL.
Is it though? To my understanding the decision was only that they will try it and see if it works on both the technical and the social levels. There was never any guarantee given that Rust will be shipped in Linux at all costs. To my understanding Linus’s approach is to see if Rust is overall viable for the kernel, and that includes the opinions of the other subsystem maintainers. Because if they are overall against it, the experiment is dead in the water.
Again, that decision has already been made. There’s no point questioning it over and over. Doing so is distracting from the debates that are important now: how can leadership do better at enforcing these project wide decisions to not let the situation fester like R4L, and reviewing the Rust code without raising the same solved questions again and again (concern trolling).
If you’re genuinely curious about those questions, there’s plenty of text written about it. LWN is a good place to start.
I hear you, but it is a bit in the nature of project that are very distributed in decision making with high individual authority that indeed, decisions will be questioned again and again and again at every point. Those projects need a long breath and the willingness to talk about points again and again and again.
It comes with its strength, particularly, those projects are extremely resilient and fast in other regards. You can’t have both. Pushing top-down decisions in such an environment can have other unintended effects.
You’re right that it’s in rust/, tho unclear to me if it’s properly in the rust subsystem or not. You and others may be right that this patch may still go thru, that seems to depend on Greg KH and Linus.
so he has the authority to block it, but not to reject it? him being the guy referred to above as “the guy who blocked the patches that the Asahi Linux folks wanted.”
There has been a lot of debate on other internet forums about whether these comments were uncharitable about Rust, Rust for Linux, or some other distinction that personally I think doesn’t really matter.
Personally I think it does matter when a core maintainer calls a project that’s part of the kernel “cancer” and the guy who’s in charge doesn’t seem to care.
And I also do not want another maintainer. If you want to make Linux
impossible to maintain due to a cross-language codebase do that in
your driver so that you have to do it instead of spreading this cancer
to core subsystems. (where this cancer explicitly is a cross-language
codebase and not rust itself, just to escape the flameware brigade).
The cancer is not Marcan, Asahi, R4L, or Rust. The cancer is anything-but-C. The cancer is a two-language codebase.
Still a very discouraging comment from a maintainer, and it bucks Linus’ decision that trying out Rust in the kernel is OK, but the distortion of this comment has been bugging me for the last week.
I don’t have an opinion, but I think a fair and less charged way than “cancer” of stating the position is just this: using Rust would have positive benefits, but they are far outweighed by the negatives of allowing a second language into the kernel.
The pros and cons of RfL are somewhat subjective, and the funal balance is hotly debated.
But I find it quite telling that Hellwig’s core argument (that a multi-lamguage codebase requires more work) is held by people who didn’t try doing any of that work. Whereas the kernel devs who started using Rust are explicitly saying “the API compat churn isn’t that big a deal, we can do that work for you if you want”.
We mostly hear about the drama, but it seems that the overall feeling toward Rust among kernel devs (not just RfL devs) is generally positive.
I don’t think the metaphor conveys any technical meaning beyond what I said. I don’t know what “kills the organism” is supposed to relate to. Is the “organism” the kernel? The community of kernel developers? And what does “kill” equate to in reality? It would be better to have the discussion in concrete terms rather than metaphors.
It does convey the personal outrage of the maintainer better, certainly.
Well, notice that you aren’t absorbing my actual point too well, because you’re very focused on making your own.
yes, I’m ignoring the less absurd things you’ve said because I don’t know what to take seriously when you haven’t retracted the most absurd thing.
To give him the benefit of the doubt, he could have been trying to convey the technical meaning of “growth” in the sense of “if you let Rust in, we’ll just have more Rust”. However, that’s an utterly vacuous thing to say, because obviously the whole point of the exercise was to let Rust in and see what happens, and why would you do that unless you want to have more of it if the experiment succeeds?
one characteristic of cancerous growth is that it happens whether you want it to or not.
It conveys no useful information beyond “I don’t think Rust is a good idea at all, for reasons”; in other words, “the benefits don’t outweigh the negatives”. In fact it confuses matters because it implies he thinks Rust would be OK if only it didn’t grow.
it doesn’t imply that, but I do think he would be much more OK with a little bit of Rust if the scope were somehow guaranteed not to expand. like I don’t know, maybe there’s a carve-out for testing a particular class of rust drivers in linux to help find bugs, but eventually the drivers are ported to C for use in linux and the rust version is used in a separate kernel written from scratch in rust?
I’m actually trying to be polite to the guy here, because I truly believe it was an emotional outburst, not an attempt at a technical argument. His actual technical argument (stated much more clearly elsewhere) is that there should be no Rust at all in the kernel, ever, not that once there is some, there will be more. So if “cancer” was supposed to mean “growth” as a technical argument, he misstated his own argument.
so you wouldn’t say that you want to have no cancer at all in your body, ever? if it’s a non-cancerous tumor, I have much less of a problem with it being in my body.
I don’t want to dig into the minutiae of why this metaphor is inaccurate — my point is that using this metaphor at all is unprofessional and counterproductive because of the heavy emotional baggage it comes with. Just say “the introduction of Rust anywhere in the codebase will inevitably result in a growing amount of Rust in the codebase”.
okay, so do you retract your previous statement that “I don’t think the metaphor conveys any technical meaning beyond what I said”?
my intention was not to get into minutia either, but we’ve gone back and forth three times and you still haven’t retracted that statement so I don’t know where you stand.
It’s especially odd in this context, given the history of this metaphor regarding Linux.
so our enemies allow themselves to use the metaphor to efficiently communicate with each other, but we have to forswear it.
If you don’t want it there at all, it’s redundant to also complain that it’s growing. Cancer also has many other attributes besides “growing” that are irrelevant or contradictory to the position taken.
The reason he used that particular metaphor was to express disgust and horror, not to be technically accurate. If he just wanted to evoke “growth” he could have said it was like ivy. Disgust and horror are not an appropriate device to call on in this context, IMO.
Re Mr. Ballmer, the enemies were wrong that time, so don’t you think evoking that argument by using the same metaphor muddies the rhetorical water a bit?
if I can’t tell whether you stand by your statements, it feels futile to interpret them as if you believe them. that’s my issue with continuing in light of the fact that you are just leaving “I don’t think the metaphor conveys any technical meaning beyond what I said” without clarifying whether you believe it.
“complaining that it’s growing” is completely different from identifying the growth as a factor in the technical drawbacks.
Cancer has a multitude of characteristics, of which growth is only one. If anything, it has many characteristics contradictory to the point being made: it’s an undesirable outcome of natural processes, it is typically self-generated from within rather than deliberately introduced, it is hard to eliminate because it’s nourished by the same system that sustains healthy cells…
If you have to tell me exactly which part of the metaphor I’m supposed to pay attention to in order to get the technical meaning, then you aren’t conveying technical meaning, you’re muddying the water. Just say what you mean.
The additional meaning being conveyed here is not technical, it’s emotional, and I do agree emotional meaning was conveyed.
The additional meaning being conveyed here is not technical
I take this to mean that you still stand by the statement that “I don’t think the metaphor conveys any technical meaning beyond what I said.” if you treat all technical metaphors like this, I’m glad you’re not my coworker.
If you have to tell me exactly which part of the metaphor I’m supposed to pay attention to in order to get the technical meaning, then you aren’t conveying technical meaning, you’re muddying the water.
or it could mean I am conveying technical meaning but you’re being willfully obtuse. just maybe…
Well, notice that you aren’t absorbing my actual point too well, because you’re very focused on making your own.
To give him the benefit of the doubt, he could have been trying to convey the technical meaning of “growth” in the sense of “if you let Rust in, we’ll just have more Rust”. However, that’s an utterly vacuous thing to say, because obviously the whole point of the exercise was to let Rust in and see what happens, and why would you do that unless you want to have more of it if the experiment succeeds? It conveys no useful information beyond “I don’t think Rust is a good idea at all, for reasons”; in other words, “the benefits don’t outweigh the negatives”. In fact it confuses matters because it implies he thinks Rust would be OK if only it didn’t grow.
I’m actually trying to be polite to the guy here, because I truly believe it was an emotional outburst, not an attempt at a technical argument. His actual technical argument (stated much more clearly elsewhere) is that there should be no Rust at all in the kernel, ever, not that once there is some, there will be more. So if “cancer” was supposed to mean “growth” as a technical argument, he misstated his own argument.
I could be wrong, but I get the feeling that people didn’t quite get why Hellwig used the word “cancer”. I could be wrong about this in a lot of ways, maybe I too misunderstood what Hellwid meant, maybe I misjudged how others are interpreting what he said. But allow me to add my 2 cents.
People seem to be interpreting what Hellwig said (i.e. calling R4L “cancer”) as him saying R4L is bad, evil, a terrible thing, etc. And I totally understand why many would think that, and would agree this is a terrible choice of words. But I think that Hellwig’s focus is actually on the fact that cancer spreads. A better word to use would probably be like “infectious”, or “viral”. Although I disagree with him, I think what he was saying is that, despite what Rust developers promised, Rust will spread to more corners of the kernel and increase the maintenance burden and will have a negative impact on the kernel as a whole.
I think you’re right, but that doesn’t exculpate Hellwig in the least. One of the defining characteristics of cancer is that it is malignant. It seems pretty clear that he used this metaphor because he thought that Rust would spread and that it is dangerous and harmful.
“infectious” or “viral” have different connotations. viral would be if they thought the BSDs might catch it. infectious is not as specific as cancer; cancer specifically grows from one area and consumes more and more of the things next to it. infections can do that but not necessarily.
Unrelated to the rust issue, my understanding is that such toxic language actually isn’t all that strange in the Linux kernel maintainer community. It feels like they try somewhat not to overdo it, but they aren’t that much concerned about the toxicity. More like, “I’ll try but I don’t actually care”.
It’s useful in some cases, but when the effect of it is raising the temperature of the discussion, then I would say it isn’t useful in that instance. There are other ways to convey the same meaning that wouldn’t have attracted the same amount of strong emotions. We’ve seen examples of such alternate arguments in this thread that would have been more suitable.
sure. but then don’t fixate on the word, and acknowledge that their choice to use strong language can also be an effect of the raising temperature of the discussion, for which responsibility is shared among all participants.
There’s this idea that multi-language codebases are categorically a disaster, right? What really bothers me is how small-minded it is. There are many C and C++ codebases that have introduced Rust. It’s not easy but it is possible, and it often is worth it. Some attempts have been successful and others haven’t. You can learn so much from other people’s experiences here!
Simply calling multi-language codebases a cancer closes off discussion rather then opening it. That goes against the basic curiosity that is a hallmark of engineering excellence.
Honestly everything around R4L that has been going has just cemented in me the belief that the average long time Linux maintainer is fundamentally incompetent. They’re reasonably good at One Thing, but totally incurious and scared of anything outside of that One Thing they’re good at.
Is it possible that we could take an experienced kernel maintainer’s opinion seriously instead of dismissing it immediately as incurious and ‘against’ engineering excellence? If a co-worker who specializes in a different domain came to you and kept insisting that something you know is very difficult and not worth the tradeoff is actually quite easy, would you be incurious and ‘against’ engineering excellence if you dismissed their wild suggestions without a deep discussion of the tradeoffs?
Evaluating who to take seriously and who not to is a matter of judgment cultivated over a lifetime.
If a co-worker who specializes in a different domain came to you and kept insisting that something you know is very difficult and not worth the tradeoff is actually quite easy, would you be incurious and ‘against’ engineering excellence if you dismissed their wild suggestions without a deep discussion of the tradeoffs?
Depends on many things, but in general, yes, of course, especially if they were someone I took seriously on other matters. If nothing else, it’s a chance to sharpen my arguments. If I keep having the same arguments over and over I might even step back and write an essay or two explaining what I believe and why I believe it. Here’s one that’s public that was a result of the sort of back-and-forth you’re talking about.
Reading the discussions on this elsewhere, there’s no way to describe this that wasn’t going to get accused of bias by people holding strong opinions on either marcan or on the rust for linux project.
I’m not accusing of bias (which is more or less unavoidable), but of constructing a narrative. The facts are: he exchanged words on the LKML and then wrote Mastodon posts about the exchange. The narrative you’re constructing is: he exchanged words on LKML, then because he was unsatisfied with how that discussion was going, he wrote Mastodon posts about the exchange in an attempt to gather support.
I think it would be better to leave out the guesswork about intentions, or at least clearly separate it from the factual retelling of events.
Correction: Marcan never “exchanged words” with Christoph Hellwig on the patches, which had not even been posted by him. He just replied to the thread with a tirade.
A next-gen scripting language; I feel like the current ontology of “Python, Ruby, JS and sometimes Lua” has some good room for improvement from the functional language/Rust/Gleam culture. Something with a strong-ish type system with sum types instead of 1990’s “OO everything”, gradual typing built in from the start, and still keeping an emphasis on fast iteration and smooth development. A compiler built around JIT or incremental recompilation might be nice. Julia and Crystal are steps in the right direction but personally neither really quite makes the leap from “good” to “excellent enough to want to learn and use vs just bashing something out in Python”. Roc might be going in the direction I want?
A language-agnostic program runtime and software build/distribution system that stores programs and libraries as wasm or another openly-specified bytecode, then makes CPU-portable and efficient programs by AoT-compiling them to super-optimized static native code on installation. You know, like Android does with dex. I think this solves a lot of the issues that currently exist with DLL’s. It would let people distribute code as portable binary libraries, build code with portable libraries, not have the runtime overhead calling of dynamic linking, and enable the code inlining and optimizations it disallows. But you could still do things like distribute updated libraries or make plugins for a program (again as bytecode), plop them on an existing computer, and the system would just recompile the programs that need them. The main downside is it’s pretty difficult to always go from bytecode to native code as well as you can go from source code to native code.
A browser scripting environment that enables automatic FFI generation for any language that can target wasm. This kinda exists but has the Slight Issue that any language using it essentially has to be able to talk with JS’s bonkers object model and API assumptions, which usually isn’t worth the effort vs just writing your damn thing in JS.
A web browser that takes a sane subset of HTML and the browser API, and comes with lots of conformance and test suites so other people can make other browsers using the same subset of functionality.
Something nicer than email, intended for resilience and abuse-resistance. (I’m dreaming about it, I don’t know how to do it.)
Something nicer than Discord/Zulip/whatever for chat, built around a peer-to-peer pubsub model from the start but less grotty than XMPP.
Something nicer than IPFS for sharing host-anywhere content-hashed data. And then a crypto identity system built on top of it that lets you do public-key signing and encryption of messages and stuff.
About a million video games.
…yeah, I don’t understand the people who ask “how do you come up with ideas for things to build?”
Reminds me of UNCOL, TenDRA TDF, ANDF, Taos/Taos and even LLVM bitcode. Some more successful than others, some more effective at being actually architecture-neutral than others.
Also ChrysaLisp, which is made by a former Taos engineer. It’s both a reimplementation/imagining of TaoVM, and an entire desktop environment and Lisp dialect, but you could focus just on the VM part.
I’ve (not) been working on this in the background as a long-term project for a while, now, as someone who has zero experience with writing compilers; I call it Rumland (it was initially a pie in the sky idea of an ML-inspired typed scripting language that compiles to Ruby or otherwise interoperates w/ the Ruby ecosystem, then I decided to just do the most minimal, fastest-compiling ML-inspired typed scripting language I could.)
I wanted a scripting language like a mini-Haskell that can do cold builds of at least small scale personal programs (< 100k SLOC), including linking, within tens of milliseconds on a half-decade-old mid-range machine. I wouldn’t mind compiling mid-range programs like ~500k SLOC w/ dozens of transitive dependencies that fast as well, but the main focus is personal programs of < 10k SLOC, then small (but really nice to use and good looking of course) open source projects off less than 100k SLOC.
As soon as I get it compiling a very minimal language of only ADTs, functions, basic effect tracking, imports, and a tiny stdlib, I’ll lock any further changes that don’t maintain that build performance; if I have to give up all the Haskell-inspired niceties I have in mind, I’ll do it. (It’s not a public project, but I may one day make it one.)
I’ve been keeping an eye on Roc as well, and it hits a lot of the same stuff I started this project for; I love that they do their own linking (and I love the recent growth in interest in linker performance in general, e.g. mold, wild).
neither really quite makes the leap from “good” to “excellent enough to want to learn and use vs just bashing something out in Python”.
This is the problem, though. Nothing can make a very compelling case vs. using Python, at least not enough to justify the learning cost. Of the languages you list, you can already use Gleam/Elixir for scripting. There’s also F# - I’ve personally used it as a scripting language for quite a bit. You have always been able to use lisps like Racket or Clojure (via Babashka) for scripting, and there’s a slew of neolangs that try to innovate in this space. But there’s nothing so compelling that would make you want to give up everything you have. Nobody wants to give up the comforts of a familiar language to learn one that is marginally better, and at the end of the day, that’s what all of these “next-gen” improvements are. That’s one of the reasons why I’m looking forward to the upcoming stabilization of cargo-script; Instead of a new language that has all the shiny features of rust that make it comy for me to use, I’d rather just use rust for scripting too, if I can help it.
Yeah, but I’m a language nerd, so if the language is neat then for me the learning cost is potentially negative. :-P And lately a lot of my own scripting-language needs are not “get this big program to work robustly” or even “do a lot of talking with this particular domain-specific library”, but just juggling and automating fairly basic things. It feels kinda wrong to use Elixir or F# for such things, though maybe I should work harder on giving it a go. Jank also looks like it might make me actually want to write clojure, too.
It’s just… It’s not that hard to write a shell-script-like pipeline that’s nicer than Python’s subprocess lib, and I would kick a puppy to the moon for something with the built-in facility to take a bunch of JSON, stuff it into a an in-memory table (or even a persistent one), and let me query it like the database it is instead of rolling my own indices and joins with hashtables checking for the fields I care about. …hell, that alone might be a killer feature to build a language around. There’s lots of annoying REST API’s out there that force you to do this sort of thing to get the information you want.
And lately a lot of my own scripting-language needs are not “get this big program to work robustly” or even “do a lot of talking with this particular domain-specific library”, but just juggling and automating fairly basic things. […]
It’s just… It’s not that hard to write a shell-script-like pipeline that’s nicer than Python’s subprocess lib
And in the parent comment:
A next-gen scripting language; I feel like the current ontology of “Python, Ruby, JS and sometimes Lua” has some good room for improvement from the functional language/Rust/Gleam culture. Something with a strong-ish type system with sum types instead of 1990’s “OO everything”, gradual typing built in from the start, and still keeping an emphasis on fast iteration and smooth development. A compiler built around JIT or incremental recompilation might be nice.
These strike me as requirements for very different languages: the first one is a shell language that revolves around running pipelines of commands while the second is a more traditional scripting language.
But the idea of exposing an SQLite database as a type in a shell language is interesting. I’ve been thinking of a slightly different angle: streaming an SQLite table contents as a way of providing statically-typed pipelines (still very fuzzy admittedly).
I’d say F# fits what you want, especially the last bit. You can use LINQ to query over an array, so you could do basically everything you wrote. The only sticking point would be calling subprocesses, but I’m sure there’s a library that makes it easier, or you could write the boilerplate for that once and put it on nuget.
Hmmm, thanks. I’ve used F# before for medium-sized things and it’s been pleasant but I’ve never tried using it for scripting stuff. Given that u/pyj agrees with you in the sibling comment, I’ll give it a try sometime.
Somewhat in that direction is my Next Generation Shell. Preparing arguments for calling external programs, running external programs (curl to JSON API for example), parsing their output, handling their exit codes properly, small scale data manipulation are the intended use cases.
Small scripts “should not suffer” from language capabilities (no public static void main). If you decide to have main() you get automatic parsing of command line arguments to fit main() definition, of which you can have several because it’s a multimethod.
Ciel looks really cool. I read the docs yesterday and liked almost everything I saw. I feel like I want to start a new project with it, or maybe retrofit one I’ve started but haven’t done much with yet.
There’s also F# - I’ve personally used it as a scripting language for quite a bit
I also tried using F# as a scripting replacement for Python, and had very similar experience and opinion as what you found. It was amazing to me how great it worked with .Net Core across Windows/Mac/Linux.
…yeah, I don’t understand the people who ask “how do you come up with ideas for things to build?”
I don’t ask that exactly, but I do get stuck not having enough ideas about how to implement those ideas.
About a million video games.
This in particular, I have long lists of ideas but any one of them is just discouraging to think about how to produce the intended effect. I have a resolution to do at least one game jam this year, the last two years I’ve struggled after a fairly productive 2022 (when I was between jobs).
On the next-gen scripting language thing… and having chatted with you a bit about some of the past work we’ve both done… one thing I was thinking about the other day on the way back from a flight test campaign was how much the Lua scripting in ArduPilot scares me. It’s an incredibly powerful feature but also seems like it has potential to have unbounded runtime that could cause problems.
A potential solution I scribbled down was to use a more limited VM that is amenable to ahead-of-time verification for bounded execution time and remembered https://ebpf.io/. The pseudo-C code that you use to write most eBPF code is… not great and a long way away from being an easy scripting language, but it seems like a language similar to what you’re describing could be compiled down to something like eBPF and used safely for that kind of scripting.
Ooh, yeah, scripting for high-safety systems sounds like a very interesting and very tricky proposition. Verification would be good, but you’d probably also need a good simulation framework. Do it well and there might be a pretty great market for it though, and Eris knows it’s a market that needs more convenient tools!
With email-like applications the whole problem is the high level “how to do it”. We’d want something that anyone (or a lot of potential senders) could send you an email while preventing spam and abuse. These two contradict each other. I’m hoping a “only friend of my friends” could be a reasonable mdidleground, which is adjacent to what I’m already implementing.
Something nicer than IPFS for sharing host-anywhere content-hashed data. And then a crypto identity system built on top of it that lets you do public-key signing and encryption of messages and stuff.
Something nicer than IPFS for sharing host-anywhere content-hashed data. And then a crypto identity system built on top of it that lets you do public-key signing and encryption of messages and stuff.
You might be interested in Peergos, which does cryptographic identity, signing and encryption on top of a much more efficient ipfs implementation.
I think this solves a lot of the issues that currently exist with DLL’s. It would let people distribute code as portable binary libraries, build code with portable libraries, not have the runtime overhead calling of dynamic linking, and enable the code inlining and optimizations it disallows.
This sounds interesting. What would the ABI look like across langs?
I dunno! Probably a lot like what any ABI looks like, which will be imperfect by necessity. The nice thing about wasm is you can use unmanaged memory with it, mostly unlike JVM or .NET, but then adding managed memory a la the GC proposal is a bit of a mess. So I kinda feel like the ABI is not the special part here and will never be able to be perfect, it just has to be something that does the job pretty well. An AOT compiler/optimizer should hopefully be able to make most of the ABI go away with whole-program opt when the program is turned into native code.
Something nicer than Discord/Zulip/whatever for chat, built around a peer-to-peer pubsub model from the start but less grotty than XMPP.
Not less “grotty” than XMPP, because it’s built on top of XMPP, but I recently found Applesauce, a project that plans to use https://spritely.institute/ and XMPP to build a peer-to-peer pubsub social networking platform like you describe.
I think that the way to make programming truly easier is to get rid of pointers and reference semantics. Pointers are the gotos of data structures. Represent data as pure immutable values, as is done in pure functional languages, as well as in some other language families, like APL and the array languages.
Sure, that’s nice and all. Clojure is a great example of this philosophy. However, for something like Rust it’s simply a non-option.
If nothing else, your fancy functional immutable data thingy will need a garbage collector, and we will continue to use systems languages that make indirection explicit, such as Rust, to write those.
I hate saying this, but I’d love a TL;DR summary of these articles because they’re so lengthy. I’ve seen the others in this series posted here, and I still disagree with a lot of the premises. I don’t want clickable URLs in my shell buffer, I don’t think dragging a file onto the shell window should copy the file there, etc. There are just better ways of doing most of those things (IMO).
I don’t really get the focus on terminal emulation and TTYs. Nobody’s forcing you to use that stuff. There are a lot of ways to do “terminal stuff” outside of a terminal, with or without a CLI and TUI.
The obvious way would be to use Vim or Emacs and do the work there. Emacs has a TUI and can (or can be made to) do all of the things the author is asking for, and the underlying details of TTY protocols and terminal emulation is entirely hidden away.
I’m an emacs user, so that’s the only one I have much direct experience with.
I’d imagine most editors can do it. I’ve seen a half-dozen “magit for $something” articles posted here, so probably those tools. VSCode has a shell/terminal thing.
It doesn’t mean it’s correct, though. I like the the separation of concerns, and so an editor should be an editor. What people are doing in the e.g. Neovim community is insane to me (package managers, many Lua rewrites of tools you can use in the CLI, etc.).
I think the evolution needs to come for « CLI replacements », and so far besides Cat9 I really haven’t see anything promising (besides emacs, but it comes with its own issues).
We likely consume, and expect to consume, content on the Internet in very different ways. I personally don’t deal with short-form whatevers at all. If it’s not worth at least an hour with a printout in the arm chair I simply stay away.
You are forced to use terminal emulation and ttys every step of the way (assuming Linux/BSD). The only one that successfully paved another way is Android and it worked rather well for them, and even then it pops up here and there. I know, I worked on it. The first thing your desktop has to do in order to get something in the screen is a ridiculous dance to tell the TTY to please stop doing its thing, and even then it’s there in ways that completely undoes the security model they try to implement.
That something is ‘hidden away’ doesn’t mean that it doesn’t influence every layer above it. Emacs pays through the nose to get its TUI to work. There’s a ridiculous amount of code and workarounds to present its rather modest UI.
A tangent perhaps but as a former Emacs user and as former Vim user (neither are even close to doing the things I ask for) I find it fairly bizarre to use a text editor to do shell work and vice versa. Just as I avoid trying to use a tooth brush to clean my toilet or a toilet brush to clean my teeth. They can get the job done, sortof, but at a price and with some risks.
I’m going to disagree with your last paragraph and say it’s quite natural to use a text editor to do shell work.
Most “shell work” involves navigating around, searching through, and otherwise working with text, which is exactly the toolset built into text editors. cat, grep, sed, and awk are command line tools implementing text editor functionality in a clunky (clunky for interactive use) pipeline/streaming fashion. Likewise, ls, find, cd, and other filesystem navigation is also basic text editor functionality. They all spit out text, and what better way to view and work with it than a text editor? Maybe I’ve had too much Emacs koolaid, though.
For the rest of it, I don’t care much. On a day-to-day basis I never have problems related to TTYs, and with a few (easily fixed) exceptions in the past, it’s been that way for a long time.
With that logic you can use a hex editor for everything, it’s all “just” bytes no? representations matter, interventions matter. Trivialities aside, they operate on different mechanisms and assumptions. Text editing is buffers with some intended persistence and at rather small buffer sizes. You can expect to seek and build features assuming that works; the backing store changing is an exception, not the default and so on.
The cli shell is stream processing. Its data sources are hierarchies of short lived processes that might expect something back from you to do their job and can spit out infinite amounts of data. The world can change beneath its feet between every command. To do that it gets to go through all the ugly legacy – signals, sessions, buffer bloat, line blocking and with that, serial communication.
This brings us back to never having problems; ‘works for me’-isms is just dismissed with ‘doesn’t work for me’-isms and doesn’t get you anywhere. It’s systems engineering, one doesn’t get to think like that. You don’t have to dig far to find people with serious problems with the TTY and a few vulnerabilities a year. First(?) Flatpack sandbox escape? the tty again.
Whenever they try to fix something to make embedded development bearable, you commit the Linus taboo of breaking userspace because your stack relies on hidden assumptions that all go back to the emulating the fantasy computer. To edit text. It’s holding everyone back.
Just the other week I had a coffee spill. It fried my mouse and worse, I had to make a new cup of coffee. I grabbed another rodent from the box of spare input devices. I plugged it in. Now my keyboard stopped working. I unplugged the mouse again. The kernel crashed and I lost a lot of work. Angry reboot. Plug back in. Now there’s half a second delay between keypresses every 10 seconds. 2 minutes later the machine froze up. It turned out the USB mouse exposed itself as serial rather than USB human-interface. This meant some /dev/input/eventN gets routed through the tty layer again. The display server ran the device node non-blocking, but that only goes skin deep. You still have ‘the global’ lock around in tty and a handful of mutexes on top of that. Race condition into memory corruption and panic. Then race condition into deadlock.
The point you missed is that a text editor has tools to work with text, while hex editors typically do not.
I find working inside of emacs generally easier than in the terminal because it has great tools for working with buffers of text, which is what most shell commands produce. If there were a hex editor that made it even easier, then I would indeed use it instead.
they imported the data reported in the blog post as two numpy/scipy vectors, then ran a student’s t-test to test the hypothesis that the two distributions the two vectors were sampled from are different. (docs on ttest_indhere).
Technically this is bad statistics, because you’d want to specify your confidence significance level ahead of time (typical value is 5%, which roughly corresponds to a difference of two standard deviations between the two distributions). Essentially, this test is saying that if the two variables came from the same underlying distribution, you would expect to see this data no more than 0.07% of the time. I suspect that the default for scipy uses a 5% confidence interval significance level, so statistically you can say that this data comes from the same distribution (i.e. the latency is the same between X11 and Wayland) no more than 5% of the time.
Edit: the above is sorta wrong. You don’t specify a significance level in the t-test itself, but you should specify it before performing the test, otherwise it’s hard to interpret the results based on your null hypothesis. So if we state the null hypothesis as “I expect that if X11 has the same latency as Wayland, and I measure the latency 20 times in both, I expect to get a different result no more than one time out of twenty”, then the above test is coherent and we can reject the null hypothesis.
Reading the docs some more, ~this might also be a bad test choice because itequal_var` says to not assume the underlying distributions have the same variance, which is reasonable I don’t think we have any reason to suspect a priori (i.e. Wayland may have more lag but it may be more consistent than X11, or vice-versa).
It means that if x11 andwayland are actually drawn from identical distributions, then probability of getting two sets of samples with a statistic less than or equal to what we got above is 0.06%.
In other words, the difference would be improbable if x11 and wayland were drawn from identical distributions.
The author has taken some latency measurements under x11 and Wayland. The measurements are all over the place, so eyeballing whether the sets are meaningfully or just noisy is difficult.
The poster above used Welch’s T-Test, a standard statistical test, to determine if the sets are statistically different or not. The low p value indicates that it is likely that the two sets of measurements are drawn from different distributions. Therefore the observed latencies do systematically differ between x11 and Wayland in this experiment.
There are disputes about the answer to that, but according to most theoretical statisticians (including Fisher) and to me, it would indicate that there’s probably something wrong with the statistical model you’re using. [source: closely related to the topic of my PhD and my papers]
Sure. Please DM me and tell me how much detail you want! I know you’re smart (tridactyl) but I don’t know how interested you are in disputes in theoretical statistics.
Or in case you or anyone else doesn’t want to DM me, you can read R.A. Fisher’s original writings on this, and he was enormously influential (even when he was wrong).
That there’s not enough evidence to reject the null hypothesis. That is, we wouldn’t be able to say whether x11 and wayland are drawn from an identical distribution or not.
It’s unclear to me if there’s referential integrity across these various PStates. Or transactions across them. Anyone know?
No, and this also isn’t important in Rama’s model since PStates are not your source of truth. Depots are the source of truth and the definition of the data can and should be completely normalized. Any consistency issue in your PStates can be fixed with a recompute from depot data.
In RDBMS’s, referential integrity only protects against a subset of possible data integrity issues. When you need to denormalize your schemas in an RDBMS (due to being forced to for performance reasons), maintaining data integrity becomes an application problem. And this should never be the case for a supposed source of truth.
I don’t see how normalisation or denormalization affects whether I want transactions or not.
Can a user write into the depot with a transaction? Concretely: can they observe some state in the depot or a pstate at time T1 and then write something to the depot at time T2 only if that observed state is still valid?
Or do you get around it by having the depot be an atomic message log that you write higher-level messages to, and then each pstate can get access to that log?
Or something more exotic?
Rama has two kinds of topologies, streaming and microbatching.
Microbatch topologies are cross-partition transactions for every PState in every case. Our bank transfer example relies on this.
Stream topologies are transactional for all PState changes on one partition of a module done in the same event, no matter how many.
You don’t need transactions for depots, since all data that must be atomic should just be in the same depot record. Like for bank transfer, you append one depot record that says “Transfer $10 from Alice to Bob” and not two depot records saying “Deduct $10 from Alice” and “Give $10 to Bob”.
The ACID page in the docs goes into this in more detail.
There’s latency and extra network traffic, though with a static site those should be pretty low.
The UX might get a little weird in some ways — in the case of the site menu, the menu-open state becomes part of the history, so if you open the menu and click a link and go back, the menu’s still open. Then you go back again and it’s the same page but with the menu shut.
I expected the history to fill up as you described, but it seems the implementation avoids that by making the menu close button go back in history if possible. It does that with this HTML (shown unminified):
Doesn’t work as intended on Safari / iOS, and the post filtering also wasn’t visible to me.
The code I quoted does work as intended in Safari on iPadOS 17: it prevents history from being filled when I repeatedly tap the menu button and the menu close button.
Perhaps rather than replying to me, you meant to comment on the article as a whole. I see that on iPadOS, the CSS transition animations don’t work. The page just blinks between states like any other page load.
You’re right, I misunderstood: I thought it was intended that the menu not enter the history at all, but this way is probably better.
Clever! 👍🏻
I like this idea! Do you think it’s extreme to try and implement dark/light mode using static HTML? I can’t seem to find a good workaround for a javascript-less solution to give people the option to choose to deviate from their system preference.
But it sure feels like overkill to generate a copy of each page just to avoid making someone enable JS to change the colors on their screen… which I don’t even do because I prefer everything in dark mode anyway.
There’s a CSS-only way (using a heavily restyled checkbox) to toggle other CSS attributes:
Today I learned that
light-dark()is a thing! Thanks!I’m using a similar idea for my own dark mode checkbox: https://isuffix.com (website is still being built).
GP comment might enjoy more examples of CSS
:has()in this blog post: https://www.joshwcomeau.com/css/has/I don’t understand why so many web sites implement a dark mode toggle anyway. If your page uses CSS conditionally on
prefers-color-schemeto apply a light theme or dark theme depending on the user’s system preference, why isn’t that enough?For example, if the user is looking at your page in light theme and suddenly they think their bright screen is hurting their eyes, wouldn’t they change their system preference or their browser’s preference to dark? (If they don’t solve the problem by just lowering their screen brightness.) After they do so, not only your page but all their other apps would look dark, fixing their problem more thoroughly.
For apps (native or web) the user hangs around in for a long time, I can see some reasons to allow customizing the app’s theme to differ from the system’s. A user of an image editing app might want a light or dark mode depending on the brightness of the images they edit, or a user might want to theme an app’s windows so it’s easily recognizable in their window switcher. But for the average blog website, these reasons don’t apply.
I am curious about how many people use it as well. But it certainly is easier to change by clicking a button in your window than going into your system or browser settings, which makes me think that it would be nice to add. Again, for the imagined person who decides to deviate from their system preference.
Although you’ve made me realize that even thinking about this without putting work into other, known-to-be-used accessibility features is kind of ridiculous. There is lower hanging fruit.
Here’s a concrete example. I generally keep my browser set to dark mode. However, when using dark mode, the online training portal at work switches from black text on a white background to white text on a white background. If I wanted to read the training material, I would need to go into my browser settings and switch to light mode, which then ruins any other tab I would switch to.
If there was a toggle button at the training portal, I could switch off dark mode for that specific site, making the text readable but not breaking my other tabs. Or, if the training portal at work won’t add the button, I could at least re-enable dark mode in every tab whose site had added such a toggle.
Or, hear me out, instead of adding javascript to allow users to work around its broken css, the training portal developers could fix its css?
(Browsers should have an easy per-site dork mode toggle like the reader mode toggle.)
I feel like this is something to fix with stylus or a user script, maybe?
sounds like the button fixes it
Sure, but only on sites that provide a button. It seems a little silly that one bad site should mean that you change your settings on every other site / don’t have your preferred theme on those sites.
Or the DarkReader extension or similar.
Given how widely different colour schemes can vary, even just within the broad realms of “light” and “dark”, I can imagine some users would prefer to see some sites in light mode, even if they want to see everything else in dark mode. It’s the same reason I’ve set my browser to increase the font size for certain websites, despite mostly liking the defaults.
It would be nicer if this could be done at the browser level, rather than individually for each site (i.e. if there was a toggle somewhere in the browser UI to switch toggle between light/dark mode, and if the browser could remember this preference). As it is, a lot of sites that do have this toggle need to either handle the preference server-side (not possible with static sites, unnecessary cookies), handle the preference client-side (FOUC, also unnecessary cookies), or don’t save the preference at all and have the user manually toggle with every visit. None of these options are really ideal.
That said, I still have a theme switcher on my own site, mostly because I wanted to show off that I made two different colour schemes for my website, and that I’m proud of both of them… ;)
I remember the days when you could do
<link rel="alternate stylesheet" title="thing" href="...">and the browser would provide its own nice little ui for switching. Actually, Firefox still does if you look down its menu (view -> page style), but it doesn’t remember your preference across loads or refreshes, so meh, not a good user experience. But hey, page transitions are an IE6 feature coming back again, so maybe alternate stylesheets will too someday.The prefers dark mode css thing really also ought to be a trivial button on the browser UI too. I’m pretty sure it is somewhere in the F12 things but I can’t even find it so woe on the users lol.
But on the topic in general too, like I think static html is overrated. Remember you can always generate html on demand with a trivial program on the server with these changes and still use all the same browser features…
I’ve been preparing something like this. You can do it with css pseudo selectors and a checkbox:
:root:has(#checkbox-id:checked)or so; then you use this to either ‘respect’ the system theme, or invert it.The problems I’m having with this approach:
Yeah, I think I saw the checkbox trick before, but the problems you outline make the
site/page/darkandsite/page/lightsolution seem more enticing, since they can avoid especially the state reset issue. I like the idea of respecting/inverting the system theme as a way of preserving a good default, though!Yeah, as an alternative, for the state issue I was thinking of using a cookie + choose the styles based on it, but that brings a whole host of other “issues”
Discovered this while watching Shane Mattner’s ESP32-C3 embedded Rust tutorials: https://youtu.be/vUSHaogHs1s
Possibly love at first sight. I just did
sudo apt install neovimand then followed the install instructions (https://docs.astronvim.com/#-installation) and finally enabled the community Rust pack (https://docs.astronvim.com/#-astrocommunity) and now I have a very feature-rich Rust coding environmentWhat editor did you use previously?
I love this point about avoiding enshittification.
The most infuriating experience for me was the sudden pricing change of replit. I used to default to creating a repl when testing out an idea. However they did go through an identity crisis, and it feels like they are not interested in having people like me in their audience anymore.
I ended up cooking building my own platform for home-brewed apps: https://smallweb.run. Essentially, it allows me to have a new webapp available by just creating a new folder on my disk (with an automatic https domain).
Nice take on the moderately dynamic website.
I love that this is basically self hosted val.town!
yeah val.town is one of my biggest inspiration ! I also took some ideas from https://blot.im and https://pico.sh.
If you’re interested in following the development, please join the community discord at https://discord.smallweb.run !
Smallweb is a neat idea! Thanks for sharing :)
Link to “Optimized memcpy” is broken, @andrewrk
A standard “worst case” for allocators is producer-consumer: in a loop, one thread allocates, sends the pointer to the other thread, which then deallocates. I think for this scenario the presented allocator will give infinite growth of memory usage?
If I am reading the code correctly (which is not given!), in the producer-consumer scenario, producer will fill its slab, while consumer will fill its free list. Then, producer will rotate to the next
Thread, and fill its slab. Eventually, producer will “steal” consumer’s thread (once both have the same thread index, they’ll contend for mutex and one will be edited to the next id), and re-use all entities from the free list.But then this repeats! Once stolen free list is exhausted, we allocate one more slab, and do a merry-go-round again!
Yup, that’s the case that snmalloc and mimalloc tune for because improving it can double the throughput in a lot of transaction-processing workloads. Thread caching approaches are the worst here because the allocating thread always misses in its cache (no freed objects) and the freeing thread ends up having to walk its cache list periodically and properly free everything.
It’s also worth noting that benchmarking allocators is incredibly hard. For example, we found that snmalloc was much worse than everything else on one microbenchmark. We eventually tracked the issue down to cache contention. Our allocation pattern meant that the metadata ended up in a small handful of cache associativity sets and so we were using a small fraction of the total cache. We added some extra logic to add some jitter to the displacements to avoid this. The benchmark got faster, but everything else got slower. What made the benchmark special? It wasn’t ever touching any of the memory that it allocated and so almost 100% of the cache was available for the allocator to use. Making the benchmark touch even 20% of each allocation eliminated the gap between snmalloc and the other fast allocators. In practice, if you’re allocating a load of memory and then not touching it, the right fix is not to make the allocator faster for that workload, it’s to not allocate the memory (outside a few niche use cases such as kernel memory that you need to preallocate so an interrupt handler can use it in a situation where it can’t acquire locks).
This example, right?
It’s a great example because I didn’t think of it until someone pointed it out to me.
There is a performance data point of that example program in the readme file in that repository.
Your analysis is almost spot-on, but consider that it first tries to rotate before allocating another slab. So it is more likely that the rotation finds another freelist, and continues reusing memory from there.
At first I thought that adding rotation to the free function would improve the situation. When I tried it, I found that in the linked example it indeed reduced peak memory usage in exchange for performance, but in real world scenarios, it actually performed worse than the simple free implementation on all metrics.
Rrright! I got confused by this likely:
https://github.com/ziglang/zig/blob/5ad91a646a753cc3eecd8751e61cf458dadd9ac4/lib/std/heap/SmpAllocator.zig#L151
Feels like it should be unlikely, as enter there at most once.
So, yeah, I think this actually fixes it! Because we rotate first, I think what’ll happen is producer thread “chasing” after consumer thread! Nifty!
Still, I’d adjusted that example to say
while (true)and left it running overnight, no make sure it doesn’t OOM eventually.Hm, thinking about this for another five minutes, I think this still breaks if you have, say, 4 producer threads and one consumer thread? In this situation, producer will be more likely to rotate over another producer. While sometimes we will steal consumer’s free list, we should still fall back to slab alllocation infinitely often? I’d run a benchmark, but I have a rule against laptops in bed :P
Yup, there it is,
O(∞)total memory usage for a program thatO(1)memory at any given moment in time: https://github.com/andrewrk/CarmensPlayground/pull/1Great find. That prompts the question, is the next step to try to mitigate such problems in SmpAllocator, or keep it how it is, document its performance characteristics under different workloads, and create another allocator that is designed for mpsc? 🤔
I’m leaning towards the latter, since choosing an allocator tailored to the application is something Zig is uniquely positioned to offer.
My gut feeling answer is to just wait for some community member to write a cool pure-zig allocator that can become the default recommendation, but, at this point, I’ve been waiting for seven years for that to happen in Rust, and that didn’t quite materialize (to be fair, for a long time writing allocators in rust was hamstrung by slow threadlocals, but that has been fixed years ago as well).
With my TigerBeetle hat on: never call free, problem solved :P
I think the default allocator should perform tolerably well in all situations. SmpAllocator performs very badly in the benchmark that matklad presented, so I don’t think it should be the default. I think it would be fine to document the performance characteristics of SmpAllocator and leave it as an option.
I guess this can be said more simply: in a producer-consumer scenario, if producer and consumer have different
thread_index, then slab allocation happens eventually. If they have the same index, contention happens eventually and the indexes become different.This idea reminds me of my statistical genetics days. There’s an (in)famous piece of software, PLINK, which is just gobsmackingly fast. It achieves this by “compressing” genotypes into two-bits. The high school understanding of genetics as AA, Aa, aA, aa, can roughly be made to work even with modern sequencing datasets. Usually you can’t tell Aa from aA so you really just have AA, aA, aa (hom-ref, het, hom-alt). You can clearly stuff that into two bits: 00, 01, and 10. We can repurpose 11 for “no data” or N/A.
Now that you’ve got 32 genotypes per u64, you can start implementing all sorts of operations very quickly. Mean genotype? You can compute that with a couple masks and pop counts, followed by one division. Count of homozygous alternates? Masks and pop counts. Correlation between two genomic positions? I bet PLINK even has a linear least squares implementation written in terms of bitfiddles.
The trouble is PLINK is a special purpose binary tool. What I actually want is a library of tools for working with compressed arrays so I can use these ideas to build new tools. I hope Vortex becomes such a toolkit.
There is the BitArray in Julia, which is supposed to work directly with the underlying bits as bools, but abstracting that away so you can write code as if it was a normal array. I can’t vouch for how well optimized the code is in practice though.
As a warning, in C++ there’s a specialization for
std::vector<bool>that does something similar but breaks many of the assumptions that you expect to be true from a genericstd::vector<T>. The major one: It isn’t necessarily stored in a contiguous array, whereas the generic version is. This means that you can’t really take the address of a start and end element to a function that expects them to be contiguous, among other things. It was more than a decade ago that I ran into these kinds of issues (before I’d learned to very carefully read cppreference.com all the time) but it was definitely surprising to learn about, even just to learn that there was a specialization for it.The addressing problem is not from a non-contiguous array, it’s from being a bitset and the obvious impossibility of addressing individual bits. As a result indexing into a
vector<bool>returns a proxy object which breaks all kinds of assumptions about the interaction between vector and its T.Having a separate dedicated bitset causes none of the issues, because people don’t expect it to behave exactly like an array, and more importantly can not use it in generic contexts.
time for fractional addresses :P
Yeah, I wish they’d made it a separate type in C++ to make it clear that you can’t expect it to behave the same as the generic version. It’s a tricky design issue because lots of stuff does work the same, like iterating over it or accessing elements by index but then other parts… don’t.
Those concerns don’t apply to the Julia implementation. It uses a contiguous array and folks don’t use vector memory addresses directly anyway.
For genetic data, the work for other kinds of packed arrays is already done by the BioJulia team here.
I think their code is flexible enough that you could define a new alphabet for a different domain and still use the optimised implementations for LongSequence.
I was planning to make the move from zsh to nushell as a Rust-based shell because of its structured data capabilities, but fish might be a more mature option to try out something new. Does anyone know how the Rust rewrite of fish compares to nushell in terms of performance?
Nushell was noticeably slower than fish last I used it (maybe 6+ months ago?), but unless Nushell got job control, I don’t consider them direct competitors. Nushell is something I keep checking to use in place of things like PowerShell for my use cases. fish is awesome just being my interactive shell.
Ah I see :( I was mainly wondering if the performance benefits of not having to pass data as text between commands may outweigh the relative immaturity compared to fish
It is the command-line tools that you use, not the shell, which impose passing text around. This has always been true. For example, you can build up database-like operations (with database-like performance or possibly even better with properly PGO SIMD compilation optimizations..) using streams where there is no parsing on a per input byte basis or formatting on a per output byte basis in a couple thousand lines of Nim.
It can be helpful, of course, for very small data (where the time to fork & exec seem like large costs) to have things built in to the shell, and then the shell can matter. For such tiny data performance only really matters if there is an “outer loop” over a great many situations, though.
Fast enough at what? I don’t remember using fish and being annoyed by anything being slow.
To the contrary I’ve switched to fish on a few very low spec machines because it’s noticeably faster than other shells, even before the rust rewrite.
(From their website)
I couldn’t find any published research showing that Hedy is particularly good versus other approaches.
There is some limited evidence that kids can learn with it, but that would also be true of many other approaches.
In one of their research articles they claim that Hedy is the first gradual programming language, which is a bit annoying cos racket has had teaching languages that gradually introduce concepts for much longer, and they probably weren’t the first.
I feel like a lot of Meyer’s stuff follows a pattern: he denigrates other people’s approaches to problems because they’re not his approach, and his approach is best by the virtue of being his approach. One example from this paper:
Meyer mocks natural semantics without ever explaining the actual point of it: as a basis for tooling. Sure, conditional expression is intuitive to a 6-year-old, but I can’t force a 6-year-old inside my type checker!
This is especially ironic, because here’s Meyer’s later definition of the “conditional expression”
if C then p else q:Where C’ = “the complement of C” (not defined, but later indicated to be “¬C”), ⌊p⌋ = “Rounded version of p” ⟨post_p / Pre_p, Pre_p⟩,
r / Xis the set{[x, y] ∈ R | x ∈ X]. I’m not fully clear on what it means to “round” a postcondition; he only defines “rounding” a whole program.Now compare this to what was so bad one could only “shake one’s head”:
Which is easier to understand? Which is easier to programmatically evaluate?
To be fair, we’ve had lots of training and years to grok the Kahn notation, and 12 hours to digest the Meyer one.
This is at least a very authentic Meyer paper, we can be pretty sure it’s not written by an LLM, which I find somewhat refreshing :)
Sounds like an interesting direction to me, rants about Hoare et al. notwithstanding, and cool to see Isabelle-proofs as part of the release.
I think
⌊post_p⌋ = post_⌊p⌋
Which I think is this?
post_p / (Pre_p ∩ C)
but yeah, rounding is defined in the article only for programs, not for postconditions.
I’m skeptical about what this article contributes to the field (has this really not been done before?), but I could believe that building a theory of computation directly on set theory would be convenient for reusing set theory ideas and tools in computation theory.
This silly post about kebab shop reviews is not very good statistics.
If the hypothesis is that “the closer to the train station, the worse the kebab”, then google reviews are clearly a sample of all reviews and it is the p-value of the correlation test that the experimenter should be looking at (again, comparing against a threshold identified before calculation), but also the google reviews are likely a biased sample, etc, etc.
If the hypothesis is that “average google review of kebab shops in the study area is correlated with distance from the nearest train station” vs “they’re not”, and we assume that data collection has gathered essentially all of the reviews (so it’s not a sample), then a threshold for the correlation coefficient should really be identified before calculation.
But then, without built-in understanding of utf8, is this supposed to only cover us-ascii? The same string might be encoded in multiple ways in uncode + utf8.
Example: in unicode, å can be either the direct
U+00E5or a composition ofU+0061 + U+030A, both render the same. That means different lengths.I didn’t know of any languages canonicalising unicode strings before comparing them, but it turns out that Swift does this.
Do you know of any other languages that do?
It works for any encoding if you canonicalize to the right flavour of bytestring first.
I think this gets into the discussion at the top of the article, which is that
switch (color)should obviously be semantically equivalent toif color == ... else if color == ..., but the definition of==is ambiguous. Pointer and string equality are two answers, what you’re talking about is whether two byte strings of valid utf encoding represent the same sequence of glyphs. (*)That’s useful in some contexts but it’s more specialized than just “match on this set of byte strings”.
(*) to be super pedantic,
{ U+00E5 }and{ U+0061, U+030A }are different strings, but encode the same glyph. Text is hard.If you need unicode semantics you can use a library like zg https://codeberg.org/atman/zg
My brother in christ, please just let me switch on strings.
The word “just” is sitting on top of a lot of complexity in this statement.
Pardon my lack of zig experience but I wouldnt expect any reference type to work in switch in a systems language (similar to C)
I would count rust as a systems language and it works in magch there (not just for &str but any reference).
Any constant reference.
Do you want equivalent but differently encoded unicode strings to be considered equal?
No, just raw byte-for-byte comparison.
None of the workarounds in the article seem to address that either.
Do you think that not allowing users to switch on strings will avoid canonicalisation issues somehow?
Can users define their own string type and make switch work on it?
would it make sense to allow an optional comparator to the switch statement? e.g.
switch (mystr, &std.mem.equal) { ... }I had been imagining something like the ability to extend
switchto operate on any aggregate type that has a hash-function definition available, but having a comparator generalizes that, and even supports some interesting use-cases like canonicalization-in-comparison (as @andrewrk asked about) as well as even fancier things like edit distance matching!I find this idea quite compelling…
All the best,
-HG
Sounds like pattern matching
Can anyone explain what the kernel maintainer drama they’re talking about is?
Most of Asahi’s linux work is in Rust, such as their new GPU drivers for Apple Silicon. Part of that requires access to the DMA subsystem. An asahi maintainer other than Marcan wanted to upstream Rust bindings to DMA. These would have been part of the “Rust for Linux” subsystem, which explicitly is exempted from the usual kernel stability guarantees. The maintainer of DMA in Linux objected to them upstreaming those bindings, as he felt he would then be obligated to maintain the Rust code, and he did not want to maintain Rust. This was combined with some uncharitable comments on the Rust for Linux project’s goals. There has been a lot of debate on other internet forums about whether these comments were uncharitable about Rust, Rust for Linux, or some other distinction that personally I think doesn’t really matter.
Marcan and that maintainer then exchanged words on the LKML. Unsatisfied, Marcan appealed to his Mastodon for support. Linus then stepped in with a reply heavily condemning the appeal to social media and not really addressing the complaints that led to that point. Marcan later resigned from his kernel maintainerships, and now also from the Asahi project.
I think this understates the position of the maintainer. He sent a message to LKML that states it pretty clearly: he thinks a multi-language kernel is a terrible mistake. And I can understand his position — which to me demonstrates that there is a leadership failure happening, because this kind of fundamental conflict shouldn’t be allowed to go on this long. “Disagree and Commit”, as they say at Amazon.
https://lwn.net/ml/all/20250131075751.GA16720@lst.de/
I’ll go further, it misstates the maintainers position.
The maintainer never indicated “he felt he would then be obligated to maintain the Rust code”.
The maintainers “NAK” occurred in response to the following post
I.e. after the explicit statement that the rust people, not Hellwig, would be maintaining it separately.
(And then we reach the post you quoted, where he makes it clear that his reasons are that he doesn’t want Rust in Linux at all, not that he doesn’t want to maintain it)
Yes, I understated the understatement. :) His stated goal is to use his influence and control over the DMA subsystem to prevent Rust (edit: or any other non-C language), from being in the kernel at all. As his NAK says:
Classic LKML rhetorical style here, to use a loaded word like “cancer” to make what is actually a reasonable technical point.
It is not a reasonable technical point! It demonstrates a fundamental incuriosity about technical matters. Categorical technical statements are rarely reasonable.
I disagree in general, but not in specifics. I’ve spoken to Chris Hellman at lengths (literally 6 hours) about the topic of Rust and it was not combative. He was curious. It’s hard to sum up at length, but his disagreements are rooted in specifics and he’s a hardliner about the single source language thing. But not for the sake of being a hardliner. His hard-line stance comes out of different weighting of the issues at hand, not an outright dismissal.
Those people existing is to be expected and it doesn’t invalidate their opinions.
Obviously not a fan of LKML rhetoric, but yep, you get the tone you built over years.
You can disagree, but for the rest of us who haven’t had these super private, intimate exchanges, where we get to “really know” someone, what he puts into the public is our perception of him. To that end, he’s called the project a cancer, and declared he’s going to do whatever he can to stop it which is not what a curios person does. He comes across instead as a religious zealot or “hardliner” in your terms.
That’s fair, but it’s also a categorical mistake to read a (brusque and inappropriate) email and judge on a whole person for days and weeks on the internet. I’m not saying you “really need to know someone”, but you also need to be aware that hundreds of hours are spent on reading what those statements may mean.
Wanting to keep a large, complex, codebase in a single language is a very reasonable technical point.
Using the term “cancer” to describe it the adoption of another language, though, is inflammatory and probably didn’t help matters.
A general inclination towards this is reasonable (and one I agree with). A categorical statement is not.
Again, many other C and/or C++ projects have introduced Rust into them. Some have succeeded, while others have not. It is worth learning from them.
I’ve personally had to make the decision whether to allow an additional language into a large codebase, and there are some major negatives to consider that are unrelated to the technical aspects of the language. The maintainer said three times in the above excerpts that he doesn’t have a negative opinion of Rust itself.
I’ve always heard that Google allows only four languages in their codebase, and one of those is only because they invented it. :)
Yes, I was part of the first Rust team at Meta so I’m quite aware of the upsides and downsides! That’s why I said “categorical”.
By “categorical”, you mean saying “no second language could ever have upsides that overcome the downsides of having two languages”? I agree that’s a pretty conservative position. It’s hard to even imagine another candidate for second language, though (I dunno, Zig maybe?), so I didn’t take it as literally categorical.
This is the quote:
This is quite categorical!
I think Zig sees itself as a candidate second language for C projects, but that’s probably beside the point; nobody’s trying to bring it into the kernel.
Indeed, but that call was already decided when Rust support was first merged.
The goal of the experiment is to explore those tradeoffs. You can’t do that if you don’t merge things!
And yet they are one of the companies pushing Rust for Linux… They understand the tradeoff. The rust for linux people understand the tradeoff. They just think it is worth it.
Totally. But that does seem like a valid engineering question people could disagree about.
However, this is exactly the kind of huge impactful question that should get escalated to the chief architect. And if there’s a sanctioned scouting effort to gather experience to figure out the right answer, somebody found to be actively subverting that effort should be…corrected…by the chief architect.
It’s interesting that they won’t allow it in their internal codebase but they’re happy to let the Linux maintainers take on the burden of adding it there.
What’s your source for Google not allowing Rust in their internal code base? Google publishes crate audits for google3: https://github.com/google/rust-crate-audits/blob/main/manual-sources/google3-audits.toml
I’m not sure where you get that from - I worked at Google and literally all the code I wrote was in rust
In the google3 main repo?
Is the Linux kernel a place for curiosity or for consistency?
R4L is officially an “experiment” with goal to explore tradeoffs. I.E. satisfy curiosity.
This whole debate is so tired. The “including Rust” ship has, at least temporarily, sailed, and maintainers blocking it based on not wanting Rust/a second language doesn’t make sense when the whole point is to explore that option.
Maybe all the code will get deleted at some point, but the code needs to be merged first to then decide that based on how well it works in practice.
Why does it need to be merged though? If it’s officially an experiment, maintain it in a separate R4L tree, and everybody’s happy?
To have a thorough experiment, it must be merged into the official tree. One must be able to see how this interacts with a diverse set of hardware and software configurations, and also how this affects the development process inside the kernel community. Some maintainers are afraid of how the introduction of Rust would affect them in the future and the maintainability of the code base. Without it being in the official tree, there wouldn’t be any conclusion on those points.
It sounds like some of the kernel maintainers at least don’t want to have to pay the cost of this large-scale experiment? Like they don’t feel it’s worth the downsides of massively complicating the kernel into a multi-language codebase? Like they understand the potential benefits of memory safety but they feel they are outweighed by the risks?
These individuals believe that they know the result of the experimental before it’s been run. But they do not, because they cannot, by definition. Of course they don’t want to pay the cost, because change is hard, and deep down they know that there’s a chance that the experiment is successful, and thus their responsibilities will either change or they will have to cede responsibility to someone more capable.
Whether there is benefit to Rust in theory, there is little question. It provably solves problems that the kernel has. Whether it can solve those problems without causing larger ones is why the experiment is required. Blocking the experiment for essentially any reason is more or less indefenisble, because the entire point is that they don’t know.
That sounds way too confident and assured of a statement to me. You are discounting the opinion of subject matter experts as if they have no idea what they’re talking about. If someone came to you at your workplace and said that they want to run an experiment to convert your codebase to a new language, and they want you to help maintain and support that experiment while also delivering on all your existing obligations, would you be so quick to accept that? I don’t know about you, but personally I would and have pushed back on rewrite suggestions as a bad idea, despite the perceived benefits (for example, Management thinks that rewriting in JavaScript will make it easier to hire people for the project in the future).
Would rewriting in JavaScript have possibly made me redundant? Maybe. But it would also be massively expensive, cause huge churn, and have a real possibility of failing as a project. We can’t just ignore the very real possibility of risks because of the expected benefits.
There is no interpretation in which it’s too assured. My position is that the result of the experiment is unknown, and that the potential benefits are known, both of which are hard facts.
I have run such experiments in my workplace before. Some have succeeded and some have failed. The successes have massively outweighed the impact of the failures.
Assuming that you know the outcome before you begin is the pinnacle of hubris, which we see time and again with these entrenched maintainers. They may be domain experts, but they deserve every ounce of criticism that they receive. Closed mindedness is not how progress is achieved.
My point is that there must be some threshold that you use to decide whether some experiment is going to be useful to run or not even worth the effort because your experience and expertise tells you otherwise? Or would you accept every experiment that anyone proposed? Eg, suppose someone wanted to model your entire technical stack with TLA+ for high assurance purposes? On paper it sounds like a great idea–formally verify your concurrency properties–but you don’t see how a reasonable project lead might say ‘While this project could certainly bring great benefits, it’s not worth the tradeoff with our team size and capabilities right now. That may change in the future’?
Some threshold must exist, yes. Presumably, that threshold should be somewhere below “the progenitor of the system has decide that the experiment should be run”, which is currently the case of RFL.
Individual system maintainers should not and must not be able to usurp the direction of the project as a whole.
Your core misunderstanding is that we are coming from a tabula rasa. We are not, and if your position is that Linus’s previous endorsements shouldn’t stand, then we’re having an entirely different (and much less tenable) conversation.
Is it though? To my understanding the decision was only that they will try it and see if it works on both the technical and the social levels. There was never any guarantee given that Rust will be shipped in Linux at all costs. To my understanding Linus’s approach is to see if Rust is overall viable for the kernel, and that includes the opinions of the other subsystem maintainers. Because if they are overall against it, the experiment is dead in the water.
where do the interests of actual linux users factor into this?
I like it when drivers work correctly and don’t break every other time I update the kernel due to weird concurrency issues.
then you should update your kernel twice as often, then it will be every fourth time instead of every other time.
Again, that decision has already been made. There’s no point questioning it over and over. Doing so is distracting from the debates that are important now: how can leadership do better at enforcing these project wide decisions to not let the situation fester like R4L, and reviewing the Rust code without raising the same solved questions again and again (concern trolling).
If you’re genuinely curious about those questions, there’s plenty of text written about it. LWN is a good place to start.
I hear you, but it is a bit in the nature of project that are very distributed in decision making with high individual authority that indeed, decisions will be questioned again and again and again at every point. Those projects need a long breath and the willingness to talk about points again and again and again.
It comes with its strength, particularly, those projects are extremely resilient and fast in other regards. You can’t have both. Pushing top-down decisions in such an environment can have other unintended effects.
It’s not something to take up (period, but even more so) with the R4L devs.
Doing that is like berating a cashier for the price of an item.
It sounds like the decision has actually not been made, just dictated by ‘top management’ and now getting pushback from the rank-and-file?
The guy who blocked it is ‘top management’, inasmuch as that exists for Linux.
sorry, the guy who blocked what? has the decision been made?
The guy who blocked the patches that the Asahi Linux folks wanted, which I think were to allow writing some rust drivers.
The guy who blocked has the authority to reject patches in their subsystem.
It wasn’t a patch in their subsystem though, it was in
rust/and used the DMA subsystem.You’re right that it’s in
rust/, tho unclear to me if it’s properly in the rust subsystem or not. You and others may be right that this patch may still go thru, that seems to depend on Greg KH and Linus.but did they have the authority to reject the patch?
I don’t believe so.
so he has the authority to block it, but not to reject it? him being the guy referred to above as “the guy who blocked the patches that the Asahi Linux folks wanted.”
So the original statement, “that decision has already been made,” is supposed to mean that the decision not to allow that patch has already been made?
ThinkChaos was talking about the decision to do the Rust for Linux (R4L) experiment.
I was talking about the decision to block the patch.
Sorry for the confusion.
how could the decision have been made, if the guy who blocked the patches is ‘top management’, inasmuch as that exists for Linux?
how was the decision made?
I sure hope it’s for both! If we cannot find a balance between the two, then we have failed as engineers.
Personally I think it does matter when a core maintainer calls a project that’s part of the kernel “cancer” and the guy who’s in charge doesn’t seem to care.
Marcan got a lot of mileage out of theatrically misinterpreting the “cancer” comment, and you’re carrying it here.
The actual comment:
The cancer is not Marcan, Asahi, R4L, or Rust. The cancer is anything-but-C. The cancer is a two-language codebase.
Still a very discouraging comment from a maintainer, and it bucks Linus’ decision that trying out Rust in the kernel is OK, but the distortion of this comment has been bugging me for the last week.
R4L is a project to make Linux a two-language code base. So he quite explicitly called the goal of the project cancer.
Yep, that’s what the commenter above is referring to. He thinks that project goal is unworthy. Or, well, in his words, cancer.
I don’t have an opinion, but I think a fair and less charged way than “cancer” of stating the position is just this: using Rust would have positive benefits, but they are far outweighed by the negatives of allowing a second language into the kernel.
The pros and cons of RfL are somewhat subjective, and the funal balance is hotly debated.
But I find it quite telling that Hellwig’s core argument (that a multi-lamguage codebase requires more work) is held by people who didn’t try doing any of that work. Whereas the kernel devs who started using Rust are explicitly saying “the API compat churn isn’t that big a deal, we can do that work for you if you want”.
We mostly hear about the drama, but it seems that the overall feeling toward Rust among kernel devs (not just RfL devs) is generally positive.
but then you lose the meaning conveyed by the metaphor, of something that grows beyond its intended scope until it kills the organism.
I don’t think the metaphor conveys any technical meaning beyond what I said. I don’t know what “kills the organism” is supposed to relate to. Is the “organism” the kernel? The community of kernel developers? And what does “kill” equate to in reality? It would be better to have the discussion in concrete terms rather than metaphors.
It does convey the personal outrage of the maintainer better, certainly.
thread depth limit reached; continuing here.
yes, I’m ignoring the less absurd things you’ve said because I don’t know what to take seriously when you haven’t retracted the most absurd thing.
one characteristic of cancerous growth is that it happens whether you want it to or not.
it doesn’t imply that, but I do think he would be much more OK with a little bit of Rust if the scope were somehow guaranteed not to expand. like I don’t know, maybe there’s a carve-out for testing a particular class of rust drivers in linux to help find bugs, but eventually the drivers are ported to C for use in linux and the rust version is used in a separate kernel written from scratch in rust?
so you wouldn’t say that you want to have no cancer at all in your body, ever? if it’s a non-cancerous tumor, I have much less of a problem with it being in my body.
I agree that “kill” could have multiple meanings.
I think “growing beyond its intended scope” is technical meaning that is not included in “the benefits outweigh the negatives.” you disagree?
Cancer has no intended scope, right? “Allowing into the kernel” conveys that there shouldn’t be any in the kernel.
sure. so do you think “growing” is a technically meaningful word that is not implied in “the benefits outweigh the negatives”?
I don’t want to dig into the minutiae of why this metaphor is inaccurate — my point is that using this metaphor at all is unprofessional and counterproductive because of the heavy emotional baggage it comes with. Just say “the introduction of Rust anywhere in the codebase will inevitably result in a growing amount of Rust in the codebase”.
It’s especially odd in this context, given the history of this metaphor regarding Linux.
okay, so do you retract your previous statement that “I don’t think the metaphor conveys any technical meaning beyond what I said”?
my intention was not to get into minutia either, but we’ve gone back and forth three times and you still haven’t retracted that statement so I don’t know where you stand.
so our enemies allow themselves to use the metaphor to efficiently communicate with each other, but we have to forswear it.
If you don’t want it there at all, it’s redundant to also complain that it’s growing. Cancer also has many other attributes besides “growing” that are irrelevant or contradictory to the position taken.
The reason he used that particular metaphor was to express disgust and horror, not to be technically accurate. If he just wanted to evoke “growth” he could have said it was like ivy. Disgust and horror are not an appropriate device to call on in this context, IMO.
Re Mr. Ballmer, the enemies were wrong that time, so don’t you think evoking that argument by using the same metaphor muddies the rhetorical water a bit?
if I can’t tell whether you stand by your statements, it feels futile to interpret them as if you believe them. that’s my issue with continuing in light of the fact that you are just leaving “I don’t think the metaphor conveys any technical meaning beyond what I said” without clarifying whether you believe it.
“complaining that it’s growing” is completely different from identifying the growth as a factor in the technical drawbacks.
Cancer has a multitude of characteristics, of which growth is only one. If anything, it has many characteristics contradictory to the point being made: it’s an undesirable outcome of natural processes, it is typically self-generated from within rather than deliberately introduced, it is hard to eliminate because it’s nourished by the same system that sustains healthy cells…
If you have to tell me exactly which part of the metaphor I’m supposed to pay attention to in order to get the technical meaning, then you aren’t conveying technical meaning, you’re muddying the water. Just say what you mean.
The additional meaning being conveyed here is not technical, it’s emotional, and I do agree emotional meaning was conveyed.
I take this to mean that you still stand by the statement that “I don’t think the metaphor conveys any technical meaning beyond what I said.” if you treat all technical metaphors like this, I’m glad you’re not my coworker.
or it could mean I am conveying technical meaning but you’re being willfully obtuse. just maybe…
Well, notice that you aren’t absorbing my actual point too well, because you’re very focused on making your own.
To give him the benefit of the doubt, he could have been trying to convey the technical meaning of “growth” in the sense of “if you let Rust in, we’ll just have more Rust”. However, that’s an utterly vacuous thing to say, because obviously the whole point of the exercise was to let Rust in and see what happens, and why would you do that unless you want to have more of it if the experiment succeeds? It conveys no useful information beyond “I don’t think Rust is a good idea at all, for reasons”; in other words, “the benefits don’t outweigh the negatives”. In fact it confuses matters because it implies he thinks Rust would be OK if only it didn’t grow.
I’m actually trying to be polite to the guy here, because I truly believe it was an emotional outburst, not an attempt at a technical argument. His actual technical argument (stated much more clearly elsewhere) is that there should be no Rust at all in the kernel, ever, not that once there is some, there will be more. So if “cancer” was supposed to mean “growth” as a technical argument, he misstated his own argument.
I don’t see the misinterpretation, lonjil said
Not
Or whatever phrase you would need to make your objection stand.
I could be wrong, but I get the feeling that people didn’t quite get why Hellwig used the word “cancer”. I could be wrong about this in a lot of ways, maybe I too misunderstood what Hellwid meant, maybe I misjudged how others are interpreting what he said. But allow me to add my 2 cents.
People seem to be interpreting what Hellwig said (i.e. calling R4L “cancer”) as him saying R4L is bad, evil, a terrible thing, etc. And I totally understand why many would think that, and would agree this is a terrible choice of words. But I think that Hellwig’s focus is actually on the fact that cancer spreads. A better word to use would probably be like “infectious”, or “viral”. Although I disagree with him, I think what he was saying is that, despite what Rust developers promised, Rust will spread to more corners of the kernel and increase the maintenance burden and will have a negative impact on the kernel as a whole.
I think technical leaders should focus on clarity rather than everyone else trying to do exegesis on what they write.
That’s very fair, I was not trying to defend Hellwig’s communication skills in the least here.
I think you’re right, but that doesn’t exculpate Hellwig in the least. One of the defining characteristics of cancer is that it is malignant. It seems pretty clear that he used this metaphor because he thought that Rust would spread and that it is dangerous and harmful.
yes, which is a perfectly valid technical position and to disallow a maintainer from expressing that position is completely insane.
“infectious” or “viral” have different connotations. viral would be if they thought the BSDs might catch it. infectious is not as specific as cancer; cancer specifically grows from one area and consumes more and more of the things next to it. infections can do that but not necessarily.
actively expelling the cancer probably would have drawn even more ire.
Unrelated to the rust issue, my understanding is that such toxic language actually isn’t all that strange in the Linux kernel maintainer community. It feels like they try somewhat not to overdo it, but they aren’t that much concerned about the toxicity. More like, “I’ll try but I don’t actually care”.
The only reason they’ve toned it down is due to financial pressure from major funders of LF.
The fact that many of those LF funders are also the employers of major Linux maintainers is also relevant, I feel.
I think it’s kind of crazy to label a vivid and extremely useful metaphor as “toxic.”
It’s useful in some cases, but when the effect of it is raising the temperature of the discussion, then I would say it isn’t useful in that instance. There are other ways to convey the same meaning that wouldn’t have attracted the same amount of strong emotions. We’ve seen examples of such alternate arguments in this thread that would have been more suitable.
sure. but then don’t fixate on the word, and acknowledge that their choice to use strong language can also be an effect of the raising temperature of the discussion, for which responsibility is shared among all participants.
There’s this idea that multi-language codebases are categorically a disaster, right? What really bothers me is how small-minded it is. There are many C and C++ codebases that have introduced Rust. It’s not easy but it is possible, and it often is worth it. Some attempts have been successful and others haven’t. You can learn so much from other people’s experiences here!
Simply calling multi-language codebases a cancer closes off discussion rather then opening it. That goes against the basic curiosity that is a hallmark of engineering excellence.
Honestly everything around R4L that has been going has just cemented in me the belief that the average long time Linux maintainer is fundamentally incompetent. They’re reasonably good at One Thing, but totally incurious and scared of anything outside of that One Thing they’re good at.
Do you suppose there might be some kind of incentive structure behind this pattern?
That seems likely, though I don’t think I can usefully speculate on the details.
Is it possible that we could take an experienced kernel maintainer’s opinion seriously instead of dismissing it immediately as incurious and ‘against’ engineering excellence? If a co-worker who specializes in a different domain came to you and kept insisting that something you know is very difficult and not worth the tradeoff is actually quite easy, would you be incurious and ‘against’ engineering excellence if you dismissed their wild suggestions without a deep discussion of the tradeoffs?
Evaluating who to take seriously and who not to is a matter of judgment cultivated over a lifetime.
Depends on many things, but in general, yes, of course, especially if they were someone I took seriously on other matters. If nothing else, it’s a chance to sharpen my arguments. If I keep having the same arguments over and over I might even step back and write an essay or two explaining what I believe and why I believe it. Here’s one that’s public that was a result of the sort of back-and-forth you’re talking about.
Your post strayed away from a factual re-telling of facts and towards narrativization here.
Reading the discussions on this elsewhere, there’s no way to describe this that wasn’t going to get accused of bias by people holding strong opinions on either marcan or on the rust for linux project.
I’m not accusing of bias (which is more or less unavoidable), but of constructing a narrative. The facts are: he exchanged words on the LKML and then wrote Mastodon posts about the exchange. The narrative you’re constructing is: he exchanged words on LKML, then because he was unsatisfied with how that discussion was going, he wrote Mastodon posts about the exchange in an attempt to gather support.
I think it would be better to leave out the guesswork about intentions, or at least clearly separate it from the factual retelling of events.
marcan has posted about this, so it’s hardly guesswork: https://lore.kernel.org/rust-for-linux/208e1fc3-cfc3-4a26-98c3-a48ab35bb9db@marcan.st/
Correction: Marcan never “exchanged words” with Christoph Hellwig on the patches, which had not even been posted by him. He just replied to the thread with a tirade.
A next-gen scripting language; I feel like the current ontology of “Python, Ruby, JS and sometimes Lua” has some good room for improvement from the functional language/Rust/Gleam culture. Something with a strong-ish type system with sum types instead of 1990’s “OO everything”, gradual typing built in from the start, and still keeping an emphasis on fast iteration and smooth development. A compiler built around JIT or incremental recompilation might be nice. Julia and Crystal are steps in the right direction but personally neither really quite makes the leap from “good” to “excellent enough to want to learn and use vs just bashing something out in Python”. Roc might be going in the direction I want?
A language-agnostic program runtime and software build/distribution system that stores programs and libraries as wasm or another openly-specified bytecode, then makes CPU-portable and efficient programs by AoT-compiling them to super-optimized static native code on installation. You know, like Android does with dex. I think this solves a lot of the issues that currently exist with DLL’s. It would let people distribute code as portable binary libraries, build code with portable libraries, not have the runtime overhead calling of dynamic linking, and enable the code inlining and optimizations it disallows. But you could still do things like distribute updated libraries or make plugins for a program (again as bytecode), plop them on an existing computer, and the system would just recompile the programs that need them. The main downside is it’s pretty difficult to always go from bytecode to native code as well as you can go from source code to native code.
A browser scripting environment that enables automatic FFI generation for any language that can target wasm. This kinda exists but has the Slight Issue that any language using it essentially has to be able to talk with JS’s bonkers object model and API assumptions, which usually isn’t worth the effort vs just writing your damn thing in JS.
A web browser that takes a sane subset of HTML and the browser API, and comes with lots of conformance and test suites so other people can make other browsers using the same subset of functionality.
Something nicer than email, intended for resilience and abuse-resistance. (I’m dreaming about it, I don’t know how to do it.)
Something nicer than Discord/Zulip/whatever for chat, built around a peer-to-peer pubsub model from the start but less grotty than XMPP.
Something nicer than IPFS for sharing host-anywhere content-hashed data. And then a crypto identity system built on top of it that lets you do public-key signing and encryption of messages and stuff.
About a million video games.
…yeah, I don’t understand the people who ask “how do you come up with ideas for things to build?”
Reminds me of UNCOL, TenDRA TDF, ANDF, Taos/Taos and even LLVM bitcode. Some more successful than others, some more effective at being actually architecture-neutral than others.
Also ChrysaLisp, which is made by a former Taos engineer. It’s both a reimplementation/imagining of TaoVM, and an entire desktop environment and Lisp dialect, but you could focus just on the VM part.
Oooh, thanks! Sounds like there’s lots of prior art to learn the mistakes of then. :D
I’ve (not) been working on this in the background as a long-term project for a while, now, as someone who has zero experience with writing compilers; I call it Rumland (it was initially a pie in the sky idea of an ML-inspired typed scripting language that compiles to Ruby or otherwise interoperates w/ the Ruby ecosystem, then I decided to just do the most minimal, fastest-compiling ML-inspired typed scripting language I could.)
I wanted a scripting language like a mini-Haskell that can do cold builds of at least small scale personal programs (< 100k SLOC), including linking, within tens of milliseconds on a half-decade-old mid-range machine. I wouldn’t mind compiling mid-range programs like ~500k SLOC w/ dozens of transitive dependencies that fast as well, but the main focus is personal programs of < 10k SLOC, then small (but really nice to use and good looking of course) open source projects off less than 100k SLOC.
As soon as I get it compiling a very minimal language of only ADTs, functions, basic effect tracking, imports, and a tiny stdlib, I’ll lock any further changes that don’t maintain that build performance; if I have to give up all the Haskell-inspired niceties I have in mind, I’ll do it. (It’s not a public project, but I may one day make it one.)
I’ve been keeping an eye on Roc as well, and it hits a lot of the same stuff I started this project for; I love that they do their own linking (and I love the recent growth in interest in linker performance in general, e.g. mold, wild).
This is the problem, though. Nothing can make a very compelling case vs. using Python, at least not enough to justify the learning cost. Of the languages you list, you can already use Gleam/Elixir for scripting. There’s also F# - I’ve personally used it as a scripting language for quite a bit. You have always been able to use lisps like Racket or Clojure (via Babashka) for scripting, and there’s a slew of neolangs that try to innovate in this space. But there’s nothing so compelling that would make you want to give up everything you have. Nobody wants to give up the comforts of a familiar language to learn one that is marginally better, and at the end of the day, that’s what all of these “next-gen” improvements are. That’s one of the reasons why I’m looking forward to the upcoming stabilization of
cargo-script; Instead of a new language that has all the shiny features of rust that make it comy for me to use, I’d rather just use rust for scripting too, if I can help it.Yeah, but I’m a language nerd, so if the language is neat then for me the learning cost is potentially negative. :-P And lately a lot of my own scripting-language needs are not “get this big program to work robustly” or even “do a lot of talking with this particular domain-specific library”, but just juggling and automating fairly basic things. It feels kinda wrong to use Elixir or F# for such things, though maybe I should work harder on giving it a go. Jank also looks like it might make me actually want to write clojure, too.
It’s just… It’s not that hard to write a shell-script-like pipeline that’s nicer than Python’s
subprocesslib, and I would kick a puppy to the moon for something with the built-in facility to take a bunch of JSON, stuff it into a an in-memory table (or even a persistent one), and let me query it like the database it is instead of rolling my own indices and joins with hashtables checking for the fields I care about. …hell, that alone might be a killer feature to build a language around. There’s lots of annoying REST API’s out there that force you to do this sort of thing to get the information you want.And in the parent comment:
These strike me as requirements for very different languages: the first one is a shell language that revolves around running pipelines of commands while the second is a more traditional scripting language.
But the idea of exposing an SQLite database as a type in a shell language is interesting. I’ve been thinking of a slightly different angle: streaming an SQLite table contents as a way of providing statically-typed pipelines (still very fuzzy admittedly).
I’d say F# fits what you want, especially the last bit. You can use LINQ to query over an array, so you could do basically everything you wrote. The only sticking point would be calling subprocesses, but I’m sure there’s a library that makes it easier, or you could write the boilerplate for that once and put it on nuget.
Hmmm, thanks. I’ve used F# before for medium-sized things and it’s been pleasant but I’ve never tried using it for scripting stuff. Given that u/pyj agrees with you in the sibling comment, I’ll give it a try sometime.
Have you played with kdb+/q at all?
Never heard of it, I’ll have to check it out! Thanks.
DuckDB is great for that.
Somewhat in that direction is my Next Generation Shell. Preparing arguments for calling external programs, running external programs (curl to JSON API for example), parsing their output, handling their exit codes properly, small scale data manipulation are the intended use cases.
Small scripts “should not suffer” from language capabilities (no public static void main). If you decide to have main() you get automatic parsing of command line arguments to fit main() definition, of which you can have several because it’s a multimethod.
https://github.com/ngs-lang/ngs
https://github.com/ngs-lang/ngs/wiki/Use-Cases
Hope this helps
and also Common Lisp, specially with http://ciel-lang.org/ which can be seen as Babashka-like (without the platform limitations):
it’s still alhpa, we are refining it and adding more features.
Ciel looks really cool. I read the docs yesterday and liked almost everything I saw. I feel like I want to start a new project with it, or maybe retrofit one I’ve started but haven’t done much with yet.
I also tried using F# as a scripting replacement for Python, and had very similar experience and opinion as what you found. It was amazing to me how great it worked with .Net Core across Windows/Mac/Linux.
I don’t ask that exactly, but I do get stuck not having enough ideas about how to implement those ideas.
This in particular, I have long lists of ideas but any one of them is just discouraging to think about how to produce the intended effect. I have a resolution to do at least one game jam this year, the last two years I’ve struggled after a fairly productive 2022 (when I was between jobs).
Hmmm, do we need «What software you’d try implementing, but would want to discuss implementation design first» threads?
On the next-gen scripting language thing… and having chatted with you a bit about some of the past work we’ve both done… one thing I was thinking about the other day on the way back from a flight test campaign was how much the Lua scripting in ArduPilot scares me. It’s an incredibly powerful feature but also seems like it has potential to have unbounded runtime that could cause problems.
A potential solution I scribbled down was to use a more limited VM that is amenable to ahead-of-time verification for bounded execution time and remembered https://ebpf.io/. The pseudo-C code that you use to write most eBPF code is… not great and a long way away from being an easy scripting language, but it seems like a language similar to what you’re describing could be compiled down to something like eBPF and used safely for that kind of scripting.
Ooh, yeah, scripting for high-safety systems sounds like a very interesting and very tricky proposition. Verification would be good, but you’d probably also need a good simulation framework. Do it well and there might be a pretty great market for it though, and Eris knows it’s a market that needs more convenient tools!
It’s relatively easy to take https://github.com/n0-computer/iroh nowadays, and either built-in https://www.iroh.computer/proto/iroh-blobs , or just something custom with
baofor incremental hash verification. Depending on what you want it to do exactly, could be a little spin on top of the minimal demo app they have: https://github.com/n0-computer/sendmeThese are two things I could try to incorporate into https://github.com/dpc/rostra
With email-like applications the whole problem is the high level “how to do it”. We’d want something that anyone (or a lot of potential senders) could send you an email while preventing spam and abuse. These two contradict each other. I’m hoping a “only friend of my friends” could be a reasonable mdidleground, which is adjacent to what I’m already implementing.
Sounds rather like Veilid
You might be interested in Peergos, which does cryptographic identity, signing and encryption on top of a much more efficient ipfs implementation.
https://book.peergos.org
This sounds interesting. What would the ABI look like across langs?
I dunno! Probably a lot like what any ABI looks like, which will be imperfect by necessity. The nice thing about wasm is you can use unmanaged memory with it, mostly unlike JVM or .NET, but then adding managed memory a la the GC proposal is a bit of a mess. So I kinda feel like the ABI is not the special part here and will never be able to be perfect, it just has to be something that does the job pretty well. An AOT compiler/optimizer should hopefully be able to make most of the ABI go away with whole-program opt when the program is turned into native code.
Not less “grotty” than XMPP, because it’s built on top of XMPP, but I recently found Applesauce, a project that plans to use https://spritely.institute/ and XMPP to build a peer-to-peer pubsub social networking platform like you describe.
Your link is broken. Here’s the right one: https://spritely.institute
Thank you for catching my mistake!
rakudo (perl6) is pretty cool.
Tangentially, @doug-moen wrote:
Sure, that’s nice and all. Clojure is a great example of this philosophy. However, for something like Rust it’s simply a non-option.
If nothing else, your fancy functional immutable data thingy will need a garbage collector, and we will continue to use systems languages that make indirection explicit, such as Rust, to write those.
You can get some of this with Elixir or Gleam, but you’re always going to have references creeping in (foreign keys in databases, URLs, names, etc).
My first choice would have been Android TV or even LineageOS Android TV.
I have Jellyfin and other apps on LineageOS Android TV (on a jailbroken Amazon Fire Stick), and I’ve mostly been very happy with it.
Pushcx did list the Nvidia Shield as an option, which is Android TV.
And LineageOS doesn’t have a supported release of Android TV.
Yes but he just mentioned “The NVIDIA Shield is not bad”
I hate saying this, but I’d love a TL;DR summary of these articles because they’re so lengthy. I’ve seen the others in this series posted here, and I still disagree with a lot of the premises. I don’t want clickable URLs in my shell buffer, I don’t think dragging a file onto the shell window should copy the file there, etc. There are just better ways of doing most of those things (IMO).
I don’t really get the focus on terminal emulation and TTYs. Nobody’s forcing you to use that stuff. There are a lot of ways to do “terminal stuff” outside of a terminal, with or without a CLI and TUI.
The obvious way would be to use Vim or Emacs and do the work there. Emacs has a TUI and can (or can be made to) do all of the things the author is asking for, and the underlying details of TTY protocols and terminal emulation is entirely hidden away.
Honestly, besides
emacs, do you have anything else on your mind?I’m an emacs user, so that’s the only one I have much direct experience with.
I’d imagine most editors can do it. I’ve seen a half-dozen “magit for $something” articles posted here, so probably those tools. VSCode has a shell/terminal thing.
Vscode’s terminal is a regular terminal emulator.
It doesn’t mean it’s correct, though. I like the the separation of concerns, and so an editor should be an editor. What people are doing in the e.g. Neovim community is insane to me (package managers, many Lua rewrites of tools you can use in the CLI, etc.).
I think the evolution needs to come for « CLI replacements », and so far besides Cat9 I really haven’t see anything promising (besides emacs, but it comes with its own issues).
We likely consume, and expect to consume, content on the Internet in very different ways. I personally don’t deal with short-form whatevers at all. If it’s not worth at least an hour with a printout in the arm chair I simply stay away.
You are forced to use terminal emulation and ttys every step of the way (assuming Linux/BSD). The only one that successfully paved another way is Android and it worked rather well for them, and even then it pops up here and there. I know, I worked on it. The first thing your desktop has to do in order to get something in the screen is a ridiculous dance to tell the TTY to please stop doing its thing, and even then it’s there in ways that completely undoes the security model they try to implement.
That something is ‘hidden away’ doesn’t mean that it doesn’t influence every layer above it. Emacs pays through the nose to get its TUI to work. There’s a ridiculous amount of code and workarounds to present its rather modest UI.
A tangent perhaps but as a former Emacs user and as former Vim user (neither are even close to doing the things I ask for) I find it fairly bizarre to use a text editor to do shell work and vice versa. Just as I avoid trying to use a tooth brush to clean my toilet or a toilet brush to clean my teeth. They can get the job done, sortof, but at a price and with some risks.
I’m going to disagree with your last paragraph and say it’s quite natural to use a text editor to do shell work.
Most “shell work” involves navigating around, searching through, and otherwise working with text, which is exactly the toolset built into text editors. cat, grep, sed, and awk are command line tools implementing text editor functionality in a clunky (clunky for interactive use) pipeline/streaming fashion. Likewise, ls, find, cd, and other filesystem navigation is also basic text editor functionality. They all spit out text, and what better way to view and work with it than a text editor? Maybe I’ve had too much Emacs koolaid, though.
For the rest of it, I don’t care much. On a day-to-day basis I never have problems related to TTYs, and with a few (easily fixed) exceptions in the past, it’s been that way for a long time.
With that logic you can use a hex editor for everything, it’s all “just” bytes no? representations matter, interventions matter. Trivialities aside, they operate on different mechanisms and assumptions. Text editing is buffers with some intended persistence and at rather small buffer sizes. You can expect to seek and build features assuming that works; the backing store changing is an exception, not the default and so on. The cli shell is stream processing. Its data sources are hierarchies of short lived processes that might expect something back from you to do their job and can spit out infinite amounts of data. The world can change beneath its feet between every command. To do that it gets to go through all the ugly legacy – signals, sessions, buffer bloat, line blocking and with that, serial communication.
This brings us back to never having problems; ‘works for me’-isms is just dismissed with ‘doesn’t work for me’-isms and doesn’t get you anywhere. It’s systems engineering, one doesn’t get to think like that. You don’t have to dig far to find people with serious problems with the TTY and a few vulnerabilities a year. First(?) Flatpack sandbox escape? the tty again.
Previous maintainer of the subsystem? Original gangster Alan Cox - https://lwn.net/Articles/343828 quit (incidentally triggered by a bug from Emacs losing data while trying to be a shell) Greg tried to step up, well - https://www.youtube.com/watch?v=g4sZUBS57OQ
Whenever they try to fix something to make embedded development bearable, you commit the Linus taboo of breaking userspace because your stack relies on hidden assumptions that all go back to the emulating the fantasy computer. To edit text. It’s holding everyone back.
Just the other week I had a coffee spill. It fried my mouse and worse, I had to make a new cup of coffee. I grabbed another rodent from the box of spare input devices. I plugged it in. Now my keyboard stopped working. I unplugged the mouse again. The kernel crashed and I lost a lot of work. Angry reboot. Plug back in. Now there’s half a second delay between keypresses every 10 seconds. 2 minutes later the machine froze up. It turned out the USB mouse exposed itself as serial rather than USB human-interface. This meant some /dev/input/eventN gets routed through the tty layer again. The display server ran the device node non-blocking, but that only goes skin deep. You still have ‘the global’ lock around in tty and a handful of mutexes on top of that. Race condition into memory corruption and panic. Then race condition into deadlock.
The point you missed is that a text editor has tools to work with text, while hex editors typically do not.
I find working inside of emacs generally easier than in the terminal because it has great tools for working with buffers of text, which is what most shell commands produce. If there were a hex editor that made it even easier, then I would indeed use it instead.
Therefore we reject the null hypothesis that
x11andwaylandare drawn from identical distributions.What does this mean?
they imported the data reported in the blog post as two numpy/scipy vectors, then ran a student’s t-test to test the hypothesis that the two distributions the two vectors were sampled from are different. (docs on
ttest_indhere).Technically this is bad statistics, because you’d want to specify your
confidencesignificance level ahead of time (typical value is 5%, which roughly corresponds to a difference of two standard deviations between the two distributions). Essentially, this test is saying that if the two variables came from the same underlying distribution, you would expect to see this data no more than 0.07% of the time. I suspect that the default for scipy uses a 5%confidence intervalsignificance level, so statistically you can say that this data comes from the same distribution (i.e. the latency is the same between X11 and Wayland) no more than 5% of the time.Edit: the above is sorta wrong. You don’t specify a significance level in the t-test itself, but you should specify it before performing the test, otherwise it’s hard to interpret the results based on your null hypothesis. So if we state the null hypothesis as “I expect that if X11 has the same latency as Wayland, and I measure the latency 20 times in both, I expect to get a different result no more than one time out of twenty”, then the above test is coherent and we can reject the null hypothesis.
Reading the docs some more, ~this might also be a bad test choice because it
I don’t think we have any reasonto suspect a priori (i.e. Wayland may have more lag but it may be more consistent than X11, or vice-versa).Edits: my stats is rusty AF.
Just a correction,
equal_val=Falsemeans we use Welsch’s t-test which drops the equal-variance assumption.I usually use p<0.01. I didn’t put that in since the p-value was below most typical thresholds (except those darn physicists).
from what I remember from statistics class, this means that there is a 0.06% chance that the distribution of both sets of values is the same
in other words the difference is highly unlikely to be due to statistical variance
It means that if
x11andwaylandare actually drawn from identical distributions, then probability of getting two sets of samples with a statistic less than or equal to what we got above is 0.06%.In other words, the difference would be improbable if
x11andwaylandwere drawn from identical distributions.The author has taken some latency measurements under x11 and Wayland. The measurements are all over the place, so eyeballing whether the sets are meaningfully or just noisy is difficult.
The poster above used Welch’s T-Test, a standard statistical test, to determine if the sets are statistically different or not. The low p value indicates that it is likely that the two sets of measurements are drawn from different distributions. Therefore the observed latencies do systematically differ between x11 and Wayland in this experiment.
Interesting! What would a high p value indicate?
There are disputes about the answer to that, but according to most theoretical statisticians (including Fisher) and to me, it would indicate that there’s probably something wrong with the statistical model you’re using. [source: closely related to the topic of my PhD and my papers]
Could you expand on this or share a link?
Sure. Please DM me and tell me how much detail you want! I know you’re smart (tridactyl) but I don’t know how interested you are in disputes in theoretical statistics.
Or in case you or anyone else doesn’t want to DM me, you can read R.A. Fisher’s original writings on this, and he was enormously influential (even when he was wrong).
That there’s not enough evidence to reject the null hypothesis. That is, we wouldn’t be able to say whether
x11andwaylandare drawn from an identical distribution or not.