I love the simple and straightforward nature of the tool and its presentation. Even though it requires a bit of legwork, it’s very transparent and I have enough understanding and information to adapt it to my use case. Thanks for this!
I understand the rationale for including the original emoji (unicode wants to be a superset of existing character sets) but they should have been put in a code space reserved for backwards compatibility with bad ideas, not made such a big part of unicode.
At this point, there’s a strong argument for a new character set that is a subset of unicode that removes all of the things that are not text. We already have mechanisms for embedding images in text. Even in the ‘90s, instant messaging systems were able to avoid sending common images by having a pre-defined set of pictures that they referenced with short identifiers. This was a solved problem before Unicode got involved and it’s made text processing an increasingly complicated mess, shoving image rendering into text pipelines for no good reason.
The web could have defined a URL encoding scheme for emoji from an agreed set, or even a shorthand tag with graceful fallback (e.g. <emoji img="gb-flag;flag;>Union Flag</emoji>, which would render a British flag if you have an image for gb-flag, a generic flag if you don’t, have ‘Union Flag’ as the alt text or the fall back if you don’t support emoji). With the explicit description and fallback, you avoid the things like ‘I’m going to shoot you with a 🔫’ being rendered as ‘I’m going to shoot you with a {image of a gun}’ or ‘I’m going to shoot you with a {image of a water pistol}’ depending on the platform: if you didn’t have the water-pistol image, you’d fall back to the text, not show the pistol image.
Like it or not, emoji are a big part of culture now. They genuinely help convey emotion in a fairly intuitive manner through text, way better than obscure tone indicators. I mean, what’s more understandable?
“Are you going to leave me stranded? 😛”
“Are you going to leave me stranded? [/j]”
It definitely changes the meaning of the text. They’re here to stay, and being in Unicode means they got standardized, and it wouldn’t have happened otherwise.
Of course there’s issue with different icon sets having different designs (like how Samsung’s 😬 was completely different from everyone else’s), but those tend to get resolved eventually.
Like it or not, emoji are a big part of culture now. They genuinely help convey emotion in a fairly intuitive manner through text, way better than obscure tone indicators.
Except they don’t. Different in groups assign different meanings to different ones. Try asking someone for an aubergine using emoji some time and see what happens.
“Are you going to leave me stranded? 😛”
This is culturally specific. It’s an extra set of things that people learning English need to learn. This meaning for sticking out your tongue is not even universal across European cultures. And that’s one of the top ten most common reaction emoji, once you get deeper into the hundreds of others the meaning is even further removed. How would you interpret the difference between 🐶 and 🐕 in a sentence?
Of course there’s issue with different icon sets having different designs (like how Samsung’s 😬 was completely different from everyone else’s), but those tend to get resolved eventually.
That’s an intrinsic property of using unicode code points. They are abstract identifiers that tell you how to find a glyph. The glyphs can be different. A Chalkboard A and a Times A are totally different pictures because that’s an intrinsic property of text. If Android has a gun and iOS has a waterpistol for their pistol emoji, that’s totally fine for characters but a problem for images.
😱 Sure emojis are ambiguous . And different groups can use them differently. But that doesn’t mean they don’t convey meaning? The fact that they are so widely used should point towards them being useful no? 😉
I never said that embedding images in text is not useful. I said that they are not text, do not have the properties of text, and treating them as text causes more problems than it solves.
I disagree. At best, they are precursors of an ideographic script. For a writing system, there has to be some kind of broad consensus on semantics and there isn’t for most emoji beyond ‘that is a picture of X’.
Please describe to me the semantics of the letter “р”.
For alphabetic writing systems, the semantics of individual letters is defined by their use in words. The letter ‘p’ is a component in many of the words in this post and yours.
Thank you! (That was actually U+0440 CYRILLIC SMALL LETTER ER, which only featured once in both posts, but no matter.)
the semantics of individual letters is defined by their use in words
The thing is, I disagree. “e” as a letter itself doesn’t have ‘semantics’, only the words including it do[1]. What’s the semantics of the letter “e” in “lobster”? An answer to this question isn’t even wrong. It gets worse when different writing systems interpret the same characters differently: if I write “CCP”, am I referring to the games company CCP Games? Or was I abbreviating сoветская социалистическая республика? What is the semantics of a letter you cannot even identify the system of?
Emoji are given meaning of different complexity by their use in a way that begins to qualify them as logographic. Most other writing systems didn’t start out this way, but that doesn’t make them necessarily more valid.
[1]: The claim doesn’t even hold in traditional logographic writing systems which by all rights should favor your argument. What is the semantics of the character 湯? Of the second stroke of that character? Again, answers aren’t even wrong unless you defer to the writing system to begin with, in which case there’s no argument about (in)validity.
Yes, but their original point is that we should be able to compose emojis like we compose words, as in the old days of phpBB and instant messaging. :mrgreen:
Just a nit: people do compose emojis - I see sequences of emojis all the time. People send messages entirely of emojis that other people (not necessarily me) understand.
Yeah, and also there’s nothing wrong with that, it’s something any group can and should be able to do. I have no entitlement to be able to understand what other people say to each other (you didn’t claim that, so this isn’t an attack on you. I am just opposed to the “I don’t like / use / understand emojis how other people use them therefore they are bad” sentiment that surfaces periodically).
But not of characters and rarely true even of ideographs in languages that use them (there are exceptions but a language is not useful unless there is broad agreement on meaning). It’s not even true of most words, for the same reason: you can’t use a language for communication unless people ascribe the same meaning to words. Things like slang and jargon rarely change more than a small fraction of the common vocabulary (Clockwork Orange aside).
Without getting into the philosophy of what language is, I think this skit best illustrates what I mean (as an added bonus, emoji would have resolved the ambiguities in the text).
Note I’m not arguing for emoji to be in Unicode, I’m just nitpicking the idea that the problem with them is ambiguity.
Socrates would like to have a chat with you. I won’t go through the philosophical tooth-pulling that he would have enjoyed, but suffice it to say that most people are talking past each other and that most social constructions are not well-founded.
I suspect that this is a matter of perspective; try formalizing something the size of English Prime (or, in my case, Lojban) and see how quickly your intuitions fail.
I understand the rationale for including the original emoji (unicode wants to be a superset of existing character sets) but they should have been put in a code space reserved for backwards compatibility with bad ideas, not made such a big part of unicode.
Except emoji have been absolutely stellar for Unicode: not only are they a huge driver of adoption of unicode (and and through UTF8) because they’re actively desirable to a large proportion of the population, they’ve also been a huge driver of improvements to all sorts of useful unicode features which renderers otherwise tend to ignore despite their usefulness to the rendering of actual text, again because they’re highly desirable and platforms which did not support them got complaints. I fully credit emoji with mysql finally getting their heads out of their ass and releasing a non-broken UTF8 (in 2010 or so). That’s why said unicode consortium has been actively leveraging emoji to force support for more complex compositions.
And the reality is there ain’t that much difference between “image rendering” and “text pipeline”. Rendering “an image” is much easier than properly rendering complex scripts like arabic, devanagari, or burmese (or Z̸̠̽a̷͍̟̱͔͛͘̚ĺ̸͎̌̄̌g̷͓͈̗̓͌̏̉o̴̢̺̹̕), even ignoring that you can use text presentation if you don’t feel like adding colors to your pileline.
Even in the ‘90s, instant messaging systems were able to avoid sending common images by having a pre-defined set of pictures that they referenced with short identifiers.
After all what’s better than one standard if not fifteen?
This was a solved problem before Unicode got involved and it’s made text processing an increasingly complicated mess, shoving image rendering into text pipelines for no good reason.
This problem was solved by adding icons in text. Dingbats are as old as printing, and the Zapf Dingbats which unicode inherited date back to the late 70s.
The web
Because nobody could ever want icons outside the web, obviously. As demonstrated by Lucida Icons having never existed.
subset of unicode that removes all of the things that are not text
It sounds like you disagree solidly with some of Unicode’s practices so maybe this is not so appealing, but FWIW the Unicode character properties would be very handy for defining the subset you’d like to include or exclude. Most languages seem to have a stdlib interface to them, so you could pretty easily promote an ideal of how user input like comment boxes should be sanitized and offer your ideal code for devs to pick up and reuse.
new character set that is a subset of unicode that removes all of the things that are not text
and who’d be the gatekeeper on what the text is and isn’t? What would they say about the ancient Egyptian hieroglyphs? Are they text? If yes, why, they are pictures. If no, why, they encode a language.
It might be a shallow dissimilar, but people trying to tell others what forms of writing text are worthy of being supported by text rendering pipelines gets me going.
If the implementation is really so problematic, treat emojis as complicated ligatures and render them black and white.
and who’d be the gatekeeper on what the text is and isn’t? What would they say about the ancient Egyptian hieroglyphs? Are they text? If yes, why, they are pictures. If no, why, they encode a language.
Hieroglyphics encode a (dead) language. There are different variations on the glyphs depending on who drew them (and what century they lived in) and so they share the property that there is a tight(ish, modulo a few thousand years of drift) coupling between an abstract hieroglyph and meaning and a loose coupling between that abstract hieroglyph and a concrete image that represents it. Recording them as text is useful for processing them because you want to extract the abstract characters and process them.
The same is true of Chinese (though traditional vs simplified made this a bit more complex and the unicode decisions to represent Kanji and Chinese text using the same code points has complicated things somewhat): you can draw the individual characters in different ways (within certain constraints) and convey the same meaning.
In contrast, emoji do not convey abstract meaning, they are tightly coupled to the images that are used to represent them. This was demonstrated very clearly by the pistol debacle. Apple decided that a real pistol image was bad because it was used in harassment and decided to replace the image that they rendered with a water pistol. This led to the exact same string being represented by glyphs that conveyed totally different meaning. This is because the glyph not the character encodes meaning for emoji. If you parsed the string as text, there is no possible way of extracting meaning without also knowing the font that is used.
Since the image is the meaningful bit, not the character, we should store these things as images and use any of the hundreds of images-and-text formats that we already have.
More pragmatically: unicode represents writing schemes. If a set of images have acquired a significant semantic meaning over time, then they may count as a writing system and so can be included. Instead, things are being added in the emoji space as new things that no one is using yet, to try to define a writing scheme (largely for marketing reasons, so that ‘100 new emoji!’ can be a bullet point on new Android or iOS releases).
It might be a shallow dissimilar, but people trying to tell others what forms of writing text are worthy of being supported by text rendering pipelines gets me going.
It’s not just (or even mostly) about the rendering pipelines (though it is annoying there because emoji are totally unlike anything else and have required entirely new feature to be added to font formats to support them), it’s about all of the other things that process text. A core idea of unicode is that text has meaningful semantics distinct from the glyps that they represent. Text is a serialisation of language and can be used to process that language in a somewhat abstract representation. What, aside from rendering, can you do with processing of emoji as text that is useful? Can you sort them according to the current locale meaningfully, for example (seriously, how should 🐕 and 🍆 be sorted - they’re in Unicode and so that has to be specified for every locale)? Can you translate them into a different language? Can you extract phonemes from them? Can you, in fact, do anything useful with them that you couldn’t do if you embedded them as images with alt text?
Statistically, no-one cares about hieroglyphics, but lots of people care about being able to preserve emojis intact. So text rendering pipelines need to deal with emojis, which means we get proper hieroglyphics (and other Unicode) “for free”.
Plus, being responsible for emoji gives the Unicode Consortium the sort of PR coverage most organizations spend billions to achieve. If this helps them get even more ancient writing systems implemented, it’s a net good.
What, aside from rendering, can you do with processing of emoji as text that is useful?
Today, I s/☑️/✅/g a text file.
Can you sort them according to the current locale meaningfully, for example (seriously, how should 🐕 and 🍆 be sorted - they’re in Unicode and so that has to be specified for every locale)?
At this point, there’s a strong argument for a new character set that is a subset of unicode that removes all of the things that are not text.
All that’s missing from this sentence to set off all the 🚩 🚩 🚩 is a word like “just” or “simply”.
Others have started poking at your definition of “text”, and are correct to do so – are hieroglyphs “text”? how about ideograms? logograms? – but really the problem is that while you may feel you have a consistent rule for demarcating “text” from “images” (or any other “not text” things), standards require getting a bunch of other people to agree with your rule. And that’s going to be difficult, because any such rule will be arbitrary. Yours, for example, mostly seem to count certain very-image-like things as “text” if they’ve been around long enough (Chinese logograms, Egyptian hieroglyphics) while counting other newer ones as “not text” (emoji). So one might reasonably ask you where the line is: how old does the usage have to be in order to make the jump from “image” to “text”? And since you seem to be fixated on a requirement that emoji should always render the same on every platform, what are you going to do about all the variant letter and letter-like characters that are already in Unicode? Do we really need both U+03A9 GREEK LETTER CAPITAL OMEGA and U+2126 OHM SIGN?
So one might reasonably ask you where the line is: how old does the usage have to be in order to make the jump from “image” to “text”?
Do they serialise language? They’re text. Emoji are not a writing system. They might be a precursor to a writing system (most ideographic writing systems started with pictures and were then formalised) but that doesn’t happen until people ascribe common meaning to them beyond ‘this is a picture of X’.
And since you seem to be fixated on a requirement that emoji should always render the same on every platform, what are you going to do about all the variant letter and letter-like characters that are already in Unicode?
That’s the opposite of my point. Unicode code points represent an abstraction. They are not supposed to require an exact glyph. There are some things in Unicode to allow lossless round tripping through existing character encodings that could be represented as sequences of combining diacritics. They’re not idea in a pure-Unicode world but they are essential for Unicode’s purpose: being able to represent all text in a form amenable to processing.
For each character, there is a large space of possible glyphs that a reader will recognise. The letter A might be anything from a monospaced block character to a curved illustrated drop character from an illuminated manuscript. The picture is not closely coupled to the meaning and changing the picture within that space does not alter the semantics. Emoji do not have that property. They cause confusion when slightly different glyphs are used. Buzzfeed and similar places are full of ‘funny’ exchanges from people interpreting emoji differently, often because they see slightly different glyphs.
The way that emoji are used assumes that the receiver of a message will see exactly the same glyph that the sender sends. That isn’t necessary for any writing system. If I send Unicode of English, Greek, Icelandic, Chinese, or ancient Egyptian, the reader’s understanding will not change if they change fonts (as long as the fonts don’t omit glyphs for characters in that space). If someone sends a Unicode message containing emoji, they don’t have that guarantee because there is no abstract semantics associated with them. I send a picture of a dog, you see a different dog, I make a reference to a feature of that dog and that feature isn’t present in your font, you are confused. Non-geeks in my acquaintance refer to them as ‘little pictures’ and think of them in the same way as embedded GIFs. Treating them as characters causes problems but does not solve any problems.
Do they serialise language? They’re text. Emoji are not a writing system. They might be a precursor to a writing system (most ideographic writing systems started with pictures and were then formalised) but that doesn’t happen until people ascribe common meaning to them beyond ‘this is a picture of X’.
I think this is going to end up being a far deeper and more complex rabbit hole than the tone of your comment anticipates. Plenty of things that are in Unicode today, and that you undoubtedly would consider to be “text”, do not hold up to this criterion.
For example, any character that has regional/dialect/language-specific variations in pronunciation seems to be right out by your rules. So consider, say, Spanish, where in some dialects the sound of something like the “c” in “Barcelona” is /s/ and in others it’s /θ/. It seems hard to say that speakers of different dialects agree on what that character stands for.
At this point, I feel like the cat is out of the bag; people are used to being able to use emoji in almost any text-entry context. Text rendering pipelines are now stuck supporting these icons. With that being the case, wouldn’t it be way more complexity to layer another parsing scheme on top of Unicode in order to represent emoji? I can see the argument that they shouldn’t have been put in there in the first place, but it doesn’t seem like it would be worth it to try to remove them now that they’re already there.
On the technical side, for what it does, it’s actually pretty good. Async calls and .await don’t need to do heap allocations. The Future trait is relatively simple (API surface of C++ coroutines seems much larger IMHO). Ability to use futures with a custom executor makes it possible to use them, efficiently, on top of other languages’ runtimes, or for clever applications like fuzzing protocols deterministically.
Usability-wise, like many things in Rust, they have a steep frustrating learning curve, and a couple more usability issues than other Rust features. But once you learn how to use them, they’re fine. They don’t have any inherent terrible gotchas. Cancellation at any await point is the closest to a gotcha, but you can get away with that since Rust uses destructors and guards so much. In contrast with old node-style callbacks, which had issues no matter how experienced you were: subtlety of immediate vs deferred callbacks, risk of forgetting to call a callback on some code path or when an exception is thrown, need to deal with possibility of having a callback called more than once, etc.
The famous color article implied that async should be an implementation detail to hide, but that’s merely a particular design choice, which chooses implicit magic over clarity and guarantees. Rust doesn’t hide error propagation. Rust doesn’t hide heap allocations, and Rust doesn’t hide await points. This is a feature, because Rust is used for low-level code where it can be absolutely necessary to have locks, threads, and syscalls behave in specific predictable ways. Having them sometimes invisibly replaced with something completely else would be as frightening as having UB.
async Rust is fine for what it accomplishes, but is generally harder to use than normal Rust, and normal Rust was already changeling for many developers. Which would be fine if async Rust was only used where warranted, but it actually became the default and dominant dialect of Rust.
[Async Rust] actually became the default and dominant dialect of Rust.
Is it? I’m just dabbling into Rust and not very experienced by any means, but at least so far i’ve gotten the impression that async stuff is mostly found in some “leaf” and specialized crates, and that more “core” crates like regex or serde are just normal non-async Rust.
I never got the impression of async Rust being dominant, much less the default. For which i’m grateful really, as i too share the impression that the async stuff in Rust is quite complicated and difficult to grok, and i would prefer to avoid it if possible.
The famous color article implied that async should be an implementation detail to hide, but that’s merely a particular design choice, which chooses implicit magic over clarity and guarantees.
async/await is the callee telling the caller how it wants to be executed/scheduled.
Languages with no async/await let the caller decide how the callee should be executed/scheduled.
In Erlang/Elixir, any function can be given to spawn(), or Task.start().
In Go, any function can be given to go ....
It’s up to the caller to determine how the function should be scheduled. async/await tend to contaminate your whole code base. I’ve seen many Python and some Rust+tokio codebases where everything up to main() was an async function. At that point, can we just get rid of the keyword?
You’ve given examples of higher-level languages with a fat runtime. This includes golang, which requires special care around FFI.
This “implicit async is fine, you don’t need the await syntax” is another variant of “implicit allocations/GC are fine, you don’t need lifetime syntax”.
True in most cases, but Rust is just not that kind of language, on purpose. It’s designed to “contaminate” codebases with very explicit code, and give low-level control and predictable performance of everything.
You’ve given examples of higher-level languages with a fat runtime.
Aren’t we considering tokio and all the other third-party libraries that you need to bring in in order to schedule async code, a fat runtime?
The comparison with lifetimes is unfair. Here, we are adding the async keyword to almost every function, and the await keyword to almost every function call. With lifetimes, there is far more options. But if we were just adding ‘static to every reference as a lifetime, then yes, I would ask if it’s really needed.
The thing that makes Rust different from e.g. Go: being able to choose your runtime. And it doesn’t even need to be fat, there are embedded runtimes or libraries such as Smol.
No, I wouldn’t put tokio in the same category. With something like Erlang your whole program lives in Erlang’s environment. It’s opaque, and sits between you and the OS, so it’s not a surprise when it inserts “magic” into that interaction.
But Rust is for systems programming. It’s not supposed to intervene where you don’t ask it to. Things that Rust does are transparent, and you’re free to program for the underlying OS.
In case of Rust you use tokio as a library. Tokio is dominant, but not necessary. I’ve worked on projects that used custom async executors. You can use async in embedded programming. You can use async in the kernel. It’s a builder for state machines, not a VM.
In relation to lifetimes I did not mean just the amount of tokens you add to the source code, but the fact that Rust requires you to be precise and explicit about details that could have been abstracted away (with a GC). Rust’s types have “colors” in many ways: owned or borrowed, shared or mutable, copy or non-copy. Rust could have had “colorless” types: everything shared mutable copyable, but chose to have all this complexity and syntax for a reason.
Rust could ship a green thread runtime in libstd (again). It could hide the async and await keywords and insert glue code automatically. It would be nicer syntactically. It would be easier to use. It would be a good tradeoff for majority of programs — just like benefits of adding a GC. But Rust choses to be low level and offer transparency, predictability, and control instead. In low-level programming knowing function’s mode of execution is as important as knowing whether it mutates or frees memory.
So we are told, but the home page talks about applications like command-line tools and network services. Seemingly the same kind of ‘systems programming’ that Rob Pike was saying Go is good for.
All of coreutils are in C. Nfs, sshd, samba, ntpd are in C. Nginx, apache are in C. Rust wants to be where C is.
There is a large overlap with Golang, and not every Rust program had to be in Rust, like not every C program had to be in C. But Rust is sticking to being low-level enough to be able to replace C everywhere.
All those tools are written in C because they want to be able to run wherever you can run a C compiler. Tools that are written in Rust most often don’t care about having that level of portability.
Rust vs C platform support is a complex topic. Rust tools generally have better portability across contemporary platforms. Windows often has native support, whereas in POSIX-centric C projects that’s typically “MS sucks, use WSL?” (and I don’t count having to build Linux binaries in a Linux VM as portability to Windows).
Rust’s platform support is not too bad these days. I think it covers everything than Debian supports except IA64 and SH4. And Rust also has some support for S390x, AVR, m68k, CUDA, BPF. Haiku, VxWorks, PlayStation 1, 3DS, and Apple Watch.
Rust doesn’t support platforms that LLVM/clang can’t find maintainers for, and these are generally pretty niche and/or obsolete. I think C projects support targets like Itanium or Dreamcast more out of tradition and a sense of pride, rather than any real-world need for them.
GCC backend (rustc_codegen_gcc) for Rust is in the works, so eventually Rust won’t be limited by LLVM’s platform support.
Languages with no async/await let the caller decide how the callee should be executed/scheduled.
In Go, everything is a goroutine, so in a sense everything is already contaminated with async/await. Code that blocks is automatically an ‘await point’, unless you use the go keyword. So I don’t think the semantics around caller/callee are not any different from async Rust with e.g. tokio::main as an entry point. The difference is you have to manually mark await points. However you do get better control, e.g. tokio::select! can act on any async future, not just channel reads..
I’m not a fan of what feels like needless hostility (confrontational tone?) in the article, and was expecting to hate it going in, but it does make some good points.
There’s an important distinction between a future—which does nothing until awaited—and a task, which spawns work in the runtime’s thread pool… returning a future that marks its completion.
I feel like this point in particular does not get attention when talking about async in languages and took me a long while to get the mental model for.
To whatever challenges teaching Rust has, async adds a whole new set.
I disagree with this opinion. In any language with native async infrastructure built-in, I’ve had to learn how it works pretty intimately to effectively use it. My worst experiences have been with Python’s asyncio while the easiest was probably F#.
I disagree with this opinion. In any language with native async infrastructure built-in, I’ve had to learn how it works pretty intimately to effectively use it.
I don’t think you’re disagreeing? The article is essentially saying that you have to learn async along with the rest of the language, and you are also saying that you had to learn async with the rest of the language.
In languages with concurrency and no async/await (erlang, elixir, go, …), the choice of the scheduling model of your code is determined at the call site. The callee should not care about how it is executed.
let x = fetch(...).await;
tokio::spawn(async { fetch(...).await; });
You have the same amount of control of scheduling. If you’re referring to being unable to call an async method from a sync context, this is technically also true in Go, but since everything runs in a goroutine everything is always an async context.
What makes Rust harder is the semantics around moving and borrowing but also the “different concrete type for each async expression” nature of the generated state machines. For example this is easy in go, but painful in Rust:
handlers[e] = func(...) ...
// later
x := handlers[event]
go x()
The real scary thing is that it took users weeks to notice that it shipped, despite that it wasn’t obfuscated in any way. This shows how risky the ecosystem is, without enough eyes reviewing published crates. If any high profile crate author gets infected with malware that injects itself into crates, it’s going to be an apocalypse for Rust.
I think it’s only a sign that we’re unaware until this hits a sandboxed / reproducable build system. I guess that’s currently distribution packaging or projects that otherwise use Nix or Bazel to build.
If the complaint is that binaries are more difficult to audit than source, and no one is auditing, then it should make no difference either way from a security perspective.
I think “weeks” is a bit of an exaggeration. People were openly discussing it at least a week after release. It’s true though that it didn’t blow up on social media until weeks later and many people didn’t realise until then.
If it had been a security issue or it was done by someone much less reputable than the author of serde or if the author did not respond then I suspect rustsec may have been more motivated to post an advisory.
Something that I might have expected to see included in this comment, and that I instead will provide myself, is a plug for bothering to review the code in one’s (prospective) dependencies, or to import reviews from trusted other people (or, put differently, to limit oneself to dependencies that one is able and willing to review or that someone one trusts has reviewed).
I recall that kornel at least used to encourage the use of cargo-crev, and their Lib.rs now also shows reviews from the newer and more streamlined cargo-vet.
I note that the change adding the blob to Serde was reviewed and approved through cargo-vet by someone at Mozilla. I don’t think that necessarily means these reviewing measures would not be useful in a situation that isn’t as much a drill (i.e., with a blob more likely to be malicious).
Yeah - my recollection of crev is that libraries like serde often got reviews like “it’s serde, might as well be the stdlib, I trust this without reviewing it as the chances of it being malicious are basically zero”
What a ridiculous thing to have even happened in the first place, let alone refusing to acknowledge there could possibly be an issue for so long. I glad it’s been fixed but would make me think twice about using serde. I’m sure it’ll be fine, who’s ever heard of a security issue in a codec anyway?
Remember that there are real human being maintaining serde. It is not, in fact, blindingly obvious to all developers that the pre-compiled blobs were bad; on this site there were loud voices on both sides. Can you imagine suddenly getting caught in the crosshairs of angry developers like that? When I imagine it, it feels bad, and I’m liable to get defensive about it.
It may also have been a failed attempt at fixing something you’ve heard people complain about all the time, probably even about your code that slows down peoples builds (*). So yeah it was a bad idea in hindsight, but we don’t need more burned out maintainers from this. And I say this as someone who is openly disappointed by this happening.
(*) I’m not going to discuss how much time it actually saved.
Yeah, basically the biggest gains are offset by process creation being surprisingly slow. I’m working on a follow-up article where I talk about that in detail.
I posted your piece because it was the first one that explained in detail what the hell was going on, specifically how serde works. Looking forward to a followup.
That’s how it started, then they centralized everything with one team that doles out the “managed CI” offering, with their own global library and controls. Any competing infra gets flagged and audited hardcore until you give up by attrition.
This seems to only be checking the performance under –release. Most compilation is done without –release, meaning that most of the proc macro will not be optimized.
As someone who packages software, I think it’s worth noting that packagers expect different things than end users, though they are compatible.
One of my wishes is to avoid blobs from a vendor, since we can’t always recompile those in the build process to work with the architectures we support.
(The other big difference is the DESTDIR env var. End users don’t generally care, but it becomes essential when preparing a package)
I therefore understand those who support their end users, before getting packaged.
The real human being maintaining serde knew about the pushback that would happen and did it on purpose to prove a point in a pre-RFC he submitted. I don’t feel particularly bad about him getting pushback for using half the Rust ecosystem as his guinea pigs. (In fact I would like to see more of it.)
The real human being maintaining serde knew about the pushback that would happen and did it on purpose to prove a point in a pre-RFC he submitted.
What’s the reason to believe in this over any other explanation of the situation? E.g. that pushback was unexpected and that the RFC is the result of the pushback, rather than a cause?
I consider dtolnay a competent open source maintainer who understands the people who run his code well, and I would expect any competent open source maintainer to expect such pushback.
But how that necessary leads to “on purpose to prove a point”?
I don’t think dtolnay expected exactly zero pushback. But, given that some people in this thread argue quite a reasonable point that binaries are actually almost as fine as source, it is plausible that only bounded pushback was expected.
“Someone else is always auditing the code and will save me from anything bad in a macro before it would ever run on my machines.” (At one point serde_derive ran an untrusted binary for over 4 weeks across 12 releases before almost anyone became aware. This was plain-as-day code in the crate root; I am confident that professionally obfuscated malicious code would be undetected for years.)
I don’t see someone competent casually pushing such a controversial change, casually saying that this is now the only supported way to use serde, casually pushing a complete long pre-RFC that uses the controversial change to advance it, and then casually reverting the change in the span of a few days. That takes preparation and foresight.
I actually respect this move. It is exactly the kind of move I would do if I had goodwill to burn and was frustrated with the usual formal process, and it takes boldness and courage to pull it off the way he did it. I also think the pushback is entirely appropriate and the degree of it was quite mild.
Aha, thanks! I think that’s a coherent story to infer from this evidence (and I was wondering if there might be some missing bits I don’t know).
From where I stand, I wouldn’t say that this explanation looks completely implausible, but I do find it unlikely.
For me, the salient bits are:
what it says on the tin. dtolnay didn’t write a lot of responses in the discussion, but what they have written is more or less what I have expected to see from a superb maintainer acting in good faith.
there wasn’t any previous Wasm macro work that was stalled, and that required nefarious plans to get it unstuck.
really, literally everyone wants sandboxed Wasm proc macros. There can’t be any more support for this feature already. What is lacking is not motivation or support but (somewhat surprisingly) a written-down RFC for how to move forward and (expectedly) implementation effort to make it come true.
dtolnay likes doing crazy things! Like how all crates follow 1.0.x versions, or watt, or deref-based specialization in anyhow. So, “because I can” seems like enough motivation here.
“if you don’t like a feature in this crate, don’t use the crate or do the groundwork to make implementing this feature better” feels like a normal mode of operation for widely used OSS projects with sole maintainers. I’ve said as much with respect to MSRV of my once_cell crate.
I agree that there are multiple intepretations possible and that yours also follows from the evidence available. The reason I think it’s reasonable to consider something deeper to be going on is: every single Rust controversy I’ve discussed with key Rust people had a lot more going on than was there on the surface. Case in point: dtolnay was also the one thus far unnamed by anyone speaking for the project person who was involved in ThePHD’s talk being downgraded from a keynote. If I see someone acting surreptitiously in one case I will expect that to repeat.
O_o that’s news to me, thanks. It didn’t occur that dtopnay might have been involved there (IIRC, they aren’t a team lead of any top-level team, so I assume weren’t a member of the notorious leadership chat)
Calling me anonymous is pretty funny, considering I’ve called myself “whitequark” for close to 15 years at this point and shipped several world-class projects under it.
whitequark would be pretty well known to an old Rust team member such as matklad, having been one themself, so no, not anonymous… buut we don’t know this is the same whitequark, so yes, still anonymous.
I mean, I wrote both Rust language servers/IDEs that everyone is using and whitequark wrote the Ruby parser everyone is using (and also smaltcp). I think we know perfectly fine who we are talking with. One us might be secretly a Labrador in a trench coat, but that doesn’t have any bearing on the discussion, and speculation on that topic is hugely distasteful.
In terms of Rust team membership, I actually don’t know which team whitequark was on, but they are definitely on the alumni page right now. I was on the cargo team and TL for the IDE team.
You were talking about “Rust teams” and the only way I’ve seen that term used is to indicate those under the “Rust Project”. Neither person is on a Rust team or an alumni.
Tbh, it more reeks of desperation to make people’s badly configured CI flows faster. I think that a conspiratorial angle hasn’t been earned yet for this and that we should go for the most likely option: it was merely a desperate attempt to make unoptimized builds faster.
I think this is hard to justify when someone comes to you with a security issue, when your response is “fork it, not my problem”, and then closing the issue, completely dismissing the legitimate report. I understand humans are maintaining it, humans maintain all software I use in fact, and I’m not ok with deciding “Oh, a human was involved, I guess we should let security bad practices slide”. I, and I’m sure many others, are not frustrated because they didn’t understand the security implications, but because they were summarily dismissed and rejected, when they had dire implications for all their users. From my understanding, Serde is a) extremely popular in the Rust world, and b) deals in one of the most notoriously difficult kinds of code to secure, so seeing the developers’ reaction to a security issue is very worrying for the community as a whole.
The thing is, its not unambiguous whether this is a security issue. “Shipping precompiled binaries is not significantly more insecure than shipping source code” is an absolutely reasonable stance to have. I even think it is true if we consider only first-order effects and the current state of rust packaging&auditing.
Note also that concerns were not “completely dismissed”. Dismissal looks like “this is not a problem”. What was said was rather “fixing this problem is out of scope for the library, if you want to see it fixed, work on the underlying infrastructure”. Reflecting on my own behavior in this discussion, I might be overly sensitive here, but to me there’s a world of difference between a dismissal, and an acknowledgment with disagreement on priorities.
let alone refusing to acknowledge there could possibly be an issue for so long.
This is perhaps a reasonable take-away from all the internet discussions about the topic, but I don’t think this actually reflects what did happen.
The maintainer was responsive on the issue and they very clearly articulated that:
they are aware that the change makes it harder to build software for some users
they are aware that the change concerns some users from the security point of view
non-the-less, the change is an explicit design decision for the library
the way to solve this typical open-source dilemma is to allocate the work with the party needs the fruits of the work
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I do think it was a mistake to not predict that this change will be this controversial (or to proceed with controversial change without preliminary checks with wider community).
But, given that a mistake had been made, the handling of the situation was exemplary. Everything that needed fixing was fixed, promptly.
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I’m still waiting to hear what “security concern” there was here. Other language-package ecosystems have been shipping precompiled binaries in packages for years now; why is it such an apocalyptically awful thing in Rust and only Rust?
The main thing is loss of auditing ability — with the opaque binaries, you can not just look at the package tarbal from crates.io and read the source. It is debatable how important that is: in practice, as this very story demonstrates, few people look at the tarballs. OTOH, “can you look at tarballs” is an ecosystem-wide property — if we lose it, we won’t be able to put the toothpaste back into the tube.
This is amplified by the fact that this is build time code — people are in general happier with sandbox the final application, then with sandboxing the sprawling build infra.
With respect to other languages — of course! But also note how other languages are memory unsafe for decades…
The main thing is loss of auditing ability — with the opaque binaries, you can not just look at the package tarbal from crates.io and read the source.
It’s not that hard to verify the provenance of a binary. And it appears that for some time after serde switched to shipping the precompiled macros, exactly zero people actually were auditing it (based on how long it took for complaints to be registered about it).
OTOH, “can you look at tarballs” is an ecosystem-wide property — if we lose it, we won’t be able to put the toothpaste back into the tube.
The ecosystem having what boils down to a social preference for source-only does not imply that binary distributions are automatically/inherently a security issue.
With respect to other languages — of course! But also note how other languages are memory unsafe for decades…
My go-to example of a language that often ships precompiled binaries in packages is Python. Which is not exactly what I think of when I think “memory unsafe for decades”.
It’s not that hard to verify the provenance of a binary.
Verifying provenance and auditing source are orthogonal. If you have trusted provenance, you can skip auditing the source. If you audited the source, you don’t care about the provenance.
It’s a question which one is more practically important, but to weight this tradeoff, you need to acknowledge its existence.
This sounds like:
People say that they claim about auditing, and probably some people are, but it’s also clear that majority don’t actually audit source code. So they benefits of audits are vastly overstated, and we need to care about provenance and trusted publishing.
This doesn’t sound like:
There’s absolutely ZERO security benefits here whatsoever
I don’t know where your last two blockquotes came from, but they didn’t come from my comment that you were replying to, and I won’t waste my time arguing with words that have been put in my mouth by force.
That’s how I read your reply: as an absolute refusal to acknowledge that source auditing is a thing, rather than as a nuanced comparison of auditing in theory vs auditing in practice.
It might not have been your intention to communicate that, but that was my take away from what’s actually written.
In the original github thread, someone went to great lengths to try to reproduce the shipped binary, and just couldn’t do it. So it is very reasonable to assume that either they had something in their build that differed from the environment used to build it, or that he binary was malicious, and without much deeper investigation, it’s nearly impossible to tell which is the answer. If it was trivial to reproduce to build with source code you could audit yourself, then there’s far less of a problem.
Rust doesn’t really do reproducible builds, though, so I’m not sure why people expected to be able to byte-for-byte reproduce this.
Also, other language-package ecosystems really have solved this problem – in the Python world, for example, PyPI supports a verifiable chain all the way from your source repo to the uploaded artifact. You don’t need byte-for-byte reproducibility when you have that.
I guesss I should clarify that in GP comment the problem is misalignment between maintainer’s and user’s view of the issue. This is a problem irrespective of ground truth value of security.
Maybe other language package ecosystems are also wrong to be distributing binaries, and have security concerns that are not being addressed because people in those ecosystems are not making as much of a fuss about it.
If there were some easy way to exploit the mere use of precompiled binaries, someone would have by now. The incentives to use such an exploit are just way too high not to.
Anecdotally, I almost always see Python malware packaged as source code. I think that could change at any time to compiled binaries fwiw, just a note.
I don’t think attackers choosing binary payloads would mean anything for anyone really. The fundamental problem isn’t solved by reproducible builds - those only help if someone is auditing the code.
The fundamental problem is that your package manager has near-arbitrary rights on your computer, and dev laptops tend to be very privileged at companies. I can likely go from ‘malicious build script’ to ‘production access’ in a few hours (if I’m being slow and sneaky) - that’s insane. Why does a build script have access to my ssh key files? To my various tokens? To my ~/.aws/ folder? Insane. There’s zero reason for those privileges to be handed out like that.
The real solution here is to minimize impact. I’m all for reproducible builds because I think they’re neat and whatever, sure, people can pretend that auditing is practical if that’s how they want to spend their time. But really the fundamental concept of “running arbitrary code as your user” is just broken, we should fix that ASAP.
The fundamental problem is that your package manager has near-arbitrary rights on your computer
Like I’ve pointed out to a couple people, this is actually a huge advantage for Python’s “binary” (.whl) package format, because its install process consists solely of unpacking the archive and moving files to their destinations. It’s the “source” format that can ship a setup.py running arbitrary code at install time. So telling pip to exclusively install from .whl (with --only-binary :all:) is generally a big security win for Python deployments.
(and I put “binary” in scare quotes because, for people who aren’t familiar with it, a Python .whl package isn’t required to contain compiled binaries; it’s just that the .whl format is the one that allows shipping those, as well as shipping ordinary Python source code files)
Anecdotally, I almost always see Python malware packaged as source code. I think that could change at any time to compiled binaries fwiw, just a note.
Agree. But that’s a different threat, it has nothing to do with altered binaries.
I don’t think attackers choosing binary payloads would mean anything for anyone really. The fundamental problem isn’t solved by reproducible builds - those only help if someone is auditing the code.
Code auditing is worthless if you’re not sure the binary you’re running on your machine has been produced from the source code you’ve audited. This source <=> binary mapping is precisely where source bootstrapping + reproducible builds are helping.
The real solution here is to minimize impact. I’m all for reproducible builds because I think they’re neat and whatever, sure, people can pretend that auditing is practical if that’s how they want to spend their time. But really the fundamental concept of “running arbitrary code as your user” is just broken, we should fix that ASAP.
This is a false dichotomy. I think we agree on the fact we want code audit + binary reproducibility + proper sandboxing.
Agree. But that’s a different threat, it has nothing to do with altered binaries.
Well, we disagree, because I think they’re identical in virtually every way.
Code auditing is worthless if you’re not sure the binary you’re running on your machine has been produced from the source code you’ve audited. This source <=> binary mapping is precisely where source bootstrapping + reproducible builds are helping.
I’m highly skeptical of the value behind code auditing to begin with, so anything that relies on auditing to have value is already something I’m side eyeing hard tbh.
I think we agree on the fact we want code audit + binary reproducibility + proper sandboxing.
I think where we disagree on the weights. I barely care about binary reproducibility, I frankly don’t think code auditing is practical, and I think sandboxing is by far the most important, cost effective measure to improve security and directly address the issues.
I am familiar with the concept of reproducible builds. Also, as far as I’m aware, Rust’s current tooling is incapable of producing reproducible binaries.
And in theory there are many attack vectors that might be present in any form of software distribution, whether source or binary.
What I’m looking for here is someone who will step up and identify a specific security vulnerability that they believe actually existed in serde when it was shipping precompiled macros, but that did not exist when it was shipping those same macros in source form. “Someone could compromise the maintainer or the project infrastructure”, for example, doesn’t qualify there, because both source and binary distributions can be affected by such a compromise.
Aren’t there links in the original github issue to exactly this being done in the NPM and some other ecosystem? Yes this is a security problem, and yes it has been exploited in the real world.
What I’m looking for here is someone who will step up and identify a specific security vulnerability that they believe actually existed in serde when it was shipping precompiled macros, but that did not exist when it was shipping those same macros in source form. “Someone could compromise the maintainer or the project infrastructure”, for example, doesn’t qualify there, because both source and binary distributions can be affected by such a compromise.
If you have proof of an actual concrete vulnerability in serde of that nature, I invite you to show it.
The existence of an actual exploit is not necessary to be able to tell that something is a serious security concern. It’s like laying an AR-15 in the middle of the street and claiming there’s nothing wrong with it because no one has picked it up and shot someone with it. This is the opposite of a risk assessment, this is intentionally choosing to ignore clear risks.
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening. To me this is up with with Sony’s rootkit back in the day, completely unexpected, unauthorised behaviour that no reasonable person would expect, nor would they look out for it because it is just such an unreasonable thing to do to your users.
I think the point is that if precompiled macros are an AR-15 laying in the street, then source macros are an AR-15 with a clip next to it. It doesn’t make sense to raise the alarm about one but not the other.
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening.
I think this is extreme. No additional accessing of any kind was done. Binaries don’t have additional abilities that build.rs does not have. It’s not at all comparable to installing a rootkit. The precompiled macros did the same thing that the source macros did.
The existence of an actual exploit is not necessary to be able to tell that something is a serious security concern. It’s like laying an AR-15 in the middle of the street and claiming there’s nothing wrong with it because no one has picked it up and shot someone with it. This is the opposite of a risk assessment, this is intentionally choosing to ignore clear risks.
Once again, other language package ecosystems routinely ship precompiled binaries. Why have those languages not suffered the extreme consequences you seem to believe inevitably follow from shipping binaries?
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening.
Even the most extreme prosecutors in the US never dreamed of taking laws like CFAA this far.
To me this is up with with Sony’s rootkit back in the day, completely unexpected, unauthorised behaviour that no reasonable person would expect, nor would they look out for it because it is just such an unreasonable thing to do to your users.
I think you should take a step back and consider what you’re actually advocating for here. For one thing, you’ve just invalidated the “without any warranty” part of every open-source software license, because you’re declaring that you expect and intend to legally enforce a rule on the author that the software will function in certain ways and not in others. And you’re also opening the door to even more, because it’s not that big a logical or legal leap from liability for a technical choice you dislike to liability for, say, an accidental bug.
The author of serde didn’t take over your computer, or try to. All that happened was serde started shipping a precompiled form of something you were going to compile anyway, much as other language package managers already do and have done for years. You seem to strongly dislike that, but dislike does not make something a security vulnerability and certainly does not make it a literal crime.
I think that what actually is happening in other language ecosystems is that while there are precompiled binaries sihpped along some installation methods, for other installation methods those are happening by source.
So you still have binary distribution for people who want that, and you have the source distribution for others.
I have not confirmed this but I believe that this might be the case for Python packages hosted on debian repos, for example. Packages on PyPI tend to have source distributions along with compiled ones, and the debian repos go and build packages themselves based off of their stuff rather than relying on the package developers’ compiled output.
When I release a Python library, I provide the source and a binary. A linux package repo maintainer could build the source code rather than using my built binary. If they do that, then the thing they “need to trust” is the source code, and less trust is needed on myself (on top of extra benefits like source code access allowing them to fix things for their distribution mechanisms)
So you still have binary distribution for people who want that, and you have the source distribution for others.
I don’t know of anyone who actually wants the sdists from PyPI. Repackagers don’t go to PyPI, they go to the actual source repository. And a variety of people, including both me and a Python core developer, strongly recommend always invoking pip with the --only-binary :all: flag to force use of .whl packages, which have several benefits:
When combined with --require-hashes and --no-deps, you get as close to perfectly byte-for-byte reproducible installs as is possible with the standard Python packaging toolchain.
You will never accidentally compile, or try to compile, something at runtime.
You will never run any type of install-time scripting, since a .whl has no scripting hooks (as opposed to an sdist, which can run arbitrary code at install time via its setup.py).
I mean there are plenty of packages with actual native dependencies who don’t ship every permutation of platform/Python version wheel needed, and there the source distribution is available. Though I think that happens less and less since the number of big packages with native dependencies is relatively limited.
But the underlying point is that with an option of compiling everything “from source” available as an official thing from the project, downstream distributors do not have to do things like, say, confirm that the project’s vendored compiled binary is in fact compiled from the source being pointed at.
Install-time scripting is less of an issue in this thought process (after all, import-time scripting is a thing that can totally happen!). It should feel a bit obvious that a bunch of source files is easier to look through to figure out issues rather than “oh this part is provided by this pre-built binary”, at least it does to me.
I’m not arguing against binary distributions, just think that if you have only the binary distribution suddenly it’s a lot harder to answer a lot of questions.
But the underlying point is that with an option of compiling everything “from source” available as an official thing from the project, downstream distributors do not have to do things like, say, confirm that the project’s vendored compiled binary is in fact compiled from the source being pointed at.
As far as I’m aware, it was possible to build serde “from source” as a repackager. It did not produce a binary byte-for-byte identical to the one being shipped first-party, but as I understand it producing a byte-for-byte identical binary is not something Rust’s current tooling would have supported anyway. In other words, the only sense in which “binary only” was true was for installing from crates.io.
So any arguments predicated on “you have only the binary distribution” don’t hold up.
Hmm, I felt like I read repackagers specifically say that the binary was a problem (I think it was more the fact that standard tooling didn’t allow for both worlds to exist). But this is all a bit moot anyways
I don’t know of anyone who actually wants the sdists from PyPI.
It’s a useful fallback when there are no precompiled binaries available for your specific OS/Arch/Python version combination. For example when pip installing from a ARM Mac there are still cases where precompiled binaries are not available, there were a lot more closer to the M1 release.
When I say I don’t know of anyone who wants the sdist, read as “I don’t know anyone who, if a wheel were available for their target platform, would then proceed to explicitly choose an sdist over that wheel”.
Also, not for nothing, most of the discussion has just been assuming that “binary blob = inherent automatic security vulnerability” without really describing just what the alleged vulnerability is. When one person asserts existence of a thing (such as a security vulnerability) and another person doubts that existence, the burden of proof is on the person asserting existence, but it’s also perfectly valid for the doubter to point to prominent examples of use of binary blobs which have not been exploited despite widespread deployment and use, as evidence in favor of “not an inherent automatic security vulnerability”
Yeah, this dynamic has been infuriating. In what threat model is downloading source code from the internet and executing it different from downloading compiled code from the internet and executing it? The threat is the “from the internet” part, which you can address by:
Hash-pinning the artifacts, or
Copying them to a local repository (and locking down internet access).
Anyone with concerns about this serde change should already be doing one or both of these things, which also happen to make builds faster and more reliable (convenient!).
Yeah, hashed/pinned dependency trees have been around forever in other languages, along with tooling to automate their creation and maintenance. It doesn’t matter at that point whether the artifact is a precompiled binary, because you know it’s the artifact you expected to get (and have hopefully pre-vetted).
Downloading source code from the internet gives you the possibility to audit it, downloading a binary makes this nearly impossible without whipping out a disassembler and hoping that if it is malicious, they haven’t done anything to obfuscate that in the compiled binary. There is a “these languages are turing complete, therefore they are equivalent” argument to be made, but I’d rather read Rust than assembly to understand behaviour.
The point is that if there were some easy way to exploit the mere use of precompiled binaries, the wide use of precompiled binaries in other languages would have been widely exploited already. Therefore it is much less likely that the mere presence of a precompiled binary in a package is inherently a security vulnerability.
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I’m confused about this point. Is anyone going to fix crates.io so this can’t happen again?
Assuming that this is a security problem (which I’m not interested in arguing about), it seems like the vulnerability is in the packaging infrastructure, and serde just happened to exploit that vulnerability for a benign purpose. It doesn’t go away just because serde decides to stop exploiting it.
I don’t think it’s an easy problem to fix: ultimately, package registry is just a storage for files, and you can’t control what users put there.
There’s an issue open about sanitizing permission bits of the downloaded files (which feels like a good thing to do irrespective of security), but that’s going to be a minor speed bump at most, as you can always just copy the file over with the executable bit.
A proper fix here would be fully sandboxed builds, but:
POSIX doesn’t have “make a sandbox” API, so implementing isolation in a nice way is hard.
There’s a bunch of implementation work needed to allow use-cases that currently escape the sandbox (wasm proc macros and metabuild).
I’m sure it’ll be fine, who’s ever heard of a security issue in a codec anyway?
Who’s ever heard of a security issue caused by a precompiled binary shipping in a dependency? Like, maybe it’s happened a few times? I can think of one incident where a binary was doing analytics, not outright malware, but that’s it.
I’m confused at the idea that if we narrow the scope to “a precompiled binary dependency” we somehow invalidate the risk. Since apparently “curl $FOO > sh” is a perfectly cromulent way to install things these days among some communities, in my world (30+ year infosec wonk) we really don’t get to split hairs over ‘binary v. source’ or even ‘target v. dependency’.
I’m not sure I get your point. You brought up codec vulns, which are irrelevant to the binary vs source discussion. I brought that back to the actual threat, which is an attack that requires a precompiled binary vs source code. I’ve only seen (in my admittedly only 10 Years of infosec work) such an attack one time, and it was hardly an attack and instead just shady monetization.
This is the first comment I’ve made in this thread, so I didn’t bring up codecs. Sorry if that impacts your downplaying supply chain attacks, something I actually was commenting on.
Ah, then forget what I said about “you” saying that. I didn’t check who had commented initially.
As for downplaying supply chain attacks, not at all. I consider them to be a massive problem and I’ve actively advocated for sandboxed build processes, having even spoken with rustc devs about the topic.
What I’m downplaying is the made up issue that a compiled binary is significantly different from source code for the threat of “malicious dependency”.
So not only do you not pay attention enough to see who said what, you knee-jerk responded without paying attention to what I did say. Maybe in another 10 years…
Because I can curl $FOO > foo.sh; vi foo.sh then can choose to chmod +x foo.sh; ./foo.sh. I can’t do that with an arbitrary binary from the internet without whipping out Ghidra and hoping my RE skills are good enough to spot malicious code. I might also miss it in some downloaded Rust or shell code, but the chances are significantly lower than in the binary. Particularly when the attempt from people in the original issue thread to reproduce the binary failed, so no one knows what’s in it.
No one, other than these widely publicisedinstances in NPM, as well as PyPi and Ruby, as pointed out in the original github issue. I guess each language community needs to rediscover basic security issues on their own, long live NIH.
I hadn’t dived into them, they were brought up in the original thread, and shipping binaries in those languages (other than python with wheels) is not really common (but would be equally problematic). But point taken, shouldn’t trust sources without verifying them (how meta).
But the question here is “Does a binary make a difference vs source code?” and if you’re saying “well history shows us that attackers like binaries more” and then history does not show that, you can see my issue right?
But what’s more, even if attackers did use binaries more, would we care? Maybe, but it depends on why. If it’s because binaries are so radically unauditable, and source code is so vigilitantly audited, ok sure. But I’m realllly doubtful that that would be the reason.
I love this. I’m currently working on something similar and this came at the perfect time. Do you have any more sources on alignment? I’m not super informed but I’m curious since most other sources seem to make a big deal out of alignment (e.g. here).
Another reason to align the heap would be to support >1GB sizes with the same relative pointer, e.g. you could do 4GB with 4 byte alignment. But then I guess it wouldn’t be small anymore.
I do not have any other references on alignment; if you find any, please post them to lobste.rs!
Adding alignment does expand the reach of pointers. It would make even-smaller pointers more practical: for instance you could access 16MB instead of 4MB with 24-bit values (assuming 1 bit is still used for a tag.)
It’s probably worth noting that this is a problem only for large projects. Yes, a CI build of LLVM for us takes 20 minutes (16 core Azure VM), but LLVM is one of the largest C++ codebases that I work on and even then incremental builds are often a few seconds (linking debug builds takes longer because of the vast quantity of debug info).
There’s always going to be a trade off between build speed and performance: moving work from run time to compile time is always going to slow down compile speed. The big problem with C++ builds is the amount of duplicated work. Templates are instantiated at use point and that instantiations is not shared between compilation units. I wouldn’t be surprised if around 20% of compilation units in LLVM have duplicate copies of the small vector class instantiated on an LLVM value. Most of these are inlined and then all except one of the remainder is thrown away in the final link. The compilation database approach and modules can reduce this by caching the instantiations and sharing them across compilation units. This is something that a language with a more overt module system can definitely improve because dependency tracking is simpler. If a generic collection and a type over which it’s instantiated are in separate files then it’s trivial to tell (with a useful over approximation) when the instantiations needs to be regenerated: when either of the files has changed. With C++, you need to check that no other tokens in the source file have altered the shape of the AST. C++ modules help a lot here, by providing includable units that generate a consistent AST independent of their use order. Rust has had a similar abstraction from the start (as have most non-C languages).
I think that’s true for C++ but I use Rust for small projects and I definitely find compile times to still be a problem. Rust has completely replaced my usage of C++ but is strictly worse in terms of compilation time for my projects. Even incremental builds can be really slow because if you depend on a lot of crates (which is incredibly easy to do transitively), linking time will dominate. mold helps a lot here.
I think this is always going to be a problem regardless of the compiler or optimizations because it’s just too tempting to use advanced macros / type level logic or pull in too many dependencies. In C++ I would just avoid templates and other slow to compile patterns, but in Rust these utilities are too powerful and convenient to ignore. Most of the Rust versions of a given library will try to do something cool with types and macros that’s usually great but you end up paying for it in compile times. I hope the Rust community starts to see compile times as a more pressing concern and designing crates with that in mind.
I use Rust for small projects and I definitely find compile times to still be a problem.
This is my experience. Unless we’re talking about ultra tiny workspaces, any small to medium sized project hurts, especially compared to build times with other tool chains like the Go compiler.
I think zig does offer more safety than the author lets on. It definitely offers flexibility, a good bootstrap story, and will work to make memory mistakes visible to the programmer.
Rust is better for critical stuff than C. Zig seems nicer to use, to me, than both.
I think Zig’s approach to memory safety is not reasonable for any new system language. Temporal memory errors is about the single biggest problem large code bases, and Zig is the only new language that does not make any real attempt to fix this. We’ve got decades of evidence demonstrating that manual memory management is fundamentally unsafe as it requires no mistakes by a developer, but is also infeasible to analyze without language level support for tracking object life time. There are numerous ways a language can do this, from zero runtime overhaad (rust w/borrow check) through to a full tracing GC - though I believe the latter simply is not feasible in kernel level code so seems impractical for a general “systems language”.
I know whenever I bring up temporal safety someone chimes in with references to fuzzing and debug allocators, but we know that those are not sufficient: C/C++ projects have access to fuzzers and debug allocators, and the big/security sensitive ones aggressively fuzz, and use aggressive debug allocators while doing so, and those do not magically remove all security bugs. In fact a bunch of things the “debug” allocators do in Zig are already the default behaviour of the macOS system allocator for example, and the various webkit and blink allocators are similarly aggressive in release builds.
I think it depends on your priorities and values and what you’re trying to do. Rust does not guarantee memory safety, only safe Rust does. If what you’re doing requires you to use a lot of unsafe Rust, that’s not necessarily better than something like Zig or even plain C. In fact, writing unsafe Rust is more difficult than writing C or Zig, and the ergonomics can be terrible. See for why wlroots-rs was abandoned for example.
From my experience, I’ve found Rust to do great when no unsafe code or complex lifetimes are involved, such as writing a web server. It also works great when you can design the memory model with lifetimes and the borrow checker in mind. When you can’t, or you have to deal with an existing memory model (such as when interacting with C), it can fare very poorly in terms of ergonomics and be more difficult to maintain.
Rust does not guarantee memory safety, only safe Rust does.
And that’s great, because people are writing very small amounts of code in unsafe Rust.
I’ve also implemented a programming language VM in Rust and found that you can usually define good abstractions around small amounts of unsafe code. It isn’t always easy (and wlroots legitimately seems like a hard case, but I don’t have enough domain expertise to confirm) but given how rarely you need to do it it seems manageable.
But I think focusing on a few hard cases is missing the forest for the trees. If you can write the vast majority of code in a pleasant, safe and convenient language it very quickly makes the small part that you do need to be unsafe or fight lifetimes worth it.
I would love to see a language that does even better and handles more cases. But until that comes along I would strongly prefer to use Rust which helps me write correct code than the alternatives just because in a few sections it doesn’t help me.
But I think focusing on a few hard cases is missing the forest for the trees. If you can write the vast majority of code in a pleasant, safe and convenient language it very quickly makes the small part that you do need to be unsafe or fight lifetimes worth it.
That’s fine, but we don’t need to have only one be all end all programming language. We can have different options for different situations, and that’s what Zig and other languages provide in contrast to Rust.
I really enjoy using Zig more than using Rust. I have to work with existing C codebases, and Rust seems to hold C code at arms length, where Zig will let me integrate more fluidly. Both Rust and Zig seem like great replacements for C, but Zig seems to play more nicely with existing software.
I enjoy using Zig a lot more than Rust as well. While Rust has many fancy features, I am frequently fighting with them. And I’m not talking about the borrow checker, but rather limitations in the type system, tooling, or features being incomplete.
But all that aside I think where I would use Zig in a commercial setting is limited, only because it is too easy to introduce memory use issues, and I don’t think the language is doing enough, at least yet, to prevent them.
This. I’ve been using Rust way more than Zig, and even used Rust professionally, for about 3 years, and I have to say Zig really gets a lot right. Zig’s memory safety system may not be on-par with Rust but that’s because it opts to make unsafe code easier to write without bugs. Traversing the AST (which is available via std.zig.Ast), you can get the same borrow checker system in place with Zig 🙂 Now that Zig is maturing with the bootstrapping issue figured out, I bet we’ll see this sooner than later, as it’s a real need for some.
Zig is builder friendly. You ever try C interop with Rust? Rust is enterprise friendly.
But lifetimes in rust are based on the control-flow graph not on the AST. Lexical lifetimes got replaced by non-lexical lifetimes back in 2018. Also there are lifetimes annotations, how would you get those?
I know neither zig nor rust, and do not plan to, but: usually, a control-flow graph (along with more complicated structures besides) is built from an ast; is there some reason you can not do this for zig?
It’s like the “draw two circles and then just draw the rest of the owl” meme.
You can make CFG from AST easily, but then liveness and escape analysis is the hard part that hasn’t been solved for half a century, and even Rust needs the crutch of restrictive and manual lifetime annotations.
Traversing the AST, you can get the same borrow checker system in place with Zig
Do you have more reading on this? Because my understanding is that Rust’s borrow checker isn’t possible to infer all lifetimes from the AST alone. Hence it’s infamous lifetime annotations ('a). While Rust inference is generally pretty good, there are definitely plenty of scenarios I’ve run into where the inferred lifetimes are wrong, or ambiguous in which both cases require the additional syntax.
If you could prove the safety (or lack thereof) of mutable aliasing entirely through syntactic analysis, the issues we have with C would be a lot less glaring. Having a parser in the standard library is incredibly cool, as is comptime, but you still have to build a static verifier for borrow checking your language/runtime of choice.
I do think there’s a good opportunity for conventions and libraries to emerge in the Zig ecosystem that make writing code that’s “safe by default” a lot easier. You could absolutely force allocations and lookups in your code to use something like a slotmap to prevent race-y mutation from happening behind the scenes.
Unfortunately that confidence doesn’t extend to 3rd-party libraries. Trying to transparently interface with arbitrary C code (which may in turn drag in embedded assembler, Fortran, or who knows what) means bringing all of the memory safety issues of C directly into your code base. See: literal decades of issues in image format decoding libraries, HTML parsers, etc. that get reused in nominally “memory-safe” languages.
All of which is precisely why Rust does keep C/C++ at arm’s length, no matter how appealing trivial interop would be. Mozilla, MS, and Google/Android have custodianship over hundreds of millions of lines of C++ code, and yet they also understand and want the safety benefits Rust provides enough to deal with the more complicated and toilsome integration story vs. something like Zig or D.
Right right, I made a big mistake to say “same”. Others mention annotations - those could be created in comments or in novel ways Im sure ! We’re starting to see things like CheckedC so yea, my thoughts still stand…
But there isn’t enough information in the AST without annotations. Else you wouldn’t need mut and lifetime annotations in rust. The reason for those annotations is that the compiler couldn’t proof that the safe rust code doesn’t introduce new memory bugs (safe code can still trigger memory bugs introduced by unsafe code) without those annotations.
The AST describes the whole program, that doesn’t mean that every property that is true for your code can be automatically inferred by an general algorithm. A well known example would be if your program computes in a finite amount of time.
Zig’s comptime code generation is an example of this approach. From what I’ve seen it works pretty well in practice. It has one big property that I don’t like though: whether or not your code typechecks is tested when the macro/template is expanded, not when it is defined. This means that you can write a comptime function that generates invalid code in some cases, and you never find out about it as long as those cases never happen.
I think it’s important to note that when you do find a case that generates invalid code, it is a comptime error. So, you can guarantee all usages of your function are valid if the project compiles.
This can also be applied to platform-specific blocks:
if (on linux) {
// linux specific stuff
} else if (on windows) {
// windows specific stuff
} else {
@compileError("unsupportedPlatform");
}
If you were to call this function on an unsupported platform, it would be a compile error. If you don’t, you can compile the project fine. I think this property of the language is really cool, and I think the Zig standard library uses it for a few different things like skipping tests on certain platforms (others compilers embed this as metadata in the file).
For small macros/templates this is perfectly fine because it’s easy to make sure all the code is tested. But when you start using it for large and complicated things that may have unused code paths lurking in unexpected places, it starts to worry me. Maybe it works fine in practice most of the time, but the idea leaves a bad taste in my mouth.
It did for me at first - why on earth wouldn’t you want all of your code checked?? - but I think I’m being converted. Partial evaluation like this just feels so powerful. I’ve even ended up emulating these ideas in Java for work.
C++ templates have the same property that they are checked at compile-time but on use-sites, and I think this is widely understood to have turned into a usability hell, which is basically the reason for the work on concepts. For example, it is easy for developers of comptime logic in libraries to miss errors that happen on their clients, or to end up with code that is effectively not checked/tested inside the library.
I’m not saying that Zig comptime has the exact same downsides as C++ templates (it has the benefit of being a properly designed compilation-level language, rather than something that grew organically in different directions than the base language), but I also suspect that this class of issues is a problem at large and I share the blog post’s reservations about running wild with the idea.
On the other hand, it is true that the benefits in terms of reduced complexity of the language (compared to a much richer time system) are significant. Language design is a balancing act.
Uhu, with C++/Zig style generics, it’s very hard for the user (or the author) to state precisely what the API of an utility is, which should make SemVer significantly harder in practice.
The second problem is tooling: when you have T: type, there isn’t much an IDE can correctly do with completions, goto definition, etc. Some heuristics are possible (eg, looking at the example instantiations), but that’s hard to implement (precise solutions are simpler).
Regarding tooling: Smalltalk/Squeak/Pharo people would say that you don’t need static analysis to have good tooling, as long as you rephrase the tooling to be about exploring the state of the program as it runs, rather than without running it. I could see how this might possibly work with Zig – collect global traces of successful generic instantiations (for the testsuite in the project, or for your whole set of installed packages, or over the whole history of your past usage of the code…) and show their shared structure to the user.
collect global traces of successful generic instantiations (for the testsuite in the project, or for your whole set of installed packages, or over the whole history of your past usage of the code…) and show their shared structure to the user.
This is something that we already did with the old implementation of autodoc and that plan to eventually also support in the new version as something that you should expect to be able to rely on when browsing documentation for a package.
One advantage Zig has over plain C++ templates is that you can do arbitrary comptime checks and emit clean, precise error messages. For example, instead of just blowing up when you instantiate with a type that doesn’t have a given method, you can check for it and fail compilation with type foo must have bar(x: i32) method. You can even implement something akin to concepts in userspace.
Yeah but I have been burned enough times by pulling out code that compiled 6 months ago to discover, oops, it no longer builds. Even in Rust. There are few things more frustrating. “I need to fix this bug that I just found in an old prod system, it’s a 15 minute fix and and oops now I get to spend three hours trying to figure out which dependency three levels down in my dep tree didn’t follow semver and broke its API.” Or which file doesn’t build with a newer compiler’s interpretation of C++17, or which Magical Build Flag didn’t get saved in the build docs, or why the new binary needs to load libfuckyou.so.0.7.3 but the old one doesn’t, or…
So if possible I would prefer to not add yet more ways of scattering around hidden build landmines in large and complex programs. But as I said, I need to play with it more and get a better feel for how it actually works out in practice.
My first thought was the double 0 problem you usually get from ones complement, but this actually reserves a sentinel “Nan” value.
The problem there is that part of what makes twos complement so great is that the representation also makes basic addition, subtraction, shifts, etc very efficient to implement.
The other problem here is the gratuitous and unnecessary use of undefined behavior. UB is bad, and is meant to exist for things that cannot be specified, e.g using an object after it has been freed, accessing memory through the wrong type, etc. obviously the C and C++ committees decided to abandon that and create numerous security vulnerabilities over the years. If you don’t want to constrain the implementation the correct course of action is unspecified behavior. That is the implementation can choose what to do, but it must always do that, the compiler can optimize according to that but it can’t go “that’s UB so I’ll just remove it”, which I also think is a bogus interpretation of what UB means but that’s what we have.
The desire to say “NaN is UB” means that you can’t test for NaN, because by definition it isn’t defined so the compiler can remove any NaN checks you insert. You might say “we’ll add a built in with defined behaviour”, but that doesn’t help because you gave the compiler the option to remove intermediate NaNs. Eg if(__builtin_isnan(0/0)) doesn’t trip because 0/0 can’t produce a NaN (that would be UB), so therefore the value passed to isnan can’t be NaN, so therefore isnan is false. This is obviously a dumb example, but it’s fundamentally the behavior you see for anything UB. Imagine isnan(x), which looks reasonable, but if x got it’s value from any arithmetic in the scope of the optimizer (which can include inlining) then isnan is always false.
It’s not giving up the ease of two’s complement addition core operation (ie full adders) as I understood it (after re-reading, it’s late, I on first skim thought they were advocating sign-magnitude) but rather to reserve ~0 as a not-numerically-valid NaN. However efficiency would still be lost of course in that the implementation would need to have checks for NaNness of input, and the overflow/underflow detection would be more complex as well.
But this only matters if we decide to build processors this way to support the representation/interpretation, otherwise it’s going to be software layers above.
I made a mistake in my previous post, it’s not ~0 that is the reserved value, ~0 is still the representation of -1. It is the interpretation of {msb=1, others=0} that is changed to INT_NAN rather than a valid number, but my interpretation is that beyond that, the representation remains two’s complement-like (4 bit example):
The problem isn’t the NAN value being tested it is the preceding arithmetic, because of what UB means. Take
x = y / z;
if (__builtin_isnan(x)) doThingA()
else doThingB()
Because arithmetic producing INT_NAN is UB, then the compiler is free to assume that no arithmetic produce it, which gives the compiler the ability to convert the above into:
x = y / z
doThingB()
Because the product “cannot be NAN” the isnans can be removed.
I think the idea is that you’d only ever assign INT_NAN when you want to set the sentinel value.
However, unlike floating point arithmetic where e.g. 0/0 can result in NaN, under my model, no arithmetic operation on integers can result in INT_NAN as overflow is undefined behavior.
You wouldn’t be checking for INT_NAN after arithmetic: you’d only check before, to see if you have an empty optional. It’s not true that the compiler can remove every NaN check, only the ones immediately following arithmetic. And those don’t make sense in this model since there’s nothing meaningful you could do at that point. If you’re representing optional<int>, there’s no reason to check if your optional is suddenly empty after an arithmetic operation. If you want to prevent overflow from resulting in a NaN, you can insert a precondition before doing any arithmetic.
You wouldn’t be checking for INT_NAN after arithmetic: you’d only check before, to see if you have an empty optional.
No - that handles a parameter passed to you, it does not handle general arithmetic, unless you pre-check everything, e.g:
x = a() + b() / c()
would become
_a = a()
if (isnan(_a)) ???
_b = b()
if (isnan(_b)) ???
_c = c()
if (isnan(_c)) ???
if (_c == 0) ???
x = _a + _b / _c
This is assuming of course the optional case, but this runs into the standard footgun that comes from the frankly stupid at this point belief that having arithmetic edge cases be undefined rather than unspecified, e.g.
In a world where arithmetic on nan is undefined behaviour, as the author specified, the isnan() and printf can be removed. Because we’ve said “arithmetic on nan is undefined behaviour” rather than “unspecified”, the compiler can do the following logic:
1. "1 * _a" is UB if _a is nan. Therefore _a cannot be nan.
2. "1 * _a" is "_a" by standard arithmetic
3. due to 1, isnan(_a) is false
4. if (isnan(_a)) will never be taken due to 3
5. no one uses _a so we can flatten
So we get the above code lowered to
return a()
This is how compilers reason about UB. It’s not “I have proved this path leads to UB so that’s an error”, it’s instead the much worse “I have proved that this path leads to UB, therefore this path cannot happen”.
Yes, this logic has led to security bugs, in numerous projects, and yet language committees continue to assign “UB” to specifiable behaviour.
A very easy way to create serialisers/deserialisers in JS is to write functions which create or consume a plain object representation, and then use JSON to create a string representation. So there’s no better reason than that it’s just easier. The alternatives seem to be to add a big serialisation library and complicate the build process with code generation from schema files , or write an ad-hoc schema-style format where the “schema” is represented as the state control flow in the code. Making a more efficient schemaless format with the JSON object model just seemed easier.
Oddly, the author acknowledges this, but then doesn’t really explore the implications:
Yet it is not a force of nature; in reality it is a system deliberately designed by humans to advance a ruthless end, without admitting to it. Behind the door lies the human controlling it. In this way the machine becomes an abstraction and a disguise for human ruthlessness.
…which is disappointing, since that opens up an opportunity to talk about how computing systems could be designed to be more humane, which is immediately squandered in favor of advocating for crypto-“anarchism” whose success is “beyond dispute” and whose ruthlessness is “clearly” good.
Which is to say, I agree with you. Furthermore, it’s not just computing itself which obscures the underlying human ruthlessness or integrity (although in a mundane, banality-of-evil sense, it definitely does). But, at a deeper level, what obscures the human element is the belief that computing is inherently anything.
Absolutely. There are plenty of examples of humane systems:
Automated doors which will detect an obstruction and either stop what they’re doing or default open.
Highly visible, trivially operated emergency off switches.
Free and open source software which allows end users to inspect in massive detail what their system is doing, and to correct anything they consider a bug.
Telephone services which immediately go straight to a human. They still exist, and more companies should brag about them.
Rounded corners, on both physical and digital products.
These all exist because someone looked at the existing MVP, decided it wasn’t good enough, and designed (or enforced) something more humane. All of us, as users and creators, “simply” need to encourage the proliferation of such systems.
I’d agree with you that exploring the human angle is key to fixing this problem. This post is tagged “philosophy”, but I’d as soon tag it “religion” (if such a tag existed and if Lobsters was the place for such debates) because it and these comments are really speaking about our view of humans and morality. Where the author sees “ruthlessness” from a machine following its programming in the subway, others may see life giving opportunity in the biomedical field, for example.
What computers, and machines in general, therefore possess, is this: the power of unlimited, total ruthlessness, ruthlessness greater than even the most warped and terrible human is capable of, or can even comprehend.
I don’t believe this can be true because a human, potentially a warped and terrible one, had to intentionally create the machine. In other words, if someone made a machine to destroy humanity, they were doing so intentionally and they are worse than the machine itself because they understand what humanity is, the machine does not (nod to the sentience debate).
In this regard one wonders if there has ever been a human who truly desired infinite integrity in full and prescient knowledge of its consequences.
This hypothetical question is the perfect example that this is really a religious debate, not just an ethics in engineering one. 1-2 billion people in the world would answer that question with “Yes, His name is Jesus.” We can’t consider integrity in machines without dealing with integrity in the humans that create and use them and we can’t deal with that without first knowing where we each come from in our beliefs. I sincerely am not posting this to start a religious flame war, merely point out that machines aren’t what is in question in this post.
This post is tagged “philosophy”, but I’d as soon tag it “religion” (if such a tag existed and if Lobsters was the place for such debates) because it and these comments are really speaking about our view of humans and morality. Where the author sees “ruthlessness” from a machine following its programming in the subway, others may see life giving opportunity in the biomedical field, for example.
This hypothetical question is the perfect example that this is really a religious debate
I just want to say that these types of questions and discussions are definitely philosophical, even if people turn to religion for the answer.
What surrprised me about Tainter’s analysis (and I haven’t read his entire book yet) is that he sees complexity as a method by which societies gain efficiency. This is very different from the way software developers talk about complexity (as ‘bloat’, ‘baggage’, ‘legacy’, ‘complication’), and made his perspective seem particularly fresh.
I don’t mean to sound dismissive – Tainter’s works are very well documented, and he makes a lot of valid points – but it’s worth keeping in mind that grand models of history have made for extremely attractive pop history books, but really poor explanations of historical phenomena. Tainter’s Collapse of Complex Societies, while obviously based on a completely different theory (and one with far less odious consequences in the real world) is based on the same kind of scientific thinking that brought us dialectical materialism.
His explanation of the fall of the evolution and the eventual fall of the Roman Empire makes a number of valid points about the Empire’s economy and about some of the economic interests behind the Empire’s expansion, no doubt. However, explaining even the expansion – let alone the fall! – of the Roman Empire strictly in terms of energy requirements is about as correct as explaining it in terms of class struggle.
Yes, some particular military expeditions were specifically motivated by the desire to get more grain or more cows. But many weren’t – in fact, some of the greatest Roman wars, like (some of) the Roman-Parthian wars, were not driven specifically by Roman desire to get more grains or cows. Furthermore, periods of rampant, unsustainably rising military and infrastructure upkeep costs were not associated only with expansionism, but also with mounting outside pressure (ironically, sometimes because the energy per capita on the other side of the Roman border really sucked, and the Huns made it worse on everyone). The increase of cost and decrease in efficiency, too, are not a matter of half-rational historical determinism – they had economic as well as cultural and social causes that rationalising things in terms of energy not only misses, but distorts to the point of uselessness. The breakup of the Empire was itself a very complex social, cultural and military story which is really not something that can be described simply in terms of the dissolution of a central authority.
That’s also where this mismatch between “bloat” and “features” originates. Describing program features simply in terms of complexity is a very reductionist model, which accounts only for the difficulty of writing and maintaining it, not for its usefulness, nor for the commercial environment in which it operates and the underlying market forces. Things are a lot more nuanced than “complexity = good at first, then bad”: critical features gradually become unneeded (see Xterm’s many emulation modes, for example), markets develop in different ways and company interests align with them differently (see Microsoft’s transition from selling operating systems and office programs to renting cloud servers) and so on.
However, explaining even the expansion – let alone the fall! – of the Roman Empire strictly in terms of energy requirements is about as correct as explaining it in terms of class struggle.
Of course. I’m long past the age where I expect anyone to come up with a single, snappy explanation for hundreds of years of human history.
But all models are wrong, only some are useful. Especially in our practice, where we often feel overwhelmed by complexity despite everyone’s best efforts, I think it’s useful to have a theory about the origins and causes of complexity, even if only for emotional comfort.
Especially in our practice, where we often feel overwhelmed by complexity despite everyone’s best efforts, I think it’s useful to have a theory about the origins and causes of complexity, even if only for emotional comfort.
Indeed! The issue I take with “grand models” like Tainter’s and the way they are applied in grand works like Collapse of Complex Societies is that they are ambitiously applied to long, grand processes across the globe without an exploration of the limits (and assumptions) of the model.
To draw an analogy with our field: IMHO the Collapse of… is a bit like taking Turing’s machine as a model and applying it to reason about modern computers, without noting the differences between modern computers and Turing machines. If you cling to it hard enough, you can hand-wave every observed performance bottleneck in terms of the inherent inefficiency of a computer reading instructions off a paper tape, even though what’s actually happening is cache misses and hard drives getting thrashed by swapping. We don’t fall into this fallacy because we understand the limits of Turing’s model – in fact, Turing himself explicitly mentioned many (most?) of them, even though he had very little prior art in terms of alternative implementations, and explicitly formulated his model to apply only to some specific aspects of computation.
Like many scholars at the intersections of economics and history in his generation, Tainter doesn’t explore the limits of his model too much. He came up with a model that explains society-level processes in terms of energy output per capita and upkeep cost and, without noting where these processes are indeed determined solely (or primarily) by energy output per capita and upkeep post, he proceeded to apply it to pretty much all of history. If you cling to this model hard enough you can obviously explain anything with it – the model is explicitly universal – even things that have nothing to do with energy output per capita or upkeep cost.
In this regard (and I’m parroting Walter Benjamin’s take on historical materialism here) these models are quasi-religious and are very much like a mechanical Turk. From the outside they look like history masterfully explaining things, but if you peek inside, you’ll find our good ol’ friend theology, staunchly applying dogma (in this case, the universal laws of complexity, energy output per capita and upkeep post) to any problem you throw its way.
Without an explicit understanding of their limits, even mathematical models in exact sciences are largely useless – in fact, a big part of early design work is figuring out what models apply. Descriptive models in humanistic disciplines are no exception. If you put your mind to it, you can probably explain every Cold War decision in terms of Vedic ethics or the I Ching, but that’s largely a testament to one’s creativity, not to their usefulness.
Furthermore, periods of rampant, unsustainably rising military and infrastructure upkeep costs were not associated only with expansionism, but also with mounting outside pressure (ironically, sometimes because the energy per capita on the other side of the Roman border really sucked, and the Huns made it worse on everyone).
Not to mention all the periods of rampant rising military costs due to civil war. Those aren’t wars about getting more energy!
Tainter’s Collapse of Complex Societies, while obviously based on a completely different theory (and one with far less odious consequences in the real world) is based on the same kind of scientific thinking that brought us dialectical materialism.
Sure. This is all about a framing of events that happened; it’s not predictive, as much as it is thought-provoking.
Thought-provoking, grand philosophy was certainly a part of philosophy but became especially popular (some argue that it was Nathaniel Bacon who really brought forth the idea of predicting progress) during the Industrial Era with the rise of what is known as the modernist movement. Modernist theories often differed but frequently shared a few characteristics such as grand narratives of history and progress, definite ideas of the self, a strong belief in progress, a belief that order was superior to chaos, and often structuralist philosophies. Modernism had a strong belief that everything could be measured, modeled, categorized, and predicted. It was an understandable byproduct of a society rigorously analyzing their surroundings for the first time.
Modernism flourished in a lot of fields in the late 19th early 20th century. This was the era that brought political philosophies like the Great Society in the US, the US New Deal, the eugenics movement, biological determinism, the League of Nations, and other grand social and political engineering ideas. It was embodied in the Newtonian physics of the day and was even used to explain social order in colonizing imperialist nation-states. Marx’s dialectical materialism and much of Hegel’s materialism was steeped in this modernist tradition.
In the late 20th century, modernism fell into a crisis. Theories of progress weren’t bearing fruit. Grand visions of the future, such as Marx’s dialectical materialism, diverged significantly from actual lived history and frequently resulted in a magnitude of horrors. This experience was repeated by eugenics, social determinism, and fascist movements. Planck and Einstein challenged the neat Newtonian order that had previously been conceived. Gödel’s Incompleteness Theorem showed us that there are statements we cannot evaluate the validity of. Moreover many social sciences that bought into modernist ideas like anthropology, history, and urban planning were having trouble making progress that agreed with the grand modernist ideas that guided their work. Science was running into walls as to what was measurable and what wasn’t. It was in this crisis that postmodernism was born, when philosophers began challenging everything from whether progress and order were actually good things to whether humans could ever come to mutual understanding at all.
Since then, philosophy has mostly abandoned the concept of modeling and left that to science. While grand, evocative theories are having a bit of a renaissance in the public right now, philosophers continue to be “stuck in the hole of postmodernism.” Philosophers have raised central questions about morality, truth, and knowledge that have to be answered before large, modernist philosophies gain hold again.
I don’t understand this, because my training has been to consider models (simplified ways of understanding the world) as only having any worth if they are predictive and testable i.e. allow us to predict how the whole works and what it does based on movements of the pieces.
Models with predictive values in history (among other similar fields of study, including, say, cultural anthropology) were very fashionable at one point. I’ve only mentioned dialectical materialism because it’s now practically universally recognized to have been not just a failure, but a really atrocious one, so it makes for a good insult, and it shares the same fallacy with energy economic models, so it’s a doubly good jab. But there was a time, as recent as the first half of the twentieth century, when people really thought they could discern “laws of history” and use them to predict the future to some degree.
Unfortunately, this has proven to be, at best, beyond the limits of human understanding and comprehension. This is especially difficult to do in the study of history, where sources are imperfect and have often been lost (case in point: there are countless books we know the Romans wrote because they’re mentioned or quoted by ancient authors, but we no longer have them). Our understanding of these things can change drastically with the discovery of new sources. The history of religion provides a good example, in the form of our understanding of Gnosticism, which was forever altered by the discovery of the Nag Hammadi library, to the point where many works published prior to this discovery and the dissemination of its text are barely of historical interest now.
That’s not to say that developing a theory of various historical phenomenons is useless, though. Even historical materialism, misguided as they were (especially in their more politicized formulations), were not without value. They forced an entire generation of historians to think more about things that they never really thought about before. It is certainly incorrect to explain everything in terms of class struggle, competition for resources and the means of production, and the steady march from primitive communism to the communist mode of production – but it is also true that competition for resources and the means of production were involved in some events and processes, and nobody gave much thought to that before the disciples of Marx and Engels.
This is true here as well (although I should add that, unlike most materialistic historians, Tainter is most certainly not an idiot, not a war criminal, and not high on anything – I think his works display an unhealthy attachment for historical determinism, but he most certainly doesn’t belong in the same gallery as Lenin and Mao). His model is reductionist to the point where you can readily apply much of the criticism of historical materialism to it as well (which is true of a lot of economic models if we’re being honest…). But it forced people to think of things in a new way. Energy economics is not something that you’re tempted to think about when considering pre-industrial societies, for example.
These models don’t really have predictive value and they probably can’t ever gain one. But they do have an exploratory value. They may not be able to tell you what will happen tomorrow, but they can help you think about what’s happening today in more ways than one, from more angles, and considering more factors, and possibly understand it better.
That’s something historians don’t do anymore. There was a period where people tried to predict the future development of history, and then the whole discipline gave up. It’s a bit like what we are witnessing in the Economics field: there are strong calls to stop attributing predictive value to macroeconomic models because after a certain scale, they are just over-fitting to existing patterns, and they fail miserably after a few years.
Well, history is not math, right? It’s a way of writing a story backed by a certain amount of evidence. You can use a historical model to make predictions, sure, but the act of prediction itself causes changes.
(OP here.) I totally agree, and this is something I didn’t explore in my essay. Tainter doesn’t see complexity as always a problem: at first, it brings benefits! That’s why people do it. But there are diminishing returns and maintenance costs that start to outstrip the marginal benefits.
Maybe one way this could apply to software: imagine I have a simple system, just a stateless input/output. I can add a caching layer in front, which could win a huge performance improvement. But now I have to think about cache invalidation, cache size, cache expiry, etc. Suddenly there are a lot more moving parts to understand and maintain in the future. And the next performance improvement will probably not be anywhere near as big, but it will require more work because you have to understand the existing system first.
In Tainter’s view, a society of subsistence farmers, where everyone grows their own crops, makes their own tools, teaches their own children, etc. is not very complex. Add a blacksmith (division of labour) to that society, and you gain efficiency, but introduce complexity.
I think the biggest benefit of this is that - if I’m not mistaken - the new UI will be compatible with Wayland (as far as I know, Fleet’s interface already is Wayland compatible). This could finally allow me to move away from X for good.
In addition to fractional scaling, it modernizes and simplifies the whole graphics stack in a way akin to something like Aero when whatever major version of windows did that. Features I appreciate are:
Much less tearing
HiDPI is much better, probably partly due to the fractional scaling
More secure (though I’m taking others at their word)
X development has slowed, so I’m less confident that code rot will not create stability or security issues
I think for you it wouldn’t make a whole lot of difference, unless you decide to run the Linux version of your JetBrains product using an X server to display the window (which I wouldn’t recommend considering there’s native Windows versions and their products seem to have excellent WSL support).
For a Linux user… it’s kind of a mixed bag. There’s some security benefits to Wayland, but mostly the reason I see this as a good thing is that most desktop environment seem to be moving X into maintenance mode, and doing all new interesting development exclusively on Wayland. Gesture support and fractional scaling come to mind.
Go’s and Zig’s defer are rather different beasts
Go runs defered statements at the end of the function, Zig at the end of scope.
ant to lock a mutex insife a loop? Can’t use Go defer for that..
destructors can’t take arguments or return values
While most destructions only release acquired resources, passing an argument to a defered call can be very useful in many cases
hidden code
all defer code is visible in the scope. Look for all lines starting with defer in the current scope and you have all the calls.
Looking for destructors means looking how drop is implemented for all the types in the scopes.
Go’s and Zig’s defer are rather different beasts Go runs defered statements at the end of the function, Zig at the end of scope. ant to lock a mutex insife a loop? Can’t use Go defer for that..
This distinction doesn’t really matter in a language with first-class lambdas. If you want to unlock a mutex at the end of a loop iteration with Go, create and call a lambda in the loop that uses defer internally.
destructors can’t take arguments or return values
But constructors can. If you implement a Defer class to use RAII, it takes a lambda in the constructor and calls it in the destructor.
hidden code all defer code is visible in the scope
I’m not sure I buy that argument, given that the code in defer is almost always calling another function. The code inside the constructor for the object whose cleanup you are defering is also not visible in the calling function.
hidden code all defer code is visible in the scope
I’m not sure I buy that argument, given that the code in defer is almost always calling another function. The code inside the constructor for the object whose cleanup you are defering is also not visible in the calling function.
The point is that as a reader of zig, you can look at the function and see all the code which can be executed. You can see the call and breakpoint that line. As a reader of c++, it’s a bit more convoluted to breakpoint on destructors.
This can work sometimes, but other times packing pointers in a struct just so you can drop it later is wasteful. This happens a lot with for example the Vulkan API where a lot of the vkDestroy* functions take multiple arguments. I’m a big fan of RAII but it’s not strictly better.
At least in C++, most of this all goes away after inlining. First the constructor and destructor are both inlined in the enclosing scope. This turns the capture of the arguments in the constructor into local assignments in a structure in the current stack frame. Then scalar replacement of aggregates runs and splits the structure into individual allocas in the first phase and then into SSA values in the second. At this point, the ‘captured’ values are just propagated directly into the code from the destructor.
If you want to unlock a mutex at the end of a loop iteration with Go, create and call a lambda in the loop that uses defer internally.
Note that Go uses function scope for defer. So this will actually acquire locks slowly then release them all at the end of function. This is very likely not what you want and can even risk deadlocks.
Is a lambda not a function in Go? I wouldn’t expect defer in a lambda to release the lock at the end of the enclosing scope, because what happens if the lambda outlives the function?
The first point is minor, and not really changing the overall picture of leaking by default.
Destruction with arguments is sometimes useful indeed, but there are workarounds. Sometimes you can take arguments when constructing the object. In the worst case you can require an explicit function call to drop with arguments (just like defer does), but still use the default drop to either catch bugs (log or panic when the right drop has been forgotten) or provide a sensible default, e.g. delete a temporary file if temp_file.keep() hasn’t been called.
Automatic drop code is indeed implicit and can’t be grepped for, but you have to consider the trade-off: a forgotten defer is also invisible and can’t be grepped for either. This is the change in default: by default there may be drop code you may not be aware of, instead of by default there may be a leak you may not be aware of.
destructors can’t take arguments or return values. While most destructions only release acquired resources, passing an argument to a deferred call can be very useful in many cases.
Yes, more than useful:
Zero-cost abstraction in terms of state: A deferred call doesn’t artificially require objects to contain all state needed by their destructors. State is generally bad, especially references, and especially long lived objects that secretly know about each other.
Dependencies are better when they are explicit: If one function needs to run before another, letting it show (in terms of what arguments they require) is a good thing: It makes wrong code look wrong (yes, destruction order is a common problem in C++) and prevents it from compiling if you have lifetimes like Rust.
Expressiveness: In the harsh reality we live in, destructors can fail.
I think the right solution is explicit destructors: Instead of the compiler inserting invisible destructor calls, the compiler fails if you don’t. This would be a natural extension to an explicit language like C – it would only add safety. Not only that: It fits well with defer too – syntactic sugar doesn’t matter, because it just solves the «wrong default» problem. But more than anything, I think it would shine in a language with lifetimes, like Rust, where long lived references are precisely what you don’t want to mess with.
I think there’s one more potential attack that’s missing: a vulnerability or misconfiguration in etcd (or wherever else you store your secrets) that allows reading without requiring root access or physical access to the machine. In that scenario, encrypted secrets can provide another layer of security assuming the decryption is done via a separate mechanism.
I also don’t think “most people aren’t using feature X of Vault” is that strong an argument. You can’t dismiss a tool by insisting people aren’t using it correctly.
Yeah the dismissiveness of vault is mainly just me ranting. Maybe should have been an aside or footnote because the argument doesn’t rely on people misusing Vault. A properly configured Vault instance is what I ultimately compared to plain Kubernetes Secrets.
I agree that Vault is a complicated beast (I used to manage Vault at a previous employer) but USP for Vault must be the dynamic secret with TTLs right?
So even if you could read the secret from the RAMdisk on the node/pod it would not be usable unless you timed it so that you read it exactly before the service read it but after the injector well injected it.
My understanding is that while Vault can perform automatic password rotation, it can’t e.g. configure Redis or MySQL to change the password automatically. You could build something that does that for every secret-consuming application, but now vault is relegated to being a random password generator and again could be replaced with plain kubernetes secrets, /dev/urandom, and a cronjob.
I think the real value from Vault is the policies, not just storage. If a deployment is not taking advantage of that, then yes it’s no better than etcd or anything else.
By forever maintaining two implementations of the compiler - one in C, one in Zig. This way you will always be able to bootstrap from source in three steps:
Use system C compiler to build C implementation from source. We call this stage1. stage1 is only capable of outputting C code.
Use stage1 to build the Zig implementation to .c code. Use system C compiler to build from this .c code. We call this stage2.
Use stage2 to build the Zig implementation again. The output is our final zig binary to ship to the user. At this point, if you build the Zig implementation again, you get back the same binary.
I’m curious, is there some reason you don’t instead write a backend for the Zig implementation of the compiler to output C code? That seems like it would be easier than maintaining an entirely separate compiler. What am I missing?
The above post says they wanted two separate compilers, one written in C and one in Zig. I’m wondering why they just have one compiler written in Zig that can also output C code as a target. Have it compile itself to C, zip up the C code, and now you have a bootstrap compiler that can build on any system with a C compiler.
In the above linked Zig Roadmap video, Andrew explains that their current plan is halfway between what you are saying and what was said above. They plan to have the Zig compiler output ‘ugly’ C, then they will manually clean up those C files and version control them, and as they add new features to the Zig source, they will port those features to the C codebase.
I just watched this talk and learned a bit more. It does seem like the plan is to use the C backend to compile the Zig compiler to C. What interests me though is there will be a manual cleanup process and then two separate codebases will be maintained. I’m curious why an auto-generated C compiler wouldn’t be good enough for bootstrapping without manual cleanup.
Generated source code usually isn’t considered to be acceptable from an auditing/chain of trust point of view. Don’t expect the C code generated by the Zig compiler’s C backend to be normal readable C, expect something closer to minified js in style but without the minification aspect. Downloading a tarball of such generated C source should be considered equivalent to downloading an opaque binary to start the bootstrapping process.
Being able to trust a compiler toolchain is extremely important from a security perspective, and the Zig project believes that this extra work is worth it.
It would work fine, but it wouldn’t be legitimate as a bootstrappable build because the build would rely on a big auto-generated artifact. An auto-generated artifact isn’t source code. The question is: what do you need to build Zig, other than source code?
It could be reasonable to write and maintain a relatively simple Zig interpreter that’s just good enough to run the Zig compiler, if the interpreter is written in a language that builds cleanly from C… like Lua, or JavaScript using Fabrice Bellard’s QuickJS.
The issue is not to be completely free of all bootstrap seeds. The issue is to avoid making new ones. C is the most widely accepted and practical bootstrap target. What do you think is a better alternative?
C isn’t necessarily a bad choice today, but I think it needs to be explicitly acknowledged in this kind of discussion. C isn’t better at being bootstrapped than Zig, many just happen to have chosen it in their seed.
A C compiler written in Zig or Rust to allow bootstrapping old code without encouraging new C code to be written could be a great project, for example.
The issue is building from maintained source code with a widely accepted bootstrapping base, like a C compiler.
The Zig plan is to compile the compiler to C using its own C backend, once, and then refactor that output into something to maintain as source code. This compiler would only need to have the C backend.
The Go compiler used to be written in C. Around 1.4 they switched to a Go compiler written in Go. If you were setting up an entirely new platform (and not use cross compiling), i believe the recommended steps are still get a C compiler working, build Go 1.4, then update from 1.4 to latest.
Theoretically. But from a practical point of view? Yes, there are systems like Redox (Rust), but in most cases the C compiler is an inevitable piece of the puzzle (the bootstrapping chain) when building an operating system. And in such cases, I would (when focused on simplicity) rather prefer a language that depends just on C (that I already have) instead of a sequence of previous versions of its own compilers. (and I say that as someone, who does most of his work in Java – which is terrible from the bootstrapping point-of-view)
However, I do not object much against the dependence on previous versions of your compiler. It is often the way to go, because you want to write your compiler in a higher language instead of some old-school C and because you create a language and you believe in its qualities, you use it also for writing the compiler. What I do not understand is why someone (not this particular case, I saw this pattern before many times) present the “self-hosted” as an advantage…
The self-hosted Zig compiler provides much faster compile times and is easier to hack, allowing language development to move forward. In theory the gains could be done in a different language, but some of the kind of optimizations used are exactly the kind of thing Zig is good at. See this talk for some examples: https://media.handmade-seattle.com/practical-data-oriented-design/.
It’s possible to have an “async” system for systems programming that isn’t colored. Zig solved this problem. In the case of zig, I can write a single function to be called from the Erlang virtual machine’s FFI either in an async fashion to use it’s the vm’s preemptibility, or I can call the same function to be run in its own thread (this is non-async), so you can explore the tradeoffs without having to write your function twice.
I really like what Andrew did with color-free async in Zig, but it’s not a panacea. There are limitations. You can’t use multiple different event loops (one blocking, one non-blocking) in the same binary, for example. Unless I’ve misunderstood something.
You absolutely can use multiple event loops in the same binary. It isn’t obvious how to do it cleanly, if you ask me, though, or I haven’t hit upon “the correct pattern” for it
I’m not convinced that solves a real language problem since the caller has to opt in to the desired behavior and the callee has to support both modes of operation. It doesn’t allow you to just call an async function from a sync context and have it automatically work. Zig allows you to put both versions in one function, but now you don’t know which async functions you can safely call from sync without reading the documentation or looking at the implementation. Is there something I’m missing here?
I love the simple and straightforward nature of the tool and its presentation. Even though it requires a bit of legwork, it’s very transparent and I have enough understanding and information to adapt it to my use case. Thanks for this!
I understand the rationale for including the original emoji (unicode wants to be a superset of existing character sets) but they should have been put in a code space reserved for backwards compatibility with bad ideas, not made such a big part of unicode.
At this point, there’s a strong argument for a new character set that is a subset of unicode that removes all of the things that are not text. We already have mechanisms for embedding images in text. Even in the ‘90s, instant messaging systems were able to avoid sending common images by having a pre-defined set of pictures that they referenced with short identifiers. This was a solved problem before Unicode got involved and it’s made text processing an increasingly complicated mess, shoving image rendering into text pipelines for no good reason.
The web could have defined a URL encoding scheme for emoji from an agreed set, or even a shorthand tag with graceful fallback (e.g.
<emoji img="gb-flag;flag;>Union Flag</emoji>, which would render a British flag if you have an image for gb-flag, a generic flag if you don’t, have ‘Union Flag’ as the alt text or the fall back if you don’t support emoji). With the explicit description and fallback, you avoid the things like ‘I’m going to shoot you with a 🔫’ being rendered as ‘I’m going to shoot you with a {image of a gun}’ or ‘I’m going to shoot you with a {image of a water pistol}’ depending on the platform: if you didn’t have thewater-pistolimage, you’d fall back to the text, not show the pistol image.Like it or not, emoji are a big part of culture now. They genuinely help convey emotion in a fairly intuitive manner through text, way better than obscure tone indicators. I mean, what’s more understandable?
“Are you going to leave me stranded? 😛”
“Are you going to leave me stranded? [/j]”
It definitely changes the meaning of the text. They’re here to stay, and being in Unicode means they got standardized, and it wouldn’t have happened otherwise.
Of course there’s issue with different icon sets having different designs (like how Samsung’s 😬 was completely different from everyone else’s), but those tend to get resolved eventually.
Except they don’t. Different in groups assign different meanings to different ones. Try asking someone for an aubergine using emoji some time and see what happens.
This is culturally specific. It’s an extra set of things that people learning English need to learn. This meaning for sticking out your tongue is not even universal across European cultures. And that’s one of the top ten most common reaction emoji, once you get deeper into the hundreds of others the meaning is even further removed. How would you interpret the difference between 🐶 and 🐕 in a sentence?
That’s an intrinsic property of using unicode code points. They are abstract identifiers that tell you how to find a glyph. The glyphs can be different. A Chalkboard A and a Times A are totally different pictures because that’s an intrinsic property of text. If Android has a gun and iOS has a waterpistol for their pistol emoji, that’s totally fine for characters but a problem for images.
😱 Sure emojis are ambiguous . And different groups can use them differently. But that doesn’t mean they don’t convey meaning? The fact that they are so widely used should point towards them being useful no? 😉
I never said that embedding images in text is not useful. I said that they are not text, do not have the properties of text, and treating them as text causes more problems than it solves.
Emoji are not alphabets, syllabaries, abugidas, or abjads. But they are ideograms, which qualifies them as a written script.
I disagree. At best, they are precursors of an ideographic script. For a writing system, there has to be some kind of broad consensus on semantics and there isn’t for most emoji beyond ‘that is a picture of X’.
Please describe to me the semantics of the letter “р”.
For alphabetic writing systems, the semantics of individual letters is defined by their use in words. The letter ‘p’ is a component in many of the words in this post and yours.
Thank you! (That was actually U+0440 CYRILLIC SMALL LETTER ER, which only featured once in both posts, but no matter.)
The thing is, I disagree. “e” as a letter itself doesn’t have ‘semantics’, only the words including it do[1]. What’s the semantics of the letter “e” in “lobster”? An answer to this question isn’t even wrong. It gets worse when different writing systems interpret the same characters differently: if I write “CCP”, am I referring to the games company CCP Games? Or was I abbreviating сoветская социалистическая республика? What is the semantics of a letter you cannot even identify the system of?
Emoji are given meaning of different complexity by their use in a way that begins to qualify them as logographic. Most other writing systems didn’t start out this way, but that doesn’t make them necessarily more valid.
[1]: The claim doesn’t even hold in traditional logographic writing systems which by all rights should favor your argument. What is the semantics of the character 湯? Of the second stroke of that character? Again, answers aren’t even wrong unless you defer to the writing system to begin with, in which case there’s no argument about (in)validity.
This is true of words as well.
Yes, but their original point is that we should be able to compose emojis like we compose words, as in the old days of phpBB and instant messaging. :mrgreen:
Just a nit: people do compose emojis - I see sequences of emojis all the time. People send messages entirely of emojis that other people (not necessarily me) understand.
The fact that an in-group can construct a shared language using emoji that’s basically opaque to outsiders is probably a big part of their appeal.
Yeah, and also there’s nothing wrong with that, it’s something any group can and should be able to do. I have no entitlement to be able to understand what other people say to each other (you didn’t claim that, so this isn’t an attack on you. I am just opposed to the “I don’t like / use / understand emojis how other people use them therefore they are bad” sentiment that surfaces periodically).
That’s fair, I’m just nitpicking a specific point (that happens to be a pet peeve of mine).
But not of characters and rarely true even of ideographs in languages that use them (there are exceptions but a language is not useful unless there is broad agreement on meaning). It’s not even true of most words, for the same reason: you can’t use a language for communication unless people ascribe the same meaning to words. Things like slang and jargon rarely change more than a small fraction of the common vocabulary (Clockwork Orange aside).
Without getting into the philosophy of what language is, I think this skit best illustrates what I mean (as an added bonus, emoji would have resolved the ambiguities in the text).
Note I’m not arguing for emoji to be in Unicode, I’m just nitpicking the idea that the problem with them is ambiguity.
Socrates would like to have a chat with you. I won’t go through the philosophical tooth-pulling that he would have enjoyed, but suffice it to say that most people are talking past each other and that most social constructions are not well-founded.
I suspect that this is a matter of perspective; try formalizing something the size of English Prime (or, in my case, Lojban) and see how quickly your intuitions fail.
Except emoji have been absolutely stellar for Unicode: not only are they a huge driver of adoption of unicode (and and through UTF8) because they’re actively desirable to a large proportion of the population, they’ve also been a huge driver of improvements to all sorts of useful unicode features which renderers otherwise tend to ignore despite their usefulness to the rendering of actual text, again because they’re highly desirable and platforms which did not support them got complaints. I fully credit emoji with mysql finally getting their heads out of their ass and releasing a non-broken UTF8 (in 2010 or so). That’s why said unicode consortium has been actively leveraging emoji to force support for more complex compositions.
And the reality is there ain’t that much difference between “image rendering” and “text pipeline”. Rendering “an image” is much easier than properly rendering complex scripts like arabic, devanagari, or burmese (or Z̸̠̽a̷͍̟̱͔͛͘̚ĺ̸͎̌̄̌g̷͓͈̗̓͌̏̉o̴̢̺̹̕), even ignoring that you can use text presentation if you don’t feel like adding colors to your pileline.
After all what’s better than one standard if not fifteen?
This problem was solved by adding icons in text. Dingbats are as old as printing, and the Zapf Dingbats which unicode inherited date back to the late 70s.
Because nobody could ever want icons outside the web, obviously. As demonstrated by Lucida Icons having never existed.
It sounds like you disagree solidly with some of Unicode’s practices so maybe this is not so appealing, but FWIW the Unicode character properties would be very handy for defining the subset you’d like to include or exclude. Most languages seem to have a stdlib interface to them, so you could pretty easily promote an ideal of how user input like comment boxes should be sanitized and offer your ideal code for devs to pick up and reuse.
and who’d be the gatekeeper on what the text is and isn’t? What would they say about the ancient Egyptian hieroglyphs? Are they text? If yes, why, they are pictures. If no, why, they encode a language.
It might be a shallow dissimilar, but people trying to tell others what forms of writing text are worthy of being supported by text rendering pipelines gets me going.
If the implementation is really so problematic, treat emojis as complicated ligatures and render them black and white.
Hieroglyphics encode a (dead) language. There are different variations on the glyphs depending on who drew them (and what century they lived in) and so they share the property that there is a tight(ish, modulo a few thousand years of drift) coupling between an abstract hieroglyph and meaning and a loose coupling between that abstract hieroglyph and a concrete image that represents it. Recording them as text is useful for processing them because you want to extract the abstract characters and process them.
The same is true of Chinese (though traditional vs simplified made this a bit more complex and the unicode decisions to represent Kanji and Chinese text using the same code points has complicated things somewhat): you can draw the individual characters in different ways (within certain constraints) and convey the same meaning.
In contrast, emoji do not convey abstract meaning, they are tightly coupled to the images that are used to represent them. This was demonstrated very clearly by the pistol debacle. Apple decided that a real pistol image was bad because it was used in harassment and decided to replace the image that they rendered with a water pistol. This led to the exact same string being represented by glyphs that conveyed totally different meaning. This is because the glyph not the character encodes meaning for emoji. If you parsed the string as text, there is no possible way of extracting meaning without also knowing the font that is used.
Since the image is the meaningful bit, not the character, we should store these things as images and use any of the hundreds of images-and-text formats that we already have.
More pragmatically: unicode represents writing schemes. If a set of images have acquired a significant semantic meaning over time, then they may count as a writing system and so can be included. Instead, things are being added in the emoji space as new things that no one is using yet, to try to define a writing scheme (largely for marketing reasons, so that ‘100 new emoji!’ can be a bullet point on new Android or iOS releases).
It’s not just (or even mostly) about the rendering pipelines (though it is annoying there because emoji are totally unlike anything else and have required entirely new feature to be added to font formats to support them), it’s about all of the other things that process text. A core idea of unicode is that text has meaningful semantics distinct from the glyps that they represent. Text is a serialisation of language and can be used to process that language in a somewhat abstract representation. What, aside from rendering, can you do with processing of emoji as text that is useful? Can you sort them according to the current locale meaningfully, for example (seriously, how should 🐕 and 🍆 be sorted - they’re in Unicode and so that has to be specified for every locale)? Can you translate them into a different language? Can you extract phonemes from them? Can you, in fact, do anything useful with them that you couldn’t do if you embedded them as images with alt text?
Statistically, no-one cares about hieroglyphics, but lots of people care about being able to preserve emojis intact. So text rendering pipelines need to deal with emojis, which means we get proper hieroglyphics (and other Unicode) “for free”.
Plus, being responsible for emoji gives the Unicode Consortium the sort of PR coverage most organizations spend billions to achieve. If this helps them get even more ancient writing systems implemented, it’s a net good.
Today, I
s/☑️/✅/ga text file.Do I have the book for you!
We can’t even do that with a lot of text! 😝
All that’s missing from this sentence to set off all the 🚩 🚩 🚩 is a word like “just” or “simply”.
Others have started poking at your definition of “text”, and are correct to do so – are hieroglyphs “text”? how about ideograms? logograms? – but really the problem is that while you may feel you have a consistent rule for demarcating “text” from “images” (or any other “not text” things), standards require getting a bunch of other people to agree with your rule. And that’s going to be difficult, because any such rule will be arbitrary. Yours, for example, mostly seem to count certain very-image-like things as “text” if they’ve been around long enough (Chinese logograms, Egyptian hieroglyphics) while counting other newer ones as “not text” (emoji). So one might reasonably ask you where the line is: how old does the usage have to be in order to make the jump from “image” to “text”? And since you seem to be fixated on a requirement that emoji should always render the same on every platform, what are you going to do about all the variant letter and letter-like characters that are already in Unicode? Do we really need both U+03A9 GREEK LETTER CAPITAL OMEGA and U+2126 OHM SIGN?
etc.
Do they serialise language? They’re text. Emoji are not a writing system. They might be a precursor to a writing system (most ideographic writing systems started with pictures and were then formalised) but that doesn’t happen until people ascribe common meaning to them beyond ‘this is a picture of X’.
That’s the opposite of my point. Unicode code points represent an abstraction. They are not supposed to require an exact glyph. There are some things in Unicode to allow lossless round tripping through existing character encodings that could be represented as sequences of combining diacritics. They’re not idea in a pure-Unicode world but they are essential for Unicode’s purpose: being able to represent all text in a form amenable to processing.
For each character, there is a large space of possible glyphs that a reader will recognise. The letter A might be anything from a monospaced block character to a curved illustrated drop character from an illuminated manuscript. The picture is not closely coupled to the meaning and changing the picture within that space does not alter the semantics. Emoji do not have that property. They cause confusion when slightly different glyphs are used. Buzzfeed and similar places are full of ‘funny’ exchanges from people interpreting emoji differently, often because they see slightly different glyphs.
The way that emoji are used assumes that the receiver of a message will see exactly the same glyph that the sender sends. That isn’t necessary for any writing system. If I send Unicode of English, Greek, Icelandic, Chinese, or ancient Egyptian, the reader’s understanding will not change if they change fonts (as long as the fonts don’t omit glyphs for characters in that space). If someone sends a Unicode message containing emoji, they don’t have that guarantee because there is no abstract semantics associated with them. I send a picture of a dog, you see a different dog, I make a reference to a feature of that dog and that feature isn’t present in your font, you are confused. Non-geeks in my acquaintance refer to them as ‘little pictures’ and think of them in the same way as embedded GIFs. Treating them as characters causes problems but does not solve any problems.
I think this is going to end up being a far deeper and more complex rabbit hole than the tone of your comment anticipates. Plenty of things that are in Unicode today, and that you undoubtedly would consider to be “text”, do not hold up to this criterion.
For example, any character that has regional/dialect/language-specific variations in pronunciation seems to be right out by your rules. So consider, say, Spanish, where in some dialects the sound of something like the “c” in “Barcelona” is /s/ and in others it’s /θ/. It seems hard to say that speakers of different dialects agree on what that character stands for.
At this point, I feel like the cat is out of the bag; people are used to being able to use emoji in almost any text-entry context. Text rendering pipelines are now stuck supporting these icons. With that being the case, wouldn’t it be way more complexity to layer another parsing scheme on top of Unicode in order to represent emoji? I can see the argument that they shouldn’t have been put in there in the first place, but it doesn’t seem like it would be worth it to try to remove them now that they’re already there.
Neat use of Janet. You could also save the whole environment so you don’t have to stuff everything into
stuff.I actually tried that. The solution is found after I already wrote the article. The Janet’s official documentation about embedding is not working.
The solution is already in the source code. I didn’t mention it because it’s complicated.
Zig side:
Janet side:
(make-image (curenv))I think Rust’s
asyncis fine.On the technical side, for what it does, it’s actually pretty good. Async calls and
.awaitdon’t need to do heap allocations. TheFuturetrait is relatively simple (API surface of C++ coroutines seems much larger IMHO). Ability to use futures with a custom executor makes it possible to use them, efficiently, on top of other languages’ runtimes, or for clever applications like fuzzing protocols deterministically.Usability-wise, like many things in Rust, they have a steep frustrating learning curve, and a couple more usability issues than other Rust features. But once you learn how to use them, they’re fine. They don’t have any inherent terrible gotchas. Cancellation at any await point is the closest to a gotcha, but you can get away with that since Rust uses destructors and guards so much. In contrast with old node-style callbacks, which had issues no matter how experienced you were: subtlety of immediate vs deferred callbacks, risk of forgetting to call a callback on some code path or when an exception is thrown, need to deal with possibility of having a callback called more than once, etc.
The famous color article implied that async should be an implementation detail to hide, but that’s merely a particular design choice, which chooses implicit magic over clarity and guarantees. Rust doesn’t hide error propagation. Rust doesn’t hide heap allocations, and Rust doesn’t hide await points. This is a feature, because Rust is used for low-level code where it can be absolutely necessary to have locks, threads, and syscalls behave in specific predictable ways. Having them sometimes invisibly replaced with something completely else would be as frightening as having UB.
asyncRust is fine for what it accomplishes, but is generally harder to use than normal Rust, and normal Rust was already changeling for many developers. Which would be fine if async Rust was only used where warranted, but it actually became the default and dominant dialect of Rust.Is it? I’m just dabbling into Rust and not very experienced by any means, but at least so far i’ve gotten the impression that async stuff is mostly found in some “leaf” and specialized crates, and that more “core” crates like
regexorserdeare just normal non-async Rust.I never got the impression of async Rust being dominant, much less the default. For which i’m grateful really, as i too share the impression that the async stuff in Rust is quite complicated and difficult to grok, and i would prefer to avoid it if possible.
async/awaitis the callee telling the caller how it wants to be executed/scheduled.Languages with no
async/awaitlet the caller decide how the callee should be executed/scheduled.In Erlang/Elixir, any function can be given to
spawn(), orTask.start(). In Go, any function can be given togo ....It’s up to the caller to determine how the function should be scheduled. async/await tend to contaminate your whole code base. I’ve seen many Python and some Rust+tokio codebases where everything up to
main()was an async function. At that point, can we just get rid of the keyword?You’ve given examples of higher-level languages with a fat runtime. This includes golang, which requires special care around FFI.
This “implicit async is fine, you don’t need the await syntax” is another variant of “implicit allocations/GC are fine, you don’t need lifetime syntax”.
True in most cases, but Rust is just not that kind of language, on purpose. It’s designed to “contaminate” codebases with very explicit code, and give low-level control and predictable performance of everything.
Aren’t we considering tokio and all the other third-party libraries that you need to bring in in order to schedule async code, a fat runtime?
The comparison with lifetimes is unfair. Here, we are adding the async keyword to almost every function, and the await keyword to almost every function call. With lifetimes, there is far more options. But if we were just adding ‘static to every reference as a lifetime, then yes, I would ask if it’s really needed.
You don’t need Tokio to drive async code. You can have it for your ELF binary, and use the JS runtime when compiling to WASM.
So, replacing the fat runtime that is tokio by the fat runtime that is JS. How does that dismiss the examples of high-level languages I gave?
The thing that makes Rust different from e.g. Go: being able to choose your runtime. And it doesn’t even need to be fat, there are embedded runtimes or libraries such as Smol.
No, I wouldn’t put tokio in the same category. With something like Erlang your whole program lives in Erlang’s environment. It’s opaque, and sits between you and the OS, so it’s not a surprise when it inserts “magic” into that interaction.
But Rust is for systems programming. It’s not supposed to intervene where you don’t ask it to. Things that Rust does are transparent, and you’re free to program for the underlying OS.
In case of Rust you use tokio as a library. Tokio is dominant, but not necessary. I’ve worked on projects that used custom async executors. You can use async in embedded programming. You can use async in the kernel. It’s a builder for state machines, not a VM.
In relation to lifetimes I did not mean just the amount of tokens you add to the source code, but the fact that Rust requires you to be precise and explicit about details that could have been abstracted away (with a GC). Rust’s types have “colors” in many ways: owned or borrowed, shared or mutable, copy or non-copy. Rust could have had “colorless” types: everything shared mutable copyable, but chose to have all this complexity and syntax for a reason.
Rust could ship a green thread runtime in libstd (again). It could hide the async and await keywords and insert glue code automatically. It would be nicer syntactically. It would be easier to use. It would be a good tradeoff for majority of programs — just like benefits of adding a GC. But Rust choses to be low level and offer transparency, predictability, and control instead. In low-level programming knowing function’s mode of execution is as important as knowing whether it mutates or frees memory.
So we are told, but the home page talks about applications like command-line tools and network services. Seemingly the same kind of ‘systems programming’ that Rob Pike was saying Go is good for.
All of coreutils are in C. Nfs, sshd, samba, ntpd are in C. Nginx, apache are in C. Rust wants to be where C is.
There is a large overlap with Golang, and not every Rust program had to be in Rust, like not every C program had to be in C. But Rust is sticking to being low-level enough to be able to replace C everywhere.
All those tools are written in C because they want to be able to run wherever you can run a C compiler. Tools that are written in Rust most often don’t care about having that level of portability.
Rust vs C platform support is a complex topic. Rust tools generally have better portability across contemporary platforms. Windows often has native support, whereas in POSIX-centric C projects that’s typically “MS sucks, use WSL?” (and I don’t count having to build Linux binaries in a Linux VM as portability to Windows).
Rust’s platform support is not too bad these days. I think it covers everything than Debian supports except IA64 and SH4. And Rust also has some support for S390x, AVR, m68k, CUDA, BPF. Haiku, VxWorks, PlayStation 1, 3DS, and Apple Watch.
Rust doesn’t support platforms that LLVM/clang can’t find maintainers for, and these are generally pretty niche and/or obsolete. I think C projects support targets like Itanium or Dreamcast more out of tradition and a sense of pride, rather than any real-world need for them.
GCC backend (rustc_codegen_gcc) for Rust is in the works, so eventually Rust won’t be limited by LLVM’s platform support.
Erlang doesn’t allow local state in functions. You need
async/awaitto be able to comprehend how and when local state can change.Go doesn’t have a good excuse.
In Go, everything is a goroutine, so in a sense everything is already contaminated with
async/await. Code that blocks is automatically an ‘await point’, unless you use thegokeyword. So I don’t think the semantics around caller/callee are not any different from async Rust with e.g.tokio::mainas an entry point. The difference is you have to manually mark await points. However you do get better control, e.g.tokio::select!can act on any async future, not just channel reads..Select in rust is not without its problems - it can trigger a panic and adds cancellation safety issues.
I’m not a fan of what feels like needless hostility (confrontational tone?) in the article, and was expecting to hate it going in, but it does make some good points.
I feel like this point in particular does not get attention when talking about
asyncin languages and took me a long while to get the mental model for.I disagree with this opinion. In any language with native async infrastructure built-in, I’ve had to learn how it works pretty intimately to effectively use it. My worst experiences have been with Python’s
asynciowhile the easiest was probably F#.I don’t think you’re disagreeing? The article is essentially saying that you have to learn async along with the rest of the language, and you are also saying that you had to learn async with the rest of the language.
I think the difference is they’re making it sound like some uniquely difficult thing in Rust, and I disagree that it’s some Rust-only problem.
It’s an async/await problem.
In languages with concurrency and no async/await (erlang, elixir, go, …), the choice of the scheduling model of your code is determined at the call site. The callee should not care about how it is executed.
In go:
In Rust:
You have the same amount of control of scheduling. If you’re referring to being unable to call an async method from a sync context, this is technically also true in Go, but since everything runs in a goroutine everything is always an async context.
What makes Rust harder is the semantics around moving and borrowing but also the “different concrete type for each async expression” nature of the generated state machines. For example this is easy in go, but painful in Rust:
may i ask what you used to figure out how to think about concurrency in F# ?
A lot of experience getting C#/F# async interop working and the .NET documentation/ecosystem is pretty great these days.
https://learn.microsoft.com/en-us/dotnet/fsharp/tutorials/async
In F# 6, they made the C# async primitives work seamlessly so you’re no longer wrapping anything in tasks to wait on it on the F# side.
The real scary thing is that it took users weeks to notice that it shipped, despite that it wasn’t obfuscated in any way. This shows how risky the ecosystem is, without enough eyes reviewing published crates. If any high profile crate author gets infected with malware that injects itself into crates, it’s going to be an apocalypse for Rust.
Maybe this is also a sign that the complaints themselves were incoherent?
I think it’s only a sign that we’re unaware until this hits a sandboxed / reproducable build system. I guess that’s currently distribution packaging or projects that otherwise use Nix or Bazel to build.
But it highlights how little else is sandboxed.
Exactly, these complaints are incoherent unless you were already doing the things that would cause you to notice the change!
I’m not sure that “No one’s looking anyway, so it’s totally fine” is the right takeaway from this.
If the complaint is that binaries are more difficult to audit than source, and no one is auditing, then it should make no difference either way from a security perspective.
it is perfectly coherent to advocate for other people.
I think “weeks” is a bit of an exaggeration. People were openly discussing it at least a week after release. It’s true though that it didn’t blow up on social media until weeks later and many people didn’t realise until then.
If it had been a security issue or it was done by someone much less reputable than the author of serde or if the author did not respond then I suspect rustsec may have been more motivated to post an advisory.
Something that I might have expected to see included in this comment, and that I instead will provide myself, is a plug for bothering to review the code in one’s (prospective) dependencies, or to import reviews from trusted other people (or, put differently, to limit oneself to dependencies that one is able and willing to review or that someone one trusts has reviewed).
I recall that kornel at least used to encourage the use of
cargo-crev, and their Lib.rs now also shows reviews from the newer and more streamlinedcargo-vet.I note that the change adding the blob to Serde was reviewed and approved through
cargo-vetby someone at Mozilla. I don’t think that necessarily means these reviewing measures would not be useful in a situation that isn’t as much a drill (i.e., with a blob more likely to be malicious).Yeah - my recollection of crev is that libraries like serde often got reviews like “it’s serde, might as well be the stdlib, I trust this without reviewing it as the chances of it being malicious are basically zero”
What a ridiculous thing to have even happened in the first place, let alone refusing to acknowledge there could possibly be an issue for so long. I glad it’s been fixed but would make me think twice about using serde. I’m sure it’ll be fine, who’s ever heard of a security issue in a codec anyway?
Remember that there are real human being maintaining
serde. It is not, in fact, blindingly obvious to all developers that the pre-compiled blobs were bad; on this site there were loud voices on both sides. Can you imagine suddenly getting caught in the crosshairs of angry developers like that? When I imagine it, it feels bad, and I’m liable to get defensive about it.It may also have been a failed attempt at fixing something you’ve heard people complain about all the time, probably even about your code that slows down peoples builds (*). So yeah it was a bad idea in hindsight, but we don’t need more burned out maintainers from this. And I say this as someone who is openly disappointed by this happening.
(*) I’m not going to discuss how much time it actually saved.
This overview by @cadey implied it did not save much time at all, basically only if you were running a CI setup without a cache.
https://xeiaso.net/blog/serde-precompiled-stupid
Running a CI setup without a cache is, for better or for worse, very common
Yeah, basically the biggest gains are offset by process creation being surprisingly slow. I’m working on a follow-up article where I talk about that in detail.
I posted your piece because it was the first one that explained in detail what the hell was going on, specifically how serde works. Looking forward to a followup.
My workplace is way too big. This describes our CI setup. Only the blessed JVM gets to have cached CI builds.
This is how shadow IT begins. Has anyone started running/sharing their locally setup CI for your project yet?
That’s how it started, then they centralized everything with one team that doles out the “managed CI” offering, with their own global library and controls. Any competing infra gets flagged and audited hardcore until you give up by attrition.
This seems to only be checking the performance under –release. Most compilation is done without –release, meaning that most of the proc macro will not be optimized.
As someone who packages software, I think it’s worth noting that packagers expect different things than end users, though they are compatible.
One of my wishes is to avoid blobs from a vendor, since we can’t always recompile those in the build process to work with the architectures we support.
(The other big difference is the DESTDIR env var. End users don’t generally care, but it becomes essential when preparing a package)
I therefore understand those who support their end users, before getting packaged.
The real human being maintaining serde knew about the pushback that would happen and did it on purpose to prove a point in a pre-RFC he submitted. I don’t feel particularly bad about him getting pushback for using half the Rust ecosystem as his guinea pigs. (In fact I would like to see more of it.)
What’s the reason to believe in this over any other explanation of the situation? E.g. that pushback was unexpected and that the RFC is the result of the pushback, rather than a cause?
I consider dtolnay a competent open source maintainer who understands the people who run his code well, and I would expect any competent open source maintainer to expect such pushback.
But how that necessary leads to “on purpose to prove a point”?
I don’t think dtolnay expected exactly zero pushback. But, given that some people in this thread argue quite a reasonable point that binaries are actually almost as fine as source, it is plausible that only bounded pushback was expected.
The excerpt from the RFC is:
I don’t see someone competent casually pushing such a controversial change, casually saying that this is now the only supported way to use serde, casually pushing a complete long pre-RFC that uses the controversial change to advance it, and then casually reverting the change in the span of a few days. That takes preparation and foresight.
I actually respect this move. It is exactly the kind of move I would do if I had goodwill to burn and was frustrated with the usual formal process, and it takes boldness and courage to pull it off the way he did it. I also think the pushback is entirely appropriate and the degree of it was quite mild.
Aha, thanks! I think that’s a coherent story to infer from this evidence (and I was wondering if there might be some missing bits I don’t know).
From where I stand, I wouldn’t say that this explanation looks completely implausible, but I do find it unlikely.
For me, the salient bits are:
I agree that there are multiple intepretations possible and that yours also follows from the evidence available. The reason I think it’s reasonable to consider something deeper to be going on is: every single Rust controversy I’ve discussed with key Rust people had a lot more going on than was there on the surface. Case in point: dtolnay was also the one thus far unnamed by anyone speaking for the project person who was involved in ThePHD’s talk being downgraded from a keynote. If I see someone acting surreptitiously in one case I will expect that to repeat.
O_o that’s news to me, thanks. It didn’t occur that dtopnay might have been involved there (IIRC, they aren’t a team lead of any top-level team, so I assume weren’t a member of the notorious leadership chat)
Maybe take hearsay from an anonymous Internet catgirl with a grain of salt.
Calling me anonymous is pretty funny, considering I’ve called myself “whitequark” for close to 15 years at this point and shipped several world-class projects under it.
whitequark would be pretty well known to an old Rust team member such as matklad, having been one themself, so no, not anonymous… buut we don’t know this is the same whitequark, so yes, still anonymous.
Hm? Neither of them are Rust team members, unless they are represented under different names in the Project.
I mean, I wrote both Rust language servers/IDEs that everyone is using and whitequark wrote the Ruby parser everyone is using (and also smaltcp). I think we know perfectly fine who we are talking with. One us might be secretly a Labrador in a trench coat, but that doesn’t have any bearing on the discussion, and speculation on that topic is hugely distasteful.
In terms of Rust team membership, I actually don’t know which team whitequark was on, but they are definitely on the alumni page right now. I was on the cargo team and TL for the IDE team.
Thank you for all the context I was missing. Is it just oversight you aren’t on the alumni for those team pages?
Turns out there were at least two bugs in the teams repo with respect to me, thanks for pointing this out!
I’m glad my bickering had at least some positive outcome :)
Probably! I think https://github.com/rust-lang/team/commit/458c784dda91392b710d36661f440de40fdac316should have added me as one, not sure why that didn’t happen
I don’t know what you mean by “the Project”, but the source of truth for Rust team membership is https://github.com/rust-lang/team.
You were talking about “Rust teams” and the only way I’ve seen that term used is to indicate those under the “Rust Project”. Neither person is on a Rust team or an alumni.
https://www.rust-lang.org/governance
That is what I meant, yes. Those pages are generated from the Git repo I linked. Ctrl-F on https://www.rust-lang.org/governance/teams/compiler and https://www.rust-lang.org/governance/teams/alumni.
A find would tell you matklad was not on a team. He was “just” a contributor. No real data exists about whitequark.
Tbh, it more reeks of desperation to make people’s badly configured CI flows faster. I think that a conspiratorial angle hasn’t been earned yet for this and that we should go for the most likely option: it was merely a desperate attempt to make unoptimized builds faster.
I think this is hard to justify when someone comes to you with a security issue, when your response is “fork it, not my problem”, and then closing the issue, completely dismissing the legitimate report. I understand humans are maintaining it, humans maintain all software I use in fact, and I’m not ok with deciding “Oh, a human was involved, I guess we should let security bad practices slide”. I, and I’m sure many others, are not frustrated because they didn’t understand the security implications, but because they were summarily dismissed and rejected, when they had dire implications for all their users. From my understanding, Serde is a) extremely popular in the Rust world, and b) deals in one of the most notoriously difficult kinds of code to secure, so seeing the developers’ reaction to a security issue is very worrying for the community as a whole.
The thing is, its not unambiguous whether this is a security issue. “Shipping precompiled binaries is not significantly more insecure than shipping source code” is an absolutely reasonable stance to have. I even think it is true if we consider only first-order effects and the current state of rust packaging&auditing.
Note also that concerns were not “completely dismissed”. Dismissal looks like “this is not a problem”. What was said was rather “fixing this problem is out of scope for the library, if you want to see it fixed, work on the underlying infrastructure”. Reflecting on my own behavior in this discussion, I might be overly sensitive here, but to me there’s a world of difference between a dismissal, and an acknowledgment with disagreement on priorities.
This is perhaps a reasonable take-away from all the internet discussions about the topic, but I don’t think this actually reflects what did happen.
The maintainer was responsive on the issue and they very clearly articulated that:
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I do think it was a mistake to not predict that this change will be this controversial (or to proceed with controversial change without preliminary checks with wider community).
But, given that a mistake had been made, the handling of the situation was exemplary. Everything that needed fixing was fixed, promptly.
I’m still waiting to hear what “security concern” there was here. Other language-package ecosystems have been shipping precompiled binaries in packages for years now; why is it such an apocalyptically awful thing in Rust and only Rust?
The main thing is loss of auditing ability — with the opaque binaries, you can not just look at the package tarbal from crates.io and read the source. It is debatable how important that is: in practice, as this very story demonstrates, few people look at the tarballs. OTOH, “can you look at tarballs” is an ecosystem-wide property — if we lose it, we won’t be able to put the toothpaste back into the tube.
This is amplified by the fact that this is build time code — people are in general happier with sandbox the final application, then with sandboxing the sprawling build infra.
With respect to other languages — of course! But also note how other languages are memory unsafe for decades…
It’s not that hard to verify the provenance of a binary. And it appears that for some time after serde switched to shipping the precompiled macros, exactly zero people actually were auditing it (based on how long it took for complaints to be registered about it).
The ecosystem having what boils down to a social preference for source-only does not imply that binary distributions are automatically/inherently a security issue.
My go-to example of a language that often ships precompiled binaries in packages is Python. Which is not exactly what I think of when I think “memory unsafe for decades”.
Verifying provenance and auditing source are orthogonal. If you have trusted provenance, you can skip auditing the source. If you audited the source, you don’t care about the provenance.
It’s a question which one is more practically important, but to weight this tradeoff, you need to acknowledge its existence.
This sounds like:
This doesn’t sound like:
I don’t know where your last two blockquotes came from, but they didn’t come from my comment that you were replying to, and I won’t waste my time arguing with words that have been put in my mouth by force.
That’s how I read your reply: as an absolute refusal to acknowledge that source auditing is a thing, rather than as a nuanced comparison of auditing in theory vs auditing in practice.
It might not have been your intention to communicate that, but that was my take away from what’s actually written.
Once again, I don’t intend to waste my time arguing with someone who just puts words in my mouth.
In the original github thread, someone went to great lengths to try to reproduce the shipped binary, and just couldn’t do it. So it is very reasonable to assume that either they had something in their build that differed from the environment used to build it, or that he binary was malicious, and without much deeper investigation, it’s nearly impossible to tell which is the answer. If it was trivial to reproduce to build with source code you could audit yourself, then there’s far less of a problem.
Rust doesn’t really do reproducible builds, though, so I’m not sure why people expected to be able to byte-for-byte reproduce this.
Also, other language-package ecosystems really have solved this problem – in the Python world, for example, PyPI supports a verifiable chain all the way from your source repo to the uploaded artifact. You don’t need byte-for-byte reproducibility when you have that.
Ah yes, garbage collected languages are famously ‘memory unsafe for decades’
I guesss I should clarify that in GP comment the problem is misalignment between maintainer’s and user’s view of the issue. This is a problem irrespective of ground truth value of security.
Maybe other language package ecosystems are also wrong to be distributing binaries, and have security concerns that are not being addressed because people in those ecosystems are not making as much of a fuss about it.
If there were some easy way to exploit the mere use of precompiled binaries, someone would have by now. The incentives to use such an exploit are just way too high not to.
There are ways to exploit binary releases. It’s certainly not easy, but this has definitely been exploited in the wild.
You can read this page https://reproducible-builds.org/docs/buy-in/ to get a high-level history of the “reproducible build” (and bootstrapping) movement.
Anecdotally, I almost always see Python malware packaged as source code. I think that could change at any time to compiled binaries fwiw, just a note.
I don’t think attackers choosing binary payloads would mean anything for anyone really. The fundamental problem isn’t solved by reproducible builds - those only help if someone is auditing the code.
The fundamental problem is that your package manager has near-arbitrary rights on your computer, and dev laptops tend to be very privileged at companies. I can likely go from ‘malicious build script’ to ‘production access’ in a few hours (if I’m being slow and sneaky) - that’s insane. Why does a build script have access to my ssh key files? To my various tokens? To my ~/.aws/ folder? Insane. There’s zero reason for those privileges to be handed out like that.
The real solution here is to minimize impact. I’m all for reproducible builds because I think they’re neat and whatever, sure, people can pretend that auditing is practical if that’s how they want to spend their time. But really the fundamental concept of “running arbitrary code as your user” is just broken, we should fix that ASAP.
Like I’ve pointed out to a couple people, this is actually a huge advantage for Python’s “binary” (
.whl) package format, because its install process consists solely of unpacking the archive and moving files to their destinations. It’s the “source” format that can ship asetup.pyrunning arbitrary code at install time. So tellingpipto exclusively install from.whl(with--only-binary :all:) is generally a big security win for Python deployments.(and I put “binary” in scare quotes because, for people who aren’t familiar with it, a Python
.whlpackage isn’t required to contain compiled binaries; it’s just that the.whlformat is the one that allows shipping those, as well as shipping ordinary Python source code files)Agree. But that’s a different threat, it has nothing to do with altered binaries.
Code auditing is worthless if you’re not sure the binary you’re running on your machine has been produced from the source code you’ve audited. This source <=> binary mapping is precisely where source bootstrapping + reproducible builds are helping.
This is a false dichotomy. I think we agree on the fact we want code audit + binary reproducibility + proper sandboxing.
Well, we disagree, because I think they’re identical in virtually every way.
I’m highly skeptical of the value behind code auditing to begin with, so anything that relies on auditing to have value is already something I’m side eyeing hard tbh.
I think where we disagree on the weights. I barely care about binary reproducibility, I frankly don’t think code auditing is practical, and I think sandboxing is by far the most important, cost effective measure to improve security and directly address the issues.
I am familiar with the concept of reproducible builds. Also, as far as I’m aware, Rust’s current tooling is incapable of producing reproducible binaries.
And in theory there are many attack vectors that might be present in any form of software distribution, whether source or binary.
What I’m looking for here is someone who will step up and identify a specific security vulnerability that they believe actually existed in
serdewhen it was shipping precompiled macros, but that did not exist when it was shipping those same macros in source form. “Someone could compromise the maintainer or the project infrastructure”, for example, doesn’t qualify there, because both source and binary distributions can be affected by such a compromise.Aren’t there links in the original github issue to exactly this being done in the NPM and some other ecosystem? Yes this is a security problem, and yes it has been exploited in the real world.
I’m going to quote my other comment:
If you have proof of an actual concrete vulnerability in
serdeof that nature, I invite you to show it.The existence of an actual exploit is not necessary to be able to tell that something is a serious security concern. It’s like laying an AR-15 in the middle of the street and claiming there’s nothing wrong with it because no one has picked it up and shot someone with it. This is the opposite of a risk assessment, this is intentionally choosing to ignore clear risks.
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening. To me this is up with with Sony’s rootkit back in the day, completely unexpected, unauthorised behaviour that no reasonable person would expect, nor would they look out for it because it is just such an unreasonable thing to do to your users.
I think the point is that if precompiled macros are an AR-15 laying in the street, then source macros are an AR-15 with a clip next to it. It doesn’t make sense to raise the alarm about one but not the other.
I think this is extreme. No additional accessing of any kind was done. Binaries don’t have additional abilities that
build.rsdoes not have. It’s not at all comparable to installing a rootkit. The precompiled macros did the same thing that the source macros did.Once again, other language package ecosystems routinely ship precompiled binaries. Why have those languages not suffered the extreme consequences you seem to believe inevitably follow from shipping binaries?
Even the most extreme prosecutors in the US never dreamed of taking laws like CFAA this far.
I think you should take a step back and consider what you’re actually advocating for here. For one thing, you’ve just invalidated the “without any warranty” part of every open-source software license, because you’re declaring that you expect and intend to legally enforce a rule on the author that the software will function in certain ways and not in others. And you’re also opening the door to even more, because it’s not that big a logical or legal leap from liability for a technical choice you dislike to liability for, say, an accidental bug.
The author of
serdedidn’t take over your computer, or try to. All that happened wasserdestarted shipping a precompiled form of something you were going to compile anyway, much as other language package managers already do and have done for years. You seem to strongly dislike that, but dislike does not make something a security vulnerability and certainly does not make it a literal crime.I think that what actually is happening in other language ecosystems is that while there are precompiled binaries sihpped along some installation methods, for other installation methods those are happening by source.
So you still have binary distribution for people who want that, and you have the source distribution for others.
I have not confirmed this but I believe that this might be the case for Python packages hosted on debian repos, for example. Packages on PyPI tend to have source distributions along with compiled ones, and the debian repos go and build packages themselves based off of their stuff rather than relying on the package developers’ compiled output.
When I release a Python library, I provide the source and a binary. A linux package repo maintainer could build the source code rather than using my built binary. If they do that, then the thing they “need to trust” is the source code, and less trust is needed on myself (on top of extra benefits like source code access allowing them to fix things for their distribution mechanisms)
I don’t know of anyone who actually wants the sdists from PyPI. Repackagers don’t go to PyPI, they go to the actual source repository. And a variety of people, including both me and a Python core developer, strongly recommend always invoking
pipwith the--only-binary :all:flag to force use of.whlpackages, which have several benefits:--require-hashesand--no-deps, you get as close to perfectly byte-for-byte reproducible installs as is possible with the standard Python packaging toolchain..whlhas no scripting hooks (as opposed to an sdist, which can run arbitrary code at install time via itssetup.py).I misread that as “sadists from PyPi” and could not help but agree.
I mean there are plenty of packages with actual native dependencies who don’t ship every permutation of platform/Python version wheel needed, and there the source distribution is available. Though I think that happens less and less since the number of big packages with native dependencies is relatively limited.
But the underlying point is that with an option of compiling everything “from source” available as an official thing from the project, downstream distributors do not have to do things like, say, confirm that the project’s vendored compiled binary is in fact compiled from the source being pointed at.
Install-time scripting is less of an issue in this thought process (after all, import-time scripting is a thing that can totally happen!). It should feel a bit obvious that a bunch of source files is easier to look through to figure out issues rather than “oh this part is provided by this pre-built binary”, at least it does to me.
I’m not arguing against binary distributions, just think that if you have only the binary distribution suddenly it’s a lot harder to answer a lot of questions.
As far as I’m aware, it was possible to build
serde“from source” as a repackager. It did not produce a binary byte-for-byte identical to the one being shipped first-party, but as I understand it producing a byte-for-byte identical binary is not something Rust’s current tooling would have supported anyway. In other words, the only sense in which “binary only” was true was for installing fromcrates.io.So any arguments predicated on “you have only the binary distribution” don’t hold up.
Hmm, I felt like I read repackagers specifically say that the binary was a problem (I think it was more the fact that standard tooling didn’t allow for both worlds to exist). But this is all a bit moot anyways
It’s a useful fallback when there are no precompiled binaries available for your specific OS/Arch/Python version combination. For example when pip installing from a ARM Mac there are still cases where precompiled binaries are not available, there were a lot more closer to the M1 release.
When I say I don’t know of anyone who wants the sdist, read as “I don’t know anyone who, if a wheel were available for their target platform, would then proceed to explicitly choose an sdist over that wheel”.
Argumentum ad populum does not make the choice valid.
Also, not for nothing, most of the discussion has just been assuming that “binary blob = inherent automatic security vulnerability” without really describing just what the alleged vulnerability is. When one person asserts existence of a thing (such as a security vulnerability) and another person doubts that existence, the burden of proof is on the person asserting existence, but it’s also perfectly valid for the doubter to point to prominent examples of use of binary blobs which have not been exploited despite widespread deployment and use, as evidence in favor of “not an inherent automatic security vulnerability”
Yeah, this dynamic has been infuriating. In what threat model is downloading source code from the internet and executing it different from downloading compiled code from the internet and executing it? The threat is the “from the internet” part, which you can address by:
Anyone with concerns about this serde change should already be doing one or both of these things, which also happen to make builds faster and more reliable (convenient!).
Yeah, hashed/pinned dependency trees have been around forever in other languages, along with tooling to automate their creation and maintenance. It doesn’t matter at that point whether the artifact is a precompiled binary, because you know it’s the artifact you expected to get (and have hopefully pre-vetted).
Downloading source code from the internet gives you the possibility to audit it, downloading a binary makes this nearly impossible without whipping out a disassembler and hoping that if it is malicious, they haven’t done anything to obfuscate that in the compiled binary. There is a “these languages are turing complete, therefore they are equivalent” argument to be made, but I’d rather read Rust than assembly to understand behaviour.
The point is that if there were some easy way to exploit the mere use of precompiled binaries, the wide use of precompiled binaries in other languages would have been widely exploited already. Therefore it is much less likely that the mere presence of a precompiled binary in a package is inherently a security vulnerability.
I’m confused about this point. Is anyone going to fix crates.io so this can’t happen again?
Assuming that this is a security problem (which I’m not interested in arguing about), it seems like the vulnerability is in the packaging infrastructure, and serde just happened to exploit that vulnerability for a benign purpose. It doesn’t go away just because serde decides to stop exploiting it.
I don’t think it’s an easy problem to fix: ultimately, package registry is just a storage for files, and you can’t control what users put there.
There’s an issue open about sanitizing permission bits of the downloaded files (which feels like a good thing to do irrespective of security), but that’s going to be a minor speed bump at most, as you can always just copy the file over with the executable bit.
A proper fix here would be fully sandboxed builds, but:
Who’s ever heard of a security issue caused by a precompiled binary shipping in a dependency? Like, maybe it’s happened a few times? I can think of one incident where a binary was doing analytics, not outright malware, but that’s it.
I’m confused at the idea that if we narrow the scope to “a precompiled binary dependency” we somehow invalidate the risk. Since apparently “curl $FOO > sh” is a perfectly cromulent way to install things these days among some communities, in my world (30+ year infosec wonk) we really don’t get to split hairs over ‘binary v. source’ or even ‘target v. dependency’.
I’m not sure I get your point. You brought up codec vulns, which are irrelevant to the binary vs source discussion. I brought that back to the actual threat, which is an attack that requires a precompiled binary vs source code. I’ve only seen (in my admittedly only 10 Years of infosec work) such an attack one time, and it was hardly an attack and instead just shady monetization.
This is the first comment I’ve made in this thread, so I didn’t bring up codecs. Sorry if that impacts your downplaying supply chain attacks, something I actually was commenting on.
Ah, then forget what I said about “you” saying that. I didn’t check who had commented initially.
As for downplaying supply chain attacks, not at all. I consider them to be a massive problem and I’ve actively advocated for sandboxed build processes, having even spoken with rustc devs about the topic.
What I’m downplaying is the made up issue that a compiled binary is significantly different from source code for the threat of “malicious dependency”.
So not only do you not pay attention enough to see who said what, you knee-jerk responded without paying attention to what I did say. Maybe in another 10 years…
Because I can
curl $FOO > foo.sh; vi foo.shthen can choose tochmod +x foo.sh; ./foo.sh. I can’t do that with an arbitrary binary from the internet without whipping out Ghidra and hoping my RE skills are good enough to spot malicious code. I might also miss it in some downloaded Rust or shell code, but the chances are significantly lower than in the binary. Particularly when the attempt from people in the original issue thread to reproduce the binary failed, so no one knows what’s in it.No one, other than these widely publicised instances in NPM, as well as PyPi and Ruby, as pointed out in the original github issue. I guess each language community needs to rediscover basic security issues on their own, long live NIH.
Am I missing something? Both links involve malicious source files, not binaries.
I hadn’t dived into them, they were brought up in the original thread, and shipping binaries in those languages (other than python with wheels) is not really common (but would be equally problematic). But point taken, shouldn’t trust sources without verifying them (how meta).
But the question here is “Does a binary make a difference vs source code?” and if you’re saying “well history shows us that attackers like binaries more” and then history does not show that, you can see my issue right?
But what’s more, even if attackers did use binaries more, would we care? Maybe, but it depends on why. If it’s because binaries are so radically unauditable, and source code is so vigilitantly audited, ok sure. But I’m realllly doubtful that that would be the reason.
There’s some interesting meat to think about here in the context of package management, open source, burden on maintainers, varying interest groups.
I love this. I’m currently working on something similar and this came at the perfect time. Do you have any more sources on alignment? I’m not super informed but I’m curious since most other sources seem to make a big deal out of alignment (e.g. here).
Another reason to align the heap would be to support >1GB sizes with the same relative pointer, e.g. you could do 4GB with 4 byte alignment. But then I guess it wouldn’t be small anymore.
I do not have any other references on alignment; if you find any, please post them to lobste.rs!
Adding alignment does expand the reach of pointers. It would make even-smaller pointers more practical: for instance you could access 16MB instead of 4MB with 24-bit values (assuming 1 bit is still used for a tag.)
It’s probably worth noting that this is a problem only for large projects. Yes, a CI build of LLVM for us takes 20 minutes (16 core Azure VM), but LLVM is one of the largest C++ codebases that I work on and even then incremental builds are often a few seconds (linking debug builds takes longer because of the vast quantity of debug info).
There’s always going to be a trade off between build speed and performance: moving work from run time to compile time is always going to slow down compile speed. The big problem with C++ builds is the amount of duplicated work. Templates are instantiated at use point and that instantiations is not shared between compilation units. I wouldn’t be surprised if around 20% of compilation units in LLVM have duplicate copies of the small vector class instantiated on an LLVM value. Most of these are inlined and then all except one of the remainder is thrown away in the final link. The compilation database approach and modules can reduce this by caching the instantiations and sharing them across compilation units. This is something that a language with a more overt module system can definitely improve because dependency tracking is simpler. If a generic collection and a type over which it’s instantiated are in separate files then it’s trivial to tell (with a useful over approximation) when the instantiations needs to be regenerated: when either of the files has changed. With C++, you need to check that no other tokens in the source file have altered the shape of the AST. C++ modules help a lot here, by providing includable units that generate a consistent AST independent of their use order. Rust has had a similar abstraction from the start (as have most non-C languages).
I think that’s true for C++ but I use Rust for small projects and I definitely find compile times to still be a problem. Rust has completely replaced my usage of C++ but is strictly worse in terms of compilation time for my projects. Even incremental builds can be really slow because if you depend on a lot of crates (which is incredibly easy to do transitively), linking time will dominate. mold helps a lot here.
I think this is always going to be a problem regardless of the compiler or optimizations because it’s just too tempting to use advanced macros / type level logic or pull in too many dependencies. In C++ I would just avoid templates and other slow to compile patterns, but in Rust these utilities are too powerful and convenient to ignore. Most of the Rust versions of a given library will try to do something cool with types and macros that’s usually great but you end up paying for it in compile times. I hope the Rust community starts to see compile times as a more pressing concern and designing crates with that in mind.
This is my experience. Unless we’re talking about ultra tiny workspaces, any small to medium sized project hurts, especially compared to build times with other tool chains like the Go compiler.
I think zig does offer more safety than the author lets on. It definitely offers flexibility, a good bootstrap story, and will work to make memory mistakes visible to the programmer.
Rust is better for critical stuff than C. Zig seems nicer to use, to me, than both.
I think Zig’s approach to memory safety is not reasonable for any new system language. Temporal memory errors is about the single biggest problem large code bases, and Zig is the only new language that does not make any real attempt to fix this. We’ve got decades of evidence demonstrating that manual memory management is fundamentally unsafe as it requires no mistakes by a developer, but is also infeasible to analyze without language level support for tracking object life time. There are numerous ways a language can do this, from zero runtime overhaad (rust w/borrow check) through to a full tracing GC - though I believe the latter simply is not feasible in kernel level code so seems impractical for a general “systems language”.
I know whenever I bring up temporal safety someone chimes in with references to fuzzing and debug allocators, but we know that those are not sufficient: C/C++ projects have access to fuzzers and debug allocators, and the big/security sensitive ones aggressively fuzz, and use aggressive debug allocators while doing so, and those do not magically remove all security bugs. In fact a bunch of things the “debug” allocators do in Zig are already the default behaviour of the macOS system allocator for example, and the various webkit and blink allocators are similarly aggressive in release builds.
I think it depends on your priorities and values and what you’re trying to do. Rust does not guarantee memory safety, only safe Rust does. If what you’re doing requires you to use a lot of unsafe Rust, that’s not necessarily better than something like Zig or even plain C. In fact, writing unsafe Rust is more difficult than writing C or Zig, and the ergonomics can be terrible. See for why wlroots-rs was abandoned for example.
From my experience, I’ve found Rust to do great when no unsafe code or complex lifetimes are involved, such as writing a web server. It also works great when you can design the memory model with lifetimes and the borrow checker in mind. When you can’t, or you have to deal with an existing memory model (such as when interacting with C), it can fare very poorly in terms of ergonomics and be more difficult to maintain.
And that’s great, because people are writing very small amounts of code in unsafe Rust.
I’ve also implemented a programming language VM in Rust and found that you can usually define good abstractions around small amounts of unsafe code. It isn’t always easy (and wlroots legitimately seems like a hard case, but I don’t have enough domain expertise to confirm) but given how rarely you need to do it it seems manageable.
But I think focusing on a few hard cases is missing the forest for the trees. If you can write the vast majority of code in a pleasant, safe and convenient language it very quickly makes the small part that you do need to be unsafe or fight lifetimes worth it.
I would love to see a language that does even better and handles more cases. But until that comes along I would strongly prefer to use Rust which helps me write correct code than the alternatives just because in a few sections it doesn’t help me.
That’s fine, but we don’t need to have only one be all end all programming language. We can have different options for different situations, and that’s what Zig and other languages provide in contrast to Rust.
I really enjoy using Zig more than using Rust. I have to work with existing C codebases, and Rust seems to hold C code at arms length, where Zig will let me integrate more fluidly. Both Rust and Zig seem like great replacements for C, but Zig seems to play more nicely with existing software.
I enjoy using Zig a lot more than Rust as well. While Rust has many fancy features, I am frequently fighting with them. And I’m not talking about the borrow checker, but rather limitations in the type system, tooling, or features being incomplete.
But all that aside I think where I would use Zig in a commercial setting is limited, only because it is too easy to introduce memory use issues, and I don’t think the language is doing enough, at least yet, to prevent them.
This. I’ve been using Rust way more than Zig, and even used Rust professionally, for about 3 years, and I have to say Zig really gets a lot right. Zig’s memory safety system may not be on-par with Rust but that’s because it opts to make unsafe code easier to write without bugs. Traversing the AST (which is available via std.zig.Ast), you can get the same borrow checker system in place with Zig 🙂 Now that Zig is maturing with the bootstrapping issue figured out, I bet we’ll see this sooner than later, as it’s a real need for some.
Zig is builder friendly. You ever try C interop with Rust? Rust is enterprise friendly.
But lifetimes in rust are based on the control-flow graph not on the AST. Lexical lifetimes got replaced by non-lexical lifetimes back in 2018. Also there are lifetimes annotations, how would you get those?
I know neither zig nor rust, and do not plan to, but: usually, a control-flow graph (along with more complicated structures besides) is built from an ast; is there some reason you can not do this for zig?
It’s like the “draw two circles and then just draw the rest of the owl” meme.
You can make CFG from AST easily, but then liveness and escape analysis is the hard part that hasn’t been solved for half a century, and even Rust needs the crutch of restrictive and manual lifetime annotations.
Do you have more reading on this? Because my understanding is that Rust’s borrow checker isn’t possible to infer all lifetimes from the AST alone. Hence it’s infamous lifetime annotations (
'a). While Rust inference is generally pretty good, there are definitely plenty of scenarios I’ve run into where the inferred lifetimes are wrong, or ambiguous in which both cases require the additional syntax.It would absolutely be a learning experience for it to be explained to me why :) I don’t see why not: the AST describes the full program.
If you could prove the safety (or lack thereof) of mutable aliasing entirely through syntactic analysis, the issues we have with C would be a lot less glaring. Having a parser in the standard library is incredibly cool, as is
comptime, but you still have to build a static verifier for borrow checking your language/runtime of choice.I do think there’s a good opportunity for conventions and libraries to emerge in the Zig ecosystem that make writing code that’s “safe by default” a lot easier. You could absolutely force allocations and lookups in your code to use something like a slotmap to prevent race-y mutation from happening behind the scenes.
Unfortunately that confidence doesn’t extend to 3rd-party libraries. Trying to transparently interface with arbitrary C code (which may in turn drag in embedded assembler, Fortran, or who knows what) means bringing all of the memory safety issues of C directly into your code base. See: literal decades of issues in image format decoding libraries, HTML parsers, etc. that get reused in nominally “memory-safe” languages.
All of which is precisely why Rust does keep C/C++ at arm’s length, no matter how appealing trivial interop would be. Mozilla, MS, and Google/Android have custodianship over hundreds of millions of lines of C++ code, and yet they also understand and want the safety benefits Rust provides enough to deal with the more complicated and toilsome integration story vs. something like Zig or D.
Right right, I made a big mistake to say “same”. Others mention annotations - those could be created in comments or in novel ways Im sure ! We’re starting to see things like CheckedC so yea, my thoughts still stand…
But there isn’t enough information in the AST without annotations. Else you wouldn’t need mut and lifetime annotations in rust. The reason for those annotations is that the compiler couldn’t proof that the safe rust code doesn’t introduce new memory bugs (safe code can still trigger memory bugs introduced by unsafe code) without those annotations. The AST describes the whole program, that doesn’t mean that every property that is true for your code can be automatically inferred by an general algorithm. A well known example would be if your program computes in a finite amount of time.
Doesn’t zig still require manually freeing memory at the “right time”?
I think it’s important to note that when you do find a case that generates invalid code, it is a comptime error. So, you can guarantee all usages of your function are valid if the project compiles.
This can also be applied to platform-specific blocks:
If you were to call this function on an unsupported platform, it would be a compile error. If you don’t, you can compile the project fine. I think this property of the language is really cool, and I think the Zig standard library uses it for a few different things like skipping tests on certain platforms (others compilers embed this as metadata in the file).
It did for me at first - why on earth wouldn’t you want all of your code checked?? - but I think I’m being converted. Partial evaluation like this just feels so powerful. I’ve even ended up emulating these ideas in Java for work.
C++ templates have the same property that they are checked at compile-time but on use-sites, and I think this is widely understood to have turned into a usability hell, which is basically the reason for the work on concepts. For example, it is easy for developers of comptime logic in libraries to miss errors that happen on their clients, or to end up with code that is effectively not checked/tested inside the library. I’m not saying that Zig comptime has the exact same downsides as C++ templates (it has the benefit of being a properly designed compilation-level language, rather than something that grew organically in different directions than the base language), but I also suspect that this class of issues is a problem at large and I share the blog post’s reservations about running wild with the idea. On the other hand, it is true that the benefits in terms of reduced complexity of the language (compared to a much richer time system) are significant. Language design is a balancing act.
Uhu, with C++/Zig style generics, it’s very hard for the user (or the author) to state precisely what the API of an utility is, which should make SemVer significantly harder in practice.
The second problem is tooling: when you have
T: type, there isn’t much an IDE can correctly do with completions, goto definition, etc. Some heuristics are possible (eg, looking at the example instantiations), but that’s hard to implement (precise solutions are simpler).I agree, there was a discussion along these lines in a previous post about Zig.
Regarding tooling: Smalltalk/Squeak/Pharo people would say that you don’t need static analysis to have good tooling, as long as you rephrase the tooling to be about exploring the state of the program as it runs, rather than without running it. I could see how this might possibly work with Zig – collect global traces of successful generic instantiations (for the testsuite in the project, or for your whole set of installed packages, or over the whole history of your past usage of the code…) and show their shared structure to the user.
This is something that we already did with the old implementation of autodoc and that plan to eventually also support in the new version as something that you should expect to be able to rely on when browsing documentation for a package.
One advantage Zig has over plain C++ templates is that you can do arbitrary comptime checks and emit clean, precise error messages. For example, instead of just blowing up when you instantiate with a type that doesn’t have a given method, you can check for it and fail compilation with
type foo must have bar(x: i32) method. You can even implement something akin to concepts in userspace.Yeah but I have been burned enough times by pulling out code that compiled 6 months ago to discover, oops, it no longer builds. Even in Rust. There are few things more frustrating. “I need to fix this bug that I just found in an old prod system, it’s a 15 minute fix and and oops now I get to spend three hours trying to figure out which dependency three levels down in my dep tree didn’t follow semver and broke its API.” Or which file doesn’t build with a newer compiler’s interpretation of C++17, or which Magical Build Flag didn’t get saved in the build docs, or why the new binary needs to load
libfuckyou.so.0.7.3but the old one doesn’t, or…So if possible I would prefer to not add yet more ways of scattering around hidden build landmines in large and complex programs. But as I said, I need to play with it more and get a better feel for how it actually works out in practice.
My first thought was the double 0 problem you usually get from ones complement, but this actually reserves a sentinel “Nan” value.
The problem there is that part of what makes twos complement so great is that the representation also makes basic addition, subtraction, shifts, etc very efficient to implement.
The other problem here is the gratuitous and unnecessary use of undefined behavior. UB is bad, and is meant to exist for things that cannot be specified, e.g using an object after it has been freed, accessing memory through the wrong type, etc. obviously the C and C++ committees decided to abandon that and create numerous security vulnerabilities over the years. If you don’t want to constrain the implementation the correct course of action is unspecified behavior. That is the implementation can choose what to do, but it must always do that, the compiler can optimize according to that but it can’t go “that’s UB so I’ll just remove it”, which I also think is a bogus interpretation of what UB means but that’s what we have.
The desire to say “NaN is UB” means that you can’t test for NaN, because by definition it isn’t defined so the compiler can remove any NaN checks you insert. You might say “we’ll add a built in with defined behaviour”, but that doesn’t help because you gave the compiler the option to remove intermediate NaNs. Eg if(__builtin_isnan(0/0)) doesn’t trip because 0/0 can’t produce a NaN (that would be UB), so therefore the value passed to isnan can’t be NaN, so therefore isnan is false. This is obviously a dumb example, but it’s fundamentally the behavior you see for anything UB. Imagine isnan(x), which looks reasonable, but if x got it’s value from any arithmetic in the scope of the optimizer (which can include inlining) then isnan is always false.
It’s not giving up the ease of two’s complement addition core operation (ie full adders) as I understood it (after re-reading, it’s late, I on first skim thought they were advocating sign-magnitude) but rather to reserve ~0 as a not-numerically-valid NaN. However efficiency would still be lost of course in that the implementation would need to have checks for NaNness of input, and the overflow/underflow detection would be more complex as well. But this only matters if we decide to build processors this way to support the representation/interpretation, otherwise it’s going to be software layers above.
This is what I understood and how I still interpret it on re-read. The clearest indication I found is this phrase:
10000000would be-127using ones-complement but-0using sign-and-magnitude.I made a mistake in my previous post, it’s not ~0 that is the reserved value, ~0 is still the representation of -1. It is the interpretation of {msb=1, others=0} that is changed to INT_NAN rather than a valid number, but my interpretation is that beyond that, the representation remains two’s complement-like (4 bit example):
I see it now. Is this a novel idea? Does it have a name?
I think only arithmetic is meant to be UB on INT_NAN. You can still check for it, otherwise it can’t be used as a sentinel value.
The problem isn’t the NAN value being tested it is the preceding arithmetic, because of what UB means. Take
Because arithmetic producing INT_NAN is UB, then the compiler is free to assume that no arithmetic produce it, which gives the compiler the ability to convert the above into:
doThingB()
Because the product “cannot be NAN” the isnans can be removed.
I think the idea is that you’d only ever assign INT_NAN when you want to set the sentinel value.
You wouldn’t be checking for INT_NAN after arithmetic: you’d only check before, to see if you have an empty optional. It’s not true that the compiler can remove every NaN check, only the ones immediately following arithmetic. And those don’t make sense in this model since there’s nothing meaningful you could do at that point. If you’re representing
optional<int>, there’s no reason to check if your optional is suddenly empty after an arithmetic operation. If you want to prevent overflow from resulting in a NaN, you can insert a precondition before doing any arithmetic.No - that handles a parameter passed to you, it does not handle general arithmetic, unless you pre-check everything, e.g:
would become
This is assuming of course the optional case, but this runs into the standard footgun that comes from the frankly stupid at this point belief that having arithmetic edge cases be undefined rather than unspecified, e.g.
In a world where arithmetic on nan is undefined behaviour, as the author specified, the isnan() and printf can be removed. Because we’ve said “arithmetic on nan is undefined behaviour” rather than “unspecified”, the compiler can do the following logic:
So we get the above code lowered to
This is how compilers reason about UB. It’s not “I have proved this path leads to UB so that’s an error”, it’s instead the much worse “I have proved that this path leads to UB, therefore this path cannot happen”.
Yes, this logic has led to security bugs, in numerous projects, and yet language committees continue to assign “UB” to specifiable behaviour.
Side question for the author, what’s the motivation for schemaless in a video game context?
A very easy way to create serialisers/deserialisers in JS is to write functions which create or consume a plain object representation, and then use JSON to create a string representation. So there’s no better reason than that it’s just easier. The alternatives seem to be to add a big serialisation library and complicate the build process with code generation from schema files , or write an ad-hoc schema-style format where the “schema” is represented as the state control flow in the code. Making a more efficient schemaless format with the JSON object model just seemed easier.
The machine does nothing else than what a human tells it to do. So ruthlessness of a machine is actually ruthlessness of humans.
Oddly, the author acknowledges this, but then doesn’t really explore the implications:
…which is disappointing, since that opens up an opportunity to talk about how computing systems could be designed to be more humane, which is immediately squandered in favor of advocating for crypto-“anarchism” whose success is “beyond dispute” and whose ruthlessness is “clearly” good.
Which is to say, I agree with you. Furthermore, it’s not just computing itself which obscures the underlying human ruthlessness or integrity (although in a mundane, banality-of-evil sense, it definitely does). But, at a deeper level, what obscures the human element is the belief that computing is inherently anything.
Absolutely. There are plenty of examples of humane systems:
These all exist because someone looked at the existing MVP, decided it wasn’t good enough, and designed (or enforced) something more humane. All of us, as users and creators, “simply” need to encourage the proliferation of such systems.
I’d agree with you that exploring the human angle is key to fixing this problem. This post is tagged “philosophy”, but I’d as soon tag it “religion” (if such a tag existed and if Lobsters was the place for such debates) because it and these comments are really speaking about our view of humans and morality. Where the author sees “ruthlessness” from a machine following its programming in the subway, others may see life giving opportunity in the biomedical field, for example.
I don’t believe this can be true because a human, potentially a warped and terrible one, had to intentionally create the machine. In other words, if someone made a machine to destroy humanity, they were doing so intentionally and they are worse than the machine itself because they understand what humanity is, the machine does not (nod to the sentience debate).
This hypothetical question is the perfect example that this is really a religious debate, not just an ethics in engineering one. 1-2 billion people in the world would answer that question with “Yes, His name is Jesus.” We can’t consider integrity in machines without dealing with integrity in the humans that create and use them and we can’t deal with that without first knowing where we each come from in our beliefs. I sincerely am not posting this to start a religious flame war, merely point out that machines aren’t what is in question in this post.
I just want to say that these types of questions and discussions are definitely philosophical, even if people turn to religion for the answer.
Ethics is absolutely a branch of philosophy.
Blame not the tools but the systems which created them and in which they are used.
What surrprised me about Tainter’s analysis (and I haven’t read his entire book yet) is that he sees complexity as a method by which societies gain efficiency. This is very different from the way software developers talk about complexity (as ‘bloat’, ‘baggage’, ‘legacy’, ‘complication’), and made his perspective seem particularly fresh.
I don’t mean to sound dismissive – Tainter’s works are very well documented, and he makes a lot of valid points – but it’s worth keeping in mind that grand models of history have made for extremely attractive pop history books, but really poor explanations of historical phenomena. Tainter’s Collapse of Complex Societies, while obviously based on a completely different theory (and one with far less odious consequences in the real world) is based on the same kind of scientific thinking that brought us dialectical materialism.
His explanation of the fall of the evolution and the eventual fall of the Roman Empire makes a number of valid points about the Empire’s economy and about some of the economic interests behind the Empire’s expansion, no doubt. However, explaining even the expansion – let alone the fall! – of the Roman Empire strictly in terms of energy requirements is about as correct as explaining it in terms of class struggle.
Yes, some particular military expeditions were specifically motivated by the desire to get more grain or more cows. But many weren’t – in fact, some of the greatest Roman wars, like (some of) the Roman-Parthian wars, were not driven specifically by Roman desire to get more grains or cows. Furthermore, periods of rampant, unsustainably rising military and infrastructure upkeep costs were not associated only with expansionism, but also with mounting outside pressure (ironically, sometimes because the energy per capita on the other side of the Roman border really sucked, and the Huns made it worse on everyone). The increase of cost and decrease in efficiency, too, are not a matter of half-rational historical determinism – they had economic as well as cultural and social causes that rationalising things in terms of energy not only misses, but distorts to the point of uselessness. The breakup of the Empire was itself a very complex social, cultural and military story which is really not something that can be described simply in terms of the dissolution of a central authority.
That’s also where this mismatch between “bloat” and “features” originates. Describing program features simply in terms of complexity is a very reductionist model, which accounts only for the difficulty of writing and maintaining it, not for its usefulness, nor for the commercial environment in which it operates and the underlying market forces. Things are a lot more nuanced than “complexity = good at first, then bad”: critical features gradually become unneeded (see Xterm’s many emulation modes, for example), markets develop in different ways and company interests align with them differently (see Microsoft’s transition from selling operating systems and office programs to renting cloud servers) and so on.
Of course. I’m long past the age where I expect anyone to come up with a single, snappy explanation for hundreds of years of human history.
But all models are wrong, only some are useful. Especially in our practice, where we often feel overwhelmed by complexity despite everyone’s best efforts, I think it’s useful to have a theory about the origins and causes of complexity, even if only for emotional comfort.
Indeed! The issue I take with “grand models” like Tainter’s and the way they are applied in grand works like Collapse of Complex Societies is that they are ambitiously applied to long, grand processes across the globe without an exploration of the limits (and assumptions) of the model.
To draw an analogy with our field: IMHO the Collapse of… is a bit like taking Turing’s machine as a model and applying it to reason about modern computers, without noting the differences between modern computers and Turing machines. If you cling to it hard enough, you can hand-wave every observed performance bottleneck in terms of the inherent inefficiency of a computer reading instructions off a paper tape, even though what’s actually happening is cache misses and hard drives getting thrashed by swapping. We don’t fall into this fallacy because we understand the limits of Turing’s model – in fact, Turing himself explicitly mentioned many (most?) of them, even though he had very little prior art in terms of alternative implementations, and explicitly formulated his model to apply only to some specific aspects of computation.
Like many scholars at the intersections of economics and history in his generation, Tainter doesn’t explore the limits of his model too much. He came up with a model that explains society-level processes in terms of energy output per capita and upkeep cost and, without noting where these processes are indeed determined solely (or primarily) by energy output per capita and upkeep post, he proceeded to apply it to pretty much all of history. If you cling to this model hard enough you can obviously explain anything with it – the model is explicitly universal – even things that have nothing to do with energy output per capita or upkeep cost.
In this regard (and I’m parroting Walter Benjamin’s take on historical materialism here) these models are quasi-religious and are very much like a mechanical Turk. From the outside they look like history masterfully explaining things, but if you peek inside, you’ll find our good ol’ friend theology, staunchly applying dogma (in this case, the universal laws of complexity, energy output per capita and upkeep post) to any problem you throw its way.
Without an explicit understanding of their limits, even mathematical models in exact sciences are largely useless – in fact, a big part of early design work is figuring out what models apply. Descriptive models in humanistic disciplines are no exception. If you put your mind to it, you can probably explain every Cold War decision in terms of Vedic ethics or the I Ching, but that’s largely a testament to one’s creativity, not to their usefulness.
Not to mention all the periods of rampant rising military costs due to civil war. Those aren’t wars about getting more energy!
Sure. This is all about a framing of events that happened; it’s not predictive, as much as it is thought-provoking.
Thought-provoking, grand philosophy was certainly a part of philosophy but became especially popular (some argue that it was Nathaniel Bacon who really brought forth the idea of predicting progress) during the Industrial Era with the rise of what is known as the modernist movement. Modernist theories often differed but frequently shared a few characteristics such as grand narratives of history and progress, definite ideas of the self, a strong belief in progress, a belief that order was superior to chaos, and often structuralist philosophies. Modernism had a strong belief that everything could be measured, modeled, categorized, and predicted. It was an understandable byproduct of a society rigorously analyzing their surroundings for the first time.
Modernism flourished in a lot of fields in the late 19th early 20th century. This was the era that brought political philosophies like the Great Society in the US, the US New Deal, the eugenics movement, biological determinism, the League of Nations, and other grand social and political engineering ideas. It was embodied in the Newtonian physics of the day and was even used to explain social order in colonizing imperialist nation-states. Marx’s dialectical materialism and much of Hegel’s materialism was steeped in this modernist tradition.
In the late 20th century, modernism fell into a crisis. Theories of progress weren’t bearing fruit. Grand visions of the future, such as Marx’s dialectical materialism, diverged significantly from actual lived history and frequently resulted in a magnitude of horrors. This experience was repeated by eugenics, social determinism, and fascist movements. Planck and Einstein challenged the neat Newtonian order that had previously been conceived. Gödel’s Incompleteness Theorem showed us that there are statements we cannot evaluate the validity of. Moreover many social sciences that bought into modernist ideas like anthropology, history, and urban planning were having trouble making progress that agreed with the grand modernist ideas that guided their work. Science was running into walls as to what was measurable and what wasn’t. It was in this crisis that postmodernism was born, when philosophers began challenging everything from whether progress and order were actually good things to whether humans could ever come to mutual understanding at all.
Since then, philosophy has mostly abandoned the concept of modeling and left that to science. While grand, evocative theories are having a bit of a renaissance in the public right now, philosophers continue to be “stuck in the hole of postmodernism.” Philosophers have raised central questions about morality, truth, and knowledge that have to be answered before large, modernist philosophies gain hold again.
I don’t understand this, because my training has been to consider models (simplified ways of understanding the world) as only having any worth if they are predictive and testable i.e. allow us to predict how the whole works and what it does based on movements of the pieces.
You’re not thinking like a philosopher ;-)
Neither are you ;-)
https://plato.stanford.edu/entries/pseudo-science/ https://plato.stanford.edu/entries/popper/
Models with predictive values in history (among other similar fields of study, including, say, cultural anthropology) were very fashionable at one point. I’ve only mentioned dialectical materialism because it’s now practically universally recognized to have been not just a failure, but a really atrocious one, so it makes for a good insult, and it shares the same fallacy with energy economic models, so it’s a doubly good jab. But there was a time, as recent as the first half of the twentieth century, when people really thought they could discern “laws of history” and use them to predict the future to some degree.
Unfortunately, this has proven to be, at best, beyond the limits of human understanding and comprehension. This is especially difficult to do in the study of history, where sources are imperfect and have often been lost (case in point: there are countless books we know the Romans wrote because they’re mentioned or quoted by ancient authors, but we no longer have them). Our understanding of these things can change drastically with the discovery of new sources. The history of religion provides a good example, in the form of our understanding of Gnosticism, which was forever altered by the discovery of the Nag Hammadi library, to the point where many works published prior to this discovery and the dissemination of its text are barely of historical interest now.
That’s not to say that developing a theory of various historical phenomenons is useless, though. Even historical materialism, misguided as they were (especially in their more politicized formulations), were not without value. They forced an entire generation of historians to think more about things that they never really thought about before. It is certainly incorrect to explain everything in terms of class struggle, competition for resources and the means of production, and the steady march from primitive communism to the communist mode of production – but it is also true that competition for resources and the means of production were involved in some events and processes, and nobody gave much thought to that before the disciples of Marx and Engels.
This is true here as well (although I should add that, unlike most materialistic historians, Tainter is most certainly not an idiot, not a war criminal, and not high on anything – I think his works display an unhealthy attachment for historical determinism, but he most certainly doesn’t belong in the same gallery as Lenin and Mao). His model is reductionist to the point where you can readily apply much of the criticism of historical materialism to it as well (which is true of a lot of economic models if we’re being honest…). But it forced people to think of things in a new way. Energy economics is not something that you’re tempted to think about when considering pre-industrial societies, for example.
These models don’t really have predictive value and they probably can’t ever gain one. But they do have an exploratory value. They may not be able to tell you what will happen tomorrow, but they can help you think about what’s happening today in more ways than one, from more angles, and considering more factors, and possibly understand it better.
That’s something historians don’t do anymore. There was a period where people tried to predict the future development of history, and then the whole discipline gave up. It’s a bit like what we are witnessing in the Economics field: there are strong calls to stop attributing predictive value to macroeconomic models because after a certain scale, they are just over-fitting to existing patterns, and they fail miserably after a few years.
Well, history is not math, right? It’s a way of writing a story backed by a certain amount of evidence. You can use a historical model to make predictions, sure, but the act of prediction itself causes changes.
(OP here.) I totally agree, and this is something I didn’t explore in my essay. Tainter doesn’t see complexity as always a problem: at first, it brings benefits! That’s why people do it. But there are diminishing returns and maintenance costs that start to outstrip the marginal benefits.
Maybe one way this could apply to software: imagine I have a simple system, just a stateless input/output. I can add a caching layer in front, which could win a huge performance improvement. But now I have to think about cache invalidation, cache size, cache expiry, etc. Suddenly there are a lot more moving parts to understand and maintain in the future. And the next performance improvement will probably not be anywhere near as big, but it will require more work because you have to understand the existing system first.
I’m not sure it’s so different.
A time saving or critically important feature for me may be a “bloated” waste of bits for somebody else.
In Tainter’s view, a society of subsistence farmers, where everyone grows their own crops, makes their own tools, teaches their own children, etc. is not very complex. Add a blacksmith (division of labour) to that society, and you gain efficiency, but introduce complexity.
I think the biggest benefit of this is that - if I’m not mistaken - the new UI will be compatible with Wayland (as far as I know, Fleet’s interface already is Wayland compatible). This could finally allow me to move away from X for good.
Fleet doesn’t seem to run natively on Wayland, at least not by default.
Note the issue isn’t just running in a Wayland environment, which IDEA already does, but running natively as a Wayland client (ie, no XWayland).
Oh, weird, I had read somewhere that Fleet’s interface was based on Skia and that it’d be Wayland ready by simply switching the backend.
As a Windows user, what does this functionally mean? I only ever interact with Linux through a terminal.
In addition to fractional scaling, it modernizes and simplifies the whole graphics stack in a way akin to something like Aero when whatever major version of windows did that. Features I appreciate are:
I think for you it wouldn’t make a whole lot of difference, unless you decide to run the Linux version of your JetBrains product using an X server to display the window (which I wouldn’t recommend considering there’s native Windows versions and their products seem to have excellent WSL support).
For a Linux user… it’s kind of a mixed bag. There’s some security benefits to Wayland, but mostly the reason I see this as a good thing is that most desktop environment seem to be moving X into maintenance mode, and doing all new interesting development exclusively on Wayland. Gesture support and fractional scaling come to mind.
In practical sense fractional scaling doesn’t really work for X11 while it’s OK on Wayland.
There are multiple points here I disagree with:
This distinction doesn’t really matter in a language with first-class lambdas. If you want to unlock a mutex at the end of a loop iteration with Go, create and call a lambda in the loop that uses
deferinternally.But constructors can. If you implement a
Deferclass to use RAII, it takes a lambda in the constructor and calls it in the destructor.I’m not sure I buy that argument, given that the code in
deferis almost always calling another function. The code inside the constructor for the object whose cleanup you aredefering is also not visible in the calling function.The point is that as a reader of zig, you can look at the function and see all the code which can be executed. You can see the call and breakpoint that line. As a reader of c++, it’s a bit more convoluted to breakpoint on destructors.
As someone that works daily with several hundred lines functions, that sounds like a con way more than a pro.
This can work sometimes, but other times packing pointers in a struct just so you can drop it later is wasteful. This happens a lot with for example the Vulkan API where a lot of the
vkDestroy*functions take multiple arguments. I’m a big fan of RAII but it’s not strictly better.At least in C++, most of this all goes away after inlining. First the constructor and destructor are both inlined in the enclosing scope. This turns the capture of the arguments in the constructor into local assignments in a structure in the current stack frame. Then scalar replacement of aggregates runs and splits the structure into individual
allocas in the first phase and then into SSA values in the second. At this point, the ‘captured’ values are just propagated directly into the code from the destructor.Note that Go uses function scope for defer. So this will actually acquire locks slowly then release them all at the end of function. This is very likely not what you want and can even risk deadlocks.
Is a lambda not a function in Go? I wouldn’t expect
deferin a lambda to release the lock at the end of the enclosing scope, because what happens if the lambda outlives the function?Sorry, I misread what you said. I was thinking
defer func() { ... }()notfunc() { defer ... }().Sorry, I should have put some code - it’s much clearer what I meant from your post.
The first point is minor, and not really changing the overall picture of leaking by default.
Destruction with arguments is sometimes useful indeed, but there are workarounds. Sometimes you can take arguments when constructing the object. In the worst case you can require an explicit function call to drop with arguments (just like
deferdoes), but still use the default drop to either catch bugs (log or panic when the right drop has been forgotten) or provide a sensible default, e.g. delete a temporary file iftemp_file.keep()hasn’t been called.Automatic drop code is indeed implicit and can’t be grepped for, but you have to consider the trade-off: a forgotten
deferis also invisible and can’t be grepped for either. This is the change in default: by default there may be drop code you may not be aware of, instead of by default there may be a leak you may not be aware of.Yes, more than useful:
I think the right solution is explicit destructors: Instead of the compiler inserting invisible destructor calls, the compiler fails if you don’t. This would be a natural extension to an explicit language like C – it would only add safety. Not only that: It fits well with
defertoo – syntactic sugar doesn’t matter, because it just solves the «wrong default» problem. But more than anything, I think it would shine in a language with lifetimes, like Rust, where long lived references are precisely what you don’t want to mess with.You could run an anonymous function within a loop in Go, just to get the per-loop defer. Returning a value in a defer is also possible.
Good writeup.
I think there’s one more potential attack that’s missing: a vulnerability or misconfiguration in etcd (or wherever else you store your secrets) that allows reading without requiring root access or physical access to the machine. In that scenario, encrypted secrets can provide another layer of security assuming the decryption is done via a separate mechanism.
I also don’t think “most people aren’t using feature X of Vault” is that strong an argument. You can’t dismiss a tool by insisting people aren’t using it correctly.
Yeah the dismissiveness of vault is mainly just me ranting. Maybe should have been an aside or footnote because the argument doesn’t rely on people misusing Vault. A properly configured Vault instance is what I ultimately compared to plain Kubernetes Secrets.
I agree that Vault is a complicated beast (I used to manage Vault at a previous employer) but USP for Vault must be the dynamic secret with TTLs right? So even if you could read the secret from the RAMdisk on the node/pod it would not be usable unless you timed it so that you read it exactly before the service read it but after the injector well injected it.
Are we talking about https://www.hashicorp.com/resources/painless-password-rotation-hashicorp-vault ?
My understanding is that while Vault can perform automatic password rotation, it can’t e.g. configure Redis or MySQL to change the password automatically. You could build something that does that for every secret-consuming application, but now vault is relegated to being a random password generator and again could be replaced with plain kubernetes secrets, /dev/urandom, and a cronjob.
I think the real value from Vault is the policies, not just storage. If a deployment is not taking advantage of that, then yes it’s no better than etcd or anything else.
How will you ensure that you can still build zig from sources in the future?
By forever maintaining two implementations of the compiler - one in C, one in Zig. This way you will always be able to bootstrap from source in three steps:
.ccode. Use system C compiler to build from this.ccode. We call this stage2.https://github.com/ziglang/zig-bootstrap
I’m curious, is there some reason you don’t instead write a backend for the Zig implementation of the compiler to output C code? That seems like it would be easier than maintaining an entirely separate compiler. What am I missing?
That is the current plan as far as I’m aware
The above post says they wanted two separate compilers, one written in C and one in Zig. I’m wondering why they just have one compiler written in Zig that can also output C code as a target. Have it compile itself to C, zip up the C code, and now you have a bootstrap compiler that can build on any system with a C compiler.
In the above linked Zig Roadmap video, Andrew explains that their current plan is halfway between what you are saying and what was said above. They plan to have the Zig compiler output ‘ugly’ C, then they will manually clean up those C files and version control them, and as they add new features to the Zig source, they will port those features to the C codebase.
I just watched this talk and learned a bit more. It does seem like the plan is to use the C backend to compile the Zig compiler to C. What interests me though is there will be a manual cleanup process and then two separate codebases will be maintained. I’m curious why an auto-generated C compiler wouldn’t be good enough for bootstrapping without manual cleanup.
Generated source code usually isn’t considered to be acceptable from an auditing/chain of trust point of view. Don’t expect the C code generated by the Zig compiler’s C backend to be normal readable C, expect something closer to minified js in style but without the minification aspect. Downloading a tarball of such generated C source should be considered equivalent to downloading an opaque binary to start the bootstrapping process.
Being able to trust a compiler toolchain is extremely important from a security perspective, and the Zig project believes that this extra work is worth it.
That makes a lot of sense! Thank you for the clear and detailed response :)
It would work fine, but it wouldn’t be legitimate as a bootstrappable build because the build would rely on a big auto-generated artifact. An auto-generated artifact isn’t source code. The question is: what do you need to build Zig, other than source code?
It could be reasonable to write and maintain a relatively simple Zig interpreter that’s just good enough to run the Zig compiler, if the interpreter is written in a language that builds cleanly from C… like Lua, or JavaScript using Fabrice Bellard’s QuickJS.
Except that you can’t bootstrap C, so you’re back where you started?
The issue is not to be completely free of all bootstrap seeds. The issue is to avoid making new ones. C is the most widely accepted and practical bootstrap target. What do you think is a better alternative?
C isn’t necessarily a bad choice today, but I think it needs to be explicitly acknowledged in this kind of discussion. C isn’t better at being bootstrapped than Zig, many just happen to have chosen it in their seed.
A C compiler written in Zig or Rust to allow bootstrapping old code without encouraging new C code to be written could be a great project, for example.
This is in fact being worked on: https://github.com/Vexu/arocc
Or do like Golang. For bootstrap you need to:
Build the Zig compiler to Wasm, then run it to cross-compile the new compiler. Wasm is forever.
I certainly hope that’s true, but in reality wasm has existed for 5 years and C has existed for 50.
The issue is building from maintained source code with a widely accepted bootstrapping base, like a C compiler.
The Zig plan is to compile the compiler to C using its own C backend, once, and then refactor that output into something to maintain as source code. This compiler would only need to have the C backend.
I mean, if it is, then it should have the time to grow some much needed features.
https://dl.acm.org/doi/10.1145/3426422.3426978
It’s okay if you don’t know because it’s not your language, but is this how Go works? I know there’s some kind of C bootstrap involved.
The Go compiler used to be written in C. Around 1.4 they switched to a Go compiler written in Go. If you were setting up an entirely new platform (and not use cross compiling), i believe the recommended steps are still get a C compiler working, build Go 1.4, then update from 1.4 to latest.
How do we build C compilers from source?
Bootstrapping a C compiler is usually much easier than bootstrapping a chain of some-other-language compilers.
Only if you accept a c compiler in your bootstrap seed and don’t accept a some-other-language compiler in your seed.
Theoretically. But from a practical point of view? Yes, there are systems like Redox (Rust), but in most cases the C compiler is an inevitable piece of the puzzle (the bootstrapping chain) when building an operating system. And in such cases, I would (when focused on simplicity) rather prefer a language that depends just on C (that I already have) instead of a sequence of previous versions of its own compilers. (and I say that as someone, who does most of his work in Java – which is terrible from the bootstrapping point-of-view)
However, I do not object much against the dependence on previous versions of your compiler. It is often the way to go, because you want to write your compiler in a higher language instead of some old-school C and because you create a language and you believe in its qualities, you use it also for writing the compiler. What I do not understand is why someone (not this particular case, I saw this pattern before many times) present the “self-hosted” as an advantage…
The self-hosted Zig compiler provides much faster compile times and is easier to hack, allowing language development to move forward. In theory the gains could be done in a different language, but some of the kind of optimizations used are exactly the kind of thing Zig is good at. See this talk for some examples: https://media.handmade-seattle.com/practical-data-oriented-design/.
But you could make a C compiler (or a C interpreter) from scratch relatively easily.
It’s possible to have an “async” system for systems programming that isn’t colored. Zig solved this problem. In the case of zig, I can write a single function to be called from the Erlang virtual machine’s FFI either in an async fashion to use it’s the vm’s preemptibility, or I can call the same function to be run in its own thread (this is non-async), so you can explore the tradeoffs without having to write your function twice.
I really like what Andrew did with color-free async in Zig, but it’s not a panacea. There are limitations. You can’t use multiple different event loops (one blocking, one non-blocking) in the same binary, for example. Unless I’ve misunderstood something.
You absolutely can use multiple event loops in the same binary. It isn’t obvious how to do it cleanly, if you ask me, though, or I haven’t hit upon “the correct pattern” for it
I’m not convinced that solves a real language problem since the caller has to opt in to the desired behavior and the callee has to support both modes of operation. It doesn’t allow you to just call an async function from a sync context and have it automatically work. Zig allows you to put both versions in one function, but now you don’t know which async functions you can safely call from sync without reading the documentation or looking at the implementation. Is there something I’m missing here?