I wish people would stop crapping on Han unification if they can’t read Hanzi/Kanji. It is totally appropriate that those characters in the article are the same. If they were if different, it would be like 4 joined up and 4 in two strokes being separated, or 7 with a slash in the middle being separated. They’re different ways to write the same thing, and have all been used in Japan within the last 100 years.
There were serious issues. Unicode has eventually added dedicated code points undoing worst cases of the unification.
It’s not just about readability, but a cultural issue. People can read a foreign way of writing a character, but it’s not how they write it.
China and Japan culturally care a lot about calligraphy. To them it’s not just a font. Number of strokes does matter, even the direction of strokes is significant. Japan even has a set of traditional variants of characters that are used only in people’s names.
As a Chinese person, my cultural feeling is that I would identify Kanji characters with Hanzi characters. Many characters do look the same and are written the same. Differences in stroke order or the direction of certain strokes feel more like “inessential” font differences. Visibly different forms between Kanji and Hanzi are similar to simplified vs traditional: more elaborate font differences, but still essentially the same characters.
One interesting angle is how Kanji characters are read in Chinese: they’re just read like native Chinese characters, completely ignoring the Japanese pronunciation and any shape differences. For example, the protagonist of Slam Dunk, 桜木花道 is always read as yīng mù huā dào in Mandarin Chinese, despite that (1) 桜 is written 樱 in Chinese (2) 花 is written slightly differently and (3) the Japanese pronunciation, Sakuragi Hanamichi, being kunyomi, bears no resemblance to yīng mù huā dào.
On a meta level, there’s no distinction between Hanzi and Kanji in Chinese: they’re both 汉字, pronounced hàn zì. I don’t know for sure whether Japanese people have this distinction, but it’s probably illuminating to see that the Japanese wiki page 漢字 encompasses the Chinese character system in all its regional variants.
Thanks for your input. There are 新字体 and 国字 (和製漢字), but as far as I know this distinction is only academic. Kunyomi/onyomi/etc. distinction is impossible to ignore, but that’s more related to words’ etymology than writing.
Visibly different forms between Kanji and Hanzi are similar to simplified vs traditional: more elaborate font differences, but still essentially the same characters.
Normally, I’m the first to say that the differences between simplified and traditional are overblown; however, I think it’s also eliding a bit to claim they’re essentially the same.
My mental model is that simplified originally was a surjective function. (That’s not true anymore.) But, while characters like 電／电 are onto and 只／隻 are grammatically awkward, characters like 复 can be downright misleading.
n.b. these differences matter less to Mandarin speakers, since simplified was made for it. (e.g. characters homophonous in Mandarin were merged) But the Japanese (and Korean, but that’s a different story) simplification projects came to different conclusions because they’re for different cultures and languages.
There were a few bugs and ghost characters in the process, which is to be expected when you’re digitizing tens of thousands of characters, but the basic idea of unification is sound. I had a friend who wrote her family name 櫻木 instead of the usual 桜木. Well sure, enough, that comes through on Lobsters because both variants are encoded. So too is the common 高 vs 髙 variation. The point is to be able to encode both variants where you would need them both in a single text without having to duplicate everything or worse (do we need variants each of Japanese, Korean, and Vietnamese? for pre-War and post-War Japanese? for various levels of cursive? etc.). It was a success.
Calligraphy can’t be represented in text at all. For that you need a vector image format, not a text encoding.
You’re saying unification is sound, but you can spell your friend’s name correctly only because these characters weren’t unified.
do we need variants each of Japanese, Korean, and Vietnamese? for pre-War and post-War Japanese
Yes! Unicode has Middle English, Old Church Slavonic with its one-off ꙮ, even Phoenician and Hieroglyphs. There’s old Kana in there already. East Asia should be able to encode their historical texts too.
UCS-2 was meant to be limited to contemporary characters due to 16-bit limit, but Unicode changed course to include everything.
CJK having a font dependency is reminiscent of legacy code pages.
I’ve worked on a website developed in Hong Kong, and it did track locale and set lang and language-specific font stacks to distinguish between the zh-xx and jp variants. They do care.
I’ve mentioned calligraphy not in technical sense, but cultural. The characters and their strokes are a valued tradition.
People may disagree about strokes of some complex characters, or there may be older and newer ways to draw a character, but that doesn’t mean the differences don’t matter.
I think the technical solution of mapping “characters” to code points to glyphs, applied to both alphabets and logograms suggests a perspective/commonality that isn’t the best way to view the issue.
You could also think of the character variants as differences in spelling. In English you have US and GB spellings, as well as historical spellings, and also some words with multiple spellings co-existing. These are the same mutually intelligible words, but if you did “English word unification”, it’d annoy some people.
I’m not familiar with this issue, but why not just add markers for similar characters to distinguish the cultural variant to be used where it is relevant?
Re Han unification - what do native speakers think? I assume there’s a diversity of opinion.
I’ve also thought for a while that “Greco” unification would be good - we would lose the attacks where words usually written in one script are written in the identical looking letter from another script.
Last I looked into the discussion about Han unification, I got the feeling that people in China (and maybe Japan) were annoyed that their input was not specifically requested before and during the discussion to proceed with Han unification. But I really don’t know enough about these scripts to have an opinion.
Regarding Greek letters, is this a common attack vector? What characters are most often used? From the Cyrillic set?
Painting “East Asian engineers” as a unitary body here is doing a lot of lifting.
The vast majority of pre-Unicode encoding schemes were both under unique resource constraints (ASCII compat? Fixed / variable length encoding? National language policy?) and designed for specific domains.
But to wit: Big5 unified some characters, HKSCS then deunified them because they weren’t the same in Cantonese.
Painting “East Asian engineers” as a unitary body here is doing a lot of lifting.
Apologies, that was not my intent. I meant that a lot of unification decisions have been made by engineers who are familiar with the issues, rather than, say, American engineers with little knowledge of Han characters.
I think Han unification was basically a good idea, but the problem is that it’s unclear where to draw the line. In fact, many Chinese characters that seem like they should be unified are separate in Unicode just because they are separate in JIS X 0208/0212/0213. Hooray for round-trip convertibility (sarcasm intended)!
From what I understand, Asian people get it much worse: many Chinese, Japanese, and Korean logograms that are written very differently get assigned the same code point
The logograms are not “very different”. Any educated person would see them as the same.
Eich also had about ten years of experience with language design and compiler developer and was explicitly hired by Netscape to put a programming language in the browser (pg 7). Originally this was supposed Scheme, but then Netscape signed a deal with Sun and agreed to make it more “Java-like”.
Wow wonder what that alternate timeline where Scheme is used instead of JS is like.
If that’s true, then C#, Lua, D, Erlang, Haskell, PHP, Scala, Go, Objective-C, Python, Ruby, and Smalltalk are basically Scheme. In other words, if JavaScript is Scheme by that criteria, then every language is Scheme, which of course means none of them are.
For the purpose of learning one language coming from another today, perhaps the comparison isn’t very useful. But to understand how JavaScript was designed at the time, it is. This OP and the context in which Crockford compares JavaScript to Scheme is much more about the latter. In 1995 when JavaScript was released, roughly one third of the languages on this list hadn’t been invented yet, and about another third were released that same year. Most of them looked very different when they were first released from how they look today. Compared with other mainstream languages as they existed at that time, JavaScript’s functions (first class and anonymize-able) brought it a lot closer to Scheme than, for example, Smalltalk or Objective-C. A lot of languages have slowly climbed into the a semi-functional, multi-paradigmatic soup over the years. C#, for example, which came along a good 5 years after JavaScript, has only recently supported what it calls “lambda expressions.”
I know that’s a popular refrain, but it’s just not true. C# was started as a way to write portable (in the sense of “can run on Windows, Windows NT on various CPUs, and Windows CE on various CPUs”) COM code. You can see that clearly in C# 1: it’s AOT, not a JIT. The global assembly cache (GAC) was a major point of emphasis, because it was trying to replace the COM registry. Transparent COM interop was a big selling point.
Beyond that, C# 1 and Java were meaningfully different. C# supported pointers, delegates/function pointers, unsigned types, checked arithmetic, structs, virtual and non-virtual methods, and more that Java did not have, but that you really need/want if you’re trying to replace writing COM objects. These differences have accelerated over time, but have been pretty strong from the beginning.
Microsoft invested heavily into Visual J++, which was an Embrace-Extend of Java. Sun sued. As a part of the settlement, Microsoft agreed to conform to Sun’s compliance tests. Anders put forward COOL (C-like Object-Oriented Language) as a replacement vehicle for Microsoft’s language goals:
When you go back and look, C# version 1.0, released with Visual Studio .NET 2002, looked a lot like Java. As part of its stated design goals for ECMA, it sought to be a “simple, modern, general-purpose object-oriented language.” At the time, looking like Java meant it achieved those early design goals.
J++ (and J/Direct) ended up in .NET and J#… and were deprecated in short order.
For obvious legal reasons, Microsoft could never say something like “C# is our Java-like.” But everyone writing code on the Windows platform at the time knew what was happening.
C# was originally begun as a clean room implementation of the Java spec. It was called Project Cool. (I didn’t work at Microsoft, but had recently left Redmond and had lots of friends working there.)
It took a lot of turns on the way to it’s 1.0 release, so it’s unfair to say that it remained just a clean room Java. However, that is how it began.
I think a fundamental issue here is that there were multiple projects started around the same time that got merged into .NET. I’m aware of Cool, but the peeps I know were coming from the COM side and knew things as NGWS (which I think was Next Generation Windows Services, but it’s been too long, and I mostly just knew the initialism). From their perspective, the J++ team got repurposed to help COM 3/DCOM 2/etc. (Managed C++ being a great example of this.) From the perspective of the J++ team, they probably saw their view affirmed. I don’t actually know what’s correct, but I respect that different teams probably had different perspectives, and mine may be wrong
I don’t think your perspective is wrong. C# and .NET wasn’t planned to be “that way” from the beginning. It emerged far different than the (multiple separate) ideas that began it. The clean room JVM was one of those ideas. Anders (being his own force of nature) had a completely different set of ideas that he was trying to figure out how to get in. There were definitely influences from the COM side (they had heavily influenced J++ as well) and NGWS (yup, you got the acronym right). There was also a Sun law suit in the middle of all of that, which freaked out the execs and did damage to a lot of technical plans that were in progress.
C# 1.0 was definitely not Java. But a dramatic portion of it was drawn from Java – some directly, and in a few cases (nonvirtual methods by default? really?!? 🤦♂️) things done differently in response to Java.
Thanks for expanding. I’m mostly looking at it from an application development perspective, and to me the ecosystems show strong similarities - object-orientated, huge availability of libraries (i.e. you “compose” programs using ready-made components), garbage collected, heavy reliance on IDEs to navigate…
I’m not complaining, btw. I like C#/.net well enough. But to me it does look like MS was worried that Java was gonna eat their lunch when it came to Windows application programming, and they needed something sexier than Visual Basic.
C#, for example, which came along a good 5 years after JavaScript, has only recently supported what it calls “lambda expressions.”
…the heck? Even if you want to claim that delegates don’t count (and I think they should, but whatever), lambda expressions proper came with C# 3 in 2007. That was a key component of LINQ. How is that “recent”? C++ didn’t get them until either 11 or 14, and I wouldn’t even call those recent, either; it’s been over a decade.
Fair enough. My recollection of when they were added is somewhat warped by how long it took for them to be adopted where I worked at the time. There was some backlash from OOP purists. Nevertheless, 2007 is still 12 years after JavaScript, so my main point stands.
So the people who invented Scheme, and taught people how to write Scheme, apparently think they can get across at least some of the same concepts in JavaScript.
It’s true that tail calls are a big deal, and so are macros. But I’d argue a bit against “distaste for mutation” as a difference – since there is a fair amount of mutation in SICP, and it’s somewhat idiomatic to write JS with little mutation.
But yeah I guess if you’re writing tail recursive JS, people are going to tell you to stop doing that, because it will eventually blow up in production. I guess for pedagogical code it’s fine.
(My argument towards my “shitbag” point about the latter opinion is that it is not a rational one, it is a baseless “political” belief that stands in the way of others’ happiness; therefore, shitbag-ness. Rational and sound defenses of gay marriage are readily available for anyone wishing to look, one example: https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article=1178&context=faculty_articles IMHO if you can objectively defeat a belief on rational grounds, it is no longer a political belief that you are entitled to, but an oppressively false one)
It is an indictment of our field that this opinion is even controversial. Of course XML is better than YAML. YAML is a pile of garbage that should be abandoned completely. XML at least has a lot of decent engineering behind it for specific purposes.
You’re right. The principles should be laid out: Ease of reasoning about configuration file formats is vastly more important than conveniences for writing specific values. Implicit conversion among types beyond very basic lifting of integer types is a bad idea, especially for configuration file formats. Grammars for configuration file formats should be simple enough to write a complete, correct grammar as a one day project.
XML is kind of a weird animal because it’s playing the role equivalent to text for JSON. The principles above apply to the DTD you apply to your XML schema.
There is no implicit type conversion going on on the YAML side. no is a boolean in YAML, just like false is a boolean in JSON. If a YAML parser converts it to a string, that’s the parser’s problem.
No, @xigoi is right, strictly speaking. The parser is where this conversion is going on. Only if it cannot read it as anything else, it reads unquoted literals as if they were quoted strings. Of course, to a user that is neither here nor there: the rules need to be memorized to be able to use unquoted literals correctly.
the rules need to be memorized to be able to use unquoted literals correctly
You’ll have a better time if you just use quotes by default… I don’t understand the appeal of unquoted literals in YAML
This, for me, is the root of it. YAML is fine as long as you are explicit. Now what it takes to be explicit is going to be driven by what types you intend to use. It seem, to me, that the majority of yaml use cases intend to use only a handful of scalar types and a handful collection types. That small set of types, not coincidentally, is basically the same as what you get in JSON and properly formed JSON is always valid YAML. So I would assert that if you use YAML and explicitly quote string values that you are effectively getting a slightly looser JSON parser which happens to allow you to write a flavor of JSON which is much easier for human concerns; I.E. less picky about trailing commas, supports comments, and is easier on the eyes with some of its constructs.
Of course, we’ve got a whole shitload of options these days, so I wouldn’t be surprised if some other markup/serialization format is better in any given specific domain. Different tools for different jobs…
One thing I will absolutely agree with is that YAML is awful when used as a basis for psuedo-DSLs, as you see in things like ansible and a lot of CICD systems.
I think we basically agree, but in my opinion one should accept that people are lazy (or forgetful) and use shortcuts, or even copy/paste bad examples. This is like saying sloppiness in PHP or JS is not a problem because one can always use ===.
Most people don’t have the discipline to be explicit all the time (some don’t have the discipline to ever be explicit), therefore it’s probably safer to avoid tools with overly prominent inbuilt footguns entirely.
Rubbish, the list of semantic surprises in YAML is long and established. The problems with XML boil down to “my fingies are sore fwom all the typing” and fashion.
One of the most talented developers I know can only work for 2-3 hours a day on a good day because of RSI. I don’t think your patronising take carries the weight you think it does.
There are other issues with XML. Handling entities is complex as are the rules for name spacing. Writing an XML parser is complex so most people use libxml2, which is a massive library that doesn’t have a great security track record. For most YAML use cases (where the input data is trusted) this doesn’t matter too much. Parsing YAML is also incredibly hard so everyone uses the same YAML parser library.
Problems in a specific parser can’t be called problems in the format itself. For what it’s worth YAML’s popular parsers have also had horrible security problems in the past.
If you have a minute to go into detail, I’m interested in what I’ve missed that makes namespaces complicated, I found them pleasing when used correctly, and frankly used so infrequently that it hardly ever came up, outside of specific formats that used xml as a container, for example MXML. But this knowledge is old now in my case, so I probably just missed the use case that you’re referring to.
The entity expansions should never have been a thing, that much I’m sure we can all agree on. DTDs were a mistake, but XSD cleaned most of that up; but unless you were building general XML tooling you could in most cases ignore schemas and custom entities completely.
What’s good about XML (aside from how much support and tooling it once had) is IMO:
The consistency with which the tree structure is defined. I don’t know why “modern” markups are all obsessed with the idea that the end of a node should be implied by what’s around it, rather than clearly marked, but I can’t stand it.
A clear separation of attributes and children.
Consistency in results, in that there are no “clever” re-interpretations of text.
Yeah that’s rough as guts 🤣 I’ve never seen somebody override the current namespace in the middle of the document, I never even considered that as something you could do. Nobody should have done this, ever.
It’s almost 20 years since I did anything serious with XML, but I seem to remember the namespace things let you define alternative names for tags to avoid conflicts, so you had to parse tags as their qualified name, their unqualified name in the current namespace (or the parents?) or their aliased name.
A lot of the security issues of libxml2 were due to the inherent complexity of the format. There are a lot of JSON parsers because the format is simple. You can write a JSON parser in a couple of hundred lines of code if you have a decent string library. A compliant XML parser is at least one, probably two, orders of magnitude more complex. That significantly increases the probability that it will have bugs.
I’m also not sure I agree on the ‘clear separation of attributes and children’ thing. XML formats that I’ve worked with have never managed to be completely consistent here. Attributes are unstructured key-value pairs, children are trees, but there are a lot of cases where it’s unclear whether you should put something in an attribute or a child. Things using XML for text markup have to follow the rule that cdata is text that is marked up by surrounding text, but things using XML as a structured data transport often end up accidentally leaking implementation details of their first implementation’s data structures into this decision.
I have zero real world issues with YAML, honestly. I’ll take YAML over XML every day of the week for config files I have to edit manually. Do I prefer a subset like StrictYAML? Yep. Do I still prefer YAML over anything else? Also yep.
This sounds like all the folks who have “no real issues” with MySQL, or PHP (or dare I say it, JavaScript). Somehow the issues with YAML seem more commonly accepted as “objectively” bad, whereas the others tend to get defended more fiercely. I wonder why!
At a previous employer we had a crazy bug that took a while to track down, and when we did it turned out the root cause was YAML parsing something as a float over a string yet the syntax highlighting parsed it as a string.
I wasn’t the developer on the case, so I don’t remember the exact specifics, but it boiled down to something like we used a object ID which was a hex string, and the ID in question was something along the lines of:
oid: 123E456
Which according to the YAML spec allows scientific notation. Of course this could be chalked up to a bug in the syntax highlighting or failure on our part to not use quotation marks but he results were the same; a difficult to track down bug downstream.
Don’t like how often people forget this. I do wish there were some sort of emojis to represent languages without nations like Malayalam or the many north american indigenous languages.
The ISO codes can sometimes suffice, but the best badge seems to be the native language’s name in its native script (or both). It could be cute to a have a single codepoint symbol but the labeling with the script is clear to its readers.
I haven’t looked into this, but surely the various federally recognized tribes in the US have their own flags? I take your point though that it doesn’t solve the overall problem.
On the other hand, an interface that displays language names raises questions about which language’s name takes precedence, and then when you settle on “display each language as it names itself” raises further questions about how to handle the fact that not all those languages read left-to-right (hello Arabic, Hebrew, etc.) or even horizontally (hello Mongolian). And that’s before getting into how to select particular dialects or variants like en-US versus en-GB, or zh-TW, etc.
Flags or other “graphical” representations can at least push the issues out of the technical and into the political realm.
Yes, lets casually marginalize swaths of folks so we can say “not our problem” /s. Wikipedia, et al. seem just fine with their language pickers, etc. without needing to piss a bunch of folks off for software makers’ naïve/insensitive decisions since the can often afford to skip the political implications as “doesn’t affect me”.
Yes, lets casually marginalize swaths of folks so we can say “not our problem”
Yes, let’s just casually sneer at people and accuse them of evil, that’ll certainly win them over to doing what you want.
Like it or not, using a graphical language picker does avoid some rendering issues, and those rendering issues may not be under the control of the programmer writing the language picker, which can sometimes make it a least-worst-option approach.
Good job. You invented an exception that might happen in some rare cases (‘rare’ because the web & other places don’t seem to have issues in practice at present), but you haven’t said anything that refutes the main point that, as a heuristic, avoid flags as the symbol for languages since it is the wrong literally the wrong symbol for the idea—and often has negative side effects on users such as being reminded they don’t fit in as immigrants/expats to their new home, leaving minority languages out as unimportant, reinforcing some dialect as more correct/prestigious, plastering a colonizer’s banner in your UI, etc.
You could get away with ignorance, but reminding others that it in almost all cases it is the wrong thing to do means if you know but are casually doing it anyways is malice. A PSA for good help folks avoid inflaming issues in the “political realm” is a positive move as many developers come from the background of privileges that makes them accidentally miss the implications of their choices—and I don’t see the value in your devil’s advocacy.
I mean, just be concrete. I’m in the US. Most of the posts on the site I manage are in English. Some are translated into Spanish. What flags should we use? The flag of the UK has nothing to do with our content (we’re US-focused). The flag of Spain doesn’t really have anything to do with most of our Spanish readership, who are immigrants or children of immigrants from Latin America. We could do the flags of the US and Mexico, but that would be annoying to people from other Latin American countries, and for that matter a lot of Mexicans grow up speaking indigenous languages, which aren’t represented by the flag and we don’t have support for.
There are cases where flags are appropriate, like picking your Amazon store based on what items are available and shippable, and cases where it’s not, like picking the language for site chrome.
I mean think about the UK flag… does Wales speak English or Welsh? Scotland English or Scots? Northern Ireland speak Gaelic dialects? Why not use the English red+white flag? What? …you say many folks don’t recognize it & others confuse it with the Georgian flag. The UK is #6 in the world for number of English speakers. There’s no language governing body. It just doesn’t make a lot of sense when you break it down.
picking your Amazon store based
That is a geographical region, not a language so. With taxes usually tied to nations, so the flag makes sense.
I understand the rationale for including the original emoji (unicode wants to be a superset of existing character sets) but they should have been put in a code space reserved for backwards compatibility with bad ideas, not made such a big part of unicode.
At this point, there’s a strong argument for a new character set that is a subset of unicode that removes all of the things that are not text. We already have mechanisms for embedding images in text. Even in the ‘90s, instant messaging systems were able to avoid sending common images by having a pre-defined set of pictures that they referenced with short identifiers. This was a solved problem before Unicode got involved and it’s made text processing an increasingly complicated mess, shoving image rendering into text pipelines for no good reason.
The web could have defined a URL encoding scheme for emoji from an agreed set, or even a shorthand tag with graceful fallback (e.g. <emoji img="gb-flag;flag;>Union Flag</emoji>, which would render a British flag if you have an image for gb-flag, a generic flag if you don’t, have ‘Union Flag’ as the alt text or the fall back if you don’t support emoji). With the explicit description and fallback, you avoid the things like ‘I’m going to shoot you with a 🔫’ being rendered as ‘I’m going to shoot you with a {image of a gun}’ or ‘I’m going to shoot you with a {image of a water pistol}’ depending on the platform: if you didn’t have the water-pistol image, you’d fall back to the text, not show the pistol image.
Like it or not, emoji are a big part of culture now. They genuinely help convey emotion in a fairly intuitive manner through text, way better than obscure tone indicators. I mean, what’s more understandable?
“Are you going to leave me stranded? 😛”
“Are you going to leave me stranded? [/j]”
It definitely changes the meaning of the text. They’re here to stay, and being in Unicode means they got standardized, and it wouldn’t have happened otherwise.
Of course there’s issue with different icon sets having different designs (like how Samsung’s 😬 was completely different from everyone else’s), but those tend to get resolved eventually.
Like it or not, emoji are a big part of culture now. They genuinely help convey emotion in a fairly intuitive manner through text, way better than obscure tone indicators.
Except they don’t. Different in groups assign different meanings to different ones. Try asking someone for an aubergine using emoji some time and see what happens.
“Are you going to leave me stranded? 😛”
This is culturally specific. It’s an extra set of things that people learning English need to learn. This meaning for sticking out your tongue is not even universal across European cultures. And that’s one of the top ten most common reaction emoji, once you get deeper into the hundreds of others the meaning is even further removed. How would you interpret the difference between 🐶 and 🐕 in a sentence?
Of course there’s issue with different icon sets having different designs (like how Samsung’s 😬 was completely different from everyone else’s), but those tend to get resolved eventually.
That’s an intrinsic property of using unicode code points. They are abstract identifiers that tell you how to find a glyph. The glyphs can be different. A Chalkboard A and a Times A are totally different pictures because that’s an intrinsic property of text. If Android has a gun and iOS has a waterpistol for their pistol emoji, that’s totally fine for characters but a problem for images.
😱 Sure emojis are ambiguous . And different groups can use them differently. But that doesn’t mean they don’t convey meaning? The fact that they are so widely used should point towards them being useful no? 😉
I never said that embedding images in text is not useful. I said that they are not text, do not have the properties of text, and treating them as text causes more problems than it solves.
I disagree. At best, they are precursors of an ideographic script. For a writing system, there has to be some kind of broad consensus on semantics and there isn’t for most emoji beyond ‘that is a picture of X’.
Please describe to me the semantics of the letter “р”.
For alphabetic writing systems, the semantics of individual letters is defined by their use in words. The letter ‘p’ is a component in many of the words in this post and yours.
Thank you! (That was actually U+0440 CYRILLIC SMALL LETTER ER, which only featured once in both posts, but no matter.)
the semantics of individual letters is defined by their use in words
The thing is, I disagree. “e” as a letter itself doesn’t have ‘semantics’, only the words including it do[1]. What’s the semantics of the letter “e” in “lobster”? An answer to this question isn’t even wrong. It gets worse when different writing systems interpret the same characters differently: if I write “CCP”, am I referring to the games company CCP Games? Or was I abbreviating сoветская социалистическая республика? What is the semantics of a letter you cannot even identify the system of?
Emoji are given meaning of different complexity by their use in a way that begins to qualify them as logographic. Most other writing systems didn’t start out this way, but that doesn’t make them necessarily more valid.
[1]: The claim doesn’t even hold in traditional logographic writing systems which by all rights should favor your argument. What is the semantics of the character 湯? Of the second stroke of that character? Again, answers aren’t even wrong unless you defer to the writing system to begin with, in which case there’s no argument about (in)validity.
Yes, but their original point is that we should be able to compose emojis like we compose words, as in the old days of phpBB and instant messaging. :mrgreen:
Just a nit: people do compose emojis - I see sequences of emojis all the time. People send messages entirely of emojis that other people (not necessarily me) understand.
Yeah, and also there’s nothing wrong with that, it’s something any group can and should be able to do. I have no entitlement to be able to understand what other people say to each other (you didn’t claim that, so this isn’t an attack on you. I am just opposed to the “I don’t like / use / understand emojis how other people use them therefore they are bad” sentiment that surfaces periodically).
But not of characters and rarely true even of ideographs in languages that use them (there are exceptions but a language is not useful unless there is broad agreement on meaning). It’s not even true of most words, for the same reason: you can’t use a language for communication unless people ascribe the same meaning to words. Things like slang and jargon rarely change more than a small fraction of the common vocabulary (Clockwork Orange aside).
Without getting into the philosophy of what language is, I think this skit best illustrates what I mean (as an added bonus, emoji would have resolved the ambiguities in the text).
Note I’m not arguing for emoji to be in Unicode, I’m just nitpicking the idea that the problem with them is ambiguity.
Socrates would like to have a chat with you. I won’t go through the philosophical tooth-pulling that he would have enjoyed, but suffice it to say that most people are talking past each other and that most social constructions are not well-founded.
I suspect that this is a matter of perspective; try formalizing something the size of English Prime (or, in my case, Lojban) and see how quickly your intuitions fail.
I understand the rationale for including the original emoji (unicode wants to be a superset of existing character sets) but they should have been put in a code space reserved for backwards compatibility with bad ideas, not made such a big part of unicode.
Except emoji have been absolutely stellar for Unicode: not only are they a huge driver of adoption of unicode (and and through UTF8) because they’re actively desirable to a large proportion of the population, they’ve also been a huge driver of improvements to all sorts of useful unicode features which renderers otherwise tend to ignore despite their usefulness to the rendering of actual text, again because they’re highly desirable and platforms which did not support them got complaints. I fully credit emoji with mysql finally getting their heads out of their ass and releasing a non-broken UTF8 (in 2010 or so). That’s why said unicode consortium has been actively leveraging emoji to force support for more complex compositions.
And the reality is there ain’t that much difference between “image rendering” and “text pipeline”. Rendering “an image” is much easier than properly rendering complex scripts like arabic, devanagari, or burmese (or Z̸̠̽a̷͍̟̱͔͛͘̚ĺ̸͎̌̄̌g̷͓͈̗̓͌̏̉o̴̢̺̹̕), even ignoring that you can use text presentation if you don’t feel like adding colors to your pileline.
Even in the ‘90s, instant messaging systems were able to avoid sending common images by having a pre-defined set of pictures that they referenced with short identifiers.
After all what’s better than one standard if not fifteen?
This was a solved problem before Unicode got involved and it’s made text processing an increasingly complicated mess, shoving image rendering into text pipelines for no good reason.
This problem was solved by adding icons in text. Dingbats are as old as printing, and the Zapf Dingbats which unicode inherited date back to the late 70s.
The web
Because nobody could ever want icons outside the web, obviously. As demonstrated by Lucida Icons having never existed.
subset of unicode that removes all of the things that are not text
It sounds like you disagree solidly with some of Unicode’s practices so maybe this is not so appealing, but FWIW the Unicode character properties would be very handy for defining the subset you’d like to include or exclude. Most languages seem to have a stdlib interface to them, so you could pretty easily promote an ideal of how user input like comment boxes should be sanitized and offer your ideal code for devs to pick up and reuse.
new character set that is a subset of unicode that removes all of the things that are not text
and who’d be the gatekeeper on what the text is and isn’t? What would they say about the ancient Egyptian hieroglyphs? Are they text? If yes, why, they are pictures. If no, why, they encode a language.
It might be a shallow dissimilar, but people trying to tell others what forms of writing text are worthy of being supported by text rendering pipelines gets me going.
If the implementation is really so problematic, treat emojis as complicated ligatures and render them black and white.
and who’d be the gatekeeper on what the text is and isn’t? What would they say about the ancient Egyptian hieroglyphs? Are they text? If yes, why, they are pictures. If no, why, they encode a language.
Hieroglyphics encode a (dead) language. There are different variations on the glyphs depending on who drew them (and what century they lived in) and so they share the property that there is a tight(ish, modulo a few thousand years of drift) coupling between an abstract hieroglyph and meaning and a loose coupling between that abstract hieroglyph and a concrete image that represents it. Recording them as text is useful for processing them because you want to extract the abstract characters and process them.
The same is true of Chinese (though traditional vs simplified made this a bit more complex and the unicode decisions to represent Kanji and Chinese text using the same code points has complicated things somewhat): you can draw the individual characters in different ways (within certain constraints) and convey the same meaning.
In contrast, emoji do not convey abstract meaning, they are tightly coupled to the images that are used to represent them. This was demonstrated very clearly by the pistol debacle. Apple decided that a real pistol image was bad because it was used in harassment and decided to replace the image that they rendered with a water pistol. This led to the exact same string being represented by glyphs that conveyed totally different meaning. This is because the glyph not the character encodes meaning for emoji. If you parsed the string as text, there is no possible way of extracting meaning without also knowing the font that is used.
Since the image is the meaningful bit, not the character, we should store these things as images and use any of the hundreds of images-and-text formats that we already have.
More pragmatically: unicode represents writing schemes. If a set of images have acquired a significant semantic meaning over time, then they may count as a writing system and so can be included. Instead, things are being added in the emoji space as new things that no one is using yet, to try to define a writing scheme (largely for marketing reasons, so that ‘100 new emoji!’ can be a bullet point on new Android or iOS releases).
It might be a shallow dissimilar, but people trying to tell others what forms of writing text are worthy of being supported by text rendering pipelines gets me going.
It’s not just (or even mostly) about the rendering pipelines (though it is annoying there because emoji are totally unlike anything else and have required entirely new feature to be added to font formats to support them), it’s about all of the other things that process text. A core idea of unicode is that text has meaningful semantics distinct from the glyps that they represent. Text is a serialisation of language and can be used to process that language in a somewhat abstract representation. What, aside from rendering, can you do with processing of emoji as text that is useful? Can you sort them according to the current locale meaningfully, for example (seriously, how should 🐕 and 🍆 be sorted - they’re in Unicode and so that has to be specified for every locale)? Can you translate them into a different language? Can you extract phonemes from them? Can you, in fact, do anything useful with them that you couldn’t do if you embedded them as images with alt text?
Statistically, no-one cares about hieroglyphics, but lots of people care about being able to preserve emojis intact. So text rendering pipelines need to deal with emojis, which means we get proper hieroglyphics (and other Unicode) “for free”.
Plus, being responsible for emoji gives the Unicode Consortium the sort of PR coverage most organizations spend billions to achieve. If this helps them get even more ancient writing systems implemented, it’s a net good.
What, aside from rendering, can you do with processing of emoji as text that is useful?
Today, I s/☑️/✅/g a text file.
Can you sort them according to the current locale meaningfully, for example (seriously, how should 🐕 and 🍆 be sorted - they’re in Unicode and so that has to be specified for every locale)?
At this point, there’s a strong argument for a new character set that is a subset of unicode that removes all of the things that are not text.
All that’s missing from this sentence to set off all the 🚩 🚩 🚩 is a word like “just” or “simply”.
Others have started poking at your definition of “text”, and are correct to do so – are hieroglyphs “text”? how about ideograms? logograms? – but really the problem is that while you may feel you have a consistent rule for demarcating “text” from “images” (or any other “not text” things), standards require getting a bunch of other people to agree with your rule. And that’s going to be difficult, because any such rule will be arbitrary. Yours, for example, mostly seem to count certain very-image-like things as “text” if they’ve been around long enough (Chinese logograms, Egyptian hieroglyphics) while counting other newer ones as “not text” (emoji). So one might reasonably ask you where the line is: how old does the usage have to be in order to make the jump from “image” to “text”? And since you seem to be fixated on a requirement that emoji should always render the same on every platform, what are you going to do about all the variant letter and letter-like characters that are already in Unicode? Do we really need both U+03A9 GREEK LETTER CAPITAL OMEGA and U+2126 OHM SIGN?
So one might reasonably ask you where the line is: how old does the usage have to be in order to make the jump from “image” to “text”?
Do they serialise language? They’re text. Emoji are not a writing system. They might be a precursor to a writing system (most ideographic writing systems started with pictures and were then formalised) but that doesn’t happen until people ascribe common meaning to them beyond ‘this is a picture of X’.
And since you seem to be fixated on a requirement that emoji should always render the same on every platform, what are you going to do about all the variant letter and letter-like characters that are already in Unicode?
That’s the opposite of my point. Unicode code points represent an abstraction. They are not supposed to require an exact glyph. There are some things in Unicode to allow lossless round tripping through existing character encodings that could be represented as sequences of combining diacritics. They’re not idea in a pure-Unicode world but they are essential for Unicode’s purpose: being able to represent all text in a form amenable to processing.
For each character, there is a large space of possible glyphs that a reader will recognise. The letter A might be anything from a monospaced block character to a curved illustrated drop character from an illuminated manuscript. The picture is not closely coupled to the meaning and changing the picture within that space does not alter the semantics. Emoji do not have that property. They cause confusion when slightly different glyphs are used. Buzzfeed and similar places are full of ‘funny’ exchanges from people interpreting emoji differently, often because they see slightly different glyphs.
The way that emoji are used assumes that the receiver of a message will see exactly the same glyph that the sender sends. That isn’t necessary for any writing system. If I send Unicode of English, Greek, Icelandic, Chinese, or ancient Egyptian, the reader’s understanding will not change if they change fonts (as long as the fonts don’t omit glyphs for characters in that space). If someone sends a Unicode message containing emoji, they don’t have that guarantee because there is no abstract semantics associated with them. I send a picture of a dog, you see a different dog, I make a reference to a feature of that dog and that feature isn’t present in your font, you are confused. Non-geeks in my acquaintance refer to them as ‘little pictures’ and think of them in the same way as embedded GIFs. Treating them as characters causes problems but does not solve any problems.
Do they serialise language? They’re text. Emoji are not a writing system. They might be a precursor to a writing system (most ideographic writing systems started with pictures and were then formalised) but that doesn’t happen until people ascribe common meaning to them beyond ‘this is a picture of X’.
I think this is going to end up being a far deeper and more complex rabbit hole than the tone of your comment anticipates. Plenty of things that are in Unicode today, and that you undoubtedly would consider to be “text”, do not hold up to this criterion.
For example, any character that has regional/dialect/language-specific variations in pronunciation seems to be right out by your rules. So consider, say, Spanish, where in some dialects the sound of something like the “c” in “Barcelona” is /s/ and in others it’s /θ/. It seems hard to say that speakers of different dialects agree on what that character stands for.
At this point, I feel like the cat is out of the bag; people are used to being able to use emoji in almost any text-entry context. Text rendering pipelines are now stuck supporting these icons. With that being the case, wouldn’t it be way more complexity to layer another parsing scheme on top of Unicode in order to represent emoji? I can see the argument that they shouldn’t have been put in there in the first place, but it doesn’t seem like it would be worth it to try to remove them now that they’re already there.
I’m porting biscuit-java to Kotlin so I can use it in a multiplatform (Android and iOS) app. It’s got a compiler and datalog interpreter in there, so it’s been major nerd snipping fun.
I’ve been using this exact analogy the last month explaining to people the issues with Web Environment Integrity.
I remember why devices were considered yours when you bought them & you were expected to tinker & administer them. Now everything has become an appliance.
I was looking at a new phone as mine is starting to show serious signs of wear. I almost pulled the trigger on a Zenfone 9 after the 10 launch looking to slap on OmniROM + microG to find out back in May ASUS yanked the unlock tool & the servers required to get the unlock keys. ASUS says they’ll be up again after “maintenance” in Q3, but it’s almost Q4… and I have little reason to believe they ever will return. (I guess Sony is the only one left with OLED, 3.5 mm headphone jack, with a vaguely compact offering)
I switched banks a couple years ago after my banking app started detecting root even with Magisk Hide. I cited that they don’t tell me how to use my laptop so why can they with my general compute phone. Last month the new bank finally caught up. Either its the custom ROM or the root, but updating to the latest hide root option, I had no such luck. The web experience can work but it’s awful & not optimized (including checks for Netscape Navigator 4, disabled pasted, etc.). I wouldn’t be surprised if they yanked the website soonish which makes me wonder if I should go ahead with my plan to build a UserScript to alleviate some of the pain. Even then I’m reliant on SMS as the only 2FA & changing numbers (normal for prepaid) involves paperwork & a week of waiting. I asked them how I would do transfers abroad if roaming without SMS & the staff bit their tongue in the middle of saying do a transfer thru the (mobile) app.
The online stores are ruining it too. I picked up some Nike gear on sale. Due to the shoddy 3rd-party code in the checkout that didn’t handle users blocking fingerprinting + ad/tracking blocking I couldn’t get thru. I brought it up with their customer service that said I should disable my anti-virus, make the purchase, then re-enable …Funny, since I don’t download random BS & am on Linux, I don’t use an anti-virus, but why do I always need to use a less secure setup for payment? If you want to know it’s me, I already authenticated & you could add TOTP or FIDO2 2FA if you wanted to make sure instead of reading all my sensors & canvas. I ended up having a friend who doesn’t care about his privacy buy it on my behalf for a Hong Kong pie. At least it scrambles in false data for his personal footprint.
Things are only going to get increasingly worse. The complaints I have all came from just last month.
I switched banks a couple years ago after my banking app started detecting root even with Magisk Hide. I cited that they don’t tell me how to use my laptop so why can they with my general compute phone
They’re not telling you how to use your devices. They’re telling you the terms on which they’ll allow you to access their systems, which they own as surely and completely as you own yours, and over which they have rights just as strong as your rights over your devices.
You do not have an inherent right to access their systems, after all. And you don’t, strictly speaking require that access to use a bank. You can manage your banking in person at a branch, over the phone with their customer service, etc.
A similar argument can be made with how workers are supposed to work in a factory:
Communist workers: we work here, we ought to dictate how we work!
Capitalist owner: you work in my factory, I ought to dictate how you work!
Sure, corporations would like to secure their connections, and minimise their own costs. But there is a point where that desire to do so actively infringes on the users, especially if the service they provide have no easily accessible alternative, or the switching costs are non-trivial. At some point they are actively extorting compliance.
And you don’t, strictly speaking require that access to use a bank.
I don’t, strictly speaking require to buy things online. Realistically though…
What this misses is that access to digital financial services is pretty much a prerequisite for existing in western societies. You can make an argument in the positive, sure: that you don’t have some god-given right to access their services. But in the negative space, when every equivalent service has similar such restrictions and use of those services are on the critical path to your continued well-being, it’s difficult to interpret that as anything other than authoritarian. It’s not conceptually different to the use of the term ‘wage-slavery’, used to describe situations in which there’s a superficial veneer of choice but, in practice, all reasonable employment options are monopolised by exploitative employers.
What this misses is that access to digital financial services is pretty much a prerequisite for existing in western societies.
This is true. But demanding that the existing financial world accept customers using rooted phones will move the needle towards more equitable access to financial services not a whit.
Maybe. I’ll just state that if the price for everyone having equitable access to financial services is that the devices used for access are “locked down”, that’s a price I’m personally ok with paying.
Equitable access to financial services using any device presupposes that everyone can afford the devices, and one of the ways the devices could be (more) affordable is by letting their owners maintain them after they’re abandoned by their manufacturers. User freedom extends the life (and, frequently, capability) of hardware.
In general, I think ceding control of our window into most of modern society would be a mistake, no matter what. But even if you don’t, it must surely be a bit concerning that we’re ceding that control to profit-making companies which have repeatedly demonstrated their willingness to exploit that position for money.
In many ways I wish they gave out something akin to those Trezor / Ledger cryptocurrency hardware wallets (not an endorsement)–a dedicated, purpose-built device with all the 2FA protections & account info on a portable, ‘dumb’ piece of hardware as opposed to co-opting mobile phones which have other, general-compute needs. If it was something I could slap on a keyring and could, I dunno, do NFC transfers between two accounts & only used a host device for network access (without tracking/analytic nonsense), then I could be into it.
Someone needing to use the bank’s website on their phone instead of the bank’s app on their phone is not “a superficial veneer of choice”, in my opinion. They continue to have meaningful access to the bank, at which point the rest of your argument falls apart (since it’s predicated on the lack of such access).
Also, as noted elsewhere, I am willing to accept that as the tradeoff of vastly increased security for the average user. Even if a worldwide law were passed requiring that every single device everywhere be a 100% Free software/hardware StallmanPhone™, the number of people who would actually make use of all that glorious freedom would likely round to zero, relative to the total population. And what we’re talking about here is not denying them participation in society; they still can bank on their phones, just not via the app. I am willing to accept those sorts of occasional speed bumps as the cost of vastly improved conditions for everyone else. You presumably are not, but I don’t see how any argument that claims to be on the side of the masses can choose not to make this sort of tradeoff.
(you will recall that I argued we should break up Google as a solution to “web attestation”, since a broken-up Google would not be able to enforce its will the way the current Google does)
I mean sure technically speaking about usage terms being whatever they decide to write—they could write you can’t use the service without a banana on your head if they wanted. But why are the rules so strict (crossing the line as many folks see it) & different versus the web browser experience? In this case, at least until Google forces WEI, the regulations on my machine are… that I have a browser that support TLS? (They don’t even support TOTP/FIDO2) With the website, there’s nothing against root/admin, modifying hosts, ad blocking, etc. so why the shift when it comes to the general-compute of a smart phone? These are two very different standards. One could say they would have done it all along if they had the ability, but we’ve had 2 decades of online banking being just fine.
Also why isn’t the “risk” of root my burden? I have done few modifications with root: modifying hosts, setting up a firewall, and eliminating the faux shutter sound of the camera—with ⅔ making my device safer. The workaround is the web, but I am afraid website will start disappearing saying they only support mobile applications “for my safety”. I do go to branches frequently, almost exclusively use cash for real world purchases, use the physical ATM/debit card over digital scans, but more shopping even here has moved online which requires dealing with the digital banking situation.
There probably is a mundane reason for it that has to do with some compliance checklist somewhere.
But the real answer is to ask why you think you have a right to press on that question. You’d probably be very offended if someone else kept pressing you and pressing you about why you set up and manage your devices and systems the way you do. It’s none of anyone else’s business, after all, because they’re yours and you own them and that gives you the freedom to manage them how you like without needing to account for your preferences to anyone else. And exactly the same thing is true of the bank and its systems. If the bank only wants to allow access from apps whose integrity can be attested, then that’s the bank’s choice and they don’t need to account for that choice to you.
Or, more succinctly: it is not possible to have a just and coherent philosophy where you get to tell people “my systems, my rules” but others do not.
But the real answer is to ask why you think you have a right to press on that
question. You’d probably be very offended if someone else kept pressing you and
pressing you about why you set up and manage your devices and systems the way
you do.
That’s the thing. They do keep pressing and have been doing so for
decades, and yeah, I’m offended. Your argument is utterly specious and
without merit, because it is founded on a false equivalency. There is a
huge imbalance of power between people and large corporations or
organizations of any kind. Requiring approved devices is how monopolies
gain strength. Doing things like rooting devices is how people can take
back power from monopolies. Would you find it acceptable for your bank
to require that you visit a branch with an approved vehicle or method of
transportation? “Sorry. You didn’t arrive in a Tesla newer than 2020,
so no service for you.” “Oh, you came on foot? You must be a threat.
No service.” Sorry but I’m not buying it.
As I recall, you and I have gone round and round on this exact issue in the past with no useful result. But I’ll try again.
I think the flaw in your worldview is that it’s based on binary thinking rather than on tradeoffs, and tradeoffs are really what’s happening.
To take an example, if I were to walk into my local bank branch one day and ask to wire or withdraw in cash a significant amount of money, they probably would not immediately comply, and would first ask me a series of questions about why I want to do that. Not because they’re evil and hate my freedom, and not because the government hates my freedom and wants surveillance of my transactions (anything large enough gets auto-reported to them anyway). But because, unfortunately, there are tons of scammers out there and tons of scam victims who get conned into making big withdrawals or transfers to the scammer, and bank staff are trained to watch for things that match the pattern and ask questions before the money goes anywhere.
Similarly, staff at many stores are now trained to question large quantities or denominations of gift cards, because a lot of scams use them as a form of currency.
Does this make life marginally harder for me, a bona fide citizen going about my lawful business? Sure. But is it a worthwhile tradeoff? Probably it is; those minor obstacles actually are pretty effective at stopping situations where people lose what they can’t afford to lose, and dealing with them cost me no more than a moment or two of my time.
The same is broadly true, in my opinion, of modern computing security. Is it harder for someone who wants to run custom and customized everything to do so? Sure. But it hasn’t been made harder out of some evil motive to destroy freedom; it’s been made harder because it’s a tradeoff. In return for some legitimate uses being more difficult, a lot of attack vectors can be significantly mitigated or even eliminated.
Does this mean that if you root your phone you might need to access the bank via a web browser instead of their app, since it no longer trusts the environment it finds itself running in? Yes. Does this mean you might need to check some boxes in a deeply-concealed preferences panel, or click through some warnings when running random code you downloaded off the internet on your laptop? Yes.
But it’s a minor inconvenience and, despite dubious popular quotations, it doesn’t actually “give up” any “essential liberty” and does in fact achieve better security overall. It’s the same reason that Facebook started logging a huge scary warning when you open the browser console, telling you that whatever someone told you paste in there was going to hack you – is it annoyingly paternalistic to someone who’s sufficiently technically competent? Absolutely. But it’s probably saved a lot of people a lot of pain, and that’s a tradeoff I am willing to make.
Does this mean that if you root your phone you might need to access the bank via a web browser instead of their app, since it no longer trusts the environment it finds itself running in? Yes. Does this mean you might need to check some boxes in a deeply-concealed preferences panel, or click through some warnings when running random code you downloaded off the internet on your laptop? Yes.
If it was just that I could live with it. But if rooting my phone means I can no longer access my bank at all, then it and I have a serious problem.
I’m pretty sure I could still “bank” even without a cell phone at all, but I would still be seriously pissed off if I could no longer buy things online. I would switch banks over this.
OK, story time: my bank pretty required me to install an proprietary app on my phone so I can buy things online. I am not aware of any alternative, not even SMS 2FA. I haven’t tested what happens if I root my phone, or (gasp!) switch to a flip phone.
What I’m telling you is, my bank suggested I might not be able to buy things online without an Android or iOS phone.
I have questions about how that situation was reached. Is it actually a blanket policy of the bank that all customers are required to install the bank’s app and that no method of payment is available other than that app?
Or does the bank offer other options – such as a debit card that could be entered into an online form, or the ability to link an account number or debit card to a popular payments system like Paypal or Apple Pay (or whatever Google’s payment system is called nowadays) – but those options had already been rejected by you and the app was offered as a last resort?
The method I use (IIRC it was the only one that was presented to me), is to enter my credit card info into some web form (no need for an app for that), and then I receive a 2FA push notification on my banking app, on which I have to enter a 6-digit PIN code of my choosing to validate.
The notification also appear when I simply try to login to the banking website. I could use the banking app if I knew my banking password, but I don’t: it’s 20 random characters stored in my local password database, and I’m not simplifying it for such a minor convenience.
I also have an array of PIN codes written on a small card that I can use as recovery codes, but but as far as I know they can’t be used for online payments. Perhaps I could contact my bank and cry about having broken my phone or something, but no guarantee.
For some reason my payments on Steam don’t trigger the 2FA.
Linking my credit card to a FAANG is unholy enough that it didn’t even cross my mind. But I don’t recall this option having ever been presented to me. I’m not sure this is even possible since my bank is French.
Here in Thailand most online purchases require scanning a QR code with some garbled code for the app (no 2FA beyond logging into the app required). When you can use a form online, you will always be redirected to an iframe asking for SMS 2FA (no TOTP, no FIDO2, no proprietary app option), but this already assumes you have a SIM & aren’t roaming. On their investment accounts you can opt into some form of proprietary TOTP that is tied to their app, which also won’t run, but that luckily hasn’t been rolled to the general banking (but I could see a future where that is enforced, rather than allowing general TOTP that doesn’t require anything specific). WEI can enforce the same anti-consumer policies onto the web (& banks can/will use this before Google is “broken up” & even if they were Google could still shill for for this pro-DRM garbage corpos love, so bad argument).
WEI can enforce the same anti-consumer policies onto the web
Fuck. Fuck fuck fuck.
We want reliable 2FA that is sufficiently resilient to most hacks. SMS is vulnerable to SIM swapping, and requires a phone. Not good. TOTP doesn’t require a phone (or even a separate device), but it still not phishing resistant. Besides, online payments involve giving your credit card number to some third party, so FIDO2 can’t work there…
Wait a minute, something could work, if not now, at least in the near future:
Customer gives away their credit card number.
Vendor does what it does to require payment.
Customer logs in to their banking website (FIDO2).
Banking website shows pending payment requests.
Customer confirms the payment.
Or better yet, have the bank issue a temporary credit card number, good for one payment of a specified sum (or at least up to some limit).
As someone who has had a phone break in a foreign country with sky-high import fees, I know it. I went to buy a phone online to be shipped to my home upon the return in 5 or so days. I couldn’t do QR scanning for checkout because I didn’t have a phone. I couldn’t do credit/debit cards because SMS 2FA iframe required I was a) in the country & b) had a phone to receive SMS. The fact most phones are brittle & carried everywhere seems absurd they want proprietary apps or these new “private access tokens” that live on a phone and/or in someone else’s cloud.
Banks would likely save money & security headache by giving out FIDO2 keys with every account, but they don’t think about it that way.
…But ideally we’d pick a system that removed the credit card middlemen altogether skimming money for dubious value when direct bank transfers absolutely could be implemented (popular here, & credit cards usually aren’t accepted).
Yeah, facial scanning is something I refuse to use. I don’t trust whoever is holding onto the info or how it could be used on my while asleep or unconscious. Do they demand it?
I think consumers should have the upper hand over banks & corporations in such push-pull scenarios. Attesting the integrity of your app & its updates but not my whole system—games have done this for a while but not prevented you from modifying your PC. I don’t think it’s an unreasonable ask—and should be in line with right to repair where the corporations said you can’t touch the hardware or your warranty is void, etc. Are users wanting right to repair inconsistent in their logic? Consumers could pick another device, except that it got to the point where reasonably they couldn’t. I could argue host blocking and firewalls repair the software side of my system. In the vein, I did switch banks since the previous started checking for custom ROMs/root, but I looked around & I don’t think any of the options available anymore. We lost power.
Are users wanting right to repair inconsistent in their logic?
No, because they’re (usually) not asking for right to someone else’s systems. You are asking for access to the bank’s systems, and the bank is laying out the terms on which that access will be granted. And trying to measure the relative “power” of the parties is not a useful approach and likely would not produce the outcomes you want. For example, should you lose the right to root your phone once you’re above a certain level of net worth? Above a certain level of social influence? Should others be able to just yoink your devices out of your pocket and claim them if the wealth or power disparity between you and them is large enough? These are questions that your approach has to consider and have answers for, and I don’t think you’re going to like where they end up.
Blocking users based on income (which was never suggested) is the same anti-consumer sentiment as blocking a user from being the admin or super user of their device–the same way we’d call out companies making it harder to replace car/motorcycle/tractor parts, or voiding warranty for replacing them ourselves, or locking down the computers in those systems because of “safety”. This isn’t about access to their server’s hardware but access to the APIs (usually simple HTTP) with that system regardless of the OS setup on the client side. They can set some terms about the clients/authentication, but the entire device on the other end? Nah. Hardware was easier for laggard politicians to fight for, but it’s going to be more difficult getting them on board for the freedom of the software too as they largely go unaware of something they don’t buy off a shelf or understand that can modify the OS. Arguing that banks get to be the only ones setting terms for your device (which with Google’s WEI they could close all internet access) is the sort of pro-corporate take I’m not interested in entertaining.
The messaging of the article/infographic is about “boiling the frog”, and how slowly we’ve had more & more of our ability to do what we want with the thing we own away by the corporations who, without regulation, we never get prevent them eroding our technology. This balance isn’t fair, and each individual chip away has been maybe-sorta good, but cumulatively have been detrimental.
Blocking users based on income (which was never suggested)
You seem to think some people deserve the “upper hand” over others, based on relative economic power. I’m not sure how else to characterize that than making one’s technology-related rights inversely correlate to one’s wealth.
This isn’t about access to their server’s hardware but access to the APIs (usually simple HTTP) with that system regardless of the OS setup on the client side.
So is a DDoS OK because it doesn’t actually damage the other party’s server hardware? One can easily do harm via simple network connection without the need for physical contact to the hardware on the other end. And people who run networked services have both a right to defend against such attacks, and a right to set out the terms on which they will allow networked access to their services.
Arguing that banks get to be the only ones setting terms for your device (which with Google’s WEI they could close all internet access) is the sort of pro-corporate take I’m not interested in entertaining.
This is the opposite of the argument I’m making. I’m arguing that the bank has rights to their own hardware, systems, etc. which are equal in force and scope to your rights to your hardware, systems, etc. You don’t seem to like that very much, though, and seem to be arguing, as noted above, that one’s rights to one’s devices (etc.) should be reduced in proportion to one’s economic standing. That’s the only way to get to a stance where you get to enforce policies about your devices but the bank is not able to enforce policies about theirs. And it leads to the sorts of hypotheticals I already raised, where your rights might have to become null against someone of lower economic standing than yourself. I don’t think that’s what you actually want, of course, but it is the logical consequence of what you’re arguing.
I don’t know how you jump from all devices getting to run what the owners decides implies anything about income. One of the best ways to keep old devices running because you can’t afford (or just don’t want to) upgrade is moving to LineageOS or similar or postmarketOS where you can at least keep software updates rolling—especially since the manufacturers don’t see a profit incentive to bother with updates for perfectly fine hardware. To think an individual has any amount of press against a bank/corporation & that there is anything equal in that fight & still trying to hammer it out as fair leads me to believe you are just playing devil’s advocate. Consumers/workers can & should unite against such restrictive policies either by legislation or otherwise—just like right to repair.
Feel free to respond further—but I won’t. I wasn’t going to but you keep pulling this notion out of your butt that I would be arguing it favor of folks with more economic power.
I think consumers should have the upper hand over banks & corporations in such push-pull scenarios.
From the context of that and other things you’ve posted, it seems you think there should be some sort of sliding scale of digital rights versus economic power, where people/entities with more economic power get fewer rights. This, at least, seems to be how you justify giving yourself more of a right to control your own systems and devices than you’re willing to grant to other entities, such as banks, to control their own systems and devices.
What I keep saying, over and over, is that this is a problematic approach, and leads to questions about whether someone with less economic power than you can override your choices and your rights to your devices/systems, the same way you seem to want to override the choices and rights of entities more economically powerful than yourself.
Reduced to its simplest form, suppose we have three people: Alice, Bob, and Carol.
Carol does not want to let Bob access her systems in a certain way. Bob argues “ah, but Carol is significantly more economically powerful than I am, therefore Carol’s rights to make such choices are reduced, and I shall be permitted to access those systems whether Carol wants me to or not. However, I will retain the right to control my own devices and systems.”
But then Alice comes along and says “Bob is significantly more economically powerful than I am, therefore Bob’s rights are reduced and I will make use of Bob’s systems whether he wants me to or not”.
My argument is you are Bob and you are getting upset when presented with the hypothetical of Alice.
Please see some of the other comment chains where I’ve been pointing out how difficult (honestly, I’d say impossible) it is to actually cleanly separate businesses from individuals.
Yes, you can separate them even in sole-proprietorships or taxing under your national ID & folks are pointing it out, but you are having an out-of-touch-Skinner moment where you think everyone else is wrong.
The tax agency has no issues because the tax agency has the concept of taxpayers who fall into multiple categories.
The sliding scale of digital rights being posed here has issues for several reasons, but in the current iteration it has issues because it assumes every entity is at most exactly one of (“business”, “individual”).
But the real answer is to ask why you think you have a right to press on that question. You’d probably be very offended if someone else kept pressing you and pressing you about why you set up and manage your devices and systems the way you do.
This feels like you’re equating people and corporations. I wouldn’t. Human rights are vastly more important than corporations rights. In fact, in fact, I’d say corporation rights only matter as far as they can be derived from human rights. There’s also a difference in scale: suppliers have a wider reach than users. What they do have more consequences, so they warrant more scrutiny.
it is not possible to have a just and coherent philosophy where you get to tell people “my systems, my rules” but others do not.
If it was people on both sides, sure. But it’s not: it’s a user on one side, and a business on the other. We can apply different rules and still be coherent.
If it was people on both sides, sure. But it’s not: it’s a user on one side, and a business on the other. We can apply different rules and still be coherent.
According to the tax agency of the country I live in, I am both an individual and a business. And this is not a terribly unusual situation to be in.
So I don’t think your proposed distinction is one that can be coherently drawn in the first place, let alone used productively to determine policy.
Sure you are. I still submit that your individual rights and freedoms should be separate from your business rights and freedoms.
My “business” is residual royalties from a tech book I wrote years ago. How do you propose to separate that from me? Due to being a “business”, do I not have any rights over my own words?
My opinion on copyright is simple: the thing is too damn expansive. The duration at the very least should be seriously limited. 20 years after publication date, tops. After that it belongs to the public domain (attribution should still be a thing though).
You do have rights over your own words. But if you published your book more than 20 years ago, I can live with denying you any monetary benefit from it.
By “over my own words” I meant my website. Since it’s been infected by business-me on the occasions when I mentioned the book, and thus presumably individual-me’s rights were reduced automatically when that happened.
As I keep saying, your attempt to draw a clear line between businesses and individuals has problems.
As I mentioned in another comment, my “business” is royalties from a tech book I wrote.
How does that affect, say, my personal blog? I mentioned the book in a couple blog posts; does that cause the “business me” to infect the “individual me” and take away my rights to run my website the way I want?
I just don’t think there is a coherent way to draw the line people seem to want to draw here, or to do it in a way that won’t have all sorts of terrible consequences.
There are heaps of laws around commercial speech that aren’t applicable to individuals. There are also heaps of laws around things like, hmm, accessibility or privacy (GDPR, DMA), that businesses have to follow that individuals don’t.
That’s not really what’s being discussed here, though. Remember that the thing that kicked all this off was a complaint that a bank wasn’t supporting a broad a set of mobile devices with a native app as the user would have desired. The bank’s website almost certainly still works, though, and is likely to be compliant with applicable laws around accessibility, for example. Same for the bank’s physical branches, or its customer-service phone line, etc.
But the user wants to override the bank’s choices, and is asserting a right to do so on the basis that they, as an individual, are allowed to “own” their own devices and systems and exercise rights of ownership over them, but that the bank, as a business, somehow should not be allowed to own its own devices and systems or exercise rights of ownership over them.
I’m merely exploring the argument that, once you go down that road, it’s very hard as a supposed “individual” to actually enforce your “ownership” against others, because of how easy it is to wind up being labeled a “business”.
Yes, I frequently cannot use websites because I have “strict” privacy setting enabled in Firefox, and have “resist fingerprinting” enabled. Also running OpenBSD. I am frequently completely blocked from sites (actual 403 forbidden) due to Cloudflare (wtf?), and even outside of that, so many sites just don’t function when their tracking/spyware scripts can’t load. Personal computing is on such a poor trajectory I feel more discouraged about it every day :\
This is a long thread, but you’re suggesting that in the relationship between you and a bank or payment processor, the latter is an adversary? Because this is not the business relationship I’m familiar with.
Does your payment processor use pervasive tracking technology to monitor their consumer “customers” (even cart holders who never check out!) and then sell those data to their merchant customers?
Edit: a little less snarkily, what I mean is: there are multiple ways that payment providers can and do monetise you. It’s not just that they take a clip of the transaction, or charge the merchants a fee - that’s the obvious bit and entirely above board. It’s the rest that’s usually not transparent, and the tracking of users for subsequent monetisation of the business intelligence establishes a more traditionally adversarial relationship between parties.
I switched banks a couple years ago after my banking app started detecting root even with Magisk Hide. I cited that they don’t tell me how to use my laptop so why can they with my general compute phone
They’re not telling you how to use your devices. They’re telling you the terms on which they’ll allow you to access their society, which they own as surely and completely as you own your device, and over which they have rights just as strong as your rights over your devices.
You do not have an inherent right to participate in society, after all. And you don’t, strictly speaking require that participation in order to survive. You can just, like, live in the woods and eat berries, or something. Convenience has a cost, after all, and it’s your choice—as one half of an equitable, mutually beneficial partnership between you and your bank—whether to pay it.
over which they have rights just as strong as your rights over your devices.
I disagree. I think that DRM, anti circumvention laws, and the widespread adoption of proprietary firmwares has dramatically reduced the degree to which people have rights over their “own” devices.
Though I still agree with your central tenet: that people can choose their providers, and vote with their wallets if they turn evil. I just changed ISPs for that reason, myself. Capitalism works :)
I quite like Firefox’s Tab Stash extension for that, where I periodically put all my deduped open tabs into an unnamed collection showing the date of creation.
I zap all my open tabs 2-3x a week. I can think of maybe one or two things I haven’t been able to find again over the past ~15 years. I’ve come to terms with lossy information retention.
I try to do it once a day. I’ve come to realise that if it’s important and worth caring about (it almost always isn’t), I’ll either remember it or be able to find it in my browser history.
That’s it. All my tabs, gone, and I never knew how to recover them. I use Firefox on Windows, and it happens to me about once or twice a year.
Edit: sometimes I get a warning about closing a window with multiple tabs opened. I believe the reason for this warning is because there is no “undo” on this: if I go through the data is gone for good.
In Firefox, the History menu should list recently closed tabs and windows. For a whole window, Ctrl/Cmd-Shift-N should revive it.
If that fails, there should be several backups of your recent sessions in your Firefox profile directory. Check the sessionstore.js file and other files in the sessionstore-backups directory.
Double-check that you have session restore enabled in Firefox settings (“Open previous windows and tabs”).
I have. It shows the history, most notably the most recently opened pages. I haven’t found anything relating to tabs there, especially what I wanted: recently closed tabs.
I’m always slightly surprised when I meet people who actually use Haskell. I consider it one of the languages like Smalltalk or Prolog that you should primarily learn because if makes you a better programmer in any language, not because you should actually use them (Erlang is almost in that category, but there are some places where it would definitely be my go-to choice).
I’ve been using Haskell professionally for about a decade, I’m currently in my first job since uni where I’m not using it for production systems. There are few languages that all you to build as maintainable systems as Haskell. Yes there is a shallow learning curve (steep learning curves mean you learn a lot quickly), but the payoff is being able to produce systems which you can change drastically and have machine checked help with the compiler having your back constantly.
I’ve used it for processing medium data on AWS with a fairly AWS specific pipeline of services, and during my time we rearchitected the fundamentals of the how the app ran twice, from a single monolithic containerised app, to splitting it into microservices, to using AWS Lambda, all in the same code base, without any bugs in business logic throughout. We knew that we could make the change we thought we needed to make, and then follow the errors until it compiled again (I couldn’t tell you how long it took, as there was obviously a lot of concurrent deployment changes, but the Haskell side was generally done is one, maybe two two-week sprints).
We also did all this while much of the tooling we today didn’t exist, or was in its infancy; HLS is an amazing productivity booster that I wish I’d had over much of the past decade. There are warts, yes, but things have massively improved, and all of that without having the dedicated teams and financial backing other languages like Go and Rust have had behind them. Most of these tools have no more than a handful of people contributing to them, and are achieving great results.
It makes for an amazing language where concurrency matters, its threading is as cheaper or cheaper than Erlang’s, and immutable by default with very explicit mutation where needed means your business logic can be clearly expressed without fear that you’ve forgotten some state - the data you have is all you need to produce the results. Performance is also excellent; it might not be C, C++ or Rust fast, but it’s in the ballpark, and the much nicer concurrency story than those three makes up for it for me, when you’re building production systems.
Haskell’s type system is IMO its number one asset, and I disagree when people say you should only stick to the simplest types so it’s easier for others to learn. People can be taught, and we’re doing engineering, not playing with lego. Most other languages feel to me like building bridges by putting stones and logs on top of each other, Rust has figured out that you can bind them together with lime, where Haskell has moved onto using timber trusses. We haven’t as an industry figured out how to do engineering to the level of building trusses and cables from steel, and we’re a long way from finite element analysis to model out software-bridges, but Haskell gets us moving in the right direction. Engineering is all about building things which are predictable, model-able, and Haskell’s type system gives us a glimpse of that, by letting you make illegal states unrepresentable, and allowing us to precisely define what cannot be done, not just what can.
It’s always struck me as bizarre that we expect professional programming to just be easy for everyone - anyone can buy an Arduino and make it do some stuff, but I want the people designing the power grid, my motherboard, and the avionics systems in planes to meet a significantly higher standard of rigour; why don’t we expect the same from professional software engineering?
Haskell’s benefits can feel tedious in the small, but I have witnessed them absolutely pay off in the large. One aspect of the community is that we don’t tend to boast too much about it; why should we, we’re just getting the job done, we’re not trying to start a cult like some members of other language communities seem hell bent on. Facebook is processing millions of requests per second in Haskell to filter spam, and having non-programmers write that Haskell; banks around the world are internally building large systems in Haskell, and have been doing so many years at this point; NASA are using it for hard real-time monitoring of high assurance systems, and ESA are using a derivative (Clash) to design the hardware for terabit space communications laser systems. To name just a few.
I could go on much longer, but as has been the case for the past 15 years, the rumours of Haskell’s death have been greatly exaggerated yet again. It is still worth learning, it is still worth building real systems in, and is only getting better at doing it as time goes on.
It’s always struck me as bizarre that we expect professional programming to just be easy for everyone - anyone can buy an Arduino and make it do some stuff, but I want the people designing the power grid, my motherboard, and the avionics systems in planes to meet a significantly higher standard of rigour; why don’t we expect the same from professional software engineering?
I couldn’t agree more. I’m flabbergasted by the obsession with simplicity, and unwillingness to learn in our industry. I can’t believe the success of Go. To me, it is a terrible language, working with it would be a nightmare, and yet people love it for “basically being ifs and loops”. Zig is coming up hot because it’s mostly C. How can you have this job be that unwilling to learn?
Sorry for the rant comment, but that stuck with me since forever.
I couldn’t disagree more :-) I love learning new languages but there are a lots of advantages to simplicity besides being easier for novices to pick up. In my job I have to go in unfamiliar codebases all the time and edit other people’s code, and that’s a lot easier when the language stops them from being too clever. Even in my own projects, fancier languages make me feel taxed trying to decide what’s the right way to do something instead of focusing on the problem at hand. In Rust, I see long discussions all the time where someone has code that already works, but they want to compact it down to an ideal form using just the right combinators. That rarely happens with Go because you “just write a loop” and move on, saving all that mental energy. I’m not necessarily saying I prefer Go over Rust all the time, but I do in this respect. You could just write boring loops in Rust too, but it matters what the language and community shepherd you to do.
Also, I want to defend Zig! I’ve been learning it recently and really enjoying it. The fact that it’s “mostly C” is a killer feature that lets you integrate it with any existing C/C++ project or libraries. It has several innovations which fix many of C’s problems while avoiding C++ level complexity. It takes a fresh approach to memory allocation which is worth checking out for any systems programmer “willing to learn”.
The thing is, I don’t actually have a problem with Go and Zig. I don’t want to use them, but that doesn’t mean I want to see them fail. What annoys me so much, is that they are successful, while other languages that I like are not. I really like Haskell, and I’m not really using it, mostly for the reasons detailed in OP’s post (although, for me, the ecosystem is the largest problem). But all those downsides are the consequence of a small community. I’m angry because these communities are so small, while Go is big, and Zig will probably become big.
Having C-compatibility is certainly extremely useful, but you can link C-libraries in Rust too. There is a big margin between “just loops and ifs” and “too clever”. I think Haskell without most compiler extensions has a pretty reasonable level of complexity. Sure, GADT are over the top, but there is a hell of a lot of potential over Go before it becomes too clever. Give me ADT, generics and a great Macro system. Rust is close, but I think it’s still too verbose, has way too many iter() calls everywhere, no HKTs, and function signatures tend to be terrible with non-trivial generic code. Also, rust proc macros are too much effort, closures and async also have a lot of problems. I would love to have Rust with these things fixed. I don’t think that would be too complicated. But it won’t happen, because too many people already think Rust is too complicated. And THAT is what tilts me.
Coming into someone else’s code is exactly when you want these features - you have absolutely no idea what the constraints and invariants on the system are, and in a language like Go, you probably never will. In Rust or Haskell, it is at least possible to teach the compiler what should be disallowed by new developers. People look at Python or JavaScript and think they’re simple, but they’re exponentially more complex than strongly typed languages because the state space of any piece of code is actually infinite. This makes it genuinely impossible to make changes and be certain you’ve made the correct changes.
Prioritising writing your first code is a fundamentally flawed idea for a business that expects to be developing for a long time. Optimising for the ability to make changes without breaking things. “Move fast and break things” only needs to exist when you can’t avoid breaking things, and that is the case in a lot of the YOLOLangs we see today - but powerful type systems are finally taking their rightful place with languages like Rust, Typescript becoming popular and learning from the lessons Haskell has been teaching for decades.
If I prioritize productivity, getting things done fast, I’d say Go. If I prioritize correctness, getting things done ‘right’ (contra Worse is Better), I’d say Rust, which I imagine would protect better from the mistakes that the programmers will make. Because I prioritize correctness and have very unimpressive productivity, I imagine I would be fired and never get to make that decision. :-)
We’re talking about real humans here so no 100% that’s not going to happen. Though I think as a tech lead I could drag a small team through the onboarding curve with Rust. Just remember: .clone() is your friend.
I’m the other way around; I think that Haskell memes are misleading at best and damaging at worst, but Haskell itself seems like a reasonable language. It’s one of the few popular languages to not be set-oriented, using DCPO instead of Set for its collection of types.
Yeah. Haskell’s semantics are not necessarily categorical, but if they were, then they seem to fit well with CPOs of some sort. To me, this alone is quite enough to merit study.
I can assure you Haskell has its place as a go-to language. I’ve used it professionally and for fun for 8 years now and it’s great for fleshing out (and keeping up to date) complex data domains, be it the core of a SaaS backend or a compiler. Not to mention its concurrency story (e.g. the stm library) which is pure magic.
It was my default for a while, because at the time (2007 or so?) it was the best safe language to write Unix daemons in. It also had the best C FFI of any of the mature languages at the time.
I still use Hakyll for my website, but I don’t write it much anymore besides that. I don’t write much code outside work at all these days.
I read Learn You a Haskell and thought it was a pretty cool language. So I was excited to try it out, but once I took a good look at the ecosystem and tooling around the language I was so put off I just couldn’t bring myself to continue. I’m really glad I read the book though.
We almost exclusively use Haskell at bitnomial. We’re a CFTC-regulated futures/options exchange. We’re able to build amazingly reliable distributed systems at scale with a pretty small team. It’s definitely not the best choice for everyone, but it’s worked phenomenally for us. Complex distributed systems with lots of concurrency and millions of dollars riding on us getting the logic right are made tractable with Haskell.
Not OP. But I know people that would use Erlang and, given the choice, not write Elixir at all.
Personally I tend to write my libraries in Erlang and applications in Elixir. The main reason for this is that it is harder to use Elixir code from an Erlang application, but super simple to use Erlang code in an Elixir application. Obviously this does not work all the time. For example, if I was writing an Ecto adapter, I would write it in Elixir.
I do not know the current status of it, but I know there were issues with the process at one point. It is also less ideal actually calling Elixir code from Erlang. A normal Erlang function call would look like
my_module:my_function()
Whereas calling an Elixir function from Erlang is
'Elixir.MyModule':my_function()
And then the opposite, calling an Elixir function from Elixir,
MyModule.my_function()
And an Erlang function from Elixir,
:my_module.my_function()
And that is only calling functions. There are other differences that you would need to be aware of as well which makes calling between languages less ergonomic than is ideal.
I’ve never used Elixir. I am one of the three people in the world who like Prolog and so Elixir’s move away from Prolog-like syntax isn’t a selling point for me. Last time I looked at Elixir, they had made some things better and some worse in my highly subjective judgement.
On the subject of simplicity, I think one axis that, IME, divides people 50/50 is whether we’re talking about intentional or extensional simplicity. I.e. which one do you think is simpler in a car; Automatic transmission or manual transmission? Automatic transmission has simpler external “semantics”, while manual transmission has a simpler implementation.
Some people think Nix is unnecessary complexity because it wraps everything and does unspeakable things, but I think Nix is essential simplicity, because it does these things to make their semantics simpler.
You can’t make something simpler by piling on complexity.
This is a tautology from an extensional perspective and largely irrelevant from an intensional one. Thank you for demonstrating the other side of the argument.
If you watch the Defcon talk, Dildog talks about the code not being entirely ready, and the docs not being done. He mentions he needs a month or so to wrap things up. Clearly, it wont be 100% at that point, but it will be more than now.
This seems to be largely missing the point of a TPM. The lack of programmability is not a bug, it is a feature. The device provides a set of interfaces that are generic, so the same TPM can be used with multiple operating systems easily and so that the behaviour of those interfaces can be relied on. I can have a system that dual boots Windows and Linux and will allow Linux to unlock my LUKS-encrypted root volume and Windows to unlock my BitLocker-encrypted root volume, and allow Linux to expose my SSH keys and Windows to expose my Windows Hello credentials, but without allowing either operating system to access the other’s state and, crucially, guaranteeing that compromising either does not allow the attacker to exfiltrate keys, only to perform online attacks.
Relying on DICE just punts the problem to another hardware root of trust. You can’t rely on an inductive system to prove your base case. If every machine comes with its own programmable device then I need to have some mechanism for my OS to trust ones that have been validated and, most importantly, when a bug is found in a particular version, I need a mechanism for anti-rollback so that I can guarantee that an attacker can’t bypass my security by just installing an old version of the firmware. I can’t do that validation in my OS because it relies on secure boot for integrity and a secure boot chain gives me an attestation that the only things that can compromise my boot are things earlier in the chain, so the thing I’m trying to validate is in the set of things that can compromise my chain. So now I need a root of trust that can boot my TPM replacement and can load its firmware.
It’s very easy to design a simpler alternative to a TPM if you avoid solving any of the hard problems (it’s almost certainly possible to design something better that does solve the hard problems, because TPM is, uh, not the best spec in the world). Many of the techniques in the article actually describe how TPMs are implemented, but providing direct access to the keys to the programmable bit of the RoT is something that no reputable RoT has done for a good ten years because it is terrible for security. A device built along these lines would be vulnerable to a load of power and probably timing side channels and likely to glitch injection attacks and a load of other things that are in scope for the TPM’s threat model.
The lack of programmability is not a bug, it is a feature.
As far as I understand it’s only a feature for Treacherous Computing, which requires a Root of Trust that is outside the control of the end user. If there’s one feature of the TPM I really really don’t care about, it’s this one.
Besides, you can achieve that with DICE anyway: the manufacturer can just provision a TPM-like firmware at the factory and issue a certificate for that. Users could change the firmware, but only factory-approved firmware would enjoy a factory-issued certificate.
The device provides a set of interfaces that are generic
“Generic” is not quite the right word. “Exhaustive” comes closer. The TPM for instance doesn’t have generic support for arbitrary signature algorithms. It has specific support for RSA, ECDSA, Ed25519… That’s just hard coding a gazillion use cases and hoping we covered enough to call that “generic”.
[…] and, crucially, guaranteeing that compromising either does not allow the attacker to exfiltrate keys, only to perform online attacks.
This has nothing to do with DICE. Just write firmware that don’t leak their own CDI and you’ll be okay. Which is pretty easy to do when each piece of firmware you write is focused on one single use case. Besides, manufacturers today write firmware hoping it won’t leak the root secret that is stored in fuses. They have the exact same problem you seem to criticise DICE for, only worse: if the root secret is leaked and it can’t be re-fused, the chip is scrap.
Relying on DICE just punts the problem to another hardware root of trust. You can’t rely on an inductive system to prove your base case.
But we do have a base case: it’s the bootloader, that derives the CDI from the root UDS and the firmware it loads. That bootloader is tiny and provided by the manufacturer. Surely they can shake out all the bugs from a C program that hardly requires a couple hundred lines? At these sizes even the compiled binary could be audited.
Firmware with actual functionality is another matter, but as I said in the article, they’re easier to make than one generic firmware to address all use cases. And if one has a bug that leaks its own CDI, we can correct the bug and the new version will have its new uncompromised CDI.
If every machine comes with its own programmable device then I need to have some mechanism for my OS to trust ones that have been validated
Ah, Treacherous Computing. As I’ve said, just add that capability to the manufacturer’s default firmware. That way the OS can check the firmware’s key against the manufacturer’s certificate.
Well, strictly speaking the OS is a bit late to check anything, and Secure Boot doesn’t work to begin with. If however the hard drive is encrypted by keys derived from the CDI, decryption is only possible when the approved firmware runs (else we get useless keys). Then the OS running at all is proof that we were running the correct firmware, and if the DICE chip is integrated into the main CPU (so MitM is not possible), its manufacturer issued firmware can refuse to give up the encryption keys if the bootloader isn’t right.
most importantly, when a bug is found in a particular version, I need a mechanism for anti-rollback so that I can guarantee that an attacker can’t bypass my security by just installing an old version of the firmware.
Assuming updates are even a thing… old firmware let the new firmware in, checks it is signed by the manufacturer, checks that it is a newer version, new firmware is swapped in, new public key (yeah, the CDI changed, so…) is sent to the manufacturer (on a secure channel of course), and the manufacturer issues a new certificate.
To prevent rollbacks my first thought would be a persistent increment-only hardware counter, that would be hashed together with the CDI instead of using the CDI directly. If every firmware version does this, incrementing the counter instantly change the keys of older versions. With the wrong keys they’re no longer approved, and rollback attempts are detected as soon as we check the certificate. We don’t even have to explicitly revoke old keys with that approach.
That was the first idea that popped up, we can probably do better.
It’s very easy to design a simpler alternative to a TPM if you avoid solving any of the hard problems
The problems the TPM solve aren’t hard, they’re many.
Hardware-wise, manufacturers of current secure elements can easily add DICE capability at no loss of security. They have the hardware secret sauce to do it. Unlike Tillitis, they don’t have to use an FPGA whose only threat model was making a token effort to protect their customers’ proprietary bitstream from IP theft.
Software-wise, DICE automatically makes thing easier by (i) allowing us to only address the uses cases we care about, and (ii) addressing them separately. Even if end users are derps, the manufacturer can publish a bunch of official firmware for the most popular use cases. They’re already doing that after all.
providing direct access to the keys to the programmable bit of the RoT is something that no reputable RoT has done for a good ten years because it is terrible for security
Which is why DICE does not do this. DICE firmware does not, I repeat, does not access the root key of the device. Only the tiny bootloader does. The main firmware only get access to a derived key that is specific to it, and it alone. Compromising the root key of the device from the main firmware is flat out impossible.
Sorry for re-stating the obvious, but it’s hard to interpret what you just wrote under the assumption that you understood that, I’m not sure what’s your point here.
A device built along these lines would be vulnerable to a load of power and probably timing side channels and likely to glitch injection attacks and a load of other things that are in scope for the TPM’s threat model.
No, you’re just talking out of your ass here. You need to explain how data can flow from secrets to the side channels with the DICE approach, in a way that they do not with the regular approach. You also need to be aware of the actual threat model: the timing side channel is almost always relevant, but in practice easy to address — even Daniel J. Bernstein said so. Glitch attacks and power analysis are harder, but they require physical access and as such are out of many threat models. They’re important when the user is the enemy, but you know what I think of Treacherous Computing.
For instance, it’s reasonable to assume that under the normal operation of the HSM, its immediate environment is trusted enough not to purposefully inject glitches or analyse power. It’s not always a valid assumption, but not everybody is making a credit card to be plugged into many untrusted terminals.
Then there’s the question of what can be extracted at which point. The bootloader is fixed, and hashes the program and secret together. We already know of ways to mitigate the power side channel, it’s to hash the UDS in its own block, and hash the rest of the program starting on the next block. (Ideally we’d want a constant power hash, but for RAX designs, constant power addition is very expensive, so let’s settle for the mitigation instead.)
Glitching the loading of the program would just yield a different CDI, not sure we can do much there. And once the program is loaded, access to the UDS is closed off, so no amount of power analysis and glitching can shake that off (assuming adequate protection of the latch, but that should be easy, compared to protecting the integrity of the entire program’s runtime state.)
As far as I understand it’s only a feature for Treacherous Computing, which requires a Root of Trust that is outside the control of the end user.
No, any form of security requires the root of trust to be out of control of the attacker. The attacker is assumed to be able to compromise the later stages of execution because the DICE model is based on inductive proofs and gives you guarantees only about the base state of a system that has unbounded states later. Each step is, effectively, promising not to do certain bad things (e.g. GRUB promises not to lie about the kernel that it has loaded) but once you get to a general purpose OS you can run arbitrary code. Once you get to this point, you lower layers are expected to lock down the system such that an attacker cannot do specified bad things. For example, the TPM may have unlocked certain keys (often implicitly by using a KDF over a secret mixed with some PCR values) but does not permit that key to be exfiltrated, so an attacker who compromises my OS cannot steal my SSH keys or disk encryption keys (which means that they can do online attacks during their compromise).
This has nothing to do with DICE. Just write firmware that don’t leak their own CDI and you’ll be okay. Which is pretty easy to do when each piece of firmware you write is focused on one single use case. Besides, manufacturers today write firmware hoping it won’t leak the root secret that is stored in fuses.
That is absolutely untrue for the hardware RoTs that I’ve worked with. They have fixed function hardware that stores keys and expose a set of operations on them (including ACLs that authorise what operations a key can be used for and what derived keys can be used for). They run the TPM stack on the programmable core, but they assume that this can be compromised. An attacker who compromises the TPM stack has very little more access than someone who compromises the first-stage boot loader on the host.
Assuming updates are even a thing… old firmware let the new firmware in, checks it is signed by the manufacturer, checks that it is a newer version, new firmware is swapped in, new public key (yeah, the CDI changed, so…) is sent to the manufacturer (on a secure channel of course), and the manufacturer issues a new certificate.
Okay, so now you need your device to be able to make network connections or you need to rely on the OS to provide that channel. Now you have a very wide attack surface. Being able to support this use case was actually one of the motivations for CHERIoT because it isn’t feasible with existing hardware within the threat models of people who rely on TPMs.
Glitch attacks and power analysis are harder, but they require physical access and as such are out of many threat models. They’re important when the user is the enemy, but you know what I think of Treacherous Computing.
Okay, I think this is where we disagree. The primary use case for a TPM, for me, is protecting my data if my machine is stolen. There’s no point using disk encryption if an attacker also has access to the key. You seem to think that DRM (‘repeating the phrase ‘Treacherous Computing’ does not inspire confidence, you seem to use the phrase to dismiss all of the things that people have done to improve security by assuming malicious intent) is the only threat model where attackers have physical access. I think that attackers with physical access (whether that’s via an ‘evil maid’ attack or outright theft) or who have compromised my OS are the only use cases where I want a hardware root of trust. If a system isn’t robust in those two cases, then it fails my threat model. By moving access to keys and crypto algorithms out of fixed-function units and my moving more of the code on the programmable core into the TCB, you make it harder to defend against this threat model. I’m not sure what your threat model actually is, you might have a good solution for it, but it isn’t the one that I care about.
any form of security requires the root of trust to be out of control of the attacker.
Okay.
The attacker is assumed to be able to compromise the later stages of execution because the DICE model is based on inductive proofs and gives you guarantees only about the base state of a system that has unbounded states later.
I don’t like this induction analogy, but even then we do have a base case: the DICE bootloader that is a fixed function. The firmware on top has its own key and therefore can be integrated in the root of trust (one with actual functionality this time), and, well… just like a TPM it’s not supposed to reveal its own secrets.
If we can’t do bug-free firmware, we’re kinda doomed anyway. Speaking of which…
That is absolutely untrue for the hardware RoTs that I’ve worked with. They have fixed function hardware that stores keys and expose a set of operations on them (including ACLs that authorise what operations a key can be used for and what derived keys can be used for).
You feel like you’re contradicting me, but I feel like you’re making my case for me: those fixed function they wrote, they have to make sure they don’t have any bug that would expose the secrets, right? This is exactly analogous to the firmware I spoke of. Nothing stops this firmware to provide fixed functions to the untrusted host…
…unless you need the variable functions on top to be trusted anyway, kind of security by layers. We’d need something like a 2-stage execution environment, where the first can set up some functions, and the second is denied access to the CDI of the first stage, but can issue commands to it nonetheless. I guess DICE isn’t enough for that. Whether this complexity is worth the trouble is another question though.
The primary use case for a TPM, for me, is protecting my data if my machine is stolen.
I use a password for that. It’s easier for me to trust a secret that’s not even on my machine. (Though ideally I’d use a password and a secure element.)
I think that attackers with physical access (whether that’s via an ‘evil maid’ attack or outright theft) or who have compromised my OS are the only use cases where I want a hardware root of trust.
Protection against theft should be straightforward. Evil Maids however are very powerful: a key logger, something that intercepts the video signals… I’m afraid those would be hard to detect, and mitigating them would pretty much require every major component of the laptop to contain a secure element so all communications between component can be encrypted and authenticated. This is no picnic.
My, if I ever come to actually fear an Evil Maid, there’s no way I’m leaving my laptop at the hotel.
By moving access to keys and crypto algorithms out of fixed-function units and my moving more of the code on the programmable core into the TCB, you make it harder to defend against this threat model.
Only to the extent the firmware is more likely to have bugs than fixed-function units. Keeping the firmware small and specialised improves our odds dramatically. Because of course, the attacker can’t change DICE firmware without changing the keys. If they do, the disk simply won’t decrypt. Their only chance is to exploit a bug in the firmware.
I don’t like this induction analogy, but even then we do have a base case:
Then you probably don’t like DICE, since the entire model is based on inductive proofs of security.
the DICE bootloader that is a fixed function. The firmware on top has its own key and therefore can be integrated in the root of trust (one with actual functionality this time), and, well… just like a TPM it’s not supposed to reveal its own secrets.
I am really struggling to understand how the security claims that you’re making map back to things that the security proofs about DICE give you.
You feel like you’re contradicting me, but I feel like you’re making my case for me: those fixed function they wrote, they have to make sure they don’t have any bug that would expose the secrets, right?
Right, and they do that by being implemented in a substrate that is amenable to formal verification, including side-channel resistance. In particular, it has no arbitrary control flow (everything is dataflow down wires, control signals are data), and it has almost no dynamic dataflow (values can flow only to places where there are wires). Doing the same thing on a general-purpose programmable core that does not provide hardware memory safety is vastly harder. It is easy to validate that keys cannot be used for the wrong purpose because those accesses have values coming from the relevant bit in the ACL and are anded with that value. There is no way of getting a key except via a wire from the register file that contains the key.
…unless you need the variable functions on top to be trusted anyway, kind of security by layers. We’d need something like a 2-stage execution environment, where the first can set up some functions, and the second is denied access to the CDI of the first stage, but can issue commands to it nonetheless. I guess DICE isn’t enough for that. Whether this complexity is worth the trouble is another question though.
And then you decide that you need to protect two different things at the higher level from each other. And two different things at the lower level. And now you have TrustZone and two privilege modes. Only now you have shared resources between different trust domains, so you have side channels. Again, there’s a reason people stopped building hardware RoTs like this! The TPM is intentionally inflexible because as soon as you start allowing programmable functionality you need multiple trust domains, as soon as you have multiple trust domains you need both fine-grained sharing and strong isolation and those are both hard problems.
I use a password for that. It’s easier for me to trust a secret that’s not even on my machine.
So you enter the password into your bootloader and now it’s in memory. Anyone who manages to compromise your OS once can exfiltrate it. The entire point of a security coprocessor is to ensure that keys are not accessible to someone who compromises your OS.
But, again, you haven’t given me a threat model. It seems like you’re happy with the level of security that you get from no hardware and you’re unhappy with the level of security that you get from a TPM, but somehow want something different. You’ve written a crypto library, so I presume you know what a threat model looks like: what capabilities to you assume your attacker has? What are your attackers’ goals? What tools will they be able to construct from their base capabilities towards that goal? How do you prevent them from doing so? If you start from a threat model, we can have a reasonable discussion about whether your approach addresses it, but it seems like you’re starting from a bunch of technologies that people building secure elements stopped using ages ago because they aren’t sufficient for the threat models for these devices, and saying that they’re a better way of building secure elements.
it seems like you’re starting from a bunch of technologies that people building secure elements stopped using ages ago
To be honest I don’t believe you. Please give me evidence.
I am really struggling to understand how the security claims that you’re making map back to things that the security proofs about DICE give you.
Okay, it’s simple: you start with two things:
Trusted DICE hardware/bootloader
Trusted firmware to load on top
You load the firmware on the hardware, and this hardware+firmware pair constitutes an HSM. That’s a bit strange, but strictly speaking DICE hardware is not by itself an HSM. It only becomes so when loaded with a particular piece of firmware. (In practice the bootloader alone doesn’t really count, since it doesn’t provide any functionality users want. Users want actual cryptographic services, loading firmware is only a means to that end.) Anyway, with trusted hardware and trusted firmware, you now have a trusted HSM. Ask for its public key, and register that as “trusted”.
Now let’s say you have those 3 things:
A trusted public key
An untrusted piece of DICE-looking hardware
An untrusted piece of firmware
Testing whether you can trust this hardware+firmware pair is simple:
load the firmware into the hardware (you now have an HSM).
Demand proof from your HSM that it has the private key matching the trusted public key.
And that’s it. Notice the similarity between that and a fixed-function HSM: it’s the same, you just need to start by loading firmware you don’t trust yet. The actual verification after that is exactly the same as for a classic HSM.
Of course this all hinges on the guarantee that changing the firmware unpredictably changes the CDI. That we trust the hardware+bootloader to provide that guarantee. If you have reasons to distrust this guarantee I’d like to know why.
Now what about Dragons?
Right, and they do that by being implemented in a substrate that is amenable to formal verification, including side-channel resistance.
You’re clearly assuming that firmware is not amenable to formal verification. This is blatantly false: even regular cryptographic libraries can be formally analysed, and some have been. Including side channel resistance. And since firmware runs on a specified piece of hardware, that verification is even easier.
In particular, it has no arbitrary control flow (everything is dataflow down wires, control signals are data), and it has almost no dynamic dataflow (values can flow only to places where there are wires).
No, you’re making a category error there. Programs (barring the self modifying relics) have a fixed control flow, just like hardware. When we talk about arbitrary control flow we don’t talk about programs having it, we’re talking about our ability to make or load arbitrary programs. Once the program is chosen the control flow is frozen.
Same thing for the dynamic flow: cryptographic libraries have almost no dynamic flow, and that’s what make them secure against the most important side channels. So while some pieces of firmware can have dynamic data flow, not all of them have, and obviously a sane cryptographic engineer would only trust the ones that has as little of it as is reasonable.
Doing the same thing on a general-purpose programmable core that does not provide hardware memory safety is vastly harder.
Again a category error, I believe. This is not about guaranteeing anything about arbitrary firmware, we want to guarantee that a particular piece of firmware is void of memory errors (among other errors). Hardware memory safety tricks like ASLR or executable XOR writeable help, but you can completely sidestep the problem and write your firmware in Rust.
What you call “vastly harder” is at worst a speed bump.
To be honest I don’t believe you. Please give me evidence.
Most of the details of these things are protected by NDAs, but if you let me know which manufacturers you’ve talked to I can point you in the right direction in their designs, if we’ve worked with any of the same people.
Demand proof from your HSM that it has the private key matching the trusted public key.
That works but only because you’re sidestepping all of the hard problems. As a user, I don’t care about the code identity I care about the guarantees. That’s always the hard bit in any attestation system and if you tie software on the host to specific versions of the RoT then you end up with something very fragile. In the RoT designs I’m familiar with, this is sidestepped entirely as a problem. In a TPM, the useful guarantees I get are all about negative properties: it’s not programmable and so I know you cannot do things that are not exposed, I don’t have to prove a completeness property over some general-purpose code (ignoring firmware TPMs, which were the worst idea ever and were all broken by Spectre if not before).
You’re clearly assuming that firmware is not amenable to formal verification. This is blatantly false: even regular cryptographic libraries can be formally analysed, and some have been.
Yes and no. I’ve worked with the EverCrypt team and they got further than most. When we tried to use their code in production, we discovered that their proofs of temporal safety were holding only because they never freed memory (not great for a resource-constrained environment). Those proofs also depend on compiling with CompCert and may or may not hold with other compilers. Most importantly, they hold only if no code in the system that they are linked with ever hits undefined behaviour. If, for example, you have a memory safety bug in the parser for mailbox messages from your host, none of the proof in EverCrypt hold because you have just violated one of their axioms.
With the Low* work, they are able to prove that the implementations are constant time, in conjunction with a modest microarchitectural model. They are not able to prove anything about power because most ALUs have data-dependent power consumption. The techniques used to harden against these in hardware (e.g. running identical pipelines in parallel that that XOR’d inputs so that the power is always uniform) are simply not expressive in instruction sets. The Low* based proofs also depend on their being no speculative execution (if I can train a branch predictor to go to the wrong place, it doesn’t matter that there are no timing leaks in the correct path) but that’s probably fine.
Programs (barring the self modifying relics) have a fixed control flow, just like hardware. When we talk about arbitrary control flow we don’t talk about programs having it, we’re talking about our ability to make or load arbitrary programs. Once the program is chosen the control flow is frozen.
No, pretty much every program will use the branch and link register instruction. Proving anything about control flow is now a data flow problem: you have to prove that the set of values that will reach the input of that instruction is constrained. This is possible only by assuming that you have constrained memory access, so you’re returning to the ‘memory safety is an axiom’ world. Now, with CHERIoT, we can make that guarantee, but you won’t find it in any other embedded ISA.
So while some pieces of firmware can have dynamic data flow, not all of them have, and obviously a sane cryptographic engineer would only trust the ones that has as little of it as is reasonable.
This only helps the verification if you have strong compartmentalisation. Again, not something provided by most non-CHERIoT embedded systems.
Again a category error, I believe. This is not about guaranteeing anything about arbitrary firmware, we want to guarantee that a particular piece of firmware is void of memory errors (among other errors). Hardware memory safety tricks like ASLR or executable XOR writeable help, but you can completely sidestep the problem and write your firmware in Rust.
ASLR is totally inappropriate, it barely works when you have a full 32-bit address space, you have no chance of getting useful levels of entropy doing it without an MMU. Even if it did work, it’s a probabilistic defence and so is inappropriate for building a foundation for formal verification. Immutable code is the default on embedded devices (many are Harvard architecture) but you’re still vulnerable to code reuse. You can write in Rust, but you can’t verify anything about timing in Rust because the abstract machine isn’t in Rust. Anything involving your assembly routines and anything involving I/O will be in unsafe Rust code, so there’s still scope for bugs even if you verify all of the safe Rust code (Rust verification tools are improving but they’re nowhere near the level that you’d require for a system like this).
I say vastly harder based on experiences of teams that I have worked with. They have shipped hardware RoTs and formally verified crypto implementations. The software verification effort took more people and took longer. The hardware RoT was deployed, at scale, to people with physical access some of whom had a large financial incentive to break it. It is still doing fine with no compromises. The verified crypto implementation was broken within a week of internal testing by finding bugs in the integration with the surrounding system. I consider the thing that multiple vendors have done and that has stood up to attacks in production vastly easier than the thing that hypothetically could be done but has not actually worked any time someone has tried it, but maybe that’s just me. If you want to prove me wrong and ship a formally verified firmware stack for a RoT that is resistant to timing and power side channels and glitching attacks, I’d be very happy to see it: even if I never use it, the tools that you’d have to build along the way would be game changers for the industry.
But you still haven’t told me what problem you’re trying to solve or what your threat model is.
Most of the details of these things are protected by NDAs
Fuck, I forgot about that. Evidence not available, then. Fuck. Still, I care less about implementation details than I care about the user-visible interface. Do those poor sods also put these under NDA? Like, they sell a chip with an ISA, and then they tell you “this ISA is a corporate/state secret, sign the NDA please”?
I bet they do. Fuck them.
But you still haven’t told me what problem you’re trying to solve or what your threat model is.
Same threats as anyone else. I’m just trying to have something simpler achieve the same goals. Because as a user, I just can’t stand these huge TPM specs. Specs that apparently had a critical vulnerability, discovered March of this year. Not in the hardware, not in the software. In the spec.
I say vastly harder based on experiences of teams that I have worked with.
I have an hypothesis: software teams are drooling incompetents compared to hardware teams. Maybe it’s less about the actual difficulty of software, and more about how hardware teams understand the stakes of what they’re doing and are trained (and tooled) accordingly, while software teams simply don’t and aren’t. I still remember this blog post by someone who worked with hardware teams, and then noticed how software teams just didn’t know how to test their stuff.
you can’t verify anything about timing in Rust
I’m aware of at least one language that can (forgot the name, it’s designed specifically for cryptography), and I believe cryptographic libraires have been written in it. Good point about having to verify the compilers as well though. That obviously has been done, though whether it was done with the relevant verified language(s) I don’t know.
No, pretty much every program will use the branch and link register instruction.
Jump and link to a hard coded constant. Unless indirect calls are involved, but that would typically mean the main program calls arbitrary code… Could be used internally of course, and we’d found we could still account for the jump and link destination… or just avoid indirect calls indirectly, if that makes verification easier.
Now if it’s something as trivial as “call foo if condition1, call bar if condition 2”, well… first of all that’s not constant time, but let’s say we’re doing signature verification and don’t care about leaking data just yet: a fixed function hardware equivalent would do exactly the same. How could it not, at some point there is stuff to do.
The techniques used to harden against these in hardware (e.g. running identical pipelines in parallel that that XOR’d inputs so that the power is always uniform) are simply not expressive in instruction sets.
Which means constant power can’t be achieved in software, I’m aware. This is a definite disadvantage, and it does exclude some use cases (like credit cards). I maintain that you don’t always need to address the energy side channel. Or even electromagnetic emissions.
That works but only because you’re sidestepping all of the hard problems. As a user, I don’t care about the code identity I care about the guarantees. That’s always the hard bit in any attestation system and if you tie software on the host to specific versions of the RoT then you end up with something very fragile. In the RoT designs I’m familiar with, this is sidestepped entirely as a problem.
How as a user would you have any guarantee? There’s only one way I’m aware of, it goes in 2 steps:
Trust the manufacturer to give you a secure HSM (any guarantee you may hope for is entirely dependent on how trustworthy the manufacturer actually is).
Use public key cryptography to verify the identity of that HSM.
That’s true of any HSM, DICE or not. And what are you on about with “tying to a specific version of the RoT”? Isn’t that what happens anyway? Doesn’t the manufacturer have to sign the entirety of the chip, firmware included, regardless of the approach taken? The only additional difficulty I see with DICE is that since changing the firmware changes the keys, updates to the firmware are harder. But that’s quite moot if the alternative is fixed functions etched in immutable hardware: can’t update those at all.
But then DICE does an advantage: someone other than the manufacturer could write the firmware and vouch for it. A downstream user (or trusted secondary shop) can then load the firmware and register its keys.
In a TPM, the useful guarantees I get are all about negative properties: it’s not programmable […]
Neither is DICE firmware. A dice HSM is not programmable by default, it has a “fixed-function” firmware, identical except in implementation details to fixed-function hardware.
One could load arbitrary firmware onto their DICE hardware, but only one such firmware has been signed and can be checked for identity. It can’t be swapped for anything else. Programs aren’t general purpose, the CPU running them is. Freeze the program and you get a special purpose appliance.
(ignoring firmware TPMs, which were the worst idea ever and were all broken by Spectre if not before)..
Perhaps, but since fusing the TPM to the main execution chip is the only way to have actual hardware security… that can be better than a discrete chip, even one that has a level-magical hardware security.
Fuck, I forgot about that. Evidence not available, then. Fuck. Still, I care less about implementation details than I care about the user-visible interface. Do those poor sods also put these under NDA? Like, they sell a chip with an ISA, and then they tell you “this ISA is a corporate/state secret, sign the NDA please”?
The ISA for the programmable part is often open but that’s not the interesting bit. The interfaces to the fixed-function units and the protection models are often NDAs. Some of the security also depends on exactly what you do in your EDA tools. Apparently if you try to fab Pluton with the default Cadence config it happily optimises away a load of mitigations. RoT vendors don’t like talking about their security features in public because it paints a target on them, they’re happy to talk about them to (potential) customers.
That’s less true with things like OpenTitan or Calyptra, but I haven’t looked at either in detail. I am pushing for both to adopt the CHERIoT Ibex as the programmable component because it enables some quite interesting things. We can statically verify (in the binary) which compartments in a firmware image have access to which bits of MMIO space, so we can allow (for example) an Intel-provided compartment to have access to back-door interfaces to the CPU that let it attest to CPU state, but not have access to the key storage, and that no component except the Intel-provided compartment has access to this bit of the state, so only Intel can compromise the security of the Intel processor using the RoT.
Same threats as anyone else.
So, to be clear, the attacker capabilities are:
Has physical access.
Can control power to the device.
Can retry things an unlimited number of times and measure timing.
Can retry things an unlimited number of times and measure power.
Can try installing other firmware versions and then run anything that the device does not prevent.
The attacker’s goal is to exfiltrate a key that will allow them to produce an emulation of the device that is indistinguishable from the real device, from the perspective of software.
The requirements for software are such that I must be able to:
Identify that the device is the device that it thinks it is.
Perform signing and encryption operations from a host without access to keys held in the device.
Restrict what keys can be used for which purposes.
Upgrade the firmware of the device without losing access to keys.
Share the device between mutually distrusting hosts without granting any of them the ability to impersonate another (including accessing keys that are private to that host but sharing ones that are private to the device owner).
Upgrade the firmware of the device on one system without another system losing access to its keys.
I probably missed something.
But it sounds like you also want to be able to run arbitrary code on the device to perform OS-specific functionality (again, this is actually one of the target use cases for CHERIoT because it’s really hard).
I’m just trying to have something simpler achieve the same goals. Because as a user, I just can’t stand these huge TPM specs. Specs that apparently had a critical vulnerability, discovered March of this year. Not in the hardware, not in the software. In the spec.
I don’t think we disagree that TPM is a clusterfuck, but I think it’s an interface that you can run on a secure system (we’ve actually run the TPM reference stack on a CHERIoT implementation), I just don’t think that you’re solving most of the problems that I want a RoT to solve.
I have an hypothesis: software teams are drooling incompetents compared to hardware teams
Absolutely not in this case. The folks on the EverCrypt project were some of the smartest people I’ve met (the kind of people I’d trust to build the theorem prover that I depend on, not just the kind that I’d trust to use it correctly).
AMD and Intel routinely mess up security critical things.
I’m aware of at least one language that can (forgot the name, it’s designed specifically for cryptography), and I believe cryptographic libraires have been written in it.
F* / Low*, which is what EverCrypt used.
Good point about having to verify the compilers as well though. That obviously has been done, though whether it was done with the relevant verified language(s) I don’t know.
Kind of. Formally verified compilers come with a lot of caveats. CompCert, for example, does not guarantee anything if your input code contains any undefined behaviour. This is fine for F*, which generates C code from an ML dialect that guarantees no UB, but makes CompCert pretty useless for C code written by humans. Even then, it’s not completely clear how well this works because both F* and CompCert have a formal model of C semantics but it might not be the same formal model of C semantics and any mismatch can invalidate the proofs.
Formal verification for hardware also comes with some caveats. My favourite hardware security PoC was a TrustZone exploit in a verified Arm core. They ran two wires close together so that if you rapidly toggled the value in a register you’d induce a current in a wire that led to the S state bit and would let you enter S state from unprivileged code. The core was correct at the RTL layer but not correct with respect to analogue effects. RoT designs typically also include mitigations at these layers but they have to be designed based on the specific circuits and it’s really hard to put them in general-purpose cores. The threat model for an AES engine is much easier to express than the threat model for an add instruction (are either of the operands of the add secret? It depends on the surrounding code. Is it preferable for a glitch to give a lower or a higher value for the result of an add? It depends on the surrounding code).
This is why you typically put the programmable logic outside of the trust. The bit that runs the TPM stack is not part of the TCB for key confidentiality or integrity. It is responsible for some bits of PCR state manipulation and command parsing, but if you compromise it then you still can’t exfiltrate keys. You might be able to add different things to PCR state, but that’s about it.
Jump and link to a hard coded constant. Unless indirect calls are involved, but that would typically mean the main program calls arbitrary code
Return instructions are also computed jumps (jump to the link register). A single stack bug can hijack control flow this way. Hence memory safety being a prerequisite for CFI.
a fixed function hardware equivalent would do exactly the same. How could it not, at some point there is stuff to do.
It’s about the negative cases. It’s easy to verify that a key stored in a key register never travels somewhere else: don’t put wires anywhere other than the input to the crypto engines. You can statically enumerate all of the possible dataflow paths. It’s much harder to verify that a value stored at some location in memory is never read by anything that can lead to I/O because you have to ensure that no load instruction that leads to I/O can read that address. That’s a global alias analysis problem.
How as a user would you have any guarantee? There’s only one way I’m aware of, it goes in 2 steps:
It’s a hard problem. It’s simpler if you can enforce constraints. If the TCB logic is fixed function, your proof obligations on the programmable bit are much less. Ideally, the programmable bit is in the TCB for availability and not confidentiality or integrity, so you just punt on it entirely. Now you have a much simpler problem of getting an attestation over the device, rather than the device plus a software stack. The attestation is a claim from the manufacturer that it provides some security guarantees.
That’s what I mean about the guarantees. Identity is a building block for attestation, it’s not sufficient. The important thing is that manufacturer X makes claims about software Y running on hardware Z. If these claims are untrue, you can sue them. That’s what you’re getting from attestation: accountability that you can use to build legal liability. In the worst case, you need to universally qualify these claims over all values of Y because the OS doesn’t want to carry a list of valid firmware versions. Often, you make slightly weaker claims that rely on the ability to have monotonic versions of Y (e.g. ‘I promise that this gives security guarantees as long as you’re running the latest firmware and, if there are security bugs in the firmware I will fix them within a week’), which is where you get the requirements to be able to do secure update with anti-rollback protection.
But then DICE does an advantage: someone other than the manufacturer could write the firmware and vouch for it. A downstream user (or trusted secondary shop) can then load the firmware and register its keys.
DICE (with slight tweaks) is used in most of these things already. The problem is not the firmware, it’s the space of things that a firmware image can possibly do and how you get a guarantee that your firmware is secure in the presence of all of the capabilities listed for attackers above. And it’s about ensuring that things like:
If I find a bug in firmware version X, I can’t then deploy firmware version X to get the keys from someone who wants to be running version X+1.
If I find a bug in firmware version X, I can’t extract all of the keys before the user has a chance to upgrade to X+1.
Those are the hard problems. DICE is a small part of solving them, it is not sufficient by itself.
Imagine I have one of these devices and I have an emulator for one that gives me complete visibility into all of the secret keys. How do you enable me to move the keys from firmware X on the real device to firmware X+N on the real device but not to:
Firmware X-1 (which has a known vulnerability that leaks keys) on the device.
Firmware X on the emulator.
Firmware X+N on the emulator.
DICE is an important building block for enforcing this kind of policy, but it’s nowhere near sufficient by itself, and without being able to enforce that policy you have no security for your secrets and may as well just store them in the kernel.
This gets even harder when you’re using it as part of your secure boot chain (which is what I want from a root of trust) because any change to the DICE signature for the RoT’s firmware will change the PCR values for the next steps of the boot. I can’t rely on code that loads after the RoT’s firmware to verify its attestation because it is able to tamper with any of that code, so I must be able to provide an attestation over trust properties from the RoT initialisation that I can use for the PCR values, not an attestation simply of identity.
Perhaps, but since fusing the TPM to the main execution chip is the only way to have actual hardware security… that can be better than a discrete chip, even one that has a level-magical hardware security.
This is why the good designs have a separate core in the same package as the main chip. If you want to see it done really well, take a look at the Xbox One. There’s not much public, but you can probably find people who have reverse engineered bits of it.
Looks like we’re converging. Thanks for the detailed reply.
So, to be clear, the attacker capabilities are:
Has physical access.
Can control power to the device.
Can retry things an unlimited number of times and measure timing.
Can retry things an unlimited number of times and measure power.
Can try installing other firmware versions and then run anything that the device does not prevent.
Well if the NSA gets a hold of Snowden’s computer I guess that’s what we have. I do reckon under this model though that there’s no escaping the energy side channel. Which kind of means masking everything that ever accesses the secrets, and… my that’s expensive. And we can probably forget about RAX designs (I’m told masking addition is expensive), and stick to stuff like AES and SHA-3 instead.
One thing that seems impossible to mitigate though is bignum arithmetic. How can we deal with elliptic curves or RSA when everything must draw constant power? Do we switch to curves with binary fields? I’m told the security literature is less sold on their mathematical security than prime fields, but that’s the only way I can think of to avoid multiplying stuff.
Or we bite the bullet and make a general purpose CPU where every operation that is naturally constant time, is masked to be constant energy as well. Addition, multiplication, bit manipulation… all constant energy, guaranteed by the CPU. And while we’re at it stay in-order and remove speculative execution, cryptographic code is unlikely to benefit from out of order speculative cores anyway. Making a CPU like this is probably a bear, but this could have the tremendous advantage of making constant time code automatically constant energy.
I won’t dispute that is a hard problem, and I understand the appeal to limit constant energy hardware to specialised operations instead.
The requirements for software are such that I must be able to:
Identify that the device is the device that it thinks it is.
Perform signing and encryption operations from a host without access to keys held in the device.
Restrict what keys can be used for which purposes.
Upgrade the firmware of the device without losing access to keys.
Share the device between mutually distrusting hosts without granting any of them the ability to impersonate another (including accessing keys that are private to that host but sharing ones that are private to the device owner).
Upgrade the firmware of the device on one system without another system losing access to its keys.
I’m not sold on everything here. Specifically:
Restrict what keys can be used for which purposes.
If the device derives one key pair for encryption, and another key pair for signature I kind of get your wish. But if I want to perform signatures for Debian packages and personal emails, I could allow users to use different keys if I accept a domain separation string, but I can’t prevent them from using the same domain separation string for two different purposes.
Also, isn’t the guarantee that different domain separation strings causes us to use unrelated keys enough?
Upgrade the firmware of the device without losing access to keys.
Upgrade the firmware of the device on one system without another system losing access to its keys.
It’s a nice to have if firmware can’t be expected to be bug free, but I’m not entirely sure which use case requires that not only the keys are preserved, but the upgrade can happen in an hostile and offline environment. For instance, if we have encrypted secrets and need to change the firmware, we could decrypt the thing, change the firmware, then encrypt the thing back. That simple procedure only works in a trusted environment, but I’m not sure how much of a show stopper this really is.
The ISA for the programmable part is often open but that’s not the interesting bit. The interfaces to the fixed-function units and the protection models are often NDAs.
Protection models are implementation details, so that’s not too bad. An open ISA is good. The interface to the fixed-function units however is part of the ISA: the firmware needs to call it one war or another.
I would almost forgive the vendor for not disclosing the exact algorithm used under the hood. Is it AES-GCM? ChaPoly? I don’t really care if I know where to put the plaintext, where to retrieve the ciphertext, and how big the authentication tag is. But I do need to know how to interact with the unit, so I hope this particular part is not under NDA.
But it sounds like you also want to be able to run arbitrary code on the device to perform OS-specific functionality
Yeah, I want the end user to have a choice what firmware they use. The primary use case would be the ability to use different cryptographic primitives (the old one is broken, or the new one is faster, whatever).
I also want to enable end users to load arbitrary firmware they would then trust on first use. Yes, this means their entire computer, including the insecure parts, is assumed trusted as well. But there’s a difference between trusting your computer now, and trusting it for 2 years straight working abroad and leaving your laptop at the hotel. It’s a freedom/security trade-off mostly.
I don’t think we disagree that TPM is a clusterfuck
Thanks. But there’s something more fundamental about it, I think: the TPM specifically supports a lot of cryptographic primitives, and a lot of use cases. Because it kinda has to: too many users to satisfy there. Doing something similar for a single customer having a precise use case in mind would automatically divide the size of the specs by a couple orders of magnitude.
At the same time though, the generality (if not genericness) of the TPM is desirable, and DICE seems to be a nice way to keep that generality while keeping things simple. Well, simple if you punt on the firmware, at some point some one does have to write bug-free code. Hence the “which is harder, software or hardware?”. I’ve always thought the two were comparable, and bet on tailored software being much smaller than kitchen-sink hardware, and as such, proportionally easier.
If I find a bug in firmware version X, I can’t then deploy firmware version X to get the keys from someone who wants to be running version X+1.
It’s kind of cheating, but DICE gives you that out of the box: firmware X and X+1 are different, so they get different keys. One could load firmware X and exploit a bug to get its keys, but this wouldn’t reveal the keys of X+1.
If I find a bug in firmware version X, I can’t extract all of the keys before the user has a chance to upgrade to X+1.
Depend what you mean. Keys accessible from X are toast, and there’s no way the user can perform an upgrade from X to X+1 in an untrusted environment. But if user can consider themselves “safe enough” to trust the environment around the chip, they can perform the upgrade and just assume no MitM screws up the update process.
How do you enable me to move the keys from firmware X on the real device to firmware X+N on the real device but not to:
Firmware X-1 (which has a known vulnerability that leaks keys) on the device.
Firmware X on the emulator.
Firmware X+N on the emulator.
We may have a solution:
Run old firmware.
Load new firmware in memory, and any associated certificate.
Have the old firmware check the new firmware’s certificate and version number.
If it all checks out (new firmware is newer and certificate is valid), then:
Put the CDI in a special persistent-ish region (it only needs to survive one single reboot).
Lock that special region with the new firmware’s hash.
Load new firmware
If the hash of the new firmware matches the lock, give it reading access. Otherwise just wipe the special region.
New firmware does whatever it needs to migrate from the old CDI to its own new CDI.
Maybe we could use fixed-function units instead, but I’m deliberately avoiding those to minimise hardware requirements. In any case, I agree DICE is not enough. Though if my special region trick works, it’s pretty close. Now, assuming no exploitable bug in the relevant firmware:
Can’t downgrade, the old firmware will prevent that.
Can’t move to the emulator, that would mean extracting the CDI, and the CDI doesn’t leave the device to begin with.
If I’m using version X and want to upgrade to version X+2, skipping X+1 because that one has a critical key extraction vulnerability, I’m probably screwed if I can’t get access to a revocation list (and most likely a reliable clock as well), I’m liable to be tricked into upgrading into X+1. The only mitigation I see here is upgrading regularly, hoping exploits for X+1 don’t have time to get in effect before I upgrade to X+2.
Oh, and one huge red flag about my approach: the CDI is transferred from one firmware to the next, so the very mechanism we use to mitigate vulnerabilities, is itself a source of vulnerabilities! I really don’t like that kind of… security/security trade-off.
Sorry if I missed something, my screen is no longer big enough to fit a useful subset of your comment and my reply on them and so you’re relying on my attention span, which is severely impacted by the nice weather.
Well if the NSA gets a hold of Snowden’s computer I guess that’s what we have. I do reckon under this model though that there’s no escaping the energy side channel. Which kind of means masking everything that ever accesses the secrets, and… my that’s expensive. And we can probably forget about RAX designs (I’m told masking addition is expensive), and stick to stuff like AES and SHA-3 instead.
These attacks are now feasible with hardware that costs a couple of thousand dollars. Five years ago it cost tens of thousands of dollars. Within the lifetime of the device, I expect it to be hundreds of dollars.
One thing that seems impossible to mitigate though is bignum arithmetic. How can we deal with elliptic curves or RSA when everything must draw constant power? Do we switch to curves with binary fields? I’m told the security literature is less sold on their mathematical security than prime fields, but that’s the only way I can think of to avoid multiplying stuff.
It’s non-trivial but there are techniques for building constant-power large multipliers. I expect to see some of these things exposed in a slightly more generic way as people start caring more about post-quantum security (for devices with a 10+ year lifetime, support for post-quantum encryption is now a requirement, but no one knows what the right algorithm is).
Or we bite the bullet and make a general purpose CPU where every operation that is naturally constant time, is masked to be constant energy as well. Addition, multiplication, bit manipulation… all constant energy, guaranteed by the CPU. And while we’re at it stay in-order and remove speculative execution, cryptographic code is unlikely to benefit from out of order speculative cores anyway. Making a CPU like this is probably a bear, but this could have the tremendous advantage of making constant time code automatically constant energy.
It’s not just that it’s hard, the problem is that it impacts power / clock frequency for everything. With a clean split, you don’t care too much about leaks from the general-purpose core because that’s outside your TCB for confidentiality and integrity and so you can make it fast / efficient and you can make the fixed-function bits fast and secure but less power efficient.
If the device derives one key pair for encryption, and another key pair for signature I kind of get your wish. But if I want to perform signatures for Debian packages and personal emails, I could allow users to use different keys if I accept a domain separation string, but I can’t prevent them from using the same domain separation string for two different purposes.
It’s mostly about defence in depth (which is a good principle for the whole system). For example, for WebAuthn, you really want to have a single secret that’s used with a KDF and some other data to generate a key that’s used for signing. You want to enforce the policy that the secret used with the KDF never leaves the device and is not used except as input to a KDF. You also want to enforce a policy on the derived keys that they also never leave the device and are used only for signing. This makes it harder for a compromised OS to leak the key (especially if you also throw in some rate limiting).
It’s a nice to have if firmware can’t be expected to be bug free, but I’m not entirely sure which use case requires that not only the keys are preserved, but the upgrade can happen in an hostile and offline environment.
My original post had one: I am dual-booting Linux and Windows, using BitKeeper for encrypting my NTFS partition and LUKS2 for my ext4 one. Linux doesn’t trust Windows, Windows doesn’t trust Linux. With the TPM today, both can hold disk encryption keys (which can be further protected by a PIN / password) that the other can’t use. If an attacker installs malware that compromises the NT kernel and gets full access to the TPM interface, they still can’t decrypt my Linux partition.
From this use case, it follows that either Linux or Windows should be able to upgrade the firmware on the device, without destroying the utility for the other. If everything that’s using the device needs to cooperate in updates then I have a difficult operational problem and the likely outcome is people don’t update the firmware and keep running vulnerable versions.
Given that even formally verified software isn’t bug free (formal verification aims to ensure that all of your bugs exist in the spec, sometimes it just guarantees that they exist outside of your abstract machine), I think assuming that the firmware contains bugs is a safe axiom.
Protection models are implementation details, so that’s not too bad. An open ISA is good. The interface to the fixed-function units however is part of the ISA: the firmware needs to call it one war or another.
Probably quibbling about semantics, but to me the ISA is the set of instructions that run. Interfaces to things that run outside of the main pipeline may be architectural but they’re not part of the instruction set architecture.
I would almost forgive the vendor for not disclosing the exact algorithm used under the hood. Is it AES-GCM? ChaPoly? I don’t really care if I know where to put the plaintext, where to retrieve the ciphertext, and how big the authentication tag is. But I do need to know how to interact with the unit, so I hope this particular part is not under NDA.
Typically you don’t even get the datasheets for these things without an NDA. The algorithms used may be in marketing material (because people looking for FIPS compliance read those before bothering to sign the NDA). How keys are protected and the security model is definitely NDA’d in most cases. In a few cases because the offerings are utter crap and everyone would point and laugh, others because they’re doing quite clever things that they don’t want competitors to copy.
Thanks. But there’s something more fundamental about it, I think: the TPM specifically supports a lot of cryptographic primitives, and a lot of use cases. Because it kinda has to: too many users to satisfy there. Doing something similar for a single customer having a precise use case in mind would automatically divide the size of the specs by a couple orders of magnitude.
Maybe. On the other hand, if you get a chance to look at what Apple does in their Secure Element, you might long for the days of something as simple as a TPM. Feature creep is really easy. A couple of requests that I’ve seen for people to run on the RoT:
OS-free firmware updates. Fetch firmware for all devices, validate them, and install them. Requires an SR-IOV VF from the NIC assigned to the RoT and requires the RoT to run a full network stack. If you can do it, it’s really nice because it lets you sidestep things like the NVIDIA vulnerability from yesterday.
Fingerprint recognition logic for unlocking keys without the OS being involved. Requires a connection to the fingerprint reader (oh, that’s a USB device, so now we need a USB stack) and to run some small ML model (so might also need to be control plane for an ML accelerator).
Once you have a secure box, everything wants to live in the secure box. See also: TrustZone.
At the same time though, the generality (if not genericness) of the TPM is desirable, and DICE seems to be a nice way to keep that generality while keeping things simple. Well, simple if you punt on the firmware, at some point some one does have to write bug-free code. Hence the “which is harder, software or hardware?”. I’ve always thought the two were comparable, and bet on tailored software being much smaller than kitchen-sink hardware, and as such, proportionally easier.
I still don’t think that DICE gives you enough unless you can ensure that your firmware is bug free. You need some signing infrastructure, attestations, and a trust model on top. And that’s complicated.
It’s kind of cheating, but DICE gives you that out of the box: firmware X and X+1 are different, so they get different keys. One could load firmware X and exploit a bug to get its keys, but this wouldn’t reveal the keys of X+1.
But that’s not solving the problem. I need to be able to get the keys from firmware X+1 because otherwise a firmware upgrade locks me out of everything I’m using this device for.
We may have a solution:
That kind-of works. The problem is that you can brick your device (or, at least, lose access to all keys) by installing a buggy firmware that doesn’t allow updates. You need to build some kind of A/B model on top. Normally this is done by running firmware A, it installs firmware in slot B. Set a flag so that B boots next but A boots after that. B then boots and runs some checks. B then updates the flag so that B always boots next and deletes A.
With your model, the first time B boots, it would be able to decrypt the keys and reencrypt with its CDI. This makes me super nervous because that’s the point the firmware has access to I/O and to the keys, so I just need to find one buggy firmware that’s newer than the current one to exfiltrate all of your keys. You really need to make sure that you keep up to date with updates (a few iPhone jailbreaks have worked this way: don’t upgrade for a while, wait for a vulnerability to be found, upgrade to the vulnerable version, run exploit).
This does meet my requirement for an untrusted OS being able to do the upgrade though and it’s probably easy for the driver on the host to do a periodic version check and push out updates if there’s an old version so you need to actively cooperate with an attacker to allow the delayed-upgrade attacks.
I’m happy now that there is a process, but it’s taken a thread four times longer than your original post to get there. This is the kind of thing that keeps me coming back here, thanks for your patience!
I still think you want to do a lot more in fixed-function logic if you want something secure against someone with more than a few hundred dollars to throw at breaking it though.
I’m happy now that there is a process, but it’s taken a thread four times longer than your original post to get there. This is the kind of thing that keeps me coming back here, thanks for your patience!
Thank you for yours. I found this process in no small part thank to this discussion, and I’ve learned a few things too. I don’t think there’s much I seriously disagree with any more, so I’ll just reply with some of my thoughts.
my reply on them and so you’re relying on my attention span, which is severely impacted by the nice weather.
Heat wave ongoing at home, my nights are no cooler than 28°C… and I’m kind of high on sleep deprivation. Next summer we’ll definitely install an A/C.
Probably quibbling about semantics, but to me the ISA is the set of instructions that run. Interfaces to things that run outside of the main pipeline may be architectural but they’re not part of the instruction set architecture.
Makes sense. I prefer to us a slightly more expansive definition: the ISA is everything I need to know to make software for a piece of hardware. To me it’s a slightly more useful definition because it really defines the contours of what I want to know, and the limits of what I believe is acceptable for an NDA.
Typically you don’t even get the datasheets for these things without an NDA.
Fuck them I guess, then? I want full ownership of what I buy, and that includes the right to explain how to use it. Hopefully these NDAs only happen in business-to-business transactions, where those freedom considerations matter a lot less.
These attacks are now feasible with hardware that costs a couple of thousand dollars. Five years ago it cost tens of thousands of dollars. Within the lifetime of the device, I expect it to be hundreds of dollars.
Okay, this is so much worse than I thought. I guess I can assume any police department or dedicated criminal have those, or will soon. Great. Now hardware security requires power side channel resistance. That sets expectations I guess. Good to know regardless.
It’s non-trivial but there are techniques for building constant-power large multipliers.
That’s good. Though I’m not entirely sold on the utility of huge multipliers. For instance when I compare Libsodium (that uses 128-bit multipliers) with Monocypher (that stops at 64 bits), the advantage of the bigger multipliers is only about 2x. Now the actual measure is screwed up by the crazy complex out of order architecture, I don’t know how many multiplier units there are in my CPU, and haven’t looked at the assembly. Still, I suspect that roughly speaking, the speed of bignum arithmetic is roughly proportional to the length of your biggest multiplier. (Schoolbook multiplication suggests a quadratic relation instead, but bigger multipliers are most likely slower.)
It may therefore be enough to use smaller multipliers and chain them or loop with them (with a hardware control unit, microcode, or even firmware). And if we can have constant power multipliers, then constant power everything is not so far out of reach. Though again, given the consequences on hardware design, this probably means sticking to a simple and weak CPU.
I am dual-booting Linux and Windows, using BitKeeper for encrypting my NTFS partition and LUKS2 for my ext4 one.
Ah, that one. It slipped my mind. That looks legitimate indeed. Still, I would like to try and cop out of this one by using small firmware.
The idea is simple: what if the firmware in this particular case is used only for OS boot, and maybe there’s a standard so everybody agrees on, if not a single cipher-suite, say a very small set thereof? If the thing does nothing more than measuring code and giving the go/no go, then all you need is a KDF? That’s a couple hundred lines of code at worst, significantly smaller than even TweetNaCl. So we throw all the formal methods we can, like writing a certified F* compiler that outputs RISC-V code directly, test the hell out of this thing… and maybe we’ll never need to update it?
Perhaps I’m grasping at straws, but I did say it was a cop-out.
I still don’t think that DICE gives you enough unless you can ensure that your firmware is bug free. You need some signing infrastructure, attestations, and a trust model on top. And that’s complicated.
Bug-free firmware is (almost?) as critical as bug-free fixed functions, no doubt about that. Which is why I want to keep it as small as possible. Just please don’t destroy my dreams…
On the other hand, if you get a chance to look at what Apple does in their Secure Element, you might long for the days of something as simple as a TPM. […] Once you have a secure box, everything wants to live in the secure box.
…You just destroyed my dreams. Well, we both saw that thread on no one actually wanting simplicity. No solution there, except perhaps implanting a bomb in their hearts that will explode if someone manages to exploit a bug in the wild.
To be honest, excluding cryptographic primitives and communication library, to me the maximum acceptable firmware size is around 50 lines of C code. 200 as an absolute maximum. If it has to be any higher, this seriously impacts the trust I have in it. As for the size of the cryptographic code, anything bigger than Monocypher is a bust in my opinion. We may not have the same tastes with respect to fixed-function units vs firmware, but I do agree on one thing: what we trust the keys with should be really, really, really small…
…even if in practice it won’t be.
That kind-of works. The problem is that you can brick your device (or, at least, lose access to all keys) by installing a buggy firmware that doesn’t allow updates. You need to build some kind of A/B model on top.
I’m guessing that it’s easy to test that the new firmware still allows update. A/B is safer in that respect, but that’s still more stuff to add to the thing, and as always I want the bare minimum.
This makes me super nervous because that’s the point the firmware has access to I/O and to the keys, so I just need to find one buggy firmware that’s newer than the current one to exfiltrate all of your keys. You really need to make sure that you keep up to date with updates (a few iPhone jailbreaks have worked this way: don’t upgrade for a while, wait for a vulnerability to be found, upgrade to the vulnerable version, run exploit).
I’m nervous for the same reason. The iPhone jail can take hike though. I guess most users would have regular control over their machine, and can update regularly… except they can no longer do that once their computer is stolen by a determined adversary.
Hmm, so hardware security mandates that no firmware, current or future, can ever be exploited into exfiltrating the keys. So we want to update it as infrequently as possible. This means minimising the reasons for updates, so the API of the firmware has to be very stable. Which it is more likely to be if we manage to keep it small. Again.
Makes sense. I prefer to us a slightly more expansive definition: the ISA is everything I need to know to make software for a piece of hardware. To me it’s a slightly more useful definition because it really defines the contours of what I want to know, and the limits of what I believe is acceptable for an NDA.
That’s still architecture (as opposed to microarchitecture, which may change between versions or across vendors) but it’s not instruction set architecture. The difference between the two is important in a lot of cases (for example, the Arm GIC specification is architectural, but it’s not instruction set architecture).
That’s good. Though I’m not entirely sold on the utility of huge multipliers. For instance when I compare Libsodium (that uses 128-bit multipliers) with Monocypher (that stops at 64 bits), the advantage of the bigger multipliers is only about 2x.
As you say, this makes more difference on smaller in-order cores. It’s also worth noting that a few multiplication units have special cases for multiplies by zero, which means that you may find that splitting into smaller parts introduces power and timing side channels.
It’s far more important for post-quantum algorithms though. These seem to involve a lot of huge (on the order of KiBs) numbers that need multiplying so having a nicely pipelined big number multiplier can improve performance and let you do all of the power / perf optimisations that you want in your normal multiply (assuming you have one - a lot of embedded cores lack hardware multiple or divide).
The idea is simple: what if the firmware in this particular case is used only for OS boot, and maybe there’s a standard so everybody agrees on, if not a single cipher-suite, say a very small set thereof? If the thing does nothing more than measuring code and giving the go/no go, then all you need is a KDF? That’s a couple hundred lines of code at worst, significantly smaller than even TweetNaCl. So we throw all the formal methods we can, like writing a certified F* compiler that outputs RISC-V code directly, test the hell out of this thing… and maybe we’ll never need to update it?
Unfortunately, that’s exactly what the TPM was supposed to be (though it also needed to define the communication protocol, which is a critical part of the system). Once you’ve covered all of the use cases, I think you’ll end up with something almost as complex as the TPM spec. Actually, possibly worse because doing it now people would insist on some post-quantium signature algorithms and the ability to plug in new ones after there’s consensus on the right ones to use.
To be honest, excluding cryptographic primitives and communication library, to me the maximum acceptable firmware size is around 50 lines of C code. 200 as an absolute maximum. If it has to be any higher, this seriously impacts the trust I have in it. As for the size of the cryptographic code, anything bigger than Monocypher is a bust in my opinion. We may not have the same tastes with respect to fixed-function units vs firmware, but I do agree on one thing: what we trust the keys with should be really, really, really small…
The bit I’d be most worried in is the protocol parsing and I/O code and you’d be lucky to get that down to 200 lines of code. Fortunately, verification of protocol parsing is quite easy. One of the spinoffs from EverCrypt is a thing that lets you define a binary protocol and will then generate C serialisers and deserialisers. As long as you sprinkle enough volatile in there that the compiler doesn’t introduce TOCTOU bugs, you’re probably fine.
But I think you’re coming around to part of my world view which is that the key thing that you need from the hardware is support for fine-grained compartmentalisation. If you have compartmentalisation, you can make the amount of code that has access to the keys tiny and move the rest of it out of your TCB for key confidentiality.
I’m nervous for the same reason. The iPhone jail can take hike though. I guess most users would have regular control over their machine, and can update regularly… except they can no longer do that once their computer is stolen by a determined adversary.
That’s the problem and why I consider most confidential computing things to be dual-use technologies: it’s very hard to build something that can be used for letting me run code on a computer that I’ve rented without the owner being able to corrupt or inspect it, but doesn’t allow DRM-like applications. Once you think of them as dual-use, that leads to treating them in law like other dual-use technologies and regulating the use, not the technology. I’d personally love to see fair use strengthened in statute law such that any form of DRM that prevents the end user from exercising their fair use rights is, if not corrected within a small time window, grounds for immediate revocation of copyright. If you rely on vigilante justice then you don’t get the protection of law.
Hmm, so hardware security mandates that no firmware, current or future, can ever be exploited into exfiltrating the keys. So we want to update it as infrequently as possible. This means minimising the reasons for updates, so the API of the firmware has to be very stable. Which it is more likely to be if we manage to keep it small. Again.
You should take a look at the CHERIoT platform. I think we’re providing you with a lot of the building blocks that you need. Oh, and the first consumer hardware (hopefully shipping next year) will come with OpenTitan so you get some fixed-function crypto bits and a DICE implementation out of the box, in addition to object granularity memory safety and (up to) function-granularity compartmentalisation.
That’s still architecture (as opposed to microarchitecture, which may change between versions or across vendors) but it’s not instruction set architecture. The difference between the two is important in a lot of cases (for example, the Arm GIC specification is architectural, but it’s not instruction set architecture).
Oh, what I called ISA you call “architecture”. Makes sense. Besides, all I want is a shortcut to point to what I mean, so “architecture” it is.
It’s also worth noting that a few multiplication units have special cases for multiplies by zero, which means that you may find that splitting into smaller parts introduces power and timing side channels.
Crap, I forgot about those. Well obviously if we design a CPU to use in a secure element we wouldn’t use that kind of shortcut.
It’s far more important for post-quantum algorithms though. These seem to involve a lot of huge (on the order of KiBs) numbers that need multiplying
I didn’t know about those. If most such algs do indeed multiply huge numbers together, huge multipliers have more value than I thought.
The bit I’d be most worried in is the protocol parsing and I/O code and you’d be lucky to get that down to 200 lines of code.
Okay, let’s check what Tillitis have done with their bootloader. It’s one of their biggest firmware from what I could gather. So, their protocol parser seems to be about 200 lines, give or take, though their coding style makes it artificially high. I think my coding style would squeeze it in 100. They have a 150 lines standard lib, which I don’t think I can meaningfully reduce. They have a BLAKE2s implementation for the KDF of course. Their main is fairly big, over 300 lines.
OK, maybe I was a little optimist there. I’ll need to try stuff out, maybe there’s a way to compress their code further, or at least put the common bits in a library. About that: one neat thing the TKey does, is give access to parts of the code that made the bootloader. Most notably BLAKE2s. That way programs may avoid bringing some of their own code, making not only their source code smaller, but their binary as well, and increase available RAM in the process.
The same thing can be done with the basic communication facilities. You could have a simple API that let you send and receive messages of limited size, and handle the tag & size for you. If it’s good enough everyone would use it, and the only job left is parsing what goes inside the messages. Which is easy if we’re sticking to reasonable fixed or TLV formats. Which we can: we’re controlling both sides of the channel.
I’ll need to experiment to know for sure.
That’s the problem and why I consider most confidential computing things to be dual-use technologies
That’s an excellent point, that actually influences an unrelated essay I may write soon: in the Andor StarWars series (highly recommended by the way), there’s the notion of imperial tech. Widespread, convenient, but ultimately serves the Empire before its denizens. Like that radio whose spyware got Cassian spotted at some point. An effective rebellion needs to do away with those, which is why Nemik uses an old, hard to use navigator to help with a heist: it’s free from Empire influence.
The problem however with that dichotomy is that it fails to account for dual use. There only so much time for a Cory Doctorow lecture in a StarWars series after all. But it does call into questions some of the ideas I had for our real world, and I need to think about it.
You should take a look at the CHERIoT platform.
Will do.
But I think you’re coming around to part of my world view which is that the key thing that you need from the hardware is support for fine-grained compartmentalisation.
I do indeed. Though I still have a problem with how it might complicate the architecture of that hardware. Here’s an idea I’ve just got: divide the firmware into isolated compartments. A given compartment can do 2 things:
Access keys unique to that compartment.
Call other compartments (with an RPC mechanism, compartments need memory protection from each other).
And then there’s one “main” compartment, that can also do I/O.
How to use this? If we want to keep things easy we could put all the firmware in the main compartment. No isolation, but easier prototyping. But if we want things to be secure, then we write one compartment for what would otherwise be a fixed-function unit (one compartment for X25519, one for EdDSA…), isolate more complex protocols in their own compartments as well (the Noise XK compartment would for instance call in the X25519 compartment), and the main compartment would only do the I/O and parsing.
Many of the techniques in the article actually describe how TPMs are implemented, but providing direct access to the keys to the programmable bit of the RoT is something that no reputable RoT has done for a good ten years because it is terrible for security.
Is that what it’s doing? I didn’t read the design that way…
If you want to plug in arbitrary crypto algorithms, you need to run them on the programmable core, not in fixed function units. If you want to run them on the programmable core then you need it to have raw access to keys. If the programmable core has access to keys then you have no way of protecting the, from bugs in the firmware. If you have persistent keys then one buggy or malicious firmware image can leak keys stored by others.
Damn, you really did miss the central point of my entire post. Please re-read it, and tell me where you get lost, or what specific point you think is incorrect. I’ve had feedback about my article not being crystal clear so that may be on me. Unfortunately I don’t know how I can make it better yet.
In the mean time, I can answer more directly:
If you want to plug in arbitrary crypto algorithms, you need to run them on the programmable core, not in fixed function units.
Correct so far.
If you want to run them on the programmable core then you need it to have raw access to keys.
Not quite. The programmable core does not need access to root keys. It does need access to some key, but DICE makes sure this key is independent from the root key, and unique to the particular firmware being loaded.
If you have persistent keys then one buggy or malicious firmware image can leak keys stored by others.
No, it cannot. DICE makes it flat out impossible.
Programmable firmware cannot read the root key, so it cannot leak it. It cannot read (or compute) the derived keys of other firmware, so it cannot leak those either. The only key it can leak is its own. Malicious firmware can leak their own key, but that doesn’t do anything. Buggy firmware is more problematic (their own secret is trusted until we find the bug), but then you fix it, and the new firmware automatically gets a new key.
I hope this helps clear things up, because we can’t have a meaningful discussion if you don’t understand this point.
Not quite. The programmable core does not need access to root keys. It does need access to some key, but DICE makes sure this key is independent from the root key, and unique to the particular firmware being loaded.
Only if you have fixed-function hardware doing key derivation based on PCRs. And that means that you depend on fixed-function implementations of PCRs and a KDF, at an absolute minimum, and you can’t plug in arbitrary KDFs.
You seem to be currently assuming that the thing in the PCR is the entire firmware image. As other have pointed out, that complicates your update process because now you need your new firmware to export your keys, wrapped in a key that the new firmware can decrypt.
I’m not even sure how you would do that because the only key that the new firmware can trust is one that’s derived from the secret and the PCR (anything not derived from the device secret could be run on an emulator and used to leak your keys) and what you really want is something where the old version can derive a public key and the new version can derive a secret key. Again, this is something that you can do if you have asymmetric crypto in hardware (as with a typical TPM implementation), because you can allow arbitrary code on the fixed-function unit to derive a key pair from the secret and an arbitrary value, which then doesn’t grant you access to the secret key and lets you access the secret key only if you use a PCR value as the input, but now you’re relying on more key derivation in hardware.
More conventional uses of DICE use a hash of the signing key as the value embedded in the PCR, which means that two firmware images signed by the same key can have access to the same keys. But that means you need signature verification in hardware[1] and that same logic is responsible for anti-replay. Anti-replay is hard if you want to allow multiple different firmwares to be installed by different users. This is normally implemented with two things:
A counter in NVRAM. Each firmware comes with a signed value and will not be loaded if the value is less than in NVRAM. Firmware is allowed to store a larger value in this space so that it can allow downgrade until it’s ensured that the new version works.
A unary value stored in fuses. This is used in conjunction with a second higher-bits version value in the firmware. Blow a fuse and you burn all of the lower-bits values. This can be done a limited number of times and is intended only for when you accidentally install sign firmware that can either maliciously set the NVRAM value to the maximum or a few other really bad cases.
These are easy only with a single signing entity. If you want different signatures then you probably can’t do the fuse thing and the NVRAM value would need to be one of a bank (if you allowed a finite number of firmware sources to be protected).
You’re also pushing a lot of complexity into software here and it’s really unclear to me what your goal is.
[1] Or initial-boot firmware in ROM, which amounts to almost the same thing.
Not quite. The programmable core does not need access to root keys. It does need access to some key, but DICE makes sure this key is independent from the root key, and unique to the particular firmware being loaded.
Only if you have fixed-function hardware doing key derivation based on PCRs. And that means that you depend on fixed-function implementations of PCRs and a KDF, at an absolute minimum, and you can’t plug in arbitrary KDFs.
Yes! That’s how DICE works! There’s a fixed bootloader with a fixed KDF inside that does the key derivation.
And indeed I can’t plug an arbitrary KDF at this stage. But I don’t care, because what KDF is used is immaterial to my perception of the CDI. If I put the same firmware on a different HSM it will get a different random CDI, and won’t even be able to tell that v1.0.2 and v1.0.3 are using different KDFs or not.
You seem to be currently assuming that the thing in the PCR is the entire firmware image.
Well… I think I do? The important part isn’t where the image is stored, it’s the fact that it is measured in its entirety.
As other have pointed out, that complicates your update process because now you need your new firmware to export your keys, wrapped in a key that the new firmware can decrypt.
We can’t change the firmware without changing the keys, so yes, the update process is made more complicated: any wrapped keys must be unwrapped, stored somewhere safe, and wrapped again… not sure how to best do it, short of having explicit support for this in the fixed functions (which I really want to minimise). As for the firmware keys themselves, they’ll be gone so if anything is derived from them we need to rotate them.
One thing the fixed function could do is reserve a small region of RAM to store arbitrary data to pass along the next firmware. Something like the following:
Old firmware stores sensitive cleartext data in the special region.
Old firmware declares the hash of the next firmware.
Old firmware asks the HSM to trigger a reset.
Reboot. The fixed function bootloader now has control.
Untrusted new firmware is loaded.
Fixed bootloader hashes new firmware.
If the new firmware hash matches what was declared in step (2), unlock the special region. Wipe it otherwise.
Derive the new CDI
Start new firmware.
This should void the need for the asymmetric big guns, thus minimising the amount of code in the fixed function. The price we pay for that is preserving state between firmware loads, and to be honest I’m not entirely comfortable with that.
Also note that the old firmware must be able to authenticate the new firmware (most likely by checking a certificate), else I could just use the upgrade path to upload malicious firmware. Oh, and if the old firmware has a vulnerability there, we’re kinda doomed.
More conventional uses of DICE use a hash of the signing key as the value embedded in the PCR, which means that two firmware images signed by the same key can have access to the same keys.
If the firmware is signed instead of locally measured this is no longer DICE. Though if we limit ourselves to manufacturer-provided firmware this could be a valid alternative. I do think however that arbitrary, user provided firmware, is too powerful to pass up.
I would really like to understand what problems you are solving with this, because it’s clearly targeting a different threat model and a different set of problems to a TPM. This makes it very hard for me to have a useful opinion because it definitely doesn’t solve the problems that a TPM solves, but it might solve some other interesting problems. I just don’t know what they are.
A security token, similar to YubiKey, only more flexible. But those are (i) discrete, and (ii) should be carried on my person instead of tied to my computer. Like, well… a key. Main threat model: fishing, stolen laptop (assuming the key isn’t next to it).
A security token, similar to the TPM, only simpler and more flexible. Something that’s always on my computer, that, despite being flexible to the point of allowing arbitrary code, can reasonably guarantee that its own private keys will never leak. Online attacks are something for my OS to do, I just want to make sure my keys don’t have to be rotated every time a CVE pops up.
A root of trust, that helps guarantee that the firmware & software I want to execute, is the one to execute. Something that guarantees that unwanted bootloader or kernel modifications don’t survive a reboot. The main threat model is software attacks (vulnerabilities in my programs, OS, or me clicking on the wrong thing), but I would like some resistance against physical access too though that’s impossible with discrete chips (the main execution unit and the secure element must have a secure channel between them, or they must be one and the same).
Important thing about the root of trust: I need to be able to tell the machine to trust arbitrary software. My software my choice. Letting big companies dictate what I’m allowed to execute on my machine is not a goal.
I think this is a silly thing for people to lose their minds over. Other languages ship binary components, it’s not even uncommon - pyc or .so files get shipped frequently in Python.
This has very little impact on security. Everyone is calling this “a vulnerability” and they’re being overdramatic and inaccurate. This is not a vulnerability. This changes one thing; that if you were someone who audited their crate (most of you are not) you can’t determine that the binary comes from the source code that you’ve audited. Or, well, you can - just compile the source code into a binary and use that - but you can’t do it for arbitrary binaries. Unfortunately replacing the binary is not a straightforward process, but that’s ok - go ahead and push cargo to support managing native dependencies directly.
This mostly messes up build systems that weren’t expecting this. But this is also one of the more build-system friendly methods to get the speed up. Ultimately, if serde wanted to cache the crate’s binary output, that would not play well with some build systems.
People are bringing up reproducibility. The main issue of reproducible builds is that, if they were practical, one could say “this binary came from that source”, allowing a sort of “web of trust”. Of course, then you have to trust that web of trust, it only helps catch things too late, etc. Far better approaches would be to allow for things like package signing, permissions for build scripts, etc. Ask the cargo team for those.
To be candid, this was handled very poorly. It should have been obvious that this was going to have blow back, or potentially break build systems, etc. For such a core crate I think there should have been far more care put into ensuring a safe rollout where everyone knew what to expect. I resent that a popular crate is being wielded as a weapon to push cargo into implementing some feature, at the expense of the entire language and ecosystem - this was seriously the last thing Rust needed. As someone trying to get Rust to be a supported language at my company I find it extremely annoying that I’m likely going to have to address this at work, and it’s one more thing I’ll have to deal with when I try to get a major multi billion dollar company using Rust formally.
But people are severely overreacting to what is basically a non issue.
I disagree on all counts but the fact that it was poorly handled. The popularity of the crate, and the sensitivity of the change (whether or not you consider it to be valid, a lot of people do) warranted a much more consensual approach. More to the points:
The fact that other languages do it is irrelevant.
Downloading random blobs to be executed locally is a vulnerability. Examples of binary blobs being subverted abound. At the very least it arbitrarily expands the scope of the binary providers one has to trust. Pushing the model to the extreme, one would have to trust every single crate author to provide safe binaries were they to all use binary blobs.
Speed is a straw man. We extensively use serde in our very large production codebase, and the macro generation time is a very small fraction of the entire compile time. In release mode, it’s absurdly non-existent.
Reproducibility is a poor attempt at solving a problem that should not exist to begin with. Just don’t force people to use binary blobs.
Downloading random blobs to be executed locally is a vulnerability. Examples of binary blobs being subverted abound. At the very least it arbitrarily expands the scope of the binary providers one has to trust. Pushing the model to the extreme, one would have to trust every single crate author to provide safe binaries were they to all use binary blobs.
Repeating what was already said above: in the Python world, people distribute pre-compiled binary blobs in packages that include extensions written in C (or other compiled languages), because building those things yourself from source is an absolute pain (the go-to example being the extremely popular NumPy/SciPy combo, which requires something like four different compilers and some Fortran libraries in order to build from source).
Binaries in Python packages also are not (currently) byte-for-byte reproducible.
I’m not going to engage with the troll-ness here, but I will point out that Python should be looked to by people interested in packaging, because Python has Seen Some Shit™. If a newer language or tool is running into a problem, odds are Python has seen it before, and there’s stuff you can go look up and research and mine for ideas and solutions.
Also, Python packaging has been solid and good for years now, and most of what people complain about now is the lack of high-level project/workspace management flows, not package install/uninstall/etc.
I agree 100% that Python is an excellent example of what to avoid. I swear I am not trolling
Let me give you a counterexample: I recall a time when Ruby did not have a package manager. Bundler was developed, quickly adopted the by the community, upstreamed into the stdlib. Bundler is a good package manager and there are no competing standards.
By contrast, I do not work in Python, and I have on multiple occasions had to fix Python environments against my will. The messiness of Python’s package management has a blast radius larger than its own developer community. I have had to fix Python packaging environments more frequently than Ruby and I work in Ruby full-time.
As someone who has had to suffer that extended blast radius as well at $WORK, I do sympathize, but I will say that to me, the precompiled binaries significantly tamp it down. It’s even become a somewhat … creative … delivery vehicle for build tools like cmake and clang-format, which I’m not sure what to think about, but it’s fun not to depend on the distro to get the latest and greatest.
And still, there are quite some Python packages that are darn hard to rebuild if you don’t want to or can’t use prebuilt wheels (e.g. if you are making derivations for Python packages in Nix).
Binary wheels are exactly the problem, because 99% of the users can rely on binary wheels, there is very little incentive to make a wheel easy to build outside the specific build environment that a project/company uses.
I don’t deny that binary wheels don’t solve many problems. It is unreasonable to expect that users build, say PyTorch, on their own machine.
But having used Python and Rust professionally for years, I’d rather like Python packaging to move closer to Rust packaging than the other way around. There is just so much less friction (yeah, Rust had a clean slate).
I would wager that at least the majority, and probably the significant majority, of the perceived ease/simplicity of “Rust packaging” compared to Python comes down to exactly two things:
People generally do not do as much “outsourcing” to other languages in Rust as in Python. People do wrap existing C code with Rust, yes, but the level of wrapping and bundling and resultant management of libraries in other languages is something Rust just doesn’t seem to do on anywhere near the scale of Python, and is a source of massive complexity for what otherwise would be a single-language-focused packaging toolchain.
Static versus dynamic linking. Python, and other languages which do runtime dynamic linking as the default/preferred approach, have to figure out ways to isolate projects’ dependencies from each other at runtime. Rust can just build a statically-linked binary containing all the dependencies, and not have to worry.
If and when Rust changes on those two factors, its packaging toolchain will likely become significantly more complex, by necessity.
Rust is not Python and its ecosystem has evolved different implicit contracts between upstream and downstream.
It’s far too easy to morph the skeleton of your argument into “We don’t need Rust because we have C” on a basis of “Other well-established languages do X. Therefore not improving the state of things is acceptable.”
The claim being made is that shipping binaries in a package is somehow inherently insecure, or significantly less secure, than only shipping source. The counterpoint to that claim is other package ecosystems which do exactly that without the claimed attendant security issues.
(ironically, a Python .whl package – the kind that can contain binary blobs – is arguably much better security-wise than the older source-only format, because a source-only package can and indeed has to be able to run arbitrary code on the target machine at install time in order to build any extensions it relies on, while a .whl is literally just a zipped archive whose install process consists of unzipping and moving the files to their destinations, with no ability to run install-time code)
Ironically, PyPI has been introducing more trustable/auditable flows for package publishing where you can track the provenance of a given build, but people got really angry because the first integration they did was with GitHub Actions.
The fact that other languages do it is irrelevant.
It’s relevant when people are calling this “disgusting” as if it’s some crazy, wild, out there thing that’s SO insane to consider.
Downloading random blobs to be executed locally is a vulnerability
Words have meaning! Like, NO, it is not a vulnerability. That’s just coopting a word with a negative connotation to apply it where it’s inappropriate because you don’t like the other thing. It is insane in this context to call this a vulnerability. Please stop.
Speed is a straw man.
What? How can it be a straw man? It was the point of this. Speed isn’t “an argument that I’ve set up to attack” ie a straw man, it is the point of this PR.
and the macro generation time is a very small fraction of the entire compile time.
It’s not just the compilation time of the serde crate, it’s also that in debug mode the macros will run without any optimizations, making them very slow as well. Maybe that doesn’t impact you, it impacts others. Besides, if native binary support were built in this wouldn’t be an issue.
Reproducibility is a poor attempt at solving a problem that should not exist to begin with.
Well I can agree that reprodicibility is a bad attempt to solve this problem at least.
The idea is not crazy and has technical merit. For those calling it “disgusting”, I usually ignore emotional reactions.
Words have meaning! Like, NO, it is not a vulnerability
Words have meanings, and in my threat model, it is a vulnerability. YMMV obviously.
What? How can it be a straw man?
Compilation speed is not the purview of the serde crate. When using a crate for a particular purpose, I’m not giving carte blanche to the author of that crate to alter my development process as they see fit. Is it a problem for the author? fine if they fix it locally. Did they come up with a revolutionary idea worth sharing? fine, propose it as a PR and have a discussion about it. Is it so good that it can benefit build tools ? fine share it with the cargo team. But leave others build processes alone.
It’s not just the compilation time of the serde crate
Not the purview of the crate author. It would be the purview of the cargo team. With a problem statement, a proposed solution, and a discussion. Even if it was a problem for everyone but me the chosen solution would still be half-baked and ill-suited.
Did they come up with a revolutionary idea worth sharing? fine, propose it as a PR and have a discussion about it. Is it so good that it can benefit build tools ? fine share it with the cargo team. But leave others build processes alone.
Don’t be naive, they don’t owe you anything and don’t have to discuss anything with anyone. If they own the repo they can do whatever they feel like it. If you don’t like it you have many options, like: pin last version you like, stop using that crate (including any other that uses it), pay them do fix your problem or even fork it and do a better job. This is a problem you can solve on your own instead you’ve decided to tell the maintainer what they should do. That’s not how it works.
they don’t owe you anything and don’t have to discuss anything with anyone
I believe there’s more nuance to this in general. Maintainers owe at least two things to the users:
First, truthful communication about the nature of software. You can’t say “production-ready & secure” in your Readme, if it actually is “buggy & vulnerable”. It’s ok to push arbitrary low-quality code to GitHub, it’s not to mislead users into believing it is fit for production use.
Second, if you communicate that your project is dependable, you then can not abruptly renege on that promise.
An important observation here is that, although the license say “WITHOUT WARRANTY OF ANY KIND”, that is a statement about what’s legal, not what’s ethical. Breaking the two rules above is legal, but is not ethical.
Specializing to this particular case, I do consider the situation to be a pretty significant breakage of the implicit contract. serde is not just someone’s hobby project. It is well understood, by both users and maintainers (or at least I, as a user, totally got this understanding, with no evidence to the contrary prior to this situation), that serde is one of the foundational crates of the Rust ecosystem.
Specializing to this particular case, I do consider the situation to be a pretty significant breakage of the implicit contract
If you want to hold people accountable for breaking it, make it an explicit contract.
By way of comparison, I’m a fan of baseball, but one aspect of it I can’t stand is people – often coaches and commentators of older generations – who insist the game is to be played according to certain “unwritten rules”. If you want people to play by a set of rules, write them down and enforce them. “Unwritten” rules are a mechanism for arbitrariness, and so too are “implicit contracts”.
I believe there’s more nuance to this in general. Maintainers owe at least two things to the users:
No, they don’t. It’s their own time they are giving up for free, and complaints that they do that in the wrong way are inappropriate.
They have all the right in the world to solve compilation time problems as they see fit, without waiting for some ideal solution that the rust project may come up with, some time in the distant future.
But even if we assume we live in a world you described:
First, truthful communication about the nature of software. You can’t say “production-ready & secure” in your Readme, if it actually is “buggy & vulnerable”. It’s ok to push arbitrary low-quality code to GitHub, it’s not to mislead users into believing it is fit for production use.
Second, if you communicate that your project is dependable, you then can not abruptly renege on that promise.
I don’t see where dtolnay stopped doing that with the serde framework. Shipping binary blobs as part of packages is nothing new. No one is saying that Python or Ruby is not production ready so please don’t try to make it look like serde stopped being production ready or that authors are not truthful in “communication about the nature of software”.
No, they don’t. It’s their own time they are giving up for free, and complaints that they do that in the wrong way are inappropriate.
I don’t think the world is in absolutes like that. Yes, it’s their crate and their time, so they ultimately make the decisions. But if your crate is such a fundamental cornerstone of the Rust community, it is also good to take issues that other people have into account. Maybe dtolney did that and decided to do that anyway, and that is in their right. Just as other people can disagree and criticise the decision, being open source doesn’t relinquish it. Just keep the discussions technical and try to keep the drama out.
If they end up keeping the binary download and a substantial part of the community disagrees, it is probably unavoidable that serde-derive gets forked. Personally, I think that crates that end up getting used by neatly everything should probably maintained with the Rust org itself.
Just keep the discussions technical and try to keep the drama out.
I don’t know why Rust seems more prone to unnecessary drama. Is it something in the personality of the people who use Rust? Is it the culture that all voices have to be heard? Is it the communication medium? GitHub certainly makes it easy for anyone to post in the issue and the drama noise quickly overwhelms the technical signal. (Sometimes I wonder if mailing-list-based projects are more immune to this kind of briggading because of the greater effort needed for a person entirely uninvolved in the project to voice their opinion.)
How is what I am donating relevant to the argument?
Because it’s the argument you are making. You are suggesting a group of volunteers take work over from another group of volunteers without any extra resources.
I don’t agree, there are Rust developers paid to work on Rust and their companies may be interested in doing the same.
But here you have it: I have donated to Rust in the past. So I guess I am allowed to have an opinion.
(I didn’t mention it in the earlier comment, because I still don’t think it is relevant, and I don’t want to live in a world where only people with disposable income can have an opinion.)
I don’t see the point of your comment. You said there are no absolutes, then agreed that the maintainer absolutely gets to decide what to do with their project. Your comment cancels itself out.
It’s their own time they are giving up for free, and complaints that they do that in the wrong way are inappropriate.
There are many points between the absolutes: the maintainer does the work for free and criticizing them is inappropriate and the maintainer should always do what the downstream users say.
Like: it’s ok for the maintainer to make their choices and it’s fair for people to criticize the choices as long as it’s done in a fair way without personal attacks and measured.
serde is one of the foundational crates of the Rust ecosystem
Is this is a sign the crate or functionality within should really be part of the Rust official libraries. Official rust tooling depends on them. The community treats it like it is “official”, when nothing indicates it is.
Is this evidence that the “small stdlib” approach doesn’t really work? Or at least, only works as long as the planets align.
The crates listed in https://crates.io/teams/github:rust-lang:libs are also maintained by the Rust team (or, at minimum, they’re functionally maintained by one person who has given the rest of the team the power to kick them out if they lose it) without forcing them to update in lockstep with the compiler the way std has to.
Words have meanings, and in my threat model, it is a vulnerability. YMMV obviously.
No, just no. Like, seriously, stop with this. It is not a vulnerability. There is no CVE you could file for this. There is no vulnerability class you could attribute. IT is a design decision that impacts nothing about the threat model - attackers in the same position can do the same exact things (execute code on your system), the only salient different is that if you audit the source code you can not tie that audited source code to a binary generated by someone else. That is NOT a vulnerability.
Not the purview of the crate author. It would be the purview of the cargo team. With a problem statement, a proposed solution, and a discussion. Even if it was a problem for everyone but me the chosen solution would still be half-baked and ill-suited.
FWIW I agree to an extent and I definitely think the crate author overstepped here.
No, just no. Like, seriously, stop with this. It is not a vulnerability. There is no CVE you could file for this. There is no vulnerability class you could attribute. IT is a design decision that impacts nothing about the threat model - attackers in the same position can do the same exact things (execute code on your system), the only salient different is that if you audit the source code you can not tie that audited source code to a binary generated by someone else. That is NOT a vulnerability.
If the attacker has access to the developer’s signing keys, their ability to execute code on your system without detection is mitigated by source-based distribution. Relying on non-reproducable binaries makes you much more vulnerable to such an attack. Yes, words have meanings, and “vulnerability” has a wider scope than what fits into a CVE.
The full quote:
(I) A flaw or weakness in a system’s design, implementation, or
operation and management that could be exploited to violate the
system’s security policy. (See: harden.)
“that could be exploited to violate the system’s security policy”.
This change to use a binary artifact is not a vulnerability. It changes nothing about what the attacker can or can not exploit.
This is like saying “using jwt is a vulnerability because you could have used something else and i don’t like jwt tokens”.
I have never heard, nor would I ever accept, someone saying “The attack is literally identical except that in one case the detection method is different, therefor it is a vulnerability”.
That’s is an insane position, especially when the “different” detection method is much more impractical. Detection is a crucial element is many threat models.
Detection is a crucial element is many threat models.
OK, sure. You’re talking to a Detection Engineer/ Incident Responder who founded a company to build a SIEM lol I’m obviously well aware of how important detection is.
That doesn’t change anything - “worse auditability” is not a vulnerability. You can say it’s worse, or less safe, or whatever, I don’t care. Calling it a “vulnerability” is wrong.
So if someone pushed an update that disables one of your detection systems so that the only fallback involves requires reverse engineering, you wouldn’t say they introduced a vulnerability?
Yes. A change to software that changes how you would detect an attack, or even one that removes the ability to detect an attack, is not a vulnerability. If you want to make up another word for that feel free though.
I’ll give some additional context. In general the issue of “detection” is punted to a separate system. For example, I may write rules in a SIEM to detect an issue based on the logs that a system produces.
a) If my rules have bypasses it is not considered a vulnerability in the service, it would be, at most, a “vulnerability” in the rule (although that is not the nomenclature used - we call them bypasses)
b) If the service changes the format of its logs or something, breaking my detection logic, that is not considered a vulnerability of the service. It’s up to the detection system to account for that, manage the updates, etc.
c) If the service produces a stable audit log, and then it breaks that audit log, that could maybe be considered a vulnerability technically, although it would almost certainly not be called one and instead most people would call it a bypass. But yeah, that would be a vulnerability in the service, if you want to use that terminology.
None of these situations are even close to what serde is doing, which is just changing some internal behavior that inconveniences auditors. That is not a vulnerability. It’s at best close to (a) or (b), neither of which are vulnerabilities.
If you can’t accept this I’m sorry, you’re wrong and that’s it. You can be wrong if you want to be, I don’t care, but this isn’t really a matter of opinion or discussion or nuance.
edit: I’ll also note that having vulnerability encompass “a detection broken” is entirely useless. What would you do with that information? Nothing with the software itself, you’d go and change the detection. Would everyone be notified for every single change to software that might impact some arbitrary detection implemented in a totally different system? Who is this serving?
a) If my rules have bypasses it is not considered a vulnerability in the service, it would be, at most, a “vulnerability” in the rule (although that is not the nomenclature used - we call them bypasses)
b) If the service changes the format of its logs or something, breaking my detection logic, that is not considered a vulnerability of the service. It’s up to the detection system to account for that, manage the updates, etc.
c) If the service produces a stable audit log, and then it breaks that audit log, that could maybe be considered a vulnerability technically, although it would almost certainly not be called one and instead most people would call it a bypass. But yeah, that would be a vulnerability in the service, if you want to use that terminology.
I think “in/of the service” is doing some pretty heavy lifting here.
Not really, because even if you were sayin g”but it’s sitll a vulnerability, just elsewhere”, I still justify why:
a) That’s not the case (we don’t call it a vuln), except perhaps if being pedantic about (c), which doesn’t apply because serde never made some sort of security policy around its own auditability
b) It’s definitely not what people are saying - people are saying serde is vulnerable, and I think it would be an incredible shift in goal posts to say otherwise
In no way does my point rely on where the vulnerability is, it’s just trying to give your argument the best possible interpretation by saying “maybe, if pedantic, in one specific case that odesn’t apply, you could call it a vuln in another system”.
Not really, because even if you were sayin g”but it’s sitll a vulnerability, just elsewhere”, I still justify why:
a) That’s not the case (we don’t call it a vuln), except perhaps if being pedantic about (c), which doesn’t apply because serde never made some sort of security policy around its own auditability
If we are talking about a vulnerability elsewhere, why are security policies issued by serde the only relevant ones?
b) It’s definitely not what people are saying - people are saying serde is vulnerable, and I think it would be an incredible shift in goal posts to say otherwise
The one person in this thread said “downloading random blobs to be executed locally is a vulnerability.” I don’t think they were claiming that serde downloads and executes binary blobs.
Honestly, it feels like the same form letter as the actix-web kerfuffle, just different mad libs filled in.
Someone builds trust as a purveyor of the thing representing some aspect of what makes Rust great.
It’s discovered that they have a more laissez faire attitude toward some aspect of security than many people in the ecosystem do (In actix-web’s case, unsafe. In Serde’s case, auditability. In both cases, a perceived violation of “Pick the safest default, even if that means a bit less performance.” Yes, I know integer overflow/underflow in release builds is a glaring exception.)
Said someone responds with a WONTFIX of some kind.
Many people feel betrayed and overreact as they feel the need to defend a language and ecosystem they’ve become emotionally invested in against what they perceive as a potential existential threat.
I see a somewhat different commonality. In both cases, an individual developer, who obviously takes pride in their work, went to unusual lengths to optimize their own software on some easily measurable metric. In Actix’s case, this meant coming out on top of benchmarks. For serde, it meant reducing compile time. In both cases, the community decided that the end didn’t justify the means.
The lesson I take from this, which I will keep in mind for my own AccessKit project, is that instead of going to great lengths to move one’s own project ahead of the rest of the ecosystem, it’s better to do what we can to move the whole ecosystem forward, and accept the limitations of that approach. We may be able to get our own project where we want faster by taking the bold step alone, but in the long run, it’s better to move everyone forward together.
*nod* “Do it properly and move the whole ecosystem forward, even if it means taking longer to get there” has been a recurring theme for Rust.
In fact, it’s something I’ve contrasted with other ecosystems. For example, that’s why Rust was focusing on slowly but steadily improving LLVM-related things, while Go had to backpedal on bypassing libSystem.dylib (the ABI stability surface for the XNU kernel used in Darwin, macOS, and iOS, equivalent to ntdll.dll) in their efforts to just unilaterally rewrite or bypass anything that was a drag on their goal to break new ground for easy cross-building and easy deployment.
Please don’t wave this “10x” figure like a magic wand. Like in every performance-related issue, reality is much more nuanced.
If using a binary blob will improve the time of your code generation, in any reasonably-sized project that time is only a fraction of the overall compilation time, especially in release mode.
The end user should retain the ability to opt-in to at least evaluate whether or not using a pre-compiled binary is beneficial to their use case.
It is the fact that users are robbed of this ability and force-fed a questionable practice for dubious reason that is being criticized, and in my opinion, justly so.
The end user should retain the ability to opt-in to at least evaluate whether or not using a pre-compiled binary is beneficial to their use case.
I’m in agreement this should have been a feature flag. I’m in disagreement that “the community” has decided anything. It’s weird this picked up traction several weeks after it happened and is seemingly some dogpile issue now.
It’s weird this picked up traction several weeks after it happened and is seemingly some dogpile issue now.
I don’t know about others, but I only learned of it now. I only pay attention to Serde changelogs when the Dependabot PRs start rolling in, and my last wave of them took my projects to 1.0.168.
(So it’s possible we’ll see another wave of outrage on September 1st, when the next wave of Dependabot PRs rolls in and the people who’ve been living in a hole see and investigate the “Experiment with precompiling the serde_derive macros to reduce build time” line in their changelogs.)
EDIT: Ouch. It looks like I also have a couple of projects where it reached 1.0.180 because I was too tired and distracted to realize that “precompile the proc macros” shouldn’t automatically be dismissed some kind of dependency sequencing hack that still ultimately builds from source. (To be fair, that does roughly describe the equivalent or better speed-up you can get by depending on serde_derive directly, instead of through the derive feature on the serde crate as a build sequencing hack.)
I think the actix-web thing is way different because it was marketing itself as “safe” and was using “unsafe” incorrectly all over the place. People found actual vulnerabilities in it and they got wontfix’d.
I think people were reacting pretty reasonably in the case of actix. I think people are totally blowing this situation out of proportion.
It will also be pretty hard to get full blown crate-level binary support. There is no stable ABI for rust code itself, there are top-level decisions which can change everything (global allocator) and thus building for even every tier 1 target of rustc itself won’t be enough (as shown by various build problems in this “experiment”).
I’m disappointed that something titled “experiment” landed “opt-out” (if you want to call it like that) in a minor patch release. And then when people have broken builds or concerns, it’s left like that until a suitable opt-in solution is found..
I’m honestly disappointed by at least two people here who are/were part of rust teams and active for a very long time. This whole situation got handled pretty poorly, compared to some other, less controversial changes in the rust compiler. (Meaning the experience exists, and the crate is relevant enough it might also be rustc itself.)
I understand the intention: Shipping an optimized binary for a core-tool nearly everyone uses, kind of like the rust-compiler installation, which ships an optimized std binary. But the execution was not the best.
this is a silly thing for people to lose their minds over
Yes, it is. But I mostly ignore the nasty comments, because high-interest Github issues are increasingly like Twitter or YouTube comments - they’re full of ugly noise.
Note that it’s the policy of all dtolnay crates to only bump patch version. That is, anyhow, semver, serde, syn, thiserror, etc. do not follow semver with respect to minor versions.
This is strange, but defensible — there’s no physical difference in how Cargo treats minor vs patch versions, so an argument could be made that it’s simpler to just stick to patches (you can convey more info in minor, but, as that’s not machine checked, the value of that info is relatively minor).
It does mean though that you should always use serde = "1.0.x" in your Cargo.toml, rather than serde = "1", as the latter breaks minimal versions.
Updating the images changes the key, so… you need to revoke the old key and use the new one. If you need the old key for the migration process (encrypted disk, wrapped keys…), use the old image do decrypt, and encrypt again with the new image.
The above was also my main issue when reading the post. I thought that having a persistent keypair that is tied to the machine is an explicit feature of a TPM. Derive subkeys all you want but that root should remain static.
So when your secret suddenly changes due to software updates and you have no way of obtaining the correct secret without running an old, now known-insecure copy of your program .. you have a problem, no?
Correct, I do have a problem. But I believe the solution is easy: control the environment so the update process itself doesn’t leak the old keys. Because the old firmware is vulnerable doesn’t mean my update program has to exploit it.
The problem doesn’t go away entirely with the classical approach: if your old firmware is vulnerable you need to assume its keys have been leaked, so you can no longer talk to your TPM over an untrusted channel to perform the update. You need the controlled environment just the same. And on top of that you need to change the possibly leaked old keys, and cross fingers that you have enough fuses left to blow.
My issue with SRFI’s has always been that, when I go to a Scheme’s website and it says It implements R5RS with SRFI 17, SRFI 91, and SRFI 26 I don’t know what that means.
But If see R6RS, or R7RS-small I have a pretty good idea what is in there. Personally I like R6RS, and Racket’s implementation the best, so that is what I use.
Some people (including myself and it would appear @Decabytes as well) find it harder to reason about and discuss ecosystems like this where there is a large universe of possible combinations. There is no single reference or document that covers what is supported. Instead there are now various opaque identifiers that must be mentally juggled and compared and remembered. (It’s great that there’s a single place to look up the meaning of each SRFI, but I don’t think that solves comprehension.)
If you are making some software, you can no longer say “works with Scheme R[number]RS implementations”, but you instead have to list out the SRFIs you use, which may or may not be supported by the user’s favoured implementation. Then you have to repeat that complexity juggling with other libraries you may also want to use.
It’s a general issue that tends to arise with any ecosystems arranged in this way. It prioritises implementer flexibility and experimentation over user comprehension of what’s supported. (Maybe that’s okay, maybe it’s not! Probably like all things … it depends on context, each person’s preference, etc.)
People have made similar complaints about the XMPP ecosystem with its XEPs, which is also an ecosystem of optional extensions.
The Haskell situation is maybe a little better in that—practically speaking—there is only one Haskell compiler in widespread use. You could argue that this is a net loss for the Haskell ecosystem! But the fact that most (all?) language extensions are enabled on a per-module basis means that compatibility comes down to asking “which version of GHC are you using?” rather than needing to ask “does your compiler support TemplateHaskell? And DerivingVia? How about MultiWayIf?”
(I’ll add that the Haskell language extensions are referred to by name. If common usage in the Scheme community is to talk about “SFRI 26,” it does seem like Haskell puts less of a burden on one’s memory when talking about these things.)
so if you write a program which uses SRFIs X Y and Z you just need to check if X and Y and Z are in that list. This is a very deterministic, black and white, well defined thing.
You don’t need to memorize the numbers btw. I don’t know why people think that.
It prioritises implementer flexibility and experimentation over user comprehension of what’s supported
It’s explicitly listed out what is is supported, it is very easy for a user to look at the list and see that the things on the list are supported.
But I also really don’t think that’s true at all. the SRFI process is to enable users to have understandable libraries of code that they can use, potentially coordinated across implementations. implementors doing exploration would not take the time to specify stable APIs like that, write documentation on them necessarily. I think you have this backwards.
Cross checking several implementations against a list of specs before I start writing a program sounds complicated to me but is “very easy” for typical scheme users tells me I’m not smart enough to enjoy this language
They are all there, but what’s there doesn’t necessarily mean the implementations are actually compliant. There’s often caveats “Our implementation just re-exports core foo in place of srfi-foo and differs in semantics” — or they won’t tell you that, and it’ll be different.
That’s a totally different question? It doesn’t make sense to write that as a reply to my comment that provides a list of the SRFIs.
If a low quality implementation is providing an incorrect implementation of a specification then that is obviously a bug. I don’t know what that has to do with me though.
Yes I don’t have them all memorized because there are so many and what Schemes implement what varies. I have much better knowledge of revised reports because they are self contained groupings of specific functionality.
SRFIs would be better if they targeted more practical problems. There’s a fairly recent JSON SRFI, which is great. But, generally, most are experimental language features that have no business being standardized, most of the time, imho.
Then, there’s the terminal interface SRFI, which sounds great, but doesn’t have a fallback pure scheme implementation. So you’re at the will of implementations / library authors to build them out, which is non-trivial, and likely not fully cross compatible anyway.
The community is too small, and the ecosystem too fractured. :/
Not every SRFI can be implemented in pure scheme, but that’s an advantage not a disadvantage: since they can specify things that would otherwise be impossible to add as an external library. It lets you know that you need an implementation to provide you SRFI-x.
The community is too small, and the ecosystem too fractured
This is ironic isn’t it. For such a small community to be so fractured…
SRFIs are more about an API spec, not the implementation. They commonly have a reference implementation, of course.
Package managers in scheme aren’t standardized… :) until r6rs you didn’t even have a common way to do libraries across implementations outside of load so… it’s all very tricky. Very very tricky. :)
Well, not my community, but I would want a standards process to end up with me being able to use code written under the standard in any compatible implementation with ideally zero changes. This is how C and Fortran compilers work.
Going beyond C, I’d expect standardisation to establish a common system for specifying modules and packages, project dependencies and their versions and a registry of compatible packages for people to use.
If you get all of that then you should be able to switch implementations quite easily and a common registry and package format would encourage wider code reuse rather than the current fracturing that lispers complain about.
Is that just not something schemers are interested in? (Genuine question) And if they’re not, then what’s the point of the standardisation process?
Agree completely: things like a JSON library are great to have. useful to specify as a SRFI but not part of the language itself. The whole value of scheme is to have a tiny core language that it is so flexible that you can add things like this as a library.
SRFIs would be better if they targeted more practical problems. There’s a fairly recent JSON SRFI, which is great. But, generally, most are experimental language features that have no business being standardized, most of the time, imho.
I feel this too. A lot of the newer SRFIs feel very “galaxy brain” slash “language design by the back door” slash “the SRFI author is probably the only one who wants this” but are unlikely to positively impact my day-to-day life writing Scheme programs tbh
In general I feel that there are many very smart people churning out API designs that I don’t actually like nor want to use. Maybe I’m not smart enough to appreciate them. If so that’s OK. Aesthetically, many of the designs feel very “R6RS / abstract data types / denotational semantics” focused. Which is fine I guess, but I don’t personally enjoy using APIs designed that way very much, nor do I think they’re going to “make fetch Scheme happen” for the modal industry programmer anyway
ultimately folks are free to do whatever they want with their free time so I’m not mad about it, I’m happy to just keep plugging along using (mostly unchanging) older R5RS implementations and porting code to my own toy module system, etc and relying on portable code as much as possible
FWIW I thought R6RS was “fine” until they broke all my code using DEFINE-RECORD-TYPE because something something PL nerds denotational semantics etc. I have appreciated that the R7RS-small work I’m aware of thus far doesn’t break my code in the same way R6RS did
I believe it was R6RS that included the first operational semantics for a standard Scheme. Previously R4RS had included a denotational semantics, which R5RS had left unchanged despite there being changes in the text that required a change to the semantics.
In neither R4-, R5- nor R6RS did one need to read the semantics to make effective use of the language.
A lot of the newer SRFIs feel very “galaxy brain” slash “language design by the back door” slash “the SRFI author is probably the only one who wants this” but are unlikely to positively impact my day-to-day life writing Scheme programs tbh
FWIW, I agree. The newer SRFIs seem very much aimed at comprehensiveness instead of ergonomics. If you look at some of the older SRFIs they seem a lot more focused and minimal.
On one hand this is technically interesting. On the other hand, it’s definitely against the Terms of Service (without permission).
Permissions and Restrictions
You may access and use the Service as made available to you, as long as you comply with this Agreement and applicable law. You may view or listen to Content for your personal, non-commercial use. You may also show YouTube videos through the embeddable YouTube player.
The following restrictions apply to your use of the Service. You are not allowed to:
access, reproduce, download, distribute, transmit, broadcast, display, sell, license, alter, modify or otherwise use any part of the Service or any Content except: (a) as expressly authorized by the Service; or (b) with prior written permission from YouTube and, if applicable, the respective rights holders;
circumvent, disable, fraudulently engage with, or otherwise interfere with any part of the Service (or attempt to do any of these things), including security-related features or features that (a) prevent or restrict the copying or other use of Content or (b) limit the use of the Service or Content;
On the third hand, nobody cares about the terms of service. I’m sure by using an adblocker online I violate half the terms of service (that I don’t even see) for websites I visit. They want it one way. But it’s the other way.
I feel like the algorithms list is a bit much. Many of them look like interview question knowledge, not programming knowledge
I recognize many from college CS, and then never used them or saw them in code in the last 20 years
e.g. Floyd-Warshall, Bellman-Ford (I’d be very interested if someone can point me to open source code that uses these, and things like it)
I happen to know that Python’s substr in s uses something like Boyer-Moore or Knuth-Morris Pratt, but honestly I don’t remember which, and it doesn’t matter. (I didn’t know this for years)
Ditto Aho-Corasick – kinda fun, but I doubt most people need to know it. It’s useful if you’re writing certain kinds of libraries, which almost nobody is
I remember an accusation on a Google mailing list that dynamic programming interview questions (dynamic programming being mentioned in the post) are basically a shibboleth for “did you study CS at a certain set of schools” . Then there was an argument about dynamic programming vs. memoization and people couldn’t agree on the difference (indicating that most people don’t really use it, or remember what it is)
For most programmers, I would lean toward something more like what Rich Hickey says – just learn to use maps and vectors, and learn to divide programs into functions. You can get really far with appropriate choices there. It can also be done surprisingly badly (if you really read people’s code who are just starting)
And also learn how to use very basic static type relationships to express the domain, what you intend (i.e. learn both dynamic and static languages).
Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don’t get fancy. (Even if n does get big, use Rule 2 first.)
Rule 4. Fancy algorithms are buggier than simple ones, and they’re much harder to implement. Use simple algorithms as well as simple data structures.
Also the list is too weighted towards serial algorithms (like college courses are). There are no parallel algorithms or data structures, and these days parallelism and concurrency are what will get you real speedups – e.g. learning how to either share or copy data effectively
I will say that compilers have many data structures that aren’t used in “every day” code, but most people aren’t writing compilers.
I agree with learning to think with invariants though – often if I ask people explain WHY the code is correct, they don’t have an answer … And when they need to explain, they stumble across a bug. Every time I write code I think about the invariants of data structures and write comments
I’m old fashioned and use classes and methods – they are great for maintaining invariants
Yes thanks for acknowledging that. It doesn’t strictly relate to your post, but these days people get bashed over the head with “algorithms” stuff due to needing to interview… So I just offer a balancing opinion on whether you need that to write good software.
I guess the short answer is that math is useful, and I’ve worked with people don’t know what they don’t know (CS knowledge), but also I think it’s a relatively small part of writing good software
Systems software is an exception, and you’re clearly deep in that space, and I found many things on the blog useful (including in the update!)
Hm very nice post off the cuff! I will say one bad thing and then a few good things …
Everybody knows why any regular expression can be complied to an equivalent finite state machine
Picking on this one part – You’re living in a bubble :) Everyone doesn’t know this!
Most programmers don’t even know what a “regular language” vs. “regex” is, the former being the thing that’s equivalent to a NFA/DFA. I’ve made the point on the Oil blog that it’s actually good to know this from an engineering perspective, and it doesn’t really “land” for most readers.
But actually if you ask me “why”, right now I don’t remember the answer. I know I wrote a huge shell lexer that depends on this fact being true
I vaguely remember the shape of Thompson’s construction, but don’t know if that counts. I know I could find a book that tells me why, but that is very low on my priority list – it doesn’t help get software done.
(And I know there is the derivatives method, one of the few math-y things I’ve looked into in the last 5 years, but never implemented it, or Thompson’s construction)
Like I said I’m just offering some different perspectives.
As a person who used to do more math, I would have thought this advice is more actionable - https://missing.csail.mit.edu/ – i.e. learn how to use Unix and version control. Learning all the practical stuff takes longer than learning algorithms IME. (I think Julia Evans posts are popular because they have many good practical tips – the “tacit knowledge” as you say – that is where the real difficulty in making software is, IMO)
And the other contrasting opinion I would give is basically along the lines of “learning the basics”.
There is the famous quote “I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times”
I would apply that to programming … IME, the most effective people are not the ones who know a bazillion different techniques and tools and languages, but have really just mastered the basics of using functions / data / classes / whatever to model problems, AND mastered a single language (in addition to knowing others).
I guess that’s just another way of saying “tacit knowledge”
Did you know that grow factor 2 has a problem that the size after n reallocations is larger then the sum total of all previous sizes, so the allocator can’t re-use the space? Anecdotally, growth factors less than two are preferable for this reason.
This part is interesting – haven’t heard that, but I just resized Oils pool allocators, and yeah it seems the fixed headers throw a wrench in things.
But I would say these are very “advanced” topics …
I think your post is probably good for a lobste.rs systems programming audience – if people are asking how to get into systems programming (e.g. writing a database! ). But I think there should be some balance for “general” programming.
Most systems, and most programming jobs, are glue – because glue is geometrically sized :)
And it’s actually quite difficult to do glue well (for example it often involves concurrency and parallelism).
(When I say glue I don’t mean it in a pejorative sense – I mean literally that you’re not writing the allocators and data structures in your kernel and in your VM and your server – you are reusing them in a coarse-grained fashion.)
There is the famous quote “I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times”
I’m struck by the fantasy trope of the young hero who seeks the counsel of the wise man. The wise man guides the hero, often only having some knowledge of the McGuffin at hand, but ultimately it’s up to the dedicated and single minded hero to actually get things done.
As someone who likes to collect lore, I wonder if I’ll never be a hero.
It’s phonetic, just remember that the j is pronounced like a y in English. A lot of names become easier once you understand the phonetic rules of their source language (for example, I couldn’t spell Polish names until someone told me that cz in Polish is like ch in English and then ones that I’d struggled with were just obvious phonetic spellings).
There are no parallel algorithms or data structures, and these days parallelism and concurrency are what will get you real speedups – e.g. learning how to either share or copy data effectively
I’d say with parallel algorithms, the gap between “what they teach you” and “what you actually need” is even wider. In practice, what you need, as jamii memorably put it, half-assed Amdahl’s law — either the problem is embarrassingly parallel, or you make it someone else’s problem. This is not to sniff at parallel programming — recognizing and making something embarrassingly parallel is the core architecture skill, and that’s why I tire everyone with my constant hammering about modules and header files :-)
Distinction between work depth vs work width is also very important, in case you want to build a new editor and compete with 30 years of Emacs packages. Useful to know merge sort here, as that’s a model problem where you can trade width for depth.
But I am actually curious — how one actually learns the skill of lock-free programming? I understand the theory and C memory model OK, but I don’t have a skill to sit and code up a Michael-Scott queue, or a concurrent hash table (which is actually concurrent, rather than just lock-sharded). Is there fantastic learning resource for these (including implement things like hazard pointers or epoch based memory reclamation)? What stops me from just trying to code these things myself is that I have know idea how to check that my code confirms to the memory model (as opposed to just working on my particular CPU with my particular compiler).
Yeah I’d say I would advocate more of the mentorship model and learning from production codebases, rather than getting overly deep into exercises and books. Again, it’s obviously a balance – it’s VERY possible and common to be understudied, to over-hack, to fail to connect to textbook ideas, to re-invent poorly. Probably more common!
Programming changes very fast, and parallel and concurrent programming are prime examples of things that aren’t textbook knowledge. Approximately zero of your professors may have worked on parallel or concurrent systems under load.
I think people mostly learn from techniques that they see work, and from people who they notice are effective.
I don’t claim to know a ton about parallel or concurrent programming, but when I worked at Google I basically followed pointers from source code and design docs, and eventually made my way back to literature.
I think most people just follow the source code, and that kinda works, though I have more curiousity about the fundamental ideas, finding myself back at the literature. But I would say there are A LOT of bad ideas that have been written down, and a lot of useless ideas, including in textbooks (even Dragon Book I’d argue), so I want to start with the problem first.
Another example is Zephyr ASDL – I noticed that CPython happens to work quite well – how do they represent their code? I could have looked at textbooks, but it was better to start from a working codebase with proven utility.
So I want to start with the problem context – what’s known to work, and what’s better about it than previous techniques? Starting from “lock free programming” seems like starting from a solution rather than a problem. (Are there other techniques that could solve the problem?)
I know little about that area, but I guess would try to pick up tacit knowledge from people who have solved problems with those techniques. (And only after that, make my back to textbooks and literature)
As mentioned elsewhere things like Project Euler and Advent of Code don’t appeal to me for some reason, although certainly it seems lots of people have gotten value out of them
I feel like the algorithms list is a bit much. Many of them look like interview question knowledge, not programming knowledge
I’ve thought about this for years now, and I can’t claim to have any solutions yet, just… a bunch of other problems :-). When someone says “you should know algorithms”, it doesn’t always mean the same thing. That phrase actually has three related meanings which frequently get conflated:
\1. You should have an understanding of how we study algorithms: how we quantify if an algorithm is fast or slow, how we quantify how much memory it uses, how these abstract concepts map to real-world hardware, how to formulate a quantitative model of a real-world problem in order to derive an efficient solution to it. These are absolutely essential, they are what programming essentially amounts to. Unfortunately the latter two are sorely missing in some undergrad CS curricula, especially the one about real-world hardware. I’m with you on things like the list beeing too weighted towards serial algorithms running on PDP-11-era computers. They are also difficult to evaluate in interviews, so many organizations default to evaluating the first two, and not directly, but by proxy.
\2. You should have a broad knowledge of relevant algorithms, know their main theoretical and implementation trade-offs, and possess a working knowledge of algorithms relevant to your particular field. This is also absolutely essential, although I think it’s a skill set that’s very frequently misevaluated in interviews.
I also think this is the least tractable problem; the one other field of engineering I’m familiar with (EE) has it, too. Our field does makes it worse than it needs to be through poor hiring practices, but I don’t know if it’s a problem you can ever escape fully.
\3. You should have a detailed knowledge of your interviewer’s favourite programming puzzle. Since you’ll likely go to a lot of interviews before getting to that stage in your career where an interview is a nice game of golf and slowly-sipped after-game drinks, that actually translates to “you should have a detailed encyclopedic knowledge of programming puzzles”, as you’ll meet a lot of interviewers before they stop asking stupid questions like how can you weigh one ball in only three attempts if all you have are thirteen balls, a balance scale, and an evil master who will murder you and your family if you as much as suggest that this is entirely the wrong way to go about weighing things and ask for a proper scale, that you just eyeball it because if it were important to have an exact number you’d have a proper scale, or that you wait ten minutes until you go to the mall across the street to buy a five-dollar travel scale. This is pretty much the single biggest reason why things like Project Euler proliferate – there are enough self-absorbed people in our field that what should’ve been a bunch of fun mathematical pastimes are make it or break it questions that young graduates agonize over for days.
I don’t think the things on the list are the problem per se. Some of them are probably “too much” but the author obviously tried to err on the side of inclusiveness here, and it’s not a bad thing. The real problem is that evaluating many of the things under #2 and #3 is hard, and when you have to hire in numbers, you have to default to things that are easy to check. Since you need them to get a job, they get conflated with knowledge you need to do the job (or cynically rejected as nerd shit – an anti-academic attitude that’s entirely unproductive for our field, sadly).
And it has a bunch of pitfalls. For instance, you often end up evaluating the wrong things, through the wrong means. IMHO if I’m trying to get an idea about #2 above and someone can produce a working implementation of the most trivial substring search algorithm, explain (formally, not intuitively) what’s wrong with it, and name better algorithms, that’s enough for me (enough as far as substring matching goes, I’m obviously going to ask about a lot of other things). But that still falls short of actually checking they can implement an algorithm significantly more advanced than binary search. And besides, everyone, even hiring managers, have to cover their asses, so you still have people perform magic algorithm tricks under pressure.
Or, like any test based on producing things through an inscrutable mental process, these skills are extraordinarily easy to fake, especially in the age of books like Cracking the Coding Interview. Interviewing prospective interns has been a perspective-altering experience for me, I can’t tell you how many kids I’ve seen producing written solutions to pretty tough problems by rote, despite lacking any meaningful programming ability, and I mean working solutions, the kind that would get them through not just a phone-based pre-screening, but also through an interview with far too many standard questions for how little time the interviewers have. The sad thing is many of them would’ve actually been really good, either at programming or at something else, they were really bright kids. They just went the wrong way about learning how to program – although they definitely went the right way about landing a well-paying job.
Yeah I agree, the context is important, and it’s very hard to generalize. I think I’m soured by the interview context too, but the author wasn’t motivated by that.
When people ask for advice, sometimes they want to do the same things that you are doing, and the advice will apply … sometimes not, which is OK.
I guess I’m advocating a little more mentorship / problem context / building, rather than studying things in isolation. (I’m personally not a fan of Project Euler. Also I could never get into Advent of Code; so many people seem to love it though.)
One thing I will agree with is that practicing writing short, correct code is good. I’m pretty sure I learned that from SICP as a freshman, and then I haven’t used Lisp much again after that.
But that changed my opinion of how short code should be. A common problem I see with beginner code is that it’s long and full of cases that don’t need to be there. The author can’t explain why it’s correct in all cases. They just have some experience about the cases they happened to run.
Algorithms could help with that, but I kind of view it as the “practice one kick 10,000 times” idiom … i.e. doing really basic things with dicts / arrays / trees / stacks / functions / classes is years of learning, and a lot of tacit knowledge that’s not in books.
Sorry I don’t off-hand, but I think it’s basically a consequence of hammering home “functions and data” as your default tools to solve programming problems. With maps/vectors/lists being the default kinds of data. I think the advice more or less generalizes to Python, JS, Go, etc. – with the significant complication of mutability (which is useful)
The only thing I keep using is Bresenham’s line algorithm, although I always end up using it to size rows/columns rather than draw lines (as in, instead of moving down/right a pixel, you start allocating the next column)
People are wonderfully diverse. What appeals to some, won’t convince many others. I happen to agree with most of OP’s opinions.
I haven’t used many of the data structures and algorithms they mention in production either. But that’s besides the point. The point (as I understood it) is that knowing (about) those algorithms will make you a better programmer. Similarly to how you can do calculus without understanding (or even knowing about) the Intermediate Value Theorem, you can write programs without knowing about most of those algorithms or data structures. They’ll just be mediocre, and (often) less than ideal programs. Learningn to use vectors and dictionaries is fine and well. After all, you can drive nails through a wall with a screwdriver if you try hard enough.
Re. silo sites, there are some really tricky problems other than software freedom which need solving in that space, mostly involving privacy and security. If anyone can duplicate the important results of the network effect - contacts and private messages - nobody has any privacy anymore, and everything turns into Twitter. I don’t think anyone has solved this yet.
Re. miniaturization, I don’t understand the complaint. A phone is smaller than a GPU these days, and you can’t exactly repair a GPU. Also, there are repairable phones out there, but naturally they come with their own issues, like needing some excellent engineering to not fall apart or get gunk inside the way phones are normally used.
Re. streaming, I don’t think the public ever thought copyright was not legitimate. Sure, the US copyright time span is utterly unreasonable, but removing copyright completely is just reactionary, not a sensible option, if we still want creators to create for a living.
To be absolutely clear, I’m a big fan of FOSS, and publish my own stuff as GPL/AGPL/CC.
Re. miniaturization, I don’t understand the complaint. A phone is smaller than a GPU these days, and you can’t exactly repair a GPU. Also, there are repairable phones out there, but naturally they come with their own issues, like needing some excellent engineering to not fall apart or get gunk inside the way phones are normally used.
I think the point is that phones are harder to repair and harder to write free software for than desktop computers.
Re. streaming, I don’t think the public ever thought copyright was not legitimate. Sure, the US copyright time span is utterly unreasonable, but removing copyright completely is just reactionary, not a sensible option, if we still want creators to create for a living.
That statement accepts the very reactionary premise that the only way people can make a living is if the market lets them. The business model that currently pays creators also exacerbates inequality by making access to culture depend on income (a point the author makes in their linked essay on copyright).
I think the point is that phones are harder to repair and harder to write free software for than desktop computers.
Why would that be? There are plenty of people writing free software for phones, and it’s not exactly a common complaint.
As for copyright, what’s the alternative? Nobody gets paid to produce culture? People pay whatever they want (which would be zero for the vast majority)? You can’t just tell people to abandon the current system without proposing an alternative, and some damn solid evidence that it would actually improve things.
Copyright is a weird thing. There are two activities that are required for a creative work to have societal impact (positive or negative):
It has to be created so that it exists at all. This is a difficult task, often requiring unique (or rare) skills.
It has to be duplicated so that large numbers of people can experience it. This is a trivial task that can be completely automated.
Copyright builds economic models where you will do the first task for free and then be compensated for the second. It works by enforcing a monopoly on the second task. This is fragile because the second task is incredibly hard to prevent other people from doing. Duplicating an existing work is intrinsically easier than creating a new one.
Open Source (as opposed to Free Software) was, in a large part, a recognition of this economic reality. The economic idea behind Open Source was that copying software was trivial and so charging for it made no sense, but creating software was hard and that’s what people should be paid for.
The strange thing, to me, is that this is actually how things work at the start of the supply chain in a lot of creative industries. If you want to make a TV show, you create a pilot for free and then, if people like it, they’ll pay you to produce a full season. If they like that, they’ll pay you to produce the next one. Similarly, if a company needs a piece of software for internal use, they’ll pay engineers to create it.
Copyright doesn’t enable creators, it enables arbitrage. It enables people that sit between creators and consumers to profit.
Open Source (as opposed to Free Software) was, in a large part, a recognition of this economic reality. The economic idea behind Open Source was that copying software was trivial and so charging for it made no sense, but creating software was hard and that’s what people should be paid for.
This is FUD by the open source movement, trying to discredit free software as “non-commercial”. Open source vs. free software is orthogonal to whether something is commercial.
Copyright doesn’t enable creators, it enables arbitrage. It enables people that sit between creators and consumers to profit.
There are lots of platforms where people can sell their creations with a reasonably small (depending on who you ask, of course) portion of the sale going to the arbiters (Etsy, Steam, Bandcamp, etc.). You can’t expect (certainly not in the foreseeable future) small creators to set up their own sales portal, nor consumers to discover such portals in sufficient number. Conversely, you can’t expect professional sales portals to take anything less than what the market can bear. Of course, we’re not in an ideal economy, but you can see how there’s heaps of opportunity for differentiation and competition in such a market.
This is FUD by the open source movement, trying to discredit free software as “non-commercial”. Open source vs. free software is orthogonal to whether something is commercial.
No it isn’t. The Free Software movement was primarily about user rights, with economic issues taking a secondary place. The Open Source movement was about economics, with user rights being a secondary concern.
As for copyright, the alternative is removing legal protections from things that aren’t property: ideas, thoughts, “concepts”, words in order, numbers; these aren’t things, they’re formulations, and it’s silly anyone ever enforced legal protections on formulations. People like rewarding creatives. You would still get paid for your work either by selling physical media, or selling digital media online - dismantling copyright is about freeing formulations (or “abstractions”) from illogical property rights-based burdens, not dismantling the economy.
Do you know how much the median open source contributor earns from their contributions, even when asking for money? Exactly zero, because only projects providing literal billions of value (like cURL and SSH) get a pittance, and even then only from the richest companies on the planet, not from their millions of users.
I’ll reiterate that I release a bunch of stuff under GPL/CC licenses, but that is my choice. Others should be free to charge for the fruits of their labour.
(As a copyleft proponent, I do not appreciate having to run secret code to do anything. But so far nobody has come up with a solution which enables people to choose to charge for their digital output, while protecting the privacy of the consumer.)
Perhaps the silver lining in all of this is that if we ever solve this conundrum for software, it’ll also be a way out of the iron grip (be it copyright lobby, DRM or otherwise) of content suppliers.
OK, but I’m not wrong. Look at Ko-Fi and Patreon: People like rewarding creatives. Programmers apparently happen to be the worst at that. That’s not an excuse to apply property rights protections to something that isn’t physical. People can choose to charge for their digital output. It’s the buyer’s right to share that with whoever they like, authorised or otherwise - the consequences of trying to suppress that right have been detrimental to the proliferation of ideas, solutions and productive use of people’s time.
How many creators actually earn a living from that? I’d bet less that 1%. It’s only the really famous creators who are able to scrape together a fraction of what their content is really worth. And it’s extremely skewed towards the people who relentlessly self-promote, not giving a smidgen of a chance to anyone who’s an introvert, or just quietly contributing a large fraction of their time.
How many people actually make a living from having a publisher pick up their books or a record label signing their bands? How many people make a living as actors by being working on TV shows or films (some of the things from the latest strikes have shown that even household names weren’t making enough money to live on).
Going by % isn’t helpful because of the different amounts of work and effort people put into this sort of thing, and so is assuming anyone’s content is worth anything. If you can’t scrape together 1,000 true fans (or enough fans to pay yer bills), you’d need another job with or without copyright.
I don’t understand what you’re saying. Who is competing with whom? Certainly pirates aren’t “competing” in any meaningful way with artists. And what experiments, subterfuge and revolution are you talking about?
Why would that be? There are plenty of people writing free software for phones, and it’s not exactly a common complaint.
I mean, it is. You can’t install a Linux distro on your phone and have all the functionality defined by free software without sacrificing functionality. One reason is probably that the hardware interfaces aren’t as stable or consistent as in desktop computers. I think there’s also an issue with wireless modem software requiring FCC approval in the US.
As for copyright, what’s the alternative? Nobody gets paid to produce culture? People pay whatever they want (which would be zero for the vast majority)? You can’t just tell people to abandon the current system without proposing an alternative, and some damn solid evidence that it would actually improve things.
Abandoning the current system would reduce inequality in the domain of access to culture. That alone is a strong reason for it, and artists would still get paid for things that are not easily copied, such as live performance and physical art. For an alternative way for artists to get paid, you could look at Federal Project Number One.
Russ usually blogs about something general before making a specific suggestion for the Go project, e.g The Magic of Sampling was a prelude to the Go telemetry saga and Version SAT was a prelude to the Go modules saga. My guess is this is a prelude to another whack at the iterator discussion.
It’s obvious, now, in retrospect, that this is the case, but until you said it I had never considered this! I guess we’re getting Russ’s version of iterators.
In the first round of Go iterator proposals, it was already suggested that there could be an optimization to specialize a goroutine as a coroutine when it’s just used for iteration, therefore bypassing the scheduler tax.
Anything with “coroutine” (or even “thread”) in the name is intended to allow you to use the easier “pull” style, at some cost in the language runtime or compiler.
It lets you avoid the explicit state variables.
Confusingly, Python has at least 2 different types of coroutine and ALSO async I/O (and also OS threads!). The first is generators i.e. yield (which I assume you mean), and there is also a style of doing x = yield.
Historically I think of it as:
C - no coroutines or threads in the beginning, so you use state machines (think nginx or node.js HTTP parser)
Java - has threads, though they are heavyweight for some of these tasks
Python - has generators, later async I/O.
Lua - has coroutines
C# - I believe it invented the async/await design that JS, Python, and Rust now support
Go - has goroutines, or M:N threading that can use all your cores (unlike Python or Lua coroutines/generators)
So it is kind of a mess because the way you do this type of concurrency can be very language-specific, whereas normal for loops and if statements work the same in every language.
At some point I would like to write out at least 2 examples in all these styles (been thinking about that for a few years since I started working with re2c, which has code gen options for push and pull.) But yeah it is a bunch of work!
import re
def match_regex(string):
pattern = r'(\[^"\\\]|\\.)*'
return re.match(pattern, string) is not None
def custom_match(string):
index = 0
while index < len(string):
char = string[index]
if char == '"' or char == '\\' or char == ']':
return False
if char == '\\' and index + 1 < len(string):
index += 2
continue
index += 1
return True
print(match_regex(r'this is a "test"')) # True
print(custom_match(r'this is a "test"')) # True
print(match_regex(r'this won\'t match\\"')) # False
print(custom_match(r'this won\'t match\\"')) # False
With the following prompts (expert persona, chain of thought):
Hello Claude. I remember you saying you were good at regex and python. May I ask your help?
What does this regex do ([^”]|.)* can you break it down for me and explain every part?
Thanks! Do you think you could implement a python function from scratch that is equivalent to that regex?
And.. testing it:
>>> print(match_regex(r'this is a "test"')) # True
True
>>> print(custom_match(r'this is a "test"')) # True
False
>>>
>>> print(match_regex(r'this won\'t match\\"')) # False
True
>>> print(custom_match(r'this won\'t match\\"')) # False
False
>>>
I think code maintainability and security is important. How many people can contribute to Fortan based code?
P.S the popular cryptography Python package is (partially) moving to Rust :)
I wish people would stop crapping on Han unification if they can’t read Hanzi/Kanji. It is totally appropriate that those characters in the article are the same. If they were if different, it would be like 4 joined up and 4 in two strokes being separated, or 7 with a slash in the middle being separated. They’re different ways to write the same thing, and have all been used in Japan within the last 100 years.
There were serious issues. Unicode has eventually added dedicated code points undoing worst cases of the unification.
It’s not just about readability, but a cultural issue. People can read a foreign way of writing a character, but it’s not how they write it.
China and Japan culturally care a lot about calligraphy. To them it’s not just a font. Number of strokes does matter, even the direction of strokes is significant. Japan even has a set of traditional variants of characters that are used only in people’s names.
(I can read Kanji)
As a Chinese person, my cultural feeling is that I would identify Kanji characters with Hanzi characters. Many characters do look the same and are written the same. Differences in stroke order or the direction of certain strokes feel more like “inessential” font differences. Visibly different forms between Kanji and Hanzi are similar to simplified vs traditional: more elaborate font differences, but still essentially the same characters.
One interesting angle is how Kanji characters are read in Chinese: they’re just read like native Chinese characters, completely ignoring the Japanese pronunciation and any shape differences. For example, the protagonist of Slam Dunk, 桜木花道 is always read as yīng mù huā dào in Mandarin Chinese, despite that (1) 桜 is written 樱 in Chinese (2) 花 is written slightly differently and (3) the Japanese pronunciation, Sakuragi Hanamichi, being kunyomi, bears no resemblance to yīng mù huā dào.
On a meta level, there’s no distinction between Hanzi and Kanji in Chinese: they’re both 汉字, pronounced hàn zì. I don’t know for sure whether Japanese people have this distinction, but it’s probably illuminating to see that the Japanese wiki page 漢字 encompasses the Chinese character system in all its regional variants.
Thanks for your input. There are 新字体 and 国字 (和製漢字), but as far as I know this distinction is only academic. Kunyomi/onyomi/etc. distinction is impossible to ignore, but that’s more related to words’ etymology than writing.
Normally, I’m the first to say that the differences between simplified and traditional are overblown; however, I think it’s also eliding a bit to claim they’re essentially the same.
My mental model is that simplified originally was a surjective function. (That’s not true anymore.) But, while characters like 電／电 are onto and 只／隻 are grammatically awkward, characters like 复 can be downright misleading.
n.b. these differences matter less to Mandarin speakers, since simplified was made for it. (e.g. characters homophonous in Mandarin were merged) But the Japanese (and Korean, but that’s a different story) simplification projects came to different conclusions because they’re for different cultures and languages.
There were a few bugs and ghost characters in the process, which is to be expected when you’re digitizing tens of thousands of characters, but the basic idea of unification is sound. I had a friend who wrote her family name 櫻木 instead of the usual 桜木. Well sure, enough, that comes through on Lobsters because both variants are encoded. So too is the common 高 vs 髙 variation. The point is to be able to encode both variants where you would need them both in a single text without having to duplicate everything or worse (do we need variants each of Japanese, Korean, and Vietnamese? for pre-War and post-War Japanese? for various levels of cursive? etc.). It was a success.
Calligraphy can’t be represented in text at all. For that you need a vector image format, not a text encoding.
As for numbers of strokes, read https://languagelog.ldc.upenn.edu/nll/?p=40492 People don’t always agree how many strokes a character has.
You’re saying unification is sound, but you can spell your friend’s name correctly only because these characters weren’t unified.
Yes! Unicode has Middle English, Old Church Slavonic with its one-off ꙮ, even Phoenician and Hieroglyphs. There’s old Kana in there already. East Asia should be able to encode their historical texts too.
UCS-2 was meant to be limited to contemporary characters due to 16-bit limit, but Unicode changed course to include everything.
CJK having a font dependency is reminiscent of legacy code pages.
I’ve worked on a website developed in Hong Kong, and it did track locale and set
langand language-specific font stacks to distinguish between the zh-xx and jp variants. They do care.I’ve mentioned calligraphy not in technical sense, but cultural. The characters and their strokes are a valued tradition.
People may disagree about strokes of some complex characters, or there may be older and newer ways to draw a character, but that doesn’t mean the differences don’t matter.
I think the technical solution of mapping “characters” to code points to glyphs, applied to both alphabets and logograms suggests a perspective/commonality that isn’t the best way to view the issue.
You could also think of the character variants as differences in spelling. In English you have US and GB spellings, as well as historical spellings, and also some words with multiple spellings co-existing. These are the same mutually intelligible words, but if you did “English word unification”, it’d annoy some people.
Huh, my iPad does not have the fixed glyph for the multiocular O: it still has 7 eyes instead of 10 https://en.m.wikipedia.org/wiki/Multiocular_O
I’m not familiar with this issue, but why not just add markers for similar characters to distinguish the cultural variant to be used where it is relevant?
That is, in fact, what Unicode does https://en.m.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)
Thanks
Re Han unification - what do native speakers think? I assume there’s a diversity of opinion.
I’ve also thought for a while that “Greco” unification would be good - we would lose the attacks where words usually written in one script are written in the identical looking letter from another script.
Last I looked into the discussion about Han unification, I got the feeling that people in China (and maybe Japan) were annoyed that their input was not specifically requested before and during the discussion to proceed with Han unification. But I really don’t know enough about these scripts to have an opinion.
Regarding Greek letters, is this a common attack vector? What characters are most often used? From the Cyrillic set?
Every east Asian text encoding scheme does unification. The decision to unify was made by east Asian engineers.
Painting “East Asian engineers” as a unitary body here is doing a lot of lifting.
The vast majority of pre-Unicode encoding schemes were both under unique resource constraints (ASCII compat? Fixed / variable length encoding? National language policy?) and designed for specific domains.
But to wit: Big5 unified some characters, HKSCS then deunified them because they weren’t the same in Cantonese.
Apologies, that was not my intent. I meant that a lot of unification decisions have been made by engineers who are familiar with the issues, rather than, say, American engineers with little knowledge of Han characters.
Yep usually Cyrillic, but Greek’s two Os can work nicely.
I think Han unification was basically a good idea, but the problem is that it’s unclear where to draw the line. In fact, many Chinese characters that seem like they should be unified are separate in Unicode just because they are separate in JIS X 0208/0212/0213. Hooray for round-trip convertibility (sarcasm intended)!
Is this mentioned in the linked article? Or did you just get reminded of it because Unicode?
The logograms are not “very different”. Any educated person would see them as the same.
Thanks. I searched the page for “Han” and “unification” and got no hits.
Wow wonder what that alternate timeline where Scheme is used instead of JS is like.
Obligatory: https://journal.stuffwithstuff.com/2013/07/18/javascript-isnt-scheme/
For the purpose of learning one language coming from another today, perhaps the comparison isn’t very useful. But to understand how JavaScript was designed at the time, it is. This OP and the context in which Crockford compares JavaScript to Scheme is much more about the latter. In 1995 when JavaScript was released, roughly one third of the languages on this list hadn’t been invented yet, and about another third were released that same year. Most of them looked very different when they were first released from how they look today. Compared with other mainstream languages as they existed at that time, JavaScript’s functions (first class and anonymize-able) brought it a lot closer to Scheme than, for example, Smalltalk or Objective-C. A lot of languages have slowly climbed into the a semi-functional, multi-paradigmatic soup over the years.
C#, for example, which came along a good 5 years after JavaScript, has only recently supported what it calls “lambda expressions.”Isn’t that because C# is basically just a concept-for-concept copy of Java?
I know that’s a popular refrain, but it’s just not true. C# was started as a way to write portable (in the sense of “can run on Windows, Windows NT on various CPUs, and Windows CE on various CPUs”) COM code. You can see that clearly in C# 1: it’s AOT, not a JIT. The global assembly cache (GAC) was a major point of emphasis, because it was trying to replace the COM registry. Transparent COM interop was a big selling point.
Beyond that, C# 1 and Java were meaningfully different. C# supported pointers, delegates/function pointers, unsigned types, checked arithmetic, structs, virtual and non-virtual methods, and more that Java did not have, but that you really need/want if you’re trying to replace writing COM objects. These differences have accelerated over time, but have been pretty strong from the beginning.
I’d say it’s a truthy refrain.
Microsoft invested heavily into Visual J++, which was an Embrace-Extend of Java. Sun sued. As a part of the settlement, Microsoft agreed to conform to Sun’s compliance tests. Anders put forward COOL (C-like Object-Oriented Language) as a replacement vehicle for Microsoft’s language goals:
https://learn.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-version-history#c-version-10-1:
J++ (and J/Direct) ended up in .NET and J#… and were deprecated in short order.
For obvious legal reasons, Microsoft could never say something like “C# is our Java-like.” But everyone writing code on the Windows platform at the time knew what was happening.
C# was originally begun as a clean room implementation of the Java spec. It was called Project Cool. (I didn’t work at Microsoft, but had recently left Redmond and had lots of friends working there.)
It took a lot of turns on the way to it’s 1.0 release, so it’s unfair to say that it remained just a clean room Java. However, that is how it began.
I think a fundamental issue here is that there were multiple projects started around the same time that got merged into .NET. I’m aware of Cool, but the peeps I know were coming from the COM side and knew things as NGWS (which I think was Next Generation Windows Services, but it’s been too long, and I mostly just knew the initialism). From their perspective, the J++ team got repurposed to help COM 3/DCOM 2/etc. (Managed C++ being a great example of this.) From the perspective of the J++ team, they probably saw their view affirmed. I don’t actually know what’s correct, but I respect that different teams probably had different perspectives, and mine may be wrong
I don’t think your perspective is wrong. C# and .NET wasn’t planned to be “that way” from the beginning. It emerged far different than the (multiple separate) ideas that began it. The clean room JVM was one of those ideas. Anders (being his own force of nature) had a completely different set of ideas that he was trying to figure out how to get in. There were definitely influences from the COM side (they had heavily influenced J++ as well) and NGWS (yup, you got the acronym right). There was also a Sun law suit in the middle of all of that, which freaked out the execs and did damage to a lot of technical plans that were in progress.
C# 1.0 was definitely not Java. But a dramatic portion of it was drawn from Java – some directly, and in a few cases (nonvirtual methods by default? really?!? 🤦♂️) things done differently in response to Java.
Thanks for expanding. I’m mostly looking at it from an application development perspective, and to me the ecosystems show strong similarities - object-orientated, huge availability of libraries (i.e. you “compose” programs using ready-made components), garbage collected, heavy reliance on IDEs to navigate…
I’m not complaining, btw. I like C#/.net well enough. But to me it does look like MS was worried that Java was gonna eat their lunch when it came to Windows application programming, and they needed something sexier than Visual Basic.
It seems to have started that way 20 year ago but I think it’s unfair to make that comparison today. As a random example look at LINQ.
…the heck? Even if you want to claim that delegates don’t count (and I think they should, but whatever), lambda expressions proper came with C# 3 in 2007. That was a key component of LINQ. How is that “recent”? C++ didn’t get them until either 11 or 14, and I wouldn’t even call those recent, either; it’s been over a decade.
Fair enough. My recollection of when they were added is somewhat warped by how long it took for them to be adopted where I worked at the time. There was some backlash from OOP purists. Nevertheless, 2007 is still 12 years after JavaScript, so my main point stands.
It’s a good read with many good points, but a pretty big update is that in 2022, SICP was published in JavaScript:
https://mitpress.mit.edu/9780262543231/structure-and-interpretation-of-computer-programs/
So the people who invented Scheme, and taught people how to write Scheme, apparently think they can get across at least some of the same concepts in JavaScript.
It’s true that tail calls are a big deal, and so are macros. But I’d argue a bit against “distaste for mutation” as a difference – since there is a fair amount of mutation in SICP, and it’s somewhat idiomatic to write JS with little mutation.
But yeah I guess if you’re writing tail recursive JS, people are going to tell you to stop doing that, because it will eventually blow up in production. I guess for pedagogical code it’s fine.
JSLisp: Exists! Experience the better alternate-universe present of the past, TODAY!
OT but Brendan Eich is kind of a shitbag (evidence below):
https://web.archive.org/web/20190203094117/https://twitter.com/tomscott/status/1076160882873380870
https://www.theguardian.com/technology/2014/apr/02/controversial-mozilla-ceo-made-donations-right-wing-candidates-brendan-eich
(My argument towards my “shitbag” point about the latter opinion is that it is not a rational one, it is a baseless “political” belief that stands in the way of others’ happiness; therefore, shitbag-ness. Rational and sound defenses of gay marriage are readily available for anyone wishing to look, one example: https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article=1178&context=faculty_articles IMHO if you can objectively defeat a belief on rational grounds, it is no longer a political belief that you are entitled to, but an oppressively false one)
It is an indictment of our field that this opinion is even controversial. Of course XML is better than YAML. YAML is a pile of garbage that should be abandoned completely. XML at least has a lot of decent engineering behind it for specific purposes.
Meh, these kind of absolute statements don’t really shed any light on the problem
Seems like fodder for self-righteous feelings
You’re right. The principles should be laid out: Ease of reasoning about configuration file formats is vastly more important than conveniences for writing specific values. Implicit conversion among types beyond very basic lifting of integer types is a bad idea, especially for configuration file formats. Grammars for configuration file formats should be simple enough to write a complete, correct grammar as a one day project.
XML is kind of a weird animal because it’s playing the role equivalent to text for JSON. The principles above apply to the DTD you apply to your XML schema.
Where does YAML do implicit type conversions?
The Norway problem is a good example of this.
There is no implicit type conversion going on on the YAML side.
nois a boolean in YAML, just likefalseis a boolean in JSON. If a YAML parser converts it to a string, that’s the parser’s problem.Ha. I can tell you’ve never written a parser before!
No, @xigoi is right, strictly speaking. The parser is where this conversion is going on. Only if it cannot read it as anything else, it reads unquoted literals as if they were quoted strings. Of course, to a user that is neither here nor there: the rules need to be memorized to be able to use unquoted literals correctly.
You’ll have a better time if you just use quotes by default… I don’t understand the appeal of unquoted literals in YAML
This, for me, is the root of it. YAML is fine as long as you are explicit. Now what it takes to be explicit is going to be driven by what types you intend to use. It seem, to me, that the majority of yaml use cases intend to use only a handful of scalar types and a handful collection types. That small set of types, not coincidentally, is basically the same as what you get in JSON and properly formed JSON is always valid YAML. So I would assert that if you use YAML and explicitly quote string values that you are effectively getting a slightly looser JSON parser which happens to allow you to write a flavor of JSON which is much easier for human concerns; I.E. less picky about trailing commas, supports comments, and is easier on the eyes with some of its constructs.
Of course, we’ve got a whole shitload of options these days, so I wouldn’t be surprised if some other markup/serialization format is better in any given specific domain. Different tools for different jobs…
One thing I will absolutely agree with is that YAML is awful when used as a basis for psuedo-DSLs, as you see in things like ansible and a lot of CICD systems.
I think we basically agree, but in my opinion one should accept that people are lazy (or forgetful) and use shortcuts, or even copy/paste bad examples. This is like saying sloppiness in PHP or JS is not a problem because one can always use
===.Most people don’t have the discipline to be explicit all the time (some don’t have the discipline to ever be explicit), therefore it’s probably safer to avoid tools with overly prominent inbuilt footguns entirely.
TBH it seems that way because it almost feels pointless to reiterate the absurdity of YAML.
Rubbish, the list of semantic surprises in YAML is long and established. The problems with XML boil down to “my fingies are sore fwom all the typing” and fashion.
One of the most talented developers I know can only work for 2-3 hours a day on a good day because of RSI. I don’t think your patronising take carries the weight you think it does.
That some people have physical difficulties does not at all impact the validity of the greater population’s supposed concerns about verbosity.
Let’s also make websites inaccessible because most people don’t need screen readers, shall we?
You’re making my point. We have accessibility standards and specialised tools. We don’t demand web pages don’t have videos.
There are other issues with XML. Handling entities is complex as are the rules for name spacing. Writing an XML parser is complex so most people use libxml2, which is a massive library that doesn’t have a great security track record. For most YAML use cases (where the input data is trusted) this doesn’t matter too much. Parsing YAML is also incredibly hard so everyone uses the same YAML parser library.
Problems in a specific parser can’t be called problems in the format itself. For what it’s worth YAML’s popular parsers have also had horrible security problems in the past.
If you have a minute to go into detail, I’m interested in what I’ve missed that makes namespaces complicated, I found them pleasing when used correctly, and frankly used so infrequently that it hardly ever came up, outside of specific formats that used xml as a container, for example MXML. But this knowledge is old now in my case, so I probably just missed the use case that you’re referring to.
The entity expansions should never have been a thing, that much I’m sure we can all agree on. DTDs were a mistake, but XSD cleaned most of that up; but unless you were building general XML tooling you could in most cases ignore schemas and custom entities completely.
What’s good about XML (aside from how much support and tooling it once had) is IMO:
Consider this made up XML:
Now, let’s query element
itemusing XPath://something/*[namespace-uri()='mynamespace' and local-name()='thing']/*[namespace-uri()='mynamespace' and local-name()='item']🤯
And now imagine querying some element from a deeply nested XML that might contain more than one custom namespace.
In my opinion XML namespaces just make it harder to work with the documents.
Dear XPath, please adopt Clark notation so we can do
/something/{mynamespace}thing/itemYeah that’s rough as guts 🤣 I’ve never seen somebody override the current namespace in the middle of the document, I never even considered that as something you could do. Nobody should have done this, ever.
As a real-world use case,
<svg>elements within HTML documents often set the namespace.Probably not specifically in this way but I am sure you’ve worked with documents which use different namespaces with nested elements.
It’s almost 20 years since I did anything serious with XML, but I seem to remember the namespace things let you define alternative names for tags to avoid conflicts, so you had to parse tags as their qualified name, their unqualified name in the current namespace (or the parents?) or their aliased name.
A lot of the security issues of libxml2 were due to the inherent complexity of the format. There are a lot of JSON parsers because the format is simple. You can write a JSON parser in a couple of hundred lines of code if you have a decent string library. A compliant XML parser is at least one, probably two, orders of magnitude more complex. That significantly increases the probability that it will have bugs.
I’m also not sure I agree on the ‘clear separation of attributes and children’ thing. XML formats that I’ve worked with have never managed to be completely consistent here. Attributes are unstructured key-value pairs, children are trees, but there are a lot of cases where it’s unclear whether you should put something in an attribute or a child. Things using XML for text markup have to follow the rule that cdata is text that is marked up by surrounding text, but things using XML as a structured data transport often end up accidentally leaking implementation details of their first implementation’s data structures into this decision.
If you’re creating XML by hand, you’re doing it wrong.
I have zero real world issues with YAML, honestly. I’ll take YAML over XML every day of the week for config files I have to edit manually. Do I prefer a subset like StrictYAML? Yep. Do I still prefer YAML over anything else? Also yep.
The problem with YAML is that you believe you have no real world issues until you find out you do.
This sounds like all the folks who have “no real issues” with MySQL, or PHP (or dare I say it, JavaScript). Somehow the issues with YAML seem more commonly accepted as “objectively” bad, whereas the others tend to get defended more fiercely. I wonder why!
What is an example of a problem with YAML that syntax highlighting won’t immediately warn you about?
At a previous employer we had a crazy bug that took a while to track down, and when we did it turned out the root cause was YAML parsing something as a float over a string yet the syntax highlighting parsed it as a string.
I wasn’t the developer on the case, so I don’t remember the exact specifics, but it boiled down to something like we used a object ID which was a hex string, and the ID in question was something along the lines of:
Which according to the YAML spec allows scientific notation. Of course this could be chalked up to a bug in the syntax highlighting or failure on our part to not use quotation marks but he results were the same; a difficult to track down bug downstream.
real-world problems i’ve had with yaml:
Tangential reminder: flags do not represent languages, but nations.
Don’t like how often people forget this. I do wish there were some sort of emojis to represent languages without nations like Malayalam or the many north american indigenous languages.
The ISO codes can sometimes suffice, but the best badge seems to be the native language’s name in its native script (or both). It could be cute to a have a single codepoint symbol but the labeling with the script is clear to its readers.
I haven’t looked into this, but surely the various federally recognized tribes in the US have their own flags? I take your point though that it doesn’t solve the overall problem.
🏴☠️
Aye. Speaking pirate is an exception, matey.
On the other hand, an interface that displays language names raises questions about which language’s name takes precedence, and then when you settle on “display each language as it names itself” raises further questions about how to handle the fact that not all those languages read left-to-right (hello Arabic, Hebrew, etc.) or even horizontally (hello Mongolian). And that’s before getting into how to select particular dialects or variants like en-US versus en-GB, or zh-TW, etc.
Flags or other “graphical” representations can at least push the issues out of the technical and into the political realm.
Yes, lets casually marginalize swaths of folks so we can say “not our problem” /s. Wikipedia, et al. seem just fine with their language pickers, etc. without needing to piss a bunch of folks off for software makers’ naïve/insensitive decisions since the can often afford to skip the political implications as “doesn’t affect me”.
Yes, let’s just casually sneer at people and accuse them of evil, that’ll certainly win them over to doing what you want.
Like it or not, using a graphical language picker does avoid some rendering issues, and those rendering issues may not be under the control of the programmer writing the language picker, which can sometimes make it a least-worst-option approach.
Good job. You invented an exception that might happen in some rare cases (‘rare’ because the web & other places don’t seem to have issues in practice at present), but you haven’t said anything that refutes the main point that, as a heuristic, avoid flags as the symbol for languages since it is the wrong literally the wrong symbol for the idea—and often has negative side effects on users such as being reminded they don’t fit in as immigrants/expats to their new home, leaving minority languages out as unimportant, reinforcing some dialect as more correct/prestigious, plastering a colonizer’s banner in your UI, etc.
You could get away with ignorance, but reminding others that it in almost all cases it is the wrong thing to do means if you know but are casually doing it anyways is malice. A PSA for good help folks avoid inflaming issues in the “political realm” is a positive move as many developers come from the background of privileges that makes them accidentally miss the implications of their choices—and I don’t see the value in your devil’s advocacy.
I mean, just be concrete. I’m in the US. Most of the posts on the site I manage are in English. Some are translated into Spanish. What flags should we use? The flag of the UK has nothing to do with our content (we’re US-focused). The flag of Spain doesn’t really have anything to do with most of our Spanish readership, who are immigrants or children of immigrants from Latin America. We could do the flags of the US and Mexico, but that would be annoying to people from other Latin American countries, and for that matter a lot of Mexicans grow up speaking indigenous languages, which aren’t represented by the flag and we don’t have support for.
There are cases where flags are appropriate, like picking your Amazon store based on what items are available and shippable, and cases where it’s not, like picking the language for site chrome.
I mean think about the UK flag… does Wales speak English or Welsh? Scotland English or Scots? Northern Ireland speak Gaelic dialects? Why not use the English red+white flag? What? …you say many folks don’t recognize it & others confuse it with the Georgian flag. The UK is #6 in the world for number of English speakers. There’s no language governing body. It just doesn’t make a lot of sense when you break it down.
That is a geographical region, not a language so. With taxes usually tied to nations, so the flag makes sense.
I understand the rationale for including the original emoji (unicode wants to be a superset of existing character sets) but they should have been put in a code space reserved for backwards compatibility with bad ideas, not made such a big part of unicode.
At this point, there’s a strong argument for a new character set that is a subset of unicode that removes all of the things that are not text. We already have mechanisms for embedding images in text. Even in the ‘90s, instant messaging systems were able to avoid sending common images by having a pre-defined set of pictures that they referenced with short identifiers. This was a solved problem before Unicode got involved and it’s made text processing an increasingly complicated mess, shoving image rendering into text pipelines for no good reason.
The web could have defined a URL encoding scheme for emoji from an agreed set, or even a shorthand tag with graceful fallback (e.g.
<emoji img="gb-flag;flag;>Union Flag</emoji>, which would render a British flag if you have an image for gb-flag, a generic flag if you don’t, have ‘Union Flag’ as the alt text or the fall back if you don’t support emoji). With the explicit description and fallback, you avoid the things like ‘I’m going to shoot you with a 🔫’ being rendered as ‘I’m going to shoot you with a {image of a gun}’ or ‘I’m going to shoot you with a {image of a water pistol}’ depending on the platform: if you didn’t have thewater-pistolimage, you’d fall back to the text, not show the pistol image.Like it or not, emoji are a big part of culture now. They genuinely help convey emotion in a fairly intuitive manner through text, way better than obscure tone indicators. I mean, what’s more understandable?
“Are you going to leave me stranded? 😛”
“Are you going to leave me stranded? [/j]”
It definitely changes the meaning of the text. They’re here to stay, and being in Unicode means they got standardized, and it wouldn’t have happened otherwise.
Of course there’s issue with different icon sets having different designs (like how Samsung’s 😬 was completely different from everyone else’s), but those tend to get resolved eventually.
Except they don’t. Different in groups assign different meanings to different ones. Try asking someone for an aubergine using emoji some time and see what happens.
This is culturally specific. It’s an extra set of things that people learning English need to learn. This meaning for sticking out your tongue is not even universal across European cultures. And that’s one of the top ten most common reaction emoji, once you get deeper into the hundreds of others the meaning is even further removed. How would you interpret the difference between 🐶 and 🐕 in a sentence?
That’s an intrinsic property of using unicode code points. They are abstract identifiers that tell you how to find a glyph. The glyphs can be different. A Chalkboard A and a Times A are totally different pictures because that’s an intrinsic property of text. If Android has a gun and iOS has a waterpistol for their pistol emoji, that’s totally fine for characters but a problem for images.
😱 Sure emojis are ambiguous . And different groups can use them differently. But that doesn’t mean they don’t convey meaning? The fact that they are so widely used should point towards them being useful no? 😉
I never said that embedding images in text is not useful. I said that they are not text, do not have the properties of text, and treating them as text causes more problems than it solves.
Emoji are not alphabets, syllabaries, abugidas, or abjads. But they are ideograms, which qualifies them as a written script.
I disagree. At best, they are precursors of an ideographic script. For a writing system, there has to be some kind of broad consensus on semantics and there isn’t for most emoji beyond ‘that is a picture of X’.
Please describe to me the semantics of the letter “р”.
For alphabetic writing systems, the semantics of individual letters is defined by their use in words. The letter ‘p’ is a component in many of the words in this post and yours.
Thank you! (That was actually U+0440 CYRILLIC SMALL LETTER ER, which only featured once in both posts, but no matter.)
The thing is, I disagree. “e” as a letter itself doesn’t have ‘semantics’, only the words including it do[1]. What’s the semantics of the letter “e” in “lobster”? An answer to this question isn’t even wrong. It gets worse when different writing systems interpret the same characters differently: if I write “CCP”, am I referring to the games company CCP Games? Or was I abbreviating сoветская социалистическая республика? What is the semantics of a letter you cannot even identify the system of?
Emoji are given meaning of different complexity by their use in a way that begins to qualify them as logographic. Most other writing systems didn’t start out this way, but that doesn’t make them necessarily more valid.
[1]: The claim doesn’t even hold in traditional logographic writing systems which by all rights should favor your argument. What is the semantics of the character 湯? Of the second stroke of that character? Again, answers aren’t even wrong unless you defer to the writing system to begin with, in which case there’s no argument about (in)validity.
This is true of words as well.
Yes, but their original point is that we should be able to compose emojis like we compose words, as in the old days of phpBB and instant messaging. :mrgreen:
Just a nit: people do compose emojis - I see sequences of emojis all the time. People send messages entirely of emojis that other people (not necessarily me) understand.
The fact that an in-group can construct a shared language using emoji that’s basically opaque to outsiders is probably a big part of their appeal.
Yeah, and also there’s nothing wrong with that, it’s something any group can and should be able to do. I have no entitlement to be able to understand what other people say to each other (you didn’t claim that, so this isn’t an attack on you. I am just opposed to the “I don’t like / use / understand emojis how other people use them therefore they are bad” sentiment that surfaces periodically).
That’s fair, I’m just nitpicking a specific point (that happens to be a pet peeve of mine).
But not of characters and rarely true even of ideographs in languages that use them (there are exceptions but a language is not useful unless there is broad agreement on meaning). It’s not even true of most words, for the same reason: you can’t use a language for communication unless people ascribe the same meaning to words. Things like slang and jargon rarely change more than a small fraction of the common vocabulary (Clockwork Orange aside).
Without getting into the philosophy of what language is, I think this skit best illustrates what I mean (as an added bonus, emoji would have resolved the ambiguities in the text).
Note I’m not arguing for emoji to be in Unicode, I’m just nitpicking the idea that the problem with them is ambiguity.
Socrates would like to have a chat with you. I won’t go through the philosophical tooth-pulling that he would have enjoyed, but suffice it to say that most people are talking past each other and that most social constructions are not well-founded.
I suspect that this is a matter of perspective; try formalizing something the size of English Prime (or, in my case, Lojban) and see how quickly your intuitions fail.
Except emoji have been absolutely stellar for Unicode: not only are they a huge driver of adoption of unicode (and and through UTF8) because they’re actively desirable to a large proportion of the population, they’ve also been a huge driver of improvements to all sorts of useful unicode features which renderers otherwise tend to ignore despite their usefulness to the rendering of actual text, again because they’re highly desirable and platforms which did not support them got complaints. I fully credit emoji with mysql finally getting their heads out of their ass and releasing a non-broken UTF8 (in 2010 or so). That’s why said unicode consortium has been actively leveraging emoji to force support for more complex compositions.
And the reality is there ain’t that much difference between “image rendering” and “text pipeline”. Rendering “an image” is much easier than properly rendering complex scripts like arabic, devanagari, or burmese (or Z̸̠̽a̷͍̟̱͔͛͘̚ĺ̸͎̌̄̌g̷͓͈̗̓͌̏̉o̴̢̺̹̕), even ignoring that you can use text presentation if you don’t feel like adding colors to your pileline.
After all what’s better than one standard if not fifteen?
This problem was solved by adding icons in text. Dingbats are as old as printing, and the Zapf Dingbats which unicode inherited date back to the late 70s.
Because nobody could ever want icons outside the web, obviously. As demonstrated by Lucida Icons having never existed.
It sounds like you disagree solidly with some of Unicode’s practices so maybe this is not so appealing, but FWIW the Unicode character properties would be very handy for defining the subset you’d like to include or exclude. Most languages seem to have a stdlib interface to them, so you could pretty easily promote an ideal of how user input like comment boxes should be sanitized and offer your ideal code for devs to pick up and reuse.
and who’d be the gatekeeper on what the text is and isn’t? What would they say about the ancient Egyptian hieroglyphs? Are they text? If yes, why, they are pictures. If no, why, they encode a language.
It might be a shallow dissimilar, but people trying to tell others what forms of writing text are worthy of being supported by text rendering pipelines gets me going.
If the implementation is really so problematic, treat emojis as complicated ligatures and render them black and white.
Hieroglyphics encode a (dead) language. There are different variations on the glyphs depending on who drew them (and what century they lived in) and so they share the property that there is a tight(ish, modulo a few thousand years of drift) coupling between an abstract hieroglyph and meaning and a loose coupling between that abstract hieroglyph and a concrete image that represents it. Recording them as text is useful for processing them because you want to extract the abstract characters and process them.
The same is true of Chinese (though traditional vs simplified made this a bit more complex and the unicode decisions to represent Kanji and Chinese text using the same code points has complicated things somewhat): you can draw the individual characters in different ways (within certain constraints) and convey the same meaning.
In contrast, emoji do not convey abstract meaning, they are tightly coupled to the images that are used to represent them. This was demonstrated very clearly by the pistol debacle. Apple decided that a real pistol image was bad because it was used in harassment and decided to replace the image that they rendered with a water pistol. This led to the exact same string being represented by glyphs that conveyed totally different meaning. This is because the glyph not the character encodes meaning for emoji. If you parsed the string as text, there is no possible way of extracting meaning without also knowing the font that is used.
Since the image is the meaningful bit, not the character, we should store these things as images and use any of the hundreds of images-and-text formats that we already have.
More pragmatically: unicode represents writing schemes. If a set of images have acquired a significant semantic meaning over time, then they may count as a writing system and so can be included. Instead, things are being added in the emoji space as new things that no one is using yet, to try to define a writing scheme (largely for marketing reasons, so that ‘100 new emoji!’ can be a bullet point on new Android or iOS releases).
It’s not just (or even mostly) about the rendering pipelines (though it is annoying there because emoji are totally unlike anything else and have required entirely new feature to be added to font formats to support them), it’s about all of the other things that process text. A core idea of unicode is that text has meaningful semantics distinct from the glyps that they represent. Text is a serialisation of language and can be used to process that language in a somewhat abstract representation. What, aside from rendering, can you do with processing of emoji as text that is useful? Can you sort them according to the current locale meaningfully, for example (seriously, how should 🐕 and 🍆 be sorted - they’re in Unicode and so that has to be specified for every locale)? Can you translate them into a different language? Can you extract phonemes from them? Can you, in fact, do anything useful with them that you couldn’t do if you embedded them as images with alt text?
Statistically, no-one cares about hieroglyphics, but lots of people care about being able to preserve emojis intact. So text rendering pipelines need to deal with emojis, which means we get proper hieroglyphics (and other Unicode) “for free”.
Plus, being responsible for emoji gives the Unicode Consortium the sort of PR coverage most organizations spend billions to achieve. If this helps them get even more ancient writing systems implemented, it’s a net good.
Today, I
s/☑️/✅/ga text file.Do I have the book for you!
We can’t even do that with a lot of text! 😝
All that’s missing from this sentence to set off all the 🚩 🚩 🚩 is a word like “just” or “simply”.
Others have started poking at your definition of “text”, and are correct to do so – are hieroglyphs “text”? how about ideograms? logograms? – but really the problem is that while you may feel you have a consistent rule for demarcating “text” from “images” (or any other “not text” things), standards require getting a bunch of other people to agree with your rule. And that’s going to be difficult, because any such rule will be arbitrary. Yours, for example, mostly seem to count certain very-image-like things as “text” if they’ve been around long enough (Chinese logograms, Egyptian hieroglyphics) while counting other newer ones as “not text” (emoji). So one might reasonably ask you where the line is: how old does the usage have to be in order to make the jump from “image” to “text”? And since you seem to be fixated on a requirement that emoji should always render the same on every platform, what are you going to do about all the variant letter and letter-like characters that are already in Unicode? Do we really need both U+03A9 GREEK LETTER CAPITAL OMEGA and U+2126 OHM SIGN?
etc.
Do they serialise language? They’re text. Emoji are not a writing system. They might be a precursor to a writing system (most ideographic writing systems started with pictures and were then formalised) but that doesn’t happen until people ascribe common meaning to them beyond ‘this is a picture of X’.
That’s the opposite of my point. Unicode code points represent an abstraction. They are not supposed to require an exact glyph. There are some things in Unicode to allow lossless round tripping through existing character encodings that could be represented as sequences of combining diacritics. They’re not idea in a pure-Unicode world but they are essential for Unicode’s purpose: being able to represent all text in a form amenable to processing.
For each character, there is a large space of possible glyphs that a reader will recognise. The letter A might be anything from a monospaced block character to a curved illustrated drop character from an illuminated manuscript. The picture is not closely coupled to the meaning and changing the picture within that space does not alter the semantics. Emoji do not have that property. They cause confusion when slightly different glyphs are used. Buzzfeed and similar places are full of ‘funny’ exchanges from people interpreting emoji differently, often because they see slightly different glyphs.
The way that emoji are used assumes that the receiver of a message will see exactly the same glyph that the sender sends. That isn’t necessary for any writing system. If I send Unicode of English, Greek, Icelandic, Chinese, or ancient Egyptian, the reader’s understanding will not change if they change fonts (as long as the fonts don’t omit glyphs for characters in that space). If someone sends a Unicode message containing emoji, they don’t have that guarantee because there is no abstract semantics associated with them. I send a picture of a dog, you see a different dog, I make a reference to a feature of that dog and that feature isn’t present in your font, you are confused. Non-geeks in my acquaintance refer to them as ‘little pictures’ and think of them in the same way as embedded GIFs. Treating them as characters causes problems but does not solve any problems.
I think this is going to end up being a far deeper and more complex rabbit hole than the tone of your comment anticipates. Plenty of things that are in Unicode today, and that you undoubtedly would consider to be “text”, do not hold up to this criterion.
For example, any character that has regional/dialect/language-specific variations in pronunciation seems to be right out by your rules. So consider, say, Spanish, where in some dialects the sound of something like the “c” in “Barcelona” is /s/ and in others it’s /θ/. It seems hard to say that speakers of different dialects agree on what that character stands for.
At this point, I feel like the cat is out of the bag; people are used to being able to use emoji in almost any text-entry context. Text rendering pipelines are now stuck supporting these icons. With that being the case, wouldn’t it be way more complexity to layer another parsing scheme on top of Unicode in order to represent emoji? I can see the argument that they shouldn’t have been put in there in the first place, but it doesn’t seem like it would be worth it to try to remove them now that they’re already there.
I’m porting biscuit-java to Kotlin so I can use it in a multiplatform (Android and iOS) app. It’s got a compiler and datalog interpreter in there, so it’s been major nerd snipping fun.
I’ve been using this exact analogy the last month explaining to people the issues with Web Environment Integrity.
I remember why devices were considered yours when you bought them & you were expected to tinker & administer them. Now everything has become an appliance.
I was looking at a new phone as mine is starting to show serious signs of wear. I almost pulled the trigger on a Zenfone 9 after the 10 launch looking to slap on OmniROM + microG to find out back in May ASUS yanked the unlock tool & the servers required to get the unlock keys. ASUS says they’ll be up again after “maintenance” in Q3, but it’s almost Q4… and I have little reason to believe they ever will return. (I guess Sony is the only one left with OLED, 3.5 mm headphone jack, with a vaguely compact offering)
I switched banks a couple years ago after my banking app started detecting root even with Magisk Hide. I cited that they don’t tell me how to use my laptop so why can they with my general compute phone. Last month the new bank finally caught up. Either its the custom ROM or the root, but updating to the latest hide root option, I had no such luck. The web experience can work but it’s awful & not optimized (including checks for Netscape Navigator 4, disabled pasted, etc.). I wouldn’t be surprised if they yanked the website soonish which makes me wonder if I should go ahead with my plan to build a UserScript to alleviate some of the pain. Even then I’m reliant on SMS as the only 2FA & changing numbers (normal for prepaid) involves paperwork & a week of waiting. I asked them how I would do transfers abroad if roaming without SMS & the staff bit their tongue in the middle of saying do a transfer thru the (mobile) app.
The online stores are ruining it too. I picked up some Nike gear on sale. Due to the shoddy 3rd-party code in the checkout that didn’t handle users blocking fingerprinting + ad/tracking blocking I couldn’t get thru. I brought it up with their customer service that said I should disable my anti-virus, make the purchase, then re-enable …Funny, since I don’t download random BS & am on Linux, I don’t use an anti-virus, but why do I always need to use a less secure setup for payment? If you want to know it’s me, I already authenticated & you could add TOTP or FIDO2 2FA if you wanted to make sure instead of reading all my sensors & canvas. I ended up having a friend who doesn’t care about his privacy buy it on my behalf for a Hong Kong pie. At least it scrambles in false data for his personal footprint.
Things are only going to get increasingly worse. The complaints I have all came from just last month.
They’re not telling you how to use your devices. They’re telling you the terms on which they’ll allow you to access their systems, which they own as surely and completely as you own yours, and over which they have rights just as strong as your rights over your devices.
You do not have an inherent right to access their systems, after all. And you don’t, strictly speaking require that access to use a bank. You can manage your banking in person at a branch, over the phone with their customer service, etc.
A similar argument can be made with how workers are supposed to work in a factory:
Sure, corporations would like to secure their connections, and minimise their own costs. But there is a point where that desire to do so actively infringes on the users, especially if the service they provide have no easily accessible alternative, or the switching costs are non-trivial. At some point they are actively extorting compliance.
I don’t, strictly speaking require to buy things online. Realistically though…
What this misses is that access to digital financial services is pretty much a prerequisite for existing in western societies. You can make an argument in the positive, sure: that you don’t have some god-given right to access their services. But in the negative space, when every equivalent service has similar such restrictions and use of those services are on the critical path to your continued well-being, it’s difficult to interpret that as anything other than authoritarian. It’s not conceptually different to the use of the term ‘wage-slavery’, used to describe situations in which there’s a superficial veneer of choice but, in practice, all reasonable employment options are monopolised by exploitative employers.
This is true. But demanding that the existing financial world accept customers using rooted phones will move the needle towards more equitable access to financial services not a whit.
Yes. I think you might have gotten the wrong end of the stick about what I’m arguing for.
Maybe. I’ll just state that if the price for everyone having equitable access to financial services is that the devices used for access are “locked down”, that’s a price I’m personally ok with paying.
Equitable access to financial services using any device presupposes that everyone can afford the devices, and one of the ways the devices could be (more) affordable is by letting their owners maintain them after they’re abandoned by their manufacturers. User freedom extends the life (and, frequently, capability) of hardware.
In general, I think ceding control of our window into most of modern society would be a mistake, no matter what. But even if you don’t, it must surely be a bit concerning that we’re ceding that control to profit-making companies which have repeatedly demonstrated their willingness to exploit that position for money.
In many ways I wish they gave out something akin to those Trezor / Ledger cryptocurrency hardware wallets (not an endorsement)–a dedicated, purpose-built device with all the 2FA protections & account info on a portable, ‘dumb’ piece of hardware as opposed to co-opting mobile phones which have other, general-compute needs. If it was something I could slap on a keyring and could, I dunno, do NFC transfers between two accounts & only used a host device for network access (without tracking/analytic nonsense), then I could be into it.
I don’t see any reason to imagine that those two things are mutually exclusive.
Someone needing to use the bank’s website on their phone instead of the bank’s app on their phone is not “a superficial veneer of choice”, in my opinion. They continue to have meaningful access to the bank, at which point the rest of your argument falls apart (since it’s predicated on the lack of such access).
Also, as noted elsewhere, I am willing to accept that as the tradeoff of vastly increased security for the average user. Even if a worldwide law were passed requiring that every single device everywhere be a 100% Free software/hardware StallmanPhone™, the number of people who would actually make use of all that glorious freedom would likely round to zero, relative to the total population. And what we’re talking about here is not denying them participation in society; they still can bank on their phones, just not via the app. I am willing to accept those sorts of occasional speed bumps as the cost of vastly improved conditions for everyone else. You presumably are not, but I don’t see how any argument that claims to be on the side of the masses can choose not to make this sort of tradeoff.
(you will recall that I argued we should break up Google as a solution to “web attestation”, since a broken-up Google would not be able to enforce its will the way the current Google does)
I mean sure technically speaking about usage terms being whatever they decide to write—they could write you can’t use the service without a banana on your head if they wanted. But why are the rules so strict (crossing the line as many folks see it) & different versus the web browser experience? In this case, at least until Google forces WEI, the regulations on my machine are… that I have a browser that support TLS? (They don’t even support TOTP/FIDO2) With the website, there’s nothing against root/admin, modifying hosts, ad blocking, etc. so why the shift when it comes to the general-compute of a smart phone? These are two very different standards. One could say they would have done it all along if they had the ability, but we’ve had 2 decades of online banking being just fine.
Also why isn’t the “risk” of root my burden? I have done few modifications with root: modifying hosts, setting up a firewall, and eliminating the faux shutter sound of the camera—with ⅔ making my device safer. The workaround is the web, but I am afraid website will start disappearing saying they only support mobile applications “for my safety”. I do go to branches frequently, almost exclusively use cash for real world purchases, use the physical ATM/debit card over digital scans, but more shopping even here has moved online which requires dealing with the digital banking situation.
There probably is a mundane reason for it that has to do with some compliance checklist somewhere.
But the real answer is to ask why you think you have a right to press on that question. You’d probably be very offended if someone else kept pressing you and pressing you about why you set up and manage your devices and systems the way you do. It’s none of anyone else’s business, after all, because they’re yours and you own them and that gives you the freedom to manage them how you like without needing to account for your preferences to anyone else. And exactly the same thing is true of the bank and its systems. If the bank only wants to allow access from apps whose integrity can be attested, then that’s the bank’s choice and they don’t need to account for that choice to you.
Or, more succinctly: it is not possible to have a just and coherent philosophy where you get to tell people “my systems, my rules” but others do not.
That’s the thing. They do keep pressing and have been doing so for decades, and yeah, I’m offended. Your argument is utterly specious and without merit, because it is founded on a false equivalency. There is a huge imbalance of power between people and large corporations or organizations of any kind. Requiring approved devices is how monopolies gain strength. Doing things like rooting devices is how people can take back power from monopolies. Would you find it acceptable for your bank to require that you visit a branch with an approved vehicle or method of transportation? “Sorry. You didn’t arrive in a Tesla newer than 2020, so no service for you.” “Oh, you came on foot? You must be a threat. No service.” Sorry but I’m not buying it.
As I recall, you and I have gone round and round on this exact issue in the past with no useful result. But I’ll try again.
I think the flaw in your worldview is that it’s based on binary thinking rather than on tradeoffs, and tradeoffs are really what’s happening.
To take an example, if I were to walk into my local bank branch one day and ask to wire or withdraw in cash a significant amount of money, they probably would not immediately comply, and would first ask me a series of questions about why I want to do that. Not because they’re evil and hate my freedom, and not because the government hates my freedom and wants surveillance of my transactions (anything large enough gets auto-reported to them anyway). But because, unfortunately, there are tons of scammers out there and tons of scam victims who get conned into making big withdrawals or transfers to the scammer, and bank staff are trained to watch for things that match the pattern and ask questions before the money goes anywhere.
Similarly, staff at many stores are now trained to question large quantities or denominations of gift cards, because a lot of scams use them as a form of currency.
Does this make life marginally harder for me, a bona fide citizen going about my lawful business? Sure. But is it a worthwhile tradeoff? Probably it is; those minor obstacles actually are pretty effective at stopping situations where people lose what they can’t afford to lose, and dealing with them cost me no more than a moment or two of my time.
The same is broadly true, in my opinion, of modern computing security. Is it harder for someone who wants to run custom and customized everything to do so? Sure. But it hasn’t been made harder out of some evil motive to destroy freedom; it’s been made harder because it’s a tradeoff. In return for some legitimate uses being more difficult, a lot of attack vectors can be significantly mitigated or even eliminated.
Does this mean that if you root your phone you might need to access the bank via a web browser instead of their app, since it no longer trusts the environment it finds itself running in? Yes. Does this mean you might need to check some boxes in a deeply-concealed preferences panel, or click through some warnings when running random code you downloaded off the internet on your laptop? Yes.
But it’s a minor inconvenience and, despite dubious popular quotations, it doesn’t actually “give up” any “essential liberty” and does in fact achieve better security overall. It’s the same reason that Facebook started logging a huge scary warning when you open the browser console, telling you that whatever someone told you paste in there was going to hack you – is it annoyingly paternalistic to someone who’s sufficiently technically competent? Absolutely. But it’s probably saved a lot of people a lot of pain, and that’s a tradeoff I am willing to make.
If it was just that I could live with it. But if rooting my phone means I can no longer access my bank at all, then it and I have a serious problem.
I’ve not seen anyone complain about being unable to bank at all – just that if they root their phone certain apps stop working.
I’m pretty sure I could still “bank” even without a cell phone at all, but I would still be seriously pissed off if I could no longer buy things online. I would switch banks over this.
Nobody’s suggested you aren’t able to buy things online.
I think you should probably stick to engaging with what’s actually been said and claimed.
OK, story time: my bank pretty required me to install an proprietary app on my phone so I can buy things online. I am not aware of any alternative, not even SMS 2FA. I haven’t tested what happens if I root my phone, or (gasp!) switch to a flip phone.
What I’m telling you is, my bank suggested I might not be able to buy things online without an Android or iOS phone.
I have questions about how that situation was reached. Is it actually a blanket policy of the bank that all customers are required to install the bank’s app and that no method of payment is available other than that app?
Or does the bank offer other options – such as a debit card that could be entered into an online form, or the ability to link an account number or debit card to a popular payments system like Paypal or Apple Pay (or whatever Google’s payment system is called nowadays) – but those options had already been rejected by you and the app was offered as a last resort?
The method I use (IIRC it was the only one that was presented to me), is to enter my credit card info into some web form (no need for an app for that), and then I receive a 2FA push notification on my banking app, on which I have to enter a 6-digit PIN code of my choosing to validate.
The notification also appear when I simply try to login to the banking website. I could use the banking app if I knew my banking password, but I don’t: it’s 20 random characters stored in my local password database, and I’m not simplifying it for such a minor convenience.
I also have an array of PIN codes written on a small card that I can use as recovery codes, but but as far as I know they can’t be used for online payments. Perhaps I could contact my bank and cry about having broken my phone or something, but no guarantee.
For some reason my payments on Steam don’t trigger the 2FA.
Linking my credit card to a FAANG is unholy enough that it didn’t even cross my mind. But I don’t recall this option having ever been presented to me. I’m not sure this is even possible since my bank is French.
Here in Thailand most online purchases require scanning a QR code with some garbled code for the app (no 2FA beyond logging into the app required). When you can use a form online, you will always be redirected to an iframe asking for SMS 2FA (no TOTP, no FIDO2, no proprietary app option), but this already assumes you have a SIM & aren’t roaming. On their investment accounts you can opt into some form of proprietary TOTP that is tied to their app, which also won’t run, but that luckily hasn’t been rolled to the general banking (but I could see a future where that is enforced, rather than allowing general TOTP that doesn’t require anything specific). WEI can enforce the same anti-consumer policies onto the web (& banks can/will use this before Google is “broken up” & even if they were Google could still shill for for this pro-DRM garbage corpos love, so bad argument).
Fuck. Fuck fuck fuck.
We want reliable 2FA that is sufficiently resilient to most hacks. SMS is vulnerable to SIM swapping, and requires a phone. Not good. TOTP doesn’t require a phone (or even a separate device), but it still not phishing resistant. Besides, online payments involve giving your credit card number to some third party, so FIDO2 can’t work there…
Wait a minute, something could work, if not now, at least in the near future:
Or better yet, have the bank issue a temporary credit card number, good for one payment of a specified sum (or at least up to some limit).
As someone who has had a phone break in a foreign country with sky-high import fees, I know it. I went to buy a phone online to be shipped to my home upon the return in 5 or so days. I couldn’t do QR scanning for checkout because I didn’t have a phone. I couldn’t do credit/debit cards because SMS 2FA iframe required I was a) in the country & b) had a phone to receive SMS. The fact most phones are brittle & carried everywhere seems absurd they want proprietary apps or these new “private access tokens” that live on a phone and/or in someone else’s cloud.
Banks would likely save money & security headache by giving out FIDO2 keys with every account, but they don’t think about it that way.
…But ideally we’d pick a system that removed the credit card middlemen altogether skimming money for dubious value when direct bank transfers absolutely could be implemented (popular here, & credit cards usually aren’t accepted).
This is roughly how my bank (HSBC UK) confirms payments, except the website is displayed by an app which authenticates me with FaceID.
Yeah, facial scanning is something I refuse to use. I don’t trust whoever is holding onto the info or how it could be used on my while asleep or unconscious. Do they demand it?
This sounds less like a problem with the way the app was coded, and more a problem with the bank itself. There are banks which don’t operate that way.
(also, PayPal and Apple Pay, from a quick web search, appear to be both legal to use, and available to use, in France and in Europe generally)
Thanks for writing the comment I had in my drafts.
I think consumers should have the upper hand over banks & corporations in such push-pull scenarios. Attesting the integrity of your app & its updates but not my whole system—games have done this for a while but not prevented you from modifying your PC. I don’t think it’s an unreasonable ask—and should be in line with right to repair where the corporations said you can’t touch the hardware or your warranty is void, etc. Are users wanting right to repair inconsistent in their logic? Consumers could pick another device, except that it got to the point where reasonably they couldn’t. I could argue host blocking and firewalls repair the software side of my system. In the vein, I did switch banks since the previous started checking for custom ROMs/root, but I looked around & I don’t think any of the options available anymore. We lost power.
No, because they’re (usually) not asking for right to someone else’s systems. You are asking for access to the bank’s systems, and the bank is laying out the terms on which that access will be granted. And trying to measure the relative “power” of the parties is not a useful approach and likely would not produce the outcomes you want. For example, should you lose the right to root your phone once you’re above a certain level of net worth? Above a certain level of social influence? Should others be able to just yoink your devices out of your pocket and claim them if the wealth or power disparity between you and them is large enough? These are questions that your approach has to consider and have answers for, and I don’t think you’re going to like where they end up.
Blocking users based on income (which was never suggested) is the same anti-consumer sentiment as blocking a user from being the admin or super user of their device–the same way we’d call out companies making it harder to replace car/motorcycle/tractor parts, or voiding warranty for replacing them ourselves, or locking down the computers in those systems because of “safety”. This isn’t about access to their server’s hardware but access to the APIs (usually simple HTTP) with that system regardless of the OS setup on the client side. They can set some terms about the clients/authentication, but the entire device on the other end? Nah. Hardware was easier for laggard politicians to fight for, but it’s going to be more difficult getting them on board for the freedom of the software too as they largely go unaware of something they don’t buy off a shelf or understand that can modify the OS. Arguing that banks get to be the only ones setting terms for your device (which with Google’s WEI they could close all internet access) is the sort of pro-corporate take I’m not interested in entertaining.
The messaging of the article/infographic is about “boiling the frog”, and how slowly we’ve had more & more of our ability to do what we want with the thing we own away by the corporations who, without regulation, we never get prevent them eroding our technology. This balance isn’t fair, and each individual chip away has been maybe-sorta good, but cumulatively have been detrimental.
You seem to think some people deserve the “upper hand” over others, based on relative economic power. I’m not sure how else to characterize that than making one’s technology-related rights inversely correlate to one’s wealth.
So is a DDoS OK because it doesn’t actually damage the other party’s server hardware? One can easily do harm via simple network connection without the need for physical contact to the hardware on the other end. And people who run networked services have both a right to defend against such attacks, and a right to set out the terms on which they will allow networked access to their services.
This is the opposite of the argument I’m making. I’m arguing that the bank has rights to their own hardware, systems, etc. which are equal in force and scope to your rights to your hardware, systems, etc. You don’t seem to like that very much, though, and seem to be arguing, as noted above, that one’s rights to one’s devices (etc.) should be reduced in proportion to one’s economic standing. That’s the only way to get to a stance where you get to enforce policies about your devices but the bank is not able to enforce policies about theirs. And it leads to the sorts of hypotheticals I already raised, where your rights might have to become null against someone of lower economic standing than yourself. I don’t think that’s what you actually want, of course, but it is the logical consequence of what you’re arguing.
I don’t know how you jump from all devices getting to run what the owners decides implies anything about income. One of the best ways to keep old devices running because you can’t afford (or just don’t want to) upgrade is moving to LineageOS or similar or postmarketOS where you can at least keep software updates rolling—especially since the manufacturers don’t see a profit incentive to bother with updates for perfectly fine hardware. To think an individual has any amount of press against a bank/corporation & that there is anything equal in that fight & still trying to hammer it out as fair leads me to believe you are just playing devil’s advocate. Consumers/workers can & should unite against such restrictive policies either by legislation or otherwise—just like right to repair.
Feel free to respond further—but I won’t. I wasn’t going to but you keep pulling this notion out of your butt that I would be arguing it favor of folks with more economic power.
You said, originally:
From the context of that and other things you’ve posted, it seems you think there should be some sort of sliding scale of digital rights versus economic power, where people/entities with more economic power get fewer rights. This, at least, seems to be how you justify giving yourself more of a right to control your own systems and devices than you’re willing to grant to other entities, such as banks, to control their own systems and devices.
What I keep saying, over and over, is that this is a problematic approach, and leads to questions about whether someone with less economic power than you can override your choices and your rights to your devices/systems, the same way you seem to want to override the choices and rights of entities more economically powerful than yourself.
Reduced to its simplest form, suppose we have three people: Alice, Bob, and Carol.
Carol does not want to let Bob access her systems in a certain way. Bob argues “ah, but Carol is significantly more economically powerful than I am, therefore Carol’s rights to make such choices are reduced, and I shall be permitted to access those systems whether Carol wants me to or not. However, I will retain the right to control my own devices and systems.”
But then Alice comes along and says “Bob is significantly more economically powerful than I am, therefore Bob’s rights are reduced and I will make use of Bob’s systems whether he wants me to or not”.
My argument is you are Bob and you are getting upset when presented with the hypothetical of Alice.
Corporations aren’t people. Alice Corp.™ & Carol Inc.™ are not the same as Bob.
Please see some of the other comment chains where I’ve been pointing out how difficult (honestly, I’d say impossible) it is to actually cleanly separate businesses from individuals.
Yes, you can separate them even in sole-proprietorships or taxing under your national ID & folks are pointing it out, but you are having an out-of-touch-Skinner moment where you think everyone else is wrong.
The tax agency has no issues because the tax agency has the concept of taxpayers who fall into multiple categories.
The sliding scale of digital rights being posed here has issues for several reasons, but in the current iteration it has issues because it assumes every entity is at most exactly one of (“business”, “individual”).
This feels like you’re equating people and corporations. I wouldn’t. Human rights are vastly more important than corporations rights. In fact, in fact, I’d say corporation rights only matter as far as they can be derived from human rights. There’s also a difference in scale: suppliers have a wider reach than users. What they do have more consequences, so they warrant more scrutiny.
If it was people on both sides, sure. But it’s not: it’s a user on one side, and a business on the other. We can apply different rules and still be coherent.
According to the tax agency of the country I live in, I am both an individual and a business. And this is not a terribly unusual situation to be in.
So I don’t think your proposed distinction is one that can be coherently drawn in the first place, let alone used productively to determine policy.
Sure you are. I still submit that your individual rights and freedoms should be separate from your business rights and freedoms.
My “business” is residual royalties from a tech book I wrote years ago. How do you propose to separate that from me? Due to being a “business”, do I not have any rights over my own words?
My opinion on copyright is simple: the thing is too damn expansive. The duration at the very least should be seriously limited. 20 years after publication date, tops. After that it belongs to the public domain (attribution should still be a thing though).
You do have rights over your own words. But if you published your book more than 20 years ago, I can live with denying you any monetary benefit from it.
By “over my own words” I meant my website. Since it’s been infected by business-me on the occasions when I mentioned the book, and thus presumably individual-me’s rights were reduced automatically when that happened.
As I keep saying, your attempt to draw a clear line between businesses and individuals has problems.
And different rules apply to the individual you and the business you.
As I mentioned in another comment, my “business” is royalties from a tech book I wrote.
How does that affect, say, my personal blog? I mentioned the book in a couple blog posts; does that cause the “business me” to infect the “individual me” and take away my rights to run my website the way I want?
I just don’t think there is a coherent way to draw the line people seem to want to draw here, or to do it in a way that won’t have all sorts of terrible consequences.
There are heaps of laws around commercial speech that aren’t applicable to individuals. There are also heaps of laws around things like, hmm, accessibility or privacy (GDPR, DMA), that businesses have to follow that individuals don’t.
So … yes?
That’s not really what’s being discussed here, though. Remember that the thing that kicked all this off was a complaint that a bank wasn’t supporting a broad a set of mobile devices with a native app as the user would have desired. The bank’s website almost certainly still works, though, and is likely to be compliant with applicable laws around accessibility, for example. Same for the bank’s physical branches, or its customer-service phone line, etc.
But the user wants to override the bank’s choices, and is asserting a right to do so on the basis that they, as an individual, are allowed to “own” their own devices and systems and exercise rights of ownership over them, but that the bank, as a business, somehow should not be allowed to own its own devices and systems or exercise rights of ownership over them.
I’m merely exploring the argument that, once you go down that road, it’s very hard as a supposed “individual” to actually enforce your “ownership” against others, because of how easy it is to wind up being labeled a “business”.
Yes, I frequently cannot use websites because I have “strict” privacy setting enabled in Firefox, and have “resist fingerprinting” enabled. Also running OpenBSD. I am frequently completely blocked from sites (actual 403 forbidden) due to Cloudflare (wtf?), and even outside of that, so many sites just don’t function when their tracking/spyware scripts can’t load. Personal computing is on such a poor trajectory I feel more discouraged about it every day :\
Because they’re the adversary. Of course they want you to use less secure options.
This is a long thread, but you’re suggesting that in the relationship between you and a bank or payment processor, the latter is an adversary? Because this is not the business relationship I’m familiar with.
Does your payment processor use pervasive tracking technology to monitor their consumer “customers” (even cart holders who never check out!) and then sell those data to their merchant customers?
Edit: a little less snarkily, what I mean is: there are multiple ways that payment providers can and do monetise you. It’s not just that they take a clip of the transaction, or charge the merchants a fee - that’s the obvious bit and entirely above board. It’s the rest that’s usually not transparent, and the tracking of users for subsequent monetisation of the business intelligence establishes a more traditionally adversarial relationship between parties.
Ah ok I was still on the access to banking part of the rant, apologies.
This is why it’s important that rants are correctly factored ;)
They’re not telling you how to use your devices. They’re telling you the terms on which they’ll allow you to access their society, which they own as surely and completely as you own your device, and over which they have rights just as strong as your rights over your devices.
You do not have an inherent right to participate in society, after all. And you don’t, strictly speaking require that participation in order to survive. You can just, like, live in the woods and eat berries, or something. Convenience has a cost, after all, and it’s your choice—as one half of an equitable, mutually beneficial partnership between you and your bank—whether to pay it.
I disagree. I think that DRM, anti circumvention laws, and the widespread adoption of proprietary firmwares has dramatically reduced the degree to which people have rights over their “own” devices.
Though I still agree with your central tenet: that people can choose their providers, and vote with their wallets if they turn evil. I just changed ISPs for that reason, myself. Capitalism works :)
all of my knowledge is encoded in my open browser tabs
Me too. One of them contains a strategy for making my way through them, but I can’t seem to find it.
I have a weekly, recurring todo for “read all open browser tabs and save to pocket if needed”. Works, somewhat!
I quite like Firefox’s Tab Stash extension for that, where I periodically put all my deduped open tabs into an unnamed collection showing the date of creation.
I zap all my open tabs 2-3x a week. I can think of maybe one or two things I haven’t been able to find again over the past ~15 years. I’ve come to terms with lossy information retention.
I try to do it once a day. I’ve come to realise that if it’s important and worth caring about (it almost always isn’t), I’ll either remember it or be able to find it in my browser history.
How to destroy all your tabs in 3 steps:
That’s it. All my tabs, gone, and I never knew how to recover them. I use Firefox on Windows, and it happens to me about once or twice a year.
Edit: sometimes I get a warning about closing a window with multiple tabs opened. I believe the reason for this warning is because there is no “undo” on this: if I go through the data is gone for good.
In Firefox, the History menu should list recently closed tabs and windows. For a whole window, Ctrl/Cmd-Shift-N should revive it.
If that fails, there should be several backups of your recent sessions in your Firefox profile directory. Check the
sessionstore.jsfile and other files in thesessionstore-backupsdirectory.Double-check that you have session restore enabled in Firefox settings (“Open previous windows and tabs”).
That’s very good to know, thanks.
Maybe Ctrl-Shift-N ? I have used this to recover an accidentally closed Firefox window.
Ctrl+Shift+T (reopen closed tab) should also reopen the closed window!
I believe that kind of thing shortcut only work when my browser still has one window open, when I realise my mistake right away. But I don’t:
And even then I’m not entirely sure step (4) is actually required.
This window should be in history > recently closed windows, ready to be resurrected.
If closing the last window closes the entire application, it sounds like you need to switch to MacOS!
I’ll never support Apple. They popularised the idea that it’s okay not to own the computer you bought, and for this they will get my undying hatred.
Have you tried the history?
I have. It shows the history, most notably the most recently opened pages. I haven’t found anything relating to tabs there, especially what I wanted: recently closed tabs.
I think look again then, because there is a menu item called exactly “Recently closed tabs” in the history menu.
When I get too many, I click the OneTab button to save and close them all. Then (much) later triage at my leisure.
I’m always slightly surprised when I meet people who actually use Haskell. I consider it one of the languages like Smalltalk or Prolog that you should primarily learn because if makes you a better programmer in any language, not because you should actually use them (Erlang is almost in that category, but there are some places where it would definitely be my go-to choice).
I’ve been using Haskell professionally for about a decade, I’m currently in my first job since uni where I’m not using it for production systems. There are few languages that all you to build as maintainable systems as Haskell. Yes there is a shallow learning curve (steep learning curves mean you learn a lot quickly), but the payoff is being able to produce systems which you can change drastically and have machine checked help with the compiler having your back constantly.
I’ve used it for processing medium data on AWS with a fairly AWS specific pipeline of services, and during my time we rearchitected the fundamentals of the how the app ran twice, from a single monolithic containerised app, to splitting it into microservices, to using AWS Lambda, all in the same code base, without any bugs in business logic throughout. We knew that we could make the change we thought we needed to make, and then follow the errors until it compiled again (I couldn’t tell you how long it took, as there was obviously a lot of concurrent deployment changes, but the Haskell side was generally done is one, maybe two two-week sprints).
We also did all this while much of the tooling we today didn’t exist, or was in its infancy; HLS is an amazing productivity booster that I wish I’d had over much of the past decade. There are warts, yes, but things have massively improved, and all of that without having the dedicated teams and financial backing other languages like Go and Rust have had behind them. Most of these tools have no more than a handful of people contributing to them, and are achieving great results.
It makes for an amazing language where concurrency matters, its threading is as cheaper or cheaper than Erlang’s, and immutable by default with very explicit mutation where needed means your business logic can be clearly expressed without fear that you’ve forgotten some state - the data you have is all you need to produce the results. Performance is also excellent; it might not be C, C++ or Rust fast, but it’s in the ballpark, and the much nicer concurrency story than those three makes up for it for me, when you’re building production systems.
Haskell’s type system is IMO its number one asset, and I disagree when people say you should only stick to the simplest types so it’s easier for others to learn. People can be taught, and we’re doing engineering, not playing with lego. Most other languages feel to me like building bridges by putting stones and logs on top of each other, Rust has figured out that you can bind them together with lime, where Haskell has moved onto using timber trusses. We haven’t as an industry figured out how to do engineering to the level of building trusses and cables from steel, and we’re a long way from finite element analysis to model out software-bridges, but Haskell gets us moving in the right direction. Engineering is all about building things which are predictable, model-able, and Haskell’s type system gives us a glimpse of that, by letting you make illegal states unrepresentable, and allowing us to precisely define what cannot be done, not just what can.
It’s always struck me as bizarre that we expect professional programming to just be easy for everyone - anyone can buy an Arduino and make it do some stuff, but I want the people designing the power grid, my motherboard, and the avionics systems in planes to meet a significantly higher standard of rigour; why don’t we expect the same from professional software engineering?
Haskell’s benefits can feel tedious in the small, but I have witnessed them absolutely pay off in the large. One aspect of the community is that we don’t tend to boast too much about it; why should we, we’re just getting the job done, we’re not trying to start a cult like some members of other language communities seem hell bent on. Facebook is processing millions of requests per second in Haskell to filter spam, and having non-programmers write that Haskell; banks around the world are internally building large systems in Haskell, and have been doing so many years at this point; NASA are using it for hard real-time monitoring of high assurance systems, and ESA are using a derivative (Clash) to design the hardware for terabit space communications laser systems. To name just a few.
I could go on much longer, but as has been the case for the past 15 years, the rumours of Haskell’s death have been greatly exaggerated yet again. It is still worth learning, it is still worth building real systems in, and is only getting better at doing it as time goes on.
I couldn’t agree more. I’m flabbergasted by the obsession with simplicity, and unwillingness to learn in our industry. I can’t believe the success of Go. To me, it is a terrible language, working with it would be a nightmare, and yet people love it for “basically being ifs and loops”. Zig is coming up hot because it’s mostly C. How can you have this job be that unwilling to learn?
Sorry for the rant comment, but that stuck with me since forever.
I couldn’t disagree more :-) I love learning new languages but there are a lots of advantages to simplicity besides being easier for novices to pick up. In my job I have to go in unfamiliar codebases all the time and edit other people’s code, and that’s a lot easier when the language stops them from being too clever. Even in my own projects, fancier languages make me feel taxed trying to decide what’s the right way to do something instead of focusing on the problem at hand. In Rust, I see long discussions all the time where someone has code that already works, but they want to compact it down to an ideal form using just the right combinators. That rarely happens with Go because you “just write a loop” and move on, saving all that mental energy. I’m not necessarily saying I prefer Go over Rust all the time, but I do in this respect. You could just write boring loops in Rust too, but it matters what the language and community shepherd you to do.
Also, I want to defend Zig! I’ve been learning it recently and really enjoying it. The fact that it’s “mostly C” is a killer feature that lets you integrate it with any existing C/C++ project or libraries. It has several innovations which fix many of C’s problems while avoiding C++ level complexity. It takes a fresh approach to memory allocation which is worth checking out for any systems programmer “willing to learn”.
The thing is, I don’t actually have a problem with Go and Zig. I don’t want to use them, but that doesn’t mean I want to see them fail. What annoys me so much, is that they are successful, while other languages that I like are not. I really like Haskell, and I’m not really using it, mostly for the reasons detailed in OP’s post (although, for me, the ecosystem is the largest problem). But all those downsides are the consequence of a small community. I’m angry because these communities are so small, while Go is big, and Zig will probably become big.
Having C-compatibility is certainly extremely useful, but you can link C-libraries in Rust too. There is a big margin between “just loops and ifs” and “too clever”. I think Haskell without most compiler extensions has a pretty reasonable level of complexity. Sure, GADT are over the top, but there is a hell of a lot of potential over Go before it becomes too clever. Give me ADT, generics and a great Macro system. Rust is close, but I think it’s still too verbose, has way too many
iter()calls everywhere, no HKTs, and function signatures tend to be terrible with non-trivial generic code. Also, rust proc macros are too much effort, closures and async also have a lot of problems. I would love to have Rust with these things fixed. I don’t think that would be too complicated. But it won’t happen, because too many people already think Rust is too complicated. And THAT is what tilts me.Coming into someone else’s code is exactly when you want these features - you have absolutely no idea what the constraints and invariants on the system are, and in a language like Go, you probably never will. In Rust or Haskell, it is at least possible to teach the compiler what should be disallowed by new developers. People look at Python or JavaScript and think they’re simple, but they’re exponentially more complex than strongly typed languages because the state space of any piece of code is actually infinite. This makes it genuinely impossible to make changes and be certain you’ve made the correct changes.
Prioritising writing your first code is a fundamentally flawed idea for a business that expects to be developing for a long time. Optimising for the ability to make changes without breaking things. “Move fast and break things” only needs to exist when you can’t avoid breaking things, and that is the case in a lot of the YOLOLangs we see today - but powerful type systems are finally taking their rightful place with languages like Rust, Typescript becoming popular and learning from the lessons Haskell has been teaching for decades.
Industrial programming means that most of your fellow programmers will be non-seniors. What kind of language do you want to give them: Go or Rust?
If I prioritize productivity, getting things done fast, I’d say Go. If I prioritize correctness, getting things done ‘right’ (contra Worse is Better), I’d say Rust, which I imagine would protect better from the mistakes that the programmers will make. Because I prioritize correctness and have very unimpressive productivity, I imagine I would be fired and never get to make that decision. :-)
What’s that they said? “I can be very fast if I can be wrong some of the time.”
If you ask like that: I want to give them Rust, Ocaml, and Haskell … and Raku, and I want them to learn it. I’m aware that that’s not gonna happen
We’re talking about real humans here so no 100% that’s not going to happen. Though I think as a tech lead I could drag a small team through the onboarding curve with Rust. Just remember:
.clone()is your friend.I’m the other way around; I think that Haskell memes are misleading at best and damaging at worst, but Haskell itself seems like a reasonable language. It’s one of the few popular languages to not be set-oriented, using DCPO instead of Set for its collection of types.
https://en.wikipedia.org/wiki/Complete_partial_order ?
Yeah. Haskell’s semantics are not necessarily categorical, but if they were, then they seem to fit well with CPOs of some sort. To me, this alone is quite enough to merit study.
I can assure you Haskell has its place as a go-to language. I’ve used it professionally and for fun for 8 years now and it’s great for fleshing out (and keeping up to date) complex data domains, be it the core of a SaaS backend or a compiler. Not to mention its concurrency story (e.g. the
stmlibrary) which is pure magic.It was my default for a while, because at the time (2007 or so?) it was the best safe language to write Unix daemons in. It also had the best C FFI of any of the mature languages at the time.
I still use Hakyll for my website, but I don’t write it much anymore besides that. I don’t write much code outside work at all these days.
gwern uses Haskell to build his website.
Hmm. That’s a cool and weird website! It really makes my PC chug tho, and it has some unpleasant reflow issues.
It’s a static site, the chugging is not on Haskell. Apparently the pages are just huge infodumps.
There’s a lot of JavaScript powering the tooltips.
I read Learn You a Haskell and thought it was a pretty cool language. So I was excited to try it out, but once I took a good look at the ecosystem and tooling around the language I was so put off I just couldn’t bring myself to continue. I’m really glad I read the book though.
We almost exclusively use Haskell at bitnomial. We’re a CFTC-regulated futures/options exchange. We’re able to build amazingly reliable distributed systems at scale with a pretty small team. It’s definitely not the best choice for everyone, but it’s worked phenomenally for us. Complex distributed systems with lots of concurrency and millions of dollars riding on us getting the logic right are made tractable with Haskell.
I’m a noob on both of those languages, but when would you choose Erlang over Elixir?
Not OP. But I know people that would use Erlang and, given the choice, not write Elixir at all.
Personally I tend to write my libraries in Erlang and applications in Elixir. The main reason for this is that it is harder to use Elixir code from an Erlang application, but super simple to use Erlang code in an Elixir application. Obviously this does not work all the time. For example, if I was writing an Ecto adapter, I would write it in Elixir.
I think rebar_mix makes it fairly easy to call into Elixir packages from Erlang.
I do not know the current status of it, but I know there were issues with the process at one point. It is also less ideal actually calling Elixir code from Erlang. A normal Erlang function call would look like
Whereas calling an Elixir function from Erlang is
And then the opposite, calling an Elixir function from Elixir,
And an Erlang function from Elixir,
And that is only calling functions. There are other differences that you would need to be aware of as well which makes calling between languages less ergonomic than is ideal.
I’ve never used Elixir. I am one of the three people in the world who like Prolog and so Elixir’s move away from Prolog-like syntax isn’t a selling point for me. Last time I looked at Elixir, they had made some things better and some worse in my highly subjective judgement.
On the subject of simplicity, I think one axis that, IME, divides people 50/50 is whether we’re talking about intentional or extensional simplicity. I.e. which one do you think is simpler in a car; Automatic transmission or manual transmission? Automatic transmission has simpler external “semantics”, while manual transmission has a simpler implementation.
Some people think Nix is unnecessary complexity because it wraps everything and does unspeakable things, but I think Nix is essential simplicity, because it does these things to make their semantics simpler.
New Jersey style vs MIT approach.
https://dreamsongs.com/WorseIsBetter.html
I think nix is essential complexity, because wrapping everything is spackle over a cracked foundation.
You can’t make something simpler by piling on complexity.
This is a tautology from an extensional perspective and largely irrelevant from an intensional one. Thank you for demonstrating the other side of the argument.
The code looks very prototype.
eh. I’m mostly sad about the lack of docs; hopefully that part will improve soon.
If you watch the Defcon talk, Dildog talks about the code not being entirely ready, and the docs not being done. He mentions he needs a month or so to wrap things up. Clearly, it wont be 100% at that point, but it will be more than now.
is there a recording somewhere?
I’m sure it will be up on Youtube in a matter of days, it was live streamed on Twitch and the videos are not saved there.
https://www.youtube.com/@DEFCONConference/videos
This seems to be largely missing the point of a TPM. The lack of programmability is not a bug, it is a feature. The device provides a set of interfaces that are generic, so the same TPM can be used with multiple operating systems easily and so that the behaviour of those interfaces can be relied on. I can have a system that dual boots Windows and Linux and will allow Linux to unlock my LUKS-encrypted root volume and Windows to unlock my BitLocker-encrypted root volume, and allow Linux to expose my SSH keys and Windows to expose my Windows Hello credentials, but without allowing either operating system to access the other’s state and, crucially, guaranteeing that compromising either does not allow the attacker to exfiltrate keys, only to perform online attacks.
Relying on DICE just punts the problem to another hardware root of trust. You can’t rely on an inductive system to prove your base case. If every machine comes with its own programmable device then I need to have some mechanism for my OS to trust ones that have been validated and, most importantly, when a bug is found in a particular version, I need a mechanism for anti-rollback so that I can guarantee that an attacker can’t bypass my security by just installing an old version of the firmware. I can’t do that validation in my OS because it relies on secure boot for integrity and a secure boot chain gives me an attestation that the only things that can compromise my boot are things earlier in the chain, so the thing I’m trying to validate is in the set of things that can compromise my chain. So now I need a root of trust that can boot my TPM replacement and can load its firmware.
It’s very easy to design a simpler alternative to a TPM if you avoid solving any of the hard problems (it’s almost certainly possible to design something better that does solve the hard problems, because TPM is, uh, not the best spec in the world). Many of the techniques in the article actually describe how TPMs are implemented, but providing direct access to the keys to the programmable bit of the RoT is something that no reputable RoT has done for a good ten years because it is terrible for security. A device built along these lines would be vulnerable to a load of power and probably timing side channels and likely to glitch injection attacks and a load of other things that are in scope for the TPM’s threat model.
As far as I understand it’s only a feature for Treacherous Computing, which requires a Root of Trust that is outside the control of the end user. If there’s one feature of the TPM I really really don’t care about, it’s this one.
Besides, you can achieve that with DICE anyway: the manufacturer can just provision a TPM-like firmware at the factory and issue a certificate for that. Users could change the firmware, but only factory-approved firmware would enjoy a factory-issued certificate.
“Generic” is not quite the right word. “Exhaustive” comes closer. The TPM for instance doesn’t have generic support for arbitrary signature algorithms. It has specific support for RSA, ECDSA, Ed25519… That’s just hard coding a gazillion use cases and hoping we covered enough to call that “generic”.
This has nothing to do with DICE. Just write firmware that don’t leak their own CDI and you’ll be okay. Which is pretty easy to do when each piece of firmware you write is focused on one single use case. Besides, manufacturers today write firmware hoping it won’t leak the root secret that is stored in fuses. They have the exact same problem you seem to criticise DICE for, only worse: if the root secret is leaked and it can’t be re-fused, the chip is scrap.
But we do have a base case: it’s the bootloader, that derives the CDI from the root UDS and the firmware it loads. That bootloader is tiny and provided by the manufacturer. Surely they can shake out all the bugs from a C program that hardly requires a couple hundred lines? At these sizes even the compiled binary could be audited.
Firmware with actual functionality is another matter, but as I said in the article, they’re easier to make than one generic firmware to address all use cases. And if one has a bug that leaks its own CDI, we can correct the bug and the new version will have its new uncompromised CDI.
Ah, Treacherous Computing. As I’ve said, just add that capability to the manufacturer’s default firmware. That way the OS can check the firmware’s key against the manufacturer’s certificate.
Well, strictly speaking the OS is a bit late to check anything, and Secure Boot doesn’t work to begin with. If however the hard drive is encrypted by keys derived from the CDI, decryption is only possible when the approved firmware runs (else we get useless keys). Then the OS running at all is proof that we were running the correct firmware, and if the DICE chip is integrated into the main CPU (so MitM is not possible), its manufacturer issued firmware can refuse to give up the encryption keys if the bootloader isn’t right.
Assuming updates are even a thing… old firmware let the new firmware in, checks it is signed by the manufacturer, checks that it is a newer version, new firmware is swapped in, new public key (yeah, the CDI changed, so…) is sent to the manufacturer (on a secure channel of course), and the manufacturer issues a new certificate.
To prevent rollbacks my first thought would be a persistent increment-only hardware counter, that would be hashed together with the CDI instead of using the CDI directly. If every firmware version does this, incrementing the counter instantly change the keys of older versions. With the wrong keys they’re no longer approved, and rollback attempts are detected as soon as we check the certificate. We don’t even have to explicitly revoke old keys with that approach.
That was the first idea that popped up, we can probably do better.
The problems the TPM solve aren’t hard, they’re many.
Hardware-wise, manufacturers of current secure elements can easily add DICE capability at no loss of security. They have the hardware secret sauce to do it. Unlike Tillitis, they don’t have to use an FPGA whose only threat model was making a token effort to protect their customers’ proprietary bitstream from IP theft.
Software-wise, DICE automatically makes thing easier by (i) allowing us to only address the uses cases we care about, and (ii) addressing them separately. Even if end users are derps, the manufacturer can publish a bunch of official firmware for the most popular use cases. They’re already doing that after all.
Which is why DICE does not do this. DICE firmware does not, I repeat, does not access the root key of the device. Only the tiny bootloader does. The main firmware only get access to a derived key that is specific to it, and it alone. Compromising the root key of the device from the main firmware is flat out impossible.
Sorry for re-stating the obvious, but it’s hard to interpret what you just wrote under the assumption that you understood that, I’m not sure what’s your point here.
No, you’re just talking out of your ass here. You need to explain how data can flow from secrets to the side channels with the DICE approach, in a way that they do not with the regular approach. You also need to be aware of the actual threat model: the timing side channel is almost always relevant, but in practice easy to address — even Daniel J. Bernstein said so. Glitch attacks and power analysis are harder, but they require physical access and as such are out of many threat models. They’re important when the user is the enemy, but you know what I think of Treacherous Computing.
For instance, it’s reasonable to assume that under the normal operation of the HSM, its immediate environment is trusted enough not to purposefully inject glitches or analyse power. It’s not always a valid assumption, but not everybody is making a credit card to be plugged into many untrusted terminals.
Then there’s the question of what can be extracted at which point. The bootloader is fixed, and hashes the program and secret together. We already know of ways to mitigate the power side channel, it’s to hash the UDS in its own block, and hash the rest of the program starting on the next block. (Ideally we’d want a constant power hash, but for RAX designs, constant power addition is very expensive, so let’s settle for the mitigation instead.)
Glitching the loading of the program would just yield a different CDI, not sure we can do much there. And once the program is loaded, access to the UDS is closed off, so no amount of power analysis and glitching can shake that off (assuming adequate protection of the latch, but that should be easy, compared to protecting the integrity of the entire program’s runtime state.)
No, any form of security requires the root of trust to be out of control of the attacker. The attacker is assumed to be able to compromise the later stages of execution because the DICE model is based on inductive proofs and gives you guarantees only about the base state of a system that has unbounded states later. Each step is, effectively, promising not to do certain bad things (e.g. GRUB promises not to lie about the kernel that it has loaded) but once you get to a general purpose OS you can run arbitrary code. Once you get to this point, you lower layers are expected to lock down the system such that an attacker cannot do specified bad things. For example, the TPM may have unlocked certain keys (often implicitly by using a KDF over a secret mixed with some PCR values) but does not permit that key to be exfiltrated, so an attacker who compromises my OS cannot steal my SSH keys or disk encryption keys (which means that they can do online attacks during their compromise).
That is absolutely untrue for the hardware RoTs that I’ve worked with. They have fixed function hardware that stores keys and expose a set of operations on them (including ACLs that authorise what operations a key can be used for and what derived keys can be used for). They run the TPM stack on the programmable core, but they assume that this can be compromised. An attacker who compromises the TPM stack has very little more access than someone who compromises the first-stage boot loader on the host.
Okay, so now you need your device to be able to make network connections or you need to rely on the OS to provide that channel. Now you have a very wide attack surface. Being able to support this use case was actually one of the motivations for CHERIoT because it isn’t feasible with existing hardware within the threat models of people who rely on TPMs.
Okay, I think this is where we disagree. The primary use case for a TPM, for me, is protecting my data if my machine is stolen. There’s no point using disk encryption if an attacker also has access to the key. You seem to think that DRM (‘repeating the phrase ‘Treacherous Computing’ does not inspire confidence, you seem to use the phrase to dismiss all of the things that people have done to improve security by assuming malicious intent) is the only threat model where attackers have physical access. I think that attackers with physical access (whether that’s via an ‘evil maid’ attack or outright theft) or who have compromised my OS are the only use cases where I want a hardware root of trust. If a system isn’t robust in those two cases, then it fails my threat model. By moving access to keys and crypto algorithms out of fixed-function units and my moving more of the code on the programmable core into the TCB, you make it harder to defend against this threat model. I’m not sure what your threat model actually is, you might have a good solution for it, but it isn’t the one that I care about.
Okay.
I don’t like this induction analogy, but even then we do have a base case: the DICE bootloader that is a fixed function. The firmware on top has its own key and therefore can be integrated in the root of trust (one with actual functionality this time), and, well… just like a TPM it’s not supposed to reveal its own secrets.
If we can’t do bug-free firmware, we’re kinda doomed anyway. Speaking of which…
You feel like you’re contradicting me, but I feel like you’re making my case for me: those fixed function they wrote, they have to make sure they don’t have any bug that would expose the secrets, right? This is exactly analogous to the firmware I spoke of. Nothing stops this firmware to provide fixed functions to the untrusted host…
…unless you need the variable functions on top to be trusted anyway, kind of security by layers. We’d need something like a 2-stage execution environment, where the first can set up some functions, and the second is denied access to the CDI of the first stage, but can issue commands to it nonetheless. I guess DICE isn’t enough for that. Whether this complexity is worth the trouble is another question though.
I use a password for that. It’s easier for me to trust a secret that’s not even on my machine. (Though ideally I’d use a password and a secure element.)
Protection against theft should be straightforward. Evil Maids however are very powerful: a key logger, something that intercepts the video signals… I’m afraid those would be hard to detect, and mitigating them would pretty much require every major component of the laptop to contain a secure element so all communications between component can be encrypted and authenticated. This is no picnic.
My, if I ever come to actually fear an Evil Maid, there’s no way I’m leaving my laptop at the hotel.
Only to the extent the firmware is more likely to have bugs than fixed-function units. Keeping the firmware small and specialised improves our odds dramatically. Because of course, the attacker can’t change DICE firmware without changing the keys. If they do, the disk simply won’t decrypt. Their only chance is to exploit a bug in the firmware.
Then you probably don’t like DICE, since the entire model is based on inductive proofs of security.
I am really struggling to understand how the security claims that you’re making map back to things that the security proofs about DICE give you.
Right, and they do that by being implemented in a substrate that is amenable to formal verification, including side-channel resistance. In particular, it has no arbitrary control flow (everything is dataflow down wires, control signals are data), and it has almost no dynamic dataflow (values can flow only to places where there are wires). Doing the same thing on a general-purpose programmable core that does not provide hardware memory safety is vastly harder. It is easy to validate that keys cannot be used for the wrong purpose because those accesses have values coming from the relevant bit in the ACL and are anded with that value. There is no way of getting a key except via a wire from the register file that contains the key.
And then you decide that you need to protect two different things at the higher level from each other. And two different things at the lower level. And now you have TrustZone and two privilege modes. Only now you have shared resources between different trust domains, so you have side channels. Again, there’s a reason people stopped building hardware RoTs like this! The TPM is intentionally inflexible because as soon as you start allowing programmable functionality you need multiple trust domains, as soon as you have multiple trust domains you need both fine-grained sharing and strong isolation and those are both hard problems.
So you enter the password into your bootloader and now it’s in memory. Anyone who manages to compromise your OS once can exfiltrate it. The entire point of a security coprocessor is to ensure that keys are not accessible to someone who compromises your OS.
But, again, you haven’t given me a threat model. It seems like you’re happy with the level of security that you get from no hardware and you’re unhappy with the level of security that you get from a TPM, but somehow want something different. You’ve written a crypto library, so I presume you know what a threat model looks like: what capabilities to you assume your attacker has? What are your attackers’ goals? What tools will they be able to construct from their base capabilities towards that goal? How do you prevent them from doing so? If you start from a threat model, we can have a reasonable discussion about whether your approach addresses it, but it seems like you’re starting from a bunch of technologies that people building secure elements stopped using ages ago because they aren’t sufficient for the threat models for these devices, and saying that they’re a better way of building secure elements.
To be honest I don’t believe you. Please give me evidence.
Okay, it’s simple: you start with two things:
You load the firmware on the hardware, and this hardware+firmware pair constitutes an HSM. That’s a bit strange, but strictly speaking DICE hardware is not by itself an HSM. It only becomes so when loaded with a particular piece of firmware. (In practice the bootloader alone doesn’t really count, since it doesn’t provide any functionality users want. Users want actual cryptographic services, loading firmware is only a means to that end.) Anyway, with trusted hardware and trusted firmware, you now have a trusted HSM. Ask for its public key, and register that as “trusted”.
Now let’s say you have those 3 things:
Testing whether you can trust this hardware+firmware pair is simple:
And that’s it. Notice the similarity between that and a fixed-function HSM: it’s the same, you just need to start by loading firmware you don’t trust yet. The actual verification after that is exactly the same as for a classic HSM.
Of course this all hinges on the guarantee that changing the firmware unpredictably changes the CDI. That we trust the hardware+bootloader to provide that guarantee. If you have reasons to distrust this guarantee I’d like to know why.
Now what about Dragons?
You’re clearly assuming that firmware is not amenable to formal verification. This is blatantly false: even regular cryptographic libraries can be formally analysed, and some have been. Including side channel resistance. And since firmware runs on a specified piece of hardware, that verification is even easier.
No, you’re making a category error there. Programs (barring the self modifying relics) have a fixed control flow, just like hardware. When we talk about arbitrary control flow we don’t talk about programs having it, we’re talking about our ability to make or load arbitrary programs. Once the program is chosen the control flow is frozen.
Same thing for the dynamic flow: cryptographic libraries have almost no dynamic flow, and that’s what make them secure against the most important side channels. So while some pieces of firmware can have dynamic data flow, not all of them have, and obviously a sane cryptographic engineer would only trust the ones that has as little of it as is reasonable.
Again a category error, I believe. This is not about guaranteeing anything about arbitrary firmware, we want to guarantee that a particular piece of firmware is void of memory errors (among other errors). Hardware memory safety tricks like ASLR or executable XOR writeable help, but you can completely sidestep the problem and write your firmware in Rust.
What you call “vastly harder” is at worst a speed bump.
Most of the details of these things are protected by NDAs, but if you let me know which manufacturers you’ve talked to I can point you in the right direction in their designs, if we’ve worked with any of the same people.
That works but only because you’re sidestepping all of the hard problems. As a user, I don’t care about the code identity I care about the guarantees. That’s always the hard bit in any attestation system and if you tie software on the host to specific versions of the RoT then you end up with something very fragile. In the RoT designs I’m familiar with, this is sidestepped entirely as a problem. In a TPM, the useful guarantees I get are all about negative properties: it’s not programmable and so I know you cannot do things that are not exposed, I don’t have to prove a completeness property over some general-purpose code (ignoring firmware TPMs, which were the worst idea ever and were all broken by Spectre if not before).
Yes and no. I’ve worked with the EverCrypt team and they got further than most. When we tried to use their code in production, we discovered that their proofs of temporal safety were holding only because they never freed memory (not great for a resource-constrained environment). Those proofs also depend on compiling with CompCert and may or may not hold with other compilers. Most importantly, they hold only if no code in the system that they are linked with ever hits undefined behaviour. If, for example, you have a memory safety bug in the parser for mailbox messages from your host, none of the proof in EverCrypt hold because you have just violated one of their axioms.
With the Low* work, they are able to prove that the implementations are constant time, in conjunction with a modest microarchitectural model. They are not able to prove anything about power because most ALUs have data-dependent power consumption. The techniques used to harden against these in hardware (e.g. running identical pipelines in parallel that that XOR’d inputs so that the power is always uniform) are simply not expressive in instruction sets. The Low* based proofs also depend on their being no speculative execution (if I can train a branch predictor to go to the wrong place, it doesn’t matter that there are no timing leaks in the correct path) but that’s probably fine.
No, pretty much every program will use the branch and link register instruction. Proving anything about control flow is now a data flow problem: you have to prove that the set of values that will reach the input of that instruction is constrained. This is possible only by assuming that you have constrained memory access, so you’re returning to the ‘memory safety is an axiom’ world. Now, with CHERIoT, we can make that guarantee, but you won’t find it in any other embedded ISA.
This only helps the verification if you have strong compartmentalisation. Again, not something provided by most non-CHERIoT embedded systems.
ASLR is totally inappropriate, it barely works when you have a full 32-bit address space, you have no chance of getting useful levels of entropy doing it without an MMU. Even if it did work, it’s a probabilistic defence and so is inappropriate for building a foundation for formal verification. Immutable code is the default on embedded devices (many are Harvard architecture) but you’re still vulnerable to code reuse. You can write in Rust, but you can’t verify anything about timing in Rust because the abstract machine isn’t in Rust. Anything involving your assembly routines and anything involving I/O will be in unsafe Rust code, so there’s still scope for bugs even if you verify all of the safe Rust code (Rust verification tools are improving but they’re nowhere near the level that you’d require for a system like this).
I say vastly harder based on experiences of teams that I have worked with. They have shipped hardware RoTs and formally verified crypto implementations. The software verification effort took more people and took longer. The hardware RoT was deployed, at scale, to people with physical access some of whom had a large financial incentive to break it. It is still doing fine with no compromises. The verified crypto implementation was broken within a week of internal testing by finding bugs in the integration with the surrounding system. I consider the thing that multiple vendors have done and that has stood up to attacks in production vastly easier than the thing that hypothetically could be done but has not actually worked any time someone has tried it, but maybe that’s just me. If you want to prove me wrong and ship a formally verified firmware stack for a RoT that is resistant to timing and power side channels and glitching attacks, I’d be very happy to see it: even if I never use it, the tools that you’d have to build along the way would be game changers for the industry.
But you still haven’t told me what problem you’re trying to solve or what your threat model is.
Fuck, I forgot about that. Evidence not available, then. Fuck. Still, I care less about implementation details than I care about the user-visible interface. Do those poor sods also put these under NDA? Like, they sell a chip with an ISA, and then they tell you “this ISA is a corporate/state secret, sign the NDA please”?
I bet they do. Fuck them.
Same threats as anyone else. I’m just trying to have something simpler achieve the same goals. Because as a user, I just can’t stand these huge TPM specs. Specs that apparently had a critical vulnerability, discovered March of this year. Not in the hardware, not in the software. In the spec.
I have an hypothesis: software teams are drooling incompetents compared to hardware teams. Maybe it’s less about the actual difficulty of software, and more about how hardware teams understand the stakes of what they’re doing and are trained (and tooled) accordingly, while software teams simply don’t and aren’t. I still remember this blog post by someone who worked with hardware teams, and then noticed how software teams just didn’t know how to test their stuff.
I’m aware of at least one language that can (forgot the name, it’s designed specifically for cryptography), and I believe cryptographic libraires have been written in it. Good point about having to verify the compilers as well though. That obviously has been done, though whether it was done with the relevant verified language(s) I don’t know.
Jump and link to a hard coded constant. Unless indirect calls are involved, but that would typically mean the main program calls arbitrary code… Could be used internally of course, and we’d found we could still account for the jump and link destination… or just avoid indirect calls indirectly, if that makes verification easier.
Now if it’s something as trivial as “call foo if condition1, call bar if condition 2”, well… first of all that’s not constant time, but let’s say we’re doing signature verification and don’t care about leaking data just yet: a fixed function hardware equivalent would do exactly the same. How could it not, at some point there is stuff to do.
Which means constant power can’t be achieved in software, I’m aware. This is a definite disadvantage, and it does exclude some use cases (like credit cards). I maintain that you don’t always need to address the energy side channel. Or even electromagnetic emissions.
How as a user would you have any guarantee? There’s only one way I’m aware of, it goes in 2 steps:
That’s true of any HSM, DICE or not. And what are you on about with “tying to a specific version of the RoT”? Isn’t that what happens anyway? Doesn’t the manufacturer have to sign the entirety of the chip, firmware included, regardless of the approach taken? The only additional difficulty I see with DICE is that since changing the firmware changes the keys, updates to the firmware are harder. But that’s quite moot if the alternative is fixed functions etched in immutable hardware: can’t update those at all.
But then DICE does an advantage: someone other than the manufacturer could write the firmware and vouch for it. A downstream user (or trusted secondary shop) can then load the firmware and register its keys.
Neither is DICE firmware. A dice HSM is not programmable by default, it has a “fixed-function” firmware, identical except in implementation details to fixed-function hardware.
One could load arbitrary firmware onto their DICE hardware, but only one such firmware has been signed and can be checked for identity. It can’t be swapped for anything else. Programs aren’t general purpose, the CPU running them is. Freeze the program and you get a special purpose appliance.
Perhaps, but since fusing the TPM to the main execution chip is the only way to have actual hardware security… that can be better than a discrete chip, even one that has a level-magical hardware security.
The ISA for the programmable part is often open but that’s not the interesting bit. The interfaces to the fixed-function units and the protection models are often NDAs. Some of the security also depends on exactly what you do in your EDA tools. Apparently if you try to fab Pluton with the default Cadence config it happily optimises away a load of mitigations. RoT vendors don’t like talking about their security features in public because it paints a target on them, they’re happy to talk about them to (potential) customers.
That’s less true with things like OpenTitan or Calyptra, but I haven’t looked at either in detail. I am pushing for both to adopt the CHERIoT Ibex as the programmable component because it enables some quite interesting things. We can statically verify (in the binary) which compartments in a firmware image have access to which bits of MMIO space, so we can allow (for example) an Intel-provided compartment to have access to back-door interfaces to the CPU that let it attest to CPU state, but not have access to the key storage, and that no component except the Intel-provided compartment has access to this bit of the state, so only Intel can compromise the security of the Intel processor using the RoT.
So, to be clear, the attacker capabilities are:
The attacker’s goal is to exfiltrate a key that will allow them to produce an emulation of the device that is indistinguishable from the real device, from the perspective of software.
The requirements for software are such that I must be able to:
I probably missed something.
But it sounds like you also want to be able to run arbitrary code on the device to perform OS-specific functionality (again, this is actually one of the target use cases for CHERIoT because it’s really hard).
I don’t think we disagree that TPM is a clusterfuck, but I think it’s an interface that you can run on a secure system (we’ve actually run the TPM reference stack on a CHERIoT implementation), I just don’t think that you’re solving most of the problems that I want a RoT to solve.
Absolutely not in this case. The folks on the EverCrypt project were some of the smartest people I’ve met (the kind of people I’d trust to build the theorem prover that I depend on, not just the kind that I’d trust to use it correctly).
AMD and Intel routinely mess up security critical things.
F* / Low*, which is what EverCrypt used.
Kind of. Formally verified compilers come with a lot of caveats. CompCert, for example, does not guarantee anything if your input code contains any undefined behaviour. This is fine for F*, which generates C code from an ML dialect that guarantees no UB, but makes CompCert pretty useless for C code written by humans. Even then, it’s not completely clear how well this works because both F* and CompCert have a formal model of C semantics but it might not be the same formal model of C semantics and any mismatch can invalidate the proofs.
Formal verification for hardware also comes with some caveats. My favourite hardware security PoC was a TrustZone exploit in a verified Arm core. They ran two wires close together so that if you rapidly toggled the value in a register you’d induce a current in a wire that led to the S state bit and would let you enter S state from unprivileged code. The core was correct at the RTL layer but not correct with respect to analogue effects. RoT designs typically also include mitigations at these layers but they have to be designed based on the specific circuits and it’s really hard to put them in general-purpose cores. The threat model for an AES engine is much easier to express than the threat model for an add instruction (are either of the operands of the add secret? It depends on the surrounding code. Is it preferable for a glitch to give a lower or a higher value for the result of an add? It depends on the surrounding code).
This is why you typically put the programmable logic outside of the trust. The bit that runs the TPM stack is not part of the TCB for key confidentiality or integrity. It is responsible for some bits of PCR state manipulation and command parsing, but if you compromise it then you still can’t exfiltrate keys. You might be able to add different things to PCR state, but that’s about it.
Return instructions are also computed jumps (jump to the link register). A single stack bug can hijack control flow this way. Hence memory safety being a prerequisite for CFI.
It’s about the negative cases. It’s easy to verify that a key stored in a key register never travels somewhere else: don’t put wires anywhere other than the input to the crypto engines. You can statically enumerate all of the possible dataflow paths. It’s much harder to verify that a value stored at some location in memory is never read by anything that can lead to I/O because you have to ensure that no load instruction that leads to I/O can read that address. That’s a global alias analysis problem.
It’s a hard problem. It’s simpler if you can enforce constraints. If the TCB logic is fixed function, your proof obligations on the programmable bit are much less. Ideally, the programmable bit is in the TCB for availability and not confidentiality or integrity, so you just punt on it entirely. Now you have a much simpler problem of getting an attestation over the device, rather than the device plus a software stack. The attestation is a claim from the manufacturer that it provides some security guarantees.
That’s what I mean about the guarantees. Identity is a building block for attestation, it’s not sufficient. The important thing is that manufacturer X makes claims about software Y running on hardware Z. If these claims are untrue, you can sue them. That’s what you’re getting from attestation: accountability that you can use to build legal liability. In the worst case, you need to universally qualify these claims over all values of Y because the OS doesn’t want to carry a list of valid firmware versions. Often, you make slightly weaker claims that rely on the ability to have monotonic versions of Y (e.g. ‘I promise that this gives security guarantees as long as you’re running the latest firmware and, if there are security bugs in the firmware I will fix them within a week’), which is where you get the requirements to be able to do secure update with anti-rollback protection.
DICE (with slight tweaks) is used in most of these things already. The problem is not the firmware, it’s the space of things that a firmware image can possibly do and how you get a guarantee that your firmware is secure in the presence of all of the capabilities listed for attackers above. And it’s about ensuring that things like:
Those are the hard problems. DICE is a small part of solving them, it is not sufficient by itself.
Imagine I have one of these devices and I have an emulator for one that gives me complete visibility into all of the secret keys. How do you enable me to move the keys from firmware X on the real device to firmware X+N on the real device but not to:
DICE is an important building block for enforcing this kind of policy, but it’s nowhere near sufficient by itself, and without being able to enforce that policy you have no security for your secrets and may as well just store them in the kernel.
This gets even harder when you’re using it as part of your secure boot chain (which is what I want from a root of trust) because any change to the DICE signature for the RoT’s firmware will change the PCR values for the next steps of the boot. I can’t rely on code that loads after the RoT’s firmware to verify its attestation because it is able to tamper with any of that code, so I must be able to provide an attestation over trust properties from the RoT initialisation that I can use for the PCR values, not an attestation simply of identity.
This is why the good designs have a separate core in the same package as the main chip. If you want to see it done really well, take a look at the Xbox One. There’s not much public, but you can probably find people who have reverse engineered bits of it.
Looks like we’re converging. Thanks for the detailed reply.
Well if the NSA gets a hold of Snowden’s computer I guess that’s what we have. I do reckon under this model though that there’s no escaping the energy side channel. Which kind of means masking everything that ever accesses the secrets, and… my that’s expensive. And we can probably forget about RAX designs (I’m told masking addition is expensive), and stick to stuff like AES and SHA-3 instead.
One thing that seems impossible to mitigate though is bignum arithmetic. How can we deal with elliptic curves or RSA when everything must draw constant power? Do we switch to curves with binary fields? I’m told the security literature is less sold on their mathematical security than prime fields, but that’s the only way I can think of to avoid multiplying stuff.
Or we bite the bullet and make a general purpose CPU where every operation that is naturally constant time, is masked to be constant energy as well. Addition, multiplication, bit manipulation… all constant energy, guaranteed by the CPU. And while we’re at it stay in-order and remove speculative execution, cryptographic code is unlikely to benefit from out of order speculative cores anyway. Making a CPU like this is probably a bear, but this could have the tremendous advantage of making constant time code automatically constant energy.
I won’t dispute that is a hard problem, and I understand the appeal to limit constant energy hardware to specialised operations instead.
I’m not sold on everything here. Specifically:
If the device derives one key pair for encryption, and another key pair for signature I kind of get your wish. But if I want to perform signatures for Debian packages and personal emails, I could allow users to use different keys if I accept a domain separation string, but I can’t prevent them from using the same domain separation string for two different purposes.
Also, isn’t the guarantee that different domain separation strings causes us to use unrelated keys enough?
It’s a nice to have if firmware can’t be expected to be bug free, but I’m not entirely sure which use case requires that not only the keys are preserved, but the upgrade can happen in an hostile and offline environment. For instance, if we have encrypted secrets and need to change the firmware, we could decrypt the thing, change the firmware, then encrypt the thing back. That simple procedure only works in a trusted environment, but I’m not sure how much of a show stopper this really is.
Protection models are implementation details, so that’s not too bad. An open ISA is good. The interface to the fixed-function units however is part of the ISA: the firmware needs to call it one war or another.
I would almost forgive the vendor for not disclosing the exact algorithm used under the hood. Is it AES-GCM? ChaPoly? I don’t really care if I know where to put the plaintext, where to retrieve the ciphertext, and how big the authentication tag is. But I do need to know how to interact with the unit, so I hope this particular part is not under NDA.
Yeah, I want the end user to have a choice what firmware they use. The primary use case would be the ability to use different cryptographic primitives (the old one is broken, or the new one is faster, whatever).
I also want to enable end users to load arbitrary firmware they would then trust on first use. Yes, this means their entire computer, including the insecure parts, is assumed trusted as well. But there’s a difference between trusting your computer now, and trusting it for 2 years straight working abroad and leaving your laptop at the hotel. It’s a freedom/security trade-off mostly.
Thanks. But there’s something more fundamental about it, I think: the TPM specifically supports a lot of cryptographic primitives, and a lot of use cases. Because it kinda has to: too many users to satisfy there. Doing something similar for a single customer having a precise use case in mind would automatically divide the size of the specs by a couple orders of magnitude.
At the same time though, the generality (if not genericness) of the TPM is desirable, and DICE seems to be a nice way to keep that generality while keeping things simple. Well, simple if you punt on the firmware, at some point some one does have to write bug-free code. Hence the “which is harder, software or hardware?”. I’ve always thought the two were comparable, and bet on tailored software being much smaller than kitchen-sink hardware, and as such, proportionally easier.
It’s kind of cheating, but DICE gives you that out of the box: firmware X and X+1 are different, so they get different keys. One could load firmware X and exploit a bug to get its keys, but this wouldn’t reveal the keys of X+1.
Depend what you mean. Keys accessible from X are toast, and there’s no way the user can perform an upgrade from X to X+1 in an untrusted environment. But if user can consider themselves “safe enough” to trust the environment around the chip, they can perform the upgrade and just assume no MitM screws up the update process.
We may have a solution:
Maybe we could use fixed-function units instead, but I’m deliberately avoiding those to minimise hardware requirements. In any case, I agree DICE is not enough. Though if my special region trick works, it’s pretty close. Now, assuming no exploitable bug in the relevant firmware:
If I’m using version X and want to upgrade to version X+2, skipping X+1 because that one has a critical key extraction vulnerability, I’m probably screwed if I can’t get access to a revocation list (and most likely a reliable clock as well), I’m liable to be tricked into upgrading into X+1. The only mitigation I see here is upgrading regularly, hoping exploits for X+1 don’t have time to get in effect before I upgrade to X+2.
Oh, and one huge red flag about my approach: the CDI is transferred from one firmware to the next, so the very mechanism we use to mitigate vulnerabilities, is itself a source of vulnerabilities! I really don’t like that kind of… security/security trade-off.
Sorry if I missed something, my screen is no longer big enough to fit a useful subset of your comment and my reply on them and so you’re relying on my attention span, which is severely impacted by the nice weather.
These attacks are now feasible with hardware that costs a couple of thousand dollars. Five years ago it cost tens of thousands of dollars. Within the lifetime of the device, I expect it to be hundreds of dollars.
It’s non-trivial but there are techniques for building constant-power large multipliers. I expect to see some of these things exposed in a slightly more generic way as people start caring more about post-quantum security (for devices with a 10+ year lifetime, support for post-quantum encryption is now a requirement, but no one knows what the right algorithm is).
It’s not just that it’s hard, the problem is that it impacts power / clock frequency for everything. With a clean split, you don’t care too much about leaks from the general-purpose core because that’s outside your TCB for confidentiality and integrity and so you can make it fast / efficient and you can make the fixed-function bits fast and secure but less power efficient.
It’s mostly about defence in depth (which is a good principle for the whole system). For example, for WebAuthn, you really want to have a single secret that’s used with a KDF and some other data to generate a key that’s used for signing. You want to enforce the policy that the secret used with the KDF never leaves the device and is not used except as input to a KDF. You also want to enforce a policy on the derived keys that they also never leave the device and are used only for signing. This makes it harder for a compromised OS to leak the key (especially if you also throw in some rate limiting).
My original post had one: I am dual-booting Linux and Windows, using BitKeeper for encrypting my NTFS partition and LUKS2 for my ext4 one. Linux doesn’t trust Windows, Windows doesn’t trust Linux. With the TPM today, both can hold disk encryption keys (which can be further protected by a PIN / password) that the other can’t use. If an attacker installs malware that compromises the NT kernel and gets full access to the TPM interface, they still can’t decrypt my Linux partition.
From this use case, it follows that either Linux or Windows should be able to upgrade the firmware on the device, without destroying the utility for the other. If everything that’s using the device needs to cooperate in updates then I have a difficult operational problem and the likely outcome is people don’t update the firmware and keep running vulnerable versions.
Given that even formally verified software isn’t bug free (formal verification aims to ensure that all of your bugs exist in the spec, sometimes it just guarantees that they exist outside of your abstract machine), I think assuming that the firmware contains bugs is a safe axiom.
Probably quibbling about semantics, but to me the ISA is the set of instructions that run. Interfaces to things that run outside of the main pipeline may be architectural but they’re not part of the instruction set architecture.
Typically you don’t even get the datasheets for these things without an NDA. The algorithms used may be in marketing material (because people looking for FIPS compliance read those before bothering to sign the NDA). How keys are protected and the security model is definitely NDA’d in most cases. In a few cases because the offerings are utter crap and everyone would point and laugh, others because they’re doing quite clever things that they don’t want competitors to copy.
Maybe. On the other hand, if you get a chance to look at what Apple does in their Secure Element, you might long for the days of something as simple as a TPM. Feature creep is really easy. A couple of requests that I’ve seen for people to run on the RoT:
Once you have a secure box, everything wants to live in the secure box. See also: TrustZone.
I still don’t think that DICE gives you enough unless you can ensure that your firmware is bug free. You need some signing infrastructure, attestations, and a trust model on top. And that’s complicated.
But that’s not solving the problem. I need to be able to get the keys from firmware X+1 because otherwise a firmware upgrade locks me out of everything I’m using this device for.
That kind-of works. The problem is that you can brick your device (or, at least, lose access to all keys) by installing a buggy firmware that doesn’t allow updates. You need to build some kind of A/B model on top. Normally this is done by running firmware A, it installs firmware in slot B. Set a flag so that B boots next but A boots after that. B then boots and runs some checks. B then updates the flag so that B always boots next and deletes A.
With your model, the first time B boots, it would be able to decrypt the keys and reencrypt with its CDI. This makes me super nervous because that’s the point the firmware has access to I/O and to the keys, so I just need to find one buggy firmware that’s newer than the current one to exfiltrate all of your keys. You really need to make sure that you keep up to date with updates (a few iPhone jailbreaks have worked this way: don’t upgrade for a while, wait for a vulnerability to be found, upgrade to the vulnerable version, run exploit).
This does meet my requirement for an untrusted OS being able to do the upgrade though and it’s probably easy for the driver on the host to do a periodic version check and push out updates if there’s an old version so you need to actively cooperate with an attacker to allow the delayed-upgrade attacks.
I’m happy now that there is a process, but it’s taken a thread four times longer than your original post to get there. This is the kind of thing that keeps me coming back here, thanks for your patience!
I still think you want to do a lot more in fixed-function logic if you want something secure against someone with more than a few hundred dollars to throw at breaking it though.
Thank you for yours. I found this process in no small part thank to this discussion, and I’ve learned a few things too. I don’t think there’s much I seriously disagree with any more, so I’ll just reply with some of my thoughts.
Heat wave ongoing at home, my nights are no cooler than 28°C… and I’m kind of high on sleep deprivation. Next summer we’ll definitely install an A/C.
Makes sense. I prefer to us a slightly more expansive definition: the ISA is everything I need to know to make software for a piece of hardware. To me it’s a slightly more useful definition because it really defines the contours of what I want to know, and the limits of what I believe is acceptable for an NDA.
Fuck them I guess, then? I want full ownership of what I buy, and that includes the right to explain how to use it. Hopefully these NDAs only happen in business-to-business transactions, where those freedom considerations matter a lot less.
Okay, this is so much worse than I thought. I guess I can assume any police department or dedicated criminal have those, or will soon. Great. Now hardware security requires power side channel resistance. That sets expectations I guess. Good to know regardless.
That’s good. Though I’m not entirely sold on the utility of huge multipliers. For instance when I compare Libsodium (that uses 128-bit multipliers) with Monocypher (that stops at 64 bits), the advantage of the bigger multipliers is only about 2x. Now the actual measure is screwed up by the crazy complex out of order architecture, I don’t know how many multiplier units there are in my CPU, and haven’t looked at the assembly. Still, I suspect that roughly speaking, the speed of bignum arithmetic is roughly proportional to the length of your biggest multiplier. (Schoolbook multiplication suggests a quadratic relation instead, but bigger multipliers are most likely slower.)
It may therefore be enough to use smaller multipliers and chain them or loop with them (with a hardware control unit, microcode, or even firmware). And if we can have constant power multipliers, then constant power everything is not so far out of reach. Though again, given the consequences on hardware design, this probably means sticking to a simple and weak CPU.
Ah, that one. It slipped my mind. That looks legitimate indeed. Still, I would like to try and cop out of this one by using small firmware.
The idea is simple: what if the firmware in this particular case is used only for OS boot, and maybe there’s a standard so everybody agrees on, if not a single cipher-suite, say a very small set thereof? If the thing does nothing more than measuring code and giving the go/no go, then all you need is a KDF? That’s a couple hundred lines of code at worst, significantly smaller than even TweetNaCl. So we throw all the formal methods we can, like writing a certified F* compiler that outputs RISC-V code directly, test the hell out of this thing… and maybe we’ll never need to update it?
Perhaps I’m grasping at straws, but I did say it was a cop-out.
Bug-free firmware is (almost?) as critical as bug-free fixed functions, no doubt about that. Which is why I want to keep it as small as possible. Just please don’t destroy my dreams…
…You just destroyed my dreams. Well, we both saw that thread on no one actually wanting simplicity. No solution there, except perhaps implanting a bomb in their hearts that will explode if someone manages to exploit a bug in the wild.
To be honest, excluding cryptographic primitives and communication library, to me the maximum acceptable firmware size is around 50 lines of C code. 200 as an absolute maximum. If it has to be any higher, this seriously impacts the trust I have in it. As for the size of the cryptographic code, anything bigger than Monocypher is a bust in my opinion. We may not have the same tastes with respect to fixed-function units vs firmware, but I do agree on one thing: what we trust the keys with should be really, really, really small…
…even if in practice it won’t be.
I’m guessing that it’s easy to test that the new firmware still allows update. A/B is safer in that respect, but that’s still more stuff to add to the thing, and as always I want the bare minimum.
I’m nervous for the same reason. The iPhone jail can take hike though. I guess most users would have regular control over their machine, and can update regularly… except they can no longer do that once their computer is stolen by a determined adversary.
Hmm, so hardware security mandates that no firmware, current or future, can ever be exploited into exfiltrating the keys. So we want to update it as infrequently as possible. This means minimising the reasons for updates, so the API of the firmware has to be very stable. Which it is more likely to be if we manage to keep it small. Again.
That’s still architecture (as opposed to microarchitecture, which may change between versions or across vendors) but it’s not instruction set architecture. The difference between the two is important in a lot of cases (for example, the Arm GIC specification is architectural, but it’s not instruction set architecture).
As you say, this makes more difference on smaller in-order cores. It’s also worth noting that a few multiplication units have special cases for multiplies by zero, which means that you may find that splitting into smaller parts introduces power and timing side channels.
It’s far more important for post-quantum algorithms though. These seem to involve a lot of huge (on the order of KiBs) numbers that need multiplying so having a nicely pipelined big number multiplier can improve performance and let you do all of the power / perf optimisations that you want in your normal multiply (assuming you have one - a lot of embedded cores lack hardware multiple or divide).
Unfortunately, that’s exactly what the TPM was supposed to be (though it also needed to define the communication protocol, which is a critical part of the system). Once you’ve covered all of the use cases, I think you’ll end up with something almost as complex as the TPM spec. Actually, possibly worse because doing it now people would insist on some post-quantium signature algorithms and the ability to plug in new ones after there’s consensus on the right ones to use.
The bit I’d be most worried in is the protocol parsing and I/O code and you’d be lucky to get that down to 200 lines of code. Fortunately, verification of protocol parsing is quite easy. One of the spinoffs from EverCrypt is a thing that lets you define a binary protocol and will then generate C serialisers and deserialisers. As long as you sprinkle enough volatile in there that the compiler doesn’t introduce TOCTOU bugs, you’re probably fine.
But I think you’re coming around to part of my world view which is that the key thing that you need from the hardware is support for fine-grained compartmentalisation. If you have compartmentalisation, you can make the amount of code that has access to the keys tiny and move the rest of it out of your TCB for key confidentiality.
That’s the problem and why I consider most confidential computing things to be dual-use technologies: it’s very hard to build something that can be used for letting me run code on a computer that I’ve rented without the owner being able to corrupt or inspect it, but doesn’t allow DRM-like applications. Once you think of them as dual-use, that leads to treating them in law like other dual-use technologies and regulating the use, not the technology. I’d personally love to see fair use strengthened in statute law such that any form of DRM that prevents the end user from exercising their fair use rights is, if not corrected within a small time window, grounds for immediate revocation of copyright. If you rely on vigilante justice then you don’t get the protection of law.
You should take a look at the CHERIoT platform. I think we’re providing you with a lot of the building blocks that you need. Oh, and the first consumer hardware (hopefully shipping next year) will come with OpenTitan so you get some fixed-function crypto bits and a DICE implementation out of the box, in addition to object granularity memory safety and (up to) function-granularity compartmentalisation.
Oh, what I called ISA you call “architecture”. Makes sense. Besides, all I want is a shortcut to point to what I mean, so “architecture” it is.
Crap, I forgot about those. Well obviously if we design a CPU to use in a secure element we wouldn’t use that kind of shortcut.
I didn’t know about those. If most such algs do indeed multiply huge numbers together, huge multipliers have more value than I thought.
Okay, let’s check what Tillitis have done with their bootloader. It’s one of their biggest firmware from what I could gather. So, their protocol parser seems to be about 200 lines, give or take, though their coding style makes it artificially high. I think my coding style would squeeze it in 100. They have a 150 lines standard lib, which I don’t think I can meaningfully reduce. They have a BLAKE2s implementation for the KDF of course. Their main is fairly big, over 300 lines.
OK, maybe I was a little optimist there. I’ll need to try stuff out, maybe there’s a way to compress their code further, or at least put the common bits in a library. About that: one neat thing the TKey does, is give access to parts of the code that made the bootloader. Most notably BLAKE2s. That way programs may avoid bringing some of their own code, making not only their source code smaller, but their binary as well, and increase available RAM in the process.
The same thing can be done with the basic communication facilities. You could have a simple API that let you send and receive messages of limited size, and handle the tag & size for you. If it’s good enough everyone would use it, and the only job left is parsing what goes inside the messages. Which is easy if we’re sticking to reasonable fixed or TLV formats. Which we can: we’re controlling both sides of the channel.
I’ll need to experiment to know for sure.
That’s an excellent point, that actually influences an unrelated essay I may write soon: in the Andor StarWars series (highly recommended by the way), there’s the notion of imperial tech. Widespread, convenient, but ultimately serves the Empire before its denizens. Like that radio whose spyware got Cassian spotted at some point. An effective rebellion needs to do away with those, which is why Nemik uses an old, hard to use navigator to help with a heist: it’s free from Empire influence.
The problem however with that dichotomy is that it fails to account for dual use. There only so much time for a Cory Doctorow lecture in a StarWars series after all. But it does call into questions some of the ideas I had for our real world, and I need to think about it.
Will do.
I do indeed. Though I still have a problem with how it might complicate the architecture of that hardware. Here’s an idea I’ve just got: divide the firmware into isolated compartments. A given compartment can do 2 things:
And then there’s one “main” compartment, that can also do I/O.
How to use this? If we want to keep things easy we could put all the firmware in the main compartment. No isolation, but easier prototyping. But if we want things to be secure, then we write one compartment for what would otherwise be a fixed-function unit (one compartment for X25519, one for EdDSA…), isolate more complex protocols in their own compartments as well (the Noise XK compartment would for instance call in the X25519 compartment), and the main compartment would only do the I/O and parsing.
Is that what it’s doing? I didn’t read the design that way…
If you want to plug in arbitrary crypto algorithms, you need to run them on the programmable core, not in fixed function units. If you want to run them on the programmable core then you need it to have raw access to keys. If the programmable core has access to keys then you have no way of protecting the, from bugs in the firmware. If you have persistent keys then one buggy or malicious firmware image can leak keys stored by others.
Damn, you really did miss the central point of my entire post. Please re-read it, and tell me where you get lost, or what specific point you think is incorrect. I’ve had feedback about my article not being crystal clear so that may be on me. Unfortunately I don’t know how I can make it better yet.
In the mean time, I can answer more directly:
Correct so far.
Not quite. The programmable core does not need access to root keys. It does need access to some key, but DICE makes sure this key is independent from the root key, and unique to the particular firmware being loaded.
No, it cannot. DICE makes it flat out impossible.
Programmable firmware cannot read the root key, so it cannot leak it. It cannot read (or compute) the derived keys of other firmware, so it cannot leak those either. The only key it can leak is its own. Malicious firmware can leak their own key, but that doesn’t do anything. Buggy firmware is more problematic (their own secret is trusted until we find the bug), but then you fix it, and the new firmware automatically gets a new key.
I hope this helps clear things up, because we can’t have a meaningful discussion if you don’t understand this point.
Only if you have fixed-function hardware doing key derivation based on PCRs. And that means that you depend on fixed-function implementations of PCRs and a KDF, at an absolute minimum, and you can’t plug in arbitrary KDFs.
You seem to be currently assuming that the thing in the PCR is the entire firmware image. As other have pointed out, that complicates your update process because now you need your new firmware to export your keys, wrapped in a key that the new firmware can decrypt.
I’m not even sure how you would do that because the only key that the new firmware can trust is one that’s derived from the secret and the PCR (anything not derived from the device secret could be run on an emulator and used to leak your keys) and what you really want is something where the old version can derive a public key and the new version can derive a secret key. Again, this is something that you can do if you have asymmetric crypto in hardware (as with a typical TPM implementation), because you can allow arbitrary code on the fixed-function unit to derive a key pair from the secret and an arbitrary value, which then doesn’t grant you access to the secret key and lets you access the secret key only if you use a PCR value as the input, but now you’re relying on more key derivation in hardware.
More conventional uses of DICE use a hash of the signing key as the value embedded in the PCR, which means that two firmware images signed by the same key can have access to the same keys. But that means you need signature verification in hardware[1] and that same logic is responsible for anti-replay. Anti-replay is hard if you want to allow multiple different firmwares to be installed by different users. This is normally implemented with two things:
These are easy only with a single signing entity. If you want different signatures then you probably can’t do the fuse thing and the NVRAM value would need to be one of a bank (if you allowed a finite number of firmware sources to be protected).
You’re also pushing a lot of complexity into software here and it’s really unclear to me what your goal is.
[1] Or initial-boot firmware in ROM, which amounts to almost the same thing.
Yes! That’s how DICE works! There’s a fixed bootloader with a fixed KDF inside that does the key derivation.
And indeed I can’t plug an arbitrary KDF at this stage. But I don’t care, because what KDF is used is immaterial to my perception of the CDI. If I put the same firmware on a different HSM it will get a different random CDI, and won’t even be able to tell that v1.0.2 and v1.0.3 are using different KDFs or not.
Well… I think I do? The important part isn’t where the image is stored, it’s the fact that it is measured in its entirety.
We can’t change the firmware without changing the keys, so yes, the update process is made more complicated: any wrapped keys must be unwrapped, stored somewhere safe, and wrapped again… not sure how to best do it, short of having explicit support for this in the fixed functions (which I really want to minimise). As for the firmware keys themselves, they’ll be gone so if anything is derived from them we need to rotate them.
One thing the fixed function could do is reserve a small region of RAM to store arbitrary data to pass along the next firmware. Something like the following:
This should void the need for the asymmetric big guns, thus minimising the amount of code in the fixed function. The price we pay for that is preserving state between firmware loads, and to be honest I’m not entirely comfortable with that.
Also note that the old firmware must be able to authenticate the new firmware (most likely by checking a certificate), else I could just use the upgrade path to upload malicious firmware. Oh, and if the old firmware has a vulnerability there, we’re kinda doomed.
If the firmware is signed instead of locally measured this is no longer DICE. Though if we limit ourselves to manufacturer-provided firmware this could be a valid alternative. I do think however that arbitrary, user provided firmware, is too powerful to pass up.
I would really like to understand what problems you are solving with this, because it’s clearly targeting a different threat model and a different set of problems to a TPM. This makes it very hard for me to have a useful opinion because it definitely doesn’t solve the problems that a TPM solves, but it might solve some other interesting problems. I just don’t know what they are.
I want a couple things:
A security token, similar to YubiKey, only more flexible. But those are (i) discrete, and (ii) should be carried on my person instead of tied to my computer. Like, well… a key. Main threat model: fishing, stolen laptop (assuming the key isn’t next to it).
A security token, similar to the TPM, only simpler and more flexible. Something that’s always on my computer, that, despite being flexible to the point of allowing arbitrary code, can reasonably guarantee that its own private keys will never leak. Online attacks are something for my OS to do, I just want to make sure my keys don’t have to be rotated every time a CVE pops up.
A root of trust, that helps guarantee that the firmware & software I want to execute, is the one to execute. Something that guarantees that unwanted bootloader or kernel modifications don’t survive a reboot. The main threat model is software attacks (vulnerabilities in my programs, OS, or me clicking on the wrong thing), but I would like some resistance against physical access too though that’s impossible with discrete chips (the main execution unit and the secure element must have a secure channel between them, or they must be one and the same).
Important thing about the root of trust: I need to be able to tell the machine to trust arbitrary software. My software my choice. Letting big companies dictate what I’m allowed to execute on my machine is not a goal.
I’m going to reiterate a few of my points.
I think this is a silly thing for people to lose their minds over. Other languages ship binary components, it’s not even uncommon - pyc or .so files get shipped frequently in Python.
This has very little impact on security. Everyone is calling this “a vulnerability” and they’re being overdramatic and inaccurate. This is not a vulnerability. This changes one thing; that if you were someone who audited their crate (most of you are not) you can’t determine that the binary comes from the source code that you’ve audited. Or, well, you can - just compile the source code into a binary and use that - but you can’t do it for arbitrary binaries. Unfortunately replacing the binary is not a straightforward process, but that’s ok - go ahead and push
cargoto support managing native dependencies directly.This mostly messes up build systems that weren’t expecting this. But this is also one of the more build-system friendly methods to get the speed up. Ultimately, if serde wanted to cache the crate’s binary output, that would not play well with some build systems.
People are bringing up reproducibility. The main issue of reproducible builds is that, if they were practical, one could say “this binary came from that source”, allowing a sort of “web of trust”. Of course, then you have to trust that web of trust, it only helps catch things too late, etc. Far better approaches would be to allow for things like package signing, permissions for build scripts, etc. Ask the cargo team for those.
To be candid, this was handled very poorly. It should have been obvious that this was going to have blow back, or potentially break build systems, etc. For such a core crate I think there should have been far more care put into ensuring a safe rollout where everyone knew what to expect. I resent that a popular crate is being wielded as a weapon to push
cargointo implementing some feature, at the expense of the entire language and ecosystem - this was seriously the last thing Rust needed. As someone trying to get Rust to be a supported language at my company I find it extremely annoying that I’m likely going to have to address this at work, and it’s one more thing I’ll have to deal with when I try to get a major multi billion dollar company using Rust formally.But people are severely overreacting to what is basically a non issue.
I disagree on all counts but the fact that it was poorly handled. The popularity of the crate, and the sensitivity of the change (whether or not you consider it to be valid, a lot of people do) warranted a much more consensual approach. More to the points:
The fact that other languages do it is irrelevant.
Downloading random blobs to be executed locally is a vulnerability. Examples of binary blobs being subverted abound. At the very least it arbitrarily expands the scope of the binary providers one has to trust. Pushing the model to the extreme, one would have to trust every single crate author to provide safe binaries were they to all use binary blobs.
Speed is a straw man. We extensively use serde in our very large production codebase, and the macro generation time is a very small fraction of the entire compile time. In release mode, it’s absurdly non-existent.
Reproducibility is a poor attempt at solving a problem that should not exist to begin with. Just don’t force people to use binary blobs.
Repeating what was already said above: in the Python world, people distribute pre-compiled binary blobs in packages that include extensions written in C (or other compiled languages), because building those things yourself from source is an absolute pain (the go-to example being the extremely popular NumPy/SciPy combo, which requires something like four different compilers and some Fortran libraries in order to build from source).
Binaries in Python packages also are not (currently) byte-for-byte reproducible.
And the world has yet to end due to this.
I see your point, but here’s a counterpoint: nobody should be looking to Python as an exemplar of package management.
I’m not going to engage with the troll-ness here, but I will point out that Python should be looked to by people interested in packaging, because Python has Seen Some Shit™. If a newer language or tool is running into a problem, odds are Python has seen it before, and there’s stuff you can go look up and research and mine for ideas and solutions.
Also, Python packaging has been solid and good for years now, and most of what people complain about now is the lack of high-level project/workspace management flows, not package install/uninstall/etc.
I agree 100% that Python is an excellent example of what to avoid. I swear I am not trolling
Let me give you a counterexample: I recall a time when Ruby did not have a package manager. Bundler was developed, quickly adopted the by the community, upstreamed into the stdlib. Bundler is a good package manager and there are no competing standards.
By contrast, I do not work in Python, and I have on multiple occasions had to fix Python environments against my will. The messiness of Python’s package management has a blast radius larger than its own developer community. I have had to fix Python packaging environments more frequently than Ruby and I work in Ruby full-time.
As someone who has had to suffer that extended blast radius as well at $WORK, I do sympathize, but I will say that to me, the precompiled binaries significantly tamp it down. It’s even become a somewhat … creative … delivery vehicle for build tools like cmake and clang-format, which I’m not sure what to think about, but it’s fun not to depend on the distro to get the latest and greatest.
And still, there are quite some Python packages that are darn hard to rebuild if you don’t want to or can’t use prebuilt wheels (e.g. if you are making derivations for Python packages in Nix).
Binary wheels are exactly the problem, because 99% of the users can rely on binary wheels, there is very little incentive to make a wheel easy to build outside the specific build environment that a project/company uses.
I don’t deny that binary wheels don’t solve many problems. It is unreasonable to expect that users build, say PyTorch, on their own machine.
But having used Python and Rust professionally for years, I’d rather like Python packaging to move closer to Rust packaging than the other way around. There is just so much less friction (yeah, Rust had a clean slate).
I would wager that at least the majority, and probably the significant majority, of the perceived ease/simplicity of “Rust packaging” compared to Python comes down to exactly two things:
If and when Rust changes on those two factors, its packaging toolchain will likely become significantly more complex, by necessity.
Yeah. I agree with both points. Additionally, the Python ecosystem has a lot of legacy to deal with. But the ecosystem has made large strides.
Rust is not Python and its ecosystem has evolved different implicit contracts between upstream and downstream.
It’s far too easy to morph the skeleton of your argument into “We don’t need Rust because we have C” on a basis of “Other well-established languages do X. Therefore not improving the state of things is acceptable.”
The claim being made is that shipping binaries in a package is somehow inherently insecure, or significantly less secure, than only shipping source. The counterpoint to that claim is other package ecosystems which do exactly that without the claimed attendant security issues.
(ironically, a Python
.whlpackage – the kind that can contain binary blobs – is arguably much better security-wise than the older source-only format, because a source-only package can and indeed has to be able to run arbitrary code on the target machine at install time in order to build any extensions it relies on, while a.whlis literally just a zipped archive whose install process consists of unzipping and moving the files to their destinations, with no ability to run install-time code)It depends on who’s saying it. A lot of people (myself included) are saying it’s less auditable.
Ironically, PyPI has been introducing more trustable/auditable flows for package publishing where you can track the provenance of a given build, but people got really angry because the first integration they did was with GitHub Actions.
It’s relevant when people are calling this “disgusting” as if it’s some crazy, wild, out there thing that’s SO insane to consider.
Words have meaning! Like, NO, it is not a vulnerability. That’s just coopting a word with a negative connotation to apply it where it’s inappropriate because you don’t like the other thing. It is insane in this context to call this a vulnerability. Please stop.
What? How can it be a straw man? It was the point of this. Speed isn’t “an argument that I’ve set up to attack” ie a straw man, it is the point of this PR.
It’s not just the compilation time of the serde crate, it’s also that in debug mode the macros will run without any optimizations, making them very slow as well. Maybe that doesn’t impact you, it impacts others. Besides, if native binary support were built in this wouldn’t be an issue.
Well I can agree that reprodicibility is a bad attempt to solve this problem at least.
The idea is not crazy and has technical merit. For those calling it “disgusting”, I usually ignore emotional reactions.
Words have meanings, and in my threat model, it is a vulnerability. YMMV obviously.
Compilation speed is not the purview of the
serdecrate. When using a crate for a particular purpose, I’m not giving carte blanche to the author of that crate to alter my development process as they see fit. Is it a problem for the author? fine if they fix it locally. Did they come up with a revolutionary idea worth sharing? fine, propose it as a PR and have a discussion about it. Is it so good that it can benefit build tools ? fine share it with the cargo team. But leave others build processes alone.Not the purview of the crate author. It would be the purview of the cargo team. With a problem statement, a proposed solution, and a discussion. Even if it was a problem for everyone but me the chosen solution would still be half-baked and ill-suited.
Don’t be naive, they don’t owe you anything and don’t have to discuss anything with anyone. If they own the repo they can do whatever they feel like it. If you don’t like it you have many options, like: pin last version you like, stop using that crate (including any other that uses it), pay them do fix your problem or even fork it and do a better job. This is a problem you can solve on your own instead you’ve decided to tell the maintainer what they should do. That’s not how it works.
I believe there’s more nuance to this in general. Maintainers owe at least two things to the users:
First, truthful communication about the nature of software. You can’t say “production-ready & secure” in your Readme, if it actually is “buggy & vulnerable”. It’s ok to push arbitrary low-quality code to GitHub, it’s not to mislead users into believing it is fit for production use.
Second, if you communicate that your project is dependable, you then can not abruptly renege on that promise.
An important observation here is that, although the license say “WITHOUT WARRANTY OF ANY KIND”, that is a statement about what’s legal, not what’s ethical. Breaking the two rules above is legal, but is not ethical.
Specializing to this particular case, I do consider the situation to be a pretty significant breakage of the implicit contract. serde is not just someone’s hobby project. It is well understood, by both users and maintainers (or at least I, as a user, totally got this understanding, with no evidence to the contrary prior to this situation), that serde is one of the foundational crates of the Rust ecosystem.
If you want to hold people accountable for breaking it, make it an explicit contract.
By way of comparison, I’m a fan of baseball, but one aspect of it I can’t stand is people – often coaches and commentators of older generations – who insist the game is to be played according to certain “unwritten rules”. If you want people to play by a set of rules, write them down and enforce them. “Unwritten” rules are a mechanism for arbitrariness, and so too are “implicit contracts”.
The world runs on arbitrary mechanisms. You wouldn’t like it if we had to encode every social expectation into a legal contract.
No, they don’t. It’s their own time they are giving up for free, and complaints that they do that in the wrong way are inappropriate.
They have all the right in the world to solve compilation time problems as they see fit, without waiting for some ideal solution that the rust project may come up with, some time in the distant future.
I don’t think this is a controversial point of view, see as an example: https://github.com/serde-rs/serde/pull/2580#issuecomment-1685034248
But even if we assume we live in a world you described:
I don’t see where dtolnay stopped doing that with the serde framework. Shipping binary blobs as part of packages is nothing new. No one is saying that Python or Ruby is not production ready so please don’t try to make it look like serde stopped being production ready or that authors are not truthful in “communication about the nature of software”.
I don’t think the world is in absolutes like that. Yes, it’s their crate and their time, so they ultimately make the decisions. But if your crate is such a fundamental cornerstone of the Rust community, it is also good to take issues that other people have into account. Maybe dtolney did that and decided to do that anyway, and that is in their right. Just as other people can disagree and criticise the decision, being open source doesn’t relinquish it. Just keep the discussions technical and try to keep the drama out.
If they end up keeping the binary download and a substantial part of the community disagrees, it is probably unavoidable that
serde-derivegets forked. Personally, I think that crates that end up getting used by neatly everything should probably maintained with the Rust org itself.I don’t know why Rust seems more prone to unnecessary drama. Is it something in the personality of the people who use Rust? Is it the culture that all voices have to be heard? Is it the communication medium? GitHub certainly makes it easy for anyone to post in the issue and the drama noise quickly overwhelms the technical signal. (Sometimes I wonder if mailing-list-based projects are more immune to this kind of briggading because of the greater effort needed for a person entirely uninvolved in the project to voice their opinion.)
And how much money are you donating to the Rust org?
How is what I am donating relevant to the argument? Next step, I am going to straw man you and ask:
See, these discussion with personal attacks do not get anywhere, so let’s stop it here.
Because it’s the argument you are making. You are suggesting a group of volunteers take work over from another group of volunteers without any extra resources.
I don’t agree, there are Rust developers paid to work on Rust and their companies may be interested in doing the same.
But here you have it: I have donated to Rust in the past. So I guess I am allowed to have an opinion.
(I didn’t mention it in the earlier comment, because I still don’t think it is relevant, and I don’t want to live in a world where only people with disposable income can have an opinion.)
I don’t see the point of your comment. You said there are no absolutes, then agreed that the maintainer absolutely gets to decide what to do with their project. Your comment cancels itself out.
The commenter I reacted to said
There are many points between the absolutes: the maintainer does the work for free and criticizing them is inappropriate and the maintainer should always do what the downstream users say.
Like: it’s ok for the maintainer to make their choices and it’s fair for people to criticize the choices as long as it’s done in a fair way without personal attacks and measured.
Is this is a sign the crate or functionality within should really be part of the Rust official libraries. Official rust tooling depends on them. The community treats it like it is “official”, when nothing indicates it is.
Is this evidence that the “small stdlib” approach doesn’t really work? Or at least, only works as long as the planets align.
The size of
stdis misleading.The crates listed in https://crates.io/teams/github:rust-lang:libs are also maintained by the Rust team (or, at minimum, they’re functionally maintained by one person who has given the rest of the team the power to kick them out if they lose it) without forcing them to update in lockstep with the compiler the way
stdhas to.(Thanks to burntsushi of
regexfame for pointing that link out)No, just no. Like, seriously, stop with this. It is not a vulnerability. There is no CVE you could file for this. There is no vulnerability class you could attribute. IT is a design decision that impacts nothing about the threat model - attackers in the same position can do the same exact things (execute code on your system), the only salient different is that if you audit the source code you can not tie that audited source code to a binary generated by someone else. That is NOT a vulnerability.
FWIW I agree to an extent and I definitely think the crate author overstepped here.
If the attacker has access to the developer’s signing keys, their ability to execute code on your system without detection is mitigated by source-based distribution. Relying on non-reproducable binaries makes you much more vulnerable to such an attack. Yes, words have meanings, and “vulnerability” has a wider scope than what fits into a CVE.
a) Even if that were true it’s not a vulnerability, it’s just attack surface. This is like saying that build.rs is a vulnerability.
b) That’s not true -
build.rswould be a perfectly fine place for the attacker to put their payloadsNot this wide.
“Without an increased risk of detection” is doing some pretty heavy lifting imo
There is a definition for this word and weakness in design is included in it: https://datatracker.ietf.org/doc/html/rfc4949
The full quote: (I) A flaw or weakness in a system’s design, implementation, or operation and management that could be exploited to violate the system’s security policy. (See: harden.)
“that could be exploited to violate the system’s security policy”.
This change to use a binary artifact is not a vulnerability. It changes nothing about what the attacker can or can not exploit.
This is like saying “using jwt is a vulnerability because you could have used something else and i don’t like jwt tokens”.
Just for the reference, I fully agree with your point that characterizing this as a vulnerability is seriously wrong.
For reference, I fully agree with your point that there’s a social contract that maintainers should uphold even if they aren’t legally obligated.
Of course? Which is why I included it from the start?
I have never heard, nor would I ever accept, someone saying “The attack is literally identical except that in one case the detection method is different, therefor it is a vulnerability”.
That’s is an insane position, especially when the “different” detection method is much more impractical. Detection is a crucial element is many threat models.
OK, sure. You’re talking to a Detection Engineer/ Incident Responder who founded a company to build a SIEM lol I’m obviously well aware of how important detection is.
That doesn’t change anything - “worse auditability” is not a vulnerability. You can say it’s worse, or less safe, or whatever, I don’t care. Calling it a “vulnerability” is wrong.
So if someone pushed an update that disables one of your detection systems so that the only fallback involves requires reverse engineering, you wouldn’t say they introduced a vulnerability?
I’ve been pretty clear that I do not consider the change from serde to be a vulnerability. Because it isn’t.
And you justified it by saying a change that requires a different detection method is not a vulnerability, apparently regardless of threat model.
Yes. A change to software that changes how you would detect an attack, or even one that removes the ability to detect an attack, is not a vulnerability. If you want to make up another word for that feel free though.
I’ll give some additional context. In general the issue of “detection” is punted to a separate system. For example, I may write rules in a SIEM to detect an issue based on the logs that a system produces.
a) If my rules have bypasses it is not considered a vulnerability in the service, it would be, at most, a “vulnerability” in the rule (although that is not the nomenclature used - we call them bypasses)
b) If the service changes the format of its logs or something, breaking my detection logic, that is not considered a vulnerability of the service. It’s up to the detection system to account for that, manage the updates, etc.
c) If the service produces a stable audit log, and then it breaks that audit log, that could maybe be considered a vulnerability technically, although it would almost certainly not be called one and instead most people would call it a bypass. But yeah, that would be a vulnerability in the service, if you want to use that terminology.
None of these situations are even close to what serde is doing, which is just changing some internal behavior that inconveniences auditors. That is not a vulnerability. It’s at best close to (a) or (b), neither of which are vulnerabilities.
If you can’t accept this I’m sorry, you’re wrong and that’s it. You can be wrong if you want to be, I don’t care, but this isn’t really a matter of opinion or discussion or nuance.
edit: I’ll also note that having vulnerability encompass “a detection broken” is entirely useless. What would you do with that information? Nothing with the software itself, you’d go and change the detection. Would everyone be notified for every single change to software that might impact some arbitrary detection implemented in a totally different system? Who is this serving?
I think “in/of the service” is doing some pretty heavy lifting here.
Not really, because even if you were sayin g”but it’s sitll a vulnerability, just elsewhere”, I still justify why:
a) That’s not the case (we don’t call it a vuln), except perhaps if being pedantic about (c), which doesn’t apply because serde never made some sort of security policy around its own auditability
b) It’s definitely not what people are saying - people are saying serde is vulnerable, and I think it would be an incredible shift in goal posts to say otherwise
In no way does my point rely on where the vulnerability is, it’s just trying to give your argument the best possible interpretation by saying “maybe, if pedantic, in one specific case that odesn’t apply, you could call it a vuln in another system”.
If we are talking about a vulnerability elsewhere, why are security policies issued by serde the only relevant ones?
The one person in this thread said “downloading random blobs to be executed locally is a vulnerability.” I don’t think they were claiming that serde downloads and executes binary blobs.
I think this is a pointless conversation. We can agree to disagree, I think I’ve done more than enough to try to explain this to you and others.
I don’t know what we disagree about now. It appears that you were arguing with someone in your head.
Honestly, it feels like the same form letter as the actix-web kerfuffle, just different mad libs filled in.
unsafe. In Serde’s case, auditability. In both cases, a perceived violation of “Pick the safest default, even if that means a bit less performance.” Yes, I know integer overflow/underflow in release builds is a glaring exception.)I see a somewhat different commonality. In both cases, an individual developer, who obviously takes pride in their work, went to unusual lengths to optimize their own software on some easily measurable metric. In Actix’s case, this meant coming out on top of benchmarks. For serde, it meant reducing compile time. In both cases, the community decided that the end didn’t justify the means.
The lesson I take from this, which I will keep in mind for my own AccessKit project, is that instead of going to great lengths to move one’s own project ahead of the rest of the ecosystem, it’s better to do what we can to move the whole ecosystem forward, and accept the limitations of that approach. We may be able to get our own project where we want faster by taking the bold step alone, but in the long run, it’s better to move everyone forward together.
*nod* “Do it properly and move the whole ecosystem forward, even if it means taking longer to get there” has been a recurring theme for Rust.
In fact, it’s something I’ve contrasted with other ecosystems. For example, that’s why Rust was focusing on slowly but steadily improving LLVM-related things, while Go had to backpedal on bypassing
libSystem.dylib(the ABI stability surface for the XNU kernel used in Darwin, macOS, and iOS, equivalent tontdll.dll) in their efforts to just unilaterally rewrite or bypass anything that was a drag on their goal to break new ground for easy cross-building and easy deployment.Disagree. A 10x improvement in compile times that uses serde code is a fine tradeoff to me.
Please don’t wave this “10x” figure like a magic wand. Like in every performance-related issue, reality is much more nuanced.
If using a binary blob will improve the time of your code generation, in any reasonably-sized project that time is only a fraction of the overall compilation time, especially in release mode.
The end user should retain the ability to opt-in to at least evaluate whether or not using a pre-compiled binary is beneficial to their use case.
It is the fact that users are robbed of this ability and force-fed a questionable practice for dubious reason that is being criticized, and in my opinion, justly so.
I’m in agreement this should have been a feature flag. I’m in disagreement that “the community” has decided anything. It’s weird this picked up traction several weeks after it happened and is seemingly some dogpile issue now.
I don’t know about others, but I only learned of it now. I only pay attention to Serde changelogs when the Dependabot PRs start rolling in, and my last wave of them took my projects to 1.0.168.
(So it’s possible we’ll see another wave of outrage on September 1st, when the next wave of Dependabot PRs rolls in and the people who’ve been living in a hole see and investigate the “Experiment with precompiling the serde_derive macros to reduce build time” line in their changelogs.)
EDIT: Ouch. It looks like I also have a couple of projects where it reached 1.0.180 because I was too tired and distracted to realize that “precompile the proc macros” shouldn’t automatically be dismissed some kind of dependency sequencing hack that still ultimately builds from source. (To be fair, that does roughly describe the equivalent or better speed-up you can get by depending on
serde_derivedirectly, instead of through thederivefeature on theserdecrate as a build sequencing hack.)I think the actix-web thing is way different because it was marketing itself as “safe” and was using “unsafe” incorrectly all over the place. People found actual vulnerabilities in it and they got wontfix’d.
I think people were reacting pretty reasonably in the case of actix. I think people are totally blowing this situation out of proportion.
I understand Cargo as intentionally designed to be a from-source build system. This change subverts that.
Yeah, but if we had binary support natively we’d have way faster compile times.
For sure, though we’d lose other things.
That’s why I said it was intentional.
It will also be pretty hard to get full blown crate-level binary support. There is no stable ABI for rust code itself, there are top-level decisions which can change everything (global allocator) and thus building for even every tier 1 target of rustc itself won’t be enough (as shown by various build problems in this “experiment”).
I was low-key relieved to see that the precompiled binary is skipped for the architectures my team uses. (ARM variants)
I’m disappointed that something titled “experiment” landed “opt-out” (if you want to call it like that) in a minor patch release. And then when people have broken builds or concerns, it’s left like that until a suitable opt-in solution is found..
I’m honestly disappointed by at least two people here who are/were part of rust teams and active for a very long time. This whole situation got handled pretty poorly, compared to some other, less controversial changes in the rust compiler. (Meaning the experience exists, and the crate is relevant enough it might also be rustc itself.)
I understand the intention: Shipping an optimized binary for a core-tool nearly everyone uses, kind of like the rust-compiler installation, which ships an optimized std binary. But the execution was not the best.
Yes, it is. But I mostly ignore the nasty comments, because high-interest Github issues are increasingly like Twitter or YouTube comments - they’re full of ugly noise.
Note that it’s the policy of all dtolnay crates to only bump patch version. That is, anyhow, semver, serde, syn, thiserror, etc. do not follow semver with respect to minor versions.
This is strange, but defensible — there’s no physical difference in how Cargo treats minor vs patch versions, so an argument could be made that it’s simpler to just stick to patches (you can convey more info in minor, but, as that’s not machine checked, the value of that info is relatively minor).
It does mean though that you should always use
serde = "1.0.x"in your Cargo.toml, rather thanserde = "1", as the latter breaks minimal versions.How do security updates work if the derived secret is bound to the image?
… I guess if images were able to call into each other, then a newer image could wrap an older image. That doesn’t solve feature upgrades though. Hmm.
Updating the images changes the key, so… you need to revoke the old key and use the new one. If you need the old key for the migration process (encrypted disk, wrapped keys…), use the old image do decrypt, and encrypt again with the new image.
I reckon it’s bloody tedious.
The above was also my main issue when reading the post. I thought that having a persistent keypair that is tied to the machine is an explicit feature of a TPM. Derive subkeys all you want but that root should remain static.
So when your secret suddenly changes due to software updates and you have no way of obtaining the correct secret without running an old, now known-insecure copy of your program .. you have a problem, no?
Correct, I do have a problem. But I believe the solution is easy: control the environment so the update process itself doesn’t leak the old keys. Because the old firmware is vulnerable doesn’t mean my update program has to exploit it.
The problem doesn’t go away entirely with the classical approach: if your old firmware is vulnerable you need to assume its keys have been leaked, so you can no longer talk to your TPM over an untrusted channel to perform the update. You need the controlled environment just the same. And on top of that you need to change the possibly leaked old keys, and cross fingers that you have enough fuses left to blow.
I never saw the point of R7RS-large when SRFIs exist. It should not be an issue to decouple the language specification from the libraries.
My issue with SRFI’s has always been that, when I go to a Scheme’s website and it says It implements R5RS with SRFI 17, SRFI 91, and SRFI 26 I don’t know what that means.
But If see R6RS, or R7RS-small I have a pretty good idea what is in there. Personally I like R6RS, and Racket’s implementation the best, so that is what I use.
Some people (including myself and it would appear @Decabytes as well) find it harder to reason about and discuss ecosystems like this where there is a large universe of possible combinations. There is no single reference or document that covers what is supported. Instead there are now various opaque identifiers that must be mentally juggled and compared and remembered. (It’s great that there’s a single place to look up the meaning of each SRFI, but I don’t think that solves comprehension.)
If you are making some software, you can no longer say “works with Scheme R[number]RS implementations”, but you instead have to list out the SRFIs you use, which may or may not be supported by the user’s favoured implementation. Then you have to repeat that complexity juggling with other libraries you may also want to use.
It’s a general issue that tends to arise with any ecosystems arranged in this way. It prioritises implementer flexibility and experimentation over user comprehension of what’s supported. (Maybe that’s okay, maybe it’s not! Probably like all things … it depends on context, each person’s preference, etc.)
People have made similar complaints about the XMPP ecosystem with its XEPs, which is also an ecosystem of optional extensions.
How I feel about Haskell language extensions.
The Haskell situation is maybe a little better in that—practically speaking—there is only one Haskell compiler in widespread use. You could argue that this is a net loss for the Haskell ecosystem! But the fact that most (all?) language extensions are enabled on a per-module basis means that compatibility comes down to asking “which version of GHC are you using?” rather than needing to ask “does your compiler support TemplateHaskell? And DerivingVia? How about MultiWayIf?”
(I’ll add that the Haskell language extensions are referred to by name. If common usage in the Scheme community is to talk about “SFRI 26,” it does seem like Haskell puts less of a burden on one’s memory when talking about these things.)
There is, for example, chicken has the exact list of SRFIs it implements here https://wiki.call-cc.org/supported-standards#srfis
so if you write a program which uses SRFIs X Y and Z you just need to check if X and Y and Z are in that list. This is a very deterministic, black and white, well defined thing.
You don’t need to memorize the numbers btw. I don’t know why people think that.
It’s explicitly listed out what is is supported, it is very easy for a user to look at the list and see that the things on the list are supported.
But I also really don’t think that’s true at all. the SRFI process is to enable users to have understandable libraries of code that they can use, potentially coordinated across implementations. implementors doing exploration would not take the time to specify stable APIs like that, write documentation on them necessarily. I think you have this backwards.
Your assertion that this whole system is “very easy” has convinced me to avoid Scheme ¯_(ツ)_/¯ it doesn’t sound easy to me
What problem are you having with this system?
Cross checking several implementations against a list of specs before I start writing a program sounds complicated to me but is “very easy” for typical scheme users tells me I’m not smart enough to enjoy this language
There’s been some kind of confusion, you are now talking about a different problem.
It’s changed from checking if a set of numbers is a subset of another, to finding an intersection of multiple sets.
Great observation, it immediately brought to mind OpenGL extensions back in the day. What a nightmare.
I really enjoyed this question.
Its just a monoid in the category of endofunctors. Whats the problem?
I flagged this comment as unkind.
Alright.
Not sure about Decabytes, but for me: yes. The few times I’ve touched Scheme these numbers have been opaque and confusing.
They are all here https://srfi.schemers.org/
They are all there, but what’s there doesn’t necessarily mean the implementations are actually compliant. There’s often caveats “Our implementation just re-exports core
fooin place ofsrfi-fooand differs in semantics” — or they won’t tell you that, and it’ll be different.Ah, the joys of SRFIng.
That’s a totally different question? It doesn’t make sense to write that as a reply to my comment that provides a list of the SRFIs.
If a low quality implementation is providing an incorrect implementation of a specification then that is obviously a bug. I don’t know what that has to do with me though.
Yes I don’t have them all memorized because there are so many and what Schemes implement what varies. I have much better knowledge of revised reports because they are self contained groupings of specific functionality.
it’s precise
unclear would be stuff like “this has a bunch of list utilities and most of the file io functions you are used to from other places”
SRFIs would be better if they targeted more practical problems. There’s a fairly recent JSON SRFI, which is great. But, generally, most are experimental language features that have no business being standardized, most of the time, imho.
Then, there’s the terminal interface SRFI, which sounds great, but doesn’t have a fallback pure scheme implementation. So you’re at the will of implementations / library authors to build them out, which is non-trivial, and likely not fully cross compatible anyway.
The community is too small, and the ecosystem too fractured. :/
Not every SRFI can be implemented in pure scheme, but that’s an advantage not a disadvantage: since they can specify things that would otherwise be impossible to add as an external library. It lets you know that you need an implementation to provide you SRFI-x.
This is ironic isn’t it. For such a small community to be so fractured…
The lisp eats itself.
https://en.wikipedia.org/wiki/Ouroboros
That comment reminded me of the German guy dreaming of that and realizing it fits the data on the structure of benzene.
Why is a JSON lib part of the standardisation process anyway? I don’t get the value-add vs adding the lib to the package manager.
Probably the answer is that compatibility isn’t reliable enough for that or there is no package manager, or something like that?
SRFIs are more about an API spec, not the implementation. They commonly have a reference implementation, of course.
Package managers in scheme aren’t standardized… :) until r6rs you didn’t even have a common way to do libraries across implementations outside of
loadso… it’s all very tricky. Very very tricky. :)@river, also.
Well, not my community, but I would want a standards process to end up with me being able to use code written under the standard in any compatible implementation with ideally zero changes. This is how C and Fortran compilers work.
Going beyond C, I’d expect standardisation to establish a common system for specifying modules and packages, project dependencies and their versions and a registry of compatible packages for people to use.
If you get all of that then you should be able to switch implementations quite easily and a common registry and package format would encourage wider code reuse rather than the current fracturing that lispers complain about.
Is that just not something schemers are interested in? (Genuine question) And if they’re not, then what’s the point of the standardisation process?
It is something some of us are very interested in.
But it is also something that some implementors are explicitly against.
Which means we all end up screwed and it makes the standards useless and harms the language and community in the long term.
The goal was the get cooperation and interoperability, but it sadly didn’t end up happening.
What a shame. Thanks for clearing that up for me.
Agree completely: things like a JSON library are great to have. useful to specify as a SRFI but not part of the language itself. The whole value of scheme is to have a tiny core language that it is so flexible that you can add things like this as a library.
I feel this too. A lot of the newer SRFIs feel very “galaxy brain” slash “language design by the back door” slash “the SRFI author is probably the only one who wants this” but are unlikely to positively impact my day-to-day life writing Scheme programs tbh
In general I feel that there are many very smart people churning out API designs that I don’t actually like nor want to use. Maybe I’m not smart enough to appreciate them. If so that’s OK. Aesthetically, many of the designs feel very “R6RS / abstract data types / denotational semantics” focused. Which is fine I guess, but I don’t personally enjoy using APIs designed that way very much, nor do I think they’re going to “make
fetchScheme happen” for the modal industry programmer anywayultimately folks are free to do whatever they want with their free time so I’m not mad about it, I’m happy to just keep plugging along using (mostly unchanging) older R5RS implementations and porting code to my own toy module system, etc and relying on portable code as much as possible
FWIW I thought R6RS was “fine” until they broke all my code using DEFINE-RECORD-TYPE because something something PL nerds denotational semantics etc. I have appreciated that the R7RS-small work I’m aware of thus far doesn’t break my code in the same way R6RS did
I believe it was R6RS that included the first operational semantics for a standard Scheme. Previously R4RS had included a denotational semantics, which R5RS had left unchanged despite there being changes in the text that required a change to the semantics.
In neither R4-, R5- nor R6RS did one need to read the semantics to make effective use of the language.
FWIW, I agree. The newer SRFIs seem very much aimed at comprehensiveness instead of ergonomics. If you look at some of the older SRFIs they seem a lot more focused and minimal.
denotational semantics is not the enemy here.
many of us were very unhappy with R6RS.
On one hand this is technically interesting. On the other hand, it’s definitely against the Terms of Service (without permission).
On the third hand, nobody cares about the terms of service. I’m sure by using an adblocker online I violate half the terms of service (that I don’t even see) for websites I visit. They want it one way. But it’s the other way.
How do you feel about GPL license violations? I mean open source licenses are roughly equivalent to terms of service.
I dislike them, because good things are good and bad things are bad.
It’s a bit rich for Google to complain about bot scraping.
Does a TOS apply if I never accepted it?
It’s factual information. Do with it what you will.
I feel like the algorithms list is a bit much. Many of them look like interview question knowledge, not programming knowledge
I recognize many from college CS, and then never used them or saw them in code in the last 20 years
e.g. Floyd-Warshall, Bellman-Ford (I’d be very interested if someone can point me to open source code that uses these, and things like it)
I happen to know that Python’s
substr in suses something like Boyer-Moore or Knuth-Morris Pratt, but honestly I don’t remember which, and it doesn’t matter. (I didn’t know this for years)Ditto Aho-Corasick – kinda fun, but I doubt most people need to know it. It’s useful if you’re writing certain kinds of libraries, which almost nobody is
I remember an accusation on a Google mailing list that dynamic programming interview questions (dynamic programming being mentioned in the post) are basically a shibboleth for “did you study CS at a certain set of schools” . Then there was an argument about dynamic programming vs. memoization and people couldn’t agree on the difference (indicating that most people don’t really use it, or remember what it is)
For most programmers, I would lean toward something more like what Rich Hickey says – just learn to use maps and vectors, and learn to divide programs into functions. You can get really far with appropriate choices there. It can also be done surprisingly badly (if you really read people’s code who are just starting)
And also learn how to use very basic static type relationships to express the domain, what you intend (i.e. learn both dynamic and static languages).
I lean more towards this advice: http://users.ece.utexas.edu/~adnan/pike.html
Also the list is too weighted towards serial algorithms (like college courses are). There are no parallel algorithms or data structures, and these days parallelism and concurrency are what will get you real speedups – e.g. learning how to either share or copy data effectively
I will say that compilers have many data structures that aren’t used in “every day” code, but most people aren’t writing compilers.
I agree with learning to think with invariants though – often if I ask people explain WHY the code is correct, they don’t have an answer … And when they need to explain, they stumble across a bug. Every time I write code I think about the invariants of data structures and write comments
I’m old fashioned and use classes and methods – they are great for maintaining invariants
Too many thoughts for this narrow text area!
https://matklad.github.io/2023/08/13/role-of-algorithms.html
Also, this goes without saying in my head, but maybe not entirely obvious:
The question of whether you should know algorithms is unrelated to the question of whether you should ask algorithms question on the interview
There’s perhaps a quantum physics joke here: trying to measure knowledge sadly destroys the system.
Yes thanks for acknowledging that. It doesn’t strictly relate to your post, but these days people get bashed over the head with “algorithms” stuff due to needing to interview… So I just offer a balancing opinion on whether you need that to write good software.
I guess the short answer is that math is useful, and I’ve worked with people don’t know what they don’t know (CS knowledge), but also I think it’s a relatively small part of writing good software
Systems software is an exception, and you’re clearly deep in that space, and I found many things on the blog useful (including in the update!)
Hm very nice post off the cuff! I will say one bad thing and then a few good things …
Picking on this one part – You’re living in a bubble :) Everyone doesn’t know this!
Most programmers don’t even know what a “regular language” vs. “regex” is, the former being the thing that’s equivalent to a NFA/DFA. I’ve made the point on the Oil blog that it’s actually good to know this from an engineering perspective, and it doesn’t really “land” for most readers.
But actually if you ask me “why”, right now I don’t remember the answer. I know I wrote a huge shell lexer that depends on this fact being true
I vaguely remember the shape of Thompson’s construction, but don’t know if that counts. I know I could find a book that tells me why, but that is very low on my priority list – it doesn’t help get software done.
(And I know there is the derivatives method, one of the few math-y things I’ve looked into in the last 5 years, but never implemented it, or Thompson’s construction)
Like I said I’m just offering some different perspectives.
As a person who used to do more math, I would have thought this advice is more actionable - https://missing.csail.mit.edu/ – i.e. learn how to use Unix and version control. Learning all the practical stuff takes longer than learning algorithms IME. (I think Julia Evans posts are popular because they have many good practical tips – the “tacit knowledge” as you say – that is where the real difficulty in making software is, IMO)
And the other contrasting opinion I would give is basically along the lines of “learning the basics”.
There is the famous quote “I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times”
I would apply that to programming … IME, the most effective people are not the ones who know a bazillion different techniques and tools and languages, but have really just mastered the basics of using functions / data / classes / whatever to model problems, AND mastered a single language (in addition to knowing others).
I guess that’s just another way of saying “tacit knowledge”
This part is interesting – haven’t heard that, but I just resized Oils pool allocators, and yeah it seems the fixed headers throw a wrench in things.
But I would say these are very “advanced” topics …
I think your post is probably good for a lobste.rs systems programming audience – if people are asking how to get into systems programming (e.g. writing a database! ). But I think there should be some balance for “general” programming.
Most systems, and most programming jobs, are glue – because glue is geometrically sized :)
And it’s actually quite difficult to do glue well (for example it often involves concurrency and parallelism).
(When I say glue I don’t mean it in a pejorative sense – I mean literally that you’re not writing the allocators and data structures in your kernel and in your VM and your server – you are reusing them in a coarse-grained fashion.)
Yes, of course! As with almost anything else on the blog, the target audience is an earlier version of me!
I’m struck by the fantasy trope of the young hero who seeks the counsel of the wise man. The wise man guides the hero, often only having some knowledge of the McGuffin at hand, but ultimately it’s up to the dedicated and single minded hero to actually get things done.
As someone who likes to collect lore, I wonder if I’ll never be a hero.
:)
Finally a mnemonic for how to spell Dijkstra’s name.
It’s phonetic, just remember that the j is pronounced like a y in English. A lot of names become easier once you understand the phonetic rules of their source language (for example, I couldn’t spell Polish names until someone told me that cz in Polish is like ch in English and then ones that I’d struggled with were just obvious phonetic spellings).
That only helps if you’ve heard it spoken 😉 I don’t think I’ve ever heard a native Dutch speaker pronounce his name.
BTW isn’t IJ considered a ligature in Dutch? That’s another rule, like “i before e except after c” that at least narrows down the search space.
The Dutch vowel ij is pronounced more like the name of the letter a than the name of the letter i.
I’d say with parallel algorithms, the gap between “what they teach you” and “what you actually need” is even wider. In practice, what you need, as jamii memorably put it, half-assed Amdahl’s law — either the problem is embarrassingly parallel, or you make it someone else’s problem. This is not to sniff at parallel programming — recognizing and making something embarrassingly parallel is the core architecture skill, and that’s why I tire everyone with my constant hammering about modules and header files :-)
Distinction between work depth vs work width is also very important, in case you want to build a new editor and compete with 30 years of Emacs packages. Useful to know merge sort here, as that’s a model problem where you can trade width for depth.
But I am actually curious — how one actually learns the skill of lock-free programming? I understand the theory and C memory model OK, but I don’t have a skill to sit and code up a Michael-Scott queue, or a concurrent hash table (which is actually concurrent, rather than just lock-sharded). Is there fantastic learning resource for these (including implement things like hazard pointers or epoch based memory reclamation)? What stops me from just trying to code these things myself is that I have know idea how to check that my code confirms to the memory model (as opposed to just working on my particular CPU with my particular compiler).
Yeah I’d say I would advocate more of the mentorship model and learning from production codebases, rather than getting overly deep into exercises and books. Again, it’s obviously a balance – it’s VERY possible and common to be understudied, to over-hack, to fail to connect to textbook ideas, to re-invent poorly. Probably more common!
Programming changes very fast, and parallel and concurrent programming are prime examples of things that aren’t textbook knowledge. Approximately zero of your professors may have worked on parallel or concurrent systems under load.
I think people mostly learn from techniques that they see work, and from people who they notice are effective.
I don’t claim to know a ton about parallel or concurrent programming, but when I worked at Google I basically followed pointers from source code and design docs, and eventually made my way back to literature.
I’ve mentioned SEDA by Matt Welsh, that was a big one - http://matt-welsh.blogspot.com/2010/07/retrospective-on-seda.html (sometimes it’s not the solution itself, but how people describe it, and the related work, that clicks)
Everything of this type that I learned was on the job, I don’t think schools or text books will tech you it - https://news.ycombinator.com/item?id=37045329
I think most people just follow the source code, and that kinda works, though I have more curiousity about the fundamental ideas, finding myself back at the literature. But I would say there are A LOT of bad ideas that have been written down, and a lot of useless ideas, including in textbooks (even Dragon Book I’d argue), so I want to start with the problem first.
Another example is Zephyr ASDL – I noticed that CPython happens to work quite well – how do they represent their code? I could have looked at textbooks, but it was better to start from a working codebase with proven utility.
It of course helps to have people explicitly write about it – https://eli.thegreenplace.net/2014/06/04/using-asdl-to-describe-asts-in-compilers But that’s oral knowledge, not textbook knowledge. (Notably Bendersky is a Python core dev – i.e. a first person source! )
So I want to start with the problem context – what’s known to work, and what’s better about it than previous techniques? Starting from “lock free programming” seems like starting from a solution rather than a problem. (Are there other techniques that could solve the problem?)
I know little about that area, but I guess would try to pick up tacit knowledge from people who have solved problems with those techniques. (And only after that, make my back to textbooks and literature)
As mentioned elsewhere things like Project Euler and Advent of Code don’t appeal to me for some reason, although certainly it seems lots of people have gotten value out of them
I’ve thought about this for years now, and I can’t claim to have any solutions yet, just… a bunch of other problems :-). When someone says “you should know algorithms”, it doesn’t always mean the same thing. That phrase actually has three related meanings which frequently get conflated:
\1. You should have an understanding of how we study algorithms: how we quantify if an algorithm is fast or slow, how we quantify how much memory it uses, how these abstract concepts map to real-world hardware, how to formulate a quantitative model of a real-world problem in order to derive an efficient solution to it. These are absolutely essential, they are what programming essentially amounts to. Unfortunately the latter two are sorely missing in some undergrad CS curricula, especially the one about real-world hardware. I’m with you on things like the list beeing too weighted towards serial algorithms running on PDP-11-era computers. They are also difficult to evaluate in interviews, so many organizations default to evaluating the first two, and not directly, but by proxy.
\2. You should have a broad knowledge of relevant algorithms, know their main theoretical and implementation trade-offs, and possess a working knowledge of algorithms relevant to your particular field. This is also absolutely essential, although I think it’s a skill set that’s very frequently misevaluated in interviews.
I also think this is the least tractable problem; the one other field of engineering I’m familiar with (EE) has it, too. Our field does makes it worse than it needs to be through poor hiring practices, but I don’t know if it’s a problem you can ever escape fully.
\3. You should have a detailed knowledge of your interviewer’s favourite programming puzzle. Since you’ll likely go to a lot of interviews before getting to that stage in your career where an interview is a nice game of golf and slowly-sipped after-game drinks, that actually translates to “you should have a detailed encyclopedic knowledge of programming puzzles”, as you’ll meet a lot of interviewers before they stop asking stupid questions like how can you weigh one ball in only three attempts if all you have are thirteen balls, a balance scale, and an evil master who will murder you and your family if you as much as suggest that this is entirely the wrong way to go about weighing things and ask for a proper scale, that you just eyeball it because if it were important to have an exact number you’d have a proper scale, or that you wait ten minutes until you go to the mall across the street to buy a five-dollar travel scale. This is pretty much the single biggest reason why things like Project Euler proliferate – there are enough self-absorbed people in our field that what should’ve been a bunch of fun mathematical pastimes are make it or break it questions that young graduates agonize over for days.
I don’t think the things on the list are the problem per se. Some of them are probably “too much” but the author obviously tried to err on the side of inclusiveness here, and it’s not a bad thing. The real problem is that evaluating many of the things under #2 and #3 is hard, and when you have to hire in numbers, you have to default to things that are easy to check. Since you need them to get a job, they get conflated with knowledge you need to do the job (or cynically rejected as nerd shit – an anti-academic attitude that’s entirely unproductive for our field, sadly).
And it has a bunch of pitfalls. For instance, you often end up evaluating the wrong things, through the wrong means. IMHO if I’m trying to get an idea about #2 above and someone can produce a working implementation of the most trivial substring search algorithm, explain (formally, not intuitively) what’s wrong with it, and name better algorithms, that’s enough for me (enough as far as substring matching goes, I’m obviously going to ask about a lot of other things). But that still falls short of actually checking they can implement an algorithm significantly more advanced than binary search. And besides, everyone, even hiring managers, have to cover their asses, so you still have people perform magic algorithm tricks under pressure.
Or, like any test based on producing things through an inscrutable mental process, these skills are extraordinarily easy to fake, especially in the age of books like Cracking the Coding Interview. Interviewing prospective interns has been a perspective-altering experience for me, I can’t tell you how many kids I’ve seen producing written solutions to pretty tough problems by rote, despite lacking any meaningful programming ability, and I mean working solutions, the kind that would get them through not just a phone-based pre-screening, but also through an interview with far too many standard questions for how little time the interviewers have. The sad thing is many of them would’ve actually been really good, either at programming or at something else, they were really bright kids. They just went the wrong way about learning how to program – although they definitely went the right way about landing a well-paying job.
Yeah I agree, the context is important, and it’s very hard to generalize. I think I’m soured by the interview context too, but the author wasn’t motivated by that.
When people ask for advice, sometimes they want to do the same things that you are doing, and the advice will apply … sometimes not, which is OK.
I guess I’m advocating a little more mentorship / problem context / building, rather than studying things in isolation. (I’m personally not a fan of Project Euler. Also I could never get into Advent of Code; so many people seem to love it though.)
One thing I will agree with is that practicing writing short, correct code is good. I’m pretty sure I learned that from SICP as a freshman, and then I haven’t used Lisp much again after that.
But that changed my opinion of how short code should be. A common problem I see with beginner code is that it’s long and full of cases that don’t need to be there. The author can’t explain why it’s correct in all cases. They just have some experience about the cases they happened to run.
Algorithms could help with that, but I kind of view it as the “practice one kick 10,000 times” idiom … i.e. doing really basic things with dicts / arrays / trees / stacks / functions / classes is years of learning, and a lot of tacit knowledge that’s not in books.
Well said.
Do you have a reference for that Hickey “quote”?
Sorry I don’t off-hand, but I think it’s basically a consequence of hammering home “functions and data” as your default tools to solve programming problems. With maps/vectors/lists being the default kinds of data. I think the advice more or less generalizes to Python, JS, Go, etc. – with the significant complication of mutability (which is useful)
Big list here! Wow I didn’t realize there were so many - https://github.com/tallesl/Rich-Hickey-fanclub
If you can’t find the Hickey source, Kernighan and Pike say the same in The Practice of Programming.
The only thing I keep using is Bresenham’s line algorithm, although I always end up using it to size rows/columns rather than draw lines (as in, instead of moving down/right a pixel, you start allocating the next column)
People are wonderfully diverse. What appeals to some, won’t convince many others. I happen to agree with most of OP’s opinions.
I haven’t used many of the data structures and algorithms they mention in production either. But that’s besides the point. The point (as I understood it) is that knowing (about) those algorithms will make you a better programmer. Similarly to how you can do calculus without understanding (or even knowing about) the Intermediate Value Theorem, you can write programs without knowing about most of those algorithms or data structures. They’ll just be mediocre, and (often) less than ideal programs. Learningn to use vectors and dictionaries is fine and well. After all, you can drive nails through a wall with a screwdriver if you try hard enough.
Re. silo sites, there are some really tricky problems other than software freedom which need solving in that space, mostly involving privacy and security. If anyone can duplicate the important results of the network effect - contacts and private messages - nobody has any privacy anymore, and everything turns into Twitter. I don’t think anyone has solved this yet.
Re. miniaturization, I don’t understand the complaint. A phone is smaller than a GPU these days, and you can’t exactly repair a GPU. Also, there are repairable phones out there, but naturally they come with their own issues, like needing some excellent engineering to not fall apart or get gunk inside the way phones are normally used.
Re. streaming, I don’t think the public ever thought copyright was not legitimate. Sure, the US copyright time span is utterly unreasonable, but removing copyright completely is just reactionary, not a sensible option, if we still want creators to create for a living.
To be absolutely clear, I’m a big fan of FOSS, and publish my own stuff as GPL/AGPL/CC.
I think the point is that phones are harder to repair and harder to write free software for than desktop computers.
That statement accepts the very reactionary premise that the only way people can make a living is if the market lets them. The business model that currently pays creators also exacerbates inequality by making access to culture depend on income (a point the author makes in their linked essay on copyright).
Why would that be? There are plenty of people writing free software for phones, and it’s not exactly a common complaint.
As for copyright, what’s the alternative? Nobody gets paid to produce culture? People pay whatever they want (which would be zero for the vast majority)? You can’t just tell people to abandon the current system without proposing an alternative, and some damn solid evidence that it would actually improve things.
Copyright is a weird thing. There are two activities that are required for a creative work to have societal impact (positive or negative):
Copyright builds economic models where you will do the first task for free and then be compensated for the second. It works by enforcing a monopoly on the second task. This is fragile because the second task is incredibly hard to prevent other people from doing. Duplicating an existing work is intrinsically easier than creating a new one.
Open Source (as opposed to Free Software) was, in a large part, a recognition of this economic reality. The economic idea behind Open Source was that copying software was trivial and so charging for it made no sense, but creating software was hard and that’s what people should be paid for.
The strange thing, to me, is that this is actually how things work at the start of the supply chain in a lot of creative industries. If you want to make a TV show, you create a pilot for free and then, if people like it, they’ll pay you to produce a full season. If they like that, they’ll pay you to produce the next one. Similarly, if a company needs a piece of software for internal use, they’ll pay engineers to create it.
Copyright doesn’t enable creators, it enables arbitrage. It enables people that sit between creators and consumers to profit.
This is FUD by the open source movement, trying to discredit free software as “non-commercial”. Open source vs. free software is orthogonal to whether something is commercial.
There are lots of platforms where people can sell their creations with a reasonably small (depending on who you ask, of course) portion of the sale going to the arbiters (Etsy, Steam, Bandcamp, etc.). You can’t expect (certainly not in the foreseeable future) small creators to set up their own sales portal, nor consumers to discover such portals in sufficient number. Conversely, you can’t expect professional sales portals to take anything less than what the market can bear. Of course, we’re not in an ideal economy, but you can see how there’s heaps of opportunity for differentiation and competition in such a market.
No it isn’t. The Free Software movement was primarily about user rights, with economic issues taking a secondary place. The Open Source movement was about economics, with user rights being a secondary concern.
As for copyright, the alternative is removing legal protections from things that aren’t property: ideas, thoughts, “concepts”, words in order, numbers; these aren’t things, they’re formulations, and it’s silly anyone ever enforced legal protections on formulations. People like rewarding creatives. You would still get paid for your work either by selling physical media, or selling digital media online - dismantling copyright is about freeing formulations (or “abstractions”) from illogical property rights-based burdens, not dismantling the economy.
Do you know how much the median open source contributor earns from their contributions, even when asking for money? Exactly zero, because only projects providing literal billions of value (like cURL and SSH) get a pittance, and even then only from the richest companies on the planet, not from their millions of users.
I’ll reiterate that I release a bunch of stuff under GPL/CC licenses, but that is my choice. Others should be free to charge for the fruits of their labour.
(As a copyleft proponent, I do not appreciate having to run secret code to do anything. But so far nobody has come up with a solution which enables people to choose to charge for their digital output, while protecting the privacy of the consumer.)
Perhaps the silver lining in all of this is that if we ever solve this conundrum for software, it’ll also be a way out of the iron grip (be it copyright lobby, DRM or otherwise) of content suppliers.
OK, but I’m not wrong. Look at Ko-Fi and Patreon: People like rewarding creatives. Programmers apparently happen to be the worst at that. That’s not an excuse to apply property rights protections to something that isn’t physical. People can choose to charge for their digital output. It’s the buyer’s right to share that with whoever they like, authorised or otherwise - the consequences of trying to suppress that right have been detrimental to the proliferation of ideas, solutions and productive use of people’s time.
How many creators actually earn a living from that? I’d bet less that 1%. It’s only the really famous creators who are able to scrape together a fraction of what their content is really worth. And it’s extremely skewed towards the people who relentlessly self-promote, not giving a smidgen of a chance to anyone who’s an introvert, or just quietly contributing a large fraction of their time.
How many people actually make a living from having a publisher pick up their books or a record label signing their bands? How many people make a living as actors by being working on TV shows or films (some of the things from the latest strikes have shown that even household names weren’t making enough money to live on).
Going by % isn’t helpful because of the different amounts of work and effort people put into this sort of thing, and so is assuming anyone’s content is worth anything. If you can’t scrape together 1,000 true fans (or enough fans to pay yer bills), you’d need another job with or without copyright.
The present system applies violence to kill nascent competition. So instead of experiments and solid evidence, we get subterfuge and revolution.
We didn’t learn from the 1900s.
I don’t understand what you’re saying. Who is competing with whom? Certainly pirates aren’t “competing” in any meaningful way with artists. And what experiments, subterfuge and revolution are you talking about?
Alternative systems that compete with copyright.
I mean, it is. You can’t install a Linux distro on your phone and have all the functionality defined by free software without sacrificing functionality. One reason is probably that the hardware interfaces aren’t as stable or consistent as in desktop computers. I think there’s also an issue with wireless modem software requiring FCC approval in the US.
Abandoning the current system would reduce inequality in the domain of access to culture. That alone is a strong reason for it, and artists would still get paid for things that are not easily copied, such as live performance and physical art. For an alternative way for artists to get paid, you could look at Federal Project Number One.
Russ usually blogs about something general before making a specific suggestion for the Go project, e.g The Magic of Sampling was a prelude to the Go telemetry saga and Version SAT was a prelude to the Go modules saga. My guess is this is a prelude to another whack at the iterator discussion.
It’s obvious, now, in retrospect, that this is the case, but until you said it I had never considered this! I guess we’re getting Russ’s version of iterators.
Yeah these two posts by Bob Nystrom use the same tree iteration example:
https://journal.stuffwithstuff.com/2013/01/13/iteration-inside-and-out/
https://journal.stuffwithstuff.com/2013/02/24/iteration-inside-and-out-part-2/
Russ’s Go post talks about comparing two trees. The Nystrom post talks about interleaving two trees with their iterators.
The terminology is a little weird though. Nystrom calls them “internal” and “external”.
Russ calls it “control flow as code” vs “control flow as data”.
I find it more intuitive to use “pull vs. push” for the two styles:
Though there are even more than 2 styles depending on the language … would be interesting to write them all out.
Where does a generator coroutine fit in those categories?
A coroutine generator is a pull iterator.
In the first round of Go iterator proposals, it was already suggested that there could be an optimization to specialize a goroutine as a coroutine when it’s just used for iteration, therefore bypassing the scheduler tax.
Anything with “coroutine” (or even “thread”) in the name is intended to allow you to use the easier “pull” style, at some cost in the language runtime or compiler.
It lets you avoid the explicit state variables.
Confusingly, Python has at least 2 different types of coroutine and ALSO async I/O (and also OS threads!). The first is generators i.e.
yield(which I assume you mean), and there is also a style of doingx = yield.Historically I think of it as:
So it is kind of a mess because the way you do this type of concurrency can be very language-specific, whereas normal
forloops andifstatements work the same in every language.At some point I would like to write out at least 2 examples in all these styles (been thinking about that for a few years since I started working with
re2c, which has code gen options for push and pull.) But yeah it is a bunch of work!Note to myself: ChatGPT and Bard produce buggy solutions for this query:
😂
I’m slightly surprised you got that – I got these wrong solutions, at least in the spirit of the question:
Bard excerpt:
wrong because it’s treating the regex chars as plain chars.
ChatGPT excerpt:
Uhh it’s handling
\\twiceIt’s extremely annoying to have code that “looks kinda correct” but is totally wrong. They are very good at convincing people of that
I’m convinced that most people using AI coding assistants are not testing their code …
Here’s what claude gave me:
With the following prompts (expert persona, chain of thought):
And.. testing it:
Wrong because the regex grew an extra level of escaping e.g. for
\], and that leaked into the imperative version too …Fortran is good enough (check numpy/scipy).
There’s a whole world of CPU bound tasks that aren’t numerical programming.
Example in the article was fibonacci number…
I think code maintainability and security is important. How many people can contribute to Fortan based code? P.S the popular cryptography Python package is (partially) moving to Rust :)
Hopefully not many, I like my paycheck.
You should sabotage PyO3/Rust then :P
They are sabotaging themselfs, no need for me =)