license all content you transmit through Firefox to Mozilla (“When you upload or input information through Firefox, you hereby grant us a nonexclusive, royalty-free, worldwide license to use that information to help you navigate, experience, and interact with online content as you indicate with your use of Firefox. -https://www.mozilla.org/en-US/about/legal/terms/firefox)
allow Mozilla to both sell your private information
If you’re already using Firefox, I can confirm that porting your profile over to Librewolf (https://librewolf.net) is relatively painless, and the only issues you’ll encounter are around having the resist fingerprinting setting turned on by default (which you can choose to just disable if you don’t like the trade-offs). I resumed using Firefox in 2016 and just switched away upon this shift in policy, and I do so sadly and begrudgingly, but you’d be crazy to allow Mozilla to cross these lines without switching away.
If you’re a macOS + Littlesnitch user, I can also recommend setting Librewolf to not allow communication to any Mozilla domain other than addons.mozilla.org, just in case.
👋 I respect your opinion and LibreWolf is a fine choice; however, it shares the same problem that all “forks” have and that I thought I made clear in the article…
Developing Firefox costs half a billion per year. There’s overhead in there for sure, but you couldn’t bring that down to something more manageable, like 100 million per year, IMO, without making it completely uncompetitive to Chrome, whose estimate cost exceeds 1 billion per year. The harsh reality is that you’re still using Mozilla’s work and if Mozilla goes under, LibreWolf simply ceases to exist because it’s essentially Firefox + settings. So you’re not really sticking it to the man as much as you’d like.
There are 3 major browser engines left (minus the experiments still in development that nobody uses). All 3 browser engines are, in fact, funded by Google’s Ads and have been for almost the past 2 decades. And any of the forks would become unviable without Apple’s, Google’s or Mozilla’s hard work, which is the reality we are in.
Not complaining much, but I did mention the recent controversy you’re referring to and would’ve preferred comments on what I wrote, on my reasoning, not on the article’s title.
I do what I can and no more, which used to mean occasionally being a Firefox advocate when I could, giving Mozilla as much benefit of the doubt as I could muster, paying for an MDN subscription, and sending some money their way when possible. Now it means temporarily switching to Librewolf, fully acknowledging how unsustainable that is, and waiting for a more sustainable option to come along.
I don’t disagree with the economic realities you mentioned and I don’t think any argument you made is bad or wrong. I’m just coming to a different conclusion: If Firefox can’t take hundreds of millions of dollars from Google every year and turn that into a privacy respecting browser that doesn’t sell my data and doesn’t prohibit me from visiting whatever website I want, then what are we even doing here? I’m sick of this barely lesser of two evils shit. Burn it to the fucking ground.
I think “barely lesser of two evils” is just way off the scale, and I can’t help but feel that it is way over-dramatized.
Also, what about the consequences of having a chrome-only web? Many websites are already “Hyrum’s lawed” to being usable only in Chrome, developers only test for Chrome, the speed of development is basically impossible to follow as is.
Firefox is basically the only thing preventing the most universal platform from becoming a Google-product.
Well there’s one other: Apple. Their hesitance to allow non-Safari browsers on iOS is a bigger bulwark against a Chrome-only web than Firefox at this point IMO.
I’m a bit afraid that the EU is in the process of breaking that down though. If proper Chrome comes over to iOS and it becomes easy to install, I’m certain that Google will start their push to move iOS users over.
I know it’s not exactly the same but Safari is also in the WebKit family and Safari is nether open source nor cross platform nor anywhere close to Firefox in many technical aspects (such as by far having the most functional and sane developer tools of any browser it there).
Pretty much the same here: I used to use Firefox, I have influenced some people in the past to at least give Firefox a shot, some people ended up moving to it from Chrome based on my recommendations. But Mozilla insists on breaking trust roughly every year, so when the ToS came around, there was very little goodwill left and I have permanently switched to LibreWolf.
Using a fork significantly helps my personal short-term peace of mind: whenever Mozilla makes whatever changes they’re planning to make which requires them to have a license to any data I input into Firefox, I trust that I will hear about those changes before LibreWolf incorporates them, and there’s a decent chance that LibreWolf will rip them out and keep them out for a few releases as I assess the situation. If I’m using Firefox directly, there’s a decent probability that I’ll learn about those changes after Firefox updates itself to include them. Hell, for all I know, Firefox is already sending enough telemetry to Mozilla that someone there decided to make money off it and that’s why they removed the “Mozilla will doesn’t and will never sell your data” FAQ item; maybe LibreWolf ripping out telemetry is protecting me against Mozilla right now, I don’t know.
Long term, what I personally do doesn’t matter. The fact that Mozilla has lost so much good-will that long-term Firefox advocates are switching away should be terrifying to Mozilla and citizens of the Web broadly, but my personal actions here have close to 0 effect on that. I could turn into a disingenuous Mozilla shill but I don’t exactly think I’d be able to convince enough people to keep using Firefox to cancel out Mozilla’s efforts to sink their own brand.
If Firefox is just one of three browsers funded by Google which don’t respect user privacy, then what’s the point of it?
People want Firefox and Mozilla to be an alternative to Google’s crap. If they’re not going to be the alternative, instead choosing to copy every terrible idea Google has, then I don’t see why Mozilla is even needed.
Well to be fair to Mozilla, they’re pushing back against some web standard ideas Google has. They’ve come out against things like WebUSB and WebHID for example.
How the heck do they spend that much? At ~20M LoC, they’re spending 25K per line of code a year. While details are hard to find, I think that puts them way above the industry norms.
I’m pretty sure that’s off by 3 orders of magnitude; OP’s figure would be half a US billion, i.e. half a milliard. That means 500M / 20M = 25 $/LOC. Not 25K.
I see your point, but by that same logic, shouldn’t we all then switch to Librewolf? If Firefox’s funding comes from Google, instead of its user base, then even if a significant portion of Firefox’s users switch, it can keep on getting funded, and users who switched can get the privacy non-exploitation they need?
TL;DR 90% of Mozilla’s revenue comes from ad partnerships (Google) and Apple received ca. 19 Bn $ per annum to keep Google as the default search engine.
Where did you get those numbers? Are you referring to the whole effort, (legal, engineering, marketing, administration, etc) ot just development?
That’s an absolutely bonkers amount of money, and while i absolutely believe it, im also kind of curious what other software products are in a similar league
Yeah, that seems like legal butt-covering. If someone in a criminalizing jurisdiction accesses these materials and they try to sue to the browser, Mozilla can say the user violated TOS.
This is just a lie. It’s just a lie. Firefox is gratis, and it’s FLOSS. These stupid paragraphs about legalese are just corporate crap every business of a certain size has to start qualifying so they can’t get their wallet gaped by lawyers in the future. Your first bullet point sucks - you don’t agree to the Acceptable Use Policy to use Firefox, you agree to it when using Mozilla services, i.e. Pocket or whatever. Similarly, your second bulletpoint is completely false, that paragraph doesn’t even exist:
You give Mozilla the rights necessary to operate Firefox. This includes processing your data as we describe in the Firefox Privacy Notice. It also includes a nonexclusive, royalty-free, worldwide license for the purpose of doing as you request with the content you input in Firefox. This does not give Mozilla any ownership in that content.
The text was recently clarified because of the inane outrage over basic legalese. And Mozilla isn’t selling your information. That’s not something they can casually lie about and there’s no reason to lie about it unless they want to face lawsuits from zealous legal types in the future. Why constantly lie to attack Mozilla? Are you being paid to destroy Free Software?
Consciously lying should be against Lobsters rules.
Let’s really look at what’s written here, because either u/altano or u/WilhelmVonWeiner is correct, not both.
The question we want to answer: do we “agree to an acceptable use policy” when we use Firefox? Let’s look in the various terms of service agreements (Terms Of Use, Terms Of Service, Mozilla Accounts Privacy). We see that it has been changed. It originally said:
“When you upload or input information through Firefox, you hereby grant us a nonexclusive, royalty-free, worldwide license to use that information to help you navigate, experience, and interact with online content as you indicate with your use of Firefox.”
Note that this makes no distinction between Firefox as a browser and services offered by Mozilla. The terms did make a distinction between Firefox as distributed by Mozilla and Firefox source code, but that’s another matter. People were outraged, and rightfully so, because you were agreeing to an acceptable use policy to use Firefox, the binary from Mozilla. Period.
That changed to:
“You give Mozilla the rights necessary to operate Firefox. This includes processing your data as we describe in the Firefox Privacy Notice. It also includes a nonexclusive, royalty-free, worldwide license for the purpose of doing as you request with the content you input in Firefox. This does not give Mozilla any ownership in that content.”
Are the legally equivalent, but they’re just using “nicer”, “more acceptable” language? No. The meaning is changed in important ways, and this is probably what you’re referring to when you say, “you don’t agree to the Acceptable Use Policy to use Firefox, you agree to it when using Mozilla services”
However, the current terms still say quite clearly that we agree to the AUP for Mozilla Services when we use Firefox whether or not we use Mozilla Services. The claim that “you don’t agree to the Acceptable Use Policy to use Firefox” is factually incorrect.
So is it OK for u/WilhelmVonWeiner to say that u/altano is lying, and call for censure? No. First, it’s disingenuous for u/WilhelmVonWeiner to pretend that the original wording didn’t exist. Also, the statement, “Similarly, your second bulletpoint is completely false, that paragraph doesn’t even exist:” is plainly false, because we can see that paragraph verbatim here:
So if u/WilhelmVonWeiner is calling someone out for lying, they really shouldn’t lie themselves, or they should afford others enough benefit of the doubt to distinguish between lying and being mistaken. After all, is u/WilhelmVonWeiner lying, or just mistaken here?
I’m all for people venting when someone is clearly in the wrong, but it seems that u/WilhelmVonWeiner is not only accusing others of lying, but is perhaps lying or at very least being incredibly disingenuous themselves.
Oh - and I take exception to this in particular:
“every business of a certain size has to start qualifying so they can’t get their wallet gaped by lawyers”
Being an apologist for large organizations that are behaving poorly is the kind of behavior we expect on Reddit or on the orange site, but not here. We do not want to or should we need to engage with people who do not make good faith arguments.
Consciously lying should be against Lobsters rules.
This is a pretty rude reply so I’m not going to respond to the specifics.
Mozilla has edited their acceptable use policy and terms of service to do damage control and so my exact quotes might not be up anymore, but yeah sure, assume that everyone quoting Mozilla is just a liar instead of that explanation if you want.
Sorry for being rude. It was unnecessary of me and I apologise, I was agitated. I strongly disagree with your assessment of what Mozilla is doing as “damage control” - they are doing what is necessary to legally protect the Mozilla Foundation and Corporation from legal threats by clarifying how they use user data. It is false they are selling your private information. It is false they have a nonexclusive … license to everything you do using Firefox. It is false that you have to agree to the Acceptable Use Policy to use Firefox. It’s misinformation, it’s FUD and it’s going to hurt one of the biggest FLOSS nonprofits and alternate web browsers.
It is false that you have to agree to the Acceptable Use Policy to use Firefox.
So people can judge for them selves, the relevant quote from the previous Terms of Use was:
Your use of Firefox must follow Mozilla’s Acceptable Use Policy, and you agree that you will not use Firefox to infringe anyone’s rights or violate any applicable laws or regulations.
This is a pretty incendiary comment and I would expect any accusation of outright dishonesty to come with evidence that they know they’re wrong. I am not taking a position on who has the facts straight, but I don’t see how you could prove altano is lying. Don’t attribute to malice what can be explained by…simply being incorrect.
FYI this is a change made in response to the recent outrage, the original version of the firefox terms included
Your use of Firefox must follow Mozilla’s Acceptable Use Policy, and you agree that you will not use Firefox to infringe anyone’s rights or violate any applicable laws or regulations.
Your locale is forced to en-US, your timezone is UTC, your system is set to Windows. It will put canvas behind a prompt and randomizes some pixels such that fingerprinting based on rendering is a bit harder. It will also disable using SVG and fonts that you have installed on your systems
Btw, I don’t recommend anyone using resist fingerprinting. This is the “hard mode” that is known to break a lot of pages and has no site-specific settings. Only global on or off. A lot of people turn it on and then end up hating Firefox and switching browsers because their web experience sucks and they don’t know how to turn it off. This is why we now show a rather visible info bar in settings under privacy/security when you turn this on and that’s also why we are working on a new mode that can spoof only specific APIs and only on specific sites. More to come.
Yes, if everyone is running a custom set of spoofs you’d end up being unique again. The intent for the mechanism is for us to be able to experiment and test out a variety of sets before we know what works (in terms of webcompat). In the end, we want everyone to look as uniform as possible
It breaks automatic dark mode and sites don’t remember their zoom setting. Dates are also not always localized correctly. That’s what I’ve noticed so far at least.
If you have clang-tidy available, there’s a lint to prevent assignments in if conditionals. Swapping the operands just looks too uncanny to me and you have to convince everyone to do it.
Both GCC and clang have warnings for the assignment in conditional mistake, and I believe they’re included in -Wextra. There really isn’t any need for this awkwardness.
One exception: I use reversed comparisons when the left side would otherwise be very long and potentially make the comparison hard to spot.
if (5 == function(lots, of, very + long + argument, expressions))
So what is actually wrong with a program that has an Rc live across wait points? As in if we disable static checks and just ran the program (like a C compiler) could there ever be thread contention and undefined behaviour? It seems like only one thread would use the Rc at a time… so an Arc seems like overkill at first blush.
Can the cores of a CPU really be so out of sync that the refcount set from before the work stealing is not reflected correctly in the core that picks up the work? How would that happen? Maybe If it was stealing it back and had an earlier version of the future in cache? If so, you’d have similar issues with the other plain old data in the future… so it can’t be that.
I can’t tell if this is a compiler false positive (completeness issue) or saving us from actual UB.
The compiler simply enforces what the type promises, and in this case Rc says that it isn’t safe to send elsewhere. You could have some kind of RcSendable type constructible from an Rc with a refcount of one, or you could have some kind of structure containing Rcs that guarantees that they can’t be leaked to some other part of the program, and have them be Send, but making Rc itself Send in a limited set of circumstances would be difficult, for questionable gain.
Keep in mind that it’s impossible to make a compiler that allows all correct programs but rejects all incorrect programs. So since Rust wants to reject all programs that have UB, it must also reject some programs that don’t have UB. Efforts are ongoing to increase the number of correct programs that Rust allows, but adding special logic to the compiler to allow fresh Rcs to be Send seems not worth it.
So what is actually wrong with a program that has an Rc live across wait points?
The problem is elsewhere.
An Rc automatically frees its contents. It uses a refcount which is adjusted when the Rc is cloned or dropped. If Rc were Send then you could clone it into multiple threads. The refcount adjustments don’t use atomic instructions so they are likely to go wrong and cause use-after-free errors.
If a send an Rc that I own to another thread, won’t it be moved (neither cloned nor dropped - the refcount stays constant)?
(And then my question was if every last clone of a given Rc was sent/moved as part of a single owned value, in this case a future, to another thread, mightn’t that be technically valid?)
You can’t statically prove it’s the last reference though, so the type system has to disallow the general case where it might not be the last reference. If you only need the one reference, perhaps don’t use an Rc?
That would avoid the need to reallocate the inner value. It seems like their current implementation has exactly the same memory representation (with the small exception that AtomicUsize can have more stringent alignment requirements than usize, although it should be trivial to just make sure the RcInner struct is aligned properly)
I believe system allocators frequently align to at least 8 bytes, which means that in practice the RcInner should already end up aligned suitably for ArcInner.
Given that, you could implement this yourself. This assumes of course that RcInner and ArcInner don’t ever change layouts (or if they do, that the layouts stay identical).
fn rc_to_arc<T>(mut rc: Rc<T>) -> Result<Arc<T>, Rc<T>> {
// first, check to make sure we have unique ownership
if Rc::get_mut(&mut rc).is_none() {
return Err(rc);
}
// next, grab the raw pointer value
let p = Rc::into_raw(rc);
// check to make sure it's aligned for AtomicUsize.
// this pointer points to the value, not the header,
// but the header's size is a multiple of the AtomicUsize
// alignment and so if the pointer is aligned, so is
// the header.
if (p as *const AtomicUsize).is_aligned() {
// the memory layout of RcInner and ArcInner is identical.
Ok(unsafe { Arc::from_raw(p) })
} else {
Err(unsafe { Rc::from_raw(p) })
}
}
You definitely can have “memory value is V0, CPU 1 writes V1, CPU 2 reads V0”, and you’re exactly right that applies to any memory location.
If you want to ensure writes made by one CPU are visible to another with certainty you need to issue instructions for that. Otherwise your write may be in a cache or memory queue not visible to the other CPU, or your read may be from a stale cache
As far as I remember, Rust targets some abstract common denominator of existing memory models. This allows the compiler to make valid choices while checking the higher level code (what LLVM does on the lower level is highly platform specific though). That memory model is quite conservative, thus the errors. What would happen in reality for the potentially racy construct is sort of UB as it’s naturally not defined :)
So any value that is Send and that is moved to another thread might have some instructions added so that it is read coherently?
That sounds fine in this specific case so long as the same instructions were applied to RcInner. But that is pointed to with a NonNull, for which the compiler doesn’t want to mess with and which isn’t Send…
You make this sound like it’s automatic. The guts of Rc use raw pointers, which aren’t Send, therefore Rc is not Send. If you take something like Arc, the guts are also not Send, but Arc (unsafely) implements Send explicitly as a manual promise. Arc’s methods are manually coded so that the overall type behaves in an atomic/coherent way.
The whole point of Rc is that it doesn’t go to all that trouble, which has a cost as well, but it still allows an object to be referenced from multiple locations which are accessible to only one thread. (Including objects of types which themselves are not Send.) Yes you could modify Rc to implement Send. Congratulations, you implemented Arc.
Many primitive types such as i32 are also Send, and so are types derived from them. Not because the compiler inserts any special instructions or something, but because the mechanism by which it’s moved from one thread to another is assumed to be safe (e.g. channel, mutex, etc. - safe here usually meaning it uses a memory barrier of some kind), and there’s nothing about the type itself that needs special treatment.
Is a marginal slowdown really that important in CI, as opposed to something being run on a laptop where the developer is interactively waiting for npm publish to complete?
(To be fair, I’m kind of just stirring the pot - an obvious retort to this question might be, actually, private packages tend to be huge and this would balloon the time on the order of tens of minutes. I don’t know whether that’s true or not.)
You could also make it opt-out. Default to slow, strong compression, but if it causes a major performance regression on your deployment, toggle a flag and you’re back on the old behaviour.
I haven’t followed Swift much recently, but the original Swift was simple because it punted on all of the hard problems. It had no mechanism for error handling (there were some small mods towards NSError). It had no support for concurrency and the everything-is-shared-mutable-state model made that really hard to fix (the structured concurrency and actor extensions are trying). It had no support for shared libraries (Swift’s shared library support now is really nicely designed and it’s the only modern language that I think has addressed this problem well, but that inevitably came with complexity).
For macOS development, Objective-C++ is actually a very nice language. C++ for local code that is specialised over strong types, Objective-C for late-bound interfaces that are exposed across module boundaries,
I have never used Objective C++, but from afar it sounds horrifying. Take a monstrously, ineffably complex programming language and another quite complex programming language and mash them together into an amalgamation. Can anybody hope to make sense of it?
Speaking as someone who used it for years, it actually works quite well. Obj-C is not very complex, and its OO stuff is pretty separable from the C part. Obj-C++ extends the C part.
I’ll also point out that modern Objective-C’s ARC (automatic reference counting) composes very well with C++’s object lifetime rules. In pure Objective-C with ARC,
is not allowed, because NSString* is an ARC pointer and the compiler needs to be able to reason about its initialisation and invalidation. The semantics of C structs don’t allow for that, which can make implementing custom data structures tricky: you either have to do it all in the context of Objective-C classes (which has certain overheads), use indirection, or you turn off ARC in the relevant source files and do manual reference counting.
The same code quoted above will compile on Objective-C++ however, because the ARC pointer is treated as having the usual set of C++ constructors, destructor, assignment operator, and so on - it’s not a POD (plain old data) type. This means the struct also gets implicit constructor/destructor/operator implementations when not explicitly implemented.
You can therefore shove pointers to Objective-C objects into all sorts of C++ data structures, that have been implemented without special Objective-C support, including the STL. It all composes rather nicely.
(A significant proportion of my work in recent years has been working on macOS device drivers and related apps, daemons/agents, etc.; I’ve mostly been using Objective-C++ on that, although I’ve recently introduced Rust in one part of such a project. My limited contact with Swift has been exceedingly frustrating, so I’ve avoided it where possible; it never appealed to me in the first place due to the reasons David mentioned, and the practicalities around extremely poor forward- and backwards-compatibility were a nightmare to deal with on a project where it was forced upon me.)
ARC certainly makes this easy, though prior to ARC I implemented a C++ smart pointer class for Objective-C references that did the relevant retain and release operation, which meant that this was already possible in Objective-C++ without ARC, it just became nicer because you didn’t need to use the smart pointer.
Definitely, you weren’t the only one to implement such a smart pointer class. I guess my point was more that modern Objective-C actually composes better with C++ libraries than it does with C libraries and data structures.
I also used it for years and we made Mac apps that spoke to hardware devices through the IOKit kernel driver API (which is in C++). It was indeed quite nice.
There is a non-trivial amount of ObjC++ in Firefox, too, or at least there was last time I checked. For TenFourFox I used ObjC++ as glue code to connect up libraries.
To add to what others have said: a lot of the pain in C++ comes from trying to do things that are easy in Objective-C, and vice versa. With C++, it’s easy to create rich types with no run-time overhead, but that create tight coupling. With Objective-C, you have late-bound interfaces everywhere, but avoiding dynamic dispatch is very hard / hacky. The combination means that you can completely avoid things like raw C pointers. You can used C++ collections inside a module, Objective-C ones across the boundaries.
You should give Obj-C a try, I think! It’s a surprisingly thin layer on top of C, giving a lot of bang for the buck for writing dynamic (but fast!) programs. It’s quite unique in that you have two extremes: C on the one hand, and a fully dynamic OO runtime (you can determine implementation at runtime). Overall syntax is outdated and weird (smalltalk influence), but it is still unmatched in that niche.
It’s not bad in practice. The use case is, you need Objective-C system frameworks but you can’t do without particular C++ libraries. You still spend the bulk of application code in one language.
I had a game that used Bullet physics in this way. I migrated most of the code to Swift after it was introduced, but I kept some Objective-C++ in order to keep Bullet. These days Swift has direct C++ interop, both for library use and for gradual migration of C++ projects to Swift.
I worked with making an iOS app at one point and I found while thinking in Lisp like patterns it seemed to get out my way if I wanted it to. But that is a beginner and greenfield perspective for sure.
I don’t think it sounds too bad, but I haven’t used it myself.
My understanding is it’s just the extra OOP bits from Objective-C overlayed on C++, similar to how it was overlayed on C in the first place. Basically just a second, independent object system. I understand why people wouldn’t like that, but it doesn’t sound too different than C++/CLI or using C++/JNI.
With the caveat that I’ve read their design docs, but not actually used it in anger:
They make a clear distinction between ABI-stable and ABI-unstable shapes of structures. Within a library, there are no ABI guarantees. At a library boundary, you have a choice whether you want to sacrifice some performance for the ability to change a layout later, or sacrifice flexibility for performance. This is a per-structure choice. Depending on the choice that you make, the compiler either lowers to something similar to Objective-C non-fragile ivars, or C struct fields.
They actually have a language notion of a library boundary. This means that generics can be statically reified within a library, but fall back to dynamic dispatch across library boundaries. Contrast this with C++ where templates either have to live in headers (and then end up copied in every compilation unit, including the implementation, and it’s a violation of the one-definition rule to link two libraries that use different versions of the same template) or they are private to a library. The Swift model gracefully falls back. Things may be faster inside a library, but they still work from outside, and the ABI doesn’t leak implementation details of the generics, only their interfaces.
but the original Swift was simple because it punted on all of the hard problems
Hmm…I’d say it was already incredibly complicated despite punting on a lot of hard problems and largely because it tried to codify “solutions” to non-problems into the language. Which never works, because reality, even code reality, is way too messy for that.
As an example, I wrote about the mess that is initialization back in 2014, so right after Swift apepared. There was some pushback from a prominent member of the Swift team saying that my goal of simplicity just wasn’t compatible with some of the goals they had. Hmm….
There was also a great rant by a prominent member of the community about Swift being just a mess of special cases upon special cases. I think he left the Apple ecosystem, and he’s not the only one. Alas I can’t find it and I don’t remember the name.
Anyway, I predicted back then that because they had so many language features for initialization it would never actually work out and there would be a renaissance of builder and factory method patterns and there would be even more language features for initialization. Five years later: hello SwiftUI! :-)
So the failure of Swift now isn’t surprising, the trajectory was already set in stone the day it launched and there wasn’t really much one could have done about it afterward…much less so since the same faulty reasoning that led to the initial problems was still present and guided later evolution.
Anyway, I predicted back then that because they had so many language features for initialization it would never actually work out and there would be a renaissance of builder and factory method patterns and there would be even more language features for initialization. Five years later: hello SwiftUI! :-)
I think this is an instance of correlation not being causation? My understanding is that the actual cause of SwiftUI is the successful design of Flutter (which gave raise to both SwiftUI and Kotlin Compose), and it is relatively orthogonal to language machinery.
Case in point, Kotlin’s initialization story is much more tame than Swit’s one (as it doesn’t try/doesn’t need to prove initialization safety statically), but it also converged on the essentially same design (or rather, vice-verse, IIRC Kotlin work in the area predate’s Swift’s).
Not to disagree with your wider point on constructors, which I agree with, just to point out that SwiftUI is not I think a particularly strong argument here.
I think you might want to have a look at the actual article. Swift introduced yet more special syntax for the part of SwiftUI that creates the view tree. So yet more language features for yet another way of constructing views^Wobjects^Wstructs.
I have read the articles! If I understand your argument correctly, it says that the fact that they needed to add new stuff to support Swift UI means that the original rules were inadequate. My counter-argument is even languages that don’t have Swift-style maze of initialization rule add special cases to support SwiftUI patterns. Ergo, adding stuff for SwiftUI is orthogonal to your normal way to initialize objects. In other words I claim that, in counter-factual where Swift doesn’t have complicated initialization rules and uses Java/Go style “everything is null to start with” or Rust/ML style “everything starts with all the parts specified”, it would have added more or less the same features still for SwiftUI.
The story is even more illustrative with Kotlin — it was specifically designed for DSLs like SwiftUI/Compose. The whole language, with its second-class-lambdas, extensions, and coming out-of-fashion implicit this, works towards that goal. And yet, when the actual UIs started to be implemented, it was quickly apparent that no one wants to write +button(), and a bit more of compiler special sauce is needed for nice surface syntax.
I must be a lousy communicator, because you seem to have misunderstood the article almost completely.
The point was not that Swift has the wrong initialization rules or too many of them. The point is, as it says in the title: “Remove features for greater power”. The many initialization rules are not the source of the problem, they are a symptom of the problem.
First rule of baking programming conventions into the language: Don’t do it!
The second rule of baking programming conventions into the language (experts only): Don’t do it yet!
The problem is trying to bake this stuff into the language. As a consequence, you get 30 pages of initialization rules. As a further consequence, those 30 pages will be forever insufficient.
So for me, the supposed counter-point you bring with Kotlin actually supports my point. You write:
it was specifically designed for DSLs like SwiftUI/Compose. The whole language, with its second-class-lambdas, extensions, and coming out-of-fashion implicit this, works towards that goal
So they baked a whole bunch of features into the language to support the DSL use case. What was the title of the blog post again?
“Remove features for greater power”
So they added a lot of features into the language especially for this use-case and it didn’t even work out for this particular use-case. Surprise surprise!
First rule of baking programming conventions into the language: Don’t do it!
The second rule of baking programming conventions into the language (experts only): Don’t do it yet!
I simply don’t think the static/compiler-oriented mindset is compatible with the sorts of things these languages are trying to do. You put way too much into the language/compiler, and you do it way too early.
Ruby has had a bunch of these kinds of frameworks, and as far as I know they did not require any changes to the language. Because Ruby had fewer but more flexible features to start with.
With Objective-S I seem to be violating that rule, because it certainly does put things into the language. Or at least seems to do so. What I am doing, however, is following the second rule: “don’t do it yet”. (With quite a bit of trepidation, because it is “experts only”).
And I am not actually baking all that much into the language. I am baking a bit of useful surface syntax and the associated metaobject-protocol into the language. What lies behind those metaobject protocols is quite flexible.
So far this appears to strike a good balance between providing some syntactic convenience and compiler support while not making the mistake of baking way too much into the language.
Indeed! I misunderstood your original comment as meaning that SwiftUI is a downstream consequence of initialization rules. I agree that both are rather the result of the lack of expressiveness, which doesn’t allow the user to “do it yourself” in userland code. The Kotlin example was exactly to illustrate that point.
There was also a great rant by a prominent member of the community about Swift being just a mess of special cases upon special cases. I think he left the Apple ecosystem, and he’s not the only one. Alas I can’t find it and I don’t remember the name.
Crucially, the vast majority of this is incidental complexity, not essential complexity. Swift is a crescendo of special cases stopping just short of the general; the result is complexity in the semantics, complexity in the behaviour (i.e. bugs), and complexity in use (i.e. workarounds).
How to fix:
What, then, should be removed? It’s probably too late to remove any of that, …
So I certainly wasn’t the only one who accurately predicted the current sad state of affairs. It was extremely obvious to a lot of people.
I am if the opinion that the only avenue for making a well rounded language that lasts is to put all of the skill points towards a small handful of skills (“fast”, “fast to compile”, “ease of dynamic linking”, “easy to use”, “easy to learn”, whatever) where each “skill” can be thought as a different axis in a graph. Go all in on a few niches. This helps with clarity of purpose for the project itself, a clear pitch for prospective users, and makes it possible to show the benefits for that niche. Without doing this the evolution of the project ends up rudderless, and adoption can stagnate either slowing down or killing the project. Once you’ve reached critical mass, fight for dear life not to abandon those hard earned early design decisions that allowed you to carve a niche, but go all-in on a handful of other axis as well. Rinse and repeat. A language can start as easy to use and once it has a clear base experience it can start trying to produce faster binary. Or it can start as fast to compile and afterwards try to produce optimized code. Or it can start producing super optimized code and later try to become faster to compile or easier to use. All the early design decisions will hamper the later pivot, and that’s ok! No language will ever be all things for everyone (barring some unrealistic investment in monstrous amounts of research, development and maintenance).
Swift had a clear initial view of what it wanted to accomplish, and did so successfully (I believe, don’t have personal experience with using Swift, nor do I keep up with its development). It is now on its expansion era, to allow for niches that previously were either difficult or impossible to cater to. It is a delicate process that can easily go haywire because of subtle interactions between completely unrelated features can make the result worse than the sum of its parts. But I can’t blame them for trying.
I could maybe argue that sync Rust had excellent “early design decisions that allowed you to carve a niche”, but the async related problems are now occupying a vast majority of the brain time of the lang teams. While I don’t think we’re heading to 217 keywords, I certainly feel uneasy seeing “keyword generics” and similar efforts.
“Do not obey in advance”. It’s highly unlikely that you’ll ever hear anything about it, but even if you did, it would be something like a cease and desist letter. You can decide then what to do about it.
Considering there is criminal liability for noncompliance, I don’t think “it would be something like a cease and desist letter” is useful advice at all. Even if you reckon you’re safe from this in your country, you’d potentially have to very carefully consider any future travel and whether you’re entering jurisdictions which might begin extradition proceedings, etc.
I’m no expert on the CRA, but it was my understanding that after public consultation, liability has been explicitly excluded for non-commercial offerings, including open source contributors. In other words, you won’t be accidentally held liable for a security vulnerability in your code that someone downloaded from your website or repository for free. You don’t generally “accidentally” sell software for money, so you can price any costs for compliance into whatever offering you have, or alternatively, it’s not that difficult to exclude selling to markets where you don’t consider compliance with local laws proportionate.
The OSA is pretty unusual in that there’s no carve-out for small-scale or non-commercial operations, and in that it effectively applies globally to almost any publicly accessible website or digital service with a community component.
The page doesn’t explain why the CPU needed a separate, incompatible “protect mode” to access more memory. I learned more about that from the following Wikipedia pages, which I have summarized.
Real mode, also called real address mode, was once the only mode of Intel CPUs, so operating systems already supported it. In this mode, addresses always correspond to real locations in memory. For backwards compatibility, x86 CPUs still start up in real mode, though modern operating systems turn on protected mode when booting so as to be able to access more than 1 MiB of memory.
Protected mode, also called protected virtual address mode, enables virtual memory, paging, and other features. The mode was backwards incompatible with real mode because it reinterpreted some bits of a memory address to point to memory indirectly via an entry inside a descriptor table. It also reinterpreted two bits as defining the privilege of the memory request.
Protected mode was introduced in the Intel 80286 processor, the one mentioned in the article. The Wikipedia article on the processor has sections on its protected mode and its OS support.
Also… “protect mode”? I had never heard anyone call it that way, ever. It has always been “protected mode” to me. (Sorry, had to mention it somewhere; not related to what you asked.)
Also… “protect mode”? I had never heard anyone call it that way, ever.
I have discovered this since sharing the post. :-( Apparently my shorthand form that I’ve been using for over a third of a century is Bad and Wrong and Incorrect. My mistake.
Not your fault! The article is full of “protect mode” as well, so I assumed the title here was intentional. I suppose someone edited the article content before publishing it and “fixed” protected with protect?
I was an early adopter of Windows – I used Windows 2 in production on my own work computer in 1988/1989, and I told my boss and my boss’ boss at work that Windows 3 was going to be huge and they should both stock up with as many copies as they could get, and that they should let me train the staff on Windows.
They laughed at me. They did not.
We sold out of all 17 (seventeen) copies of Windows 3.0 before 9AM on the day of release, and the company was flooded with enquiries. There was huge interest and nobody else other than me had ever used Windows as an environment. (We sold apps with Runtime Windows, such as Excel, Pagemaker, and the radical Omnis database, AFAIK the first GUI database for the PC, in which my company developed a bespoke app.) But the company regarded Windows as a joke product.
After 6 months of overwork due to being the only person who could support all the customers with Windows 3.0, I quit.
But, yes, at the time, around the start of the 1990s, that was a common term: Windows ran in “protect mode” and it was “protect mode” that let it use XMS as more program RAM and so on. Few other “protect mode” OSes existed: OS/2 1.x, SCO Xenix, and Novell Netware were about it.
Yes, protected mode might have been more accurate, but it was known by the shorter name.
This is not history to me. I am at work right now (and should not be commenting here) and this was an earlier stage of my job. I only switched to being a Linux guy in the 21st century. The first part of my career was as an expert Windows support guy.
As another person who lived through those times, I agree that “protect mode” was commonly heard, though it was also common to hear “protected mode”.
Not that this is by any means definitive, considering how common shortening of such things in a config file context is/was, but this conversation made me think of the OS/2 CONFIG.SYS setting: “PROTECTONLY=NO”. I always looked forward to the day that every app was built for OS/2 and I’d be able to flip that to YES. Still waiting!
It does SMP, it can run thousands of DOS and 16-bit Windows apps, dozens of OS/2 apps, and some Win32 and FOSS/xNix apps. It’s tiny and amazingly quick compared to any modern OS, and I very much include Linux in that. No Linux comes close in performance and about the only thing that even could compete is TinyCoreLinux running from RAM. Alpine Linux is tiny and runs well on early 64-bit kit such as a high-end Core 2 Duo, and ArcaOS absolutely stomps it. It boots faster, apps load faster, it’s more responsive, and it takes less space on disk or RAM.
An ArcaOS with PAE support in the kernel would deliver quite a lot of what I want from an OS even now.
I just don’t have much use for it… :-/
It’s blisteringly quick on low-end kit, but then, so is Windows XP, especially XP64, and XP has a lot more apps and is much easier to get running. It’s just horribly insecure.
Lol I enjoyed the article, thanks. I bet OS/2 (er, ArcaOS) does indeed run pretty quick on modern hardware compared to how it ran on my IBM PS/2 Model 70 that packed an almost unimaginably large 6 MB of RAM!
I’ll always love OS/2, both for it being the first “real OS” that I ever ran (and igniting a life-long interest in systems software) and for the lessons I learned as a teenager watching people on Usenet fall further and further into denial that it was all over for it (because of Macro$oft and the corrupt Ziff-Davis, natch). Oh to go back to a time with such low-stakes internet brainrot.
I recently did something that I felt guilty about, because I enjoyed it too much: put XP on a recent machine and went online. It was so fast and responsive, and at that time, you could still get free XP antivirus. You can get tools to find drivers, modern browsers that can handle 202x social networks etc…
It is a forbidden pleasure.
By comparison with modern OSes, WinXP is sleek and fast. Especially XP64 in 8GB of RAM.
I reviewed MS Office 97 when it came out. I hated it: bloated, sluggish, buggy. Now, it’s tiny and fast. It runs perfectly on WINE, including installing the service releases. Time makes a lot of difference. But it’s one suite.
XP makes one look at modern Windows and Linux and realise how appallingly sluggish everything is today.
But ArcaOS flies on machines that are low-end for XP. It can use up to 64 cores, it can just about do wifi and multihead true-colour accelerated graphics, USB2, UEFI… it has browsers, it has the yum command and repos with FOSS ports. It is surprisingly capable for a late-1990s OS.
The only big thing it can’t do is access >4GB RAM, except for a big fast RAMdisk.
It reminded me why I thought XP was a bloated mess in 2001.
My father’s company got its first Windows 3 install bundled with an app. A vector drawing program, I think it was called MetaDesign, decided it was cheaper to write a Windows 3 app and bundle a copy of Windows for DOS users than to write or license a vector drawing library and printer drivers for DOS. Before that, they used GEM, but once they had Windows installed it was easy to sell them more Windows apps.
I never ran 3.0 in protected mode though, only ever on real mode on an 8086 or 386 Enhanced Mode on an 80386 or newer. I was quite happy with that, because the 80286’s segmentation model was a thing that you teach in computer architecture courses as a cautionary tale.
This was a standard thing to do in the Windows 2 era. It was known as “runtime Windows” and the result was most of a Windows installation, but rejigged: there was no WIN.COM and instead users ran the binary of the app itself from DOS, which loaded Windows with the app as “shell”. Quit the app, Windows shut down.
It worked fine. I had a client in London, Mentorn – they’re still around – whose accounts department ran on Excel, but which didn’t use Windows for anything else. So it was Excel + runtime Windows.
Other apps that did this were Pagemaker and Omnis, the first Windows database.
Snag: it was the plain unaugmented Windows, not Windows/286 or Windows/386. So, it could only use 640kB of RAM and run in Real Mode.
If you installed several such apps, each got its own copy of Runtime Windows.
But if you then installed a full copy of standalone Windows (2 or later) on the machine, you could browse to the app’s directory, run the binary, and it ran normally under full Windows, including interop with other apps. The app was stock and it was Windows that was modified, AFAICT.
I do not recall ever seeing Runtime Windows 3. With Windows 2’s fugly MS-DOS Executive replaced by the perfectly usable Program Manager, Win3 made a quite decent DOS shell that users liked, and some used it with no other apps.
I never ran 3.0 in protected mode though
(!)
It was a thing.
Mainly for me on 386 machines with only 1MB of RAM. Win3 & 3.1 wanted 2MB minimum for 386 Enhanced Mode and if they had less they’d load in Standard Mode by default.
(286s were mostly too slow to comfortably run Windows. Indeed Windows 3 was a major driver of adoption of the 386 via the budget 386SX with a 16-bit bus.
((Aside to the aside: clock-for-lock, the 80286 was quicker than the 80386. But the normal speed of 286 PCs was 8MHz to 10MHz with a few 12MHz. I know of a single 20MHz 286 machine that was ever made, by Dell. Many 10-12MHz 286s had slower RAM and so ran with wait states. CPU cache only became a thing with high-end 386DX machines and most 286s had no cache, so if the RAM couldn’t keep up, the CPU just had to go slower.
(((Most IBM 286 PS/2 models shipped with slow RAM and wait states: in other words, the 286-based PS/2s for which IBM crippled OS/2 1.x and destroyed its chances in the market were lousy slow 286s and unsuitable for running OS/2 1.x, which was about the most demandingh 286 OS ever released, bigger and slower than Netware 2, or Xenix 286, or Concurrent DOS 286.)))
/* I am nesting way too deep here */
The 386SX ran from 16MHz to up to 33 or even 40MHz in some late models. So, 386SX machines were in fact faster than 286s, not because the CPU was inherently quicker – it wasn’t – but because it was clocked faster.
Also, few non-IBM 286s had VGA. Windows on EGA wasn’t very pleasant.))
So, there were lots of decent 386 machines, both 386DX and some 386SX, that were specced to run DOS and only had a meg or two of RAM. I saw some with 1.5 or 1.6MB RAM: 640kB plus 1MB of extended memory. Odd, but cheap.
You could force them to run in Enhanced Mode. I forget the switch now. win /e or something. It worked. You got pre-emption of DOS apps in V86 mode “VMs”, you got virtual memory. It was dog slow but it worked. It wasn’t very stable but Windows 3 wasn’t very stable.
Secondly, Windows could only run in Standard Mode if under another 386 multitasker. So, it ran in Standard Mode under DESQview/386, and DESQview/X, and Concurrent DOS 386, and a few other things.
So it was very much a thing and quite common.)
When I posted here on Lobsters in the past that Windows/286 ran in Standard Mode a lot of people got angry and contradicted me. It seems to me that there still is not a categorical, ex-cathedra statement of exactly how Windows 2 in 286 mode worked.
But Windows 3 in Standard Mode was quite good and made effective use of up to 15.6MB of XMS if you had it – on a 286.
This was a standard thing to do in the Windows 2 era
It’s possible that I misremember. It could have been Windows 2. I think 3.0 was the first one that they rolled out company-wide.
It was a thing.
I know, I just never owned a 286, and the 386s I owned had 4 or 5 MiB of RAM, so there was no point in pretending to be a 286. I went straight from 3.0 in Real mode to 3.1 in Enhanced Mode.
Windows 3.0 ran on my Amstrad PC1640 HD20 (with the hard disk upgraded to a massive 40 MB!). With 640 KiB of RAM, you could run Windows and 1-2 apps. It had an EGA monitor, so you mostly ran one app maximised anyway, which reduced the value of multitasking.
Give it a try. As I recall, it is merely running in “real mode” with the HMA enabled, and with a special version of USER in the HMA. At least that is what the docs included with it say.
What windows/286 does appear to have is an EMS 4.0 / EEMS aware KERNEL, and so it can possibly page below 640K if the h/w implements it. i.e. if the board, chipset and memory cards are set for minimal base memory, and EMS “backfill”.
Or at least that is the impression I got a number of years ago when I had a try at disassembling some of its KERNEL.* I looks like that was removed/deleted/replaced when the Windows 3.0 scheme with support for DPMI was eventually created. Which would make sense, as it is simpler.
I’d have to suggest that you’d need to offer proof for windows/286 operating in “standard” i.e. 286 protected mode, as that goes against everything I’ve seen written in the past. So maybe take those images from the site above, and get them working in such a mode.
A place I worked at during 88 - 89 had a copy of Windows/386 and Windows/286, and us developers had Acer 286 machines with (usually) 1M RAM and EGA (some lucky folks had 2M). As I recall, none of us tried running those particular Windows versions. Trying Windows/386 would have been pointless (despite there being a handfull of 386 machines), and we did give plain windows a go. Yes it was painful in EGA (or even CGA) mode.
On the other hand, Windows/286 knows how to use LIM 4.0 boards to store and swap executable code. Older versions didn’t even know how to use LIM EMS 3.0.
Give it a try. As I recall, it is merely running in “real mode” with the HMA enabled, and with a special version of USER in the HMA.
Interesting stuff. Thanks, and for the PC Mag link.
I am not bothered enough to try to run up some kind of 286 PC emulator to try it out. I do not miss Windows 2.x at all and have no urge to return.
It’s very interesting to me, though, that pivotal parts of the development of IT in the 1980s and 1990s is being forgotten now… because without knowing this stuff, some things about how it is today make no sense at all.
This industry needs to know its history much better.
Thanks for sharing the post. I suggested a title change to “Protected…” and if a few others do as well, we can stop discussing your shorthand and it’ll be easier to stick to the substance of the article. I enjoyed the post you shared from landley.net as well.
I find this tremendously interesting, because my head was firmly in Mac space at this time, so I missed much of it.
This is one of those undocumented behavior distinctions you can very well experiment to find out, and which may appear stable, but which make me want to assume the worst case across the whole domain. If I’m riding on an abstracted pay service, I can’t depend on such a fine detail of behavior to stay constant.
This piece is kind of interesting, but I think its core thesis is pretty much nonsense. You don’t need to have been there when software was first written in order to understand it. Humans are capable of learning things.
I have worked with software that probably couldn’t have survived a complete change of team, and I will say this: It’s usually the worst code at the company, it’s often itself a rewrite of what the company started with, and I always get the impression it’s being held back by the original developers who are still with it. Without these first-generation programmers, any software in danger of becoming unlearnable would necessarily be simplified or replaced.
You don’t need to have been there when software was first written in order to understand it. Humans are capable of learning things.
I think that’s a bit of a straw man; the article doesn’t say that the software itself is incomprehensible to others. With enough effort you can look at the software and understand what it does. What you can’t do after the fact is understand the context in which it was written; why was it done that way? What alternatives were considered and discarded? How has the context changed since those decisions were initially made? That’s what they mean when they talk about theory-building.
In theory you could write this stuff down, but I have never seen this actually happen in an effective way. (Probably because people keep thinking of the software itself as the point rather than the theory it embodies.)
I considered this, but looking at the article, it almost seems to take care not to talk about why. And, in any case, my experience is that people forget the context at such a rate that by ten or so years out, reverse-engineering it from the code is at least as reliable as asking the authors. Anyway, reading again, I still think this is about more than just context.
I think on balance I agree with the article. As @technomancy says, it’s about the theory the software embodies. Code is just one facet of that theory, and can never capture the tacit knowledge, ambiguities and personal relationships which all play a part in a software system.
However, I do agree with @edk- that the article dances around this point. Perhaps it’s intrinsically a bit of an abstract argument, but I couldn’t help but feel that more concrete writing would have helped.
This appears to be an excerpt from a book, so perhaps the rest of the book goes into detail on this point. I’ve added it to my list, but not bought/read it yet.
For some reason, there is a widespread default mindset (at least in the part of the industry I’ve seen) that “only those who built it can understand it.”
It doesn’t even depend on code quality (though I am a firm believer that any code written by a human can be understood by a human).
You can have a module that is clearly structured and spiced with comments about “why this solution is chosen,” or “when we’ll need X, it can be changed that way,” or “this is the assumption here; if it breaks, the assumption was wrong”… And still, when something is broken or needs update, people would hack around the module or treat it as a black box that “I’ve tried to pass this data, do you know why it doesn’t work? oh, I experimented for six more hours and it seems I guessed why!” or ask in a chat “who knows how this constant is used (used once in a codebase, with a clear comment why)” etc. etc.
It is like, through the years, the overall stance of a developer has switched from “I’ll understand the crap out of this codebase, disassemble it into smallest bits, and will rewrite it my way!!!” (an attitude that met with a lot of grounded critique) to “Nobody should understand your crap, either you support it forever, or it is thrown away in its entirety.”
I don’t think it’s true that only those who built it can understand it, but the effort required to understand a legacy codebase from scratch & safely make changes is enormous and this problem affects FOSS as well. I’ve been dealing with this for the TLA+ tools - specifically the parser - which when I joined the project was a pile of 20+-year-old Java code with everybody who touched it gone from the project for a decade or more. Past a certain point the code ceases to become source code in some sense - people will only deal with it at the API level and everything within is indistinguishable from a binary blob that cannot be changed. The process of shedding light onto that part of the codebase required writing over 300 round-trip parse tests to semi-exhaustively document its behavior, and even with that monumental effort I still only really have a handle on the syntax component of the parser, let alone the semantic checker. But that isn’t all. You may have developed a mental model of the codebase, but who is going to review your PRs? It then becomes a social enterprise of either convincing people that your tests are thorough enough to catch any regressions or giving them some understanding of the codebase as well.
Compare that with being the original author, where you basically have total ownership & can make rapid dictatorial changes to a component often without any real code review. The difference in effort is 1-2 orders of magnitude.
Then consider the scenario of me leaving. Sure all the tests I wrote are still there, but do people have a grasp of how thorough the test coverage is to gauge how safe their changes are? I would not be surprised if it took five years after me leaving for basic changes to the parser to happen again.
The only thing I was trying to say is that “only original author can fully understand that” becomes industry’s self-fulfilling prophecy, creating a feedback loop between people not trying to read others’ code (and not giving the feedback that it lacks some background information or clear structure), and people not thinking of their code as a way to communicate everything they know, because “nobody will try to read it anyway, the important thing is that it works.”
It manifests in many things, including the changed stance for code reviews, where “you left a lot of comments” starts to be universally seen as “you are nitpicking and stalling the development,” and disincentivizes those who are really trying to read the code and comment of the things that aren’t clear enough or lack an explanation of the non-obvious design choices.
okay, I’ll take the alternate stance here. I worked on the back end of a large triple AAA video game that was always online. I worked on it for roughly 6 years before I moved to another company.
I have very good documentation, very clear objectives. It was very simple infrastructure - as simple as it could be made. The “why” of decisions was documented and weaved consistently into the fabric of the solution.
I hired into my new company! my successor. Expecting him to have experience with the same problems and my original infrastructure sought to solve.
he didn’t, he didn’t learn how or why certain things were how they were. my expectation of his ability to solve problems that I had already solved because he would’ve had experience with them was completely incorrect.
had the system failed catastrophically he would’ve been unable to fix it and that was not discovered even after working there for three years
For some reason, there is a widespread default mindset (at least in the part of the industry I’ve seen) that “only those who built it can understand it.”
There are levels of understanding and documentation is variable, but there are almost always some things that don’t make it into documentation. For example, the approaches that you discarded because they didn’t work may not be written down. The requirements that were implicit ten years ago and were so obvious that they didn’t need writing down, but which are now gone, may be omitted, and they influenced part of the design.
With enough archeology, you often can reconstruct the thought processes, but that will take enormous amounts of effort. If you were there (and have a good memory), you can usually just recall things.
The problem (for me) is that people start taking those contextual truths and applying them unconditionally to any situation. Like, even without looking frequently, “I wouldn’t even start to try reading through the module (where choices of approach and limitations might be visible in code or well-documented); I’ll treat it as a black box or delegate it to the module author, regardless of the current organization structure.”
The situations I am quoting in the previous comment (“who knows how this constant is used?” in chat, regardless of the fact that the constant is used once in a codebase, with a clear comment why and what’s the meaning) are all real and somewhat disturbing. Might depend on the corner of the industry and the kind of team one is working with, of course.
I completely agree with the second half of your post. I might just be a grumpy old person at this point, but the mindset seems to have shifted a lot in the last twenty years.
For example, back then there was a common belief that software should run on i386 and 64-bit SPARC so that you knew it handled big vs little endian, 32- vs 64-bit pointers, strong vs weak alignment requirements, and strong vs weak memory models. It also had to run on one BSD and one SysV variant to make sure it wasn’t making any assumptions beyond POSIX (using OS-specific features was fine, as long as you had fallback). This was a mark of code quality and something that people did because they knew platforms changed over time and wanted to make sure that their code could adapt.
Now, I see projects that support macOS and Linux refusing FreeBSD patches because they come with too much maintenance burden, when really they’re just highlighting poor platform abstractions.
Similarly, back then people cared a lot about API stability and, to a lesser degree, ABI stability (the latter mostly because computers were slow and recompiling everything in your dependency tree might be an overnight job or a whole-weekend thing). Maintaining stable APIs and having graceful deprecation policies was just what you did as part of software engineering. Then the ‘move fast and break things’ or ‘we can refactor our monorepo and code outside doesn’t matter’ mindsets are common.
The problem (for me) is that people start taking those contextual truths and applying them unconditionally to any situation.
That seems like a meta-problem that’s orthogonal to the original article’s thesis. It strikes me as an instance of the H L Mencken quote, “For every complex problem there is a solution which is clear, simple and wrong.”
I’m not sure the overall attitude has changed over the years. I suspect the nuance required for dealing with the problem of software longevity and legacy code is something that is currently mainly learned the hard way, rather than being taught. As such, many inexperienced practitioners will lack the awareness or tools to deal with it; combined with the rapid growth and thus younger-skewing demographics of the industry, I guess it means those with the requisite experience are in the minority. But has this situation really ever been different?
In any case, none of this is an argument against the thesis of the original text - you can certainly argue it’s a little vague (possibly because it’s a short excerpt from a book) and perhaps overly absolutist. (I’d argue the extent of the problem scales non-linearly with the size of the code on the one hand, and you can to some extent counteract it by proactive development practices.)
FWIW, as a contractor/consultant, I’d say the majority of my projects over the last years have been of the “we have this legacy code, the person/team who wrote it is/are no longer around” kind to some degree. My approach is definitely not to assume that I will never understand the existing code. In fact, I have found a variety of tactics for tackling the task of making sense of existing code. Again, I suspect most of these are not taught. But all of them are much less efficient than just picking the brains of a person who already has a good mental model of the code and the problem it solves. (It is fiendishly difficult to say with any reliability in retrospect whether it would have been cheaper to just start over from scratch on any such project. I do suspect it can shake out either way and depends a lot on the specifics.)
okay, I’ll take the alternate stance here. I worked on the back end of a large triple AAA video game that was always online. I worked on it for roughly 6 years before I moved to another company.
I have very good documentation, very clear objectives. It was very simple infrastructure - as simple as it could be made. The “why” of decisions was documented and weaved consistently into the fabric of the solution.
I hired into my new company! my successor. Expecting him to have experience with the same problems and my original infrastructure sought to solve.
he didn’t, he didn’t learn how or why certain things were how they were. my expectation of his ability to solve problems that I had already solved because he would’ve had experience with them was completely incorrect.
had the system failed catastrophically he would’ve been unable to fix it and that was not discovered even after working there for three years
Without these first-generation programmers, any software in danger of becoming unlearnable would necessarily be simplified or replaced.
I agree with your primary criticism–it is certainly true that software can be understood without the original creators.
However, your assessment of what will happen is very optimistic. It is entirely possible that what will happen is that new programmers will be brought in. They will only have time to make basic bug-fixes, which will be kludges. If asked to add new functionality, there will be copy paste. When they do try to buck the trend of increasing kludges, they will break things because they do not fully understand the software.
So I agree, any software should be understandable, but it will take investment in rebuilding a theory of how it works, and rewriting, or refactoring the software to make it workable for the new programmers. This will only happen if management understands that they have a lump of poorly understood software and trusts the developers to play the long game of improving the software.
The optimism is really just extended pessimism: I claim that, if you keep doing that, at some point all changes will break more than they fix, and either someone will take a hatchet to it or it will have to be abandoned.
It’s not that far off, only a little exaggerated. Yes, you can understand code you didn’t write, but you can’t understand it in the same way as one of its authors, until you’ve rewritten a chunk of it yourself. Yes, a team (or a solo developer) can maintain inherited software, but they’re going to have an adjustment period in which they’ll be inclined to “bolt-on” or “wrapper” solutions because they have trepidation about touching the core code. And it’s fair to say that that adjustment period ends, not after some period of staring at the code, but after making enough changes to it — not only that some part of it becomes their own, but that they run into enough challenges that the constraints that shaped the existing code start to make sense.
I wish I’d thought of this in my first comment, but the article is basically a long-winded way to say “the worst memory is better than the best documentation”. I’ll just leave that there.
but they’re going to have an adjustment period in which they’ll be inclined to “bolt-on” or “wrapper” solutions because they have trepidation about touching the core code
I can believe this happens sometimes but I don’t think it’s necessary. I’ve picked up legacy projects and within days made changes to them that I’d stand by today. Codebases take time to learn, and working on them helps, but finding one’s way around a new program, figuring out why things are the way they are, and building an intuition for how things should look, are all skills that one can develop.
Anyway I think even your version of the point largely refutes the original. Learning by doing is still just learning, not magic. In particular it doesn’t require an unbroken chain of acculturation. Even if the team behind some software all leaves at once, it’s not doomed.
I would also argue that in some cases the original authors of a program hold it back. The constraints that shaped the existing code aren’t always relevant decades down the track. Some the authors will simply be wrong about things. Removing the code from most of its context can be a good thing when it allows the project to go in a new direction. Also, especially for code that’s difficult to maintain… the original authors are the reason that is so—and as long as the chain of first-generation programmers remains intact, the path of least resistance to full facility with the code is to be trained to think like them. Breaking that local maximum might not be the worst thing.
Perhaps the problem with churn is that it’s not a clean break. You get an endless stream of second-generation programmers who try to build in the image of what came before, but always leave before they achieve mastery. I dunno.
I think it’s very accurate that the founders and early employees have the deepest knowledge of the system though. Yea, new people can come in and learn it, but it’s never quite to the same level. Anecdotally of course.
My mental model of this is that, roughly, the money you pour into a paid programmer accumulates. It’s not “linear”, but if a programmer goes, it takes away the accumulated value. I believe some accounting practices model similar stuff.
The converse also applies: if your company relies heavily on a piece of open-source software, hiring a core maintainer of (or experienced contributor to) that software onto your staff is probably a bargain.
…Web Browsers from ports that are patched to implement pledge(2) and unveil(8). […] FreeBSD 14.1, AFAIK, does not implement such feature.
I suppose the idiomatic method for securing processes on FreeBSD is capsicum(4). And at least for Firefox, it looks like someone has been working on adding support but they ran into some tricky cases that apparently aren’t well supported.
I guess for retrofitting huge, complex code bases, pledge and unveil or jails are probably easier to get working. I wonder how this affects things like file picker dialogs for choosing uploads, etc. If they’re implemented in-process, I guess they get to “see” the veiled or jailed file system - which means you don’t get any mysterious permission issues, but you also can’t upload arbitrary files. If they were implemented via IPC to some desktop environment process, the user could see the usual file system hierarchy to select arbitrary files, and if those files were sent back to the browser process as file descriptors, it would actually work. (I think the latter is how sandboxed apps are permitted to read and write arbitrary files on macOS, with Apple-typical disregard for slightly more complex requirements than just picking one file.)
I guess for retrofitting huge, complex code bases, pledge and unveil or jails are probably easier to get working.
In regard to pledge+unveil, it’s extremely simple to actually implement in code (though considerations for where+what in the code would be more complex) and the predefined promises make it pretty easy.
I wonder how this affects things like file picker dialogs for choosing uploads, etc.
For pledge+unveil, from memory, the dialog just cannot browse/see outside of the unveil’d paths. For browsers there’s a few files /etc/<browser>/unveil.* that lists the paths each kind of browser process is allowed access. Included here is ~/Downloads and common XDG-dirs for example, which allows for most file picking stuff to work fine for most users.
In regard to pledge+unveil, it’s extremely simple to actually implement in code (though considerations for where+what in the code would be more complex) and the predefined promises make it pretty easy.
One advantage is also that you can progressively enhance this over time. You can keep adding restrictions every time you fix instances of code which would previously have been violating a pledge. Capsicum would appear to require a more top-down approach. (I can’t help but wonder if you could add a kind of compatibility mode where it allows open() and similar as long as the provided path traverses a directory for which the process holds an appropriate file descriptor through which you’d normally be expected to call openat(). Or maybe that already exists, I really need to get hands-on with this one day.)
Included here is ~/Downloads and common XDG-dirs for example, which allows for most file picking stuff to work fine for most users.
I can see that the alternative would probably require a more holistic approach at the desktop environment level to implement well, but defining these directories statically up front seems like an awkward compromise. Various XDG directories contain precisely the kind of data you’d want to protect from exfiltration via a compromised process.
Capsicum lets you handle things like the downloads directory by passing a file descriptor with CAP_CREATE. This can be used with openat with the O_CREAT flag, but doesn’t let you open existing files in that directory. This is all visible in the code, so you don’t need to cross reference external policy files.
If you run a Capsicum app with ktrace, you can see every system call that Capsicum blocks, so it’s easy to fix them. With a default-deny policy and no access to global namespaces, it’s easy to write least-privilege software with Capsicum. I have not had that experience with any of the other sandboxing frameworks I’ve tried.
Capsicum lets you handle things like the downloads directory by passing a file descriptor with CAP_CREATE. This can be used with openat with the O_CREAT flag, but doesn’t let you open existing files in that directory. This is all visible in the code, so you don’t need to cross reference external policy files.
The download side is easier to handle as long as you’re keeping to a single download directory. I expect uploads to be somewhat more annoying UX wise: it’s rather unusual to have an “uploads” directory where the user would first copy any files they want to upload to a website, then select them in the “Browse…” dialog.
One slightly less annoying option is to agree on blanket read access to a bunch of stuff under $HOME, which appears to be what’s used here in practice, but it leaves you vulnerable to data exfiltration, which surely is one of the attack scenarios this whole exercise is trying to defend against.
Anything more comprehensive I can come up with will necessarily be a multi-process arrangement where the less-sandboxed process sends file descriptors of what the user selects to the heavily sandboxed one. And where drag & drop of a file sends (a) file descriptor(s) rather than just (a) path(s).
To be clear, I’m not saying this would be a bad system! I think it’d be great to have this in a desktop environment. Just that it’s a little tricky to retrofit onto a giant ball of code you’ve never even looked inside before.
If you run a Capsicum app with ktrace, you can see every system call that Capsicum blocks, so it’s easy to fix them. With a default-deny policy and no access to global namespaces, it’s easy to write least-privilege software with Capsicum. I have not had that experience with any of the other sandboxing frameworks I’ve tried.
That’s good to know - in contrast, dealing with the Sandbox on Apple’s platforms is super annoying as you’re mostly reduced to reading system log tea leaves when things aren’t working - macOS dtrace is falling apart more and more with every release and sometimes requires disabling the very security features you’re trying to debug.
But tracing only just begins to address the stated problem of retrofitting a large existing code base. If everything including dependencies including transitive ones is using open rather than openat, but open is completely non-functional, I suspect that might be rather a chore to get fixed. I mean it’s feasible if you actually “own” most of the project, but modifying something as big and unknown as a web browser in this way is quite an undertaking even if you can eventually get it all upstreamed.
The download side is easier to handle as long as you’re keeping to a single download directory. I expect uploads to be somewhat more annoying UX wise: it’s rather unusual to have an “uploads” directory where the user would first copy any files they want to upload to a website, then select them in the “Browse…” dialog.
Capsicum was designed to support this via the powerbox model (just as on macOS: it was designed to be able to more cleanly support the sandboxing model Apple was developing at the time). When you want to upload a file, the file dialog runs as a service in another process that has access to anything and gives file descriptors to selected files. You can also implement the same thing on top of a drag and drop protocol.
Alex Richardson did some Qt / KDE patches to support this and they worked well. Not sure what happened to them.
But tracing only just begins to address the stated problem of retrofitting a large existing code base. If everything including dependencies including transitive ones is using open rather than openat, but open is completely non-functional, I suspect that might be rather a chore to get fixed.
Alex Richardson did some Qt / KDE patches to support this and they worked well. Not sure what happened to them.
Good to know it’s been done and the code presumably is still out there somewhere. Something to keep note of in case I end up doing any UNIX desktop work. (And it sounds like this was done as part of an academic research project, so probably worth trying to get hold of any other published artifacts from that - perhaps part of this project?)
The nice thing about this is that open is a replaceable symbol. For example, in one project where I want to use some existing libraries in a Capsicum sandbox I simply replace open with something that calls openat with the right base depending on the prefix.
Providing this fallback compatibility wrapper as a user space libc override is a nifty technique, thanks for sharing!
I can see that the alternative would probably require a more holistic approach at the desktop environment level to implement well, but defining these directories statically up front seems like an awkward compromise.
I kind of agree. It did feel unintuitive to me at first. Worth noting these files are owned by root and normal users cannot write to them by default.
Various XDG directories contain precisely the kind of data you’d want to protect from exfiltration via a compromised process.
That’s right – from what I remember it was pretty well locked down though.
When I see the effort some people put, mostly on their spare time, to give people the ability to run extremely closed-source software onto extremely closed-source hardware, with FLOSS in the middle, I’m just amazed… in a good way.
I’m happy that they’re doing it, and I’m not… These people have my absolute respect.
What makes you call the hardware extremely closed source? I mean all the firmware is closed source of course but that’s the case for pretty much all hardware, what makes Mac hardware unusually closed source? It’s not especially locked down, all the mechanisms which are used to install and boot different operating systems are mechanisms intentionally included by Apple after all, it’s not like Asahi relies on some kind of jailbreaking
This might be ignorance from my part, I’m happy to be corrected.
But I’ve always assumed that Mac didn’t use any standard BIOS, and the new ones were using some non-standard UEFI boot process. My understanding was that, while it’s true that their custom firmware allow you to boot another OS, this is something your OS has to put effort into supporting. This is because Apple don’t care about you, as opposed to Intel/AMD/Asus/… who actively contribute to the Linux kernel for their hardware support. My understanding, as well, is that there is very little free documentation available on how to support Apple’s hardware, hence my “extremely closed source”.
Why else would we need the efforts of the Asahi team, if the hardware (which, I admit, I used as a metonymy for hardware+firmware) was standard?
The firmware and bootloader become fairly uninteresting once you’ve successfully booted the operating system, and while there are lots of fiddly bits to take care of to get things running smoothly, once you’re booted, you’re booted. (usually) Booting is not usually a herculean effort, as long as the hardware and firmware vendors haven’t tried to actively block you. Which Apple hasn’t, apparently. (Assuming secure boot is disabled.)
That said, Apple’s GPUs are entirely undocumented, so building a high-quality GPU driver for these things purely based on reverse-engineering (intercepting and tracing what I/O the macOS GPU driver performs on the GPU), especially in this kind of time frame is beyond impressive.
Other devices and SoC subsystems are also mostly undocumented of course and need their own drivers but the complexity of most devices is not in the same league as a GPU; USB is almost certainly a standard implementation though (perhaps with some smaller hacks) so that’s one big driver stack they won’t have to have reimplemented.
USB is almost certainly a standard implementation though
Correct! Most Apple peripherals are custom (though some have fun heritage like the Samsung UART, lol) but core USB is the same Synopsys DesignWare block you see in nearly everything (that’s not Intel nor AMD) these days.
That’s just the host/device controller though. To make it work you also need a USB PHY driver (Apple custom), a USB-PD controller driver (Partly incompatible Apple variant of an existing TI chip), and the I2C controller driver for the bus that chip talks through (which is actually a variant of an old PASemi controller).
Yeah, by “core” USB I meant the host/device controller :)
Frankly this is exactly why I like ACPI. With ACPI, all that glue-ish stuff around a generic core interface like XHCI mostly/hopefully can be written once in AML, instead of having to write and maintain drivers in each kernel. Which, yeah, from a Linux-centric perspective that last part is not a feature, but I personally like the BSDs for example, and I also hope that “the future” will be more Redox-shaped or something, so I’m always biased in favor of things that make OS diversity easier!
Heh, I was working on some low-level USB code on a microcontroller for a client. I’m staring the code and having this moment of deja vu “weird… I’ve never used this brand of chip before but for some reason a lot of this feels very familiar.”
I had worked on USB code before on different chips though. I opened up the datasheets from an old project and this new project and started looking at the USB registers. Sure enough, exactly the same names, exactly the same bit layout, exactly the same memory addresses. Google some of the names and find data sheets from multiple vendors and then find one from Synopsis. Sure helped debug things when I had my own reference implementation to compare against!
Booting other OS’s is intentionally supported, and doing so without compromising security for people who aren’t doing so is a significant amount of work. There’s a bunch of stuff describing how non-apple OS’s are installed without trivially undermining secure boot, and I think back in the early days of Asahi marcan talked about it a bunch.
Fortunately, heap allocation is not necessary to use C++ coroutines. However, C++20 coroutines do require dynamic allocation (memory that is allocated at runtime). Pigweed’s pw::async2::Coro API allocates memory using an pw::Allocator.
I don’t see a meaningful distinction between “heap allocation” and “dynamic allocation”, other than the scope of the allocator: whether it’s OS-provided and global, or custom and specific to some subsystem. Either way, it has the same effect.
What’s the benefit to using Pigweed’s custom allocator over the global operator new / malloc? If anything it’s a drawback, because in the case that multiple subsystems implement their own allocators, memory is wasted because each one is holding on to free space that the others can’t access.
Nevertheless, there’s a lot of interesting info here on the innards of C++ coroutines!
I don’t see a meaningful distinction between “heap allocation” and “dynamic allocation”, other than the scope of the allocator: whether it’s OS-provided and global, or custom and specific to some subsystem. Either way, it has the same effect.
A very meaningful distinction is that heap allocation, e.g. dynamic storage, will at some point call ::operator new, and has some chance of ending up doing a system call, which has implications in terms of latency for instance as this increases the chances for a thread to get scheduled-out. So far this was entirely precluding the use of coroutines in e.g. real-time audio processing loops unless you could assert at compile-time that HALO went into effect and every memory allocation got optimized out.
If anything it’s a drawback, because in the case that multiple subsystems implement their own allocators, memory is wasted because each one is holding on to free space that the others can’t access.
it’s definitely the only sane way to design some systems when you really want to make sure you aren’t ever going to take a lock.
There are a few reasons why someone might prefer a custom allocator over a global new or malloc.
In some cases, it’s a hard requirement: not all platforms provide a global allocator. This is fairly common in embedded systems.
Custom allocators can be beneficial for reliability, too– the exact amount of memory available for the purpose of allocating a particular coroutine or set of coroutines can be fixed, and failure to allocate from that dedicated pool of memory can be handled gracefully, rather than allowing the coroutine(s) to gradually expand into memory that is dedicated for other tasks.
Finally, using a custom allocator can sometimes improve performance by taking advantage of cache locality or certain allocation patterns. For example, bump allocators and arena allocators can provide more efficient allocation because they only deallocate at the end of certain scopes or lifetimes. This makes allocation just a simple pointer increment and makes deallocation a total no-op until the end of the arena’s lifetime.
You’re preaching to the choir! I agree custom heap allocators can be useful. But they’re still heaps. I found your article a bit misleading because you imply you’ve found a way to run C++coroutines without heap allocation, which you haven’t. You’ve just built your own heap.
Sorry for the misdirection! I didn’t intend to be misleading– this is an unfortunately common terminology mismatch. I’m using “heap allocation” to mean specifically the provided malloc/free/new/delete, and “dynamic allocation” to refer to the more general case of runtime memory allocation.
I do also discuss in this post why dynamic allocation is needed and how I hope its use can be avoided in the future, so I think it is still relevant to those hoping to avoid dynamic allocation completely (though I’m sad that it isn’t currently possible today).
In some cases, it’s a hard requirement: not all platforms provide a global allocator. This is fairly common in embedded systems.
But this requires you to bring along an allocator. And if you are going to bring along an allocator, why not just implement the global new and delete overloads?
The reason that I would want to avoid global new / delete on embedded platforms is that allocation is more likely to fail (because memory is tightly constrained) and so I need the caller to handle the failure case and I’m on an embedded platform that doesn’t support exceptions so it needs to handle allocation failure by the return address.
Your article mentions this about half way down:
Allow recovering from allocation failure without using exceptions.
From the API surface, it looks as if you provide a hook in the promise object that means that allocation failure can return a constant. This is really nice, you can have a coroutine call return something like an ErrorOr<T> and then return the error value if it fails to allocate space for the coroutine.
It would probably be nice to lead with that. I had to read a lot to figure out that you’ve actually solved the bit of the problem that I cared about.
Yes, handling the failure case gracefully is super important! RE ErrorOr<T>, Pigweed provides a pw::Result<T> which is exactly this– either a T or a pw::Status (our error code).
Coro<T> requires that T be convertible from pw::Status so that it can produce a value of T from pw::Status::Internal() in the case of allocation failure.
Custom allocators can be beneficial for reliability, too– the exact amount of memory available for the purpose of allocating a particular coroutine or set of coroutines can be fixed, and failure to allocate from that dedicated pool of memory can be handled gracefully, rather than allowing the coroutine(s) to gradually expand into memory that is dedicated for other tasks.
I’m going to flip this statement on its head for a sec. Using a fixed-sized arena allocator or something like that is great for having guarantees that you’re not going to blow up your heap. But… do we know at compile-time how big the allocations are going to be? Instead of using a fixed-sized arena, would it be possible to use something like a freelist allocator for each type that needs to get allocated? Like I could declare in my code that I’m going to assign a static allocation for 10 Coro objects and 30 frames; the allocator could return those from the freelist and return them to the freelist upon deallocation (both O(1) without needing to ever end the arena’s lifetime).
The size of the coroutine state will depend on the captured state (automatic variables at suspension points) so much like a lambda (closure type), its size is known by the compiler at compile time, but as far as I’m aware there’s no clear way to obtain it in your code at compile time, for example to be used as a template argument.
Yeah, this isn’t possible today, sadly– I discuss this a bit at the end of the post. If C++ offered a way to inspect the size of the coroutine of a specific function implementation, there would be no need for dynamic memory allocation: we could pre-reserve exactly the required amount of space. This would have huge benefits both for reliability and efficiency– no more need to guess at and adjust the right size for your allocators.
Little late getting back to this but I’ve had a little time to dig into some of the WGxx documents and I understand quite well why we can’t do this (yet anyway, hopefully someday!) For using it on an embedded system though, the HALO thing scares me a whole lot more than having to pre-reserve some space for an allocator (which also scares me less than having hidden/implicit malloc behind the scenes). On most of the smaller-sized embedded systems I work on, running out of space in a bounded arena and returning an error is a manageable thing. Changing the function that sets up a coroutine a little bit, resulting in HALO being available and significantly changing how much stack space is required… that’s spooky! At least with running out of arena space there’s well-defined error semantics. Walking off the end of a fixed-sized stack usually just means corruption.
I’m still digging a little more… it seems like there is a way to ensure HALO is disabled but having it potentially ok by default seems like a footgun for embedded, and we’ve got lots of footguns already!
(I’m still super intrigued through. This looks super cool!)
AFAIR the issue is that the size of the coroutine is only known by the compiler after optimization passes (as it depends on how local variables are stored, which lifetimes overlap or not, what can be reused) but the sizeof(...) operator is typically implemented in the frontend. The compiler writers said they would not be able to implement a coroutine proposal with frontend known size without heroic efforts. This was discussed extensively during the design of C++ coroutines, including a competing “core coroutine” proposal appearing and ending up dying on this same issue. Some alternatives were explored, such as types that did not support sizeof(…) but still could be stored on the stack, but it the end storing the coroutine on a heap with an escape hatch for ellision was deemed the best compromise.
A more general case of the example above would be if a were a variable-length array with x as the size. I would absolutely never write such a thing in embedded code (and typically not in normal C++ if there’s a chance an attacker can influence the savour of `x‘) but it is permitted.
Yeah I wasn’t thinking about the general case too much, just the specific example, and assuming the statically sized array was the point that seemed problematic. You’re right it’s not always possible… in most languages.
Rust does know the size of each Future, but it doesn’t let you have dynamically sized values on the stack (on stable). And using that unstable feature, it’s not possible to hold a dynamically sized value across await points (#61335) as await points are where the size needs to be known to be stored in the generated Future’s enum variant.
“Safe memory reclamation” is special jargon in this context.
If you are using RCU, hazard pointers, or other similar lock-free techniques, then a write to a datastructure cannot immediately free() any memory that was removed, because the memory can still be in use by concurrent readers. You have to wait for a grace period or quiescent period to pass before it can be freed. You don’t want to do this as part of the write transaction because that will ruin your write throughput: grace periods are relatively long and can accommodate multiple write transactions.
So you need to hang on to the to-be-freed memory somehow and reclaim it later. Sometimes the whole concurrent algorithm is easier if you guarantee type-stable memory by using a region of some kind. Sometimes you can save significant amounts of memory (a pointer per node) by using a GC instead of a to-be-freed list.
I’m not sure how custom allocators help here. They are not invoked until after the destructor has run, so prolonging the lifetime of the allocation using a custom allocator doesn’t help because the object is gone (you may get an arbitrary bit pattern if you try to read it). Things like RCU work fine without a custom allocator because they defer destruction, not deletion.
They are not invoked until after the destructor has run,
Note that C++20 adds “destroying delete” which lets you define allocators that take care of the destruction themselves. But as linked article says it’s not standardized for the allocator methods on coroutine promise types >_<
Interesting. The Solaris VMEM allocator gets some really nice performance benefits by resetting deallocated objects to a partially initialised state (allocation often happens on hot paths, deallocation is easy to defer. You can defer not needing an object far more easily than you can defer needing one). I believe that could be represented in C++20 now, because delete would not call the destructor and would just add the object to the pool, leaving it as a valid object of the correct type.
I don’t know how you can do things like data-structure-specialized GC or type-stable memory without a custom allocator. You don’t need those techniques for RCU (RCU was an example to set the scene) but they are sometimes useful for concurrent data structures. I also dunno how this would fit with the C++ facilities for custom allocators — I guess you have to avoid destruction until the memory is about to be freed (especially for type-stable memory)
Type-stable memory can be done with custom allocators, but it’s often better to do it with a class overriding operators new and delete. You can use CRTP to implement a superclass that provides a custom memory pool for each type.
I could be wrong but my recollection is that they use alloca which allocates arbitrary stack space instead of using actual allocators (which typically involve a system call in the worst case).
We have two main groups: this one focused on a more classic approach to writing embedded software, and another focused on using the chip with asynchronous Rust and the Embassy framework.
Can someone explain the difference between the “classic” and async/Embassy approach?
I suspect it is sync vs async, but I suspect the classic would be higher level since it presumably has an embedded RTOS underneath it providing the sync abstraction, while Embassy presumably has a thinner runtime and leverages Rust’s async to support multitasking.
So how does one do multitasking with classic? Like how do you run a bluetooth or wifi module while also doing the other things your application needs to do?
Independent pieces of hardware do stuff in the background when you’re not interacting with them, so you either need to check in every so often manually, or use interrupts. A common pattern is to have a big loop that goes around doing everything that needs to be done, and then sleeps when there’s nothing to be done, to be waken up by an interrupt or something else. You might write a bunch of state machines that you drive forward when the correct events come in.
That’s why async is so exciting for embedded. You can write what looks and feels like abstracted task code for an RTOS, but it compiles into a fairly predictable event loop with state machines, just like what many people write by hand.
That makes sense. When I’ve dabbled with embedded programming the “classic” approach usually involved pretty low-level management of network peripherals—you had to know the state machine for each peripheral and how to handle its events and then compose that state machine with the state machines of other peripherals and your application and it was always really painful especially if you misunderstood something about a peripheral’s state machine. Embassy looks more or less like what I’ve been wanting—I can’t wait to try it out.
A few of my own patches made it into this release!
Modernised Hypervisor.framework (HVF) API usage on x86-64 macOS hosts, which gives double-digit percentage point performance improvements on most workloads.
Minor fix to handling of translucent mouse cursors via VNC.
Unfortunately a bunch of others so far haven’t been accepted, I‘ll try again for the 9.2 cycle:
x2apic support when using HVF, which yields double-digit percentage perf gain for VMs with more than 1 core. (Patch has been entirely ignored in the mailing list despite being very simple.)
3D graphics acceleration with macOS guests. (Rejected because you currently can only use it with x86-64 guests on x86-64 hosts; suggestion was I include patches which will allow arm64 macOS guests to boot in Qemu. I’m currently stuck on XHCI not working correctly with those due to a suspected interrupt issue. If anyone knows how MSIs are supposed to work with PCIe devices on arm64 I’d love to know.)
Passing through guest pointers to native host mouse cursors on macOS hosts using the Cocoa UI. (Diverging opinions on whether this is desired behaviour.)
I’ve also been working on support for HVF’s in-kernel APIC support, but that turns out to be buggy, so I’m currently trying to figure out if there’s some baseline feature set that does work reliably…
I don’t quite get the appeal of the textual prompt about discarding the key. Overlay text breaks the immersion to some extent. If it really is useless now and there’s some emotional backstory, surely the player character could for example convey the emotion through gesture? Shake their fist and fling the key away? (I don’t really go for horror games, so haven’t played RE2 and don’t know what sense of relief the author is alluding to here.)
I liked Far Cry 2’s approach to the map. The first-person player character literally pulls out a map from somewhere and holds it in his¹ hand alongside the handheld GPS device. The interesting bit here is that the game keeps going on around you. You can run around while holding the map, even though it’s obscuring most of your field of view. If you encounter baddies (or they encounter you) while you’re staring at the map, they’ll quite happily start shooting you while you’re still fumbling to grab your weapon.
The same mechanic works while you’re driving too. Much as driving while holding up a map in one hand is rather hazardous in real life, the game tends to punish you for doing this as well, as tempting as it frequently is.
The rest of the HUD is similarly understated and disappears most of the time, though a health bar and number of health syringes will show up when you’ve taken damage. (And IIRC the screen will momentarily be tinted red when taking a hit.)
Later games in the series started cluttering the screen a lot more unfortunately. (They also undeniably tidied up some of the more unpolished game mechanics, but in my opinion never quite recreated the atmosphere of danger that Far Cry 2 conveyed. Nor the moral ambivalence.)
¹ Unfortunately, all playable characters are male, even though key NPCs you meet in the game will be chosen from the pool of player characters plus a woman who for some reason is not playable.
the relief from the player character comes from the fact that in RE/2 you have very limited inventory space. Even just carrying a key around is a whole ordeal, and when you open a door and still have to hold onto the key, that means that you have new challenges ahead, and you don’t even get an extra inventory slot (which you could use to, for example, pick up healing items).
There are absurd situations you can end up in where you have no free inventory slots, but no healing items, and are hurt, and there is a healing item right in front of you and you can’t use it. It’s fodder for a lot of jokes, but it’s also an amazing source of tension.
None of this really makes me a better programmer, but it’s fun to think about.
I love those sorts of things, depending on the game. Helldivers has an interesting middle-ground to this; there’s a HUD, but it’s quite minimal and digging for more information tends to have real consequences. Your weapon has no targeting reticle, unless you actually aim down the sights of it. The ammo counter is a numberless “full/kinda full/empty” bar, unless you bring up the context menu that lets you switch weapon modes and such; only then does it say “23/30 rounds”. You have a local minimap, but zooming out on it to see the full map means your character literally looks down at the PDA strapped to their arm and stops aiming their weapon at the horrible bug-monsters coming to claw out their eyeballs. In general you can be fighting for your life or figuring out what the hell you’re doing with your life, but not both.
They do a good job of using semi-artificial inconveniences like that to add to the gritty dystopia of the game, and to make the decision making and frenetic pacing more hectic. This now makes me wonder what the game might look like if even more of the information in it were implicit rather than given in the HUD.
By the way, the term that describes a UI element like, “You pull a map out and the game doesn’t pause” where it’s internal to the world is “diegetic.” I’d go so far as to say that diegetic UI elements are always superior to virtual ones. The post makes mention of how Isaac’s suit in Dead Space shows his health on the spine, which is also a diegetic UI element.
diegetic UI elements are always superior to virtual ones
I think “superior” is doing a lot of work in that sentence. Diagetic elements are always more immersive, but not every game is about being as immersive as possible.
Imagine in real life if you could have a map that stopped time when you used it and filled your entire field of vision, you would probably find that a lot more useful than a paper map! So I don’t think it’s as simple as saying diagetic is better every time.
Diegetic doesn’t always mean “time doesn’t stop when you look at a map” so much as, “the map is a fundamental part of the world rather than an arbitrary menu.”
This blog announces or reviews a ton of SBCs and micro-PCs, but I thought this one in particular was interesting and might be newsworthy here, as it’s basically “Raspberry Pi 5 but it’s x86, can run Windows, and btw supports fast SSDs.” A cute detail is that there’s a RPi2400 chip on board to drive the GPIO pins.
That’s the usual way to upload firmware to an RP2040 and not specific to this board. As far as I’m aware this functionality is part of the bootrom inside the RP2040 and completely hardwired.
(Incidentally, I’ve previously used an RPi Pico as a kind of GPIO breakout board in an x86 SFF PC based custom appliance as well. They are so small, and the decent USB support makes it a good choice.)
In-line as in, it shows up as a USB mass storage device connected to the x86 computer? Yes, that’s what I’d expect. The RP2040’s USB port must surely be wired directly to a USB 2.0 host port on the x86 SoC, so it’ll show up as whatever type of USB device the code running on the RP2040 exposes. In the case of the naked bootrom with BOOTSEL held, this is a USB MSC. With the stock firmware, it would presumably show up as some kind of GPIO-controlling USB device. (HID?) That’s assuming it even ships with any kind of useful firmware on the RP2040. (The “documentation” link on Radxa’s X4 page seems broken, I’ve not researched beyond that.) With TinyUSB, you can program it to expose whatever USB interface you like.
The tricky bit would be resetting or power cycling the RP2040 in isolation without having to power down the host SoC. The “BOOTSEL” button on an RPi Pico normally only has an effect during very early startup, before the bootrom has loaded the current firmware. Perhaps the GPIO connector exposes the RUN pin (30) which can be used to hard reboot an RP2040. (On my SFF PC based build I run both BOOTSEL and RUN contacts to externally accessible push buttons for exactly this reason.) Another option is to have a software reset trigger in your RP2040 firmware. This’ll only work if you didn’t make any mistakes in coding your firmware. (Now who would do such a thing. 😅) Otherwise you’d have to power cycle it, which presumably means power cycling the whole board.
I received an email saying that I was eligible for a new crypto offering that was targeting open-source developers. They claimed they had checked my GitHub username, and that I was in. They offered $200 to do it for me, all I needed to do was send them the private SSH key associated with my GitHub account, so they could claim the reward. One important detail is that it had to be the private SSH key associated with my account at the time the contract was added to the blockchain, so I couldn’t just make a new temporary one.
I investigated it and the crypto distribution was real. I was able to claim it myself, and I sold the tokens as soon as I was able, for almost $3000.
I did! But by doing it myself I didn’t have to send anyone my private SSH key, I just had to sign a challenge token with it, which they verified with the public key that they had collected from GitHub.
The process was still sketchy, they had a Docker container that you would have to run. The container would read the private SSH key (which had to be mounted to the container) and run a program to generate the response token. They did recommend running the container without a network attached, and I turned off my wifi just in case as well. The container would die after printing out the response token, so I think it was pretty safe.
That’s pretty nuts. I have to ask: what’s in it for them? And why the super unsafe delivery if it’s some altruistic cause?
I’ve been receiving those emails as well and assumed they were scams aimed at injecting malicious code into my github repos or exfiltrating code from private repos. I have zero practical experience with cryptocurrency, so I guess I’ll let this “opportunity” slide as it sounds like more hassle and risk than it’s worth even if it’s not a scam.
I think they gain by creating traction among developers, that would be my guess. And since they created the token when they give it away they’re not losing money, it’s all fictitious anyway. :)
They seem to assume that the only reason packets ever drop because the link is full of traffic. Besides the fact that dropping packets in collisions is part of Ethernet, you will get dropped packets sooner or later any time you use wifi, cellular internet, damaged/degraded wired networks, long-distance wired networks, or wired networks that wander near nasty sources of interference. So, you know, always. TCP does a great job of covering this up when a) latencies are low, and b) ~99% of packets still get through. (Tried to find the article that graphed this but I can’t; I think it was from Google or Cloudflare?)
QUIC is pretty good, definitely use it in 99% of the places you’d normally use UDP. But the question is not “good data” vs “bad data”, it is “mostly-good data now” vs “perfect data maybe half a second from now”. Especially for video telecoms and stuff like that, a garbled frame or two and some weird distortion for a few seconds is better than someone just cutting off and waiting a painful few seconds for TCP and the video/audio codec to get their collective shit together before you get to say “sorry you cut out there, can you repeat that?”
Modern wireless standards do an impressive amount of forward error correction and so will provide an abstraction that looks like a reliable link even with a lot of frames being dropped. This is critical for satellite links, where a retransmit is hundreds of ms of latency, but is increasingly important for WiFi at high data rates. The faster your link speed, the more data you need to buffer. 802.11bn is supposed to scale to 100 Gb/s. If it takes 1 ms to retransmit, you’d need a 100 Mb buffer to avoid slowdown for a single dropped packet (or you’ll have a smaller window and so get back pressure and slow down). If you have enough error correction, you just get the frames arriving slightly out of order once you’ve received enough to reconstruct the dropped one.
I think modern wireless protocols dynamically adapt the size of their error correction codes when they detect loss, so your speed just gracefully degrades in most cases, but you don’t see huge latency spikes.
Interesting, isn’t that wasteful when a higher layer is ok with some packet loss? But OTOH if the higher level is doing its own retries, and the least reliable link is the first one, it makes sense to retry early where the latency cost is lower.
Maybe the link should do a small number of retries, to get a good balance of latency and reliability.
Not a deep expert on this, but I seem to remember TCP assumes packet loss is down to bandwidth limitation or congestion, and will respond by reducing the transmission rate right down, only slowly ramping back up. The typical packet loss scenario in WiFi is that there‘s some momentary interference, or the geometry of the link has changed, so the peers need to switch to less sensitive QAM encodings, etc. Until that happens, a whole batch of packets will have been dropped, which TCP interprets as catastrophic bandwidth reduction even if the wireless link recovers very quickly. So the higher layer doesn’t cope very well with the specific pattern of loss.
The conclusion of the article surprised me: it recommends a fairly complicated collection of mechanisms. I thought QUIC was going to get support for lossy streams so that it could avoid creating a backlog of wasted work when the network conditions suddenly change.
The ideals of this post are dead. Firefox is neither private nor free. Do not use Firefox in 2025.
Mozilla has done an about face and now demands that Firefox users:
See https://lobste.rs/s/de2ab1/firefox_adds_terms_use for more discussion.
If you’re already using Firefox, I can confirm that porting your profile over to Librewolf (https://librewolf.net) is relatively painless, and the only issues you’ll encounter are around having the resist fingerprinting setting turned on by default (which you can choose to just disable if you don’t like the trade-offs). I resumed using Firefox in 2016 and just switched away upon this shift in policy, and I do so sadly and begrudgingly, but you’d be crazy to allow Mozilla to cross these lines without switching away.
If you’re a macOS + Littlesnitch user, I can also recommend setting Librewolf to not allow communication to any Mozilla domain other than addons.mozilla.org, just in case.
👋 I respect your opinion and LibreWolf is a fine choice; however, it shares the same problem that all “forks” have and that I thought I made clear in the article…
Developing Firefox costs half a billion per year. There’s overhead in there for sure, but you couldn’t bring that down to something more manageable, like 100 million per year, IMO, without making it completely uncompetitive to Chrome, whose estimate cost exceeds 1 billion per year. The harsh reality is that you’re still using Mozilla’s work and if Mozilla goes under, LibreWolf simply ceases to exist because it’s essentially Firefox + settings. So you’re not really sticking it to the man as much as you’d like.
There are 3 major browser engines left (minus the experiments still in development that nobody uses). All 3 browser engines are, in fact, funded by Google’s Ads and have been for almost the past 2 decades. And any of the forks would become unviable without Apple’s, Google’s or Mozilla’s hard work, which is the reality we are in.
Not complaining much, but I did mention the recent controversy you’re referring to and would’ve preferred comments on what I wrote, on my reasoning, not on the article’s title.
I do what I can and no more, which used to mean occasionally being a Firefox advocate when I could, giving Mozilla as much benefit of the doubt as I could muster, paying for an MDN subscription, and sending some money their way when possible. Now it means temporarily switching to Librewolf, fully acknowledging how unsustainable that is, and waiting for a more sustainable option to come along.
I don’t disagree with the economic realities you mentioned and I don’t think any argument you made is bad or wrong. I’m just coming to a different conclusion: If Firefox can’t take hundreds of millions of dollars from Google every year and turn that into a privacy respecting browser that doesn’t sell my data and doesn’t prohibit me from visiting whatever website I want, then what are we even doing here? I’m sick of this barely lesser of two evils shit. Burn it to the fucking ground.
I think “barely lesser of two evils” is just way off the scale, and I can’t help but feel that it is way over-dramatized.
Also, what about the consequences of having a chrome-only web? Many websites are already “Hyrum’s lawed” to being usable only in Chrome, developers only test for Chrome, the speed of development is basically impossible to follow as is.
Firefox is basically the only thing preventing the most universal platform from becoming a Google-product.
Well there’s one other: Apple. Their hesitance to allow non-Safari browsers on iOS is a bigger bulwark against a Chrome-only web than Firefox at this point IMO.
I’m a bit afraid that the EU is in the process of breaking that down though. If proper Chrome comes over to iOS and it becomes easy to install, I’m certain that Google will start their push to move iOS users over.
I know it’s not exactly the same but Safari is also in the WebKit family and Safari is nether open source nor cross platform nor anywhere close to Firefox in many technical aspects (such as by far having the most functional and sane developer tools of any browser it there).
Pretty much the same here: I used to use Firefox, I have influenced some people in the past to at least give Firefox a shot, some people ended up moving to it from Chrome based on my recommendations. But Mozilla insists on breaking trust roughly every year, so when the ToS came around, there was very little goodwill left and I have permanently switched to LibreWolf.
Using a fork significantly helps my personal short-term peace of mind: whenever Mozilla makes whatever changes they’re planning to make which requires them to have a license to any data I input into Firefox, I trust that I will hear about those changes before LibreWolf incorporates them, and there’s a decent chance that LibreWolf will rip them out and keep them out for a few releases as I assess the situation. If I’m using Firefox directly, there’s a decent probability that I’ll learn about those changes after Firefox updates itself to include them. Hell, for all I know, Firefox is already sending enough telemetry to Mozilla that someone there decided to make money off it and that’s why they removed the “Mozilla will doesn’t and will never sell your data” FAQ item; maybe LibreWolf ripping out telemetry is protecting me against Mozilla right now, I don’t know.
Long term, what I personally do doesn’t matter. The fact that Mozilla has lost so much good-will that long-term Firefox advocates are switching away should be terrifying to Mozilla and citizens of the Web broadly, but my personal actions here have close to 0 effect on that. I could turn into a disingenuous Mozilla shill but I don’t exactly think I’d be able to convince enough people to keep using Firefox to cancel out Mozilla’s efforts to sink their own brand.
If Firefox is just one of three browsers funded by Google which don’t respect user privacy, then what’s the point of it?
People want Firefox and Mozilla to be an alternative to Google’s crap. If they’re not going to be the alternative, instead choosing to copy every terrible idea Google has, then I don’t see why Mozilla is even needed.
Well to be fair to Mozilla, they’re pushing back against some web standard ideas Google has. They’ve come out against things like WebUSB and WebHID for example.
How the heck do they spend that much? At ~20M LoC, they’re spending 25K per line of code a year. While details are hard to find, I think that puts them way above the industry norms.
I’m pretty sure that’s off by 3 orders of magnitude; OP’s figure would be half a US billion, i.e. half a milliard. That means 500M / 20M = 25 $/LOC. Not 25K.
I see your point, but by that same logic, shouldn’t we all then switch to Librewolf? If Firefox’s funding comes from Google, instead of its user base, then even if a significant portion of Firefox’s users switch, it can keep on getting funded, and users who switched can get the privacy non-exploitation they need?
I gathered some numbers on that here: https://untested.sonnet.io/notes/defaults-matter-dont-assume-consent/#h-dollar510000000
TL;DR 90% of Mozilla’s revenue comes from ad partnerships (Google) and Apple received ca. 19 Bn $ per annum to keep Google as the default search engine.
Where did you get those numbers? Are you referring to the whole effort, (legal, engineering, marketing, administration, etc) ot just development?
That’s an absolutely bonkers amount of money, and while i absolutely believe it, im also kind of curious what other software products are in a similar league
doesn’t seem like a particularly grave concern to me
That page says “Services”. Does it apply to Firefox or the VPN?
The sexuality and violence thing I suspect is so that they are covered for use in Saudi Arabia and Missouri.
Yeah, that seems like legal butt-covering. If someone in a criminalizing jurisdiction accesses these materials and they try to sue to the browser, Mozilla can say the user violated TOS.
i assume it applies mostly to Bugzilla / Mozilla Connect / Phabricator / etc
This is just a lie. It’s just a lie. Firefox is gratis, and it’s FLOSS. These stupid paragraphs about legalese are just corporate crap every business of a certain size has to start qualifying so they can’t get their wallet gaped by lawyers in the future. Your first bullet point sucks - you don’t agree to the Acceptable Use Policy to use Firefox, you agree to it when using Mozilla services, i.e. Pocket or whatever. Similarly, your second bulletpoint is completely false, that paragraph doesn’t even exist:
The text was recently clarified because of the inane outrage over basic legalese. And Mozilla isn’t selling your information. That’s not something they can casually lie about and there’s no reason to lie about it unless they want to face lawsuits from zealous legal types in the future. Why constantly lie to attack Mozilla? Are you being paid to destroy Free Software?
Consciously lying should be against Lobsters rules.
Let’s really look at what’s written here, because either u/altano or u/WilhelmVonWeiner is correct, not both.
The question we want to answer: do we “agree to an acceptable use policy” when we use Firefox? Let’s look in the various terms of service agreements (Terms Of Use, Terms Of Service, Mozilla Accounts Privacy). We see that it has been changed. It originally said:
“When you upload or input information through Firefox, you hereby grant us a nonexclusive, royalty-free, worldwide license to use that information to help you navigate, experience, and interact with online content as you indicate with your use of Firefox.”
Note that this makes no distinction between Firefox as a browser and services offered by Mozilla. The terms did make a distinction between Firefox as distributed by Mozilla and Firefox source code, but that’s another matter. People were outraged, and rightfully so, because you were agreeing to an acceptable use policy to use Firefox, the binary from Mozilla. Period.
That changed to:
“You give Mozilla the rights necessary to operate Firefox. This includes processing your data as we describe in the Firefox Privacy Notice. It also includes a nonexclusive, royalty-free, worldwide license for the purpose of doing as you request with the content you input in Firefox. This does not give Mozilla any ownership in that content.”
Are the legally equivalent, but they’re just using “nicer”, “more acceptable” language? No. The meaning is changed in important ways, and this is probably what you’re referring to when you say, “you don’t agree to the Acceptable Use Policy to use Firefox, you agree to it when using Mozilla services”
However, the current terms still say quite clearly that we agree to the AUP for Mozilla Services when we use Firefox whether or not we use Mozilla Services. The claim that “you don’t agree to the Acceptable Use Policy to use Firefox” is factually incorrect.
So is it OK for u/WilhelmVonWeiner to say that u/altano is lying, and call for censure? No. First, it’s disingenuous for u/WilhelmVonWeiner to pretend that the original wording didn’t exist. Also, the statement, “Similarly, your second bulletpoint is completely false, that paragraph doesn’t even exist:” is plainly false, because we can see that paragraph verbatim here:
https://www.mozilla.org/en-US/about/legal/terms/firefox/
So if u/WilhelmVonWeiner is calling someone out for lying, they really shouldn’t lie themselves, or they should afford others enough benefit of the doubt to distinguish between lying and being mistaken. After all, is u/WilhelmVonWeiner lying, or just mistaken here?
I’m all for people venting when someone is clearly in the wrong, but it seems that u/WilhelmVonWeiner is not only accusing others of lying, but is perhaps lying or at very least being incredibly disingenuous themselves.
Oh - and I take exception to this in particular:
“every business of a certain size has to start qualifying so they can’t get their wallet gaped by lawyers”
Being an apologist for large organizations that are behaving poorly is the kind of behavior we expect on Reddit or on the orange site, but not here. We do not want to or should we need to engage with people who do not make good faith arguments.
This is a pretty rude reply so I’m not going to respond to the specifics.
Mozilla has edited their acceptable use policy and terms of service to do damage control and so my exact quotes might not be up anymore, but yeah sure, assume that everyone quoting Mozilla is just a liar instead of that explanation if you want.
EDIT:
https://blog.mozilla.org/en/products/firefox/update-on-terms-of-use/
Sorry for being rude. It was unnecessary of me and I apologise, I was agitated. I strongly disagree with your assessment of what Mozilla is doing as “damage control” - they are doing what is necessary to legally protect the Mozilla Foundation and Corporation from legal threats by clarifying how they use user data. It is false they are selling your private information. It is false they have a nonexclusive … license to everything you do using Firefox. It is false that you have to agree to the Acceptable Use Policy to use Firefox. It’s misinformation, it’s FUD and it’s going to hurt one of the biggest FLOSS nonprofits and alternate web browsers.
So people can judge for them selves, the relevant quote from the previous Terms of Use was:
Source: http://archive.today/btoQM
The updated terms make no mention of the Acceptable Use Policy.
This is a pretty incendiary comment and I would expect any accusation of outright dishonesty to come with evidence that they know they’re wrong. I am not taking a position on who has the facts straight, but I don’t see how you could prove altano is lying. Don’t attribute to malice what can be explained by…simply being incorrect.
that’s not binding to firefox. that’s binding to mozilla services like websites and other services. https://www.mozilla.org/en-US/about/legal/terms/mozilla/ links to the acceptable use page for instance. whereas the firefox one does not. https://www.mozilla.org/en-US/about/legal/terms/firefox/
firefox is fine. your other points are also largely incorrect.
FYI this is a change made in response to the recent outrage, the original version of the firefox terms included
Which has now been removed.
What are the trade-offs for resisting fingerprinting? Does it disable certain CSS features, or?
Your locale is forced to en-US, your timezone is UTC, your system is set to Windows. It will put canvas behind a prompt and randomizes some pixels such that fingerprinting based on rendering is a bit harder. It will also disable using SVG and fonts that you have installed on your systems
Btw, I don’t recommend anyone using resist fingerprinting. This is the “hard mode” that is known to break a lot of pages and has no site-specific settings. Only global on or off. A lot of people turn it on and then end up hating Firefox and switching browsers because their web experience sucks and they don’t know how to turn it off. This is why we now show a rather visible info bar in settings under privacy/security when you turn this on and that’s also why we are working on a new mode that can spoof only specific APIs and only on specific sites. More to come.
Now that I know about it, I’m really looking forward to the new feature!
I’m using CanvasBlocker but its performance and UX could use some love.
This is the kind of thing Mozilla still does that sets it very far appart from the rest. Thanks!
heh, I wonder how many bits of entropy will be there in roughly “which of the spoofs are enabled”? :D
Yes, if everyone is running a custom set of spoofs you’d end up being unique again. The intent for the mechanism is for us to be able to experiment and test out a variety of sets before we know what works (in terms of webcompat). In the end, we want everyone to look as uniform as possible
It breaks automatic dark mode and sites don’t remember their zoom setting. Dates are also not always localized correctly. That’s what I’ve noticed so far at least.
If you have
clang-tidyavailable, there’s a lint to prevent assignments inifconditionals. Swapping the operands just looks too uncanny to me and you have to convince everyone to do it.Both GCC and clang have warnings for the assignment in conditional mistake, and I believe they’re included in
-Wextra. There really isn’t any need for this awkwardness.One exception: I use reversed comparisons when the left side would otherwise be very long and potentially make the comparison hard to spot.
Interesting.
So what is actually wrong with a program that has an Rc live across wait points? As in if we disable static checks and just ran the program (like a C compiler) could there ever be thread contention and undefined behaviour? It seems like only one thread would use the Rc at a time… so an Arc seems like overkill at first blush.
Can the cores of a CPU really be so out of sync that the refcount set from before the work stealing is not reflected correctly in the core that picks up the work? How would that happen? Maybe If it was stealing it back and had an earlier version of the future in cache? If so, you’d have similar issues with the other plain old data in the future… so it can’t be that.
I can’t tell if this is a compiler false positive (completeness issue) or saving us from actual UB.
The compiler simply enforces what the type promises, and in this case
Rcsays that it isn’t safe to send elsewhere. You could have some kind ofRcSendabletype constructible from anRcwith a refcount of one, or you could have some kind of structure containingRcs that guarantees that they can’t be leaked to some other part of the program, and have them beSend, but makingRcitselfSendin a limited set of circumstances would be difficult, for questionable gain.Keep in mind that it’s impossible to make a compiler that allows all correct programs but rejects all incorrect programs. So since Rust wants to reject all programs that have UB, it must also reject some programs that don’t have UB. Efforts are ongoing to increase the number of correct programs that Rust allows, but adding special logic to the compiler to allow fresh
Rcs to beSendseems not worth it.The problem is elsewhere.
An Rc automatically frees its contents. It uses a refcount which is adjusted when the Rc is cloned or dropped. If Rc were Send then you could clone it into multiple threads. The refcount adjustments don’t use atomic instructions so they are likely to go wrong and cause use-after-free errors.
If a send an Rc that I own to another thread, won’t it be moved (neither cloned nor dropped - the refcount stays constant)?
(And then my question was if every last clone of a given Rc was sent/moved as part of a single owned value, in this case a future, to another thread, mightn’t that be technically valid?)
You can’t statically prove it’s the last reference though, so the type system has to disallow the general case where it might not be the last reference. If you only need the one reference, perhaps don’t use an
Rc?As an aside, it would be nice to have functions on
Rc<T>/Arc<T>to go between the two types, but only if their reference count is 1.That would avoid the need to reallocate the inner value. It seems like their current implementation has exactly the same memory representation (with the small exception that
AtomicUsizecan have more stringent alignment requirements thanusize, although it should be trivial to just make sure theRcInnerstruct is aligned properly)I believe system allocators frequently align to at least 8 bytes, which means that in practice the
RcInnershould already end up aligned suitably forArcInner.Given that, you could implement this yourself. This assumes of course that
RcInnerandArcInnerdon’t ever change layouts (or if they do, that the layouts stay identical).That said, this is a very niche use-case.
It’s pessimistic because it can’t prove at compile time that those things are safe at run time.
Moving an Rc across threads isn’t necessarily the problem, it’s what happens to the Rc before and after, how it is shared.
You definitely can have “memory value is V0, CPU 1 writes V1, CPU 2 reads V0”, and you’re exactly right that applies to any memory location.
If you want to ensure writes made by one CPU are visible to another with certainty you need to issue instructions for that. Otherwise your write may be in a cache or memory queue not visible to the other CPU, or your read may be from a stale cache
As far as I remember, Rust targets some abstract common denominator of existing memory models. This allows the compiler to make valid choices while checking the higher level code (what LLVM does on the lower level is highly platform specific though). That memory model is quite conservative, thus the errors. What would happen in reality for the potentially racy construct is sort of UB as it’s naturally not defined :)
So any value that is
Sendand that is moved to another thread might have some instructions added so that it is read coherently?That sounds fine in this specific case so long as the same instructions were applied to
RcInner. But that is pointed to with aNonNull, for which the compiler doesn’t want to mess with and which isn’tSend…Am I on the right track?
You make this sound like it’s automatic. The guts of
Rcuse raw pointers, which aren’tSend, thereforeRcis notSend. If you take something likeArc, the guts are also notSend, butArc(unsafely)implementsSendexplicitly as a manual promise.Arc’s methods are manually coded so that the overall type behaves in an atomic/coherent way.The whole point of
Rcis that it doesn’t go to all that trouble, which has a cost as well, but it still allows an object to be referenced from multiple locations which are accessible to only one thread. (Including objects of types which themselves are notSend.) Yes you could modifyRcto implementSend. Congratulations, you implementedArc.Many primitive types such as
i32are alsoSend, and so are types derived from them. Not because the compiler inserts any special instructions or something, but because the mechanism by which it’s moved from one thread to another is assumed to be safe (e.g. channel, mutex, etc. - safe here usually meaning it uses a memory barrier of some kind), and there’s nothing about the type itself that needs special treatment.Is this really that much of a downside? I can’t imagine publishing to be something that is done often enough to warrant concern about this.
It’s probably an issue for companies that publish private packages many times a day from CI.
Is a marginal slowdown really that important in CI, as opposed to something being run on a laptop where the developer is interactively waiting for
npm publishto complete?(To be fair, I’m kind of just stirring the pot - an obvious retort to this question might be, actually, private packages tend to be huge and this would balloon the time on the order of tens of minutes. I don’t know whether that’s true or not.)
You could also make it opt-out. Default to slow, strong compression, but if it causes a major performance regression on your deployment, toggle a flag and you’re back on the old behaviour.
I haven’t followed Swift much recently, but the original Swift was simple because it punted on all of the hard problems. It had no mechanism for error handling (there were some small mods towards NSError). It had no support for concurrency and the everything-is-shared-mutable-state model made that really hard to fix (the structured concurrency and actor extensions are trying). It had no support for shared libraries (Swift’s shared library support now is really nicely designed and it’s the only modern language that I think has addressed this problem well, but that inevitably came with complexity).
For macOS development, Objective-C++ is actually a very nice language. C++ for local code that is specialised over strong types, Objective-C for late-bound interfaces that are exposed across module boundaries,
I have never used Objective C++, but from afar it sounds horrifying. Take a monstrously, ineffably complex programming language and another quite complex programming language and mash them together into an amalgamation. Can anybody hope to make sense of it?
Speaking as someone who used it for years, it actually works quite well. Obj-C is not very complex, and its OO stuff is pretty separable from the C part. Obj-C++ extends the C part.
I’ll also point out that modern Objective-C’s ARC (automatic reference counting) composes very well with C++’s object lifetime rules. In pure Objective-C with ARC,
is not allowed, because
NSString*is an ARC pointer and the compiler needs to be able to reason about its initialisation and invalidation. The semantics of C structs don’t allow for that, which can make implementing custom data structures tricky: you either have to do it all in the context of Objective-C classes (which has certain overheads), use indirection, or you turn off ARC in the relevant source files and do manual reference counting.The same code quoted above will compile on Objective-C++ however, because the ARC pointer is treated as having the usual set of C++ constructors, destructor, assignment operator, and so on - it’s not a POD (plain old data) type. This means the
structalso gets implicit constructor/destructor/operator implementations when not explicitly implemented.You can therefore shove pointers to Objective-C objects into all sorts of C++ data structures, that have been implemented without special Objective-C support, including the STL. It all composes rather nicely.
(A significant proportion of my work in recent years has been working on macOS device drivers and related apps, daemons/agents, etc.; I’ve mostly been using Objective-C++ on that, although I’ve recently introduced Rust in one part of such a project. My limited contact with Swift has been exceedingly frustrating, so I’ve avoided it where possible; it never appealed to me in the first place due to the reasons David mentioned, and the practicalities around extremely poor forward- and backwards-compatibility were a nightmare to deal with on a project where it was forced upon me.)
ARC certainly makes this easy, though prior to ARC I implemented a C++ smart pointer class for Objective-C references that did the relevant retain and release operation, which meant that this was already possible in Objective-C++ without ARC, it just became nicer because you didn’t need to use the smart pointer.
Definitely, you weren’t the only one to implement such a smart pointer class. I guess my point was more that modern Objective-C actually composes better with C++ libraries than it does with C libraries and data structures.
This made me curious. What was your use-case? iOS apps? Mac apps? Something else?
I also used it for years and we made Mac apps that spoke to hardware devices through the IOKit kernel driver API (which is in C++). It was indeed quite nice.
Mac apps, libraries for use in Mac/iOS apps.
There is a non-trivial amount of ObjC++ in Firefox, too, or at least there was last time I checked. For TenFourFox I used ObjC++ as glue code to connect up libraries.
To add to what others have said: a lot of the pain in C++ comes from trying to do things that are easy in Objective-C, and vice versa. With C++, it’s easy to create rich types with no run-time overhead, but that create tight coupling. With Objective-C, you have late-bound interfaces everywhere, but avoiding dynamic dispatch is very hard / hacky. The combination means that you can completely avoid things like raw C pointers. You can used C++ collections inside a module, Objective-C ones across the boundaries.
You should give Obj-C a try, I think! It’s a surprisingly thin layer on top of C, giving a lot of bang for the buck for writing dynamic (but fast!) programs. It’s quite unique in that you have two extremes: C on the one hand, and a fully dynamic OO runtime (you can determine implementation at runtime). Overall syntax is outdated and weird (smalltalk influence), but it is still unmatched in that niche.
It’s not bad in practice. The use case is, you need Objective-C system frameworks but you can’t do without particular C++ libraries. You still spend the bulk of application code in one language.
I had a game that used Bullet physics in this way. I migrated most of the code to Swift after it was introduced, but I kept some Objective-C++ in order to keep Bullet. These days Swift has direct C++ interop, both for library use and for gradual migration of C++ projects to Swift.
Probably people who have actually used it.
I worked with making an iOS app at one point and I found while thinking in Lisp like patterns it seemed to get out my way if I wanted it to. But that is a beginner and greenfield perspective for sure.
I don’t think it sounds too bad, but I haven’t used it myself.
My understanding is it’s just the extra OOP bits from Objective-C overlayed on C++, similar to how it was overlayed on C in the first place. Basically just a second, independent object system. I understand why people wouldn’t like that, but it doesn’t sound too different than C++/CLI or using C++/JNI.
Can you tell us more about why Swift’s shared library support is well-designed?
With the caveat that I’ve read their design docs, but not actually used it in anger:
They make a clear distinction between ABI-stable and ABI-unstable shapes of structures. Within a library, there are no ABI guarantees. At a library boundary, you have a choice whether you want to sacrifice some performance for the ability to change a layout later, or sacrifice flexibility for performance. This is a per-structure choice. Depending on the choice that you make, the compiler either lowers to something similar to Objective-C non-fragile ivars, or C struct fields.
They actually have a language notion of a library boundary. This means that generics can be statically reified within a library, but fall back to dynamic dispatch across library boundaries. Contrast this with C++ where templates either have to live in headers (and then end up copied in every compilation unit, including the implementation, and it’s a violation of the one-definition rule to link two libraries that use different versions of the same template) or they are private to a library. The Swift model gracefully falls back. Things may be faster inside a library, but they still work from outside, and the ABI doesn’t leak implementation details of the generics, only their interfaces.
Wonderful explanation, thank you!
An overview from Rust folks: https://faultlore.com/blah/swift-abi/
Hmm…I’d say it was already incredibly complicated despite punting on a lot of hard problems and largely because it tried to codify “solutions” to non-problems into the language. Which never works, because reality, even code reality, is way too messy for that.
As an example, I wrote about the mess that is initialization back in 2014, so right after Swift apepared. There was some pushback from a prominent member of the Swift team saying that my goal of simplicity just wasn’t compatible with some of the goals they had. Hmm….
There was also a great rant by a prominent member of the community about Swift being just a mess of special cases upon special cases. I think he left the Apple ecosystem, and he’s not the only one. Alas I can’t find it and I don’t remember the name.
Anyway, I predicted back then that because they had so many language features for initialization it would never actually work out and there would be a renaissance of builder and factory method patterns and there would be even more language features for initialization. Five years later: hello SwiftUI! :-)
So the failure of Swift now isn’t surprising, the trajectory was already set in stone the day it launched and there wasn’t really much one could have done about it afterward…much less so since the same faulty reasoning that led to the initial problems was still present and guided later evolution.
I think this is an instance of correlation not being causation? My understanding is that the actual cause of SwiftUI is the successful design of Flutter (which gave raise to both SwiftUI and Kotlin Compose), and it is relatively orthogonal to language machinery.
Case in point, Kotlin’s initialization story is much more tame than Swit’s one (as it doesn’t try/doesn’t need to prove initialization safety statically), but it also converged on the essentially same design (or rather, vice-verse, IIRC Kotlin work in the area predate’s Swift’s).
Not to disagree with your wider point on constructors, which I agree with, just to point out that SwiftUI is not I think a particularly strong argument here.
I think you might want to have a look at the actual article. Swift introduced yet more special syntax for the part of SwiftUI that creates the view tree. So yet more language features for yet another way of constructing views^Wobjects^Wstructs.
The more general problems with SwiftUI (and related) are another issue, which I talk about a little bit here: UIs Are Not Pure Functions of the Model - React.js and Cocoa Side by Side
Last I checked the inspiration for Flutter and SwiftUI etc. was React.
I have read the articles! If I understand your argument correctly, it says that the fact that they needed to add new stuff to support Swift UI means that the original rules were inadequate. My counter-argument is even languages that don’t have Swift-style maze of initialization rule add special cases to support SwiftUI patterns. Ergo, adding stuff for SwiftUI is orthogonal to your normal way to initialize objects. In other words I claim that, in counter-factual where Swift doesn’t have complicated initialization rules and uses Java/Go style “everything is null to start with” or Rust/ML style “everything starts with all the parts specified”, it would have added more or less the same features still for SwiftUI.
The story is even more illustrative with Kotlin — it was specifically designed for DSLs like SwiftUI/Compose. The whole language, with its second-class-lambdas, extensions, and coming out-of-fashion implicit
this, works towards that goal. And yet, when the actual UIs started to be implemented, it was quickly apparent that no one wants to write+button(), and a bit more of compiler special sauce is needed for nice surface syntax.I must be a lousy communicator, because you seem to have misunderstood the article almost completely.
The point was not that Swift has the wrong initialization rules or too many of them. The point is, as it says in the title: “Remove features for greater power”. The many initialization rules are not the source of the problem, they are a symptom of the problem.
The problem is trying to bake this stuff into the language. As a consequence, you get 30 pages of initialization rules. As a further consequence, those 30 pages will be forever insufficient.
So for me, the supposed counter-point you bring with Kotlin actually supports my point. You write:
So they baked a whole bunch of features into the language to support the DSL use case. What was the title of the blog post again?
So they added a lot of features into the language especially for this use-case and it didn’t even work out for this particular use-case. Surprise surprise!
I simply don’t think the static/compiler-oriented mindset is compatible with the sorts of things these languages are trying to do. You put way too much into the language/compiler, and you do it way too early.
Ruby has had a bunch of these kinds of frameworks, and as far as I know they did not require any changes to the language. Because Ruby had fewer but more flexible features to start with.
https://github.com/AndyObtiva/glimmer
With Objective-S I seem to be violating that rule, because it certainly does put things into the language. Or at least seems to do so. What I am doing, however, is following the second rule: “don’t do it yet”. (With quite a bit of trepidation, because it is “experts only”).
And I am not actually baking all that much into the language. I am baking a bit of useful surface syntax and the associated metaobject-protocol into the language. What lies behind those metaobject protocols is quite flexible.
So far this appears to strike a good balance between providing some syntactic convenience and compiler support while not making the mistake of baking way too much into the language.
Indeed! I misunderstood your original comment as meaning that SwiftUI is a downstream consequence of initialization rules. I agree that both are rather the result of the lack of expressiveness, which doesn’t allow the user to “do it yourself” in userland code. The Kotlin example was exactly to illustrate that point.
Found it: Which features overcomplicate Swift? What should be removed?, by Rob Rix.
How to fix:
So I certainly wasn’t the only one who accurately predicted the current sad state of affairs. It was extremely obvious to a lot of people.
And silly me: of course I had referenced it an another one of my posts: The Curious Case of Swift’s Adoption of Smalltalk Keyword Syntax
It’s another example of complexity begetting more complexity, special cases begetting more special cases.
I am if the opinion that the only avenue for making a well rounded language that lasts is to put all of the skill points towards a small handful of skills (“fast”, “fast to compile”, “ease of dynamic linking”, “easy to use”, “easy to learn”, whatever) where each “skill” can be thought as a different axis in a graph. Go all in on a few niches. This helps with clarity of purpose for the project itself, a clear pitch for prospective users, and makes it possible to show the benefits for that niche. Without doing this the evolution of the project ends up rudderless, and adoption can stagnate either slowing down or killing the project. Once you’ve reached critical mass, fight for dear life not to abandon those hard earned early design decisions that allowed you to carve a niche, but go all-in on a handful of other axis as well. Rinse and repeat. A language can start as easy to use and once it has a clear base experience it can start trying to produce faster binary. Or it can start as fast to compile and afterwards try to produce optimized code. Or it can start producing super optimized code and later try to become faster to compile or easier to use. All the early design decisions will hamper the later pivot, and that’s ok! No language will ever be all things for everyone (barring some unrealistic investment in monstrous amounts of research, development and maintenance).
Swift had a clear initial view of what it wanted to accomplish, and did so successfully (I believe, don’t have personal experience with using Swift, nor do I keep up with its development). It is now on its expansion era, to allow for niches that previously were either difficult or impossible to cater to. It is a delicate process that can easily go haywire because of subtle interactions between completely unrelated features can make the result worse than the sum of its parts. But I can’t blame them for trying.
How do you see that wrt to Rust and async?
I could maybe argue that sync Rust had excellent “early design decisions that allowed you to carve a niche”, but the async related problems are now occupying a vast majority of the brain time of the lang teams. While I don’t think we’re heading to 217 keywords, I certainly feel uneasy seeing “keyword generics” and similar efforts.
“Do not obey in advance”. It’s highly unlikely that you’ll ever hear anything about it, but even if you did, it would be something like a cease and desist letter. You can decide then what to do about it.
Considering there is criminal liability for noncompliance, I don’t think “it would be something like a cease and desist letter” is useful advice at all. Even if you reckon you’re safe from this in your country, you’d potentially have to very carefully consider any future travel and whether you’re entering jurisdictions which might begin extradition proceedings, etc.
Yes, but that’s a common thing nowadays. Did you know there’s criminal liability for supply chain consumers coming? (CRA?)
The thing here is: there’s an escalation ladder here. Ofcom does a very poor job at disclosing this ladder.
I’m no expert on the CRA, but it was my understanding that after public consultation, liability has been explicitly excluded for non-commercial offerings, including open source contributors. In other words, you won’t be accidentally held liable for a security vulnerability in your code that someone downloaded from your website or repository for free. You don’t generally “accidentally” sell software for money, so you can price any costs for compliance into whatever offering you have, or alternatively, it’s not that difficult to exclude selling to markets where you don’t consider compliance with local laws proportionate.
The OSA is pretty unusual in that there’s no carve-out for small-scale or non-commercial operations, and in that it effectively applies globally to almost any publicly accessible website or digital service with a community component.
The page doesn’t explain why the CPU needed a separate, incompatible “protect mode” to access more memory. I learned more about that from the following Wikipedia pages, which I have summarized.
Real mode, also called real address mode, was once the only mode of Intel CPUs, so operating systems already supported it. In this mode, addresses always correspond to real locations in memory. For backwards compatibility, x86 CPUs still start up in real mode, though modern operating systems turn on protected mode when booting so as to be able to access more than 1 MiB of memory.
Protected mode, also called protected virtual address mode, enables virtual memory, paging, and other features. The mode was backwards incompatible with real mode because it reinterpreted some bits of a memory address to point to memory indirectly via an entry inside a descriptor table. It also reinterpreted two bits as defining the privilege of the memory request.
Protected mode was introduced in the Intel 80286 processor, the one mentioned in the article. The Wikipedia article on the processor has sections on its protected mode and its OS support.
May I suggest two articles I wrote at the beginning of the year that go deep into this topic? :)
One is https://blogsystem5.substack.com/p/from-0-to-1-mb-in-dos and talks about the real mode limitations and difficulties mentioned in the article. The other is https://blogsystem5.substack.com/p/beyond-the-1-mb-barrier-in-dos and talks about protected mode and the DOS extenders also mentioned in the article. Hope they are a useful summary to later navigate other Wikipedia content!
Also… “protect mode”? I had never heard anyone call it that way, ever. It has always been “protected mode” to me. (Sorry, had to mention it somewhere; not related to what you asked.)
Me neither, and I’ve known about protected mode since the 90s.
I have discovered this since sharing the post. :-( Apparently my shorthand form that I’ve been using for over a third of a century is Bad and Wrong and Incorrect. My mistake.
Not your fault! The article is full of “protect mode” as well, so I assumed the title here was intentional. I suppose someone edited the article content before publishing it and “fixed” protected with protect?
Does a double take
So it is.
OK, so, I think I retract my apology, then.
I was an early adopter of Windows – I used Windows 2 in production on my own work computer in 1988/1989, and I told my boss and my boss’ boss at work that Windows 3 was going to be huge and they should both stock up with as many copies as they could get, and that they should let me train the staff on Windows.
They laughed at me. They did not.
We sold out of all 17 (seventeen) copies of Windows 3.0 before 9AM on the day of release, and the company was flooded with enquiries. There was huge interest and nobody else other than me had ever used Windows as an environment. (We sold apps with Runtime Windows, such as Excel, Pagemaker, and the radical Omnis database, AFAIK the first GUI database for the PC, in which my company developed a bespoke app.) But the company regarded Windows as a joke product.
After 6 months of overwork due to being the only person who could support all the customers with Windows 3.0, I quit.
But, yes, at the time, around the start of the 1990s, that was a common term: Windows ran in “protect mode” and it was “protect mode” that let it use XMS as more program RAM and so on. Few other “protect mode” OSes existed: OS/2 1.x, SCO Xenix, and Novell Netware were about it.
Yes, protected mode might have been more accurate, but it was known by the shorter name.
This is not history to me. I am at work right now (and should not be commenting here) and this was an earlier stage of my job. I only switched to being a Linux guy in the 21st century. The first part of my career was as an expert Windows support guy.
As another person who lived through those times, I agree that “protect mode” was commonly heard, though it was also common to hear “protected mode”.
Not that this is by any means definitive, considering how common shortening of such things in a config file context is/was, but this conversation made me think of the OS/2 CONFIG.SYS setting: “PROTECTONLY=NO”. I always looked forward to the day that every app was built for OS/2 and I’d be able to flip that to YES. Still waiting!
Thank you!
Tried ArcaOS? :-)
Honestly, it’s pretty good. I tried it.
It does SMP, it can run thousands of DOS and 16-bit Windows apps, dozens of OS/2 apps, and some Win32 and FOSS/xNix apps. It’s tiny and amazingly quick compared to any modern OS, and I very much include Linux in that. No Linux comes close in performance and about the only thing that even could compete is TinyCoreLinux running from RAM. Alpine Linux is tiny and runs well on early 64-bit kit such as a high-end Core 2 Duo, and ArcaOS absolutely stomps it. It boots faster, apps load faster, it’s more responsive, and it takes less space on disk or RAM.
An ArcaOS with PAE support in the kernel would deliver quite a lot of what I want from an OS even now.
I just don’t have much use for it… :-/
It’s blisteringly quick on low-end kit, but then, so is Windows XP, especially XP64, and XP has a lot more apps and is much easier to get running. It’s just horribly insecure.
Lol I enjoyed the article, thanks. I bet OS/2 (er, ArcaOS) does indeed run pretty quick on modern hardware compared to how it ran on my IBM PS/2 Model 70 that packed an almost unimaginably large 6 MB of RAM!
I’ll always love OS/2, both for it being the first “real OS” that I ever ran (and igniting a life-long interest in systems software) and for the lessons I learned as a teenager watching people on Usenet fall further and further into denial that it was all over for it (because of Macro$oft and the corrupt Ziff-Davis, natch). Oh to go back to a time with such low-stakes internet brainrot.
Thanks!
Yes, I agree.
I recently did something that I felt guilty about, because I enjoyed it too much: put XP on a recent machine and went online. It was so fast and responsive, and at that time, you could still get free XP antivirus. You can get tools to find drivers, modern browsers that can handle 202x social networks etc…
It is a forbidden pleasure.
By comparison with modern OSes, WinXP is sleek and fast. Especially XP64 in 8GB of RAM.
I reviewed MS Office 97 when it came out. I hated it: bloated, sluggish, buggy. Now, it’s tiny and fast. It runs perfectly on WINE, including installing the service releases. Time makes a lot of difference. But it’s one suite.
XP makes one look at modern Windows and Linux and realise how appallingly sluggish everything is today.
But ArcaOS flies on machines that are low-end for XP. It can use up to 64 cores, it can just about do wifi and multihead true-colour accelerated graphics, USB2, UEFI… it has browsers, it has the
yumcommand and repos with FOSS ports. It is surprisingly capable for a late-1990s OS.The only big thing it can’t do is access >4GB RAM, except for a big fast RAMdisk.
It reminded me why I thought XP was a bloated mess in 2001.
My father’s company got its first Windows 3 install bundled with an app. A vector drawing program, I think it was called MetaDesign, decided it was cheaper to write a Windows 3 app and bundle a copy of Windows for DOS users than to write or license a vector drawing library and printer drivers for DOS. Before that, they used GEM, but once they had Windows installed it was easy to sell them more Windows apps.
I never ran 3.0 in protected mode though, only ever on real mode on an 8086 or 386 Enhanced Mode on an 80386 or newer. I was quite happy with that, because the 80286’s segmentation model was a thing that you teach in computer architecture courses as a cautionary tale.
This was a standard thing to do in the Windows 2 era. It was known as “runtime Windows” and the result was most of a Windows installation, but rejigged: there was no
WIN.COMand instead users ran the binary of the app itself from DOS, which loaded Windows with the app as “shell”. Quit the app, Windows shut down.It worked fine. I had a client in London, Mentorn – they’re still around – whose accounts department ran on Excel, but which didn’t use Windows for anything else. So it was Excel + runtime Windows.
Other apps that did this were Pagemaker and Omnis, the first Windows database.
Snag: it was the plain unaugmented Windows, not Windows/286 or Windows/386. So, it could only use 640kB of RAM and run in Real Mode.
If you installed several such apps, each got its own copy of Runtime Windows.
But if you then installed a full copy of standalone Windows (2 or later) on the machine, you could browse to the app’s directory, run the binary, and it ran normally under full Windows, including interop with other apps. The app was stock and it was Windows that was modified, AFAICT.
I do not recall ever seeing Runtime Windows 3. With Windows 2’s fugly MS-DOS Executive replaced by the perfectly usable Program Manager, Win3 made a quite decent DOS shell that users liked, and some used it with no other apps.
(!)
It was a thing.
Mainly for me on 386 machines with only 1MB of RAM. Win3 & 3.1 wanted 2MB minimum for 386 Enhanced Mode and if they had less they’d load in Standard Mode by default.
(286s were mostly too slow to comfortably run Windows. Indeed Windows 3 was a major driver of adoption of the 386 via the budget 386SX with a 16-bit bus.
((Aside to the aside: clock-for-lock, the 80286 was quicker than the 80386. But the normal speed of 286 PCs was 8MHz to 10MHz with a few 12MHz. I know of a single 20MHz 286 machine that was ever made, by Dell. Many 10-12MHz 286s had slower RAM and so ran with wait states. CPU cache only became a thing with high-end 386DX machines and most 286s had no cache, so if the RAM couldn’t keep up, the CPU just had to go slower.
(((Most IBM 286 PS/2 models shipped with slow RAM and wait states: in other words, the 286-based PS/2s for which IBM crippled OS/2 1.x and destroyed its chances in the market were lousy slow 286s and unsuitable for running OS/2 1.x, which was about the most demandingh 286 OS ever released, bigger and slower than Netware 2, or Xenix 286, or Concurrent DOS 286.)))
/* I am nesting way too deep here */
The 386SX ran from 16MHz to up to 33 or even 40MHz in some late models. So, 386SX machines were in fact faster than 286s, not because the CPU was inherently quicker – it wasn’t – but because it was clocked faster.
Also, few non-IBM 286s had VGA. Windows on EGA wasn’t very pleasant.))
So, there were lots of decent 386 machines, both 386DX and some 386SX, that were specced to run DOS and only had a meg or two of RAM. I saw some with 1.5 or 1.6MB RAM: 640kB plus 1MB of extended memory. Odd, but cheap.
You could force them to run in Enhanced Mode. I forget the switch now.
win /eor something. It worked. You got pre-emption of DOS apps in V86 mode “VMs”, you got virtual memory. It was dog slow but it worked. It wasn’t very stable but Windows 3 wasn’t very stable.Secondly, Windows could only run in Standard Mode if under another 386 multitasker. So, it ran in Standard Mode under DESQview/386, and DESQview/X, and Concurrent DOS 386, and a few other things.
So it was very much a thing and quite common.)
When I posted here on Lobsters in the past that Windows/286 ran in Standard Mode a lot of people got angry and contradicted me. It seems to me that there still is not a categorical, ex-cathedra statement of exactly how Windows 2 in 286 mode worked.
But Windows 3 in Standard Mode was quite good and made effective use of up to 15.6MB of XMS if you had it – on a 286.
It’s possible that I misremember. It could have been Windows 2. I think 3.0 was the first one that they rolled out company-wide.
I know, I just never owned a 286, and the 386s I owned had 4 or 5 MiB of RAM, so there was no point in pretending to be a 286. I went straight from 3.0 in Real mode to 3.1 in Enhanced Mode.
Windows 3.0 ran on my Amstrad PC1640 HD20 (with the hard disk upgraded to a massive 40 MB!). With 640 KiB of RAM, you could run Windows and 1-2 apps. It had an EGA monitor, so you mostly ran one app maximised anyway, which reduced the value of multitasking.
See here: https://winworldpc.com/product/windows-20/windows-286
Give it a try. As I recall, it is merely running in “real mode” with the HMA enabled, and with a special version of USER in the HMA. At least that is what the docs included with it say.
What windows/286 does appear to have is an EMS 4.0 / EEMS aware KERNEL, and so it can possibly page below 640K if the h/w implements it. i.e. if the board, chipset and memory cards are set for minimal base memory, and EMS “backfill”.
Or at least that is the impression I got a number of years ago when I had a try at disassembling some of its KERNEL.* I looks like that was removed/deleted/replaced when the Windows 3.0 scheme with support for DPMI was eventually created. Which would make sense, as it is simpler.
I’d have to suggest that you’d need to offer proof for windows/286 operating in “standard” i.e. 286 protected mode, as that goes against everything I’ve seen written in the past. So maybe take those images from the site above, and get them working in such a mode.
A place I worked at during 88 - 89 had a copy of Windows/386 and Windows/286, and us developers had Acer 286 machines with (usually) 1M RAM and EGA (some lucky folks had 2M). As I recall, none of us tried running those particular Windows versions. Trying Windows/386 would have been pointless (despite there being a handfull of 386 machines), and we did give plain windows a go. Yes it was painful in EGA (or even CGA) mode.
Interesting stuff. Thanks, and for the PC Mag link.
I am not bothered enough to try to run up some kind of 286 PC emulator to try it out. I do not miss Windows 2.x at all and have no urge to return.
It’s very interesting to me, though, that pivotal parts of the development of IT in the 1980s and 1990s is being forgotten now… because without knowing this stuff, some things about how it is today make no sense at all.
This industry needs to know its history much better.
Thanks for sharing the post. I suggested a title change to “Protected…” and if a few others do as well, we can stop discussing your shorthand and it’ll be easier to stick to the substance of the article. I enjoyed the post you shared from landley.net as well.
I find this tremendously interesting, because my head was firmly in Mac space at this time, so I missed much of it.
These days, it’s already the firmware (UEFI) that enters protected mode. It’s also the only mode supported by long mode. (aka 64-bit mode, aka x86-64)
Is it my impression or the data he presents does show better latency in at least two of the presented plots?
I agree. In particular, for
seems rather a stretch - most of the blue-only area is around 50µs, while almost all of the orange-only area is above 70µs.
To be fair, for
they say
This is one of those undocumented behavior distinctions you can very well experiment to find out, and which may appear stable, but which make me want to assume the worst case across the whole domain. If I’m riding on an abstracted pay service, I can’t depend on such a fine detail of behavior to stay constant.
I would also assume that this is a sintetic benchmark. Would the results be the same on, say, 50% network saturation? How about 80%?
This piece is kind of interesting, but I think its core thesis is pretty much nonsense. You don’t need to have been there when software was first written in order to understand it. Humans are capable of learning things.
I have worked with software that probably couldn’t have survived a complete change of team, and I will say this: It’s usually the worst code at the company, it’s often itself a rewrite of what the company started with, and I always get the impression it’s being held back by the original developers who are still with it. Without these first-generation programmers, any software in danger of becoming unlearnable would necessarily be simplified or replaced.
I think that’s a bit of a straw man; the article doesn’t say that the software itself is incomprehensible to others. With enough effort you can look at the software and understand what it does. What you can’t do after the fact is understand the context in which it was written; why was it done that way? What alternatives were considered and discarded? How has the context changed since those decisions were initially made? That’s what they mean when they talk about theory-building.
In theory you could write this stuff down, but I have never seen this actually happen in an effective way. (Probably because people keep thinking of the software itself as the point rather than the theory it embodies.)
I considered this, but looking at the article, it almost seems to take care not to talk about why. And, in any case, my experience is that people forget the context at such a rate that by ten or so years out, reverse-engineering it from the code is at least as reliable as asking the authors. Anyway, reading again, I still think this is about more than just context.
I think on balance I agree with the article. As @technomancy says, it’s about the theory the software embodies. Code is just one facet of that theory, and can never capture the tacit knowledge, ambiguities and personal relationships which all play a part in a software system.
However, I do agree with @edk- that the article dances around this point. Perhaps it’s intrinsically a bit of an abstract argument, but I couldn’t help but feel that more concrete writing would have helped.
This appears to be an excerpt from a book, so perhaps the rest of the book goes into detail on this point. I’ve added it to my list, but not bought/read it yet.
For some reason, there is a widespread default mindset (at least in the part of the industry I’ve seen) that “only those who built it can understand it.”
It doesn’t even depend on code quality (though I am a firm believer that any code written by a human can be understood by a human).
You can have a module that is clearly structured and spiced with comments about “why this solution is chosen,” or “when we’ll need X, it can be changed that way,” or “this is the assumption here; if it breaks, the assumption was wrong”… And still, when something is broken or needs update, people would hack around the module or treat it as a black box that “I’ve tried to pass this data, do you know why it doesn’t work? oh, I experimented for six more hours and it seems I guessed why!” or ask in a chat “who knows how this constant is used (used once in a codebase, with a clear comment why)” etc. etc.
It is like, through the years, the overall stance of a developer has switched from “I’ll understand the crap out of this codebase, disassemble it into smallest bits, and will rewrite it my way!!!” (an attitude that met with a lot of grounded critique) to “Nobody should understand your crap, either you support it forever, or it is thrown away in its entirety.”
I don’t think it’s true that only those who built it can understand it, but the effort required to understand a legacy codebase from scratch & safely make changes is enormous and this problem affects FOSS as well. I’ve been dealing with this for the TLA+ tools - specifically the parser - which when I joined the project was a pile of 20+-year-old Java code with everybody who touched it gone from the project for a decade or more. Past a certain point the code ceases to become source code in some sense - people will only deal with it at the API level and everything within is indistinguishable from a binary blob that cannot be changed. The process of shedding light onto that part of the codebase required writing over 300 round-trip parse tests to semi-exhaustively document its behavior, and even with that monumental effort I still only really have a handle on the syntax component of the parser, let alone the semantic checker. But that isn’t all. You may have developed a mental model of the codebase, but who is going to review your PRs? It then becomes a social enterprise of either convincing people that your tests are thorough enough to catch any regressions or giving them some understanding of the codebase as well.
Compare that with being the original author, where you basically have total ownership & can make rapid dictatorial changes to a component often without any real code review. The difference in effort is 1-2 orders of magnitude.
Then consider the scenario of me leaving. Sure all the tests I wrote are still there, but do people have a grasp of how thorough the test coverage is to gauge how safe their changes are? I would not be surprised if it took five years after me leaving for basic changes to the parser to happen again.
The only thing I was trying to say is that “only original author can fully understand that” becomes industry’s self-fulfilling prophecy, creating a feedback loop between people not trying to read others’ code (and not giving the feedback that it lacks some background information or clear structure), and people not thinking of their code as a way to communicate everything they know, because “nobody will try to read it anyway, the important thing is that it works.”
It manifests in many things, including the changed stance for code reviews, where “you left a lot of comments” starts to be universally seen as “you are nitpicking and stalling the development,” and disincentivizes those who are really trying to read the code and comment of the things that aren’t clear enough or lack an explanation of the non-obvious design choices.
okay, I’ll take the alternate stance here. I worked on the back end of a large triple AAA video game that was always online. I worked on it for roughly 6 years before I moved to another company.
I have very good documentation, very clear objectives. It was very simple infrastructure - as simple as it could be made. The “why” of decisions was documented and weaved consistently into the fabric of the solution.
I hired into my new company! my successor. Expecting him to have experience with the same problems and my original infrastructure sought to solve.
he didn’t, he didn’t learn how or why certain things were how they were. my expectation of his ability to solve problems that I had already solved because he would’ve had experience with them was completely incorrect.
had the system failed catastrophically he would’ve been unable to fix it and that was not discovered even after working there for three years
There are levels of understanding and documentation is variable, but there are almost always some things that don’t make it into documentation. For example, the approaches that you discarded because they didn’t work may not be written down. The requirements that were implicit ten years ago and were so obvious that they didn’t need writing down, but which are now gone, may be omitted, and they influenced part of the design.
With enough archeology, you often can reconstruct the thought processes, but that will take enormous amounts of effort. If you were there (and have a good memory), you can usually just recall things.
This is all true, of course.
The problem (for me) is that people start taking those contextual truths and applying them unconditionally to any situation. Like, even without looking frequently, “I wouldn’t even start to try reading through the module (where choices of approach and limitations might be visible in code or well-documented); I’ll treat it as a black box or delegate it to the module author, regardless of the current organization structure.”
The situations I am quoting in the previous comment (“who knows how this constant is used?” in chat, regardless of the fact that the constant is used once in a codebase, with a clear comment why and what’s the meaning) are all real and somewhat disturbing. Might depend on the corner of the industry and the kind of team one is working with, of course.
I completely agree with the second half of your post. I might just be a grumpy old person at this point, but the mindset seems to have shifted a lot in the last twenty years.
For example, back then there was a common belief that software should run on i386 and 64-bit SPARC so that you knew it handled big vs little endian, 32- vs 64-bit pointers, strong vs weak alignment requirements, and strong vs weak memory models. It also had to run on one BSD and one SysV variant to make sure it wasn’t making any assumptions beyond POSIX (using OS-specific features was fine, as long as you had fallback). This was a mark of code quality and something that people did because they knew platforms changed over time and wanted to make sure that their code could adapt.
Now, I see projects that support macOS and Linux refusing FreeBSD patches because they come with too much maintenance burden, when really they’re just highlighting poor platform abstractions.
Similarly, back then people cared a lot about API stability and, to a lesser degree, ABI stability (the latter mostly because computers were slow and recompiling everything in your dependency tree might be an overnight job or a whole-weekend thing). Maintaining stable APIs and having graceful deprecation policies was just what you did as part of software engineering. Then the ‘move fast and break things’ or ‘we can refactor our monorepo and code outside doesn’t matter’ mindsets are common.
That seems like a meta-problem that’s orthogonal to the original article’s thesis. It strikes me as an instance of the H L Mencken quote, “For every complex problem there is a solution which is clear, simple and wrong.”
I’m not sure the overall attitude has changed over the years. I suspect the nuance required for dealing with the problem of software longevity and legacy code is something that is currently mainly learned the hard way, rather than being taught. As such, many inexperienced practitioners will lack the awareness or tools to deal with it; combined with the rapid growth and thus younger-skewing demographics of the industry, I guess it means those with the requisite experience are in the minority. But has this situation really ever been different?
In any case, none of this is an argument against the thesis of the original text - you can certainly argue it’s a little vague (possibly because it’s a short excerpt from a book) and perhaps overly absolutist. (I’d argue the extent of the problem scales non-linearly with the size of the code on the one hand, and you can to some extent counteract it by proactive development practices.)
FWIW, as a contractor/consultant, I’d say the majority of my projects over the last years have been of the “we have this legacy code, the person/team who wrote it is/are no longer around” kind to some degree. My approach is definitely not to assume that I will never understand the existing code. In fact, I have found a variety of tactics for tackling the task of making sense of existing code. Again, I suspect most of these are not taught. But all of them are much less efficient than just picking the brains of a person who already has a good mental model of the code and the problem it solves. (It is fiendishly difficult to say with any reliability in retrospect whether it would have been cheaper to just start over from scratch on any such project. I do suspect it can shake out either way and depends a lot on the specifics.)
okay, I’ll take the alternate stance here. I worked on the back end of a large triple AAA video game that was always online. I worked on it for roughly 6 years before I moved to another company.
I have very good documentation, very clear objectives. It was very simple infrastructure - as simple as it could be made. The “why” of decisions was documented and weaved consistently into the fabric of the solution.
I hired into my new company! my successor. Expecting him to have experience with the same problems and my original infrastructure sought to solve.
he didn’t, he didn’t learn how or why certain things were how they were. my expectation of his ability to solve problems that I had already solved because he would’ve had experience with them was completely incorrect.
had the system failed catastrophically he would’ve been unable to fix it and that was not discovered even after working there for three years
I agree with your primary criticism–it is certainly true that software can be understood without the original creators.
However, your assessment of what will happen is very optimistic. It is entirely possible that what will happen is that new programmers will be brought in. They will only have time to make basic bug-fixes, which will be kludges. If asked to add new functionality, there will be copy paste. When they do try to buck the trend of increasing kludges, they will break things because they do not fully understand the software.
So I agree, any software should be understandable, but it will take investment in rebuilding a theory of how it works, and rewriting, or refactoring the software to make it workable for the new programmers. This will only happen if management understands that they have a lump of poorly understood software and trusts the developers to play the long game of improving the software.
The optimism is really just extended pessimism: I claim that, if you keep doing that, at some point all changes will break more than they fix, and either someone will take a hatchet to it or it will have to be abandoned.
It’s not that far off, only a little exaggerated. Yes, you can understand code you didn’t write, but you can’t understand it in the same way as one of its authors, until you’ve rewritten a chunk of it yourself. Yes, a team (or a solo developer) can maintain inherited software, but they’re going to have an adjustment period in which they’ll be inclined to “bolt-on” or “wrapper” solutions because they have trepidation about touching the core code. And it’s fair to say that that adjustment period ends, not after some period of staring at the code, but after making enough changes to it — not only that some part of it becomes their own, but that they run into enough challenges that the constraints that shaped the existing code start to make sense.
I wish I’d thought of this in my first comment, but the article is basically a long-winded way to say “the worst memory is better than the best documentation”. I’ll just leave that there.
I can believe this happens sometimes but I don’t think it’s necessary. I’ve picked up legacy projects and within days made changes to them that I’d stand by today. Codebases take time to learn, and working on them helps, but finding one’s way around a new program, figuring out why things are the way they are, and building an intuition for how things should look, are all skills that one can develop.
Anyway I think even your version of the point largely refutes the original. Learning by doing is still just learning, not magic. In particular it doesn’t require an unbroken chain of acculturation. Even if the team behind some software all leaves at once, it’s not doomed.
I would also argue that in some cases the original authors of a program hold it back. The constraints that shaped the existing code aren’t always relevant decades down the track. Some the authors will simply be wrong about things. Removing the code from most of its context can be a good thing when it allows the project to go in a new direction. Also, especially for code that’s difficult to maintain… the original authors are the reason that is so—and as long as the chain of first-generation programmers remains intact, the path of least resistance to full facility with the code is to be trained to think like them. Breaking that local maximum might not be the worst thing.
Perhaps the problem with churn is that it’s not a clean break. You get an endless stream of second-generation programmers who try to build in the image of what came before, but always leave before they achieve mastery. I dunno.
I think it’s very accurate that the founders and early employees have the deepest knowledge of the system though. Yea, new people can come in and learn it, but it’s never quite to the same level. Anecdotally of course.
My mental model of this is that, roughly, the money you pour into a paid programmer accumulates. It’s not “linear”, but if a programmer goes, it takes away the accumulated value. I believe some accounting practices model similar stuff.
The converse also applies: if your company relies heavily on a piece of open-source software, hiring a core maintainer of (or experienced contributor to) that software onto your staff is probably a bargain.
I suppose the idiomatic method for securing processes on FreeBSD is capsicum(4). And at least for Firefox, it looks like someone has been working on adding support but they ran into some tricky cases that apparently aren’t well supported.
I guess for retrofitting huge, complex code bases, pledge and unveil or jails are probably easier to get working. I wonder how this affects things like file picker dialogs for choosing uploads, etc. If they’re implemented in-process, I guess they get to “see” the veiled or jailed file system - which means you don’t get any mysterious permission issues, but you also can’t upload arbitrary files. If they were implemented via IPC to some desktop environment process, the user could see the usual file system hierarchy to select arbitrary files, and if those files were sent back to the browser process as file descriptors, it would actually work. (I think the latter is how sandboxed apps are permitted to read and write arbitrary files on macOS, with Apple-typical disregard for slightly more complex requirements than just picking one file.)
In regard to pledge+unveil, it’s extremely simple to actually implement in code (though considerations for where+what in the code would be more complex) and the predefined promises make it pretty easy.
For pledge+unveil, from memory, the dialog just cannot browse/see outside of the unveil’d paths. For browsers there’s a few files
/etc/<browser>/unveil.*that lists the paths each kind of browser process is allowed access. Included here is~/Downloadsand common XDG-dirs for example, which allows for most file picking stuff to work fine for most users.One advantage is also that you can progressively enhance this over time. You can keep adding restrictions every time you fix instances of code which would previously have been violating a pledge. Capsicum would appear to require a more top-down approach. (I can’t help but wonder if you could add a kind of compatibility mode where it allows
open()and similar as long as the provided path traverses a directory for which the process holds an appropriate file descriptor through which you’d normally be expected to callopenat(). Or maybe that already exists, I really need to get hands-on with this one day.)I can see that the alternative would probably require a more holistic approach at the desktop environment level to implement well, but defining these directories statically up front seems like an awkward compromise. Various XDG directories contain precisely the kind of data you’d want to protect from exfiltration via a compromised process.
Capsicum lets you handle things like the downloads directory by passing a file descriptor with
CAP_CREATE. This can be used withopenatwith theO_CREATflag, but doesn’t let you open existing files in that directory. This is all visible in the code, so you don’t need to cross reference external policy files.If you run a Capsicum app with ktrace, you can see every system call that Capsicum blocks, so it’s easy to fix them. With a default-deny policy and no access to global namespaces, it’s easy to write least-privilege software with Capsicum. I have not had that experience with any of the other sandboxing frameworks I’ve tried.
The download side is easier to handle as long as you’re keeping to a single download directory. I expect uploads to be somewhat more annoying UX wise: it’s rather unusual to have an “uploads” directory where the user would first copy any files they want to upload to a website, then select them in the “Browse…” dialog.
One slightly less annoying option is to agree on blanket read access to a bunch of stuff under $HOME, which appears to be what’s used here in practice, but it leaves you vulnerable to data exfiltration, which surely is one of the attack scenarios this whole exercise is trying to defend against.
Anything more comprehensive I can come up with will necessarily be a multi-process arrangement where the less-sandboxed process sends file descriptors of what the user selects to the heavily sandboxed one. And where drag & drop of a file sends (a) file descriptor(s) rather than just (a) path(s).
To be clear, I’m not saying this would be a bad system! I think it’d be great to have this in a desktop environment. Just that it’s a little tricky to retrofit onto a giant ball of code you’ve never even looked inside before.
That’s good to know - in contrast, dealing with the Sandbox on Apple’s platforms is super annoying as you’re mostly reduced to reading system log tea leaves when things aren’t working - macOS
dtraceis falling apart more and more with every release and sometimes requires disabling the very security features you’re trying to debug.But tracing only just begins to address the stated problem of retrofitting a large existing code base. If everything including dependencies including transitive ones is using
openrather thanopenat, butopenis completely non-functional, I suspect that might be rather a chore to get fixed. I mean it’s feasible if you actually “own” most of the project, but modifying something as big and unknown as a web browser in this way is quite an undertaking even if you can eventually get it all upstreamed.Capsicum was designed to support this via the powerbox model (just as on macOS: it was designed to be able to more cleanly support the sandboxing model Apple was developing at the time). When you want to upload a file, the file dialog runs as a service in another process that has access to anything and gives file descriptors to selected files. You can also implement the same thing on top of a drag and drop protocol.
Alex Richardson did some Qt / KDE patches to support this and they worked well. Not sure what happened to them.
The nice thing about this is that open is a replaceable symbol. For example, in one project where I want to use some existing libraries in a Capsicum sandbox I simply replace
openwith something that callsopenatwith the right base depending on the prefix.It would be fairly easy to do something similar for the XDG paths and have pre-opened file descriptors for each with a sensible set of permissions.
Good to know it’s been done and the code presumably is still out there somewhere. Something to keep note of in case I end up doing any UNIX desktop work. (And it sounds like this was done as part of an academic research project, so probably worth trying to get hold of any other published artifacts from that - perhaps part of this project?)
Providing this fallback compatibility wrapper as a user space libc override is a nifty technique, thanks for sharing!
I kind of agree. It did feel unintuitive to me at first. Worth noting these files are owned by root and normal users cannot write to them by default.
That’s right – from what I remember it was pretty well locked down though.
When I see the effort some people put, mostly on their spare time, to give people the ability to run extremely closed-source software onto extremely closed-source hardware, with FLOSS in the middle, I’m just amazed… in a good way.
I’m happy that they’re doing it, and I’m not… These people have my absolute respect.
What makes you call the hardware extremely closed source? I mean all the firmware is closed source of course but that’s the case for pretty much all hardware, what makes Mac hardware unusually closed source? It’s not especially locked down, all the mechanisms which are used to install and boot different operating systems are mechanisms intentionally included by Apple after all, it’s not like Asahi relies on some kind of jailbreaking
This might be ignorance from my part, I’m happy to be corrected.
But I’ve always assumed that Mac didn’t use any standard BIOS, and the new ones were using some non-standard UEFI boot process. My understanding was that, while it’s true that their custom firmware allow you to boot another OS, this is something your OS has to put effort into supporting. This is because Apple don’t care about you, as opposed to Intel/AMD/Asus/… who actively contribute to the Linux kernel for their hardware support. My understanding, as well, is that there is very little free documentation available on how to support Apple’s hardware, hence my “extremely closed source”.
Why else would we need the efforts of the Asahi team, if the hardware (which, I admit, I used as a metonymy for hardware+firmware) was standard?
The firmware and bootloader become fairly uninteresting once you’ve successfully booted the operating system, and while there are lots of fiddly bits to take care of to get things running smoothly, once you’re booted, you’re booted. (usually) Booting is not usually a herculean effort, as long as the hardware and firmware vendors haven’t tried to actively block you. Which Apple hasn’t, apparently. (Assuming secure boot is disabled.)
That said, Apple’s GPUs are entirely undocumented, so building a high-quality GPU driver for these things purely based on reverse-engineering (intercepting and tracing what I/O the macOS GPU driver performs on the GPU), especially in this kind of time frame is beyond impressive.
Other devices and SoC subsystems are also mostly undocumented of course and need their own drivers but the complexity of most devices is not in the same league as a GPU; USB is almost certainly a standard implementation though (perhaps with some smaller hacks) so that’s one big driver stack they won’t have to have reimplemented.
Correct! Most Apple peripherals are custom (though some have fun heritage like the Samsung UART, lol) but core USB is the same Synopsys DesignWare block you see in nearly everything (that’s not Intel nor AMD) these days.
That’s just the host/device controller though. To make it work you also need a USB PHY driver (Apple custom), a USB-PD controller driver (Partly incompatible Apple variant of an existing TI chip), and the I2C controller driver for the bus that chip talks through (which is actually a variant of an old PASemi controller).
Yeah, by “core” USB I meant the host/device controller :)
Frankly this is exactly why I like ACPI. With ACPI, all that glue-ish stuff around a generic core interface like XHCI mostly/hopefully can be written once in AML, instead of having to write and maintain drivers in each kernel. Which, yeah, from a Linux-centric perspective that last part is not a feature, but I personally like the BSDs for example, and I also hope that “the future” will be more Redox-shaped or something, so I’m always biased in favor of things that make OS diversity easier!
Heh, I was working on some low-level USB code on a microcontroller for a client. I’m staring the code and having this moment of deja vu “weird… I’ve never used this brand of chip before but for some reason a lot of this feels very familiar.”
I had worked on USB code before on different chips though. I opened up the datasheets from an old project and this new project and started looking at the USB registers. Sure enough, exactly the same names, exactly the same bit layout, exactly the same memory addresses. Google some of the names and find data sheets from multiple vendors and then find one from Synopsis. Sure helped debug things when I had my own reference implementation to compare against!
Booting other OS’s is intentionally supported, and doing so without compromising security for people who aren’t doing so is a significant amount of work. There’s a bunch of stuff describing how non-apple OS’s are installed without trivially undermining secure boot, and I think back in the early days of Asahi marcan talked about it a bunch.
I don’t see a meaningful distinction between “heap allocation” and “dynamic allocation”, other than the scope of the allocator: whether it’s OS-provided and global, or custom and specific to some subsystem. Either way, it has the same effect.
What’s the benefit to using Pigweed’s custom allocator over the global operator new / malloc? If anything it’s a drawback, because in the case that multiple subsystems implement their own allocators, memory is wasted because each one is holding on to free space that the others can’t access.
Nevertheless, there’s a lot of interesting info here on the innards of C++ coroutines!
A very meaningful distinction is that heap allocation, e.g. dynamic storage, will at some point call ::operator new, and has some chance of ending up doing a system call, which has implications in terms of latency for instance as this increases the chances for a thread to get scheduled-out. So far this was entirely precluding the use of coroutines in e.g. real-time audio processing loops unless you could assert at compile-time that HALO went into effect and every memory allocation got optimized out.
it’s definitely the only sane way to design some systems when you really want to make sure you aren’t ever going to take a lock.
There are a few reasons why someone might prefer a custom allocator over a global
newormalloc.In some cases, it’s a hard requirement: not all platforms provide a global allocator. This is fairly common in embedded systems.
Custom allocators can be beneficial for reliability, too– the exact amount of memory available for the purpose of allocating a particular coroutine or set of coroutines can be fixed, and failure to allocate from that dedicated pool of memory can be handled gracefully, rather than allowing the coroutine(s) to gradually expand into memory that is dedicated for other tasks.
Finally, using a custom allocator can sometimes improve performance by taking advantage of cache locality or certain allocation patterns. For example, bump allocators and arena allocators can provide more efficient allocation because they only deallocate at the end of certain scopes or lifetimes. This makes allocation just a simple pointer increment and makes deallocation a total no-op until the end of the arena’s lifetime.
You’re preaching to the choir! I agree custom heap allocators can be useful. But they’re still heaps. I found your article a bit misleading because you imply you’ve found a way to run C++coroutines without heap allocation, which you haven’t. You’ve just built your own heap.
Sorry for the misdirection! I didn’t intend to be misleading– this is an unfortunately common terminology mismatch. I’m using “heap allocation” to mean specifically the provided malloc/free/new/delete, and “dynamic allocation” to refer to the more general case of runtime memory allocation.
I do also discuss in this post why dynamic allocation is needed and how I hope its use can be avoided in the future, so I think it is still relevant to those hoping to avoid dynamic allocation completely (though I’m sad that it isn’t currently possible today).
But this requires you to bring along an allocator. And if you are going to bring along an allocator, why not just implement the global
newanddeleteoverloads?The reason that I would want to avoid global new / delete on embedded platforms is that allocation is more likely to fail (because memory is tightly constrained) and so I need the caller to handle the failure case and I’m on an embedded platform that doesn’t support exceptions so it needs to handle allocation failure by the return address.
Your article mentions this about half way down:
From the API surface, it looks as if you provide a hook in the promise object that means that allocation failure can return a constant. This is really nice, you can have a coroutine call return something like an
ErrorOr<T>and then return the error value if it fails to allocate space for the coroutine.It would probably be nice to lead with that. I had to read a lot to figure out that you’ve actually solved the bit of the problem that I cared about.
Yes, handling the failure case gracefully is super important! RE
ErrorOr<T>, Pigweed provides apw::Result<T>which is exactly this– either aTor apw::Status(our error code).Coro<T>requires thatTbe convertible frompw::Statusso that it can produce a value ofTfrompw::Status::Internal()in the case of allocation failure.Sounds nice. I might have a look and see if it can be ported to CHERIoT.
I’m going to flip this statement on its head for a sec. Using a fixed-sized arena allocator or something like that is great for having guarantees that you’re not going to blow up your heap. But… do we know at compile-time how big the allocations are going to be? Instead of using a fixed-sized arena, would it be possible to use something like a freelist allocator for each type that needs to get allocated? Like I could declare in my code that I’m going to assign a static allocation for 10
Coroobjects and 30 frames; the allocator could return those from the freelist and return them to the freelist upon deallocation (both O(1) without needing to ever end the arena’s lifetime).The size of the coroutine state will depend on the captured state (automatic variables at suspension points) so much like a lambda (closure type), its size is known by the compiler at compile time, but as far as I’m aware there’s no clear way to obtain it in your code at compile time, for example to be used as a template argument.
Yeah, this isn’t possible today, sadly– I discuss this a bit at the end of the post. If C++ offered a way to inspect the size of the coroutine of a specific function implementation, there would be no need for dynamic memory allocation: we could pre-reserve exactly the required amount of space. This would have huge benefits both for reliability and efficiency– no more need to guess at and adjust the right size for your allocators.
Little late getting back to this but I’ve had a little time to dig into some of the WGxx documents and I understand quite well why we can’t do this (yet anyway, hopefully someday!) For using it on an embedded system though, the HALO thing scares me a whole lot more than having to pre-reserve some space for an allocator (which also scares me less than having hidden/implicit malloc behind the scenes). On most of the smaller-sized embedded systems I work on, running out of space in a bounded arena and returning an error is a manageable thing. Changing the function that sets up a coroutine a little bit, resulting in HALO being available and significantly changing how much stack space is required… that’s spooky! At least with running out of arena space there’s well-defined error semantics. Walking off the end of a fixed-sized stack usually just means corruption.
I’m still digging a little more… it seems like there is a way to ensure HALO is disabled but having it potentially ok by default seems like a footgun for embedded, and we’ve got lots of footguns already!
(I’m still super intrigued through. This looks super cool!)
AFAIR the issue is that the size of the coroutine is only known by the compiler after optimization passes (as it depends on how local variables are stored, which lifetimes overlap or not, what can be reused) but the
sizeof(...)operator is typically implemented in the frontend. The compiler writers said they would not be able to implement a coroutine proposal with frontend known size without heroic efforts. This was discussed extensively during the design of C++ coroutines, including a competing “core coroutine” proposal appearing and ending up dying on this same issue. Some alternatives were explored, such as types that did not support sizeof(…) but still could be stored on the stack, but it the end storing the coroutine on a heap with an escape hatch for ellision was deemed the best compromise.Ahhhhh interesting, that makes sense. I’m going to put it on my list of things to play with :). Cool project!
.. is it though ? how does that work if you have
It’s interested in the upper bound as common when you’re pre-allocating. So compute both, take the largest.
A more general case of the example above would be if
awere a variable-length array withxas the size. I would absolutely never write such a thing in embedded code (and typically not in normal C++ if there’s a chance an attacker can influence the savour of `x‘) but it is permitted.exactly, after all you can have recursive calls or alloca so there’s no way to precompute the entire required stack size statically
Yeah I wasn’t thinking about the general case too much, just the specific example, and assuming the statically sized array was the point that seemed problematic. You’re right it’s not always possible… in most languages.
Rust does know the size of each Future, but it doesn’t let you have dynamically sized values on the stack (on stable). And using that unstable feature, it’s not possible to hold a dynamically sized value across await points (#61335) as await points are where the size needs to be known to be stored in the generated Future’s enum variant.
Another reason is to support safe memory reclamation for a concurrent data structure.
Not sure what you mean by this? Just about all system allocators are thread-safe.
“Safe memory reclamation” is special jargon in this context.
If you are using RCU, hazard pointers, or other similar lock-free techniques, then a write to a datastructure cannot immediately free() any memory that was removed, because the memory can still be in use by concurrent readers. You have to wait for a grace period or quiescent period to pass before it can be freed. You don’t want to do this as part of the write transaction because that will ruin your write throughput: grace periods are relatively long and can accommodate multiple write transactions.
So you need to hang on to the to-be-freed memory somehow and reclaim it later. Sometimes the whole concurrent algorithm is easier if you guarantee type-stable memory by using a region of some kind. Sometimes you can save significant amounts of memory (a pointer per node) by using a GC instead of a to-be-freed list.
I’m not sure how custom allocators help here. They are not invoked until after the destructor has run, so prolonging the lifetime of the allocation using a custom allocator doesn’t help because the object is gone (you may get an arbitrary bit pattern if you try to read it). Things like RCU work fine without a custom allocator because they defer destruction, not deletion.
Note that C++20 adds “destroying delete” which lets you define allocators that take care of the destruction themselves. But as linked article says it’s not standardized for the allocator methods on coroutine promise types >_<
https://stackoverflow.com/questions/67595789/what-is-destroying-operator-delete-in-c20
Interesting. The Solaris VMEM allocator gets some really nice performance benefits by resetting deallocated objects to a partially initialised state (allocation often happens on hot paths, deallocation is easy to defer. You can defer not needing an object far more easily than you can defer needing one). I believe that could be represented in C++20 now, because delete would not call the destructor and would just add the object to the pool, leaving it as a valid object of the correct type.
I don’t know how you can do things like data-structure-specialized GC or type-stable memory without a custom allocator. You don’t need those techniques for RCU (RCU was an example to set the scene) but they are sometimes useful for concurrent data structures. I also dunno how this would fit with the C++ facilities for custom allocators — I guess you have to avoid destruction until the memory is about to be freed (especially for type-stable memory)
Type-stable memory can be done with custom allocators, but it’s often better to do it with a class overriding operators new and delete. You can use CRTP to implement a superclass that provides a custom memory pool for each type.
I could be wrong but my recollection is that they use
allocawhich allocates arbitrary stack space instead of using actual allocators (which typically involve a system call in the worst case).Can someone explain the difference between the “classic” and async/Embassy approach?
I interpreted it as:
But perhaps sync (classic) versus async (Embassy) is the key difference being alluded to
I suspect it is sync vs async, but I suspect the classic would be higher level since it presumably has an embedded RTOS underneath it providing the sync abstraction, while Embassy presumably has a thinner runtime and leverages Rust’s async to support multitasking.
It’s just a hardware abstraction layer, definitely no RTOS there. It doesn’t do multitasking, just abstractions for interacting with the hardware.
So how does one do multitasking with classic? Like how do you run a bluetooth or wifi module while also doing the other things your application needs to do?
Independent pieces of hardware do stuff in the background when you’re not interacting with them, so you either need to check in every so often manually, or use interrupts. A common pattern is to have a big loop that goes around doing everything that needs to be done, and then sleeps when there’s nothing to be done, to be waken up by an interrupt or something else. You might write a bunch of state machines that you drive forward when the correct events come in.
That’s why async is so exciting for embedded. You can write what looks and feels like abstracted task code for an RTOS, but it compiles into a fairly predictable event loop with state machines, just like what many people write by hand.
That makes sense. When I’ve dabbled with embedded programming the “classic” approach usually involved pretty low-level management of network peripherals—you had to know the state machine for each peripheral and how to handle its events and then compose that state machine with the state machines of other peripherals and your application and it was always really painful especially if you misunderstood something about a peripheral’s state machine. Embassy looks more or less like what I’ve been wanting—I can’t wait to try it out.
Write it all manually with hardware timers and interrupts.
A few of my own patches made it into this release!
Unfortunately a bunch of others so far haven’t been accepted, I‘ll try again for the 9.2 cycle:
I’ve also been working on support for HVF’s in-kernel APIC support, but that turns out to be buggy, so I’m currently trying to figure out if there’s some baseline feature set that does work reliably…
I don’t quite get the appeal of the textual prompt about discarding the key. Overlay text breaks the immersion to some extent. If it really is useless now and there’s some emotional backstory, surely the player character could for example convey the emotion through gesture? Shake their fist and fling the key away? (I don’t really go for horror games, so haven’t played RE2 and don’t know what sense of relief the author is alluding to here.)
I liked Far Cry 2’s approach to the map. The first-person player character literally pulls out a map from somewhere and holds it in his¹ hand alongside the handheld GPS device. The interesting bit here is that the game keeps going on around you. You can run around while holding the map, even though it’s obscuring most of your field of view. If you encounter baddies (or they encounter you) while you’re staring at the map, they’ll quite happily start shooting you while you’re still fumbling to grab your weapon.
Far Cry 2 screenshot with player holding the map while standing
The same mechanic works while you’re driving too. Much as driving while holding up a map in one hand is rather hazardous in real life, the game tends to punish you for doing this as well, as tempting as it frequently is.
Far Cry 2 screenshot with player holding up the map while driving
The rest of the HUD is similarly understated and disappears most of the time, though a health bar and number of health syringes will show up when you’ve taken damage. (And IIRC the screen will momentarily be tinted red when taking a hit.)
Later games in the series started cluttering the screen a lot more unfortunately. (They also undeniably tidied up some of the more unpolished game mechanics, but in my opinion never quite recreated the atmosphere of danger that Far Cry 2 conveyed. Nor the moral ambivalence.)
¹ Unfortunately, all playable characters are male, even though key NPCs you meet in the game will be chosen from the pool of player characters plus a woman who for some reason is not playable.
the relief from the player character comes from the fact that in RE/2 you have very limited inventory space. Even just carrying a key around is a whole ordeal, and when you open a door and still have to hold onto the key, that means that you have new challenges ahead, and you don’t even get an extra inventory slot (which you could use to, for example, pick up healing items).
There are absurd situations you can end up in where you have no free inventory slots, but no healing items, and are hurt, and there is a healing item right in front of you and you can’t use it. It’s fodder for a lot of jokes, but it’s also an amazing source of tension.
None of this really makes me a better programmer, but it’s fun to think about.
I love those sorts of things, depending on the game. Helldivers has an interesting middle-ground to this; there’s a HUD, but it’s quite minimal and digging for more information tends to have real consequences. Your weapon has no targeting reticle, unless you actually aim down the sights of it. The ammo counter is a numberless “full/kinda full/empty” bar, unless you bring up the context menu that lets you switch weapon modes and such; only then does it say “23/30 rounds”. You have a local minimap, but zooming out on it to see the full map means your character literally looks down at the PDA strapped to their arm and stops aiming their weapon at the horrible bug-monsters coming to claw out their eyeballs. In general you can be fighting for your life or figuring out what the hell you’re doing with your life, but not both.
They do a good job of using semi-artificial inconveniences like that to add to the gritty dystopia of the game, and to make the decision making and frenetic pacing more hectic. This now makes me wonder what the game might look like if even more of the information in it were implicit rather than given in the HUD.
By the way, the term that describes a UI element like, “You pull a map out and the game doesn’t pause” where it’s internal to the world is “diegetic.” I’d go so far as to say that diegetic UI elements are always superior to virtual ones. The post makes mention of how Isaac’s suit in Dead Space shows his health on the spine, which is also a diegetic UI element.
I think “superior” is doing a lot of work in that sentence. Diagetic elements are always more immersive, but not every game is about being as immersive as possible.
Imagine in real life if you could have a map that stopped time when you used it and filled your entire field of vision, you would probably find that a lot more useful than a paper map! So I don’t think it’s as simple as saying diagetic is better every time.
Diegetic doesn’t always mean “time doesn’t stop when you look at a map” so much as, “the map is a fundamental part of the world rather than an arbitrary menu.”
This blog announces or reviews a ton of SBCs and micro-PCs, but I thought this one in particular was interesting and might be newsworthy here, as it’s basically “Raspberry Pi 5 but it’s x86, can run Windows, and btw supports fast SSDs.” A cute detail is that there’s a RPi2400 chip on board to drive the GPIO pins.
“but there’s also a boot selection switch that lets you put the RP2040 into USB mass storage mode for firmware uploads.” Oh that’s cool
That’s the usual way to upload firmware to an RP2040 and not specific to this board. As far as I’m aware this functionality is part of the bootrom inside the RP2040 and completely hardwired.
(Incidentally, I’ve previously used an RPi Pico as a kind of GPIO breakout board in an x86 SFF PC based custom appliance as well. They are so small, and the decent USB support makes it a good choice.)
I interpreted it as you could flash the 2040 in-line from the main board
In-line as in, it shows up as a USB mass storage device connected to the x86 computer? Yes, that’s what I’d expect. The RP2040’s USB port must surely be wired directly to a USB 2.0 host port on the x86 SoC, so it’ll show up as whatever type of USB device the code running on the RP2040 exposes. In the case of the naked bootrom with BOOTSEL held, this is a USB MSC. With the stock firmware, it would presumably show up as some kind of GPIO-controlling USB device. (HID?) That’s assuming it even ships with any kind of useful firmware on the RP2040. (The “documentation” link on Radxa’s X4 page seems broken, I’ve not researched beyond that.) With TinyUSB, you can program it to expose whatever USB interface you like.
The tricky bit would be resetting or power cycling the RP2040 in isolation without having to power down the host SoC. The “BOOTSEL” button on an RPi Pico normally only has an effect during very early startup, before the bootrom has loaded the current firmware. Perhaps the GPIO connector exposes the RUN pin (30) which can be used to hard reboot an RP2040. (On my SFF PC based build I run both BOOTSEL and RUN contacts to externally accessible push buttons for exactly this reason.) Another option is to have a software reset trigger in your RP2040 firmware. This’ll only work if you didn’t make any mistakes in coding your firmware. (Now who would do such a thing. 😅) Otherwise you’d have to power cycle it, which presumably means power cycling the whole board.
I received an email saying that I was eligible for a new crypto offering that was targeting open-source developers. They claimed they had checked my GitHub username, and that I was in. They offered $200 to do it for me, all I needed to do was send them the private SSH key associated with my GitHub account, so they could claim the reward. One important detail is that it had to be the private SSH key associated with my account at the time the contract was added to the blockchain, so I couldn’t just make a new temporary one.
I investigated it and the crypto distribution was real. I was able to claim it myself, and I sold the tokens as soon as I was able, for almost $3000.
What the hell. Even after explaining it it sounds like a scam. Did you at least rotate all your keys?
I did! But by doing it myself I didn’t have to send anyone my private SSH key, I just had to sign a challenge token with it, which they verified with the public key that they had collected from GitHub.
The process was still sketchy, they had a Docker container that you would have to run. The container would read the private SSH key (which had to be mounted to the container) and run a program to generate the response token. They did recommend running the container without a network attached, and I turned off my wifi just in case as well. The container would die after printing out the response token, so I think it was pretty safe.
This is the token, if anyone’s interested: https://claim.fluence.network/wallet
That’s pretty nuts. I have to ask: what’s in it for them? And why the super unsafe delivery if it’s some altruistic cause?
I’ve been receiving those emails as well and assumed they were scams aimed at injecting malicious code into my github repos or exfiltrating code from private repos. I have zero practical experience with cryptocurrency, so I guess I’ll let this “opportunity” slide as it sounds like more hassle and risk than it’s worth even if it’s not a scam.
I think they gain by creating traction among developers, that would be my guess. And since they created the token when they give it away they’re not losing money, it’s all fictitious anyway. :)
They seem to assume that the only reason packets ever drop because the link is full of traffic. Besides the fact that dropping packets in collisions is part of Ethernet, you will get dropped packets sooner or later any time you use wifi, cellular internet, damaged/degraded wired networks, long-distance wired networks, or wired networks that wander near nasty sources of interference. So, you know, always. TCP does a great job of covering this up when a) latencies are low, and b) ~99% of packets still get through. (Tried to find the article that graphed this but I can’t; I think it was from Google or Cloudflare?)
QUIC is pretty good, definitely use it in 99% of the places you’d normally use UDP. But the question is not “good data” vs “bad data”, it is “mostly-good data now” vs “perfect data maybe half a second from now”. Especially for video telecoms and stuff like that, a garbled frame or two and some weird distortion for a few seconds is better than someone just cutting off and waiting a painful few seconds for TCP and the video/audio codec to get their collective shit together before you get to say “sorry you cut out there, can you repeat that?”
Except that Ethernet and WiFi handle detecting the drops and doing retries, etc, at the link level. They don’t rely on TCP semantics
Modern wireless standards do an impressive amount of forward error correction and so will provide an abstraction that looks like a reliable link even with a lot of frames being dropped. This is critical for satellite links, where a retransmit is hundreds of ms of latency, but is increasingly important for WiFi at high data rates. The faster your link speed, the more data you need to buffer. 802.11bn is supposed to scale to 100 Gb/s. If it takes 1 ms to retransmit, you’d need a 100 Mb buffer to avoid slowdown for a single dropped packet (or you’ll have a smaller window and so get back pressure and slow down). If you have enough error correction, you just get the frames arriving slightly out of order once you’ve received enough to reconstruct the dropped one.
I think modern wireless protocols dynamically adapt the size of their error correction codes when they detect loss, so your speed just gracefully degrades in most cases, but you don’t see huge latency spikes.
Interesting, isn’t that wasteful when a higher layer is ok with some packet loss? But OTOH if the higher level is doing its own retries, and the least reliable link is the first one, it makes sense to retry early where the latency cost is lower.
Maybe the link should do a small number of retries, to get a good balance of latency and reliability.
Not a deep expert on this, but I seem to remember TCP assumes packet loss is down to bandwidth limitation or congestion, and will respond by reducing the transmission rate right down, only slowly ramping back up. The typical packet loss scenario in WiFi is that there‘s some momentary interference, or the geometry of the link has changed, so the peers need to switch to less sensitive QAM encodings, etc. Until that happens, a whole batch of packets will have been dropped, which TCP interprets as catastrophic bandwidth reduction even if the wireless link recovers very quickly. So the higher layer doesn’t cope very well with the specific pattern of loss.
Re. TCP goodput, are you thinking of the Mathis equation? https://lobste.rs/s/5zjwgs/golang_is_evil_on_shitty_networks_2022#c_qxm49s
The conclusion of the article surprised me: it recommends a fairly complicated collection of mechanisms. I thought QUIC was going to get support for lossy streams so that it could avoid creating a backlog of wasted work when the network conditions suddenly change.
The author is working on a spec for media streaming over QUIC, so they’re certainly aware of this.
Yeah, yet they talk about everything except it.