I don’t think I would write “Go’s error handling is awesome”, but it’s probably the least bad thing I’ve used. The main alternatives to exceptions that I’ve used have been C’s return values which are typically pretty terrible and Rust’s which are better in principle (less boilerplate at call sites, no unhandled-error bugs, and similar non-problems), but somewhat worse than Go’s in practice:
My biggest grievance with Rust error handling is the choice between (1) the sheer burden of managing your own error trait implementations (2) the unidiomatic-ness of using anyhow in libraries and (3) the absurd amount of time I spend debugging macro expansion errors from crates like thiserror. Maybe there’s some (4) option where you return some dynamic boxed error and pray callers never need to introspect it?
And I don’t think either Rust or Go have a particularly compelling solution for attaching stack traces automatically (which is occasionally helpful) or for displaying additional error context beyond the stack trace (i.e., all of the parameters that were passed down in a structured format)? Maybe Rust has something here–I’m somewhat less familiar.
I’m also vaguely aware that Zig and Swift do errors-as-values, but I haven’t used them either. Maybe the people who have found something better than Go could enlighten us?
We use a custom error type in Go that lets you add fields, like loggers do, and just stores the values in a map[string]interface{}. This doesn’t solve the problem of capturing all parameter values, or stack traces, but it’s pretty good in practice. This is particularly true since errors tend to eventually end up in log messages so you can do something like log.WithFields(err.Fields).Error("something terrible happened"). If we could nest those fields based on the call stack I would probably never complain again.
Best solution Ive used is polymorphic variants from ocaml. They force you to at least acknowledge the error exists and you can easily compose different errors together.
Exceptions are better on all counts though, so even that is a faulty conclusion in my opinion.
Exceptions do the correct thing by default, auto-unwrap in the success case, auto-bubble up on the error case with as fine-grained scoping as necessary (try-catch blocks). Go’s error handling is just a tad bit higher than the terrible mess that is C’s errno.
Edit: Re cryptic stack traces: most exception-based languages can trivially chain exceptions as well, so you get both a clear chain of cause and a stack trace. I swear if Java would simply default to printing out the message of each exception in the chain at the top in an easier to read matter and only print the stack trace after, people would like it better.
I was hoping this article would compare if err != nil to more modern approaches (Rust’s ?) and not just Java-style exceptions, but unfortunately it doesn’t.
I’d be more interested to read an article that weighs the approaches against each other.
One point the article misses is how value-based error handling works really nicely when you don’t have constructors (either in your language, or at least your codebase, in case you’re using C++.)
I’ve been pretty disappointed by Rust’s approach to error handling. It improves upon two “problems” in Go which IMHO are not actually problems in practice: if err != nil boilerplate and unhandled return values, but making good error types is fairly hard–either you manually maintain a bunch of implementations of the Error trait (which is a truly crushing amount of boilerplate) or you use something like anyhow to punt on errors (which is generally considered to be poor practice for library code) or you use some crate that generates the boilerplate for you via macros. The latter seems idyllic, but in practice I spend about as much time debugging macro errors as I would spend just maintaining the implementations manually.
In Go, the error implementation is just a single method called Error() that returns a string. Annotating that error, whether in a library or a main package, is just fmt.Errorf("calling the endpoint: %w", err). I don’t think either of them do a particularly good job of automating stack trace stuff, and I’m not sure about Rust, but at least Go does not have a particularly good solution for getting more context out of the error beyond the error message–specifically parameter values (if I’m passing a bunch of identifiers, filepaths, etc down the call stack that would be relevant for debugging, you have to pack them into the error message and they often show up several times in the error message or not at all because few people have a good system for attaching that metadata exactly once).
A lot of people have a smug sense of superiority about their language’s approach to error handling, which (beyond the silliness of basing one’s self-esteem on some programming language feature) always strikes me as silly because even the best programming languages are not particularly good at it, or at least not as good as I imagine it ought to be.
a bunch of implementations of the Error trait (which is a truly crushing amount of boilerplate)
you usually just need to impl Display, which I wouldn’t call a “crushing” amount of boilerplate
or you use some crate that generates the boilerplate for you via macros.
thiserror is pretty good, although tbqh just having an enum Error and implementing display for it is good enough. I’ve done some heavy lifting with error handling before but that’s usually to deal with larger issues, like making sure errors are Clone + Serialize + Deserialize and can keep stacktraces across FFI boundaries.
It’s pretty rarely “just” impl Display though, right? If you want automatic conversions from some upstream types you need to implement From, for example. You could not do it, but then you’re shifting the boilerplate to every call site. Depending on other factors, you likely also need Debug and Error. There are likely others as well that I’m not thinking about.
#[derive(Debug)] and impl Display makes the impl of Error trivial (impl Error for E {}). If you’re wrapping errors then you probably want to implement source(). thiserror is a nice crate for doing everything with macros, and it’s not too heavy so the debugging potential is pretty low.
One advantage of map_err(...) everywhere instead of implementing From is that it gives you access to file!() and line!() macros so you can get stack traces out of your normal error handling.
thiserror is a nice crate for doing everything with macros, and it’s not too heavy so the debugging potential is pretty low.
I’ve used thiserror and a few other crates, and I still spend a lot more time than I’d like debugging macro expansions. To the point where I waffle between using it and maintaining the trait implementations by hand. I’m not sure which of the two is less work on balance, but I know that I spend wayyy more time trying to make good error types in Rust than I do with Go (and I’d like to reiterate that I think there’s plenty of room for improvement on Go’s side).
One advantage of map_err(…) everywhere instead of implementing From is that it gives you access to file!() and line!() macros so you can get stack traces out of your normal error handling.
Maybe I should try this more. I guess I wish there was clear, agreed-upon guidance for how to do error handling in Rust. It seems like lots of people have subtly different ideas about how to do it–you mentioned just implementing Display while others encourage thiserror and someone else in this thread suggested Box<dyn Error> while others suggest anyhow.
The rule of thumb I’ve seen is anyhow for applications and thiserror or your own custom error type for libraries, and if thiserror doesn’t fit your needs (for example, needing clone-able or serializable errors, stack traces, etc). Most libraries I’ve seen either use thiserror if they’re wrapping a bunch of other errors, or just have their own error type which is usually not too complex.
That’s too bad, I genuinely enjoy learning about new (to me) ways of solving these problems, I just dislike the derisive fervor with which these conversations take place.
You discount anyhow as punting on errors, but Go’s Error() with a string is the same strategy.
If you want that, you don’t even need anyhow. Rust’s stdlib has Box<dyn Error>. It supports From<String>, so you can use .map_err(|err| format!("calling the endpoint: {err}")). There’s downcast() and .source() for chaining errors and getting errors with data, if there’s more than a string (but anyhow does that better with .context()).
One source of differences in different languages’ error handling complexity is whether you think errors are just generic failures with some human-readable context for logging/debugging (Go makes this easy), or you think errors have meaning that should be distinguishable in code and handled by code (Rust assumes this). The latter is inherently more complicated, because it’s doing more. You can do it either way in either language, of course, it’s just a question of what seems more idiomatic.
I don’t think I agree. It’s perfectly idiomatic in Go to define your own error types and then to handle them distinctly in code up the stack. The main difference is that Rust typically uses enums (closed set) rather than Go’s canonical error interface (open set). I kind of think an open set is more appropriate because it gives upstream functions more flexibility to add error cases in the future without breaking the API, and of course Rust users can elect into open set semantics–they just have to do it a little more thoughtfully. The default in Go seems a little more safe in this regard, and Go users can opt into closed set semantics when appropriate (although I’m genuinely not sure off the top of my head when you need closed set semantics for errors?). I’m sure there are other considerations I’m not thinking of as well–it’s interesting stuff to think about!
Maybe “idiomatic” isn’t quite the right word and I just mean “more common”. As I say, you can do both ways in both languages. But I see a lot of Go code that propagates errors by just adding a string to the trace, rather than translating them into a locally meaningful error type. (E.g.,
return fmt.Errorf("Couldn't do that: %w", err)
so the caller can’t distinguish the errors without reading the strings, as opposed to
return &ErrCouldntDoThat{err} // or equivalent
AFAIK the %w feature was specifically designed to let you add strings to a human-readable trace without having to distinguish errors.
Whereas I see a lot of Rust code defining a local error type and an impl From to wrap errors in local types. (Whether that’s done manually or via a macro.)
Maybe it’s just what code I’m looking at. And of course, one could claim people would prefer the first way in Rust, if it had a stdlib way to make a tree of untyped error strings.
But I see a lot of Go code that propagates errors by just adding a string to the trace, rather than translating them into a locally meaningful error type
Right, we usually add a string when we’re just passing it up the call stack, so we can attach contextual information to the error message as necessary (I don’t know why you would benefit from a distinct error type in this case?). We create a dedicated error type when there’s something interesting that a caller might want to switch on (e.g., resource not found versus resource exists).
AFAIK the %w feature was specifically designed to let you add strings to a human-readable trace without having to distinguish errors.
It returns a type that wraps some other error, but you can still check the underlying error type with errors.Is() and errors.As(). So I might have an API that returns *FooNotFoundErr and its caller might wrap it in fmt.Errorf("fetching foo: %w", err), and the toplevel caller might do if errors.As(err, &fooNotFoundErr) { return http.StatusNotFound }.
Whereas I see a lot of Rust code defining a local error type and an impl From to wrap errors in local types. (Whether that’s done manually or via a macro.)
I think this is just the open-vs-closed set thing? I’m curious where we disagree: In Go, fallible functions return an error which is an open set of error types, sort of like Box<dyn Error>, and so we don’t need a distinct type for each function that represents the unique set of errors it could return. And since we’re not creating a distinct error type for each fallible function, we may still want to annotate it as we pass it up the call stack, so we have fmt.Errorf() much like Rust has anyhow! (but we can use fmt.Errorf() inside libraries as well as applications precisely because concrete error types aren’t part of the API). If you have to make an error type for each function’s return, then you don’t need fmt.Errorf() because you just add the annotation on your custom type, but when you don’t need to create custom types, you realize that you still want to annotate your errors.
I tend to agree that rusts error handling is both better and worse. In day to day use I can typically get away with anyhow or dyn Error but it’s honestly a mess, and one that I really dread when it starts barking at me.
On the other hand… I think being able to chain ‘?’ blocks is a god send for legibility, I think Result is far superior to err.
I certainly bias towards Rusts overall but it’s got real issues.
There is one thing to be said against ?: it does not encourage the addition of contextual information, which can make diagnosing an error more difficult when e.g. it gets expect-ed (or logged out) half a dozen frames above with no indication of the path it took.
However I that is hardly unsolvable. You could have e.g. ?("text") which wraps with text and returns, and ?(unwrapped) which direct returns (the keyword being there to encourage wrapping, one could even imagine extending this to more keywords e.g. ?(panic)` would be your unwrap).
Oh yeah I’m not saying it’s not possible to decorate things (it very much is), just pointing out that the incentives are not necessarily in that direction.
If I was a big applications writer / user of type-erased errors, I’d probably add a wrapping method or two to Result (if I was to use “raw” boxed error, as IIRC anyhow has something like that already).
I’ve often wondered if people would like Java exceptions more if it only supported checked exceptions. You still have the issue of exceptions being a parallel execution flow / go-to, but you lose the issue of random exceptions crashing programs. In my opinion it would make the language easier to write, because the compiler would force you to think about all the ways your program could fail at each level of abstraction. Programs would be more verbose, but maybe it would force us to think more about exception classes.
Tl;Dr Java would be fine if we removed RuntimeException?
No, Go has unchecked exceptions. They’re called “panics”.
What makes Go better than Java is that you return the error interface instead of a concrete error type, which means you can add a new error to an existing method without breaking all your callers and forcing them to update their own throws declarations.
Rust, for example, has a good compromise of using option types and pattern matching to find error conditions, leveraging some nice syntactic sugar to achieve similar results.
I’m also personally quite fond of error handling in Swift.
Rust, Zig, and Swift all have interesting value-oriented results. Swift more so since it added, well, Result and the ability to convert errors to that.
No matter how many times Go people try to gaslight me, I will not accept this approach to error-handling as anything approaching good. Here’s why:
Go’s philosophy regarding error handling forces developers to incorporate errors as first class citizens of most functions they write.
[…]
Most linters or IDEs will catch that you’re ignoring an error, and it will certaintly be visible to your teammates during code review.
Why must you rely on a linter or IDE to catch this mistake? Because the compiler doesn’t care if you do this.
If you care about correctness, you should want a compiler that considers handling errors part of its purview. This approach is no better than a dynamic language.
The fact that the compiler doesn’t catch it when you ignore an error return has definitely bitten me before. doTheThing() on its own looks like a perfectly innocent line of code, and the compiler won’t even warn on it, but it might be swallowing an error.
I learned that the compiler doesn’t treat unused function results as errors while debugging a bug in production; an operation which failed was treated as if it succeeded and therefore wasn’t re-tried as it should. I had been programming in Go for many years at that point, but it had never occurred to me that silently swallowing an error in Go could possibly be so easy as just calling a function in the normal way. I had always done _ = doTheThing() if I needed to ignore an error, out of the assumption that of course unused error returns is a compile error.
Because errors aren’t special to the Go compiler, and Go doesn’t yell at you if you ignore any return value. It’s probably not the most ideal design decision, but in practice it’s not really a problem. Most functions return something that you have to handle, so when you see a naked function call it stands out like a sore thumb. I obviously don’t have empirical evidence, but in my decade and a half of using Go collaboratively, this has never been a real pain point whether with junior developers or otherwise. It seems like it mostly chafes people who already had strong negative feelings toward Go.
Is there a serious argument behind the sarcasm as to how this is comparable to array bounds checks? Do you have any data about the vulnerabilities that have arisen in Go due to unhandled errors?
Because the programmer made an intentional decision to ignore the error. It won’t let you call a function that returns an error with out assigning it to something, that would be a compile time error. If the programmer decides to ignore it, that’s on the programmer (and so beware 3rd party code).
Now perhaps it might be a good idea for the compiler to insert code when assigned to _ that panics if the result is non-nil. Doesn’t really help at runtime, but at least it would fail loudly so they could be found.
I’ve spent my own share of time tracking down bugs because something appeared to be working but the error/exception was swallowed somewhere without a trace.
huh… til. I always assumed you needed to use the result, probably because of single vs multiple returns needing both being a compile time error. Thanks.
Because the programmer made an intentional decision to ignore the error.
f.Write(s)
is not an intentional decision to ignore the error. Neither is
_, err := f.Write(s)
Yet the go compiler will never flag the first one, and may not flag the second one depending on err being used elsewhere in the scope (e.g. in the unheard of case where you have two different possibly error-ing calls in the same scope and you check the other one).
Yet the go compiler will never flag the first one, and may not flag the second one depending on err being used elsewhere in the scope (e.g. in the unheard of case where you have two different possibly error-ing calls in the same scope and you check the other one).
_, err := f.Write(s) is a compiler error if err already exists (no new variables on left side of :=), and if err doesn’t already exist and you aren’t handling it, you get a different error (declared and not used: err). I think you would have to assign a new variable t, err := f.Write(s) and then take care to handle t in order to silently ignore the err, but yeah, with some work you can get Go to silently swallow it in the variable declaration case.
Because they couldn’t be arsed to add this in v0, and they can’t be arsed to work on it for cmd/vet, and there are third-party linters which do it, so it’s all good. Hopefully you don’t suffer from unknown unknowns and you know you should use one of these linters before you get bit, and they don’t get abandoned.
(TBF you need both that and errcheck, because the unused store one can’t catch ignoring return values entirely).
I don’t really care. Generally speaking, I would expect compilers to either warn or error on an implicitly swallowed error. The Go team could fix this issue by either adding warnings for this case specifically (going back on their decision to avoid warnings), or by making it a compile error, I don’t care which.
This is slightly more nuanced. Go project ships both go build and go vet. go vet is an isomorphic to how Rust handles warnings (that warnings apply to you, not your dependencies).
So there would be nothing wrong per se if this was caught by go vet and not go build.
What is the issue though, is that this isn’t caught by first-party go vet, and requires third party errcheck.
Meh plenty of code bases don’t regularly run go vet. This is a critical enough issue that it should be made apparent as part of any normal build, either as a warning or an error.
If you care about correctness, you should want a compiler that considers handling errors part of its purview. This approach is no better than a dynamic language.
I agree with you that it’s better for this to be a compiler error, but (1) I’ll never understand why this is such a big deal–I’m sure it’s caused bugs, but I don’t think I’ve ever seen one in the dozen or so years of using Go and (2) I don’t think many dynamic languages have tooling that could catch unhandled errors so I don’t really understand the “no better than a dynamic language” claim. I also suspect that the people who say good things about Go’s error handling are making a comparison to exceptions in other languages rather than to Rust’s approach to errors-as-values (which has its own flaws–no one has devised a satisfactory error handling system as far as I’m aware).
The fact that these bugs seem so rare and that the mitigation seems so trivial makes me feel like this is (yet another) big nothingburger.
The most common response to my critique of Go’s error-handling is always some variation on “this never happens”, which I also do not accept because I have seen this happen. In production. So good for you, if you have not; but I know from practice this is an issue of concern.
I don’t think many dynamic languages have tooling that could catch unhandled errors so I don’t really understand the “no better than a dynamic language” claim.
Relying on the programmer to comprehensively test inputs imperatively in a million little checks at runtime is how dynamic languages handle errors. This is how Go approached error-handling, with the added indignity of unnecessary verbosity. At least in Ruby you can write single-line guard clauses.
I don’t really follow your dismissal of Rust since you didn’t actually make an argument, but personally I consider Rust’s Option type the gold standard of error-handling so far. The type system forces you to deal with the possiblity of failure in order to access the inner value. This is objectively better at preventing “trivial” errors than what Go provides.
The most common response to my critique of Go’s error-handling is always some variation on “this never happens”, which I also do not accept because I have seen this happen. In production. So good for you, if you have not; but I know from practice this is an issue of concern.
I’m sure it has happened before, even in production. I think most places run linters in CI which default to checking errors, and I suspect if someone wasn’t doing this and experienced a bug in production, they would just turn on the linter and move on with life. Something so exceedingly rare and so easily mitigated does not meet my threshold for “issue of concern”.
Relying on the programmer to comprehensively test inputs imperatively in a million little checks at runtime is how dynamic languages handle errors
That’s how all languages handle runtime errors. You can’t handle them at compile time. But your original criticism was that Go is no better than a dynamic language with respect to detecting unhandled errors, which seems untrue to me because I’m not aware of any dynamic languages with these kinds of linters. Even if such a linter exists for some dynamic language, I’m skeptical that they’re so widely used that it merits elevating the entire category of dynamic languages.
I don’t really follow your dismissal of Rust since you didn’t actually make an argument, but personally I consider Rust’s Option type the gold standard of error-handling so far. The type system forces you to deal with the possiblity of failure in order to access the inner value. This is objectively better at preventing “trivial” errors than what Go provides.
I didn’t dismiss Rust, I was suggesting that you may have mistaken the article as some sort of criticism of Rust’s error handling. But I will happily register complaints with Rust’s error handling as well–while it does force you to check errors and is strictly better than Go in that regard, this is mostly a theoretical victory insofar as these sorts of bugs are exceedingly rare in Go even without strict enforcement, and Rust makes you choose between the verbosity of managing your own error types, debugging macro expansion errors from crates like thiserror, or punting altogether and doing the bare minimum to provide recoverable error information. I have plenty of criticism for Go’s approach to error handling, but pushing everything into an error interface and switching on the dynamic type gets the job done.
For my money, Rust has the better theoretical approach and Go has the better practical approach, and I think both of them could be significantly improved. They’re both the best I’m aware of, and yet it’s so easy for me to imagine something better (automatic stack trace annotations, capturing and formatting relevant context variables, etc). Neither of them seems so much better in relative or absolute terms that their proponents should express superiority or derision.
Fair enough. It’s a pity things like this are so difficult to answer empirically, and we must rely on our experiences. I am very curious how many orgs are bitten by this and how frequently.
Enabling a linter is different from doing “a million little checks at runtime”. This behaviour is not standard because you can use Go for many reasons other than writing production-grade services, and you don’t want to clutter your terminal with unchecked error warnings.
I admit that it would be better if this behaviour were part of go vet rather than an external linter.
The strange behaviour here is not “Go people are trying to gaslight me”, but people like you coming and complaining about Go’s error handling when you have no interest in the language at all.
Enabling a linter is different from doing “a million little checks at runtime”.
You can’t lint your way out of this problem. The Go type system is simply not good enough to encapsulate your program’s invarients, so even if your inputs pass a type check you still must write lots of imperative checks to ensure correctness.
Needing to do this ad-hoc is strictly less safe than relying on the type system to check this for you. err checks are simply one example of this much larger weakness in the language.
The strange behaviour here is not “Go people are trying to gaslight me”, but people like you coming and complaining about Go’s error handling when you have no interest in the language at all.
I have to work with it professionally, so I absolutely do have an interest in this. And I wouldn’t feel the need to develop this critique of it publicly if there weren’t a constant drip feed of stories telling me how awesome this obviously poor feature is.
Your views about how bad Go’s type system is are obviously not supported by the facts, otherwise Go programs would be full of bugs (or full of minuscule imperative checks) with respect to your_favourite_language.
I understand your point about being forced to use a tool in your $job that you don’t like, that happened to me with Java, my best advice to you is to just change $job instead of complaining under unrelated discussions.
Your views about how bad Go’s type system is are obviously not supported by the facts, otherwise Go programs would be full of bugs (or full of minuscule imperative checks)
They are full of bugs, and they are full of miniscule imperative checks. The verbosity of all the if err != nil checks is one of the first things people notice. Invoke “the facts” without bringing any isn’t meaningfully different than subjective opinion.
Your comments amount to “shut up and go away” and I refuse. To publish a blog post celebrating a language feature, and to surface it on a site of professionals, is to invite comment and critique. I am doing this, and I am being constructive by articulating specific downsides to this language decision and its impacts. This is relevant information that people use to evaluate languages and should be part of the conversation.
If if err != nil checks are the “minuscle imperative checks” you complain about, I have no problem with that.
That you have “facts” about Go programs having worse technical quality (and bug count) than any other language I seriously doubt, at most you have anecdotes.
And the only anecdote you’ve been able to come up with so far is that you’ve found “production bugs” caused by unchecked errors that can be fixed by a linter. Being constructive would mean indicating how the language should change to address your perceived problem, not implying that the entire language should be thrown out the window. If that’s how you feel, just avoid commenting on random Go post.
Yeah, I have seen it happen maybe twice in eight years of using Go professionally, but I have seen it complained about in online comment sections countless times. :-)
If I were making a new language today, I wouldn’t copy Go’s error handling. It would probably look more like Zig. But I also don’t find it to be a source of bugs in practice.
Everyone who has mastered a language builds up muscle memory of how to avoid the Bad Parts. Every language has them. This is not dispositive to the question of whether a particular design is good or not.
Not seeing a problem as a bug in production doesn’t tell you much. It usually just means that the developers spent more writing tests or doing manual testing - and this is just not visible to you. The better the compiler and type-system, the fewer tests you need for the same quality.
Not seeing a problem as a bug in production doesn’t tell you much
Agreed, but I wasn’t talking about just production–I don’t recall seeing a bug like this in any environment, at any stage.
It usually just means that the developers spent more writing tests or doing manual testing - and this is just not visible to you.
In a lot of cases I am the developer, or I’m working closely with junior developers, so it is visible to me.
The better the compiler and type-system, the fewer tests you need for the same quality.
Of course with Go we don’t need to write tests for unhandled errors any more than with Rust, we just use a linter. And even when static analysis isn’t an option, I disagree with the logic that writing tests is always slower. Not all static analysis is equal, and in many cases it’s not cheap from a developer velocity perspective. Checking for errors is very cheap from a developer velocity perspective, but pacifying the borrow checker is not. In many cases, you can write a test or two in the time it would take to satisfy rustc and in some cases I’ve even introduced bugs precisely because my attention was so focused on the borrow checker and not on the domain problem (these were bugs in a rewrite from an existing Go application which didn’t have the bugs to begin with despite not having the hindsight benefit that the Rust rewrite enjoyed). I’m not saying Rust is worse or static analysis is bad, but that the logic that more static analysis necessarily improves quality or velocity is overly simplistic, IMHO.
Of course with Go we don’t need to write tests for unhandled errors any more than with Rust, we just use a linter.
I just want to emphasize that It’s not the same thing - as you also hint to in the next sentence.
I disagree with the logic that writing tests is always slower.
I didn’t say that writing tests is always slower or that using the compiler to catch these things is necessarily always better. I’m not a Rust developer btw. and Rust’s errorhandling is absolutely not the current gold-standard by my own judgement.
I just want to emphasize that It’s not the same thing - as you also hint to in the next sentence.
It kind of is the same thing: static analysis. The only difference is that the static analysis is broken out into two tools instead of one, so slightly more care needs to be taken to ensure the linter is run in CI or locally or wherever appropriate. To be clear, I think Rust is strictly better for having it in the compiler–I mostly just disagree with the implications in this thread that if the compiler isn’t doing the static analysis then the situation is no better than a dynamic language.
I didn’t say that writing tests is always slower or that using the compiler to catch these things is necessarily always better.
What did you mean when you said “It usually just means that the developers spent more writing tests or doing manual testing … The better the compiler and type-system, the fewer tests you need for the same quality.” if not an argument about more rigorous static analysis saving development time? Are we just disagreeing about “always”?
I mostly just disagree with the implications in this thread that if the compiler isn’t doing the static analysis then the situation is no better than a dynamic language.
Ah I see - that is indeed an exaggeration that I don’t share.
Are we just disagreeing about “always”?
First that, but it also in general has other disadvantages. For instance, writing tests or doing manual tests is often easy to do. Learning how to deal with a complex time system is not. Go was specifically created to get people to contribute fast.
Just one example that shows that it’s not so easy to decide which way is more productive.
Swallowing errors is the very worst option there is. Even segfaulting is better, you know at least something is up in that case.
Dynamic languages usually just throw an exception and those have way better behavior (you can’t forget, an empty catch is a deliberate sign to ignore an error, not an implicit one like with go), at least some handler further up will log something and more importantly the local block that experienced the error case won’t just continue executing as if nothing happened.
Time and time again wlroots proves how solid it is as a project. Really outstanding work!
It’s just a shame that Wayland didn’t dare to define such things on the protocol level in the first place. I mean, given the rock-sold colour space support in macOS, any sane engineer designing a new display manager/compositor in the 2010’s would have put colour management as a design-centerpiece. Libraries like Little CMS prove that you don’t even need to do much in terms of colour transformations by hand; simply define your surfaces in a sufficiently large working colour space and do the transformations ad-hoc.
From what I remember back then, the only thing the Wayland engineers seemed to care about was going down to the lowest common denominator and ‘no flickering’ (which they saw in X in some cases).
For instance, it is not possible to portably place an application window ‘at the top’, given one may not dare to assume this even though 99.99% of all displays support this. It would have made more sense to have ‘feature flags’ for displays or have more strict assumptions on the coordinate space.
In the end, a wayland compositor requires close to 50.000 LOC of boilerplate, which wlroots gracefully provides, and this boilerplate is fragile as you depend on proprietary interfaces and extensions. You can write a basic X display manager in 500 LOC only based on the stable X libraries. With all of X’s flaws, this is still a strong point today.
In the end, a wayland compositor requires close to 50.000 LOC of boilerplate, which wlroots gracefully provides, and this boilerplate is fragile as you depend on proprietary interfaces and extensions. You can write a basic X display manager in 500 LOC only based on the stable X libraries. With all of X’s flaws, this is still a strong point today.
This instinctually bothers me too, but I don’t think it’s actually correct. The reason that your X display manager can be 500 LOC is because of the roughly 370 LOC in Xorg. The dominance of wlroots feels funny to me based on my general dislike for monocultures, but if you think of wlroots as just “the guts of Xorg, but in ‘window manager userland’”, it actually is not that much worse than Xorg and maybe even better.
I don’t really get your criticism. Wayland is used on a lot of devices, including car displays and KIOSK-like installations. Does an application window even make sense if you only have a single application displayed at all times? Should Wayland not scale down to such setups?
Especially that it has an actually finely-working extension system so that such a functionality can be trivially added (either as a standard if it’s considered widely useful, or as a custom extension if it only makes sense for a single server implementation).
A Wayland compositors’ 50 thousands LOC is the whole thing. It’s not boilerplate, it’s literally a whole display server communicating in a shared “language” with clients, sitting on top core Linux kernel APIs. That’s it. Your 500 LOC comparison under X is just a window manager plugin, just because it operates as a separate binary it is essentially the same as a tiling window manager plugin for Gnome.
It’s just a shame that Wayland didn’t dare to define such things on the protocol level in the first place.
Then it would have taken 2× as long to get it out of the door and gain any adoption at all.
Routine reminder that the entire F/OSS ecosystem worth of manpower and funding is basically a rounding error compared to what Apple can pour into macOS in order to gain “rock-solid colour space support” from day zero.
For instance, it is not possible to portably place an application window ‘at the top’, given one may not dare to assume this even though 99.99% of all displays support this. It would have made more sense to have ‘feature flags’ for displays or have more strict assumptions on the coordinate space.
license all content you transmit through Firefox to Mozilla (“When you upload or input information through Firefox, you hereby grant us a nonexclusive, royalty-free, worldwide license to use that information to help you navigate, experience, and interact with online content as you indicate with your use of Firefox. -https://www.mozilla.org/en-US/about/legal/terms/firefox)
allow Mozilla to both sell your private information
If you’re already using Firefox, I can confirm that porting your profile over to Librewolf (https://librewolf.net) is relatively painless, and the only issues you’ll encounter are around having the resist fingerprinting setting turned on by default (which you can choose to just disable if you don’t like the trade-offs). I resumed using Firefox in 2016 and just switched away upon this shift in policy, and I do so sadly and begrudgingly, but you’d be crazy to allow Mozilla to cross these lines without switching away.
If you’re a macOS + Littlesnitch user, I can also recommend setting Librewolf to not allow communication to any Mozilla domain other than addons.mozilla.org, just in case.
👋 I respect your opinion and LibreWolf is a fine choice; however, it shares the same problem that all “forks” have and that I thought I made clear in the article…
Developing Firefox costs half a billion per year. There’s overhead in there for sure, but you couldn’t bring that down to something more manageable, like 100 million per year, IMO, without making it completely uncompetitive to Chrome, whose estimate cost exceeds 1 billion per year. The harsh reality is that you’re still using Mozilla’s work and if Mozilla goes under, LibreWolf simply ceases to exist because it’s essentially Firefox + settings. So you’re not really sticking it to the man as much as you’d like.
There are 3 major browser engines left (minus the experiments still in development that nobody uses). All 3 browser engines are, in fact, funded by Google’s Ads and have been for almost the past 2 decades. And any of the forks would become unviable without Apple’s, Google’s or Mozilla’s hard work, which is the reality we are in.
Not complaining much, but I did mention the recent controversy you’re referring to and would’ve preferred comments on what I wrote, on my reasoning, not on the article’s title.
I do what I can and no more, which used to mean occasionally being a Firefox advocate when I could, giving Mozilla as much benefit of the doubt as I could muster, paying for an MDN subscription, and sending some money their way when possible. Now it means temporarily switching to Librewolf, fully acknowledging how unsustainable that is, and waiting for a more sustainable option to come along.
I don’t disagree with the economic realities you mentioned and I don’t think any argument you made is bad or wrong. I’m just coming to a different conclusion: If Firefox can’t take hundreds of millions of dollars from Google every year and turn that into a privacy respecting browser that doesn’t sell my data and doesn’t prohibit me from visiting whatever website I want, then what are we even doing here? I’m sick of this barely lesser of two evils shit. Burn it to the fucking ground.
I think “barely lesser of two evils” is just way off the scale, and I can’t help but feel that it is way over-dramatized.
Also, what about the consequences of having a chrome-only web? Many websites are already “Hyrum’s lawed” to being usable only in Chrome, developers only test for Chrome, the speed of development is basically impossible to follow as is.
Firefox is basically the only thing preventing the most universal platform from becoming a Google-product.
Well there’s one other: Apple. Their hesitance to allow non-Safari browsers on iOS is a bigger bulwark against a Chrome-only web than Firefox at this point IMO.
I’m a bit afraid that the EU is in the process of breaking that down though. If proper Chrome comes over to iOS and it becomes easy to install, I’m certain that Google will start their push to move iOS users over.
I know it’s not exactly the same but Safari is also in the WebKit family and Safari is nether open source nor cross platform nor anywhere close to Firefox in many technical aspects (such as by far having the most functional and sane developer tools of any browser it there).
Pretty much the same here: I used to use Firefox, I have influenced some people in the past to at least give Firefox a shot, some people ended up moving to it from Chrome based on my recommendations. But Mozilla insists on breaking trust roughly every year, so when the ToS came around, there was very little goodwill left and I have permanently switched to LibreWolf.
Using a fork significantly helps my personal short-term peace of mind: whenever Mozilla makes whatever changes they’re planning to make which requires them to have a license to any data I input into Firefox, I trust that I will hear about those changes before LibreWolf incorporates them, and there’s a decent chance that LibreWolf will rip them out and keep them out for a few releases as I assess the situation. If I’m using Firefox directly, there’s a decent probability that I’ll learn about those changes after Firefox updates itself to include them. Hell, for all I know, Firefox is already sending enough telemetry to Mozilla that someone there decided to make money off it and that’s why they removed the “Mozilla will doesn’t and will never sell your data” FAQ item; maybe LibreWolf ripping out telemetry is protecting me against Mozilla right now, I don’t know.
Long term, what I personally do doesn’t matter. The fact that Mozilla has lost so much good-will that long-term Firefox advocates are switching away should be terrifying to Mozilla and citizens of the Web broadly, but my personal actions here have close to 0 effect on that. I could turn into a disingenuous Mozilla shill but I don’t exactly think I’d be able to convince enough people to keep using Firefox to cancel out Mozilla’s efforts to sink their own brand.
If Firefox is just one of three browsers funded by Google which don’t respect user privacy, then what’s the point of it?
People want Firefox and Mozilla to be an alternative to Google’s crap. If they’re not going to be the alternative, instead choosing to copy every terrible idea Google has, then I don’t see why Mozilla is even needed.
Well to be fair to Mozilla, they’re pushing back against some web standard ideas Google has. They’ve come out against things like WebUSB and WebHID for example.
How the heck do they spend that much? At ~20M LoC, they’re spending 25K per line of code a year. While details are hard to find, I think that puts them way above the industry norms.
I’m pretty sure that’s off by 3 orders of magnitude; OP’s figure would be half a US billion, i.e. half a milliard. That means 500M / 20M = 25 $/LOC. Not 25K.
I see your point, but by that same logic, shouldn’t we all then switch to Librewolf? If Firefox’s funding comes from Google, instead of its user base, then even if a significant portion of Firefox’s users switch, it can keep on getting funded, and users who switched can get the privacy non-exploitation they need?
TL;DR 90% of Mozilla’s revenue comes from ad partnerships (Google) and Apple received ca. 19 Bn $ per annum to keep Google as the default search engine.
Where did you get those numbers? Are you referring to the whole effort, (legal, engineering, marketing, administration, etc) ot just development?
That’s an absolutely bonkers amount of money, and while i absolutely believe it, im also kind of curious what other software products are in a similar league
Yeah, that seems like legal butt-covering. If someone in a criminalizing jurisdiction accesses these materials and they try to sue to the browser, Mozilla can say the user violated TOS.
This is just a lie. It’s just a lie. Firefox is gratis, and it’s FLOSS. These stupid paragraphs about legalese are just corporate crap every business of a certain size has to start qualifying so they can’t get their wallet gaped by lawyers in the future. Your first bullet point sucks - you don’t agree to the Acceptable Use Policy to use Firefox, you agree to it when using Mozilla services, i.e. Pocket or whatever. Similarly, your second bulletpoint is completely false, that paragraph doesn’t even exist:
You give Mozilla the rights necessary to operate Firefox. This includes processing your data as we describe in the Firefox Privacy Notice. It also includes a nonexclusive, royalty-free, worldwide license for the purpose of doing as you request with the content you input in Firefox. This does not give Mozilla any ownership in that content.
The text was recently clarified because of the inane outrage over basic legalese. And Mozilla isn’t selling your information. That’s not something they can casually lie about and there’s no reason to lie about it unless they want to face lawsuits from zealous legal types in the future. Why constantly lie to attack Mozilla? Are you being paid to destroy Free Software?
Consciously lying should be against Lobsters rules.
Let’s really look at what’s written here, because either u/altano or u/WilhelmVonWeiner is correct, not both.
The question we want to answer: do we “agree to an acceptable use policy” when we use Firefox? Let’s look in the various terms of service agreements (Terms Of Use, Terms Of Service, Mozilla Accounts Privacy). We see that it has been changed. It originally said:
“When you upload or input information through Firefox, you hereby grant us a nonexclusive, royalty-free, worldwide license to use that information to help you navigate, experience, and interact with online content as you indicate with your use of Firefox.”
Note that this makes no distinction between Firefox as a browser and services offered by Mozilla. The terms did make a distinction between Firefox as distributed by Mozilla and Firefox source code, but that’s another matter. People were outraged, and rightfully so, because you were agreeing to an acceptable use policy to use Firefox, the binary from Mozilla. Period.
That changed to:
“You give Mozilla the rights necessary to operate Firefox. This includes processing your data as we describe in the Firefox Privacy Notice. It also includes a nonexclusive, royalty-free, worldwide license for the purpose of doing as you request with the content you input in Firefox. This does not give Mozilla any ownership in that content.”
Are the legally equivalent, but they’re just using “nicer”, “more acceptable” language? No. The meaning is changed in important ways, and this is probably what you’re referring to when you say, “you don’t agree to the Acceptable Use Policy to use Firefox, you agree to it when using Mozilla services”
However, the current terms still say quite clearly that we agree to the AUP for Mozilla Services when we use Firefox whether or not we use Mozilla Services. The claim that “you don’t agree to the Acceptable Use Policy to use Firefox” is factually incorrect.
So is it OK for u/WilhelmVonWeiner to say that u/altano is lying, and call for censure? No. First, it’s disingenuous for u/WilhelmVonWeiner to pretend that the original wording didn’t exist. Also, the statement, “Similarly, your second bulletpoint is completely false, that paragraph doesn’t even exist:” is plainly false, because we can see that paragraph verbatim here:
So if u/WilhelmVonWeiner is calling someone out for lying, they really shouldn’t lie themselves, or they should afford others enough benefit of the doubt to distinguish between lying and being mistaken. After all, is u/WilhelmVonWeiner lying, or just mistaken here?
I’m all for people venting when someone is clearly in the wrong, but it seems that u/WilhelmVonWeiner is not only accusing others of lying, but is perhaps lying or at very least being incredibly disingenuous themselves.
Oh - and I take exception to this in particular:
“every business of a certain size has to start qualifying so they can’t get their wallet gaped by lawyers”
Being an apologist for large organizations that are behaving poorly is the kind of behavior we expect on Reddit or on the orange site, but not here. We do not want to or should we need to engage with people who do not make good faith arguments.
Consciously lying should be against Lobsters rules.
This is a pretty rude reply so I’m not going to respond to the specifics.
Mozilla has edited their acceptable use policy and terms of service to do damage control and so my exact quotes might not be up anymore, but yeah sure, assume that everyone quoting Mozilla is just a liar instead of that explanation if you want.
Sorry for being rude. It was unnecessary of me and I apologise, I was agitated. I strongly disagree with your assessment of what Mozilla is doing as “damage control” - they are doing what is necessary to legally protect the Mozilla Foundation and Corporation from legal threats by clarifying how they use user data. It is false they are selling your private information. It is false they have a nonexclusive … license to everything you do using Firefox. It is false that you have to agree to the Acceptable Use Policy to use Firefox. It’s misinformation, it’s FUD and it’s going to hurt one of the biggest FLOSS nonprofits and alternate web browsers.
It is false that you have to agree to the Acceptable Use Policy to use Firefox.
So people can judge for them selves, the relevant quote from the previous Terms of Use was:
Your use of Firefox must follow Mozilla’s Acceptable Use Policy, and you agree that you will not use Firefox to infringe anyone’s rights or violate any applicable laws or regulations.
This is a pretty incendiary comment and I would expect any accusation of outright dishonesty to come with evidence that they know they’re wrong. I am not taking a position on who has the facts straight, but I don’t see how you could prove altano is lying. Don’t attribute to malice what can be explained by…simply being incorrect.
FYI this is a change made in response to the recent outrage, the original version of the firefox terms included
Your use of Firefox must follow Mozilla’s Acceptable Use Policy, and you agree that you will not use Firefox to infringe anyone’s rights or violate any applicable laws or regulations.
Your locale is forced to en-US, your timezone is UTC, your system is set to Windows. It will put canvas behind a prompt and randomizes some pixels such that fingerprinting based on rendering is a bit harder. It will also disable using SVG and fonts that you have installed on your systems
Btw, I don’t recommend anyone using resist fingerprinting. This is the “hard mode” that is known to break a lot of pages and has no site-specific settings. Only global on or off. A lot of people turn it on and then end up hating Firefox and switching browsers because their web experience sucks and they don’t know how to turn it off. This is why we now show a rather visible info bar in settings under privacy/security when you turn this on and that’s also why we are working on a new mode that can spoof only specific APIs and only on specific sites. More to come.
Yes, if everyone is running a custom set of spoofs you’d end up being unique again. The intent for the mechanism is for us to be able to experiment and test out a variety of sets before we know what works (in terms of webcompat). In the end, we want everyone to look as uniform as possible
It breaks automatic dark mode and sites don’t remember their zoom setting. Dates are also not always localized correctly. That’s what I’ve noticed so far at least.
I am a proper Rust n00b ramping up as we speak. Even before getting into conceptual challenges of understanding lifetimes, I think it’s important to mention that the syntax was truly jarring for me. In literally every other language I’ve learned, a single unclosed quote means that I have written my program incorrectly in a very fundamental way. I’ve been programming for over 10 years so there’s a lot of muscle memory to undo here. Sorry if this bikeshed-y topic has been discussed to death, but since the article explicitly covers why lifetimes are hard to learn and doesn’t mention this point, I figured it’s fair game to mention again.
I personally like the notation, but I could see how it looks jarring. FWIW, Rust probably borrowed the notation from ML, where single quotes are used to denote type variables. For example, the signature of map in Standard ML is:
val map : (’a -> ’b) -> ’a list -> ’b list.
I got nerd-sniped thinking about how old the use of ' as a sigil might be. It’s used in lisp; I think it’s oldest use might go back to MACLISP in 1966. I think older dialects of lisps required that you say (quote foo) instead of 'foo. See section 1.3 “Notational Conventions” on page 3 of the MACLISP manual (large PDF warning).
Is there a reason that it’s only for lifetimes and not all type parameters? My guess would be because it makes them easy to distinguish (and since type parameters are more common, they get the shorter case), but I could be wrong of course.
I recently saw it in reading the various Cyclone papers, where they introduced it for identifying differing regions, with those having different lifetimes. However, I believe Cyclone was itself drawing inspiration from ML (or Caml).
It also has mention of “borrowing” in what may be a similar fashion.
The ' is an odd one. I had another skim of the history of Standard ML but it isn’t very interested in questions of notation. However, it reminded me that ML type variables are often rendered in print as α, β instead of 'a, 'b, which made me think this might be a kind of stropping. “Stropping” comes from apostrophe, and prefix ' was one form it could take. (Stropping varied a lot depending on the print and punch equipment available at each site.) But this is a wild guess.
I got nerd-sniped thinking about how old the use of ’ as a sigil might be. It’s used in lisp
Oh, yeah, it took me a good six months to stop typing a ( right after the '. The joke about Greenspun’s tenth rule practically writes itself at this point :-).
I’m only mentioning this for the laughs. As far as I’m concerned, any syntax that’s sufficiently different from that of a Turing tarpit is fine. I mean, yeah, watching the compiler try to figure out what the hell I had in mind there was hilarious, but hardly the reason why I found lifetimes so weird to work with.
I haven’t heard this particular challenge before. I came to Rust long after I learned Lisp and Standard ML, so it never occurred to me that it would be jarring, but if you’ve only worked in recent members of the ALGOL family I can see that being the case.
What do you mean muscle memory? Do you usually double your quotes manually? Does your IDE not do it for you?
Not trying to “attack” you or anything, genuinely curious as these kind of syntactical changes are more-or-less invisible to me when writing code due to a good IDE “typing” what I intend based on context for the most part.
As for a single single quote being jarring, I believe it does have some history in LISPs for “unquote” or for symbols. Possibly, the latter case was the inspiration in case of Rust?
Edit: I see there is a much better discussion in the sibling thread regarding its origin.
Ah yes, now that you mention it, “muscle memory” is not the right phrase here. I didn’t mean muscle memory in the (proper) sense of a command that you’ve been using for years, and then now you need to use slightly differently. What I meant was that for years, whenever I saw a single quote, I expected another quote to close it somewhere later in the code. And now I have to undo that expectation.
May not be completely relevant to the article, but I always wanted to write a blog post titled something like “Visual programming is here, right before our eyes”.
I really enjoyed learning about CellPond, but I don’t believe that we have to go to such “depths” for visual programming - a featureful IDE with a well-supported language is pretty much already that.
Programs in most PLs are not 1D text, they are 2D text with some rudimentary visual structuring, like blocks, indentation, alignment etc.
This is turbo-charged by a good IDE that makes operations operate on symbolic structures and not on a list of characters. E.g. something as simple as putting the caret on a variable will immediately “bring up another layer of information” as a graph showcasing the declaration and uses of it. I can also select an expression (and can expand it syntactically) and choose to store that expression as a new node. Intellij for example will even ask if I want to replace the same expression at different places, effectively merging nodes. And I would argue that experienced developers look at code just like that - we are making use of our visual cortex, we are pattern matching on blocks/expressions, etc. We are not reading like you would read a book, not at all.
I can also see all the usages of a function right away, click on its definition, its overrides, parent class/interface right from my IDE. These are semantically relevant operations on specific types of nodes, not text.
Also, writing is just a very efficient and accurate visual encoding of arbitrary ideas, so it only makes sense to make use of it where a more specific encoding can’t be applied (sort of a fallback). “Nodes” (blocks of code) being text-based doesn’t take away from “visual programming” in my book. I would actually prefer to *extend” ordinary programming languages with non-textual blocks, so that I can replace, say, a textual state machine with a literal, interactive FSM model, and my IDE will let me play with it. (If we look at it from far away, Intellij again has a feature that lets you test regular expressions by simply writing a sample text, and it showcases which part matches and which doesn’t). Because a graph of a state machine is more “efficient”/“human-friendly” than a textual representation. But I think it’s faulty to think that we could have a representation that will apply equally well for different areas - this is exactly what human writing is.
I care a lot if it takes 10 seconds to start! I quit vim a lot while I’m working. In part because it takes 0 seconds to start. Though the Haskell language server kinda ruins that.
This is simple vs. easy. It definitely is very simple mathematics, pretty basic set theory (not Venn diagram basic, as it needs functions&relations, but nothing even remotely crazy). It probably isn’t easy for most people, as I think this “foundations of mathematics” stuff is not frequently taught, and it does require somewhat more abstract thinking.
Perhaps even the question itself is a type error.
Sure, programming is mostly done via Turing-complete languages, and a Turing-completeness is surprisingly easy/trivial to reach. (Doesn’t make “the sum of its parts” any easier though, two trivial function’s composition may not end up as a trivial function.)
As for why I consider it a type error: programming is the whole act and art of building and maintaining computable functions with certain properties, if we want to be very abstract.
Naming variables, encoding business logic/constraints in a safe way, focusing on efficiency are all parts of programming itself. The computing model is just one (small) part of it all.
The computing model is just one (small) part of it all.
Related:
“(Alan) Turing’s work is very important; there is no question about that. He illuminated many fundamental questions. In a sense, however, it is unfortunate that he chose to write in terms of ‘computability’ especially in view of development, subsequent to Turing’s work, of modern computers. As a theory of ‘computability’, Turing’s work is quite important, as a theory of ‘computations’, it is worthless.
It has even been shown that the two approaches are equivalent fixed point algorithms; the first gives the greatest and the latter the least fixed point
The author definitely has more experience with GCs than I do, and perhaps it’s only for this article’s sake that they focus so much on pointers themselves, but.. I feel this is a bit misguided.
A garbage collector (ref counting included) operates semantically on handles. In the ref counting case that handle will as an internal detail increment/decrement a counter. In the tracing case it may have write or read-barriers, but the point is that these are objects that have their own semantics by which they can be accessed. It’s an entirely separate implementation detail if they are represented as an integer corresponding to a memory address/region or something else. There are languages that can express express these semantics natively (I believe RAII is a core primitive here), but we have seen user-level APIs for ref counting (e.g. glib), and runtimes may choose to simply handle it themselves e.g. by generating the correct code via a JIT compiler.
Of course a runtime’s assumptions can be violated, but as the author even points out, the JVM for example hands out handles for FFI purposes, so not sure how much of a problem this actually is. As for the concrete design, I don’t have enough experience to provide feedback - I do like the aspect that code can be monomorphic, but given that tracing GCs have to trace the heap, I feel the runtime checks for that phase would be a negative tradeoff.
Also, to continue with their analogy, if you reach for native code, you effectively call a burgler to your block. Even if you only hand them a key and don’t tell them which house is yours and have an alarm system so you have to know the password, they might just break the window in.
Not really. PostmarketOS has put in the work to get it functional, but AFAIK it’s not being upstreamed because Alpine is not interested.
See https://wiki.postmarketos.org/wiki/Systemd and relevant entries in the blog.
Well….
Grub reimplements several filesystems (btrfs, xfs, ext* etcetcetc), image formats, encryption schemes and misc utilities.
A good example is how luks2 support in Grub doesn’t support things like different argon implementations.
Or… You could not. If you focus on EFI you could use an EFI stub to boot your kernel. Or boot a UKI directly. None of it needs systemd.
Booting directly into an UKI/EFI binary is not particularly user-friendly for most users, which is the missing context of the quote. The statement isn’t about needingsystemd, it’s about offering a replacement to grub.
Booting directly into an UKI/EFI binary is not particularly user-friendly for most users
I’d say it’s the opposite. An average user doesn’t need to pick a kernel to boot. They use the kernel provided by their distribution and don’t want to see any of the complexities before they can get to the browser/email/office suite/games. It’s us—nerds—want to build our own kernles and pick one on every boot. Supermajority of users don’t need multiple kernels. They don’t know what a kernel is, don’t need to know, and don’t want know. They don’t want to know the difference between a bootloader and a kernel. They don’t care to know what UKI even is. They’re happy if it works and doesn’t bother them with cryptic test on screen changing too fast to read even a word of it. From this perspective—average user’s point of view—UKI/EFI stub is pretty good as long as it works.
If you want to provide reliable systems for users you to provide rollback support for updates. For this to work you do need to have an intermediate loader. This is further complicated by the Secure Boot that needs to be supported on modern systems.
I’d say it’s the opposite. An average user doesn’t need to pick a kernel to boot.
sd-boot works just fine by skipping the menu entirely unless prompted. Ensuring that the common use-case does not need to interact, nor see, any menu at boot.
Isn’t this built in to UEFI? EFI provides a boot order and a separate Next Boot entry. On upgrade system can set Next Boot to the new UKI and on successful boot into it system can permanently add it to the front of boot order list.
This is further complicated by the Secure Boot that needs to be supported on modern systems.
I’m not certain what you’re referring to here. If you say that this makes new kernel installation a bit more complicated than plopping a UKI built on CI into EFI partition than sure, but this is a distro issue, not UEFI/UKI issue. Either way it should be transparent for users, especially for the ones who don’t care about any of this.
On upgrade system can set Next Boot to the new UKI and on successful boot into it system can permanently add it to the front of boot order list.
UEFI is not terribly reliable and this is all bad UX. You really want a bootloader in between here.
This allows you to do a bit more fine grained fallback bootloading from the Boot Loader Specification which is implemented by sd-boot and partially implemented by grub.
I’m not certain what you’re referring to here. If you say that this makes new kernel installation a bit more complicated than plopping a UKI built on CI into EFI partition than sure, but this is a distro issue, not UEFI/UKI issue. Either way it should be transparent for users, especially for the ones who don’t care about any of this.
UKIs are not going to be directly signed for Secure Boot, this is primarily handled by shim. This means that we are forced to be forced through a loop where shim is always booted first which then would boot your-entry.efi.
However, this makes the entire boot mechanism of UEFI moot as you are always going to have the same entry on all boots.
UEFI is not terribly reliable and this is all bad UX.
I used to boot my machines with just the kernel as EFI stub and encountered so many issues with implementations that I switched to sd-boot.[^1]
Things such as:
Forgetting boot entries whilst not having any mechanism to add them back.
Not storing arguments.
Forgetting arguments if the list order was changed.
Forgetting arguments on EFI upgrades.
Deduplicating logic that only could store one entry per EFI partition.
I’m surprised any of them did anything more than boot “Bootmgfw.efi” since that seems to be all they’re tested with.
Fortunately, nowadays you can store the arguments in UKI (and Discoverable Partitions Specification eliminates many of those). The rest was still terrible UX if I got anywhere off the rails.
[^1]: Even on SBCs, Das U-Boot implements enough of EFI to use sd-boot, which is a far better experience than the standard way of booting and upgrading kernels on most SBCs.
I’m one of those using an unusual window manager (sawfish) with xfce4. I’m starting to dread the inevitable day that I’ll have to change back to some mainstream system that will of course include all the pain points that led me to my weird setup in the first place. Fine, so X11 needs work, but replacing it with something as antimodular as wayland seems to be is just depressing. Something must be done (I agree!); this is something (er, yes?); therefore this must be done (no, wait!!). We’ve seen it with systemd and wayland recently, and linux audio infrastructure back in the day; I wonder what will be next :-(
First, reports of X’s death are greatly exaggerated. Like the author said in this link, it probably isn’t actually going to break for many years to come. So odds are good you can just keep doing what you’re doing and not worry too much about it.
But, even if it did die, and forking it and keeping it going proves impossible, you might be able to go a while with a fullscreen xwayland rootful window and still run your same window manager in there. If applications force the wayland on you, maybe you can nest a wayland compositor in the xwayland instance to run that program under your window manager too. I know how silly that sounds - three layers of stuff - but there’s a good chance it’d work, and patching up the holes there may be a reasonable course of action.
So I don’t think the outlook is super bleak yet either.
I’m not convinced that building something like those kinds of window managers on top of wlroots is significantly more difficult than building those kinds of window managers on top of X11. There’s a pretty healthy ecosystem of wlroots-based compositors which fill that need at least for me, and I believe that whatever is missing from the Wayland ecosystem is just missing because nobody has made it yet, not because it couldn’t be made. Therefore, I don’t think your issue is with “antimodularity”, but with the fact that there isn’t 40 years of history.
I assure you, my issue is with antimodularity. The ICCCM and EWMH are protocols, not code. I can’t think of much tighter a coupling than having to link with and operate within a 60,000-line C-language API, ABI and runtime. (From a fault isolation perspective alone, this is a disaster. If an X window manager crashes, you restart it. If the wayland compositor crashes, that’s the end of the session.) You also don’t have to use C or any kind of FFI to write an X window manager. They can be very small and simple indeed (e.g. tinywm in 42 lines of C or 29 lines of Python; a similarly-sized port to Scheme). You’d be hard pressed to achieve anything similar using wlroots; you’d be pushing your line budget before you’d finished writing the makefile.
There is nothing inherent in Wayland that would prevent decoupling the window manager part from the server part. Window managers under X are perfectly fit to be trivial plugins, and this whole topic is a bit overblown.
Also, feel free to write a shim layer to wlroots so you can program against it in anything you want.
I’m talking about modularity. Modularity is what makes it feasible to maintain a slightly-different ecosystem adjacent to another piece of software. Without modularity, it’s not feasible to do so. Obviously there’s nothing inherently preventing me from doing a bunch of programming to get back to where I started. But then there’s nothing inherently preventing me from designing and implementing a graphics server from scratch in any way I like. Or writing my own operating system. There’s a point beyond which one just stops, and learns to stop worrying and love the bomb. Does that sound overblown? Perhaps it always seems a bit overblown when software churn breaks other people’s software.
With all due respect, I feel your points are a bit demagogue without going into a specific tradeoff.
The wayland protocol is very simple, it is basically a trivial IPC over Linux-native DRM buffers and features. Even adding a plug-in on top of wlroots would still result in much less complexity than what one would have with X. Modularity is not an inherent goal we should strive for, it is a tradeoff with some positives and negatives.
Modular software by definition has more surface area, so more complexity, more code, more possibility of bugs. Whether it’s worth it in this particular case depends on how likely we consider a window manager “plugin” to crash. In my personal opinion, this is extremely rare - they have a quite small scope and it’s much more likely to have a bug in the display server part, at which point X will fail in the exact same manner as a “non-modular” Wayland server.
I’m not sure exactly what pain points led you to your current setup, but I don’t think the outlook is that bleak. There are some interestingly customizable, automatable, and nerd-driven options out there. Sway and hyprland being the best-known but there are more niche ones if you go looking.
I use labwc and while it’s kind of enough, it’s years away from being anywhere near something like Sawfish (or any X11 window manager that’s made it past version 0.4 or so). Sawfish may be a special case due to how customizable and scriptable it is but basically everything other than KWin and Gnome’s compositor are basically in the TWM stage of their existence :).
Tiling compositors fare a little better because there’s more prior art on them (wlroots is what Sway is built on) and, in many ways, they’re simpler to write. Both fare pretty badly because there’s a lot of variation; and, at least the last time I tried it (but bear in mind that was like six years ago?) there was a lot more code to write, even with wlroots.
There are some interestingly customizable, automatable, and nerd-driven options out there.
Ah, sorry, I misread that via this part:
There are some interestingly customizable, automatable, and nerd-driven options out there.
…as in, there are, but none of the stacking ones are anywhere near the point where they’re on-par with window managers that went past the tinkering stage today. It’s certainly reasonable to expect it to get better; Wayland compositors are the cool thing to build now, X11 WMs are not :). labwc, in fact, is “almost” there, and it’s a fairly recent development.
Just digging into Hyprland and it’s pretty nice. The keybindings make sense, especially when switching windows, the mouse will move to the focused window. Necessary? Probably not. But it’s a really nice QoL improvement for a new Linux user like myself.
I’m pretty much in the same situation as you. I’m running XFCE (not with sawfish, but with whatever default WM that it ships with). I didn’t really ever explore using Wayland since X11 is working for me so I found no reason to switch yet. For a while it seems like if you wanted a “lightweight” desktop environment you’re stuck with tiling Wayland environments like Sway. I still prefer stacking-based desktop environments, but don’t really want to run something as heavy as GNOME or KDE. I’ll probably eventually switch to LXQt which will get Wayland support soon.
Great article! The scientific method at work. It’s always good to question and revalidate old results, especially since hardware evolution (caches, branch prediction…) can change the “laws of nature”, or ant least physical constants, out from under us.
It sounds to me like the choice of benchmarks could be an issue. I’ve been rereading various GC papers recently; they frequently seem to use compilers, like javac, as benchmarks, which seems reasonable.
It sounds to me like the choice of benchmarks could be an issue. I’ve been rereading various GC papers recently; they frequently seem to use compilers, like javac, as benchmarks, which seems reasonable.
I’d have thought compilers were pretty unrepresentative for GC. They create a big AST as they parse, which exists as long as the program is valid. Then they walk it and generate code, some of which may be shortlived, but something like javac doesn’t really do any optimisation (it used to, then HotSpot spent the first bit of the load having to undo the javac optimisation) and is little more than a streaming write of the AST. Most programs have a much broader set of lifetimes than compilers.
Hans Boehm has a nice anecdote about adding the BDW GC to GCC. In his first version, it made GCC quite a lot faster. It turned out that it was equivalent to just never calling free: the compiler was so short-lived that the threshold for triggering GC was never reached. LLVM learned from this and mostly uses arena allocators, and has an option (on by default in release builds, I believe) for clang not to ever free things. Clangd needs to reclaim memory because it’s a long-lived thing, but most objects in clang are live for most of the duration that the tool runs and so it’s faster to simply let exit be your GC.
For what it’s worth, ZGC and Shenandoah both experienced a large throughput boost after moving to a generational GC (they were already low-latency=bounded pause time), on real world benchmarks as well.
I would like to mention GraalOS as a related development, as it’s not too well known (not affiliated, just follow GraalVM)
It would allow the safe hosting of applications without virtualization, in a way maybe similar to CGI, but with Graal-supported languages as scripts (JVM languages, Javascript, Python, Ruby, but also LLVM languages).
Is there anyone here that understands tail calls (tail call recursion?) well enough to explain why it is such a big deal, but in simple terms a lowly Python coder can grok? My attempts to understand it lead me here:
When a function gets called, you build a whole new context (a “frame”?) for it to run in.
The frame encapsulates the variables in each function call from its caller and callees (functions called inside the frame).
Creating a frame is heavy.
The deepest frames, i.e., ones that don’t call another function, are called tail calls.
Since tail calls don’t call other functions, they don’t benefit from being in their own frame because there’s nothing to subsequently encapsulate them from.
Tail call optimization replaces the creation of the tail frame with an inline execution of the function.
The inline execution is much faster than making a new frame.
Assuming that’s right, the thing I can’t wrap my head around is aren’t tail calls edge cases? The surface area of a sphere (the “tail calls”) grows much slower than the volume (the non-tail function calls) so shouldn’t optimizing for it be a tiny improvement in execution across all calls?
You have got some of the key ideas, but what’s important here is the specific context: this is about optimizing CPython’s interpreter loop.
First, a small correction: functions that don’t call other functions are usually called “leaf functions”. Tail calls are different: a tail call is when the last thing that a non-leaf function does is call another function, eg,
return somefunc(some, args);
Normally what happens is the outer func calls somefunc() which pushes a frame on the stack, does its thing, pops its frame, then returns, then the outer func pops its frame and returns. If the compiler does “tail call optimization” then what happens is the outer func pops its frame, then does a goto to somefunc(), which runs as before, and when it returns it goes straight to the outer func’s caller.
Tail call optimization turns a tail call into a kind of goto-with-arguments.
Back to the context. A classic bytecode interpreter loop looks like (in C-ish syntax):
for (;;) {
switch (bytecode[program_counter]) {
case OP_ADD:
// add some numbers
program_counter++;
continue;
case OP_JUMP:
// work out where
program_counter = jump_target;
continue;
// other opcodes etc usw
}
}
There are a couple of reasons why this isn’t as good as it could be:
Each opcode involves two branches, one at the top of the loop to dispatch to the appropriate opcode, and one back to the top when the opcode is done;
The dispatch branch is super unpredictable, so the CPU has a hard time making it fast.
These issues can be addressed using computed goto, which is what the CPython interpreter normally uses. The opcode dispatch is copied to the end of every opcode implementation, using an explicit jump table. Normally the compiler will compile a switch statement to a jump table behind the scenes, but by making it explicit the interpreter gets better control over the shape of the compiled code.
It is initialized like jump_table[OP_ADD] = &IMP_ADD using a GNU C extension that allows you to take the address of a label and use it in a computed goto statement.
This is better, but compilers still find it hard to do a good job with this kind of interpreter loop. Part of the problem is that the loop is one huge function containing the hot parts of the implementations of every opcode. It’s too big with too many variations for the compiler to understand things like which variables should stay in registers.
This new interpreter loop uses tail calls as a goto-with-arguments to help the compiler. Each opcode is implemented by its own function, and the dispatch is a tail call to a function pointer.
The tail call arguments are the variables that need to stay in registers. The release notes talk about compiling with the “preserve_none” calling convention, which helps to ensure the tail call is more like a goto-with-arguments-in-registers and avoid putting variables in memory on the stack.
This time the fun_table contains standard function pointers.
That’s the basic idea, but it has some cool computer science connections.
The idea of a function call as goto-with-arguments was popularized by the “lambda-the-ultimate” papers which talked about using the lambda calculus and continuation passing style when compiling Scheme. Low-level compiler intermediate representations such as continuation-passing style and A-normal form are literally goto-with-arguments-in-registers. The classic static single assignment form (SSA) can be viewed as a relatively simple transformation of CPS.
So the idea is to be able to program the compiler’s back end more directly, with less interference from the front end’s ideas about structured programming, to get the interpreter loop closer to what an expert assembly language programmer might write.
Thanks for the great explanation! I think I am mostly following it, although I have one follow-up:
The tail call arguments are the variables that need to stay in registers. The release notes talk about compiling with the “preserve_none” calling convention, which helps to ensure the tail call is more like a goto-with-arguments-in-registers and avoid putting variables in memory on the stack.
Would it ever be the case that an opcode could have too many required arguments to fit in the remaining registers? Or would that never be the case because these opcodes map to instructions for a given architecture and the arguments map to parameter registers?
The arguments to these tail calls take the place of the local variables that would have been declared above the for(;;) loop in the classic version. They are the variables that are used by basically all the opcodes: the bytecode pointer, the program counter, interpreter stack, etc. So it’s a fixed list that doesn’t depend on the opcode. INTERP_ARGS was supposed to suggest a pair of C macros that contain the boilerplate for opcode function declarations and calls - the same boilerplate each time. The goto-with-arguments is being used as goto-and-here-are-our-local-variables.
The arguments to an opcode are typically fetched from following elements in the bytecode array, or they might be implicit on the interpreter stack. In the first example here https://docs.python.org/3/library/dis.html the right hand column shows an argument for each opcode, except for RETURN_VALUE which implicitly uses the value on top of the stack.
It’s a bit confusing because we’re dealing with two levels, the C interpreter and the bytecode it is interpreting. I discussed the C call stack when talking about tail calls, but that’s an implementation detail that isn’t directly visible in the interpreter source code; but there’s also an explicit interpreter stack used for bytecode expression evaluation, eg, OP_ADD typically pops two values, adds them, and pushes the result.
I don’t know anything about Python internals, but on the Java front certain instructions are very high-level, e.g. they might cause “expensive” class loading involving IO, so they definitely not map to instructions.
I think parent commenter mostly meant that writing the code this way would “implicitly assure” that the compiler will prioritize arguments (like program counter) to fit in registers, over non-arguments, of which there might be several depending on complexity of the instruction, and otherwise all the variables would compete for the same registers.
Slightly unrelated, but you seem to be well-versed in the topic: while I was writing a JVM interpreter, I was researching the difference between this “classic interpreter loop” as you outlined above vs the version that copies the ‘opcode dispatch’ part to the end of each opcode implementation (Wikipedia seems to call it token threading), and while your reasoning (the dispatch branch being unpredictable) seems sound and this is exactly the same I have read/heard from others, a very experienced colleague and my minimal benchmark showed that this may not actually result in improved performance on modern hardware.
I am asking if anyone has more conclusive “proof” that this is still the case on modern hardware? I made a tiny language with a couple of basic opcodes and wrote an interpreter in Zig for both versions, making sure that function calls implementing the opcodes are inlined, and the interpreter loop was actually faster on the few (hand-written) toy example programs I wrote in this “language”. My assumption is that my language was just way too simple and there might have been some i-cache effect at play, which would not be the case for a more realistic language?
Also, I haven’t had a compiler for this “language” and the kind of program I could write by hand might have been way too simplistic as well to count as “proper benchmark”. Anyway, I am looking to find some time to benchmark this better.
You also need either __attribute__((musttail)) or [[clang:musttail]].
C’s usual calling conventions make it difficult to do tail call optimization in general, so it needs an explicit annotation. (It’s almost hopeless in C++ because destructors make most things that look like tail calls into non-tail calls.)
(Edit: lol @fanf posted a great explanation while I was typing this out and brings up some great points about computed gotos)
Note that this optimization is for the interpreter implementation (i.e. the C interpreter code), not Python semantics. Tail calls in Python code will still always allocate a stack frame.
Basically, this change improves the way the main CPython bytecode interpreter loop works. A simple way to implement a bytecode interpreter loop is:
while ((op := bytecode[current_instruction]):
match op:
case ADD:
# ... add logic ...
case CALL:
# ... call func ...
# etc..
current_instruction += 1
However, this approach has some problems. We end up with a giant function that’s hard for a C compiler to optimize (there are a ton of branches (which the branch predictor doesn’t like), lots of local variables (which makes register allocation difficult since we have a ton of variables and not enough registers to fit them into), any inlined nested calls will bloat the whole function further (negatively impacting CPU instruction cache), etc). At minimum, we have to jump twice: once when switch on the opcode, then again at the bottom of the loop to goto the top.
Since the functions are smaller, the compiler can go ham with optimizations. Since each function has the same call signature, all the compiler needs to do is jmp (i.e. goto) the next function directly without any cleanup/resetting for an outer loop. Since it’s just a jump (because of the tailcall optimizations), we don’t allocate a stack frame and have eliminated the jump at the bottom of the loop to get back to the top.
Saving on creating stack frames is definitely part of it. And you’re right about tail recursion being an edge case (compared to an explicit loop in Python). Tail calls, however, not so much. return obj.Method() happens all the time.
Further, having tail call optimization allows you to implement “subroutine threaded” interpreters, in which the interpreter loop is a tail recursion based “call the next instruction.” This is often quite fast, though not generally faster than “direct threaded code.”
Further further, you could compile to continuation passing style, which turns your program into a series of tail calls, that could then be optimized. In this case, all the control flow logic could be simplified by using the continuations, too.
The parentheses in the following sentence threw me off, and I missed a subtlety of what it’s saying:
you’re right about tail recursion being an edge case… Tail calls, however, not so much.
For those not aware, “tail recursion” and “tail call” are different concepts: a “tail call” is anything of the form return f(...) (or equivalent for method calls, lambdas, etc.). @apg is saying this form is common.
In contrast, “tail recursion” is when a function makes a tail call to itself, e.g. a Python function like:
@apg is saying that it’s rare to see such tail recursion in Python (after all, it’s not super common for Python functions to call themselves at all; let alone as a tail call).
Those ideas can also be generalised:
A tail call is a function call which appears in “tail position”. We can apply that idea of “tail position” to other language constructs besides function calls.
We can generalise the idea of tail recursion beyond functions which immediately call themselves, to a set of mutually-recursive functions which make tail calls to each other. For example, (is_even, is_odd) = (lambda n: is_odd(n-1) if n else True, lambda n: is_even(n-1) if n else False)
One modification I’d make to your expansion: “tail position” is best explained with the concept “the call is the last thing before returning.”
def loop(x)
if x > 0:
return x + next(x - 1)
return x
The call to next() is not in tail position. The addition operation is in tail position, and so this can’t be optimized. To account for this, you’ll often rewrite this code:
def loop(x, accum=0):
if x > 0:
return next(x - 1, accum+x)
return accum
Now, the addition happens before the call to next and the call is the last thing done before returning. This can be optimized.
In contrast, “tail recursion” is when a function makes a tail call to itself,
Right.
In functional programming languages, you usually have to use recursion to write a loop. Languages like Scheme and SML commonly support “proper” tail calls, i.e. tail call optimization is guaranteed. The effect is that tail recursion compiles into code that is as efficient as a while loop in an imperative language.
str is a dynamically sized type which is basically the same as [T], but for string data (bytes which are guaranteed to be UTF-8). &String is a reference to a String which is allocated on the heap, whereas an &str could point to a string allocated somewhere else (in particular, string literals are not allocated on the heap).
If you understand the relationship between Vec<T> and &[T], the relationship between String and &str is the same.
A lot of people have given you some great answers, and I’m a bit impartial here, but I really like the diagrams in the Rust book if you’re having trouble visualizing the differences between &String and &str:
A String is an owned type, it’s is a length, a capacity, and a pointer to an exclusive buffer which it frees on drop (or when realloc-inc). An &String is a reference to that specific structure.
But lots of string-like are not that e.g. a string literal is a segment of the binary, or you can get a string from a slice of binary data.
This is the same relationship as Vec, &Vec, &[], and [] (String/str are wrappers to those which guarantee the underlying buffer always contains valid UTF8).
Nobody seems to have mentioned this particular point explicitly: &str can represent substrings of an existing string, whereas &String cannot. (That’s what “slice” means, of course, but the implications weren’t obvious to me when I worked through this the first time.)
This helps to support highly expressive yet efficient programming. For example, you can idiomatically iterate over .lines or .split without any heap allocations (neither from the iterator, nor from the substrings). You might have already done it without realizing what you didn’t have to do!
The reason you almost never see str (except in signatures where a reference is composed later) is because it’s a slice, which means it doesn’t have a statically known size. Rust requires dynamically sized types to be accessed though a reference, since references are always sized (more precisely, they’re pointers, and pointers are sized).
A String is essentially Box<str> with some additional metadata (actual length, capacity). As such a reference like &String doesn’t point to the slice itself; it points to the (heap) structure containing the slice.
TL;DR: both exist because they express different concepts / have different ABI consequences. Comparatively, a &str is to a char * as a &String is to a &std::string.
Conceptually, str is to [u8] what String is to Vec<u8>. In other words, str is pretty much an array of bytes, [u8], but with the additional restriction that it has to be valid utf8.
So, as https://lobste.rs/s/cbvnzl/string_vs_str#c_jqzhu7 already mentioned, the difference between &str and &String is analogous to the difference between a slice of bytes, &[u8], and &Vec<u8>[^footnote below] mostly, which is not perfect).
It just so happens that str is rarely constructed.
I think the type of "quoted strings" could have been str, but then you’d have to use &"quoted string" instead of "quoted string" almost all the time. So, they just made its type &str.
[^footnote below]: &[u8], and &Vec<u8>, btw, can also be annoying to distinguish; this is something I wish Rust did better, but I don’t have a better design in mind. Rust’s solution is AsRef, but it often adds to the confusion, since &String can often act like &str, but also often refuses to act as &str depending on what you are doing.
Because Rust in some sort made the mistake of assuming most people want to append to strings all the time, so String carries excess capacity. I think the better call would have been to say that str and &str are the types to represent an owned fix length string and a reference to it, and have a StringBuilder for the uncommon case where someone actually needs to append to a string.
That’s only true if you’ve appended to it. (since it uses the same capacity doubling strategy as Vec, and its initial capacity is the same as whatever you try to put in it)
I think the better call would have been to say that str and &str are the types to represent an owned fix length string and a reference to it
That’s literally what we already have today.
and have a StringBuilder for the uncommon case where someone actually needs to append to a string.
That’d be a weird name IMO. String is just a Vec with UTF-8 data. Should Vec be ArrayBuilder? In any case, languages that let you append to things called String are supremely common, so it’s not like the current naming is anything anyone would be confused by.
You’re speaking about a different form of capacity, @mitsuhiko is taking about how String is a (pointer, len, capacity) and how it could be (pointer, len) if it were immutable.
That’d be a weird name IMO.
I personally have joked that I want a Rust 2.0 only to turn String into StrBuf. This would make it consistent with Path and PathBuf,
Should Vec be ArrayBuilder?
I also agree that trying to achieve consistency in one place can still leave other consistencies. I think that Vec is common enough that ArrayBuf would be a bit odd.
In any case, languages that let you append to things called String are supremely common
Java is a very prominent example of String being immutable and StringBuffer being mutable. C# has String and StringBuilder. In Python, strings are immutable, but with dynamic typing, it feels like they’re mutable. Ruby almost went the same way for Ruby 3.0, but Matz was worried about breaking code. Go has immutable strings.
I’m not so sure it’s clear which is more popular.
That said, all of this is really moot for Rust, and I also agree with @withoutboats that it’s also pretty irrelevant to the question asked.
You’re speaking about a different form of capacity, @mitsuhiko is taking about how String is a (pointer, len, capacity) and how it could be (pointer, len) if it were immutable.
That’s how I interpreted it initially, but the word “excess” made me question that. “excess capacity” to me, means more capacity than you’re using. But I concede that your interpretation is reasonable.
I personally have joked that I want a Rust 2.0 only to turn String into StrBuf. This would make it consistent with Path and PathBuf,
I like that.
I’m not so sure it’s clear which is more popular.
Oh, certainly. I’m just saying that it’s common enough that I don’t think it should be a point of confusion.
I was thinking more of how &str doesn’t implement Add in Rust, that is, in Python you can
That has more to do with explicit costs, the Python version semantically allocates 4 different strings, although in practice it gets constant folded by the peephole optimiser to only allocate one (and it’s a constant in the code object).
This is really more of an API choice than an inherent property of static typing.
Indeed, after all you can write the “python” code in Go or Java or haskell (by modifying it a bit in the latter case).
Thanks for clarifying. Yes, that makes sense - I hadn’t considered that you can concatenate strings like that in Python, which admittedly does have a very mutable feel.
You’re speaking about a different form of capacity, @mitsuhiko is taking about how String is a (pointer, len, capacity) and how it could be (pointer, len) if it were immutable.
But you’d still need a type to represent a stack allocated string, so str and &str wouldn’t disappear, right? All you’d do is add an immutable String type, which would only add to the confusion.
That’s not exactly what I understand. I’m not him though so I could be wrong. You’re not wrong that you’d still need some sort of slice type. What I understand is that he’s saying the types should be like this:
String turns into StringBuilder
str is repurposed to mean what we call String today.
&str is repurposed to mean what we call &String today.
He didn’t say what the name of today’s str should be. I think the difficulty there is one of the problems with this specific naming scheme, and that’s not even getting into how str is the only language type here (and it almost wasn’t, and maybe that would have been the right call too, but it’s not completely obvious).
&str is layout compatible to be able to point to both a str (Box<str>) and a slice of utf-8. It would be possible to get away with one reference type for both things here.
But in the first case (str (Box<str>)) you have a narrow pointer to a place holding a wide pointer (the box) while in the second case you have a wide pointer to a slice.
Or if what you meant by “point to a str (Box<str>)” was pointing to the data owned by the box, now the type is special and inconsistent with the rest of the language. That is to say, if str is a box owning a slice, and &str is pointer to a slice, then &T would refer to a pointer to T in most cases, but not when T is str.
&str is (length, borrowed_ptr). str is itself (length, owned_ptr). They are layout compatible, the difference is just that for the owned one you know you need to free the memory on drop.
As for “type is special” it’s only special if you want to make it so. Fat pointers are a core affordance of the language and various conversions between fat pointers exist today.
So you’re saying what is str today would effectively be [char]? Then once again, all you’re doing is adding a type and renaming everything else, right?
String turns into StringBuilder
str is repurposed to mean what we call String today.
So String becomes two different types, one of which used to be convenient but loses all convenience and the other is just inefficient (because I assume in that scheme str is an alias for Box<str> and StringBuilder loses all string-like behaviour).
And I assume Vec gets renamed to ArrayBuilder, and you have to convert back and forth between that and the aliased Box<[]>?
str is a dynamically sized type, so it has to be behind a pointer; the owned version of it is Box<str>. Of course, this does already exist, so all you’re really doing is grinding axes about how String isn’t called StringBuilder. Can’t you see how this has nothing to do with the person’s question?
Technically yes it can be, though the sort of things you can do with this are limited because UTF-8 is a variable width encoding and you can’t replace a character with a character of a different width without possibly growing the String. Here’s an example of a method that can mutate a str in place: https://doc.rust-lang.org/std/primitive.str.html#method.make_ascii_uppercase
The easiest way to get an &mut str is to start with a String (though there are others); you can’t get an &mut str from a string literal, which is the other easy way to get a &str.
I am definitely not well versed in rust, but I believe part of the reason why you can’t have mutable references to string literals is that they are often part of the read-only region of the binary (of course, this is only a possible optimization, not a necessity). One can rather easily cause a segfault in C by doing just that.
This is true but also not really fundamental. Compare to slice literals
&[1, 2, 3] - Allocated in read-only memory, lives forever
&mut [1, 2, 3] - [1, 2, 3] copied to the stack and mutable borrowed, lives until it is implicitly (roughly at the end of the stack frame).
The machinery is there, we’re just lacking the syntax to say “I want a mutable version of this string literal”. Which we’re lacking because it wouldn’t be very useful.
That’s correct, a string literal is generally stored in .rodata / .text, is global (as different occurrences of the same literal may be dedup’d), and can be shared far and wide. There’s no way modifying one in place would be safe by Rust’s semantics.
Because lexical shadowing cause lots of bugs in C and C++, hence D and Zig forbid it.
comptime is probably not as interesting as it looks
You would have to compare comptime to the swath of C++ templates it replaces.
Memory safety is highly underestimated
Because it turns out memory safety is highly overrated by some, and is only one of the desirable properties of a language, and probably behind learnability and productivity in terms of priority.
Because lexical shadowing cause lots of bugs in C and C++, hence D and Zig forbid it.
The point in the post is that there are languages that solved it.
comptime is probably not as interesting as it looks
You would have to compare comptime to the swath of C++ templates it replaces.
The post directly refers to C++ templates and doesn’t think it’s a valid solution.
comptime is one of those features that feel like a candy, but has a bitter taste to it. It feels like C++ templates back again, and the community will not address that.
Sure, people may disagree, but please be more specific. The post particularly lays out the point where they feel like they are recommitting previous mistakes.
Memory safety is highly underestimated
Because it turns out memory safety is highly overrated by some, and is only one of the desirable properties of a language, and probably behind learnability and productivity in terms of priority.
Memory safety is a tangible objective with a reasonably good definition. I have yet (as a trainer and someone very interested in that space) to find a tangible definition of learnability and productivity. Productivity is often mistaken with familiarity, learnability is often very biased to what the speaker knows.
Capers Jones FP are metrics of business value of a program. It’s a good productivity metric the whole process of programming. Sometimes, you can find productivity metrics that give you the relationship of FP to LOC on a programming language basis, but even this is very far away from a measure of programmer productivity on a language and are influenced by other effects, like surrounding domain or domain expertise of the programmer.
I looked at the 259 pages document you linked quite hard. The term “Learnability” is mentioned once. It seems to be a very good meta-study, however, reading this:
In this study, I have deliberately avoided taking any position as to the conclusions one should make regarding the actual design decisions. For example, I do not offer any analysis on whether static or dynamic typing is better. That task belongs properly to focused systematic literature reviews and is beyond the scope of any mapping study.
I’m not sure it supports your point, especially not at a whole language scale.
Because it turns out memory safety is highly overrated by some, and is only one of the desirable properties of a language, and probably behind learnability and productivity in terms of priority.
To me, memory safety is like the baseline, the bare minimum. Anything without it simply isn’t worth using for any reason.
When people say things like your parent, what they always mean is “by default.” Even programs written in GC’d languages can have memory safety issues via FFI, but we still call them memory safe. Unsafe is like an FFI to a (slightly larger) language.
Maybe, but I really wish engineers, at least, wouldn’t talk that way. Engineering almost always involves trade-offs, and the maximalist stance (“bare minimum”, “for any reason”) make those harder to identify and discuss. For example, the follow-up comment is much more interesting (IMO) than the one I responded to.
So, rereading your comment, I think I missed a word:
Is unsafe Rust not worth using for any reason?
Emphasis mine. I thought you were saying (like I hear often on the internet) that the existence of unsafe Rust means that Rust is not a memory safe language, and was trying to talk about that definition, not about the judgement.
I think doing something potentially unsafe can be worth it, if you are willing to pay the cost of spending a lot of time making sure the code is correct. For large C, C++, or Zig code bases, this isn’t really viable. Easily 10xes your development time.
I see unsafe Rust the same way I see the code generating assembly inside a compiler. Rustc or the Zig compiler can contain bugs that make them emit incorrect code. Same with Go or javac or the JIT inside the HotSpot VM. So you are never free of unsafe code. But the unsafety can be contained, be placed inside a box. You can design an interface around unsafety that doesn’t permit incorrect use. Then you can spend a lot of time proving it correct. And after that, you can write an infinite amount of code depending on that unsafe module without worry!
Essentially, I view unsafe in Rust as a method of adding a new primitive to the language, the same way you could add a new feature to Python by writing C code for the Python interpreter. The novelty is just that you write the new feature in the same language as you use the feature in.
If 1% of your code is unsafe modules with safe interfaces, and you 10x the cost of developing those parts by being very careful and proving the code correct, your overall development cost only went up by 9%. That’s a much lower cost, and thus what I mentioned at the start, being willing to pay the cost of doing it right, becomes much smaller. It will be worth it in many more situations.
That got a little rambly but I hope I communicated my stance correctly.
For large C, C++, or Zig code bases, this isn’t really viable. Easily 10xes your development time.
An effect that large probably requires poor coding practices. Stuff like global variables, shared state between threads, lack of separation between effects and computation… When you make such a mess of your code base, OK, you need strong static guarantees, like “no arbitrary code execution is ever possible no matter how buggy my program is” — that is, memory safety. Now don’t get me wrong, I love static guarantees. I love when my compiler is disciplined so I don’t have to be. But even then you want a nice and maintainable program.
My hypothesis, is that if you do the right thing, that is, properly decomposing your program into deep modules and avoid cutting corners too often, is that memory safety doesn’t boost your productivity nearly as much as a whopping 10x. Especially if you go for some safety, like just adding bounds checks.
I will concede that in my experience, we rarely do the right thing.
I don’t know, when Java introduced a wider audience to GC, software development has definitely experienced a huge boost. Maybe 10x just in itself is not a correct figure. But you don’t only ship software, you also maintain it, fix bugs etc - and here, guaranteed memory safety can easily have an order of magnitude advantage, especially as the software grows.
The important point, to me, is reducing the potential for harm, and there are many ways to do that. For example, memory unsafety can be mitigated by running code in a sandbox of some kind (such as by compiling and running in a WASM environment), or by only running it against trusted input in simple scenarios (my local programs I use for experimenting), or by aggressively using fuzzers and tools like valgrind, ASAN, LSAN, and static analyzers like Coverity. All of these are valid approaches for mitigating memory unsafe code, and depending on my risk model, I can be OK with some level of memory unsafety.
Further, there are domains in which harm is more likely to come from too-loose type systems than it is from memory unsafety. I can implement my CRUD app in pure memory-safe Python, but that won’t protect me from a SQL injection… Something that enforces that the database only interacts with SanitizedString’s might. On the other hand, in exploratory development, too strong of a type system might slow you down too much to maintain flow.
Anyways, I generally don’t like maximalist stances. If multiple reasonably popular approaches exist in the wild, there are more than likely understandable motivations for each of them.
Can you explain what exactly memory safety means to you?
I’m writing a lot of code that is 100% “memory safe” (in my view), but would not be possible to be written in memory safe language conveniently (like Rust).
They probably mean the definition you get if you stop at the summary of the Wikipedia page, which is easy to misinterpret as saying that they have to be enforced by the toolchain (in which I include things like linters and theorem provers) or VM having compile-time or runtime guard rails which cannot suffer from false negatives.
That’s why i’m asking. The “summary definition” of Wikipedia isn’t really helpful at all.
The chapter “Classification of memory safety errors” is much better in listing potential problems, but those are language dependent.
Zig (in safe modes!) will have checks against Buffer overflow and Buffer over-read, and has increased awareness for Uninitialized variables, as undefined is an explicit decision by the programmer. This doesn’t mean Undefined variables are a non-problem, but it’s less likely as you have to actively decide not to initialize.
<pedantic>Use-after-free, Double-free, Mismatched-free are not possible in Zig either, as the language has no concept of memory allocation.</pedantic> This doesn’t mean a userland allocator won’t suffer from this problems, but the GeneralPurposeAllocator has mitigations to detect and prevent these problems.
In C, you actually have a problem with the built-in support for malloc and free. Clang, for example, knows that malloc is special, and can be considered “pure” and returning a unique object. This is how the function is defined in the C standard.
This yields interesting problems with the assumptions of the programmers and what the compiler actually does.
This is because malloc will always return a pointer to a new object, so it can never alias with another result of mallocever. Also malloc won’t have side effects besides yielding a new object. So eliding the invocation of malloc is fine, as we don’t use the result.
If we do remove the glob = p1; line, the compiler will just make the function return true; and won’t emit any calls to mallocat all!
As Zig has no builtin allocator in the language (but it’s a userland concept inside the standard library), the compiler can’t make the assumptions above and thus actually integrate the checks.
I feel you, but the React stack nowadays is so deep and diverse that it feels like it’s gonna be soon affecting browsers’ architecture. They literary solved all the hard problems in a rather elegant way. I compare to my days with PHP3 and jQuery 🙂
It basically made functional UIs mainstream, which greatly improved testability, and correctness.
I do remember the millions of websites/small GUIs where you could easily end up in inconsistent states (like a checkbox not being in sync with another related state) and while UI bugs are by no means “over”, I personally experience less bugs of this kind.
(Unfortunately, API calls are still a frequent source of errors and those are often not handled properly by UIs)
A good key escrow service so I could recover from losing my keys. Bad thing: I don’t think there’s any good way to do this.
An easy way to do mutual encrypted backups for end users. So I can recommend people to buy a specific NAS, then install this backup software, sync with some friend, then backup up data mutually without being able to access the other’s data.
Something like YunoHost, but better. I started evaluating it, but while good, it still has rough edges. Self-hosting is more important than even, and making it easier is important.
A combination of software to browse the web in “readable” mode. There are many attempts to do this, but you’d still get a ton of blank pages for websites that require JS to load. Chromium has a CLI option to load a site, execute the JS, then dump the DOM as HTML, which should address this. This would make browsing on lightweight devices nice (esp. if you could offload the JS execution to a separate system).
Something like a Linux distro, but with WASM binaries and focusing on non-server apps. So I could create portable environments and run them off my home folder on Windows, macOS, and Linux. (And others.)
For the Nas : https://hexos.com/
I don’t know what state it’s in now - it’s pretty new. But Linus from LTT invested in them specifically because he wanted a simple Nas system with buddy backup.
An easy way to do mutual encrypted backups for end users. So I can recommend people to buy a specific NAS, then install this backup software, sync with some friend, then backup up data mutually without being able to access the other’s data.
For those who happen to be using ZFS anyway, ZFS is exactly this. I know it’s sadly difficult to install in some operating systems.
A good key escrow service so I could recover from losing my keys. Bad thing: I don’t think there’s any good way to do this.
Funny you mention this, but I spent a few months designing an open-source key escrow via social recovery for $work the year before last. Sadly it wasn’t released. But it’s very possible!
In theory this can be done. You can do n-m secret sharing so you have m contacts and n are required to agree to recover your account, but although I think in practice it would work, it seems unsatisfactory to me.
I already trust some parties with very sensitive things- even a bank safe deposit box might be enough, but I kinda wish there was something nicer.
I am familiar with Bitwarden’s “timed” access. It just doesn’t feel “complete”.
I’d rather print the passwords too, in addition of a USB drive. I think the safe deposit box feels right, because I already entrust my bank with my money. Trusting them with IT matters is different (too many banks with funky security policies), though, so in theory I would trust them to do key escrow for me- or even provide me a password manager, but in practice, it would not feel good either.
A browser plugin to add transcripts/alt text to random websites. If I install this, before sharing an inaccessible comic strip (looking at you, xkcd and PA!), I can write my own transcript. Then, if I share it with vision-impaired users using the same browser plugin, they get my transcript.
I’d like to have an open-source port scanner with a web interface. E.g. like http://www.dnstools.ch/port-scanner.html , but as open source, so that I can install it on my own server, and people can use it to scan their own PC for open ports.
Background is that there are some network (German Freifunk, or the LAN at events like Chaos Communication Congress) which don’t have client isolation, so open TCP/UDP ports on a laptop can be accessed by other clients on the same LAN (this is different from home internet connections, where the clients are often behind the firewall provided by the DSL/cable router). I want to set up a server+website in such LANs so visitors can easily scan their laptops/phones for open ports.
Bonus points if it also does other client scans (e.g. for insecure service settings); basically the scans from Nessus but with a simple web frontend.
That seems reductive. There are rainmap, rainmap-lite, and WebMap that prove it can be done and isn’t especially hard to implement, but none of those are anything like “a few lines of CGI script.”
I think, more than any implementation challenges, the real obstacle to writing something like this is that nobody could sell it. If you tried to offer it as a service, you’d be kicked off your host before your first customer finished their trial period. And if everyone who wants to use it needs to stand up a web server themselves in the environment where they want to use it, well, that’s actually harder than standing up a box where you just run nmap, nmapfe, etc.
It’s something I’d like to have too, but there are just no places where it makes sense to me.
TL;DR: Because the only alternative I’m aware of are exceptions.
I don’t think I would write “Go’s error handling is awesome”, but it’s probably the least bad thing I’ve used. The main alternatives to exceptions that I’ve used have been C’s return values which are typically pretty terrible and Rust’s which are better in principle (less boilerplate at call sites, no unhandled-error bugs, and similar non-problems), but somewhat worse than Go’s in practice:
My biggest grievance with Rust error handling is the choice between (1) the sheer burden of managing your own error trait implementations (2) the unidiomatic-ness of using
anyhowin libraries and (3) the absurd amount of time I spend debugging macro expansion errors from crates likethiserror. Maybe there’s some (4) option where you return some dynamic boxed error and pray callers never need to introspect it?And I don’t think either Rust or Go have a particularly compelling solution for attaching stack traces automatically (which is occasionally helpful) or for displaying additional error context beyond the stack trace (i.e., all of the parameters that were passed down in a structured format)? Maybe Rust has something here–I’m somewhat less familiar.
I’m also vaguely aware that Zig and Swift do errors-as-values, but I haven’t used them either. Maybe the people who have found something better than Go could enlighten us?
We use a custom error type in Go that lets you add fields, like loggers do, and just stores the values in a
map[string]interface{}. This doesn’t solve the problem of capturing all parameter values, or stack traces, but it’s pretty good in practice. This is particularly true since errors tend to eventually end up in log messages so you can do something likelog.WithFields(err.Fields).Error("something terrible happened"). If we could nest those fields based on the call stack I would probably never complain again.That’s pretty awesome. I’ll play around with this. Thanks for sharing.
Best solution Ive used is polymorphic variants from ocaml. They force you to at least acknowledge the error exists and you can easily compose different errors together.
Exceptions are better on all counts though, so even that is a faulty conclusion in my opinion.
Exceptions do the correct thing by default, auto-unwrap in the success case, auto-bubble up on the error case with as fine-grained scoping as necessary (try-catch blocks). Go’s error handling is just a tad bit higher than the terrible mess that is C’s errno.
Edit: Re cryptic stack traces: most exception-based languages can trivially chain exceptions as well, so you get both a clear chain of cause and a stack trace. I swear if Java would simply default to printing out the message of each exception in the chain at the top in an easier to read matter and only print the stack trace after, people would like it better.
I was hoping this article would compare
if err != nilto more modern approaches (Rust’s?) and not just Java-style exceptions, but unfortunately it doesn’t.I’d be more interested to read an article that weighs the approaches against each other.
One point the article misses is how value-based error handling works really nicely when you don’t have constructors (either in your language, or at least your codebase, in case you’re using C++.)
I’ve been pretty disappointed by Rust’s approach to error handling. It improves upon two “problems” in Go which IMHO are not actually problems in practice:
if err != nilboilerplate and unhandled return values, but making good error types is fairly hard–either you manually maintain a bunch of implementations of the Error trait (which is a truly crushing amount of boilerplate) or you use something likeanyhowto punt on errors (which is generally considered to be poor practice for library code) or you use some crate that generates the boilerplate for you via macros. The latter seems idyllic, but in practice I spend about as much time debugging macro errors as I would spend just maintaining the implementations manually.In Go, the error implementation is just a single method called
Error()that returns a string. Annotating that error, whether in a library or a main package, is justfmt.Errorf("calling the endpoint: %w", err). I don’t think either of them do a particularly good job of automating stack trace stuff, and I’m not sure about Rust, but at least Go does not have a particularly good solution for getting more context out of the error beyond the error message–specifically parameter values (if I’m passing a bunch of identifiers, filepaths, etc down the call stack that would be relevant for debugging, you have to pack them into the error message and they often show up several times in the error message or not at all because few people have a good system for attaching that metadata exactly once).A lot of people have a smug sense of superiority about their language’s approach to error handling, which (beyond the silliness of basing one’s self-esteem on some programming language feature) always strikes me as silly because even the best programming languages are not particularly good at it, or at least not as good as I imagine it ought to be.
you usually just need to impl
Display, which I wouldn’t call a “crushing” amount of boilerplatethiserror is pretty good, although tbqh just having an
enum Errorand implementing display for it is good enough. I’ve done some heavy lifting with error handling before but that’s usually to deal with larger issues, like making sure errors are Clone + Serialize + Deserialize and can keep stacktraces across FFI boundaries.It’s pretty rarely “just” impl Display though, right? If you want automatic conversions from some upstream types you need to implement From, for example. You could not do it, but then you’re shifting the boilerplate to every call site. Depending on other factors, you likely also need Debug and Error. There are likely others as well that I’m not thinking about.
#[derive(Debug)]andimpl Displaymakes the impl ofErrortrivial (impl Error for E {}). If you’re wrapping errors then you probably want to implementsource().thiserroris a nice crate for doing everything with macros, and it’s not too heavy so the debugging potential is pretty low.One advantage of
map_err(...)everywhere instead of implementingFromis that it gives you access tofile!()andline!()macros so you can get stack traces out of your normal error handling.I’ve used
thiserrorand a few other crates, and I still spend a lot more time than I’d like debugging macro expansions. To the point where I waffle between using it and maintaining the trait implementations by hand. I’m not sure which of the two is less work on balance, but I know that I spend wayyy more time trying to make good error types in Rust than I do with Go (and I’d like to reiterate that I think there’s plenty of room for improvement on Go’s side).Maybe I should try this more. I guess I wish there was clear, agreed-upon guidance for how to do error handling in Rust. It seems like lots of people have subtly different ideas about how to do it–you mentioned just implementing
Displaywhile others encouragethiserrorand someone else in this thread suggestedBox<dyn Error>while others suggestanyhow.The rule of thumb I’ve seen is
anyhowfor applications andthiserroror your own custom error type for libraries, and ifthiserrordoesn’t fit your needs (for example, needing clone-able or serializable errors, stack traces, etc). Most libraries I’ve seen either use thiserror if they’re wrapping a bunch of other errors, or just have their own error type which is usually not too complex.Surprisingly, you don’t see people mention Common Lisp’s condition system in these debates
That’s too bad, I genuinely enjoy learning about new (to me) ways of solving these problems, I just dislike the derisive fervor with which these conversations take place.
You discount
anyhowas punting on errors, but Go’sError()with a string is the same strategy.If you want that, you don’t even need
anyhow. Rust’s stdlib hasBox<dyn Error>. It supportsFrom<String>, so you can use.map_err(|err| format!("calling the endpoint: {err}")). There’sdowncast()and.source()for chaining errors and getting errors with data, if there’s more than a string (butanyhowdoes that better with.context()).Ah, I didn’t know about downcast(). Thanks for the correction.
One source of differences in different languages’ error handling complexity is whether you think errors are just generic failures with some human-readable context for logging/debugging (Go makes this easy), or you think errors have meaning that should be distinguishable in code and handled by code (Rust assumes this). The latter is inherently more complicated, because it’s doing more. You can do it either way in either language, of course, it’s just a question of what seems more idiomatic.
I don’t think I agree. It’s perfectly idiomatic in Go to define your own error types and then to handle them distinctly in code up the stack. The main difference is that Rust typically uses enums (closed set) rather than Go’s canonical error interface (open set). I kind of think an open set is more appropriate because it gives upstream functions more flexibility to add error cases in the future without breaking the API, and of course Rust users can elect into open set semantics–they just have to do it a little more thoughtfully. The default in Go seems a little more safe in this regard, and Go users can opt into closed set semantics when appropriate (although I’m genuinely not sure off the top of my head when you need closed set semantics for errors?). I’m sure there are other considerations I’m not thinking of as well–it’s interesting stuff to think about!
Maybe “idiomatic” isn’t quite the right word and I just mean “more common”. As I say, you can do both ways in both languages. But I see a lot of Go code that propagates errors by just adding a string to the trace, rather than translating them into a locally meaningful error type. (E.g.,
so the caller can’t distinguish the errors without reading the strings, as opposed to
AFAIK the
%wfeature was specifically designed to let you add strings to a human-readable trace without having to distinguish errors.Whereas I see a lot of Rust code defining a local error type and an
impl Fromto wrap errors in local types. (Whether that’s done manually or via a macro.)Maybe it’s just what code I’m looking at. And of course, one could claim people would prefer the first way in Rust, if it had a stdlib way to make a tree of untyped error strings.
Right, we usually add a string when we’re just passing it up the call stack, so we can attach contextual information to the error message as necessary (I don’t know why you would benefit from a distinct error type in this case?). We create a dedicated error type when there’s something interesting that a caller might want to switch on (e.g., resource not found versus resource exists).
It returns a type that wraps some other error, but you can still check the underlying error type with
errors.Is()anderrors.As(). So I might have an API that returns*FooNotFoundErrand its caller might wrap it infmt.Errorf("fetching foo: %w", err), and the toplevel caller might doif errors.As(err, &fooNotFoundErr) { return http.StatusNotFound }.I think this is just the open-vs-closed set thing? I’m curious where we disagree: In Go, fallible functions return an
errorwhich is an open set of error types, sort of likeBox<dyn Error>, and so we don’t need a distinct type for each function that represents the unique set of errors it could return. And since we’re not creating a distinct error type for each fallible function, we may still want to annotate it as we pass it up the call stack, so we havefmt.Errorf()much like Rust hasanyhow!(but we can use fmt.Errorf() inside libraries as well as applications precisely because concrete error types aren’t part of the API). If you have to make an error type for each function’s return, then you don’t needfmt.Errorf()because you just add the annotation on your custom type, but when you don’t need to create custom types, you realize that you still want to annotate your errors.This is true, usually you create a specific error type on the fly when you understand that the caller needs to distinguish it.
I tend to agree that rusts error handling is both better and worse. In day to day use I can typically get away with anyhow or dyn Error but it’s honestly a mess, and one that I really dread when it starts barking at me.
On the other hand… I think being able to chain ‘?’ blocks is a god send for legibility, I think Result is far superior to err.
I certainly bias towards Rusts overall but it’s got real issues.
There is one thing to be said against
?: it does not encourage the addition of contextual information, which can make diagnosing an error more difficult when e.g. it gets expect-ed (or logged out) half a dozen frames above with no indication of the path it took.However I that is hardly unsolvable. You could have e.g.
?("text") which wraps with text and returns, and ?(unwrapped) which direct returns (the keyword being there to encourage wrapping, one could even imagine extending this to more keywords e.g.?(panic)` would be your unwrap).In a chain i’ll just map_err which as soon as the chain is multiline looks and works well. Inline it’s not excellent ha.
Oh yeah I’m not saying it’s not possible to decorate things (it very much is), just pointing out that the incentives are not necessarily in that direction.
If I was a big applications writer / user of type-erased errors, I’d probably add a wrapping method or two to
Result(if I was to use “raw” boxed error, as IIRC anyhow has something like that already).I’ve often wondered if people would like Java exceptions more if it only supported checked exceptions. You still have the issue of exceptions being a parallel execution flow / go-to, but you lose the issue of random exceptions crashing programs. In my opinion it would make the language easier to write, because the compiler would force you to think about all the ways your program could fail at each level of abstraction. Programs would be more verbose, but maybe it would force us to think more about exception classes.
Tl;Dr Java would be fine if we removed RuntimeException?
You’d need to make checked exceptions not horrendous to use to start with e.g. genericity exception transparency, etc…
It would also necessarily yield a completely different language, consider what would happen if NPEs were checked.
Basically, Kotlin, so yeah, totally agree with you.
No, Go has unchecked exceptions. They’re called “panics”.
What makes Go better than Java is that you return the error interface instead of a concrete error type, which means you can add a new error to an existing method without breaking all your callers and forcing them to update their own throws declarations.
The creator of C# explains the issue well here: https://www.artima.com/articles/the-trouble-with-checked-exceptions#part2
You can just throw Exception (or even a generic) in Java just fine, though if all you want is an “error interface”.
Java’s problem with checked exceptions is simply that checked exceptions would probably require effect types to be ergonomic.
Looks like it’s been updated:
I’m also personally quite fond of error handling in Swift.
Rust, Zig, and Swift all have interesting value-oriented results. Swift more so since it added, well, Result and the ability to convert errors to that.
Zig’s not really value oriented. It’s more like statically typed error codes.
No matter how many times Go people try to gaslight me, I will not accept this approach to error-handling as anything approaching good. Here’s why:
Why must you rely on a linter or IDE to catch this mistake? Because the compiler doesn’t care if you do this.
If you care about correctness, you should want a compiler that considers handling errors part of its purview. This approach is no better than a dynamic language.
The fact that the compiler doesn’t catch it when you ignore an error return has definitely bitten me before.
doTheThing()on its own looks like a perfectly innocent line of code, and the compiler won’t even warn on it, but it might be swallowing an error.I learned that the compiler doesn’t treat unused function results as errors while debugging a bug in production; an operation which failed was treated as if it succeeded and therefore wasn’t re-tried as it should. I had been programming in Go for many years at that point, but it had never occurred to me that silently swallowing an error in Go could possibly be so easy as just calling a function in the normal way. I had always done
_ = doTheThing()if I needed to ignore an error, out of the assumption that of course unused error returns is a compile error.Does anyone know the reason why the Go compiler allows ignored errors?
Because errors aren’t special to the Go compiler, and Go doesn’t yell at you if you ignore any return value. It’s probably not the most ideal design decision, but in practice it’s not really a problem. Most functions return something that you have to handle, so when you see a naked function call it stands out like a sore thumb. I obviously don’t have empirical evidence, but in my decade and a half of using Go collaboratively, this has never been a real pain point whether with junior developers or otherwise. It seems like it mostly chafes people who already had strong negative feelings toward Go.
It’s similar to array bounds checks in c - not really a problem.
I hope this is sarcasm.
Yes
Is there a serious argument behind the sarcasm as to how this is comparable to array bounds checks? Do you have any data about the vulnerabilities that have arisen in Go due to unhandled errors?
Because the programmer made an intentional decision to ignore the error. It won’t let you call a function that returns an error with out assigning it to something, that would be a compile time error. If the programmer decides to ignore it, that’s on the programmer (and so beware 3rd party code).
Now perhaps it might be a good idea for the compiler to insert code when assigned to _ that panics if the result is non-nil. Doesn’t really help at runtime, but at least it would fail loudly so they could be found.
I’ve spent my own share of time tracking down bugs because something appeared to be working but the error/exception was swallowed somewhere without a trace.
This is incorrect: https://go.dev/play/p/k7ErZU5QYCu
huh… til. I always assumed you needed to use the result, probably because of single vs multiple returns needing both being a compile time error. Thanks.
To be fair I was just as certain as you that of course Go requires using the return values, until I had to debug this production bug. No worries.
is not an intentional decision to ignore the error. Neither is
Yet the go compiler will never flag the first one, and may not flag the second one depending on
errbeing used elsewhere in the scope (e.g. in the unheard of case where you have two different possibly error-ing calls in the same scope and you check the other one).Yeah, I always thought the _ was required, I learned something today!
I do have a few places with err and err2, it does kind of suck - I should probably breakup those functions.
_, err := f.Write(s)is a compiler error iferralready exists (no new variables on left side of :=), and iferrdoesn’t already exist and you aren’t handling it, you get a different error (declared and not used: err). I think you would have to assign a new variablet, err := f.Write(s)and then take care to handletin order to silently ignore theerr, but yeah, with some work you can get Go to silently swallow it in the variable declaration case.Because they couldn’t be arsed to add this in v0, and they can’t be arsed to work on it for cmd/vet, and there are third-party linters which do it, so it’s all good. Hopefully you don’t suffer from unknown unknowns and you know you should use one of these linters before you get bit, and they don’t get abandoned.
(TBF you need both that and errcheck, because the unused store one can’t catch ignoring return values entirely).
Considering how much effort the Go team puts in basically everything, this language makes it very hard to to take you serious.
Yes, except for things that they decide not to be arsed about. I can confirm this as a very real experience of dealing with Go.
Which is fair enough.
Sure, but then it is equally fair to criticize them for it.
The go compiler doesn’t do warnings, only errors. Linters do warnings, and do warn about unchecked errors.
I don’t really care. Generally speaking, I would expect compilers to either warn or error on an implicitly swallowed error. The Go team could fix this issue by either adding warnings for this case specifically (going back on their decision to avoid warnings), or by making it a compile error, I don’t care which.
This is slightly more nuanced. Go project ships both
go buildandgo vet.go vetis an isomorphic to how Rust handles warnings (that warnings apply to you, not your dependencies).So there would be nothing wrong per se if this was caught by
go vetand notgo build.What is the issue though, is that this isn’t caught by first-party
go vet, and requires third partyerrcheck.Meh plenty of code bases don’t regularly run
go vet. This is a critical enough issue that it should be made apparent as part of any normal build, either as a warning or an error.And that’s perfectly fine given that Go is pleasurable even for quick and dirt prototypes, fun side projects, and so on.
I agree with you that it’s better for this to be a compiler error, but (1) I’ll never understand why this is such a big deal–I’m sure it’s caused bugs, but I don’t think I’ve ever seen one in the dozen or so years of using Go and (2) I don’t think many dynamic languages have tooling that could catch unhandled errors so I don’t really understand the “no better than a dynamic language” claim. I also suspect that the people who say good things about Go’s error handling are making a comparison to exceptions in other languages rather than to Rust’s approach to errors-as-values (which has its own flaws–no one has devised a satisfactory error handling system as far as I’m aware).
The fact that these bugs seem so rare and that the mitigation seems so trivial makes me feel like this is (yet another) big nothingburger.
The most common response to my critique of Go’s error-handling is always some variation on “this never happens”, which I also do not accept because I have seen this happen. In production. So good for you, if you have not; but I know from practice this is an issue of concern.
Relying on the programmer to comprehensively test inputs imperatively in a million little checks at runtime is how dynamic languages handle errors. This is how Go approached error-handling, with the added indignity of unnecessary verbosity. At least in Ruby you can write single-line guard clauses.
I don’t really follow your dismissal of Rust since you didn’t actually make an argument, but personally I consider Rust’s
Optiontype the gold standard of error-handling so far. The type system forces you to deal with the possiblity of failure in order to access the inner value. This is objectively better at preventing “trivial” errors than what Go provides.I’m sure it has happened before, even in production. I think most places run linters in CI which default to checking errors, and I suspect if someone wasn’t doing this and experienced a bug in production, they would just turn on the linter and move on with life. Something so exceedingly rare and so easily mitigated does not meet my threshold for “issue of concern”.
That’s how all languages handle runtime errors. You can’t handle them at compile time. But your original criticism was that Go is no better than a dynamic language with respect to detecting unhandled errors, which seems untrue to me because I’m not aware of any dynamic languages with these kinds of linters. Even if such a linter exists for some dynamic language, I’m skeptical that they’re so widely used that it merits elevating the entire category of dynamic languages.
I didn’t dismiss Rust, I was suggesting that you may have mistaken the article as some sort of criticism of Rust’s error handling. But I will happily register complaints with Rust’s error handling as well–while it does force you to check errors and is strictly better than Go in that regard, this is mostly a theoretical victory insofar as these sorts of bugs are exceedingly rare in Go even without strict enforcement, and Rust makes you choose between the verbosity of managing your own error types, debugging macro expansion errors from crates like
thiserror, or punting altogether and doing the bare minimum to provide recoverable error information. I have plenty of criticism for Go’s approach to error handling, but pushing everything into an error interface and switching on the dynamic type gets the job done.For my money, Rust has the better theoretical approach and Go has the better practical approach, and I think both of them could be significantly improved. They’re both the best I’m aware of, and yet it’s so easy for me to imagine something better (automatic stack trace annotations, capturing and formatting relevant context variables, etc). Neither of them seems so much better in relative or absolute terms that their proponents should express superiority or derision.
I don’t accept your unsubstantiated assertion that this is rare, so it seems we are at an impasse.
Fair enough. It’s a pity things like this are so difficult to answer empirically, and we must rely on our experiences. I am very curious how many orgs are bitten by this and how frequently.
Couldn’t agree more (honourable mention to Zig, though).
Enabling a linter is different from doing “a million little checks at runtime”. This behaviour is not standard because you can use Go for many reasons other than writing production-grade services, and you don’t want to clutter your terminal with unchecked error warnings.
I admit that it would be better if this behaviour were part of
go vetrather than an external linter.The strange behaviour here is not “Go people are trying to gaslight me”, but people like you coming and complaining about Go’s error handling when you have no interest in the language at all.
You can’t lint your way out of this problem. The Go type system is simply not good enough to encapsulate your program’s invarients, so even if your inputs pass a type check you still must write lots of imperative checks to ensure correctness.
Needing to do this ad-hoc is strictly less safe than relying on the type system to check this for you.
errchecks are simply one example of this much larger weakness in the language.I have to work with it professionally, so I absolutely do have an interest in this. And I wouldn’t feel the need to develop this critique of it publicly if there weren’t a constant drip feed of stories telling me how awesome this obviously poor feature is.
Your views about how bad Go’s type system is are obviously not supported by the facts, otherwise Go programs would be full of bugs (or full of minuscule imperative checks) with respect to your_favourite_language.
I understand your point about being forced to use a tool in your $job that you don’t like, that happened to me with Java, my best advice to you is to just change $job instead of complaining under unrelated discussions.
They are full of bugs, and they are full of miniscule imperative checks. The verbosity of all the
if err != nilchecks is one of the first things people notice. Invoke “the facts” without bringing any isn’t meaningfully different than subjective opinion.Your comments amount to “shut up and go away” and I refuse. To publish a blog post celebrating a language feature, and to surface it on a site of professionals, is to invite comment and critique. I am doing this, and I am being constructive by articulating specific downsides to this language decision and its impacts. This is relevant information that people use to evaluate languages and should be part of the conversation.
If
if err != nilchecks are the “minuscle imperative checks” you complain about, I have no problem with that.That you have “facts” about Go programs having worse technical quality (and bug count) than any other language I seriously doubt, at most you have anecdotes.
And the only anecdote you’ve been able to come up with so far is that you’ve found “production bugs” caused by unchecked errors that can be fixed by a linter. Being constructive would mean indicating how the language should change to address your perceived problem, not implying that the entire language should be thrown out the window. If that’s how you feel, just avoid commenting on random Go post.
Yeah, I have seen it happen maybe twice in eight years of using Go professionally, but I have seen it complained about in online comment sections countless times. :-)
If I were making a new language today, I wouldn’t copy Go’s error handling. It would probably look more like Zig. But I also don’t find it to be a source of bugs in practice.
Everyone who has mastered a language builds up muscle memory of how to avoid the Bad Parts. Every language has them. This is not dispositive to the question of whether a particular design is good or not.
The happy people are just happily working on solving their real problems. not wasting time complaining.
Not seeing a problem as a bug in production doesn’t tell you much. It usually just means that the developers spent more writing tests or doing manual testing - and this is just not visible to you. The better the compiler and type-system, the fewer tests you need for the same quality.
Agreed, but I wasn’t talking about just production–I don’t recall seeing a bug like this in any environment, at any stage.
In a lot of cases I am the developer, or I’m working closely with junior developers, so it is visible to me.
Of course with Go we don’t need to write tests for unhandled errors any more than with Rust, we just use a linter. And even when static analysis isn’t an option, I disagree with the logic that writing tests is always slower. Not all static analysis is equal, and in many cases it’s not cheap from a developer velocity perspective. Checking for errors is very cheap from a developer velocity perspective, but pacifying the borrow checker is not. In many cases, you can write a test or two in the time it would take to satisfy rustc and in some cases I’ve even introduced bugs precisely because my attention was so focused on the borrow checker and not on the domain problem (these were bugs in a rewrite from an existing Go application which didn’t have the bugs to begin with despite not having the hindsight benefit that the Rust rewrite enjoyed). I’m not saying Rust is worse or static analysis is bad, but that the logic that more static analysis necessarily improves quality or velocity is overly simplistic, IMHO.
I just want to emphasize that It’s not the same thing - as you also hint to in the next sentence.
I didn’t say that writing tests is always slower or that using the compiler to catch these things is necessarily always better. I’m not a Rust developer btw. and Rust’s errorhandling is absolutely not the current gold-standard by my own judgement.
It kind of is the same thing: static analysis. The only difference is that the static analysis is broken out into two tools instead of one, so slightly more care needs to be taken to ensure the linter is run in CI or locally or wherever appropriate. To be clear, I think Rust is strictly better for having it in the compiler–I mostly just disagree with the implications in this thread that if the compiler isn’t doing the static analysis then the situation is no better than a dynamic language.
What did you mean when you said “It usually just means that the developers spent more writing tests or doing manual testing … The better the compiler and type-system, the fewer tests you need for the same quality.” if not an argument about more rigorous static analysis saving development time? Are we just disagreeing about “always”?
Ah I see - that is indeed an exaggeration that I don’t share.
First that, but it also in general has other disadvantages. For instance, writing tests or doing manual tests is often easy to do. Learning how to deal with a complex time system is not. Go was specifically created to get people to contribute fast.
Just one example that shows that it’s not so easy to decide which way is more productive.
Ah, I think we’re agreed then. “always” in particular was probably a poor choice of words on my part.
Swallowing errors is the very worst option there is. Even segfaulting is better, you know at least something is up in that case.
Dynamic languages usually just throw an exception and those have way better behavior (you can’t forget, an empty catch is a deliberate sign to ignore an error, not an implicit one like with go), at least some handler further up will log something and more importantly the local block that experienced the error case won’t just continue executing as if nothing happened.
Time and time again wlroots proves how solid it is as a project. Really outstanding work!
It’s just a shame that Wayland didn’t dare to define such things on the protocol level in the first place. I mean, given the rock-sold colour space support in macOS, any sane engineer designing a new display manager/compositor in the 2010’s would have put colour management as a design-centerpiece. Libraries like Little CMS prove that you don’t even need to do much in terms of colour transformations by hand; simply define your surfaces in a sufficiently large working colour space and do the transformations ad-hoc.
From what I remember back then, the only thing the Wayland engineers seemed to care about was going down to the lowest common denominator and ‘no flickering’ (which they saw in X in some cases).
For instance, it is not possible to portably place an application window ‘at the top’, given one may not dare to assume this even though 99.99% of all displays support this. It would have made more sense to have ‘feature flags’ for displays or have more strict assumptions on the coordinate space.
In the end, a wayland compositor requires close to 50.000 LOC of boilerplate, which wlroots gracefully provides, and this boilerplate is fragile as you depend on proprietary interfaces and extensions. You can write a basic X display manager in 500 LOC only based on the stable X libraries. With all of X’s flaws, this is still a strong point today.
This instinctually bothers me too, but I don’t think it’s actually correct. The reason that your X display manager can be 500 LOC is because of the roughly 370 LOC in Xorg. The dominance of wlroots feels funny to me based on my general dislike for monocultures, but if you think of wlroots as just “the guts of Xorg, but in ‘window manager userland’”, it actually is not that much worse than Xorg and maybe even better.
I think you mean 370k LOC.
Yes indeed, my bad.
I don’t really get your criticism. Wayland is used on a lot of devices, including car displays and KIOSK-like installations. Does an application window even make sense if you only have a single application displayed at all times? Should Wayland not scale down to such setups?
Especially that it has an actually finely-working extension system so that such a functionality can be trivially added (either as a standard if it’s considered widely useful, or as a custom extension if it only makes sense for a single server implementation).
A Wayland compositors’ 50 thousands LOC is the whole thing. It’s not boilerplate, it’s literally a whole display server communicating in a shared “language” with clients, sitting on top core Linux kernel APIs. That’s it. Your 500 LOC comparison under X is just a window manager plugin, just because it operates as a separate binary it is essentially the same as a tiling window manager plugin for Gnome.
Then it would have taken 2× as long to get it out of the door and gain any adoption at all.
Routine reminder that the entire F/OSS ecosystem worth of manpower and funding is basically a rounding error compared to what Apple can pour into macOS in order to gain “rock-solid colour space support” from day zero.
What do you mean by this? I can’t understand it.
The ideals of this post are dead. Firefox is neither private nor free. Do not use Firefox in 2025.
Mozilla has done an about face and now demands that Firefox users:
See https://lobste.rs/s/de2ab1/firefox_adds_terms_use for more discussion.
If you’re already using Firefox, I can confirm that porting your profile over to Librewolf (https://librewolf.net) is relatively painless, and the only issues you’ll encounter are around having the resist fingerprinting setting turned on by default (which you can choose to just disable if you don’t like the trade-offs). I resumed using Firefox in 2016 and just switched away upon this shift in policy, and I do so sadly and begrudgingly, but you’d be crazy to allow Mozilla to cross these lines without switching away.
If you’re a macOS + Littlesnitch user, I can also recommend setting Librewolf to not allow communication to any Mozilla domain other than addons.mozilla.org, just in case.
👋 I respect your opinion and LibreWolf is a fine choice; however, it shares the same problem that all “forks” have and that I thought I made clear in the article…
Developing Firefox costs half a billion per year. There’s overhead in there for sure, but you couldn’t bring that down to something more manageable, like 100 million per year, IMO, without making it completely uncompetitive to Chrome, whose estimate cost exceeds 1 billion per year. The harsh reality is that you’re still using Mozilla’s work and if Mozilla goes under, LibreWolf simply ceases to exist because it’s essentially Firefox + settings. So you’re not really sticking it to the man as much as you’d like.
There are 3 major browser engines left (minus the experiments still in development that nobody uses). All 3 browser engines are, in fact, funded by Google’s Ads and have been for almost the past 2 decades. And any of the forks would become unviable without Apple’s, Google’s or Mozilla’s hard work, which is the reality we are in.
Not complaining much, but I did mention the recent controversy you’re referring to and would’ve preferred comments on what I wrote, on my reasoning, not on the article’s title.
I do what I can and no more, which used to mean occasionally being a Firefox advocate when I could, giving Mozilla as much benefit of the doubt as I could muster, paying for an MDN subscription, and sending some money their way when possible. Now it means temporarily switching to Librewolf, fully acknowledging how unsustainable that is, and waiting for a more sustainable option to come along.
I don’t disagree with the economic realities you mentioned and I don’t think any argument you made is bad or wrong. I’m just coming to a different conclusion: If Firefox can’t take hundreds of millions of dollars from Google every year and turn that into a privacy respecting browser that doesn’t sell my data and doesn’t prohibit me from visiting whatever website I want, then what are we even doing here? I’m sick of this barely lesser of two evils shit. Burn it to the fucking ground.
I think “barely lesser of two evils” is just way off the scale, and I can’t help but feel that it is way over-dramatized.
Also, what about the consequences of having a chrome-only web? Many websites are already “Hyrum’s lawed” to being usable only in Chrome, developers only test for Chrome, the speed of development is basically impossible to follow as is.
Firefox is basically the only thing preventing the most universal platform from becoming a Google-product.
Well there’s one other: Apple. Their hesitance to allow non-Safari browsers on iOS is a bigger bulwark against a Chrome-only web than Firefox at this point IMO.
I’m a bit afraid that the EU is in the process of breaking that down though. If proper Chrome comes over to iOS and it becomes easy to install, I’m certain that Google will start their push to move iOS users over.
I know it’s not exactly the same but Safari is also in the WebKit family and Safari is nether open source nor cross platform nor anywhere close to Firefox in many technical aspects (such as by far having the most functional and sane developer tools of any browser it there).
Pretty much the same here: I used to use Firefox, I have influenced some people in the past to at least give Firefox a shot, some people ended up moving to it from Chrome based on my recommendations. But Mozilla insists on breaking trust roughly every year, so when the ToS came around, there was very little goodwill left and I have permanently switched to LibreWolf.
Using a fork significantly helps my personal short-term peace of mind: whenever Mozilla makes whatever changes they’re planning to make which requires them to have a license to any data I input into Firefox, I trust that I will hear about those changes before LibreWolf incorporates them, and there’s a decent chance that LibreWolf will rip them out and keep them out for a few releases as I assess the situation. If I’m using Firefox directly, there’s a decent probability that I’ll learn about those changes after Firefox updates itself to include them. Hell, for all I know, Firefox is already sending enough telemetry to Mozilla that someone there decided to make money off it and that’s why they removed the “Mozilla will doesn’t and will never sell your data” FAQ item; maybe LibreWolf ripping out telemetry is protecting me against Mozilla right now, I don’t know.
Long term, what I personally do doesn’t matter. The fact that Mozilla has lost so much good-will that long-term Firefox advocates are switching away should be terrifying to Mozilla and citizens of the Web broadly, but my personal actions here have close to 0 effect on that. I could turn into a disingenuous Mozilla shill but I don’t exactly think I’d be able to convince enough people to keep using Firefox to cancel out Mozilla’s efforts to sink their own brand.
If Firefox is just one of three browsers funded by Google which don’t respect user privacy, then what’s the point of it?
People want Firefox and Mozilla to be an alternative to Google’s crap. If they’re not going to be the alternative, instead choosing to copy every terrible idea Google has, then I don’t see why Mozilla is even needed.
Well to be fair to Mozilla, they’re pushing back against some web standard ideas Google has. They’ve come out against things like WebUSB and WebHID for example.
How the heck do they spend that much? At ~20M LoC, they’re spending 25K per line of code a year. While details are hard to find, I think that puts them way above the industry norms.
I’m pretty sure that’s off by 3 orders of magnitude; OP’s figure would be half a US billion, i.e. half a milliard. That means 500M / 20M = 25 $/LOC. Not 25K.
I see your point, but by that same logic, shouldn’t we all then switch to Librewolf? If Firefox’s funding comes from Google, instead of its user base, then even if a significant portion of Firefox’s users switch, it can keep on getting funded, and users who switched can get the privacy non-exploitation they need?
I gathered some numbers on that here: https://untested.sonnet.io/notes/defaults-matter-dont-assume-consent/#h-dollar510000000
TL;DR 90% of Mozilla’s revenue comes from ad partnerships (Google) and Apple received ca. 19 Bn $ per annum to keep Google as the default search engine.
Where did you get those numbers? Are you referring to the whole effort, (legal, engineering, marketing, administration, etc) ot just development?
That’s an absolutely bonkers amount of money, and while i absolutely believe it, im also kind of curious what other software products are in a similar league
doesn’t seem like a particularly grave concern to me
That page says “Services”. Does it apply to Firefox or the VPN?
The sexuality and violence thing I suspect is so that they are covered for use in Saudi Arabia and Missouri.
Yeah, that seems like legal butt-covering. If someone in a criminalizing jurisdiction accesses these materials and they try to sue to the browser, Mozilla can say the user violated TOS.
i assume it applies mostly to Bugzilla / Mozilla Connect / Phabricator / etc
This is just a lie. It’s just a lie. Firefox is gratis, and it’s FLOSS. These stupid paragraphs about legalese are just corporate crap every business of a certain size has to start qualifying so they can’t get their wallet gaped by lawyers in the future. Your first bullet point sucks - you don’t agree to the Acceptable Use Policy to use Firefox, you agree to it when using Mozilla services, i.e. Pocket or whatever. Similarly, your second bulletpoint is completely false, that paragraph doesn’t even exist:
The text was recently clarified because of the inane outrage over basic legalese. And Mozilla isn’t selling your information. That’s not something they can casually lie about and there’s no reason to lie about it unless they want to face lawsuits from zealous legal types in the future. Why constantly lie to attack Mozilla? Are you being paid to destroy Free Software?
Consciously lying should be against Lobsters rules.
Let’s really look at what’s written here, because either u/altano or u/WilhelmVonWeiner is correct, not both.
The question we want to answer: do we “agree to an acceptable use policy” when we use Firefox? Let’s look in the various terms of service agreements (Terms Of Use, Terms Of Service, Mozilla Accounts Privacy). We see that it has been changed. It originally said:
“When you upload or input information through Firefox, you hereby grant us a nonexclusive, royalty-free, worldwide license to use that information to help you navigate, experience, and interact with online content as you indicate with your use of Firefox.”
Note that this makes no distinction between Firefox as a browser and services offered by Mozilla. The terms did make a distinction between Firefox as distributed by Mozilla and Firefox source code, but that’s another matter. People were outraged, and rightfully so, because you were agreeing to an acceptable use policy to use Firefox, the binary from Mozilla. Period.
That changed to:
“You give Mozilla the rights necessary to operate Firefox. This includes processing your data as we describe in the Firefox Privacy Notice. It also includes a nonexclusive, royalty-free, worldwide license for the purpose of doing as you request with the content you input in Firefox. This does not give Mozilla any ownership in that content.”
Are the legally equivalent, but they’re just using “nicer”, “more acceptable” language? No. The meaning is changed in important ways, and this is probably what you’re referring to when you say, “you don’t agree to the Acceptable Use Policy to use Firefox, you agree to it when using Mozilla services”
However, the current terms still say quite clearly that we agree to the AUP for Mozilla Services when we use Firefox whether or not we use Mozilla Services. The claim that “you don’t agree to the Acceptable Use Policy to use Firefox” is factually incorrect.
So is it OK for u/WilhelmVonWeiner to say that u/altano is lying, and call for censure? No. First, it’s disingenuous for u/WilhelmVonWeiner to pretend that the original wording didn’t exist. Also, the statement, “Similarly, your second bulletpoint is completely false, that paragraph doesn’t even exist:” is plainly false, because we can see that paragraph verbatim here:
https://www.mozilla.org/en-US/about/legal/terms/firefox/
So if u/WilhelmVonWeiner is calling someone out for lying, they really shouldn’t lie themselves, or they should afford others enough benefit of the doubt to distinguish between lying and being mistaken. After all, is u/WilhelmVonWeiner lying, or just mistaken here?
I’m all for people venting when someone is clearly in the wrong, but it seems that u/WilhelmVonWeiner is not only accusing others of lying, but is perhaps lying or at very least being incredibly disingenuous themselves.
Oh - and I take exception to this in particular:
“every business of a certain size has to start qualifying so they can’t get their wallet gaped by lawyers”
Being an apologist for large organizations that are behaving poorly is the kind of behavior we expect on Reddit or on the orange site, but not here. We do not want to or should we need to engage with people who do not make good faith arguments.
This is a pretty rude reply so I’m not going to respond to the specifics.
Mozilla has edited their acceptable use policy and terms of service to do damage control and so my exact quotes might not be up anymore, but yeah sure, assume that everyone quoting Mozilla is just a liar instead of that explanation if you want.
EDIT:
https://blog.mozilla.org/en/products/firefox/update-on-terms-of-use/
Sorry for being rude. It was unnecessary of me and I apologise, I was agitated. I strongly disagree with your assessment of what Mozilla is doing as “damage control” - they are doing what is necessary to legally protect the Mozilla Foundation and Corporation from legal threats by clarifying how they use user data. It is false they are selling your private information. It is false they have a nonexclusive … license to everything you do using Firefox. It is false that you have to agree to the Acceptable Use Policy to use Firefox. It’s misinformation, it’s FUD and it’s going to hurt one of the biggest FLOSS nonprofits and alternate web browsers.
So people can judge for them selves, the relevant quote from the previous Terms of Use was:
Source: http://archive.today/btoQM
The updated terms make no mention of the Acceptable Use Policy.
This is a pretty incendiary comment and I would expect any accusation of outright dishonesty to come with evidence that they know they’re wrong. I am not taking a position on who has the facts straight, but I don’t see how you could prove altano is lying. Don’t attribute to malice what can be explained by…simply being incorrect.
that’s not binding to firefox. that’s binding to mozilla services like websites and other services. https://www.mozilla.org/en-US/about/legal/terms/mozilla/ links to the acceptable use page for instance. whereas the firefox one does not. https://www.mozilla.org/en-US/about/legal/terms/firefox/
firefox is fine. your other points are also largely incorrect.
FYI this is a change made in response to the recent outrage, the original version of the firefox terms included
Which has now been removed.
What are the trade-offs for resisting fingerprinting? Does it disable certain CSS features, or?
Your locale is forced to en-US, your timezone is UTC, your system is set to Windows. It will put canvas behind a prompt and randomizes some pixels such that fingerprinting based on rendering is a bit harder. It will also disable using SVG and fonts that you have installed on your systems
Btw, I don’t recommend anyone using resist fingerprinting. This is the “hard mode” that is known to break a lot of pages and has no site-specific settings. Only global on or off. A lot of people turn it on and then end up hating Firefox and switching browsers because their web experience sucks and they don’t know how to turn it off. This is why we now show a rather visible info bar in settings under privacy/security when you turn this on and that’s also why we are working on a new mode that can spoof only specific APIs and only on specific sites. More to come.
Now that I know about it, I’m really looking forward to the new feature!
I’m using CanvasBlocker but its performance and UX could use some love.
This is the kind of thing Mozilla still does that sets it very far appart from the rest. Thanks!
heh, I wonder how many bits of entropy will be there in roughly “which of the spoofs are enabled”? :D
Yes, if everyone is running a custom set of spoofs you’d end up being unique again. The intent for the mechanism is for us to be able to experiment and test out a variety of sets before we know what works (in terms of webcompat). In the end, we want everyone to look as uniform as possible
It breaks automatic dark mode and sites don’t remember their zoom setting. Dates are also not always localized correctly. That’s what I’ve noticed so far at least.
I am a proper Rust n00b ramping up as we speak. Even before getting into conceptual challenges of understanding lifetimes, I think it’s important to mention that the syntax was truly jarring for me. In literally every other language I’ve learned, a single unclosed quote means that I have written my program incorrectly in a very fundamental way. I’ve been programming for over 10 years so there’s a lot of muscle memory to undo here. Sorry if this bikeshed-y topic has been discussed to death, but since the article explicitly covers why lifetimes are hard to learn and doesn’t mention this point, I figured it’s fair game to mention again.
I personally like the notation, but I could see how it looks jarring. FWIW, Rust probably borrowed the notation from ML, where single quotes are used to denote type variables. For example, the signature of
mapin Standard ML is:I got nerd-sniped thinking about how old the use of
'as a sigil might be. It’s used in lisp; I think it’s oldest use might go back to MACLISP in 1966. I think older dialects of lisps required that you say(quote foo)instead of'foo. See section 1.3 “Notational Conventions” on page 3 of the MACLISP manual (large PDF warning).I am the one who proposed the notation (can’t find the reference now, but it’s there, trust me) and yes it is from ML.
Is there a reason that it’s only for lifetimes and not all type parameters? My guess would be because it makes them easy to distinguish (and since type parameters are more common, they get the shorter case), but I could be wrong of course.
Yes because types and lifetimes are different kinds, and yes because types are more common they are unmarked.
Is it?
I recently saw it in reading the various Cyclone papers, where they introduced it for identifying differing regions, with those having different lifetimes. However, I believe Cyclone was itself drawing inspiration from ML (or Caml).
It also has mention of “borrowing” in what may be a similar fashion.
http://www.cs.umd.edu/~mwh/papers/ismm.pdf
Also the syntax there seems to evolve over various papers.
The
'is an odd one. I had another skim of the history of Standard ML but it isn’t very interested in questions of notation. However, it reminded me that ML type variables are often rendered in print as α, β instead of'a,'b, which made me think this might be a kind of stropping. “Stropping” comes from apostrophe, and prefix'was one form it could take. (Stropping varied a lot depending on the print and punch equipment available at each site.) But this is a wild guess.Oh, yeah, it took me a good six months to stop typing a
(right after the'. The joke about Greenspun’s tenth rule practically writes itself at this point :-).I’m only mentioning this for the laughs. As far as I’m concerned, any syntax that’s sufficiently different from that of a Turing tarpit is fine. I mean, yeah, watching the compiler try to figure out what the hell I had in mind there was hilarious, but hardly the reason why I found lifetimes so weird to work with.
Is there also any precedent for this kind of notation in math?
ML is on my list of ur-languages to study: https://news.ycombinator.com/item?id=35816454
I can’t think of instances of
'xin math, but ofcx'is used frequently for the derivative, as well as any kind of transformed version ofx.Or if you just want another variable with a name “like”
x. This is also how it’s used in Haskell.I haven’t heard this particular challenge before. I came to Rust long after I learned Lisp and Standard ML, so it never occurred to me that it would be jarring, but if you’ve only worked in recent members of the ALGOL family I can see that being the case.
What do you mean muscle memory? Do you usually double your quotes manually? Does your IDE not do it for you?
Not trying to “attack” you or anything, genuinely curious as these kind of syntactical changes are more-or-less invisible to me when writing code due to a good IDE “typing” what I intend based on context for the most part.
As for a single single quote being jarring, I believe it does have some history in LISPs for “unquote” or for symbols. Possibly, the latter case was the inspiration in case of Rust?
Edit: I see there is a much better discussion in the sibling thread regarding its origin.
Ah yes, now that you mention it, “muscle memory” is not the right phrase here. I didn’t mean muscle memory in the (proper) sense of a command that you’ve been using for years, and then now you need to use slightly differently. What I meant was that for years, whenever I saw a single quote, I expected another quote to close it somewhere later in the code. And now I have to undo that expectation.
May not be completely relevant to the article, but I always wanted to write a blog post titled something like “Visual programming is here, right before our eyes”.
I really enjoyed learning about CellPond, but I don’t believe that we have to go to such “depths” for visual programming - a featureful IDE with a well-supported language is pretty much already that.
Programs in most PLs are not 1D text, they are 2D text with some rudimentary visual structuring, like blocks, indentation, alignment etc.
This is turbo-charged by a good IDE that makes operations operate on symbolic structures and not on a list of characters. E.g. something as simple as putting the caret on a variable will immediately “bring up another layer of information” as a graph showcasing the declaration and uses of it. I can also select an expression (and can expand it syntactically) and choose to store that expression as a new node. Intellij for example will even ask if I want to replace the same expression at different places, effectively merging nodes. And I would argue that experienced developers look at code just like that - we are making use of our visual cortex, we are pattern matching on blocks/expressions, etc. We are not reading like you would read a book, not at all.
I can also see all the usages of a function right away, click on its definition, its overrides, parent class/interface right from my IDE. These are semantically relevant operations on specific types of nodes, not text.
Also, writing is just a very efficient and accurate visual encoding of arbitrary ideas, so it only makes sense to make use of it where a more specific encoding can’t be applied (sort of a fallback). “Nodes” (blocks of code) being text-based doesn’t take away from “visual programming” in my book. I would actually prefer to *extend” ordinary programming languages with non-textual blocks, so that I can replace, say, a textual state machine with a literal, interactive FSM model, and my IDE will let me play with it. (If we look at it from far away, Intellij again has a feature that lets you test regular expressions by simply writing a sample text, and it showcases which part matches and which doesn’t). Because a graph of a state machine is more “efficient”/“human-friendly” than a textual representation. But I think it’s faulty to think that we could have a representation that will apply equally well for different areas - this is exactly what human writing is.
I care a lot if it takes 10 seconds to start! I quit vim a lot while I’m working. In part because it takes 0 seconds to start. Though the Haskell language server kinda ruins that.
I think it’s actually complex mathematics
This is simple vs. easy. It definitely is very simple mathematics, pretty basic set theory (not Venn diagram basic, as it needs functions&relations, but nothing even remotely crazy). It probably isn’t easy for most people, as I think this “foundations of mathematics” stuff is not frequently taught, and it does require somewhat more abstract thinking.
Perhaps even the question itself is a type error. Sure, programming is mostly done via Turing-complete languages, and a Turing-completeness is surprisingly easy/trivial to reach. (Doesn’t make “the sum of its parts” any easier though, two trivial function’s composition may not end up as a trivial function.)
As for why I consider it a type error: programming is the whole act and art of building and maintaining computable functions with certain properties, if we want to be very abstract.
Naming variables, encoding business logic/constraints in a safe way, focusing on efficiency are all parts of programming itself. The computing model is just one (small) part of it all.
Programming is the art of understanding a process in enough detail that a computer can be used to automate part of it.
Related:
https://lockywolf.wordpress.com/2019/02/16/a-mathematical-theory-of-systems-engineering-the-elements-by-a-wayne-wymore/
More like complicated mathematics.
Somebody has a reference for this?
Perhaps they are referencing this paper? https://www.cs.cornell.edu/courses/cs6120/2019fa/blog/unified-theory-gc/
Edit: same title, wrong link: https://courses.cs.washington.edu/courses/cse590p/05au/p50-bacon.pdf
The author definitely has more experience with GCs than I do, and perhaps it’s only for this article’s sake that they focus so much on pointers themselves, but.. I feel this is a bit misguided.
A garbage collector (ref counting included) operates semantically on handles. In the ref counting case that handle will as an internal detail increment/decrement a counter. In the tracing case it may have write or read-barriers, but the point is that these are objects that have their own semantics by which they can be accessed. It’s an entirely separate implementation detail if they are represented as an integer corresponding to a memory address/region or something else. There are languages that can express express these semantics natively (I believe RAII is a core primitive here), but we have seen user-level APIs for ref counting (e.g. glib), and runtimes may choose to simply handle it themselves e.g. by generating the correct code via a JIT compiler.
Of course a runtime’s assumptions can be violated, but as the author even points out, the JVM for example hands out handles for FFI purposes, so not sure how much of a problem this actually is. As for the concrete design, I don’t have enough experience to provide feedback - I do like the aspect that code can be monomorphic, but given that tracing GCs have to trace the heap, I feel the runtime checks for that phase would be a negative tradeoff.
Also, to continue with their analogy, if you reach for native code, you effectively call a burgler to your block. Even if you only hand them a key and don’t tell them which house is yours and have an alarm system so you have to know the password, they might just break the window in.
Oof… what is the word I’m looking for here?..
Or… You could not. If you focus on EFI you could use an EFI stub to boot your kernel. Or boot a UKI directly. None of it needs systemd.
systemd-boot is only administratively part of systemd really (and it’s pretty thin!), you can use it with other init systems
Or… You could skip it.
sure! if you don’t need to choose an entry
Can you use systemd on alpine linux?
Not really. PostmarketOS has put in the work to get it functional, but AFAIK it’s not being upstreamed because Alpine is not interested.
See https://wiki.postmarketos.org/wiki/Systemd and relevant entries in the blog.
Embrace, extend…
This is dumb.
I don’t see any competitors to systemd that can run GNOME on the market.
https://wiki.gentoo.org/wiki/GNOME/GNOME_without_systemd/Gentoo
The main desktop environment for Chimera Linux is GNOME and it uses dinit at the init and service manager.
GNOME is available for all the BSDs and they have their own rc based init system.
Well…. Grub reimplements several filesystems (btrfs, xfs, ext* etcetcetc), image formats, encryption schemes and misc utilities.
A good example is how luks2 support in Grub doesn’t support things like different argon implementations.
Booting directly into an UKI/EFI binary is not particularly user-friendly for most users, which is the missing context of the quote. The statement isn’t about needing
systemd, it’s about offering a replacement togrub.I’d say it’s the opposite. An average user doesn’t need to pick a kernel to boot. They use the kernel provided by their distribution and don’t want to see any of the complexities before they can get to the browser/email/office suite/games. It’s us—nerds—want to build our own kernles and pick one on every boot. Supermajority of users don’t need multiple kernels. They don’t know what a kernel is, don’t need to know, and don’t want know. They don’t want to know the difference between a bootloader and a kernel. They don’t care to know what UKI even is. They’re happy if it works and doesn’t bother them with cryptic test on screen changing too fast to read even a word of it. From this perspective—average user’s point of view—UKI/EFI stub is pretty good as long as it works.
If you want to provide reliable systems for users you to provide rollback support for updates. For this to work you do need to have an intermediate loader. This is further complicated by the Secure Boot that needs to be supported on modern systems.
sd-bootworks just fine by skipping the menu entirely unless prompted. Ensuring that the common use-case does not need to interact, nor see, any menu at boot.Isn’t this built in to UEFI? EFI provides a boot order and a separate Next Boot entry. On upgrade system can set Next Boot to the new UKI and on successful boot into it system can permanently add it to the front of boot order list.
I’m not certain what you’re referring to here. If you say that this makes new kernel installation a bit more complicated than plopping a UKI built on CI into EFI partition than sure, but this is a distro issue, not UEFI/UKI issue. Either way it should be transparent for users, especially for the ones who don’t care about any of this.
UEFI is not terribly reliable and this is all bad UX. You really want a bootloader in between here.
This allows you to do a bit more fine grained fallback bootloading from the Boot Loader Specification which is implemented by
sd-bootand partially implemented bygrub.See: https://uapi-group.org/specifications/specs/boot_loader_specification/#boot-counting
UKIs are not going to be directly signed for Secure Boot, this is primarily handled by
shim. This means that we are forced to be forced through a loop whereshimis always booted first which then would bootyour-entry.efi.However, this makes the entire boot mechanism of UEFI moot as you are always going to have the same entry on all boots.
I used to boot my machines with just the kernel as EFI stub and encountered so many issues with implementations that I switched to sd-boot.[^1]
Things such as:
I’m surprised any of them did anything more than boot “Bootmgfw.efi” since that seems to be all they’re tested with.
Fortunately, nowadays you can store the arguments in UKI (and Discoverable Partitions Specification eliminates many of those). The rest was still terrible UX if I got anywhere off the rails.
[^1]: Even on SBCs, Das U-Boot implements enough of EFI to use sd-boot, which is a far better experience than the standard way of booting and upgrading kernels on most SBCs.
I think you’re missing the number of users for whom the alternative is Windows, not a different kernel.
Yes, they could rely on their firmware to provide a menu for that, but they’re also pretty universally bad, and often much slower.
Personally I love grub and don’t ever see that changing har har
I’m one of those using an unusual window manager (sawfish) with xfce4. I’m starting to dread the inevitable day that I’ll have to change back to some mainstream system that will of course include all the pain points that led me to my weird setup in the first place. Fine, so X11 needs work, but replacing it with something as antimodular as wayland seems to be is just depressing. Something must be done (I agree!); this is something (er, yes?); therefore this must be done (no, wait!!). We’ve seen it with systemd and wayland recently, and linux audio infrastructure back in the day; I wonder what will be next :-(
First, reports of X’s death are greatly exaggerated. Like the author said in this link, it probably isn’t actually going to break for many years to come. So odds are good you can just keep doing what you’re doing and not worry too much about it.
But, even if it did die, and forking it and keeping it going proves impossible, you might be able to go a while with a fullscreen xwayland rootful window and still run your same window manager in there. If applications force the wayland on you, maybe you can nest a wayland compositor in the xwayland instance to run that program under your window manager too. I know how silly that sounds - three layers of stuff - but there’s a good chance it’d work, and patching up the holes there may be a reasonable course of action.
So I don’t think the outlook is super bleak yet either.
I’m not convinced that building something like those kinds of window managers on top of wlroots is significantly more difficult than building those kinds of window managers on top of X11. There’s a pretty healthy ecosystem of wlroots-based compositors which fill that need at least for me, and I believe that whatever is missing from the Wayland ecosystem is just missing because nobody has made it yet, not because it couldn’t be made. Therefore, I don’t think your issue is with “antimodularity”, but with the fact that there isn’t 40 years of history.
I assure you, my issue is with antimodularity. The ICCCM and EWMH are protocols, not code. I can’t think of much tighter a coupling than having to link with and operate within a 60,000-line C-language API, ABI and runtime. (From a fault isolation perspective alone, this is a disaster. If an X window manager crashes, you restart it. If the wayland compositor crashes, that’s the end of the session.) You also don’t have to use C or any kind of FFI to write an X window manager. They can be very small and simple indeed (e.g. tinywm in 42 lines of C or 29 lines of Python; a similarly-sized port to Scheme). You’d be hard pressed to achieve anything similar using wlroots; you’d be pushing your line budget before you’d finished writing the makefile.
The Mir compositor seems to be closer to what you’d want, with a stable API to develop your own window manager / DE on. It still runs the code in the same process, though.
Another alternative would be Arcan, which has (what I consider at least) a refinement on the X11 approach to window management with high crash resilience. While it has its own native protocols, there support for Wayland clients.
Yes, Arcan looks very interesting indeed. I’m following the project (from a bit of a distance) with interest.
There is nothing inherent in Wayland that would prevent decoupling the window manager part from the server part. Window managers under X are perfectly fit to be trivial plugins, and this whole topic is a bit overblown.
Also, feel free to write a shim layer to wlroots so you can program against it in anything you want.
I’m talking about modularity. Modularity is what makes it feasible to maintain a slightly-different ecosystem adjacent to another piece of software. Without modularity, it’s not feasible to do so. Obviously there’s nothing inherently preventing me from doing a bunch of programming to get back to where I started. But then there’s nothing inherently preventing me from designing and implementing a graphics server from scratch in any way I like. Or writing my own operating system. There’s a point beyond which one just stops, and learns to stop worrying and love the bomb. Does that sound overblown? Perhaps it always seems a bit overblown when software churn breaks other people’s software.
With all due respect, I feel your points are a bit demagogue without going into a specific tradeoff.
The wayland protocol is very simple, it is basically a trivial IPC over Linux-native DRM buffers and features. Even adding a plug-in on top of wlroots would still result in much less complexity than what one would have with X. Modularity is not an inherent goal we should strive for, it is a tradeoff with some positives and negatives.
Modular software by definition has more surface area, so more complexity, more code, more possibility of bugs. Whether it’s worth it in this particular case depends on how likely we consider a window manager “plugin” to crash. In my personal opinion, this is extremely rare - they have a quite small scope and it’s much more likely to have a bug in the display server part, at which point X will fail in the exact same manner as a “non-modular” Wayland server.
I’m not sure exactly what pain points led you to your current setup, but I don’t think the outlook is that bleak. There are some interestingly customizable, automatable, and nerd-driven options out there. Sway and hyprland being the best-known but there are more niche ones if you go looking.
I use labwc and while it’s kind of enough, it’s years away from being anywhere near something like Sawfish (or any X11 window manager that’s made it past version 0.4 or so). Sawfish may be a special case due to how customizable and scriptable it is but basically everything other than KWin and Gnome’s compositor are basically in the TWM stage of their existence :).
Tiling compositors fare a little better because there’s more prior art on them (wlroots is what Sway is built on) and, in many ways, they’re simpler to write. Both fare pretty badly because there’s a lot of variation; and, at least the last time I tried it (but bear in mind that was like six years ago?) there was a lot more code to write, even with wlroots.
I’m not saying it’s there right now, but it’s not totally dire, and I think it’s reasonable to expect it to get better.
Ah, sorry, I misread that via this part:
…as in, there are, but none of the stacking ones are anywhere near the point where they’re on-par with window managers that went past the tinkering stage today. It’s certainly reasonable to expect it to get better; Wayland compositors are the cool thing to build now, X11 WMs are not :). labwc, in fact, is “almost” there, and it’s a fairly recent development.
Just digging into Hyprland and it’s pretty nice. The keybindings make sense, especially when switching windows, the mouse will move to the focused window. Necessary? Probably not. But it’s a really nice QoL improvement for a new Linux user like myself.
I’m pretty much in the same situation as you. I’m running XFCE (not with sawfish, but with whatever default WM that it ships with). I didn’t really ever explore using Wayland since X11 is working for me so I found no reason to switch yet. For a while it seems like if you wanted a “lightweight” desktop environment you’re stuck with tiling Wayland environments like Sway. I still prefer stacking-based desktop environments, but don’t really want to run something as heavy as GNOME or KDE. I’ll probably eventually switch to LXQt which will get Wayland support soon.
Isn’t XFCE getting Wayland support at the moment?
It’s currently experimental. XFCE’s Wayland support isn’t packaged in distros yet, AFAIK.
Great article! The scientific method at work. It’s always good to question and revalidate old results, especially since hardware evolution (caches, branch prediction…) can change the “laws of nature”, or ant least physical constants, out from under us.
It sounds to me like the choice of benchmarks could be an issue. I’ve been rereading various GC papers recently; they frequently seem to use compilers, like javac, as benchmarks, which seems reasonable.
I’d have thought compilers were pretty unrepresentative for GC. They create a big AST as they parse, which exists as long as the program is valid. Then they walk it and generate code, some of which may be shortlived, but something like javac doesn’t really do any optimisation (it used to, then HotSpot spent the first bit of the load having to undo the javac optimisation) and is little more than a streaming write of the AST. Most programs have a much broader set of lifetimes than compilers.
Hans Boehm has a nice anecdote about adding the BDW GC to GCC. In his first version, it made GCC quite a lot faster. It turned out that it was equivalent to just never calling
free: the compiler was so short-lived that the threshold for triggering GC was never reached. LLVM learned from this and mostly uses arena allocators, and has an option (on by default in release builds, I believe) for clang not to ever free things. Clangd needs to reclaim memory because it’s a long-lived thing, but most objects in clang are live for most of the duration that the tool runs and so it’s faster to simply letexitbe your GC.For what it’s worth, ZGC and Shenandoah both experienced a large throughput boost after moving to a generational GC (they were already low-latency=bounded pause time), on real world benchmarks as well.
I would like to mention GraalOS as a related development, as it’s not too well known (not affiliated, just follow GraalVM)
It would allow the safe hosting of applications without virtualization, in a way maybe similar to CGI, but with Graal-supported languages as scripts (JVM languages, Javascript, Python, Ruby, but also LLVM languages).
The Oracle-written documentations/ads are pretty bad, so let me link to a HN comment which describes what it actually is: https://news.ycombinator.com/item?id=37613065
I think the idea is quite clever.
Is there anyone here that understands tail calls (tail call recursion?) well enough to explain why it is such a big deal, but in simple terms a lowly Python coder can grok? My attempts to understand it lead me here:
Assuming that’s right, the thing I can’t wrap my head around is aren’t tail calls edge cases? The surface area of a sphere (the “tail calls”) grows much slower than the volume (the non-tail function calls) so shouldn’t optimizing for it be a tiny improvement in execution across all calls?
You have got some of the key ideas, but what’s important here is the specific context: this is about optimizing CPython’s interpreter loop.
First, a small correction: functions that don’t call other functions are usually called “leaf functions”. Tail calls are different: a tail call is when the last thing that a non-leaf function does is call another function, eg,
Normally what happens is the outer func calls somefunc() which pushes a frame on the stack, does its thing, pops its frame, then returns, then the outer func pops its frame and returns. If the compiler does “tail call optimization” then what happens is the outer func pops its frame, then does a goto to somefunc(), which runs as before, and when it returns it goes straight to the outer func’s caller.
Tail call optimization turns a tail call into a kind of goto-with-arguments.
Back to the context. A classic bytecode interpreter loop looks like (in C-ish syntax):
There are a couple of reasons why this isn’t as good as it could be:
Each opcode involves two branches, one at the top of the loop to dispatch to the appropriate opcode, and one back to the top when the opcode is done;
The dispatch branch is super unpredictable, so the CPU has a hard time making it fast.
These issues can be addressed using computed goto, which is what the CPython interpreter normally uses. The opcode dispatch is copied to the end of every opcode implementation, using an explicit jump table. Normally the compiler will compile a switch statement to a jump table behind the scenes, but by making it explicit the interpreter gets better control over the shape of the compiled code.
It is initialized like
jump_table[OP_ADD] = &IMP_ADDusing a GNU C extension that allows you to take the address of a label and use it in a computed goto statement.This is better, but compilers still find it hard to do a good job with this kind of interpreter loop. Part of the problem is that the loop is one huge function containing the hot parts of the implementations of every opcode. It’s too big with too many variations for the compiler to understand things like which variables should stay in registers.
This new interpreter loop uses tail calls as a goto-with-arguments to help the compiler. Each opcode is implemented by its own function, and the dispatch is a tail call to a function pointer.
The tail call arguments are the variables that need to stay in registers. The release notes talk about compiling with the “preserve_none” calling convention, which helps to ensure the tail call is more like a goto-with-arguments-in-registers and avoid putting variables in memory on the stack.
This time the
fun_tablecontains standard function pointers.That’s the basic idea, but it has some cool computer science connections.
The idea of a function call as goto-with-arguments was popularized by the “lambda-the-ultimate” papers which talked about using the lambda calculus and continuation passing style when compiling Scheme. Low-level compiler intermediate representations such as continuation-passing style and A-normal form are literally goto-with-arguments-in-registers. The classic static single assignment form (SSA) can be viewed as a relatively simple transformation of CPS.
So the idea is to be able to program the compiler’s back end more directly, with less interference from the front end’s ideas about structured programming, to get the interpreter loop closer to what an expert assembly language programmer might write.
Thanks for the great explanation! I think I am mostly following it, although I have one follow-up:
Would it ever be the case that an opcode could have too many required arguments to fit in the remaining registers? Or would that never be the case because these opcodes map to instructions for a given architecture and the arguments map to parameter registers?
I was a bit vague there :-)
The arguments to these tail calls take the place of the local variables that would have been declared above the for(;;) loop in the classic version. They are the variables that are used by basically all the opcodes: the bytecode pointer, the program counter, interpreter stack, etc. So it’s a fixed list that doesn’t depend on the opcode. INTERP_ARGS was supposed to suggest a pair of C macros that contain the boilerplate for opcode function declarations and calls - the same boilerplate each time. The goto-with-arguments is being used as goto-and-here-are-our-local-variables.
The arguments to an opcode are typically fetched from following elements in the bytecode array, or they might be implicit on the interpreter stack. In the first example here https://docs.python.org/3/library/dis.html the right hand column shows an argument for each opcode, except for RETURN_VALUE which implicitly uses the value on top of the stack.
It’s a bit confusing because we’re dealing with two levels, the C interpreter and the bytecode it is interpreting. I discussed the C call stack when talking about tail calls, but that’s an implementation detail that isn’t directly visible in the interpreter source code; but there’s also an explicit interpreter stack used for bytecode expression evaluation, eg, OP_ADD typically pops two values, adds them, and pushes the result.
I don’t know anything about Python internals, but on the Java front certain instructions are very high-level, e.g. they might cause “expensive” class loading involving IO, so they definitely not map to instructions.
I think parent commenter mostly meant that writing the code this way would “implicitly assure” that the compiler will prioritize arguments (like program counter) to fit in registers, over non-arguments, of which there might be several depending on complexity of the instruction, and otherwise all the variables would compete for the same registers.
Slightly unrelated, but you seem to be well-versed in the topic: while I was writing a JVM interpreter, I was researching the difference between this “classic interpreter loop” as you outlined above vs the version that copies the ‘opcode dispatch’ part to the end of each opcode implementation (Wikipedia seems to call it token threading), and while your reasoning (the dispatch branch being unpredictable) seems sound and this is exactly the same I have read/heard from others, a very experienced colleague and my minimal benchmark showed that this may not actually result in improved performance on modern hardware.
I am asking if anyone has more conclusive “proof” that this is still the case on modern hardware? I made a tiny language with a couple of basic opcodes and wrote an interpreter in Zig for both versions, making sure that function calls implementing the opcodes are inlined, and the interpreter loop was actually faster on the few (hand-written) toy example programs I wrote in this “language”. My assumption is that my language was just way too simple and there might have been some i-cache effect at play, which would not be the case for a more realistic language?
Also, I haven’t had a compiler for this “language” and the kind of program I could write by hand might have been way too simplistic as well to count as “proper benchmark”. Anyway, I am looking to find some time to benchmark this better.
Yeah if you only have a couple of opcodes that’s too small to be difficult. Anton Ertl has a good discussion https://www.complang.tuwien.ac.at/forth/threaded-code.html
Is this something that GCC / CLANG will automatically do if you put the tail calls into this goto-with-arguments implementation?
What do you mean by your first “this”? And what do you mean by your second “this”?
Will the C compilers do the tail call optimization automatically if you write your C code using a goto-with-arguments style?
You also need either
__attribute__((musttail))or[[clang:musttail]].C’s usual calling conventions make it difficult to do tail call optimization in general, so it needs an explicit annotation. (It’s almost hopeless in C++ because destructors make most things that look like tail calls into non-tail calls.)
(Edit: lol @fanf posted a great explanation while I was typing this out and brings up some great points about computed gotos)
Note that this optimization is for the interpreter implementation (i.e. the C interpreter code), not Python semantics. Tail calls in Python code will still always allocate a stack frame.
Basically, this change improves the way the main CPython bytecode interpreter loop works. A simple way to implement a bytecode interpreter loop is:
However, this approach has some problems. We end up with a giant function that’s hard for a C compiler to optimize (there are a ton of branches (which the branch predictor doesn’t like), lots of local variables (which makes register allocation difficult since we have a ton of variables and not enough registers to fit them into), any inlined nested calls will bloat the whole function further (negatively impacting CPU instruction cache), etc). At minimum, we have to jump twice: once when switch on the opcode, then again at the bottom of the loop to goto the top.
A more desirable architecture is:
Since the functions are smaller, the compiler can go ham with optimizations. Since each function has the same call signature, all the compiler needs to do is
jmp(i.e. goto) the next function directly without any cleanup/resetting for an outer loop. Since it’s just a jump (because of the tailcall optimizations), we don’t allocate a stack frame and have eliminated the jump at the bottom of the loop to get back to the top.See this helpful writeup for a more detailed explanation and example C code for the protobuf interpreter: https://blog.reverberate.org/2021/04/21/musttail-efficient-interpreters.html
(Note: Haven’t read the python specifics)
Saving on creating stack frames is definitely part of it. And you’re right about tail recursion being an edge case (compared to an explicit loop in Python). Tail calls, however, not so much.
return obj.Method()happens all the time.Further, having tail call optimization allows you to implement “subroutine threaded” interpreters, in which the interpreter loop is a tail recursion based “call the next instruction.” This is often quite fast, though not generally faster than “direct threaded code.”
Further further, you could compile to continuation passing style, which turns your program into a series of tail calls, that could then be optimized. In this case, all the control flow logic could be simplified by using the continuations, too.
Lots of potential options!
The parentheses in the following sentence threw me off, and I missed a subtlety of what it’s saying:
For those not aware, “tail recursion” and “tail call” are different concepts: a “tail call” is anything of the form
return f(...)(or equivalent for method calls,lambdas, etc.). @apg is saying this form is common.In contrast, “tail recursion” is when a function makes a tail call to itself, e.g. a Python function like:
@apg is saying that it’s rare to see such tail recursion in Python (after all, it’s not super common for Python functions to call themselves at all; let alone as a tail call).
Those ideas can also be generalised:
A tail call is a function call which appears in “tail position”. We can apply that idea of “tail position” to other language constructs besides function calls.
We can generalise the idea of tail recursion beyond functions which immediately call themselves, to a set of mutually-recursive functions which make tail calls to each other. For example,
(is_even, is_odd) = (lambda n: is_odd(n-1) if n else True, lambda n: is_even(n-1) if n else False)Thanks for expanding on this!
One modification I’d make to your expansion: “tail position” is best explained with the concept “the call is the last thing before returning.”
The call to
next()is not in tail position. The addition operation is in tail position, and so this can’t be optimized. To account for this, you’ll often rewrite this code:Now, the addition happens before the call to
nextand the call is the last thing done before returning. This can be optimized.Right.
In functional programming languages, you usually have to use recursion to write a loop. Languages like Scheme and SML commonly support “proper” tail calls, i.e. tail call optimization is guaranteed. The effect is that tail recursion compiles into code that is as efficient as a while loop in an imperative language.
AKA Lambda: The Ultimate GOTO :)
Rust noob here
Stringimplies the existence of&String, and&strimplies the existence ofstr. Why don’t we just have String and &String?stris a dynamically sized type which is basically the same as[T], but for string data (bytes which are guaranteed to be UTF-8).&Stringis a reference to aStringwhich is allocated on the heap, whereas an&strcould point to a string allocated somewhere else (in particular, string literals are not allocated on the heap).If you understand the relationship between
Vec<T>and&[T], the relationship betweenStringand&stris the same.A lot of people have given you some great answers, and I’m a bit impartial here, but I really like the diagrams in the Rust book if you’re having trouble visualizing the differences between
&Stringand&str:&Stringis here: https://doc.rust-lang.org/stable/book/ch04-02-references-and-borrowing.html&stris here: https://doc.rust-lang.org/stable/book/ch04-03-slices.html#string-slicesThis also visually explains one of the reasons why
&strexists: you need two hops to get to the data with&String, and only one with&str.Both of those types also exist
The difference is that str is a slice of a String
String’s slice is different from other slices because if the variable width of Unicode, in short
A
Stringis an owned type, it’s is a length, a capacity, and a pointer to an exclusive buffer which it frees ondrop(or when realloc-inc). An&Stringis a reference to that specific structure.But lots of string-like are not that e.g. a string literal is a segment of the binary, or you can get a string from a slice of binary data.
This is the same relationship as
Vec,&Vec,&[], and[](String/strare wrappers to those which guarantee the underlying buffer always contains valid UTF8).Nobody seems to have mentioned this particular point explicitly:
&strcan represent substrings of an existing string, whereas&Stringcannot. (That’s what “slice” means, of course, but the implications weren’t obvious to me when I worked through this the first time.)This helps to support highly expressive yet efficient programming. For example, you can idiomatically iterate over
.linesor.splitwithout any heap allocations (neither from the iterator, nor from the substrings). You might have already done it without realizing what you didn’t have to do!strexists;&stris its borrowed form: https://doc.rust-lang.org/std/primitive.str.htmlThe reason you almost never see
str(except in signatures where a reference is composed later) is because it’s a slice, which means it doesn’t have a statically known size. Rust requires dynamically sized types to be accessed though a reference, since references are always sized (more precisely, they’re pointers, and pointers are sized).A
Stringis essentiallyBox<str>with some additional metadata (actual length, capacity). As such a reference like&Stringdoesn’t point to the slice itself; it points to the (heap) structure containing the slice.TL;DR: both exist because they express different concepts / have different ABI consequences. Comparatively, a
&stris to achar *as a&Stringis to a&std::string.Conceptually,
stris to[u8]whatStringis toVec<u8>. In other words,stris pretty much an array of bytes,[u8], but with the additional restriction that it has to be valid utf8.So, as https://lobste.rs/s/cbvnzl/string_vs_str#c_jqzhu7 already mentioned, the difference between
&strand&Stringis analogous to the difference between a slice of bytes,&[u8], and&Vec<u8>[^footnote below] mostly, which is not perfect).It just so happens that
stris rarely constructed.I think the type of
"quoted strings"could have beenstr, but then you’d have to use&"quoted string"instead of"quoted string"almost all the time. So, they just made its type&str.[^footnote below]:
&[u8], and&Vec<u8>, btw, can also be annoying to distinguish; this is something I wish Rust did better, but I don’t have a better design in mind. Rust’s solution isAsRef, but it often adds to the confusion, since&Stringcan often act like&str, but also often refuses to act as&strdepending on what you are doing.Because Rust in some sort made the mistake of assuming most people want to append to strings all the time, so String carries excess capacity. I think the better call would have been to say that
strand&strare the types to represent an owned fix length string and a reference to it, and have aStringBuilderfor the uncommon case where someone actually needs to append to a string.That’s only true if you’ve appended to it. (since it uses the same capacity doubling strategy as
Vec, and its initial capacity is the same as whatever you try to put in it)That’s literally what we already have today.
That’d be a weird name IMO.
Stringis just aVecwith UTF-8 data. ShouldVecbeArrayBuilder? In any case, languages that let you append to things calledStringare supremely common, so it’s not like the current naming is anything anyone would be confused by.You’re speaking about a different form of capacity, @mitsuhiko is taking about how String is a
(pointer, len, capacity)and how it could be(pointer, len)if it were immutable.I personally have joked that I want a Rust 2.0 only to turn
StringintoStrBuf. This would make it consistent withPathandPathBuf,I also agree that trying to achieve consistency in one place can still leave other consistencies. I think that Vec is common enough that ArrayBuf would be a bit odd.
Java is a very prominent example of String being immutable and StringBuffer being mutable. C# has String and StringBuilder. In Python, strings are immutable, but with dynamic typing, it feels like they’re mutable. Ruby almost went the same way for Ruby 3.0, but Matz was worried about breaking code. Go has immutable strings.
I’m not so sure it’s clear which is more popular.
That said, all of this is really moot for Rust, and I also agree with @withoutboats that it’s also pretty irrelevant to the question asked.
That’s how I interpreted it initially, but the word “excess” made me question that. “excess capacity” to me, means more capacity than you’re using. But I concede that your interpretation is reasonable.
I like that.
Oh, certainly. I’m just saying that it’s common enough that I don’t think it should be a point of confusion.
What does dynamic typing have to do with it? In Rust, thanks to variable shadowing, you can do:
or in Java you can do:
just the same as Python’s
Is there some other sense of Python strings “feeling mutable” I’m missing?
This is a sincere question btw, I know tone can be lost and come across terse in writing.
You’re all good, you sounded very sincere to me!
The dynamic typing bit is probably wrong, you’re correct. I was thinking more of how &str doesn’t implement Add in Rust, that is, in Python you can
but in Rust, it makes you turn
"John"into aStringfirst. This is really more of an API choice than an inherent property of static typing.The more direct equivalent Rust example would be
That has more to do with explicit costs, the Python version semantically allocates 4 different strings, although in practice it gets constant folded by the peephole optimiser to only allocate one (and it’s a constant in the code object).
Indeed, after all you can write the “python” code in Go or Java or haskell (by modifying it a bit in the latter case).
Thanks for clarifying. Yes, that makes sense - I hadn’t considered that you can concatenate strings like that in Python, which admittedly does have a very mutable feel.
But you’d still need a type to represent a stack allocated string, so
strand&strwouldn’t disappear, right? All you’d do is add an immutableStringtype, which would only add to the confusion.That’s not exactly what I understand. I’m not him though so I could be wrong. You’re not wrong that you’d still need some sort of slice type. What I understand is that he’s saying the types should be like this:
Stringturns intoStringBuilderstris repurposed to mean what we callStringtoday.&stris repurposed to mean what we call&Stringtoday.He didn’t say what the name of today’s
strshould be. I think the difficulty there is one of the problems with this specific naming scheme, and that’s not even getting into howstris the only language type here (and it almost wasn’t, and maybe that would have been the right call too, but it’s not completely obvious).That is what I intended to convey.
What type would you use for a pointer to a slice of UTF-8, then?
&stris layout compatible to be able to point to both astr(Box<str>) and a slice of utf-8. It would be possible to get away with one reference type for both things here.But in the first case (
str(Box<str>)) you have a narrow pointer to a place holding a wide pointer (the box) while in the second case you have a wide pointer to a slice.Or if what you meant by “point to a
str(Box<str>)” was pointing to the data owned by the box, now the type is special and inconsistent with the rest of the language. That is to say, ifstris a box owning a slice, and&stris pointer to a slice, then&Twould refer to a pointer toTin most cases, but not whenTisstr.If you meant neither of those, could you clarify?
&stris(length, borrowed_ptr).stris itself(length, owned_ptr). They are layout compatible, the difference is just that for the owned one you know you need to free the memory on drop.As for “type is special” it’s only special if you want to make it so. Fat pointers are a core affordance of the language and various conversions between fat pointers exist today.
So you’re saying what is
strtoday would effectively be[char]? Then once again, all you’re doing is adding a type and renaming everything else, right?No.
stris an array of bytes ([u8]) that happen to be valid UTF8.charis 32 bit long.So
Stringbecomes two different types, one of which used to be convenient but loses all convenience and the other is just inefficient (because I assume in that schemestris an alias forBox<str>andStringBuilderloses all string-like behaviour).And I assume
Vecgets renamed toArrayBuilder, and you have to convert back and forth between that and the aliasedBox<[]>?It’s not my proposal, so I don’t know how to answer any of this.
stris a dynamically sized type, so it has to be behind a pointer; the owned version of it isBox<str>. Of course, this does already exist, so all you’re really doing is grinding axes about howStringisn’t calledStringBuilder. Can’t you see how this has nothing to do with the person’s question?Is the contents of str mutable? Just not grow-able?
Technically yes it can be, though the sort of things you can do with this are limited because UTF-8 is a variable width encoding and you can’t replace a character with a character of a different width without possibly growing the String. Here’s an example of a method that can mutate a
strin place: https://doc.rust-lang.org/std/primitive.str.html#method.make_ascii_uppercaseThe easiest way to get an
&mut stris to start with aString(though there are others); you can’t get an&mut strfrom a string literal, which is the other easy way to get a&str.I am definitely not well versed in rust, but I believe part of the reason why you can’t have mutable references to string literals is that they are often part of the read-only region of the binary (of course, this is only a possible optimization, not a necessity). One can rather easily cause a segfault in C by doing just that.
This is true but also not really fundamental. Compare to slice literals
&[1, 2, 3]- Allocated in read-only memory, lives forever&mut [1, 2, 3]-[1, 2, 3]copied to the stack and mutable borrowed, lives until it is implicitly (roughly at the end of the stack frame).The machinery is there, we’re just lacking the syntax to say “I want a mutable version of this string literal”. Which we’re lacking because it wouldn’t be very useful.
That’s correct, a string literal is generally stored in .rodata / .text, is global (as different occurrences of the same literal may be dedup’d), and can be shared far and wide. There’s no way modifying one in place would be safe by Rust’s semantics.
Some of the points are highly debatable.
Because lexical shadowing cause lots of bugs in C and C++, hence D and Zig forbid it.
You would have to compare comptime to the swath of C++ templates it replaces.
Because it turns out memory safety is highly overrated by some, and is only one of the desirable properties of a language, and probably behind learnability and productivity in terms of priority.
The point in the post is that there are languages that solved it.
The post directly refers to C++ templates and doesn’t think it’s a valid solution.
Sure, people may disagree, but please be more specific. The post particularly lays out the point where they feel like they are recommitting previous mistakes.
Memory safety is a tangible objective with a reasonably good definition. I have yet (as a trainer and someone very interested in that space) to find a tangible definition of learnability and productivity. Productivity is often mistaken with familiarity, learnability is often very biased to what the speaker knows.
No, I’ve read that correctly. It describe a scoping exactly like in C++. The Rust solution does create bugs. https://rules.sonarsource.com/cpp/RSPEC-1117/?search=shadow
Please refer to Capers Jones for data points about various languages, expressed in cost per function point.
https://jyx.jyu.fi/handle/123456789/47698 languages with “pub” and “fn” as keywords avoid the fact that this reduces learnability for non-english speakers.
Capers Jones FP are metrics of business value of a program. It’s a good productivity metric the whole process of programming. Sometimes, you can find productivity metrics that give you the relationship of FP to LOC on a programming language basis, but even this is very far away from a measure of programmer productivity on a language and are influenced by other effects, like surrounding domain or domain expertise of the programmer.
I looked at the 259 pages document you linked quite hard. The term “Learnability” is mentioned once. It seems to be a very good meta-study, however, reading this:
I’m not sure it supports your point, especially not at a whole language scale.
To me, memory safety is like the baseline, the bare minimum. Anything without it simply isn’t worth using for any reason.
This must be more extreme than you mean. Unsafe Rust forgoes memory safety. Is unsafe Rust not worth using for any reason?
When people say things like your parent, what they always mean is “by default.” Even programs written in GC’d languages can have memory safety issues via FFI, but we still call them memory safe. Unsafe is like an FFI to a (slightly larger) language.
Maybe, but I really wish engineers, at least, wouldn’t talk that way. Engineering almost always involves trade-offs, and the maximalist stance (“bare minimum”, “for any reason”) make those harder to identify and discuss. For example, the follow-up comment is much more interesting (IMO) than the one I responded to.
So, rereading your comment, I think I missed a word:
Emphasis mine. I thought you were saying (like I hear often on the internet) that the existence of unsafe Rust means that Rust is not a memory safe language, and was trying to talk about that definition, not about the judgement.
My apologies.
I think doing something potentially unsafe can be worth it, if you are willing to pay the cost of spending a lot of time making sure the code is correct. For large C, C++, or Zig code bases, this isn’t really viable. Easily 10xes your development time.
I see unsafe Rust the same way I see the code generating assembly inside a compiler. Rustc or the Zig compiler can contain bugs that make them emit incorrect code. Same with Go or javac or the JIT inside the HotSpot VM. So you are never free of unsafe code. But the unsafety can be contained, be placed inside a box. You can design an interface around unsafety that doesn’t permit incorrect use. Then you can spend a lot of time proving it correct. And after that, you can write an infinite amount of code depending on that unsafe module without worry!
Essentially, I view
unsafein Rust as a method of adding a new primitive to the language, the same way you could add a new feature to Python by writing C code for the Python interpreter. The novelty is just that you write the new feature in the same language as you use the feature in.If 1% of your code is unsafe modules with safe interfaces, and you 10x the cost of developing those parts by being very careful and proving the code correct, your overall development cost only went up by 9%. That’s a much lower cost, and thus what I mentioned at the start, being willing to pay the cost of doing it right, becomes much smaller. It will be worth it in many more situations.
That got a little rambly but I hope I communicated my stance correctly.
An effect that large probably requires poor coding practices. Stuff like global variables, shared state between threads, lack of separation between effects and computation… When you make such a mess of your code base, OK, you need strong static guarantees, like “no arbitrary code execution is ever possible no matter how buggy my program is” — that is, memory safety. Now don’t get me wrong, I love static guarantees. I love when my compiler is disciplined so I don’t have to be. But even then you want a nice and maintainable program.
My hypothesis, is that if you do the right thing, that is, properly decomposing your program into deep modules and avoid cutting corners too often, is that memory safety doesn’t boost your productivity nearly as much as a whopping 10x. Especially if you go for some safety, like just adding bounds checks.
I will concede that in my experience, we rarely do the right thing.
I don’t know, when Java introduced a wider audience to GC, software development has definitely experienced a huge boost. Maybe 10x just in itself is not a correct figure. But you don’t only ship software, you also maintain it, fix bugs etc - and here, guaranteed memory safety can easily have an order of magnitude advantage, especially as the software grows.
Java also popularised huge standard libraries. It’s easier to be productive when a good portion of the work is already done.
The important point, to me, is reducing the potential for harm, and there are many ways to do that. For example, memory unsafety can be mitigated by running code in a sandbox of some kind (such as by compiling and running in a WASM environment), or by only running it against trusted input in simple scenarios (my local programs I use for experimenting), or by aggressively using fuzzers and tools like valgrind, ASAN, LSAN, and static analyzers like Coverity. All of these are valid approaches for mitigating memory unsafe code, and depending on my risk model, I can be OK with some level of memory unsafety.
Further, there are domains in which harm is more likely to come from too-loose type systems than it is from memory unsafety. I can implement my CRUD app in pure memory-safe Python, but that won’t protect me from a SQL injection… Something that enforces that the database only interacts with
SanitizedString’s might. On the other hand, in exploratory development, too strong of a type system might slow you down too much to maintain flow.Anyways, I generally don’t like maximalist stances. If multiple reasonably popular approaches exist in the wild, there are more than likely understandable motivations for each of them.
Can you explain what exactly memory safety means to you?
I’m writing a lot of code that is 100% “memory safe” (in my view), but would not be possible to be written in memory safe language conveniently (like Rust).
the things we can universally agree on:
What would that baseline contain for you?
They probably mean the definition you get if you stop at the summary of the Wikipedia page, which is easy to misinterpret as saying that they have to be enforced by the toolchain (in which I include things like linters and theorem provers) or VM having compile-time or runtime guard rails which cannot suffer from false negatives.
That’s why i’m asking. The “summary definition” of Wikipedia isn’t really helpful at all.
The chapter “Classification of memory safety errors” is much better in listing potential problems, but those are language dependent.
Zig (in safe modes!) will have checks against Buffer overflow and Buffer over-read, and has increased awareness for Uninitialized variables, as
undefinedis an explicit decision by the programmer. This doesn’t mean Undefined variables are a non-problem, but it’s less likely as you have to actively decide not to initialize.<pedantic>Use-after-free, Double-free, Mismatched-free are not possible in Zig either, as the language has no concept of memory allocation.</pedantic>This doesn’t mean a userland allocator won’t suffer from this problems, but the GeneralPurposeAllocator has mitigations to detect and prevent these problems.In C, you actually have a problem with the built-in support for
mallocandfree. Clang, for example, knows thatmallocis special, and can be considered “pure” and returning a unique object. This is how the function is defined in the C standard.This yields interesting problems with the assumptions of the programmers and what the compiler actually does.
Consider the following program:
The compiler will optimize it to this:
This is because
mallocwill always return a pointer to a new object, so it can never alias with another result ofmallocever. Alsomallocwon’t have side effects besides yielding a new object. So eliding the invocation ofmallocis fine, as we don’t use the result.If we do remove the
glob = p1;line, the compiler will just make the functionreturn true;and won’t emit any calls tomallocat all!As Zig has no builtin allocator in the language (but it’s a userland concept inside the standard library), the compiler can’t make the assumptions above and thus actually integrate the checks.
I was furious for a couple of seconds.
I feel you, but the React stack nowadays is so deep and diverse that it feels like it’s gonna be soon affecting browsers’ architecture. They literary solved all the hard problems in a rather elegant way. I compare to my days with PHP3 and jQuery 🙂
While my blog is in PHP. I really enjoy React actually. Also, I very much like this component library: https://mantine.dev/
I don’t think it’s elegant by any means. In practice.
It basically made functional UIs mainstream, which greatly improved testability, and correctness.
I do remember the millions of websites/small GUIs where you could easily end up in inconsistent states (like a checkbox not being in sync with another related state) and while UI bugs are by no means “over”, I personally experience less bugs of this kind.
(Unfortunately, API calls are still a frequent source of errors and those are often not handled properly by UIs)
Why not? Any points against? What would you use for complex web apps?
I mean react itself, not your particular pick of options inside that stack.
React itself is also cool.
A sampling of my list:
For the Nas : https://hexos.com/ I don’t know what state it’s in now - it’s pretty new. But Linus from LTT invested in them specifically because he wanted a simple Nas system with buddy backup.
OK, I’m glad I’m not the only person who thinks about buddy backups.
Let’s see how they execute, although to recommend others, I would prefer an appliance.
For those who happen to be using ZFS anyway, ZFS is exactly this. I know it’s sadly difficult to install in some operating systems.
TIL you can
zfs sendencrypted datasets. That’s very neat!OMG IT’S THE NEATEST THING IN THE WORLD i mean sorry yes
Though be aware that encrypted sends had a few bugs recently.
I do read much of the web in the fetch/convert-to-text/open-in-vim mode.
I tried Chromium’s dumping; this seems inferior to Firefox Marionette session.navigate which I use most of the time.
An observation: a lot of sites are readable when you kill both scripts and CSS; some of them require curl-impersonate though.
Funny you mention this, but I spent a few months designing an open-source key escrow via social recovery for $work the year before last. Sadly it wasn’t released. But it’s very possible!
In theory this can be done. You can do n-m secret sharing so you have m contacts and n are required to agree to recover your account, but although I think in practice it would work, it seems unsatisfactory to me.
I already trust some parties with very sensitive things- even a bank safe deposit box might be enough, but I kinda wish there was something nicer.
Two straightforward alternatives:
Put your secrets in 1Password. They have a whole emergency access thing.
Put your secrets on a USB drive and drop them off with a relative or in a safe deposit box.
I am familiar with Bitwarden’s “timed” access. It just doesn’t feel “complete”.
I’d rather print the passwords too, in addition of a USB drive. I think the safe deposit box feels right, because I already entrust my bank with my money. Trusting them with IT matters is different (too many banks with funky security policies), though, so in theory I would trust them to do key escrow for me- or even provide me a password manager, but in practice, it would not feel good either.
Oh! I forgot my latest obsession.
A browser plugin to add transcripts/alt text to random websites. If I install this, before sharing an inaccessible comic strip (looking at you, xkcd and PA!), I can write my own transcript. Then, if I share it with vision-impaired users using the same browser plugin, they get my transcript.
Xkcd already has community transcripts through explainxkcd: https://www.explainxkcd.com/wiki/index.php/Main_Page
So a plugin that pulls from that existing data source may be very feasible.
The encrypted backup thingy: you can do that with synology
Peer-to-peer with some NAT hole punching, or you need to bring your own VPN?
Your own VPN, or you can use their quickconnect service I guess
I’d like to have an open-source port scanner with a web interface. E.g. like http://www.dnstools.ch/port-scanner.html , but as open source, so that I can install it on my own server, and people can use it to scan their own PC for open ports.
Background is that there are some network (German Freifunk, or the LAN at events like Chaos Communication Congress) which don’t have client isolation, so open TCP/UDP ports on a laptop can be accessed by other clients on the same LAN (this is different from home internet connections, where the clients are often behind the firewall provided by the DSL/cable router). I want to set up a server+website in such LANs so visitors can easily scan their laptops/phones for open ports.
Bonus points if it also does other client scans (e.g. for insecure service settings); basically the scans from Nessus but with a simple web frontend.
I wonder if that could be done in a few lines of CGI script: essentially return the output of nmap on the connecting client, right?
I believe that would port-scan the server’s, not the client’s network.
That seems reductive. There are rainmap, rainmap-lite, and WebMap that prove it can be done and isn’t especially hard to implement, but none of those are anything like “a few lines of CGI script.”
I think, more than any implementation challenges, the real obstacle to writing something like this is that nobody could sell it. If you tried to offer it as a service, you’d be kicked off your host before your first customer finished their trial period. And if everyone who wants to use it needs to stand up a web server themselves in the environment where they want to use it, well, that’s actually harder than standing up a box where you just run nmap, nmapfe, etc.
It’s something I’d like to have too, but there are just no places where it makes sense to me.
This might align with your interests:
https://github.com/cldrn/rainmap-lite