Maybe a little bit, but it’s a recurring theme in AoC that you have to implement the spec as written, but not the spec as you think it means on a first read.
I think there was a rash of solutions in the early days of Dec 2022 where people were oohing and aahing over that current generation of LLMs solving the problems instantly.
It died down quite a bit as the difficulty ramped up.
Oh yeah, I got one very tedious bit of slice manipulation handed to me by copilot but for the rest it’s been mostly saving me typing debug output and the likes.
Guilty as charged. I messed around with ChatGPT on the first few problems last year. That was right after it came out, and it was pretty amazing how fast it could come up with a (typically slightly buggy) solution.
I apparently lucked into doing it the way that doesn’t run into any of the problems other people had, on a whim. (Spoiler below, stop if you don’t want them, but it’s day 1 so…)
For part 1, rather than doing a multi-match and extracting the first and last matches in the list, I did a match against the input and a match against the reversed input, which is an old trick.
For part 2, I kept the same structure rather than rewrite, which meant that I matched the reversed string against /\d|eno|owt|eerht|ruof|evif|xis|neves|thgie|enin/, and then re-reversed the capture before passing it through a string-to-num map.
And it turns out that that totally sidesteps the problem of “wait, how am I supposed to get 21 out of xxtwonexx?”
For part 1, rather than doing a multi-match and extracting the first and last matches in the list, I did a match against the input and a match against the reversed input, which is an old trick.
I took a similar approach. I’m using C++, so the natural solution was to use reverse iterators .rbegin() and .rend() that iterate through the elements of a container in reverse order. Rather than use a regex—which seemed like overkill—for part two, I just had an array of digit names I looped through and performed the appropriate search and chose the earliest one:
for (int i = 0; i < 10; i++) {
auto it = std::search(line.begin(), line.end(), digit_names[i].begin(), digit_names[i].end());
if (it <= first_digit_name_iter) {
first_digit_name_iter = it;
// ...
And in reverse:
for (int i = 0; i < 10; i++) {
auto it = std::search(line.rbegin(), line.rend(), digit_names[i].rbegin(), digit_names[i].rend());
if (it <= last_digit_name_iter) {
last_digit_name_iter = it;
// ...
My figuring on what is and isn’t “overkill” is: AoC is ranked by when you submit your solution, so that’s time to write the code plus time to run it. If something is really the wrong tool, the challenge will prove it by making your solution take an hour, or a terabyte of RAM, to run. But if I’m using a language where regexes are “right there” and they make my solution take 100ms instead of 10ms, I’m not bothered.
I like AOC because everyone can have their own goal! I’m impressed by people who can chase the leaderboard. I always personally aim for the lowest possible latency. Managed to get both parts today in under 120 microseconds including file reading and parsing.
The difficulty, IMO, is that the problematic lines aren’t in the sample.
I did overlapping regexp matches. It was easy, once I caught why my first attempt didn’t work. Another solution would be to just do a search with indexOf and lastIndexOf, for each expected words, but you have to be careful to sort the results.
There’s a subtle hint, because the first sample does include a case where there’s only one digit (the “first” and “last” match are the same, or you could say they overlap completely). When you get the part 2 spec you have an opportunity to ask yourself “hmm, what changes about overlaps now that we’re matching strings of more than one character?”. Or at least it gives you a good first place to look when things go wrong.
Apparently some peoplle tried to solve part 2 using substitution (replace words with digits and then feed that into the solution for part 1), which also suffers from problems with overlaps, but in a way that’s harder to dig yourself out of.
RustDesk scares me. Here’re some of the things I found when I looked through rustdesk/rustdesk-server’s source code last (b713303c1557f158a3b6279fcce5139190ed2937):
Lots of functionality is implemented by spawning sh and using it to run shell code
EDIT: The other part that’s really shocking is the needless amount of additional allocation, i.e. using String, Vec, .to_owned() and .to_vec() everywhere rather than borrowing as &str or as a slice.
This reminds me a lot of the Rust project I was on at my previous gig, especially all the unwrap_or_default(), checking if the string it just generated is_empty() instead of using an Option, useless use of cat and other stuff that is easily done from within Rust… I had to fix a lot of that.
I think it’s a strong signal of a team of developers who are not very familiar with Rust, working under time pressure to deliver a product, being given features to implement with instructions on how to do it in shell and not having much time or incentives to learn how to do it properly.
This reminds me a lot of the Rust project I was on at my previous gig, especially all the unwrap_or_default(), checking if the string it just generated is_empty() instead of using an Option, useless use of cat and other stuff that is easily done from within Rust… I had to fix a lot of that.
Excessive of mut is another sin the codebase commits.
I think it’s a strong signal of a team of developers who are not very familiar with Rust, working under time pressure to deliver a product, being given features to implement with instructions on how to do it in shell and not having much time or incentives to learn how to do it properly.
It probably betrays a lack of understanding of the shell commands being executed as well (at least the awk/cat/ps wrappers—I can almost forgive wrapping loginctl in a rush to ship, as D-Bus APIs are often poorly documented and annoying to work with).
I have to imagine if the developers had actually understood that the awk/cat wrappers were simply reading files and doing rudimentary parsing, then they would have at least tried to implement them in rust.
EDIT: The other part that’s really shocking is the needless amount of additional allocation, i.e. using String, Vec, .to_owned() and .to_vec() everywhere rather than borrowing as &str or as a slice.
I’m reminded of a piece of advice given in earnest to beginner Rust programmers, that if you’re having trouble dealing with lifetimes when getting a piece of code you’re writing to compile, just call clone() or otherwise go head and do the extra validations, in order to get the essential logic of your program working. After all, these extra copies and boxings of data structures are not really very different from what the runtime of an interpreted, dynamically-typed language like Python would be doing under the hood as a matter of course, in your particular code the extra overhead might not be meaningful anyway, and if it is you at least have the option of rewriting in to use more references and fewer clones once you’ve gotten the initial code working.
That’s a good advice that I need to keep reminding myself, even after years of using Rust. Unless used in the most critical code paths, clones and allocations don’t really matter, while it’s easy to waste a lot of time trying to avoid them.
Reminds me of my friend who works as tech support in a hospital. From her I have gathered two main facts:
Medical professionals are usually idiots, and
It should be a capital offense to force quarterly password changes upon cleaning staff who have zero real need or desire to touch the medical IT system but still have to use it to log timesheets
In my experience, it is never productive (but possibly cathartic) to label people as idiots.
Computer people often seem to have a hard time understanding this, but just because someone isn’t a computer expert doesn’t make them an idiot. They have better things to do than trying to work a stupid piece of machinery, and as you can read elsewhere in these comments, they have to do so under a tremendous amount of stress and often while sleep deprived.
They might be a “computer idiot”, but by that standard I’m a total idiot in most non-computer fields. And I suspect a lot of us are.
One would think a system could be designed wherein someone whose role requires one simple set of permissions (view and modify one’s own timesheet) would be exempt from the normal security standards of rotating passwords because the permissions of the role simply can’t result in any kind of harmful system breach.
There is absolutely no reason to use passwords in the hospital setting at all. Everyone should just use a card that is unlocked with a PIN code for the next 12 hours or until the person clocks out. Then the person should be able to move freely between the terminals.
I’m smiling a little bit at the history you reminded me of. https://en.wikipedia.org/wiki/Sun_Ray These units had smart card-based authentication and transparently portable sessions. You walk away from a terminal and take your card with you, plug it into a new terminal, and carry on working on whatever you were doing. Virtually zero friction other than network latency sometimes making the UX a bit laggy. In practice it worked really nicely the vast majority of the time.
I used these briefly in Bristol in 2001 through a friend from the Bristol and Bath Linux Users Group, it was really impressive, using tmux sometimes reminds me of playing with those Sun boxes.
Apologies that it took me a while to reply. How does web-based make sessions portable? It seems to me, without having put a ton of though into it, that being browser-based makes portability significantly more challenging. You can’t use a shared client environment (i.e. you need a separate Windows/Linux/OSX/whatever account for each user) and you’ll still have to manage security within the confines of a browser environment. I agree that it makes it really easy to access content from everywhere, but you’re also dealing with multi-layer authentication (Windows login, browser login, cookie expiration times, etc).
With a SunRay you could pull your smartcard, walk to the other end of the building, put your smartcard in, and have everything magically pop back up exactly how you’d left it.
Ah, right. I was assuming that at this point the software stack runs whole in the browser and the workstations would be basically kiosks.
Some equipment has a dedicated computer (radiology basically), but since there usually are only so many operators and the software might not even support multiple system users, shared OS session would probably work fine in practice.
But yeah, we might just as well just launch a container on a shared server for every user and let them run a lightweight desktop over spice.
#include <stdio.h>
int main(void) {
unsigned long long
t = 1,
f = 2,
u = 9,
o = 19,
s = 25,
c = 4217,
e = 7171939,
a = 1762195240847,
b = 399833425042215739,
x[] = { o * b , f * u * s * c * a , t * e };
puts((char *)x);
}
But in 6 years only one human has gotten my email address from this and sent me a message, so it appears to work a little too well.
The comma is a sequence point, so the evaluation order is well defined, unsigned overflow is well defined, and I thought accessing the result of a cast to char * was well defined too, right?
Oh, my bad. Still, the implementation-defined behavior makes the snippet ambiguous. And I forgot to mention that the size of long long is also implementation-defined.
Edit: Actually, on a platform where both char and long long are 64 bits, wouldn’t this call puts on a string without a zero terminator and therefore cause undefined behavior?
I’ve finally brought the bulk of my books to Denmark after moving here a few months ago, so I’m putting together some IKEA shelves, and if the weather isn’t woeful I’ll make a trip to pick up some houseplants. Hopefully it’ll all help my apartment feel a little more lived in.
I’m going to try get some writing done; I have a draft of an osdev and retrocomputing related article I want to finish.
I moved from Dublin to Aarhus. So it’s slightly colder on average, but also slightly less wet (not that you’d think so after October here), so I think I’ve a good idea of what I can expect from November (and you’re probably correct with “a bold hope” haha).
I think that the points around making things serializable is interesting and valid. But I think that it’s important to acknowledge that the rust type system does in fact involve trade-offs. Self referential structures aren’t just “object soup”. They are a natural and fundamental way of modeling information and when a language makes it awkward to express those relationships it has given up something. Rust gave it up in return for memory safety without GC which is a valid tradeoff.
They are a natural and fundamental way of modeling information and when a language makes it awkward to express those relationships it has given up something.
One could say the same thing about goto. Just s/information/code/.
The progress in programming comes from increasing global composability by eliminating local expressiveness that interferes with it.
In my experience forcing developers to express information relationships in a bigger framework of tree of ownership leads to substantially better software.
I would argue that the comparison to goto is not completely apt — the two tradeoffs offer very different benefits. In the goto case you barely lose anything (almost every usage can be converted to structured control-flow), while your reasoning abilities are greatly increased. It is no accident Dijkstra’s paper became so widespread.
Not having a GC is a valid tradeoff in some (niche) use cases, as mentioned, but it is very far from a ubiquitous one. There is nothing fundamentally bad about self-referential data, in fact I think the vast majority of existing software would be vastly more complicated without them.
almost every usage can be converted to structured control-flow
Almost every use of self-referencing can be expressed as indexes, with increased safety, ability to reason about and composability.
Not having a GC is a valid tradeoff in some (niche) use cases, as mentioned, but it is very far from a ubiquitous one.
Not having a GC makes the code have no runtime, which increases it’s composability by making it potentially usable in everything. No one will use Java code from Python, or JS, etc. while almost every language can use Rust because it doesn’t have a runtime.
It also allows practical aliasing model which enables efficient memory safety and concurrency.
Small concession, big wins.
There is nothing fundamentally bad about self-referential data
“There’s nothing fundamentally bad about goto.” Quite a few elegant goto patterns in C, that I don’t miss, but used to employ.
Dropping it too was a relatively minor concession for ability to reason and compose larger software.
Almost every use of self-referencing can be expressed as indexes, with increased safety
Moving references out of the type system does not increase safety. If you have a pointer to an object of type T, and the type system knows that it is a T*, then static and dynamic checkers (and run-time enforcement techniques) can ensure that your T* does not outlive the underlying T, or that it is explicitly invalidated if it does. In contrast, if you refer to objects via an index then you are hiding this relationship from the type system. You are doing manual memory management again. It’s not quite as bad as C, because a dangling index will point to a bounds-checked object and so will either be detectably invalid ot point to some T, but there is nothing that guarantees that it points to the T that you think it does.
If you delete an object from a graph, you now have a free slot in your index table. Do you reuse that? Now you have a mechanism for introducing use-after-free bugs. A dangling index to that slot will refer to the wrong object. Alternatively, do you avoid reusing the slot? Now you have a memory leak and have increasingly sparse data structures that become progressively more inefficient to traverse.
This kind of model works fairly well for things like the CFG in the Rust compiler, because those are short-lived structures and so it’s fine to simply burn slots for deallocated basic blocks and free the whole thing at the end (though it’s still more effort than it would be to use explicit traced references or weak references in a reference-counted environment). They work very poorly for data structures that change shape over time.
Not having a GC makes the code have no runtime, which increases it’s composability by making it potentially usable in everything. No one will use Java code from Python, or JS, etc. while almost every language can use Rust because it doesn’t have a runtime.
People happily used Objective-C from Ruby, Python, and Java (and even Haskell and Ocaml) via mechanically generated FFI, and from C/C++ via hand-written FFI. Having a runtime didn’t prevent this. Having reference counting in the runtime made it easy to build bridges that automatically surfaced Objective-C objects in these other languages. Objective-C’s perception as an Apple language was the largest barrier to this.
It also allows practical aliasing model which enables efficient memory safety and concurrency.
Except that now you’re advocating for design patterns that throw away temporal safety. Type-pooled allocators in C/C++ have the same level of temporal safety as your proposal. They’re much better than nothing (which is why Apple now uses them in the XNU kernel) but they’re a long way away from ideal.
“There’s nothing fundamentally bad about goto.” Quite a few elegant goto patterns in C, that I don’t miss, but used to employ.
C’s goto is absolutely nothing like the goto that existed prior to structured programming. It has well-defined behaviour with respect to lexically scoped objects (you can’t branch over a local variable’s declaration, for example) and it is not permitted to branch to labels in a different function.
C’s goto is absolutely nothing like the goto that existed prior to structured programming. It has well-defined behaviour with respect to lexically scoped objects (you can’t branch over a local variable’s declaration, for example) and it is not permitted to branch to labels in a different function.
You can branch over a local variable’s declaration. The following is legal:
goto L1; // legal
int x = 42;
L1:
goto L2; // legal
{
double y = 3.1419;
L2:
}
The only restriction that the C standard places on goto’s are that (1) the jump must be within the same function, and that (2) you cannot jump into a scope containing a variable-length array. (C23 6.8.6.1 “The goto statement”, draft N3096), so the following would be invalid:
s/increased/decreased. Even naive C have better tools to actually debug “dangling pointer” problems — valgrind will cry about it, but there is nothing to do if you store an index, but the associated value was replaced in the meanwhile. Let alone a managed memory language.
And come on, increased ability to reason about? What is more clear, that my Node has a parent: Node, or if it has a parent: u32? In a managed language the former has only a single invalid value (or if the language is null-aware, none), the latter has potentially 2^32, let alone the case of dangling pointers.
Not having a GC makes the code have no runtime, which increases it’s composability
Yes, which is a niche use case, which might be worth it depending on context, say, writing a specialized library. Most people want to write applications, though, where this concern is negligible, and can be circumvented by IPC.
It also allows practical aliasing model which enables efficient memory safety and concurrency
So you think a checked array access will have better optimizations by the compiler than pointers themselves? Especially that one of rust’s strong points is that it can actually “use the restrict keyword” properly, thanks to the borrow checker? This is an unfounded statement on your part.
There is definitely value in restricting the “degree of freedom” of code, see types, but as most things, it is always a tradeoff. We don’t generally use regular programs, even though they are much easier to reason about than Turing-complete ones.
And come on, increased ability to reason about? What is more clear, that my Node has a parent: Node, or if it has a parent: u32?
I think you misunderstood this. The idea is to not to just replace the pointer with an index in the object, but to implement another vector of data in the main struct:
parents: Vec<(PersonId, PersonId)>
This works if your data is immutable, so you can sort it and use ranges and binary search. If you mutate your data during runtime, using a map works better:
parents: HashMap<PersonId, PersonId>
Although if the data is kept in memory for a long time, and mutates during its lifetime, I’d consider using indices like this a bit before jumping into writing the code.
Again, you have all your data in flat structures and you do not have any nesting of data anywhere. It makes a lot of sense e.g. when modeling databases to an ORM, or when writing a compiler.
Again, you have all your data in flat structures and you do not have any nesting of data anywhere. It makes a lot of sense e.g. when modeling databases to an ORM, or when writing a compiler.
That’s an important point. If one is working with graph data, chances are high that it will need to be persisted anyway, in which case there’s barely any technology that will allow persisting the “object soup” natively (they were attempt of “object oriented databases”, but obviously it’s a terrible idea so it didn’t work out).
So now the “ease” of “object soup” has an additional difficulty of dealing with the conversion back and forth to the form I’d advocate for in the first place as clearly better.
tools to actually debug “dangling pointer” problems
It’s not a pointer. It can’t be accidentally deferenced forgetting to check if it’s still valid. Sure the logical coherence of the larger data-structure needs to be taken care of, but it’s not unlike GCed languages. In a managed language if you point a child at a new parent, but forget to update the old parent, are you going to get great results? If anything, all the “invalid values” (often including ‘generations’ in a slotmap) can actually crash the code in an orderly way and help spot the issue, instead of letting it “mostly work”.
or if it has a parent: u32
It’s parent: NodeId.
Yes, which is a niche use case, which might be worth it depending on context, say, writing a specialized library.
That’s no a niche use-case. The ubiquity of C and C++ despite their huge risks, is a result of being able to reuse them.
So you think a checked array access will have better optimizations by the compiler than pointers themselves?
Who cares. Rust does have references, including shared ones. You only really need to use indexes when you’re already working with a graph, and children need to point at parents, etc. and the whole thing is non-trivial already, and the performance overhead of “non-native-references” is akin to the cost of a JOIN in a database .
Normal relationships that fit a DAG (“tree of onwership”) are all one needs 99% of the time anyway.
Worry about all the memory barriers, extra memory usage and cache invalidation caused by a GCed-runtime that is slowing down the normal GCed code at all times, to make the GC work, not a 1% case in Rust, that is often fast enough.
There is nothing fundamentally bad about self-referential data, in fact I think the vast majority of existing software would be vastly more complicated without them.
It wasn’t obvious to me, but the reasons for Rust not supporting self-referential structs can be said to be historical, not strictly technical (see 1 and 2).
I find that restriction quite limiting in the kind of APIs I’m able to design. I often want to encapsulate two object in a single struct, where one fields references the other (e.g. rusqlite::Connection and rusqlite::Transaction). You can’t do that in current Rust because:
Fields of structs are dropped in declaration order.
Transferring ownership (i.e. moving) of a struct needs to be equivalent to memcpy().
In [2], withoutboats lays out a proposal for a Move trait that would invert the status quo: structs would be immovable by default and movable as an opt-in. Immovable types could be naturally self-referential and even structs that implement Move could manually instruct the compiler how to copy themselves, thereby solving (2). As for (1), it’s a simple convention that would need to be relaxed.
The biggest reason in my understanding why this is not happening is backward compatibility.
You’re falling into the trap of believing that there is a single path of progress towards some kind of ideal, which every language is moving towards or away from. Ideas in languages evolve the same way organisms do: In response to localised pressures, towards whatever gives them an advantage in that domain.
So to your point: goto has indeed mostly “evolved” out of most programmers workflow and languages. But it is still alive and well in certain contexts, for example kernel development.
Totally agreed, there are major tradeoffs here. One that’s gotten a lot of discussion recently: The designers of async/await had to go through an enormous amount of trouble to fit futures into the language, since they’re fundamentally self-referential types. They settled on the Pin API, which is a leading contender for the trickiest module in the whole standard library.
Another interesting one: The small-string-optimization layout in GCC’s std::string is self-referential. Rust’s String doesn’t do SSO in general, but if it did, it would have to do something like what Clang does rather than what GCC does.
I don’t think SSO is typically self-referential in the sense that is not allowed in rust. Typically SSO looks like (but more optimized to use every available byte and not hardcoded to 64 but platforms).
There is no explicit reference to the inline buffer (as that would waste a word of SSO space). Rust can handle this perfectly fine as a method like as_u8 can borrow a reference to the buffer.
Oops, I skipped over what’s probably the most important implementation detail: Clang’s string::data()has a branch to handle the small string case, but GCC’s string::data()does not.
Well said. I’m as big a member of the Rust Evangelism Strike Force as anyone, but it’s very frustrating to see so many people try to spin things that are clearly flaws/shortcomings/whatever as somehow actually good. It’s not just Rust fans, of course–I see it in all of the various programming language forums/subreddits/etc. It’s okay to admit that something sucks about the thing you like!
Using arenas in Rust is sometimes the best approach because of performance reasons, but frequently arenas are just awkward, bug prone (must keep your indexes in sync with arena updates), workarounds for Rust’s borrow rules.
In the past seven years I’ve been working with Rust, using the approach with indices described in the article has been an absolutely awesome API two times in quite large projects. It’s a very nice abstraction when you need to parse a file of code once and then refer it later. Eventually you’ll have one allocated string sitting in the bottom, tokenized in the next layer and then building a database of typed token IDs either in vectors or maps on top of the AST. The public API is built on top of the flat vectors of IDs which can then take a reference directly from the string which represents the code.
It is very efficient and extremely nice API to use. It has been causing almost zero crashes and has accelerated all development in the corresponding code bases. It’s not that I’m trying to evangelize Rust, but this way of modeling the data works very nicely for a few hard problems I’ve been needing to solve in my career.
Sure. And I guess I shouldn’t have specified “because of performance reasons”. Rather, my point was that sometimes the arena approach is actually the best approach to the problem, but sometimes it’s used because it’s the only option.
Phrased a different way: if you were writing your code in a GC’d ref-happy language and you’d still choose the arena approach, great. But if you wouldn’t choose the arena approach in the hypothetical language, then it seems hard to argue that it’s better than “object soup” or whatever. It’s like the infamous tree/graph data structures conundrum. Using trees and graphs to model data relationships is very often exactly what we want. The fact that I see so many (frankly, newbie) Rust fans try to tell people that they really don’t want/need trees is crazy.
By this, I mean the fact that MS is paying a team with Guido to speed up the language, and that we were all quite giddy when they announced that the target was a 50% gain at each release for the next 5 ones. We saw that coming, but still.
This always seemed a little optimistic to me.
In my last project, we had sum(itertools.starmap(operator.mul, zip(p, q))) in a lot of places, then we created a function for it, then we realized we had bugs because of length issues and added frenetically strict=True everywhere.
It always depends somewhat on your baseline. HHVM managed to more than double performance each year for multiple years because PHP was very slow. Starting from C, you only manage that kind of speedup when you’re doing autovectorisation for the first time or similar. A number of Python implementations are a lot faster than CPython, doing things like type-specialisation in the AST, better dispatch in the interpreter, and JITing. 50% speedup over five releases is a 750% speedup overall and that’s less than the speedup you get going from a decent bytecode interpreter to a modest JIT in most Smalltalk-like languages. It seems quite feasible for CPython to get that kind of speedup.
As far as I know, today, Rust is the only programming language (other than C) able to create efficient WebAssembly (wasm) libraries: no need to embed a big runtime or to use an over-complex compilation toolchain.
Zig is pretty good for this too, with support for wasm and wasi targets in the compiler and no large runtime needed. Just compile your library with -target wasm32-freestanding or -target was32-wasi and you’re good to go.
The language is pre-1.0 though, so it doesn’t have an advantage over rust there.
I find it a little odd the recommendation for Rust as a C or assembly replacement (even more on the latter). I 100% see Rust as a C++ replacement, in all the ways it could mean.
Otherwise, yeah, like others have mentioned the author is on the nose. I stopped using Rust after about 3 years when I realized the complexity is out the wazoo. The lack of a standard, the massive toolchain to build Rust itself, the massive man hours to create a Rust compiler, are just way too much. In comparison, look at the amount of C compilers over the years.
It’s this exact reason I switched to Zig. If you want 100% memory safety, then Go (or any GC language) really is a great choice.
Rust is a great C replacement. C features and design patterns map almost 1:1 to Rust. Writing of wrappers for C libraries is straightforward. There’s very accurate c2rust.
OTOH Rust has plenty of impedance mismatches with C++. It doesn’t have enough OOP features to translate C++ interfaces to usable Rust. Can’t emulate copy constructors. Doesn’t have placement or RVO. Templates vs generics are all different except the superficial choice of angle brackets for their syntax.
I know some people equate it like: C is small, C++ is big, Rust is big, therefore Rust == C++, but that is a false equivalence. They have their complexity for different reasons, and there’s only a small overlap in their features.
Rust is a great C replacement. C features and design patterns map almost 1:1 to Rust. Writing of wrappers for C libraries is straightforward. There’s very accurate c2rust.
Only if the library you’re wrapping’s model of ownership lines up nicely with rust’s ideas cleanly, which is not always the case.
A good write-up on this is the article Giving up on wlroots-rs by the developer behind the wlroots rust wrapper wlroots-rs, as well as the Way Cooler compositor.
This is one famous example, but OTOH there are thousands of crates successfully wrapping C dependencies. A lot of libraries have boring init + free combos, sometimes an Rc or Cow equivalent. Some need an artisanal smart pointer.
Rust is a C++ replacement, not C. Its concept of references and lifetimes is lifted from C++, and those concepts do not really exist in C (and where they do they are described differently). Rust has ML’s type system, but that doesn’t make it a child of C and not C++.
Rust took move semantics from C++, but I don’t think “lifted a couple of features from” means it’s a suitable replacement for the whole language.
I look at this from perspective of Liskov’s substitutability. Rust is close to having a superset of C features, but only a subset of C++ features, and seemingly common features (moves, references, dtors, overloading) have conflicting semantics.
Ownership exists in C, because that’s inherent to having heap allocations and free that must be called exactly once. References and lifetimes also exist when you can have pointers to the stack and non-GC pointers to interiors of allocations. These things exist even if C doesn’t use explicit terminology for them.
C doesn’t enforce lifetimes via the type system, but via threats of UB, and runtime crashes/corruption. By analogy, dynamically typed languages can’t prevent type errors at compile time, but that doesn’t mean they don’t have types or type errors.
There’s also another way to look at it, putting language theory aside. Rust has the least amount of success in GUIs and game development, which happen to be C++’s strong domains. GTK in C has decent Rust bindings. Qt is unusable. Rust was accepted in Linux, which famously rejected C++.
That’s really interesting to hear. Do you have a resource I (and readers) could check out regarding Rust-C equivalence? :o
And yes, your last paragraph is more or less the reasoning. Well that and I experienced that Rust has advanced features not like C++, but are powerful as the ones in C++ (unlike C which lacks them).
I wish projects like this would clearly say why they exist and why anyone might care to use/contribute/follow them.
My best guess from a quick search is that the authors either made it just for fun or they made it because they wanted ZFS features but don’t like the license conflict or the btrfs code.
The proposed value for Linux users seems to be “btrfs but better”.
If you’re interested in the origin, bcache itself is a stable, transparent way to split hot/cold data between multiple fast/slow drives through caching. The author at some point said effectively - you know, that’s halfway to a decent filesystem with good features, so let’s make it happen.
Note that the “Status” section is woefully out-of-date. Everything on that list is complete and stable, except for erasure coding (which is not yet stable). I don’t know about the status of DEX/PMEM.
Honestly, after fairly extensive experience with using Btrfs during 4 years of running openSUSE as my daily driver, the headline on the website tells you what you need to know:
“The COW filesystem for Linux that won’t eat your data”.
That’s it. That’s the story.
Btrfs is the most featureful FS in the Linux kernel but it is not reliable. I know lots of people are in deep denial about this, but it is not.
Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck successfully repair all types of filesystem corruption. E.g. some other software or hardware bugs can fatally damage a volume.
WARNING: Using ‘–repair’ can further damage a filesystem instead of helping if it can’t fix your particular issue.
It is extremely important that you ensure a backup has been created before invoking ‘–repair’. If any doubt open a support request first, before attempting a repair. Use this flag at your own risk.
Btrfs is not trustworthy. If you fill the volume, it will corrupt. When it is corrupted, it cannot be fixed and you will lose the contents.
OpenSUSE uses a tool called “Snapper” to automatically take snapshots before software installation. This means progressively filling the root filesystem. Unless you manually manage this and manually prune them, it will fill up, and when it fills up, it is game over.
I have been using Linux since kernel 1.0 or so in 1995 (I suspect well over one hundred distros now, if you count major versions) and in the 20th century it was routine for Linux boxes to self-destruct. If the power went at a bad moment, boom, dead.
21st century Linux is different. It is solid now.
I used openSUSE for 4 years, and in that time, I had to format and reinstall my OS about half a dozen times or more, because Btrfs self-destructed. This was not a one-off experience.
Fedora doesn’t have any snapshotting functionality, and so it’s not using Btrfs for the thing that makes Btrfs special. It’s like a moped enjoying perfect engine reliability because you pedal everywhere and never start the engine.
(Why does it use it then? I think this is why: Fedora leans heavily on Flatpak and is increasing that. Flatpak is extremely inefficient and involves a lot of file duplication. Btrfs can compress that. That means it doesn’t look so bad.)
Btrfs self-corrupts on full volumes and it does not have a working repair tool.
That is not acceptable. Therefore Btrfs is not acceptable, and that’s without considering the issues around its RAID handling and so on.
Yeah. BTRFS is kind of shit. I was recently considering adding more storage to my system and found out that I could select between two allocation policies if my disks were mismatched (or if I added one without balancing). I can use the single profile and it will only write to the bigger disk until it is full (BTRFS’ allocation policy is most free space) or I can pick RAID0 and it will stripe across all disks until eventually it is only writing to a single disks. Both cases hotspotting.
I don’t get the complaint about --repair. This is the same situation as fsck in all filesystems where it exists. If you have a corrupted filesystem, the best a tool can do is make clever guesses about the way to recover. Hopefully the guesses will be right, but they may be wrong. They’re being explicit about it.
Like I said, I have fairly extensive, repeated, personal experience of this.
Fsck is simple and it usually works. I have used it on ext2, ext3, ext4, hfs+, and probably on reiserfs, xfs, and others.
Usually it fixes some damage, maybe small, and afterwards your filesystem works again. It’s as safe or safer than CHKDSK on the DOS/Windows family. It’s part of my routine maintenance.
I have never ever seen Btrfs repair successfully fix any form of damage. None at all, not even once.
What I have seen is a healthy working volume, which I only checked out of curiosity, destroyed by it.
Don’t be misled. What the warning means is “DO NOT RUN THIS”. It will destroy the volume.
There is no way to repair a damaged Btrfs volume, even if the damage is trivial.
I’ve installed openSUSE with Btrfs a few years ago. It worked kind of well, until I’ve tried Android development; the VM was unusable (stuttering, low fps, low performance) if the VM image was stored on a Btrfs partition. When I moved the VM to an ext4 partition, everything was smooth again. I’ve tried to search for the issue on various forums, but couldn’t find any fix for this, so I’ve just resigned from openSUSE and Btrfs altogether, and didn’t look back on it ever again. I haven’t experienced any data losses but from what you write, this was just a matter of time. So it’s good I’ve resigned.
One of the benefits over btrfs today is that bcachefs supports native encryption like ZFS does, meaning you don’t need to juggle LUKS + btrfs.
I think for the next while at least btrfs or ZFS will still be preferred but I suspect that bcachefs will be good enough that the (admittedly minor) effort of ensuring OpenZFS on Linux is working such dealing with lack of packages or kernel upgrades breaking things (hello ArchLinux dkms) will make it a success since it’s natively integrated into the kernel.
A big adopting point would be if container tooling such as containerd start to natively support bcachefs the way btrfs and ZFS are.
I think bcachefs is significantly better than ZFS or BTRFS due to flexibility. You can pick storage configuration per-directory, chance storage configuration at any time, stripe sizes change automatically as you add disks and mismatched disks are not a problem.
Overal bcachefs feels like you just add your disks, add your files and it more or less does the best possible thing given your placement constraints. ZFS placement is quite rigid and specified upfront and BTRFS placement is way too stupidly simple.
The only two platforms known to actually support zfs are FreeBSD and Linux. No other filesystem than FAT has a good chance at being writable from every major OS
I can know without further confirmation that the files at the other end are exactly what I was sending, and the transit was minimal. I also can then roll back on the remote server to before that send happened, in case I want to check something that was deleted during that sync.
I’m not sure there are “other tools”. Traditionally filesystems have dump/restore for backup/restore, there’s rsync for incremental sync and restic/borg/duplicity support encryption and authentication.
Zfs send/receive works zfs snapshots and native encryption to form a rich, yet simple, solution for sync and backup. It works well ssh for network transport - or eg USB media for local backup. It supports remote mounting (mount backup on backup host) if wanted.
Like most things zfs, it’s the combination of features that creates a powerful emergent whole.
ZFS […] If you have 2 disks mirrored you can’t add a 3rd disk and still have all data stored twice.
That isn’t currently the case anymore, you can add a third disk to a mirror setup and it’ll resilver just fine to have triple copies of everything.
Though the RAID still applies (2.2 adds DRAID, which does let you expand IIRC, at massive complexity cost and has issues still, RAID Expansion is still in the pipes).
Did you make a typo? The requirement is 3 disks and 2x redundancy. I’ll admit that I didn’t look into the configuration too closely but this seemed difficult to do with ZFS. Especially if you were starting from 2 disks and wanted to live migrate to 3.
My understanding is that at the end of the day you configure your performance/redundancy into a virtual block device. You then put a more-or-less single-disk filesystem on top of that. (It is smarter than your average single-disk filesystem because it is aware of the topology but IIUC the behaviour is basically the same). You can grow by adding more disks to the virtual block device but you don’t really have much option to change the existing devices in the pool. (As you said there are some limited options such as 2 disks replicated can become N disks replicated).
bcachefs (and BTRFS for this case) are fairly flexible about this. They see that you want 2 copies and they pick two disks and put the data there. Adding more disks gives them more options to pick from. With bcachefs you can get even more flexible and say “everything in this directory shall have 4 copies” and it will replicate that data more times (well it won’t proactively replicate existing data IIUC, it will replicate new writes or on a balance operation) even if that filesystem previously had no configuration with 4x redundancy.
ZFS’ internal structures are messy, but essentially there is two types of vdev’s; RAID and Mirror.
A non-mirror vdev (ie, just a plain disk, no redundancy), is simply a special case of a RAID1 with no partners (though internally it simply presents the device directly as pool member).
Mirror vdevs can be brought up and down in redundancy by adding or removing disks. If your pool is only mirror disks, you can even split it in half and have two pools with identical data. You can also remove and add mirror vdevs at your pleasure. Even convert single-disks into a mirror configuration.
RAID vdevs are much more strict. They cannot be expanded once created, they can’t be split and cannot be removed from a pool. This is due to the internal block pointers being in a special format to handle the RAID stripes.
The RAID Expansion work doesn’t even change that, it instead allows you to rewrite a RAID vdev in a way where it has more members than before. The old data will still only exist at the previous redundancy level (ie, same parity-data ratio) and only new data will benefit from the better parity to data ratio.
The objective stated was “a filesystem that has 3 disks, and all data has 2 replicas”. Number of data replicas < number of disks. This saves 50% of your storage space if your failure budget only requires 2 replicas.
I apologize for that massive delay, but you can set this up too, though I think OP meant “2 replicas + data”. But the ZFS solution is to use copies=2 and have no mirrors. In that case, ZFS stores two copies of data on different disks, as long as this is possible. Though the caution here is that ZFS will not warn you if the copies have to be put on the same disk and existing data is not replicated. And ZFS does warn that in case of a single disk failure, it may loose an unspecified write (ie, non-replicated write will be rolled back if the only copy was on the dead disk).
You would have to ask them, but a bunch of people do complain about it.
Now that this comment the thread has helped me learn more about bcachefs, I don’t think the licensing was a huge deal. The bigger motivations are: bcachefs devs think they can do much better than ZFS or BTRFS; and also ZFS will never be first class on Linux (because Torvalds don’t want it in the kernel, etc)
Eh, Torvalds doesn’t ship binaries. It’s companies like RedHat/IBM, Amazon, Google, Samsung and Microsoft that ship kernel binaries and wouldn’t want legal exposure.
Canonical paid for & published a legal opinion and shipped binaries including ZFS for quite a while.
OpenZFS not being in the Linux kernel has the social effect that the kernel devs don’t maintain compatibility with it, which has resulted in occasional breakages, which puts a maintenance burden on distributors (which is maybe why Canonical are dropping/mothballing their ZFS interface code?).
Canonical, a small UK company, is not the same scale of legal target as big US companies like IBM, Google, Microsoft and Amazon. Some of them won’t even ship GPLv3 code let alone mixed GPLv2 & CDDL. I am not a lawyer, but I’ve been close enough to corporate lawyers IP discussions to expect that they prioritize covering their asses from huge law suits over enabling a few long tail filesystem features (from their perspective at least)…
Their value proposition is btrfs or ZFS but better, and especially faster. They think they can achieve it by using a simpler architecture inspired by databases.
The proposed value for Linux users seems to be “btrfs but better”.
That’s my rough understanding as well. Though I don’t think it’s supposed to be a direct clone of everything btrfs does — more that there’s been lots of promising filesystem ideas since ext3/4, and btrfs and bcachefs are drawing from that same pool.
I’m not 100% up to speed on why btrfs failed to take off. Is it just that it had issues with RAID not working/being unsafe?
As a relatively casual observer, my understanding is that:
Even in server and workstation applications, most usecases don’t need the advanced features of btrfs or it’s equivalent in ZFS, making the overheads not worthwhile for some users, and in some devices (embedded, phones, etc) it was not feasible as an EXT4 replacement.
COW filesystems tend to impose IO overheads that make them undesirable
Some of btrfs’s flagship features excepting COW can be replicated by existing features available to Linux users; XFS permits differential snapshots incremental backup of filesystems which can approximate snapshotting, and LVM2 allows existing filesystems to get features like block device spanning and online partition resizing.
Red Hat was the big driver of btrfs adoption, but they abruptly decided to abandon support for btrfs (even as an optional OS) in RHEL starting from RHEL 8, and Fedora consequently only uses btrfs on Workstation, not Server. My understanding is they were sponsoring a lot of the btrfs work, too.
For those who truly did need all of the options btrfs provided, ZFS provided more, didn’t have a reputation for corruption and unstability that btrfs had (probably unfairly after a few years), and had a lot more materials and documentation on management and deployment.
Red Hat in the end decided to create Stratis, a userland program which replicates some btrfs functionality as a management layer over XFS and LVM2, though I’m not sure how widely this is adopted relative to btrfs.
didn’t have a reputation for corruption and unstability that btrfs had (probably unfairly after a few years)
In my experience this reputation is not unfair at all when it comes to BTRFS’ advanced features. I’ve been running BTRFS in RAID5/6 from 2016 to beginning of 2023 and have experienced data corruption roughly every year in that period. Can’t complain though, BTRFS tools all warn you about RAID5/6 being completely broken every time you run them.
I don’t get it. Your setup was unsupported and documented as such with warnings. Why do you think it’s fair to say it’s unstable?
It’s like randomly moving some files in the system directories after clicking “yes, I know this will cause issues” and saying the system is possible to corrupt and unstable… You’re really on your own in that situation.
Which is a pretty good demonstration of why one might want to build a new filesystem vs btrfs… IMO RAID5 is a pretty fundamental feature for a modern filesystem, both ZFS and Linux md have had it working fine since forever, and and it’s been well-known to be entirely broken in btrfs for… what, seven years or more?
Coincidentally, the author apparently started work on bcachefs in 2015.
I don’t get it. Your setup was unsupported and documented as such with warnings. Why do you think it’s fair to say it’s unstable?
Because it’s unstable regardless of whether the unstableness is documented or not. Why would it be unfair to say something that is factually true? (This is a genuine question, I am wondering if I am lacking a subtlety in my understanding of the English language.)
If you’re doing something explicitly unsupported, then it shouldn’t affect how the system is described. It’s like spraying your computer with water or plugging it into 300V and saying it’s unstable because it crashes - the user guide did tell you not to do that. My phone has an option to enable some experimental GPU features which actually crash when starting games - I can’t say my phone is unstable, just because I explicitly enabled that myself. It’s a option we’re not preventing from taking, but doing those things only says something about us, not about the system.
I can’t say my phone is unstable, just because I explicitly enabled that myself.
You can’t say the phone is unstable, but you can say it’s unstable when you enable experimental GPU features, the same way I’m not saying BTRFS is unstable, I’m saying BTRFS is unstable when using is advanced features. I really don’t understand where our disconnect comes from.
Because it’s not unstable when using advanced features. Advanced features like compression, snapshots, many raid levels works just fine. It’s unstable when using experimental unstable features. And pointing that out is not meaningful, because it’s the same for an experimental unstable feature of any project by definition.
In the ActivityPub community I encountered a dev that treats URLs as opaque strings as it pertains to internal things to his application.
I can’t fathom trying to decouple the logic that paths, query params and everything else allows in favour of something so bland as a string identifier.
That’s not far off the right thing to do. A URL is three parts:
The schema
://
The stuff whose interpretation depends on the schema
Often, the best thing to do is treat the whole thing as an opaque token and pass it onto a different layer that understand the schema and can do the right thing.
Huge “it depends”: data:, jar:, blob:, and about: also do not require a double slash. But chrome: and resource: mostly do. I guess it’s fair to say that everything after the colon is specific to the URL scheme.
If anything, I wonder why the double slash was introduced at all by some schemes. Even in file: URLs it seems like a single slash should suffice and in http: it doesn’t add any value at all AFAICT. Does anyone happen to know the rationale behind it?
“The formats and protocols were designed to look as much like the existing ones as possible,” he explains, saying that HTTP was designed to look like NNTP, or Network News Transfer Protocol, which was used for Internet newsgroups. “The aim was for people who worked with the protocols to look at them and say: ‘Oh, yeah, I see what’s going on here.”
…the double slash at the front each web address came from a file system for a computer workstation called the Apollo/Domain. “The double slashes were there because, on some computer systems, that was already used to mean: ‘We’re going outside the computer now.’ The single slash was for the local file system. The double slash was for the outside.”
I also remember reading a quote that Tim just liked the look of it, but I can’t find any source for that, so maybe it’s a false memory or I heard it in person at a lecture he gave.
Yes, in Domain, local paths were Unix style /like/this, remote systems paths were like //host/usr/bin/whatever, and you could specify the protocol like protocol://whateverhost/file/path.
If HTTP(S) URLs didn’t have a double slash, and were of the form a http:server/path or https:server/path, then a colon would also suffice for that; i.e. :server/path could be treated as protocol relative.
Double slash means it’s a “hierarchical URI” which implies certain things about the structure (host, path) vs “non hierarchical” which is just schemeSpecificPart
That’s even more counterexamples, thanks! I’d previously used :// to try to spot things that are probably URLs in text, and few of those are likely to appear in human-authored text, but it’s quite depressing that there are so many exceptions to a fairly simple rule.
Actually, the // is part of the “stuff” as well. See mailto schema URIs
Now, if we use the concept of URLs as being web addresses, all the stuff has a known interpretation in the http schema, just different naming as highlighted (but also maybe different interpretation in an application, like using fragment to define an anchor or a path in an SPA website)
Ugh, that makes them even more horrible than I thought. I’d never quite thought of mailto as a URL schema but it does seem to think it’s one. And it’s decades too late to fix it.
mailto isn’t the only one that does this, it’s common in URI schemes! a few off the top of my head: magnet:, turn:, javascript:, view-source: so it’s not really something to “fix” but just to learn i think
Strictly speaking the // denotes the start of what the URL specification calls an “authority” (i.e., a domain name).
If the thing that comes after the : in a specific URI scheme isn’t a domain name, // probably shouldn’t be there.
There are some badly designed URI schemes whose designers did not appreciate this subtlety and erroneously require a // followed by something that isn’t an authority/domain name.
While the // may seem unnecessary today it actually does relate to one feature of relative URLs, namely protocol-relative addressing (e.g. <script src="//example.com/foo.js"/>). Though I suppose you could still have that syntax work even without having a // in absolute URLs.
There’s a little bit more to the authority part than just a domain name. The full syntax is,
user:pass@host:port
The login part was never used much and is mostly deprecated now, for reasons that should be clear(text). The host part can be a network address or hostname; the network address does not have to be an IP address (it can be like [weirdnet;…]) and the hostname does not have to be a domain name (it might come from hosts.txt or netbios, etc.).
To expand on the above for the benefit of others: after the use of colons in IPv6 addresses led to the need to wrap IPv6 addresses in brackets for use in URLs, a syntax was designed for other network addresses “just in case”. IPv7 does not seem like a very likely possibility, but if it were to happen, the syntax would be https://[v7.FOO]//.
I wonder if non-integer ports are also feasible? I could also imagine some implementations supporting /etc/services service name lookup in the port field. Then there’s UNIX domain sockets, something I’ve idly contemplated extending URL schemes to support. Maybe https://[UNIX./var/lib/foo]/.
As regards the user:pass@ thing, it is interesting to note that URI schemes divide into having two completely different uses for this kind of syntax. In HTTP it is for specifying the credential used to access a resource. But in other schemes like mailto: or xmpp: it is for specifying a user identity being named as a resource.
Technically these are different syntaxes of course as the mailto: and xmpp: schemes do not use the // authority syntax. But it is a distinction worth contemplating if we’re going to think deeply about URI design.
You can still have arguments as path elements or query parameters too, just using template URLs like in for example OpenSearch. This way you don’t end up locked into a URL routing scheme that’s now been hardcoded into your API documentation and every client.
Where scheme needs to follow certain rules, and schemepart needs to follow other rules governed by scheme, and the details of those rules (and their combinatorial requirements) are non-trivial! If you don’t need to interrogate a URL for details, treating it as an opaque string is a galaxy-brain maneuver, IMO.
Yeah, it’s a total mess. In CHICKEN, for this reason we split the eggs for parsing URLs into two: one “generic syntax” egg which simply parses the BNF from RFC 3986 (which is the successor to the one you linked) and one “common syntax” egg which deals with the specifics of HTTP and “suchlike” schemes. The “common syntax” egg also parses out the query string into key/value pairs, further percent-decodes path components after splitting them apart (because the insanity of the URI syntax means that for some schemes, %2F is identical to /, but for others you first need to split the path components and then decode %2F into slashes “inside” each path component), etc.
The solution isn’t very intuitively appealing, but it seems to work well enough.
Wow. The rare article that speaks to my exact situation: I use Cinnamon. I hope that Cinnamon one day supports Wayland. When it does, (poof) I will have switched to Wayland. Whatever their reasons, enough people find that bothersome, or unrelatable, such that I’ve learned to just say nothing in threads about Wayland.
I was hyped for the technical merits of Wayland in the 2010’s. I was excited when GNOME added support. I don’t wish to use GNOME, but I know it might lead to momentum in Wayland support so I’m all for it. I was glad to see KDE do so as well, though I don’t wish to use KDE. I even tried sway back in 2018 or 2019 because I had good experiences in tiling wms in the 00’s.
However, the way life has played out is that I now care more about working with my computer than working on my computer. I’m less of a productivity and environment explorer than I was ten years ago. Cinnamon works for me—works fabulously, in fact—and so I’m happy to be using whatever underlying technology makes it work so well.
I quite like Firefox’s Tab Stash extension for that, where I periodically put all my deduped open tabs into an unnamed collection showing the date of creation.
I zap all my open tabs 2-3x a week. I can think of maybe one or two things I haven’t been able to find again over the past ~15 years. I’ve come to terms with lossy information retention.
I try to do it once a day. I’ve come to realise that if it’s important and worth caring about (it almost always isn’t), I’ll either remember it or be able to find it in my browser history.
That’s it. All my tabs, gone, and I never knew how to recover them. I use Firefox on Windows, and it happens to me about once or twice a year.
Edit: sometimes I get a warning about closing a window with multiple tabs opened. I believe the reason for this warning is because there is no “undo” on this: if I go through the data is gone for good.
In Firefox, the History menu should list recently closed tabs and windows. For a whole window, Ctrl/Cmd-Shift-N should revive it.
If that fails, there should be several backups of your recent sessions in your Firefox profile directory. Check the sessionstore.js file and other files in the sessionstore-backups directory.
Double-check that you have session restore enabled in Firefox settings (“Open previous windows and tabs”).
I have. It shows the history, most notably the most recently opened pages. I haven’t found anything relating to tabs there, especially what I wanted: recently closed tabs.
So, the BIOS is stored on the card itself, right? I wonder when this will be used for malware persistence; that sounds like a pretty hard thing to catch.
Is it even possible to flash the BIOS back without letting the on-GPU firmware run?
Which is fine, provided customers have the option of being able to add trusted certs so as to run their own modded binaries.
The fact that customers aren’t given the option tells you what this sort of “security” functionality is really about: manufacturer control over the devices they sell, after they’ve sold them.
If a customer can install their own certificates, then so can malware.
Having secure boot while supporting non-manufacturer software is hard, and requires a lot of engineering effort (marcan had a long diatribe about this on the arm Macs, and seemed generally positive on the design, but as I understand it all the device firmware is loaded by iboot prior to OS startup, and is apple signed top to bottom).
Obviously it doesn’t help when you have stuff along the lines of what Nvidia is apparently doing - where the bios is setting a max frequency based on the buckets Nvidia wants to sell things at :-/
If a customer can install their own certificates, then so can malware.
This is, indeed, the hard problem and it depends a lot on your threat model. For something like a graphics card, it’s generally fine to say that people with access to the card are out of scope of the threat model, because that requires opening the case and so you can have a jumper (or even just a button or switch) that allows provisioning a new certificate. That’s harder if your threat model includes someone with temporary access to the machine (for example, if someone manages to join the cleaning crew for the building - which many companies outsource - and can have 10 minutes to poke a desktop and install malware). It’s even worse when the machine is accessible only by the attacker. The last case is our threat model for confidential cloud computing: the attacker is the cloud provider and they have the computer in their data centre, the defender is the customer and they want to get strong security guarantees.
There are a few PCIe specs that help here. IDE (not to be confused with IDE, which is totally unrelated) gives you an end-to-end encrypted communication channel to a device. TDISP (ADISP in earlier versions, possibly a newer name in the most recent ones because naming is hard) lets the device authenticate an endpoint as a software entity (for example, to tell the difference between two TDX or SEV-SNP VMs on the host) and lets the code on the host get an attestation over the code in the device. This lets you defer the decision on whether you trust the device’s firmware until the device is attached (the flows here are somewhat complex because there are two levels of trust: do you trust this device to be in the machine at all, and do you trust it to be given access to your VM?).
That doesn’t help with anything that’s in the core TCB for the system though, such as CPU microcode, where you can’t run any code until it’s loaded and so the best that you can hope for is a remote attestation quote that you can verify on a different computer and decide whether to release keys to the booted system. Generally the approach here is to make it easily auditable: if the cloud vendor does something to compromise these layers of the system, either you can easily tell that you’re running a malicious version because it’s different from what everyone else is running, or they deploy the same malware to all of their customers, ensuring that enough people have standing to sue them that they will go out of business if they are actually malicious. Though any sufficiently advanced incompetence is indistinguishable from malice, so proving malice may be hard.
For local devices, typically the best solution is to provide a provisioning step that can be done by the user once. This installs a new key that is stored somewhere else (a cheap HSM like a U2F device is ideal) that must be used to sign any new key. This, at least, requires an attacker with physical access to steal the separate device. You can also do a lighweight version of this by requiring a PIN on something like a TPM that can do rate limiting on retries.
The typical design for this is to ship it read-only, writeable if you move a jumper. Chromebooks regularly had this capability, usually with an extra screw as the jumper. Is it that hard?
The same goes for other kinds of hardware tampering, though, and if someone has physical access you can’t really prevent that. The jumper seems like a pragmatic solution.
The whole point of a lot of apple’s various peering mechanisms is specifically to ensure that having physical access to the device does not provide a silent compromise of the device.
Ah, that makes sense, different threat model then. As I recall the Chromebook would always show a scary warning at boot if the boot stuff had been modified.
Any functionality which precludes me from running the firmware I want is malware.
You are free to run whatever hardware you want, but saying that people who disagree with you are wrong, and any other model is “malware” implies a fairly abusive model of view of what choice users should have.
Restrictive code signing practices are not acceptable.
The pre-code signing model was devices had their control software burned into rom, and the only way you could get bug fixes was new hardware.
“hardware” devices have been running embedded software for decades, and no one complains about not being able to install their own software on those systems.
For secure systems your options are code signing, or the firmware is burned into an unfixable, unimprovable rom.
“You’re free to run whatever hardware you want” is a fake choice, all the pragmatically useful hardware vendors behave the same.
Also, yes, we do complain about not being able to run our own software on all sorts of embedded devices in exactly the same way, this is why “blob free” linux is a holy grail. (See also, phone basebands)
If your idea of a secure system is inherently aligned with a manufacturer that has opaque motives and is subject to coercion from governments and shareholders, it’s very different from mine.
Also, yes, we do complain about not being able to run our own software on all sorts of embedded devices in exactly the same way, this is why “blob free” linux is a holy grail. (See also, phone basebands)
The old “blob” free hardware people think of still had binary blobs, they were just burned into ROMs that were just as opaque, and just as unmodifiable, but also couldn’t receive updates (fixes, features, etc).
There has not been “blob” free hardware since the 70s
Like, has this kind of behavior from vendors gotten so ingrained in people that a system with open documentation and at least source-available blobs/firmware has actually become impossible to imagine?
This is a goalpost move, the fact that it doesn’t exist doesn’t mean it can’t or shouldn’t. You didn’t ask me where to get such a thing, you asked what an alternative to not-it is.
When legal and economic problems become technical ones, why is it always the technology that has to change for the worse?
Having secure boot while supporting non-manufacturer software is hard, and requires a lot of engineering effort
Oh, sure. But it is possible; and it’s possible to do it securely. Heck, if you wanted you could do it physically - put the certs on a socketed chip or something like a Yubikey.
Obviously it doesn’t help when you have stuff along the lines of what Nvidia is apparently doing
Which is sort of my point - I don’t believe security is the primary motivation here, or there would be an option for users to use their own certs.
Not really. The issue isn’t that the card runs unsigned firmware, but that unsigned firmware can persist through reboots.
It’d be absolutely fine if the card let you upload custom firmware, but only into the RAM.
Tampering with the vBIOS will void your graphics card’s warranty
So will removing the heat sink if you want to clean it or reapply thermal paste. But my card is from ASRock which has basically nonexistent warranty anyway.
I just memorized -czvf and -xzvf, create and extract. If I want to do something else to a file, unless I’m desperate I just extract it and do that stuff.
I also get tripped up but at the same time I try to remember source, then destination. And since I’m almost always doing soft links -s reminds me that source is first.
These “like open source but you can’t compete with us” licenses leave a real bad taste in my mouth. I don’t care for the (x)GPL(n), but those guys walk the walk. The licenses are clear and match the rhetoric. They want to convert you, and they would prefer you use other peoples’ stuff if you don’t want to share. These new licenses are…. sleazy is the best word I can think of. I don’t think it’s going to achieve the results they’re hoping for.
I suspect ten years from now, most of the companies who’re attempting this sort of thing (and haven’t folded) will have moved to a more traditional commercial license, or ended those products. The companies they’re trying to fight off with them will be still around doing the same thing.
It’s kind of funny because “like open source but you can’t compete with us” is how I view most AGPL projects, particularly when it’s backed by a commercial entity.
If Hashicorp had taken the AGPL route here, they would still be able to do literally whatever they want with the code - including using it in combination with proprietary extensions/modules (because they own it), but no-one else would have that ability.
I’m not saying I love the BSL, but it’s barely worse than the AGPL IMO.
AGPL doesn’t prevent anyone from using the software to compete with the authors. There’s a huge difference; namely the difference between being open source and being proprietary.
“You may do anything with the software as long as you release the source code” is not at all similar to “you may do anything with the software as long as you don’t compete with us”.
AGPL doesn’t prevent anyone from using the software to compete with the authors.
AGPL had that effect in practice, which is why a lot of companies adopted it. If your competitors couldn’t or wouldn’t share and license their changes or broader codebases alike, requiring they do so in order to build on your software was functionally equivalent to saying they couldn’t build on your software.
That behavior of AGPL began to break down in popular perception after the public AWS-MongoDB brouhaha. AWS argued they could offer MongoDB without sharing and licensing code they didn’t want to share and license. Mongo wrote and adopted a new variant of AGPL that more clearly required it.
I don’t mind that copyleft licenses give their copyright holders a competitive advantage. I am in no way opposed to that. I’m opposed to licenses which masquerade as open source while making it illegal to compete with the copyright holder.
It prevents them from offering extensions/whatever that are specific to them - but the project “owners” have no such limitations.
This is my point: with agpl a hypothetical hashicorp could release terraform++ which is the same agpl core plus a proprietary extension, under a proprietary license.
No contributors can do the same.
That is what I meant when I said they “can’t compete”. Anything they want to offer they have to make available upstream too, but the reverse is not true.
I mean, that’s broadly true of any copyleft licence. The licence doesn’t privilege the “project owner”, copyright law does. So I feel like that’s a bit of a red herring (although I won’t pass up the opportunity to say that I think your enemy here should be CLAs, which are what allow corporate-owned projects to accept community contributions and retain most or all of the privileges of a sole copyright holder).
Ownership privilege thing aside, you’re more or less claiming that the AGPL’s “you must free everything if you modify” requirement is so severe as to be equivalent to forbidding modification. But that is—if you allow that providing access to a SaaS product to someone in 2023 is the same kind of transaction as shipping software to them 20 years ago—a substantially weaker restriction than its pre-SaaS antecedent in GPL, which required you to free everything you provided to your customer anyway, even if you were just using it unmodified.
(Sufficiently strong) copyleft reduces the set of people who can build a given project into a proprietary product. Ideally it’d reduce it to zero, but in a world with relatively little copylefted software and lots of CLAs, there’s often one particular entity that still enjoys that option. I think that:
is better than not reducing it at all
still admits plenty of uses of the software that compete with its owners
Yeah, but only in the case where a single corporation dominates the development. Once it’s a combined effort, AGPL locks them mutually in. Especially if they encourage outside individual contributions to block relicensing even more thoroughly.
It’s the best instrument to ensure the project stays libre despite sharehoarders. Not a perfect one.
It depends somewhat. There are also variations on the AGPL, including requiring copyright assignment or a CLA that allows the main player to produce things that include proprietary extensions but doesn’t permit the same for other people.
The main thing that companies care about is freedom from lock in. The traditional way of avoiding lock in is to have a second source (IBM, famously, did this with every single component in the IBM PC except the OS because they thought operating systems were commodities, which directly led to Microsoft’s dominance of the desktop market). When you have a second source, you can always switch to them if the primary supplier changes their terms to something that you don’t like (puts up the price too much, removes features that you care about, and so on).
In theory, the AGPL guarantees that a second source can exist if there is a market for one. If you use an AGPL’d service from vendor X and they put up their price to more than you think it’s worth, vendor Y can come along and sell precisely the same thing for the cost that you are willing to pay. As long as that’s a price that’s sufficient to maintain the service, that’s fine.
The down side here is that, if vendor X is paying for the maintenance, it is cheaper for vendor Y to provide the service as long as vendor X stays in business. AGPL can lead to the kind of failure mode where vendor X pays all of the costs, vendor Y gets most of the profit, vendor X goes out of business, vendor Y then can’t keep providing the service and needs to either take vendor X’s place (and then gets pushed out by vendor Z doing the same thing) or just watches the service degrade until customers go away. This is basically the same case as the tragedy of the commons.
Bit of a tangent, but it would probably be a good idea to find a different term for this because the “tragedy of the commons” – as originally described – has some issues. For example:
Even before Hardin’s ‘The Tragedy of the Commons’ was published, however, the young political scientist Elinor Ostrom had proven him wrong.
And this is not some lone crank with an axe to grind – Ostrom received the Nobel in economics in 2009 “for her analysis of economic governance, especially the commons”.
Also, in general I know a lot of tech people (not necessarily any individual in this thread, just a general trend) like to sneer at social sciences, but Ostrom and others have done a lot of work on successful approaches to managing commons, and the FOSS world could stand to learn from that.
From a game theory perspective, “tragedy of the commons” doesn’t necessarily mean a situation where you’re doomed to ruin everything; it specifically means a situation which is lose/lose IF each individual actor plays in their own self-interest without cooperation. I don’t think that’s an inaccurate description of the situation; it’s basically saying you can’t win without some agreement to cooperate. The fallacy (which I agree needs to be debunked) is that cooperation is impossible.
Except there’s Amazon. Amazon doesn’t want vendor X do stay in business at all. And they have the money to sell at a loss until X goes bankrupt, and they don’t even need to close the source.
That just shows that current price is not a good enough signal for sustainable planning.
Imagine inheriting an orchard. You need to replant portion of the trees every year to keep it producing indefinitely. If you don’t, you immediately save some money, the trees will keep producing for some time into the future and in effect you make more than you’d make if you took proper care and replanted. After all it takes many years for new trees to start producing and you only have so much land.
At some point society will be forced to acknowledge that simplistic methods like lowest-price tendering does not work, because it is transitively extractive. Tragedy of commons would be hard to come about in a rural community. If you overgrazed, you would lose social status and people would be hesitant to lend you a helping hand, prompting you to compensate. Everyone would intuitively understand that if there was no common grass, it would be a problem. It’s just that our current society has decided to mostly prohibit non-financial forms of planning, because they are hard to check for fraudulent behavior.
It’s not AGPL, it’s the threat of jail time for a CFO who green lights a purchase of the costlier of two offerings while hiring programmers for twice the price to do it in-house would be perfectly fine.
Except that there are multiple examples of rural communities experiencing tragedy of the commons. They are no more immune than any other community. The only effective remedy I know of is an authority enforcing rules.
Oh, I am not saying explicit rules are not better. Just that while chasing “good rules” we might have somehow forgotten that “good” does not always equate with “most easily enforceable”.
Does it matter, though? I would say that adoption for the purpose of virtue signalling masking adversarial conduct is orthogonal to the adoption for the purpose of protecting user and developer interests. The license is not to blame for what people use it for.
All it takes is a single determined person at the right moment to force the organization to publish it under AGPL and accept outside contributions and increasingly the virality of the license will make it a part of the commons.
An example of project that would be extremely to close now is NextCloud. It does accept external contributions, does not ask for CLA and assumes AGPL. Every new contributor is yet another party who would have to agree to a license change.
From a practical perspective, the AGPL is a non-starter. And not just because of evil freedom-hating corporations.
Suppose that Carol is a true and fervent lover of Free Software and of software freedom. Every morning, she open-palm slams a copy of the GNU Manifesto, etc.
And suppose that Carol runs an online forum for other Free Software enthusiasts like herself. She ensures that only copyleft-licensed software is used to run it, and publishes not just the full source (under a copyleft license, of course), but also all the configuration, scripts, dev tooling, etc., because she believes so deeply in the ethos of Free Software that she goes above and beyond what is required of her by the licenses.
And suppose that Carol’s forum has two significant dependencies: PackageA which is written by Alice, and PackageB which is written by Bob. And suppose Alice gets a bit burned out as a maintainer, and hands over commit access to PackageA to Carol, knowing it will be in good hands.
Now, for sake of argument, suppose that PackageA is quite old and was originally released by Alice under GPLv2 without an “or any later version” clause. And PackageB is newer, and was released by Bob under GPLv3.
And then one day Bob relicenses PackageB to AGPL.
And then not long after, Carol finds a security issue in PackageB. Carol can report this to Bob, and submit a patch to Bob, but until Bob has incorporated the patch into his tree, it is illegal for Carol to incorporate the fix into her own running copy on her site, because she would be modifying PackageB and triggering a requirement to release all the AGPL-defined “Corresponding Source” of her forum under the AGPL. And Carol does not have the legal right to relicense PackageA. The AGPL imposes more restrictions/passes on fewer freedoms than the GPLv2, so it would be a GPLv2 violation to distribute PackageA under AGPL; she does not have the ability to do an “intermediate” relicensing to GPLv3 (which makes an exception to the no-further-restrictions clause specifically to allow AGPL); and she is not the copyright holder to PackageA, so she has no other avenue for legally relicensing.
In other words, the AGPL can create a situation where, despite the best of good will and good faith and professed belief in software freedom from all parties, and copyleft licenses on everything involved, it would still be illegal for someone to exercise the basic freedoms listed in the Free Software definition. And this is not a particularly far-fetched situation – there’s a lot of differently-licensed software out there, and even the FSF’s own licenses can accidentally create legal incompatibilities amongst themselves (as in the given example).
In this particular case, as long as Carol is not distributing appliances or containers that assemble both PackageA and PackageB into a single derivative work, I believe she would be OK just publishing the corresponding sources independently for any user to assemble themselves.
I am of the opinion that libraries should be licensed under MIT, Apache, MPL (or perhaps LGPL) since the incentive is then for everyone to contribute instead of doing the foundational work privately. Including otherwise proprietary businesses. It is the final applications that benefit from GPL / AGPL.
It was a tactical error on the part of Alice to use GPLv2-only license for a library, causing such hassle to downstream users.
It doesn’t take much to produce a situation where every individual component is 100% FSF-approved copyleft Free Software using FSF’s own licenses, but combining them makes it illegal to exercise one’s freedom, and the situation I gave is one such example.
And dismissing it by claiming that it was a “tactical error” to use what was, at the time, the FSF’s flagship recommended copyleft Free Software license is… well, it’s something. There’s a ton of GPLv2 code out there, and quite a lot of it doesn’t have the “or any later version” clause, which makes it inherently incompatible with the FSF’s newer and shinier toy.
I have seen this line of reasoning in the past. “FSF should have stuck with GPLv2.” I don’t think it adds much to the discussion.
I imagine people at FSF were not that happy about developers using GPLv2 in the GPLv2-only manner, leaving doors open to various exploits GPLv3 and AGPL addressed, blocking compatibility and damaging their reputation.
And dismissing it by claiming that it was a “tactical error” to use what was, at the time, the FSF’s flagship recommended copyleft Free Software license is… well, it’s something.
If Alice was actually listening to FSF, she would have used LGPL (a conservative choice) or GPL (more aggressively anti-proprietary), but in both cases “or any later version”. She though she was being smart when she were deleting those words from the license template. And now people who she would probably have considered legitimate users cannot use library.
So yeah, a tactical error. She should have chosen a permissive license or stuck with FSF who knew what the game actually is. And honestly, I can’t fault FSF for ignoring the compatibility issues. They are trying to defend the commons despite people who only see what’s directly in front of them. They probably count on social pressure to convince authors to update their GPLv2-only to GPLv2+.
It would be nice if she agreed to a license change.
Several times in the past I’ve pointed out the irony of the Free-Software-est of all the licenses being frequently used to enforce a decidedly un-Free-Software position: one of the major use cases of the AGPL is to ensure a company can produce proprietary add-ons and extensions while using copyright to ensure nobody else gets to compete with them on a level playing field.
Well the AGPL means “you can use it but you can’t sell software for a living” as opposed to “you can use it but you have to host it forever and can never outsource it unless it’s to us”
A particularly nasty one to start with today!
A part of me wonders whether the creator went out of his way to guard against common LLM usage.
Maybe a little bit, but it’s a recurring theme in AoC that you have to implement the spec as written, but not the spec as you think it means on a first read.
I can barely read the trite stuff about Elves as it is and I habitually skim all the text. I think that might just be enough obfuscation against LLMs.
I think there was a rash of solutions in the early days of Dec 2022 where people were oohing and aahing over that current generation of LLMs solving the problems instantly.
It died down quite a bit as the difficulty ramped up.
Oh yeah, I got one very tedious bit of slice manipulation handed to me by copilot but for the rest it’s been mostly saving me typing debug output and the likes.
Guilty as charged. I messed around with ChatGPT on the first few problems last year. That was right after it came out, and it was pretty amazing how fast it could come up with a (typically slightly buggy) solution.
I apparently lucked into doing it the way that doesn’t run into any of the problems other people had, on a whim. (Spoiler below, stop if you don’t want them, but it’s day 1 so…)
For part 1, rather than doing a multi-match and extracting the first and last matches in the list, I did a match against the input and a match against the reversed input, which is an old trick.
For part 2, I kept the same structure rather than rewrite, which meant that I matched the reversed string against
/\d|eno|owt|eerht|ruof|evif|xis|neves|thgie|enin/, and then re-reversed the capture before passing it through a string-to-num map.And it turns out that that totally sidesteps the problem of “wait, how am I supposed to get
21out ofxxtwonexx?”I just used a regular expression, with the leading group as optional. Means you always pick up the trailing “one” in “twone” first.
I looked for non-overlaping matches and got the right solution. Maybe my input never hit this “twone” edge case by luck!
I took a similar approach. I’m using C++, so the natural solution was to use reverse iterators
.rbegin()and.rend()that iterate through the elements of a container in reverse order. Rather than use a regex—which seemed like overkill—for part two, I just had an array of digit names I looped through and performed the appropriate search and chose the earliest one:And in reverse:
My figuring on what is and isn’t “overkill” is: AoC is ranked by when you submit your solution, so that’s time to write the code plus time to run it. If something is really the wrong tool, the challenge will prove it by making your solution take an hour, or a terabyte of RAM, to run. But if I’m using a language where regexes are “right there” and they make my solution take 100ms instead of 10ms, I’m not bothered.
I like AOC because everyone can have their own goal! I’m impressed by people who can chase the leaderboard. I always personally aim for the lowest possible latency. Managed to get both parts today in under 120 microseconds including file reading and parsing.
SPOILER…
The difficulty, IMO, is that the problematic lines aren’t in the sample.
I did overlapping regexp matches. It was easy, once I caught why my first attempt didn’t work. Another solution would be to just do a search with
indexOfandlastIndexOf, for each expected words, but you have to be careful to sort the results.Yeah, it’s pretty nasty for a day 1.
There’s a subtle hint, because the first sample does include a case where there’s only one digit (the “first” and “last” match are the same, or you could say they overlap completely). When you get the part 2 spec you have an opportunity to ask yourself “hmm, what changes about overlaps now that we’re matching strings of more than one character?”. Or at least it gives you a good first place to look when things go wrong.
Apparently some peoplle tried to solve part 2 using substitution (replace words with digits and then feed that into the solution for part 1), which also suffers from problems with overlaps, but in a way that’s harder to dig yourself out of.
Yes. My implementation passed all the sample tests, but it returns an incorrect value. Not easy at all.
RustDesk scares me. Here’re some of the things I found when I looked through
rustdesk/rustdesk-server’s source code last (b713303c1557f158a3b6279fcce5139190ed2937):shand using it to run shell codeshdoes not set-e,-u, or-o pipefailand doesn’t check exit codesstatic muts in multithreadedasynccodeOh my god, this is almost daily wtf worthy. It’s “useless use of cat” to a level I wouldn’t have believed existed. What’s wrong with:
EDIT: The other part that’s really shocking is the needless amount of additional allocation, i.e. using String, Vec,
.to_owned()and.to_vec()everywhere rather than borrowing as &str or as a slice.This reminds me a lot of the Rust project I was on at my previous gig, especially all the
unwrap_or_default(), checking if the string it just generatedis_empty()instead of using anOption, useless use ofcatand other stuff that is easily done from within Rust… I had to fix a lot of that.I think it’s a strong signal of a team of developers who are not very familiar with Rust, working under time pressure to deliver a product, being given features to implement with instructions on how to do it in shell and not having much time or incentives to learn how to do it properly.
Excessive of
mutis another sin the codebase commits.It probably betrays a lack of understanding of the shell commands being executed as well (at least the
awk/cat/pswrappers—I can almost forgive wrappingloginctlin a rush to ship, as D-Bus APIs are often poorly documented and annoying to work with).I have to imagine if the developers had actually understood that the
awk/catwrappers were simply reading files and doing rudimentary parsing, then they would have at least tried to implement them in rust.I’m reminded of a piece of advice given in earnest to beginner Rust programmers, that if you’re having trouble dealing with lifetimes when getting a piece of code you’re writing to compile, just call
clone()or otherwise go head and do the extra validations, in order to get the essential logic of your program working. After all, these extra copies and boxings of data structures are not really very different from what the runtime of an interpreted, dynamically-typed language like Python would be doing under the hood as a matter of course, in your particular code the extra overhead might not be meaningful anyway, and if it is you at least have the option of rewriting in to use more references and fewer clones once you’ve gotten the initial code working.That’s a good advice that I need to keep reminding myself, even after years of using Rust. Unless used in the most critical code paths, clones and allocations don’t really matter, while it’s easy to waste a lot of time trying to avoid them.
That’s the kind of thing you typically only see in datascience python code!!!
The number of
os.system(“rm -rf ../../files/../idontknowwhy/../*”)type commands I’ve seen…
I find it worth pointing out that, whereas the shell-ing out is ‘merely’ a (strong) code smell, this is prima facie Undefined Behavior.
The shell stuff is especially odd, given how easy it is to do those things directly from Rust.
While the shelling-out looks like just bad practice, the
unsafeuse does look severe. Reported upstream: https://github.com/rustdesk/rustdesk-server/issues/324Reminds me of my friend who works as tech support in a hospital. From her I have gathered two main facts:
Ah, the dichotomy of human existence.
I think you’re being unreasonable toward medical professionals. I think the points could be more succinctly stated as:
People are usually idiots (including IT professionals like us commenting on this)
It should be a capital offense to force quarterly password changes
Yeah, I can get behind that.
In my experience, it is never productive (but possibly cathartic) to label people as idiots.
Computer people often seem to have a hard time understanding this, but just because someone isn’t a computer expert doesn’t make them an idiot. They have better things to do than trying to work a stupid piece of machinery, and as you can read elsewhere in these comments, they have to do so under a tremendous amount of stress and often while sleep deprived.
They might be a “computer idiot”, but by that standard I’m a total idiot in most non-computer fields. And I suspect a lot of us are.
One would think a system could be designed wherein someone whose role requires one simple set of permissions (view and modify one’s own timesheet) would be exempt from the normal security standards of rotating passwords because the permissions of the role simply can’t result in any kind of harmful system breach.
There is absolutely no reason to use passwords in the hospital setting at all. Everyone should just use a card that is unlocked with a PIN code for the next 12 hours or until the person clocks out. Then the person should be able to move freely between the terminals.
I’m smiling a little bit at the history you reminded me of. https://en.wikipedia.org/wiki/Sun_Ray These units had smart card-based authentication and transparently portable sessions. You walk away from a terminal and take your card with you, plug it into a new terminal, and carry on working on whatever you were doing. Virtually zero friction other than network latency sometimes making the UX a bit laggy. In practice it worked really nicely the vast majority of the time.
I used these briefly in Bristol in 2001 through a friend from the Bristol and Bath Linux Users Group, it was really impressive, using tmux sometimes reminds me of playing with those Sun boxes.
Most of the systems are web-based nowadays, it would be trivial to make the sessions portable or even multi-headed.
Apologies that it took me a while to reply. How does web-based make sessions portable? It seems to me, without having put a ton of though into it, that being browser-based makes portability significantly more challenging. You can’t use a shared client environment (i.e. you need a separate Windows/Linux/OSX/whatever account for each user) and you’ll still have to manage security within the confines of a browser environment. I agree that it makes it really easy to access content from everywhere, but you’re also dealing with multi-layer authentication (Windows login, browser login, cookie expiration times, etc).
With a SunRay you could pull your smartcard, walk to the other end of the building, put your smartcard in, and have everything magically pop back up exactly how you’d left it.
Ah, right. I was assuming that at this point the software stack runs whole in the browser and the workstations would be basically kiosks.
Some equipment has a dedicated computer (radiology basically), but since there usually are only so many operators and the software might not even support multiple system users, shared OS session would probably work fine in practice.
But yeah, we might just as well just launch a container on a shared server for every user and let them run a lightweight desktop over spice.
You’d think so, yes.
This is what I use:
But in 6 years only one human has gotten my email address from this and sent me a message, so it appears to work a little too well.
Maybe everybody is using big endian systems.
Holy undefined behavior!
What’s the undefined behavior?
The comma is a sequence point, so the evaluation order is well defined, unsigned overflow is well defined, and I thought accessing the result of a cast to
char *was well defined too, right?I’m pretty sure the endiannes of integers is undefined – and in general, casting a non-
voidpointer to a different pointer.Endianness is implementation defined not undefined, and casting to
char *is explicitly allowed in the standard.This is the ultimate Turing tarpit - get email scrapers to argue about C standards instead.
Oh, my bad. Still, the implementation-defined behavior makes the snippet ambiguous. And I forgot to mention that the size of
long longis also implementation-defined.Edit: Actually, on a platform where both
charandlong longare 64 bits, wouldn’t this callputson a string without a zero terminator and therefore cause undefined behavior?I’ve finally brought the bulk of my books to Denmark after moving here a few months ago, so I’m putting together some IKEA shelves, and if the weather isn’t woeful I’ll make a trip to pick up some houseplants. Hopefully it’ll all help my apartment feel a little more lived in.
I’m going to try get some writing done; I have a draft of an osdev and retrocomputing related article I want to finish.
Welcome to Denmark!
The weather here not being woeful in November is, unfortunately, a bold hope.
Thanks!
I moved from Dublin to Aarhus. So it’s slightly colder on average, but also slightly less wet (not that you’d think so after October here), so I think I’ve a good idea of what I can expect from November (and you’re probably correct with “a bold hope” haha).
Get the retrocomputing one done B)
Get the osdev one done B)
I think that the points around making things serializable is interesting and valid. But I think that it’s important to acknowledge that the rust type system does in fact involve trade-offs. Self referential structures aren’t just “object soup”. They are a natural and fundamental way of modeling information and when a language makes it awkward to express those relationships it has given up something. Rust gave it up in return for memory safety without GC which is a valid tradeoff.
One could say the same thing about
goto. Justs/information/code/.The progress in programming comes from increasing global composability by eliminating local expressiveness that interferes with it.
In my experience forcing developers to express information relationships in a bigger framework of tree of ownership leads to substantially better software.
I would argue that the comparison to goto is not completely apt — the two tradeoffs offer very different benefits. In the goto case you barely lose anything (almost every usage can be converted to structured control-flow), while your reasoning abilities are greatly increased. It is no accident Dijkstra’s paper became so widespread.
Not having a GC is a valid tradeoff in some (niche) use cases, as mentioned, but it is very far from a ubiquitous one. There is nothing fundamentally bad about self-referential data, in fact I think the vast majority of existing software would be vastly more complicated without them.
Almost every use of self-referencing can be expressed as indexes, with increased safety, ability to reason about and composability.
Not having a GC makes the code have no runtime, which increases it’s composability by making it potentially usable in everything. No one will use Java code from Python, or JS, etc. while almost every language can use Rust because it doesn’t have a runtime.
It also allows practical aliasing model which enables efficient memory safety and concurrency.
Small concession, big wins.
“There’s nothing fundamentally bad about
goto.” Quite a few elegantgotopatterns in C, that I don’t miss, but used to employ.Dropping it too was a relatively minor concession for ability to reason and compose larger software.
Moving references out of the type system does not increase safety. If you have a pointer to an object of type T, and the type system knows that it is a
T*, then static and dynamic checkers (and run-time enforcement techniques) can ensure that yourT*does not outlive the underlying T, or that it is explicitly invalidated if it does. In contrast, if you refer to objects via an index then you are hiding this relationship from the type system. You are doing manual memory management again. It’s not quite as bad as C, because a dangling index will point to a bounds-checked object and so will either be detectably invalid ot point to some T, but there is nothing that guarantees that it points to the T that you think it does.If you delete an object from a graph, you now have a free slot in your index table. Do you reuse that? Now you have a mechanism for introducing use-after-free bugs. A dangling index to that slot will refer to the wrong object. Alternatively, do you avoid reusing the slot? Now you have a memory leak and have increasingly sparse data structures that become progressively more inefficient to traverse.
This kind of model works fairly well for things like the CFG in the Rust compiler, because those are short-lived structures and so it’s fine to simply burn slots for deallocated basic blocks and free the whole thing at the end (though it’s still more effort than it would be to use explicit traced references or weak references in a reference-counted environment). They work very poorly for data structures that change shape over time.
People happily used Objective-C from Ruby, Python, and Java (and even Haskell and Ocaml) via mechanically generated FFI, and from C/C++ via hand-written FFI. Having a runtime didn’t prevent this. Having reference counting in the runtime made it easy to build bridges that automatically surfaced Objective-C objects in these other languages. Objective-C’s perception as an Apple language was the largest barrier to this.
Except that now you’re advocating for design patterns that throw away temporal safety. Type-pooled allocators in C/C++ have the same level of temporal safety as your proposal. They’re much better than nothing (which is why Apple now uses them in the XNU kernel) but they’re a long way away from ideal.
C’s goto is absolutely nothing like the goto that existed prior to structured programming. It has well-defined behaviour with respect to lexically scoped objects (you can’t branch over a local variable’s declaration, for example) and it is not permitted to branch to labels in a different function.
You can branch over a local variable’s declaration. The following is legal:
The only restriction that the C standard places on goto’s are that (1) the jump must be within the same function, and that (2) you cannot jump into a scope containing a variable-length array. (C23 6.8.6.1 “The goto statement”, draft N3096), so the following would be invalid:
EDIT: fixed formatting.
s/increased/decreased. Even naive C have better tools to actually debug “dangling pointer” problems — valgrind will cry about it, but there is nothing to do if you store an index, but the associated value was replaced in the meanwhile. Let alone a managed memory language.
And come on, increased ability to reason about? What is more clear, that my Node has a
parent: Node, or if it has aparent: u32? In a managed language the former has only a single invalid value (or if the language is null-aware, none), the latter has potentially 2^32, let alone the case of dangling pointers.Yes, which is a niche use case, which might be worth it depending on context, say, writing a specialized library. Most people want to write applications, though, where this concern is negligible, and can be circumvented by IPC.
So you think a checked array access will have better optimizations by the compiler than pointers themselves? Especially that one of rust’s strong points is that it can actually “use the restrict keyword” properly, thanks to the borrow checker? This is an unfounded statement on your part.
There is definitely value in restricting the “degree of freedom” of code, see types, but as most things, it is always a tradeoff. We don’t generally use regular programs, even though they are much easier to reason about than Turing-complete ones.
I think you misunderstood this. The idea is to not to just replace the pointer with an index in the object, but to implement another vector of data in the main struct:
This works if your data is immutable, so you can sort it and use ranges and binary search. If you mutate your data during runtime, using a map works better:
Although if the data is kept in memory for a long time, and mutates during its lifetime, I’d consider using indices like this a bit before jumping into writing the code.
Again, you have all your data in flat structures and you do not have any nesting of data anywhere. It makes a lot of sense e.g. when modeling databases to an ORM, or when writing a compiler.
That’s an important point. If one is working with graph data, chances are high that it will need to be persisted anyway, in which case there’s barely any technology that will allow persisting the “object soup” natively (they were attempt of “object oriented databases”, but obviously it’s a terrible idea so it didn’t work out).
So now the “ease” of “object soup” has an additional difficulty of dealing with the conversion back and forth to the form I’d advocate for in the first place as clearly better.
It’s not a pointer. It can’t be accidentally deferenced forgetting to check if it’s still valid. Sure the logical coherence of the larger data-structure needs to be taken care of, but it’s not unlike GCed languages. In a managed language if you point a child at a new parent, but forget to update the old parent, are you going to get great results? If anything, all the “invalid values” (often including ‘generations’ in a slotmap) can actually crash the code in an orderly way and help spot the issue, instead of letting it “mostly work”.
It’s
parent: NodeId.That’s no a niche use-case. The ubiquity of C and C++ despite their huge risks, is a result of being able to reuse them.
Who cares. Rust does have references, including shared ones. You only really need to use indexes when you’re already working with a graph, and children need to point at parents, etc. and the whole thing is non-trivial already, and the performance overhead of “non-native-references” is akin to the cost of a
JOINin a database .Normal relationships that fit a DAG (“tree of onwership”) are all one needs 99% of the time anyway.
Worry about all the memory barriers, extra memory usage and cache invalidation caused by a GCed-runtime that is slowing down the normal GCed code at all times, to make the GC work, not a 1% case in Rust, that is often fast enough.
It wasn’t obvious to me, but the reasons for Rust not supporting self-referential structs can be said to be historical, not strictly technical (see 1 and 2).
I find that restriction quite limiting in the kind of APIs I’m able to design. I often want to encapsulate two object in a single struct, where one fields references the other (e.g.
rusqlite::Connectionandrusqlite::Transaction). You can’t do that in current Rust because:memcpy().In [2], withoutboats lays out a proposal for a
Movetrait that would invert the status quo: structs would be immovable by default and movable as an opt-in. Immovable types could be naturally self-referential and even structs that implementMovecould manually instruct the compiler how to copy themselves, thereby solving (2). As for (1), it’s a simple convention that would need to be relaxed.The biggest reason in my understanding why this is not happening is backward compatibility.
For reference, these were discussed here as https://lobste.rs/s/6fjkeh/why_async_rust and https://lobste.rs/s/tfsr57/changing_rules_rust.
In fact, every program with goto can be converted into one with only loops and conditionals https://en.wikipedia.org/wiki/Structured_program_theorem
Yes, though arguably there are a few usage patterns where
gotoor its tamer little brother,breakis much cleaner.You’re falling into the trap of believing that there is a single path of progress towards some kind of ideal, which every language is moving towards or away from. Ideas in languages evolve the same way organisms do: In response to localised pressures, towards whatever gives them an advantage in that domain.
So to your point: goto has indeed mostly “evolved” out of most programmers workflow and languages. But it is still alive and well in certain contexts, for example kernel development.
Totally agreed, there are major tradeoffs here. One that’s gotten a lot of discussion recently: The designers of async/await had to go through an enormous amount of trouble to fit futures into the language, since they’re fundamentally self-referential types. They settled on the
PinAPI, which is a leading contender for the trickiest module in the whole standard library.Another interesting one: The small-string-optimization layout in GCC’s
std::stringis self-referential. Rust’sStringdoesn’t do SSO in general, but if it did, it would have to do something like what Clang does rather than what GCC does.I don’t think SSO is typically self-referential in the sense that is not allowed in rust. Typically SSO looks like (but more optimized to use every available byte and not hardcoded to 64 but platforms).
There is no explicit reference to the inline buffer (as that would waste a word of SSO space). Rust can handle this perfectly fine as a method like
as_u8can borrow a reference to the buffer.You’re right about what’s typical, and for example that’s what Clang’s implementation does, but GCC’s
std::stringis different. Reading C++ stdlib headers is the worst, but the most direct evidence of this is that Clang’s data pointer is only present in the long representation but GCC’s data pointer is always present. Also, Clang has a fast path for bitwise copying short strings, but GCC does not, because it can’t copy the self-referential data pointer. Since this steals 8 bytes of capacity from the small representation, GCC padsstd::stringto 32 bytes.Oops, I skipped over what’s probably the most important implementation detail: Clang’s
string::data()has a branch to handle the small string case, but GCC’sstring::data()does not.Well said. I’m as big a member of the Rust Evangelism Strike Force as anyone, but it’s very frustrating to see so many people try to spin things that are clearly flaws/shortcomings/whatever as somehow actually good. It’s not just Rust fans, of course–I see it in all of the various programming language forums/subreddits/etc. It’s okay to admit that something sucks about the thing you like!
Using arenas in Rust is sometimes the best approach because of performance reasons, but frequently arenas are just awkward, bug prone (must keep your indexes in sync with arena updates), workarounds for Rust’s borrow rules.
In the past seven years I’ve been working with Rust, using the approach with indices described in the article has been an absolutely awesome API two times in quite large projects. It’s a very nice abstraction when you need to parse a file of code once and then refer it later. Eventually you’ll have one allocated string sitting in the bottom, tokenized in the next layer and then building a database of typed token IDs either in vectors or maps on top of the AST. The public API is built on top of the flat vectors of IDs which can then take a reference directly from the string which represents the code.
It is very efficient and extremely nice API to use. It has been causing almost zero crashes and has accelerated all development in the corresponding code bases. It’s not that I’m trying to evangelize Rust, but this way of modeling the data works very nicely for a few hard problems I’ve been needing to solve in my career.
Sure. And I guess I shouldn’t have specified “because of performance reasons”. Rather, my point was that sometimes the arena approach is actually the best approach to the problem, but sometimes it’s used because it’s the only option.
Phrased a different way: if you were writing your code in a GC’d ref-happy language and you’d still choose the arena approach, great. But if you wouldn’t choose the arena approach in the hypothetical language, then it seems hard to argue that it’s better than “object soup” or whatever. It’s like the infamous tree/graph data structures conundrum. Using trees and graphs to model data relationships is very often exactly what we want. The fact that I see so many (frankly, newbie) Rust fans try to tell people that they really don’t want/need trees is crazy.
Well, that was a short week /s (you may want to update the title).
Outside of work, I’ve started toying around with Tilengine to build an F-Zero-like racing game with SNES Mode 7 style graphics. It’s a lot of fun.
Whelp. Need to apply more coffee before using computer it seems.
This always seemed a little optimistic to me.
What’s your use case for
sumprod()?Isn’t that just a dot product?
I’m dumb.
It always depends somewhat on your baseline. HHVM managed to more than double performance each year for multiple years because PHP was very slow. Starting from C, you only manage that kind of speedup when you’re doing autovectorisation for the first time or similar. A number of Python implementations are a lot faster than CPython, doing things like type-specialisation in the AST, better dispatch in the interpreter, and JITing. 50% speedup over five releases is a 750% speedup overall and that’s less than the speedup you get going from a decent bytecode interpreter to a modest JIT in most Smalltalk-like languages. It seems quite feasible for CPython to get that kind of speedup.
Zig is pretty good for this too, with support for wasm and wasi targets in the compiler and no large runtime needed. Just compile your library with
-target wasm32-freestandingor-target was32-wasiand you’re good to go.The language is pre-1.0 though, so it doesn’t have an advantage over rust there.
I find it a little odd the recommendation for Rust as a C or assembly replacement (even more on the latter). I 100% see Rust as a C++ replacement, in all the ways it could mean.
Otherwise, yeah, like others have mentioned the author is on the nose. I stopped using Rust after about 3 years when I realized the complexity is out the wazoo. The lack of a standard, the massive toolchain to build Rust itself, the massive man hours to create a Rust compiler, are just way too much. In comparison, look at the amount of C compilers over the years.
It’s this exact reason I switched to Zig. If you want 100% memory safety, then Go (or any GC language) really is a great choice.
Rust is a great C replacement. C features and design patterns map almost 1:1 to Rust. Writing of wrappers for C libraries is straightforward. There’s very accurate c2rust.
OTOH Rust has plenty of impedance mismatches with C++. It doesn’t have enough OOP features to translate C++ interfaces to usable Rust. Can’t emulate copy constructors. Doesn’t have placement or RVO. Templates vs generics are all different except the superficial choice of angle brackets for their syntax.
I know some people equate it like: C is small, C++ is big, Rust is big, therefore Rust == C++, but that is a false equivalence. They have their complexity for different reasons, and there’s only a small overlap in their features.
Only if the library you’re wrapping’s model of ownership lines up nicely with rust’s ideas cleanly, which is not always the case.
A good write-up on this is the article Giving up on
wlroots-rsby the developer behind thewlrootsrust wrapperwlroots-rs, as well as the Way Cooler compositor.This is one famous example, but OTOH there are thousands of crates successfully wrapping C dependencies. A lot of libraries have boring init + free combos, sometimes an Rc or Cow equivalent. Some need an artisanal smart pointer.
Rust is a C++ replacement, not C. Its concept of references and lifetimes is lifted from C++, and those concepts do not really exist in C (and where they do they are described differently). Rust has ML’s type system, but that doesn’t make it a child of C and not C++.
Rust took move semantics from C++, but I don’t think “lifted a couple of features from” means it’s a suitable replacement for the whole language.
I look at this from perspective of Liskov’s substitutability. Rust is close to having a superset of C features, but only a subset of C++ features, and seemingly common features (moves, references, dtors, overloading) have conflicting semantics.
Ownership exists in C, because that’s inherent to having heap allocations and
freethat must be called exactly once. References and lifetimes also exist when you can have pointers to the stack and non-GC pointers to interiors of allocations. These things exist even if C doesn’t use explicit terminology for them.C doesn’t enforce lifetimes via the type system, but via threats of UB, and runtime crashes/corruption. By analogy, dynamically typed languages can’t prevent type errors at compile time, but that doesn’t mean they don’t have types or type errors.
There’s also another way to look at it, putting language theory aside. Rust has the least amount of success in GUIs and game development, which happen to be C++’s strong domains. GTK in C has decent Rust bindings. Qt is unusable. Rust was accepted in Linux, which famously rejected C++.
That’s really interesting to hear. Do you have a resource I (and readers) could check out regarding Rust-C equivalence? :o
And yes, your last paragraph is more or less the reasoning. Well that and I experienced that Rust has advanced features not like C++, but are powerful as the ones in C++ (unlike C which lacks them).
I wish projects like this would clearly say why they exist and why anyone might care to use/contribute/follow them.
My best guess from a quick search is that the authors either made it just for fun or they made it because they wanted ZFS features but don’t like the license conflict or the btrfs code.
The proposed value for Linux users seems to be “btrfs but better”.
I think https://bcachefs.org/ is ok at summarising the main points.
If you’re interested in the origin, bcache itself is a stable, transparent way to split hot/cold data between multiple fast/slow drives through caching. The author at some point said effectively - you know, that’s halfway to a decent filesystem with good features, so let’s make it happen.
I did check their site. It told me what but not why.
The patreon has a lot more about the motivation / why part and an overview of the current options: https://www.patreon.com/bcachefs/about
Note that the “Status” section is woefully out-of-date. Everything on that list is complete and stable, except for erasure coding (which is not yet stable). I don’t know about the status of DEX/PMEM.
Thanks, that’s much more helpful.
Honestly, after fairly extensive experience with using Btrfs during 4 years of running openSUSE as my daily driver, the headline on the website tells you what you need to know:
“The COW filesystem for Linux that won’t eat your data”.
That’s it. That’s the story.
Btrfs is the most featureful FS in the Linux kernel but it is not reliable. I know lots of people are in deep denial about this, but it is not.
From the Btrfs docs:
From the SUSE docs:
You want a deep examination? Try the Ars one.
Btrfs is not trustworthy. If you fill the volume, it will corrupt. When it is corrupted, it cannot be fixed and you will lose the contents.
OpenSUSE uses a tool called “Snapper” to automatically take snapshots before software installation. This means progressively filling the root filesystem. Unless you manually manage this and manually prune them, it will fill up, and when it fills up, it is game over.
I have been using Linux since kernel 1.0 or so in 1995 (I suspect well over one hundred distros now, if you count major versions) and in the 20th century it was routine for Linux boxes to self-destruct. If the power went at a bad moment, boom, dead.
21st century Linux is different. It is solid now.
I used openSUSE for 4 years, and in that time, I had to format and reinstall my OS about half a dozen times or more, because Btrfs self-destructed. This was not a one-off experience.
Fedora doesn’t have any snapshotting functionality, and so it’s not using Btrfs for the thing that makes Btrfs special. It’s like a moped enjoying perfect engine reliability because you pedal everywhere and never start the engine.
(Why does it use it then? I think this is why: Fedora leans heavily on Flatpak and is increasing that. Flatpak is extremely inefficient and involves a lot of file duplication. Btrfs can compress that. That means it doesn’t look so bad.)
Btrfs self-corrupts on full volumes and it does not have a working repair tool.
That is not acceptable. Therefore Btrfs is not acceptable, and that’s without considering the issues around its RAID handling and so on.
Yeah. BTRFS is kind of shit. I was recently considering adding more storage to my system and found out that I could select between two allocation policies if my disks were mismatched (or if I added one without balancing). I can use the single profile and it will only write to the bigger disk until it is full (BTRFS’ allocation policy is most free space) or I can pick RAID0 and it will stripe across all disks until eventually it is only writing to a single disks. Both cases hotspotting.
bcachefs on the other hand has a simple policy that uses disks evenly and causes them to self-balance over time https://bcachefs-docs.readthedocs.io/en/latest/feat-multipledevices.html
I don’t get the complaint about
--repair. This is the same situation as fsck in all filesystems where it exists. If you have a corrupted filesystem, the best a tool can do is make clever guesses about the way to recover. Hopefully the guesses will be right, but they may be wrong. They’re being explicit about it.No, it’s not. Have you tried it?
Like I said, I have fairly extensive, repeated, personal experience of this.
Fsck is simple and it usually works. I have used it on ext2, ext3, ext4, hfs+, and probably on reiserfs, xfs, and others.
Usually it fixes some damage, maybe small, and afterwards your filesystem works again. It’s as safe or safer than
CHKDSKon the DOS/Windows family. It’s part of my routine maintenance.I have never ever seen Btrfs repair successfully fix any form of damage. None at all, not even once.
What I have seen is a healthy working volume, which I only checked out of curiosity, destroyed by it.
Don’t be misled. What the warning means is “DO NOT RUN THIS”. It will destroy the volume.
There is no way to repair a damaged Btrfs volume, even if the damage is trivial.
I’ve installed openSUSE with Btrfs a few years ago. It worked kind of well, until I’ve tried Android development; the VM was unusable (stuttering, low fps, low performance) if the VM image was stored on a Btrfs partition. When I moved the VM to an ext4 partition, everything was smooth again. I’ve tried to search for the issue on various forums, but couldn’t find any fix for this, so I’ve just resigned from openSUSE and Btrfs altogether, and didn’t look back on it ever again. I haven’t experienced any data losses but from what you write, this was just a matter of time. So it’s good I’ve resigned.
VMs are usable on btrfs, you just have to make sure the file or directory has copy-on-write disabled (
chattr +C).That’s interesting. I am not a developer but I run lots of VMs. I never had a problem with that.
It is 100% possible to run openSUSE on ext4, and it works perfectly. I have run both Leap and Tumbleweed like this.
Leap is very stable and a good config is
/on ext4 and/homeon XFS. That is pretty reliable and performant in my experience.With T’weed you are at the mercy of a bad update but it’s still doable.
One of the benefits over btrfs today is that bcachefs supports native encryption like ZFS does, meaning you don’t need to juggle LUKS + btrfs.
I think for the next while at least btrfs or ZFS will still be preferred but I suspect that bcachefs will be good enough that the (admittedly minor) effort of ensuring OpenZFS on Linux is working such dealing with lack of packages or kernel upgrades breaking things (hello ArchLinux dkms) will make it a success since it’s natively integrated into the kernel.
A big adopting point would be if container tooling such as containerd start to natively support bcachefs the way btrfs and ZFS are.
I think bcachefs is significantly better than ZFS or BTRFS due to flexibility. You can pick storage configuration per-directory, chance storage configuration at any time, stripe sizes change automatically as you add disks and mismatched disks are not a problem.
Overal bcachefs feels like you just add your disks, add your files and it more or less does the best possible thing given your placement constraints. ZFS placement is quite rigid and specified upfront and BTRFS placement is way too stupidly simple.
I wrote a bit more about this in a blag post a while ago: https://kevincox.ca/2023/06/10/bcachefs-attempt/#bcachefs and a bit more words about the features https://kevincox.ca/2023/06/10/bcachefs-attempt/#summary
Any cross platform support? Native encryption? Zfs send/receive type functionality? Encryption with discard support for SSDs?
Those are some of the major zfs features for me.
Ed: I see in a sibling comment that bcachefs support native encryption.
The only two platforms known to actually support zfs are FreeBSD and Linux. No other filesystem than FAT has a good chance at being writable from every major OS
ZFS is also the native filesystem for Illumos and other descendants of Solaris.
Well, there’s opensolaris of course. NetBSD?
Looks like MacOs is limping along, and windows label itself as “beta”.
https://openzfsonosx.org/forum/viewtopic.php?f=20&t=3743
https://github.com/openzfsonwindows/openzfs/releases
Can you explain the utility of ZFS send/receive? Specifically, what it offers that other tools do not.
I can know without further confirmation that the files at the other end are exactly what I was sending, and the transit was minimal. I also can then roll back on the remote server to before that send happened, in case I want to check something that was deleted during that sync.
I’m not sure there are “other tools”. Traditionally filesystems have dump/restore for backup/restore, there’s rsync for incremental sync and restic/borg/duplicity support encryption and authentication.
Zfs send/receive works zfs snapshots and native encryption to form a rich, yet simple, solution for sync and backup. It works well ssh for network transport - or eg USB media for local backup. It supports remote mounting (mount backup on backup host) if wanted.
Like most things zfs, it’s the combination of features that creates a powerful emergent whole.
From your link:
That isn’t currently the case anymore, you can add a third disk to a mirror setup and it’ll resilver just fine to have triple copies of everything.
Though the RAID still applies (2.2 adds DRAID, which does let you expand IIRC, at massive complexity cost and has issues still, RAID Expansion is still in the pipes).
Did you make a typo? The requirement is 3 disks and 2x redundancy. I’ll admit that I didn’t look into the configuration too closely but this seemed difficult to do with ZFS. Especially if you were starting from 2 disks and wanted to live migrate to 3.
My understanding is that at the end of the day you configure your performance/redundancy into a virtual block device. You then put a more-or-less single-disk filesystem on top of that. (It is smarter than your average single-disk filesystem because it is aware of the topology but IIUC the behaviour is basically the same). You can grow by adding more disks to the virtual block device but you don’t really have much option to change the existing devices in the pool. (As you said there are some limited options such as 2 disks replicated can become N disks replicated).
bcachefs (and BTRFS for this case) are fairly flexible about this. They see that you want 2 copies and they pick two disks and put the data there. Adding more disks gives them more options to pick from. With bcachefs you can get even more flexible and say “everything in this directory shall have 4 copies” and it will replicate that data more times (well it won’t proactively replicate existing data IIUC, it will replicate new writes or on a balance operation) even if that filesystem previously had no configuration with 4x redundancy.
No typo, all data can be saved thrice.
ZFS’ internal structures are messy, but essentially there is two types of vdev’s; RAID and Mirror.
A non-mirror vdev (ie, just a plain disk, no redundancy), is simply a special case of a RAID1 with no partners (though internally it simply presents the device directly as pool member).
Mirror vdevs can be brought up and down in redundancy by adding or removing disks. If your pool is only mirror disks, you can even split it in half and have two pools with identical data. You can also remove and add mirror vdevs at your pleasure. Even convert single-disks into a mirror configuration.
RAID vdevs are much more strict. They cannot be expanded once created, they can’t be split and cannot be removed from a pool. This is due to the internal block pointers being in a special format to handle the RAID stripes.
The RAID Expansion work doesn’t even change that, it instead allows you to rewrite a RAID vdev in a way where it has more members than before. The old data will still only exist at the previous redundancy level (ie, same parity-data ratio) and only new data will benefit from the better parity to data ratio.
A mistake in understanding then?
The objective stated was “a filesystem that has 3 disks, and all data has 2 replicas”. Number of data replicas < number of disks. This saves 50% of your storage space if your failure budget only requires 2 replicas.
I apologize for that massive delay, but you can set this up too, though I think OP meant “2 replicas + data”. But the ZFS solution is to use copies=2 and have no mirrors. In that case, ZFS stores two copies of data on different disks, as long as this is possible. Though the caution here is that ZFS will not warn you if the copies have to be put on the same disk and existing data is not replicated. And ZFS does warn that in case of a single disk failure, it may loose an unspecified write (ie, non-replicated write will be rolled back if the only copy was on the dead disk).
Why would anyone be worried about the ZFS license? Oracle’s never sued anyone over IP they acquired from Sun have they?
You would have to ask them, but a bunch of people do complain about it.
Now that this comment the thread has helped me learn more about bcachefs, I don’t think the licensing was a huge deal. The bigger motivations are: bcachefs devs think they can do much better than ZFS or BTRFS; and also ZFS will never be first class on Linux (because Torvalds don’t want it in the kernel, etc)
I think that was a facetious comment; there’s certainly been at least one very high profile instance of Oracle doing exactly that.
Makes sense, thanks. I had forgotten about that case (generally not very interested in licensing disputes).
Eh, Torvalds doesn’t ship binaries. It’s companies like RedHat/IBM, Amazon, Google, Samsung and Microsoft that ship kernel binaries and wouldn’t want legal exposure.
Canonical paid for & published a legal opinion and shipped binaries including ZFS for quite a while.
OpenZFS not being in the Linux kernel has the social effect that the kernel devs don’t maintain compatibility with it, which has resulted in occasional breakages, which puts a maintenance burden on distributors (which is maybe why Canonical are dropping/mothballing their ZFS interface code?).
Canonical, a small UK company, is not the same scale of legal target as big US companies like IBM, Google, Microsoft and Amazon. Some of them won’t even ship GPLv3 code let alone mixed GPLv2 & CDDL. I am not a lawyer, but I’ve been close enough to corporate lawyers IP discussions to expect that they prioritize covering their asses from huge law suits over enabling a few long tail filesystem features (from their perspective at least)…
Some features bcachefs might have that ZFS lacks are defragmentation and “resizing flexibility” (links are to previous Lobsters comments).
Thanks, those links led me to the bcachefs manual, and that answered my questions.
https://bcachefs.org/bcachefs-principles-of-operation.pdf
Their value proposition is btrfs or ZFS but better, and especially faster. They think they can achieve it by using a simpler architecture inspired by databases.
That’s my rough understanding as well. Though I don’t think it’s supposed to be a direct clone of everything btrfs does — more that there’s been lots of promising filesystem ideas since ext3/4, and btrfs and bcachefs are drawing from that same pool.
I’m not 100% up to speed on why btrfs failed to take off. Is it just that it had issues with RAID not working/being unsafe?
As a relatively casual observer, my understanding is that:
Even in server and workstation applications, most usecases don’t need the advanced features of btrfs or it’s equivalent in ZFS, making the overheads not worthwhile for some users, and in some devices (embedded, phones, etc) it was not feasible as an EXT4 replacement.
COW filesystems tend to impose IO overheads that make them undesirable
Some of btrfs’s flagship features excepting COW can be replicated by existing features available to Linux users; XFS permits
differential snapshotsincremental backup of filesystems which can approximate snapshotting, and LVM2 allows existing filesystems to get features like block device spanning and online partition resizing.Red Hat was the big driver of btrfs adoption, but they abruptly decided to abandon support for btrfs (even as an optional OS) in RHEL starting from RHEL 8, and Fedora consequently only uses btrfs on Workstation, not Server. My understanding is they were sponsoring a lot of the btrfs work, too.
For those who truly did need all of the options btrfs provided, ZFS provided more, didn’t have a reputation for corruption and unstability that btrfs had (probably unfairly after a few years), and had a lot more materials and documentation on management and deployment.
Red Hat in the end decided to create Stratis, a userland program which replicates some btrfs functionality as a management layer over XFS and LVM2, though I’m not sure how widely this is adopted relative to btrfs.
In my experience this reputation is not unfair at all when it comes to BTRFS’ advanced features. I’ve been running BTRFS in RAID5/6 from 2016 to beginning of 2023 and have experienced data corruption roughly every year in that period. Can’t complain though, BTRFS tools all warn you about RAID5/6 being completely broken every time you run them.
I don’t get it. Your setup was unsupported and documented as such with warnings. Why do you think it’s fair to say it’s unstable?
It’s like randomly moving some files in the system directories after clicking “yes, I know this will cause issues” and saying the system is possible to corrupt and unstable… You’re really on your own in that situation.
Which is a pretty good demonstration of why one might want to build a new filesystem vs btrfs… IMO RAID5 is a pretty fundamental feature for a modern filesystem, both ZFS and Linux md have had it working fine since forever, and and it’s been well-known to be entirely broken in btrfs for… what, seven years or more?
Coincidentally, the author apparently started work on bcachefs in 2015.
Because it’s unstable regardless of whether the unstableness is documented or not. Why would it be unfair to say something that is factually true? (This is a genuine question, I am wondering if I am lacking a subtlety in my understanding of the English language.)
If you’re doing something explicitly unsupported, then it shouldn’t affect how the system is described. It’s like spraying your computer with water or plugging it into 300V and saying it’s unstable because it crashes - the user guide did tell you not to do that. My phone has an option to enable some experimental GPU features which actually crash when starting games - I can’t say my phone is unstable, just because I explicitly enabled that myself. It’s a option we’re not preventing from taking, but doing those things only says something about us, not about the system.
You can’t say the phone is unstable, but you can say it’s unstable when you enable experimental GPU features, the same way I’m not saying BTRFS is unstable, I’m saying BTRFS is unstable when using is advanced features. I really don’t understand where our disconnect comes from.
Because it’s not unstable when using advanced features. Advanced features like compression, snapshots, many raid levels works just fine. It’s unstable when using experimental unstable features. And pointing that out is not meaningful, because it’s the same for an experimental unstable feature of any project by definition.
I’ve had issues with snapshots and other raid levels too unfortunately (although less frequently than with RAID5/6).
Why don’t you run a less buggy RAID setup?
I was just having fun playing with BTRFS, I only used the raid for data I did not care about losing :)
Heh, fair enough. Hope you are doing well!
And a decent name.
bcachefssounds like something Facebook would run on their CDN nodes.Side note: sweet buddha, phoronix is full of ad cancer.
In fact, AIUI, FB is a big Btrfs user.
But then its data is highly volatile anyway.
In the ActivityPub community I encountered a dev that treats URLs as opaque strings as it pertains to internal things to his application.
I can’t fathom trying to decouple the logic that paths, query params and everything else allows in favour of something so bland as a string identifier.
That’s not far off the right thing to do. A URL is three parts:
Often, the best thing to do is treat the whole thing as an opaque token and pass it onto a different layer that understand the schema and can do the right thing.
Huge “it depends”:
data:,jar:,blob:, andabout:also do not require a double slash. Butchrome:andresource:mostly do. I guess it’s fair to say that everything after the colon is specific to the URL scheme.Another common one is
tel:If anything, I wonder why the double slash was introduced at all by some schemes. Even in
file:URLs it seems like a single slash should suffice and inhttp:it doesn’t add any value at all AFAICT. Does anyone happen to know the rationale behind it?Tim Berners Lee said “It seemed like a good idea at the time”.
Less pithily:
https://www.internethalloffame.org/2012/06/06/berners-lee-world-finally-realizes-web-belongs-no-one/
I also remember reading a quote that Tim just liked the look of it, but I can’t find any source for that, so maybe it’s a false memory or I heard it in person at a lecture he gave.
Yes, in Domain, local paths were Unix style
/like/this, remote systems paths were like//host/usr/bin/whatever, and you could specify the protocol likeprotocol://whateverhost/file/path.Windows UNC paths also came from Domain.
Hah, thanks, now it all makes sense!
I don’t know the original rationale, but the double slash makes possible protocol-relative URLs.
I’m sure theres a better link to describe them, but heres what I found with a moment’s googling: https://www.paulirish.com/2010/the-protocol-relative-url/
If HTTP(S) URLs didn’t have a double slash, and were of the form a
http:server/pathorhttps:server/path, then a colon would also suffice for that; i.e.:server/pathcould be treated as protocol relative.Double slash means it’s a “hierarchical URI” which implies certain things about the structure (host, path) vs “non hierarchical” which is just schemeSpecificPart
That’s even more counterexamples, thanks! I’d previously used :// to try to spot things that are probably URLs in text, and few of those are likely to appear in human-authored text, but it’s quite depressing that there are so many exceptions to a fairly simple rule.
Actually, the // is part of the “stuff” as well. See mailto schema URIs
Now, if we use the concept of URLs as being web addresses, all the stuff has a known interpretation in the http schema, just different naming as highlighted (but also maybe different interpretation in an application, like using fragment to define an anchor or a path in an SPA website)
Ugh, that makes them even more horrible than I thought. I’d never quite thought of
mailtoas a URL schema but it does seem to think it’s one. And it’s decades too late to fix it.mailtoisn’t the only one that does this, it’s common in URI schemes! a few off the top of my head:magnet:,turn:,javascript:,view-source:so it’s not really something to “fix” but just to learn i thinkStrictly speaking the
//denotes the start of what the URL specification calls an “authority” (i.e., a domain name).If the thing that comes after the
:in a specific URI scheme isn’t a domain name,//probably shouldn’t be there.There are some badly designed URI schemes whose designers did not appreciate this subtlety and erroneously require a
//followed by something that isn’t an authority/domain name.While the
//may seem unnecessary today it actually does relate to one feature of relative URLs, namely protocol-relative addressing (e.g.<script src="//example.com/foo.js"/>). Though I suppose you could still have that syntax work even without having a//in absolute URLs.There’s a little bit more to the authority part than just a domain name. The full syntax is,
The login part was never used much and is mostly deprecated now, for reasons that should be clear(text). The host part can be a network address or hostname; the network address does not have to be an IP address (it can be like
[weirdnet;…]) and the hostname does not have to be a domain name (it might come from hosts.txt or netbios, etc.).Indeed, my mistake.
To expand on the above for the benefit of others: after the use of colons in IPv6 addresses led to the need to wrap IPv6 addresses in brackets for use in URLs, a syntax was designed for other network addresses “just in case”. IPv7 does not seem like a very likely possibility, but if it were to happen, the syntax would be
https://[v7.FOO]//.I wonder if non-integer ports are also feasible? I could also imagine some implementations supporting /etc/services service name lookup in the port field. Then there’s UNIX domain sockets, something I’ve idly contemplated extending URL schemes to support. Maybe
https://[UNIX./var/lib/foo]/.As regards the
user:pass@thing, it is interesting to note that URI schemes divide into having two completely different uses for this kind of syntax. In HTTP it is for specifying the credential used to access a resource. But in other schemes likemailto:orxmpp:it is for specifying a user identity being named as a resource.Technically these are different syntaxes of course as the
mailto:andxmpp:schemes do not use the//authority syntax. But it is a distinction worth contemplating if we’re going to think deeply about URI design.Noooo!!!! /covers ears
You can still have arguments as path elements or query parameters too, just using template URLs like in for example OpenSearch. This way you don’t end up locked into a URL routing scheme that’s now been hardcoded into your API documentation and every client.
This is basically the right thing to do!
In general a URL is defined as nothing more than
Where scheme needs to follow certain rules, and schemepart needs to follow other rules governed by scheme, and the details of those rules (and their combinatorial requirements) are non-trivial! If you don’t need to interrogate a URL for details, treating it as an opaque string is a galaxy-brain maneuver, IMO.
Yeah, it’s a total mess. In CHICKEN, for this reason we split the eggs for parsing URLs into two: one “generic syntax” egg which simply parses the BNF from RFC 3986 (which is the successor to the one you linked) and one “common syntax” egg which deals with the specifics of HTTP and “suchlike” schemes. The “common syntax” egg also parses out the query string into key/value pairs, further percent-decodes path components after splitting them apart (because the insanity of the URI syntax means that for some schemes,
%2Fis identical to/, but for others you first need to split the path components and then decode%2Finto slashes “inside” each path component), etc.The solution isn’t very intuitively appealing, but it seems to work well enough.
Wow. The rare article that speaks to my exact situation: I use Cinnamon. I hope that Cinnamon one day supports Wayland. When it does, (poof) I will have switched to Wayland. Whatever their reasons, enough people find that bothersome, or unrelatable, such that I’ve learned to just say nothing in threads about Wayland.
I was hyped for the technical merits of Wayland in the 2010’s. I was excited when GNOME added support. I don’t wish to use GNOME, but I know it might lead to momentum in Wayland support so I’m all for it. I was glad to see KDE do so as well, though I don’t wish to use KDE. I even tried sway back in 2018 or 2019 because I had good experiences in tiling wms in the 00’s.
However, the way life has played out is that I now care more about working with my computer than working on my computer. I’m less of a productivity and environment explorer than I was ten years ago. Cinnamon works for me—works fabulously, in fact—and so I’m happy to be using whatever underlying technology makes it work so well.
This hasn’t happened yet. KDE’s Wayland support has always been mediocre and remains so at this time.
I actually find it interesting how many things we’re told are impossible on X actually work pretty ok in Qt applications.
I actually think much of wayland’s criticisms are more of gtk’s x implementation than of x itself.
It has improved markedly in recent releases, and has been my daily driver (even on my gaming PC) since early this year.
Wait, so you have a super beefy machine you can connect to remotely and you need 32 or preferably 64GB of memory? Why?
all of my knowledge is encoded in my open browser tabs
Me too. One of them contains a strategy for making my way through them, but I can’t seem to find it.
I have a weekly, recurring todo for “read all open browser tabs and save to pocket if needed”. Works, somewhat!
I quite like Firefox’s Tab Stash extension for that, where I periodically put all my deduped open tabs into an unnamed collection showing the date of creation.
I zap all my open tabs 2-3x a week. I can think of maybe one or two things I haven’t been able to find again over the past ~15 years. I’ve come to terms with lossy information retention.
I try to do it once a day. I’ve come to realise that if it’s important and worth caring about (it almost always isn’t), I’ll either remember it or be able to find it in my browser history.
How to destroy all your tabs in 3 steps:
That’s it. All my tabs, gone, and I never knew how to recover them. I use Firefox on Windows, and it happens to me about once or twice a year.
Edit: sometimes I get a warning about closing a window with multiple tabs opened. I believe the reason for this warning is because there is no “undo” on this: if I go through the data is gone for good.
In Firefox, the History menu should list recently closed tabs and windows. For a whole window, Ctrl/Cmd-Shift-N should revive it.
If that fails, there should be several backups of your recent sessions in your Firefox profile directory. Check the
sessionstore.jsfile and other files in thesessionstore-backupsdirectory.Double-check that you have session restore enabled in Firefox settings (“Open previous windows and tabs”).
That’s very good to know, thanks.
Maybe Ctrl-Shift-N ? I have used this to recover an accidentally closed Firefox window.
Ctrl+Shift+T (reopen closed tab) should also reopen the closed window!
I believe that kind of thing shortcut only work when my browser still has one window open, when I realise my mistake right away. But I don’t:
And even then I’m not entirely sure step (4) is actually required.
This window should be in history > recently closed windows, ready to be resurrected.
If closing the last window closes the entire application, it sounds like you need to switch to MacOS!
I’ll never support Apple. They popularised the idea that it’s okay not to own the computer you bought, and for this they will get my undying hatred.
Have you tried the history?
I have. It shows the history, most notably the most recently opened pages. I haven’t found anything relating to tabs there, especially what I wanted: recently closed tabs.
I think look again then, because there is a menu item called exactly “Recently closed tabs” in the history menu.
When I get too many, I click the OneTab button to save and close them all. Then (much) later triage at my leisure.
So, the BIOS is stored on the card itself, right? I wonder when this will be used for malware persistence; that sounds like a pretty hard thing to catch.
Is it even possible to flash the BIOS back without letting the on-GPU firmware run?
That’s the basis of the argument for why you want hardware to only load+run signed and verified binaries.
Which is fine, provided customers have the option of being able to add trusted certs so as to run their own modded binaries.
The fact that customers aren’t given the option tells you what this sort of “security” functionality is really about: manufacturer control over the devices they sell, after they’ve sold them.
No.
If a customer can install their own certificates, then so can malware.
Having secure boot while supporting non-manufacturer software is hard, and requires a lot of engineering effort (marcan had a long diatribe about this on the arm Macs, and seemed generally positive on the design, but as I understand it all the device firmware is loaded by iboot prior to OS startup, and is apple signed top to bottom).
Obviously it doesn’t help when you have stuff along the lines of what Nvidia is apparently doing - where the bios is setting a max frequency based on the buckets Nvidia wants to sell things at :-/
This is, indeed, the hard problem and it depends a lot on your threat model. For something like a graphics card, it’s generally fine to say that people with access to the card are out of scope of the threat model, because that requires opening the case and so you can have a jumper (or even just a button or switch) that allows provisioning a new certificate. That’s harder if your threat model includes someone with temporary access to the machine (for example, if someone manages to join the cleaning crew for the building - which many companies outsource - and can have 10 minutes to poke a desktop and install malware). It’s even worse when the machine is accessible only by the attacker. The last case is our threat model for confidential cloud computing: the attacker is the cloud provider and they have the computer in their data centre, the defender is the customer and they want to get strong security guarantees.
There are a few PCIe specs that help here. IDE (not to be confused with IDE, which is totally unrelated) gives you an end-to-end encrypted communication channel to a device. TDISP (ADISP in earlier versions, possibly a newer name in the most recent ones because naming is hard) lets the device authenticate an endpoint as a software entity (for example, to tell the difference between two TDX or SEV-SNP VMs on the host) and lets the code on the host get an attestation over the code in the device. This lets you defer the decision on whether you trust the device’s firmware until the device is attached (the flows here are somewhat complex because there are two levels of trust: do you trust this device to be in the machine at all, and do you trust it to be given access to your VM?).
That doesn’t help with anything that’s in the core TCB for the system though, such as CPU microcode, where you can’t run any code until it’s loaded and so the best that you can hope for is a remote attestation quote that you can verify on a different computer and decide whether to release keys to the booted system. Generally the approach here is to make it easily auditable: if the cloud vendor does something to compromise these layers of the system, either you can easily tell that you’re running a malicious version because it’s different from what everyone else is running, or they deploy the same malware to all of their customers, ensuring that enough people have standing to sue them that they will go out of business if they are actually malicious. Though any sufficiently advanced incompetence is indistinguishable from malice, so proving malice may be hard.
For local devices, typically the best solution is to provide a provisioning step that can be done by the user once. This installs a new key that is stored somewhere else (a cheap HSM like a U2F device is ideal) that must be used to sign any new key. This, at least, requires an attacker with physical access to steal the separate device. You can also do a lighweight version of this by requiring a PIN on something like a TPM that can do rate limiting on retries.
Are we not talking about Integrated Development Environments or not talking about Integrated Drive Electronics?
Yes.
The typical design for this is to ship it read-only, writeable if you move a jumper. Chromebooks regularly had this capability, usually with an extra screw as the jumper. Is it that hard?
Yes, it is. Because it’s often not immediately obvious that it has been tampered with.
The same goes for other kinds of hardware tampering, though, and if someone has physical access you can’t really prevent that. The jumper seems like a pragmatic solution.
The whole point of a lot of apple’s various peering mechanisms is specifically to ensure that having physical access to the device does not provide a silent compromise of the device.
Ah, that makes sense, different threat model then. As I recall the Chromebook would always show a scary warning at boot if the boot stuff had been modified.
Any functionality which precludes me from running the firmware I want is malware.
Restrictive code signing practices are not acceptable.
You are free to run whatever hardware you want, but saying that people who disagree with you are wrong, and any other model is “malware” implies a fairly abusive model of view of what choice users should have.
The pre-code signing model was devices had their control software burned into rom, and the only way you could get bug fixes was new hardware.
“hardware” devices have been running embedded software for decades, and no one complains about not being able to install their own software on those systems.
For secure systems your options are code signing, or the firmware is burned into an unfixable, unimprovable rom.
“You’re free to run whatever hardware you want” is a fake choice, all the pragmatically useful hardware vendors behave the same.
Also, yes, we do complain about not being able to run our own software on all sorts of embedded devices in exactly the same way, this is why “blob free” linux is a holy grail. (See also, phone basebands)
If your idea of a secure system is inherently aligned with a manufacturer that has opaque motives and is subject to coercion from governments and shareholders, it’s very different from mine.
The old “blob” free hardware people think of still had binary blobs, they were just burned into ROMs that were just as opaque, and just as unmodifiable, but also couldn’t receive updates (fixes, features, etc).
There has not been “blob” free hardware since the 70s
A noble goal. What alternatives are there?
Is this a real question?
Like, has this kind of behavior from vendors gotten so ingrained in people that a system with open documentation and at least source-available blobs/firmware has actually become impossible to imagine?
If I was to want to purchase such a system, where do I go and how much does it cost?
This is a goalpost move, the fact that it doesn’t exist doesn’t mean it can’t or shouldn’t. You didn’t ask me where to get such a thing, you asked what an alternative to not-it is.
When legal and economic problems become technical ones, why is it always the technology that has to change for the worse?
Oh, sure. But it is possible; and it’s possible to do it securely. Heck, if you wanted you could do it physically - put the certs on a socketed chip or something like a Yubikey.
Which is sort of my point - I don’t believe security is the primary motivation here, or there would be an option for users to use their own certs.
Not really. The issue isn’t that the card runs unsigned firmware, but that unsigned firmware can persist through reboots. It’d be absolutely fine if the card let you upload custom firmware, but only into the RAM.
So will removing the heat sink if you want to clean it or reapply thermal paste. But my card is from ASRock which has basically nonexistent warranty anyway.
I really doubt it would void a statutory warranty.
I just memorized
-czvfand-xzvf, create and extract. If I want to do something else to a file, unless I’m desperate I just extract it and do that stuff.After I had already memorised it, I heard someone refer to
-xzvfas “extract ze various files”I was introduced to
xzvfas xtract ze vucking files.I’ve learnt since that the
z(gzip) andv(verbose) aren’t really necessary, and you can probably get away withxf.On GNU tar that is. Not necessarily the case on the BSDs, at least I don’t think it is on OpenBSD.
create/extract
Zipped
Verbose
(From/to) file
You don’t even need
zwhen extracting, since moderntars will autodetect the compression type.I can’t say I’ve had problems remembering tar’s arguments, but
lngets me every time.You might find
cp -sandcp -luseful.The argument order for
lnis the same as forcpormv.That’s how I remember it. Write the arguments as if I’m making a copy of the file, but use
lninstead ofcp.The problem: I don’t use
lnvery often, so even though the argument order is consistent, I second-guess myself and pull out the manual anyway.My trick: consider, if you had to provide just one argument, which is essential. That comes first.
a bit crude but
ln <real> <fake>seems to work for me.Maybe I’m looking for something that isn’t there, but I don’t get it — is this a mnemonic based off the word “crude”?
no not really, crude because
real / fakecan be (mis)interpreted in varying contexts (you can use your imagination) .for, me, i prefer real to fake and hence,
ln <real> <fake>I also get tripped up but at the same time I try to remember source, then destination. And since I’m almost always doing soft links
-sreminds me that source is first.Ah! Easy! The link goes from symlink A to file B, so the symlink name is the source and comes first right?
Fair point :P. I think source as the source file, the actual file. And destination as the symlink
ln: last -> new
These “like open source but you can’t compete with us” licenses leave a real bad taste in my mouth. I don’t care for the (x)GPL(n), but those guys walk the walk. The licenses are clear and match the rhetoric. They want to convert you, and they would prefer you use other peoples’ stuff if you don’t want to share. These new licenses are…. sleazy is the best word I can think of. I don’t think it’s going to achieve the results they’re hoping for.
I suspect ten years from now, most of the companies who’re attempting this sort of thing (and haven’t folded) will have moved to a more traditional commercial license, or ended those products. The companies they’re trying to fight off with them will be still around doing the same thing.
It’s kind of funny because “like open source but you can’t compete with us” is how I view most AGPL projects, particularly when it’s backed by a commercial entity.
If Hashicorp had taken the AGPL route here, they would still be able to do literally whatever they want with the code - including using it in combination with proprietary extensions/modules (because they own it), but no-one else would have that ability.
I’m not saying I love the BSL, but it’s barely worse than the AGPL IMO.
AGPL doesn’t prevent anyone from using the software to compete with the authors. There’s a huge difference; namely the difference between being open source and being proprietary.
“You may do anything with the software as long as you release the source code” is not at all similar to “you may do anything with the software as long as you don’t compete with us”.
AGPL had that effect in practice, which is why a lot of companies adopted it. If your competitors couldn’t or wouldn’t share and license their changes or broader codebases alike, requiring they do so in order to build on your software was functionally equivalent to saying they couldn’t build on your software.
That behavior of AGPL began to break down in popular perception after the public AWS-MongoDB brouhaha. AWS argued they could offer MongoDB without sharing and licensing code they didn’t want to share and license. Mongo wrote and adopted a new variant of AGPL that more clearly required it.
I don’t mind that copyleft licenses give their copyright holders a competitive advantage. I am in no way opposed to that. I’m opposed to licenses which masquerade as open source while making it illegal to compete with the copyright holder.
It prevents them from offering extensions/whatever that are specific to them - but the project “owners” have no such limitations.
This is my point: with agpl a hypothetical hashicorp could release terraform++ which is the same agpl core plus a proprietary extension, under a proprietary license.
No contributors can do the same.
That is what I meant when I said they “can’t compete”. Anything they want to offer they have to make available upstream too, but the reverse is not true.
I mean, that’s broadly true of any copyleft licence. The licence doesn’t privilege the “project owner”, copyright law does. So I feel like that’s a bit of a red herring (although I won’t pass up the opportunity to say that I think your enemy here should be CLAs, which are what allow corporate-owned projects to accept community contributions and retain most or all of the privileges of a sole copyright holder).
Ownership privilege thing aside, you’re more or less claiming that the AGPL’s “you must free everything if you modify” requirement is so severe as to be equivalent to forbidding modification. But that is—if you allow that providing access to a SaaS product to someone in 2023 is the same kind of transaction as shipping software to them 20 years ago—a substantially weaker restriction than its pre-SaaS antecedent in GPL, which required you to free everything you provided to your customer anyway, even if you were just using it unmodified.
(Sufficiently strong) copyleft reduces the set of people who can build a given project into a proprietary product. Ideally it’d reduce it to zero, but in a world with relatively little copylefted software and lots of CLAs, there’s often one particular entity that still enjoys that option. I think that:
Yeah, but only in the case where a single corporation dominates the development. Once it’s a combined effort, AGPL locks them mutually in. Especially if they encourage outside individual contributions to block relicensing even more thoroughly.
It’s the best instrument to ensure the project stays libre despite sharehoarders. Not a perfect one.
You say only, but isn’t that actually the most common situtation?
It depends somewhat. There are also variations on the AGPL, including requiring copyright assignment or a CLA that allows the main player to produce things that include proprietary extensions but doesn’t permit the same for other people.
The main thing that companies care about is freedom from lock in. The traditional way of avoiding lock in is to have a second source (IBM, famously, did this with every single component in the IBM PC except the OS because they thought operating systems were commodities, which directly led to Microsoft’s dominance of the desktop market). When you have a second source, you can always switch to them if the primary supplier changes their terms to something that you don’t like (puts up the price too much, removes features that you care about, and so on).
In theory, the AGPL guarantees that a second source can exist if there is a market for one. If you use an AGPL’d service from vendor X and they put up their price to more than you think it’s worth, vendor Y can come along and sell precisely the same thing for the cost that you are willing to pay. As long as that’s a price that’s sufficient to maintain the service, that’s fine.
The down side here is that, if vendor X is paying for the maintenance, it is cheaper for vendor Y to provide the service as long as vendor X stays in business. AGPL can lead to the kind of failure mode where vendor X pays all of the costs, vendor Y gets most of the profit, vendor X goes out of business, vendor Y then can’t keep providing the service and needs to either take vendor X’s place (and then gets pushed out by vendor Z doing the same thing) or just watches the service degrade until customers go away. This is basically the same case as the tragedy of the commons.
Bit of a tangent, but it would probably be a good idea to find a different term for this because the “tragedy of the commons” – as originally described – has some issues. For example:
https://aeon.co/essays/the-tragedy-of-the-commons-is-a-false-and-dangerous-myth
And this is not some lone crank with an axe to grind – Ostrom received the Nobel in economics in 2009 “for her analysis of economic governance, especially the commons”.
Also, in general I know a lot of tech people (not necessarily any individual in this thread, just a general trend) like to sneer at social sciences, but Ostrom and others have done a lot of work on successful approaches to managing commons, and the FOSS world could stand to learn from that.
From a game theory perspective, “tragedy of the commons” doesn’t necessarily mean a situation where you’re doomed to ruin everything; it specifically means a situation which is lose/lose IF each individual actor plays in their own self-interest without cooperation. I don’t think that’s an inaccurate description of the situation; it’s basically saying you can’t win without some agreement to cooperate. The fallacy (which I agree needs to be debunked) is that cooperation is impossible.
Except there’s Amazon. Amazon doesn’t want vendor X do stay in business at all. And they have the money to sell at a loss until X goes bankrupt, and they don’t even need to close the source.
That just shows that current price is not a good enough signal for sustainable planning.
Imagine inheriting an orchard. You need to replant portion of the trees every year to keep it producing indefinitely. If you don’t, you immediately save some money, the trees will keep producing for some time into the future and in effect you make more than you’d make if you took proper care and replanted. After all it takes many years for new trees to start producing and you only have so much land.
At some point society will be forced to acknowledge that simplistic methods like lowest-price tendering does not work, because it is transitively extractive. Tragedy of commons would be hard to come about in a rural community. If you overgrazed, you would lose social status and people would be hesitant to lend you a helping hand, prompting you to compensate. Everyone would intuitively understand that if there was no common grass, it would be a problem. It’s just that our current society has decided to mostly prohibit non-financial forms of planning, because they are hard to check for fraudulent behavior.
It’s not AGPL, it’s the threat of jail time for a CFO who green lights a purchase of the costlier of two offerings while hiring programmers for twice the price to do it in-house would be perfectly fine.
Except that there are multiple examples of rural communities experiencing tragedy of the commons. They are no more immune than any other community. The only effective remedy I know of is an authority enforcing rules.
Oh, I am not saying explicit rules are not better. Just that while chasing “good rules” we might have somehow forgotten that “good” does not always equate with “most easily enforceable”.
Does it matter, though? I would say that adoption for the purpose of virtue signalling masking adversarial conduct is orthogonal to the adoption for the purpose of protecting user and developer interests. The license is not to blame for what people use it for.
All it takes is a single determined person at the right moment to force the organization to publish it under AGPL and accept outside contributions and increasingly the virality of the license will make it a part of the commons.
An example of project that would be extremely to close now is NextCloud. It does accept external contributions, does not ask for CLA and assumes AGPL. Every new contributor is yet another party who would have to agree to a license change.
From a practical perspective, the AGPL is a non-starter. And not just because of evil freedom-hating corporations.
Suppose that Carol is a true and fervent lover of Free Software and of software freedom. Every morning, she open-palm slams a copy of the GNU Manifesto, etc.
And suppose that Carol runs an online forum for other Free Software enthusiasts like herself. She ensures that only copyleft-licensed software is used to run it, and publishes not just the full source (under a copyleft license, of course), but also all the configuration, scripts, dev tooling, etc., because she believes so deeply in the ethos of Free Software that she goes above and beyond what is required of her by the licenses.
And suppose that Carol’s forum has two significant dependencies:
PackageAwhich is written by Alice, andPackageBwhich is written by Bob. And suppose Alice gets a bit burned out as a maintainer, and hands over commit access toPackageAto Carol, knowing it will be in good hands.Now, for sake of argument, suppose that
PackageAis quite old and was originally released by Alice under GPLv2 without an “or any later version” clause. AndPackageBis newer, and was released by Bob under GPLv3.And then one day Bob relicenses
PackageBto AGPL.And then not long after, Carol finds a security issue in
PackageB. Carol can report this to Bob, and submit a patch to Bob, but until Bob has incorporated the patch into his tree, it is illegal for Carol to incorporate the fix into her own running copy on her site, because she would be modifyingPackageBand triggering a requirement to release all the AGPL-defined “Corresponding Source” of her forum under the AGPL. And Carol does not have the legal right to relicensePackageA. The AGPL imposes more restrictions/passes on fewer freedoms than the GPLv2, so it would be a GPLv2 violation to distributePackageAunder AGPL; she does not have the ability to do an “intermediate” relicensing to GPLv3 (which makes an exception to the no-further-restrictions clause specifically to allow AGPL); and she is not the copyright holder toPackageA, so she has no other avenue for legally relicensing.In other words, the AGPL can create a situation where, despite the best of good will and good faith and professed belief in software freedom from all parties, and copyleft licenses on everything involved, it would still be illegal for someone to exercise the basic freedoms listed in the Free Software definition. And this is not a particularly far-fetched situation – there’s a lot of differently-licensed software out there, and even the FSF’s own licenses can accidentally create legal incompatibilities amongst themselves (as in the given example).
In this particular case, as long as Carol is not distributing appliances or containers that assemble both PackageA and PackageB into a single derivative work, I believe she would be OK just publishing the corresponding sources independently for any user to assemble themselves.
I am of the opinion that libraries should be licensed under MIT, Apache, MPL (or perhaps LGPL) since the incentive is then for everyone to contribute instead of doing the foundational work privately. Including otherwise proprietary businesses. It is the final applications that benefit from GPL / AGPL.
It was a tactical error on the part of Alice to use GPLv2-only license for a library, causing such hassle to downstream users.
It doesn’t take much to produce a situation where every individual component is 100% FSF-approved copyleft Free Software using FSF’s own licenses, but combining them makes it illegal to exercise one’s freedom, and the situation I gave is one such example.
And dismissing it by claiming that it was a “tactical error” to use what was, at the time, the FSF’s flagship recommended copyleft Free Software license is… well, it’s something. There’s a ton of GPLv2 code out there, and quite a lot of it doesn’t have the “or any later version” clause, which makes it inherently incompatible with the FSF’s newer and shinier toy.
I have seen this line of reasoning in the past. “FSF should have stuck with GPLv2.” I don’t think it adds much to the discussion.
I imagine people at FSF were not that happy about developers using GPLv2 in the GPLv2-only manner, leaving doors open to various exploits GPLv3 and AGPL addressed, blocking compatibility and damaging their reputation.
If Alice was actually listening to FSF, she would have used LGPL (a conservative choice) or GPL (more aggressively anti-proprietary), but in both cases “or any later version”. She though she was being smart when she were deleting those words from the license template. And now people who she would probably have considered legitimate users cannot use library.
So yeah, a tactical error. She should have chosen a permissive license or stuck with FSF who knew what the game actually is. And honestly, I can’t fault FSF for ignoring the compatibility issues. They are trying to defend the commons despite people who only see what’s directly in front of them. They probably count on social pressure to convince authors to update their GPLv2-only to GPLv2+.
It would be nice if she agreed to a license change.
Several times in the past I’ve pointed out the irony of the Free-Software-est of all the licenses being frequently used to enforce a decidedly un-Free-Software position: one of the major use cases of the AGPL is to ensure a company can produce proprietary add-ons and extensions while using copyright to ensure nobody else gets to compete with them on a level playing field.
That only works if nobody else is contributing to the product so there’s only one party with copyright.
Most companies do CLAs or equivalent which grant them the right to relicense contributions.
Well the AGPL means “you can use it but you can’t sell software for a living” as opposed to “you can use it but you have to host it forever and can never outsource it unless it’s to us”