It seems like the optimization is performed during the MIR -> LLVM pass.
I’m not familiar enough with the rustc codebase to find the optimization pass doing this. I’d love to read the code performing this fold at compile-time trick.
It looks to me like the llvm ir is dumped before lowering, but after optimising. If you compile in debug you’ll see that the IR is completely unoptimised.
And if you use the optimisation pipeline view of compiler explorer, you’ll see that all the tearing down is done by LLVM optimisation passes.
The inline pass actually just inlines the optimised fold (sum) in the caller, and may optimise it further. But to see the constant folding you need the optimisation trace for the callee.
What a ridiculous thing to have even happened in the first place, let alone refusing to acknowledge there could possibly be an issue for so long. I glad it’s been fixed but would make me think twice about using serde. I’m sure it’ll be fine, who’s ever heard of a security issue in a codec anyway?
Remember that there are real human being maintaining serde. It is not, in fact, blindingly obvious to all developers that the pre-compiled blobs were bad; on this site there were loud voices on both sides. Can you imagine suddenly getting caught in the crosshairs of angry developers like that? When I imagine it, it feels bad, and I’m liable to get defensive about it.
It may also have been a failed attempt at fixing something you’ve heard people complain about all the time, probably even about your code that slows down peoples builds (*). So yeah it was a bad idea in hindsight, but we don’t need more burned out maintainers from this. And I say this as someone who is openly disappointed by this happening.
(*) I’m not going to discuss how much time it actually saved.
Yeah, basically the biggest gains are offset by process creation being surprisingly slow. I’m working on a follow-up article where I talk about that in detail.
I posted your piece because it was the first one that explained in detail what the hell was going on, specifically how serde works. Looking forward to a followup.
That’s how it started, then they centralized everything with one team that doles out the “managed CI” offering, with their own global library and controls. Any competing infra gets flagged and audited hardcore until you give up by attrition.
This seems to only be checking the performance under –release. Most compilation is done without –release, meaning that most of the proc macro will not be optimized.
As someone who packages software, I think it’s worth noting that packagers expect different things than end users, though they are compatible.
One of my wishes is to avoid blobs from a vendor, since we can’t always recompile those in the build process to work with the architectures we support.
(The other big difference is the DESTDIR env var. End users don’t generally care, but it becomes essential when preparing a package)
I therefore understand those who support their end users, before getting packaged.
The real human being maintaining serde knew about the pushback that would happen and did it on purpose to prove a point in a pre-RFC he submitted. I don’t feel particularly bad about him getting pushback for using half the Rust ecosystem as his guinea pigs. (In fact I would like to see more of it.)
The real human being maintaining serde knew about the pushback that would happen and did it on purpose to prove a point in a pre-RFC he submitted.
What’s the reason to believe in this over any other explanation of the situation? E.g. that pushback was unexpected and that the RFC is the result of the pushback, rather than a cause?
I consider dtolnay a competent open source maintainer who understands the people who run his code well, and I would expect any competent open source maintainer to expect such pushback.
But how that necessary leads to “on purpose to prove a point”?
I don’t think dtolnay expected exactly zero pushback. But, given that some people in this thread argue quite a reasonable point that binaries are actually almost as fine as source, it is plausible that only bounded pushback was expected.
“Someone else is always auditing the code and will save me from anything bad in a macro before it would ever run on my machines.” (At one point serde_derive ran an untrusted binary for over 4 weeks across 12 releases before almost anyone became aware. This was plain-as-day code in the crate root; I am confident that professionally obfuscated malicious code would be undetected for years.)
I don’t see someone competent casually pushing such a controversial change, casually saying that this is now the only supported way to use serde, casually pushing a complete long pre-RFC that uses the controversial change to advance it, and then casually reverting the change in the span of a few days. That takes preparation and foresight.
I actually respect this move. It is exactly the kind of move I would do if I had goodwill to burn and was frustrated with the usual formal process, and it takes boldness and courage to pull it off the way he did it. I also think the pushback is entirely appropriate and the degree of it was quite mild.
Aha, thanks! I think that’s a coherent story to infer from this evidence (and I was wondering if there might be some missing bits I don’t know).
From where I stand, I wouldn’t say that this explanation looks completely implausible, but I do find it unlikely.
For me, the salient bits are:
what it says on the tin. dtolnay didn’t write a lot of responses in the discussion, but what they have written is more or less what I have expected to see from a superb maintainer acting in good faith.
there wasn’t any previous Wasm macro work that was stalled, and that required nefarious plans to get it unstuck.
really, literally everyone wants sandboxed Wasm proc macros. There can’t be any more support for this feature already. What is lacking is not motivation or support but (somewhat surprisingly) a written-down RFC for how to move forward and (expectedly) implementation effort to make it come true.
dtolnay likes doing crazy things! Like how all crates follow 1.0.x versions, or watt, or deref-based specialization in anyhow. So, “because I can” seems like enough motivation here.
“if you don’t like a feature in this crate, don’t use the crate or do the groundwork to make implementing this feature better” feels like a normal mode of operation for widely used OSS projects with sole maintainers. I’ve said as much with respect to MSRV of my once_cell crate.
I agree that there are multiple intepretations possible and that yours also follows from the evidence available. The reason I think it’s reasonable to consider something deeper to be going on is: every single Rust controversy I’ve discussed with key Rust people had a lot more going on than was there on the surface. Case in point: dtolnay was also the one thus far unnamed by anyone speaking for the project person who was involved in ThePHD’s talk being downgraded from a keynote. If I see someone acting surreptitiously in one case I will expect that to repeat.
O_o that’s news to me, thanks. It didn’t occur that dtopnay might have been involved there (IIRC, they aren’t a team lead of any top-level team, so I assume weren’t a member of the notorious leadership chat)
Calling me anonymous is pretty funny, considering I’ve called myself “whitequark” for close to 15 years at this point and shipped several world-class projects under it.
whitequark would be pretty well known to an old Rust team member such as matklad, having been one themself, so no, not anonymous… buut we don’t know this is the same whitequark, so yes, still anonymous.
I mean, I wrote both Rust language servers/IDEs that everyone is using and whitequark wrote the Ruby parser everyone is using (and also smaltcp). I think we know perfectly fine who we are talking with. One us might be secretly a Labrador in a trench coat, but that doesn’t have any bearing on the discussion, and speculation on that topic is hugely distasteful.
In terms of Rust team membership, I actually don’t know which team whitequark was on, but they are definitely on the alumni page right now. I was on the cargo team and TL for the IDE team.
You were talking about “Rust teams” and the only way I’ve seen that term used is to indicate those under the “Rust Project”. Neither person is on a Rust team or an alumni.
Tbh, it more reeks of desperation to make people’s badly configured CI flows faster. I think that a conspiratorial angle hasn’t been earned yet for this and that we should go for the most likely option: it was merely a desperate attempt to make unoptimized builds faster.
I think this is hard to justify when someone comes to you with a security issue, when your response is “fork it, not my problem”, and then closing the issue, completely dismissing the legitimate report. I understand humans are maintaining it, humans maintain all software I use in fact, and I’m not ok with deciding “Oh, a human was involved, I guess we should let security bad practices slide”. I, and I’m sure many others, are not frustrated because they didn’t understand the security implications, but because they were summarily dismissed and rejected, when they had dire implications for all their users. From my understanding, Serde is a) extremely popular in the Rust world, and b) deals in one of the most notoriously difficult kinds of code to secure, so seeing the developers’ reaction to a security issue is very worrying for the community as a whole.
The thing is, its not unambiguous whether this is a security issue. “Shipping precompiled binaries is not significantly more insecure than shipping source code” is an absolutely reasonable stance to have. I even think it is true if we consider only first-order effects and the current state of rust packaging&auditing.
Note also that concerns were not “completely dismissed”. Dismissal looks like “this is not a problem”. What was said was rather “fixing this problem is out of scope for the library, if you want to see it fixed, work on the underlying infrastructure”. Reflecting on my own behavior in this discussion, I might be overly sensitive here, but to me there’s a world of difference between a dismissal, and an acknowledgment with disagreement on priorities.
let alone refusing to acknowledge there could possibly be an issue for so long.
This is perhaps a reasonable take-away from all the internet discussions about the topic, but I don’t think this actually reflects what did happen.
The maintainer was responsive on the issue and they very clearly articulated that:
they are aware that the change makes it harder to build software for some users
they are aware that the change concerns some users from the security point of view
non-the-less, the change is an explicit design decision for the library
the way to solve this typical open-source dilemma is to allocate the work with the party needs the fruits of the work
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I do think it was a mistake to not predict that this change will be this controversial (or to proceed with controversial change without preliminary checks with wider community).
But, given that a mistake had been made, the handling of the situation was exemplary. Everything that needed fixing was fixed, promptly.
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I’m still waiting to hear what “security concern” there was here. Other language-package ecosystems have been shipping precompiled binaries in packages for years now; why is it such an apocalyptically awful thing in Rust and only Rust?
The main thing is loss of auditing ability — with the opaque binaries, you can not just look at the package tarbal from crates.io and read the source. It is debatable how important that is: in practice, as this very story demonstrates, few people look at the tarballs. OTOH, “can you look at tarballs” is an ecosystem-wide property — if we lose it, we won’t be able to put the toothpaste back into the tube.
This is amplified by the fact that this is build time code — people are in general happier with sandbox the final application, then with sandboxing the sprawling build infra.
With respect to other languages — of course! But also note how other languages are memory unsafe for decades…
The main thing is loss of auditing ability — with the opaque binaries, you can not just look at the package tarbal from crates.io and read the source.
It’s not that hard to verify the provenance of a binary. And it appears that for some time after serde switched to shipping the precompiled macros, exactly zero people actually were auditing it (based on how long it took for complaints to be registered about it).
OTOH, “can you look at tarballs” is an ecosystem-wide property — if we lose it, we won’t be able to put the toothpaste back into the tube.
The ecosystem having what boils down to a social preference for source-only does not imply that binary distributions are automatically/inherently a security issue.
With respect to other languages — of course! But also note how other languages are memory unsafe for decades…
My go-to example of a language that often ships precompiled binaries in packages is Python. Which is not exactly what I think of when I think “memory unsafe for decades”.
It’s not that hard to verify the provenance of a binary.
Verifying provenance and auditing source are orthogonal. If you have trusted provenance, you can skip auditing the source. If you audited the source, you don’t care about the provenance.
It’s a question which one is more practically important, but to weight this tradeoff, you need to acknowledge its existence.
This sounds like:
People say that they claim about auditing, and probably some people are, but it’s also clear that majority don’t actually audit source code. So they benefits of audits are vastly overstated, and we need to care about provenance and trusted publishing.
This doesn’t sound like:
There’s absolutely ZERO security benefits here whatsoever
I don’t know where your last two blockquotes came from, but they didn’t come from my comment that you were replying to, and I won’t waste my time arguing with words that have been put in my mouth by force.
That’s how I read your reply: as an absolute refusal to acknowledge that source auditing is a thing, rather than as a nuanced comparison of auditing in theory vs auditing in practice.
It might not have been your intention to communicate that, but that was my take away from what’s actually written.
In the original github thread, someone went to great lengths to try to reproduce the shipped binary, and just couldn’t do it. So it is very reasonable to assume that either they had something in their build that differed from the environment used to build it, or that he binary was malicious, and without much deeper investigation, it’s nearly impossible to tell which is the answer. If it was trivial to reproduce to build with source code you could audit yourself, then there’s far less of a problem.
Rust doesn’t really do reproducible builds, though, so I’m not sure why people expected to be able to byte-for-byte reproduce this.
Also, other language-package ecosystems really have solved this problem – in the Python world, for example, PyPI supports a verifiable chain all the way from your source repo to the uploaded artifact. You don’t need byte-for-byte reproducibility when you have that.
I guesss I should clarify that in GP comment the problem is misalignment between maintainer’s and user’s view of the issue. This is a problem irrespective of ground truth value of security.
Maybe other language package ecosystems are also wrong to be distributing binaries, and have security concerns that are not being addressed because people in those ecosystems are not making as much of a fuss about it.
If there were some easy way to exploit the mere use of precompiled binaries, someone would have by now. The incentives to use such an exploit are just way too high not to.
Anecdotally, I almost always see Python malware packaged as source code. I think that could change at any time to compiled binaries fwiw, just a note.
I don’t think attackers choosing binary payloads would mean anything for anyone really. The fundamental problem isn’t solved by reproducible builds - those only help if someone is auditing the code.
The fundamental problem is that your package manager has near-arbitrary rights on your computer, and dev laptops tend to be very privileged at companies. I can likely go from ‘malicious build script’ to ‘production access’ in a few hours (if I’m being slow and sneaky) - that’s insane. Why does a build script have access to my ssh key files? To my various tokens? To my ~/.aws/ folder? Insane. There’s zero reason for those privileges to be handed out like that.
The real solution here is to minimize impact. I’m all for reproducible builds because I think they’re neat and whatever, sure, people can pretend that auditing is practical if that’s how they want to spend their time. But really the fundamental concept of “running arbitrary code as your user” is just broken, we should fix that ASAP.
The fundamental problem is that your package manager has near-arbitrary rights on your computer
Like I’ve pointed out to a couple people, this is actually a huge advantage for Python’s “binary” (.whl) package format, because its install process consists solely of unpacking the archive and moving files to their destinations. It’s the “source” format that can ship a setup.py running arbitrary code at install time. So telling pip to exclusively install from .whl (with --only-binary :all:) is generally a big security win for Python deployments.
(and I put “binary” in scare quotes because, for people who aren’t familiar with it, a Python .whl package isn’t required to contain compiled binaries; it’s just that the .whl format is the one that allows shipping those, as well as shipping ordinary Python source code files)
Anecdotally, I almost always see Python malware packaged as source code. I think that could change at any time to compiled binaries fwiw, just a note.
Agree. But that’s a different threat, it has nothing to do with altered binaries.
I don’t think attackers choosing binary payloads would mean anything for anyone really. The fundamental problem isn’t solved by reproducible builds - those only help if someone is auditing the code.
Code auditing is worthless if you’re not sure the binary you’re running on your machine has been produced from the source code you’ve audited. This source <=> binary mapping is precisely where source bootstrapping + reproducible builds are helping.
The real solution here is to minimize impact. I’m all for reproducible builds because I think they’re neat and whatever, sure, people can pretend that auditing is practical if that’s how they want to spend their time. But really the fundamental concept of “running arbitrary code as your user” is just broken, we should fix that ASAP.
This is a false dichotomy. I think we agree on the fact we want code audit + binary reproducibility + proper sandboxing.
Agree. But that’s a different threat, it has nothing to do with altered binaries.
Well, we disagree, because I think they’re identical in virtually every way.
Code auditing is worthless if you’re not sure the binary you’re running on your machine has been produced from the source code you’ve audited. This source <=> binary mapping is precisely where source bootstrapping + reproducible builds are helping.
I’m highly skeptical of the value behind code auditing to begin with, so anything that relies on auditing to have value is already something I’m side eyeing hard tbh.
I think we agree on the fact we want code audit + binary reproducibility + proper sandboxing.
I think where we disagree on the weights. I barely care about binary reproducibility, I frankly don’t think code auditing is practical, and I think sandboxing is by far the most important, cost effective measure to improve security and directly address the issues.
I am familiar with the concept of reproducible builds. Also, as far as I’m aware, Rust’s current tooling is incapable of producing reproducible binaries.
And in theory there are many attack vectors that might be present in any form of software distribution, whether source or binary.
What I’m looking for here is someone who will step up and identify a specific security vulnerability that they believe actually existed in serde when it was shipping precompiled macros, but that did not exist when it was shipping those same macros in source form. “Someone could compromise the maintainer or the project infrastructure”, for example, doesn’t qualify there, because both source and binary distributions can be affected by such a compromise.
Aren’t there links in the original github issue to exactly this being done in the NPM and some other ecosystem? Yes this is a security problem, and yes it has been exploited in the real world.
What I’m looking for here is someone who will step up and identify a specific security vulnerability that they believe actually existed in serde when it was shipping precompiled macros, but that did not exist when it was shipping those same macros in source form. “Someone could compromise the maintainer or the project infrastructure”, for example, doesn’t qualify there, because both source and binary distributions can be affected by such a compromise.
If you have proof of an actual concrete vulnerability in serde of that nature, I invite you to show it.
The existence of an actual exploit is not necessary to be able to tell that something is a serious security concern. It’s like laying an AR-15 in the middle of the street and claiming there’s nothing wrong with it because no one has picked it up and shot someone with it. This is the opposite of a risk assessment, this is intentionally choosing to ignore clear risks.
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening. To me this is up with with Sony’s rootkit back in the day, completely unexpected, unauthorised behaviour that no reasonable person would expect, nor would they look out for it because it is just such an unreasonable thing to do to your users.
I think the point is that if precompiled macros are an AR-15 laying in the street, then source macros are an AR-15 with a clip next to it. It doesn’t make sense to raise the alarm about one but not the other.
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening.
I think this is extreme. No additional accessing of any kind was done. Binaries don’t have additional abilities that build.rs does not have. It’s not at all comparable to installing a rootkit. The precompiled macros did the same thing that the source macros did.
The existence of an actual exploit is not necessary to be able to tell that something is a serious security concern. It’s like laying an AR-15 in the middle of the street and claiming there’s nothing wrong with it because no one has picked it up and shot someone with it. This is the opposite of a risk assessment, this is intentionally choosing to ignore clear risks.
Once again, other language package ecosystems routinely ship precompiled binaries. Why have those languages not suffered the extreme consequences you seem to believe inevitably follow from shipping binaries?
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening.
Even the most extreme prosecutors in the US never dreamed of taking laws like CFAA this far.
To me this is up with with Sony’s rootkit back in the day, completely unexpected, unauthorised behaviour that no reasonable person would expect, nor would they look out for it because it is just such an unreasonable thing to do to your users.
I think you should take a step back and consider what you’re actually advocating for here. For one thing, you’ve just invalidated the “without any warranty” part of every open-source software license, because you’re declaring that you expect and intend to legally enforce a rule on the author that the software will function in certain ways and not in others. And you’re also opening the door to even more, because it’s not that big a logical or legal leap from liability for a technical choice you dislike to liability for, say, an accidental bug.
The author of serde didn’t take over your computer, or try to. All that happened was serde started shipping a precompiled form of something you were going to compile anyway, much as other language package managers already do and have done for years. You seem to strongly dislike that, but dislike does not make something a security vulnerability and certainly does not make it a literal crime.
I think that what actually is happening in other language ecosystems is that while there are precompiled binaries sihpped along some installation methods, for other installation methods those are happening by source.
So you still have binary distribution for people who want that, and you have the source distribution for others.
I have not confirmed this but I believe that this might be the case for Python packages hosted on debian repos, for example. Packages on PyPI tend to have source distributions along with compiled ones, and the debian repos go and build packages themselves based off of their stuff rather than relying on the package developers’ compiled output.
When I release a Python library, I provide the source and a binary. A linux package repo maintainer could build the source code rather than using my built binary. If they do that, then the thing they “need to trust” is the source code, and less trust is needed on myself (on top of extra benefits like source code access allowing them to fix things for their distribution mechanisms)
So you still have binary distribution for people who want that, and you have the source distribution for others.
I don’t know of anyone who actually wants the sdists from PyPI. Repackagers don’t go to PyPI, they go to the actual source repository. And a variety of people, including both me and a Python core developer, strongly recommend always invoking pip with the --only-binary :all: flag to force use of .whl packages, which have several benefits:
When combined with --require-hashes and --no-deps, you get as close to perfectly byte-for-byte reproducible installs as is possible with the standard Python packaging toolchain.
You will never accidentally compile, or try to compile, something at runtime.
You will never run any type of install-time scripting, since a .whl has no scripting hooks (as opposed to an sdist, which can run arbitrary code at install time via its setup.py).
I mean there are plenty of packages with actual native dependencies who don’t ship every permutation of platform/Python version wheel needed, and there the source distribution is available. Though I think that happens less and less since the number of big packages with native dependencies is relatively limited.
But the underlying point is that with an option of compiling everything “from source” available as an official thing from the project, downstream distributors do not have to do things like, say, confirm that the project’s vendored compiled binary is in fact compiled from the source being pointed at.
Install-time scripting is less of an issue in this thought process (after all, import-time scripting is a thing that can totally happen!). It should feel a bit obvious that a bunch of source files is easier to look through to figure out issues rather than “oh this part is provided by this pre-built binary”, at least it does to me.
I’m not arguing against binary distributions, just think that if you have only the binary distribution suddenly it’s a lot harder to answer a lot of questions.
But the underlying point is that with an option of compiling everything “from source” available as an official thing from the project, downstream distributors do not have to do things like, say, confirm that the project’s vendored compiled binary is in fact compiled from the source being pointed at.
As far as I’m aware, it was possible to build serde “from source” as a repackager. It did not produce a binary byte-for-byte identical to the one being shipped first-party, but as I understand it producing a byte-for-byte identical binary is not something Rust’s current tooling would have supported anyway. In other words, the only sense in which “binary only” was true was for installing from crates.io.
So any arguments predicated on “you have only the binary distribution” don’t hold up.
Hmm, I felt like I read repackagers specifically say that the binary was a problem (I think it was more the fact that standard tooling didn’t allow for both worlds to exist). But this is all a bit moot anyways
I don’t know of anyone who actually wants the sdists from PyPI.
It’s a useful fallback when there are no precompiled binaries available for your specific OS/Arch/Python version combination. For example when pip installing from a ARM Mac there are still cases where precompiled binaries are not available, there were a lot more closer to the M1 release.
When I say I don’t know of anyone who wants the sdist, read as “I don’t know anyone who, if a wheel were available for their target platform, would then proceed to explicitly choose an sdist over that wheel”.
Also, not for nothing, most of the discussion has just been assuming that “binary blob = inherent automatic security vulnerability” without really describing just what the alleged vulnerability is. When one person asserts existence of a thing (such as a security vulnerability) and another person doubts that existence, the burden of proof is on the person asserting existence, but it’s also perfectly valid for the doubter to point to prominent examples of use of binary blobs which have not been exploited despite widespread deployment and use, as evidence in favor of “not an inherent automatic security vulnerability”
Yeah, this dynamic has been infuriating. In what threat model is downloading source code from the internet and executing it different from downloading compiled code from the internet and executing it? The threat is the “from the internet” part, which you can address by:
Hash-pinning the artifacts, or
Copying them to a local repository (and locking down internet access).
Anyone with concerns about this serde change should already be doing one or both of these things, which also happen to make builds faster and more reliable (convenient!).
Yeah, hashed/pinned dependency trees have been around forever in other languages, along with tooling to automate their creation and maintenance. It doesn’t matter at that point whether the artifact is a precompiled binary, because you know it’s the artifact you expected to get (and have hopefully pre-vetted).
Downloading source code from the internet gives you the possibility to audit it, downloading a binary makes this nearly impossible without whipping out a disassembler and hoping that if it is malicious, they haven’t done anything to obfuscate that in the compiled binary. There is a “these languages are turing complete, therefore they are equivalent” argument to be made, but I’d rather read Rust than assembly to understand behaviour.
The point is that if there were some easy way to exploit the mere use of precompiled binaries, the wide use of precompiled binaries in other languages would have been widely exploited already. Therefore it is much less likely that the mere presence of a precompiled binary in a package is inherently a security vulnerability.
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I’m confused about this point. Is anyone going to fix crates.io so this can’t happen again?
Assuming that this is a security problem (which I’m not interested in arguing about), it seems like the vulnerability is in the packaging infrastructure, and serde just happened to exploit that vulnerability for a benign purpose. It doesn’t go away just because serde decides to stop exploiting it.
I don’t think it’s an easy problem to fix: ultimately, package registry is just a storage for files, and you can’t control what users put there.
There’s an issue open about sanitizing permission bits of the downloaded files (which feels like a good thing to do irrespective of security), but that’s going to be a minor speed bump at most, as you can always just copy the file over with the executable bit.
A proper fix here would be fully sandboxed builds, but:
POSIX doesn’t have “make a sandbox” API, so implementing isolation in a nice way is hard.
There’s a bunch of implementation work needed to allow use-cases that currently escape the sandbox (wasm proc macros and metabuild).
I’m sure it’ll be fine, who’s ever heard of a security issue in a codec anyway?
Who’s ever heard of a security issue caused by a precompiled binary shipping in a dependency? Like, maybe it’s happened a few times? I can think of one incident where a binary was doing analytics, not outright malware, but that’s it.
I’m confused at the idea that if we narrow the scope to “a precompiled binary dependency” we somehow invalidate the risk. Since apparently “curl $FOO > sh” is a perfectly cromulent way to install things these days among some communities, in my world (30+ year infosec wonk) we really don’t get to split hairs over ‘binary v. source’ or even ‘target v. dependency’.
I’m not sure I get your point. You brought up codec vulns, which are irrelevant to the binary vs source discussion. I brought that back to the actual threat, which is an attack that requires a precompiled binary vs source code. I’ve only seen (in my admittedly only 10 Years of infosec work) such an attack one time, and it was hardly an attack and instead just shady monetization.
This is the first comment I’ve made in this thread, so I didn’t bring up codecs. Sorry if that impacts your downplaying supply chain attacks, something I actually was commenting on.
Ah, then forget what I said about “you” saying that. I didn’t check who had commented initially.
As for downplaying supply chain attacks, not at all. I consider them to be a massive problem and I’ve actively advocated for sandboxed build processes, having even spoken with rustc devs about the topic.
What I’m downplaying is the made up issue that a compiled binary is significantly different from source code for the threat of “malicious dependency”.
So not only do you not pay attention enough to see who said what, you knee-jerk responded without paying attention to what I did say. Maybe in another 10 years…
Because I can curl $FOO > foo.sh; vi foo.sh then can choose to chmod +x foo.sh; ./foo.sh. I can’t do that with an arbitrary binary from the internet without whipping out Ghidra and hoping my RE skills are good enough to spot malicious code. I might also miss it in some downloaded Rust or shell code, but the chances are significantly lower than in the binary. Particularly when the attempt from people in the original issue thread to reproduce the binary failed, so no one knows what’s in it.
No one, other than these widely publicisedinstances in NPM, as well as PyPi and Ruby, as pointed out in the original github issue. I guess each language community needs to rediscover basic security issues on their own, long live NIH.
I hadn’t dived into them, they were brought up in the original thread, and shipping binaries in those languages (other than python with wheels) is not really common (but would be equally problematic). But point taken, shouldn’t trust sources without verifying them (how meta).
But the question here is “Does a binary make a difference vs source code?” and if you’re saying “well history shows us that attackers like binaries more” and then history does not show that, you can see my issue right?
But what’s more, even if attackers did use binaries more, would we care? Maybe, but it depends on why. If it’s because binaries are so radically unauditable, and source code is so vigilitantly audited, ok sure. But I’m realllly doubtful that that would be the reason.
So if I’m reading this article correctly, the point of this is to remove more potential sources of nondeterminism from Nix. Have there been any demonstrated benefits so far, or is this still all theoretical/robustness/WIP?
It’s mostly about running nix-built OpenGL/Cuda binaries on a foreign distribution (Ubuntu, Fedora, Debian…). You need a way to inject some sort of GPU driver to the Nix closure. You won’t be able to run a nix-built OpenGL program on a foreign distribution if you don’t do so.
NixGLHost is an alternative* approach to do this.
* Alternative to NixGL. NixGLHost is in a very very alpha stage.
One of my gripes with nixgl is that i have to run all my nix applications via nixgl. If I run a non-nix binary with nixgl it usually doesn’t go well, so i can’t run my whole user session with nixgl and have it propagate to child processes.
Is there any, for example, NIX_LD_PRELOAD one could use, that could be set system-wide, that is ignored by non-nix binaries?
To be honest, that’s not a use case I had in mind when exploring this problem space. I’d probably need more than 5 minutes to correctly think about this, take what I’m about to say with a grain of salt.
My gut instinct is that we probably don’t want to globally mix the GPU Nix closure with the host one. I guess an easy non-solution for this would be to skip the problem altogether by provisioning the Nix binaries through a Nix shell. In this Nix shell, you could safely discard the host library paths and inject the nix-specific GPU libraries directly through the LD_LIBRARY_PATH (via nix-gl or nix-gl-host).
Now, if you think about it more, the use case you’re describing seems valid UX-wise. I’m not sure what would be the best way to tackle it. The main danger is getting your libraries mixed up. NIX_LD_PRELOAD could be a nice trick, but it’s kind of a shotgun approach, you end up preloading your shim for each and every Nix program, regardless if they depend on OpenGL or not.
As long as you don’t plan to use CUDA, I think the best approach would be injecting the GPU DSOs from libglvnd. There’s already all you need to point to your EGL DSOs through the __EGL_VENDOR_LIBRARY_DIRS env variable. There’s no handy way to do that for GLX, but I wrote a small patch you could re-use to do so.
I’ll try to think more about that, cool use case, thanks for the feedback.
How would replacing bash with nushell play with bootstrapping of nix and nixpkgs? When comparing guix and nix, guix did quite a good job on that topic and there is really a minimal set of packages to build everything from scratch. I’m wondering if bringing Rust in, just to build nushell, just to build stdenv based on it, would make bootstrapping nearly impossible.
Adopting this as a default stdenv would require to push this massive bindist to the Nixpkgs bootstrap seed. That seed is already massive compared to what Guix has, I don’t think we want to degrade this further.
Rust is definitely source-bootstrapable, as a matter of fact, Guix manages to do it, there’s no reason we can’t do the same. The bootstrap chain is pretty long though. On top of what we already bootstrap (gcc, openssl, etc.), we’d need to bootstrap llvm, mrustc, then rust 1.54 -> 55 -> 56 -> 57 -> 58 -> 60 -> 61 -> 62 -> 63 -> 64 -> 65.
So yeah, pushing this to stdenv would considerably degrade the bootstrap story in any case.
From my perspective, Bash is a local optimum, I personally wouldn’t change it, it’s certainly a good balance between a language that is easy to bootstrap and a good-enough expressiveness to express builds.
If we really want to move to something more modern, Oil could be a more serious contender, they seem to take bootstrapping seriously. There’s a drafted RFC wrt. Oil adoption.
[Edit]: The question is eluded, but I don’t think the author expects this to replace stdenv, at least it’s not mentionned in the article. Don’t take this comment as an overwhelming negative “this project is worthless”. Cool hack!
This made me realize that Rust is fundamentally non-bootstrapable. It’s definitely going to produce a release every six weeks for quite a number of years, and Rust’s release N needs release N-1 to build, so the bootstrap chain, by design, grows quickly and linearly with time. So it seems that, in the limit, it is really a choice between:
Is there a reference interpreter, perhaps? I imagine that that can’t be a complete solution since LLVM is a dependency, but it would allow pure-Rust toolchains to periodically adjust their bootstraps, so to speak.
There is mrustc which is written in C++ and allows you to bootstrap Rust. There is also GCC Rust implementation in the works, that will allow bootstrapping.
In my defense, I do use the term “experimental” several times and I don’t make any suggestion of replacing stdenv. I could be wrong, but I think that flakes are going to decentralize the Nix ecosystem quite a bit. While the Nixpkgs/stdenv relationship is seemingly ironclad, I don’t see why orgs or subsets of the Nix community couldn’t adopt alternative builders. Any given Nix closure can in principle have N builders involved; they’re all just producing filesystem state after all.
As for bootstrapping, yes, the cost of something like Nushell/Nuenv is certainly higher than Bash, but it’s worth considering that (a) you don’t need things like curl, jq, and coreutils and (b) one could imagine using Rust feature flags to produce lighter-weight distributions of Nushell that cut out things that aren’t germane to a Nix build environment (like DataFrames support).
The resulting binary is about 1.3 MB now. I seem to recall that the nushell binary is something like 10 MB or 50 MB, which is typical for Rust binaries. rustc is probably much larger.
There was some debate about whether Oil’s code gen needs to be bootstrapped. That is possible, but lots of people didn’t seem to realize that the bash build in Nix is not.
It includes yacc’s generated output, which is less readable than most of Oil’s generated code.
Nushell is certainly larger! Although I would be curious how much smaller Nushell could be made. Given that it’s Rust, you could in principle use feature flags to bake in only those features that a Nix builder would be likely to use and leave out things like DataFrames support (which is quite likely the “heaviest” part of Nushell).
For sure it would make bootstrapping much harder on things like OpenBSD. Good luck if you are on an arch that doesn’t have rust or LLVM. That said, I don’t think this would replace stdenv.. for sure not any time soon!
Also the article does mention exactly what you are pointing out:
it’s nowhere near the range of systems that can run Bash.
My question is orthogonal to this, and maybe I should have specify what I mean by bootstrapping. It’s “how many things I have to build first, before I can have working nixpkgs and build things users ask for”. So if we assume that nushell runs wherever bash runs, how much more effort is to build nushell (and rust and llvm) than bash? I would guess order of magnitude more, thus really complicating the initial setup of nixpkgs (or at least getting them to install without any caches).
Are there any plans for moving Nix flakes from experimental to stable? I see that Nix flakes are all the hype now. Even the linked guide states that:
Anyone beginning their Nix journey today should center their learning and experimentation around Nix flakes.
Do you agree with this?
I’ve been in love with NixOS for some time now but I avoided experimental features so far.
The thing is, Flakes FOMO starts to creep in but I’d rather avoid putting (even more) time into learning Nix/NixOS/nixpkgs/nix-the-command if flakes are going to be deprecated next spring.
The thing is, Flakes FOMO starts to creep in but I’d rather avoid putting (even more) time into learning Nix/NixOS/nixpkgs/nix-the-command if flakes are going to be deprecated next spring.
For what it is worth, I can’t fathom a universe where Nix actually removes Flakes. The data I’m seeing and the user interviews I’ve done show an overwhelming number of new Nix users are starting with Flakes because they’re easier and it creates a framework to learn inside.
If there is some sort of redesign or flakes2, it wouldn’t kill flakes1.
However, they are still considered to be an unstable feature by the Nix main developer. The flakes code you write is considered unstable and could break without further notice after a Nix update.
I think all the knowledge you get without flakes would be still transferable to a flake-based setup. Flakes are nice for a few usecases. Most importantly, pinning and mixing packages from different sources. If you don’t have urgent needs related to these two things, you can postpone migrating to flakes.
Regarding experimental features, I find the new nix command something worth looking into. The CLI is much simpler and nicer.
Part of the issue is that the RFC committee doesn’t want to rubber-stamp flakes just because a lot of people, lead by a few influential people, are all using the same unofficial and experimental feature. It makes a mockery of the RFC process and skips over a thorough design review. If flakes are really the way forward, some of all this energy should go into getting the RFC officially pushed through.
That’s the first time I hear about work notes being some form of “temporal documentation”. It makes a lot of sense.
I however think the workflow described in this page creates a hard dependency towards a online service by hard-linking internet URLs to your immutable and long-living git history. These URLS tend to be very mutable by nature, it’s hard to predict whatever they’ll still be available or not on a long time scale. Any migration would require you to re-write your whole git history.
That’s fine for your small hobby project, but it’s likely to lead to a lot of context loss in a moderately distant future (say 15/20 years) for a semi-major project (like datasette).
The git notes UX is pretty bad, it’s a shame. They’d be a great way to store this kind of temporal documentation attached to a commit.
[Edit]: Scratch that last paragraph, git notes wouldn’t work in a context where you want to store your design before starting to work on the actual implementation.
If you’re going to use issues in the way I’m describing here you need to be VERY confident that your organization has a healthy enough respect for institutional knowledge that they’re not just going to throw it all away some day when they switch to a different issue tracking system!
One of the reasons I’ve gone all-in on GitHub Issues is that it has a really good API. I have code which exports my issues on a regular basis, both as a form of backup and as a way to run my own queries against them: https://github-to-sqlite.dogsheep.net
My plan for if I ever do work on a project that migrates to another issue tracker is to create a static HTML archive of the old issues and then run a script that rewrites the entire commit history so each commit links to the full URL to the archived issue.
Out of curiosity, did you write a package that uses the binary releases out of simplicity, or did you run into issues using the Go tooling in Nixpkgs to build from source? (If the latter, I’d love to hear about them.)
You can then pkgs.callPackage the above file somwhere in your nix config. Or better, if you feel like spending a bit more time on this: clean up your module and this derivation (add the relevant meta attributes) open a PR in Nixpkgs adding this derivation and your module.
In general, the entry point to find out this kind of information for a lanugage you’re not familiar with is the nixpkgs manual. In that particular case, here’s the relevant golang section: https://nixos.org/manual/nixpkgs/stable/#sec-language-go
But wait, isn’t there that one nonguix project that allows you to install a normal kernel and Steam?
Yeah, but talk about that in the main #guix channel and you risk getting banned. GG. You just have to know that it exists and you can’t learn that it exists without knowing someone that tells you that it exists under the table.
Has this actually happened? Getting banned for talking about nonguix?
Not sure about getting banned, per se, but it’s explicitly discouraged. The second paragraph of nonguix’s readme:
Please do NOT promote this repository on any official Guix
communication channels, such as their mailing lists or IRC channel, even in
response to support requests! This is to show respect for the Guix project’s
strict policy
against recommending nonfree software, and to avoid any unnecessary hostility.
I would recommend actually reading the help-guix archives to see how often support issues are created and how many issues users have are ignored or told they are out of place.
I admit I fucked up and misunderstood the rules. My complaint now reads:
Yeah, but talk about that in the main #guix channel and you get told to not talk about it. You just have to know that it exists and you can’t learn that it exists without knowing someone that tells you that it exists under the table, like some kind of underground software drug dealer giving you a hit of wifi card firmware. This means that knowledge of the nonguix project (which may contain tools that make it possible to use Guix at all) is hidden from users that may
need it because it allows users to install proprietary software. This limits user freedom from being able to use their computer how they want by making it a potentially untrustable underground software den instead of something that can be properly handled upstream without having to place trust in too many places.
Hehe, yeah, the FSF and SFC use outrage constantly! I get emails all the time telling me that Microsoft and Apple are teaming up to murder babies or whatever. It’s pretty much all they have left at this point, and I say this as someone who donated and generally supported their mission for many, many years (which is why I still get the emails).
When I wrote that bit I made the assumption that violating the rules of the channel could get you banned. I admit that it looks wrong in hindsight, so I am pushing a commit to amend it.
Sorry, I think my Private Cloud has a bad power supply which is having a knock-on effect of upsetting the NFS mounts on the webserver. I’m acquiring a replacement right now, and in the meantime I am going to Infrastructure-as-Code it by adding a cronjob that fixes the NFS mount.
I’ve been reading the Gemini specification, as well as this post, and my conclusion is that it’s just a worse version of HTTP 1.1 and a worse version of Markdown.
Strong disagree. TLS + HTTP 1.1 requires to perform a upgrade dance involving quite a few extra steps. The specification is also pretty empty regarding SNI management. Properly implementing that RFC is pretty demanding. There’s also a lot of blind spots left to the implementer better judgement.
In comparison, the Gemini TLS connection establishing flow is more direct and simpler to implement.
TLS aside, you mentioning
I’ve been reading the Gemini specification
sounds like a clear win to me. The baseline HTTP 1.1 RFC is already massive let alone all its required extensions to work in a modern environment.
I agree that the specification is simple to read, but the specification itself is too limited and don’t find it suitable for the real world.
For example, I prefer HTTP’s optional end-to-end encryption because when working with internal routers within an infrastructure dealing with certificates is a PITA and a completely unnecessary bump in complexity and performance overhead being inside an already secured network.
I also disagree on that “extensibility is generally a bad idea” as the article says. Extensibility can work if you do it properly, like any other thing on software engineering.
EDIT:
Or the requirement of closing the connection and re-opening it with every request, and all the handshakes that means.
For clarity about what I think could be an actual improvement: I would prefer an alternative evolution of HTTP 1.0, with proper readable specs, test suites, clearer https upgrade paths, etc; instead of an evolution of Gopher.
TLS + HTTP just requires connecting to port 443 with TLS. I’ve worked on lots of things using HTTP for >20 years and I don’t think I’ve ever seen the upgrade protocol used in real life. Is it commonly used by corporate proxies or something like that?
When I looked at it (months ago), I got the same impression. I find this article irresponsible, as Gemini does not merit the support.
Gemini’s intentions are good. The specification isn’t. For instance, not knowing the size of an object before receiving it makes it a non-starter for many of its intended purposes.
This is an idea that should be developed properly and openly, allowing for input from several technically capable individuals. Not one person’s pet project.
I’ll stick to Gopher until we can actually do this right. Gemini doesn’t have the technical merit to be seen as a possible replacement to Gopher.
It does accept input from individuals. I was able to convince the author to expand the number of status codes, to use client certificates (instead of username/password crap) and to use the full URL as a request.
Interesting, I’ve been porting nixos to the nano pi m4 v2, this looks quite a bit less painful in some ways. Added to my: try this out some day list of stuff to look at.
Conceptually speaking, Guix got a lot of things right. On the other hand, Nix precedes Guix, the opposite would have been concerning.
The documentation is amazing, they are very careful about their tooling vendor lock-in. Their clean API around the toolchains and the abstractions around derivations is the selling point to me. The language is also somehow standard and comes full batteries included tooling-wise.
There’s a catch however: the Guix packageset is much smaller, you won’t have all the nice language-specific build systems you have with Nix, overall you’re likely to miss some stuff packaging-wise. Also: no systemd (I guess it might be a selling point for some people though).
Yep yep, this just intrigued me as I’ve gotten a bit deep in the guts of how nixos sd images are built. Its honestly not too big of a deal its just probably in need of a bit of a refactor tbh for the overall function to do this stuff but it honestly just strikes me as more: this was evolved not planned. Which is fine, just not as polished as it could be.
The scheme bit made a lot more sense to me off the bat versus having to do a fair amount of digging to figure out how I can adjust the partition sizes and make sure that my custom u-boot and its spl files etc… are getting put in the “blessed” right spot for this board (still not sure i am doing it right tbh as its not booting).
And the systemd bit is water under the bridge to me, that ship has sailed. I’ve had to port/add custom derivations to nixpkgs a lot so i’m not too averse to that if needed.
My real reason for all this is i’m building a little k8s cluster out of arm boards for shits, so nixops is my ultimate goal here.
After using it for a while I started to find the Nix expression language as one of the best designed syntaxes ever. It doesn’t have separators for records or lists so it’s friendly to diff. The multiline strings are perfect. Writing nested records with “a.b.c” keys is super convenient. The lambda syntax is as simple as possible. Etc etc.
It doesn’t have separators for records or lists so it’s friendly to diff.
Records are separated with the ; symbol.
As for lists, I beg to disagree. The list symbols are separated with a whitespace, which is unfortunate since whitespace is also used to represent function application. It means you’ll have to be careful enough to wrap your function applications in parenthesis every time you’ll perform it in a list.
That’s a easy trap to fell into, especially in multi-lines statements. Add the lazy nature of the language on top of that, you can end up with stacktraces that are pretty difficult to decipher. Especially if you end up doing that in a NixOS machine description :/.
I see a lot of newcomers falling into this trap on IRC.
e.g:
let pkgs = [
htop
import ./local-pkgs.nix
ssh
]; in ...
Instead of
let pkgs = [
htop
(import ./local-pkgs.nix)
ssh
]; in ...
While it might be true that 1500 Bytes is now the de facto MTU standard on the Internet (minus whatever overhead you throw at it), everything’s not lost. The problem is not that we don’t have the link layer capabilities to offer larger MTUs, the problem is that the transport protocol has to be AWARE of it. One mechanism for finding out whether what size MTU is supported by a path over the Internet is an Algorithm called DPLPMTUD. It is currently being standardized by the IETF and is more or less complete https://tools.ietf.org/html/draft-ietf-tsvwg-datagram-plpmtud-14.
There are even plans for QUIC to implement this algorithm, so if we’ll end up with a transport that is widely deployed and also supports detection of MTUs > 1500 we’ll actually might have a chance to change the link layer defaults. Fun fact: All of the 4G networking gear actually supports jumbo frames, most of the providers just haven’t enabled the support for it since they are not aware of the issue.
I can hardly believe it… but if speedtest.net were able to send jumbo frames and most users’ browsers support receiving it, it might get deployed by ISPs as they look for benchmark karma. Amazing. I thought 1500 was as invariant as π.
I was maintainer for an AS at a previous job and set up a few BGP peers with jumbo frames (4470). I would have made this available on the customer links as well, except none of them would have been able to receive the frames. They were all configured for 1500, as is the default in any OS then or today. Many of their NICs couldn’t handle 4470 either, though I suppose that has improved now.
Even if a customer had configured their NIC to handle jumbo frames, they would have had problems with the other equipment on their local network. How do you change the MTU of your smartphone, your media box or your printer? If you set the MTU on your Ethernet interface to 4470 then your network stack is going to think it can send such large frames to any node on the same link. Path MTU discovery doesn’t fix this because there is no router in between that can send ICMP packets back to you, only L2 switches.
It is easy to test. Try to ping your gateway with ping -s 4000 192.168.0.1 (or whatever your gateway is). Then change your MTU with something like ip link set eth0 mtu 4470 and see if you can still ping your gateway. Remember to run ip link set eth0 mtu 1500 afterwards (or reboot).
I don’t think that DPLPMTUD will fix this situation and let everyone have jumbo frames. As a former network administrator reading the following paragraph, they are basically saying that jumbo frames would break my network in subtle and hard to diagnose ways:
A PL that does not acknowledge data reception (e.g., UDP and UDP-
Lite) is unable itself to detect when the packets that it sends are
discarded because their size is greater than the actual PMTU. These
PLs need to rely on an application protocol to detect this loss.
So you’re going to have people complain that their browser is working, but nothing else. I wouldn’t enable jumbo frames if DPLPMTUD was everything that was promised as a fix. That said, it looks like DPLPMTUD will be good for the Internet as a whole, but it does not really help the argument for jumbo frames.
And I don’t know if it has changed recently, but the main argument for jumbo frames at the time was actually that they would lead to fewer interrupts per second. There is some overhead per processed packet, but this has mostly been fixed in hardware now. The big routers use custom hardware that handles routing at wire speed and even consumer network cards have UDP and TCP segmentation offloading, and the drivers are not limited to one packet per interrupt. So it’s not that much of a problem anymore.
Would have been cool though and I really wanted to use it, just like I wanted to get us on the Mbone. But at least we got IPv6. Sorta. :)
If your system is set up with an mtu of 1500, then you’re already going to have to perform link mtu discovery to talk with anyone using PPPoE. Like, for example, my home DSL service.
Back when I tried running an email server on there, I actually did run into trouble with this, because some bank’s firewall blocked ICMP packets, so… I thought you’d like to know, neither of us used “jumbo” datagrams, but we still had MTU trouble, because their mail server tried to send 1500 octet packets and couldn’t detect that the DSL link couldn’t carry them. The connection timed out every time.
If your application can’t track a window of viable datagram sizes, then your application is simply wrong.
If your system is set up with an mtu of 1500, then you’re already going to have to perform link mtu discovery to talk with anyone using PPPoE. Like, for example, my home DSL service.
It’s even worse: in the current situation[1], your system’s MTU won’t matter at all. Most of the network operators are straight-up MSS-clamping your TCP packets downstream, effectively discarding your system’s MTU.
I’m very excited by this draft! Not only it will fix the UDP situation we currently have, but will also make tunneling connections way more easy. That said, it also means that if we want to benefit from that, the network administrators will need to quit mss-clamping. I suspect this to take quite some time :(
[1] PMTU won’t work in many cases. Currently, you need ICMP to perform a PMTU discovery, which is sadly filtered out by some poorly-configured endpoints. Try to ping netflix.com for instance ;)
If your system is set up with an mtu of 1500, then you’re already going to have to perform link mtu discovery to talk with anyone using PPPoE. Like, for example, my home DSL service.
Very true, one can’t assume an MTU of 1500 on the Internet. I disagree that it’s on the application to handle it:
If your application can’t track a window of viable datagram sizes, then your application is simply wrong.
The network stack is responsible for PMTUD, not the application. One can’t expect every application to track the datagram size on a TCP connection. Applications that use BSD sockets simply don’t do that, they send() and recv() and let the network stack figure out the datagram size. There’s nothing wrong with that. For UDP the situation is a little different, but IP can actually fragment large UDP datagrams and PTMUD works there too (unless, again, broken by bad configurations, hence DPLPMTUD).
I disagree that it’s on the application to handle it
Sure, fine. It’s the transport layer’s job to handle it. Just as long as it’s detected at the endpoints.
For UDP the situation is a little different, but IP can actually fragment large UDP datagrams and PTMUD works there too
It doesn’t seem like anyone likes IP fragmentation.
If you’re doing a teleconferencing app, or something similarly latency-sensitive, then you cannot afford the overhead of reconstructing fragmented packets; your whole purpose in using UDP was to avoid overhead.
If you’re building your own reliable transport layer, like uTP or QUIC, then you already have a sliding size window facility; IP fragmentation is just a redundant mechanism that adds overhead.
If you’re using TCP, or any of its clones, of course this ought to be handled by the underlying stack. They promised reliable delivery with some overhead, they should deliver on it. I kind of assumed that the “subtle breakage” that @weinholt was talking about was specifically for applications that used raw packets (like the given example of ping).
You list good reasons to avoid IP fragmentation with UDP and in practice people don’t use or advocate IP fragmentation for UDP. Broken PMTUD affects everyone… ever had an SSH session that works fine until you try to list a large directory? Chances are the packets were small enough to fit in the MTU until you listed that directory. As breakages go, that one’s not too hard to figure out. The nice thing about the suggested MTU discovery method is that it will not rely on other types of packets than those already used by the application, so it should be immune to the kind of operator who filters everything he does not understand. But it does mean some applications will need to help the network layer prevent breakage, so IMHO it doesn’t make jumbo frames more likely to become a thing. It’s also a band-aid on an otherwise broken configuration, so I think we’ll see more broken configurations in the future, with less arguments to use on the operators who can now point to how everything is “working”.
According to zmap it takes 45min to scan all of IPv4 on a Gigabit connection. That could be a slow but interesting way to reliably bootstrap the network in case of an attack.
The 45 mins scan advertised on the homepage is probably the result of a TCP SYN scan though. You’ll probably need to add an application layer scanner on top of that (zgrab?). Not sure how this will affect the overall latency of the scan :/
It’s advertising an open source project, Source Hut, but also Janet, Zig, Nim, Samurai, Sway and other open source projects I like. Projects that get very little payment or gratitude for the work they do.
Yes sr.ht is a service too, a useful one at that. They support BSD well, unlike other companies, how else are they supposed to let people know this fact? Should they be paying largely unethical companies like google for ad space? Or should they just be more subversive so people don’t complain.
Let me put it this way, if every open source project was also a business, should we hate on every single one for advertising? sr.ht didn’t game the upvotes to get on the front page, people upvoted it by themselves.
I suppose there could be a tag ‘sponsored’ so people can ignore them. Not suggesting allowing lower quality from sponsored content either, probably the inverse.
The issue is that I see a Sourcehut “ad” every few days: “Sourcehut supports OpenBSD”, “Sourcehut supports migrations from Bitbucket”, “Sourcehut supports ASCII”. Yeah … we got it … A lot of these posts don’t have a lot of meat to them and at this point, it’s just getting spammy.
Yeah … we got it … A lot of these posts don’t have a lot of meat to them and at this point, it’s just getting spammy.
They don’t always have a lot of “meat,” but posts about SourceHut represent a capitalist ideology I can actually get behind. A single proprietor, working their ass off to try to change the software world, which has gotten extremely out of hand with regards to complexity, and the marketing of products that fix the complex systems we don’t need, at all, to begin with.
What’s the difference between a SourceHut post, and an post ad that complains that as an open source author I am not compensated fairly? Hint: one should be inspiration, for the other is actually possible.
payment for the service is optional, so no it doesn’t. All the things that make Sourcehut great in my opinion are the ways in which it denies capitalist ideology. Open Source Software, optional payments, etc.
It’s optional, right now, while in Alpha. It doesn’t seem the plan is that forever. Also, if it wasn’t clear, I’m extremely in favor of this model of charging people for a service, but releasing your software under a permissive license.
Just let me other another data point here. It was thanks to the “migration from Bitbucket” post that I found out Sourcehut had a nifty script to help migrations from Bitbucket and that saved hours of work as I migrated 20+ repos effortlessly. This current post made me realize that maybe I should be paying more attention to their CI system as it looks much simpler than others I’ve used. So, in the end, I’m appreciating these blog posts a lot. Yes they are related to a commercial venture but so what? You can self-host it if you’re not into SaaS outside your control. If we set a hard line like this, then it becomes impossible to post about any commercial project at all. It is already hard to monetize FOSS projects to make them sustainable, now imagine if they are not even allowed blog posts…
Same here. This string of posts made me aware of sourcehut and when I had to migrate from bitbucket, I then gave them a hard eval. I like their human, non-shitty business model of “I give them money and they give me services”, and that their products are professionally executed and no-frills.
I don’t know how to reconcile it. These articles were very useful to me, when most product ads weren’t and I’d be disappointed if this site became a product advert platform. I think people are right for flagging it is almost-an-ad, but in this one vendor’s case I’m glad I saw them and am now a happy sourcehut customer.
But there is no other way to get large numbers of people to know about something, following your advice would be suicide.
All these conversations are done like it’s all or nothing. We allow politics/marketing/etc on Lobsters or… it never happens anywhere with massive damage to individuals and society. Realistically, this is a small site with few monetary opportunities for a SaaS charging as little as he does. If the goal is spreading the word, it’s best done on sites and platforms with large numbers of potential users and (especially) paying customers. Each act of spreading the word should maximize the number of people they reach for both societal impact and profit for sustainability.
Multiple rounds on Lobsters means, aside from the first announcement with much fan fare, the author sacrificed each time opportunities to reach new, larger audiences to show the same message again to the same small crowd. Repeating it here is the opposite of spreading the word. Especially since most here that like Sourcehut are probably already following it. Maybe even buying it. He’s preaching to the choir here more than most places.
Mind-killer or not, anyone talking about large-scale adoption of software, ideology, etc should be using proven tactics in the kinds of places that get those results. That’s what you were talking about, though. I figured he was just trying to show latest BSD-related progress on one of his favorite tech forums. More noise than signal simply because he was sharing excitement more than doing technical posts or focused marketing.
Every blog post is an ad for something. It may not be a product, directly, but it’s advertising an idea, the person, or persons the idea was thought by, the writing (which, btw can be a product) of the author, etc.
If you want to sincerely flag advertising, you might as well get offline—it’s pervasive.
it seems that an advertisement has a precise definition: an ad is directly related to a paid product, not an idea.
This is a fairly pedantic interpretation. A person promotes an idea to sell something, if even themselves. That “sale” might only come later in the form of a job offer, or support through Patreon, etc, etc.. But, to say that you can’t advertise an idea is wrong. The cigarette industry’s ad campaigns have always been about selling an image, an idea that if you smoke you become part of something bigger. Oh, and btw, you’ll probably remember the brand name, and buy that kind instead of something else.
iPods were sold on the very basis of white headphones, TO THE POINT, that people without iPods started wearing white headphones to be part of the “club.” Advertisements sell you the idea of a better life, and hopefully you’ll buy my product to get it.
You’re right, and how virtuous Sourcehut may or may not be doesn’t change that. The line between ad and article is a spectrum, but this seems to be pretty well into the ad side of things. I apologise, I’ll be more discerning in the future.
If you crack some other good places to get the word out, I’d be interested in hearing. My online circle is pretty small (lobste.rs and HN), but I’m working on something I want to ‘advertise’ the hell out of quite soon…
I’ve been trying to engage more with Reddit for this reason. I don’t really like it as a platform or see it as doing a social good, but there are users there and I’d like to be there to answer their questions. I was going to make a Twitter account, too, but they wanted my phone number and a pic of my ID and a blood sample to verify my account so I abandoned that. Finding good ways to ethically grow Sourcehut’s audience is not an entirely solved problem.
The reason Twitter – and many platforms – asks for phone numbers is because spam and trolls are a persistent problem. Ban one neo-Nazi troll tweeting obscenities at some black actor for DesTROyinG WhITe SocIEtY and they’ll create a new account faster than you can say “fuck off Nazi”.
Reddit is often toxic as hell by the way, so good luck with that.
While I also see it as an ad, I’m interested in what it being announced as a Sourcehut user. But it seems you don’t have a RSS/Atom feed for the official blog… Or is there a mailing list I missed?
Somewhat amusing that this post with an interesting fully FOSS service, is marked -29 spam, whereas an actual advertisement about Huawei making macbook clones that run Linux has only -3 spam (one of which is mine).
Said FOSS service has been on the Lobsters front page multiple times recently. I suspect the reaction is: “We get it, sr.ht exists and SirCmpwn is apparently desperate to attract a paying customerbase, but a clickbaity title for a blogspam ad on the usual suspect’s software is probably crossing the line.”
This is already happening with one specifically requiring mails to be sent from another Big Mailer Corp to hit the inbox, or requiring that senders be added to the contacts for others. Any other sender will hit spambox unconditionnally for a while before being eventually upgraded to inbox.
Anybody knows which bigcorp player he’s talking about?
My mailserver, for many months, could not send mails to outlook addresses. The outlook server replied “OK” but the mail was transparently discarded. Not inbox, not spam, not trash, nothing. As if the mail had never been sent.
I believe nowadays outlook “only” send my mails to spam.
I have had the same experience. With Gmail it was even more difficult to evade their hyper-aggressive spam filters.
I can’t call any of this “easy” and I had to struggle and learn a lot of new concepts (like DKIM, which is a pain to set up). It’s also very tricky to verify, if it fails it can fail silently; your mail is just dropped or goes to spam. I had that happen when my DNSsec signatures weren’t renewed, for example, and also when I had made a small mistake that made my DKIM invalid or not used (I don’t remember which).
You need to be an expert at mail before this stuff is “easy”. When you get redirected to the spamfolder, those hosts aren’t giving any information about why this happened, so you’re left guessing. Also, you sometimes don’t even know unless you’re in contact with the recipient in some other way than just e-mail (and sometimes people don’t bother to notify you that the mail got flagged as spam). There are tools out there that can help verify your technical setup like rDNS, SPF, DKIM etc. But it’s still annoying as fuck to get it set up. Once you’ve done the work, it basically runs itself though.
So I appreciate the article’s attempt to get more people to try hosting their own mail, I would say it’s quite one-sided and assumes a whole lot of technical sysadmin competency that the author has probably simply become blind to himself.
I had a similar problem and my solution was to route all mail to them via a separate, dedicated IP which didn’t suffer the same problem. A solution possible thanks to the flexibility of Exim. As much as these simpler MTAs seem attractive I wonder how they would cope with such scenarios.
I had this problem sending from my own mail server to Gmail addresses. After a couple of months I just gave up on my own mail server and went to mailbox.org
All these clean desks make me feel bad, but I feel like making a statement. Took the picture of my desk as I found it this morning.
https://imgur.com/a/gnVV3Sp
This is really impressive.
I played with that a bit locally. To be pedantic, it seems like this optimization is done by rustc itself, not LLVM.
If we dump the LLVM IR of the sum function, we get an already folded result:
However, if we look at the MIR stage, the function is not folded yet:
It seems like the optimization is performed during the MIR -> LLVM pass.
I’m not familiar enough with the rustc codebase to find the optimization pass doing this. I’d love to read the code performing this fold at compile-time trick.
It looks to me like the llvm ir is dumped before lowering, but after optimising. If you compile in debug you’ll see that the IR is completely unoptimised.
And if you use the optimisation pipeline view of compiler explorer, you’ll see that all the tearing down is done by LLVM optimisation passes.
Hmm, you’re right. Thanks for pointing me in the right direction.
It turns out there’s an option to dump the function LLVM-IR before all the optimization passes:
As you pointed out, we start out with a non-optimized IR, so indeed, the optimization is not done on the MIR:
There’s a lot of passes, I’m omitting a lot of those, but the main optimization is done through a “InlinerPass” optimization phase:
The inline pass actually just inlines the optimised fold (sum) in the caller, and may optimise it further. But to see the constant folding you need the optimisation trace for the callee.
Nice QUIC summary!
Fun to see this page being served by HTTP2 :)
What a ridiculous thing to have even happened in the first place, let alone refusing to acknowledge there could possibly be an issue for so long. I glad it’s been fixed but would make me think twice about using serde. I’m sure it’ll be fine, who’s ever heard of a security issue in a codec anyway?
Remember that there are real human being maintaining
serde. It is not, in fact, blindingly obvious to all developers that the pre-compiled blobs were bad; on this site there were loud voices on both sides. Can you imagine suddenly getting caught in the crosshairs of angry developers like that? When I imagine it, it feels bad, and I’m liable to get defensive about it.It may also have been a failed attempt at fixing something you’ve heard people complain about all the time, probably even about your code that slows down peoples builds (*). So yeah it was a bad idea in hindsight, but we don’t need more burned out maintainers from this. And I say this as someone who is openly disappointed by this happening.
(*) I’m not going to discuss how much time it actually saved.
This overview by @cadey implied it did not save much time at all, basically only if you were running a CI setup without a cache.
https://xeiaso.net/blog/serde-precompiled-stupid
Running a CI setup without a cache is, for better or for worse, very common
Yeah, basically the biggest gains are offset by process creation being surprisingly slow. I’m working on a follow-up article where I talk about that in detail.
I posted your piece because it was the first one that explained in detail what the hell was going on, specifically how serde works. Looking forward to a followup.
My workplace is way too big. This describes our CI setup. Only the blessed JVM gets to have cached CI builds.
This is how shadow IT begins. Has anyone started running/sharing their locally setup CI for your project yet?
That’s how it started, then they centralized everything with one team that doles out the “managed CI” offering, with their own global library and controls. Any competing infra gets flagged and audited hardcore until you give up by attrition.
This seems to only be checking the performance under –release. Most compilation is done without –release, meaning that most of the proc macro will not be optimized.
As someone who packages software, I think it’s worth noting that packagers expect different things than end users, though they are compatible.
One of my wishes is to avoid blobs from a vendor, since we can’t always recompile those in the build process to work with the architectures we support.
(The other big difference is the DESTDIR env var. End users don’t generally care, but it becomes essential when preparing a package)
I therefore understand those who support their end users, before getting packaged.
The real human being maintaining serde knew about the pushback that would happen and did it on purpose to prove a point in a pre-RFC he submitted. I don’t feel particularly bad about him getting pushback for using half the Rust ecosystem as his guinea pigs. (In fact I would like to see more of it.)
What’s the reason to believe in this over any other explanation of the situation? E.g. that pushback was unexpected and that the RFC is the result of the pushback, rather than a cause?
I consider dtolnay a competent open source maintainer who understands the people who run his code well, and I would expect any competent open source maintainer to expect such pushback.
But how that necessary leads to “on purpose to prove a point”?
I don’t think dtolnay expected exactly zero pushback. But, given that some people in this thread argue quite a reasonable point that binaries are actually almost as fine as source, it is plausible that only bounded pushback was expected.
The excerpt from the RFC is:
I don’t see someone competent casually pushing such a controversial change, casually saying that this is now the only supported way to use serde, casually pushing a complete long pre-RFC that uses the controversial change to advance it, and then casually reverting the change in the span of a few days. That takes preparation and foresight.
I actually respect this move. It is exactly the kind of move I would do if I had goodwill to burn and was frustrated with the usual formal process, and it takes boldness and courage to pull it off the way he did it. I also think the pushback is entirely appropriate and the degree of it was quite mild.
Aha, thanks! I think that’s a coherent story to infer from this evidence (and I was wondering if there might be some missing bits I don’t know).
From where I stand, I wouldn’t say that this explanation looks completely implausible, but I do find it unlikely.
For me, the salient bits are:
I agree that there are multiple intepretations possible and that yours also follows from the evidence available. The reason I think it’s reasonable to consider something deeper to be going on is: every single Rust controversy I’ve discussed with key Rust people had a lot more going on than was there on the surface. Case in point: dtolnay was also the one thus far unnamed by anyone speaking for the project person who was involved in ThePHD’s talk being downgraded from a keynote. If I see someone acting surreptitiously in one case I will expect that to repeat.
O_o that’s news to me, thanks. It didn’t occur that dtopnay might have been involved there (IIRC, they aren’t a team lead of any top-level team, so I assume weren’t a member of the notorious leadership chat)
Maybe take hearsay from an anonymous Internet catgirl with a grain of salt.
Calling me anonymous is pretty funny, considering I’ve called myself “whitequark” for close to 15 years at this point and shipped several world-class projects under it.
whitequark would be pretty well known to an old Rust team member such as matklad, having been one themself, so no, not anonymous… buut we don’t know this is the same whitequark, so yes, still anonymous.
Hm? Neither of them are Rust team members, unless they are represented under different names in the Project.
I mean, I wrote both Rust language servers/IDEs that everyone is using and whitequark wrote the Ruby parser everyone is using (and also smaltcp). I think we know perfectly fine who we are talking with. One us might be secretly a Labrador in a trench coat, but that doesn’t have any bearing on the discussion, and speculation on that topic is hugely distasteful.
In terms of Rust team membership, I actually don’t know which team whitequark was on, but they are definitely on the alumni page right now. I was on the cargo team and TL for the IDE team.
Thank you for all the context I was missing. Is it just oversight you aren’t on the alumni for those team pages?
Turns out there were at least two bugs in the teams repo with respect to me, thanks for pointing this out!
I’m glad my bickering had at least some positive outcome :)
Probably! I think https://github.com/rust-lang/team/commit/458c784dda91392b710d36661f440de40fdac316should have added me as one, not sure why that didn’t happen
I don’t know what you mean by “the Project”, but the source of truth for Rust team membership is https://github.com/rust-lang/team.
You were talking about “Rust teams” and the only way I’ve seen that term used is to indicate those under the “Rust Project”. Neither person is on a Rust team or an alumni.
https://www.rust-lang.org/governance
That is what I meant, yes. Those pages are generated from the Git repo I linked. Ctrl-F on https://www.rust-lang.org/governance/teams/compiler and https://www.rust-lang.org/governance/teams/alumni.
A find would tell you matklad was not on a team. He was “just” a contributor. No real data exists about whitequark.
Tbh, it more reeks of desperation to make people’s badly configured CI flows faster. I think that a conspiratorial angle hasn’t been earned yet for this and that we should go for the most likely option: it was merely a desperate attempt to make unoptimized builds faster.
I think this is hard to justify when someone comes to you with a security issue, when your response is “fork it, not my problem”, and then closing the issue, completely dismissing the legitimate report. I understand humans are maintaining it, humans maintain all software I use in fact, and I’m not ok with deciding “Oh, a human was involved, I guess we should let security bad practices slide”. I, and I’m sure many others, are not frustrated because they didn’t understand the security implications, but because they were summarily dismissed and rejected, when they had dire implications for all their users. From my understanding, Serde is a) extremely popular in the Rust world, and b) deals in one of the most notoriously difficult kinds of code to secure, so seeing the developers’ reaction to a security issue is very worrying for the community as a whole.
The thing is, its not unambiguous whether this is a security issue. “Shipping precompiled binaries is not significantly more insecure than shipping source code” is an absolutely reasonable stance to have. I even think it is true if we consider only first-order effects and the current state of rust packaging&auditing.
Note also that concerns were not “completely dismissed”. Dismissal looks like “this is not a problem”. What was said was rather “fixing this problem is out of scope for the library, if you want to see it fixed, work on the underlying infrastructure”. Reflecting on my own behavior in this discussion, I might be overly sensitive here, but to me there’s a world of difference between a dismissal, and an acknowledgment with disagreement on priorities.
This is perhaps a reasonable take-away from all the internet discussions about the topic, but I don’t think this actually reflects what did happen.
The maintainer was responsive on the issue and they very clearly articulated that:
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I do think it was a mistake to not predict that this change will be this controversial (or to proceed with controversial change without preliminary checks with wider community).
But, given that a mistake had been made, the handling of the situation was exemplary. Everything that needed fixing was fixed, promptly.
I’m still waiting to hear what “security concern” there was here. Other language-package ecosystems have been shipping precompiled binaries in packages for years now; why is it such an apocalyptically awful thing in Rust and only Rust?
The main thing is loss of auditing ability — with the opaque binaries, you can not just look at the package tarbal from crates.io and read the source. It is debatable how important that is: in practice, as this very story demonstrates, few people look at the tarballs. OTOH, “can you look at tarballs” is an ecosystem-wide property — if we lose it, we won’t be able to put the toothpaste back into the tube.
This is amplified by the fact that this is build time code — people are in general happier with sandbox the final application, then with sandboxing the sprawling build infra.
With respect to other languages — of course! But also note how other languages are memory unsafe for decades…
It’s not that hard to verify the provenance of a binary. And it appears that for some time after serde switched to shipping the precompiled macros, exactly zero people actually were auditing it (based on how long it took for complaints to be registered about it).
The ecosystem having what boils down to a social preference for source-only does not imply that binary distributions are automatically/inherently a security issue.
My go-to example of a language that often ships precompiled binaries in packages is Python. Which is not exactly what I think of when I think “memory unsafe for decades”.
Verifying provenance and auditing source are orthogonal. If you have trusted provenance, you can skip auditing the source. If you audited the source, you don’t care about the provenance.
It’s a question which one is more practically important, but to weight this tradeoff, you need to acknowledge its existence.
This sounds like:
This doesn’t sound like:
I don’t know where your last two blockquotes came from, but they didn’t come from my comment that you were replying to, and I won’t waste my time arguing with words that have been put in my mouth by force.
That’s how I read your reply: as an absolute refusal to acknowledge that source auditing is a thing, rather than as a nuanced comparison of auditing in theory vs auditing in practice.
It might not have been your intention to communicate that, but that was my take away from what’s actually written.
Once again, I don’t intend to waste my time arguing with someone who just puts words in my mouth.
In the original github thread, someone went to great lengths to try to reproduce the shipped binary, and just couldn’t do it. So it is very reasonable to assume that either they had something in their build that differed from the environment used to build it, or that he binary was malicious, and without much deeper investigation, it’s nearly impossible to tell which is the answer. If it was trivial to reproduce to build with source code you could audit yourself, then there’s far less of a problem.
Rust doesn’t really do reproducible builds, though, so I’m not sure why people expected to be able to byte-for-byte reproduce this.
Also, other language-package ecosystems really have solved this problem – in the Python world, for example, PyPI supports a verifiable chain all the way from your source repo to the uploaded artifact. You don’t need byte-for-byte reproducibility when you have that.
Ah yes, garbage collected languages are famously ‘memory unsafe for decades’
I guesss I should clarify that in GP comment the problem is misalignment between maintainer’s and user’s view of the issue. This is a problem irrespective of ground truth value of security.
Maybe other language package ecosystems are also wrong to be distributing binaries, and have security concerns that are not being addressed because people in those ecosystems are not making as much of a fuss about it.
If there were some easy way to exploit the mere use of precompiled binaries, someone would have by now. The incentives to use such an exploit are just way too high not to.
There are ways to exploit binary releases. It’s certainly not easy, but this has definitely been exploited in the wild.
You can read this page https://reproducible-builds.org/docs/buy-in/ to get a high-level history of the “reproducible build” (and bootstrapping) movement.
Anecdotally, I almost always see Python malware packaged as source code. I think that could change at any time to compiled binaries fwiw, just a note.
I don’t think attackers choosing binary payloads would mean anything for anyone really. The fundamental problem isn’t solved by reproducible builds - those only help if someone is auditing the code.
The fundamental problem is that your package manager has near-arbitrary rights on your computer, and dev laptops tend to be very privileged at companies. I can likely go from ‘malicious build script’ to ‘production access’ in a few hours (if I’m being slow and sneaky) - that’s insane. Why does a build script have access to my ssh key files? To my various tokens? To my ~/.aws/ folder? Insane. There’s zero reason for those privileges to be handed out like that.
The real solution here is to minimize impact. I’m all for reproducible builds because I think they’re neat and whatever, sure, people can pretend that auditing is practical if that’s how they want to spend their time. But really the fundamental concept of “running arbitrary code as your user” is just broken, we should fix that ASAP.
Like I’ve pointed out to a couple people, this is actually a huge advantage for Python’s “binary” (
.whl) package format, because its install process consists solely of unpacking the archive and moving files to their destinations. It’s the “source” format that can ship asetup.pyrunning arbitrary code at install time. So tellingpipto exclusively install from.whl(with--only-binary :all:) is generally a big security win for Python deployments.(and I put “binary” in scare quotes because, for people who aren’t familiar with it, a Python
.whlpackage isn’t required to contain compiled binaries; it’s just that the.whlformat is the one that allows shipping those, as well as shipping ordinary Python source code files)Agree. But that’s a different threat, it has nothing to do with altered binaries.
Code auditing is worthless if you’re not sure the binary you’re running on your machine has been produced from the source code you’ve audited. This source <=> binary mapping is precisely where source bootstrapping + reproducible builds are helping.
This is a false dichotomy. I think we agree on the fact we want code audit + binary reproducibility + proper sandboxing.
Well, we disagree, because I think they’re identical in virtually every way.
I’m highly skeptical of the value behind code auditing to begin with, so anything that relies on auditing to have value is already something I’m side eyeing hard tbh.
I think where we disagree on the weights. I barely care about binary reproducibility, I frankly don’t think code auditing is practical, and I think sandboxing is by far the most important, cost effective measure to improve security and directly address the issues.
I am familiar with the concept of reproducible builds. Also, as far as I’m aware, Rust’s current tooling is incapable of producing reproducible binaries.
And in theory there are many attack vectors that might be present in any form of software distribution, whether source or binary.
What I’m looking for here is someone who will step up and identify a specific security vulnerability that they believe actually existed in
serdewhen it was shipping precompiled macros, but that did not exist when it was shipping those same macros in source form. “Someone could compromise the maintainer or the project infrastructure”, for example, doesn’t qualify there, because both source and binary distributions can be affected by such a compromise.Aren’t there links in the original github issue to exactly this being done in the NPM and some other ecosystem? Yes this is a security problem, and yes it has been exploited in the real world.
I’m going to quote my other comment:
If you have proof of an actual concrete vulnerability in
serdeof that nature, I invite you to show it.The existence of an actual exploit is not necessary to be able to tell that something is a serious security concern. It’s like laying an AR-15 in the middle of the street and claiming there’s nothing wrong with it because no one has picked it up and shot someone with it. This is the opposite of a risk assessment, this is intentionally choosing to ignore clear risks.
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening. To me this is up with with Sony’s rootkit back in the day, completely unexpected, unauthorised behaviour that no reasonable person would expect, nor would they look out for it because it is just such an unreasonable thing to do to your users.
I think the point is that if precompiled macros are an AR-15 laying in the street, then source macros are an AR-15 with a clip next to it. It doesn’t make sense to raise the alarm about one but not the other.
I think this is extreme. No additional accessing of any kind was done. Binaries don’t have additional abilities that
build.rsdoes not have. It’s not at all comparable to installing a rootkit. The precompiled macros did the same thing that the source macros did.Once again, other language package ecosystems routinely ship precompiled binaries. Why have those languages not suffered the extreme consequences you seem to believe inevitably follow from shipping binaries?
Even the most extreme prosecutors in the US never dreamed of taking laws like CFAA this far.
I think you should take a step back and consider what you’re actually advocating for here. For one thing, you’ve just invalidated the “without any warranty” part of every open-source software license, because you’re declaring that you expect and intend to legally enforce a rule on the author that the software will function in certain ways and not in others. And you’re also opening the door to even more, because it’s not that big a logical or legal leap from liability for a technical choice you dislike to liability for, say, an accidental bug.
The author of
serdedidn’t take over your computer, or try to. All that happened wasserdestarted shipping a precompiled form of something you were going to compile anyway, much as other language package managers already do and have done for years. You seem to strongly dislike that, but dislike does not make something a security vulnerability and certainly does not make it a literal crime.I think that what actually is happening in other language ecosystems is that while there are precompiled binaries sihpped along some installation methods, for other installation methods those are happening by source.
So you still have binary distribution for people who want that, and you have the source distribution for others.
I have not confirmed this but I believe that this might be the case for Python packages hosted on debian repos, for example. Packages on PyPI tend to have source distributions along with compiled ones, and the debian repos go and build packages themselves based off of their stuff rather than relying on the package developers’ compiled output.
When I release a Python library, I provide the source and a binary. A linux package repo maintainer could build the source code rather than using my built binary. If they do that, then the thing they “need to trust” is the source code, and less trust is needed on myself (on top of extra benefits like source code access allowing them to fix things for their distribution mechanisms)
I don’t know of anyone who actually wants the sdists from PyPI. Repackagers don’t go to PyPI, they go to the actual source repository. And a variety of people, including both me and a Python core developer, strongly recommend always invoking
pipwith the--only-binary :all:flag to force use of.whlpackages, which have several benefits:--require-hashesand--no-deps, you get as close to perfectly byte-for-byte reproducible installs as is possible with the standard Python packaging toolchain..whlhas no scripting hooks (as opposed to an sdist, which can run arbitrary code at install time via itssetup.py).I misread that as “sadists from PyPi” and could not help but agree.
I mean there are plenty of packages with actual native dependencies who don’t ship every permutation of platform/Python version wheel needed, and there the source distribution is available. Though I think that happens less and less since the number of big packages with native dependencies is relatively limited.
But the underlying point is that with an option of compiling everything “from source” available as an official thing from the project, downstream distributors do not have to do things like, say, confirm that the project’s vendored compiled binary is in fact compiled from the source being pointed at.
Install-time scripting is less of an issue in this thought process (after all, import-time scripting is a thing that can totally happen!). It should feel a bit obvious that a bunch of source files is easier to look through to figure out issues rather than “oh this part is provided by this pre-built binary”, at least it does to me.
I’m not arguing against binary distributions, just think that if you have only the binary distribution suddenly it’s a lot harder to answer a lot of questions.
As far as I’m aware, it was possible to build
serde“from source” as a repackager. It did not produce a binary byte-for-byte identical to the one being shipped first-party, but as I understand it producing a byte-for-byte identical binary is not something Rust’s current tooling would have supported anyway. In other words, the only sense in which “binary only” was true was for installing fromcrates.io.So any arguments predicated on “you have only the binary distribution” don’t hold up.
Hmm, I felt like I read repackagers specifically say that the binary was a problem (I think it was more the fact that standard tooling didn’t allow for both worlds to exist). But this is all a bit moot anyways
It’s a useful fallback when there are no precompiled binaries available for your specific OS/Arch/Python version combination. For example when pip installing from a ARM Mac there are still cases where precompiled binaries are not available, there were a lot more closer to the M1 release.
When I say I don’t know of anyone who wants the sdist, read as “I don’t know anyone who, if a wheel were available for their target platform, would then proceed to explicitly choose an sdist over that wheel”.
Argumentum ad populum does not make the choice valid.
Also, not for nothing, most of the discussion has just been assuming that “binary blob = inherent automatic security vulnerability” without really describing just what the alleged vulnerability is. When one person asserts existence of a thing (such as a security vulnerability) and another person doubts that existence, the burden of proof is on the person asserting existence, but it’s also perfectly valid for the doubter to point to prominent examples of use of binary blobs which have not been exploited despite widespread deployment and use, as evidence in favor of “not an inherent automatic security vulnerability”
Yeah, this dynamic has been infuriating. In what threat model is downloading source code from the internet and executing it different from downloading compiled code from the internet and executing it? The threat is the “from the internet” part, which you can address by:
Anyone with concerns about this serde change should already be doing one or both of these things, which also happen to make builds faster and more reliable (convenient!).
Yeah, hashed/pinned dependency trees have been around forever in other languages, along with tooling to automate their creation and maintenance. It doesn’t matter at that point whether the artifact is a precompiled binary, because you know it’s the artifact you expected to get (and have hopefully pre-vetted).
Downloading source code from the internet gives you the possibility to audit it, downloading a binary makes this nearly impossible without whipping out a disassembler and hoping that if it is malicious, they haven’t done anything to obfuscate that in the compiled binary. There is a “these languages are turing complete, therefore they are equivalent” argument to be made, but I’d rather read Rust than assembly to understand behaviour.
The point is that if there were some easy way to exploit the mere use of precompiled binaries, the wide use of precompiled binaries in other languages would have been widely exploited already. Therefore it is much less likely that the mere presence of a precompiled binary in a package is inherently a security vulnerability.
I’m confused about this point. Is anyone going to fix crates.io so this can’t happen again?
Assuming that this is a security problem (which I’m not interested in arguing about), it seems like the vulnerability is in the packaging infrastructure, and serde just happened to exploit that vulnerability for a benign purpose. It doesn’t go away just because serde decides to stop exploiting it.
I don’t think it’s an easy problem to fix: ultimately, package registry is just a storage for files, and you can’t control what users put there.
There’s an issue open about sanitizing permission bits of the downloaded files (which feels like a good thing to do irrespective of security), but that’s going to be a minor speed bump at most, as you can always just copy the file over with the executable bit.
A proper fix here would be fully sandboxed builds, but:
Who’s ever heard of a security issue caused by a precompiled binary shipping in a dependency? Like, maybe it’s happened a few times? I can think of one incident where a binary was doing analytics, not outright malware, but that’s it.
I’m confused at the idea that if we narrow the scope to “a precompiled binary dependency” we somehow invalidate the risk. Since apparently “curl $FOO > sh” is a perfectly cromulent way to install things these days among some communities, in my world (30+ year infosec wonk) we really don’t get to split hairs over ‘binary v. source’ or even ‘target v. dependency’.
I’m not sure I get your point. You brought up codec vulns, which are irrelevant to the binary vs source discussion. I brought that back to the actual threat, which is an attack that requires a precompiled binary vs source code. I’ve only seen (in my admittedly only 10 Years of infosec work) such an attack one time, and it was hardly an attack and instead just shady monetization.
This is the first comment I’ve made in this thread, so I didn’t bring up codecs. Sorry if that impacts your downplaying supply chain attacks, something I actually was commenting on.
Ah, then forget what I said about “you” saying that. I didn’t check who had commented initially.
As for downplaying supply chain attacks, not at all. I consider them to be a massive problem and I’ve actively advocated for sandboxed build processes, having even spoken with rustc devs about the topic.
What I’m downplaying is the made up issue that a compiled binary is significantly different from source code for the threat of “malicious dependency”.
So not only do you not pay attention enough to see who said what, you knee-jerk responded without paying attention to what I did say. Maybe in another 10 years…
Because I can
curl $FOO > foo.sh; vi foo.shthen can choose tochmod +x foo.sh; ./foo.sh. I can’t do that with an arbitrary binary from the internet without whipping out Ghidra and hoping my RE skills are good enough to spot malicious code. I might also miss it in some downloaded Rust or shell code, but the chances are significantly lower than in the binary. Particularly when the attempt from people in the original issue thread to reproduce the binary failed, so no one knows what’s in it.No one, other than these widely publicised instances in NPM, as well as PyPi and Ruby, as pointed out in the original github issue. I guess each language community needs to rediscover basic security issues on their own, long live NIH.
Am I missing something? Both links involve malicious source files, not binaries.
I hadn’t dived into them, they were brought up in the original thread, and shipping binaries in those languages (other than python with wheels) is not really common (but would be equally problematic). But point taken, shouldn’t trust sources without verifying them (how meta).
But the question here is “Does a binary make a difference vs source code?” and if you’re saying “well history shows us that attackers like binaries more” and then history does not show that, you can see my issue right?
But what’s more, even if attackers did use binaries more, would we care? Maybe, but it depends on why. If it’s because binaries are so radically unauditable, and source code is so vigilitantly audited, ok sure. But I’m realllly doubtful that that would be the reason.
There’s some interesting meat to think about here in the context of package management, open source, burden on maintainers, varying interest groups.
So if I’m reading this article correctly, the point of this is to remove more potential sources of nondeterminism from Nix. Have there been any demonstrated benefits so far, or is this still all theoretical/robustness/WIP?
It’s mostly about running nix-built OpenGL/Cuda binaries on a foreign distribution (Ubuntu, Fedora, Debian…). You need a way to inject some sort of GPU driver to the Nix closure. You won’t be able to run a nix-built OpenGL program on a foreign distribution if you don’t do so.
NixGLHost is an alternative* approach to do this.
* Alternative to NixGL. NixGLHost is in a very very alpha stage.
One of my gripes with nixgl is that i have to run all my nix applications via nixgl. If I run a non-nix binary with nixgl it usually doesn’t go well, so i can’t run my whole user session with nixgl and have it propagate to child processes. Is there any, for example, NIX_LD_PRELOAD one could use, that could be set system-wide, that is ignored by non-nix binaries?
To be honest, that’s not a use case I had in mind when exploring this problem space. I’d probably need more than 5 minutes to correctly think about this, take what I’m about to say with a grain of salt.
My gut instinct is that we probably don’t want to globally mix the GPU Nix closure with the host one. I guess an easy non-solution for this would be to skip the problem altogether by provisioning the Nix binaries through a Nix shell. In this Nix shell, you could safely discard the host library paths and inject the nix-specific GPU libraries directly through the LD_LIBRARY_PATH (via nix-gl or nix-gl-host).
Now, if you think about it more, the use case you’re describing seems valid UX-wise. I’m not sure what would be the best way to tackle it. The main danger is getting your libraries mixed up. NIX_LD_PRELOAD could be a nice trick, but it’s kind of a shotgun approach, you end up preloading your shim for each and every Nix program, regardless if they depend on OpenGL or not.
As long as you don’t plan to use CUDA, I think the best approach would be injecting the GPU DSOs from libglvnd. There’s already all you need to point to your EGL DSOs through the
__EGL_VENDOR_LIBRARY_DIRSenv variable. There’s no handy way to do that for GLX, but I wrote a small patch you could re-use to do so.I’ll try to think more about that, cool use case, thanks for the feedback.
How would replacing bash with nushell play with bootstrapping of nix and nixpkgs? When comparing guix and nix, guix did quite a good job on that topic and there is really a minimal set of packages to build everything from scratch. I’m wondering if bringing Rust in, just to build nushell, just to build stdenv based on it, would make bootstrapping nearly impossible.
100% agree, this article completely eludes this central question. Bash is semi-trivial to bootstrap!
Nixpkgs does not bootstrap rustc, we’re currently using a binary distribution: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/rust/rustc.nix#L28
Adopting this as a default stdenv would require to push this massive bindist to the Nixpkgs bootstrap seed. That seed is already massive compared to what Guix has, I don’t think we want to degrade this further.
Rust is definitely source-bootstrapable, as a matter of fact, Guix manages to do it, there’s no reason we can’t do the same. The bootstrap chain is pretty long though. On top of what we already bootstrap (gcc, openssl, etc.), we’d need to bootstrap llvm, mrustc, then rust 1.54 -> 55 -> 56 -> 57 -> 58 -> 60 -> 61 -> 62 -> 63 -> 64 -> 65.
So yeah, pushing this to stdenv would considerably degrade the bootstrap story in any case.
From my perspective, Bash is a local optimum, I personally wouldn’t change it, it’s certainly a good balance between a language that is easy to bootstrap and a good-enough expressiveness to express builds.
If we really want to move to something more modern, Oil could be a more serious contender, they seem to take bootstrapping seriously. There’s a drafted RFC wrt. Oil adoption.
[Edit]: The question is eluded, but I don’t think the author expects this to replace stdenv, at least it’s not mentionned in the article. Don’t take this comment as an overwhelming negative “this project is worthless”. Cool hack!
This made me realize that Rust is fundamentally non-bootstrapable. It’s definitely going to produce a release every six weeks for quite a number of years, and Rust’s release N needs release N-1 to build, so the bootstrap chain, by design, grows quickly and linearly with time. So it seems that, in the limit, it is really a choice between:
Is there a reference interpreter, perhaps? I imagine that that can’t be a complete solution since LLVM is a dependency, but it would allow pure-Rust toolchains to periodically adjust their bootstraps, so to speak.
There is mrustc which is written in C++ and allows you to bootstrap Rust. There is also GCC Rust implementation in the works, that will allow bootstrapping.
In my defense, I do use the term “experimental” several times and I don’t make any suggestion of replacing stdenv. I could be wrong, but I think that flakes are going to decentralize the Nix ecosystem quite a bit. While the Nixpkgs/stdenv relationship is seemingly ironclad, I don’t see why orgs or subsets of the Nix community couldn’t adopt alternative builders. Any given Nix closure can in principle have N builders involved; they’re all just producing filesystem state after all.
As for bootstrapping, yes, the cost of something like Nushell/Nuenv is certainly higher than Bash, but it’s worth considering that (a) you don’t need things like curl, jq, and coreutils and (b) one could imagine using Rust feature flags to produce lighter-weight distributions of Nushell that cut out things that aren’t germane to a Nix build environment (like DataFrames support).
Yes the new Oil C++ tarball is 375 kilobytes / 90K lines of compressed, readable C++ source :)
https://www.oilshell.org/release/0.14.2/
The resulting binary is about 1.3 MB now. I seem to recall that the nushell binary is something like 10 MB or 50 MB, which is typical for Rust binaries. rustc is probably much larger.
There was some debate about whether Oil’s code gen needs to be bootstrapped. That is possible, but lots of people didn’t seem to realize that the bash build in Nix is not.
It includes yacc’s generated output, which is less readable than most of Oil’s generated code.
Nushell is certainly larger! Although I would be curious how much smaller Nushell could be made. Given that it’s Rust, you could in principle use feature flags to bake in only those features that a Nix builder would be likely to use and leave out things like DataFrames support (which is quite likely the “heaviest” part of Nushell).
For sure it would make bootstrapping much harder on things like OpenBSD. Good luck if you are on an arch that doesn’t have rust or LLVM. That said, I don’t think this would replace stdenv.. for sure not any time soon!
Also the article does mention exactly what you are pointing out:
My question is orthogonal to this, and maybe I should have specify what I mean by bootstrapping. It’s “how many things I have to build first, before I can have working nixpkgs and build things users ask for”. So if we assume that nushell runs wherever bash runs, how much more effort is to build nushell (and rust and llvm) than bash? I would guess order of magnitude more, thus really complicating the initial setup of nixpkgs (or at least getting them to install without any caches).
Are there any plans for moving Nix flakes from experimental to stable? I see that Nix flakes are all the hype now. Even the linked guide states that:
Do you agree with this?
I’ve been in love with NixOS for some time now but I avoided experimental features so far.
The thing is, Flakes FOMO starts to creep in but I’d rather avoid putting (even more) time into learning Nix/NixOS/nixpkgs/nix-the-command if flakes are going to be deprecated next spring.
For what it is worth, I can’t fathom a universe where Nix actually removes Flakes. The data I’m seeing and the user interviews I’ve done show an overwhelming number of new Nix users are starting with Flakes because they’re easier and it creates a framework to learn inside.
If there is some sort of redesign or flakes2, it wouldn’t kill flakes1.
I agree, it seems like flakes are here to stay.
However, they are still considered to be an unstable feature by the Nix main developer. The flakes code you write is considered unstable and could break without further notice after a Nix update.
I think all the knowledge you get without flakes would be still transferable to a flake-based setup. Flakes are nice for a few usecases. Most importantly, pinning and mixing packages from different sources. If you don’t have urgent needs related to these two things, you can postpone migrating to flakes.
Regarding experimental features, I find the new nix command something worth looking into. The CLI is much simpler and nicer.
Part of the issue is that the RFC committee doesn’t want to rubber-stamp flakes just because a lot of people, lead by a few influential people, are all using the same unofficial and experimental feature. It makes a mockery of the RFC process and skips over a thorough design review. If flakes are really the way forward, some of all this energy should go into getting the RFC officially pushed through.
Great article!
That’s the first time I hear about work notes being some form of “temporal documentation”. It makes a lot of sense.
I however think the workflow described in this page creates a hard dependency towards a online service by hard-linking internet URLs to your immutable and long-living git history. These URLS tend to be very mutable by nature, it’s hard to predict whatever they’ll still be available or not on a long time scale. Any migration would require you to re-write your whole git history.
That’s fine for your small hobby project, but it’s likely to lead to a lot of context loss in a moderately distant future (say 15/20 years) for a semi-major project (like datasette).
The git notes UX is pretty bad, it’s a shame. They’d be a great way to store this kind of temporal documentation attached to a commit.
[Edit]: Scratch that last paragraph, git notes wouldn’t work in a context where you want to store your design before starting to work on the actual implementation.
I touched a bit more on that here: https://simonwillison.net/2022/Oct/29/the-perfect-commit/
If you’re going to use issues in the way I’m describing here you need to be VERY confident that your organization has a healthy enough respect for institutional knowledge that they’re not just going to throw it all away some day when they switch to a different issue tracking system!
One of the reasons I’ve gone all-in on GitHub Issues is that it has a really good API. I have code which exports my issues on a regular basis, both as a form of backup and as a way to run my own queries against them: https://github-to-sqlite.dogsheep.net
My plan for if I ever do work on a project that migrates to another issue tracker is to create a static HTML archive of the old issues and then run a script that rewrites the entire commit history so each commit links to the full URL to the archived issue.
I investigated exporting issue threads to git notes but was put off that idea when I learned the GitHub doesn’t display notes any more: https://github.blog/2010-08-25-git-notes-display/
Out of curiosity, did you write a package that uses the binary releases out of simplicity, or did you run into issues using the Go tooling in Nixpkgs to build from source? (If the latter, I’d love to hear about them.)
I’m new to Nix. I took the easiest way possible. It’d would be great if you can show me how to build GtS from source.
You can then
pkgs.callPackagethe above file somwhere in your nix config. Or better, if you feel like spending a bit more time on this: clean up your module and this derivation (add the relevant meta attributes) open a PR in Nixpkgs adding this derivation and your module.[Edit]: there’s apparently somebody else working on this https://github.com/NixOS/nixpkgs/pull/202110 . You can potentially team up :)
In general, the entry point to find out this kind of information for a lanugage you’re not familiar with is the nixpkgs manual. In that particular case, here’s the relevant golang section: https://nixos.org/manual/nixpkgs/stable/#sec-language-go
Has this actually happened? Getting banned for talking about nonguix?
Not sure about getting banned, per se, but it’s explicitly discouraged. The second paragraph of nonguix’s readme:
Holy shit, that’s extremely disrespectful to users.
I would recommend actually reading the help-guix archives to see how often support issues are created and how many issues users have are ignored or told they are out of place.
I admit I fucked up and misunderstood the rules. My complaint now reads:
That’s made up, like most of that article, it’s full of misconceptions. Can’t tell whether or not this has been written in good faith.
But hey, outrage is good to attract attention. Proof to the point: I’m commenting this out of outrage.
Hehe, yeah, the FSF and SFC use outrage constantly! I get emails all the time telling me that Microsoft and Apple are teaming up to murder babies or whatever. It’s pretty much all they have left at this point, and I say this as someone who donated and generally supported their mission for many, many years (which is why I still get the emails).
Hyperbole and untruths are like pissing in your shoes to get warm; backfire once the initial heat is gone.
When I wrote that bit I made the assumption that violating the rules of the channel could get you banned. I admit that it looks wrong in hindsight, so I am pushing a commit to amend it.
Not to my knowledge. No. I’ve seen it tut-tutted but I’ve yet to see someone get banned.
That’s 100% messed up if true.
I get a 403 Forbidden.
Me too now. As they themselves say:
😁
Sorry, I think my Private Cloud has a bad power supply which is having a knock-on effect of upsetting the NFS mounts on the webserver. I’m acquiring a replacement right now, and in the meantime I am going to Infrastructure-as-Code it by adding a cronjob that fixes the NFS mount.
Me too.
You can use https://archive.md/lFfIn
I’ve been reading the Gemini specification, as well as this post, and my conclusion is that it’s just a worse version of HTTP 1.1 and a worse version of Markdown.
Strong disagree. TLS + HTTP 1.1 requires to perform a upgrade dance involving quite a few extra steps. The specification is also pretty empty regarding SNI management. Properly implementing that RFC is pretty demanding. There’s also a lot of blind spots left to the implementer better judgement.
In comparison, the Gemini TLS connection establishing flow is more direct and simpler to implement.
TLS aside, you mentioning
sounds like a clear win to me. The baseline HTTP 1.1 RFC is already massive let alone all its required extensions to work in a modern environment.
I agree that the specification is simple to read, but the specification itself is too limited and don’t find it suitable for the real world.
For example, I prefer HTTP’s optional end-to-end encryption because when working with internal routers within an infrastructure dealing with certificates is a PITA and a completely unnecessary bump in complexity and performance overhead being inside an already secured network.
I also disagree on that “extensibility is generally a bad idea” as the article says. Extensibility can work if you do it properly, like any other thing on software engineering.
EDIT: Or the requirement of closing the connection and re-opening it with every request, and all the handshakes that means.
For clarity about what I think could be an actual improvement: I would prefer an alternative evolution of HTTP 1.0, with proper readable specs, test suites, clearer https upgrade paths, etc; instead of an evolution of Gopher.
TLS + HTTP just requires connecting to port 443 with TLS. I’ve worked on lots of things using HTTP for >20 years and I don’t think I’ve ever seen the upgrade protocol used in real life. Is it commonly used by corporate proxies or something like that?
When I looked at it (months ago), I got the same impression. I find this article irresponsible, as Gemini does not merit the support.
Gemini’s intentions are good. The specification isn’t. For instance, not knowing the size of an object before receiving it makes it a non-starter for many of its intended purposes.
This is an idea that should be developed properly and openly, allowing for input from several technically capable individuals. Not one person’s pet project.
I’ll stick to Gopher until we can actually do this right. Gemini doesn’t have the technical merit to be seen as a possible replacement to Gopher.
It does accept input from individuals. I was able to convince the author to expand the number of status codes, to use client certificates (instead of username/password crap) and to use the full URL as a request.
I prefer to think of Gemini as a better version of gopher with a more sane index page format.
Interesting, I’ve been porting nixos to the nano pi m4 v2, this looks quite a bit less painful in some ways. Added to my: try this out some day list of stuff to look at.
It is!
Conceptually speaking, Guix got a lot of things right. On the other hand, Nix precedes Guix, the opposite would have been concerning.
The documentation is amazing, they are very careful about their tooling vendor lock-in. Their clean API around the toolchains and the abstractions around derivations is the selling point to me. The language is also somehow standard and comes full batteries included tooling-wise.
There’s a catch however: the Guix packageset is much smaller, you won’t have all the nice language-specific build systems you have with Nix, overall you’re likely to miss some stuff packaging-wise. Also: no systemd (I guess it might be a selling point for some people though).
Yep yep, this just intrigued me as I’ve gotten a bit deep in the guts of how nixos sd images are built. Its honestly not too big of a deal its just probably in need of a bit of a refactor tbh for the overall function to do this stuff but it honestly just strikes me as more: this was evolved not planned. Which is fine, just not as polished as it could be.
The scheme bit made a lot more sense to me off the bat versus having to do a fair amount of digging to figure out how I can adjust the partition sizes and make sure that my custom u-boot and its spl files etc… are getting put in the “blessed” right spot for this board (still not sure i am doing it right tbh as its not booting).
And the systemd bit is water under the bridge to me, that ship has sailed. I’ve had to port/add custom derivations to nixpkgs a lot so i’m not too averse to that if needed.
My real reason for all this is i’m building a little k8s cluster out of arm boards for shits, so nixops is my ultimate goal here.
Nice post! Actually nice blog altogether, I started to binge read it tonight!
I couldn’t help but to notice something a tiny bit ironic though:
After using it for a while I started to find the Nix expression language as one of the best designed syntaxes ever. It doesn’t have separators for records or lists so it’s friendly to diff. The multiline strings are perfect. Writing nested records with “a.b.c” keys is super convenient. The lambda syntax is as simple as possible. Etc etc.
Records are separated with the
;symbol.As for lists, I beg to disagree. The list symbols are separated with a whitespace, which is unfortunate since whitespace is also used to represent function application. It means you’ll have to be careful enough to wrap your function applications in parenthesis every time you’ll perform it in a list.
That’s a easy trap to fell into, especially in multi-lines statements. Add the lazy nature of the language on top of that, you can end up with stacktraces that are pretty difficult to decipher. Especially if you end up doing that in a NixOS machine description :/.
I see a lot of newcomers falling into this trap on IRC.
e.g:
Instead of
Sorry, I meant separators to mean comma-like separators where the last item doesn’t end with the separator.
The issue you mentioned is real, yeah. I still love the syntax.
While it might be true that 1500 Bytes is now the de facto MTU standard on the Internet (minus whatever overhead you throw at it), everything’s not lost. The problem is not that we don’t have the link layer capabilities to offer larger MTUs, the problem is that the transport protocol has to be AWARE of it. One mechanism for finding out whether what size MTU is supported by a path over the Internet is an Algorithm called DPLPMTUD. It is currently being standardized by the IETF and is more or less complete https://tools.ietf.org/html/draft-ietf-tsvwg-datagram-plpmtud-14. There are even plans for QUIC to implement this algorithm, so if we’ll end up with a transport that is widely deployed and also supports detection of MTUs > 1500 we’ll actually might have a chance to change the link layer defaults. Fun fact: All of the 4G networking gear actually supports jumbo frames, most of the providers just haven’t enabled the support for it since they are not aware of the issue.
Wow, it might even work.
I can hardly believe it… but if speedtest.net were able to send jumbo frames and most users’ browsers support receiving it, it might get deployed by ISPs as they look for benchmark karma. Amazing. I thought 1500 was as invariant as π.
I was maintainer for an AS at a previous job and set up a few BGP peers with jumbo frames (4470). I would have made this available on the customer links as well, except none of them would have been able to receive the frames. They were all configured for 1500, as is the default in any OS then or today. Many of their NICs couldn’t handle 4470 either, though I suppose that has improved now.
Even if a customer had configured their NIC to handle jumbo frames, they would have had problems with the other equipment on their local network. How do you change the MTU of your smartphone, your media box or your printer? If you set the MTU on your Ethernet interface to 4470 then your network stack is going to think it can send such large frames to any node on the same link. Path MTU discovery doesn’t fix this because there is no router in between that can send ICMP packets back to you, only L2 switches.
It is easy to test. Try to ping your gateway with
ping -s 4000 192.168.0.1(or whatever your gateway is). Then change your MTU with something likeip link set eth0 mtu 4470and see if you can still ping your gateway. Remember to runip link set eth0 mtu 1500afterwards (or reboot).I don’t think that DPLPMTUD will fix this situation and let everyone have jumbo frames. As a former network administrator reading the following paragraph, they are basically saying that jumbo frames would break my network in subtle and hard to diagnose ways:
So you’re going to have people complain that their browser is working, but nothing else. I wouldn’t enable jumbo frames if DPLPMTUD was everything that was promised as a fix. That said, it looks like DPLPMTUD will be good for the Internet as a whole, but it does not really help the argument for jumbo frames.
And I don’t know if it has changed recently, but the main argument for jumbo frames at the time was actually that they would lead to fewer interrupts per second. There is some overhead per processed packet, but this has mostly been fixed in hardware now. The big routers use custom hardware that handles routing at wire speed and even consumer network cards have UDP and TCP segmentation offloading, and the drivers are not limited to one packet per interrupt. So it’s not that much of a problem anymore.
Would have been cool though and I really wanted to use it, just like I wanted to get us on the Mbone. But at least we got IPv6. Sorta. :)
If your system is set up with an mtu of 1500, then you’re already going to have to perform link mtu discovery to talk with anyone using PPPoE. Like, for example, my home DSL service.
Back when I tried running an email server on there, I actually did run into trouble with this, because some bank’s firewall blocked ICMP packets, so… I thought you’d like to know, neither of us used “jumbo” datagrams, but we still had MTU trouble, because their mail server tried to send 1500 octet packets and couldn’t detect that the DSL link couldn’t carry them. The connection timed out every time.
If your application can’t track a window of viable datagram sizes, then your application is simply wrong.
It’s even worse: in the current situation[1], your system’s MTU won’t matter at all. Most of the network operators are straight-up MSS-clamping your TCP packets downstream, effectively discarding your system’s MTU.
I’m very excited by this draft! Not only it will fix the UDP situation we currently have, but will also make tunneling connections way more easy. That said, it also means that if we want to benefit from that, the network administrators will need to quit mss-clamping. I suspect this to take quite some time :(
[1] PMTU won’t work in many cases. Currently, you need ICMP to perform a PMTU discovery, which is sadly filtered out by some poorly-configured endpoints. Try to
ping netflix.comfor instance ;)Very true, one can’t assume an MTU of 1500 on the Internet. I disagree that it’s on the application to handle it:
The network stack is responsible for PMTUD, not the application. One can’t expect every application to track the datagram size on a TCP connection. Applications that use BSD sockets simply don’t do that, they send() and recv() and let the network stack figure out the datagram size. There’s nothing wrong with that. For UDP the situation is a little different, but IP can actually fragment large UDP datagrams and PTMUD works there too (unless, again, broken by bad configurations, hence DPLPMTUD).
Sure, fine. It’s the transport layer’s job to handle it. Just as long as it’s detected at the endpoints.
It doesn’t seem like anyone likes IP fragmentation.
If you’re doing a teleconferencing app, or something similarly latency-sensitive, then you cannot afford the overhead of reconstructing fragmented packets; your whole purpose in using UDP was to avoid overhead.
If you’re building your own reliable transport layer, like uTP or QUIC, then you already have a sliding size window facility; IP fragmentation is just a redundant mechanism that adds overhead.
Even DNS, which seems like it ought to be a perfect use case for DNS with packet fragmentation, doesn’t seem to work well with it in practice, and it’s being phased out in favour of just running it over TCP whenever the payload is too big. Something about it acting as a DDoS amplification mechanism, and super-unreliable on top of that.
If you’re using TCP, or any of its clones, of course this ought to be handled by the underlying stack. They promised reliable delivery with some overhead, they should deliver on it. I kind of assumed that the “subtle breakage” that @weinholt was talking about was specifically for applications that used raw packets (like the given example of
ping).You list good reasons to avoid IP fragmentation with UDP and in practice people don’t use or advocate IP fragmentation for UDP. Broken PMTUD affects everyone… ever had an SSH session that works fine until you try to list a large directory? Chances are the packets were small enough to fit in the MTU until you listed that directory. As breakages go, that one’s not too hard to figure out. The nice thing about the suggested MTU discovery method is that it will not rely on other types of packets than those already used by the application, so it should be immune to the kind of operator who filters everything he does not understand. But it does mean some applications will need to help the network layer prevent breakage, so IMHO it doesn’t make jumbo frames more likely to become a thing. It’s also a band-aid on an otherwise broken configuration, so I think we’ll see more broken configurations in the future, with less arguments to use on the operators who can now point to how everything is “working”.
According to zmap it takes 45min to scan all of IPv4 on a Gigabit connection. That could be a slow but interesting way to reliably bootstrap the network in case of an attack.
I like the idea.
The 45 mins scan advertised on the homepage is probably the result of a TCP SYN scan though. You’ll probably need to add an application layer scanner on top of that (zgrab?). Not sure how this will affect the overall latency of the scan :/
There’s also a detailed wrietup about the business card design on the very same blog.
I’d be interested to understand the hardware design of the board. The post sadly doesn’t cover that part :(
Nice ad. :|
I agree it is an ad, but…
I can get behind that.
Also at the moment according to the pricing page, payment is optional.
It’s advertising an open source project, Source Hut, but also Janet, Zig, Nim, Samurai, Sway and other open source projects I like. Projects that get very little payment or gratitude for the work they do.
Yes sr.ht is a service too, a useful one at that. They support BSD well, unlike other companies, how else are they supposed to let people know this fact? Should they be paying largely unethical companies like google for ad space? Or should they just be more subversive so people don’t complain.
Let me put it this way, if every open source project was also a business, should we hate on every single one for advertising? sr.ht didn’t game the upvotes to get on the front page, people upvoted it by themselves.
I suppose there could be a tag ‘sponsored’ so people can ignore them. Not suggesting allowing lower quality from sponsored content either, probably the inverse.
The issue is that I see a Sourcehut “ad” every few days: “Sourcehut supports OpenBSD”, “Sourcehut supports migrations from Bitbucket”, “Sourcehut supports ASCII”. Yeah … we got it … A lot of these posts don’t have a lot of meat to them and at this point, it’s just getting spammy.
They don’t always have a lot of “meat,” but posts about SourceHut represent a capitalist ideology I can actually get behind. A single proprietor, working their ass off to try to change the software world, which has gotten extremely out of hand with regards to complexity, and the marketing of products that fix the complex systems we don’t need, at all, to begin with.
What’s the difference between a SourceHut post, and an
postad that complains that as an open source author I am not compensated fairly? Hint: one should be inspiration, for the other is actually possible.payment for the service is optional, so no it doesn’t. All the things that make Sourcehut great in my opinion are the ways in which it denies capitalist ideology. Open Source Software, optional payments, etc.
It’s optional, right now, while in Alpha. It doesn’t seem the plan is that forever. Also, if it wasn’t clear, I’m extremely in favor of this model of charging people for a service, but releasing your software under a permissive license.
Just let me other another data point here. It was thanks to the “migration from Bitbucket” post that I found out Sourcehut had a nifty script to help migrations from Bitbucket and that saved hours of work as I migrated 20+ repos effortlessly. This current post made me realize that maybe I should be paying more attention to their CI system as it looks much simpler than others I’ve used. So, in the end, I’m appreciating these blog posts a lot. Yes they are related to a commercial venture but so what? You can self-host it if you’re not into SaaS outside your control. If we set a hard line like this, then it becomes impossible to post about any commercial project at all. It is already hard to monetize FOSS projects to make them sustainable, now imagine if they are not even allowed blog posts…
Same here. This string of posts made me aware of sourcehut and when I had to migrate from bitbucket, I then gave them a hard eval. I like their human, non-shitty business model of “I give them money and they give me services”, and that their products are professionally executed and no-frills.
I don’t know how to reconcile it. These articles were very useful to me, when most product ads weren’t and I’d be disappointed if this site became a product advert platform. I think people are right for flagging it is almost-an-ad, but in this one vendor’s case I’m glad I saw them and am now a happy sourcehut customer.
That is fair I guess. I’ll have to check the guidelines on things like that.
Yes. I flag those too. Advertising is a mind killer.
But there is no other way to get large numbers of people to know about something, following your advice would be suicide.
I also hate advertising, I just don’t see a way around it. I won’t argue further against banishing advertising from lobste.rs at least.
All these conversations are done like it’s all or nothing. We allow politics/marketing/etc on Lobsters or… it never happens anywhere with massive damage to individuals and society. Realistically, this is a small site with few monetary opportunities for a SaaS charging as little as he does. If the goal is spreading the word, it’s best done on sites and platforms with large numbers of potential users and (especially) paying customers. Each act of spreading the word should maximize the number of people they reach for both societal impact and profit for sustainability.
Multiple rounds on Lobsters means, aside from the first announcement with much fan fare, the author sacrificed each time opportunities to reach new, larger audiences to show the same message again to the same small crowd. Repeating it here is the opposite of spreading the word. Especially since most here that like Sourcehut are probably already following it. Maybe even buying it. He’s preaching to the choir here more than most places.
Mind-killer or not, anyone talking about large-scale adoption of software, ideology, etc should be using proven tactics in the kinds of places that get those results. That’s what you were talking about, though. I figured he was just trying to show latest BSD-related progress on one of his favorite tech forums. More noise than signal simply because he was sharing excitement more than doing technical posts or focused marketing.
Every blog post is an ad for something. It may not be a product, directly, but it’s advertising an idea, the person, or persons the idea was thought by, the writing (which, btw can be a product) of the author, etc.
If you want to sincerely flag advertising, you might as well get offline—it’s pervasive.
Not a native english speaker here. I may be wrong, but after looking at the dictionnary definition:
it seems that an advertisement has a precise definition: an ad is directly related to a paid product, not an idea.
This is a fairly pedantic interpretation. A person promotes an idea to sell something, if even themselves. That “sale” might only come later in the form of a job offer, or support through Patreon, etc, etc.. But, to say that you can’t advertise an idea is wrong. The cigarette industry’s ad campaigns have always been about selling an image, an idea that if you smoke you become part of something bigger. Oh, and btw, you’ll probably remember the brand name, and buy that kind instead of something else.
iPods were sold on the very basis of white headphones, TO THE POINT, that people without iPods started wearing white headphones to be part of the “club.” Advertisements sell you the idea of a better life, and hopefully you’ll buy my product to get it.
You’re right, and how virtuous Sourcehut may or may not be doesn’t change that. The line between ad and article is a spectrum, but this seems to be pretty well into the ad side of things. I apologise, I’ll be more discerning in the future.
If you crack some other good places to get the word out, I’d be interested in hearing. My online circle is pretty small (lobste.rs and HN), but I’m working on something I want to ‘advertise’ the hell out of quite soon…
I’ve been trying to engage more with Reddit for this reason. I don’t really like it as a platform or see it as doing a social good, but there are users there and I’d like to be there to answer their questions. I was going to make a Twitter account, too, but they wanted my phone number and a pic of my ID and a blood sample to verify my account so I abandoned that. Finding good ways to ethically grow Sourcehut’s audience is not an entirely solved problem.
The reason Twitter – and many platforms – asks for phone numbers is because spam and trolls are a persistent problem. Ban one neo-Nazi troll tweeting obscenities at some black actor for DesTROyinG WhITe SocIEtY and they’ll create a new account faster than you can say “fuck off Nazi”.
Reddit is often toxic as hell by the way, so good luck with that.
Huh…I have a twitter account and all I needed for it was an email. Maybe things have changed.
Nowadays they let you in with just an email, but after some time “block” your account and only unblock it after you give your phone number.
While I also see it as an ad, I’m interested in what it being announced as a Sourcehut user. But it seems you don’t have a RSS/Atom feed for the official blog… Or is there a mailing list I missed?
https://sourcehut.org/blog/index.xml
I’ve been meaning to make this more visible…
hold pleasedone.Somewhat amusing that this post with an interesting fully FOSS service, is marked -29 spam, whereas an actual advertisement about Huawei making macbook clones that run Linux has only -3 spam (one of which is mine).
Said FOSS service has been on the Lobsters front page multiple times recently. I suspect the reaction is: “We get it, sr.ht exists and SirCmpwn is apparently desperate to attract a paying customerbase, but a clickbaity title for a blogspam ad on the usual suspect’s software is probably crossing the line.”
Anybody knows which bigcorp player he’s talking about?
My mailserver, for many months, could not send mails to outlook addresses. The outlook server replied “OK” but the mail was transparently discarded. Not inbox, not spam, not trash, nothing. As if the mail had never been sent.
I believe nowadays outlook “only” send my mails to spam.
I have had the same experience. With Gmail it was even more difficult to evade their hyper-aggressive spam filters.
I can’t call any of this “easy” and I had to struggle and learn a lot of new concepts (like DKIM, which is a pain to set up). It’s also very tricky to verify, if it fails it can fail silently; your mail is just dropped or goes to spam. I had that happen when my DNSsec signatures weren’t renewed, for example, and also when I had made a small mistake that made my DKIM invalid or not used (I don’t remember which).
You need to be an expert at mail before this stuff is “easy”. When you get redirected to the spamfolder, those hosts aren’t giving any information about why this happened, so you’re left guessing. Also, you sometimes don’t even know unless you’re in contact with the recipient in some other way than just e-mail (and sometimes people don’t bother to notify you that the mail got flagged as spam). There are tools out there that can help verify your technical setup like rDNS, SPF, DKIM etc. But it’s still annoying as fuck to get it set up. Once you’ve done the work, it basically runs itself though.
So I appreciate the article’s attempt to get more people to try hosting their own mail, I would say it’s quite one-sided and assumes a whole lot of technical sysadmin competency that the author has probably simply become blind to himself.
I had a similar problem and my solution was to route all mail to them via a separate, dedicated IP which didn’t suffer the same problem. A solution possible thanks to the flexibility of Exim. As much as these simpler MTAs seem attractive I wonder how they would cope with such scenarios.
I had this problem sending from my own mail server to Gmail addresses. After a couple of months I just gave up on my own mail server and went to mailbox.org