I agree with a lot of this, especially the root-cause analysis. Working with stacked PRs in GitHub is painful and the problem here is fundamental to git: commits do not have an identity, trees have an identity and commits are named as the tree to which they were applied. This means that you suffer a lot of rebasing pain when you make changes to one at the top, because now the versions of those commits in other branches are wrong.
I am not convinced that the solution to this is ‘buy another third-party product’. The right solution is probably to switch to Pijul, but that’s very high friction.
GitHub also makes this harder than it needs to be by modifying commit messages on merge (even with rebase) to include the PR number. Providing a fast-forward option for PRs (Azure DevOps has this - it’s the only feature that they have that I miss on GitHub) would at least avoid having to rebase after a PR is merged if it isn’t changed. annoyingly, if you just do a fast-forward on the main branch, GitHub correctly tracks the PR metadata, so this is purely a front-end issue.
I would really like a low-friction way of saying ‘this range of commits in a branch is a PR’. I think something like that could be made quite easily if you put some kind of marker on the final commit in a sequence. I’d then be able to do rebase locally and edit and gradually pull commits into the main branch.
I’m really surprised by how developers are so addicted to trying new shiny tools, but Git has been the de-facto standard in pretty much every company for the last 10 years or even more; I can’t think of another example of such a pervasive tool. (Not saying I think it’s necessarily bad).
My biggest gripe with Pijul is that the only documented way to collaborate with others on Pijul repos is Nest and there is absolute nothing about self-hosting it. Git is super simple with that regard (just create bare Git repo on machine with installed Git that is accessible via SSH and you are done), while Pijul do not describe such functionality at all. And this blog post make it even less appealing that you need to use proprietary service for simply hosting your repo.
GitHub also makes this harder than it needs to be by modifying commit messages on merge (even with rebase) to include the PR number. Providing a fast-forward option for PRs (Azure DevOps has this - it’s the only feature that they have that I miss on GitHub) would at least avoid having to rebase after a PR is merged if it isn’t changed. annoyingly, if you just do a fast-forward on the main branch, GitHub correctly tracks the PR metadata, so this is purely a front-end issue.
I’ve seen people do custom bots (I think possible with Github Actions) to work around that - someone comments a keyword on the PR, and the bot does the fast-forward and updates the target branch. As you say, it then even looks right in the PR view as having been merged.
commits do not have an identity,
Gerrit works around that by having a Change-Id in the commit messages (automatically added when creating the commit), which allows it to track commits across rebases and other modifications. I guess you could make some tools that do similar things to identify commits/branches with outdated ancestors and make sure you rebase them properly.
Gerrit works around that by having a Change-Id in the commit messages (automatically added when creating the commit), which allows it to track commits across rebases and other modifications. I guess you could make some tools that do similar things to identify commits/branches with outdated ancestors and make sure you rebase them properly
I wish git would standardise something like this. It’s also important if you’re back-porting security fixes across branches. I really want to have a way of knowing whether a commit that introduced a bug was back-ported and then of guaranteeing that the security fix is in every branch that the feature was merged into.
I’ve been using Pijul on a project recently after doing two in darcs & the CLI UX seems unfinished in comparison. It sucks, since the data model & performance are great.
Having the keystore for identity is nice, as is channels for hopping around on different features, but
there is only the record hooks so you can’t automate stuff on the upstream server on push (i.e. kicking off a CI)
there’s no UX to send patches (i.e. via email with a leading message not suited for the commit message)
you can get used to recording opening $EDITOR but weird-TOML is an ugly format to work with for commits
diff isn’t compatible with existing patch-based tooling with no alternative/compat option & opens straight to the pager instead of letting you decide (darcs diff --diff-command "kitty +kitten diff %1 %2" can get you a wonderful highlighted diff session)
it would be nice to see aliases so I could do pijul amend instead of pijul record --amend
darcs rebase have the ability to pop off patches, amend a commit message (or code) then replay the next patches without issue; you can directly pijul record --amend … --message … up until it has a dependency, then a typo I guess is set in stone & it’s a pain unlike git rebase -i … which lets you fix commit messages & apply fixups (not sure how commit message make the code not apply)
if trying to fix/amend a quickly-caught issue, pijul unrecord --reset … to pop off commits then Pijul hangs on next push to push the amended changes… can’t easily obliterate from local machine to upstream & amend commits just get shoved atop rather than see this as something akin to git push --force-*.
I have a tendency to commit kinda early & often as well as pushing to my 2nd machine as a backup, & use amending in Git & darcs to fixup my patchsets before sending to others or the canonical repo. That said, both of these tools are great, underrated, & folks should feel encouraged to try something outside the status quo as at least it will give you a new perspective on Git (i.e. learning you a Haskell doesn’t mean you’re going to switch).
GitHub also makes this harder than it needs to be by modifying commit messages on merge (even with rebase) to include the PR number
Is this a repo setting? I have never seen a commit message modified by GitHub, as otherwise the commit hash would change and when I pull main locally, my branches wouldn’t show merged into main (due to commits differing)
When GH does actual merge with a merge commit, then original commits are unchanged and the merge commit has the PR number in message (not sure what it does when it can do a fast-forward instead of merge, though).
When you click “squash and merge”, the squashed commit’s message is obviously generated by GH (PR number + title and list of summaries of PR’s commits by default). Not sure about “rebase and merge” (where rebase itself would also alter commit IDs).
And yes, it’s frustrating when I can’t use native git mechanisms to delete my local branches that are merged. The “squash and merge” also kills ancestry, so I can’t use git subtrees in repos that use that. And it breaks some flows that rely on branch ancestry. The upside is a neat linear main branch and fewer weird merge conflicts across the team. I would still love a solution that makes main branch nice without breaking git itself.
While interesting, Pijul is more of the same. In the same way something like Elixir offers massive benefits over the normal model of taping discrete parts (DB, reddis, services) together and throwing them in containers, there are other possibilities like an AST based system (and others we’ve not thought of yet).
I’m aware of some semantic dif tools like graphtage and gumtree exploring this space, but e.g. combining it with an AST based programming language (Racket seems to be going in this direction) could be phenomenal. Indeed, we can go further instead of IDEs like VSC or even Plan9’s Acme, we could see all our code as an AST we freely manipulate.
There are also budding efforts in this direction:
a recent editor called “treefrog” (site’s dead) which allowed for tree editing commands
Structural diffing would be awesome. No longer would I need to sit in space-indented code or see uncuddled curly braces, or you could align those record key/values on = but don’t since it can lead to diff issues in the future just to touch alignment. The idea of having my preferred formatting for my needs on my end would make me happier & prevent less fighting.
This news is too convenient to generate outrage at the evil manufacturer corporation, so it makes me suspect there’s more at play here. For example, if there are third party shops that e.g. refuse to change braking pads and simply fool the firmware into accepting used ones, and the train crashes, the manufacturer might be held responsible - if not by the courts, then at least in a PR sort of way. If there’s no certification process and the repair shops are just random companies that don’t know what they’re doing, it might be actually sensible to prevent them from doing maintenance. Of course, they should have built in a different way of doing this, for example in the contracts, or with a clearer mesage in the firmware.
Why should that be solved by sneaky software and not just a contract? Possibly amended by not so sneaky software to remind you of said contract.
There are very good reasons to shun third party workshops, but what if they’re the only option at one point? The company goes bust and Poland is stuck with randomly locking trains forever?
For a long time servicing the trains was done solely by the manufacturers, as they argued that the service manual was their intellectual property. Since, the courts have decided that no, they have to provide this, and other companies could enter the train servicing market. Newag’s hand forced, they complied, with bitter taste in their mouth.
The service manual handed over neglected to mention that some of the trains will lock up if left in place for 10 days, and that they’ll have to be unlocked with an undocumented sequence of button presses in the cabin – the assumption behind this being, trains don’t just stand around unused if they’re not being serviced. After this assumption turned out to be not entirely true, a newer version of the firmware included checks on the GPS position of the trains, with hardcoded coordinates of some competitors’ workshops.
Not sure if you’ve read the more detailed article, or only the Mastodon posts, but I recommend putting it through Google Translate if you can’t read Polish.
The Newag Impuls trains breaking randomly have been going on for over a year. Mysteriously, only third-party workshops had issues with this. Newag always said “see, they’re just incompetent”. An unspoken suspicion has long formed in the industry that some underhanded tactics are at play, but until now, nobody had any idea how to go about proving this.
Newag replied in a press release that the third parties could be responsible for inserting this code into the firmware. I can’t decide who can be trusted here. Both scenarios seem plausible.
That is pretty much pointless TBH, especially that they were running against the schedule and they were close to being late on fixes. With that in mind that looks especially dumb. Also, the bogus code was found in other trains, that weren’t serviced by SPS.
Also I do not get what SPS is supposed to gain there. They aren’t the only train servicing shop in Poland, so it is not that anyone who is using NEWAG trains will go and service their trains at SPS. SPS has next to nothing to gain there, it is only NEWAG that can lose. And SPS isn’t even direct competitor of NEWAG, as NEWAG is train manufacturing company while SPS is just servicing company. It is like saying that iFixIt fight for Right to Repair is a plot to ruin reputation of Apple/Samsung/other producers so only they can service smartphones.
it might be actually sensible to prevent them from doing maintenance
3rd party repair has been one of the conditions of the train tender. Newag shouldn’t have offered their trains if they were not OK with the terms. Period.
I read the entire thing and I’m not sure what it is doing or what we should be doing.
The only command that’s in there is: nix flake init -t github:nickel-lang/organist but that’s I guess how you setup an organist project, not how you use it? Then you use it regularly with nix develop?
Update: I think if you read the README here, it becomes clear: https://github.com/nickel-lang/organist Still not really clear whether or how it’ll fill many of my development needs.
Ncl
I browsed Nickel documentation previously but still, constructs like this leave me rather mystified:
services.minio = nix-s%"
%{organist.import_nix "nixpkgs#minio"}/bin/minio server --address :9000 ./.minio-data
"%
What is happening here with the %s?
I’d say in general that Nickel may be a great idea and it looks less offputting than Nixlang but it’s still very far off from something a large audience of people can use.
Recently I saw Garn which is a “Typescript eats the entire world” approach to this problem. I’m also very sceptical of it as an abstraction layer, but the choice of language does look like it could be a winner. It reminds me a bit of CDK/Typescript which is a weird imperative/declarative hybrid alternative to the standard terrible Devops ways of defining infrastructure.
My impressions as well. I’m not sure if this competes with devenv, devbox, and others, or is some completely different thing. If former, what does it bring over other tools.
Similar thoughts. Even as a Nix user I’m confused about some of the syntax I’m unfamiliar with, and generally about what Organist is trying to be.
If it’s a layer above Nix flakes dev shell configuration like some of the other projects, it seems like a hard sell as if you can do Nickel… you probably can do Nix already, and introducing an extra layer is neither here or there. If you go json/yaml it will be dumbed down but easier to consume for non-nixers, and if you go Nix - you are 100% seamless with Nix.
BTW. I’m causally lurking into Nickel and I’m still confused w.r.t level of interoperability with Nix. Nickel presents itself like a Nix-like “configuration language”, which means … it can’t really do some of the things Nix do? Or can it? Can it transpile to Nix or something?
My take is that yes it’s competing with those tools, but in a (nearly) native nix way, nearly as it depends on Nickel tooling, but the generated flake pulls that in automatically so there’s nothing else to install.
At work I am using Devenv mostly for process support (which ironically I don’t need any more) and it fits the bill, but IS two things to install before team members can start developing (plus direnv). This would only be one thing to install.
At home I run NixOS and just use a flake for my dependencies but that doesn’t launch any services so I am kind of keen on using organist if I ever need that.
My take is that yes it’s competing with those tools, but in a (nearly) native nix way, nearly as it depends on Nickel tooling, but the generated flake pulls that in automatically so there’s nothing else to install.
It’s very cool that this works so you can have your flake contents defined in some other language entirely and don’t have to think about it (if it works).
I like idea of wearings (I need to add one to my website), but for me it looks painful, that it requires so much manual steps, often involving GitHub repository or other. I believe that most of these could be automated to some degree.
The whole point is to remove the automation, though. Webrings should be manually vetted collections of content that one or more humans have decided is cool.
While I agree with the title, the points there are mostly pretty weak IMHO.
Use “vanilla” approaches to achieve desired functionality over external frameworks
Mostly agree. Often that is not needed.
Where possible, default to defining style and behaviour with inline HTML attributes
Partially agree. Use semantic HTML, but for gods’ sake, do not use on*= attributes with inline JS.
Where libraries are necessary, use libraries that leverage html attributes over libraries built around javascript or custom syntax
Partially agree, but with stuff like Hyperscript that do not differs much from on*= attributes and that would be quite painful in larger scale IMHO. Instead try to make interactivity declarative with custom attributes (more like HTMX/Alpine.js is doing). But in general try to avoid relying on JS at all.
Steer Clear of Build Steps
Partially agree. Modern pipelines are often way too complex. However I think that some build step, especially with simple (low to zero config) tools, is useful. I mostly use PostCSS to squash CSS into single, minimised, file, as well as to provide some polyfils for older browsers, which allows me to use fancy new features now. And these features greatly simplify my CSS and working with it (cascading layers, CSS nesting, some autoprefixing for some features).
Prefer “naked” HTML to obfuscation layers that compile down to HTML
If your server side rendering engine allows you to remove boilerplate and repetitions, then it is often IMHO way better. Especially that comparison there isn’t apples to apples, as it removed i18n features from f.label :field helper.
I do not disagree with the premise, but the article does hardly any job there except saying “they are old, weak and slow while we are new, secure and fast” with a single sentence from something that most non crypto related people probably not even heard of and single bar graph. IMHO not the best article.
A screen-reader user could chime in: would this require interaction by the user? (e.g. using a command to read the spoiler content). Is that user friendly?
On the other hand, for screen reader there is currently no way to mark something as a spoiler right now. So if person with sight disabilities tries to read community channels about their favourite book series or something they are stepping into minefield of spoilers that their readers cannot prevent.
I think this is where the idea has the most potential value— being able to say “this is where spoilers end” is just as important as saying “this is where spoilers start.” Being able to know that without reading, hearing, or seeing the parts in between is ideal.
I endorse this general approach: not always, but often, cleanly generated files in the source root are easier to work with than unreadable garbage tucked in a corner of a build dir.
Additionally, for libraries, this approach can often massively simplify life for the consumers, as they don’t need to depend on code used to generate stuff.
However, I would recommend a different implementation: instead of writing GitHub action, write this as a test. Specifically:
generate source file in a test
compare with what’s on disk
if the contents is the same, do nothing
otherwise, update the file on disk, and then fail the test.
Benefits of this approach:
piggy backs on the standard go test flow instead of adding a project-specific concern
independent of a specific CI provider and works locally
The file is infrequently changed, to not pollute most commits diff.
You can use .gitattributes to disable local diffing for a file, and to tell git forges that a file is generated:
src/gen/foo.c -diff linguist-generated
With -diff, git diff will just remark something like “Binary file has changed”, and linguist-generated tells GitHub and Gitea to exclude a file from web diffs and from stats.
Couldn’t you easily end up with a “wedged” build? Say I changed the input to the code generator, for example, I’ve added a data member to a protobuf spec, and adjusted my code to reference the new member. Now my project doesn’t build because the generated code doesn’t yet have this new member. But I also cannot run the test to re-generated it because the project doesn’t build. I need to remember to change the spec, run tests, only then change the code. This screams “hack”!
Feel like the proper way to handle this is to use a proper build system. We do something similar to what you have described (keep pre-generated code up-to-date, fail the build if anything changes) for a tool that uses its own generated code: https://git.codesynthesis.com/cgit/cli/cli/tree/cli/cli/buildfile#n98
You indeed could get a broken build, but it’s roughly equivalent in frequency and annoyance to, e.g., accidentally committing a file with merge conflict markers.
If you add this to the default build task, which is run unconditionally, this has the problem that your consumers now have to run code generation as well just to build your stuff. It is a fairly frequent annoyance for Rust that a build.rs of my dependency has a boat load of dependencies, all of which could have been avoided if the author of the library would have run build.rs logic on their machine instead (which is not always possible, but occasionally is)
If you add this as a dedicated make generate task, then the you’ll need to adjust you CI as well, which is a smell pointing out that local workflows need subtle adjustment as well.
That’s why I suggest tacking onto make test: this is an already existing build system entry point for checking self-consistency of a particular commit.
If you add this to the default build task, which is run unconditionally, this has the problem that your consumers now have to run code generation as well just to build your stuff.
Not if you only enable code generation for the development builds. Our setup is:
Consumer builds use pre-generated code.
Development builds update generated code if inputs change, compare the result with pre-generated, if there are difference, copy the generated code from output to source and fail the build.
Oh, that’s curious! I don’t think I’ve ever seen distinction between “development” and “consumer” builds before!
Is this a first class concept in build2, or is it just some custom config for this specific build file? Are there any docs about other uses-cases for this separation?
I don’t think I’ve ever seen distinction between “development” and “consumer” builds before!
Well there are Rust’s dev-dependencies. Though, IMHO, the semantics (or the name) is misguided – tests are not only to be run during development.
Is this a first class concept in build2, or is it just some custom config for this specific build file?
It is a first-class concept, though a pretty thin one: we simply reserved the config.<project>.develop variable to mean “development build”. In build2 we split the package manager into two tools: for consumption and for development. The latter by default configures projects with config.<project>.develop=true.
Are there any docs about other uses-cases for this separation?
So far we have used it for pre-generated source code as well as to support additional documentation outputs that require additional/non-portable tools. For example, we can produce XHTML portably but to convert that to PDF requires html2ps and ps2pdf so we only enable this step (and require these tools) in the development builds.
EDIT: Forgot to mention, you can also use config.<project>.develop (like any other configuration variable) for conditional dependencies:
Oh, that’s curious! I don’t think I’ve ever seen distinction between “development” and “consumer” builds before!
Autotools has had this distinction, I guess basically forever, the result of a make dist can be thought of as a “consumer” build. You don’t need automake/autoconf to do the ./configure && make && make install dance, but it is likely you’ll need it during some point if you’re doing development on the project.
The end user / “release” build is a tarball that builds with on only a C++ compiler and a shell. You don’t need Python or any tools written in Python to build it.
I think there are some downsides in that I want people to open up the tarball and be able to contribute without necessarily getting git involved … But I think this is less common than it used to be. But not unheard of for one-off hacks.
Thankfully all our generated code is readable and debuggable with normal tools
I cannot overstate how good Mercurial (with extensions hggit and hg-evolve) is as a Git interface, especially for history editing such as rebasing.
Take the “split commit while rebasing” task mentioned by my sibling commenter @anordal. The Git command is apparently called ‘revise –cut REV – FILES’; in Mercurial you achieve the same with hg split --rev REV.
Demonstration: I’d like to clone the history below, rebase branch bread onto main, and meanwhile split commit 5. (Note: the short numbers are real, Mercurial has local sequential rev numbers as well as globally-unique hashes.)
o 7 Eat (bread)
o 6 Bake
o 5 Pre-heat oven + let dough rise # Needs to be split
o 4 Knead dough
o 3 Wake yeast
| o 2 Build bakery (main)
| /
o 1 Initial commit
Here come the changes. We’re going to clone the git repo, make our changes with Mercurial, and push them back to the upstream Git repo.
## Clone and enter repo
hg clone git+ssh://githost/baker/breadrepo
cd breadrepo
## Make changes
# Rebase commits 3-5 onto main. 6-7 stay behind as orphans.
hg rebase --rev '3::5' --dest main
# Interactively split the 'rise bread + pre-heat oven' commit
# The UI looks like this:
# https://ludovic.chabant.com/blog/2019/04/09/190631/
hg split --rev 'desc("Pre-heat")' # or just --rev 9 (get the number from the log)
# Rebase the rest of 'bread' onto 'tip' (the newest
# commit, in this case created by splitting)
# Rebases only remaining orphans;
# ignores commits that that already have a successor in the destination.
hg rebase --rev 5::--dest tip
## Push rebased branch to upstream
hg push -B bread -f
After the hg split, before rebasing the rest of the changes, the repo looks like this. So you can see which commits are obsolete (‘x’ nodes) or still need to be rebased (’*’ nodes).
@ 11 (tip) Let dough rise
o 10 Pre-heat oven
o 8 Knead dough
o 7 Wake yeast
| * 6 (bread) Eat
| * 5 Bake
| x 4 Pre-heat oven + let dough rise
| x 3 Knead dough
| x 2 Wake yeast
o | 1 (main) Build bakery
|/
o 0 Initial commit
Also check out Jujutsu VCS for a Git-compatible, Mercurial-inspired VCS solution.
For interactive switches, I particularly like that it has short commit/change identifiers, and that change identifiers are stable across rebases, so you can write something like jj rebase -s ab -d cd && jj new ab to rebase ab in memory and then switch to change ab.
Weird idea (I do not how it will work in practice though) - add command to git-branchless that would allow to jump to the commits using their tree hash. That has will not be unique, so we probably should exclude the merge commits, but within git-branchless-smartlog entries it should be small enough to not have a lot of conflicts. And that would provide semi-consistent hashing for “recent commits” in git-branchless.
And lastly, here’s some more settings for your ~/.hgrc.
hg l prints a pretty graphical log with colors
hg bm is an alias for hg bookmark – if you’re working with Git branch this is how you’ll create and move them.
hg newcommit creates dummy commits. Useful for quickly creating some history to experiment on.
The rest of the file sets up the default colours, and some stuff I use in the log template.
[alias]
# `hg l` shows a compact log with colors
l = log --graph --template shortgit
bm = bookmark
# Quickly create history
newcommit = !
# new rev number: current tip commit + 1
rev="$($HG nextrev)"
touch f$rev;
$HG add f$rev;
$HG commit -m "Commit $rev";
nextrev = !
echo $(( $($HG id --num --hidden -r tip) + 1 ))
[revsetalias]
## A revset queries the commit graph for matching commits.
## Define your own revset in terms of existing revsets.
## See 'hg help revsets'.
# In hg-git repos, remote branch pointers are represented as Hg tags of the form
# `default/...` or `origin/...`.
# So let's find all Hg tags containing `/`; they and their ancestors are public.
gitpublic() = ancestors(tag('re:/') or gittag())
[templates]
## Named templates
# See [colors] for label<-> color mapping.
# This setup uses green tags, light cyan bookmarks, dark cyan authors, red dates.
# Draft/unpushed commits are yellow, public/upstreamed commits are white
shortgit = '\
{rev}:(git {label("changeset.{true_phase}", gitnode|short)}) \
{maybe_bookmarks}\
{maybe_tags}\
-- \
{label("log.author", author|person)} -- \
{label("log.date", date(date, "%Y-%m-%d %H:%M"))}\n\
_ {label("log.desc", desc|firstline)}'
[templatealias]
## Define your own template keywords in terms of existing template keywords
## See 'hg help templates'
# phase detection that is robust to hg-git repos
true_phase = ifcontains(rev, revset("gitpublic()"), "public", "{phase}")
# This is what the maybe_x templates do: if x exists, print x followed by a
# space. If it doesn't, print neither the x nor the space.
# Example usage of the maybe() helper:
# hg log -r . -T '{maybe("bookmark", "<bm>", bookmarks, "</bm>")}'
# The maybe() helper is inspired by hg_prompt, which has a syntax
# `{prefix{item}suffix}` for prefixes/suffixes that should only be rendered if
# {item} is nonempty
maybe(mylabel, prefix, main, suffix) = '\
{ifeq(main, "", "", label(mylabel, prefix))}\
{label(mylabel, main)}\
{ifeq(main, "", "", label(mylabel, suffix))}'
maybe_bookmarks = maybe("log.bookmark", "", bookmarks, " ")
maybe_tags = maybe("log.tag", "", tags, " ")
[color]
## Templates label
# one of auto, ansi, win32, terminfo, debug
mode = terminfo
# Colours for each label
log.author=cyan
log.branch=green # Mercurial branch, baked into a commit
log.summary=white
log.desc=yellow
log.bookmark=cyan bold # Movable branch pointer, like Git branches
log.tag=green bold
log.topic=white bold blue_background
log.graph=blue
log.date=red
changeset.public=
changeset.secret=blue bold
changeset.draft=yellow bold
I’m in charge of the publication of Goguma on iOS. We have an issue tracking all the features not available yet on iOS: https://todo.sr.ht/~emersion/goguma/138
We are still working on notification support, as iOS has strict requirements for background tasks. Besides that, it works decently well and has good accessibility.
Had a look at it: their extension is unfortunately just sending requests to a closed source API that they host, which is supposedly doing the actual APNS requests :(
$WORK: Refactorings and cleanups of Logflare. Fixing few bugs after migration from Cowboy to Bandit.
$NON_WORK: implementation of 9p2000 for Erlang to implement something like procfs for Erlang. There exists implementation that use FUSE for that, but that requires compiling C code and that can be problematic. 9p2000 allows me to implement it in pure Erlang.
I probably posted it previously, but on macOS I have prepared launch agent, that will automatically sent your local ~/.plan to https://plan.cat on edit:
Based on the comments regarding who has access, I’d say using Cloudflare would help in this scenario. It seems that this website has decided to block entire parts of the globe to mitigate bad behavior, something using Cloudflare would enable without geoblocking.
There are a lot of reasons to not use Rust, but this post does not list them out. Speaking as someone who has used Rust professionally for four years, this is my take on these points:
Rust is (over)hyped
The rationale being that it’s stackoverflow’s most loved language and 14th most used… Its a growing language, and seeing more and more industry adoption. Currently, due to posts like this, it is a hard sell to management for projects which aren’t low-level, despite developers loving using it (and I’ve been told multiple times that people feel far more comfortable with their Rust code, despite not being experts in the language). Rust might be overhyped, but the data provided to back up this claim is just not correct.
Rust projects decay
I know this person has written a book on Rust, but… I have to question what the hell they’re talking about here. The steady release cycle of the Rust compiler has never once broken my builds, not even slightly. In fact, Rust has an entire Epoch system which allows the compiler to make backwards-incompatible changes while still being able to compile old code.
I mean, seriously, I genuinely don’t know how the author came to this conclusion based on the release cycle. Even recent releases don’t have many added features. Every project I have ever come across developed in Rust I’ve been able to build with cargo build and I’ve never once thought of the version it was developed with or what I had in my toolchain. Python 3 has literally had a series of breaking changes fairly recently and its being compared as a language doing it “better” because it has fewer releases.
Rust is still beta (despite the 1.0)
sigh. Because async traits aren’t stabilized? Even though there is a perfectly workable alternative, async-traits crate, which simply makes some performance trade-offs? I’m excited for async traits being stabilized, and its been a bummer that we haven’t had them for so long, but that doesn’t make it Beta.
The standard library is anemic
This is just an opinion the author has that I strongly disagree with (and I imagine most Rust developers would). The standard library is small, this was/is a design decision with a number of significant benefits. And they do bring 3rd party libraries into the standard library once they are shown to be stable/widely used.
async is hard
To put this more accurately, Rust forces you to write correct async code, and it turns out correct async code is hard. This is an important distinction, because a language like Go makes it just as easy to write incorrect async code as it does correct async code. Having been bitten by enough data races and other undefined behavior in my lifetime, I love Rust’s stance on async code, which is to make it hard to do incorrectly with minimal runtime overhead.
Frankly, the Rust compiler is some incredible engineering that is also pushing the bounds of what a programming language can do. I mean, seriously, as frustrating as async Rust can be to work with, it is an impressive feat of engineering which is only improving steadily. Async Rust is hard, but that is because async code is hard.
[edit] Discussed below, but technically Rust just prevents data-races in async code, and does not force you to write code which is free from race-conditions or deadlocks (both of which are correctness issues). Additionally, the “async code” I’m talking about above is multi-threaded asynchronous code with memory sharing.
Frankly, the points being made in this post are so shoddy I’m confused why this is so high on lobsters. The anti-Rust force is nearly as strong as the pro-Rust force, and neither really contributes to the dialog we have on programming languages, their feature-set and the future of what programming looks like.
Use Rust, don’t use Rust, like Rust, don’t like Rust, this post is not worth reading.
To put this more accurately, Rust forces you to write correct async code, and it turns out correct async code is hard. This is an important distinction, because a language like Go makes it just as easy to write incorrect async code as it does correct async code.
I have not written any production Rust code (yet) but the “async is hard” resonates with me. I’ve wrestled with it in C, C++, Java, and Go and it’s easy to make a mistake that you don’t discover until it’s really under load.
it’s easy to make a mistake that you don’t discover until it’s really under load.
I think you really hit the nail on the head with this point. The particularly damning thing about data-race bugs is that they are probabilistic. So you can have latent code with a 0.0001% chance of having a data-race, which can go undetected until you reach loads which make it guaranteed to occur… And at that point you just have to hope you can (a) track it down (good luck figuring out how to recreate a 0.0001% chance event) and (b) it doesn’t corrupt customer data.
There is a reason so many Rust users are so passionate, and its not because writing Rust is a lovely day in the park every day. Its because you can finally rest at night.
I’m in the process of switching to Rust professionally, dabbled for years and the biggest selling point of Rust is it’s ability to help me write working software with little or no undefined behaviour.
Most languages let you build large applications that are plagued by undefined behaviour.
Async Rust is hard, but that is because async code is hard.
I think this is debatable. Async rust introduces complexities that don’t exist in other async models which rival in ease-of-use and efficiency. It has unintuitive semantics like async fn f(&self) -> T and fn f(&self) -> impl Future<Output=T> subtly not being the same; the latter is + &'self_lifetime for the Future). It also allows odd edge-cases that could’ve been banned to simplify the model:
Future::poll() can be spuriously called and must keep stored the most recently passed in Waker. Being two words in size, you can’t atomically swap them out resulting in requiring mutual exclusion for updating and notification. Completion based async models don’t require this.
Waker is both Clone and Send, meaning it can outlive the task which owns/polls the Future. This results in the task having to be heap allocated (+ reference counted to track outstanding Wakers) if it’s spawned/scheduled generically. Contrast this with something like Zig async or rayon’s join!() which allow stack-allocated structure concurrency. Waker could’ve been tied to the lifetime of the Future, forcing leaf futures to implement proper deregistration of them but reducing task constraints.
The cancellation model is also equally simple, useful, and intricately limiting/error-prone:
Drop being cancel allows you to stop select!() between any future, not just ones that take CancellationTokens or similar like in Go/C# (neat). Unfortunately, it means everything is cancellable so a().await; b().await is no longer atomic to the callee (or “halt-safe”) whereas it is in other async models.
It also means you can’t do asynchronous cancellation since Drop is synchronous. Future’s which borrow memory and use completion-based APIs underneath (e.g. Overlapped IOCP, io_uring) are now unsound if cancelled unless you 1) move ownership of the memory to the Futures (heap alloc, ref counting, locked memory) 2) block in Drop until async cancellation occurs (can deadlock runtime: waiting to drive IO but holding IO thread).
Sure, async is hard. But it can be argued that “Rust async” is an additional type of hard.
Async rust introduces complexities that don’t exist in other async models which rival in ease-of-use and efficiency.
I don’t disagree Rust introduces an additional kind of hard: async Python is much easier to use than async Rust. I wrote in another comment how it all comes down to trade offs.
I do agree with you, there are more sharp edges in async Rust than normal Rust, but from my understanding of how other languages do it no language has a solution without trade offs that are unacceptable for Rust’s design.
Personally, I think async is the wrong paradigm, but also happens to be the best we have right now. Zig is doing interesting things to prevent having the coloring problem, but I don’t think any language is doing it perfectly.
Async Rust is hard, but that is because async code is hard.
Any data to back that exact claim? I love rust and I’m working professionally in it for last few years but I think I would still find erlang approach to async code easier.
Fair point, and to really dive into that question we have to be more specific about what exactly we’re talking about. The specific thing that is hard is multi-threaded asynchronous code with memory sharing. To give examples of why this is hard, we can just look at the tradeoffs various languages have made:
Python and Node both opted to not have multi-threading at all, and their asynchronous runtimes are single-threaded. There is work to remove the GIL from Python (which I actually haven’t been following very closely), but in general, one option is to avoid the multi-threading part entirely.
Erlang/BEAM (which I do love) makes a different tradeoff, which is removing memory sharing. Instead, Erlang/BEAM processes are all about message-passing. Personally, I agree with you, and I think the majority of asynchronous/distributed systems can work this way effectively. However, that isn’t to say it is without tradeoffs, message passing is overhead.
So essentially you have two options to avoid the dangerous shenanigans of multi-threaded asynchronous code with memory sharing, which is to essentially constrain one of the variables (multi-threading or memory sharing). Both have performance trade-offs associated with them, which may or may not be deal-breaking.
Rust lets you write multi-threaded asynchronous code with memory sharing and write it correctly. In general though I agree with you about the Erlang approach, and there isn’t really anything stopping you from writing code in that way with Rust. I haven’t been following this project too closely, but Lunatic (https://github.com/lunatic-solutions/lunatic) is a BEAM alternative for Rust, and last I checked in with it they were making great progress.
Yes, I can agree that “multi-threaded asynchronous code with memory sharing” is hard to write. That’s a much more reasonable claim.
The only thing I would disagree slightly is the assertion that rust solves this problem. That’s not really completely true, since deadlocks are still just as easy to create as in c++. For that the only sort of mainstream solution I can think of is STM in Clojure (and maybe in Haskell?).
I hadn’t heard of STM, but that is a really cool concept bringing DB transaction-notions to shared memory. Wow I need to read about this more! Though I don’t think that solves the deadlock problem globally, as if we’re considering access which is not memory (eg. network), and thus not covered by STM, then we can still deadlock.
From my understanding, solving deadlocks is akin to solving the halting problem. There just simply isn’t a way to avoid them. But you are right, Rust doesn’t solve deadlocks (nor race conditions in general), just data-races. I’ll modify my original text to clarify this a bit.
Bear in mind, though, that STM has been through a hype cycle and some people are claiming that, like String Theory, it’s in the “dead walking” phase rather than past the hype. For example, Bryan Cantrill touches on transactional memory in a post from 2008 named Concurrency’s Shysters.
So fine, the problem statement is (deeply) flawed. Does that mean that the solution is invalid? Not necessarily — but experience has taught me to be wary of crooked problem statements. And in this case (perhaps not surprisingly) I take umbrage with the solution as well. Even if one assumes that writing a transaction is conceptually easier than acquiring a lock, and even if one further assumes that transaction-based pathologies like livelock are easier on the brain than lock-based pathologies like deadlock, there remains a fatal flaw with transactional memory: much system software can never be in a transaction because it does not merely operate on memory. That is, system software frequently takes action outside of its own memory, requesting services from software or hardware operating on a disjoint memory (the operating system kernel, an I/O device, a hypervisor, firmware, another process — or any of these on a remote machine). In much system software, the in-memory state that corresponds to these services is protected by a lock — and the manipulation of such state will never be representable in a transaction. So for me at least, transactional memory is an unacceptable solution to a non-problem.
As it turns out, I am not alone in my skepticism. When we on the Editorial Advisory Board of ACM Queue sought to put together an issue on concurrency, the consensus was twofold: to find someone who could provide what we felt was much-needed dissent on TM (and in particular on its most egregious outgrowth, software transactional memory), and to have someone speak from experience on the rise of CMP and what it would mean for practitioners.
I think you’ll find Erlang much harder tbh. Have you used it much? Erlang requires that you do a lot of ‘stitching up’ for async. In Rust you just write .await, in Erlang you need to send a message, provide your actor’s name so that a response can come back, write a timeout handler in case that response never comes back, handle the fact that the response may come back after you’ve timed out, decide how you can recover from that, manage your state through recursion, provide supervisor hierarchies, etc.
Fortunately, almost all of that is abstracted away by gen_server you in practice you don’t actually do all that boilerplate work yourself, you just take advantage of the solid OTP library that ships with Erlang.
For sure I have way more experience with Rust, but I’m not really sure that all of what you listed is downside or Erlang specific. You also need to handle timeouts in rust (eg. tokio::time::timeout and something (match?) to handle the result), you might also need to handle possibility that future will be canceled. Others like recursion (which enables hot reloads) and supervisors are not obvious negatives to me.
Handling a timeout in Rust is pretty trivial. You can just say timeout(f, duration) and handle the Result right there. For an actor you have to write a generalized timeout handler and, as mentioned, deal with timeouts firing concurrent to the response firing back.
I think for the most part handling cancellation isn’t too hard, at least not for most code. Manual implementors of a Future may have to worry about it, but otherwise it’s straightforward - the future won’t be polled, the state is dropped.
In Rust you just write .await, in Erlang you need to send a message
TBH I do not see difference between these two.
provide your actor’s name so that a response can come back
You can just add self() as a part of message.
write a timeout handler in case that response never comes back,
As simple as adding after block to the receive block.
handle the fact that the response may come back after you’ve timed out
Solved in OTP 24 with erlang:monitor(process, Callee, [{alias, reply_demonitor}]).
decide how you can recover from that
In most cases you simply do not try to recover from that and instead let the caller to do that for you.
Simplest async-like receive looks like, from the docs:
server() ->
receive
{request, AliasReqId, Request} ->
Result = perform_request(Request),
AliasReqId ! {reply, AliasReqId, Result}
end,
server().
client(ServerPid, Request, Timeout) ->
AliasMonReqId = monitor(process, ServerPid, [{alias, reply_demonitor}]),
ServerPid ! {request, AliasMonReqId, Request},
%% Alias as well as monitor will be automatically deactivated if we
%% receive a reply or a 'DOWN' message since we used 'reply_demonitor'
%% as unalias option...
receive
{reply, AliasMonReqId, Result} ->
Result;
{'DOWN', AliasMonReqId, process, ServerPid, ExitReason} ->
error(ExitReason)
after
Timeout ->
demonitor(AliasMonReqId),
error(timeout)
end.
The difference is huge and kind of the whole selling point of actors. You can not share memory across actors, meaning you can not share state across actors. There is no “waiting” for an actor, for example, and there is no way to communicate “inline” with an actor. Instead you must send messages.
You can just add self() as a part of message.
Sure, I wasn’t trying to imply that this is complex. It’s just more. You can’t “just” write .await, it’s “just” add self() and “just” write a response handler and “just” write a timeout handler, etc etc etc. Actors are a very low level concurrency primitive.
As simple as adding after block to the receive block.
There’s a lot of “as simple as” and “just” to using an actor. There’s just.await in async/await. If you add a timer, you can choose to do so and use that (and even that is simpler as well).
The tradeoff is that you share state and couple your execution to the execution of other futures.
Solved in OTP 24 with erlang:monitor(process, Callee, [{alias, reply_demonitor}]).
It’s “solved” in that you have a way to handle it. In async/await it’s solved by not existing as a problem to begin with. And I say “problem” loosely - literally the point of Erlang is to expose all of these things, it’s why it’s so good for writing highly reliable systems, because it exposes the unreliability of a process.
It takes all of this additional work and abstraction layering to give you what async/await has natively. And that’s a good thing - again, Erlang is designed to give you this foundational concurrent abstraction so that you can build up. But it doesn’t change the fact that in Rust it’s “just” .await.
Sure, I wasn’t trying to imply that this is complex. It’s just more….Actors are a very low level concurrency primitive.
Sure, if you pretend you have to raw-dog actors to do concurrency in Erlang, and that OTP doesn’t exist and take care of almost all the boilerplate in gen_server etc. We could also pretend that async/await syntax doesn’t exist in Rust and we need to use callbacks. Wow, complex!
Perhaps the best example is 3.6 introducing async and await as keywords in the language (and thus breaking code which used them for variables). In Rust, this was done via the 2018 Edition with the Epoch system.
The difference is a Python 3.6+ interpreter can’t run code from Python 3.5 using async/await as non-keywords, while Rust can compile a mix of 2015 Edition and 2018 Edition code with some using async/await as non-keywords and others with it.
Nix is only as deterministic as it’s builders and compilers that’s true. It gives you the scaffolding you need but you still have to ensure you follow the rules. If you use a builder that creates random data somewhere then you still won’t have a deterministic output. That’s true of pretty much anything that tries to solve this problem though. They are working with tools that make it a little too easy to violate the rules unexpectedly.
Yeah I was actually surprised by this, from some brief experience with a shell.nix
It seems like it is extremely easy to write Nix files “wrong” and break the properties of Nix
Whereas with Bazel, the build breaks if you do something wrong. It helps you write a correct configuration
Similar issue with Make – Make provides you zero help determining dependencies, so basically all Makefiles have subtle bugs. To be fair Ninja also doesn’t strictly enforce things, but it’s easy to read and debug, and has a few tools like a dep graph exporter.
nix-shell is a bit of a lukewarm compromise between Nix’s ability to supply arbitrary dependencies and the guarantees of the build sandbox. It’s not really trying to be hermetic, just ~humane.
I would like to hear more about your shell.nix experience. It has some issues, for sure, but I have found the platform to be extremely reliable overall.
Is that really relevant? A determined user could flash their BIOS with garbage and break anything they want. Does it matter that you can go out of your way to break your own NixOS system? The point is that as a user you’re not supposed to mess with your system manually, but you’re supposed to make changes by changing your configuration.nix. This is not the case for most other distros where you make changes to files under /etc side by side with files maintained by the distro packages.
The point is that I don’t think anyone is checking. IDK how many options in nixpkgs break this. I don’t think we need to stop the user from purposely breaking it but we should make “official” flags not break it and make it easy for users to not break this.
However, I do think there is a place for classical distributions, as there is a place for distributions like Nix, Guix, and even something like GoboLinux.
Pretty much every concern you share in that post is solved by NixOS. Some points
You don’t modify files in /etc, you use the NixOS configuration to do so
Changes are idempotent
Changes are reversible, there’s a history of config revisions called generations, or you can just remove the config and apply to roll forward
System configuration is reproducible and deterministic
Services are systemd units declared through the configuration
Packages reference their direct dependencies allowing for multiple versions of the same lib/package/etc
NixOS isn’t perfect by any means, but it is a solution you seem to be looking for. Guix probably as well, but it’s a smaller and more limited distro.
I took the multiple comments that “NixOS may do this but I’ll have to research” to mean you haven’t tried it. And this entire blog post screams to me that NixOS is a solution for you.
You don’t modify files in /etc, you use the NixOS configuration to do so
In a way, that’s second system syndrome. Suse Linux was bashed for doing something like that with YaST for a long time…
In case of NixOS, I found that new language (not the syntax, the semantics) changed too fast for my liking…
System configuration is reproducible and deterministic
With a few caveats: Software might still get updated (I suppose you can lock that down to specific revisions, but who does?). From a bugfix/security perspective this may be desirable, but it’s not entirely reproducible, and updates can always introduce new bugs or incompatibilities, too.
You’ll get the same version with the same Nixpkgs, every time. buildInputs = [ bash ] will always point at the same bash package for some given Nixpkgs. Package versioning is a different issue.
I think you’re misunderstanding the way nixpkgs works with versions and what the versions actually mean there. Check it out in practice. Unless you update the channels/flakes or whatever you use between runs, nothing will change - there’s no explicit pinning required.
The way I manage this is using several different versions of nixpkgs; a “known good” version for each package I want to lock, so for agda I would have nixpkgsAgda, the regular nixpkgs which is pinned to a stable release, and nixpkgsLatest which is pinned to master.
Most of my packages are on stable nixpkgs. Every now and then when I run nix flake update, it pulls in new versions of software. Things pinned to stable have never broken so far. Things pinned to latest are updated to their latest versions, and things pinned to specific packages never change.
While it does involve pulling in several versions of nixpkgs, I build a lot of software from source anyway so this doesn’t matter to me very much. I do hope that nixpkgs somehow fixes the growing tarball size in the future…
buildInputs = [ bash ] may or may not point at the same version
That’s a snippet of code written in the Nix expression language. bash is not a keyword in that language, it’s just some arbitrary variable name (indeed, so is buildInputs). Just like any other language, if that bash variable is defined/computed to be some pure, constant value, then it will always evaluate to that same thing. If it’s instead defined/computed to be some impure, under-determined value then it may vary between evaluations. Here’s an example of the former:
This evaluates to { buildInputs = [ «derivation /nix/store/6z1cb92fmxq2svrq3i68wxjmd6vvf904-bash-5.2-p15.drv» ]; } and (as far as I’m aware) always will do; even if github.com disappears, it may still live on if that sha256 appears in a cache!
Here’s an example of an impure value, which varies depending on the OS and CPU architecture, on the presence/contents of a NIX_PATH environment variable, the presence/contents of a ~/.config/nixpkgs/overlays.nix file, etc.
PS: I’ve avoided the word “version”, since that concept doesn’t really exist in Nix. It’s more precise to focus on the .drv file’s path, since that includes a hash (e.g. 6z1cb92fmxq2svrq3i68wxjmd6vvf904) which is the root of a Merkle tree that precisely defines that derivation and all of its transitive dependencies; whereas “version” usually refers to an optional, semi-numerical suffix on the file name (e.g. 5.2-p15) which (a) is easy to spoof, and (b) gives no indication of the chosen dependencies, compiler flags, etc. which may have a large effect on the resulting behaviour (which is ultimately what we care about).
In case of NixOS, I found that new language (not the syntax, the semantics) changed too fast for my liking…
Do you happen to have an example of such an semantic change at hand? Curious as nix is often regarded as rather conservative and trying to keep backwards-compatibility as good as possible.
Part of the reason Nix/Guix take their particular approach is specifically because the global mutable state of a traditional FHS style distribution is nigh-impossible to manipulate
in a deterministic way; not just because of the manipulation by the package manager, but because there is no delineation between mutations by the package manager and those by the user. A great example is /etc/passwd, which contains both locally created users as well as “service accounts” the distro maintainers make to separate out some unprivileged services.
it needs a lot of work to enforce 1) determinism/reproducibility, 2) non-hysterisis (when you stretch an object and it doesn’t return /exactly/ back… like hidden side effects in ansible), 3) ease of use, & 4) acceptance testing.
I think we somehow missed to consider two desirable properties of configuration systems that are nevertheless orthogonal: being declarative (which Excel, Terraform, Prolog mostly are) and being Turing-complete (which every Turing-complete programming language is, by definition). If we consider both properties, one can see that there is no hope Prolog is the answer.
Thats how Mark Burgess started (Prolog) and ended up with cfengine. I’d say the modern approach to this would be Erlang :) But I’m betting the only users will be you and I.
What a ridiculous thing to have even happened in the first place, let alone refusing to acknowledge there could possibly be an issue for so long. I glad it’s been fixed but would make me think twice about using serde. I’m sure it’ll be fine, who’s ever heard of a security issue in a codec anyway?
Remember that there are real human being maintaining serde. It is not, in fact, blindingly obvious to all developers that the pre-compiled blobs were bad; on this site there were loud voices on both sides. Can you imagine suddenly getting caught in the crosshairs of angry developers like that? When I imagine it, it feels bad, and I’m liable to get defensive about it.
It may also have been a failed attempt at fixing something you’ve heard people complain about all the time, probably even about your code that slows down peoples builds (*). So yeah it was a bad idea in hindsight, but we don’t need more burned out maintainers from this. And I say this as someone who is openly disappointed by this happening.
(*) I’m not going to discuss how much time it actually saved.
Yeah, basically the biggest gains are offset by process creation being surprisingly slow. I’m working on a follow-up article where I talk about that in detail.
I posted your piece because it was the first one that explained in detail what the hell was going on, specifically how serde works. Looking forward to a followup.
That’s how it started, then they centralized everything with one team that doles out the “managed CI” offering, with their own global library and controls. Any competing infra gets flagged and audited hardcore until you give up by attrition.
This seems to only be checking the performance under –release. Most compilation is done without –release, meaning that most of the proc macro will not be optimized.
As someone who packages software, I think it’s worth noting that packagers expect different things than end users, though they are compatible.
One of my wishes is to avoid blobs from a vendor, since we can’t always recompile those in the build process to work with the architectures we support.
(The other big difference is the DESTDIR env var. End users don’t generally care, but it becomes essential when preparing a package)
I therefore understand those who support their end users, before getting packaged.
The real human being maintaining serde knew about the pushback that would happen and did it on purpose to prove a point in a pre-RFC he submitted. I don’t feel particularly bad about him getting pushback for using half the Rust ecosystem as his guinea pigs. (In fact I would like to see more of it.)
The real human being maintaining serde knew about the pushback that would happen and did it on purpose to prove a point in a pre-RFC he submitted.
What’s the reason to believe in this over any other explanation of the situation? E.g. that pushback was unexpected and that the RFC is the result of the pushback, rather than a cause?
I consider dtolnay a competent open source maintainer who understands the people who run his code well, and I would expect any competent open source maintainer to expect such pushback.
But how that necessary leads to “on purpose to prove a point”?
I don’t think dtolnay expected exactly zero pushback. But, given that some people in this thread argue quite a reasonable point that binaries are actually almost as fine as source, it is plausible that only bounded pushback was expected.
“Someone else is always auditing the code and will save me from anything bad in a macro before it would ever run on my machines.” (At one point serde_derive ran an untrusted binary for over 4 weeks across 12 releases before almost anyone became aware. This was plain-as-day code in the crate root; I am confident that professionally obfuscated malicious code would be undetected for years.)
I don’t see someone competent casually pushing such a controversial change, casually saying that this is now the only supported way to use serde, casually pushing a complete long pre-RFC that uses the controversial change to advance it, and then casually reverting the change in the span of a few days. That takes preparation and foresight.
I actually respect this move. It is exactly the kind of move I would do if I had goodwill to burn and was frustrated with the usual formal process, and it takes boldness and courage to pull it off the way he did it. I also think the pushback is entirely appropriate and the degree of it was quite mild.
Aha, thanks! I think that’s a coherent story to infer from this evidence (and I was wondering if there might be some missing bits I don’t know).
From where I stand, I wouldn’t say that this explanation looks completely implausible, but I do find it unlikely.
For me, the salient bits are:
what it says on the tin. dtolnay didn’t write a lot of responses in the discussion, but what they have written is more or less what I have expected to see from a superb maintainer acting in good faith.
there wasn’t any previous Wasm macro work that was stalled, and that required nefarious plans to get it unstuck.
really, literally everyone wants sandboxed Wasm proc macros. There can’t be any more support for this feature already. What is lacking is not motivation or support but (somewhat surprisingly) a written-down RFC for how to move forward and (expectedly) implementation effort to make it come true.
dtolnay likes doing crazy things! Like how all crates follow 1.0.x versions, or watt, or deref-based specialization in anyhow. So, “because I can” seems like enough motivation here.
“if you don’t like a feature in this crate, don’t use the crate or do the groundwork to make implementing this feature better” feels like a normal mode of operation for widely used OSS projects with sole maintainers. I’ve said as much with respect to MSRV of my once_cell crate.
I agree that there are multiple intepretations possible and that yours also follows from the evidence available. The reason I think it’s reasonable to consider something deeper to be going on is: every single Rust controversy I’ve discussed with key Rust people had a lot more going on than was there on the surface. Case in point: dtolnay was also the one thus far unnamed by anyone speaking for the project person who was involved in ThePHD’s talk being downgraded from a keynote. If I see someone acting surreptitiously in one case I will expect that to repeat.
O_o that’s news to me, thanks. It didn’t occur that dtopnay might have been involved there (IIRC, they aren’t a team lead of any top-level team, so I assume weren’t a member of the notorious leadership chat)
Calling me anonymous is pretty funny, considering I’ve called myself “whitequark” for close to 15 years at this point and shipped several world-class projects under it.
whitequark would be pretty well known to an old Rust team member such as matklad, having been one themself, so no, not anonymous… buut we don’t know this is the same whitequark, so yes, still anonymous.
I mean, I wrote both Rust language servers/IDEs that everyone is using and whitequark wrote the Ruby parser everyone is using (and also smaltcp). I think we know perfectly fine who we are talking with. One us might be secretly a Labrador in a trench coat, but that doesn’t have any bearing on the discussion, and speculation on that topic is hugely distasteful.
In terms of Rust team membership, I actually don’t know which team whitequark was on, but they are definitely on the alumni page right now. I was on the cargo team and TL for the IDE team.
You were talking about “Rust teams” and the only way I’ve seen that term used is to indicate those under the “Rust Project”. Neither person is on a Rust team or an alumni.
Tbh, it more reeks of desperation to make people’s badly configured CI flows faster. I think that a conspiratorial angle hasn’t been earned yet for this and that we should go for the most likely option: it was merely a desperate attempt to make unoptimized builds faster.
I think this is hard to justify when someone comes to you with a security issue, when your response is “fork it, not my problem”, and then closing the issue, completely dismissing the legitimate report. I understand humans are maintaining it, humans maintain all software I use in fact, and I’m not ok with deciding “Oh, a human was involved, I guess we should let security bad practices slide”. I, and I’m sure many others, are not frustrated because they didn’t understand the security implications, but because they were summarily dismissed and rejected, when they had dire implications for all their users. From my understanding, Serde is a) extremely popular in the Rust world, and b) deals in one of the most notoriously difficult kinds of code to secure, so seeing the developers’ reaction to a security issue is very worrying for the community as a whole.
The thing is, its not unambiguous whether this is a security issue. “Shipping precompiled binaries is not significantly more insecure than shipping source code” is an absolutely reasonable stance to have. I even think it is true if we consider only first-order effects and the current state of rust packaging&auditing.
Note also that concerns were not “completely dismissed”. Dismissal looks like “this is not a problem”. What was said was rather “fixing this problem is out of scope for the library, if you want to see it fixed, work on the underlying infrastructure”. Reflecting on my own behavior in this discussion, I might be overly sensitive here, but to me there’s a world of difference between a dismissal, and an acknowledgment with disagreement on priorities.
let alone refusing to acknowledge there could possibly be an issue for so long.
This is perhaps a reasonable take-away from all the internet discussions about the topic, but I don’t think this actually reflects what did happen.
The maintainer was responsive on the issue and they very clearly articulated that:
they are aware that the change makes it harder to build software for some users
they are aware that the change concerns some users from the security point of view
non-the-less, the change is an explicit design decision for the library
the way to solve this typical open-source dilemma is to allocate the work with the party needs the fruits of the work
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I do think it was a mistake to not predict that this change will be this controversial (or to proceed with controversial change without preliminary checks with wider community).
But, given that a mistake had been made, the handling of the situation was exemplary. Everything that needed fixing was fixed, promptly.
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I’m still waiting to hear what “security concern” there was here. Other language-package ecosystems have been shipping precompiled binaries in packages for years now; why is it such an apocalyptically awful thing in Rust and only Rust?
The main thing is loss of auditing ability — with the opaque binaries, you can not just look at the package tarbal from crates.io and read the source. It is debatable how important that is: in practice, as this very story demonstrates, few people look at the tarballs. OTOH, “can you look at tarballs” is an ecosystem-wide property — if we lose it, we won’t be able to put the toothpaste back into the tube.
This is amplified by the fact that this is build time code — people are in general happier with sandbox the final application, then with sandboxing the sprawling build infra.
With respect to other languages — of course! But also note how other languages are memory unsafe for decades…
The main thing is loss of auditing ability — with the opaque binaries, you can not just look at the package tarbal from crates.io and read the source.
It’s not that hard to verify the provenance of a binary. And it appears that for some time after serde switched to shipping the precompiled macros, exactly zero people actually were auditing it (based on how long it took for complaints to be registered about it).
OTOH, “can you look at tarballs” is an ecosystem-wide property — if we lose it, we won’t be able to put the toothpaste back into the tube.
The ecosystem having what boils down to a social preference for source-only does not imply that binary distributions are automatically/inherently a security issue.
With respect to other languages — of course! But also note how other languages are memory unsafe for decades…
My go-to example of a language that often ships precompiled binaries in packages is Python. Which is not exactly what I think of when I think “memory unsafe for decades”.
It’s not that hard to verify the provenance of a binary.
Verifying provenance and auditing source are orthogonal. If you have trusted provenance, you can skip auditing the source. If you audited the source, you don’t care about the provenance.
It’s a question which one is more practically important, but to weight this tradeoff, you need to acknowledge its existence.
This sounds like:
People say that they claim about auditing, and probably some people are, but it’s also clear that majority don’t actually audit source code. So they benefits of audits are vastly overstated, and we need to care about provenance and trusted publishing.
This doesn’t sound like:
There’s absolutely ZERO security benefits here whatsoever
I don’t know where your last two blockquotes came from, but they didn’t come from my comment that you were replying to, and I won’t waste my time arguing with words that have been put in my mouth by force.
That’s how I read your reply: as an absolute refusal to acknowledge that source auditing is a thing, rather than as a nuanced comparison of auditing in theory vs auditing in practice.
It might not have been your intention to communicate that, but that was my take away from what’s actually written.
In the original github thread, someone went to great lengths to try to reproduce the shipped binary, and just couldn’t do it. So it is very reasonable to assume that either they had something in their build that differed from the environment used to build it, or that he binary was malicious, and without much deeper investigation, it’s nearly impossible to tell which is the answer. If it was trivial to reproduce to build with source code you could audit yourself, then there’s far less of a problem.
Rust doesn’t really do reproducible builds, though, so I’m not sure why people expected to be able to byte-for-byte reproduce this.
Also, other language-package ecosystems really have solved this problem – in the Python world, for example, PyPI supports a verifiable chain all the way from your source repo to the uploaded artifact. You don’t need byte-for-byte reproducibility when you have that.
I guesss I should clarify that in GP comment the problem is misalignment between maintainer’s and user’s view of the issue. This is a problem irrespective of ground truth value of security.
Maybe other language package ecosystems are also wrong to be distributing binaries, and have security concerns that are not being addressed because people in those ecosystems are not making as much of a fuss about it.
If there were some easy way to exploit the mere use of precompiled binaries, someone would have by now. The incentives to use such an exploit are just way too high not to.
Anecdotally, I almost always see Python malware packaged as source code. I think that could change at any time to compiled binaries fwiw, just a note.
I don’t think attackers choosing binary payloads would mean anything for anyone really. The fundamental problem isn’t solved by reproducible builds - those only help if someone is auditing the code.
The fundamental problem is that your package manager has near-arbitrary rights on your computer, and dev laptops tend to be very privileged at companies. I can likely go from ‘malicious build script’ to ‘production access’ in a few hours (if I’m being slow and sneaky) - that’s insane. Why does a build script have access to my ssh key files? To my various tokens? To my ~/.aws/ folder? Insane. There’s zero reason for those privileges to be handed out like that.
The real solution here is to minimize impact. I’m all for reproducible builds because I think they’re neat and whatever, sure, people can pretend that auditing is practical if that’s how they want to spend their time. But really the fundamental concept of “running arbitrary code as your user” is just broken, we should fix that ASAP.
The fundamental problem is that your package manager has near-arbitrary rights on your computer
Like I’ve pointed out to a couple people, this is actually a huge advantage for Python’s “binary” (.whl) package format, because its install process consists solely of unpacking the archive and moving files to their destinations. It’s the “source” format that can ship a setup.py running arbitrary code at install time. So telling pip to exclusively install from .whl (with --only-binary :all:) is generally a big security win for Python deployments.
(and I put “binary” in scare quotes because, for people who aren’t familiar with it, a Python .whl package isn’t required to contain compiled binaries; it’s just that the .whl format is the one that allows shipping those, as well as shipping ordinary Python source code files)
Anecdotally, I almost always see Python malware packaged as source code. I think that could change at any time to compiled binaries fwiw, just a note.
Agree. But that’s a different threat, it has nothing to do with altered binaries.
I don’t think attackers choosing binary payloads would mean anything for anyone really. The fundamental problem isn’t solved by reproducible builds - those only help if someone is auditing the code.
Code auditing is worthless if you’re not sure the binary you’re running on your machine has been produced from the source code you’ve audited. This source <=> binary mapping is precisely where source bootstrapping + reproducible builds are helping.
The real solution here is to minimize impact. I’m all for reproducible builds because I think they’re neat and whatever, sure, people can pretend that auditing is practical if that’s how they want to spend their time. But really the fundamental concept of “running arbitrary code as your user” is just broken, we should fix that ASAP.
This is a false dichotomy. I think we agree on the fact we want code audit + binary reproducibility + proper sandboxing.
Agree. But that’s a different threat, it has nothing to do with altered binaries.
Well, we disagree, because I think they’re identical in virtually every way.
Code auditing is worthless if you’re not sure the binary you’re running on your machine has been produced from the source code you’ve audited. This source <=> binary mapping is precisely where source bootstrapping + reproducible builds are helping.
I’m highly skeptical of the value behind code auditing to begin with, so anything that relies on auditing to have value is already something I’m side eyeing hard tbh.
I think we agree on the fact we want code audit + binary reproducibility + proper sandboxing.
I think where we disagree on the weights. I barely care about binary reproducibility, I frankly don’t think code auditing is practical, and I think sandboxing is by far the most important, cost effective measure to improve security and directly address the issues.
I am familiar with the concept of reproducible builds. Also, as far as I’m aware, Rust’s current tooling is incapable of producing reproducible binaries.
And in theory there are many attack vectors that might be present in any form of software distribution, whether source or binary.
What I’m looking for here is someone who will step up and identify a specific security vulnerability that they believe actually existed in serde when it was shipping precompiled macros, but that did not exist when it was shipping those same macros in source form. “Someone could compromise the maintainer or the project infrastructure”, for example, doesn’t qualify there, because both source and binary distributions can be affected by such a compromise.
Aren’t there links in the original github issue to exactly this being done in the NPM and some other ecosystem? Yes this is a security problem, and yes it has been exploited in the real world.
What I’m looking for here is someone who will step up and identify a specific security vulnerability that they believe actually existed in serde when it was shipping precompiled macros, but that did not exist when it was shipping those same macros in source form. “Someone could compromise the maintainer or the project infrastructure”, for example, doesn’t qualify there, because both source and binary distributions can be affected by such a compromise.
If you have proof of an actual concrete vulnerability in serde of that nature, I invite you to show it.
The existence of an actual exploit is not necessary to be able to tell that something is a serious security concern. It’s like laying an AR-15 in the middle of the street and claiming there’s nothing wrong with it because no one has picked it up and shot someone with it. This is the opposite of a risk assessment, this is intentionally choosing to ignore clear risks.
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening. To me this is up with with Sony’s rootkit back in the day, completely unexpected, unauthorised behaviour that no reasonable person would expect, nor would they look out for it because it is just such an unreasonable thing to do to your users.
I think the point is that if precompiled macros are an AR-15 laying in the street, then source macros are an AR-15 with a clip next to it. It doesn’t make sense to raise the alarm about one but not the other.
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening.
I think this is extreme. No additional accessing of any kind was done. Binaries don’t have additional abilities that build.rs does not have. It’s not at all comparable to installing a rootkit. The precompiled macros did the same thing that the source macros did.
The existence of an actual exploit is not necessary to be able to tell that something is a serious security concern. It’s like laying an AR-15 in the middle of the street and claiming there’s nothing wrong with it because no one has picked it up and shot someone with it. This is the opposite of a risk assessment, this is intentionally choosing to ignore clear risks.
Once again, other language package ecosystems routinely ship precompiled binaries. Why have those languages not suffered the extreme consequences you seem to believe inevitably follow from shipping binaries?
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening.
Even the most extreme prosecutors in the US never dreamed of taking laws like CFAA this far.
To me this is up with with Sony’s rootkit back in the day, completely unexpected, unauthorised behaviour that no reasonable person would expect, nor would they look out for it because it is just such an unreasonable thing to do to your users.
I think you should take a step back and consider what you’re actually advocating for here. For one thing, you’ve just invalidated the “without any warranty” part of every open-source software license, because you’re declaring that you expect and intend to legally enforce a rule on the author that the software will function in certain ways and not in others. And you’re also opening the door to even more, because it’s not that big a logical or legal leap from liability for a technical choice you dislike to liability for, say, an accidental bug.
The author of serde didn’t take over your computer, or try to. All that happened was serde started shipping a precompiled form of something you were going to compile anyway, much as other language package managers already do and have done for years. You seem to strongly dislike that, but dislike does not make something a security vulnerability and certainly does not make it a literal crime.
I think that what actually is happening in other language ecosystems is that while there are precompiled binaries sihpped along some installation methods, for other installation methods those are happening by source.
So you still have binary distribution for people who want that, and you have the source distribution for others.
I have not confirmed this but I believe that this might be the case for Python packages hosted on debian repos, for example. Packages on PyPI tend to have source distributions along with compiled ones, and the debian repos go and build packages themselves based off of their stuff rather than relying on the package developers’ compiled output.
When I release a Python library, I provide the source and a binary. A linux package repo maintainer could build the source code rather than using my built binary. If they do that, then the thing they “need to trust” is the source code, and less trust is needed on myself (on top of extra benefits like source code access allowing them to fix things for their distribution mechanisms)
So you still have binary distribution for people who want that, and you have the source distribution for others.
I don’t know of anyone who actually wants the sdists from PyPI. Repackagers don’t go to PyPI, they go to the actual source repository. And a variety of people, including both me and a Python core developer, strongly recommend always invoking pip with the --only-binary :all: flag to force use of .whl packages, which have several benefits:
When combined with --require-hashes and --no-deps, you get as close to perfectly byte-for-byte reproducible installs as is possible with the standard Python packaging toolchain.
You will never accidentally compile, or try to compile, something at runtime.
You will never run any type of install-time scripting, since a .whl has no scripting hooks (as opposed to an sdist, which can run arbitrary code at install time via its setup.py).
I mean there are plenty of packages with actual native dependencies who don’t ship every permutation of platform/Python version wheel needed, and there the source distribution is available. Though I think that happens less and less since the number of big packages with native dependencies is relatively limited.
But the underlying point is that with an option of compiling everything “from source” available as an official thing from the project, downstream distributors do not have to do things like, say, confirm that the project’s vendored compiled binary is in fact compiled from the source being pointed at.
Install-time scripting is less of an issue in this thought process (after all, import-time scripting is a thing that can totally happen!). It should feel a bit obvious that a bunch of source files is easier to look through to figure out issues rather than “oh this part is provided by this pre-built binary”, at least it does to me.
I’m not arguing against binary distributions, just think that if you have only the binary distribution suddenly it’s a lot harder to answer a lot of questions.
But the underlying point is that with an option of compiling everything “from source” available as an official thing from the project, downstream distributors do not have to do things like, say, confirm that the project’s vendored compiled binary is in fact compiled from the source being pointed at.
As far as I’m aware, it was possible to build serde “from source” as a repackager. It did not produce a binary byte-for-byte identical to the one being shipped first-party, but as I understand it producing a byte-for-byte identical binary is not something Rust’s current tooling would have supported anyway. In other words, the only sense in which “binary only” was true was for installing from crates.io.
So any arguments predicated on “you have only the binary distribution” don’t hold up.
Hmm, I felt like I read repackagers specifically say that the binary was a problem (I think it was more the fact that standard tooling didn’t allow for both worlds to exist). But this is all a bit moot anyways
I don’t know of anyone who actually wants the sdists from PyPI.
It’s a useful fallback when there are no precompiled binaries available for your specific OS/Arch/Python version combination. For example when pip installing from a ARM Mac there are still cases where precompiled binaries are not available, there were a lot more closer to the M1 release.
When I say I don’t know of anyone who wants the sdist, read as “I don’t know anyone who, if a wheel were available for their target platform, would then proceed to explicitly choose an sdist over that wheel”.
Also, not for nothing, most of the discussion has just been assuming that “binary blob = inherent automatic security vulnerability” without really describing just what the alleged vulnerability is. When one person asserts existence of a thing (such as a security vulnerability) and another person doubts that existence, the burden of proof is on the person asserting existence, but it’s also perfectly valid for the doubter to point to prominent examples of use of binary blobs which have not been exploited despite widespread deployment and use, as evidence in favor of “not an inherent automatic security vulnerability”
Yeah, this dynamic has been infuriating. In what threat model is downloading source code from the internet and executing it different from downloading compiled code from the internet and executing it? The threat is the “from the internet” part, which you can address by:
Hash-pinning the artifacts, or
Copying them to a local repository (and locking down internet access).
Anyone with concerns about this serde change should already be doing one or both of these things, which also happen to make builds faster and more reliable (convenient!).
Yeah, hashed/pinned dependency trees have been around forever in other languages, along with tooling to automate their creation and maintenance. It doesn’t matter at that point whether the artifact is a precompiled binary, because you know it’s the artifact you expected to get (and have hopefully pre-vetted).
Downloading source code from the internet gives you the possibility to audit it, downloading a binary makes this nearly impossible without whipping out a disassembler and hoping that if it is malicious, they haven’t done anything to obfuscate that in the compiled binary. There is a “these languages are turing complete, therefore they are equivalent” argument to be made, but I’d rather read Rust than assembly to understand behaviour.
The point is that if there were some easy way to exploit the mere use of precompiled binaries, the wide use of precompiled binaries in other languages would have been widely exploited already. Therefore it is much less likely that the mere presence of a precompiled binary in a package is inherently a security vulnerability.
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I’m confused about this point. Is anyone going to fix crates.io so this can’t happen again?
Assuming that this is a security problem (which I’m not interested in arguing about), it seems like the vulnerability is in the packaging infrastructure, and serde just happened to exploit that vulnerability for a benign purpose. It doesn’t go away just because serde decides to stop exploiting it.
I don’t think it’s an easy problem to fix: ultimately, package registry is just a storage for files, and you can’t control what users put there.
There’s an issue open about sanitizing permission bits of the downloaded files (which feels like a good thing to do irrespective of security), but that’s going to be a minor speed bump at most, as you can always just copy the file over with the executable bit.
A proper fix here would be fully sandboxed builds, but:
POSIX doesn’t have “make a sandbox” API, so implementing isolation in a nice way is hard.
There’s a bunch of implementation work needed to allow use-cases that currently escape the sandbox (wasm proc macros and metabuild).
I’m sure it’ll be fine, who’s ever heard of a security issue in a codec anyway?
Who’s ever heard of a security issue caused by a precompiled binary shipping in a dependency? Like, maybe it’s happened a few times? I can think of one incident where a binary was doing analytics, not outright malware, but that’s it.
I’m confused at the idea that if we narrow the scope to “a precompiled binary dependency” we somehow invalidate the risk. Since apparently “curl $FOO > sh” is a perfectly cromulent way to install things these days among some communities, in my world (30+ year infosec wonk) we really don’t get to split hairs over ‘binary v. source’ or even ‘target v. dependency’.
I’m not sure I get your point. You brought up codec vulns, which are irrelevant to the binary vs source discussion. I brought that back to the actual threat, which is an attack that requires a precompiled binary vs source code. I’ve only seen (in my admittedly only 10 Years of infosec work) such an attack one time, and it was hardly an attack and instead just shady monetization.
This is the first comment I’ve made in this thread, so I didn’t bring up codecs. Sorry if that impacts your downplaying supply chain attacks, something I actually was commenting on.
Ah, then forget what I said about “you” saying that. I didn’t check who had commented initially.
As for downplaying supply chain attacks, not at all. I consider them to be a massive problem and I’ve actively advocated for sandboxed build processes, having even spoken with rustc devs about the topic.
What I’m downplaying is the made up issue that a compiled binary is significantly different from source code for the threat of “malicious dependency”.
So not only do you not pay attention enough to see who said what, you knee-jerk responded without paying attention to what I did say. Maybe in another 10 years…
Because I can curl $FOO > foo.sh; vi foo.sh then can choose to chmod +x foo.sh; ./foo.sh. I can’t do that with an arbitrary binary from the internet without whipping out Ghidra and hoping my RE skills are good enough to spot malicious code. I might also miss it in some downloaded Rust or shell code, but the chances are significantly lower than in the binary. Particularly when the attempt from people in the original issue thread to reproduce the binary failed, so no one knows what’s in it.
No one, other than these widely publicisedinstances in NPM, as well as PyPi and Ruby, as pointed out in the original github issue. I guess each language community needs to rediscover basic security issues on their own, long live NIH.
I hadn’t dived into them, they were brought up in the original thread, and shipping binaries in those languages (other than python with wheels) is not really common (but would be equally problematic). But point taken, shouldn’t trust sources without verifying them (how meta).
But the question here is “Does a binary make a difference vs source code?” and if you’re saying “well history shows us that attackers like binaries more” and then history does not show that, you can see my issue right?
But what’s more, even if attackers did use binaries more, would we care? Maybe, but it depends on why. If it’s because binaries are so radically unauditable, and source code is so vigilitantly audited, ok sure. But I’m realllly doubtful that that would be the reason.
I agree with a lot of this, especially the root-cause analysis. Working with stacked PRs in GitHub is painful and the problem here is fundamental to git: commits do not have an identity, trees have an identity and commits are named as the tree to which they were applied. This means that you suffer a lot of rebasing pain when you make changes to one at the top, because now the versions of those commits in other branches are wrong.
I am not convinced that the solution to this is ‘buy another third-party product’. The right solution is probably to switch to Pijul, but that’s very high friction.
GitHub also makes this harder than it needs to be by modifying commit messages on merge (even with rebase) to include the PR number. Providing a fast-forward option for PRs (Azure DevOps has this - it’s the only feature that they have that I miss on GitHub) would at least avoid having to rebase after a PR is merged if it isn’t changed. annoyingly, if you just do a fast-forward on the main branch, GitHub correctly tracks the PR metadata, so this is purely a front-end issue.
I would really like a low-friction way of saying ‘this range of commits in a branch is a PR’. I think something like that could be made quite easily if you put some kind of marker on the final commit in a sequence. I’d then be able to do rebase locally and edit and gradually pull commits into the main branch.
Switching to Pijul, or simply switching from git to anything better, is just not an option for 99.9999% of people.
I’d expect more like 95-98% of people. Still a huge barrier in the short term. Then again, that number was the same for Git at some point in the past.
Very much this.
You might be surprised as well…
Microsoft’s continual embracement of git is a harbinger of it’s decline, and developers are historically very, very, fad driven.
It might not be an option for you personally, but is it an option for your boss or your bosses boss? Maybe….
I’m really surprised by how developers are so addicted to trying new shiny tools, but Git has been the de-facto standard in pretty much every company for the last 10 years or even more; I can’t think of another example of such a pervasive tool. (Not saying I think it’s necessarily bad).
My biggest gripe with Pijul is that the only documented way to collaborate with others on Pijul repos is Nest and there is absolute nothing about self-hosting it. Git is super simple with that regard (just create bare Git repo on machine with installed Git that is accessible via SSH and you are done), while Pijul do not describe such functionality at all. And this blog post make it even less appealing that you need to use proprietary service for simply hosting your repo.
I’ve seen people do custom bots (I think possible with Github Actions) to work around that - someone comments a keyword on the PR, and the bot does the fast-forward and updates the target branch. As you say, it then even looks right in the PR view as having been merged.
Gerrit works around that by having a Change-Id in the commit messages (automatically added when creating the commit), which allows it to track commits across rebases and other modifications. I guess you could make some tools that do similar things to identify commits/branches with outdated ancestors and make sure you rebase them properly.
I wish git would standardise something like this. It’s also important if you’re back-porting security fixes across branches. I really want to have a way of knowing whether a commit that introduced a bug was back-ported and then of guaranteeing that the security fix is in every branch that the feature was merged into.
I’ve been using Pijul on a project recently after doing two in darcs & the CLI UX seems unfinished in comparison. It sucks, since the data model & performance are great.
Having the keystore for identity is nice, as is channels for hopping around on different features, but
recordhooks so you can’t automate stuff on the upstream server on push (i.e. kicking off a CI)$EDITORbut weird-TOML is an ugly format to work with for commitsdiffisn’t compatible with existingpatch-based tooling with no alternative/compat option & opens straight to the pager instead of letting you decide (darcs diff --diff-command "kitty +kitten diff %1 %2"can get you a wonderful highlighted diff session)pijul amendinstead ofpijul record --amenddarcs rebasehave the ability to pop off patches, amend a commit message (or code) then replay the next patches without issue; you can directlypijul record --amend … --message …up until it has a dependency, then a typo I guess is set in stone & it’s a pain unlikegit rebase -i …which lets you fix commit messages & apply fixups (not sure how commit message make the code not apply)pijul unrecord --reset …to pop off commits then Pijul hangs on next push to push the amended changes… can’t easily obliterate from local machine to upstream & amend commits just get shoved atop rather than see this as something akin togit push --force-*.I have a tendency to commit kinda early & often as well as pushing to my 2nd machine as a backup, & use amending in Git & darcs to fixup my patchsets before sending to others or the canonical repo. That said, both of these tools are great, underrated, & folks should feel encouraged to try something outside the status quo as at least it will give you a new perspective on Git (i.e. learning you a Haskell doesn’t mean you’re going to switch).
Is this a repo setting? I have never seen a commit message modified by GitHub, as otherwise the commit hash would change and when I pull main locally, my branches wouldn’t show merged into main (due to commits differing)
When GH does actual merge with a merge commit, then original commits are unchanged and the merge commit has the PR number in message (not sure what it does when it can do a fast-forward instead of merge, though).
When you click “squash and merge”, the squashed commit’s message is obviously generated by GH (PR number + title and list of summaries of PR’s commits by default). Not sure about “rebase and merge” (where rebase itself would also alter commit IDs).
And yes, it’s frustrating when I can’t use native git mechanisms to delete my local branches that are merged. The “squash and merge” also kills ancestry, so I can’t use git subtrees in repos that use that. And it breaks some flows that rely on branch ancestry. The upside is a neat linear main branch and fewer weird merge conflicts across the team. I would still love a solution that makes main branch nice without breaking git itself.
Ah, that makes sense then - but I never ever use squash merge, so have not seen this behaviour.
While interesting, Pijul is more of the same. In the same way something like Elixir offers massive benefits over the normal model of taping discrete parts (DB, reddis, services) together and throwing them in containers, there are other possibilities like an AST based system (and others we’ve not thought of yet).
I’m aware of some semantic dif tools like graphtage and gumtree exploring this space, but e.g. combining it with an AST based programming language (Racket seems to be going in this direction) could be phenomenal. Indeed, we can go further instead of IDEs like VSC or even Plan9’s Acme, we could see all our code as an AST we freely manipulate.
There are also budding efforts in this direction:
Pijul can accept arbitrary binary diffs, so you can generate a nice accurate diff based on the AST if you want to.
Structural diffing would be awesome. No longer would I need to sit in space-indented code or see uncuddled curly braces, or you could align those record key/values on
=but don’t since it can lead to diff issues in the future just to touch alignment. The idea of having my preferred formatting for my needs on my end would make me happier & prevent less fighting.FWIW email patches do have an identity, and the default email workflow with git is essentially stacked.
But people for some reason are allergic to emails ¯\_(ツ)_/¯
This news is too convenient to generate outrage at the evil manufacturer corporation, so it makes me suspect there’s more at play here. For example, if there are third party shops that e.g. refuse to change braking pads and simply fool the firmware into accepting used ones, and the train crashes, the manufacturer might be held responsible - if not by the courts, then at least in a PR sort of way. If there’s no certification process and the repair shops are just random companies that don’t know what they’re doing, it might be actually sensible to prevent them from doing maintenance. Of course, they should have built in a different way of doing this, for example in the contracts, or with a clearer mesage in the firmware.
Why should that be solved by sneaky software and not just a contract? Possibly amended by not so sneaky software to remind you of said contract.
There are very good reasons to shun third party workshops, but what if they’re the only option at one point? The company goes bust and Poland is stuck with randomly locking trains forever?
For a long time servicing the trains was done solely by the manufacturers, as they argued that the service manual was their intellectual property. Since, the courts have decided that no, they have to provide this, and other companies could enter the train servicing market. Newag’s hand forced, they complied, with bitter taste in their mouth.
The service manual handed over neglected to mention that some of the trains will lock up if left in place for 10 days, and that they’ll have to be unlocked with an undocumented sequence of button presses in the cabin – the assumption behind this being, trains don’t just stand around unused if they’re not being serviced. After this assumption turned out to be not entirely true, a newer version of the firmware included checks on the GPS position of the trains, with hardcoded coordinates of some competitors’ workshops.
Not sure if you’ve read the more detailed article, or only the Mastodon posts, but I recommend putting it through Google Translate if you can’t read Polish.
The Newag Impuls trains breaking randomly have been going on for over a year. Mysteriously, only third-party workshops had issues with this. Newag always said “see, they’re just incompetent”. An unspoken suspicion has long formed in the industry that some underhanded tactics are at play, but until now, nobody had any idea how to go about proving this.
Newag replied in a press release that the third parties could be responsible for inserting this code into the firmware. I can’t decide who can be trusted here. Both scenarios seem plausible.
What motivation would SPS (the third-party repairer) have to brick the trains they’re being paid to maintain?
To ruin the reputation of the manufacturer so that they’re the only ones alowed to repair it.
That is pretty much pointless TBH, especially that they were running against the schedule and they were close to being late on fixes. With that in mind that looks especially dumb. Also, the bogus code was found in other trains, that weren’t serviced by SPS.
Also I do not get what SPS is supposed to gain there. They aren’t the only train servicing shop in Poland, so it is not that anyone who is using NEWAG trains will go and service their trains at SPS. SPS has next to nothing to gain there, it is only NEWAG that can lose. And SPS isn’t even direct competitor of NEWAG, as NEWAG is train manufacturing company while SPS is just servicing company. It is like saying that iFixIt fight for Right to Repair is a plot to ruin reputation of Apple/Samsung/other producers so only they can service smartphones.
3rd party repair has been one of the conditions of the train tender. Newag shouldn’t have offered their trains if they were not OK with the terms. Period.
I read the entire thing and I’m not sure what it is doing or what we should be doing.
The only command that’s in there is:
nix flake init -t github:nickel-lang/organistbut that’s I guess how you setup an organist project, not how you use it? Then you use it regularly withnix develop?Update: I think if you read the README here, it becomes clear: https://github.com/nickel-lang/organist Still not really clear whether or how it’ll fill many of my development needs.
NclI browsed Nickel documentation previously but still, constructs like this leave me rather mystified:
What is happening here with the
%s?I’d say in general that Nickel may be a great idea and it looks less offputting than Nixlang but it’s still very far off from something a large audience of people can use.
Recently I saw Garn which is a “Typescript eats the entire world” approach to this problem. I’m also very sceptical of it as an abstraction layer, but the choice of language does look like it could be a winner. It reminds me a bit of CDK/Typescript which is a weird imperative/declarative hybrid alternative to the standard terrible Devops ways of defining infrastructure.
My impressions as well. I’m not sure if this competes with devenv, devbox, and others, or is some completely different thing. If former, what does it bring over other tools.
Similar thoughts. Even as a Nix user I’m confused about some of the syntax I’m unfamiliar with, and generally about what Organist is trying to be.
If it’s a layer above Nix flakes dev shell configuration like some of the other projects, it seems like a hard sell as if you can do Nickel… you probably can do Nix already, and introducing an extra layer is neither here or there. If you go json/yaml it will be dumbed down but easier to consume for non-nixers, and if you go Nix - you are 100% seamless with Nix.
BTW. I’m causally lurking into Nickel and I’m still confused w.r.t level of interoperability with Nix. Nickel presents itself like a Nix-like “configuration language”, which means … it can’t really do some of the things Nix do? Or can it? Can it transpile to Nix or something?
My take is that yes it’s competing with those tools, but in a (nearly) native nix way, nearly as it depends on Nickel tooling, but the generated flake pulls that in automatically so there’s nothing else to install.
At work I am using Devenv mostly for process support (which ironically I don’t need any more) and it fits the bill, but IS two things to install before team members can start developing (plus direnv). This would only be one thing to install.
At home I run NixOS and just use a flake for my dependencies but that doesn’t launch any services so I am kind of keen on using organist if I ever need that.
It’s very cool that this works so you can have your flake contents defined in some other language entirely and don’t have to think about it (if it works).
You can use Devenv as a
mkShellreplacement when working with Nix Flakes, so you do not need to install anything manually.One of the article’s links is to Organist’s README – How does this differ from {insert your favorite tool} ?. In summary, Organist’s closest competitor is Devenv, and its main advantage is the consistency and power of the Nickel language.
I like idea of wearings (I need to add one to my website), but for me it looks painful, that it requires so much manual steps, often involving GitHub repository or other. I believe that most of these could be automated to some degree.
The whole point is to remove the automation, though. Webrings should be manually vetted collections of content that one or more humans have decided is cool.
While I agree with the title, the points there are mostly pretty weak IMHO.
Mostly agree. Often that is not needed.
Partially agree. Use semantic HTML, but for gods’ sake, do not use
on*=attributes with inline JS.Partially agree, but with stuff like Hyperscript that do not differs much from
on*=attributes and that would be quite painful in larger scale IMHO. Instead try to make interactivity declarative with custom attributes (more like HTMX/Alpine.js is doing). But in general try to avoid relying on JS at all.Partially agree. Modern pipelines are often way too complex. However I think that some build step, especially with simple (low to zero config) tools, is useful. I mostly use PostCSS to squash CSS into single, minimised, file, as well as to provide some polyfils for older browsers, which allows me to use fancy new features now. And these features greatly simplify my CSS and working with it (cascading layers, CSS nesting, some autoprefixing for some features).
If your server side rendering engine allows you to remove boilerplate and repetitions, then it is often IMHO way better. Especially that comparison there isn’t apples to apples, as it removed i18n features from
f.label :fieldhelper.I do not disagree with the premise, but the article does hardly any job there except saying “they are old, weak and slow while we are new, secure and fast” with a single sentence from something that most non crypto related people probably not even heard of and single bar graph. IMHO not the best article.
It is composed to tweets, so…
A screen-reader user could chime in: would this require interaction by the user? (e.g. using a command to read the
spoilercontent). Is that user friendly?On the other hand, for screen reader there is currently no way to mark something as a spoiler right now. So if person with sight disabilities tries to read community channels about their favourite book series or something they are stepping into minefield of spoilers that their readers cannot prevent.
Isn’t this already solved how we solve it in real life discussions? By adding a little warning that says “There are spoilers ahead!”?
Sure, but what if only part is spoiler? Like it often happens on Reddit or Discord.
I think this is where the idea has the most potential value— being able to say “this is where spoilers end” is just as important as saying “this is where spoilers start.” Being able to know that without reading, hearing, or seeing the parts in between is ideal.
I endorse this general approach: not always, but often, cleanly generated files in the source root are easier to work with than unreadable garbage tucked in a corner of a build dir.
Additionally, for libraries, this approach can often massively simplify life for the consumers, as they don’t need to depend on code used to generate stuff.
However, I would recommend a different implementation: instead of writing GitHub action, write this as a test. Specifically:
Benefits of this approach:
go testflow instead of adding a project-specific concernI second this. My default stance is to git ignore generated files, but in some cases it’s the pragmatic approach to commit them.
Some rule of thumb I use:
You can use
.gitattributesto disable local diffing for a file, and to tell git forges that a file is generated:src/gen/foo.c -diff linguist-generatedWith
-diff,git diffwill just remark something like “Binary file has changed”, andlinguist-generatedtells GitHub and Gitea to exclude a file from web diffs and from stats.Oh that’s a very useful tip, thank you.
Didn’t know about
linguist-generated, nice.Also, I often use
binaryinstead of-diffto also have benefits during merge.Couldn’t you easily end up with a “wedged” build? Say I changed the input to the code generator, for example, I’ve added a data member to a protobuf spec, and adjusted my code to reference the new member. Now my project doesn’t build because the generated code doesn’t yet have this new member. But I also cannot run the test to re-generated it because the project doesn’t build. I need to remember to change the spec, run tests, only then change the code. This screams “hack”!
Feel like the proper way to handle this is to use a proper build system. We do something similar to what you have described (keep pre-generated code up-to-date, fail the build if anything changes) for a tool that uses its own generated code: https://git.codesynthesis.com/cgit/cli/cli/tree/cli/cli/buildfile#n98
You indeed could get a broken build, but it’s roughly equivalent in frequency and annoyance to, e.g., accidentally committing a file with merge conflict markers.
If you add this to the default build task, which is run unconditionally, this has the problem that your consumers now have to run code generation as well just to build your stuff. It is a fairly frequent annoyance for Rust that a build.rs of my dependency has a boat load of dependencies, all of which could have been avoided if the author of the library would have run build.rs logic on their machine instead (which is not always possible, but occasionally is)
If you add this as a dedicated
make generatetask, then the you’ll need to adjust you CI as well, which is a smell pointing out that local workflows need subtle adjustment as well.That’s why I suggest tacking onto
make test: this is an already existing build system entry point for checking self-consistency of a particular commit.Not if you only enable code generation for the development builds. Our setup is:
Consumer builds use pre-generated code.
Development builds update generated code if inputs change, compare the result with pre-generated, if there are difference, copy the generated code from output to source and fail the build.
Oh, that’s curious! I don’t think I’ve ever seen distinction between “development” and “consumer” builds before!
Is this a first class concept in build2, or is it just some custom config for this specific build file? Are there any docs about other uses-cases for this separation?
Well there are Rust’s dev-dependencies. Though, IMHO, the semantics (or the name) is misguided – tests are not only to be run during development.
It is a first-class concept, though a pretty thin one: we simply reserved the
config.<project>.developvariable to mean “development build”. Inbuild2we split the package manager into two tools: for consumption and for development. The latter by default configures projects withconfig.<project>.develop=true.There is the announcement of this feature: https://build2.org/release/0.14.0.xhtml#develop
So far we have used it for pre-generated source code as well as to support additional documentation outputs that require additional/non-portable tools. For example, we can produce XHTML portably but to convert that to PDF requires
html2psandps2pdfso we only enable this step (and require these tools) in the development builds.EDIT: Forgot to mention, you can also use
config.<project>.develop(like any other configuration variable) for conditional dependencies:Autotools has had this distinction, I guess basically forever, the result of a
make distcan be thought of as a “consumer” build. You don’t need automake/autoconf to do the./configure && make && make installdance, but it is likely you’ll need it during some point if you’re doing development on the project.That’s how Oils works too
I think there are some downsides in that I want people to open up the tarball and be able to contribute without necessarily getting git involved … But I think this is less common than it used to be. But not unheard of for one-off hacks.
Thankfully all our generated code is readable and debuggable with normal tools
I cannot overstate how good Mercurial (with extensions hggit and hg-evolve) is as a Git interface, especially for history editing such as rebasing.
Take the “split commit while rebasing” task mentioned by my sibling commenter @anordal. The Git command is apparently called ‘revise –cut REV – FILES’; in Mercurial you achieve the same with
hg split --rev REV.Demonstration: I’d like to clone the history below, rebase branch
breadontomain, and meanwhile split commit 5. (Note: the short numbers are real, Mercurial has local sequential rev numbers as well as globally-unique hashes.)Here come the changes. We’re going to clone the git repo, make our changes with Mercurial, and push them back to the upstream Git repo.
After the
hg split, before rebasing the rest of the changes, the repo looks like this. So you can see which commits are obsolete (‘x’ nodes) or still need to be rebased (’*’ nodes).A lot of that (with exception to sequential commit IDs) is provided by git-branchless by @arxanas. It is really nice set of tools.
Ooh, that looks very nice. I especially like
git swfor interactively selecting what you want to switch to. Thanks for the rec!Also check out Jujutsu VCS for a Git-compatible, Mercurial-inspired VCS solution.
jj rebase -s ab -d cd && jj new abto rebaseabin memory and then switch to changeab.Weird idea (I do not how it will work in practice though) - add command to
git-branchlessthat would allow to jump to the commits using their tree hash. That has will not be unique, so we probably should exclude the merge commits, but withingit-branchless-smartlogentries it should be small enough to not have a lot of conflicts. And that would provide semi-consistent hashing for “recent commits” ingit-branchless.Want to try Mercurial as a Git client yourself? These installation instructions will get you set up and running!
Tried and tested by me; if an error has crept in let me know and I’ll help you.
Now try it out:
And lastly, here’s some more settings for your ~/.hgrc.
hg lprints a pretty graphical log with colorshg bmis an alias forhg bookmark– if you’re working with Git branch this is how you’ll create and move them.hg newcommitcreates dummy commits. Useful for quickly creating some history to experiment on.Did anybody use this on iOS? The description says:
I’m in charge of the publication of Goguma on iOS. We have an issue tracking all the features not available yet on iOS: https://todo.sr.ht/~emersion/goguma/138
We are still working on notification support, as iOS has strict requirements for background tasks. Besides that, it works decently well and has good accessibility.
Palaver has protocol to register how to send push messages for iOS. Maybe implementing something like that would be helpful.
https://github.com/cocodelabs/palaver-irc-capability
Had a look at it: their extension is unfortunately just sending requests to a closed source API that they host, which is supposedly doing the actual APNS requests :(
It seems that there is source for API, but nothing prevents anyone from reimplementing that API independently.
I fail to see how that differs from “regular Lima”.
$WORK: Refactorings and cleanups of Logflare. Fixing few bugs after migration from Cowboy to Bandit.$NON_WORK: implementation of 9p2000 for Erlang to implement something likeprocfsfor Erlang. There exists implementation that use FUSE for that, but that requires compiling C code and that can be problematic. 9p2000 allows me to implement it in pure Erlang.Is there any source code available for that?
https://plan.cat/
I probably posted it previously, but on macOS I have prepared launch agent, that will automatically sent your local
~/.planto https://plan.cat on edit:That requires that you put your auth in the
~/.netrc.This looks amazing. Will definitely give it a try.
Similar idea, but without GUI - https://hurl.dev/
What the actual…?
Same, bloody CloudFlare! I can’t believe people are still unironically using this Five-Eye-honeypot.
I co not think that this is CloudFlare related issue
Based on the comments regarding who has access, I’d say using Cloudflare would help in this scenario. It seems that this website has decided to block entire parts of the globe to mitigate bad behavior, something using Cloudflare would enable without geoblocking.
I was blocked too. This isn’t how the internet is supposed to work.
There are a lot of reasons to not use Rust, but this post does not list them out. Speaking as someone who has used Rust professionally for four years, this is my take on these points:
The rationale being that it’s stackoverflow’s most loved language and 14th most used… Its a growing language, and seeing more and more industry adoption. Currently, due to posts like this, it is a hard sell to management for projects which aren’t low-level, despite developers loving using it (and I’ve been told multiple times that people feel far more comfortable with their Rust code, despite not being experts in the language). Rust might be overhyped, but the data provided to back up this claim is just not correct.
I know this person has written a book on Rust, but… I have to question what the hell they’re talking about here. The steady release cycle of the Rust compiler has never once broken my builds, not even slightly. In fact, Rust has an entire Epoch system which allows the compiler to make backwards-incompatible changes while still being able to compile old code.
I mean, seriously, I genuinely don’t know how the author came to this conclusion based on the release cycle. Even recent releases don’t have many added features. Every project I have ever come across developed in Rust I’ve been able to build with
cargo buildand I’ve never once thought of the version it was developed with or what I had in my toolchain. Python 3 has literally had a series of breaking changes fairly recently and its being compared as a language doing it “better” because it has fewer releases.sigh. Because async traits aren’t stabilized? Even though there is a perfectly workable alternative,
async-traitscrate, which simply makes some performance trade-offs? I’m excited for async traits being stabilized, and its been a bummer that we haven’t had them for so long, but that doesn’t make it Beta.This is just an opinion the author has that I strongly disagree with (and I imagine most Rust developers would). The standard library is small, this was/is a design decision with a number of significant benefits. And they do bring 3rd party libraries into the standard library once they are shown to be stable/widely used.
To put this more accurately, Rust forces you to write correct async code, and it turns out correct async code is hard. This is an important distinction, because a language like Go makes it just as easy to write incorrect async code as it does correct async code. Having been bitten by enough data races and other undefined behavior in my lifetime, I love Rust’s stance on async code, which is to make it hard to do incorrectly with minimal runtime overhead.
Frankly, the Rust compiler is some incredible engineering that is also pushing the bounds of what a programming language can do. I mean, seriously, as frustrating as async Rust can be to work with, it is an impressive feat of engineering which is only improving steadily. Async Rust is hard, but that is because async code is hard.
[edit] Discussed below, but technically Rust just prevents data-races in async code, and does not force you to write code which is free from race-conditions or deadlocks (both of which are correctness issues). Additionally, the “async code” I’m talking about above is multi-threaded asynchronous code with memory sharing.
Frankly, the points being made in this post are so shoddy I’m confused why this is so high on lobsters. The anti-Rust force is nearly as strong as the pro-Rust force, and neither really contributes to the dialog we have on programming languages, their feature-set and the future of what programming looks like.
Use Rust, don’t use Rust, like Rust, don’t like Rust, this post is not worth reading.
I have not written any production Rust code (yet) but the “async is hard” resonates with me. I’ve wrestled with it in C, C++, Java, and Go and it’s easy to make a mistake that you don’t discover until it’s really under load.
I think you really hit the nail on the head with this point. The particularly damning thing about data-race bugs is that they are probabilistic. So you can have latent code with a 0.0001% chance of having a data-race, which can go undetected until you reach loads which make it guaranteed to occur… And at that point you just have to hope you can (a) track it down (good luck figuring out how to recreate a 0.0001% chance event) and (b) it doesn’t corrupt customer data.
There is a reason so many Rust users are so passionate, and its not because writing Rust is a lovely day in the park every day. Its because you can finally rest at night.
I’m in the process of switching to Rust professionally, dabbled for years and the biggest selling point of Rust is it’s ability to help me write working software with little or no undefined behaviour.
Most languages let you build large applications that are plagued by undefined behaviour.
I think this is debatable. Async rust introduces complexities that don’t exist in other async models which rival in ease-of-use and efficiency. It has unintuitive semantics like
async fn f(&self) -> Tandfn f(&self) -> impl Future<Output=T>subtly not being the same; the latter is+ &'self_lifetimefor the Future). It also allows odd edge-cases that could’ve been banned to simplify the model:Future::poll()can be spuriously called and must keep stored the most recently passed in Waker. Being two words in size, you can’t atomically swap them out resulting in requiring mutual exclusion for updating and notification. Completion based async models don’t require this.join!()which allow stack-allocated structure concurrency. Waker could’ve been tied to the lifetime of the Future, forcing leaf futures to implement proper deregistration of them but reducing task constraints.The cancellation model is also equally simple, useful, and intricately limiting/error-prone:
select!()between any future, not just ones that take CancellationTokens or similar like in Go/C# (neat). Unfortunately, it means everything is cancellable soa().await; b().awaitis no longer atomic to the callee (or “halt-safe”) whereas it is in other async models.Sure, async is hard. But it can be argued that “Rust async” is an additional type of hard.
I don’t disagree Rust introduces an additional kind of hard: async Python is much easier to use than async Rust. I wrote in another comment how it all comes down to trade offs.
I do agree with you, there are more sharp edges in async Rust than normal Rust, but from my understanding of how other languages do it no language has a solution without trade offs that are unacceptable for Rust’s design.
Personally, I think async is the wrong paradigm, but also happens to be the best we have right now. Zig is doing interesting things to prevent having the coloring problem, but I don’t think any language is doing it perfectly.
Any data to back that exact claim? I love rust and I’m working professionally in it for last few years but I think I would still find erlang approach to async code easier.
Fair point, and to really dive into that question we have to be more specific about what exactly we’re talking about. The specific thing that is hard is multi-threaded asynchronous code with memory sharing. To give examples of why this is hard, we can just look at the tradeoffs various languages have made:
Python and Node both opted to not have multi-threading at all, and their asynchronous runtimes are single-threaded. There is work to remove the GIL from Python (which I actually haven’t been following very closely), but in general, one option is to avoid the multi-threading part entirely.
Erlang/BEAM (which I do love) makes a different tradeoff, which is removing memory sharing. Instead, Erlang/BEAM processes are all about message-passing. Personally, I agree with you, and I think the majority of asynchronous/distributed systems can work this way effectively. However, that isn’t to say it is without tradeoffs, message passing is overhead.
So essentially you have two options to avoid the dangerous shenanigans of multi-threaded asynchronous code with memory sharing, which is to essentially constrain one of the variables (multi-threading or memory sharing). Both have performance trade-offs associated with them, which may or may not be deal-breaking.
Rust lets you write multi-threaded asynchronous code with memory sharing and write it correctly. In general though I agree with you about the Erlang approach, and there isn’t really anything stopping you from writing code in that way with Rust. I haven’t been following this project too closely, but Lunatic (https://github.com/lunatic-solutions/lunatic) is a BEAM alternative for Rust, and last I checked in with it they were making great progress.
Yes, I can agree that “multi-threaded asynchronous code with memory sharing” is hard to write. That’s a much more reasonable claim.
The only thing I would disagree slightly is the assertion that rust solves this problem. That’s not really completely true, since deadlocks are still just as easy to create as in c++. For that the only sort of mainstream solution I can think of is STM in Clojure (and maybe in Haskell?).
Fair enough, its just a bit of a mouthful :)
I hadn’t heard of STM, but that is a really cool concept bringing DB transaction-notions to shared memory. Wow I need to read about this more! Though I don’t think that solves the deadlock problem globally, as if we’re considering access which is not memory (eg. network), and thus not covered by STM, then we can still deadlock.
From my understanding, solving deadlocks is akin to solving the halting problem. There just simply isn’t a way to avoid them. But you are right, Rust doesn’t solve deadlocks (nor race conditions in general), just data-races. I’ll modify my original text to clarify this a bit.
Bear in mind, though, that STM has been through a hype cycle and some people are claiming that, like String Theory, it’s in the “dead walking” phase rather than past the hype. For example, Bryan Cantrill touches on transactional memory in a post from 2008 named Concurrency’s Shysters.
I think you’ll find Erlang much harder tbh. Have you used it much? Erlang requires that you do a lot of ‘stitching up’ for async. In Rust you just write
.await, in Erlang you need to send a message, provide your actor’s name so that a response can come back, write a timeout handler in case that response never comes back, handle the fact that the response may come back after you’ve timed out, decide how you can recover from that, manage your state through recursion, provide supervisor hierarchies, etc.Fortunately, almost all of that is abstracted away by gen_server you in practice you don’t actually do all that boilerplate work yourself, you just take advantage of the solid OTP library that ships with Erlang.
For sure I have way more experience with Rust, but I’m not really sure that all of what you listed is downside or Erlang specific. You also need to handle timeouts in rust (eg. tokio::time::timeout and something (match?) to handle the result), you might also need to handle possibility that future will be canceled. Others like recursion (which enables hot reloads) and supervisors are not obvious negatives to me.
Handling a timeout in Rust is pretty trivial. You can just say
timeout(f, duration)and handle theResultright there. For an actor you have to write a generalizedtimeouthandler and, as mentioned, deal with timeouts firing concurrent to the response firing back.I think for the most part handling cancellation isn’t too hard, at least not for most code. Manual implementors of a Future may have to worry about it, but otherwise it’s straightforward - the future won’t be polled, the state is dropped.
TBH I do not see difference between these two.
You can just add
self()as a part of message.As simple as adding
afterblock to thereceiveblock.Solved in OTP 24 with
erlang:monitor(process, Callee, [{alias, reply_demonitor}]).In most cases you simply do not try to recover from that and instead let the caller to do that for you.
Simplest async-like receive looks like, from the docs:
And that is all.
The difference is huge and kind of the whole selling point of actors. You can not share memory across actors, meaning you can not share state across actors. There is no “waiting” for an actor, for example, and there is no way to communicate “inline” with an actor. Instead you must send messages.
Sure, I wasn’t trying to imply that this is complex. It’s just more. You can’t “just” write
.await, it’s “just” add self() and “just” write a response handler and “just” write a timeout handler, etc etc etc. Actors are a very low level concurrency primitive.There’s a lot of “as simple as” and “just” to using an actor. There’s just
.awaitin async/await. If you add a timer, you can choose to do so and use that (and even that is simpler as well).The tradeoff is that you share state and couple your execution to the execution of other futures.
It’s “solved” in that you have a way to handle it. In async/await it’s solved by not existing as a problem to begin with. And I say “problem” loosely - literally the point of Erlang is to expose all of these things, it’s why it’s so good for writing highly reliable systems, because it exposes the unreliability of a process.
It takes all of this additional work and abstraction layering to give you what async/await has natively. And that’s a good thing - again, Erlang is designed to give you this foundational concurrent abstraction so that you can build up. But it doesn’t change the fact that in Rust it’s “just”
.await.Sure, if you pretend you have to raw-dog actors to do concurrency in Erlang, and that OTP doesn’t exist and take care of almost all the boilerplate in gen_server etc. We could also pretend that async/await syntax doesn’t exist in Rust and we need to use callbacks. Wow, complex!
i am curious about what are the recents python3 breaking changes.
Perhaps the best example is 3.6 introducing async and await as keywords in the language (and thus breaking code which used them for variables). In Rust, this was done via the 2018 Edition with the Epoch system.
The difference is a Python 3.6+ interpreter can’t run code from Python 3.5 using async/await as non-keywords, while Rust can compile a mix of 2015 Edition and 2018 Edition code with some using async/await as non-keywords and others with it.
It’s possible GP has reached the same age as me, where you mentally think something happened last year when it was like 3 years ago.
You already have your solution, but you haven’t tried it. NixOS.
Note that Nix isn’t quite completely deterministic due to the activation script which is a giant pile of shell commands.
(It maybe be mostly deterministic, but as a user you can add basically whatever you want to it and break this.)
Nix is only as deterministic as it’s builders and compilers that’s true. It gives you the scaffolding you need but you still have to ensure you follow the rules. If you use a builder that creates random data somewhere then you still won’t have a deterministic output. That’s true of pretty much anything that tries to solve this problem though. They are working with tools that make it a little too easy to violate the rules unexpectedly.
Yeah I was actually surprised by this, from some brief experience with a shell.nix
It seems like it is extremely easy to write Nix files “wrong” and break the properties of Nix
Whereas with Bazel, the build breaks if you do something wrong. It helps you write a correct configuration
Similar issue with Make – Make provides you zero help determining dependencies, so basically all Makefiles have subtle bugs. To be fair Ninja also doesn’t strictly enforce things, but it’s easy to read and debug, and has a few tools like a dep graph exporter.
nix-shell is a bit of a lukewarm compromise between Nix’s ability to supply arbitrary dependencies and the guarantees of the build sandbox. It’s not really trying to be hermetic, just ~humane.
I would like to hear more about your shell.nix experience. It has some issues, for sure, but I have found the platform to be extremely reliable overall.
I linked some comments from this page - https://github.com/oilshell/oil/wiki/Can-Oil-Use-Nix%3F
Is that really relevant? A determined user could flash their BIOS with garbage and break anything they want. Does it matter that you can go out of your way to break your own NixOS system? The point is that as a user you’re not supposed to mess with your system manually, but you’re supposed to make changes by changing your
configuration.nix. This is not the case for most other distros where you make changes to files under/etcside by side with files maintained by the distro packages.The point is that I don’t think anyone is checking. IDK how many options in nixpkgs break this. I don’t think we need to stop the user from purposely breaking it but we should make “official” flags not break it and make it easy for users to not break this.
I did try Nix (briefly).
However, I do think there is a place for classical distributions, as there is a place for distributions like Nix, Guix, and even something like GoboLinux.
Pretty much every concern you share in that post is solved by NixOS. Some points
NixOS isn’t perfect by any means, but it is a solution you seem to be looking for. Guix probably as well, but it’s a smaller and more limited distro.
I took the multiple comments that “NixOS may do this but I’ll have to research” to mean you haven’t tried it. And this entire blog post screams to me that NixOS is a solution for you.
In a way, that’s second system syndrome. Suse Linux was bashed for doing something like that with YaST for a long time…
In case of NixOS, I found that new language (not the syntax, the semantics) changed too fast for my liking…
With a few caveats: Software might still get updated (I suppose you can lock that down to specific revisions, but who does?). From a bugfix/security perspective this may be desirable, but it’s not entirely reproducible, and updates can always introduce new bugs or incompatibilities, too.
All packages are locked in Nix. So With a few caveats: Software might still get updated is not a problem there.
https://www.software.ac.uk/blog/2017-10-05-reproducible-environments-nix offers “download a specific release tarball of buildInput specs” to improve matters. Other than that,
buildInputs = [ bash ]may or may not point at the same version. https://github.com/NixOS/nixpkgs/issues/93327 discusses this for 3 years and there doesn’t seem to be a resolution.You’ll get the same version with the same Nixpkgs, every time.
buildInputs = [ bash ]will always point at the samebashpackage for some given Nixpkgs. Package versioning is a different issue.I count package versioning as part of the system configuration: Sure, your package names are always the same, but the content might differ wildly.
The content will be the same every time with the same system configuration, assuming you’ve pinned your Nixpkgs version.
Or using flakes.
I think you’re misunderstanding the way nixpkgs works with versions and what the versions actually mean there. Check it out in practice. Unless you update the channels/flakes or whatever you use between runs, nothing will change - there’s no explicit pinning required.
The way I manage this is using several different versions of nixpkgs; a “known good” version for each package I want to lock, so for agda I would have
nixpkgsAgda, the regular nixpkgs which is pinned to a stable release, andnixpkgsLatestwhich is pinned to master.Most of my packages are on stable nixpkgs. Every now and then when I run
nix flake update, it pulls in new versions of software. Things pinned to stable have never broken so far. Things pinned to latest are updated to their latest versions, and things pinned to specific packages never change.While it does involve pulling in several versions of nixpkgs, I build a lot of software from source anyway so this doesn’t matter to me very much. I do hope that nixpkgs somehow fixes the growing tarball size in the future…
That’s a snippet of code written in the Nix expression language.
bashis not a keyword in that language, it’s just some arbitrary variable name (indeed, so isbuildInputs). Just like any other language, if thatbashvariable is defined/computed to be some pure, constant value, then it will always evaluate to that same thing. If it’s instead defined/computed to be some impure, under-determined value then it may vary between evaluations. Here’s an example of the former:This evaluates to
{ buildInputs = [ «derivation /nix/store/6z1cb92fmxq2svrq3i68wxjmd6vvf904-bash-5.2-p15.drv» ]; }and (as far as I’m aware) always will do; even ifgithub.comdisappears, it may still live on if thatsha256appears in a cache!Here’s an example of an impure value, which varies depending on the OS and CPU architecture, on the presence/contents of a
NIX_PATHenvironment variable, the presence/contents of a~/.config/nixpkgs/overlays.nixfile, etc.I recommend sticking to the former ;)
PS: I’ve avoided the word “version”, since that concept doesn’t really exist in Nix. It’s more precise to focus on the
.drvfile’s path, since that includes a hash (e.g.6z1cb92fmxq2svrq3i68wxjmd6vvf904) which is the root of a Merkle tree that precisely defines that derivation and all of its transitive dependencies; whereas “version” usually refers to an optional, semi-numerical suffix on the file name (e.g.5.2-p15) which (a) is easy to spoof, and (b) gives no indication of the chosen dependencies, compiler flags, etc. which may have a large effect on the resulting behaviour (which is ultimately what we care about).Do you happen to have an example of such an semantic change at hand? Curious as nix is often regarded as rather conservative and trying to keep backwards-compatibility as good as possible.
Part of the reason Nix/Guix take their particular approach is specifically because the global mutable state of a traditional FHS style distribution is nigh-impossible to manipulate in a deterministic way; not just because of the manipulation by the package manager, but because there is no delineation between mutations by the package manager and those by the user. A great example is
/etc/passwd, which contains both locally created users as well as “service accounts” the distro maintainers make to separate out some unprivileged services.Relevant writing from Lennart Poettering on state in Linux distributions and how to accomplish a factory reset mechanism.
There is a place for “classical” (read: legacy) distributions, yes, in the same way there is a place for collecting stamps.
I want to switch to NixOS so badly, but I also want SELinux support.
it needs a lot of work to enforce 1) determinism/reproducibility, 2) non-hysterisis (when you stretch an object and it doesn’t return /exactly/ back… like hidden side effects in ansible), 3) ease of use, & 4) acceptance testing.
I think we somehow missed to consider two desirable properties of configuration systems that are nevertheless orthogonal: being declarative (which Excel, Terraform, Prolog mostly are) and being Turing-complete (which every Turing-complete programming language is, by definition). If we consider both properties, one can see that
there is no hopeProlog is the answer.Thats how Mark Burgess started (Prolog) and ended up with cfengine. I’d say the modern approach to this would be Erlang :) But I’m betting the only users will be you and I.
Doesn’t being Turing-complete mean that termination is not guaranteed?
Yes, that is why I want my configuration language to not be TC.
What a ridiculous thing to have even happened in the first place, let alone refusing to acknowledge there could possibly be an issue for so long. I glad it’s been fixed but would make me think twice about using serde. I’m sure it’ll be fine, who’s ever heard of a security issue in a codec anyway?
Remember that there are real human being maintaining
serde. It is not, in fact, blindingly obvious to all developers that the pre-compiled blobs were bad; on this site there were loud voices on both sides. Can you imagine suddenly getting caught in the crosshairs of angry developers like that? When I imagine it, it feels bad, and I’m liable to get defensive about it.It may also have been a failed attempt at fixing something you’ve heard people complain about all the time, probably even about your code that slows down peoples builds (*). So yeah it was a bad idea in hindsight, but we don’t need more burned out maintainers from this. And I say this as someone who is openly disappointed by this happening.
(*) I’m not going to discuss how much time it actually saved.
This overview by @cadey implied it did not save much time at all, basically only if you were running a CI setup without a cache.
https://xeiaso.net/blog/serde-precompiled-stupid
Running a CI setup without a cache is, for better or for worse, very common
Yeah, basically the biggest gains are offset by process creation being surprisingly slow. I’m working on a follow-up article where I talk about that in detail.
I posted your piece because it was the first one that explained in detail what the hell was going on, specifically how serde works. Looking forward to a followup.
My workplace is way too big. This describes our CI setup. Only the blessed JVM gets to have cached CI builds.
This is how shadow IT begins. Has anyone started running/sharing their locally setup CI for your project yet?
That’s how it started, then they centralized everything with one team that doles out the “managed CI” offering, with their own global library and controls. Any competing infra gets flagged and audited hardcore until you give up by attrition.
This seems to only be checking the performance under –release. Most compilation is done without –release, meaning that most of the proc macro will not be optimized.
As someone who packages software, I think it’s worth noting that packagers expect different things than end users, though they are compatible.
One of my wishes is to avoid blobs from a vendor, since we can’t always recompile those in the build process to work with the architectures we support.
(The other big difference is the DESTDIR env var. End users don’t generally care, but it becomes essential when preparing a package)
I therefore understand those who support their end users, before getting packaged.
The real human being maintaining serde knew about the pushback that would happen and did it on purpose to prove a point in a pre-RFC he submitted. I don’t feel particularly bad about him getting pushback for using half the Rust ecosystem as his guinea pigs. (In fact I would like to see more of it.)
What’s the reason to believe in this over any other explanation of the situation? E.g. that pushback was unexpected and that the RFC is the result of the pushback, rather than a cause?
I consider dtolnay a competent open source maintainer who understands the people who run his code well, and I would expect any competent open source maintainer to expect such pushback.
But how that necessary leads to “on purpose to prove a point”?
I don’t think dtolnay expected exactly zero pushback. But, given that some people in this thread argue quite a reasonable point that binaries are actually almost as fine as source, it is plausible that only bounded pushback was expected.
The excerpt from the RFC is:
I don’t see someone competent casually pushing such a controversial change, casually saying that this is now the only supported way to use serde, casually pushing a complete long pre-RFC that uses the controversial change to advance it, and then casually reverting the change in the span of a few days. That takes preparation and foresight.
I actually respect this move. It is exactly the kind of move I would do if I had goodwill to burn and was frustrated with the usual formal process, and it takes boldness and courage to pull it off the way he did it. I also think the pushback is entirely appropriate and the degree of it was quite mild.
Aha, thanks! I think that’s a coherent story to infer from this evidence (and I was wondering if there might be some missing bits I don’t know).
From where I stand, I wouldn’t say that this explanation looks completely implausible, but I do find it unlikely.
For me, the salient bits are:
I agree that there are multiple intepretations possible and that yours also follows from the evidence available. The reason I think it’s reasonable to consider something deeper to be going on is: every single Rust controversy I’ve discussed with key Rust people had a lot more going on than was there on the surface. Case in point: dtolnay was also the one thus far unnamed by anyone speaking for the project person who was involved in ThePHD’s talk being downgraded from a keynote. If I see someone acting surreptitiously in one case I will expect that to repeat.
O_o that’s news to me, thanks. It didn’t occur that dtopnay might have been involved there (IIRC, they aren’t a team lead of any top-level team, so I assume weren’t a member of the notorious leadership chat)
Maybe take hearsay from an anonymous Internet catgirl with a grain of salt.
Calling me anonymous is pretty funny, considering I’ve called myself “whitequark” for close to 15 years at this point and shipped several world-class projects under it.
whitequark would be pretty well known to an old Rust team member such as matklad, having been one themself, so no, not anonymous… buut we don’t know this is the same whitequark, so yes, still anonymous.
Hm? Neither of them are Rust team members, unless they are represented under different names in the Project.
I mean, I wrote both Rust language servers/IDEs that everyone is using and whitequark wrote the Ruby parser everyone is using (and also smaltcp). I think we know perfectly fine who we are talking with. One us might be secretly a Labrador in a trench coat, but that doesn’t have any bearing on the discussion, and speculation on that topic is hugely distasteful.
In terms of Rust team membership, I actually don’t know which team whitequark was on, but they are definitely on the alumni page right now. I was on the cargo team and TL for the IDE team.
Thank you for all the context I was missing. Is it just oversight you aren’t on the alumni for those team pages?
Turns out there were at least two bugs in the teams repo with respect to me, thanks for pointing this out!
I’m glad my bickering had at least some positive outcome :)
Probably! I think https://github.com/rust-lang/team/commit/458c784dda91392b710d36661f440de40fdac316should have added me as one, not sure why that didn’t happen
I don’t know what you mean by “the Project”, but the source of truth for Rust team membership is https://github.com/rust-lang/team.
You were talking about “Rust teams” and the only way I’ve seen that term used is to indicate those under the “Rust Project”. Neither person is on a Rust team or an alumni.
https://www.rust-lang.org/governance
That is what I meant, yes. Those pages are generated from the Git repo I linked. Ctrl-F on https://www.rust-lang.org/governance/teams/compiler and https://www.rust-lang.org/governance/teams/alumni.
A find would tell you matklad was not on a team. He was “just” a contributor. No real data exists about whitequark.
Tbh, it more reeks of desperation to make people’s badly configured CI flows faster. I think that a conspiratorial angle hasn’t been earned yet for this and that we should go for the most likely option: it was merely a desperate attempt to make unoptimized builds faster.
I think this is hard to justify when someone comes to you with a security issue, when your response is “fork it, not my problem”, and then closing the issue, completely dismissing the legitimate report. I understand humans are maintaining it, humans maintain all software I use in fact, and I’m not ok with deciding “Oh, a human was involved, I guess we should let security bad practices slide”. I, and I’m sure many others, are not frustrated because they didn’t understand the security implications, but because they were summarily dismissed and rejected, when they had dire implications for all their users. From my understanding, Serde is a) extremely popular in the Rust world, and b) deals in one of the most notoriously difficult kinds of code to secure, so seeing the developers’ reaction to a security issue is very worrying for the community as a whole.
The thing is, its not unambiguous whether this is a security issue. “Shipping precompiled binaries is not significantly more insecure than shipping source code” is an absolutely reasonable stance to have. I even think it is true if we consider only first-order effects and the current state of rust packaging&auditing.
Note also that concerns were not “completely dismissed”. Dismissal looks like “this is not a problem”. What was said was rather “fixing this problem is out of scope for the library, if you want to see it fixed, work on the underlying infrastructure”. Reflecting on my own behavior in this discussion, I might be overly sensitive here, but to me there’s a world of difference between a dismissal, and an acknowledgment with disagreement on priorities.
This is perhaps a reasonable take-away from all the internet discussions about the topic, but I don’t think this actually reflects what did happen.
The maintainer was responsive on the issue and they very clearly articulated that:
Afterwards, when it became obvious that the security concern is not niche, but have big implications for the whole ecosystem, the change was reverted and a lot of follow-up work landed.
I do think it was a mistake to not predict that this change will be this controversial (or to proceed with controversial change without preliminary checks with wider community).
But, given that a mistake had been made, the handling of the situation was exemplary. Everything that needed fixing was fixed, promptly.
I’m still waiting to hear what “security concern” there was here. Other language-package ecosystems have been shipping precompiled binaries in packages for years now; why is it such an apocalyptically awful thing in Rust and only Rust?
The main thing is loss of auditing ability — with the opaque binaries, you can not just look at the package tarbal from crates.io and read the source. It is debatable how important that is: in practice, as this very story demonstrates, few people look at the tarballs. OTOH, “can you look at tarballs” is an ecosystem-wide property — if we lose it, we won’t be able to put the toothpaste back into the tube.
This is amplified by the fact that this is build time code — people are in general happier with sandbox the final application, then with sandboxing the sprawling build infra.
With respect to other languages — of course! But also note how other languages are memory unsafe for decades…
It’s not that hard to verify the provenance of a binary. And it appears that for some time after serde switched to shipping the precompiled macros, exactly zero people actually were auditing it (based on how long it took for complaints to be registered about it).
The ecosystem having what boils down to a social preference for source-only does not imply that binary distributions are automatically/inherently a security issue.
My go-to example of a language that often ships precompiled binaries in packages is Python. Which is not exactly what I think of when I think “memory unsafe for decades”.
Verifying provenance and auditing source are orthogonal. If you have trusted provenance, you can skip auditing the source. If you audited the source, you don’t care about the provenance.
It’s a question which one is more practically important, but to weight this tradeoff, you need to acknowledge its existence.
This sounds like:
This doesn’t sound like:
I don’t know where your last two blockquotes came from, but they didn’t come from my comment that you were replying to, and I won’t waste my time arguing with words that have been put in my mouth by force.
That’s how I read your reply: as an absolute refusal to acknowledge that source auditing is a thing, rather than as a nuanced comparison of auditing in theory vs auditing in practice.
It might not have been your intention to communicate that, but that was my take away from what’s actually written.
Once again, I don’t intend to waste my time arguing with someone who just puts words in my mouth.
In the original github thread, someone went to great lengths to try to reproduce the shipped binary, and just couldn’t do it. So it is very reasonable to assume that either they had something in their build that differed from the environment used to build it, or that he binary was malicious, and without much deeper investigation, it’s nearly impossible to tell which is the answer. If it was trivial to reproduce to build with source code you could audit yourself, then there’s far less of a problem.
Rust doesn’t really do reproducible builds, though, so I’m not sure why people expected to be able to byte-for-byte reproduce this.
Also, other language-package ecosystems really have solved this problem – in the Python world, for example, PyPI supports a verifiable chain all the way from your source repo to the uploaded artifact. You don’t need byte-for-byte reproducibility when you have that.
Ah yes, garbage collected languages are famously ‘memory unsafe for decades’
I guesss I should clarify that in GP comment the problem is misalignment between maintainer’s and user’s view of the issue. This is a problem irrespective of ground truth value of security.
Maybe other language package ecosystems are also wrong to be distributing binaries, and have security concerns that are not being addressed because people in those ecosystems are not making as much of a fuss about it.
If there were some easy way to exploit the mere use of precompiled binaries, someone would have by now. The incentives to use such an exploit are just way too high not to.
There are ways to exploit binary releases. It’s certainly not easy, but this has definitely been exploited in the wild.
You can read this page https://reproducible-builds.org/docs/buy-in/ to get a high-level history of the “reproducible build” (and bootstrapping) movement.
Anecdotally, I almost always see Python malware packaged as source code. I think that could change at any time to compiled binaries fwiw, just a note.
I don’t think attackers choosing binary payloads would mean anything for anyone really. The fundamental problem isn’t solved by reproducible builds - those only help if someone is auditing the code.
The fundamental problem is that your package manager has near-arbitrary rights on your computer, and dev laptops tend to be very privileged at companies. I can likely go from ‘malicious build script’ to ‘production access’ in a few hours (if I’m being slow and sneaky) - that’s insane. Why does a build script have access to my ssh key files? To my various tokens? To my ~/.aws/ folder? Insane. There’s zero reason for those privileges to be handed out like that.
The real solution here is to minimize impact. I’m all for reproducible builds because I think they’re neat and whatever, sure, people can pretend that auditing is practical if that’s how they want to spend their time. But really the fundamental concept of “running arbitrary code as your user” is just broken, we should fix that ASAP.
Like I’ve pointed out to a couple people, this is actually a huge advantage for Python’s “binary” (
.whl) package format, because its install process consists solely of unpacking the archive and moving files to their destinations. It’s the “source” format that can ship asetup.pyrunning arbitrary code at install time. So tellingpipto exclusively install from.whl(with--only-binary :all:) is generally a big security win for Python deployments.(and I put “binary” in scare quotes because, for people who aren’t familiar with it, a Python
.whlpackage isn’t required to contain compiled binaries; it’s just that the.whlformat is the one that allows shipping those, as well as shipping ordinary Python source code files)Agree. But that’s a different threat, it has nothing to do with altered binaries.
Code auditing is worthless if you’re not sure the binary you’re running on your machine has been produced from the source code you’ve audited. This source <=> binary mapping is precisely where source bootstrapping + reproducible builds are helping.
This is a false dichotomy. I think we agree on the fact we want code audit + binary reproducibility + proper sandboxing.
Well, we disagree, because I think they’re identical in virtually every way.
I’m highly skeptical of the value behind code auditing to begin with, so anything that relies on auditing to have value is already something I’m side eyeing hard tbh.
I think where we disagree on the weights. I barely care about binary reproducibility, I frankly don’t think code auditing is practical, and I think sandboxing is by far the most important, cost effective measure to improve security and directly address the issues.
I am familiar with the concept of reproducible builds. Also, as far as I’m aware, Rust’s current tooling is incapable of producing reproducible binaries.
And in theory there are many attack vectors that might be present in any form of software distribution, whether source or binary.
What I’m looking for here is someone who will step up and identify a specific security vulnerability that they believe actually existed in
serdewhen it was shipping precompiled macros, but that did not exist when it was shipping those same macros in source form. “Someone could compromise the maintainer or the project infrastructure”, for example, doesn’t qualify there, because both source and binary distributions can be affected by such a compromise.Aren’t there links in the original github issue to exactly this being done in the NPM and some other ecosystem? Yes this is a security problem, and yes it has been exploited in the real world.
I’m going to quote my other comment:
If you have proof of an actual concrete vulnerability in
serdeof that nature, I invite you to show it.The existence of an actual exploit is not necessary to be able to tell that something is a serious security concern. It’s like laying an AR-15 in the middle of the street and claiming there’s nothing wrong with it because no one has picked it up and shot someone with it. This is the opposite of a risk assessment, this is intentionally choosing to ignore clear risks.
There might even be an argument to make that someone doing this has broken the law by accessing a computer system they don’t own without permission, since no one had any idea that this was even happening. To me this is up with with Sony’s rootkit back in the day, completely unexpected, unauthorised behaviour that no reasonable person would expect, nor would they look out for it because it is just such an unreasonable thing to do to your users.
I think the point is that if precompiled macros are an AR-15 laying in the street, then source macros are an AR-15 with a clip next to it. It doesn’t make sense to raise the alarm about one but not the other.
I think this is extreme. No additional accessing of any kind was done. Binaries don’t have additional abilities that
build.rsdoes not have. It’s not at all comparable to installing a rootkit. The precompiled macros did the same thing that the source macros did.Once again, other language package ecosystems routinely ship precompiled binaries. Why have those languages not suffered the extreme consequences you seem to believe inevitably follow from shipping binaries?
Even the most extreme prosecutors in the US never dreamed of taking laws like CFAA this far.
I think you should take a step back and consider what you’re actually advocating for here. For one thing, you’ve just invalidated the “without any warranty” part of every open-source software license, because you’re declaring that you expect and intend to legally enforce a rule on the author that the software will function in certain ways and not in others. And you’re also opening the door to even more, because it’s not that big a logical or legal leap from liability for a technical choice you dislike to liability for, say, an accidental bug.
The author of
serdedidn’t take over your computer, or try to. All that happened wasserdestarted shipping a precompiled form of something you were going to compile anyway, much as other language package managers already do and have done for years. You seem to strongly dislike that, but dislike does not make something a security vulnerability and certainly does not make it a literal crime.I think that what actually is happening in other language ecosystems is that while there are precompiled binaries sihpped along some installation methods, for other installation methods those are happening by source.
So you still have binary distribution for people who want that, and you have the source distribution for others.
I have not confirmed this but I believe that this might be the case for Python packages hosted on debian repos, for example. Packages on PyPI tend to have source distributions along with compiled ones, and the debian repos go and build packages themselves based off of their stuff rather than relying on the package developers’ compiled output.
When I release a Python library, I provide the source and a binary. A linux package repo maintainer could build the source code rather than using my built binary. If they do that, then the thing they “need to trust” is the source code, and less trust is needed on myself (on top of extra benefits like source code access allowing them to fix things for their distribution mechanisms)
I don’t know of anyone who actually wants the sdists from PyPI. Repackagers don’t go to PyPI, they go to the actual source repository. And a variety of people, including both me and a Python core developer, strongly recommend always invoking
pipwith the--only-binary :all:flag to force use of.whlpackages, which have several benefits:--require-hashesand--no-deps, you get as close to perfectly byte-for-byte reproducible installs as is possible with the standard Python packaging toolchain..whlhas no scripting hooks (as opposed to an sdist, which can run arbitrary code at install time via itssetup.py).I misread that as “sadists from PyPi” and could not help but agree.
I mean there are plenty of packages with actual native dependencies who don’t ship every permutation of platform/Python version wheel needed, and there the source distribution is available. Though I think that happens less and less since the number of big packages with native dependencies is relatively limited.
But the underlying point is that with an option of compiling everything “from source” available as an official thing from the project, downstream distributors do not have to do things like, say, confirm that the project’s vendored compiled binary is in fact compiled from the source being pointed at.
Install-time scripting is less of an issue in this thought process (after all, import-time scripting is a thing that can totally happen!). It should feel a bit obvious that a bunch of source files is easier to look through to figure out issues rather than “oh this part is provided by this pre-built binary”, at least it does to me.
I’m not arguing against binary distributions, just think that if you have only the binary distribution suddenly it’s a lot harder to answer a lot of questions.
As far as I’m aware, it was possible to build
serde“from source” as a repackager. It did not produce a binary byte-for-byte identical to the one being shipped first-party, but as I understand it producing a byte-for-byte identical binary is not something Rust’s current tooling would have supported anyway. In other words, the only sense in which “binary only” was true was for installing fromcrates.io.So any arguments predicated on “you have only the binary distribution” don’t hold up.
Hmm, I felt like I read repackagers specifically say that the binary was a problem (I think it was more the fact that standard tooling didn’t allow for both worlds to exist). But this is all a bit moot anyways
It’s a useful fallback when there are no precompiled binaries available for your specific OS/Arch/Python version combination. For example when pip installing from a ARM Mac there are still cases where precompiled binaries are not available, there were a lot more closer to the M1 release.
When I say I don’t know of anyone who wants the sdist, read as “I don’t know anyone who, if a wheel were available for their target platform, would then proceed to explicitly choose an sdist over that wheel”.
Argumentum ad populum does not make the choice valid.
Also, not for nothing, most of the discussion has just been assuming that “binary blob = inherent automatic security vulnerability” without really describing just what the alleged vulnerability is. When one person asserts existence of a thing (such as a security vulnerability) and another person doubts that existence, the burden of proof is on the person asserting existence, but it’s also perfectly valid for the doubter to point to prominent examples of use of binary blobs which have not been exploited despite widespread deployment and use, as evidence in favor of “not an inherent automatic security vulnerability”
Yeah, this dynamic has been infuriating. In what threat model is downloading source code from the internet and executing it different from downloading compiled code from the internet and executing it? The threat is the “from the internet” part, which you can address by:
Anyone with concerns about this serde change should already be doing one or both of these things, which also happen to make builds faster and more reliable (convenient!).
Yeah, hashed/pinned dependency trees have been around forever in other languages, along with tooling to automate their creation and maintenance. It doesn’t matter at that point whether the artifact is a precompiled binary, because you know it’s the artifact you expected to get (and have hopefully pre-vetted).
Downloading source code from the internet gives you the possibility to audit it, downloading a binary makes this nearly impossible without whipping out a disassembler and hoping that if it is malicious, they haven’t done anything to obfuscate that in the compiled binary. There is a “these languages are turing complete, therefore they are equivalent” argument to be made, but I’d rather read Rust than assembly to understand behaviour.
The point is that if there were some easy way to exploit the mere use of precompiled binaries, the wide use of precompiled binaries in other languages would have been widely exploited already. Therefore it is much less likely that the mere presence of a precompiled binary in a package is inherently a security vulnerability.
I’m confused about this point. Is anyone going to fix crates.io so this can’t happen again?
Assuming that this is a security problem (which I’m not interested in arguing about), it seems like the vulnerability is in the packaging infrastructure, and serde just happened to exploit that vulnerability for a benign purpose. It doesn’t go away just because serde decides to stop exploiting it.
I don’t think it’s an easy problem to fix: ultimately, package registry is just a storage for files, and you can’t control what users put there.
There’s an issue open about sanitizing permission bits of the downloaded files (which feels like a good thing to do irrespective of security), but that’s going to be a minor speed bump at most, as you can always just copy the file over with the executable bit.
A proper fix here would be fully sandboxed builds, but:
Who’s ever heard of a security issue caused by a precompiled binary shipping in a dependency? Like, maybe it’s happened a few times? I can think of one incident where a binary was doing analytics, not outright malware, but that’s it.
I’m confused at the idea that if we narrow the scope to “a precompiled binary dependency” we somehow invalidate the risk. Since apparently “curl $FOO > sh” is a perfectly cromulent way to install things these days among some communities, in my world (30+ year infosec wonk) we really don’t get to split hairs over ‘binary v. source’ or even ‘target v. dependency’.
I’m not sure I get your point. You brought up codec vulns, which are irrelevant to the binary vs source discussion. I brought that back to the actual threat, which is an attack that requires a precompiled binary vs source code. I’ve only seen (in my admittedly only 10 Years of infosec work) such an attack one time, and it was hardly an attack and instead just shady monetization.
This is the first comment I’ve made in this thread, so I didn’t bring up codecs. Sorry if that impacts your downplaying supply chain attacks, something I actually was commenting on.
Ah, then forget what I said about “you” saying that. I didn’t check who had commented initially.
As for downplaying supply chain attacks, not at all. I consider them to be a massive problem and I’ve actively advocated for sandboxed build processes, having even spoken with rustc devs about the topic.
What I’m downplaying is the made up issue that a compiled binary is significantly different from source code for the threat of “malicious dependency”.
So not only do you not pay attention enough to see who said what, you knee-jerk responded without paying attention to what I did say. Maybe in another 10 years…
Because I can
curl $FOO > foo.sh; vi foo.shthen can choose tochmod +x foo.sh; ./foo.sh. I can’t do that with an arbitrary binary from the internet without whipping out Ghidra and hoping my RE skills are good enough to spot malicious code. I might also miss it in some downloaded Rust or shell code, but the chances are significantly lower than in the binary. Particularly when the attempt from people in the original issue thread to reproduce the binary failed, so no one knows what’s in it.No one, other than these widely publicised instances in NPM, as well as PyPi and Ruby, as pointed out in the original github issue. I guess each language community needs to rediscover basic security issues on their own, long live NIH.
Am I missing something? Both links involve malicious source files, not binaries.
I hadn’t dived into them, they were brought up in the original thread, and shipping binaries in those languages (other than python with wheels) is not really common (but would be equally problematic). But point taken, shouldn’t trust sources without verifying them (how meta).
But the question here is “Does a binary make a difference vs source code?” and if you’re saying “well history shows us that attackers like binaries more” and then history does not show that, you can see my issue right?
But what’s more, even if attackers did use binaries more, would we care? Maybe, but it depends on why. If it’s because binaries are so radically unauditable, and source code is so vigilitantly audited, ok sure. But I’m realllly doubtful that that would be the reason.
There’s some interesting meat to think about here in the context of package management, open source, burden on maintainers, varying interest groups.