I wonder how much security review maintainers actually do. Reviews are difficult and incredibly boring. I have reviewed a bunch of crates for cargo-crev, and it was very tempting to gloss over the code and conclude “LGTM”.
Especially in traditional distros that bundle tons of C code, a backdoor doesn’t have to look like if (user == "hax0r") RUN_BACKDOOR(). It could be something super subtle like an integer overflow when calculating lengths of buffers. I don’t expect a volunteer maintainer looking at a diff to spot something like that.
I assume that as long as we keep insisting on designing complex software in languages that can introduce vulnerabilities, then such software will contain such vulnerabilities (introduced intentionally or unintentionally).
I think integer overflow is a great example.
In C & C++ unsigned integers wrap on overflow and signed integers exhibit UB on overflow. Rust could have fixed this, but overflow is unchecked by default in release builds for both signed and unsigned integers! Zig doubles down on the UB by making signed and unsigned integer overflow both undefined unless you use the overflow-specific operators!
How is it that none of these languages Rust doesn’t handle overflow safely by default?!
(edit: My information on Zig was inaccurate and false, it actually does have checks built into the type-system (error system?) of the language!)
Are the performance gains really worth letting every addition operation be a potential source of uncaught bugs and vulnerabilities? I certainly don’t think so.
Rust did fix it. In release builds, overflow is not UB. The current situation is that overflow panics in debug mode but wraps in release mode. However, it is possible for implementations to panic on overflow in release mode. See: https://github.com/rust-lang/rfcs/blob/master/text/0560-integer-overflow.md
Obviously, there is a reason for this nuanced stance. Because overflow checks presumably inhibit performance or other optimizations in the surrounding code.
To clarify what I was saying: I consider overflow and UB to both be unacceptable behavior by my standard of safety. Saying that it is an error to have integer overflow or underflow, and then not enforce that error by default for all projects feels similar to how C has an assert statement that is only enabled for debug builds (well really just when NDEBUG isn’t defined).
So far the only language that I have seen which can perform these error checks at compile time (without using something like Result) is ATS, but that language is a bit beyond my abilities.
If I remember correctly, some measurements were taken and it was found that always enabling arithmetic checks in the Rust compiler lead to something like a 5% performance decrease overall. The Rust team decided that this hit was not worth it, especially since Rust is aiming for performance parity with C++. I respect the team’s decision, but it does not align with the ideals that I would strive for in a safety-first language (although Rust’s primary goal is memory safety not everything-under-the-sun safety).
That’s fine to consider it unacceptable, but it sounded like you thought overflow was UB in Rust based on your comment. And even then, you might consider it unacceptable, but surely the lack of UB is an improvement. Of course, you can opt into checked arithmetic, but it’s quite a bit more verbose.
But yes, it seems like you understand the issue and the trade off here. I guess I’m just perplexed by your original comment where you act surprised at how Rust arrived at its current state. From what I recall, the intent was always to turn overflow checks on in release mode once technology erased the performance difference. (Or rather, at least that was the hope as I remember it at the time the RFC was written.)
Oh no I actually am glad that at least Rust has it defined. It is more like Ada in that way, where it is something that you can build static analysis tools around instead of the UB-optimization-wild-west situation like in C/C++.
From what I recall, the intent was always to turn overflow checks on in release mode once technology erased the performance difference.
Yes I believe that was in the original RFC, or at least it was discussed a bunch in the GitHub comments, but “once technology erased the performance difference” is different than “right now in shipping code after the language’s 1.0 release”.
I would say it is less of surprise that I feel and more a grumpy frustration - one of the reasons I picked up (or at least tried to pick up RIP) Rust in the first place was because I wanted a more fault-tolerant systems programming language as my daily driver (I am primarily a C developer).
But I remember being severely disappointed when I learned that overflow checks were disabled by default in release builds because a that ends up being only marginally better than everyone using -fwrapv in GCC/Clang.
I like having the option to enable the checks myself, but I just wish it was universal, because that would eliminate a whole class of errors from the mental model of a program (by default).
With the cost, it depends. Even though overflow checks are cheap themselves, they have a cost of preventing autovectorization and other optimizations that combine or reorder operations, because abort/exception is an observable side effect. Branches of the checks are trivially predictable, but they crowd out other branches out of branch predictors, reducing their overall efficiency.
OTOH overflows aren’t that dangerous in Rust. Rust doesn’t deal with bare malloc + memcpy. Array access is checked, so you may get a panic or a higher-level domain-specific problem, but not the usual buffer overflow. Rust doesn’t implicitly cast to a 32-bit int anywhere, so most of the time you have to overflow 64-bit values.
How is it that none of these languages handle overflow safely by default?!
The fact that both signed and unsigned integer overflow is (detectable) UB in Zig actually makes zig’s –release-safe build mode safer than Rust’s –release with regards to integer overflow. (Note that Rust is much safer however with regards to memory aliasing)
I stand corrected.
I just tested this out and it appears that Zig forces me to check an error!
I’ll cross out the inaccuracy above…and I think I’m going to give Zig a more rigorous look… I’m quite stunned and impressed actually. Sorry for getting grumpy about something I was wrong about.
I don’t think so either. I did a benchmark of overflow detection in expressions and it wasn’t that much of a time overhead, but it would bloat the code a bit, as you have to not only check after each instruction, but only use instructions that set the overflow bit.
The problem with this post is that when people complain about Rust programmes having too many dependencies they’re not actually complaining about them having too many dependencies, they’re complaining about them taking too long to compile and having too many untrusted dependencies.
The compile time issue has been discussed loads and discussing it here again probably wouldn’t be very productive, but it’s still worth pointing one thing out: it wouldn’t be nearly as much of an issue if you didn’t also have to build all the dependencies as well. Even quite big programmes, in my experience, don’t take very long to build. What takes forever is building them and all their recursive dependencies every time I update a programme, even when the dependencies haven’t changed.
There’s also the factor of keeping everything synced up. Programmes shouldn’t be using whatever the version of their dependencies were out last time the programme was updated by its authors. They should be using the latest version that actually exists, for obvious security reasons.
But the much more important factor is trust.
Now you can question how effective e.g. Debian is at evaluating new additions to its repositories and changes to the packages in its repositories but Debian packages are more carefully curated than Cargo packages. Packages on crates.io can be uploaded by anyone, and basically the only thing most people use to evaluate them is whether they look suspicious. If someone has good documentation and lots of downloads, most people - certainly I - just assume that they must be fairly trustworthy. That’s a much lower bar than the bar to get into Debian. The reason most people avoid any dependencies that aren’t in their system package manager is that downloading random crap from the internet does and should feel scary. Whenever I have to set up a Windows computer it feels so strange just downloading programs from random websites. For many programs the standard way to download them seems to be from Softpedia or one of those similar highly suspicious ad-supported software repositories.
Recently I installed spotifyd through the Arch User Repositories. Not very trustworthy, using the AUR, you might say, but yay shows you the PKGBUILD for the package you’re installing and it looked fine to me assuming that Spotifyd was fine.
Now the spotifyd crate has 27 direct dependencies. Of those, some are standard crates across the whole Rust ecosystem and are maintained by well-known members of the Rust community or the Rust team themselves like lazy_static, failure, futures, libc, log, serde, tokio-core, tokio-io, tokio-signal, url, percent-encoding, and env_logger. It seems fairly safe to trust them. However it’s still concerning to me that many are pre-1.0. tokio-* and futures are understandable, but libc? In fact, Alex Crichton just (as I’m typing this) posted an issue to the libc page on GitHub saying it has no active maintainers. This is a repository hosted under the rust-lang organisation on GitHub, which you’d think would mean it’d be officially supported by the Rust team, which contains a LOT of code, and which has >23m downloads.
Take chrono (0.4) for example, a library with a huge number of downloads. Its author, responding over two years ago to an issue on GitHub, lists many backwards-incompatible API changes he’d like to make before releasing the library as 1.0. That’s fine, that’s the whole point of releasing it as v0.4 as he is currently doing, but why is it considered okay for this library to have 9m downloads on crates.io and many many reverse dependencies that could all need to rewrite swathes of code in future?
daemonize (0.4) hasn’t had changes on Git for nearly a year. fern (0.5) hasn’t seen changes for four months. gethostname is a crate for getting the computer’s hostname that seems considerably complicated for what would in C be one line for each platform with some #ifdefs etc. It also lets you set the system hostname for some reason, despite the name.
hex is another crate that is version 0.4. Seems to be a pretty common problem in Rust: someone works on something for a bit then finds something else to do, but nothing stops you from using these packages in your software. It’s only got 60 commits in the repository’s history, which seems like not many, but it’s only very simple functionality. I’d suggest this should be in the standard library. Its code is ‘mostly extracted from rustc-serialize’, which suggests that the author’s suggestion that
Part of my theory is that C programs commonly omit deps by re-implementing the bits they need themselves, because for small stuff it’s far easier to just write or copy-paste your own code than to actually use a library.
highly amusing: it seems that the same thing happens in Rust anyway because, well, for small stuff it’s easier to just copy-paste or rewrite code than to use a big complicated library in Rust as well.
I’m not going to go through the rest of its dependencies because I think the point has been made: peoples’ concerns about the Cargo/npm approach to dependencies are quite well-founded. Quite reasonable programmes have huge numbers of recursive dependencies (Spotifyd required >400 dependent crates when I built it) that need to be rebuilt every time they’re updated, and the dependencies are untrusted, unverified, unstable and are not automatically kept up-to-date by my system package manager.
There’s also the factor of keeping everything synced up. Programmes shouldn’t be using whatever the version of their dependencies were out last time the programme was updated by its authors. They should be using the latest version that actually exists, for obvious security reasons.
I agree with quite a bit of your post, but this struck me as the polar opposite of security.
Maintainers get hacked. I’d rather have the version I’ve audited the source of than some random newer one.
The likelihood that you’ve successfully discovered all security flaws in a library through auditing are nearly zero. Your only defense against the flaws someone else will find first is being able to rapidly and safely upgrade to a version that fixes them.
For context… my employer very much cares about the security of our software. A lot of very smart people – and I count myself among them in this narrow regard – have weighed the alternatives, including auditing an unchanging pool of trusted code, and concluded that agile and frequent updates to the newest available version of our dependencies is the least risky approach to software security.
Realistically, distro packages are quite a bit less audited than people expect.
Remember when people were upset that Chromium in Debian downloaded the “OK Google” recognition blob when they assumed Debian had vetted all of Chromium?
Excellent counterpoints, thank you! Can I include your comment on that article as an appendix?
The trust problem is indeed part of the real problem (the other being robustness, which is related), and compile times are related because they force people to notice and think about what’s going on. There’s a few different approaches to trying to deal with trust, I think the biggest ones are crev and cargo-audit, but it’s a bit of an unsolved problem how well these things are actually going to work in the long run. So I suppose the real question is not how people use the code that makes their external libraries, and what tools they use, but rather how that code is managed and distributed – who is responsible for it. Debian can include code whose maintainer has been dead for 20 years because they have maintainers of their own, a security team, and a mechanism to consistently apply patches to the code if necessary. If a package in Debian becomes orphaned it can still be maintained by other people without requiring a fork just to be able to publish on crates.io (which I have had to do ), and if it remains orphaned then it can be removed from the next version of Debian and the rest of the system will adjust around that fact.
There’s room for counterpoints in all your examples, and you raise a number of real interesting questions: Is four months, or a year, actually that long for no changes in a library if that library performs its purpose and has no known security problems? There’s plenty of C code out there with no version number at all – how does one manage that? If a problem is discovered, who should fix it and how does that fix get communicated? What should and should not be in the standard library? Like I said at the end, these are big problems that don’t have a single solution.
Excellent counterpoints, thank you! Can I include your comment on that article as an appendix?
Thanks! I think it would be better to just link the lobsters thread and say something like ‘there was some interesting discussion here about this’. I don’t think my comment deserves to be highlighted over the other wonderful comments here.
who is responsible for it
I think this is a key point. Ultimately, there’s nothing inherent in the compilation model forcing it to be this way and there’s nothing inherent in dpkg forcing Debian to be well maintained and audited. And in fact I’m sure there are lots of bits of Debian that aren’t very well audited.
Maybe what crates.io needs is something akin to Arch’s distinction between core, extra, community and the AUR. Packages could be marked explicitly as being core (developed, maintained and supported by the Rust team, basically the ‘extended standard library’), extra (endorsed by the Rust team but not actively developed by them, closely audited for changes, I think this is what rust-lang-nursery is meant to be, roughly?), community (crates that are supported by and audited by a group of ‘trusted’ people, which would be a fairly broad group of people, much like the Arch ‘trusted users’ group, the idea being that they’re at least notionally endorsed by a group of at least notionally trusted people, but you should still be careful, but you’ll probably be fine), and other (all other crates).
Joe Bloggs creates a crate he thinks is useful. He puts it on the AUR. It goes into the ‘other’ category. Joe posts it to /r/rust-lang and people really like the design or the API or the functionality or whatever, but people have some concerns that it’s not super idiomatic. Joe fixes up most of these issues and some trusted community member adopts it into the ‘community’ category after a little while, and maybe Joe becomes a trusted community member himself after a little while.
Eventually this crate becomes so widely used that it’s considered not just a popular third party library but a foundational component of the ecosystem. It gets imported into the ‘extra’ category and now it’s guaranteed regular fuzzing, some oversight over changes, reasonably close auditing from the community and the Rust developers, etc.
Maybe eventually it becomes so core to the ecosystem that the Rust team adopts it as a ‘core’ crate, meaning it’s now actively supported and developed by the Rust team.
Or maybe it stops at any one of those stages. Being in ‘other’ doesn’t mean it’s bad, just not part of the standard ecosystem. People would be taught to be very wary of adding packages in ‘other’ without closely auditing them, because they are totally untrusted.
Obviously for something to be imported into ‘X’ its dependencies would all need to be ‘X’ or higher.
Thing is, Rust already has the beginnings of this. It does have crates that are developed by the core team. It does have blessed crates not developed by the core team but acknowledged as being used in almost every large project and important to the health of the ecosystem that are regularly fuzzed. It does have a broad set of widely used libraries that are considered trustworthy. It just needs these categories to be explicitly marked on the packages themselves and tooling for seeing how trustworthy the packages you are using actually are.
I can’t help thinking there’s a problem in your argument, though.
Your concern, if I’ve understood it, is that people are only trusting, say, a crate because a bunch of other people apparently have trusted it, too, and, presumably, those people haven’t had bad things happen as a result, or they’d have complained in places you could see. But then you seem to ascribe higher value to Linux-distro package repositories, when the only argument that can be made in support of trusting the distro’s packages is that… a bunch of other people apparently have trusted them, too, and presumably haven’t had bad things happen as a result.
I’m not sure there’s a way to qualitatively distinguish between “Debian’s packages are trustworthy because lots of people say so” and “crates.io is trustworthy because lots of people say so”. And both groups certainly face negative repercussions if the packages turn out not to be trustworthy, because both groups have staked at least some of their reputation on the claim of trustworthiness.
And keep in mind that mere presence of security issues doesn’t seem to impact the trustworthiness of, say, the Debian maintainers. Debian routinely updates packages to fix CVEs, for example, and nobody seems to think it reflects poorly on Debian that those vulnerable packages were in Debian’s repositories. In fact, quite the opposite: we explicitly don’t expect Debian package maintainers to be doing comprehensive security audits that might catch those problems up-front.
Merely restricting the number of people who can upload/add a package also doesn’t do it, because (for example) restrictive access policies occur in both good and bad projects, and we all know that.
Or if your argument is “I personally trust the Debian maintainers more than I trust the crate maintainers”, that’s an argument that’s persuasive to you personally, but not one that necessarily is persuasive to others, unless others are open to being persuaded on the basis of “other people trust them”, which then lands you back at the start.
Take chrono (0.4) for example, a library with a huge number of downloads. Its author, responding over two years ago to an issue on GitHub, lists many backwards-incompatible API changes he’d like to make before releasing the library as 1.0. That’s fine, that’s the whole point of releasing it as v0.4 as he is currently doing, but why is it considered okay for this library to have 9m downloads on crates.io and many many reverse dependencies that could all need to rewrite swathes of code in future?
Why do you need to rewrite? Just pin your dep to 0.4 and be done with it. If 1.0 is too much work, don’t update. I can’t see this as a problem to be honest. Seems at best a nitpick about changes happen, not rust specific. Look at openssl breaking changes between versions and a lot more stuff depends on that.
hex is another crate that is version 0.4. Seems to be a pretty common problem in Rust: someone works on something for a bit then finds something else to do, but nothing stops you from using these packages in your software.
I see this the opposite way: the software is DONE and doesn’t need to change which is a good thing. I find the incessant view that software needs to be constantly coddled/babied and updated an antipattern. Do you see that something needs to change in that library? Part of having so many dependencies should mean things get stable and essentially need no more updates or at best few updates. I want libraries to be stable and not get updated. Its not like if we had the equivalent of abs() in a library we would think it needs to be updated every few months. Once its done its done barring syntax changes or other changes to the overall language.
Quite reasonable programmes have huge numbers of recursive dependencies (Spotifyd required >400 dependent crates when I built it) that need to be rebuilt every time they’re updated, and the dependencies are untrusted, unverified, unstable and are not automatically kept up-to-date by my system package manager.
As a nix user I’d say this reflects more on your package manager of choice than rust itself. None of this need be an issue its more an issue in historically how package management has viewed dependencies.
If the software were done it would be 1.0 and stable. It isn’t done. It still has serious issues that need to be fixed, which is why the authors want to change the API for 1.0. That’s kind of the point of 0.x releases, they’re meant to be experimental unstable releases that can’t be relied upon.
If a crate reaches stable at 0.X.Y unexpectedly, why force evaluation of a 1.0 upgrade if there are no breaking changes to be made for it? Sure, let the next update be 1.0, but there’s no immediate reason to move, especially when you can say this in the crate’s top-level documentation and update that with just a 0.X.(Y+1) minor bump.
Because 1.0 is literally defined to mean stable for a Rust crate. That’s the definition of stable. They’re the same thing. 1.0+ = stable. They are synonyms.
recursive dependencies […] that need to be rebuilt every time they’re updated, and the dependencies are untrusted, unverified, unstable and are not automatically kept up-to-date by my system package manager.
Lack of updates for transitive deps seems like merely a lack of integration with the package manager. There could be something that watches the Cargo index, rebuilds affected packages and releases updates for them. I know package managers prefer unbundling and dynamic linking instead, but when dependencies rely on using generics and inlining, there’s no easy way around that.
I don’t understand the issue people have with dependencies per say.
As in, I think the whole fear of dependencies comes from the time of bad language package manager & repository (e.g. pypi&pip, or whatever the heck people used in the dark days when pypi&pip was considered a good tool).
I think the other part comes from understanding dependencies in the C++/C sense, where every time I added a library I hadn’t used before I’d have to spent 20 minutes figuring out what’s the best way to include it and how to make it compile.
Also, fat binaries should just become a thing. Memory and storage issues that made fat binaries not a good option are just not a thing anymore, if they are, they are no niche systems where you can just use distros specifically made for that.
As in, things like openssl should be dynamically linked, I agree with that. However the category of things which should be dynamically linked is tiny and far outweights the 99% of libraries which could just be static. Never has one uttered the words “Oh gee, I’m sure glad this package dynamically links to lboos, otherwise it might cost me an extra 5Mb of disk space and have from the old version that is 5.6% slower on this specific benchmark”.
I still hold to the belief that containers have basically become a thing due to fobia of fat binaries and the fact that most people don’t seem to even be aware that they would fix most or probably all of their dependency-related issues.
I’d go as far saying that things such as databases should be included in the binaries (unless there’s a reason for the database to be up and running when the program itself is shut down).
As in, I think the whole fear of dependencies comes from the time of bad language package manager & repository (e.g. pypi&pip, or whatever the heck people used in the dark days when pypi&pip was considered a good tool).
Counterpoint: At $WORK we use Rust for a Modbus command line tool.
The entire application is less than two hundred lines of code and has three dependencies (modbus, clap, and byteorder).
When transitive dependencies are brought in those three dependencies become ~20.
Building a release version of the application in GitLab CI takes fifteen minutes.
Building an application with four times the LOC in C using only system dependencies takes seconds.
I don’t fear dependencies because Cargo is a bad language package manager (it’s probably the best I’ve ever used).
I fear dependencies because every single dependency could lead to dramatic compilation-time-bloat.
Fifteen minutes?!? Is that really all compilation time? How long does it take locally? ripgrep has ~70 dependencies and a release build compiles in under a minute for me. Maybe two minutes when I use my laptop from 2012.
It’s compilation time plus pull-in-dependencies-from-cargo time.
It may also include downloading the rust tool-chain, but I’m not longer at my work machine so I don’t actually have the .gitlab-ci.yml or Dockerfile in front of me to check that.
I did a comparison in this comment running builds of both programs on my local machine.
The C program took 0m0.137s locally, but takes an order of magnitude longer in the CI job.
The Rust program took 0m36.862s locally, so presumably there is a similar scale factor for the Rust application’s CI job.
That sounds unrealistically long, how long does it take you to build locally if all the caches are in place and you just modify a few files ?
If it take 15 minutes to build from scratch, then that’s irrelevant in of itself, because you don’t need to run your ci build like a madman… Unless you can’t test locally, but then the problem is that you can’t test locally and you are using your CI as a dev testing tool.
$ time make clean all
# build output fluff...
# ...
# ...
real 0m0.137s
user 0m0.094s
sys 0m0.044s
$ cloc src/
4 text files.
4 unique files.
0 files ignored.
github.com/AlDanial/cloc v 1.74 T=0.01 s (397.6 files/s, 91740.2 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C 3 171 109 614
C/C++ Header 1 11 0 18
-------------------------------------------------------------------------------
SUM: 4 182 109 632
-------------------------------------------------------------------------------
With changes made to one file both the Rust and C applications take less than 0.2 sec.
I timed it at 0m0.205s real for the Rust application and 0m0.132s real for the C application, but I think both of those are low enough that I would consider their perceived compile time to basically be zero.
If it take 15 minutes to build from scratch, then that’s irrelevant in of itself, because you don’t need to run your ci build like a madman.
It is relevant to me.
If a CI job needs to run every time a change is getting merged into our development branch then I care a lot about how long it takes to run CI with those changes in place.
I don’t want to have to wait 15 min to fully context switch to another task.
I do not mean to sound ungrateful for the many quality open source libraries that allow us to write software without reinventing the wheel for every project; I just think that we should acknowledge that a dependency-bloat problem exists in modern software culture. If we do decide that the trade-offs involved in taking on that bloat end up being ultimately worth it then great, but I don’t want to pretend that the problem isn’t there.
Edit: Please don’t take this unscientific test as a comparison of the C and Rust programming languages. These applications do completely different things and were merely chosen because that had small (< 1000) lines of code and happened to show a difference between production applications that used a zero and non-zero number of dependencies outside of each language’s standard library.
Hmh, fair enough, the way I see it time spent in CI is basically nothing off my back. As in, I expect CI to always pass since I test locally, the few situations where it doesn’t are the exception.
If for some reason that’s not the case with your project though, you could get around it with having an OS image for your CI machine that just includes the cached rust libraries… I mean, that’s why C compilation is fast to some extent, because you link against already-compiled libraries… It just so happens that all standard OSes come with those precompiled libraries.
The simple solution for this though is just having a dedicated build or build+test machine, and everything gets cached there and stays there.
There’s a small overhead either way, but that overhead is unrelated to fat Vs dynamicslly linked binaries or rust Vs c.
Ya on the topic of Rust vs C I added an edit at the end of my comment above to make it clear that the test is not meant to be a language comparison.
If for some reason that’s not the case with your project though, you could get around it with having an OS image for your CI machine that just includes the cached rust libraries… I mean, that’s why C compilation is fast to some extent, because you link against already-compiled libraries… It just so happens that all standard OSes come with those precompiled libraries.
You kind of hit the nail on the head. For C/C++ it’s a lot easier because those libs come compiled and pre-packaged in most distros (basically what the article mentioned). And for the case at $WORK that often ends up being a major win.
If we can apt-get everything we would ever need in 30 seconds and not have to worry about caching or keeping machines up to date then that provides a substantial benefit to my team. I think you should always use the right tool for the job, and that tool is definitely not always going to be C or C++, but it certainly is nice not needing to worry about caching dependencies on a dedicated machine.
The complexity issues you have encountered can be called insane dependencies and as you have discovered, they are usually transitive dependencies (if you look at direct dependencies in apt show rviz it looks quite sane). This problem is mainly caused by lack of modularity and flawed architecture.
the graphs are huge, especially the one with build dependencies. Just the image sizes are intimidating: 6172×10741 and 10848×19358.
There are solutions for this, but I would rather write an article on this than a long comment…
However this is not specific to any programming language or build system and it does not mean that OS-level package system (like deb, rpm, guix etc.) are bad (indeed they are good).
I wonder how much security review maintainers actually do. Reviews are difficult and incredibly boring. I have reviewed a bunch of crates for cargo-crev, and it was very tempting to gloss over the code and conclude “LGTM”.
Especially in traditional distros that bundle tons of C code, a backdoor doesn’t have to look like
if (user == "hax0r") RUN_BACKDOOR(). It could be something super subtle like an integer overflow when calculating lengths of buffers. I don’t expect a volunteer maintainer looking at a diff to spot something like that.Reminded me of this attempted backdoor: https://freedom-to-tinker.com/2013/10/09/the-linux-backdoor-attempt-of-2003/
I assume that as long as we keep insisting on designing complex software in languages that can introduce vulnerabilities, then such software will contain such vulnerabilities (introduced intentionally or unintentionally).
I think integer overflow is a great example. In C & C++ unsigned integers wrap on overflow and signed integers exhibit UB on overflow. Rust could have fixed this, but overflow is unchecked by default in release builds for both signed and unsigned integers!
Zig doubles down on the UB by making signed and unsigned integer overflow both undefined unless you use the overflow-specific operators! How is it that none of these languages Rust doesn’t handle overflow safely by default?!(edit: My information on Zig was inaccurate and false, it actually does have checks built into the type-system (error system?) of the language!)Are the performance gains really worth letting every addition operation be a potential source of uncaught bugs and vulnerabilities? I certainly don’t think so.
Rust did fix it. In release builds, overflow is not UB. The current situation is that overflow panics in debug mode but wraps in release mode. However, it is possible for implementations to panic on overflow in release mode. See: https://github.com/rust-lang/rfcs/blob/master/text/0560-integer-overflow.md
Obviously, there is a reason for this nuanced stance. Because overflow checks presumably inhibit performance or other optimizations in the surrounding code.
To clarify what I was saying: I consider overflow and UB to both be unacceptable behavior by my standard of safety. Saying that it is an error to have integer overflow or underflow, and then not enforce that error by default for all projects feels similar to how C has an
assertstatement that is only enabled for debug builds (well really just whenNDEBUGisn’t defined). So far the only language that I have seen which can perform these error checks at compile time (without using something likeResult) is ATS, but that language is a bit beyond my abilities.If I remember correctly, some measurements were taken and it was found that always enabling arithmetic checks in the Rust compiler lead to something like a 5% performance decrease overall. The Rust team decided that this hit was not worth it, especially since Rust is aiming for performance parity with C++. I respect the team’s decision, but it does not align with the ideals that I would strive for in a safety-first language (although Rust’s primary goal is memory safety not everything-under-the-sun safety).
That’s fine to consider it unacceptable, but it sounded like you thought overflow was UB in Rust based on your comment. And even then, you might consider it unacceptable, but surely the lack of UB is an improvement. Of course, you can opt into checked arithmetic, but it’s quite a bit more verbose.
But yes, it seems like you understand the issue and the trade off here. I guess I’m just perplexed by your original comment where you act surprised at how Rust arrived at its current state. From what I recall, the intent was always to turn overflow checks on in release mode once technology erased the performance difference. (Or rather, at least that was the hope as I remember it at the time the RFC was written.)
Oh no I actually am glad that at least Rust has it defined. It is more like Ada in that way, where it is something that you can build static analysis tools around instead of the UB-optimization-wild-west situation like in C/C++.
Yes I believe that was in the original RFC, or at least it was discussed a bunch in the GitHub comments, but “once technology erased the performance difference” is different than “right now in shipping code after the language’s 1.0 release”. I would say it is less of surprise that I feel and more a grumpy frustration - one of the reasons I picked up (or at least tried to pick up RIP) Rust in the first place was because I wanted a more fault-tolerant systems programming language as my daily driver (I am primarily a C developer). But I remember being severely disappointed when I learned that overflow checks were disabled by default in release builds because a that ends up being only marginally better than everyone using
-fwrapvin GCC/Clang. I like having the option to enable the checks myself, but I just wish it was universal, because that would eliminate a whole class of errors from the mental model of a program (by default).:(
With the cost, it depends. Even though overflow checks are cheap themselves, they have a cost of preventing autovectorization and other optimizations that combine or reorder operations, because abort/exception is an observable side effect. Branches of the checks are trivially predictable, but they crowd out other branches out of branch predictors, reducing their overall efficiency.
OTOH overflows aren’t that dangerous in Rust. Rust doesn’t deal with bare
malloc+memcpy. Array access is checked, so you may get a panic or a higher-level domain-specific problem, but not the usual buffer overflow. Rust doesn’t implicitly cast to a 32-bitintanywhere, so most of the time you have to overflow 64-bit values.The fact that both signed and unsigned integer overflow is (detectable) UB in Zig actually makes zig’s –release-safe build mode safer than Rust’s –release with regards to integer overflow. (Note that Rust is much safer however with regards to memory aliasing)
I stand corrected. I just tested this out and it appears that Zig forces me to check an error!
I’ll cross out the inaccuracy above…and I think I’m going to give Zig a more rigorous look… I’m quite stunned and impressed actually. Sorry for getting grumpy about something I was wrong about.
I don’t think so either. I did a benchmark of overflow detection in expressions and it wasn’t that much of a time overhead, but it would bloat the code a bit, as you have to not only check after each instruction, but only use instructions that set the overflow bit.
The problem with this post is that when people complain about Rust programmes having too many dependencies they’re not actually complaining about them having too many dependencies, they’re complaining about them taking too long to compile and having too many untrusted dependencies.
The compile time issue has been discussed loads and discussing it here again probably wouldn’t be very productive, but it’s still worth pointing one thing out: it wouldn’t be nearly as much of an issue if you didn’t also have to build all the dependencies as well. Even quite big programmes, in my experience, don’t take very long to build. What takes forever is building them and all their recursive dependencies every time I update a programme, even when the dependencies haven’t changed.
There’s also the factor of keeping everything synced up. Programmes shouldn’t be using whatever the version of their dependencies were out last time the programme was updated by its authors. They should be using the latest version that actually exists, for obvious security reasons.
But the much more important factor is trust.
Now you can question how effective e.g. Debian is at evaluating new additions to its repositories and changes to the packages in its repositories but Debian packages are more carefully curated than Cargo packages. Packages on crates.io can be uploaded by anyone, and basically the only thing most people use to evaluate them is whether they look suspicious. If someone has good documentation and lots of downloads, most people - certainly I - just assume that they must be fairly trustworthy. That’s a much lower bar than the bar to get into Debian. The reason most people avoid any dependencies that aren’t in their system package manager is that downloading random crap from the internet does and should feel scary. Whenever I have to set up a Windows computer it feels so strange just downloading programs from random websites. For many programs the standard way to download them seems to be from Softpedia or one of those similar highly suspicious ad-supported software repositories.
Recently I installed
spotifydthrough the Arch User Repositories. Not very trustworthy, using the AUR, you might say, butyayshows you thePKGBUILDfor the package you’re installing and it looked fine to me assuming that Spotifyd was fine.Now the
spotifydcrate has 27 direct dependencies. Of those, some are standard crates across the whole Rust ecosystem and are maintained by well-known members of the Rust community or the Rust team themselves likelazy_static,failure,futures,libc,log,serde,tokio-core,tokio-io,tokio-signal,url,percent-encoding, andenv_logger. It seems fairly safe to trust them. However it’s still concerning to me that many are pre-1.0.tokio-*andfuturesare understandable, butlibc? In fact, Alex Crichton just (as I’m typing this) posted an issue to thelibcpage on GitHub saying it has no active maintainers. This is a repository hosted under therust-langorganisation on GitHub, which you’d think would mean it’d be officially supported by the Rust team, which contains a LOT of code, and which has >23m downloads.Take
chrono(0.4) for example, a library with a huge number of downloads. Its author, responding over two years ago to an issue on GitHub, lists many backwards-incompatible API changes he’d like to make before releasing the library as 1.0. That’s fine, that’s the whole point of releasing it as v0.4 as he is currently doing, but why is it considered okay for this library to have 9m downloads on crates.io and many many reverse dependencies that could all need to rewrite swathes of code in future?daemonize(0.4) hasn’t had changes on Git for nearly a year.fern(0.5) hasn’t seen changes for four months.gethostnameis a crate for getting the computer’s hostname that seems considerably complicated for what would in C be one line for each platform with some#ifdefs etc. It also lets you set the system hostname for some reason, despite the name.hexis another crate that is version 0.4. Seems to be a pretty common problem in Rust: someone works on something for a bit then finds something else to do, but nothing stops you from using these packages in your software. It’s only got 60 commits in the repository’s history, which seems like not many, but it’s only very simple functionality. I’d suggest this should be in the standard library. Its code is ‘mostly extracted from rustc-serialize’, which suggests that the author’s suggestion thathighly amusing: it seems that the same thing happens in Rust anyway because, well, for small stuff it’s easier to just copy-paste or rewrite code than to use a big complicated library in Rust as well.
I’m not going to go through the rest of its dependencies because I think the point has been made: peoples’ concerns about the Cargo/npm approach to dependencies are quite well-founded. Quite reasonable programmes have huge numbers of recursive dependencies (Spotifyd required >400 dependent crates when I built it) that need to be rebuilt every time they’re updated, and the dependencies are untrusted, unverified, unstable and are not automatically kept up-to-date by my system package manager.
I agree with quite a bit of your post, but this struck me as the polar opposite of security.
Maintainers get hacked. I’d rather have the version I’ve audited the source of than some random newer one.
The likelihood that you’ve successfully discovered all security flaws in a library through auditing are nearly zero. Your only defense against the flaws someone else will find first is being able to rapidly and safely upgrade to a version that fixes them.
For context… my employer very much cares about the security of our software. A lot of very smart people – and I count myself among them in this narrow regard – have weighed the alternatives, including auditing an unchanging pool of trusted code, and concluded that agile and frequent updates to the newest available version of our dependencies is the least risky approach to software security.
[Comment from banned user removed]
Realistically, distro packages are quite a bit less audited than people expect.
Remember when people were upset that Chromium in Debian downloaded the “OK Google” recognition blob when they assumed Debian had vetted all of Chromium?
Debian isn’t perfect at auditing packages but it’s much better than no auditing at all which is the reality for crates.io.
Excellent counterpoints, thank you! Can I include your comment on that article as an appendix?
The trust problem is indeed part of the real problem (the other being robustness, which is related), and compile times are related because they force people to notice and think about what’s going on. There’s a few different approaches to trying to deal with trust, I think the biggest ones are
crevandcargo-audit, but it’s a bit of an unsolved problem how well these things are actually going to work in the long run. So I suppose the real question is not how people use the code that makes their external libraries, and what tools they use, but rather how that code is managed and distributed – who is responsible for it. Debian can include code whose maintainer has been dead for 20 years because they have maintainers of their own, a security team, and a mechanism to consistently apply patches to the code if necessary. If a package in Debian becomes orphaned it can still be maintained by other people without requiring a fork just to be able to publish oncrates.io(which I have had to do ), and if it remains orphaned then it can be removed from the next version of Debian and the rest of the system will adjust around that fact.There’s room for counterpoints in all your examples, and you raise a number of real interesting questions: Is four months, or a year, actually that long for no changes in a library if that library performs its purpose and has no known security problems? There’s plenty of C code out there with no version number at all – how does one manage that? If a problem is discovered, who should fix it and how does that fix get communicated? What should and should not be in the standard library? Like I said at the end, these are big problems that don’t have a single solution.
Thanks! I think it would be better to just link the lobsters thread and say something like ‘there was some interesting discussion here about this’. I don’t think my comment deserves to be highlighted over the other wonderful comments here.
I think this is a key point. Ultimately, there’s nothing inherent in the compilation model forcing it to be this way and there’s nothing inherent in dpkg forcing Debian to be well maintained and audited. And in fact I’m sure there are lots of bits of Debian that aren’t very well audited.
Maybe what crates.io needs is something akin to Arch’s distinction between core, extra, community and the AUR. Packages could be marked explicitly as being core (developed, maintained and supported by the Rust team, basically the ‘extended standard library’), extra (endorsed by the Rust team but not actively developed by them, closely audited for changes, I think this is what rust-lang-nursery is meant to be, roughly?), community (crates that are supported by and audited by a group of ‘trusted’ people, which would be a fairly broad group of people, much like the Arch ‘trusted users’ group, the idea being that they’re at least notionally endorsed by a group of at least notionally trusted people, but you should still be careful, but you’ll probably be fine), and other (all other crates).
Joe Bloggs creates a crate he thinks is useful. He puts it on the AUR. It goes into the ‘other’ category. Joe posts it to /r/rust-lang and people really like the design or the API or the functionality or whatever, but people have some concerns that it’s not super idiomatic. Joe fixes up most of these issues and some trusted community member adopts it into the ‘community’ category after a little while, and maybe Joe becomes a trusted community member himself after a little while.
Eventually this crate becomes so widely used that it’s considered not just a popular third party library but a foundational component of the ecosystem. It gets imported into the ‘extra’ category and now it’s guaranteed regular fuzzing, some oversight over changes, reasonably close auditing from the community and the Rust developers, etc.
Maybe eventually it becomes so core to the ecosystem that the Rust team adopts it as a ‘core’ crate, meaning it’s now actively supported and developed by the Rust team.
Or maybe it stops at any one of those stages. Being in ‘other’ doesn’t mean it’s bad, just not part of the standard ecosystem. People would be taught to be very wary of adding packages in ‘other’ without closely auditing them, because they are totally untrusted.
Obviously for something to be imported into ‘X’ its dependencies would all need to be ‘X’ or higher.
Thing is, Rust already has the beginnings of this. It does have crates that are developed by the core team. It does have blessed crates not developed by the core team but acknowledged as being used in almost every large project and important to the health of the ecosystem that are regularly fuzzed. It does have a broad set of widely used libraries that are considered trustworthy. It just needs these categories to be explicitly marked on the packages themselves and tooling for seeing how trustworthy the packages you are using actually are.
I can’t help thinking there’s a problem in your argument, though.
Your concern, if I’ve understood it, is that people are only trusting, say, a crate because a bunch of other people apparently have trusted it, too, and, presumably, those people haven’t had bad things happen as a result, or they’d have complained in places you could see. But then you seem to ascribe higher value to Linux-distro package repositories, when the only argument that can be made in support of trusting the distro’s packages is that… a bunch of other people apparently have trusted them, too, and presumably haven’t had bad things happen as a result.
I’m not sure there’s a way to qualitatively distinguish between “Debian’s packages are trustworthy because lots of people say so” and “crates.io is trustworthy because lots of people say so”. And both groups certainly face negative repercussions if the packages turn out not to be trustworthy, because both groups have staked at least some of their reputation on the claim of trustworthiness.
And keep in mind that mere presence of security issues doesn’t seem to impact the trustworthiness of, say, the Debian maintainers. Debian routinely updates packages to fix CVEs, for example, and nobody seems to think it reflects poorly on Debian that those vulnerable packages were in Debian’s repositories. In fact, quite the opposite: we explicitly don’t expect Debian package maintainers to be doing comprehensive security audits that might catch those problems up-front.
Merely restricting the number of people who can upload/add a package also doesn’t do it, because (for example) restrictive access policies occur in both good and bad projects, and we all know that.
Or if your argument is “I personally trust the Debian maintainers more than I trust the crate maintainers”, that’s an argument that’s persuasive to you personally, but not one that necessarily is persuasive to others, unless others are open to being persuaded on the basis of “other people trust them”, which then lands you back at the start.
Why do you need to rewrite? Just pin your dep to 0.4 and be done with it. If 1.0 is too much work, don’t update. I can’t see this as a problem to be honest. Seems at best a nitpick about changes happen, not rust specific. Look at openssl breaking changes between versions and a lot more stuff depends on that.
I see this the opposite way: the software is DONE and doesn’t need to change which is a good thing. I find the incessant view that software needs to be constantly coddled/babied and updated an antipattern. Do you see that something needs to change in that library? Part of having so many dependencies should mean things get stable and essentially need no more updates or at best few updates. I want libraries to be stable and not get updated. Its not like if we had the equivalent of abs() in a library we would think it needs to be updated every few months. Once its done its done barring syntax changes or other changes to the overall language.
As a nix user I’d say this reflects more on your package manager of choice than rust itself. None of this need be an issue its more an issue in historically how package management has viewed dependencies.
If the software were done it would be 1.0 and stable. It isn’t done. It still has serious issues that need to be fixed, which is why the authors want to change the API for 1.0. That’s kind of the point of 0.x releases, they’re meant to be experimental unstable releases that can’t be relied upon.
If a crate reaches stable at 0.X.Y unexpectedly, why force evaluation of a 1.0 upgrade if there are no breaking changes to be made for it? Sure, let the next update be 1.0, but there’s no immediate reason to move, especially when you can say this in the crate’s top-level documentation and update that with just a 0.X.(Y+1) minor bump.
Because 1.0 is literally defined to mean stable for a Rust crate. That’s the definition of stable. They’re the same thing. 1.0+ = stable. They are synonyms.
The Semver FAQ actually addresses this in a fairly nice way.
Also true! I’d forgotten about that bit.
Lack of updates for transitive deps seems like merely a lack of integration with the package manager. There could be something that watches the Cargo index, rebuilds affected packages and releases updates for them. I know package managers prefer unbundling and dynamic linking instead, but when dependencies rely on using generics and inlining, there’s no easy way around that.
I don’t understand the issue people have with dependencies per say.
As in, I think the whole fear of dependencies comes from the time of bad language package manager & repository (e.g. pypi&pip, or whatever the heck people used in the dark days when pypi&pip was considered a good tool).
I think the other part comes from understanding dependencies in the C++/C sense, where every time I added a library I hadn’t used before I’d have to spent 20 minutes figuring out what’s the best way to include it and how to make it compile.
Also, fat binaries should just become a thing. Memory and storage issues that made fat binaries not a good option are just not a thing anymore, if they are, they are no niche systems where you can just use distros specifically made for that.
As in, things like openssl should be dynamically linked, I agree with that. However the category of things which should be dynamically linked is tiny and far outweights the 99% of libraries which could just be static. Never has one uttered the words “Oh gee, I’m sure glad this package dynamically links to lboos, otherwise it might cost me an extra 5Mb of disk space and have from the old version that is 5.6% slower on this specific benchmark”.
I still hold to the belief that containers have basically become a thing due to fobia of fat binaries and the fact that most people don’t seem to even be aware that they would fix most or probably all of their dependency-related issues.
I’d go as far saying that things such as databases should be included in the binaries (unless there’s a reason for the database to be up and running when the program itself is shut down).
Counterpoint: At
$WORKwe use Rust for a Modbus command line tool. The entire application is less than two hundred lines of code and has three dependencies (modbus, clap, and byteorder). When transitive dependencies are brought in those three dependencies become ~20. Building a release version of the application in GitLab CI takes fifteen minutes. Building an application with four times the LOC in C using only system dependencies takes seconds. I don’t fear dependencies because Cargo is a bad language package manager (it’s probably the best I’ve ever used). I fear dependencies because every single dependency could lead to dramatic compilation-time-bloat.Fifteen minutes?!? Is that really all compilation time? How long does it take locally? ripgrep has ~70 dependencies and a release build compiles in under a minute for me. Maybe two minutes when I use my laptop from 2012.
It’s compilation time plus pull-in-dependencies-from-cargo time. It may also include downloading the rust tool-chain, but I’m not longer at my work machine so I don’t actually have the
.gitlab-ci.ymlorDockerfilein front of me to check that.I did a comparison in this comment running builds of both programs on my local machine. The C program took
0m0.137slocally, but takes an order of magnitude longer in the CI job. The Rust program took0m36.862slocally, so presumably there is a similar scale factor for the Rust application’s CI job.That sounds unrealistically long, how long does it take you to build locally if all the caches are in place and you just modify a few files ?
If it take 15 minutes to build from scratch, then that’s irrelevant in of itself, because you don’t need to run your ci build like a madman… Unless you can’t test locally, but then the problem is that you can’t test locally and you are using your CI as a dev testing tool.
Building the two examples above locally from scratch:
Rust tool (3 primary dependencies):
C tool (0 primary dependencies):
With changes made to one file both the Rust and C applications take less than 0.2 sec. I
timed it at0m0.205s realfor the Rust application and0m0.132s realfor the C application, but I think both of those are low enough that I would consider their perceived compile time to basically be zero.It is relevant to me. If a CI job needs to run every time a change is getting merged into our development branch then I care a lot about how long it takes to run CI with those changes in place. I don’t want to have to wait 15 min to fully context switch to another task.
I do not mean to sound ungrateful for the many quality open source libraries that allow us to write software without reinventing the wheel for every project; I just think that we should acknowledge that a dependency-bloat problem exists in modern software culture. If we do decide that the trade-offs involved in taking on that bloat end up being ultimately worth it then great, but I don’t want to pretend that the problem isn’t there.
Edit: Please don’t take this unscientific test as a comparison of the C and Rust programming languages. These applications do completely different things and were merely chosen because that had small (< 1000) lines of code and happened to show a difference between production applications that used a zero and non-zero number of dependencies outside of each language’s standard library.
Hmh, fair enough, the way I see it time spent in CI is basically nothing off my back. As in, I expect CI to always pass since I test locally, the few situations where it doesn’t are the exception.
If for some reason that’s not the case with your project though, you could get around it with having an OS image for your CI machine that just includes the cached rust libraries… I mean, that’s why C compilation is fast to some extent, because you link against already-compiled libraries… It just so happens that all standard OSes come with those precompiled libraries.
The simple solution for this though is just having a dedicated build or build+test machine, and everything gets cached there and stays there.
There’s a small overhead either way, but that overhead is unrelated to fat Vs dynamicslly linked binaries or rust Vs c.
Ya on the topic of Rust vs C I added an edit at the end of my comment above to make it clear that the test is not meant to be a language comparison.
You kind of hit the nail on the head. For C/C++ it’s a lot easier because those libs come compiled and pre-packaged in most distros (basically what the article mentioned). And for the case at
$WORKthat often ends up being a major win. If we canapt-geteverything we would ever need in 30 seconds and not have to worry about caching or keeping machines up to date then that provides a substantial benefit to my team. I think you should always use the right tool for the job, and that tool is definitely not always going to be C or C++, but it certainly is nice not needing to worry about caching dependencies on a dedicated machine.The complexity issues you have encountered can be called insane dependencies and as you have discovered, they are usually transitive dependencies (if you look at direct dependencies in
apt show rvizit looks quite sane). This problem is mainly caused by lack of modularity and flawed architecture.Try also this:
the graphs are huge, especially the one with build dependencies. Just the image sizes are intimidating: 6172×10741 and 10848×19358.
There are solutions for this, but I would rather write an article on this than a long comment…
However this is not specific to any programming language or build system and it does not mean that OS-level package system (like deb, rpm, guix etc.) are bad (indeed they are good).