From the previous posts about it: http/2 support was a big one, and some other issues that meant it never fully passed the test suite, and there was no or very little activity to fix that, and dragging it around as EXPERIMENTAL with no expected path to get out of that didn’t make sense:
None of that explains what the actual problems are though. Why did it never fully pass the test suite? Why couldn’t HTTP/2 support be added, are there specific issues or just “nobody did the work yet”?
Came here to say this. Also, I’m the guy who uses curl a lot on a daily basis, but somehow I totally missed the fact that Hyper was ever there. Granted, I typically don’t compile curl from source and I certainly don’t interact with libcurl, but still. I wonder how many people were simply unaware of an alternative HTTP backend being available in the first place.
That kinda supports Daniel Stenberg’s point. If there had been user demand for it, people would have stepped up and publicized it, and then people like you would have become aware of it.
Sure, but that’s kind of the issue here - the adoption of an alternative HTTP library has not been communicated enough in the first place, at least that’s my impression, which is exactly why many of us were unaware of its existence. I’m not a curl developer and I certainly don’t follow its development mailing list, but I still keep up with the project overall and I check his blog every now and then and I also follow him on Mastodon, and yet somehow I totally missed the fact that Hyper was included in the experimental capacity.
Maybe the increasing interest in software supply chains and Software Bill of Materials (SBOM) will make such alternative backends more visible? If companies need to start ticking checkboxes on compliance forms for their software, they might actively look for these alternatives written in memory-safe languages. And maybe they would even invest developer time into this.
It’s probably going to take a while for the SBOM ship to pick up speed; but barring any revolutionary events, I guess the ship is underway and will not be stopped any more.
Seems reasonable. I mean, if no one wants it and no one wants to work on it, don’t do it. Personally, 99% of curl usage for me is curl | sh and reduces to “is the host compromised” and “is tls safe”, not “is curl memory safe” so while it’d be nice to answer that last one with “yes” it’s not something I’m overly concerned with.
It’s probably used by a ton of scripts that come installed by default with your distro, some of which can run in privileged contexts. It’s not just curl|sh either, the library is used by a lot of projects because it’s so featureful and reliable. A memory safe curl would have reduced the attack surface for a lot of people, and it’s sad to see this end.
Even in the case where it’s used by scripts, how often is it talking to an untrusted resource? Sometimes but I don’t expect it to be very often for things like setting up a docker image, etc.
Having curl be memory safe would be good and cool, in case it’s not clear that I feel that way. But if no one wants to work on it and users aren’t pushing hard for it it’s not surprising that it isn’t happening.
I don’t think that’s true at all. If I curl an HTTPS site with no redirects etc I feel fine reducing my security to “the site is compromised” or “TLS is compromised”. Neither is a memory safety issue unless something else has already broken.
Except that you’re receiving and processing bytes from unknown third parties long before you have any idea if the TLS is valid. Once the TLS validates then sure, after that point, if you trust the certificate and the site (both very big ifs!) then you might treat it as trusted from that point. But there’s a lot of space for issues before that
If I do curl https://hackedsite.com/, and curl is memory safe, then at worst it can do write unintended data to stdout or a file
If I do curl https://hackedsite.com/, and curl is not, and the attacker knows how to exploit it, then they may be able to execute arbitrary code on your machine. Like plant a root kit, advanced persistent threats, etc.
I think it is better to look at it from the ecosystem perspective than a personal perspective.
Something that is exposed to arbitrary input and does a lot of parsing, which curl is a prime example of, should definitely be memory safe. (Whether it’s written in Rust is a slightly different question, but that was the motivation here)
At the very least, the command line tool should be sandboxed. I wouldn’t be surprised if it is on OpenBSD, e.g. with pledge(), but I have not heard about that on other Unixes.
After doing all the setup, and before receiving input over the network, it should drop privileges so it can only write to the requested file / descriptor.
Even if curl had most components in Rust, it should still be sandboxed, for defense in depth. It is very widely used, like OpenSSH, and we just saw what a juicy target that was. It’s easy to imagine millions of dollars of resources on the other side.
At the very least, the command line tool should be sandboxed. I wouldn’t be surprised if it is on OpenBSD, e.g. with pledge(), but I have not heard about that on other Unixes.
I was curious and had a look at OpenBSD, since I too half-expect it by now. But! I don’t see any evidence of pledge being used, at least from the outside; no patches applied in the OpenBSD ports tree, and no such call in the upstream. Somehow I’m not convinced and want to spin up a VM just to ktrace it …
Hm, if I had time I would try to find an old CVE and exploit it … the double frees and such are very reliably caught by an ASAN build IME. And then it would be interesting to know if pledge() can stop it in production before that. I guess it might be hard to turn the CVE into a real exploit, but there are probably some repros out there
I’m not an OpenBSD user but I have been meaning to give it a whirl, after the xz / SSH / systemd issue, which affected both Debian and Red Hat …
Feeling a bit tired after six straight hours of code; going to try the VM install now, just for funsies! Will report back later. I don’t expect a sneaky pledge found its way in, but I’ve been more surprised before. I would expect sanitizers to be used by default, too, given their security-over-speed posture.
It’s a very nice system. I haven’t had the sense with any other OS that I could feasibly fit the entire thing in my head, notwithstanding the rate of change (or churn, less charitably) of most others. OpenBSD feels like a rock. A really good rock. You can turn it over in your hand for days and feel like you know every nook and cranny of it, and that’s quite a nice feeling to have for something you intend to expose to the Internet.
edit: I confirm there is no pledge made by the curl pkg on OpenBSD 7.6. Fwiw, the “native” such tool in OpenBSD is called ftp(1), which does of course pledge ("stdio rpath dns tty inet" when tested with an HTTPS URL).
https://i.ibb.co/QD4HHzD/curldeps.png this is a dependency tree generated on my ubuntu with debtree --rdeps-depth=50 --condense libcurl4t64. i do not know if it’s complete but it should help convey my point.
libcurl is incredibly widely used by applications as an HTTP client. It’s the go-to HTTP library for applications written in C and a few other languages.
I guess maybe it’s not clear from my post but I was talking specifically about why my use of curl doesn’t change much whether it’s memory safe or not. I’m very aware of what curl is and what libcurl is.
The sad fact is that pretty much no software has been “working fine for decades”. Rather, curl (and pretty much all software, this isn’t special to curl) has been “sort of working for decades, but if an anonymous hacker from the internet pokes it the wrong way it gives them complete access to your system”, for decades. That’s not fine.
Curl does an unusually good job at presenting information about this, here’s a chart.
I’ll cheer on pretty much anyone working to replace C code, but it’s true there are a handful of projects (numbering in the single digits) that have demonstrated the ability to meet the extraordinarily rigorous high bar needed to successfully develop and deploy nontrivial C code, and curl is one of them. Not that they never make mistakes, but that the mistakes are relatively rare, and they own up to them and learn from them.
I would certainly not generalize from this because I don’t think it’s realistic for others to replicate the success of curl.
Dovecot. They use an incredibly stylised C, which uses a parallel data stack for returning large things by value and giving arena-like properties, which makes most lifetime issues trivial to reason about.
SQLite and Lua are the main other ones I can think of off the top of my head. I’ve heard others say qemu belongs on this list but I can’t vouch for it myself. Potentially also openssh?
Note that both SQLite and Lua have a relatively closed development process where they don’t like to accept patches from outside the maintainer team unless they are specifically bugfixes; curl is honestly the only C project I know that welcomes outside contributions but still somehow manages to maintain a very high level of code quality.
“sort of working for decades…” I think is a bit too strong. If we follow your rationale my car has been sort of working for a decade, but if someone pokes it the wrong way with a big iron spike the whole thing falls apart.
I think this points to the various way someone can define what is a software that’s “working fine”. If it accomplishes its duty for decade with few incidents, is that working fine? Sure, it is not bullet proof and we all agree that ideally it would be impervious to attacks and free of bugs but what software is?
It’s not just the car “falls apart”. It’s more like “the car explodes”. It doesn’t just not work, it damages things. Like putting the gasoline tank somewhere where it explodes during typical car crashes.
The other is that people are actively trying to attack it. The internet is a hostile and largely lawless environment, unlike a typical western city. If I’m forced to stretch the analogy to fit it’s taking that car that explodes really easily and driving it into a warzone where you are a target. It’s not fit for that purpose.
There is software that doesn’t operate in that sort of hostile environment (like say the code that generates sqlites parser from a DSL), but most software, including curl (and sqlite proper), operates in a hostile environment where security is a requirement.
Or to put it another way, “working fine” includes being able to respond to contingencies. My car isn’t “working fine” if the airbag is broken, even though I’ve never had to use the airbag. The fact that the happy path is working fine isn’t sufficient.
but what software is
Really gets at the core of what a lot of software development improvements. We haven’t achieved this. We don’t know how to make software that “works fine” at scale, and this comes at an incredibly high cost to society. Billions or trillions of dollars lost to security incidents. Many peoples lives upended. Far more of both if you count non-security bugs as well. Projects like rust are an imperfect attempt at improving the situation, but we all know that we haven’t solved it yet.
I’m fine rewriting something to be memory safe, I’d like for this to have worked as well, I’m just more concerned with the TLS layer than the HTTP layer when it comes to curl.
But the probelm is Rust is treated by some like a magic bullet when we still see memory errors in these programs. There are other options that offer similar safety but don’t slap it all over their marketing. It’s like the crowd that noticed “X written in Rust” tagline became a meme then all just switched to “Memory-safe X”.
Ada is probably the most main stream and in the same vein as Rust, and that’s ignoring things like Zig and Hare because they have slightly different safety properties. Things slightly more far afield are ATS and FStar (esp via KaraMeL), which have really interesting applications. There are a bunch of other research and other academic languages in the space as well.
I do agree that it’s limited, but it isn’t the sole operator in the space either.
and that’s ignoring things like Zig and Hare because they have slightly different safety properties.
Which is to say, strictly worse. Hare doesn’t consider memory safety a feature and is betting on default spatial memory safe APIs being enough to make up for totally not-safe temporal memory safety.
I don’t recall Zig specifically but I recall it being somewhat similar, perhaps with hardened allocator support being encouraged.
Yeah, I agree with Ada maybe (I don’t know Ada’s safety properties when you manage memory yourself). I just wanted to point out that Zig and Hare definitely aren’t memory safe in the way that one might expect when comparing to Rust.
it should provide most of the same mechanisms Rust does, in a different style. Cyclone was similar, but you would use Regions to track allocations and had various pointers with different semantics. it’s definitely a rich space.
I do agree that Zig and especially Hare aren’t quite where Rust are tho! I suspect that’s roughly ok based on what they’re aiming for.
there’s also Cyclone, which is sadly dead, but which felt decidedly more accessible to me
I find this remark interesting because a major idea of Rust’s lifetime system was to be a more accessible (not necessarily in the same sense) version of Cyclone.
Well, admittedly I’ve written much more of other languages, but I didn’t find it difficult at the time; I do know that the Cyclone folks haven’t restarted their work since Rust has come along, since they feel it has completed what they set out to do.
You can’t even use pointers in Ada without getting UB if you do something wrong. SPARK solved this recently by adding borrow checking, but that’s the same approach as Rust.
That’s the big thing that unifies most (all?) languages from before Rust: being way more restrictive. Rust’s innovation was combining safety and convenient features in one language.
I apologize, you may know this better than I, but in order to hit that UB, you have to reach for the Unchecked Conversions for Access Types, right? Like a Unconstrained Array to Pointer via Unchecked Conversions. But yes, agreed, and I believe that’s generally because Access Types were meant to be an item you reached for with other mechanisms first. To your point however, this is a trade off on restrictiveness!
I do agree tho, Rust’s Big Idea was to make things much more accessibly safe and we are better for it, but I was originally attempting to answer what else could fill the niche, not putting Rust down in anyway. Rust absolutely has done great things, but there are other safer languages out there as well. Like what the FStar people have accomplished with KaraMeL is interesting too!
Ada is not mainstream, despite being regularly mentioned when discussing Rust safety.
It ranks very low or below threshold in most usage metrics. Can you name any Ada software I might want on my laptop or server ? Any device I might buy with Ada firmware ? AFAICT, Ada has been used for some military contracts and that’s it.
For better or worse, Rust has succeded where Ada hasn’t: being a reasonable and desirable language choice for normal projects.
Well, saying “the most mainstream” was not meant to convey “popular, well used, &c.” but rather “I wasn’t picking some language that no one has heard of.”
I don’t disagree tho, Ada is not incredibly popular, but it’s one that has a community around it, people know it enough to argue about it solely of the languages I mentioned, and it certainly has enough staying power that there’s a community to support it.
Wrt Ada projects, I think the one I’m most interested in would be Ironclad currently, but I’m sure there’s an Awesome Ada list with other things there. Whilst I understand that Ada isn’t the most popular, the question was “name anything else that fills the niche,” which is why it came to mind.
Can you name any Ada software I might want on my laptop or server
I wrote and use septum on my laptop. I wrote it in my spare time, and used it at work on large software projects, but once people heard it was written in Ada there was an instant response of “Ewww! Why’d you write it in Ada?” Works just fine for me, even though I never optimized it, sifting through massive projects (10+ million lines of code).
MISRA disallows malloc() so hopefully you know exactly how much memory your program will need to work (and by extension, no flexible array members in a structure). In fact, a lot of the standard C functions are excluded from use (no use of stdio.h or time.h, the functions abort(), getenv(), bsearch() or qsort() Each function must have one return statement, so no early returns either. No recursion, even if you can prove a bounded worse case (think balanced trees). No unions, no variable argument functions. MISRA is very restrictive standard and I doubt it comes close to how rust is used.
By now I’m honestly not sure if the unreflecting fanboys or the anti-fanboys that need to bring up how bad the fanboys are at every opportunity, even if not really related to the topic at hand, are the more annoying crowd. (I.e. if you think this complaint is on-topic here, do you think Daniel took this project on because he believed it to be a magic bullet? Or maybe because he thought it was a worthwhile thing to try?)
Rust doesn’t need curl. Rust programs can use hyper and other Rust-native libraries directly. Adding a curl-with-hyper to a Rust project increases the amount of C code.
This left curl-with-hyper to a niche of users who wanted more memory safety, but not wanted is strongly enough to switch to another language or another library.
Daniel laughs every time somebody suggests rewriting curl. Curl is huge, with lots of protocols and features, and achieving compatibility and feature parity with it will take years of work.
However, for a basic task of just getting a file over HTTP(S), there are plenty of options. Maybe it’d even make sense to create a Rust library that exposes a curl-compatible C API for clients that use only basic functionality.
I prefer BSD libfetch for the simple use cases. Most of the time I might want libcurl, libfetch does everything I need. The rest of the time, libcurl is huge and definitely does what I need (plus a load of other things I might need for other use cases).
However, for a basic task of just getting a file over HTTP(S), there are plenty of options.
For sure. I think a lot of people would have an easy time ripping curl out of various places than rewriting it.
The biggest issue I see with curl is not specifying --proto=https, allowing for http redirects. I wrote a little binary that just does all the right things in Rust + sandboxes, and suddenly that problem went away in the entire repo.
To add to your point, in one of my previous jobs, we needed to implement a feature that involved receiving emails from a wide range of email servers. After dabbling with quite a number of tools, libraries and approaches, the most reliable way to receive emails for us ended up being shelling out to the curl binary…
for the 95% use case of curl -OL example.com or curl -L example.com -o out I expect that rust-curl-lite would be fine.
its probably the right answer. curl has sprawled a fair bit.
As a user I would be more interested in switching to something 100% Rust and hybrids are not very interesting to me. Probably it was a good effort to validate hyper though.
Even from the first posts about it, announcing the funding for it, it was very much framed as an experiment though. Although maybe it failed in a surprising place I guess.
The way I imagine it happened is that the curl/libcurl maintainers got asked (probably insistently) to switch to Rust. They got a grant to get started and did some good progress on the project, while also not being as proficient in Rust as they are in C. But the project never really reached feature parity, and people didn’t really care to push it over the finish line. I would be bitter too after investing time into something because people asked for it and didn’t show up.
The author of Hyper created a C API for libcurl to use, so I’m not sure how much Rust the Curl devs had to interact with. In libcurl work had to be done to allow for switching between HTTP backends.
Note that the article mentions two other experimental curl components implemented in Rust. They might also be on the way out, but I didn’t get that impression.
Kinda indifferent about the outcome (as others said, curl is one of the few projects I trust with C) - but the real bummer for me is the realization (theirs, I mean) that even if you want to rewrite a large, apparently segregated chunk of code, you still can’t let people who only use the other language (Rust) because it’s probably half as much integration work and reading the original source. So people who just know Rust and are not good with C are out - no wonder this got no traction.
I can’t imagine that for a project like this, such an effort was entertained.
I’m not sure to the extent that curl is used, but since it’s so ephemerally invoked (in my use at least) the memory safety isn’t a huge deal to me (since it’s not executing arbitrary code or anything, leaks are ok, and it’s so popular that pointer errors would surely be found quickly)
There simply were no users asking for it and there were almost no developers interested or knowledgeable enough to work on it.
I suspect this will also be the fate of similar efforts to introduce Rust into the Linux kernel, git, and other projects. It’s always going to be an uphill battle for very little user-visible reward, and unless you can replace the old code completely and fairly quickly, it’s just too expensive to maintain both branches for long.
That’s not a criticism of Rust: the same would apply to attempts to rewrite large projects from the inside out in any language. It’s just that people who want to rewrite things usually want to rewrite them in Rust.
Well, with Git, the idea seems to mostly be that Rust is a nice language, so it’d be nice to have it as an option for new things. Since Git is split into a bunch of different executables and scripts, written in C, Shell, and Perl, there’s no particular road block to implementing a new sub-command in Rust.
With Curl it was very different. It was about using an external library written in Rust. That Rust library exposed a C interface. If you wanted to use Hyper in a C project today, you could use that C interface instead of libcurl! The problem is that they never found anyone who was interested in finishing the work and maintain it. With Linux, there’s tons of people who are interested. With Git, long time Git devs were interested in the idea, just because they seem to think it’d be a good idea. For curl, it seems like most of the needed work was in C, like making libcurl’s HTTP backend support better, improving the Hyper C API, and such. So it would possibly be harder to find people who are both qualified and interested.
With all that said, do note that Curl still supports two Rust libraries that seem like they’re not going away. So I don’t think there are really any problems here specific to Rust or two using multiple languages. To issues were specific to the specific project to have Hyper as a backend.
Rust for Linux isn’t about rewriting the whole kernel in rust, it’s about writing parts of the kernel in rust. Largely (though not entirely) new parts.
It’s not a separate branch to maintain. It’s part of the main branch.
There are already user facing things written in it, with no non rust alternative (off the top of my head: the graphics driver for Apple silicon, some QR codes for kernel panics)
Projects with code in multiple languages is nothing new. There’s no reason to believe that this will be expensive to maintain.
I wanted more details about what kind of issues were in that missing 5%
From the previous posts about it: http/2 support was a big one, and some other issues that meant it never fully passed the test suite, and there was no or very little activity to fix that, and dragging it around as EXPERIMENTAL with no expected path to get out of that didn’t make sense:
None of that explains what the actual problems are though. Why did it never fully pass the test suite? Why couldn’t HTTP/2 support be added, are there specific issues or just “nobody did the work yet”?
Came here to say this. Also, I’m the guy who uses curl a lot on a daily basis, but somehow I totally missed the fact that Hyper was ever there. Granted, I typically don’t compile curl from source and I certainly don’t interact with libcurl, but still. I wonder how many people were simply unaware of an alternative HTTP backend being available in the first place.
That kinda supports Daniel Stenberg’s point. If there had been user demand for it, people would have stepped up and publicized it, and then people like you would have become aware of it.
Sure, but that’s kind of the issue here - the adoption of an alternative HTTP library has not been communicated enough in the first place, at least that’s my impression, which is exactly why many of us were unaware of its existence. I’m not a curl developer and I certainly don’t follow its development mailing list, but I still keep up with the project overall and I check his blog every now and then and I also follow him on Mastodon, and yet somehow I totally missed the fact that Hyper was included in the experimental capacity.
Maybe the increasing interest in software supply chains and Software Bill of Materials (SBOM) will make such alternative backends more visible? If companies need to start ticking checkboxes on compliance forms for their software, they might actively look for these alternatives written in memory-safe languages. And maybe they would even invest developer time into this.
It’s probably going to take a while for the SBOM ship to pick up speed; but barring any revolutionary events, I guess the ship is underway and will not be stopped any more.
The SBOM ship will drive a massive amount of paperwork for audit teams to be happy filing away.
Seems reasonable. I mean, if no one wants it and no one wants to work on it, don’t do it. Personally, 99% of curl usage for me is
curl | shand reduces to “is the host compromised” and “is tls safe”, not “is curl memory safe” so while it’d be nice to answer that last one with “yes” it’s not something I’m overly concerned with.It’s probably used by a ton of scripts that come installed by default with your distro, some of which can run in privileged contexts. It’s not just
curl|sheither, the library is used by a lot of projects because it’s so featureful and reliable. A memory safe curl would have reduced the attack surface for a lot of people, and it’s sad to see this end.Even in the case where it’s used by scripts, how often is it talking to an untrusted resource? Sometimes but I don’t expect it to be very often for things like setting up a docker image, etc.
Having curl be memory safe would be good and cool, in case it’s not clear that I feel that way. But if no one wants to work on it and users aren’t pushing hard for it it’s not surprising that it isn’t happening.
Unless you’re curling localhost you’re using it to access an untrusted resource. So, basically always
I don’t think that’s true at all. If I curl an HTTPS site with no redirects etc I feel fine reducing my security to “the site is compromised” or “TLS is compromised”. Neither is a memory safety issue unless something else has already broken.
Except that you’re receiving and processing bytes from unknown third parties long before you have any idea if the TLS is valid. Once the TLS validates then sure, after that point, if you trust the certificate and the site (both very big ifs!) then you might treat it as trusted from that point. But there’s a lot of space for issues before that
In curl? Where? Do you mean at the TLS layer? That can use rust already and wouldn’t be covered by curl anyways.
Space at the HTTP layer? I mean, some, but curl does HTTP.
The attacks could be worse though
curl https://hackedsite.com/, and curl is memory safe, then at worst it can do write unintended data to stdout or a filecurl https://hackedsite.com/, and curl is not, and the attacker knows how to exploit it, then they may be able to execute arbitrary code on your machine. Like plant a root kit, advanced persistent threats, etc.I think it is better to look at it from the ecosystem perspective than a personal perspective.
Something that is exposed to arbitrary input and does a lot of parsing, which curl is a prime example of, should definitely be memory safe. (Whether it’s written in Rust is a slightly different question, but that was the motivation here)
Yes, of course, I would like curl to be memory safe.
Yeah although now that I look at https://curl.se/ , it’s a project to aspire to:
Although when I look at a sample bug, there does seem to be a ridiculously large surface area to curl, like https://curl.se/docs/CVE-2022-42915.html
i.e. seemingly infinite combinations of features.
At the very least, the command line tool should be sandboxed. I wouldn’t be surprised if it is on OpenBSD, e.g. with pledge(), but I have not heard about that on other Unixes.
After doing all the setup, and before receiving input over the network, it should drop privileges so it can only write to the requested file / descriptor.
Even if curl had most components in Rust, it should still be sandboxed, for defense in depth. It is very widely used, like OpenSSH, and we just saw what a juicy target that was. It’s easy to imagine millions of dollars of resources on the other side.
I was curious and had a look at OpenBSD, since I too half-expect it by now. But! I don’t see any evidence of
pledgebeing used, at least from the outside; no patches applied in the OpenBSD ports tree, and no such call in the upstream. Somehow I’m not convinced and want to spin up a VM just toktraceit …Hm, if I had time I would try to find an old CVE and exploit it … the double frees and such are very reliably caught by an ASAN build IME. And then it would be interesting to know if pledge() can stop it in production before that. I guess it might be hard to turn the CVE into a real exploit, but there are probably some repros out there
I’m not an OpenBSD user but I have been meaning to give it a whirl, after the xz / SSH / systemd issue, which affected both Debian and Red Hat …
Feeling a bit tired after six straight hours of code; going to try the VM install now, just for funsies! Will report back later. I don’t expect a sneaky pledge found its way in, but I’ve been more surprised before. I would expect sanitizers to be used by default, too, given their security-over-speed posture.
It’s a very nice system. I haven’t had the sense with any other OS that I could feasibly fit the entire thing in my head, notwithstanding the rate of change (or churn, less charitably) of most others. OpenBSD feels like a rock. A really good rock. You can turn it over in your hand for days and feel like you know every nook and cranny of it, and that’s quite a nice feeling to have for something you intend to expose to the Internet.
edit: I confirm there is no
pledgemade by thecurlpkg on OpenBSD 7.6. Fwiw, the “native” such tool in OpenBSD is calledftp(1), which does of coursepledge("stdio rpath dns tty inet"when tested with an HTTPS URL).Which scripts, which distros?
https://i.ibb.co/QD4HHzD/curldeps.png this is a dependency tree generated on my ubuntu with
debtree --rdeps-depth=50 --condense libcurl4t64. i do not know if it’s complete but it should help convey my point.left = things that depend on libcurl
right = things libcurl depends on
libcurl is incredibly widely used by applications as an HTTP client. It’s the go-to HTTP library for applications written in C and a few other languages.
I guess maybe it’s not clear from my post but I was talking specifically about why my use of curl doesn’t change much whether it’s memory safe or not. I’m very aware of what curl is and what libcurl is.
Memory-safe X is like a red flag to me at this point about rewriting something that has been working fine for decades.
The sad fact is that pretty much no software has been “working fine for decades”. Rather, curl (and pretty much all software, this isn’t special to curl) has been “sort of working for decades, but if an anonymous hacker from the internet pokes it the wrong way it gives them complete access to your system”, for decades. That’s not fine.
Curl does an unusually good job at presenting information about this, here’s a chart.
I’ll cheer on pretty much anyone working to replace C code, but it’s true there are a handful of projects (numbering in the single digits) that have demonstrated the ability to meet the extraordinarily rigorous high bar needed to successfully develop and deploy nontrivial C code, and curl is one of them. Not that they never make mistakes, but that the mistakes are relatively rare, and they own up to them and learn from them.
I would certainly not generalize from this because I don’t think it’s realistic for others to replicate the success of curl.
What are some other projects in this list?
Dovecot. They use an incredibly stylised C, which uses a parallel data stack for returning large things by value and giving arena-like properties, which makes most lifetime issues trivial to reason about.
Sqlite probably
SQLite and Lua are the main other ones I can think of off the top of my head. I’ve heard others say qemu belongs on this list but I can’t vouch for it myself. Potentially also openssh?
Note that both SQLite and Lua have a relatively closed development process where they don’t like to accept patches from outside the maintainer team unless they are specifically bugfixes; curl is honestly the only C project I know that welcomes outside contributions but still somehow manages to maintain a very high level of code quality.
As a QEMU developer, no not really. We do our best but we’re totally not on the level of curl or SQLite.
Well, thanks for your honesty!
“sort of working for decades…” I think is a bit too strong. If we follow your rationale my car has been sort of working for a decade, but if someone pokes it the wrong way with a big iron spike the whole thing falls apart.
I think this points to the various way someone can define what is a software that’s “working fine”. If it accomplishes its duty for decade with few incidents, is that working fine? Sure, it is not bullet proof and we all agree that ideally it would be impervious to attacks and free of bugs but what software is?
I think that analogy has multiple issues.
It’s not just the car “falls apart”. It’s more like “the car explodes”. It doesn’t just not work, it damages things. Like putting the gasoline tank somewhere where it explodes during typical car crashes.
The other is that people are actively trying to attack it. The internet is a hostile and largely lawless environment, unlike a typical western city. If I’m forced to stretch the analogy to fit it’s taking that car that explodes really easily and driving it into a warzone where you are a target. It’s not fit for that purpose.
There is software that doesn’t operate in that sort of hostile environment (like say the code that generates sqlites parser from a DSL), but most software, including curl (and sqlite proper), operates in a hostile environment where security is a requirement.
Or to put it another way, “working fine” includes being able to respond to contingencies. My car isn’t “working fine” if the airbag is broken, even though I’ve never had to use the airbag. The fact that the happy path is working fine isn’t sufficient.
Really gets at the core of what a lot of software development improvements. We haven’t achieved this. We don’t know how to make software that “works fine” at scale, and this comes at an incredibly high cost to society. Billions or trillions of dollars lost to security incidents. Many peoples lives upended. Far more of both if you count non-security bugs as well. Projects like rust are an imperfect attempt at improving the situation, but we all know that we haven’t solved it yet.
I’m fine rewriting something to be memory safe, I’d like for this to have worked as well, I’m just more concerned with the TLS layer than the HTTP layer when it comes to curl.
Curl even supports a Rust-based TLS backend,
rustls.But the probelm is Rust is treated by some like a magic bullet when we still see memory errors in these programs. There are other options that offer similar safety but don’t slap it all over their marketing. It’s like the crowd that noticed “X written in Rust” tagline became a meme then all just switched to “Memory-safe X”.
Name literally anything that fills Rust’s niche.
Ada is probably the most main stream and in the same vein as Rust, and that’s ignoring things like Zig and Hare because they have slightly different safety properties. Things slightly more far afield are ATS and FStar (esp via KaraMeL), which have really interesting applications. There are a bunch of other research and other academic languages in the space as well.
I do agree that it’s limited, but it isn’t the sole operator in the space either.
Which is to say, strictly worse. Hare doesn’t consider memory safety a feature and is betting on default spatial memory safe APIs being enough to make up for totally not-safe temporal memory safety.
I don’t recall Zig specifically but I recall it being somewhat similar, perhaps with hardened allocator support being encouraged.
Well, and also to avoid comments that aren’t about Ada, ATS, or the like haha.
But yes, they don’t have the exact same properties; Zig is closest, but certainly not at the same level.
Honestly, there’s also Cyclone, which is sadly dead, but which felt decidedly more accessible to me. Alas, we left that direction.
Yeah, I agree with Ada maybe (I don’t know Ada’s safety properties when you manage memory yourself). I just wanted to point out that Zig and Hare definitely aren’t memory safe in the way that one might expect when comparing to Rust.
Generally, and I’m not a hardcore Ada user, you:
it should provide most of the same mechanisms Rust does, in a different style. Cyclone was similar, but you would use Regions to track allocations and had various pointers with different semantics. it’s definitely a rich space.
I do agree that Zig and especially Hare aren’t quite where Rust are tho! I suspect that’s roughly ok based on what they’re aiming for.
I find this remark interesting because a major idea of Rust’s lifetime system was to be a more accessible (not necessarily in the same sense) version of Cyclone.
Well, admittedly I’ve written much more of other languages, but I didn’t find it difficult at the time; I do know that the Cyclone folks haven’t restarted their work since Rust has come along, since they feel it has completed what they set out to do.
You can’t even use pointers in Ada without getting UB if you do something wrong. SPARK solved this recently by adding borrow checking, but that’s the same approach as Rust.
That’s the big thing that unifies most (all?) languages from before Rust: being way more restrictive. Rust’s innovation was combining safety and convenient features in one language.
I apologize, you may know this better than I, but in order to hit that UB, you have to reach for the Unchecked Conversions for Access Types, right? Like a Unconstrained Array to Pointer via Unchecked Conversions. But yes, agreed, and I believe that’s generally because Access Types were meant to be an item you reached for with other mechanisms first. To your point however, this is a trade off on restrictiveness!
I do agree tho, Rust’s Big Idea was to make things much more accessibly safe and we are better for it, but I was originally attempting to answer what else could fill the niche, not putting Rust down in anyway. Rust absolutely has done great things, but there are other safer languages out there as well. Like what the FStar people have accomplished with KaraMeL is interesting too!
Ada is not mainstream, despite being regularly mentioned when discussing Rust safety.
It ranks very low or below threshold in most usage metrics. Can you name any Ada software I might want on my laptop or server ? Any device I might buy with Ada firmware ? AFAICT, Ada has been used for some military contracts and that’s it.
For better or worse, Rust has succeded where Ada hasn’t: being a reasonable and desirable language choice for normal projects.
Well, saying “the most mainstream” was not meant to convey “popular, well used, &c.” but rather “I wasn’t picking some language that no one has heard of.”
I don’t disagree tho, Ada is not incredibly popular, but it’s one that has a community around it, people know it enough to argue about it solely of the languages I mentioned, and it certainly has enough staying power that there’s a community to support it.
Wrt Ada projects, I think the one I’m most interested in would be Ironclad currently, but I’m sure there’s an Awesome Ada list with other things there. Whilst I understand that Ada isn’t the most popular, the question was “name anything else that fills the niche,” which is why it came to mind.
I wrote and use septum on my laptop. I wrote it in my spare time, and used it at work on large software projects, but once people heard it was written in Ada there was an instant response of “Ewww! Why’d you write it in Ada?” Works just fine for me, even though I never optimized it, sifting through massive projects (10+ million lines of code).
Jeremy Grosser has done a lot of work with RP2040
What about the subset of C defined by MISRA and similar rules?
MISRA disallows
malloc()so hopefully you know exactly how much memory your program will need to work (and by extension, no flexible array members in a structure). In fact, a lot of the standard C functions are excluded from use (no use ofstdio.hortime.h, the functionsabort(),getenv(),bsearch()orqsort()Each function must have one return statement, so no early returns either. No recursion, even if you can prove a bounded worse case (think balanced trees). No unions, no variable argument functions. MISRA is very restrictive standard and I doubt it comes close to how rust is used.By now I’m honestly not sure if the unreflecting fanboys or the anti-fanboys that need to bring up how bad the fanboys are at every opportunity, even if not really related to the topic at hand, are the more annoying crowd. (I.e. if you think this complaint is on-topic here, do you think Daniel took this project on because he believed it to be a magic bullet? Or maybe because he thought it was a worthwhile thing to try?)
(and I say that as a primarily C++ dev)
I would worry a lot less about “the crowd”. People are idiots, it’s not really a Rust issue.
Rust doesn’t need
curl. Rust programs can usehyperand other Rust-native libraries directly. Adding a curl-with-hyper to a Rust project increases the amount of C code.This left curl-with-hyper to a niche of users who wanted more memory safety, but not wanted is strongly enough to switch to another language or another library.
This sounds like a successful experiment with a negative result! Kudos to curl for being willing to experiment, and to learning from it.
I guess this didn’t go anywhere? https://seanmonstar.com/blog/hyper-in-curl-needs-a-champion/
NixOS explored the idea, but trouble is: curl is part of the bootstrap. Adding Rust to the bootstrap complicates things. https://github.com/NixOS/nixpkgs/issues/357409
If someone wants a rusty curl, I wonder why not go all the way and do a rust http cli tool that is all rust but can work with the curl test suite.
Daniel laughs every time somebody suggests rewriting curl. Curl is huge, with lots of protocols and features, and achieving compatibility and feature parity with it will take years of work.
However, for a basic task of just getting a file over HTTP(S), there are plenty of options. Maybe it’d even make sense to create a Rust library that exposes a curl-compatible C API for clients that use only basic functionality.
Very good point. Http/s would cover most of my cases :) I always think of it as an http client, but I’m wrong here. It does so much more
I prefer BSD libfetch for the simple use cases. Most of the time I might want libcurl, libfetch does everything I need. The rest of the time, libcurl is huge and definitely does what I need (plus a load of other things I might need for other use cases).
For sure. I think a lot of people would have an easy time ripping curl out of various places than rewriting it.
The biggest issue I see with curl is not specifying
--proto=https, allowing for http redirects. I wrote a little binary that just does all the right things in Rust + sandboxes, and suddenly that problem went away in the entire repo.To add to your point, in one of my previous jobs, we needed to implement a feature that involved receiving emails from a wide range of email servers. After dabbling with quite a number of tools, libraries and approaches, the most reliable way to receive emails for us ended up being shelling out to the
curlbinary…for the 95% use case of curl -OL example.com or curl -L example.com -o out I expect that rust-curl-lite would be fine. its probably the right answer. curl has sprawled a fair bit.
As a user I would be more interested in switching to something 100% Rust and hybrids are not very interesting to me. Probably it was a good effort to validate hyper though.
Sounds a bit like they didn’t believe in the project, but maybe this is bitter hindsight
Even from the first posts about it, announcing the funding for it, it was very much framed as an experiment though. Although maybe it failed in a surprising place I guess.
The way I imagine it happened is that the curl/libcurl maintainers got asked (probably insistently) to switch to Rust. They got a grant to get started and did some good progress on the project, while also not being as proficient in Rust as they are in C. But the project never really reached feature parity, and people didn’t really care to push it over the finish line. I would be bitter too after investing time into something because people asked for it and didn’t show up.
The author of Hyper created a C API for libcurl to use, so I’m not sure how much Rust the Curl devs had to interact with. In libcurl work had to be done to allow for switching between HTTP backends.
Note that the article mentions two other experimental curl components implemented in Rust. They might also be on the way out, but I didn’t get that impression.
Absolutely, I’m not saying they don’t have a right to be bitter
Kinda indifferent about the outcome (as others said, curl is one of the few projects I trust with C) - but the real bummer for me is the realization (theirs, I mean) that even if you want to rewrite a large, apparently segregated chunk of code, you still can’t let people who only use the other language (Rust) because it’s probably half as much integration work and reading the original source. So people who just know Rust and are not good with C are out - no wonder this got no traction.
I can’t imagine that for a project like this, such an effort was entertained.
I’m not sure to the extent that curl is used, but since it’s so ephemerally invoked (in my use at least) the memory safety isn’t a huge deal to me (since it’s not executing arbitrary code or anything, leaks are ok, and it’s so popular that pointer errors would surely be found quickly)
I suspect this will also be the fate of similar efforts to introduce Rust into the Linux kernel, git, and other projects. It’s always going to be an uphill battle for very little user-visible reward, and unless you can replace the old code completely and fairly quickly, it’s just too expensive to maintain both branches for long.
That’s not a criticism of Rust: the same would apply to attempts to rewrite large projects from the inside out in any language. It’s just that people who want to rewrite things usually want to rewrite them in Rust.
Well, with Git, the idea seems to mostly be that Rust is a nice language, so it’d be nice to have it as an option for new things. Since Git is split into a bunch of different executables and scripts, written in C, Shell, and Perl, there’s no particular road block to implementing a new sub-command in Rust.
With Curl it was very different. It was about using an external library written in Rust. That Rust library exposed a C interface. If you wanted to use Hyper in a C project today, you could use that C interface instead of libcurl! The problem is that they never found anyone who was interested in finishing the work and maintain it. With Linux, there’s tons of people who are interested. With Git, long time Git devs were interested in the idea, just because they seem to think it’d be a good idea. For curl, it seems like most of the needed work was in C, like making libcurl’s HTTP backend support better, improving the Hyper C API, and such. So it would possibly be harder to find people who are both qualified and interested.
With all that said, do note that Curl still supports two Rust libraries that seem like they’re not going away. So I don’t think there are really any problems here specific to Rust or two using multiple languages. To issues were specific to the specific project to have Hyper as a backend.
Rust for Linux isn’t about rewriting the whole kernel in rust, it’s about writing parts of the kernel in rust. Largely (though not entirely) new parts.
It’s not a separate branch to maintain. It’s part of the main branch.
There are already user facing things written in it, with no non rust alternative (off the top of my head: the graphics driver for Apple silicon, some QR codes for kernel panics)
Projects with code in multiple languages is nothing new. There’s no reason to believe that this will be expensive to maintain.