I disagree with the comments about rewriting in Rust. Instead of creating a separate project / branch that is maintained separately for a while, they should be incrementally rewriting functions and structures in Rust. If I were looking at my project and 40-80% of CVEs were directly attributable to a tool or pattern I was using, I would hope that I have the insight to say “this tool sucks”.
Help curl authors do better
We need to make it harder to write bad C code and easier to write correct C code.
Hasn’t this been the goal for decades? How do we actually make it harder to do wrong? Rust’s answer to this question sounds infinitely more appealing than C or C++ (including with all the linters).
they should be incrementally rewriting functions and structures in Rust
I think you and the author are more in agreement than you think. He also says they are not going to rewrite it from scratch, and their “this is what happens” points 1. and 2. are exactly “incrementally rewriting functions and structures in Rust”:
Step 1 and 2 above means that over time, the total amount of executable code in curl gradually can become more and more memory-safe. This development is happening already, just not very fast.
I think it’d be better to say they could be, not “they should be.” As he notes in the post, the current developers aren’t experts on rust or the best folks to lead such development. Also:
“The rewrite-it-in-rust mantra is mostly repeated by rust fans and people who think this is an easy answer to fixing the share of security problems that are due to C mistakes. Typically, the kind who has no desire or plans to participate in said venture.”
If you aren’t volunteering to do the work, it’s best to hypothesize about what a project could do and not what it should do. If you are volunteering to do the work, then have at it and let us know how it goes following your proposal.
I think everyone agrees that memory safety is better than not, but the author highlights that (1) Rust isn’t supported on all targets and (2) they would likely need a new team of maintainers with the experience and interest to rewrite in Rust and also the dedication of the existing team. In particular, the article notes how many of the “rewrite it in Rust” proponents are themselves willing to step up to the plate.
I would say that’s a lot less realistic, curl is a huge piece of software with a large API footprint. And furthermore even less useful: curl is also a massive brand and wide spread utility, its users are unlikely to switch, especially indirect ones (e.g. users of curl because for all intents and purposes it is php’s http client).
Finally, Jose Maria Quintero’s effort on librsvg has conclusively demonstrated that you can progressively migrate a codebase in flight.
I’ve migrated gradually a few codebases, and even wrote a C-to-Rust transpiler, and I don’t think it’s worth converting codebases this way.
This is because C has its own way of architecting programs, and has its own design patterns, which are different than how a Rust-first program would be structured. When you have a C program you don’t even realize how much of it is “C-isms”, but after a gradual conversion you get a very underwhelming codebase with “code smells” from C, which is then merely a first step of a major refactoring to be a Rust-like software.
Rust’s benefits come from more than just the borrow checker. Rust has its own design patterns, many of which are unique to Rust, and often not as flexible as an arbitrary C program, so they’re tricky to retrofit into a C-shaped codebase. A good C program is not a good Rust program.
I wonder how much curl is actually used now. When I first used it, it handled a load of different URL schemes and protocols and reimplementing that was far too much effort. These days, I rarely see it used for anything other than HTTP. I’ve used libfetch instead of libcurl because it’s a tiny fraction of the size and, even then, does more than I need.
The one sentence everyone who thinks we should rewrite everything in Rust shall take away is the following:
Dedicated long-term maintainer internet transfer library teams do not grow on trees.
Starting a rewrite is easy. Maintaining a rewrite over nearly two decades so that every car/IoT/whatever vendor includes, is the hard part! It doesn’t matter which language the rewrite is in.
150 security problems through a period of over 25 years in a library that runs in some twenty billion installations? Is that a lot? I don’t know.
An interesting question. We tend to say things like “perfect security is impossible”, but is it hard? What if we aimed higher though? I think about this sometimes. Programming languages could do a lot more; for example, a statically compiled program knows every single system call that it could ever possibly make (theoretically) - so it could set up a seccomp filter for itself that is guaranteed to work. Every program having even a basic seccomp profile would be kinda huge - most programs never require something like fork/exec, for example.
And of course memory safety.
Curl is in an interesting position. It’s not like a browser, which is developed by a huge group and has to evolve rapidly. It’s mostly solving one problem, or a subset of problems (a handful of protocols/ iterations of them). This is not a bad place for C, one can make the problem a bit more tractable, which is why I think curl has had a good track record relative to other C codebases. Focusing on patterns, tooling, and testing, is enough to make a C codebase pretty robust.
I also agree that rewriting curl is unlikely to happen. If the maintainers aren’t in a position to do it, who is? You’d need a totally new project. Allowing for rust modules makes sense. Focusing on more static/dynamic analysis does as well, building up “safe” libraries internally.
In lots of online sources people repeat that when writing code with C or C++, the share of security problems due to lack of memory-safety is in the range 60-70% of the flaws. In curl, looking per the date of when we introduced the flaws (not reported date), we were never above 50% C mistakes.
I wonder how much of this is due to the fact that the curl team has a much higher level of C skill than the maintainer of the average C codebase. If you read Daniel’s blog, I can tell his attention to detail is far higher than almost anyone I know. His writing and actions over the years have allowed him to build up a level of trustworthiness that is nearly unmatched. I can only think of a handful of other people I’d trust to the same degree to write C code that handles untrusted input.
The sources of the 70% figure I’m most aware of are Microsoft: 70% of Microsoft assigned CVEs are memory-safety, and Google:
Around 70% of our high severity security bugs are memory unsafety problems
I can’t say for sure but I would have thought the folks working at Microsoft and Google would be pretty highly skilled. Not sure how they compare to Daniel and the curl developers but MS and Google don’t strike me as being hugely less skilled.
I can’t say for sure but I would have thought the folks working at Microsoft and Google would be pretty highly skilled.
The worst developers and worst development processes that I’ve encountered in my career were at Microsoft. Some of the best developers I’ve ever worked with were there as well. I don’t think you can say anything about the quality of the organisation as a whole. I’ve never worked at Google, but I’ve collaborated with Google teams and worked on open source projects with a lot of Google contributors and seen a similar spread in quality: some people consistently produced Daily WTF-worthy patches, some produced amazingly clear and maintainable code. Typically, the folks in the former category got promoted.
I can’t say for sure but I would have thought the folks working at Microsoft and Google would be pretty highly skilled.
The pool of pretty highly skilled programmers whose consistently good contributions are consistently facilitated by quality design processes is… disappointingly small, to put it mildly.
Microsoft, Google, (Facebook, Netflix, Apple…) invest considerable resources on the kind of employee branding that leads people to believe that they have pretty highly skilled contributors but no organisation that large can hire only good programmers and organise an efficient development environment. In any case, at least not with the prevalent standards in engineering management and software development education, both academic and “post-academic”/on-the-job.
As a consequence, the skill base of contributors in large companies tends to be on a pretty wide spectrum. I’m not talking about a few bad apples, I’m talking entire teams that couldn’t code their way out of a hole in the ground, either because most of the people on the team can’t code their way out of a hole in the ground or because they’re led by someone who’s been the hero of at least one NixCraft meme.
The “average” skill set, or rather the observable “aggregate” skill set, i.e. what you can see from the outside, in terms of how good their most popular products are, isn’t very representative of the skill distribution inside the company. You can have – actually, I’ve seen – some of the least able programmers working on critical, but largely invisible code, things like libraries, firmware and so on. So you end up with a lot of bad code that’s maybe not representative for the quality of the best products that company puts out, but show up in bug counts nonetheless. It makes into production not just because it’s the product of poorly-skilled (or overworked, underslept, undermotivated…) developers but also because disapointingly many engineering management processes are tuned to produce this kind of code.
I don’t know how that compares to the curl team. But in my experience the code quality in large organisations tends to follow a pretty smooth distribution. Their aggregate results in terms of metrics like CVE count isn’t representative so much for the state of the art as for the best one can do without upsetting too many people.
Not sure how they compare to Daniel and the curl developers but MS and Google don’t strike me as being hugely less skilled.
Well, personally I believe they are hugely less skilled with some exceptions, but even if they weren’t it wouldn’t matter, because skill is only part of the equation. Context matters more.
If someone wants a change to happen in curl, the maintainers can say no if they have reservations about the quality of the implementation or the long-term cost of maintainership. Even if you took the curl maintainer himself and put him in a job writing code at MS or Google he would have to contend with requirements that are usually non-negotiable; engineering at a software company is always forced to compromise to get the product out the door.
MS and Google are huge institutions which introduces a lot more variability, so it would probably be a more apples to apples comparison to pick a team within MS or Google to compare to the curl core team.
I think it’s a combination of proficiency as well as the domain. If you have a single library to maintain that handles a few protocols you can focus your time primarily on a defined, static system. This is very different from other kinds of systems where new functionality is added or removed constantly.
Older, less changing C codebases tend to not really have tons of memory safety issues discovered in them. It’s the stuff that gets messed with a lot or has a bunch of uncoordinated devs interacting with it that’s gonna be impossible to do safely.
The way I’ve seen this said previously is like “of the high severity security bugs” or similar, so it is a bit of a biased sample to begin with, since those tend to be memory safety in the first place in an application like curl (and would prolly be sql injection in other environments, etc).
this said about half the bugs could be attributable to C, then there’s a pie chart breakdown that says, of those, 42 out of 51 are out-of-bounds related problems! (I usually bring this up in the context of my own use of the D programming language, pointing out that its automatic array bounds checks eliminate those vulnerabilities at runtime, but the same is true of many programming languages, not just Rust…. but indeed C isn’t really one of them without a lot of work. D solves it by having a struct array { type* data; size_t length;} and any access into it checks the index against length first. You could do that in C but… yeah a lot of work that’d be easy to forget.)
Admittedly, C has a hard time here because you can’t write a generic struct array type like there is in D. So instead they added an additional length parameter to those functions, making them harder to use (and prone to errors and even new vulnerabilities, in some cases - see the report).
C++ on the other hand can totally express that type, and they call it std::span. However, I think they dropped the ball on this one entirely because it still doesn’t do bounds checks, giving you UB on out-of-bounds access instead.
I note that the part of the blog post on which this thread of comments is commenting says not that curl has fewer memory safety vulnerabilities in absolute numbers per LOC than Google and Microsoft but that curl has a larger proportion of other vulnerabilities. What this makes me wonder is whether implementing Internet protocols is inherently a domain in which (non-memory-safety) logic errors are more likely to incur (non-memory-safety) vulnerabilities than in the average of everything that Google’s and Microsoft’s codebases do. I think that sounds plausible, but I don’t know enough to be sure.
In lots of online sources people repeat that when writing code with C or C++, the share of security problems due to lack of memory-safety is in the range 60-70% of the flaws. In curl, looking per the date of when we introduced the flaws (not reported date), we were never above 50% C mistakes.
This likely means that there are more bugs lurking in curl that haven’t been discovered; it doesn’t mean that curl necessarily has a lower defect rate.
That does not logically follow. It may be that other projects have a higher unknown defect rate than the battle tested curl. Or perhaps the oft quoted number is an assumption, not ever measured. Or that deliberate software development practices can deliver a C error security impacting rate as low as under 50%.
I disagree with the comments about rewriting in Rust. Instead of creating a separate project / branch that is maintained separately for a while, they should be incrementally rewriting functions and structures in Rust. If I were looking at my project and 40-80% of CVEs were directly attributable to a tool or pattern I was using, I would hope that I have the insight to say “this tool sucks”.
Hasn’t this been the goal for decades? How do we actually make it harder to do wrong? Rust’s answer to this question sounds infinitely more appealing than C or C++ (including with all the linters).
I think you and the author are more in agreement than you think. He also says they are not going to rewrite it from scratch, and their “this is what happens” points 1. and 2. are exactly “incrementally rewriting functions and structures in Rust”:
I think it’d be better to say they could be, not “they should be.” As he notes in the post, the current developers aren’t experts on rust or the best folks to lead such development. Also:
“The rewrite-it-in-rust mantra is mostly repeated by rust fans and people who think this is an easy answer to fixing the share of security problems that are due to C mistakes. Typically, the kind who has no desire or plans to participate in said venture.”
If you aren’t volunteering to do the work, it’s best to hypothesize about what a project could do and not what it should do. If you are volunteering to do the work, then have at it and let us know how it goes following your proposal.
I think everyone agrees that memory safety is better than not, but the author highlights that (1) Rust isn’t supported on all targets and (2) they would likely need a new team of maintainers with the experience and interest to rewrite in Rust and also the dedication of the existing team. In particular, the article notes how many of the “rewrite it in Rust” proponents are themselves willing to step up to the plate.
I think a more realistic approach would be to write a new library in Rust, and give it a curl-compatible API.
I would say that’s a lot less realistic, curl is a huge piece of software with a large API footprint. And furthermore even less useful: curl is also a massive brand and wide spread utility, its users are unlikely to switch, especially indirect ones (e.g. users of curl because for all intents and purposes it is php’s http client).
Finally, Jose Maria Quintero’s effort on librsvg has conclusively demonstrated that you can progressively migrate a codebase in flight.
Do you mean Federico Mena Quintero? As far as I know, he led the migration effort on librsvg.
Good god, yes indeed I do, I can’t believe I mangled his name so.
I’ve migrated gradually a few codebases, and even wrote a C-to-Rust transpiler, and I don’t think it’s worth converting codebases this way.
This is because C has its own way of architecting programs, and has its own design patterns, which are different than how a Rust-first program would be structured. When you have a C program you don’t even realize how much of it is “C-isms”, but after a gradual conversion you get a very underwhelming codebase with “code smells” from C, which is then merely a first step of a major refactoring to be a Rust-like software.
Rust’s benefits come from more than just the borrow checker. Rust has its own design patterns, many of which are unique to Rust, and often not as flexible as an arbitrary C program, so they’re tricky to retrofit into a C-shaped codebase. A good C program is not a good Rust program.
I wonder how much curl is actually used now. When I first used it, it handled a load of different URL schemes and protocols and reimplementing that was far too much effort. These days, I rarely see it used for anything other than HTTP. I’ve used libfetch instead of libcurl because it’s a tiny fraction of the size and, even then, does more than I need.
The one sentence everyone who thinks we should rewrite everything in Rust shall take away is the following:
Starting a rewrite is easy. Maintaining a rewrite over nearly two decades so that every car/IoT/whatever vendor includes, is the hard part! It doesn’t matter which language the rewrite is in.
An interesting question. We tend to say things like “perfect security is impossible”, but is it hard? What if we aimed higher though? I think about this sometimes. Programming languages could do a lot more; for example, a statically compiled program knows every single system call that it could ever possibly make (theoretically) - so it could set up a seccomp filter for itself that is guaranteed to work. Every program having even a basic seccomp profile would be kinda huge - most programs never require something like fork/exec, for example.
And of course memory safety.
Curl is in an interesting position. It’s not like a browser, which is developed by a huge group and has to evolve rapidly. It’s mostly solving one problem, or a subset of problems (a handful of protocols/ iterations of them). This is not a bad place for C, one can make the problem a bit more tractable, which is why I think curl has had a good track record relative to other C codebases. Focusing on patterns, tooling, and testing, is enough to make a C codebase pretty robust.
I also agree that rewriting curl is unlikely to happen. If the maintainers aren’t in a position to do it, who is? You’d need a totally new project. Allowing for rust modules makes sense. Focusing on more static/dynamic analysis does as well, building up “safe” libraries internally.
Roc’s platforms are probably the closest thing to this that I know of (https://www.roc-lang.org/platforms).
Indeed, Roc is likely the closest language to this in terms of automating permissions and configuration based on your actual code.
Consistently one of the best blog post authors, I aspire to write his quality of readable yet detailed posts.
I wonder how much of this is due to the fact that the curl team has a much higher level of C skill than the maintainer of the average C codebase. If you read Daniel’s blog, I can tell his attention to detail is far higher than almost anyone I know. His writing and actions over the years have allowed him to build up a level of trustworthiness that is nearly unmatched. I can only think of a handful of other people I’d trust to the same degree to write C code that handles untrusted input.
The sources of the 70% figure I’m most aware of are Microsoft: 70% of Microsoft assigned CVEs are memory-safety, and Google:
I can’t say for sure but I would have thought the folks working at Microsoft and Google would be pretty highly skilled. Not sure how they compare to Daniel and the curl developers but MS and Google don’t strike me as being hugely less skilled.
The worst developers and worst development processes that I’ve encountered in my career were at Microsoft. Some of the best developers I’ve ever worked with were there as well. I don’t think you can say anything about the quality of the organisation as a whole. I’ve never worked at Google, but I’ve collaborated with Google teams and worked on open source projects with a lot of Google contributors and seen a similar spread in quality: some people consistently produced Daily WTF-worthy patches, some produced amazingly clear and maintainable code. Typically, the folks in the former category got promoted.
The pool of pretty highly skilled programmers whose consistently good contributions are consistently facilitated by quality design processes is… disappointingly small, to put it mildly.
Microsoft, Google, (Facebook, Netflix, Apple…) invest considerable resources on the kind of employee branding that leads people to believe that they have pretty highly skilled contributors but no organisation that large can hire only good programmers and organise an efficient development environment. In any case, at least not with the prevalent standards in engineering management and software development education, both academic and “post-academic”/on-the-job.
As a consequence, the skill base of contributors in large companies tends to be on a pretty wide spectrum. I’m not talking about a few bad apples, I’m talking entire teams that couldn’t code their way out of a hole in the ground, either because most of the people on the team can’t code their way out of a hole in the ground or because they’re led by someone who’s been the hero of at least one NixCraft meme.
The “average” skill set, or rather the observable “aggregate” skill set, i.e. what you can see from the outside, in terms of how good their most popular products are, isn’t very representative of the skill distribution inside the company. You can have – actually, I’ve seen – some of the least able programmers working on critical, but largely invisible code, things like libraries, firmware and so on. So you end up with a lot of bad code that’s maybe not representative for the quality of the best products that company puts out, but show up in bug counts nonetheless. It makes into production not just because it’s the product of poorly-skilled (or overworked, underslept, undermotivated…) developers but also because disapointingly many engineering management processes are tuned to produce this kind of code.
I don’t know how that compares to the curl team. But in my experience the code quality in large organisations tends to follow a pretty smooth distribution. Their aggregate results in terms of metrics like CVE count isn’t representative so much for the state of the art as for the best one can do without upsetting too many people.
Well, personally I believe they are hugely less skilled with some exceptions, but even if they weren’t it wouldn’t matter, because skill is only part of the equation. Context matters more.
If someone wants a change to happen in curl, the maintainers can say no if they have reservations about the quality of the implementation or the long-term cost of maintainership. Even if you took the curl maintainer himself and put him in a job writing code at MS or Google he would have to contend with requirements that are usually non-negotiable; engineering at a software company is always forced to compromise to get the product out the door.
MS and Google are huge institutions which introduces a lot more variability, so it would probably be a more apples to apples comparison to pick a team within MS or Google to compare to the curl core team.
I think it’s a combination of proficiency as well as the domain. If you have a single library to maintain that handles a few protocols you can focus your time primarily on a defined, static system. This is very different from other kinds of systems where new functionality is added or removed constantly.
Older, less changing C codebases tend to not really have tons of memory safety issues discovered in them. It’s the stuff that gets messed with a lot or has a bunch of uncoordinated devs interacting with it that’s gonna be impossible to do safely.
My totally unscientific analysis of OpenBSD’s errata also concluded that the proportion of memory-safety bugs in their code is lower than the 70% figure that gets bandied around a lot.
The way I’ve seen this said previously is like “of the high severity security bugs” or similar, so it is a bit of a biased sample to begin with, since those tend to be memory safety in the first place in an application like curl (and would prolly be sql injection in other environments, etc).
And older curl blog about this: https://daniel.haxx.se/blog/2021/03/09/half-of-curls-vulnerabilities-are-c-mistakes/
this said about half the bugs could be attributable to C, then there’s a pie chart breakdown that says, of those, 42 out of 51 are out-of-bounds related problems! (I usually bring this up in the context of my own use of the D programming language, pointing out that its automatic array bounds checks eliminate those vulnerabilities at runtime, but the same is true of many programming languages, not just Rust…. but indeed C isn’t really one of them without a lot of work. D solves it by having a struct array { type* data; size_t length;} and any access into it checks the index against length first. You could do that in C but… yeah a lot of work that’d be easy to forget.)
They did try to add bounds-checking versions of many functions to the C standard library. It didn’t go very well: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1967.htm
Admittedly, C has a hard time here because you can’t write a generic
struct arraytype like there is in D. So instead they added an additional length parameter to those functions, making them harder to use (and prone to errors and even new vulnerabilities, in some cases - see the report).C++ on the other hand can totally express that type, and they call it
std::span. However, I think they dropped the ball on this one entirely because it still doesn’t do bounds checks, giving you UB on out-of-bounds access instead.I note that the part of the blog post on which this thread of comments is commenting says not that curl has fewer memory safety vulnerabilities in absolute numbers per LOC than Google and Microsoft but that curl has a larger proportion of other vulnerabilities. What this makes me wonder is whether implementing Internet protocols is inherently a domain in which (non-memory-safety) logic errors are more likely to incur (non-memory-safety) vulnerabilities than in the average of everything that Google’s and Microsoft’s codebases do. I think that sounds plausible, but I don’t know enough to be sure.
This likely means that there are more bugs lurking in
curlthat haven’t been discovered; it doesn’t mean thatcurlnecessarily has a lower defect rate.That does not logically follow. It may be that other projects have a higher unknown defect rate than the battle tested curl. Or perhaps the oft quoted number is an assumption, not ever measured. Or that deliberate software development practices can deliver a C error security impacting rate as low as under 50%.