However, I still think there is value in fuzzing compilers. Personally I find it very interesting that the
same technique on rustc, the Rust compiler, only found 8 bugs in a couple of weeks of fuzzing, and not > a single one of them was an actual segfault. I think it does say something about the nature of the ode
base, code quality, and the relative dangers of different programming languages, in case it was not
clear already. In addition, compilers (and compiler writers) should have these fuzz testing techniques
available to them, because it clearly finds bugs. Some of these bugs also point to underlying
weaknesses or to general cases where something really could go wrong in a real program. In all,
knowing about the bugs, even if they are relatively unimportant, will not hurt us.
This is a really interesting point - this kind of fuzzing gives us a test for whether the sorts of more advanced static verification that programming languages like Rust offer are actually paying off in terms of program reliability. If rustc, written in Rust, gets a “better score” when fuzzed than gcc, written in C (do they use C++?) does, that’s evidence that the work the Rust language designers put into the borrow checker and the type system and so forth was worthwhile. We can imagine similar fuzz testing for large programs in other programming languages.
It’d be interesting to know whether LLVM was also compiled with AFL’s instrumentation. Obviously any findings from GCC’s optimizers would be “expected” to be found in LLVM, not rustc.
Maybe instead compare this compiler with just the parts of rustc it was based on. That version, too. From there, there’s a difference between team size, amount of time to do reviews, and possibly talent. Those could create a big difference in bugs. However, the bugs that should always be prevented by its static types should still count given the language should prevent them.
So, I’d like to see rustc vs mrustc in a fuzzing comparison.
This is a really interesting point - this kind of fuzzing gives us a test for whether the sorts of more advanced static verification that programming languages like Rust offer are actually paying off in terms of program reliability. If rustc, written in Rust, gets a “better score” when fuzzed than gcc, written in C (do they use C++?) does, that’s evidence that the work the Rust language designers put into the borrow checker and the type system and so forth was worthwhile. We can imagine similar fuzz testing for large programs in other programming languages.
Not really - gcc and rustc are far from equivalent programs.
It’d be interesting to know whether LLVM was also compiled with AFL’s instrumentation. Obviously any findings from GCC’s optimizers would be “expected” to be found in LLVM, not rustc.
Maybe instead compare this compiler with just the parts of rustc it was based on. That version, too. From there, there’s a difference between team size, amount of time to do reviews, and possibly talent. Those could create a big difference in bugs. However, the bugs that should always be prevented by its static types should still count given the language should prevent them.
So, I’d like to see rustc vs mrustc in a fuzzing comparison.