1. 5
  1. 1

    Since Prof. Regehr only mentions mutational fuzzers, here is our status on starting from scratch.

    • Sequence of ASCII characters
      • Millers fuzzer
    • Sequence of words, separators, and white space
      • Lexical fuzzing – aka dictionary based fuzzers
    • Syntactically correct C program
      • We can kind of do this – see this for an approach without grammar and this for an approach where the grammar is mined from the program.
    • Type-correct C program
      • Current research
    • Statically conforming C program
      • Targeted by CSmith (and beyond)
    • Dynamically conforming C program
    • Model-conforming C program

    Although McKeeman reports finding bugs at levels 1-5, for the Csmith work we did not bother with any of levels 0-5. It’s not that we didn’t think test cases at these levels would trigger some bugs, but rather that we were mainly interested in sophisticated optimizer bugs that are very hard to spot until level 6. My guess, however, is that a high-quality modern compiler (GCC, Clang, Intel CC, MSVC, etc.) is probably not going to have a lot of bugs that you can find using levels 0-3.

    Unfortunately, it has been rather hard to convince our reviewers that ignoring parser bugs is reasonable :/.