The above maches my experience: usually that’s how the two setups look like.
But I want to argue that these are accidental differences – property-based testing and fuzzing is the same technique, if implemented properly. Specifically, it is possible (and rather easy) to get all of:
coverage guided program exploration
by a state-of-the-art fuzzing engines (AFL or libfuzzer)
using highly structured data as an input (eg, the set of WASM programs which pass validation)
with support for shrinking
The trick is to take raw bytes from fuzzer and use it as a finite PRNG you feed into property-style well-typed generator. This automatically gives you shrinking – by minimizing input raw bytes, you minimize the output well-typed struct.
And I think the core idea of formulating property-based testing as generated structured outputs from unstructered inputs, getting universal shrinking for free was popularized (and probably invented?) by Python’s hypothesis: https://hypothesis.works/articles/compositional-shrinking/
A colleague was asking me about fuzzing, perhaps I should send this to them along with the first part.
Apropos: is there a good discussion of the differences between fuzzing and Property-Based Testing?
I don’t think I’ve seen one, but in lieu of something more rigorous, here’s my take. It’s a bit of a fuzzy boundary, but property testing usually:
Wheras fuzzing:
Assuming this is the same thing I think libFuzzer does try to do that:
— https://www.llvm.org/docs/LibFuzzer.html
The above maches my experience: usually that’s how the two setups look like.
But I want to argue that these are accidental differences – property-based testing and fuzzing is the same technique, if implemented properly. Specifically, it is possible (and rather easy) to get all of:
The trick is to take raw bytes from fuzzer and use it as a finite PRNG you feed into property-style well-typed generator. This automatically gives you shrinking – by minimizing input raw bytes, you minimize the output well-typed struct.
I don’t know of a good article describing this perspective abstractly. https://fitzgeraldnick.com/2020/08/24/writing-a-test-case-generator.html is a great concrete implementation.
And I think the core idea of formulating property-based testing as generated structured outputs from unstructered inputs, getting universal shrinking for free was popularized (and probably invented?) by Python’s hypothesis: https://hypothesis.works/articles/compositional-shrinking/
That’s fair. It’s pretty nebulous, really. I feel like the main difference is the perspective, or goals, rather than anything concrete.