1. 9

  2. 1

    I disagree that fuzzing is not property based testing. It seems to me that one can test properties with different granularity; “does it crash” is a very coarse-grained property while “does the permutation method return a collection with exactly the same set of items as the input” is a fairly fine-grained one. People are going to write tests for coarser- or finer-grained properties depending on how much they care and how difficult it is to evaluate the property in question, and there’s no need to shame the coarser end of the spectrum - afl may be a very blunt instrument but it has an impressive pedigree of genuine problems detected and fixed.

    1. 2

      No shame implied! I thought that was pretty clear from the fact that I was describing fuzzing as the more fundamental concept and property-based testing an application of it, and the fact that my primary recommendation for getting started with Hypothesis is still “Fuzz first, worry about properties later”.

      I’m still on the fence as to whether fuzzing should be considered a type of property based testing, but my reasons for currently leaning towards the “not” side of the fence are:

      • The skills and techniques of effective fuzzing and effective property based testing are somewhat orthogonal
      • The conceptual boundaries are much clearer if you regard property based testing as an application of fuzzing
      • Because every property-based test is implicitly also fuzzing, it seems to make more sense to think of fuzzing as a thing that happens in the course of property-based testing rather than a property-based test in its own right

      Which combined seem to suggest pretty strongly that the world makes more sense if you regard them as separate but closely intertwined things