This library is nice, but I really wish there were better tools for guiding it. Generating examples can easily balloon to be the most time-consuming part of testing. Often one has to resort to hard caps (min or max, reject(), max_examples, etc) just to get the sample space down to something reasonable. IMO this weakens the library’s usefulness, since you are no longer testing as many edge cases. What I’d really like is either coverage-based fuzzing, or some other method to say “spend most of your entropy on these parameters.”
Agreed! (both on the “nice library” assertion, and on effort curating input shapes)
By pure coincidence, just yesterday, I wrapped up one increment of work to help make hypothesis integrate with fuzzers and symbolic execution tools better. (issue) There is still plenty to do, though.
My labor of love is project is a symbolic execution tool that gained (inefficient) hypothesis support late last year. The example in my writeup speaks to some of your concerns, perhaps.
The strategies in Hypothesis are such a mess, Hedgehog is much better.
Just look at the five arbitrary options that from_type will try. How am I going to know whether it will work?
Ok, let’s say that I somehow find out that from_type(typing.List[int]) is going to work. Which distribution will it have? We don’t know!
It’s the worst of both worlds:
World 1, which avoids naming things: Strategies do not document their distribution, so you can only hope it’s the right one. This is also the mistake of Arbitrary in QuickCheck. (Arbitrary instances are analogues of strategies in Hypothesis.) Hedgehog fixes this, simply by not using a lawless typeclass for this.
World 2, which names everything explicitly. In this world, I’d like to spend my mental capacity on choosing the distribution, not on calling from_type everywhere (let’s suppose it would actually work). But in Hypothesis, there is no systematized way of specifying the distribution when building the strategy. Hedgehog fixes this by having each Gen specify how it shrinks.
This library is nice, but I really wish there were better tools for guiding it. Generating examples can easily balloon to be the most time-consuming part of testing. Often one has to resort to hard caps (min or max, reject(), max_examples, etc) just to get the sample space down to something reasonable. IMO this weakens the library’s usefulness, since you are no longer testing as many edge cases. What I’d really like is either coverage-based fuzzing, or some other method to say “spend most of your entropy on these parameters.”
Example test illustrating the above problem
Agreed! (both on the “nice library” assertion, and on effort curating input shapes)
By pure coincidence, just yesterday, I wrapped up one increment of work to help make hypothesis integrate with fuzzers and symbolic execution tools better. (issue) There is still plenty to do, though.
My labor of love is project is a symbolic execution tool that gained (inefficient) hypothesis support late last year. The example in my writeup speaks to some of your concerns, perhaps.
The strategies in Hypothesis are such a mess, Hedgehog is much better.
Just look at the five arbitrary options that
from_typewill try. How am I going to know whether it will work?Ok, let’s say that I somehow find out that
from_type(typing.List[int])is going to work. Which distribution will it have? We don’t know!It’s the worst of both worlds:
World 1, which avoids naming things: Strategies do not document their distribution, so you can only hope it’s the right one. This is also the mistake of Arbitrary in QuickCheck. (Arbitrary instances are analogues of strategies in Hypothesis.) Hedgehog fixes this, simply by not using a lawless typeclass for this.
World 2, which names everything explicitly. In this world, I’d like to spend my mental capacity on choosing the distribution, not on calling
from_typeeverywhere (let’s suppose it would actually work). But in Hypothesis, there is no systematized way of specifying the distribution when building the strategy. Hedgehog fixes this by having each Gen specify how it shrinks.