I usually find occasion to use FsCheck, the F# variant of QuickCheck, about once a year. I can attest it is a powerful tool, well documented, and actively maintained.
The problem is to use it to full advantage takes a lot of thought. Frequently more thought than went into the feature you want to test. Deferring this extra effort feeds back on itself and so causes me to seldom use it. I have a relearning and reorientation curve every time I do use it. On the other hand every time I feel like I have gained a deeper insight into its power.
That said, it’s also useful for trivial test cases which don’t require as much effort, and you may be surprised at the problem edge cases they reveal.
One of the things I’m trying to do with Hypothesis is to make it more obvious that this is not the case by blurring the lines between property based testing and normal testing as much as possible. This might be harder to do in FsCheck, but it should still be feasible.
For example, I often recommend people just start by using property based testing as a structured fuzzer. It’s easy to do, typically reveals bugs, and is easy to extend in interesting directions as and when they occur to you.
Does anyone have experience with the rust port of quick check?
A fair number of people are using it: https://crates.io/crates/quickcheck/reverse_dependencies — I’m actually really happy that so many projects are using property based testing!
It’s a pretty faithful port of Haskell’s QuickCheck, and even shares a similar implementation strategy with Arbitrary and Testable traits. (Traits are similar in many respects to Haskell’s typeclasses.)
Just to make it clear BTW: My general impression of your work is that it is very good. I just haven’t put in enough time and don’t have enough familiarity with Rust (yet!) to properly commit to saying that in the post. I’d like to at some point.
TBH now that I’m looking at the reverse dependency list I’m just going to mark it as “Probably very good”
Awesome! We’re possibly making a rust API for SpiderMonkey’s Debugger API (the only interface is in JS right now, but Servo doesn’t want to support privileged JS) and the JS fuzzer has been incredibly helpful for catching and fixing bugs for the existing interface. My thinking is that to get the equivalent for the Rust interface, we should be using quickcheck.
The rating system isn’t clear. What makes one good, but another very good?
Smallcheck is ruled out at the start. Is there any evidence that a Quickcheck will be better under certain circumstances?
Mostly usability and completeness of implementation. It’s not a particularly exact system and would need a lot more work to give a really rigorous one.
Yes. There’s a lot you just can’t catch with exhaustive testing. Hypothesis pretty routinely finds bugs using its stateful testing that can’t manifest in fewer than, say, 8-15 steps. Even if each individual step were just a trivial yes/no (they’re usually much more complicated than that) you’re going to struggle to find those with exhaustive testing (65535 tests to run to cover everything up to 15 steps).
I’m not familiar with Hypothesis. Smallcheck vs. Quickcheck – it’s not clear that a random selection from a large set of finite values will be better generally than an increasingly larger set of all values. I am aware that some cases are more likely found using one method but not the other.
Given that Smallcheck and Quickcheck are complimentary, is it better to invest one’s limited time in one or the other? My sense is it’s better to invest in Smallcheck then Quickcheck. But I’m looking for a general study of this.
I don’t have a general study to give you I’m afraid. All I can report is that an extremely large fraction of the bugs found by Hypothesis in the wild could not have possibly been caught by exhaustive enumeration up to any reasonable depth. I couldn’t give you a number but my guesstimate is > 2/3, and of the remaining 1/3 most are of the form “You forgot this thing could be empty”.
I’m not aware of any empirical studies, but randomized generation of data has a long and fruitful track record of successful deployment and bug finding and I don’t really think the smallcheck work has anything nearly so convincing to show for it. If you do find something I’d be interested in reading it though.
What about https://github.com/pholser/junit-quickcheck for Java? I’ve used it only a tiny bit but it has felt very straightforward and user-friendly.
It fails the “supports example shrinking” requirement.
In general QuickCheck ports that don’t support shrinking are basically a dead end because they’re more frustrating to use than it’s worth, and if you make serious use of them you end up wasting a lot of time debugging their output.