1. 13
  1.  

  2. 3

    This is actually the main area I’ve applied property-based testing to. While I wasn’t working in python and so couldn’t use hypothesis, the idea is the same and it worked extremely well.

    1. 3

      That’s good to know, because I thought I invented the technique! Or, rather, I knew it was very unlikely that I was the first to invent the technique, but I’d not seen anything on the subject so had to independently reinvent it. Stopping other people from having to go through that is one of the things I’m trying to fix by having lots of documented examples of how to use property based testing.

      Do you have a link to anything public you can share?

      1. 1

        Unfortunately, I don’t think I can share what I wrote. I’ll double check, because it actually turned out quite well.

        In one case I was working with a JS method to transform an image so that a particular plane faced the camera. I used jsverify to test this by generating a solution, and then building a problem that would have that solution. Then, I’d run my method on the problem and check if it was close enough to the solution to be feasible.

        This of course requires that the problem has a unique solution or that you can enumerate all the possible solutions and have some rule for deciding whether the given one is good enough w.r.t. the one originally generated, but a lot of problems have unique solutions.

        On thinking about it, this isn’t quite the same optimization that you’re talking about in that blog post (numerical vs greedy, though both end up being approximations) but is still in the same vein I think.

        I’ve been wanting to write this sort of test for my current work (mostly greedy approximations), but I’m stymied by the fact that 1) the solution-first approach is hard here, and 2) all the code I’ve had in the past is in C++, which makes building models harder than hypothesis or jsverify.

        1. 2

          In one case I was working with a JS method to transform an image so that a particular plane faced the camera. I used jsverify to test this by generating a solution, and then building a problem that would have that solution. Then, I’d run my method on the problem and check if it was close enough to the solution to be feasible.

          Oh, that a nice trick. I don’t think it’s the same one at all, but I like it!

          the solution-first approach is hard here

          I do think you might find it useful to try the modify-based-on-a-solution approach described here: You don’t need to build a solution, you just need to be able to build inputs and them tweak them in a way with a known effect on the solution.

          I’ve had in the past is in C++, which makes building models harder than hypothesis or jsverify.

          Ah, well, I don’t have the Hypothesis C++ port ready yet I’m afraid so I can’t help you there :-)

          (This is seriously on the agenda, but I don’t have a time frame)

          1. 1

            Ah, well, I don’t have the Hypothesis C++ port ready yet I’m afraid so I can’t help you there :-) (This is seriously on the agenda, but I don’t have a time frame)

            iirc there is a decent one that uses template metaprogramming to build models, but I don’t want to mess with that :P

            In the future I’m hoping to switch languages, and existence of a good property-testing framework is something I’ll be factoring into my decision.

            1. 2

              iirc there is a decent one that uses template metaprogramming to build models, but I don’t want to mess with that :P

              I’m sortof of the opinion that if a property based testing system fails to see widespread adoption that’s probably a sign that it’s not decent (it may of course be a sign of really bad marketing, but given the amount of work required to write a good property based testing system it’s rare to have a good system with marketing that bad).

              Usually the culprit is that it’s a bit scary, or that it fails to feel language native. Most of the C++ attempts I’ve seen definitely fail to feel language native.

              Even if there were a decent one I’d still probably do a Hypothesis port. I need to port the engine to C at some point, at which point it’s easy to do a C++ wrapper, and Hypothesis is sufficiently better for testing imperative programs than classic QuickCheck that I think it would be a clear win.

              In the future I’m hoping to switch languages, and existence of a good property-testing framework is something I’ll be factoring into my decision.

              Long-term my goal is to make that no longer a factor which distinguishes languages :-)

    2. 1

      Cool! Question, though:

      There’s nothing about this approach that’s magic to Hypothesis, right? It would work more-or-less as well in Haskell using Quickcheck?
      I’m thinking of a property like

      prop_duplicating_not_worse xs = do
        xs' <- element xs
        knapsack xs >= knapsack xs'
      
      1. 2

        Does that work? Properties in Haskell QuickCheck don’t form a monad.

        That being said, regardless of the details, you should definitely be able to make something work. Hypothesis is unusually good at mixing test execution and data generation so you’ll probably get better example quality out of it, but this definitely isn’t a technique that only works in Hypothesis.

        1. 1

          Propertys don’t, but Gens do. Regardless, the code above did not work as written. I tried to make a quick and dirty port of your tests here. I succeeded in making a dirty but not-so-quick port. I had to add a lot of manual “when you fail, tell me this”, and the output and counterexamples still aren’t very good.

          How much work do you think it would be to port Hypothesis to Haskell (possibly as a layer on top of QuickCheck)?

          1. 1

            Propertys don’t, but Gens do

            Yeah, but as you just discovered that’s not really enough because trying to do it that way loses you shrinking, value reporting, etc.

            How much work do you think it would be to port Hypothesis to Haskell (possibly as a layer on top of QuickCheck)?

            Moderate but not infeasible. Figuring out the exact type signatures if you want to do it “properly” might be annoying, but if you’re happy to stick everything in the IO monad Haskell should be about as easy as any other vaguely sensible language (the languages I currently anticipate being annoying to port to are Go and C).

            I’m in the progress of getting Hypothesis ready to port to all the things. The internals are still in need of stabilizing a bit, but over the next couple months I should have at least one fully fledged port (probably Java, maybe Scala, maybe C++), at which point this question becomes much easier to answer.

            I’d not recommend trying to make it a layer on top of QuickCheck, but you might be able to make QuickCheck a layer on top of it. ;-)

            1. 1

              I’d not recommend trying to make it a layer on top of QuickCheck, but you might be able to make QuickCheck a layer on top of it. ;-)

              It’s not apparent to me why this is the case, but my understanding of Hypothesis comes entirely from the occasional posts of yours that I stumble onto.

              Is there a detailed comparison of Hypothesis and QuickCheck anywhere?

              1. 2

                It’s not apparent to me why this is the case, but my understanding of Hypothesis comes entirely from the occasional posts of yours that I stumble onto.

                Hypothesis operates under a very different set of assumptions than QuickCheck. In particular, Hypothesis never attempts to manipulate the values it generates at all - it works on an intermediate representation for how to generate those values, and performs shrinking, mutation, etc. entirely on that intermediate representation. This has a lot of really useful advantages, but means that it doesn’t fit well into QuickCheck’s assumptions on how things work - e.g. a Hypothesis strategy that generates an integer can’t actually shrink an integer you give it, it can only replay the generation in a simplified form.

                Is there a detailed comparison of Hypothesis and QuickCheck anywhere?

                There isn’t. It’s on my TODO list, but I’d really like an assist from someone who actually has more practical experience of using QuickCheck than I do - especially because my experiences with QuickCheck comes from ports tend to come from ports that lack some of the features of a more modern version of QuickCheck (partly because there is almost precisely zero documentation anywhere on the internet about the modern QuickCheck feature set)