1. 4
  1.  

  2. 10

    It’s hard to reproduce failures

    If this is the case you’re doing it wrong. When using faked data and randomised test order in RSpec it seeds the RNG with a value that is printed at the end of the tests so you can rerun with exactly the same order and faked values. No need for trial and error.

    1. -1

      That sounds like a useful feature that Faker does not have or appreciate the need for.

      1. 8
        1. 4

          This type of comment makes it sound like the article is meant as a hit job where the faker readme talks about setting a seed (as skade points out)

          1. 1

            I’ve now worked at three different companies that have used it and no one has raised or mentioned even this feature before, apologies for missing it. I think my point still stands.

      2. 7

        Mentions fuzzing but fails to mention property-based testing. :-(

        1. 1

          Ah, sure. I’m not as familiar with it but it’s certainly a solution! TBH I think of them in the same bucket.

          1. 2

            You might find this article and discussion helpful. My comment on the bottom gives brief background on where the names came from far as I could tell from my own reading that is.

            https://lobste.rs/s/p1flip/property_based_testing_is_fuzzing

            1. 2

              Thanks! The lobster comments basically say everything I want to say on the overlap.

              Depending on how one defines fuzzing and PBT, one is a subset of the other, or they’re equivalent. Or, as you say, in their original definitions they’re complementary regions in that shared space. Your POV is what I was getting at here.

              What I meant with my comment here was that I’m sad the author didn’t take the chance to evangelize the specificic techniques used within PBT, as QuickCheck is just mentioned as a generic fuzzer.

              What they’re criticizing is single-sample randomization, and fuzzing avoids that, but in PBT there’s also guided randomization, memorized critical examples and minimization, which further help reduce the overall flakiness and help that freak case become less of a freak and more of an identified edge case.

        2. 5

          And this is why things like the Hypothesis example database are such a good idea. Being able to have the random examples, but record which combinations particularly break your system mostly solves this (well, provided you either see the break cases on a dev’s machine or can fish them out of the build system)

          1. 3

            I’m currently using Alice + Bob as known users in my tests. I use Eve as a user that is malicious and is probing security boundaries, instead of just submitting wrong data.

            1. 2

              I agree with the post for testing, but use faker a lot for other use cases: for quickly filling a database with human-readable data examples for development use.