1. 14

  2. 9

    I agree, but I find it annoying that it’s a “feature” in test frameworks. A “parameter” is something that a function provides, and every language has functions.

    Basically the problem is that test frameworks want to control the “main” and have a somewhat rigid view of testing.

    I just use a shell script to run the same test twice with different parameters (two different versions of the binary). I’m using a shell function with “parameters”, but you can also use env vars.


    If I’m unit testing in Python, I’ll also just write an explicit loop rather than using weird test framework features.

    1. 3

      If I’m unit testing in Python, I’ll also just write an explicit loop rather than using weird test framework features.

      I find it helpful to unroll the loop: that way the stack trace tells you which example input failed.

      examples = [
          ("", 0),
          ("asdf", 4),
          ("asdfzxcv", 8),
      for input, output in examples:
          # This line failed, but which loop iteration was it?
          test_it(input, output)


      test_it("", 0)
      test_it("asdf", 4)    # This line failed :)
      test_it("asdfzxcv", 8)
      1. 3
        1. 1

          Great tips! We’re using nosetest generated test for this. In above case, it will generate 3 tests.

        2. 1

          The C++ unit testing framework catch has logging macros which take scope into account, meaning you can log the current state in each of your nested loops, but it will only print it out when a test fails: https://github.com/catchorg/Catch2/blob/master/docs/logging.md#top

        3. 2

          In rspec, the parameter feature lets you have default values for the whole suite (near the top of the file) while specifying overrides closer to where they are used.

          That’s the only time I have encountered a feature like that and thought “yes, this is better than a function call”.

          1. 2

            I too quite liked that feature in rspec, but the more I use it, the more I think more flexible scope in Common Lisp is better. Rather than being a feature of the test harness, it’s a language feature:

            CL-USER> (defvar *id* 42 "The default ID we will override in tests")
            CL-USER> (defun make-entity () (list :id *id*))
            CL-USER> (equal
                       '(:id 42))
            CL-USER> (let ((*id* 43))
                         '(:id 43)))
            CL-USER> (equal
                       '(:id 42))
            1. 1

              While I’m enthusing about Common Lisp, the language supports a similar approach to functions:

              CL-USER> (ql:quickload :cl-smtp)
              To load "cl-smtp":
                Load 1 ASDF system:
              ; Loading "cl-smtp"
              CL-USER> (flet ((cl-smtp:send-email (&rest _) (declare (ignore _)) "200 Success"))
                         (cl-smtp:send-email "mail.example.com" "from@example.com" "to@example.com" "subject" "body"))
              "200 Success"

              I forget who once said “patterns are what you do when you run out of language” … there’s a similar thing going on here with test harnesses.

              There are good, feature-rich, test harnesses and mocking/stubbing libraries for Common Lisp. But the language is powerful enough to make their implementation easier than in many other languages. And you can use the same concepts to make your own code more powerful.

          2. 7

            Yay for easier & better testing. Also, I think you’ll freakin love property based testing. (If you don’t already)

            I know didly-zip about the Java biome, but I’ve no doubt you can follow your nose. :D

            1. 5

              As always, resist the temptation to over-generalize. If your test case has conditions for specific parameters, you may be falling into this trap.

              Also, every framework has a different way for doing parameterized tests, but I prefer parameterized on python3.

              1. 4

                I use it extensively with other semi-advanced pytest features. They are great and necessary but I always feel like my coworkers cannot make sense of the tests I write. And sometimes I struggle to make sense of them, especially when the parameters are mixed with 3-4 layers of fixtures. While the intent can be made clear with comments, at least in pytest it’s very hard to highlight the tree of parameters and fixtures that is necessary for tests that have very complex, parametric setup.

                Anyway I would always pick 5 lines of testing+100 lines of fixtures to 105 lines for each test.

                1. 2

                  It’s been mentioned in passing a few times in here, but I wanted to call it out specifically: @pytest.mark.parametrize is great. The syntax takes a little getting used to (you have to feed it a string of comma-separated parameter names) but other than that it works as expected. As a bonus, you can stack multiple @pytest.mark.parameterize decorators on a test to get a cartesian product of test cases.

                  1. 2

                    In my experience, when you want to use parameterized tests, more often than not it’s time to refactor.

                    1. 1

                      I’m a big fan of parameterized tests, and quite like how nUnit handles then:

                      [TestCase("A", false, 23, Name = "Scenario 1")]
                      [TestCase("B", true, 48, Name = "Scenario 2")]
                      public void Test(string s, bool b, int i)

                      Both the test cases and the output tend to be quite readable. Of course it’s possible to go too far (which I’ve been guilty of) and pass a bunch of flags that switch test behaviour on and off to a degree that makes the test unreadable.

                      1. 1

                        I’m not a big fan of parameterized tests in Java using Intellij, because of the debugging experience.

                        My usual workflow when looking into test failures is: use Intellij to debug the test that fails. Step through the test so I can see what the error is, and poke at the state, variables, and assertions.

                        When I try to do this with a parameterized test, it runs all of them. This is frustrating because I often want to set breakpoints, which will trigger on the run for every single parameter. So I either have to go to every breakpoint and make it a conditional breakpoint (which is a pain), or know to hit “continue” a bunch of times until I get to the actual run I care about.