1. 40
  1. 5

    A broken watch is right twice a day…

    Very common form of testware failure… the framework ignores failures and reports everything is OK (whether it is or not). ie. Have a framework test that tests the framework. Yes, that can fail too, but if your devs never break anything, it’s time to get paranoid.

    Best solution? Invariant checks / conservation checks. eg. Your bank balance should == sum of income - sum of expenses. If it isn’t something somewhere is funky.

    eg. (Sum of tests run == (Sum of tests passed + Sum of tests failed)) && (Sum of tests run != 0)

    1. 8

      eg. (Sum of tests run == (Sum of tests passed + Sum of tests failed)) && (Sum of tests run != 0)

      That is good, but misses “test run died after running 10 successful tests”.

      The perl TAP (Test Anything Protocol) has the concept of a test ‘plan’ - the first thing you do is emit the number of tests you intend to run. Even If you emit passes, if you emit too few (or too many!) the run overall is a fail.

      https://perldoc.perl.org/Test::More#I-love-it-when-a-plan-comes-together

      This adds some developer burden (update the plan when you change the number of tests) but that is a light chore. I’d love for other languages to incorporate this (e.g. golang test infra).

      1. 7

        the framework ignores failures and reports everything is OK

        That’s one I’ve definitely seen a few times in the wild. Another fun one is if some subset of the tests don’t even get picked up and run in the first place. Because the file was in the wrong directory, the test class didn’t derive from TestCase or because the method prefix wasn’t a correctly spelled test_, for example (or, in Python, if due to a copy/paste mistake, you have two methods called test_foo in the same class - only one of them is actually run). These are very difficult to detect and are usually discovered “by accident” (hopefully sooner rather than later, ideally in code review).

        1. 3

          Oof. Duplicate test_foo ought to be picked up by a linter, hopefully. The rest not so much.

      2. 1

        We’re quoting Ronnie Raygun now?