1. 4
  1. 5

    I’m curious why your goal is “100% coverage” and not “great tests for tricky parts”?

    My position on code coverage has always been that, unless you have a team of extremely disciplined developers, your code is going to have a lot of less than ideal tests to sustain that number. Rather than give into this, I’ve always chosen to use the red lines in coverage output as a way to spot check for complexity and make a decision on test value for that spot. I define complexity as an estimate of cyclomatic complexity, or “gut feel” on importance–it’s kind of arbitrary.

    Curious to hear your thoughts on this.

    1. 4

      Not OP, but this has been my experience as well. High coverage targets lead to (a) tests that don’t actually test anything useful but needed to be written to boost the coverage percentage, and worse, (b) less defensive programming in the code under test because it’s easier to leave out a “this should never happen” sanity check than to figure out how to simulate the theoretically impossible state you’re guarding against.

      100% condition coverage doesn’t imply a high-quality test:

      def add(a, b):
          if a > b:
              return a + b
              return a - b
      def test_add():
          assertEqual(add(0, 0), 0)
          assertEqual(add(2, 1), 3)
      1. 2

        I used this exact example in a talk on property testing! The “proper” test would be something like

        @given(int(), int())
        def test_add_is_commutative(a, b):
            assert add(a, b) == add(b, a)

        (Though that also lets through multiplication… you also want something like a+(b+1) = (a+1)+b)

        1. 2

          Being in a regulated field, 100% coverage is usually required. Not just statement but even branch coverage. In the extreme, we had test cases which did not check anything. They just executed code and increased coverage.

        2. 1

          I love code complexity! (Well, this sounds weird, but I truly love to mess with code complexity metrics. I have even invented one!)

          I try not mix these two concepts together. We always have 100% on out open-source projects. So, this is not a decision I make every time. It is a rule. We also try to have 100% on our closed-source projects. But, sometimes it is not worth it in budget / time terms. And we can go as low as 95% of coverage.

          You might like some of the things I do about code complexity:

          1. 1

            It is true that 100% coverage tells you nothing at all about whether you can trust the test suite to catch your mistakes.

            However, 90% coverage tells me something very important - specifically, that you definitely cannot trust the test suite to catch your mistakes.

          2. 3

            We had a similar problem at work. There was a custom data structure, effectively a bounded priority queue. Boundary length is a template parameter. One method had a speed optimization, special cases for different lengths. A unit test checked with a very small boundary such that some cases were impossible to reach. Deleting the test would have resulted in 100% coverage.

            In the end, we refactored it to template specialization instead of plain if statements to resolve it.