1. 17

  2. 27

    My first job, almost 8 years ago, was sub-contracting to test software on the F-35. They subcontracted the unit testing to us, a much smaller company (“they” were actually a subcontracter of another company). We didn’t have requirements, but we did have pressure to stay on schedule, so we wrote tests anyway. We had excellent coverage – every single byte of machine code was covered as well as every possible branch (if we couldn’t cover an instruction or branch, we left an official note explaining why). But since we didn’t know what the software was supposed to do, these tests just gained coverage. Our customer was happy to pay us and report 100% test coverage to their superiors.

    1. 5

      I wonder if non-programmers understand the difference between lines-of-code test coverage and input domain test coverage.

      1. 6

        Not at all. Code is the closest thing to a physical artifact, so saying you’ve tested 100% of the code sounds great. Internal state and input domains are much more abstract concepts.

        1. 6

          In engineering I’ve met quite a few people who do, though maybe not phrased exactly like that. It helps that input domains can be roughly mapped to the non-software idea of testing equipment by establishing a set of operating conditions that it is supposed to function within, and then getting good test coverage of the conditions within those parameters (including any unusual/extreme/unlikely conditions included in the range). The key bit anyway is that you’re supposed to get good coverage of the conditions, not only of the equipment, i.e. full test coverage of every last bit of a part isn’t good enough if you test it only under one temperature or load. And somewhat analogously with software.