1. 3

Abstract: “We empirically evaluate the relationship between a test suite’s effectiveness and the (1) number of assertions, (2) assertion coverage, and (3) different types of assertions. We compose 6,700 test suites in total, using 24,000 assertions of five real-world Java projects. We find that the number of assertions in a test suite strongly correlates with its effectiveness, and this factor directly influences the relationship between test suite size and effectiveness”


  2. 2

    Empirical studies of software are important for testing our hypotheses about what works and doesn’t against large sample size. This study corroborates other finds such as from assertion- or property-based testing that they can get lots of errors with less work than just testing everything or ad-hoc. Additionally, it’s promising in light of all the tools automating generation and evaluation of tests from assertions. Shifts developers toward specifying what to test in the general case instead of themselves been test engines. Their brains should be saved for what machines can’t handle.

    Also note that a subset of the assertions can be fed into automated provers to make claims for all inputs rather than tested sample. They might also be used for compiler optimizations or certifications. I find the benefits from overlap in this area more interesting than those from one activity alone.