1. 21

  2. 4

    This might be a good article to plug in combinatorial testing (n-way, not pair-wise) since it’s pretty much exactly this with tools available. Intro slides here (pdf). They’ll say features, parameters, components… depends on what the tool is testing. It will usually do both a bunch of interactions and some technique to reduce number of test cases.

    1. 3

      This is why I tend to prefer end-to-end tests over unit tests, even if they are a pain to run.

      At $job, I work on a project with about 8-10 features [1] that have accrued over the past decade. There’s a master list of users (well, phone numbers) to features. Some features preclude the use of other features, other features can be mixed and matched. Some of the features are like:

        1. only query A
        1. only query B
        1. query both A and B at the same time
        1. query B first, then A depending upon the results from B

      The full test runs well over 14,000 individual tests (and there could probably be more, truth be told) across six programs (only two are actually being tested, but the other four are required to run the two under test). You would think that the mapping of phone numbers to features to be consistent, like feature 1 and 4 won’t be set, but alas, when I last checked a few months ago, there were combos (a few dozen out of millions of records) that should never exist and it can be fun at times [2] figuring out what exactly should the result be vs. what it actually is.

      [1] It’s somewhat nebulous, as some are kind of features, kind of environment mix, and some “features” indicate a query is normally over one protocol but can be another one due to issues.

      [2] Not really.