1. 12
  1.  

  2. 6

    Some examples of oracle testing that I’ve had through work:

    • the oracle is a “reference implementation” in another language (so property test where I’m using an FFI interface to call some JS code, compare against Python code result)
    • the oracle is the “old version” of code. Very very valuable when you’re asserting that your newer, simpler code has the same behavior as some spaghetti code. Wrap up the spaghetti code, then run comparisons. Very valuable when you are in “the current implementation is the spec” situations
    • the oracle is someone else’s implementation of a solution, that we can’t use for reasons X/Y/Z. This often leads to identifying important differences between what we want and what this other implementation wanted.
    1. 5

      the oracle is the “old version” of code. Very very valuable when you’re asserting that your newer, simpler code has the same behavior as some spaghetti code. Wrap up the spaghetti code, then run comparisons. Very valuable when you are in “the current implementation is the spec” situations

      This really stood out to me in the essay, too. Though the essay asserts that it’s uncommon, I’ve certainly needed to resort to this many times. I had no idea there was a nice name for the pattern.

      (And, what’s worse, I clicked through expecting to read about test approaches for the well known proprietary database.)

      1. 2

        I feel like the best way to identify when it’s useful is when you have little faith in the existing test suite catching your edge cases.

        The pattern is usually:

        • there’s some tests on a code path
        • user hits an untested edge case
        • you add a test
        • user hits another untested edge case
        • you add another test …
        • at one point you decide that your code has too many edge cases

        Since your own code has soooo may edge cases, deciding to refactor is basically declaring that you aren’t even sure what your code is doing (if you knew, why does your test suite keep on letting through edge cases?).

        If you have a good grasp of your domain and your tools you’re probably able to avoid this scenario by making the right kind of architectual decisions to limit the , for lack of a better word, spaghetti logic.

    2. 2

      oracle tests only make sense if the oracle is more likely to be correct than what you’re testing, which implies the oracle is simpler than the function under test. But if it’s simpler and more correct, why not just use the oracle as your implementation?

      Several more reasons:

      • the oracle covers only part of the functionality. E.g. it always correctly predicts the first element of the resulting tuple, but not the others.
      • the oracle doesn’t cover orthogonal requirements, such as security, logging, being written in a language the implementation maintainers can read, …
      • the purpose of the oracle is to be a second implementation, written independently using methodology to avoid making the same mistakes implementers are likely to have made

      This is related to my job, which is in ‘model based testing’. We provide our customers tooling and coach them in writing oracles, that are simultaneously readable, executable, specifications, and making it easy to run them against implementations. Sometimes it happens that the model replaces the implementation or is used temporarily until the implementation is done.

      1. 1

        …which implies the oracle is simpler than the function under test…

        This seems like a very erroneous assumption to me. In my experience the “known-good previous implementation” of a piece of software is likely to be scarred, scuffed, battle-tested, missing a few parts, have poor design assumptions, and also has exactly one configuration that is known to be good under almost any circumstance.

        1. 2

          This seems like a very erroneous assumption to me.

          This is covered in the article:

          Sometimes there’s a good reason to not use a simpler oracle:

          • The oracle is too slow
          • The oracle is a reference implementation
          • The oracle only works on a subset of expected inputs
          • The oracle only covers the happy path
          • The oracle is the old code you’re trying to refactor