This problem (test assertion not running) is a general problem in most testing frameworks which require you to call a ‘fail’ (directly or indirectly) as the only way to signal a problem. Sometimes you fail to call fail.
Perl produced a test system called TAP (Test Anything Protocol) - now ported to other environments: https://testanything.org/
This uses an approach where:
both positive (ok) and negative (not ok) assertions are reported
the test begins with a statement of the total number of assertions expected (you can finish with this too)
a test whose number of assertions doesn’t match the plan (too many or too few) has failed
There is a small piece of extra work (when you add a test, you have to increase the planned number), but this is minor and the approach guards against tests failing to run for any reason (if your test fails so drastically that it can’t even emit a plan, the test harness will/should report that).
Interesting. Yeah, I’ve experienced this when testing asynchronous code. Jest, for example, lets you set your expected number of assertions up front, per test, which is nice.
I guess I just wasn’t expecting this to come up during a totally synchronous test. And obvious that was a dangerous assumption to make.
This problem (test assertion not running) is a general problem in most testing frameworks which require you to call a ‘fail’ (directly or indirectly) as the only way to signal a problem. Sometimes you fail to call fail.
Perl produced a test system called TAP (Test Anything Protocol) - now ported to other environments: https://testanything.org/
This uses an approach where:
There is a small piece of extra work (when you add a test, you have to increase the planned number), but this is minor and the approach guards against tests failing to run for any reason (if your test fails so drastically that it can’t even emit a plan, the test harness will/should report that).
(Here’s one perl API to emit TAP: http://perldoc.perl.org/Test/More.html)
Interesting. Yeah, I’ve experienced this when testing asynchronous code. Jest, for example, lets you set your expected number of assertions up front, per test, which is nice.
I guess I just wasn’t expecting this to come up during a totally synchronous test. And obvious that was a dangerous assumption to make.
Thanks for the links!
Another excellent Elixir article! Your Bitcoin network client posts have been very helpful to me in using Elixir practically as well.
Thanks! That’s always encouraging to hear.