Apparently snapshot testing’s used in some front end frameworks to for component testing, but I’ve never seen anything like this before and the dev experience is literally breathtaking. What else have I been missing?
I use snapshot testing all the time, for everything! If I can’t write a fuzz/property-based test, I write snapshot-test. To the first approximation, anything more complicated than one assert(a == b) per test is worth moving to a snapshot.
Though, I don’t like the style where the snapshot is in a separate file, it makes small tests harder to read. By default, I specify snapshots inline, as a string literal, and use file only for gigantic outputs.
It’s also worth knowing that snapshot testing is very easy to implement, you don’t need a fancy library to start doing it (though, a library certainly has additional bells and whistles):
If I can’t write a fuzz/property-based test, I write snapshot-test.
Do you see property-based testing as a replacement for example-based testing?
In other words, when it’s possible to write a property-based test, do you treat that as the first line of defense against bugs in that code? This way of working appeals to me because I’d rather come up with a clear specification than a mountain of examples.
A counterargument is that it’s useful to have both: ideally you would have enough coverage from examples, but the fuzzer serves as a backstop. The examples test the (correctness of) the code, and the fuzzer tests the (completeness of) the examples.
I wonder if the best would actually be a combination:
A clear specification of what makes an output correct.
A corpus of interesting examples, which can be fed into the above spec.
An example-generator.
You could think of this as a corpus-driven property-based test. The examples make it easy to know what’s covered, and the spec makes it easy to know that the expected-outputs are actually correct.
I don’t think I can compress my full decision tree into a single commit box, but the top points are that you should thing about fuzzing/PBT/DST first, as those are high-investment/high-payoff techniques, and otherwise fallback to snapshot testing as a default low-investment technique.
Another way to refer to this technique is “golden file” testing.
I’ve used this technique for backend testing for many years, it is great, but you need to make sure your output is fully deterministic, which is a bit of pain when having generated IDs or timestamps.
The best libraries have a way to replace those values with placeholders. If not, you need to sanitize your output manually before passing it to the snapshot.
Apparently snapshot testing’s used in some front end frameworks to for component testing, but I’ve never seen anything like this before and the dev experience is literally breathtaking. What else have I been missing?
I use snapshot testing all the time, for everything! If I can’t write a fuzz/property-based test, I write snapshot-test. To the first approximation, anything more complicated than one
assert(a == b)per test is worth moving to a snapshot.Though, I don’t like the style where the snapshot is in a separate file, it makes small tests harder to read. By default, I specify snapshots inline, as a string literal, and use file only for gigantic outputs.
It’s also worth knowing that snapshot testing is very easy to implement, you don’t need a fancy library to start doing it (though, a library certainly has additional bells and whistles):
https://tigerbeetle.com/blog/2024-05-14-snapshot-testing-for-the-masses/
Do you see property-based testing as a replacement for example-based testing?
In other words, when it’s possible to write a property-based test, do you treat that as the first line of defense against bugs in that code? This way of working appeals to me because I’d rather come up with a clear specification than a mountain of examples.
A counterargument is that it’s useful to have both: ideally you would have enough coverage from examples, but the fuzzer serves as a backstop. The examples test the (correctness of) the code, and the fuzzer tests the (completeness of) the examples.
I wonder if the best would actually be a combination:
You could think of this as a corpus-driven property-based test. The examples make it easy to know what’s covered, and the spec makes it easy to know that the expected-outputs are actually correct.
I don’t think I can compress my full decision tree into a single commit box, but the top points are that you should thing about fuzzing/PBT/DST first, as those are high-investment/high-payoff techniques, and otherwise fallback to snapshot testing as a default low-investment technique.
Another way to refer to this technique is “golden file” testing.
I’ve used this technique for backend testing for many years, it is great, but you need to make sure your output is fully deterministic, which is a bit of pain when having generated IDs or timestamps.
The best libraries have a way to replace those values with placeholders. If not, you need to sanitize your output manually before passing it to the snapshot.
It reminds me of ppx_expect, which is similar but saves the snapshot inline and also supports things like snapshotting stdout.
Taken to an extreme (in a good way) is Ian Henry’s My Kind of REPL (discussion).