First of all, the idea of generating a test that checks the correctness of some other generated code is the notion of certifying compilation (also sometimes “self-certifying”). It’s one of my favorite concepts! This is more common in formal verification, for example in the Cogent language which generates proofs of verified filesystem implementations. But tests (especially property-based tests) can be seen as a proxy for such proofs, so generating tests instead is the same idea. I think it’s super cool and something we should do more of.

Second, I ran into this exact problem when trying to generate more interesting test values than random ones as inputs to property-based tests. I was looking for a combinatorial approach that could produce lots of combinations based on “categories” of values, but the solution I came up with wasn’t very generalizable and suffered from this same problem:

Seems ok, until we find out that the function returns only one combined value with only one test value per attribute

Using generators seems like a really good solution to this problem!