I evidently missed this the first time it got posted, and I’m glad it got posted again. I’ve had property based testing on my radar for a long time, and I’ve managed to apply it to some straightforward pieces of code, but this gives me another, more powerful strategy for testing more complex behaviour.
(This is generally something I struggle with: I get the idea of the various different kinds of testing, but often find it very hard to apply them (a) at all in a particularly challenging domain or runtime environment (*) or (b) particularly effectively, because the literature tends to show fairly trivial toy examples, not hairy real world problems.)
(* For example. I’d have to build my own test harness or runtime, and there’s a lot of “environment” that would need to be mocked/stubbed, or I’d need to build an entire abstraction layer to allow the code to be built outside its test-unfriendly target runtime.)
I’m in the same camp (have always been interested in property-based testing, but want to use it at work and not on tiny example functions).
The issue is that properties are difficult to think up for lots of typical business software. But, one property that you can always use is “does the implementation implement a model.” This is useful when the behavior is simple, but the implementation is relatively complex, i.e. the case in almost all web applications.
If you ask me, this is the technique that’s most likely to be applicable at work. There are a couple of technical hurdles (getting the database state to be correctly in sync and all that), but moreso people think writing a model is a weird at first. That’s the bigger hurdle. Of course, you can always show them this post too :)
For me, it makes a ton of sense. Not only do you get to generate most of your tests, which is an actual time and cost saver, but have you ever been in a meeting where you get into a really sticky scenario and no one can answer what the application should actually do in that case? Well, having a simplified executable model makes this way easier. It’s way easier to get a specific data state set up in an in-memory Typescript object vs. a cloud-native microservice application. But I’m definitely biased.
Yup, this definitely seems like it might make a big difference in terms of part (b) of my struggles.
I have yet to find a shortcut for (a) - on one macOS kernel extension project we managed to come up with a sufficient blend of mocking or abstracting kernel APIs and linkage cleverness to make it work so we could write unit and integration tests (as far as those terms are well-defined given the context) for kernel code implementing C callback interfaces. These days I spend a lot of my time writing macOS DriverKit device drivers, for which Apple has created a special codegen dialect of C++ headers (iig). They interface with hardware on the one side, and application code on the other side, via IOKit Mach IPC. Isolating any of this stuff (or even just building it in the context of a normal user space executable or library) is a nightmare.
I evidently missed this the first time it got posted, and I’m glad it got posted again. I’ve had property based testing on my radar for a long time, and I’ve managed to apply it to some straightforward pieces of code, but this gives me another, more powerful strategy for testing more complex behaviour.
(This is generally something I struggle with: I get the idea of the various different kinds of testing, but often find it very hard to apply them (a) at all in a particularly challenging domain or runtime environment (*) or (b) particularly effectively, because the literature tends to show fairly trivial toy examples, not hairy real world problems.)
(* For example. I’d have to build my own test harness or runtime, and there’s a lot of “environment” that would need to be mocked/stubbed, or I’d need to build an entire abstraction layer to allow the code to be built outside its test-unfriendly target runtime.)
I’m in the same camp (have always been interested in property-based testing, but want to use it at work and not on tiny example functions).
The issue is that properties are difficult to think up for lots of typical business software. But, one property that you can always use is “does the implementation implement a model.” This is useful when the behavior is simple, but the implementation is relatively complex, i.e. the case in almost all web applications.
If you ask me, this is the technique that’s most likely to be applicable at work. There are a couple of technical hurdles (getting the database state to be correctly in sync and all that), but moreso people think writing a model is a weird at first. That’s the bigger hurdle. Of course, you can always show them this post too :)
For me, it makes a ton of sense. Not only do you get to generate most of your tests, which is an actual time and cost saver, but have you ever been in a meeting where you get into a really sticky scenario and no one can answer what the application should actually do in that case? Well, having a simplified executable model makes this way easier. It’s way easier to get a specific data state set up in an in-memory Typescript object vs. a cloud-native microservice application. But I’m definitely biased.
Yup, this definitely seems like it might make a big difference in terms of part (b) of my struggles.
I have yet to find a shortcut for (a) - on one macOS kernel extension project we managed to come up with a sufficient blend of mocking or abstracting kernel APIs and linkage cleverness to make it work so we could write unit and integration tests (as far as those terms are well-defined given the context) for kernel code implementing C callback interfaces. These days I spend a lot of my time writing macOS DriverKit device drivers, for which Apple has created a special codegen dialect of C++ headers (
iig). They interface with hardware on the one side, and application code on the other side, via IOKit Mach IPC. Isolating any of this stuff (or even just building it in the context of a normal user space executable or library) is a nightmare.