The problem with this approach [with mocks] is it’s completely tautological.
Yes! I once deleted an entire Python class and the tests kept passing because it was mocks all the way down. Less then worthless tests, 99% of the time.
To be fair, that’s an issue an issue with how the tool is used – you can do bad work with any tool, after all. If you’re doing TDD, that’s why the first step is to write some test and make sure it fails for the right reasons.
It’s also why in mockist-style TDD, you should only mock interfaces you own, as other wise, you end up guessing at the semantics of the interface, rather than designing it yourself.
I feel like there’s probably one of those tedious arguments about super-picky distinctions between “mocks” and “stubs” and “test doubles” here, similar to the arguments about “unit” versus “integration” versus “end-to-end” tests, that’s unlikely to produce any satisfactory result. So I’d just call it a mock :)
I think what’s meant to be avoided here is the kind of mocking where you “inject” case by case behaviour to the mock right there in the test case. Like, manually adding a user to the mock response of an insert endpoint. This is distinct from somebody implementing the semantics of an external service as an independent project without any direct reference to a test suite, maybe except for defining the scope.
There’s another approach that I think is under-utilized in the industry - use real dependencies but run your tests against both the change and the last-known-good commit at the same time[^1]. If the LKG commit fails then you can chalk up it up to an external dependency failing. This obviously doesn’t fully exonerate the change, but in many scenarios that might be OK (or at least just significantly improves debug times). I would love to see a test framework with this built in.
[^1]: There are a few tricky things to consider (resource isolation etc.) but it’s not impossible.
I am curious about this problem form the other side: if you are a vendor of one of the “Theirs” thing, how can you simplify local testing for your users?
I’ve worked on a team before that published a local “mock” server of our service. It wasn’t purely a mock since it did a ton of the work the main service did, but ephemerally and not the hard stuff. Keep in mind it was a very simple service, and it had very specialized users who we had direct partnerships with.
It was a really large hassle, but it worked. You’d make a request to setup a test context scoped to a particular test, then make your calls using a connection to it. Our partners appreciated it alot, and it had the extra benefit of exposing some misunderstandings they had about our service. “Why isn’t the mock server working correctly” questions almost always ended in them not knowing what “correctly” meant.
I don’t think I’d ever suggest that approach or do it again. Even with a small service that didn’t change very often, it was ALOT of work and maintenance that never seemed finished. Constant conversations would come up on how much could be done by the mock service itself and how much should be provided by the user. There was also an interesting case where a user seemed to learn about our service via the testing tools and not the documentation leading to all kinds of really weird and superstitious beliefs about our API. Not to mention requests to support sdks for the testing tool or asks to workaround quirks in their testing environment that had nothing to do with us.
I still like the idea. But the effects of owning that were way larger than we expected.
The main downside of this approach is the combinatorics involved–doing end to end tests for your happy, common cases is one thing, but it gets way more difficult once you start to deal with corner cases, failures and their interactions.
It’s true, it takes a very large amount of effort to increase coverage of end to end tests. But, it’s also true that isolated unit tests don’t give you confidence about overall correctness. So each type has its own tradeoffs. There’s no reason to only have one type of test in a test suite.
Yes! I once deleted an entire Python class and the tests kept passing because it was mocks all the way down. Less then worthless tests, 99% of the time.
To be fair, that’s an issue an issue with how the tool is used – you can do bad work with any tool, after all. If you’re doing TDD, that’s why the first step is to write some test and make sure it fails for the right reasons. It’s also why in mockist-style TDD, you should only mock interfaces you own, as other wise, you end up guessing at the semantics of the interface, rather than designing it yourself.
Isn’t this… a mock?
I feel like there’s probably one of those tedious arguments about super-picky distinctions between “mocks” and “stubs” and “test doubles” here, similar to the arguments about “unit” versus “integration” versus “end-to-end” tests, that’s unlikely to produce any satisfactory result. So I’d just call it a mock :)
I think what’s meant to be avoided here is the kind of mocking where you “inject” case by case behaviour to the mock right there in the test case. Like, manually adding a user to the mock response of an insert endpoint. This is distinct from somebody implementing the semantics of an external service as an independent project without any direct reference to a test suite, maybe except for defining the scope.
Not really: https://martinfowler.com/articles/mocksArentStubs.html
There’s another approach that I think is under-utilized in the industry - use real dependencies but run your tests against both the change and the last-known-good commit at the same time[^1]. If the LKG commit fails then you can chalk up it up to an external dependency failing. This obviously doesn’t fully exonerate the change, but in many scenarios that might be OK (or at least just significantly improves debug times). I would love to see a test framework with this built in.
[^1]: There are a few tricky things to consider (resource isolation etc.) but it’s not impossible.
I am curious about this problem form the other side: if you are a vendor of one of the “Theirs” thing, how can you simplify local testing for your users?
I’ve worked on a team before that published a local “mock” server of our service. It wasn’t purely a mock since it did a ton of the work the main service did, but ephemerally and not the hard stuff. Keep in mind it was a very simple service, and it had very specialized users who we had direct partnerships with.
It was a really large hassle, but it worked. You’d make a request to setup a test context scoped to a particular test, then make your calls using a connection to it. Our partners appreciated it alot, and it had the extra benefit of exposing some misunderstandings they had about our service. “Why isn’t the mock server working correctly” questions almost always ended in them not knowing what “correctly” meant.
I don’t think I’d ever suggest that approach or do it again. Even with a small service that didn’t change very often, it was ALOT of work and maintenance that never seemed finished. Constant conversations would come up on how much could be done by the mock service itself and how much should be provided by the user. There was also an interesting case where a user seemed to learn about our service via the testing tools and not the documentation leading to all kinds of really weird and superstitious beliefs about our API. Not to mention requests to support sdks for the testing tool or asks to workaround quirks in their testing environment that had nothing to do with us.
I still like the idea. But the effects of owning that were way larger than we expected.
The main downside of this approach is the combinatorics involved–doing end to end tests for your happy, common cases is one thing, but it gets way more difficult once you start to deal with corner cases, failures and their interactions.
I quite like J. B. Rainsburger’s take on this, even if it is a little caustic.
It’s true, it takes a very large amount of effort to increase coverage of end to end tests. But, it’s also true that isolated unit tests don’t give you confidence about overall correctness. So each type has its own tradeoffs. There’s no reason to only have one type of test in a test suite.