Maybe? Maybe not. Who cares? Tests are fast or slow. They are either in the suite that I can run in seconds in my machine, or in the suite that takes longer, has external dependencies, and runs usually on the build server. And in either case we want them to be easy to maintain and to catch bugs. Labouring too much in semantic distinction adds little to it, in my experience.
I think the semantic distinction reveals intention and also what you’d do. Run fast tests more often, run slow tests in CI (or late or less often). Have mocks in fast/unit tests, have in-memory realism in slow/CI tests. So if you do as the article is implying, replace mocks with simulation that’s not “right”. Martin Fowler’s testing pyramid uses these terms. I think because you could have fast integration tests (compared to what idk). So I agree that I think of unit/integration as fast/slow for workflow but as well as realistic/clean-room. So usually I use the terms unit/integration but these concepts still aren’t understood. Even by me spending a lot of time on this.
Kent Beck says the unit is the test. How many people think the unit is the class? I learned this ~2 years ago. It’s in a book from 2002. :| I had this wrong for years. This helped me understand what to isolate (I think).
But yeah, it’s hard to come up with rules. And deeply understanding something immediate without the rules is probably too hard. Any rules you come up with will creak under project differences. Types sorta/kinda replace unit tests sometimes depending on the language. So the style, flow and distribution of tests changes. The thing to do is to make adjustments in reaction to pain.
Unit testing a face to make sure it blinks and smiles.
Unit testing a hand to make sure it grips.
Unit testing legs to make sure it jumps.
But then you integration test the body.
But that could be fast. So there’s a functional piece too. Because mocks can lie to you. If your mocks lie and this causes you pain, look at your mocks or add more realistic tests. Adjust to pain.
Which leads to another comment about something you said: “it’s hard to come up with rules”. It’s actually probably impossible to come up with exceptionless rules in software engineering, which is why I rather have none. The kind of rules we’re talking about are usually only short hands to more deep insights, so I’d rather throw the rules out and keep the insights, and make up my mind case by case.
Maybe it’s not an optimal strategy, but I find it leads to less bike shedding and allows for more creative thinking.
I don’t think this reflects any official definition that I’ve seen anywhere, but my personal definition of a unit test vs an integration test is the diagnostic ability of a test. There’s a spectrum from unit test to integration test and the platonic ideal of a unit test would tell me the precise expression in my source code where the big is. A realistic unit test (slightly further along the spectrum) would tell me the function / method, or at worse the class responsible for the bug.
If every component has a well-specified interface, full unit test coverage of the implementation to ensure that it meets the specification, and full unit test coverage of all consumers to ensure that they don’t rely on anything that is not part of the specification, then that would be sufficient. Integration tests exist because you rarely have even the first of these and writing the third is incredibly hard (testing negative properties is almost impossible).
If an integration test fails, it tells me that I’m missing a unit test. My first task is to try to write a unit test that fails for the same root cause, which can then be added to my test suite.
Absolutely. No reason why we should religiously adhere to a pre-existing architecture, though. We can adapt and invent as needed, as long as it is documented.
The post is less about integration testing and more about only mocking the parts of your codebase that are directly involved with nondeterministic behavior or external systems. Basically replace isTodayWednesday with isDateWednesday(date=Today())
No, I think he is really arguing more for a functional style, to the point of providing not just state and an external service as dependencies, but even system time.
I recommend reading this definition of unit testing by Martin Fowler. It goes into the difference between “sociable” and “isolated” unit testing. Most of the people who invented and championed unit testing at first wrote sociable tests, where only things like a database or HTTP server were mocked / stubbed. Isolated unit testing is a newer phenomenon, where you mock out every single dependency of the object / function under test.
For some reason, newer developers only know about isolated unit testing. Not sure why that is.
I’m going to read the rest of the article but aren’t they describing an integration testing practice more than a unit testing one?
Maybe? Maybe not. Who cares? Tests are fast or slow. They are either in the suite that I can run in seconds in my machine, or in the suite that takes longer, has external dependencies, and runs usually on the build server. And in either case we want them to be easy to maintain and to catch bugs. Labouring too much in semantic distinction adds little to it, in my experience.
I think the semantic distinction reveals intention and also what you’d do. Run fast tests more often, run slow tests in CI (or late or less often). Have mocks in fast/unit tests, have in-memory realism in slow/CI tests. So if you do as the article is implying, replace mocks with simulation that’s not “right”. Martin Fowler’s testing pyramid uses these terms. I think because you could have fast integration tests (compared to what idk). So I agree that I think of unit/integration as fast/slow for workflow but as well as realistic/clean-room. So usually I use the terms unit/integration but these concepts still aren’t understood. Even by me spending a lot of time on this.
Kent Beck says the unit is the test. How many people think the unit is the class? I learned this ~2 years ago. It’s in a book from 2002. :| I had this wrong for years. This helped me understand what to isolate (I think).
But yeah, it’s hard to come up with rules. And deeply understanding something immediate without the rules is probably too hard. Any rules you come up with will creak under project differences. Types sorta/kinda replace unit tests sometimes depending on the language. So the style, flow and distribution of tests changes. The thing to do is to make adjustments in reaction to pain.
But that could be fast. So there’s a functional piece too. Because mocks can lie to you. If your mocks lie and this causes you pain, look at your mocks or add more realistic tests. Adjust to pain.
So, first, I think @hwayne expressed it better and more succinctly, so: https://lobste.rs/s/2btkdu/favor_real_dependencies_for_unit_testing#c_usyhrv. That’s basically it, don’t mock code that you fully control and that has deterministic behavior, unless you really have to.
Which leads to another comment about something you said: “it’s hard to come up with rules”. It’s actually probably impossible to come up with exceptionless rules in software engineering, which is why I rather have none. The kind of rules we’re talking about are usually only short hands to more deep insights, so I’d rather throw the rules out and keep the insights, and make up my mind case by case.
Maybe it’s not an optimal strategy, but I find it leads to less bike shedding and allows for more creative thinking.
I don’t think this reflects any official definition that I’ve seen anywhere, but my personal definition of a unit test vs an integration test is the diagnostic ability of a test. There’s a spectrum from unit test to integration test and the platonic ideal of a unit test would tell me the precise expression in my source code where the big is. A realistic unit test (slightly further along the spectrum) would tell me the function / method, or at worse the class responsible for the bug.
If every component has a well-specified interface, full unit test coverage of the implementation to ensure that it meets the specification, and full unit test coverage of all consumers to ensure that they don’t rely on anything that is not part of the specification, then that would be sufficient. Integration tests exist because you rarely have even the first of these and writing the third is incredibly hard (testing negative properties is almost impossible).
If an integration test fails, it tells me that I’m missing a unit test. My first task is to try to write a unit test that fails for the same root cause, which can then be added to my test suite.
Every test is somewhere along the spectrum. Before Christmas, I wrote a thing that I thought was very close to the platonic ideal of a unit test to check a binary semaphore implemented with
futexon Linux and_umtx_opon FreeBSD. This test is hitting about a dozen lines of my code but it was intermittently failing on FreeBSD and I couldn’t work out why. It eventually turned out that FreeBSD libc was missing an annotation to interpose onpdforkand so would sometimes deadlock when the child process exited. My simple unit test was actually an integration test of the FreeBSD kernel, FreeBSD libc, and my tiny bit of code.Having a testing architecture helps everyone on the team stay on the same page and know what kinds of tests to write and where to put them.
Absolutely. No reason why we should religiously adhere to a pre-existing architecture, though. We can adapt and invent as needed, as long as it is documented.
The post is less about integration testing and more about only mocking the parts of your codebase that are directly involved with nondeterministic behavior or external systems. Basically replace
isTodayWednesdaywithisDateWednesday(date=Today())No, I think he is really arguing more for a functional style, to the point of providing not just state and an external service as dependencies, but even system time.
Or maybe I understand it completely backwards?
I recommend reading this definition of unit testing by Martin Fowler. It goes into the difference between “sociable” and “isolated” unit testing. Most of the people who invented and championed unit testing at first wrote sociable tests, where only things like a database or HTTP server were mocked / stubbed. Isolated unit testing is a newer phenomenon, where you mock out every single dependency of the object / function under test.
For some reason, newer developers only know about isolated unit testing. Not sure why that is.