Unit tests are one tool in a big toolbox. But what do we verify? I think you could make a distinction:
Verify that the data a program outputs satisfies some constraints given by the inputs. Unit tests, types, etc. can do this job well.
Verify the behavior that a system exhibits satisfies some constraints. Here Integration tests are useful, but fail when the system changes.
We assume that the encoding of these constraints is simpler than the program itself and have appropriate specificity, which is not always the case. Also, the ergonomics of your verification strategy may encourage you to be too specific.
The article describes a problem of being too specific something that unit testing encourages. Integration testing can also suffer from the same problem of being overly specific. It doesn’t encourage it as much i believe.
I think unit tests can be best summarized as confirming that raw computation works. Does the adder add? Does the calculator of user progress return what it should? Does your check for competing resources actually check the correct state?
Anything outside that, beyond simple matters for the “Difference Engine,” shouldn’t have a unit test.
Thanks for the comment. I hadn’t considered the notion that integration testing could suffer from the same problem. You’ve sparked some new thoughts in my brain and that’s always welcome!
In my experience, integration tests have the best ROI as they can be relatively simple to write, and test a large area of the code. They are good at spotting issues deep inside the code that might not be obvious at first, and perhaps even wouldn’t be caught by unit testing.
Unit testing on the other hand is perfect for testing library-like code, say your utility library to manage ut8 strings or date calculations, etc.
I think that there are ways of architecting the system such that unit tests become better than integration tests, namely hexagonal testing. In my experience people usually write end-to-end tests because they’re easier to write then wonder why CI is always red when they have 1000 of those. They’re flaky, and nearly as bad, slow to run, which makes people not run them after every change.
Sort of countering and corroborating that, I think usage-based testing is highest ROI as advocated by Cleanroom in 1980’s. Their theory was it’s better for software to have a 100 bugs users never see than even 5 they often do. Their perception would drive the acceptance and spread of the software. So, you look at the ways they can use the project, game out any ways to turn it into tests, and run those first. Maximizes the perceived quality.
Then, integration. Then, unit. These will improve actual quality but might take more resources. I said corroborate since the usage-based tests will likely be a form of integration testing since they test many features at once. Acceptance testing is the common term for this, though, since it’s usually black box.
I admit, I did not understand the specific point article was making.
May be because the example provided is very very narrow and not something I could relate to.
However, the article’s title is captivating and the topic is important.
I believe lack of automatically generated unit tests is an example of several glaring gaps
in software engineering tooling, practices and theory.
Formal specification of functionality intents, should automatically generate unit tests.
There is just little bit of work in that area eg for large systems [1], and for protocols with TLA+ [2]
I, personally, need to really invest time in TLA+ – i think these kinds of approaches are future of software engineering, especially as we move from ‘release-and-patch’ mentality, to more robust expectations of quality
in business applications, robotics systems, and so on.
In terms of unit tests vs integration tests.
I think unit tests, today, are almost a necessity for library developers – because they cannot anticipate all the
possible scenarios where their libraries will be used.
But for non-library applications, manual use-case-based integration+usability tests, are a much better investment of programmer’s time.
I posit that anything, treated uncritically and/or cargo culted, is a bad choice for your code. Terms like “unit test” have a brief lifetime as an interesting new category that may stimulate good new thoughts…after which they beg to be overconstrained, dogmatized, and rendered nothing but clickbait. Just write tests that test the behaviors you care about, as well as you can under the circumstances, as clearly and maintainable as possible—you know, just like the rest of the code—and don’t worry about what you call them. :)
Sure, I thought it was an interesting perspective from someone who prefers unit tests but thought that there was a better way to test the code in question.
I find it’s more important to be clear about the scope of the behavior you’re trying to test than to apply arbitrary terms to describe tests.
For example, if you’re trying to test the runtime behavior of the makefile (essentially using cmake as a library to accomplish the task of the function under test), then yes, that’s not a good test. On the other hand, if the function is supposed to make human-readable makefiles and you want to check the indentation and comments in the makefile, that might be a great way to write that test. The term “unit test” does nothing to assist that analysis.
The point of the article that I think is profoundly important is: only test the thing you’re testing, and avoid accidentally testing things you didn’t mean to test.
For example, if you’re trying to test the runtime behavior of the makefile (essentially using cmake as a library to accomplish the task of the function under test), then yes, that’s not a good test. On the other hand, if the function is supposed to make human-readable makefiles and you want to check the indentation and comments in the makefile, that might be a great way to write that test. The term “unit test” does nothing to assist that analysis.
I like this way of thinking, thanks for the insight.
I — also a unit test aficionado — posit that unit tests have no inherent value. In fact unit tests are a poor choice for verifying any behaviour that could otherwise be verified by cheaper means, e.g., types, property based testing, etc.
Technically speaking, “unit test” is about scope while “property based test” is about method and generativity. Seen this way, most PBTs are unit tests.
In practice, people usually mean “unit test” to mean manual oracle unit tests, and PBT to mean generative property tests of all scopes (but usually unit).
It’s not either/or though. Sometimes types can’t get you there. Sometimes contracts can’t get you there. Even when they can, code can be buggy anyway. My first and only Haskell project compiled and passed all tests. And yet, it didn’t work at all as intended.
Certainly. And I’m in no way suggesting that types will do everything and that they obviate the need for tests.
The reason for my — admittedly rather provocative — statement, is because it seems a prevalent idea in the industry (especially in the Software Craftsmanship community) that if unit tests aren’t your panacea, then you just aren’t unit testing hard enough. I think this is a damaging idea, and it deserves some pushback.
Some random thoughts:
Unit tests are one tool in a big toolbox. But what do we verify? I think you could make a distinction:
We assume that the encoding of these constraints is simpler than the program itself and have appropriate specificity, which is not always the case. Also, the ergonomics of your verification strategy may encourage you to be too specific.
The article describes a problem of being too specific something that unit testing encourages. Integration testing can also suffer from the same problem of being overly specific. It doesn’t encourage it as much i believe.
I think unit tests can be best summarized as confirming that raw computation works. Does the adder add? Does the calculator of user progress return what it should? Does your check for competing resources actually check the correct state?
Anything outside that, beyond simple matters for the “Difference Engine,” shouldn’t have a unit test.
Thanks for the comment. I hadn’t considered the notion that integration testing could suffer from the same problem. You’ve sparked some new thoughts in my brain and that’s always welcome!
In my experience, integration tests have the best ROI as they can be relatively simple to write, and test a large area of the code. They are good at spotting issues deep inside the code that might not be obvious at first, and perhaps even wouldn’t be caught by unit testing.
Unit testing on the other hand is perfect for testing library-like code, say your utility library to manage ut8 strings or date calculations, etc.
I think that there are ways of architecting the system such that unit tests become better than integration tests, namely hexagonal testing. In my experience people usually write end-to-end tests because they’re easier to write then wonder why CI is always red when they have 1000 of those. They’re flaky, and nearly as bad, slow to run, which makes people not run them after every change.
What is “hexagonal testing?” Because trying to Google that leads to athletics testing, not program testing.
https://vimeo.com/68375232 https://apiumhub.com/tech-blog-barcelona/hexagonal-architecture/ http://www.duncannisbet.co.uk/hexagonal-architecture-for-testers-part-2
Thanks.
Sort of countering and corroborating that, I think usage-based testing is highest ROI as advocated by Cleanroom in 1980’s. Their theory was it’s better for software to have a 100 bugs users never see than even 5 they often do. Their perception would drive the acceptance and spread of the software. So, you look at the ways they can use the project, game out any ways to turn it into tests, and run those first. Maximizes the perceived quality.
Then, integration. Then, unit. These will improve actual quality but might take more resources. I said corroborate since the usage-based tests will likely be a form of integration testing since they test many features at once. Acceptance testing is the common term for this, though, since it’s usually black box.
[Comment removed by author]
I admit, I did not understand the specific point article was making. May be because the example provided is very very narrow and not something I could relate to.
However, the article’s title is captivating and the topic is important.
I believe lack of automatically generated unit tests is an example of several glaring gaps in software engineering tooling, practices and theory.
Formal specification of functionality intents, should automatically generate unit tests. There is just little bit of work in that area eg for large systems [1], and for protocols with TLA+ [2] I, personally, need to really invest time in TLA+ – i think these kinds of approaches are future of software engineering, especially as we move from ‘release-and-patch’ mentality, to more robust expectations of quality in business applications, robotics systems, and so on.
In terms of unit tests vs integration tests. I think unit tests, today, are almost a necessity for library developers – because they cannot anticipate all the possible scenarios where their libraries will be used. But for non-library applications, manual use-case-based integration+usability tests, are a much better investment of programmer’s time.
[1] https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19990089302.pdf [2] https://www.linkedin.com/pulse/lamports-tla-spec-testing-why-youre-using-nira-amit
I, a unit test aficionado, posit that sometimes unit testing is a bad choice for your code. AMA.
I posit that anything, treated uncritically and/or cargo culted, is a bad choice for your code. Terms like “unit test” have a brief lifetime as an interesting new category that may stimulate good new thoughts…after which they beg to be overconstrained, dogmatized, and rendered nothing but clickbait. Just write tests that test the behaviors you care about, as well as you can under the circumstances, as clearly and maintainable as possible—you know, just like the rest of the code—and don’t worry about what you call them. :)
Sure, I thought it was an interesting perspective from someone who prefers unit tests but thought that there was a better way to test the code in question.
I find it’s more important to be clear about the scope of the behavior you’re trying to test than to apply arbitrary terms to describe tests.
For example, if you’re trying to test the runtime behavior of the makefile (essentially using cmake as a library to accomplish the task of the function under test), then yes, that’s not a good test. On the other hand, if the function is supposed to make human-readable makefiles and you want to check the indentation and comments in the makefile, that might be a great way to write that test. The term “unit test” does nothing to assist that analysis.
The point of the article that I think is profoundly important is: only test the thing you’re testing, and avoid accidentally testing things you didn’t mean to test.
I like this way of thinking, thanks for the insight.
I — also a unit test aficionado — posit that unit tests have no inherent value. In fact unit tests are a poor choice for verifying any behaviour that could otherwise be verified by cheaper means, e.g., types, property based testing, etc.
Property testing is a form of unit testing, isn’t it?
Technically speaking, “unit test” is about scope while “property based test” is about method and generativity. Seen this way, most PBTs are unit tests.
In practice, people usually mean “unit test” to mean manual oracle unit tests, and PBT to mean generative property tests of all scopes (but usually unit).
It’s not either/or though. Sometimes types can’t get you there. Sometimes contracts can’t get you there. Even when they can, code can be buggy anyway. My first and only Haskell project compiled and passed all tests. And yet, it didn’t work at all as intended.
Certainly. And I’m in no way suggesting that types will do everything and that they obviate the need for tests.
The reason for my — admittedly rather provocative — statement, is because it seems a prevalent idea in the industry (especially in the Software Craftsmanship community) that if unit tests aren’t your panacea, then you just aren’t unit testing hard enough. I think this is a damaging idea, and it deserves some pushback.