I like to say that if your automated tests (I avoid the term “unit” because I worry it implies mocking and suchlike) are slowing you down rather than speeding you up, you need to rethink how you are writing them.
Done right, automated tests are the biggest productivity boost for software development I know of.
I like to say that if your automated tests […] are slowing you down rather than speeding you up, you need to rethink how you are writing them.
Writing tests is almost always additional effort in the short term. There are some exceptions: Things that are very well-suited for automated tests like parsers often immediately benefit from a good test suite, and this is also where test driven development can make sense. Either way, automated tests often pay off in the long term because they increase the stability and robustness of the code and allow you to recognize mistakes earlier, but when you have to ship a feature or a bugfix as soon as possible, they will delay you. This tradeoff is important to keep in the back of your mind when you organize and plan your tasks.
In an individual project, I agree that it increases effort but in a collaborative project I think it reduces it slightly. Code reviewer time is generally a more scarce resource than code author time. When I review a PR that has tests, I don’t need to think about whether the code is correct for the cases that are tested. I can look at the tests and ask ‘is this actually the right answer?’ And ‘are there important cases that are not tested?’ Either of these may result in a request to add more tests. After that, I can focus on the things that actually benefit from a human: is this a sensible implementation (is it accidentally quadratic?), is it sufficiently documented that someone else will be able to understand it in a year’s time, and so on. This significantly reduces code reviewer load.
The additional effort can be pretty tiny if you already have a good test harness and fixture set in place. For a lot of my projects the tests genuinely only take a few minutes to update when I tweak an existing feature.
The trick is to learn the right habits and bake them into the project from as early as possible.
Writing tests is almost always additional effort in the short term.
Additional effort compared to what? No reasonable project would merge a PR that didn’t include tests. Not necessarily 100% coverage, of course — but at least 50% coverage, certainly.
when you have to ship a feature or a bugfix as soon as possible, [tests] will delay you.
At absolute worst, including tests will ~double the time to resolution. Is that delay ever actually significant? I’ve never once seen a situation where that would represent a problem.
automated tests often pay off in the long term because they increase the stability and robustness of the code and allow you to recognize mistakes earlier
Agreed! But “the long term” is measured in hours/days, not weeks/months. Code without tests is unreliable, impossible to refactor with confidence, and so (broadly) unmaintainable. Testability is the best proxy for quality.
I find your reasoning more religious than based on reason. No offense. A couple of things sound like unfounded assertions.
At absolute worst, including tests will ~double the time to resolution.
At absolut worst? I would say that in the world of business applications that essentially expose databases, not even at absolute best. By the time the developer is ready with mocks and other non sense, he already wrote more code than the subject of the test. In my experience, the typical PR includes 25% of code if even that mutch.
Code without tests is unreliable, impossible to refactor with confidence
You mean like the tests themselves? Where are the tests for your tests? If they are so critical, why are you so confident of their code but not of the code they are testing? It is a serious question.
I suspect the reason why people rave about tests is mostly because they work with large/entangled systems plagged with poor separation of concerns, specially IO and non local state, where it becomes impossible to just run any function against a test vector. Test is a big del for them because of the insane investment it requires.
If you use a functional programming language, or of you are the author of, say a compiler, or a library that implements a bunch of algorithms, testing becomes a matter of just calling your code, and is just a part of the developer workflow. They don’t need to defend it nor motivate it, they just do it all day effortless.
Integration tests are a gift to your customers (they break before you break the customer).
Unit tests are a gift to future maintainers (they don’t just say “something broke” they tell you where in the code to look and hints about uses and edge cases that might not be top of mind while refactoring or adding new features).
This is a wonderful explanation of Zigs way of thinking:
Make it easy to do the right thing, inteoduce friction on the bad paths (but dont forbid to use them).
This manifests in making unit testing easier than actually building a program, because you also have an allocator ready to use (which even does leak checking!).
Also theres a magic tool for UTs that can make allocations that will happen in a path also fail,so you can easily check for memleaks in all paths
The first job I ever had was developing mobile games in Unity3D at a completely new company with no pre-established human capital. Even though there was a senior programmer (the only other programmer), he gave little guidance because he had no experience with Unity3D whatsoever. The experience was hellish in so far as we failed deadline after deadline after deadline. I did 90% of the code and the problem I faced repeatedly was that it was so easy to make a change that would break something, either the code, or break expectations about how the scene was set-up.
The more code there was, the more it felt extremely dangerous to change anything and it would require a significant amount of manual testing to confirm that nothing was broken. I asked the senior programmer whether unit tests were an important thing that I should start doing and his response was ambivalent. Combined with the pressure I was under, I don’t recall ever writing a unit test while on the job.
After being miserable for an extended period of time, I quit that job.
Later, while polishing my skills for the next job, I wrote my first unit test. The day I started believing in unit tests was the day I wrote my first unit test. It became extremely obvious that unit testing was the answer to my previous painful struggles. It helped me form an intuitive understanding of when unit testing is most likely to defend against regressions.
For me personally unit tests usually pay for themselves in increased velocity within an hour. I don’t know if I’m just that sloppy or unit testing is just that effective, but I can tell you I have been on incidents where downtime cost 6-7 figures per minute and I took time to:
reproduce an issue
Write a test for it
Verify the test failed
fix code
Verify the test passed
Before pushing a new release. Because if you feel like you don’t have time to write a test you certainly don’t have time to deploy twice.
Then I moved to Google and found python code with straight up misspelled property names and unit tests that only passed in new York City (guess what time zone the main developer that feature lived in).
The dumbest possible tests will catch an unbelievable amount of mistakes. And I don’t trust any assumptions I can’t programmatically verify.
Graphics is an area where meaningful unit tests are quite hard to write, so while I generally find unit tests very useful, they don’t work as well in every area.
Early in my career I simply didn’t understand people who couldn’t see the value in unit tests, and I thought they were wrong. I took pride in using some techniques from “working effectively with legacy code” and managed to get a few hairy “untestable” things testable.
After a few more years, I still don’t understand people who don’t see the value in unit tests, but I’m willing to give them the benefit of the doubt that not all types of software development are the same. Some of these programs needlessly rely on one existing cohesive implementation of an API, but I’ve come to realize that creating your own inner platform around that API just for testing can sometimes be the wrong choice. I’m humble enough to admit that I’ve made some projects worse by adding interfaces for testing that had a negative ROI.
That being said, one of the biggest benefits of unit tests is they can be self contained runnable examples of how to use your API. The author found that having those examples is helpful because they can highlight problems. I’ve seen people dismissive of useless tests that “just create an object” or even “just import a module”, but in a dynamic language like python the value of those tests is definitely far from zero.
Many languages, from python to go, actually support a flavor of unit test that takes this to the next level via doctests and go example tests.
i have started at my current workplace as regular end-to-customer support engineer and there was basically no automated testing at all, ~11 years ago. Our QA “team” was doing manual tests, and given the amount of the functionality and third party products we integrated to, it was nearly impossible to have 100% test coverage. Sadly, the development team were some really old-school C guys (and so are today) and didn’t spend a dime on thinking about tests, or writing their own unit/integration tests. As today the code base is a real C monlith with code lines beeing almost 30 years old. It was common sense to ship the software with grave bugs “we cant test everything we have too less manpower”
And as support engineer, i suffered. Great times. If it wasnt for the nice workplace, i would have quit my job. It was so obvious that, if we had a basic amount of integration tests available, we would have to go through so much less trouble and losing customers.
The last 3 years i have quit the support team and worked on a CI pipeline, which now has a cloc of about 22188 lines of python code. As our software also deeply integrates into hardware devices (tape drives) we cant simply use docker for pristine test-environments, but settled with vagrant/vagrant-libvirt to spin up virtual machines for all our supported operating systems. It is integrated into our jenkins release pipeline and spans about ~50 virtual machines for each build, taking about ~3 hours to finish after each build, reporting included.
It has been a blast since then, especially if we do big transitions (as happened from python2.x to python3) my framework has reported so much troubles what would have been shipped to our customers a few years prior.
The hard part was, and still is, teaching our old-school developers to adopt the workflow, but its doing better each year that passes. Also, it is a great deal about COMMUNICATION, its so often that our developers tend to re-work functionality, which is not clearly communicated and then tests fail, which would have been adopted (in an ideal world, by the developers themselfes). Test driven development just seems hard to each for old school devs.
For non-library code (by library I mean something C++ Boost), what class of coding problems a hand-developed unit-test can catch that a hand-developed automated non-unit test cannot catch?
Edge cases: some conditions are easier to trigger from within the system. For example, in TigerBeetle we have a grid of 128kb blocks on disk, and a bit set-based in-memory index of free blocks, which uses something like 1.0001 bits per on-disk block. This bitset data structures had some funky bugs, which are only triggered for relatively large data files, and only for particular usage patterns. Triggering this bug in an integrated test is 128000 costlier than in a unit test which tests free-floating free set in isolation, where you don’t actually have blocks backing the bits.
Another case is when a bug can be caught by an integrated test, but it’s just less scaffolding to poke at the relevant code from inside. This most often happens in parsers, which have simple interface, but are often hidden in the guts of the system. As another TigerBeetle example, we allow passing ip addresses on the command line. You could test ip-parsing code by exercising the entire parser, but it’s just simpler to slap a bunch of table-driven unit test on top of fn parseIP.
I like to say that if your automated tests (I avoid the term “unit” because I worry it implies mocking and suchlike) are slowing you down rather than speeding you up, you need to rethink how you are writing them.
Done right, automated tests are the biggest productivity boost for software development I know of.
Writing tests is almost always additional effort in the short term. There are some exceptions: Things that are very well-suited for automated tests like parsers often immediately benefit from a good test suite, and this is also where test driven development can make sense. Either way, automated tests often pay off in the long term because they increase the stability and robustness of the code and allow you to recognize mistakes earlier, but when you have to ship a feature or a bugfix as soon as possible, they will delay you. This tradeoff is important to keep in the back of your mind when you organize and plan your tasks.
In an individual project, I agree that it increases effort but in a collaborative project I think it reduces it slightly. Code reviewer time is generally a more scarce resource than code author time. When I review a PR that has tests, I don’t need to think about whether the code is correct for the cases that are tested. I can look at the tests and ask ‘is this actually the right answer?’ And ‘are there important cases that are not tested?’ Either of these may result in a request to add more tests. After that, I can focus on the things that actually benefit from a human: is this a sensible implementation (is it accidentally quadratic?), is it sufficiently documented that someone else will be able to understand it in a year’s time, and so on. This significantly reduces code reviewer load.
The additional effort can be pretty tiny if you already have a good test harness and fixture set in place. For a lot of my projects the tests genuinely only take a few minutes to update when I tweak an existing feature.
The trick is to learn the right habits and bake them into the project from as early as possible.
Additional effort compared to what? No reasonable project would merge a PR that didn’t include tests. Not necessarily 100% coverage, of course — but at least 50% coverage, certainly.
At absolute worst, including tests will ~double the time to resolution. Is that delay ever actually significant? I’ve never once seen a situation where that would represent a problem.
Agreed! But “the long term” is measured in hours/days, not weeks/months. Code without tests is unreliable, impossible to refactor with confidence, and so (broadly) unmaintainable. Testability is the best proxy for quality.
I find your reasoning more religious than based on reason. No offense. A couple of things sound like unfounded assertions.
At absolut worst? I would say that in the world of business applications that essentially expose databases, not even at absolute best. By the time the developer is ready with mocks and other non sense, he already wrote more code than the subject of the test. In my experience, the typical PR includes 25% of code if even that mutch.
You mean like the tests themselves? Where are the tests for your tests? If they are so critical, why are you so confident of their code but not of the code they are testing? It is a serious question.
I suspect the reason why people rave about tests is mostly because they work with large/entangled systems plagged with poor separation of concerns, specially IO and non local state, where it becomes impossible to just run any function against a test vector. Test is a big del for them because of the insane investment it requires. If you use a functional programming language, or of you are the author of, say a compiler, or a library that implements a bunch of algorithms, testing becomes a matter of just calling your code, and is just a part of the developer workflow. They don’t need to defend it nor motivate it, they just do it all day effortless.
[Comment removed by author]
Integration tests are a gift to your customers (they break before you break the customer).
Unit tests are a gift to future maintainers (they don’t just say “something broke” they tell you where in the code to look and hints about uses and edge cases that might not be top of mind while refactoring or adding new features).
Not just to the customers. We have been expanding our integration tests lately and we see benefits across the board:
Downsides are:
I like “The Three Questions” mentioned on the page, that one should ask oneself when a bug happens:
In Zig it’s easier to write tests than to Run It And See, or print/log: just put a
test "it does the thing" { ... }next to the function.And because I’m lazy and do things the easy way, I have lots of tests now.
This is a wonderful explanation of Zigs way of thinking: Make it easy to do the right thing, inteoduce friction on the bad paths (but dont forbid to use them).
This manifests in making unit testing easier than actually building a program, because you also have an allocator ready to use (which even does leak checking!).
Also theres a magic tool for UTs that can make allocations that will happen in a path also fail,so you can easily check for memleaks in all paths
Very similar to what Next Generation Shell does. It just made sense to place the tests adjacent to the methods they test.
https://github.com/ngs-lang/ngs/blob/eb176b16d5095af2d1324ba2c5079ba818e47e51/lib/stdlib.ngs
The first job I ever had was developing mobile games in Unity3D at a completely new company with no pre-established human capital. Even though there was a senior programmer (the only other programmer), he gave little guidance because he had no experience with Unity3D whatsoever. The experience was hellish in so far as we failed deadline after deadline after deadline. I did 90% of the code and the problem I faced repeatedly was that it was so easy to make a change that would break something, either the code, or break expectations about how the scene was set-up.
The more code there was, the more it felt extremely dangerous to change anything and it would require a significant amount of manual testing to confirm that nothing was broken. I asked the senior programmer whether unit tests were an important thing that I should start doing and his response was ambivalent. Combined with the pressure I was under, I don’t recall ever writing a unit test while on the job.
After being miserable for an extended period of time, I quit that job.
Later, while polishing my skills for the next job, I wrote my first unit test. The day I started believing in unit tests was the day I wrote my first unit test. It became extremely obvious that unit testing was the answer to my previous painful struggles. It helped me form an intuitive understanding of when unit testing is most likely to defend against regressions.
For me personally unit tests usually pay for themselves in increased velocity within an hour. I don’t know if I’m just that sloppy or unit testing is just that effective, but I can tell you I have been on incidents where downtime cost 6-7 figures per minute and I took time to:
Before pushing a new release. Because if you feel like you don’t have time to write a test you certainly don’t have time to deploy twice.
Then I moved to Google and found python code with straight up misspelled property names and unit tests that only passed in new York City (guess what time zone the main developer that feature lived in).
The dumbest possible tests will catch an unbelievable amount of mistakes. And I don’t trust any assumptions I can’t programmatically verify.
Graphics is an area where meaningful unit tests are quite hard to write, so while I generally find unit tests very useful, they don’t work as well in every area.
Early in my career I simply didn’t understand people who couldn’t see the value in unit tests, and I thought they were wrong. I took pride in using some techniques from “working effectively with legacy code” and managed to get a few hairy “untestable” things testable.
After a few more years, I still don’t understand people who don’t see the value in unit tests, but I’m willing to give them the benefit of the doubt that not all types of software development are the same. Some of these programs needlessly rely on one existing cohesive implementation of an API, but I’ve come to realize that creating your own inner platform around that API just for testing can sometimes be the wrong choice. I’m humble enough to admit that I’ve made some projects worse by adding interfaces for testing that had a negative ROI.
That being said, one of the biggest benefits of unit tests is they can be self contained runnable examples of how to use your API. The author found that having those examples is helpful because they can highlight problems. I’ve seen people dismissive of useless tests that “just create an object” or even “just import a module”, but in a dynamic language like python the value of those tests is definitely far from zero.
Many languages, from python to go, actually support a flavor of unit test that takes this to the next level via doctests and go example tests.
i have started at my current workplace as regular end-to-customer support engineer and there was basically no automated testing at all, ~11 years ago. Our QA “team” was doing manual tests, and given the amount of the functionality and third party products we integrated to, it was nearly impossible to have 100% test coverage. Sadly, the development team were some really old-school C guys (and so are today) and didn’t spend a dime on thinking about tests, or writing their own unit/integration tests. As today the code base is a real C monlith with code lines beeing almost 30 years old. It was common sense to ship the software with grave bugs “we cant test everything we have too less manpower”
And as support engineer, i suffered. Great times. If it wasnt for the nice workplace, i would have quit my job. It was so obvious that, if we had a basic amount of integration tests available, we would have to go through so much less trouble and losing customers.
The last 3 years i have quit the support team and worked on a CI pipeline, which now has a cloc of about 22188 lines of python code. As our software also deeply integrates into hardware devices (tape drives) we cant simply use docker for pristine test-environments, but settled with vagrant/vagrant-libvirt to spin up virtual machines for all our supported operating systems. It is integrated into our jenkins release pipeline and spans about ~50 virtual machines for each build, taking about ~3 hours to finish after each build, reporting included.
It has been a blast since then, especially if we do big transitions (as happened from python2.x to python3) my framework has reported so much troubles what would have been shipped to our customers a few years prior.
The hard part was, and still is, teaching our old-school developers to adopt the workflow, but its doing better each year that passes. Also, it is a great deal about COMMUNICATION, its so often that our developers tend to re-work functionality, which is not clearly communicated and then tests fail, which would have been adopted (in an ideal world, by the developers themselfes). Test driven development just seems hard to each for old school devs.
For non-library code (by library I mean something C++ Boost), what class of coding problems a hand-developed unit-test can catch that a hand-developed automated non-unit test cannot catch?
(assuming “unit” here means “small extent” https://matklad.github.io/2022/07/04/unit-and-integration-tests.html)
Edge cases: some conditions are easier to trigger from within the system. For example, in TigerBeetle we have a grid of 128kb blocks on disk, and a bit set-based in-memory index of free blocks, which uses something like 1.0001 bits per on-disk block. This bitset data structures had some funky bugs, which are only triggered for relatively large data files, and only for particular usage patterns. Triggering this bug in an integrated test is 128000 costlier than in a unit test which tests free-floating free set in isolation, where you don’t actually have blocks backing the bits.
Another case is when a bug can be caught by an integrated test, but it’s just less scaffolding to poke at the relevant code from inside. This most often happens in parsers, which have simple interface, but are often hidden in the guts of the system. As another TigerBeetle example, we allow passing ip addresses on the command line. You could test ip-parsing code by exercising the entire parser, but it’s just simpler to slap a bunch of table-driven unit test on top of
fn parseIP.(as a disclaimer, I am generally in the “unit tests are a scam” team https://matklad.github.io/2021/05/31/how-to-test.html)