First, automated regression testing is not new. The first reference to it in the literature that I’m aware of is from the System/360 project.
Second, the goal is to establish and maintain some proof of correctness. For somethings like visual design, manual testing is the best because correctness is all about the human’s interaction with the artifact. Type systems provide tools for correctness, as do linters. Tests and model checkers also provide data about correctness, but instead of deductive proof, it’s inductive proof. This is what scientists do for everything. We don’t work with a statistical view of the results. We assert a logical statement based on the test results. If it turns out our assertion was wrong, we go back and learn how our practice of selecting cases failed, but I, at least, use the propositions I infer from the results of tests the same way I use compiler errors or lint failures or model checkers.
You must reverse-engineer the existing code, understanding not just what it does but why it does it.
I find one way to preserve intellectual control over code is to write detailed comments that explain the overall design, why it was chosen, implications, etc. And this applies both in the large (overall system/module design) but also in the small (a function implementation, etc). These days its not uncommon for the source files that I write to be 50% comments. While this requires effort and discipline, the result is invaluable when you revisit the code after a couple of years break and have only the fizziest recollection of what might be going on.
One trick I have here is forcing myself to write file-level comments. Function-level comments do not always help: they explain what the function does, sometimes tell you why it exists, but rarely discuss whether it should exist in the first place. When commenting any specific fragment of code, the mind-state naturally gravitates to describing the code, not the context.
At the file level, you don’t have any immediate code to attach to, so it’s usually easier to trick yourself into writing about the surrounding context and high-level purpose, rather than the low-level minutia.
Interesting. I do sometimes do this to provide module-level comments in Haddocks, but don’t routinely go to this level. I have recently also become a big fan of GHC-style notes as a way to have cross-referenceable pieces of documentation that aren’t a function- or file-level comment.
I’ve also gravitated to writing lots of file-level comments without explicitly realize that’s what I’ve been doing. You should write a blogpost about it!
It seems that today, with all of our tests, we allow the code to grow woolly and complicated in ways that, in the past, with no automated regression tests, we never could have tolerated. We were forced to keep the code simple because if we didn’t, the complexity of the special cases wouldn’t fit in our heads.
We were also forced to keep code simple because the computers of the past had much less memory and disk space than today. My first computer came with 16K of RAM (16,384 bytes). That’s not much for a program of any complexity. Even 64K (65,536 bytes) isn’t that much in the grand scheme of things (if, on average, an instruction takes 2 bytes, then a program can only ever have 32,767 instructions, with no space left over for any data—the editor I use, not large by today’s standards, has over 116,000 instructions and takes over 350,000 bytes just for the instructions).
Also, computers were a lot slower back in the day. I recall the breathless announcements when computers hit 33MHz. If you think your tests are slow today, they would have been glacial back then.
All this to say, we were forced to keep the code simple because the computers at the time were simpler.
I don’t think the correlation is so clear-cut. When we had 16k of RAM, we had to do manual memory management, didn’t we? Isn’t being able to write list.append("∀βcd∃") simpler than having to implement a linked-list for a char[] (and figuring out how to represent non-ASCII characters)?
True, somethings are easier now, like spell checking used to be a major engineering feat and now it’s just a few lines. Supporting non-ASCII is either easy (because of UTF-8) or insanely complex depending on what we want to do with said characters. It used to be one could use a simple text editor [1] for programming, and now we expect way more from our programming environments than what could be done in the past.
Just how much code resides behind the simple list.append() and network.connect() calls?
[1] My first real editor I used was 40k in size. It was programmable. It could handle (very slowly) files larger than memory. And it was version 1.0. Limitations? Sure—lines were limited to 255 characters, and had to end with CRLF. But I never did find a bug in it.
While I hate the modern bloat, e.g. Thinking Forth e.g. defends strong testing in a simple and modern paradigm, so I think the author goes too far. Testing is perhaps a crutch, but not the cause for poor code organization etc.
I’m not sold on the claim in the title, but this notion of intellectual control aptly names what we’re fighting for when we simplify. This is a heuristic for how easy the software is to change and how confident we can be that it works.
I’m not sure I buy this.
First, automated regression testing is not new. The first reference to it in the literature that I’m aware of is from the System/360 project.
Second, the goal is to establish and maintain some proof of correctness. For somethings like visual design, manual testing is the best because correctness is all about the human’s interaction with the artifact. Type systems provide tools for correctness, as do linters. Tests and model checkers also provide data about correctness, but instead of deductive proof, it’s inductive proof. This is what scientists do for everything. We don’t work with a statistical view of the results. We assert a logical statement based on the test results. If it turns out our assertion was wrong, we go back and learn how our practice of selecting cases failed, but I, at least, use the propositions I infer from the results of tests the same way I use compiler errors or lint failures or model checkers.
Nice article.
I find one way to preserve intellectual control over code is to write detailed comments that explain the overall design, why it was chosen, implications, etc. And this applies both in the large (overall system/module design) but also in the small (a function implementation, etc). These days its not uncommon for the source files that I write to be 50% comments. While this requires effort and discipline, the result is invaluable when you revisit the code after a couple of years break and have only the fizziest recollection of what might be going on.
One trick I have here is forcing myself to write file-level comments. Function-level comments do not always help: they explain what the function does, sometimes tell you why it exists, but rarely discuss whether it should exist in the first place. When commenting any specific fragment of code, the mind-state naturally gravitates to describing the code, not the context.
At the file level, you don’t have any immediate code to attach to, so it’s usually easier to trick yourself into writing about the surrounding context and high-level purpose, rather than the low-level minutia.
Interesting. I do sometimes do this to provide module-level comments in Haddocks, but don’t routinely go to this level. I have recently also become a big fan of GHC-style notes as a way to have cross-referenceable pieces of documentation that aren’t a function- or file-level comment.
In OpenZFS we call these “grand theory” comments.
I’ve also gravitated to writing lots of file-level comments without explicitly realize that’s what I’ve been doing. You should write a blogpost about it!
We were also forced to keep code simple because the computers of the past had much less memory and disk space than today. My first computer came with 16K of RAM (16,384 bytes). That’s not much for a program of any complexity. Even 64K (65,536 bytes) isn’t that much in the grand scheme of things (if, on average, an instruction takes 2 bytes, then a program can only ever have 32,767 instructions, with no space left over for any data—the editor I use, not large by today’s standards, has over 116,000 instructions and takes over 350,000 bytes just for the instructions).
Also, computers were a lot slower back in the day. I recall the breathless announcements when computers hit 33MHz. If you think your tests are slow today, they would have been glacial back then.
All this to say, we were forced to keep the code simple because the computers at the time were simpler.
I don’t think the correlation is so clear-cut. When we had 16k of RAM, we had to do manual memory management, didn’t we? Isn’t being able to write
list.append("∀βcd∃")simpler than having to implement a linked-list for achar[](and figuring out how to represent non-ASCII characters)?True, somethings are easier now, like spell checking used to be a major engineering feat and now it’s just a few lines. Supporting non-ASCII is either easy (because of UTF-8) or insanely complex depending on what we want to do with said characters. It used to be one could use a simple text editor [1] for programming, and now we expect way more from our programming environments than what could be done in the past.
Just how much code resides behind the simple
list.append()andnetwork.connect()calls?[1] My first real editor I used was 40k in size. It was programmable. It could handle (very slowly) files larger than memory. And it was version 1.0. Limitations? Sure—lines were limited to 255 characters, and had to end with CRLF. But I never did find a bug in it.
While I hate the modern bloat, e.g. Thinking Forth e.g. defends strong testing in a simple and modern paradigm, so I think the author goes too far. Testing is perhaps a crutch, but not the cause for poor code organization etc.
I’m not sold on the claim in the title, but this notion of intellectual control aptly names what we’re fighting for when we simplify. This is a heuristic for how easy the software is to change and how confident we can be that it works.