First of all, this conventional wisdom assumes that there is a single universal definition of “better”; but there isn’t one. If you work on a project as a team–which is almost all of us–it’s really hard to get everyone to agree on something; be it the definition of clean code, or even the type of whitespace to be used for indentation.
Having a really small team size (N) makes consensus easier. It amazes me how much process is compensation for having more than a couple of people (at most) work on something. The failure mode for a small team is bus factor. The failure mode for larger teams is silent accumulation of technical debt due to lack of consensus and coordination.
I don’t think there’s a neat solution to this problem, but I do notice that many problems go away as N approaches 1.
Which is why high productivity is so incredibly important. Get the highest productivity people (see Brooks’s chief programmer teams) and do everything to help them be more productive.
The failure mode for a small team is bus factor. The failure mode for larger teams is silent accumulation of technical debt due to lack of consensus and coordination.
I’m not sure that this concept has ever been made as clear as this quote. While not particularly a new observation, the profundity of this quote is one of those that I feel will grow to be just as important over time as Knuth’s famous quote on optimization.
Stopping all other development to make a code base consistent implies telling all developers that are not involved in the change to wait. The larger the team, the more expensive this is.
This would imply that working on software without irreversibly accumulating tech dept is impossible with a large team. Maybe that is actually a pretty fundamental truth.
Thanks for that link, I see a lot of value in this idea!
However, this assumes a team of max 5 to 7 people (10 assistants/facilitators for a single lead engineer seems overkill) and single simultaneous focus.
I currently work in a company with about a 100 engineers working on a single piece of software. Although there are extremely high code quality standards, all the latest CI practices and rigurous review processes, the issue described in the original article is clearly present, the code base is in constant flux and various global concepts are being applied simultaneously.
And I don’t see any other possible approach with a team this size.
That would be ideal, however you probably want to see a level of conformance to the same code conventions across all modules or even across all products the company provides, for example in order to make it easier for new hires to work on several parts of the product or on several related products at once. And that takes you back to where you started: multiple simultaneous gradual and never quite complete implementation of global refactorings.
On the other hand, perhaps conformance to conventions across modules and products to facilitate flexible employability is an ambition you could choose to forfeit in order to have a way to prevent tech debt from accumulating. I wonder if there is a way to measure some kind of optimum here.
One argument in favour of the boy Scout rule not brought up is that code that is being changed tends to be code that is going to change. Some parts of the code are just changed very often. By cleaning things up in the part of the code you visit, you also automatically clean up the code exactly where it will need to be clean in the future.
Seconding. It reminds me of the book Software Design X-Rays, which shows you how to examine your codebase via its commit history in order to find those hotspots most worth refactoring.
At my last team I used the “paying down credit card debt” analogy. You allocate a portion of your budget to it (hopefully enough to avoid the balance increasing), but you also still spend money on food every month.
The cynical view of this is that you kind of need to rate limit “fixing all the things” because software developers often conflate “technical debt” with “> 6 months old and not using the latest framework or pattern from my favorite blog”. The questions I usually use to guide these conversations are:
What quantifiable pain does $X cause to us in terms of velocity or outages or security exposure?
If we take action $Y, what leads us to believe that the pain will lessen?
So basically you have to convince me there’s a problem, and you have to convince me the recommended solution will actually fix the problem and that we can expect it to be cost effective.
A good followup to “Things you should never do” is Lessons from 6 software rewrite stories, which is about the ways some companies are able to do complete rewrites.
“do it in one fell swoop” I think is often so unpractical that I would not consider this advice for anything but the smallest of code bases.
Adopting habits as a team (and individuals) that continuously improve a code base is the way to go in my experience. Yes it might not be obvious or easy to agree on those habits, but it’s doable. Making things better as you go is indeed a moving target and never ending endeavor, but that is the nature of living, used software/code where requirements/features evolve.
I disagree with the first point - that there is no universal “better”. There are objectively worse and better things but the examples in the post didn’t mention them.
The article says, for example, that someone can prefer one kind of indentation and someone else might prefer another. But the difference itself is not essential in this case, that’s why there is no better or worse. Going back to scouts - its like arguing whether to place the fire place on the left side of the road or on the right side of the road. It’s hard to see why one should be preferred over others. And so there is no “better” or “worse”. But there are different set of situations and decisions. Like: should we put out the fire before leaving the place? Should we pack the trash? This is where “better” and “worse” is clear.
This also transfers to coding. And its not only about clear cases like bugs or new features. If you improve the execution speed of some process without changing any of its behaviour - it is for the better. If you identify a potential memory leak and fix it from happening - that is better. If you see that a variable for something has one name in one file and another name in another file, fixing them to be the same will end in better codebase. If you write a documentation for a part where it was missing… etc.
This “nothing can be objectively better than anything else” can only logically end with not having any code at all because who can say that any code is better than no code.
At larger companies, typically you do have some people working on specific cleanups while others work on features. And even in a monorepo you don’t typically do global cleanups in one commit. But to avoid inconsistency you do want each cleanup to finish, so don’t start on a big cleanup unless you have the time and buy-in from management to finish the job.
I like to perform really large changes (eg major version framework upgrades) by writing a one-off script to do it. Avoids generating merge conflicts / keeping long lived feature branches.
This makes sense conventionally; if you make a small part better consistently, the whole will get better in the end. In this post, I will challenge that.
Don’t bother challenging that. The accumulation of improvements is not worse than the unimproved.
It’s true, as is the converse. Which is why the prospect of deliberately not cleaning up after ourselves – globally or locally – is a silly one. Big, old, complex code bases are difficult to maintain in the best of worlds and refactoring them only in large swaths is economically infeasible for what I assume is the majority of teams.
I like to incur the cost of rewrites on each feature added/worked (maybe full, but mostly partial). All the teams that I have worked fear rewrites and justify such fear with the uncertainty and doubt that our common sense typically deduces from them (or even claimed past experiences - which is rare). Personally I love doing rewrites and have had great success in my personal projects. Rewrites are a driving force. Agility in doing rewrites is a great skill to have but unfortunately commonly frowned upon.
Having a really small team size (N) makes consensus easier. It amazes me how much process is compensation for having more than a couple of people (at most) work on something. The failure mode for a small team is bus factor. The failure mode for larger teams is silent accumulation of technical debt due to lack of consensus and coordination.
I don’t think there’s a neat solution to this problem, but I do notice that many problems go away as N approaches 1.
Two might be the sweet spot.
Which is why high productivity is so incredibly important. Get the highest productivity people (see Brooks’s chief programmer teams) and do everything to help them be more productive.
I’m not sure that this concept has ever been made as clear as this quote. While not particularly a new observation, the profundity of this quote is one of those that I feel will grow to be just as important over time as Knuth’s famous quote on optimization.
Stopping all other development to make a code base consistent implies telling all developers that are not involved in the change to wait. The larger the team, the more expensive this is.
This would imply that working on software without irreversibly accumulating tech dept is impossible with a large team. Maybe that is actually a pretty fundamental truth.
https://wiki.c2.com/?SurgicalTeam
Thanks for that link, I see a lot of value in this idea!
However, this assumes a team of max 5 to 7 people (10 assistants/facilitators for a single lead engineer seems overkill) and single simultaneous focus.
I currently work in a company with about a 100 engineers working on a single piece of software. Although there are extremely high code quality standards, all the latest CI practices and rigurous review processes, the issue described in the original article is clearly present, the code base is in constant flux and various global concepts are being applied simultaneously.
And I don’t see any other possible approach with a team this size.
Since scaling pressure of team size impacts systems, we should probably architect to reduce the size of the teams we need. More modularization.
That would be ideal, however you probably want to see a level of conformance to the same code conventions across all modules or even across all products the company provides, for example in order to make it easier for new hires to work on several parts of the product or on several related products at once. And that takes you back to where you started: multiple simultaneous gradual and never quite complete implementation of global refactorings.
On the other hand, perhaps conformance to conventions across modules and products to facilitate flexible employability is an ambition you could choose to forfeit in order to have a way to prevent tech debt from accumulating. I wonder if there is a way to measure some kind of optimum here.
One argument in favour of the boy Scout rule not brought up is that code that is being changed tends to be code that is going to change. Some parts of the code are just changed very often. By cleaning things up in the part of the code you visit, you also automatically clean up the code exactly where it will need to be clean in the future.
Seconding. It reminds me of the book Software Design X-Rays, which shows you how to examine your codebase via its commit history in order to find those hotspots most worth refactoring.
Interestingly this runs very counter to some common wisdom. See “Things you should never do” by Joel https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
At my last team I used the “paying down credit card debt” analogy. You allocate a portion of your budget to it (hopefully enough to avoid the balance increasing), but you also still spend money on food every month.
The cynical view of this is that you kind of need to rate limit “fixing all the things” because software developers often conflate “technical debt” with “> 6 months old and not using the latest framework or pattern from my favorite blog”. The questions I usually use to guide these conversations are:
So basically you have to convince me there’s a problem, and you have to convince me the recommended solution will actually fix the problem and that we can expect it to be cost effective.
This is a very old post at this point, and I think some historical context about Fog Creek is worth pointing out:
Because of Joel’s inflexible position on this, Fog Creek ended up CREATING THEIR OWN PROGRAMMING LANGUAGE and then porting the whole codebase to C# with their hand-rolled compiler.
I don’t consider that to be any less risky than a full-scale rewrite.
A good followup to “Things you should never do” is Lessons from 6 software rewrite stories, which is about the ways some companies are able to do complete rewrites.
exactly.
For the above reasons, the original post is IMO poor advice.
“do it in one fell swoop” I think is often so unpractical that I would not consider this advice for anything but the smallest of code bases.
Adopting habits as a team (and individuals) that continuously improve a code base is the way to go in my experience. Yes it might not be obvious or easy to agree on those habits, but it’s doable. Making things better as you go is indeed a moving target and never ending endeavor, but that is the nature of living, used software/code where requirements/features evolve.
I disagree with the first point - that there is no universal “better”. There are objectively worse and better things but the examples in the post didn’t mention them.
The article says, for example, that someone can prefer one kind of indentation and someone else might prefer another. But the difference itself is not essential in this case, that’s why there is no better or worse. Going back to scouts - its like arguing whether to place the fire place on the left side of the road or on the right side of the road. It’s hard to see why one should be preferred over others. And so there is no “better” or “worse”. But there are different set of situations and decisions. Like: should we put out the fire before leaving the place? Should we pack the trash? This is where “better” and “worse” is clear.
This also transfers to coding. And its not only about clear cases like bugs or new features. If you improve the execution speed of some process without changing any of its behaviour - it is for the better. If you identify a potential memory leak and fix it from happening - that is better. If you see that a variable for something has one name in one file and another name in another file, fixing them to be the same will end in better codebase. If you write a documentation for a part where it was missing… etc.
This “nothing can be objectively better than anything else” can only logically end with not having any code at all because who can say that any code is better than no code.
At larger companies, typically you do have some people working on specific cleanups while others work on features. And even in a monorepo you don’t typically do global cleanups in one commit. But to avoid inconsistency you do want each cleanup to finish, so don’t start on a big cleanup unless you have the time and buy-in from management to finish the job.
I like to perform really large changes (eg major version framework upgrades) by writing a one-off script to do it. Avoids generating merge conflicts / keeping long lived feature branches.
Don’t bother challenging that. The accumulation of improvements is not worse than the unimproved.
I think what they’re actually trying to get at is that local improvements can be global detriments.
See also: microservices.
It’s true, as is the converse. Which is why the prospect of deliberately not cleaning up after ourselves – globally or locally – is a silly one. Big, old, complex code bases are difficult to maintain in the best of worlds and refactoring them only in large swaths is economically infeasible for what I assume is the majority of teams.
I like to incur the cost of rewrites on each feature added/worked (maybe full, but mostly partial). All the teams that I have worked fear rewrites and justify such fear with the uncertainty and doubt that our common sense typically deduces from them (or even claimed past experiences - which is rare). Personally I love doing rewrites and have had great success in my personal projects. Rewrites are a driving force. Agility in doing rewrites is a great skill to have but unfortunately commonly frowned upon.