I did not care for this article, mostly because of the original sin that even the author admits in the beginning:
So I may be arguing with a strawman, but it’s a strawman I have seen deployed with some regularity over the years.
The strawman in question?
“don’t use new or interesting technology, and if you have to, only use a little bit”
Why is this bad?
[…] it is reactionary. At its worst, it borders on anti-intellectualism. I have heard it used on more than one occasion as a thought-terminating cliche to discard a potentially promising new tool.
The problem with arguing against a strawman–and not really talking about the actual thing–is that you can easily miss important context. The context is: innovation tokens are part of a global optimization strategy for a business.
I contend that any stable, successful business–setting aside the mutants evolved to live with delightful plumage and fucked-up metabolisms fueled by the transient funding bloom of ZIRP-driven investment–is inherently reactionary. And that’s fine.
I’ve kicked around a definition on this for a while, going back to old IRC sessions years ago on #lobsters, but in case somebody wants to take the idea and run with it:
“In engineering, a revolutionary approach requires that any technique which might make things better must be entertained. A reactionary approach requires that any technique which might make things worse must be avoided.”
(I would argue that most teams are neither wholly revolutionary or reactionary, but rather liberal and conservative –the less extreme positions, following the normal revolutionary -> liberal -> centrist -> conservative -> reactionary spectrum.)
Bringing this back to the essay, the thing about a business being reactionary (and we’ll handwave slightly and say that business conservatism is similar to the engineering conservatism I mentioned above) is that they seek to aggressively limit surprises. From this, all else follows–you constrain innovation tokens not because you want to be grug but because you do not want to be surprised by a bet that fails to pay off and fucks up things for the rest of the business.
(Ctrl-f in the article for “business”, note zero hits. This is a clue about how strawmanny the article is.)
It’s not operational overhead that innovation tokens seek to limit, as the author suggests; many businesses live just fine with razor-thin margins or comically byzantine processes. It’s that any tech project has some amount of uncertainty built into it, and the more innovation tokens you spend the more variance into expected payoff you introduce.
~
Businesses aren’t totally risk averse–sometimes, the entire business is predicated on a huge risky bet, such as a new drug panning out or an oil deposit actually existing or whatever–but they tend to aggressively limit where they take risks. To wit:
I knew somebody who used to spec out pumps for oil rigs. The thing about oil rigs is that they cost approximately one metric fuckton of dollars per day to run, but that’s okay because they generate about ten fucktons in return (and seriously, if you ever want to get a feeling for how small-potatoes investment in tech actually is, go look at oil and gas). Anyways, he ran the numbers for some fancy turbine pumps for mud or something, and they needed to rent them. Renting each of the pumps was like a hundred thousand dollars per day. Eyewatering. He put in the request, and the oil company said “well shit, might as well have two.” They were already placing such a big bet on the well they were drilling, and the rig was so expensive, it made no sense to have a deadlined pump risk the whole enterprise.
But you know what they didn’t spend tokens on? A better messaging system than email. A better, new project management system. A more ergonomic accounting system instead of GAAP.
~
To give you an idea of how much the article loses the plot:
The author suggests a Haskell microservice as a conservative choice, and then proceeds to talk about all the extra CI and other stuff that that creates and all the overhead, and then lo-and-behold the reveal where we should’ve just written a Haskell monolith all along. And that’s fine, and I even agree with it in that case, but it’s kayfabe. It’s a ruse.
The thing is, if the team already knows Haskell and already has a tool that will solve their problem, they aren’t spending an innovation token. It’s not because of avoiding a boundary, it’s because they know the tools.
The author goes on to crow about how cool they are, how clever, how slick, for eschewing cron in order to avoid a boundary. They then list a pile of questions as though that somehow makes their case–I argue instead that that list of questions is in fact evidence that cron is the correct choice.
Here’s the thing: those questions? They each have clear answers. I can Google or Stack Overflow them if I need to. I can copy-and-paste how they were solved on other projects. The mere fact that we have that list of questions implies a degree of understanding of the trade-offs of using cron, and that is precisely the derisking the business wants to see.
(“But friendlysock, what if they think there’s less risk in their home-rolled solve that only they maintain than there is in using cron?” Well then no shit use that instead. “But you said cron was the correct choice?” Well, you just gave me new information about the risks involved and I updated my position. Welcome to the magical world of project management. “But what if they estimated the risks wrong?” Congrats, you have discovered the ur-problem of managing software projects and developers. )
~
I get that the author probably got told “no” to some fit-for-purpose technology at some point in their career (go look at their about page, it’s pretty obvious this probably came up) by somebody who was too dull to understand it. I definitely have run into folks that will say “no we can’t do this” just because something is new to them, or because they’re heavily invested in their stack. I’ve been there, it grates.
But.
“Shitty stupid midwits can block good tech decisions out of self-interest or ignorance” is an orthogonal problem to “how do we compartmentalize the risks on a project”.
tl,dr; Innovation tokens are not about complexity, they are about compartmentalizing risk–don’t lose the plot.
Edit: happy to talk about or listen to people vent on this topic in DMs. For a professional engagement, I may also be available for the right price.
I agree with the idea that avoiding surprises is very important, and means that sometimes you’ll want to choose a solution precisely because it’s a known quantity. It doesn’t matter how good the new technology actually is, if you can’t reliably estimate what the consequences of using it will be.
(Ctrl-f in the article for “business”, note zero hits. This is a clue about how strawmanny the article is.)
I really think this is not justified. Operational overhead is a business concern.
It was not meant as an idle swipe–the original 2015 article about innovation tokens is rooted in the business case, almost to a painful degree. By contrast, I think that the article here touches on operational overhead but mostly from a perspective of how it makes developers more innovative or happier…and businesses give a shit about neither of those things except as they gate paying off the core gambles they make in the market.
Plenty of successful businesses are more than happy–successful, even, by any measure!_—to make decisions that have lasting operational overhead so long as they themselves don’t have to pay it within the amortization window they care about. Sometimes, they accomplish this most simply by forbidding additional maintenance or by skipping best practices. This is a valid and successful business strategy, if a distasteful technical one.
And really, that’s the core issue I have with the article, and that’s why I pointed out what I did: it focuses too much on a developer perspective of the world, when arguably the whole point of innovation tokens was to enable global optimization across the business. If you aren’t approaching this as fundamentally a business concern, then of course any treatment of innovation tokens will seem ill-fitting at best.
I think that the article here touches on operational overhead but mostly from a perspective of how it makes developers more innovative or happier…and businesses give a shit about neither of those things except as they gate paying off the core gambles they make in the market.
From the article:
Furthermore, some operational overhead is inevitable. You will need to hire people to mitigate it. More popular, more mature projects will have a bigger talent pool to hire from, so your training costs will be lower, and those training costs are part of your operational cost too.
Rationing innovation tokens therefore can work as a reasonable heuristic, or proxy metric, for avoiding a mess of complex operational problems associated with dependencies that are expensive to operate and hard to hire for.
As for the idea that new technology just exists to make developers excited, the article says:
Sometimes, especially in 2024 as opposed to 2015, the point of a new, shiny piece of tech is to address operational issues that the more boring, older one had.
You can argue that’s wrong, you could even try to say it’s a rationalization, but it’s just not true that this is arguing for making developers happy above all else.
for avoiding a mess of complex operational problems associated with dependencies that are expensive to operate and hard to hire for.
Again, operational problems are not what businesses care about, first and foremost. They care about reducing risk and cutting costs, in roughly that order, and the second is arguable. There are lots of successful businesses that take the complicated, hard, operational complexity things–Epic, HFT firms, SAP, whatever–and reduce the risk of doing their stuff to the point where they can reliably bill customers in a way that eats that cost.
If you can spend 10 million to make 10 billion, literally nobody cares if technically you could’ve done it for 8 million at twice the risk of failing. Operational complexity is a red herring. Operational complexity can be bounded and incorporated into the customer price.
it’s just not true that this is arguing for making developers happy above all else.
Look at your quote again–it’s exactly catering to making developers excited, because this time the castle devops tooling won’t sink into the swamp. This time, dear developer, this tooling is here to fix the problems of the last version–never you mind that we’ve probably created additional problems. This churn, it’s necessary, you see.
My experience is that developers–especially the ZIRP mutants–are incredibly precious about their tools and perceived ergonomics, and while one time in fifty this leads to great tooling and perhaps a seismic shift in the Pareto frontier of what they can accomplish, the other forty-nine times it just introduces project risk and distraction. As mentioned previously, though, it’s usually pretty hard to spot which of the two buckets any particular attempt is going to fall into.
Sometimes, especially in 2024 as opposed to 2015, the point of a new, shiny piece of tech is to address operational issues that the more boring, older one had.
You can argue that’s wrong, you could even try to say it’s a rationalization, but it’s just not true that this is arguing for making developers happy above all else.
It seems to be both to me. I don’t understand even in theory what is supposed to have changed from 2015 to 2024. The problem we’re talking about was always a human and cultural one, not a tech one. Developers liking new toys, developers liking resume padding, developers over-engineering solutions… this was never about the new tech itself not living up to the hype. I mean that might be a cherry on top, but it’s incidental to the underlying phenomenon.
Project A will eventually cost about a million dollars and produce value of around $1.1 million.
Project B will eventually cost about a million dollars and produce value of more than $50 million.
What’s immediately apparent is that control is really important for Project A but almost not at all important for Project B. This leads us to the odd conclusion that strict control is something that matters a lot on relatively useless projects and much less on useful projects. It suggests that the more you focus on control, the more likely you’re working on a project that’s striving to deliver something of relatively minor value.
Indeed. I’ve been observing a similar thing in the last few years at work–people are a lot grumpier about stumbling with simple things than with moonshots. If you screw up a microwave burrito you look a lot stupider than if you fail to set a souffle–and you get angrier if your bank were to misplace 5% of an account than if your stock portfolio swung 10% during a dip.
(Also, Tom DeMarco is a great unsung hero in the canon, and I wish he was read more likely–or at least got the same treatment as Brooks in the sense of “everybody references, few read”.)
Thanks! Very germane. McKinley talks about costs and marginal utility. Conversely, DeMarco said, in effect: I wrote a book on measuring and controlling software costs like any other engineering discipline. It doesn’t work. All software projects are more like teenagers: the more control you try to exert, the more it resists, so maybe try Agile? And try not to waste money on the low-value projects.
DeMarco’s advice seems sensible. Mind-changing, even. I might only suggest with regard to that last point, which is the one you quoted, that a low-value project (measuring “value” here like measuring the weather with a wet finger) can be a great place to try out a new tool or pattern precisely because its scope is small.
Great analysis. I actually liked idea of boundary tokens, but a few other specific points I strongly disagreed with:
If there’s a really good library in Haskell and a really good library in Ruby and you really do want to use them both, maybe it makes sense to actually have multiple services.
I feel like the chance of this making sense is effectively zero, and listening to the heuristic the article is trying to debunk is a good shortcut for understanding that.
As your team gets larger and more mature, the need to bring in more tools, and the ability to handle the associated overhead, will only increase over time.
On the contrary, what should increase is your willingness to push back against this growing “need” with all of your might at every opportunity.
But the place that the cost comes in the most is at the boundary between tools, not in the operational deficiencies of any one particular tool.
I feel like this is mostly true, but some tools really are a pain in their own right.
I like the meat of this perspective. Personally, I might have framed it as “innovation tokens are one heuristic, here’s another”.
I think it’s wise to be clear about what is “expected knowledge” for an organization or project and trying to pick solutions that are within or nearby. Introducing new technologies outside of this domain puts a burden on engineers to expand their knowledge in order to effectively maintain or operate the new technology. There’s nothing inherently wrong with that but it’s something that has a cost/benefit tradeoff.
If you have an organization of people very familiar with Go and Postgres and you introduce Temporal, you can add capabilities to your system but you will also have people spending more time and energy on maintenance and operations. You need to make an explicit decision about whether the benefits outweigh the costs. This seems to be what the “boundary token” is getting at and I think that’s really useful. I also find innovation tokens a useful heuristic.
I was going to write this comment, and was pleasantly surprised to find that you’d already written it! =)
In agreement: I don’t think boundary tokens displace innovation tokens. I think they’re orthogonal, and I will use them (or the rough idea) for decision-making in the future. I already had similar rules of thumb in my quiver, like “do whatever it takes to get the team to choose the incumbent language instead of a different one for $NEW_DECISION,” but the boundary idea adds helpful nuance and razoring for those deep discussions. Perhaps the big takeaway is that token-thinking is a family of thought, or, say, a set of lenses, and ye olde Innovation Token was just the first one discovered and popularized.
Dogma like “innovation tokens” is probably what I dislike the most about this field. Evaluating software using shallow properties like “it’s new” is not a good idea and it leads to lazy decision making where a software that could be better in literally every way will not be chosen because it is new. But instead of evaluating it people appeal to the stereotypes listed in the article and more.
much more nuanced in its description of the problem with “innovation” than the subsequent memetic mutation of the concept.
This is a big problem with snappy terms. They mutate into their worst, most memetic form. “Premature optimization is the root of all evil” is a good example but there are endless examples of programmers taking on dogmatic “wisdom” with an uncritical mind.
The post resonates with me. The “at worst it’s anti-intellectualism” is understated, I think. It simply is “anti-intellectual” because it de-facto dismisses critical thinking, instead appealing to a phrase.
That said, I don’t think the result here is good.
rather than “innovation tokens”, consider “boundary tokens”.
How about neither? How about, when you evaluate a technology, you evaluate the technology. You have context on what your team can/ can not do, must do, etc. How will the technology fit into that? Are there risks associated with one approach? Can those be mitigated? If we do it one way here, and another way there, will we pay a price for the inconsistency (the principal advocated in the article)?
Like, what if, and hear me out, people thought about things?
Basically, I think the article couldn’t be more correct on calling out the flawed nature for people to be intellectually lazy and unrigorous, I think the article then comes up with just another quick heuristic that will fail in new ways just like any heuristic will. I guess it’s all heuristics since the question is always “can we predict our success if we choose path X versus path Y”, so I suppose my problem is just using one heuristic. “This thing is really new and different” is one thing to keep in mind, “This thing is really different from our existing systems” is worth keeping in mind. Challenging assumptions about whether a “boring” technology is a good fit is one more thing to do. At the end of the day, the issue is trying to come up with quick answers using simplistic, universal heuristics.
Evaluating software using shallow properties like “it’s new” is not a good idea and it leads to lazy decision making where a software that could be better in literally every way will not be chosen because it is new.
It’s not as shallow as you might think. New typically means:
Some of the gnarlier bugs will not have been shaken out, especially if nobody has started to use the project in exactly the same usage as you have. You might be putting it under heavier load than anyone has ever before, or making it deal with larger amounts of data than anyone has ever done.
Documentation and knowledge won’t be as widespread. Think things like tutorials, stack overflow and of course hiring experienced folks is more difficult.
Limits, misfeatures, footguns and pitfalls will be less understood. There’s no lore yet about what to avoid.
It might not be available for your OS in a ready made package (arguably nowadays not as important with Docker and language-specific package managers)
The development team might burn out and leave the project, or decide it was a flawed approach and do a major v2 rewrite, leaving you stranded. The old rule that a project being alive for N years predicting it will last another N years has some truth to it. You don’t want to know how many cool/promising projects I’ve seen or used that don’t even have a web presence anymore.
In general, you have less data about how the project handles releases, security issues, political crises etc because the project has built up no track record yet.
New projects can be really good, supplant existing projects and become entrenched of course, but all in all, there’s considerable risk in using new projects that simply don’t exist in choosing older projects.
Edit I just realised, it’s not that they’re older per se, but older and still popular. They survived the popularity contest and have enough staying power to still be around. For new projects, it’s impossible to know whether they will be in the category of those that stay or those that peter out.
Right, that’s all fine and I agree entirely that “new” is a proxy word for things that may correlate with “new”. My point is that by appealing to this proxy as if the proxy itself is the thing to consider, and not the correlations, you reject critically thinking about those correlations. For example, instead of saying “we shouldn’t use this project, it’s new” one could say “we shouldn’t use this project, I’m concerned that we haven’t seen enough from the dev team to be confident in its future, and it would be costly for us to migrate later because XYZ”.
I agree in principle, but this is about policy. In most cases, “new” implies all of these things, and most of them are inherent due to the project being new, so using “new” as a shorthand for these things is useful. And it makes for a good policy to avoid too many of such projects. You can still use new projects, but you have to be strategic about it because you don’t want to get dragged down by too many of such “unknowns”. I find the “innovation tokens” concept useful for that reason.
Having said that, there are also good reasons why existing tech should be avoided. The more data you have about a project, the more you should know whether to avoid it. Still, many teams keep choosing shitty “solutions” they’re already familiar with, possibly out of fear for trying something new. That’s the other side of the coin.
Personally, I don’t have too much patience with shitty tools and will be on the lookout for a replacement after the nth time I get burned. Maybe that’s why I find the innovation tokens idea useful, as a guardrail against trying too much new stuff at once. Then again, I do tend to stick with the tools I like for longer than might be advisable while there are newer alternatives that are more powerful and also potentially really nice.
In most cases, “new” implies all of these things, and most of them are inherent due to the project being new, so using “new” as a shorthand for these things is useful.
Well I’m not so convinced. What does “new” mean? I’ll give you an example - Quickwit is company that provides an ElasticSearch-like system for search. QuickWit, as a company, is just a few years old. Is that new? It’s been years of serving customers - is that new? In many ways yes, at least in some ways no. QuickWit is built on Tantivy, which has been built for even longer than the company has been around. Is that new? Is that old?
It’s not clear to me what “new” even means in a field where software that’s younger than I am is considered extremely old, or where software itself is made up of many different components of different ages.
I definitely reject that these properties are inherent, I accept that they correlate and some correlate strongly.
In terms of avoiding unknowns, that’s just one more thing to consider. And I agree entirely that much of what you can know about a technology is how painful it can be to operate or use it for your use case. Choosing something newer that is intended to solve that problem could be completely reasonable.
I feel like I have a completely different interpretation of innovation tokens to the author. So their argument feels a little backwards to me.
Let’s say you’re going to make a web app. There’s a tool in Haskell that you really like for a critical part of your app’s problem domain. You don’t want to spend more than one innovation token though, and everything in Haskell is inherently innovative, so you write a little service that just does that one part and you write the rest of your app in Ruby
Forget whether Haskell is more innovate than Ruby, IMO if your app is already written in Haskell, choosing to spin up something else in Ruby IS spending an innovation token.
First, you will have to find a team that likes both Ruby and Haskell and sees no problem using both. If you are not familiar with the cultural proclivities of these languages, suffice it to say that this is unlikely. Hiring for Haskell programmers is hard because there are fewer of them than Ruby programmers, but hiring for polyglot Haskell/Ruby programmers who are happy to do either is going to be really hard.
To me, this problem is caused by spending innovation tokens.
The article talks about how he’s disagreeing with the common conception of “innovation tokens” which is different from the way they’re presented in the original article. I think you’re right, but your conception is different from the widely-held one that he’s arguing against.
Yeah, your interpretation matches my own. I haven’t seen the alternative definition that Glyph is arguing against myself, but maybe I’ve just not been participating in conversations where “innovation token” is interpreted in a different way.
If you want a glimpse into the world I am talking about, do a search for LinkedIn posts from MBAs talking about innovation tokens. That is the extreme end of the spectrum of course, but the type of thinking on display in those posts trickles down to less-experienced engineers as dogma.
I am trying to communicate an insight that many senior engineers already have about this style of novelty budgeting, and indeed, McKinley has it himself. I said so when I mentioned I’d be restating some of his own caveats. But the reason I don’t love the original article, despite the fact that it makes some good points, is because the idea of innovation tokens as written cuts against the intuition we have about it, because if you take it literally then each piece of non-boring technology has its own price. And many readers do take it quite literally.
I like the warning against the “innovation tokens” heuristic turning into slogan (reminded me of a similar take on “worse is better”).
I wasn’t convinced by the Haskell + Ruby example they give, though. The “save innovation tokens” approach would have been to try really hard to use only Ruby, and if they really needed Haskell then use it everywhere. If they were already using Haskell, then adding Ruby to the mix would be spending another innovation token, in terms of the cognitive load put on the organization.
The way I read it, the Choose boring tech essay had more emphasis in reducing the diversity of the stack than in the innovation tokens idea. I think both articles agree in spirit: operational overhead costs more than development overhead. As an heuristic I prefer “minimize cognitive load on the organization” to both innovation and boundary tokens.
I broadly agree here but was amused by this example:
but hiring for polyglot Haskell/Ruby programmers who are happy to do either is going to be really hard.
This is exactly my stack. I find Haskell and Ruby to be similar enough to fit into the same part of my brain, since they’re both based on rampant use of polymorphism. I don’t hire Haskell or Ruby programmers I hire programmers familiar with my domain and if they are smart enough to learn the domain then something small like programming language should be quick to pick up.
There are a couple of peanuts I was able to pick out of this:
Innovation tokens are just one way to look at operational overhead.
The author proposes boundary tokens as another form of operational overhead worth minimizing.
The rest is… less savory, starting with the ad-hominems:
I believe it is an attractive nuisance in computer programming discourse.
So anyone who uses the term “innovation token” is, in legalese, a child?
…it is reactionary. At its worst, it borders on anti-intellectualism.
And the original article, which contrasts Donald Rumsfeld with Socrates, is reactionary anti-intellectualism?
Then there’s the belabored, contrived example wherein someone spends an innovation token on building a Haskell service in a Ruby shop without consulting one’s teammates. McKinley makes it pretty clear that the decision to spend an innovation token in a team is a team decision:
Technology choices don’t happen in isolation. They have a scope that touches your entire team, organization, and the system that emerges from the sum total of your choices.
The only other criticism I saw that did not strike me as a misrepresentation of the original concept is this one:
You are going to have to train people on your stack no matter what. If a technology is adding a lot of value, it’s absolutely worth hiring for general ability and making a plan to teach people about it. You are going to have to do this with the core technology of your product anyway.
It depends on how what “a lot of value” means. Even if only one person is familiar enough with a new technology that they can quickly prototype a demonstration that quantifies how much “a lot” is, it’s not really that new and probably isn’t an innovation token. If “a lot of value” is really just “I really really wanna,” a new hire is absolutely going to have a harder time getting up to speed than if one had just stuck with the boring technology.
Attractive nuisance doctrines apply to adults in many jurisdictions as well. There are just more cases involving children because of children’s poorer judgement and the nature of the attractiveness of cool-looking hazards to children vs adults in a real property context.
Also, I have added an epilogue to make the “team” dynamic much more explicit. I was assuming that the decision maker in all my examples was a team (or even a multi-team org, given that the example described growing multiple teams). But I had left it implied that the whole point of the terms was for team discussions, and leaving that unclear was really undermining the point.
I did not care for this article, mostly because of the original sin that even the author admits in the beginning:
The strawman in question?
Why is this bad?
The problem with arguing against a strawman–and not really talking about the actual thing–is that you can easily miss important context. The context is: innovation tokens are part of a global optimization strategy for a business.
I contend that any stable, successful business–setting aside the mutants evolved to live with delightful plumage and fucked-up metabolisms fueled by the transient funding bloom of ZIRP-driven investment–is inherently reactionary. And that’s fine.
I’ve kicked around a definition on this for a while, going back to old IRC sessions years ago on #lobsters, but in case somebody wants to take the idea and run with it:
“In engineering, a revolutionary approach requires that any technique which might make things better must be entertained. A reactionary approach requires that any technique which might make things worse must be avoided.”
(I would argue that most teams are neither wholly revolutionary or reactionary, but rather liberal and conservative –the less extreme positions, following the normal revolutionary -> liberal -> centrist -> conservative -> reactionary spectrum.)
Bringing this back to the essay, the thing about a business being reactionary (and we’ll handwave slightly and say that business conservatism is similar to the engineering conservatism I mentioned above) is that they seek to aggressively limit surprises. From this, all else follows–you constrain innovation tokens not because you want to be grug but because you do not want to be surprised by a bet that fails to pay off and fucks up things for the rest of the business.
(Ctrl-f in the article for “business”, note zero hits. This is a clue about how strawmanny the article is.)
It’s not operational overhead that innovation tokens seek to limit, as the author suggests; many businesses live just fine with razor-thin margins or comically byzantine processes. It’s that any tech project has some amount of uncertainty built into it, and the more innovation tokens you spend the more variance into expected payoff you introduce.
~
Businesses aren’t totally risk averse–sometimes, the entire business is predicated on a huge risky bet, such as a new drug panning out or an oil deposit actually existing or whatever–but they tend to aggressively limit where they take risks. To wit:
I knew somebody who used to spec out pumps for oil rigs. The thing about oil rigs is that they cost approximately one metric fuckton of dollars per day to run, but that’s okay because they generate about ten fucktons in return (and seriously, if you ever want to get a feeling for how small-potatoes investment in tech actually is, go look at oil and gas). Anyways, he ran the numbers for some fancy turbine pumps for mud or something, and they needed to rent them. Renting each of the pumps was like a hundred thousand dollars per day. Eyewatering. He put in the request, and the oil company said “well shit, might as well have two.” They were already placing such a big bet on the well they were drilling, and the rig was so expensive, it made no sense to have a deadlined pump risk the whole enterprise.
But you know what they didn’t spend tokens on? A better messaging system than email. A better, new project management system. A more ergonomic accounting system instead of GAAP.
~
To give you an idea of how much the article loses the plot:
The author suggests a Haskell microservice as a conservative choice, and then proceeds to talk about all the extra CI and other stuff that that creates and all the overhead, and then lo-and-behold the reveal where we should’ve just written a Haskell monolith all along. And that’s fine, and I even agree with it in that case, but it’s kayfabe. It’s a ruse.
The thing is, if the team already knows Haskell and already has a tool that will solve their problem, they aren’t spending an innovation token. It’s not because of avoiding a boundary, it’s because they know the tools.
The author goes on to crow about how cool they are, how clever, how slick, for eschewing cron in order to avoid a boundary. They then list a pile of questions as though that somehow makes their case–I argue instead that that list of questions is in fact evidence that cron is the correct choice.
Here’s the thing: those questions? They each have clear answers. I can Google or Stack Overflow them if I need to. I can copy-and-paste how they were solved on other projects. The mere fact that we have that list of questions implies a degree of understanding of the trade-offs of using cron, and that is precisely the derisking the business wants to see.
(“But friendlysock, what if they think there’s less risk in their home-rolled solve that only they maintain than there is in using cron?” Well then no shit use that instead. “But you said cron was the correct choice?” Well, you just gave me new information about the risks involved and I updated my position. Welcome to the magical world of project management. “But what if they estimated the risks wrong?” Congrats, you have discovered the ur-problem of managing software projects and developers. )
~
I get that the author probably got told “no” to some fit-for-purpose technology at some point in their career (go look at their about page, it’s pretty obvious this probably came up) by somebody who was too dull to understand it. I definitely have run into folks that will say “no we can’t do this” just because something is new to them, or because they’re heavily invested in their stack. I’ve been there, it grates.
But.
“Shitty stupid midwits can block good tech decisions out of self-interest or ignorance” is an orthogonal problem to “how do we compartmentalize the risks on a project”.
tl,dr; Innovation tokens are not about complexity, they are about compartmentalizing risk–don’t lose the plot.
Edit: happy to talk about or listen to people vent on this topic in DMs. For a professional engagement, I may also be available for the right price.
I agree with the idea that avoiding surprises is very important, and means that sometimes you’ll want to choose a solution precisely because it’s a known quantity. It doesn’t matter how good the new technology actually is, if you can’t reliably estimate what the consequences of using it will be.
I really think this is not justified. Operational overhead is a business concern.
It was not meant as an idle swipe–the original 2015 article about innovation tokens is rooted in the business case, almost to a painful degree. By contrast, I think that the article here touches on operational overhead but mostly from a perspective of how it makes developers more innovative or happier…and businesses give a shit about neither of those things except as they gate paying off the core gambles they make in the market.
Plenty of successful businesses are more than happy–successful, even, by any measure!_—to make decisions that have lasting operational overhead so long as they themselves don’t have to pay it within the amortization window they care about. Sometimes, they accomplish this most simply by forbidding additional maintenance or by skipping best practices. This is a valid and successful business strategy, if a distasteful technical one.
And really, that’s the core issue I have with the article, and that’s why I pointed out what I did: it focuses too much on a developer perspective of the world, when arguably the whole point of innovation tokens was to enable global optimization across the business. If you aren’t approaching this as fundamentally a business concern, then of course any treatment of innovation tokens will seem ill-fitting at best.
Your comment:
From the article:
As for the idea that new technology just exists to make developers excited, the article says:
You can argue that’s wrong, you could even try to say it’s a rationalization, but it’s just not true that this is arguing for making developers happy above all else.
Again, operational problems are not what businesses care about, first and foremost. They care about reducing risk and cutting costs, in roughly that order, and the second is arguable. There are lots of successful businesses that take the complicated, hard, operational complexity things–Epic, HFT firms, SAP, whatever–and reduce the risk of doing their stuff to the point where they can reliably bill customers in a way that eats that cost.
If you can spend 10 million to make 10 billion, literally nobody cares if technically you could’ve done it for 8 million at twice the risk of failing. Operational complexity is a red herring. Operational complexity can be bounded and incorporated into the customer price.
Look at your quote again–it’s exactly catering to making developers excited, because this time the
castledevops tooling won’t sink into the swamp. This time, dear developer, this tooling is here to fix the problems of the last version–never you mind that we’ve probably created additional problems. This churn, it’s necessary, you see.My experience is that developers–especially the ZIRP mutants–are incredibly precious about their tools and perceived ergonomics, and while one time in fifty this leads to great tooling and perhaps a seismic shift in the Pareto frontier of what they can accomplish, the other forty-nine times it just introduces project risk and distraction. As mentioned previously, though, it’s usually pretty hard to spot which of the two buckets any particular attempt is going to fall into.
It seems to be both to me. I don’t understand even in theory what is supposed to have changed from 2015 to 2024. The problem we’re talking about was always a human and cultural one, not a tech one. Developers liking new toys, developers liking resume padding, developers over-engineering solutions… this was never about the new tech itself not living up to the hype. I mean that might be a cherry on top, but it’s incidental to the underlying phenomenon.
Tom DeMarco - Software Engineering: An Idea Whose Time Has Come and Gone?
Indeed. I’ve been observing a similar thing in the last few years at work–people are a lot grumpier about stumbling with simple things than with moonshots. If you screw up a microwave burrito you look a lot stupider than if you fail to set a souffle–and you get angrier if your bank were to misplace 5% of an account than if your stock portfolio swung 10% during a dip.
(Also, Tom DeMarco is a great unsung hero in the canon, and I wish he was read more likely–or at least got the same treatment as Brooks in the sense of “everybody references, few read”.)
I can’t see the full context because it’s behind a paywall. How does one know the cost of a software project or its eventual value until it’s dead?
I found a PDF on the web: http://www.cs.uni.edu/~wallingf/teaching/172/resources/demarco-on-se.pdf
Thanks! Very germane. McKinley talks about costs and marginal utility. Conversely, DeMarco said, in effect: I wrote a book on measuring and controlling software costs like any other engineering discipline. It doesn’t work. All software projects are more like teenagers: the more control you try to exert, the more it resists, so maybe try Agile? And try not to waste money on the low-value projects.
DeMarco’s advice seems sensible. Mind-changing, even. I might only suggest with regard to that last point, which is the one you quoted, that a low-value project (measuring “value” here like measuring the weather with a wet finger) can be a great place to try out a new tool or pattern precisely because its scope is small.
Great analysis. I actually liked idea of boundary tokens, but a few other specific points I strongly disagreed with:
I feel like the chance of this making sense is effectively zero, and listening to the heuristic the article is trying to debunk is a good shortcut for understanding that.
On the contrary, what should increase is your willingness to push back against this growing “need” with all of your might at every opportunity.
I feel like this is mostly true, but some tools really are a pain in their own right.
I like the meat of this perspective. Personally, I might have framed it as “innovation tokens are one heuristic, here’s another”.
I think it’s wise to be clear about what is “expected knowledge” for an organization or project and trying to pick solutions that are within or nearby. Introducing new technologies outside of this domain puts a burden on engineers to expand their knowledge in order to effectively maintain or operate the new technology. There’s nothing inherently wrong with that but it’s something that has a cost/benefit tradeoff.
If you have an organization of people very familiar with Go and Postgres and you introduce Temporal, you can add capabilities to your system but you will also have people spending more time and energy on maintenance and operations. You need to make an explicit decision about whether the benefits outweigh the costs. This seems to be what the “boundary token” is getting at and I think that’s really useful. I also find innovation tokens a useful heuristic.
I was going to write this comment, and was pleasantly surprised to find that you’d already written it! =)
In agreement: I don’t think boundary tokens displace innovation tokens. I think they’re orthogonal, and I will use them (or the rough idea) for decision-making in the future. I already had similar rules of thumb in my quiver, like “do whatever it takes to get the team to choose the incumbent language instead of a different one for $NEW_DECISION,” but the boundary idea adds helpful nuance and razoring for those deep discussions. Perhaps the big takeaway is that token-thinking is a family of thought, or, say, a set of lenses, and ye olde Innovation Token was just the first one discovered and popularized.
Dogma like “innovation tokens” is probably what I dislike the most about this field. Evaluating software using shallow properties like “it’s new” is not a good idea and it leads to lazy decision making where a software that could be better in literally every way will not be chosen because it is new. But instead of evaluating it people appeal to the stereotypes listed in the article and more.
This is a big problem with snappy terms. They mutate into their worst, most memetic form. “Premature optimization is the root of all evil” is a good example but there are endless examples of programmers taking on dogmatic “wisdom” with an uncritical mind.
The post resonates with me. The “at worst it’s anti-intellectualism” is understated, I think. It simply is “anti-intellectual” because it de-facto dismisses critical thinking, instead appealing to a phrase.
That said, I don’t think the result here is good.
How about neither? How about, when you evaluate a technology, you evaluate the technology. You have context on what your team can/ can not do, must do, etc. How will the technology fit into that? Are there risks associated with one approach? Can those be mitigated? If we do it one way here, and another way there, will we pay a price for the inconsistency (the principal advocated in the article)?
Like, what if, and hear me out, people thought about things?
Basically, I think the article couldn’t be more correct on calling out the flawed nature for people to be intellectually lazy and unrigorous, I think the article then comes up with just another quick heuristic that will fail in new ways just like any heuristic will. I guess it’s all heuristics since the question is always “can we predict our success if we choose path X versus path Y”, so I suppose my problem is just using one heuristic. “This thing is really new and different” is one thing to keep in mind, “This thing is really different from our existing systems” is worth keeping in mind. Challenging assumptions about whether a “boring” technology is a good fit is one more thing to do. At the end of the day, the issue is trying to come up with quick answers using simplistic, universal heuristics.
It’s not as shallow as you might think. New typically means:
New projects can be really good, supplant existing projects and become entrenched of course, but all in all, there’s considerable risk in using new projects that simply don’t exist in choosing older projects.
Edit I just realised, it’s not that they’re older per se, but older and still popular. They survived the popularity contest and have enough staying power to still be around. For new projects, it’s impossible to know whether they will be in the category of those that stay or those that peter out.
Right, that’s all fine and I agree entirely that “new” is a proxy word for things that may correlate with “new”. My point is that by appealing to this proxy as if the proxy itself is the thing to consider, and not the correlations, you reject critically thinking about those correlations. For example, instead of saying “we shouldn’t use this project, it’s new” one could say “we shouldn’t use this project, I’m concerned that we haven’t seen enough from the dev team to be confident in its future, and it would be costly for us to migrate later because XYZ”.
I agree in principle, but this is about policy. In most cases, “new” implies all of these things, and most of them are inherent due to the project being new, so using “new” as a shorthand for these things is useful. And it makes for a good policy to avoid too many of such projects. You can still use new projects, but you have to be strategic about it because you don’t want to get dragged down by too many of such “unknowns”. I find the “innovation tokens” concept useful for that reason.
Having said that, there are also good reasons why existing tech should be avoided. The more data you have about a project, the more you should know whether to avoid it. Still, many teams keep choosing shitty “solutions” they’re already familiar with, possibly out of fear for trying something new. That’s the other side of the coin.
Personally, I don’t have too much patience with shitty tools and will be on the lookout for a replacement after the nth time I get burned. Maybe that’s why I find the innovation tokens idea useful, as a guardrail against trying too much new stuff at once. Then again, I do tend to stick with the tools I like for longer than might be advisable while there are newer alternatives that are more powerful and also potentially really nice.
Well I’m not so convinced. What does “new” mean? I’ll give you an example - Quickwit is company that provides an ElasticSearch-like system for search. QuickWit, as a company, is just a few years old. Is that new? It’s been years of serving customers - is that new? In many ways yes, at least in some ways no. QuickWit is built on Tantivy, which has been built for even longer than the company has been around. Is that new? Is that old?
It’s not clear to me what “new” even means in a field where software that’s younger than I am is considered extremely old, or where software itself is made up of many different components of different ages.
I definitely reject that these properties are inherent, I accept that they correlate and some correlate strongly.
In terms of avoiding unknowns, that’s just one more thing to consider. And I agree entirely that much of what you can know about a technology is how painful it can be to operate or use it for your use case. Choosing something newer that is intended to solve that problem could be completely reasonable.
I feel like I have a completely different interpretation of innovation tokens to the author. So their argument feels a little backwards to me.
Forget whether Haskell is more innovate than Ruby, IMO if your app is already written in Haskell, choosing to spin up something else in Ruby IS spending an innovation token.
To me, this problem is caused by spending innovation tokens.
Am I just wrong in my interpretation?
The article talks about how he’s disagreeing with the common conception of “innovation tokens” which is different from the way they’re presented in the original article. I think you’re right, but your conception is different from the widely-held one that he’s arguing against.
Yeah, your interpretation matches my own. I haven’t seen the alternative definition that Glyph is arguing against myself, but maybe I’ve just not been participating in conversations where “innovation token” is interpreted in a different way.
If you want a glimpse into the world I am talking about, do a search for LinkedIn posts from MBAs talking about innovation tokens. That is the extreme end of the spectrum of course, but the type of thinking on display in those posts trickles down to less-experienced engineers as dogma.
I am trying to communicate an insight that many senior engineers already have about this style of novelty budgeting, and indeed, McKinley has it himself. I said so when I mentioned I’d be restating some of his own caveats. But the reason I don’t love the original article, despite the fact that it makes some good points, is because the idea of innovation tokens as written cuts against the intuition we have about it, because if you take it literally then each piece of non-boring technology has its own price. And many readers do take it quite literally.
I like the warning against the “innovation tokens” heuristic turning into slogan (reminded me of a similar take on “worse is better”).
I wasn’t convinced by the Haskell + Ruby example they give, though. The “save innovation tokens” approach would have been to try really hard to use only Ruby, and if they really needed Haskell then use it everywhere. If they were already using Haskell, then adding Ruby to the mix would be spending another innovation token, in terms of the cognitive load put on the organization.
The way I read it, the Choose boring tech essay had more emphasis in reducing the diversity of the stack than in the innovation tokens idea. I think both articles agree in spirit: operational overhead costs more than development overhead. As an heuristic I prefer “minimize cognitive load on the organization” to both innovation and boundary tokens.
I broadly agree here but was amused by this example:
This is exactly my stack. I find Haskell and Ruby to be similar enough to fit into the same part of my brain, since they’re both based on rampant use of polymorphism. I don’t hire Haskell or Ruby programmers I hire programmers familiar with my domain and if they are smart enough to learn the domain then something small like programming language should be quick to pick up.
There are a couple of peanuts I was able to pick out of this:
The rest is… less savory, starting with the ad-hominems:
So anyone who uses the term “innovation token” is, in legalese, a child?
And the original article, which contrasts Donald Rumsfeld with Socrates, is reactionary anti-intellectualism?
Then there’s the belabored, contrived example wherein someone spends an innovation token on building a Haskell service in a Ruby shop without consulting one’s teammates. McKinley makes it pretty clear that the decision to spend an innovation token in a team is a team decision:
The only other criticism I saw that did not strike me as a misrepresentation of the original concept is this one:
It depends on how what “a lot of value” means. Even if only one person is familiar enough with a new technology that they can quickly prototype a demonstration that quantifies how much “a lot” is, it’s not really that new and probably isn’t an innovation token. If “a lot of value” is really just “I really really wanna,” a new hire is absolutely going to have a harder time getting up to speed than if one had just stuck with the boring technology.
Attractive nuisance doctrines apply to adults in many jurisdictions as well. There are just more cases involving children because of children’s poorer judgement and the nature of the attractiveness of cool-looking hazards to children vs adults in a real property context.
https://www.abcdelaware.com/attractive-nuisance/
Also, I have added an epilogue to make the “team” dynamic much more explicit. I was assuming that the decision maker in all my examples was a team (or even a multi-team org, given that the example described growing multiple teams). But I had left it implied that the whole point of the terms was for team discussions, and leaving that unclear was really undermining the point.
Resent your implication that I have better judgement than a child.
Looking at the top of the thread it seems like I am catching some some resentment either way, so I might as well be accurate ;-)