What surrprised me about Tainter’s analysis (and I haven’t read his entire book yet) is that he sees complexity as a method by which societies gain efficiency. This is very different from the way software developers talk about complexity (as ‘bloat’, ‘baggage’, ‘legacy’, ‘complication’), and made his perspective seem particularly fresh.
I don’t mean to sound dismissive – Tainter’s works are very well documented, and he makes a lot of valid points – but it’s worth keeping in mind that grand models of history have made for extremely attractive pop history books, but really poor explanations of historical phenomena. Tainter’s Collapse of Complex Societies, while obviously based on a completely different theory (and one with far less odious consequences in the real world) is based on the same kind of scientific thinking that brought us dialectical materialism.
His explanation of the fall of the evolution and the eventual fall of the Roman Empire makes a number of valid points about the Empire’s economy and about some of the economic interests behind the Empire’s expansion, no doubt. However, explaining even the expansion – let alone the fall! – of the Roman Empire strictly in terms of energy requirements is about as correct as explaining it in terms of class struggle.
Yes, some particular military expeditions were specifically motivated by the desire to get more grain or more cows. But many weren’t – in fact, some of the greatest Roman wars, like (some of) the Roman-Parthian wars, were not driven specifically by Roman desire to get more grains or cows. Furthermore, periods of rampant, unsustainably rising military and infrastructure upkeep costs were not associated only with expansionism, but also with mounting outside pressure (ironically, sometimes because the energy per capita on the other side of the Roman border really sucked, and the Huns made it worse on everyone). The increase of cost and decrease in efficiency, too, are not a matter of half-rational historical determinism – they had economic as well as cultural and social causes that rationalising things in terms of energy not only misses, but distorts to the point of uselessness. The breakup of the Empire was itself a very complex social, cultural and military story which is really not something that can be described simply in terms of the dissolution of a central authority.
That’s also where this mismatch between “bloat” and “features” originates. Describing program features simply in terms of complexity is a very reductionist model, which accounts only for the difficulty of writing and maintaining it, not for its usefulness, nor for the commercial environment in which it operates and the underlying market forces. Things are a lot more nuanced than “complexity = good at first, then bad”: critical features gradually become unneeded (see Xterm’s many emulation modes, for example), markets develop in different ways and company interests align with them differently (see Microsoft’s transition from selling operating systems and office programs to renting cloud servers) and so on.
However, explaining even the expansion – let alone the fall! – of the Roman Empire strictly in terms of energy requirements is about as correct as explaining it in terms of class struggle.
Of course. I’m long past the age where I expect anyone to come up with a single, snappy explanation for hundreds of years of human history.
But all models are wrong, only some are useful. Especially in our practice, where we often feel overwhelmed by complexity despite everyone’s best efforts, I think it’s useful to have a theory about the origins and causes of complexity, even if only for emotional comfort.
Especially in our practice, where we often feel overwhelmed by complexity despite everyone’s best efforts, I think it’s useful to have a theory about the origins and causes of complexity, even if only for emotional comfort.
Indeed! The issue I take with “grand models” like Tainter’s and the way they are applied in grand works like Collapse of Complex Societies is that they are ambitiously applied to long, grand processes across the globe without an exploration of the limits (and assumptions) of the model.
To draw an analogy with our field: IMHO the Collapse of… is a bit like taking Turing’s machine as a model and applying it to reason about modern computers, without noting the differences between modern computers and Turing machines. If you cling to it hard enough, you can hand-wave every observed performance bottleneck in terms of the inherent inefficiency of a computer reading instructions off a paper tape, even though what’s actually happening is cache misses and hard drives getting thrashed by swapping. We don’t fall into this fallacy because we understand the limits of Turing’s model – in fact, Turing himself explicitly mentioned many (most?) of them, even though he had very little prior art in terms of alternative implementations, and explicitly formulated his model to apply only to some specific aspects of computation.
Like many scholars at the intersections of economics and history in his generation, Tainter doesn’t explore the limits of his model too much. He came up with a model that explains society-level processes in terms of energy output per capita and upkeep cost and, without noting where these processes are indeed determined solely (or primarily) by energy output per capita and upkeep post, he proceeded to apply it to pretty much all of history. If you cling to this model hard enough you can obviously explain anything with it – the model is explicitly universal – even things that have nothing to do with energy output per capita or upkeep cost.
In this regard (and I’m parroting Walter Benjamin’s take on historical materialism here) these models are quasi-religious and are very much like a mechanical Turk. From the outside they look like history masterfully explaining things, but if you peek inside, you’ll find our good ol’ friend theology, staunchly applying dogma (in this case, the universal laws of complexity, energy output per capita and upkeep post) to any problem you throw its way.
Without an explicit understanding of their limits, even mathematical models in exact sciences are largely useless – in fact, a big part of early design work is figuring out what models apply. Descriptive models in humanistic disciplines are no exception. If you put your mind to it, you can probably explain every Cold War decision in terms of Vedic ethics or the I Ching, but that’s largely a testament to one’s creativity, not to their usefulness.
Furthermore, periods of rampant, unsustainably rising military and infrastructure upkeep costs were not associated only with expansionism, but also with mounting outside pressure (ironically, sometimes because the energy per capita on the other side of the Roman border really sucked, and the Huns made it worse on everyone).
Not to mention all the periods of rampant rising military costs due to civil war. Those aren’t wars about getting more energy!
Tainter’s Collapse of Complex Societies, while obviously based on a completely different theory (and one with far less odious consequences in the real world) is based on the same kind of scientific thinking that brought us dialectical materialism.
Sure. This is all about a framing of events that happened; it’s not predictive, as much as it is thought-provoking.
Thought-provoking, grand philosophy was certainly a part of philosophy but became especially popular (some argue that it was Nathaniel Bacon who really brought forth the idea of predicting progress) during the Industrial Era with the rise of what is known as the modernist movement. Modernist theories often differed but frequently shared a few characteristics such as grand narratives of history and progress, definite ideas of the self, a strong belief in progress, a belief that order was superior to chaos, and often structuralist philosophies. Modernism had a strong belief that everything could be measured, modeled, categorized, and predicted. It was an understandable byproduct of a society rigorously analyzing their surroundings for the first time.
Modernism flourished in a lot of fields in the late 19th early 20th century. This was the era that brought political philosophies like the Great Society in the US, the US New Deal, the eugenics movement, biological determinism, the League of Nations, and other grand social and political engineering ideas. It was embodied in the Newtonian physics of the day and was even used to explain social order in colonizing imperialist nation-states. Marx’s dialectical materialism and much of Hegel’s materialism was steeped in this modernist tradition.
In the late 20th century, modernism fell into a crisis. Theories of progress weren’t bearing fruit. Grand visions of the future, such as Marx’s dialectical materialism, diverged significantly from actual lived history and frequently resulted in a magnitude of horrors. This experience was repeated by eugenics, social determinism, and fascist movements. Planck and Einstein challenged the neat Newtonian order that had previously been conceived. Gödel’s Incompleteness Theorem showed us that there are statements we cannot evaluate the validity of. Moreover many social sciences that bought into modernist ideas like anthropology, history, and urban planning were having trouble making progress that agreed with the grand modernist ideas that guided their work. Science was running into walls as to what was measurable and what wasn’t. It was in this crisis that postmodernism was born, when philosophers began challenging everything from whether progress and order were actually good things to whether humans could ever come to mutual understanding at all.
Since then, philosophy has mostly abandoned the concept of modeling and left that to science. While grand, evocative theories are having a bit of a renaissance in the public right now, philosophers continue to be “stuck in the hole of postmodernism.” Philosophers have raised central questions about morality, truth, and knowledge that have to be answered before large, modernist philosophies gain hold again.
I don’t understand this, because my training has been to consider models (simplified ways of understanding the world) as only having any worth if they are predictive and testable i.e. allow us to predict how the whole works and what it does based on movements of the pieces.
Models with predictive values in history (among other similar fields of study, including, say, cultural anthropology) were very fashionable at one point. I’ve only mentioned dialectical materialism because it’s now practically universally recognized to have been not just a failure, but a really atrocious one, so it makes for a good insult, and it shares the same fallacy with energy economic models, so it’s a doubly good jab. But there was a time, as recent as the first half of the twentieth century, when people really thought they could discern “laws of history” and use them to predict the future to some degree.
Unfortunately, this has proven to be, at best, beyond the limits of human understanding and comprehension. This is especially difficult to do in the study of history, where sources are imperfect and have often been lost (case in point: there are countless books we know the Romans wrote because they’re mentioned or quoted by ancient authors, but we no longer have them). Our understanding of these things can change drastically with the discovery of new sources. The history of religion provides a good example, in the form of our understanding of Gnosticism, which was forever altered by the discovery of the Nag Hammadi library, to the point where many works published prior to this discovery and the dissemination of its text are barely of historical interest now.
That’s not to say that developing a theory of various historical phenomenons is useless, though. Even historical materialism, misguided as they were (especially in their more politicized formulations), were not without value. They forced an entire generation of historians to think more about things that they never really thought about before. It is certainly incorrect to explain everything in terms of class struggle, competition for resources and the means of production, and the steady march from primitive communism to the communist mode of production – but it is also true that competition for resources and the means of production were involved in some events and processes, and nobody gave much thought to that before the disciples of Marx and Engels.
This is true here as well (although I should add that, unlike most materialistic historians, Tainter is most certainly not an idiot, not a war criminal, and not high on anything – I think his works display an unhealthy attachment for historical determinism, but he most certainly doesn’t belong in the same gallery as Lenin and Mao). His model is reductionist to the point where you can readily apply much of the criticism of historical materialism to it as well (which is true of a lot of economic models if we’re being honest…). But it forced people to think of things in a new way. Energy economics is not something that you’re tempted to think about when considering pre-industrial societies, for example.
These models don’t really have predictive value and they probably can’t ever gain one. But they do have an exploratory value. They may not be able to tell you what will happen tomorrow, but they can help you think about what’s happening today in more ways than one, from more angles, and considering more factors, and possibly understand it better.
That’s something historians don’t do anymore. There was a period where people tried to predict the future development of history, and then the whole discipline gave up. It’s a bit like what we are witnessing in the Economics field: there are strong calls to stop attributing predictive value to macroeconomic models because after a certain scale, they are just over-fitting to existing patterns, and they fail miserably after a few years.
Well, history is not math, right? It’s a way of writing a story backed by a certain amount of evidence. You can use a historical model to make predictions, sure, but the act of prediction itself causes changes.
(OP here.) I totally agree, and this is something I didn’t explore in my essay. Tainter doesn’t see complexity as always a problem: at first, it brings benefits! That’s why people do it. But there are diminishing returns and maintenance costs that start to outstrip the marginal benefits.
Maybe one way this could apply to software: imagine I have a simple system, just a stateless input/output. I can add a caching layer in front, which could win a huge performance improvement. But now I have to think about cache invalidation, cache size, cache expiry, etc. Suddenly there are a lot more moving parts to understand and maintain in the future. And the next performance improvement will probably not be anywhere near as big, but it will require more work because you have to understand the existing system first.
In Tainter’s view, a society of subsistence farmers, where everyone grows their own crops, makes their own tools, teaches their own children, etc. is not very complex. Add a blacksmith (division of labour) to that society, and you gain efficiency, but introduce complexity.
In my experience this goes wrong at the requirements phase: customers want the product to do more and more (e.g. Zawinski’s law), without any regard for the impact on internal consistency and performance. Of course, it’s our job as software engineers/architects to maintain these qualities, but that’s made difficult by these pressures. This, incidentally, is also one of the reasons open source software can be of much higher quality and less likely to collapse like the article says, since the developers are also the architects, and are more willing to take a stand and say “no” because their paycheck doesn’t depend on it.
Also, dropping unused features is rarely something the customer will initiate. After all, they paid for having it developed and “it may come in handy again some day”. The only reason it’s typically considered if the maintenance burden is too high, and communicated effectively to the customer, and the customer effectively understands and communicates which features can be dropped.
Well, yes. But you have probably faced engineering teams that act as seemingly loving to pile complexity for complexity’s sake. After all, the term “over-engineered” came from somewhere :)
The parallel between societies and software is a great find! The big thing that I disagree with though is:
and a fresh-faced team is brought in to, blessedly, design a new system from scratch.
(…)
you have to admit that this system works.
My experience is the opposite. No customer is willing to work with a reduced feature set, and the old software has accumulated a large undocumented set of specific features. The new-from-scratch version will have to somehow reproduce all of that, all the while having to keep up with patching done to the old system that is still running as the new system is under development. In other words, the new system will never be completed.
In short, we have no way to escape complexity at all. Once it’s there, it stays. The only thing we can do to keep ourselves from collapse as described in the article is avoid creating complexity in the first place. But as I think is stated correctly, that is not something most organisations are particularly good at.
No customer is willing to work with a reduced feature set…
Sure they are, because the price for the legacy system keeps going up. They eventually bite the bullet. That’s been my experience, anyway. The evidence is that products DO actually go away, in fact, we complain about Google doing it too much!
Yes, some things stay around basically forever, but those are things that are so valuable (to someone) that someone is willing to pay dearly to keep them running. Meanwhile, the rest of the world moves on to the new systems.
Absent vandals ransacking offices, perhaps this is what ‘collapse’ means in the context of software; the point where its added value can no longer fund its maintenance.
Cost is one way to look at it, but it’s much harder to make this argument in situations like SaaS. The cost imposed on the customer is much more indirect than when it’s software the customer directly operates. You need to have a deprecation process that can move customers onto the supported things in a reasonable fashion. When this is done well, there is continual evaluation to reduce the bleeding from new adoption of a feature that’s going away while migration paths are considered.
I think the best model for looking at this overall is the Jobs To Be Done (JTBD) framework. Like many management tools, it can actually be explained in to a software engineer on a single page rather than requiring a book, but people like to opine.
You split out the jobs that customers need done which are sometimes much removed from the original intent of a feature. These can then be mapped onto a solution, or the solution can be re-envisioned. Many people don’t get to the bottom of the actual job the customer is currently doing and then they deprecate with alternatives that only partially suit task.
My experience is the opposite. No customer is willing to work with a reduced feature set
Not from the same vendor. But if they’re lucky enough not to be completely locked in, once the first vendor’s system is sufficiently bloated and slow and buggy, they might be willing to consider going to the competition.
It’s still kind of a rewrite, but the difference this time is that one company might go under while another rises. (If the first company is big enough, they might also buy the competition…)
If you were hooked by the first sentence of the article mentioning The Collapse of Complex Societies, you might get quickly addicted to The Fall of Civilizations podcast by Paul Cooper. It’s one of my favorites. I love learning about how things fail(ed).
One of my favourite books, and I’ve long thought Tainter’s analysis could be applied to software, perhaps even more so than in societal collapse. This is a good writeup.
What surrprised me about Tainter’s analysis (and I haven’t read his entire book yet) is that he sees complexity as a method by which societies gain efficiency. This is very different from the way software developers talk about complexity (as ‘bloat’, ‘baggage’, ‘legacy’, ‘complication’), and made his perspective seem particularly fresh.
I don’t mean to sound dismissive – Tainter’s works are very well documented, and he makes a lot of valid points – but it’s worth keeping in mind that grand models of history have made for extremely attractive pop history books, but really poor explanations of historical phenomena. Tainter’s Collapse of Complex Societies, while obviously based on a completely different theory (and one with far less odious consequences in the real world) is based on the same kind of scientific thinking that brought us dialectical materialism.
His explanation of the fall of the evolution and the eventual fall of the Roman Empire makes a number of valid points about the Empire’s economy and about some of the economic interests behind the Empire’s expansion, no doubt. However, explaining even the expansion – let alone the fall! – of the Roman Empire strictly in terms of energy requirements is about as correct as explaining it in terms of class struggle.
Yes, some particular military expeditions were specifically motivated by the desire to get more grain or more cows. But many weren’t – in fact, some of the greatest Roman wars, like (some of) the Roman-Parthian wars, were not driven specifically by Roman desire to get more grains or cows. Furthermore, periods of rampant, unsustainably rising military and infrastructure upkeep costs were not associated only with expansionism, but also with mounting outside pressure (ironically, sometimes because the energy per capita on the other side of the Roman border really sucked, and the Huns made it worse on everyone). The increase of cost and decrease in efficiency, too, are not a matter of half-rational historical determinism – they had economic as well as cultural and social causes that rationalising things in terms of energy not only misses, but distorts to the point of uselessness. The breakup of the Empire was itself a very complex social, cultural and military story which is really not something that can be described simply in terms of the dissolution of a central authority.
That’s also where this mismatch between “bloat” and “features” originates. Describing program features simply in terms of complexity is a very reductionist model, which accounts only for the difficulty of writing and maintaining it, not for its usefulness, nor for the commercial environment in which it operates and the underlying market forces. Things are a lot more nuanced than “complexity = good at first, then bad”: critical features gradually become unneeded (see Xterm’s many emulation modes, for example), markets develop in different ways and company interests align with them differently (see Microsoft’s transition from selling operating systems and office programs to renting cloud servers) and so on.
Of course. I’m long past the age where I expect anyone to come up with a single, snappy explanation for hundreds of years of human history.
But all models are wrong, only some are useful. Especially in our practice, where we often feel overwhelmed by complexity despite everyone’s best efforts, I think it’s useful to have a theory about the origins and causes of complexity, even if only for emotional comfort.
Indeed! The issue I take with “grand models” like Tainter’s and the way they are applied in grand works like Collapse of Complex Societies is that they are ambitiously applied to long, grand processes across the globe without an exploration of the limits (and assumptions) of the model.
To draw an analogy with our field: IMHO the Collapse of… is a bit like taking Turing’s machine as a model and applying it to reason about modern computers, without noting the differences between modern computers and Turing machines. If you cling to it hard enough, you can hand-wave every observed performance bottleneck in terms of the inherent inefficiency of a computer reading instructions off a paper tape, even though what’s actually happening is cache misses and hard drives getting thrashed by swapping. We don’t fall into this fallacy because we understand the limits of Turing’s model – in fact, Turing himself explicitly mentioned many (most?) of them, even though he had very little prior art in terms of alternative implementations, and explicitly formulated his model to apply only to some specific aspects of computation.
Like many scholars at the intersections of economics and history in his generation, Tainter doesn’t explore the limits of his model too much. He came up with a model that explains society-level processes in terms of energy output per capita and upkeep cost and, without noting where these processes are indeed determined solely (or primarily) by energy output per capita and upkeep post, he proceeded to apply it to pretty much all of history. If you cling to this model hard enough you can obviously explain anything with it – the model is explicitly universal – even things that have nothing to do with energy output per capita or upkeep cost.
In this regard (and I’m parroting Walter Benjamin’s take on historical materialism here) these models are quasi-religious and are very much like a mechanical Turk. From the outside they look like history masterfully explaining things, but if you peek inside, you’ll find our good ol’ friend theology, staunchly applying dogma (in this case, the universal laws of complexity, energy output per capita and upkeep post) to any problem you throw its way.
Without an explicit understanding of their limits, even mathematical models in exact sciences are largely useless – in fact, a big part of early design work is figuring out what models apply. Descriptive models in humanistic disciplines are no exception. If you put your mind to it, you can probably explain every Cold War decision in terms of Vedic ethics or the I Ching, but that’s largely a testament to one’s creativity, not to their usefulness.
Not to mention all the periods of rampant rising military costs due to civil war. Those aren’t wars about getting more energy!
Sure. This is all about a framing of events that happened; it’s not predictive, as much as it is thought-provoking.
Thought-provoking, grand philosophy was certainly a part of philosophy but became especially popular (some argue that it was Nathaniel Bacon who really brought forth the idea of predicting progress) during the Industrial Era with the rise of what is known as the modernist movement. Modernist theories often differed but frequently shared a few characteristics such as grand narratives of history and progress, definite ideas of the self, a strong belief in progress, a belief that order was superior to chaos, and often structuralist philosophies. Modernism had a strong belief that everything could be measured, modeled, categorized, and predicted. It was an understandable byproduct of a society rigorously analyzing their surroundings for the first time.
Modernism flourished in a lot of fields in the late 19th early 20th century. This was the era that brought political philosophies like the Great Society in the US, the US New Deal, the eugenics movement, biological determinism, the League of Nations, and other grand social and political engineering ideas. It was embodied in the Newtonian physics of the day and was even used to explain social order in colonizing imperialist nation-states. Marx’s dialectical materialism and much of Hegel’s materialism was steeped in this modernist tradition.
In the late 20th century, modernism fell into a crisis. Theories of progress weren’t bearing fruit. Grand visions of the future, such as Marx’s dialectical materialism, diverged significantly from actual lived history and frequently resulted in a magnitude of horrors. This experience was repeated by eugenics, social determinism, and fascist movements. Planck and Einstein challenged the neat Newtonian order that had previously been conceived. Gödel’s Incompleteness Theorem showed us that there are statements we cannot evaluate the validity of. Moreover many social sciences that bought into modernist ideas like anthropology, history, and urban planning were having trouble making progress that agreed with the grand modernist ideas that guided their work. Science was running into walls as to what was measurable and what wasn’t. It was in this crisis that postmodernism was born, when philosophers began challenging everything from whether progress and order were actually good things to whether humans could ever come to mutual understanding at all.
Since then, philosophy has mostly abandoned the concept of modeling and left that to science. While grand, evocative theories are having a bit of a renaissance in the public right now, philosophers continue to be “stuck in the hole of postmodernism.” Philosophers have raised central questions about morality, truth, and knowledge that have to be answered before large, modernist philosophies gain hold again.
I don’t understand this, because my training has been to consider models (simplified ways of understanding the world) as only having any worth if they are predictive and testable i.e. allow us to predict how the whole works and what it does based on movements of the pieces.
You’re not thinking like a philosopher ;-)
Neither are you ;-)
https://plato.stanford.edu/entries/pseudo-science/ https://plato.stanford.edu/entries/popper/
Models with predictive values in history (among other similar fields of study, including, say, cultural anthropology) were very fashionable at one point. I’ve only mentioned dialectical materialism because it’s now practically universally recognized to have been not just a failure, but a really atrocious one, so it makes for a good insult, and it shares the same fallacy with energy economic models, so it’s a doubly good jab. But there was a time, as recent as the first half of the twentieth century, when people really thought they could discern “laws of history” and use them to predict the future to some degree.
Unfortunately, this has proven to be, at best, beyond the limits of human understanding and comprehension. This is especially difficult to do in the study of history, where sources are imperfect and have often been lost (case in point: there are countless books we know the Romans wrote because they’re mentioned or quoted by ancient authors, but we no longer have them). Our understanding of these things can change drastically with the discovery of new sources. The history of religion provides a good example, in the form of our understanding of Gnosticism, which was forever altered by the discovery of the Nag Hammadi library, to the point where many works published prior to this discovery and the dissemination of its text are barely of historical interest now.
That’s not to say that developing a theory of various historical phenomenons is useless, though. Even historical materialism, misguided as they were (especially in their more politicized formulations), were not without value. They forced an entire generation of historians to think more about things that they never really thought about before. It is certainly incorrect to explain everything in terms of class struggle, competition for resources and the means of production, and the steady march from primitive communism to the communist mode of production – but it is also true that competition for resources and the means of production were involved in some events and processes, and nobody gave much thought to that before the disciples of Marx and Engels.
This is true here as well (although I should add that, unlike most materialistic historians, Tainter is most certainly not an idiot, not a war criminal, and not high on anything – I think his works display an unhealthy attachment for historical determinism, but he most certainly doesn’t belong in the same gallery as Lenin and Mao). His model is reductionist to the point where you can readily apply much of the criticism of historical materialism to it as well (which is true of a lot of economic models if we’re being honest…). But it forced people to think of things in a new way. Energy economics is not something that you’re tempted to think about when considering pre-industrial societies, for example.
These models don’t really have predictive value and they probably can’t ever gain one. But they do have an exploratory value. They may not be able to tell you what will happen tomorrow, but they can help you think about what’s happening today in more ways than one, from more angles, and considering more factors, and possibly understand it better.
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
[Comment removed by moderator pushcx: Political history flamewar.]
That’s something historians don’t do anymore. There was a period where people tried to predict the future development of history, and then the whole discipline gave up. It’s a bit like what we are witnessing in the Economics field: there are strong calls to stop attributing predictive value to macroeconomic models because after a certain scale, they are just over-fitting to existing patterns, and they fail miserably after a few years.
Well, history is not math, right? It’s a way of writing a story backed by a certain amount of evidence. You can use a historical model to make predictions, sure, but the act of prediction itself causes changes.
(OP here.) I totally agree, and this is something I didn’t explore in my essay. Tainter doesn’t see complexity as always a problem: at first, it brings benefits! That’s why people do it. But there are diminishing returns and maintenance costs that start to outstrip the marginal benefits.
Maybe one way this could apply to software: imagine I have a simple system, just a stateless input/output. I can add a caching layer in front, which could win a huge performance improvement. But now I have to think about cache invalidation, cache size, cache expiry, etc. Suddenly there are a lot more moving parts to understand and maintain in the future. And the next performance improvement will probably not be anywhere near as big, but it will require more work because you have to understand the existing system first.
I’m not sure it’s so different.
A time saving or critically important feature for me may be a “bloated” waste of bits for somebody else.
In Tainter’s view, a society of subsistence farmers, where everyone grows their own crops, makes their own tools, teaches their own children, etc. is not very complex. Add a blacksmith (division of labour) to that society, and you gain efficiency, but introduce complexity.
In my experience this goes wrong at the requirements phase: customers want the product to do more and more (e.g. Zawinski’s law), without any regard for the impact on internal consistency and performance. Of course, it’s our job as software engineers/architects to maintain these qualities, but that’s made difficult by these pressures. This, incidentally, is also one of the reasons open source software can be of much higher quality and less likely to collapse like the article says, since the developers are also the architects, and are more willing to take a stand and say “no” because their paycheck doesn’t depend on it.
Also, dropping unused features is rarely something the customer will initiate. After all, they paid for having it developed and “it may come in handy again some day”. The only reason it’s typically considered if the maintenance burden is too high, and communicated effectively to the customer, and the customer effectively understands and communicates which features can be dropped.
As an engineer, I find this to be an insult.
Well, yes. But you have probably faced engineering teams that act as seemingly loving to pile complexity for complexity’s sake. After all, the term “over-engineered” came from somewhere :)
The parallel between societies and software is a great find! The big thing that I disagree with though is:
My experience is the opposite. No customer is willing to work with a reduced feature set, and the old software has accumulated a large undocumented set of specific features. The new-from-scratch version will have to somehow reproduce all of that, all the while having to keep up with patching done to the old system that is still running as the new system is under development. In other words, the new system will never be completed.
In short, we have no way to escape complexity at all. Once it’s there, it stays. The only thing we can do to keep ourselves from collapse as described in the article is avoid creating complexity in the first place. But as I think is stated correctly, that is not something most organisations are particularly good at.
Sure they are, because the price for the legacy system keeps going up. They eventually bite the bullet. That’s been my experience, anyway. The evidence is that products DO actually go away, in fact, we complain about Google doing it too much!
Yes, some things stay around basically forever, but those are things that are so valuable (to someone) that someone is willing to pay dearly to keep them running. Meanwhile, the rest of the world moves on to the new systems.
Absent vandals ransacking offices, perhaps this is what ‘collapse’ means in the context of software; the point where its added value can no longer fund its maintenance.
Cost is one way to look at it, but it’s much harder to make this argument in situations like SaaS. The cost imposed on the customer is much more indirect than when it’s software the customer directly operates. You need to have a deprecation process that can move customers onto the supported things in a reasonable fashion. When this is done well, there is continual evaluation to reduce the bleeding from new adoption of a feature that’s going away while migration paths are considered.
I think the best model for looking at this overall is the Jobs To Be Done (JTBD) framework. Like many management tools, it can actually be explained in to a software engineer on a single page rather than requiring a book, but people like to opine.
You split out the jobs that customers need done which are sometimes much removed from the original intent of a feature. These can then be mapped onto a solution, or the solution can be re-envisioned. Many people don’t get to the bottom of the actual job the customer is currently doing and then they deprecate with alternatives that only partially suit task.
Not from the same vendor. But if they’re lucky enough not to be completely locked in, once the first vendor’s system is sufficiently bloated and slow and buggy, they might be willing to consider going to the competition.
It’s still kind of a rewrite, but the difference this time is that one company might go under while another rises. (If the first company is big enough, they might also buy the competition…)
If you were hooked by the first sentence of the article mentioning The Collapse of Complex Societies, you might get quickly addicted to The Fall of Civilizations podcast by Paul Cooper. It’s one of my favorites. I love learning about how things fail(ed).
One of my favourite books, and I’ve long thought Tainter’s analysis could be applied to software, perhaps even more so than in societal collapse. This is a good writeup.