This is a common pattern in the service-oriented computer industry, sadly, and oftentimes it’s not even deliberate, it’s just how things happen:
According to him, unstable solutions requiring frequent maintenance meant more revenue.
Technologies that are good enough to be useful and reliable, but obscure and/or janky enough to require expertise disseminated largely by word of mouth and/or frequent servicing and customisation tend to spawn very lucrative service industries around them. These, in turn, result in a large expertise base and lots of options for outsourcing, and quickly become popular.
For example, up until early 2.6 or so (and even later for some subsystems) Linux wasn’t particularly good in terms of portability, at least not compared to NetBSD, the poster child of BSD portability. Assembling a board-specific Linux system was pretty awful, and the various solutions proposed to alleviate that in the world of embedded systems (e.g. Yocto) were so janky that they looked like an improvement only because many of the alternatives looked like torture. But it had enough compelling advantages that people did it anyway, and required enough effort, across large enough teams, that it quickly built a large base of experts that could then outcompete any NetBSD shop.
Yocto, pretty much the de facto option in the corporate world, is kind of similar. Early on, it was a peculiar pile of quirks, and its documentation was worse than useless – it was the long form equivalent of those “// increment i by one” comments above lines that say i++ as it described almost no useful idioms, and it was woefully incomplete. The only way to learn it was to join a team that was using it and look at what other people did, and the first thing you did in almost every task was look at an older git commit for something that looked similar and try to replicate it. It was more flexible than most other solutions, and more easily extended, but it owed much of its easy extensibility to its rather freeform nature: it had no syntax and sanity checks, of any kind, on anything, so you spent forever hunting down bugs caused by things like spaces between list items or configuration keys called ENABLE_FROBNICATION instead of FROBNICATION_ENABLE.
So it was enthusiastically adopted by lots of corporate players 10-15 years ago, but it was also obscure enough and janky enough that anything based on it quickly ended up requiring ample, easily-outsourced maintenance. That, in turn, spawned a very lucrative massaging-Yocto-BSPs-until-they-build service industry that wielded astronomical marketing budgets and could quickly displace lots of in-house development efforts based on other, more robust, but less technically accessible solutions, through sheer workforce availability.
It doesn’t all have to be bad. Linux’ portability story today is actually very good, especially if you consider where it was twenty years ago. Yocto was still pretty dreadful last time I touched it, but many of its internal shortcomings were fixed over time, and it developed a lot of excellent tooling that isolates you from the remaining brittleness.
Unfortunately, I do suspect this isn’t true in every corner of the computer industry. I’ve met a lot of folks “upstairs” who held poorly-performing IT and security audit teams in very high esteem because they were busy all the time – mostly staying on top of problems they couldn’t reliably fix and, respectively, producing fancy-looking reports that were mostly useless and unactionable but facilitated finger-pointing.
Nice and heartening story. I wonder what it’s like for true systems consultants nowadays. Whether it’s still possible to pull one’s weight and override the client’s wishes/opinions about implementation in the interest of fulfilling their wishes about outcomes.
Somehow I get the impression that nowadays, if you’re not moving customers “to the cloud” or at least to K8S, you’re not finding gigs easily. If anyone has experiences to the contrary, please let me know.
I’m sure that server was still running despite the shitty hardware because it saw so few reboots. The only surprising thing is that the hard disk didn’t die
Is that really so surprising? I am not doing that at Backblaze scale of course, but regarding consumer hard drives so far if they don’t die in the first three years and aren’t in some horrible environment (being thrown around a lot or not properly installed, but level (either completely horizontal or vertical) I never had a failing drive. All the ones that failed did so pretty quickly (< 3 years) or due to physical circumstances. Talking about spinning ones here.
I also used to buy old workstations for fun. These are not consumer grade of course, but they simply never died, even though these workstations were ancient and started out at either a CAD company or university and had other owners in between.
If anything, I’ve found that old (>3y) drives seem to be more likely to fail if they aren’t in regular use. Even if they do start developing issues, they tend to be of of the “a few bad sectors here and there” kind, which you might not even discover for a long time if the load is pretty light.
Missing a little bit of context. Were system updates automatically applied? How many critical security updates that require a reboot has NetBSD had in the last decade?
Got a Mac IIci here running NetBSD, which it’s done since 1999. It had to get a recap and a new hard disk, but it’s still going, doing its thing (printing, internal DNS, internal DHCP, internal AppleShare).
This is a common pattern in the service-oriented computer industry, sadly, and oftentimes it’s not even deliberate, it’s just how things happen:
Technologies that are good enough to be useful and reliable, but obscure and/or janky enough to require expertise disseminated largely by word of mouth and/or frequent servicing and customisation tend to spawn very lucrative service industries around them. These, in turn, result in a large expertise base and lots of options for outsourcing, and quickly become popular.
For example, up until early 2.6 or so (and even later for some subsystems) Linux wasn’t particularly good in terms of portability, at least not compared to NetBSD, the poster child of BSD portability. Assembling a board-specific Linux system was pretty awful, and the various solutions proposed to alleviate that in the world of embedded systems (e.g. Yocto) were so janky that they looked like an improvement only because many of the alternatives looked like torture. But it had enough compelling advantages that people did it anyway, and required enough effort, across large enough teams, that it quickly built a large base of experts that could then outcompete any NetBSD shop.
Yocto, pretty much the de facto option in the corporate world, is kind of similar. Early on, it was a peculiar pile of quirks, and its documentation was worse than useless – it was the long form equivalent of those “// increment i by one” comments above lines that say
i++as it described almost no useful idioms, and it was woefully incomplete. The only way to learn it was to join a team that was using it and look at what other people did, and the first thing you did in almost every task was look at an oldergitcommit for something that looked similar and try to replicate it. It was more flexible than most other solutions, and more easily extended, but it owed much of its easy extensibility to its rather freeform nature: it had no syntax and sanity checks, of any kind, on anything, so you spent forever hunting down bugs caused by things like spaces between list items or configuration keys calledENABLE_FROBNICATIONinstead ofFROBNICATION_ENABLE.So it was enthusiastically adopted by lots of corporate players 10-15 years ago, but it was also obscure enough and janky enough that anything based on it quickly ended up requiring ample, easily-outsourced maintenance. That, in turn, spawned a very lucrative massaging-Yocto-BSPs-until-they-build service industry that wielded astronomical marketing budgets and could quickly displace lots of in-house development efforts based on other, more robust, but less technically accessible solutions, through sheer workforce availability.
It doesn’t all have to be bad. Linux’ portability story today is actually very good, especially if you consider where it was twenty years ago. Yocto was still pretty dreadful last time I touched it, but many of its internal shortcomings were fixed over time, and it developed a lot of excellent tooling that isolates you from the remaining brittleness.
Unfortunately, I do suspect this isn’t true in every corner of the computer industry. I’ve met a lot of folks “upstairs” who held poorly-performing IT and security audit teams in very high esteem because they were busy all the time – mostly staying on top of problems they couldn’t reliably fix and, respectively, producing fancy-looking reports that were mostly useless and unactionable but facilitated finger-pointing.
Nice and heartening story. I wonder what it’s like for true systems consultants nowadays. Whether it’s still possible to pull one’s weight and override the client’s wishes/opinions about implementation in the interest of fulfilling their wishes about outcomes. Somehow I get the impression that nowadays, if you’re not moving customers “to the cloud” or at least to K8S, you’re not finding gigs easily. If anyone has experiences to the contrary, please let me know.
I’m sure that server was still running despite the shitty hardware because it saw so few reboots. The only surprising thing is that the hard disk didn’t die
Is that really so surprising? I am not doing that at Backblaze scale of course, but regarding consumer hard drives so far if they don’t die in the first three years and aren’t in some horrible environment (being thrown around a lot or not properly installed, but level (either completely horizontal or vertical) I never had a failing drive. All the ones that failed did so pretty quickly (< 3 years) or due to physical circumstances. Talking about spinning ones here.
I also used to buy old workstations for fun. These are not consumer grade of course, but they simply never died, even though these workstations were ancient and started out at either a CAD company or university and had other owners in between.
If anything, I’ve found that old (>3y) drives seem to be more likely to fail if they aren’t in regular use. Even if they do start developing issues, they tend to be of of the “a few bad sectors here and there” kind, which you might not even discover for a long time if the load is pretty light.
Missing a little bit of context. Were system updates automatically applied? How many critical security updates that require a reboot has NetBSD had in the last decade?
From the Article it seems systems were never updated.
Got a Mac IIci here running NetBSD, which it’s done since 1999. It had to get a recap and a new hard disk, but it’s still going, doing its thing (printing, internal DNS, internal DHCP, internal AppleShare).
It’s interesting that the author does not seem to draw the conclusion that consumer grade hardware is sufficient despite having this experience.