1. 24
  1.  

  2. 9

    A classic! I know that the point of this story is that over-engineering is bad and to remain practical, and that is how I read it way back when. But now I read it and think its a commentary on the ludicrous way that we structure our companies. We have the people in charge so detached from reality that they refuse to even talk to the line workers! (I mean, that is really what the story is about, right?)

    And of course upper management, rather than talk to the engineering department to actually figure out what is needed, decides engineering is “too busy” and goes with an external consultant (because its not like the engineering team will then have to maintain it…). And this consultant wants to milk the company for what its worth (and damn right they should), so they try to push the most complicated and expensive solution they can. Middle managers love this – they get to manage an $8M project!

    In this case the CEO actually followed up on the solution and saw the end result being a complete waste of time – but in my experience this is the least realistic part of the story! There is no way in hell the CEO would check in on things, everyone would just pat themselves on the back all the way down the chain.

    Even if the company hadn’t gone with an external consultant, I would not be surprised if the in-house engineering came up with a similarly over-engineered solution. Upper management thinks its a big project, so they are willing to put up real resources, and rather than selling a simple solution, opportunistic engineers can sell a complicated and expensive one (boosting their resume and status at the company). I see it all the time. Usually its not even done maliciously: every engineer will take full advantage of an exciting project rather than drudge through the usual tasks, and I am definitely guilty of this at times.

    I’ve been at startups for the past few years because I thought there would be less bureaucracy, but that has only been the case at one of the startups I was at. Unfortunately it was also the only one not doing super well (very relative – it was the only profitable one, just not growing whatsoever. Additionally it was the only one not completely over-leveraged with VC funding… Hm…).

    Perhaps I am naive, but there must be a better way to structure a company, right?

    1. 3

      I would not be surprised if the in-house engineering came up with a similarly over-engineered solution. Upper management thinks its a big project, so they are willing to put up real resources

      Middle management are subject to a simple force: If only two or three people get assigned to the plan, and it fails, their bosses are going to have pretty intense questions about why you didn’t use the rest of your team. They need a plan complex enough to let them bring a large group to bear on the problem to protect themselves.

      1. 1

        Exactly! Everyone has incentives that I feel lead to overcomplicated solutions and ultimately hurt the company.

    2. 6

      I internalize the story as an argument for pager duty and on-call support roles.

      The problem that upper management faces is that QA can be kind of bad and is reflected in the large scale metrics on customer impact. The cost of bad QA can absolutely be real to the bottom line but not evident in the people in the process. Either you pay it in customer support, reputation, or a user exodus. In this way, it’s no different than what I work on where services or products sometimes go down.

      The solution they created is actually like a system status pager duty. When a detectable error occurs, manual intervention occurs. This applied pressure to certain people in the pipeline to deal with the issues. It sucks to shutdown a processing pipeline and might even be not cost effective, but it forces the pain onto an internal individual instead of your external customers. I.e. we should be catching these problems before our customers realize it’s a problem. Along those lines, while I hate pager duty and on call support roles, I think I’ve also felt empowered to fix some of the worst parts of our system because I now have a concrete goal with my work, to reduce my pages. It turns out necessity ins the mother of invention. I end up solving problems when those problems are on me.

      1. 5

        Along those lines, while I hate pager duty and on call support roles, I think I’ve also felt empowered to fix some of the worst parts of our system because I now have a concrete goal with my work, to reduce my pages. It turns out necessity ins the mother of invention. I end up solving problems when those problems are on me.

        I’ve found this true as well. It’s easy as a developer to become isolated from the actual code that you’re writing, even at a small company. Sure the bugs bother you and you fix them as quickly as you can, but for me at least that always feels pretty abstract. Being forced to deal with an actual customer’s real life problem that was caused because you were being a little lazy one evening two months ago is an eye opener.

        3.5% of today’s users experienced a minor bug

        feels a lot less impactful than

        Jan in Brooklyn is having trouble putting in her data because someone broke the uploader code and she really needs it done by the end of the day because she’s got a big meeting tomorrow and needs to present the results.

      2. 6

        While it’s a cool story and all, if the employee would not have felt the pain via alerts, he would have not found a better workaround. So yes, at the end, $8,000,020.0 well spent.