This was the perfect reminder for starting my Friday morning. I’m going through a similar situation right now. Historically I’ve jumped in and pulled the heroic “fix it at the 11th hour” bullshit that the article talks about at the end after having the system fail in the ways I’ve predicted it would; this time around we’re communicating clear requirements to the other team. While I’ll still likely be keeping an eye on things and preparing for that 11th hour firefighting, I’m going to be taking a more passive approach in the lead-up and let the root causes see the light of day before just fixing it.
Yeah. You will destroy yourself if you try to fix everything on overtime. When things stay broken for a while, people suddenly get some appreciation for all the systems that “magically “ just work all the time. Suddenly there’s talk about scheduled maintenance and which resources are assigned to keep things running.
On my current project with working with an external team, I’ve been taking a more stern approach when things fall apart. Writing up very detailed reports on what all went wrong leading up to this, and action items that need to be taken so that we never have to deal with this again.
They’ve taken to those much more than up front stuff, and things have actually gotten better.
This made me very sad. I’ve been through this a couple of times and I still haven’t figured out how to convince the product/business side of org to buy into engineering work. Things have to break terribly before there’s participation. No amount of “I’ve seen this before” and “it’s going to break in this specific ways” helped me so far.
It pains me because I’m supposedly hired for my expertise and experience and then it’s disregarded. A huge waste of time and effort goes into education (and sometimes re-education) of the other parts of the org. What baffles me is that supermajority of the people are smart, cooperative, and trusting and yet I can’t convince them in this particular kind of situations. As if it’s too abstract or foreign for them.
That’s one problem with “let it fail”, actually. If people higher up in the chain of command refuse to understand that their decisions led to the failure, the outcome of letting it fail may not be a change in management decisions — it may be a summary termination of the people who created and operated the system that failed. Sure, it’s extremely bad for the company but getting fired isn’t a good outcome either.
So people who really need their jobs can’t often afford to just let it fail — their have to effectively shield their managers from the consequences of their bad decisions to keep their jobs. There is no simple solution to that.
I think it can actually be much much worse than that. I’ve seen this lead to projects getting canceled because one small team sets the rest of the team up for failure.
I work in games and the way the money works is; the team pitches game ideas (often with prototypes) to publishers (Microsoft, EA, Sony, etc), if a contract is signed there’s milestone deliverables every couple months to show that you’re still making progress and they keep signing the cheques. If you don’t deliver on what you promised contractually, the whole project could be dissolved. Like any large software projects, you need different teams in charge of different pieces, and there will be dependencies on those pieces. So it’s very easy for one team to cause widespread organizational distress…
This is not quite the same thing. At least the way I see it. To me it looks like lack of understanding of the engineering side of the business by the rest of the org. They view engineering as a black box of arcane witchcraft: in go feature requests and out go the features. No one wants to know how the sausage is made. At the same time everything is driven by customer value and so anything that doesn’t directly and immediately influence that or the bottom line is viewed as unnecessary overhead.
At times it gets almost surreal. I’ve seen requests for a feature estimate where an engineer will think aloud and among other things would say something like “and then a few days for writing tests” and would immediately interrupted with the request not to write them, or not right now. And on another occasion an engineer would propose to redo our AWS tagging strategy and tweak clean up policies to the blank stare of CEO but once they mention that we can save like $20k/y that way the ticket gets the highest priority.
One way the linked article can be relevant is that it’s likely those people have been through the same conversation before but they still don’t believe it’s important or they just hope that with enough will they can kick the problem further down the road as they have many times before.
This might not be exactly your issue, but a pattern I’ve seen is that an engineer will say something along the lines of, “You can have feature X in two weeks, or one week with lots of tech debt.” The PM/business guy/whatever will invariably want it on the shorter time scale, despite the warnings given by the engineer.
The problem is trust of course they want it faster. They don’t care about tech debt, and honestly it’s not their job. They can’t reason about it. It’s our job as engineers to make these tradeoffs.
Wouldn’t a business person understand the concept of debt? If they make a decision to go into debt and at some point comes the time to repay it it’s entirely on them if they didn’t plan for it. There’s always a pressure for news features or bug fixes or whatever that is not debt repayment. There’s only so much that can be worked into schedule by the engineering. Small amounts of debt can be recuperated that way. But that doesn’t work that well for big debt. Let’s say, after third business pivot a certain amount of fundamental engineering might be in order and going into tech debt default at that point is mostly on the business/product.
Debt isn’t bad for a business. Only too much of it.
Every time you put something on a credit card you are taking on debt. “Yes this is the cost of moving at a reasonable pace, and we can reconcile it later.”
A great idea in theory, if you know that it won’t result in all the blame being shoved onto specific people who get fired, or the people in charge will learn from it instead of just waste tons of time and money on a dead-end, or your coworkers will actually realize what’s going on and work to support you on it.
Excellent advice, applicable in a lot of situations.
One that comes to mind is under-performing peers: a lot of people react to them by picking up their slack, which makes management completely oblivious to the issue and unwilling to address it. Letting people above you “feel the pain” is a good way to convince them there is an issue, and push them into action.
What a great manager. Earlier in my career I’ve been bewildered as to why we were going down paths that I could see plain as day would not meet their objectives. Had somewhat actually sat down with me and explained the broader context… well I might still be working there!
This was the perfect reminder for starting my Friday morning. I’m going through a similar situation right now. Historically I’ve jumped in and pulled the heroic “fix it at the 11th hour” bullshit that the article talks about at the end after having the system fail in the ways I’ve predicted it would; this time around we’re communicating clear requirements to the other team. While I’ll still likely be keeping an eye on things and preparing for that 11th hour firefighting, I’m going to be taking a more passive approach in the lead-up and let the root causes see the light of day before just fixing it.
Yeah. You will destroy yourself if you try to fix everything on overtime. When things stay broken for a while, people suddenly get some appreciation for all the systems that “magically “ just work all the time. Suddenly there’s talk about scheduled maintenance and which resources are assigned to keep things running.
On my current project with working with an external team, I’ve been taking a more stern approach when things fall apart. Writing up very detailed reports on what all went wrong leading up to this, and action items that need to be taken so that we never have to deal with this again. They’ve taken to those much more than up front stuff, and things have actually gotten better.
This made me very sad. I’ve been through this a couple of times and I still haven’t figured out how to convince the product/business side of org to buy into engineering work. Things have to break terribly before there’s participation. No amount of “I’ve seen this before” and “it’s going to break in this specific ways” helped me so far.
It pains me because I’m supposedly hired for my expertise and experience and then it’s disregarded. A huge waste of time and effort goes into education (and sometimes re-education) of the other parts of the org. What baffles me is that supermajority of the people are smart, cooperative, and trusting and yet I can’t convince them in this particular kind of situations. As if it’s too abstract or foreign for them.
That’s one problem with “let it fail”, actually. If people higher up in the chain of command refuse to understand that their decisions led to the failure, the outcome of letting it fail may not be a change in management decisions — it may be a summary termination of the people who created and operated the system that failed. Sure, it’s extremely bad for the company but getting fired isn’t a good outcome either.
So people who really need their jobs can’t often afford to just let it fail — their have to effectively shield their managers from the consequences of their bad decisions to keep their jobs. There is no simple solution to that.
I think it can actually be much much worse than that. I’ve seen this lead to projects getting canceled because one small team sets the rest of the team up for failure.
I work in games and the way the money works is; the team pitches game ideas (often with prototypes) to publishers (Microsoft, EA, Sony, etc), if a contract is signed there’s milestone deliverables every couple months to show that you’re still making progress and they keep signing the cheques. If you don’t deliver on what you promised contractually, the whole project could be dissolved. Like any large software projects, you need different teams in charge of different pieces, and there will be dependencies on those pieces. So it’s very easy for one team to cause widespread organizational distress…
https://www.ufried.com/blog/continuous_amnesia_issue/
This is not quite the same thing. At least the way I see it. To me it looks like lack of understanding of the engineering side of the business by the rest of the org. They view engineering as a black box of arcane witchcraft: in go feature requests and out go the features. No one wants to know how the sausage is made. At the same time everything is driven by customer value and so anything that doesn’t directly and immediately influence that or the bottom line is viewed as unnecessary overhead.
At times it gets almost surreal. I’ve seen requests for a feature estimate where an engineer will think aloud and among other things would say something like “and then a few days for writing tests” and would immediately interrupted with the request not to write them, or not right now. And on another occasion an engineer would propose to redo our AWS tagging strategy and tweak clean up policies to the blank stare of CEO but once they mention that we can save like $20k/y that way the ticket gets the highest priority.
One way the linked article can be relevant is that it’s likely those people have been through the same conversation before but they still don’t believe it’s important or they just hope that with enough will they can kick the problem further down the road as they have many times before.
This might not be exactly your issue, but a pattern I’ve seen is that an engineer will say something along the lines of, “You can have feature X in two weeks, or one week with lots of tech debt.” The PM/business guy/whatever will invariably want it on the shorter time scale, despite the warnings given by the engineer.
The problem is trust of course they want it faster. They don’t care about tech debt, and honestly it’s not their job. They can’t reason about it. It’s our job as engineers to make these tradeoffs.
Wouldn’t a business person understand the concept of debt? If they make a decision to go into debt and at some point comes the time to repay it it’s entirely on them if they didn’t plan for it. There’s always a pressure for news features or bug fixes or whatever that is not debt repayment. There’s only so much that can be worked into schedule by the engineering. Small amounts of debt can be recuperated that way. But that doesn’t work that well for big debt. Let’s say, after third business pivot a certain amount of fundamental engineering might be in order and going into tech debt default at that point is mostly on the business/product.
Debt isn’t bad for a business. Only too much of it.
Every time you put something on a credit card you are taking on debt. “Yes this is the cost of moving at a reasonable pace, and we can reconcile it later.”
I agree. The problems start when later comes. Business seem too often convinced that later is indefinitely in the future.
A great idea in theory, if you know that it won’t result in all the blame being shoved onto specific people who get fired, or the people in charge will learn from it instead of just waste tons of time and money on a dead-end, or your coworkers will actually realize what’s going on and work to support you on it.
Apply with discretion.
Excellent advice, applicable in a lot of situations. One that comes to mind is under-performing peers: a lot of people react to them by picking up their slack, which makes management completely oblivious to the issue and unwilling to address it. Letting people above you “feel the pain” is a good way to convince them there is an issue, and push them into action.
What a great manager. Earlier in my career I’ve been bewildered as to why we were going down paths that I could see plain as day would not meet their objectives. Had somewhat actually sat down with me and explained the broader context… well I might still be working there!