This was causing service disruptions all over the place.
Regardless of whether or not rolling ahead with version B fixed the problem “in the end”, the team doing this completely blew all their service level objectives in the meanwhile during the transition period.
I’m not sure this is the best example to support the point the author is trying to make. Yes, there were service disruptions, but it sounds like everything was 100% in the end and they were running the newer service/code base/whatever. They obviously got lucky though.
Here’s what could have gone wrong: Hours later, version B is discovered to have a huge, directly customer-impacting bug, and rollback to A is now impossible without a break-before-make outage.
Sure, but it seems like it didn’t. My point was that the story the author chose was not a great example of why you shouldn’t do something. A better story would have been the situation you described.
Hope is not a strategy. They saw a problem during the rollout, hoped that it was the only problem, and rushed to complete the rollout. It might not have caused an outage, but I’m sure it caused a lot of stress for the people involved. Our jobs and lives don’t need to be this way.
I think you’re talking past each other a bit. Hope is a very poor strategy indeed, but it’s also true that, as a rhetorical device, using an example where everything turned out okay despite doing the wrong thing makes for a weak story.
At the very least the author could have taken some time to describe a worst-case scenario. Otherwise readers who don’t already know how bad this is are left asking themselves, “What’s wrong with picture? Why was this so bad if everything turned out okay?”
It doesn’t matter if this particular occasion ended up not too badly, that’s just bad engineering. And these articles are great cautionary tales for all of us, so I don’t see your comment being of any help at all. Nobody is disagreeing that they worked it out somehow.
Regardless of whether or not rolling ahead with version B fixed the problem “in the end”, the team doing this completely blew all their service level objectives in the meanwhile during the transition period.
I’m not sure this is the best example to support the point the author is trying to make. Yes, there were service disruptions, but it sounds like everything was 100% in the end and they were running the newer service/code base/whatever. They obviously got lucky though.
Here’s what could have gone wrong: Hours later, version B is discovered to have a huge, directly customer-impacting bug, and rollback to A is now impossible without a break-before-make outage.
Sure, but it seems like it didn’t. My point was that the story the author chose was not a great example of why you shouldn’t do something. A better story would have been the situation you described.
I agree that that would have made a better story, even to just highlight that possibility.
Hope is not a strategy. They saw a problem during the rollout, hoped that it was the only problem, and rushed to complete the rollout. It might not have caused an outage, but I’m sure it caused a lot of stress for the people involved. Our jobs and lives don’t need to be this way.
I think you’re talking past each other a bit. Hope is a very poor strategy indeed, but it’s also true that, as a rhetorical device, using an example where everything turned out okay despite doing the wrong thing makes for a weak story.
At the very least the author could have taken some time to describe a worst-case scenario. Otherwise readers who don’t already know how bad this is are left asking themselves, “What’s wrong with picture? Why was this so bad if everything turned out okay?”
Hindsight 20/20, can’t live on that.
It doesn’t matter if this particular occasion ended up not too badly, that’s just bad engineering. And these articles are great cautionary tales for all of us, so I don’t see your comment being of any help at all. Nobody is disagreeing that they worked it out somehow.