“Lock all the nerds in a room and don’t let them leave until they figure it out” is something I have actually heard working many years in ecommerce, particularly on Black Friday. It strikes me as management theater. If the pandemic lockdown taught me anything it’s that a few people who know and care about the affected systems working in an environment of their choosing can work much more quickly than an under-ventilated roomful of random people who don’t know each other and would understandably rather be anywhere else.
Also…
Unfortunately, production (many many machines) was running the last release which had been cut WELL before that point.
I have been burnt by this many times, especially in deployment workflows where there is no way to tell for sure what’s running on prod. (I wonder in this case whether the release was tagged. In some shops the “release” is just whatever the commit was at deployment time.) It’s now my first question in a prod crisis: “What version is running on prod?” (And if I have any reason to doubt it, “How do we know?”)
It seems like collectively CI workflows are converging on using a branch (i.e. main) as a mirror of what’s on production, but there is still some scattered resistance, to which I would argue that knowing what’s on prod and being able to roll it back quickly is almost always more important than whatever safety people think they’re buying themselves by making prod deployments convoluted and opaque.
“Lock all the nerds in a room and don’t let them leave until they figure it out” is something I have actually heard working many years in ecommerce, particularly on Black Friday. It strikes me as management theater. If the pandemic lockdown taught me anything it’s that a few people who know and care about the affected systems working in an environment of their choosing can work much more quickly than an under-ventilated roomful of random people who don’t know each other and would understandably rather be anywhere else.
Also…
I have been burnt by this many times, especially in deployment workflows where there is no way to tell for sure what’s running on prod. (I wonder in this case whether the release was tagged. In some shops the “release” is just whatever the commit was at deployment time.) It’s now my first question in a prod crisis: “What version is running on prod?” (And if I have any reason to doubt it, “How do we know?”)
It seems like collectively CI workflows are converging on using a branch (i.e.
main) as a mirror of what’s on production, but there is still some scattered resistance, to which I would argue that knowing what’s on prod and being able to roll it back quickly is almost always more important than whatever safety people think they’re buying themselves by making prod deployments convoluted and opaque.