I agree with the conclusion. People make mistakes, it’s true, it’s fine. But part of designing a system that has people as a part of its functioning is insulating the system from peoples’ mistakes, ideally without being too inconvenient. I’ve heard second-hand stories that in the 1940’s and 50’s aviation industry, lots of fatal crashes happened and were chalked up to “pilot error”. In the 1960’s or thereabouts, that stopped being considered an acceptable answer, and requirements for pilots got a lot stricter – and lots and lots of long-tail hardware and control issues started being ferreted out of aircraft.
That said, high reliability has costs. It probably isn’t a coincidence that this is also around the time that lots of independent aircraft companies started going out of business or merging into bigger ones, and new ones stopped being created at as fast a rate.
Had a similar experience when I was super green. I had two shells open, one to a test Oracle DB and another to its production counterpart. Accidentally rebooted production. Doh!
I once worked with a DBA who setup his sessions specifically to avoid this scenario. Production windows were white text on a red background. QA was yellow on black. Dev was green on black. Worked for him as he was always working with full access regardless of environment.
I agree with the conclusions of this article. Always connect to production with read only credentials if the intent is to simply inspect data. This isn’t always possible when dealing with things like PG parameters but generally works if you want to just run queries. Although depending on the DB even a bad query can cause issues depending on indexes and such… If the DB is on RDS then one can work with a snapshot. It’s also reasonable to ask why nobody questioned the dev needing full access to production.
What I’ve always though about is the butt clenching factor. If you’re logged into a production system and not “clenching your butt cheeks” level nervous and double checking commands before hitting enter, then you’re not taking the task seriously enough. This has served me well for 20 plus years. Even now, I prefer to be pairing on screen share for a sanity check when typing potentially destructive commands. Shimming a level of “peer review” in is better than dealing with restorations and “postmortems”.
I once worked with a DBA who setup his sessions specifically to avoid this scenario.
Oh yeah, totally a useful and underrated technique. For me, prod systems have a red shell prompt, standard terminal/utility/jump-server systems are green, and my non-work systems are a variety of other colors so I don’t get confused by which window is my desktop and which is SSH’ed into some random thing.
What I’ve always though about is the butt clenching factor. If you’re logged into a production system and not “clenching your butt cheeks” level nervous and double checking commands before hitting enter, then you’re not taking the task seriously enough.
Wise words, and ones that I should probably pay attention to more. Familiarity breeds contempt and all that.
It’s peculiar, but at some deep level, all fear must be the same.
Which says a similar thing about courage, I expect. IIRC from reading about this some years ago, the ancient Greeks didn’t distinguish between physical and moral courage.
There is also --i-am-a-dummy mysql CLI option to prevent LIMIT-less UPDATEs/DELETEs, although I’m not sure if it would have helped in the TRUNCATE TABLE scenario.
I agree with the conclusion. People make mistakes, it’s true, it’s fine. But part of designing a system that has people as a part of its functioning is insulating the system from peoples’ mistakes, ideally without being too inconvenient. I’ve heard second-hand stories that in the 1940’s and 50’s aviation industry, lots of fatal crashes happened and were chalked up to “pilot error”. In the 1960’s or thereabouts, that stopped being considered an acceptable answer, and requirements for pilots got a lot stricter – and lots and lots of long-tail hardware and control issues started being ferreted out of aircraft.
That said, high reliability has costs. It probably isn’t a coincidence that this is also around the time that lots of independent aircraft companies started going out of business or merging into bigger ones, and new ones stopped being created at as fast a rate.
Had a similar experience when I was super green. I had two shells open, one to a test Oracle DB and another to its production counterpart. Accidentally rebooted production. Doh!
I once worked with a DBA who setup his sessions specifically to avoid this scenario. Production windows were white text on a red background. QA was yellow on black. Dev was green on black. Worked for him as he was always working with full access regardless of environment.
I agree with the conclusions of this article. Always connect to production with read only credentials if the intent is to simply inspect data. This isn’t always possible when dealing with things like PG parameters but generally works if you want to just run queries. Although depending on the DB even a bad query can cause issues depending on indexes and such… If the DB is on RDS then one can work with a snapshot. It’s also reasonable to ask why nobody questioned the dev needing full access to production.
What I’ve always though about is the butt clenching factor. If you’re logged into a production system and not “clenching your butt cheeks” level nervous and double checking commands before hitting enter, then you’re not taking the task seriously enough. This has served me well for 20 plus years. Even now, I prefer to be pairing on screen share for a sanity check when typing potentially destructive commands. Shimming a level of “peer review” in is better than dealing with restorations and “postmortems”.
I always give the admins I write a different color scheme in dev vs prod.
Oh yeah, totally a useful and underrated technique. For me, prod systems have a red shell prompt, standard terminal/utility/jump-server systems are green, and my non-work systems are a variety of other colors so I don’t get confused by which window is my desktop and which is SSH’ed into some random thing.
Wise words, and ones that I should probably pay attention to more. Familiarity breeds contempt and all that.
Which says a similar thing about courage, I expect. IIRC from reading about this some years ago, the ancient Greeks didn’t distinguish between physical and moral courage.
There is also
--i-am-a-dummymysql CLI option to preventLIMIT-lessUPDATEs/DELETEs, although I’m not sure if it would have helped in theTRUNCATE TABLEscenario.I didn’t know about that! Thanks!