1. 3

    Jeebus, how were these engineers hired?!

    Around 10:00 AM, someone asked when we were going to start the switch, and Mike chimed in helpfully, “We’ve already started reprovisioning the v3 servers.” We had so little capacity that we had decided to reimage all our existing servers and then reprovision them in the new software stack. This was clever from the perspective of reducing our costs, but the optimism it entailed was tinged with madness.

    They literally had no rollback plan. None at all.

    1. 2

      While I’m usually a lot more conservative in my ops work, upon occasion it makes sense to take a no-prisoners approach.

      In glorious people’s startup it takes more courage to rollback than to deploy! O_O

      1. 1

        It’s like the kids say nowadays, YOLO.

      1. 4

        I use Leuchtturm notebook: http://www.amazon.com/Leuchtturm-Medium-Notebook-Squared-LBL12/dp/B002CV5H4Y

        The pages are numbered, and comes with a blank index at the beginning. Its pretty awesome.

        As for my “random note during the day”, I use Remember the Milk on my phone https://www.rememberthemilk.com. They have a single icon widget that lets you instantly jot down anything that gets added to a list for future review.

        1. 2

          Thanks, this looks like exactly the kind of notebook for me.

        1. 2

          Unfortunately, long weekends and holidays are when infrastructure, websites, etc are the most vulnerable. Often there is a skeleton crew left, and often they come from the bottom of the engineering totem pole.

          The ability to firefight is both hampered technically and logistically. I feel sorry for however much time Linode wasted escalating to the senior NOC network engineers.

          1. 1

            I can’t even begin to describe how much of a loss this has been. We honestly had (and have) developers who think that changing all the line-ending whitespace was really important. Or that whether there are some “legacy” K&R-style prototypes actually matters. These people go around committing non-functional changes, which makes source management (and specifically patching and merging) really painful, and which, unfortunately, also often breaks things accidentally. What’s worse is that sometimes they don’t even bother trying to compile it, much less test it. I wish I was making this up; I wish more that I had made these people go away a long time ago.

            Unfortunately these people exist everywhere. They seek to make themselves look busy/important, while changing the code to make themselves feel more comfortable. Its the Peter Principle in code.