1. 44

    Let’s see. I’ve:

    • brought down credit card processing for a national retailer
    • cut off all Internet access to a major United States military installation (twice!)
    • helped bring down an entire automobile manufacturing facility
    • helped break all internet access via cell phone for one of the major US cell networks
    • shipped incorrectly optimized code that caused the (important, air-gapped) system to become accidentally quadratic

    So, you know. Be careful.

    1. 21

      I can relate to this, though I wouldn’t say I’m on the same level as you, there’s for me a long way to go. Two of my most significant achievements:

      • Brought down the debit card processing for a multinational bank issuer (only on one country sadly, and just for 2 hours).
      • Deleted the entire payment records database for a local private university on a semester, nobody noticed it, and I managed to restore it using the only backup in existence: an Excel sheet I was using to verify data.
      1. 9

        I managed to restore it using the only backup in existence: an Excel sheet I was using to verify data.

        Deeply in awe right now.

        1. 7

          To make things more exciting, that Excel file was just in memory, so a complete computer crash would have left me helpless. I learnt my lesson and do double checks on data edition, it is funny now how I make sure about doing things right even when I’m just changing a value for parameter on a file.

      2. 14

        Sounds like you’ve had a productive and interesting career, then!

        1. 15

          plot twist: he’s an intern

        2. 5

          Did you face hard consequences

          1. 15

            brought down credit card processing for a national retailer

            No. I fixed it before it became a problem and explained to the rest of my team what had happened.

            cut off all Internet access to a major United States military installation (twice!)

            First time: no, because they installed the update without testing it in their environment first. Resulted in a lot of paperwork on their end, though.

            Second time: whoo boy. I had written the compiler that turned network intrusion signatures into code that could run on our devices. I messed up the code generator for one part, so that in certain rare circumstances, an and would be an or…which meant that certain signatures would suddenly start matching just about everything. Some customers had it set up that certain signature matches would result in blocked traffic. You can see where this is headed.

            The compiler had actually been pretty extensively tested, but the problem manifested on an older configuration that didn’t have a device still in our testing pool (I know, I know).

            I had to spend a couple of days doing non-stop calls with my boss to various impacted customers, apologizing, answering their questions, and basically (and deservedly) eating crow.

            helped bring down an entire automobile manufacturing facility

            helped break all internet access via cell phone for one of the major US cell networks

            These two ended up being a lot of noise and phone calls but, ultimately, the customer had deployed stuff into their environment without testing it first. The issues on our side were from being too aggressive with what we defined as “malicious network traffic”.

            shipped incorrectly optimized code that caused the (important, air-gapped) system to become accidentally quadratic

            Not from the customer or my company, no, but from myself, very much so. I just about had a nervous breakdown, seriously. It got bad enough that I had resolved to quit as soon as I figured out what the problem was (I certainly wasn’t going to quit and leave the problem for someone else), and had convinced myself that I was just terrible at my job and had been faking it all these years. I was miserable, working long hours every night for weeks trying to figure out the problem, constantly preoccupied, not enjoying time with my family.

            Finally figured out the problem, got the fix in, and ended up staying and being reassured that I didn’t suck, which was nice.

            (Moral of this last story, database query optimizers can sometimes make the wrong decision so don’t assume that the solutions they pick in your test environment are the ones they’re going to pick in the field…)

            I actually got a job offer during that time from what might have been my dream job. I turned it down because I didn’t want to leave my current company in a bad state. I don’t know if I made the right decision, but I’m happy, so I suppose that all worked out…

            1. 2

              Thanks for sharing.

        1. 1

          https://angristan.xyz/ - I write about what I like, mainly tech-related stuff (linux, containers, zfs) and more, like my experience as an exchange student in Korea!

          1. 1

            You have some good technical content out there on your blog. I’m reading NextDNS on your blog. Hoping my brain will understand everything so that I can use this service :-)

            1. 1

              Great! Let me know if you have any questions!

          1. 0

            Interesting, I didn’t know about BuildKit!

            1. 1

              In the end I went with the LG 27UL600 instead of the HP Z27 because the LG has 99% sRGB Colour space coverage (the HP has around 70%) and it was over 200 CHF cheaper. Thanks for all the feedback!

              1. 2

                Good choice! I have the 27UL600-W as well. Great monitor.