1. 14

  2. 15

    MySQL is a database with many, many quirks, but it is so widely used, that you should still just use it most of the time… Shit on it all you want, but there are very few PHP issues that someone else hasn’t already encountered.

    Haskell and PostgreSQL are also boring, but in my experience cause fewer headaches.

    In fact, using better-but-still-boring stuff has been amazing leverage for my business.

    1. 12
      1. Buy Almost Always Beats Build

      I’ve spent far too much time playing integration engineer, when building a tightly written solution would have solved the company’s problem much faster, more reliably, and in a more targeted fashion.

      1. 2

        Indeed, “buy” inevitably almost always means: expect it to take a week to integrate? Quadruple that, it always takes longer than you expect. But build can also be a black hole of time sink as well, but buy always scares me at times.

        1. 1

          This cost of integration is especially true when you need to write new features, and must code around this bought codebase that you may or may not have enough control over to modify to your purposes.

          On another note, the article is… provocative, and I feel it’s only exposing a very narrow view of a particular (small?) type of business. There are big considerations for most of their points that, because they don’t seem to apply to the author’s work, they dismiss them for the whole industry.

          1. 2

            There are big considerations for most of their points that, because they don’t seem to apply to the author’s work, they dismiss them for the whole industry.

            I find this far too common in blog posts, I’d like more people to list their priors and situation. I can guarantee that at some scales the Buy versus Build scales tip.

            Probably boring to say: Buy versus Build depends upon your circumstances, team, ability, and requirements. There is no silver bullet and you must evaluate every decision on its own merits. And don’t be afraid to be wrong and change your plan later if the situation changes.

      2. 6

        A more direct heading for this section would be “Non-Production Environments are Bullshit”. Environments like staging or pre-prod are a fucking lie.

        Ah, to live in a world where “the live thing” is the only thing that matters and traffic also the only thing that matters.

        In the environment I work in right now (and no, of course staging is not 100% like production, but it gets most of the production “traffic” mirrored to it) the fast deploys aren’t acceptable not because we don’t dare to do it but because it’s on a shop floor in a factory and the customers don’t tolerate even a second of downtime on the conveyor belt.

        The systems also don’t get much traffic in the sense of “this is webscale” - the problem indeed is when we can deploy in a production break because everything will stand still.

        This is an area where Chaos Engineering can be helpful. Injecting failures into production can improve confidence in how to respond when a system starts behaving in unexpected ways.

        Sure, a 500 on a website can be remedied with a quick refresh, too bad the industry doesn’t like a multi-thousand EUR thing being trashed because someone wanted to test failure.

        And yes, under certain circumstances you might be able to argue your way into getting all that fancy stuff that has come up the last 5-10 years (mostly in the internet world) into an industrial plant, but I doubt it will happen soon.

        Lastly, regarding

        Engineers should operate their code.

        I don’t like this. Responsibility needs agency. If I am just the small cog in the dev team and cannot build the solutions I think are the best, or most reliable (of course if a portion of the team agrees, not because I am the special super developer) but downright stupid decisions are made (sometimes they may be valid tradeoffs) from management, or from customer’s ecosystem / infrastructure - then nope, no fucking way.


        Right now I have a kafka installation where the VMs hosting said kafka nodes crash all the time. If this affects my code running there? Not my problem, I would have deployed a stable kafka setup if I depend on it. Sadly we’re not allowed to do that.

        Back in a very different job we developed a web app, certain performance criteria were put into the contract, after the fact it was decided to host this on a pair of EC2 instances. They behaved a little odd and were horribly slow. The customer for whom we were building the web app wouldn’t let us analyze and redo the EC2 instances. After a while I was fed up and grabbed a several year old non-developer laptop (some i5 from 2010 with 2 GB RAM) and ran both of our frontend nodes (virtualized) on it and aced the performance requirements. I certainly would not have wanted to be on call for that EC2 setup.

        Overall a mostly good article, but some of the insights need either a big caveat or some perspective shift. This sounds like “We built this non-critical thing on the web (in a product company)”.

        1. 4

          I am a researcher, neither a developer nor a product builder; that being said, I find many arguments in this write-up very appealing. To some extent, I think it is because one side-effect of applying this is that it keep everything at a human size.

          Providing a way to let a developers feel connected with the end-result of their work looks important to me.

          1. 4

            if you’re not on-call for your code, who is?

            While each developer needs to have ownership over the code they write, the team also needs to have ownership over their feature sets. You don’t want your teams to be isolated engineers fighting their own battles. You want your teams to act as teams and fight battles together. Especially in production, your team’s main objective should be solving the problem, not finding out who to pass the issue off to. Obviously, you should pull the people in with the the most domain knowledge to solve the problem as fast as possible, but you never want to get caught casting blame and passing the buck.

            QA Gates Make Quality Worse

            I have worked in two distinct QA structures. One of them seems to be what the author is describing: a dedicated QA team that catches code tossed over the wall. This was a miserable way to work and did cause a lot of conflict that was described. But, I’ve also worked on teams with embedded QEs. This proved to have far better results. The QE assigned to the team was knowledgeable about the features being produced and worked closely during the planning to determine what the testing scenarios should be. This allows them to provide other insight and scenarios that the engineer may not have written tests for. It also helps with standardizing a lot of testing procedures within the teams for each given product.

            Boring Technology is Great

            I love this advice. Unless there is a business need for new and shiny, stick with what you know. It reduces friction allowing products to be produced faster with a higher quality.

            Things Will Always Break

            There’s a pretty big dichotomy here. You have to accept some risk when deploying new code and it is inevitable you will deploy bugs to production. That doesn’t mean you shouldn’t analyze the risks and ensure you’re mitigating the risks that could have the greatest negative impact. This is especially true when you’re dealing with systems that have a effect on physical objects. If you deploy a high-risk change set that requires physical labor and manual intervention to correct the cost to the company or clients is far greater than one that can be corrected by just another deploy. These need to be taken into account. There are times when you have no choice but to deploy those high-risk change sets, but you should have done your leg work already and identified it as a risk and be ready with a plan in case it goes wrong. But, if you’re able to reasonably mitigate this risk, you should do so ahead of time.