1. 47
  1. 27


    I’m going to quote myself from about five years ago:

    Lots of people will give you guff about “devops” and how it means that you no longer need people to run your business infrastructure because it is all a simple matter of programming.

    I cannot emphasize enough how wrong this is.

    What devops does is apply the lessons learned in software development to the processes of running computer infrastructure. As a direct result, you can improve your processes by minimizing human involvement in the execution of processes. What you cannot do is improve your processes without knowing what those processes are, how they are executed, and what might go wrong.

    Ask any developer: can you solve a business problem without understanding it? No, you cannot. Programming is the expression of decision-making in a formal automation. You cannot make (good, or even reasonable) decisions without understanding the process.

    The naive devops-uber-alles view concentrates on building an MVP, a minimum viable product, and then elaborating on it in customer-desired ways. The ops part is relegated solely to deployment, the process of getting the application running on one or more servers.

    Here is what operations brings to the table:

    statistics collection
    systems monitoring
    alerting of humans based on the above
    load balancing
    strategic deployment decisions (e.g. multiple vendors, geographic distribution)
    backup and restore decisions and methods
    security policy
    security implementation
    security testing

    In modern development, it is considered good to write tests to ensure that the code you write does the right thing. In modern devops, you need to implement runtime tests to assure yourself that the computers you are running on are still behaving as you desire.

    Without operations knowledge, you cannot do this. Without development methods, you cannot do this.

    1. 7

      Lots of people will give you guff about “devops” and how it means that you no longer need people to run your business infrastructure because it is all a simple matter of programming.

      Couldn’t agree more. I think one of the worst things to ever happen to this industry is when people glommed onto the word who had exactly zero concept of what the actual ideas was trying to do. The results have ranged from annoying to disastrous.

      When I see people confused about what the term means, I like to throw Gene Kim’s The Phoenix Project at them, because it lays out the problems being solved and why the Dev+Ops philosophy evolved to help solve them.

      Most of the people causing the problem here have never read the book and never will because it’s far too much effort.

      1. 3

        I’m happy “The Phoenix Project” was suggested to me a few years back. I’m glad it hasn’t been forgotten. Might be worth a re-read.

        1. 2

          Same here. The book really resonated with me when it came out because I felt like the organizations I was working in at the time REALLY suffered massive productivity losses due to stupid STUPID silos.

        2. 1

          For the little it’s worth I read the Phoenix project, twice actually. And it’s mostly about Agile methods over “devops”.

          Though as I say in the post; it was originally Agile Systems Administration so I guess that makes sense.

          1. 1

            That’s interesting. I see agile as a methodology encouraging rapid iterations based on constant feedback from stakeholders.

            I’ve always seen DevOps as a philosophy / methodology that seeks to break down the silos that keep people from innovating everywhere they could and strives to bring some of the discipline of software engineering into an operations world where “I don’t code” used to almost be a badge of honor.

            I see lots of similarities between the two but I see devops acting at an organization wide level where I see agile in its purest and most original form acting at the level of an individual practitioner.

        3. 4

          I think it’s a more subtle trap than that.

          The danger is not so much that in the absence of an operations team you have no statistics collection or backups or alerting or so forth. Those things can be, and often are, put in place by developers in that kind of environment. Sometimes reactively rather than proactively if the dev team doesn’t have much prior experience wearing the ops hat, and often with a barely-adequate minimum of functionality, but it’ll still happen and it’ll often be good enough to get you good word-of-mouth from your initial set of customers.

          The danger is more that you’ll do those things poorly without knowing it, and if your product ends up succeeding, it will come back to bite you because you’ll think those operational details are already taken care of and don’t need further attention. Once you’re successful, the stakes will be a lot higher. At least when you don’t do it at all, you know you haven’t done it at all. It can be a lot harder to realize you’ve done a crappy job, or to justify the cost of replacing a quick-and-dirty placeholder.

          That’s the trap. “We need to hire an ops person now because we can’t launch the product at all without their expertise” is a convincing argument you can make to management. (In the days before AWS and friends, that was pretty much a given.) “We can launch the product well enough for customers to try out, and we can cobble together basic alerting and backups, but it’ll all kind of suck and we might be super sorry at some point if we don’t hire an ops person to do a competent job of it,” I can say from my own frustrating experience, can be an impossibly hard sell, especially at scrappy cash-constrained startups but also in a large organization with tight budgets.

          At startups in particular, I think it gets even tougher when management can, in good faith, point to examples of companies that didn’t hire any ops people at all and ended up getting acquired before the shoddy sort-of-ops practices caused a major disaster. Kind of hard to talk someone out of rolling the dice at a company whose entire existence is a roll of the dice.

        4. 9

          I like the other category of DevOps person: the AWS/GCP/Azure embedded sales representative.

          1. 5

            Closely followed by the pricing consultant who shows you ways to reduce your bills by 25% or more.

            Occasionally both employed by the same firm.

          2. 4

            To give my own impression (and it’s really just subjective impressions, so please don’t understand that as judging or some hard facts on anything), when I look over the last decade in that job the changes are:

            • You call it DevOps (maybe SRE) if you want to have a better income
            • You outsource more and more work to cloud providers, pay more for it and spend time configuring what has been outsourced. Instead of some spec sheet/contract for the details on what you wish for, you use something like Terraform to define what you want for them.
            • Various problems that used to be solved decades ago (esp. around scaling, security, monitoring at large) are now new again, because there is some shiny new tool that wants to do it differently

            It took me a while to realize that I’m essentially the cloud provider equivalent to a vacuum salesman. Of course I need some technical knowledge, but the salesman also needs to know some basics about vacuum cleaners (which is easy, because they are a common household item).

            Initially a lot of what one does feels like the tech job one got into, but at some time I realized it really wasn’t. The things that used to be interesting are now the outsourced parts, a lot of what one does is very high level and the focus lies on talking to abstractions (APIs, etc.), work around problems with those abstractions, etc.

            The “skill” is to know some (end user) products, not to work technologies. This in my view is similar to a sales consultant. That’s is not a bad thing. There’s nothing wrong with working like a Xerox, Microsoft, etc. partner/consultancy always did. However, if your goal of starting a certain career path was to work on technologies and not essentially selling them and write contracts it’s understandable that you can get frustrated. And depending on the exact situation this might not be obvious, because you are clearly working with tech. It can be a matter of perspective, but maybe being unhappy and frustrated with that job, when it feels like one shouldn’t be, maybe that’s why.

            I’m really curious on how things will develop. It seems like another layer of abstraction emerging. S3-Buckets are now mostly an unofficial standard, and many companies offer them, the same is true for Kubernetes (even though that one is still a lot less stable). Terraform also does act as a standard for defining orders for cloud providers

            On top of that there’s various Heroku clones created by big cloud providers, I think that’s mainly, because new companies start out as devs only. Depending on how things develop, maybe that makes the DevOps role (and similar) less important for companies in the long run.

            I think that could mean that the position of an SRE or DevOps person (tech salesman with sysadmin experience) would slowly move closer to the cloud provider, where I assume is also the closest thing to a Sysadmin/Op. After all it didn’t disappear, it just got outsourced.

            1. 4

              “devops” and “fullstack” are “we need a name for when we try to hire one person to do the work of three people so we can save money (read: pay executives more money)”

              1. 2

                I always find the term ‘full stack’ amusing because it’s always said by folks that work on such a tiny bit of the stack. My group does CPU ISA design and prototype implementation, compiler implementation and language design, OS design and implementation, and occasionally does some application things. Occasionally ‘full stack developers’ see our job ads and quickly realise that they don’t actually know anything about any of the layers of the stack that their code runs on.

                1. 1

                  That’s one way of looking at it. Another way is “we need these three people to all work together as a team rather than belonging to separate silos in the organization”.

                  If a company adopts your definition, one should run away from it quickly (or ask for more money up front).

                  1. 2

                    The point I’m making is that it’s literally impossible to know the definition because it differs from person to person.

                    1. 2

                      Interviewer: Any questions?

                      Me: Sure. This is considered a devops role. Can you tell me about the philosophy of devops at your company? How does it work here?

                      1. 1

                        “What does DevOps mean to you” is actually something I ask.

                        But why should I ask it, and why should it get to that stage?

                        We should just fix the terminology to be universally agreed.

                        1. 1

                          This is now reminding me of the XKCD Standards comic lol

                2. 2

                  You see the same with “fullstack developers”, who are usually either backend or frontend developers who can also do the other thing a bit.

                  I’ve actually done quite a bit of sysadmin stuff on production systems over the years, as well as a lot JavaScript frontend stuff. But at the end of the day I’m I’m mostly just a backend/systems programmer who can also do a bit of the other stuff.

                  1. 2

                    IME most “fullstacks” are better at frontend than those who would cling to being “frontend-only” because they still care and are willing to learn things.

                    My big beef about the word “fullstack” is it presumes webdev is the only dev. Most “fullstacks” are missing experience in many stacks, such as OS, embedded, rich client, or networks other than http

                    1. 2

                      I agree with your sentiments as I was thinking about how I should update my CV. This devolved into thoughts on how other people perceive a person’s skills and how I’ve tried to think about them when hiring.

                      The model from Football Manager (formerly, Championship Manager) for player attributes came to mind. It provides a summary of:

                      • All attributes (which are not that useful alone)
                      • Possible playing positions with their strongest, along with a style suggestion (are they more frontend than backend?)
                      • Then they provide a summary visualization of strengths with implied weakness in the radar chart (what’s the correct name for this?)
                      • I haven’t played recently, but I think in the game it caters for the multiple stacks by changing the context to other playing positions and rolling up relevant attributes into the radar chart.

                      This could be applied to any set of roles you are hiring and the summary visualization can be overlaid for comparison. You could also use it as a way to communicate the requirements for a preferred hire. It’s all food for thought, but I think it’s an interesting way to think about skillsets given how complex we’ve made the requirements.