1. 10
  1. 21

    There is no small irony to me that somebody who sells monitoring tools for a living is pushing to ship more often better faster stronger.

    Some of us don’t like, say, debugging a deploy while cooking stirfry.

    Also, the bit about policies vs judgement rings hollow to me–policies let your eng org have reproducible results and shields the folks who made mistakes. They can and have been used badly, but used well they save everybody innovation tokens.

    1. 11

      I guess the author doesn’t even realize that not every developer is working with web? My deploy is pushing a binary to Google play, Apple store or Steam. By the time it gets through review and I get user reports about things being broken it will be deep into Saturday.

      1. 8

        Two good reasons not to deploy on Fridays, at least in my environment (in-house tool development):

        • Your deploy can be absolutely perfect technologically speaking and still break someone’s experience by making a seemingly minor change. You do not want the director, already cranky because they’ve stayed late on a Friday, unable to figure out that the button for that thing is over there now.
        • You should have some time every week to try out ideas that might not work, experiment with that new library, and do other speculative things that may or may not pay off. If you don’t deploy your main products on Fridays anyway, that creates space for that kind of work.
        1. 5

          Thanks for all your (well, for most of your) comments! A few things,

          • I don’t sell monitoring tools for a living. Thanks tho!
          • My opinions are not those of my company. This is a personal article. (If I sold CI/CD software, maybe all the snark about my “marketing strategy” would make more sense.)
          • Yes, the title and many of my arguments are ridiculously hyperbolic. I was leaning perhaps too hard against all the assholes on twitter who were hyperbolically accusing me of not giving a shit about people or weekends. Sorry!
          • Most of you are completely failing to engage with my actual point, which is that deploy hygiene is worth investing in, because it lets you move faster and with more confidence.

          Many of you are suffering under the burden of truly dysfunctional orgs and frequently-broken deploys and miserable on call rotations, much of which could be addressed by zeroing in on how you ship software, and dedicating some cycles to improving it – instead of simply whining about how out of touch I am or how hard these things are or how test coverage is never perfect.

          Also, please stop acting like deploys are the only thing that cause your systems to break, or like shipping only adds risk. Also please stop acting like a mandatory freeze on progress for 20% of your work hours is a small price to pay for a dubious reward which runs counterproductive to its own stated purpose. As I said.

          One of my favorite articles, worth reading: https://www.intercom.com/blog/shipping-is-your-companys-heartbeat/

          1. 2

            I don’t sell monitoring tools for a living. Thanks tho!

            That was a (hopefully understandable) mistake that I made. We were looking at using Honeycomb at a prior gig to augment our ops suite, and it got bucketed in my heads with monitoring/ops tools.It’s technically a service for observability. Your talk on it was really neat and interesting and what got us interested in it.

            Most of you are completely failing to engage with my actual point, which is that deploy hygiene is worth investing in, because it lets you move faster and with more confidence.

            Agreed!

            Many of you are suffering under the burden of truly dysfunctional orgs and frequently-broken deploys and miserable on call rotations, much of which could be addressed by zeroing in on how you ship software, and dedicating some cycles to improving it

            This is all true! At a previous gig, we went from having weekly outages to having maybe one every few months because we were able to get enough covering fire for the business to stop banging on about features and let us fix glaring issues. Issues like having no monitoring, period. Horrifying stuff.

            That said, you show a remarkable lack of empathy for folks like me who were and are in situations like that.

            In a large org, in a better world, in a different system, sure. It’d be awesome to say “hey business, just give us a few days and a few dollars to go build something to make it easier and faster to fix things”. But, that’s not the opportunity a lot of us get. Using hyperbole to tell us that our (hard-earned) methods, suboptimal though they may be, are garbage and then dismissing the (real and valid) concerns and lived experiences of us as whining is not a good look.

            If you want to sell a product, don’t tell somebody their thing is shit. Tell them how your product can make their existing stuff even better.

            Also, please stop acting like deploys are the only thing that cause your systems to break, or like shipping only adds risk.

            Once a system is running at steady-state, the errors and issues experienced tend to be things we’ve seen, the customers know how to deal with, and support can paper over until things’re properly fixed. Code changes tend to be start/stop in nature, and even a simple fix that is “obviously” safe for a deploy can result in new and exciting failures. Those failures usually are riskier than whatever they were fixing, because they’re something the org hasn’t learned how to compensate for.

            Also please stop acting like a mandatory freeze on progress for 20% of your work hours is a small price to pay for a dubious reward which runs counterproductive to its own stated purpose.

            Not having to debug a production issue while cooking stirfry is not a dubious reward. :)

            It’s not a freeze on progress for 20% of the work; it’s a hedge to help avoid having to spend 2/7 of my week and 100% of the time I’d rather be using for not work fixing something!

            If you want to take the extreme approach in the other direction, why doesn’t every commit get pushed into production immediately? That’d be really cool and some companies I think even do something similar with feature branches and CD and A/B testing. But, a lot of teams (I’d bet more than 90%) don’t do that because it’s hard (expensive) to provision maintain with limited engineering resources when the business is breathing down their necks.

            ~

            A lot of this difference in outlook is actually brought in to sharp focus by your linked article about shipping as an org’s heartbeat. Code deploys show signs of life and progress on a product.

            Extending the analogy, though, there is such a thing as tachycardia, whereby a heart beating too quickly result in inefficient pumping and actually harms the body. We want our software shipping at clean regular intervals, without arrhythmia, so that the rest of the company body can get the most value from it.

          2. 5

            It’s all about Return Of Investment. For small-to-medium organisations spending a lot of time (and thus money) on advanced CI/CD pipelines often just isn’t good ROI.

            It’s also not an easy problem which requires specific expertise that many don’t have. What this may mean for smaller organisations is that after the person who set up the entire shebang leaves the rest of the company is left with this … thing that no one understands.

            1. 8

              The title is bad enough that I’m not reading the article.

              1. 4

                It’s helpful to understand the frame of reference the Author has here. Charity was an eng/ops person at Facebook, before going to found her own startup. FB culture, as we all know, is “deploy constantly” and “move fast”. Her message isn’t a terrible one, “Make sure you can deploy anytime, all the time, without fear” but it’s clearly not applicable in many cases. Personally, it feels like she’s leaning WAY TOO hard on this, because it’s provided her some marketing movement.

                1. 2

                  FB only very recently achieved continuous deploys, and I never really worked much on FB systems. My frame of reference is much more Parse, Linden Lab, Honeycomb etc. Not sure where you think there’s any marketing value to Honeycomb; I feel passionately about this topic because I have seen it transform teams and their productivity, when they make even marginal improvements to their pipeline.

                2. 4

                  The author is completely blind to the fact that it is a good idea, simply because it is nice for the people you employ. It also shows that you as a company respect their time off and the fact that their lives might not revolve around work.

                  Frankly I think that the whole virtue-signalling argument is useless. It doesn’t matter that it is counter productive. As a person working in your company, it is just simply very nice to have one day of the week, right before the two days off, on which you don’t have to worry about stuff breaking because you’ve had one solid day of field-testing already.

                  Also: Please try to remember that tests are not always adequate, and that not every engineer at your company is capable of delivering perfect code all the time. You’ll be glad that you’ve built this kind of safeguard into your organization when you are responsible if something goes wrong, especially, if it’s on Friday.

                  And I did not even get started about “real mission critical systems” where failure simply is not an option. Systems like that cannot be deployed more than once or twice a month and a deployment must happen in the scheduled time window.

                  In my humble opinion, it doesn’t matter whether or not your application is a web application, something mission critical, or something else, because It is a mark of a professional, to know why, when and how to act, but what differentiates a regular professional from a good professional, is that the good professional also knows why, when and how not to act, even if it impacts productivity, morale and work ethic.

                  No matter how urgent the patient’s condition is, you won’t find a qualified doctor or surgeon whom rushes in to treat the patient, without washing his or her hands and putting on hygiene gloves. They also try to avoid doing complicated surgical procedures on Thursdays and Fridays by the way.

                  I think the same principles should apply to every piece of software, and therefore I also think that the CI/CD methodology is inherently naïve and fundamentally flawed, while it also throws your employees into a meat grinder at the same time.

                  It’s not like your users will know that the feature X has been delayed to next week….. And if they do, please fire some people in your sales and/or marketing department for making promises before the product is finished.

                  1. 3

                    I don’t understand what this has to do with CI/CD or monitoring. Often a push to production means data migrations. Even if you’re not using a relational DB or even if you are using a relational DB and not pushing a schema change, new code may produce data that the old code may not understand.

                    And if that happens and breaks stuff after a few hours, a rollback + DB recovery from a snapshot won’t be an option either. What do you do then? You may have to push an emergency fix. So the question is; Do you want to do that on a Sunday morning?

                    1. 2

                      Ermm.. do you want to be doing that on a Tuesday night? Wednesday night? I don’t want to get paged out of hours period, therefore by your logic I can deploy never.

                      1. 2

                        It makes me sad to receive this kind of a message on lobste.rs… I makes me sad because you know exactly what I’m going to say next: “I can deploy never” is not an option, so you pick the next best option of deploying some time that has the least likelihood of causing trouble when you don’t want to work.

                        Besides, if you’re on a remote multi-national team, you might even have someone within their working hours 24 hours a day during week days.

                        So, the rational comparison is the statistical benefit of not deploying on Friday vs. making Friday’s commits wait for another business day before they go to production. In my experience, commits get clogged for many reasons and occasionally much longer than a day. Requesting a code review usually means a couple of days.

                        Also note that deploying on Monday doesn’t necessarily mean chunking Friday’s commits together with Monday’s commits. You don’t have to merge into master from the tip of dev. First thing on Monday morning, you can merge the last commit of Friday morning, and at 10:00 AM you can merge a few more etc. if that’s what concerns you.

                        So, I support your message on CI/CD being super important, automate as much as you can! But I wouldn’t support Friday evening commits regardless of how good your CI/CD infrastructure is.

                        1. 0

                          I specifically said that deploying on Friday night, or any night right before you go home, is stupid. Ok, so you didn’t read what I wrote at all. ☺️ Cheers.

                    2. 3

                      The author is clearly working with a precise audience that needs this message desperately and is taking desperate measures to make sure they don’t overlook it. While it’s amusingly hyperbolic I think really the heart of it is, if you don’t think it applies to you please take a moment to consider if it should. Lots of people don’t think CI/CD applies to them, when really they might see a lot of value from it. For those of you who can’t or shouldn’t, don’t.

                      1. 2

                        I think the broad advice given in this article is good (for web deployment anyway) - developers should be able to ship code quickly and with confidence, and if they can’t or feel like they can’t, that’s a problem that is worth fixing in and of itself.

                        That said, what most people actually mean by “don’t deploy on Friday” is “don’t deploy right before normal working hours end”, and it seems like the author doesn’t actually disagree with this advice. Way down the page, in the section labeled “Good Judgement Matters More Than Rules”:

                        Don’t ship before you walk out the door on any day.

                        Don’t ship big, gnarly features right before the weekend, if you aren’t going to be around to watch them.

                        Advising people to make a judgment call about how complicated their feature is, and if it’s too complicated not to deploy it near the end of normal working hours, is really a difference in degree and not kind from “don’t deploy on Fridays”.

                        1. 2

                          I’m sure there are other things you can do with a Friday than write probably breaking changes. Things like documentation, and tests. Maybe simple refactoring.

                          It is not exactly like murdering puppies. At least that activity has no long term rewards like better documentation.

                          I think read only Fridays are more similar to OpenBSD’s freezes before cutting new versions. Code gets in, just not features, and the platform is more stable for it.