Threads for kylerisse

  1. 1

    Interesting alternative to dependabot.

    1. 5

      Echo my gratitude for NOCOWS.

      Had a similar experience to the author when I needed to build and maintain home grown Mesos and Docker Swarm clusters. It was much nicer to work with Ansible to coordinate that effort across nodes than the tool of choice at the time. Then it became the main tool, I dove in deep, and I’ve never looked back.

      1. 10

        I love little tricks like this where a particular bit of language can help you write readable code. Another one I really like is making sure commit messages complete the phrase “if merged, this commit will…”.

        1. 5

          “if merged, this commit will add the ability to delete users”


          This commit adds the ability to delete users.

          Even better:

          Adds the ability to delete users.

          1. 12

            I think you misunderstood @dstaley’s comment. They’re not suggesting you write “If merged, this commit will add the ability to delete users” as the commit message, but “Add the ability to delete users”, using the imperative mood.

            What you wrote is also good, I think, and it’s down to personal taste whether you want the imperative mood (”Add the ability to delete users”) or present tense (”Adds the ability to delete users”).

            1. 2

              Ah, my bad. He’s proposing the tpope git message style, which is what I use. I misread it as advice for a verbose message body.

            2. 1

              Just wait until someone decides all commit messages need to be stories.

              “As a committer on this project, I want to add the ability to delete users, so I can…”

            3. 3

              Yes please, great rule. I work with a group that enforces religious use of rebasing and useful commit messages in PRs. We don’t allow GitHub to do squash and merge. We follow your “if merged…” completion style, it just wasn’t defined in those terms. Well stated. So much nicer than what I’ve seen previously.

            1. 1

              This is always a fun one for this topic:

              Don’t take this as a knock on datadog. They make a great product. Much respect.

              1. 2

                The value is that they bundle all the integrations into a single binary, and it’s easy for them to do because they can pull in the client lib.

                1. 1

                  Indeed, and it‘a very useful. Although this isn’t the agent, but rather their tracer. So it gets pulled in to your binary, but same idea. It’s just jarring to see your go.mod blow up as a result of pulling parts of this in for the first time.

              1. 2

                Short little post about Fallacies of Distributed Systems. These are some thoughts I feel often get downplayed when engineers are designing microservices.

                1. 1

                  These are spot on. Thanks for posting.

                1. 8

                  Had a similar experience when I was super green. I had two shells open, one to a test Oracle DB and another to its production counterpart. Accidentally rebooted production. Doh!

                  I once worked with a DBA who setup his sessions specifically to avoid this scenario. Production windows were white text on a red background. QA was yellow on black. Dev was green on black. Worked for him as he was always working with full access regardless of environment.

                  I agree with the conclusions of this article. Always connect to production with read only credentials if the intent is to simply inspect data. This isn’t always possible when dealing with things like PG parameters but generally works if you want to just run queries. Although depending on the DB even a bad query can cause issues depending on indexes and such… If the DB is on RDS then one can work with a snapshot. It’s also reasonable to ask why nobody questioned the dev needing full access to production.

                  What I’ve always though about is the butt clenching factor. If you’re logged into a production system and not “clenching your butt cheeks” level nervous and double checking commands before hitting enter, then you’re not taking the task seriously enough. This has served me well for 20 plus years. Even now, I prefer to be pairing on screen share for a sanity check when typing potentially destructive commands. Shimming a level of “peer review” in is better than dealing with restorations and “postmortems”.

                  1. 2

                    I always give the admins I write a different color scheme in dev vs prod.

                    1. 1

                      I once worked with a DBA who setup his sessions specifically to avoid this scenario.

                      Oh yeah, totally a useful and underrated technique. For me, prod systems have a red shell prompt, standard terminal/utility/jump-server systems are green, and my non-work systems are a variety of other colors so I don’t get confused by which window is my desktop and which is SSH’ed into some random thing.

                      What I’ve always though about is the butt clenching factor. If you’re logged into a production system and not “clenching your butt cheeks” level nervous and double checking commands before hitting enter, then you’re not taking the task seriously enough.

                      Wise words, and ones that I should probably pay attention to more. Familiarity breeds contempt and all that.

                    1. 3

                      This is cool to see. The use v5.36 defaults seem reasonable. The n-at-a-time iteration over a hash is pretty neat. I like the addition of defer, and by extension finally blocks. There are a few other things that look interesting.

                      Seems like the Perl Steering Council is having a positive impact on moving the language forward.

                      I still hold out hope that raku will gain in popularity but I guess improving Perl5, and eventually Perl7, is the next best option. Better yet, why not both?

                      1. 22

                        can confirm the grinding frustration will continue. the best way to mitigate it is to iterate as fast as you can. take a zero tolerance policy to slow modify-compile-execute-reflect loops.

                        it doesn’t matter if it’s a binary on your desktop or some rube goldbergian enterprise fizzbuzz on cloud.

                        if you can iterate in 1s, it’s less frustrating. way less. it can even become fun.

                        all of my github projects are about faster iteration, so i can be less frustrated. the loop shrinking will continue until morale improves. this is the way.

                        faster loop on aws:

                        faster loop on webdev:

                        faster loop on services:

                        faster loop on browser testing:

                        faster loop on react:

                        faster loop on docker containers:

                        faster loop on distributed data processing:

                        1. 1

                          This is a great philosophy, thanks for sharing your projects

                          1. 2

                            of course! life is too short for needless frustration.

                        1. 9

                          there is only 1 hard problem in cs: state.

                          naming things and cache invalidation are state problems.

                          1. 1

                            Haha, I love this. You simplified one of my favorite quotes.

                          1. 23

                            Interesting, thanks for writing.

                            The problem you run into with Ansible (as an example of a stateless solution) is that removing resources from your config doesn’t ensure they’re removed from the cloud without some care. So say I create a VPC in the YAML, run Ansible and it gets built, then I remove it from my YAML and run Ansible again, the VPC will continue to exist indefinitely without adding some other steps.

                            By contrast, Terraform with state typically ensures that the resources will be cleaned up once removed from the HCL.

                            In theory you’ll end up with much more cruft over time the stateless way. Whether or not that is more painful than working with terraform state is a compelling question. I think it depends on the team size and apply process.

                            1. 4

                              This is exactly correct. When Terraform was very early on, I think 0.3 or some such? Well before data resources – I initially did an on-prem to AWS migration solely using Terraform.

                              Unfortunately, it wasn’t quite up to snuff so after a few months I rewrote it all in Ansible, which ended up being far, far more verbose and had all the problems you listed. From an operator pov it ‘felt good,’ though.

                              Had I to do it again, I would likely use Ansible to ‘bootstrap’ all the Terraform stuff (s3 buckets and iam users/roles/keys) and do the rest with TF. Shooting for 100% Terraform (or really 100% only one tool) is usually the wrong path.

                              1. 4

                                At a previous company we had a bootstrap terraform repo that would setup the basics for the rest of the automation, like backend buckets and some roles. The bootstrap terraform used local state, committed to the repository. It worked well enough.

                                1. 2

                                  My approach is generally to use a bootstrap Terraform stack which creates the buckets and so on that Terraform needs, then change the bootstrap stack to use the bucket it created as its state backend. Having some backups is useful (as with all statefiles) but it’s generally and easy and safe approach.

                                2. 2

                                  That’s thought provoking. I wonder if it would be reasonable to run Ansible (or some other stateless tool) in a mode where it went looking for things that didn’t exist and removed them. The flaw there would be that no Ansible config sets up an entire system from first principles, but assumes there are some tools already in place.

                                  Maybe git or other source control could be used to store take on the state burden to detect removal.

                                  1. 7

                                    The downside of that idea is that it is extremely common to have multiple Terraform workspaces managing resources at the same provider. If you did “destroy all resources that aren’t in your configuration” you’d end up removing, say, all the QA infrastructure every time someone did an update to production if the two are managed separately.

                                    1. 2

                                      terraform et al are great in theory. it’s in practice where they often fall apart. it’s one of those domains where sanity must be enforced though disciplined conventions, which is hard.

                                      1. 1

                                        If such culture existed, then ansible, terraform et al would probably never spring to existence. This usage is very much motivated by a mindset of throwing a flashy tool at a problem, rather than understanding the fundamental problem and how to control it.

                                        For example, using yet another AWS service out of their line up of hundreds, is a choice that is rarely questioned. Then that device has its own challenges… Ok, AWS offers yet another service to ‘solve’ them, and so on. There is no time for discipline in this reality… Just keep adding stuff and hiring engineers to cope with the system untill the next company mass lay off.

                                    2. 2

                                      This is a classical problem with stateless tools, going as far back as make.

                                      1. 2

                                        This is also a thing in Puppet (another stateless system). The answer is that you have to put the absence of the resource in the config (in Puppet terms, ensure => absent). Then when the config has been applied everywhere it needs to be, you can delete that “tombstone” resource. For some resources there is also a purge attribute that means to delete all unmanaged resources found in a scope (like a directory).