1. 2

    Has monit moved away from pid files yet? I wrote Inspeqtor because I wanted something that worked with upstart and systemd, not rc.d.

    1. 1

      hi Mike, big fan of your work here!

      I haven’t tried this, but apparently you can match on a string - https://serverfault.com/questions/270316/monit-check-process-without-pidfile/285596 - albeit with limitations to the first match.

      Forgive the dumb question, but what’s the relationship (or lack of relationship) between pid files and upstart/systemd.

      1. 3

        PID files are known to be a racy hack, part of legacy Sys-V/Unix. “Modern” (anything in the last decade) daemons should be following the “new-style” notes here:

        https://www.freedesktop.org/software/systemd/man/daemon.html

        Integrating daemons into the init system and the OS means you pick up a lot of nice functionality for free, e.g. log management and tooling, start on boot, restart policy, etc.

    1. 1

      Both what to do and what not to do are important concerns. But for many people avoiding mistakes becomes the primary driver, and they fail to look up and get the whole picture because they’re busy avoiding mistakes. Excellence of design must sometimes be allowed to take precedence over mechanical mistake-avoidance.

      An even more sure route to good programming also takes into account why things go wrong, and crucially, why things go right when they do. It is critical to analyse second-order effects because frequently they are the only way to influence systems.

      1. 1

        I’m intrigued by your approach of analyzing why things go right. Perhaps due to temperament, this is something I’ve rarely done. I’d be curious to know more if you have any public write-ups (e.g. blog posts) you’re able to share.

        1. 2

          I have unfortunately never gotten around to writing down my take on it, but this article from Safety Differently is as good a start as any: https://safetydifferently.com/why-do-things-go-right/

          1. 1

            Cheers

      1. 1

        OP here: For an off-topic, behind-the-scenes mini freak-out I had, I couldn’t for the life of me decide if calling the post “how to make less” as opposed to “fewer” mistakes was itself a mistake. I went back and forth editing the title.

        Generally speaking, “fewer” would be the correct word, since the mistakes are countable. Yet somehow it sounded off to my native ears — stiff, cludgy. Perhaps the insertion of the adjective “dumb” in-between is the cause.

        What do you think?

        1. 3

          I think you’re right, the “dumb” turns it into a miniature garden path sentence - “less dumb” reads better than “fewer dumb”, but then it has to be re-parsed as “dumb mistakes”.

          I like the “less dumb” interpretation though - compare “how to make a smaller number of silly mistakes” with “how to make mistakes that are less silly but just as numerous”. Every dumb mistake wastes time you could have spent making a mistake that teaches you something.

          1. 1

            “Garden path sentences” is a great explanatory concept — thanks for bringing it to my world.

            Very much agree with the idea that if you’re gonna make a mistake, it might as well be a solid one :)

        1. 2

          I commited code that broke the build today because I doubly defined something. The worst thing is that I proof-read it. It built, but I didn’t want to run all tests because it was only a six-line diff and I didn’t want to wait for over an hour. It also passed review.

          Let’s say I’m having trouble focusing working from home with the heat.

          1. 1

            This highlights how friction can be an issue — if your tests take a full hour, it must be very trying to one’s patience. Is this something your team could theoretically speed up by throwing more hardware at?

            1. 1

              I’ve thought of this as well. The simple answer is no – the tests are already run in parallel. The more detailed answer is no – but we could at least run the tests on a different machine than the development machine so that I could check out another branch and do some actual work while the tests are running.

              In this particular instance, I could and should have run a specific testsuite which only takes about a minute but is a quick smoke test.

              1. 1

                Got it.

                The context for my question was based on a mistake I made in the past. Basically I was too cheap to pay for sufficient parallelization on our continuous integration server (CircleCI), despite my head engineer saying the tests could be run 5x+ faster. Considering the cost of an engineer’s time vs. a few extra docker container instances, it was a false economy.

                1. 1

                  With that many tests, ideally during development you run only the unit tests for the files/modules you have edited, and possibly their transitive dependencies. Bonus points if your testing tool watches for file changes and then takes a snapshot of changed files and runs tests on that, instead of on your development directory, allowing you to continue making changes.

                  Another possibility is to push your complete commits to side branches, and defer merging to your main branch to a bot that rebases them, tests them, and then merges them (assuming Git). Something like bors (combined with git-test), if you’re using Github.

                  1. 1

                    Unfortunately, there are no unit tests, and not everything is nicely modular. Every test is an integration test. The test suite consists of hundreds of thousands of integration tests.

                    This sounds horrible, but there are practical reasons that this is the way things work (and they are not all ‘no time to rewrite everything’).

                    Another possibility is to push your complete commits to side branches, and defer merging to your main branch to a bot that rebases them, tests them, and then merges them (assuming Git). Something like bors (combined with git-test), if you’re using Github.

                    We’re not, but that is a terrific idea. I just wonder how well it plays with the way things are currently set-up.

            1. 3

              These are some good tips. I recently have added more of this sort of rigor to my development process, with basically the realization that I don’t want to known as they guy who turns over code that breaks in an obvious case.

              1. 1

                Yea, it’s mortifying to have to explain that to your team / boss / customers. The programming equivalent of a terrible typo in a mass email.

              1. 2

                From a tech point of view, I’m curious about learning how to debug key-bindings at a holistic level in macos. Some binding stopped working in my vim instance, which is run in zsh, which is run in iterm, which is run on macos. I want to find a workflow for quickly figuring out the culprit in situations like this.

                For fun, I’m inviting a few friends over for a home-cooked dinner and getting my friends to bring musical instruments too. Afterwards we’ll put the chords up on a screen for our favorite songs and have a bit of a sing-song.

                1. 9

                  One of my favorite Vim books is still Drew Neil’s Practical Vim. That book is a great read and it changed how I use Vim. I’ve bought it for a lot of my co workers.

                  1. 7

                    Author of screencast here. Cannot agree more about Practical Vim being the best resource out there. I’m such a fan that when he was in town I booked a day of Vim training with him.

                    1. 2

                      I need to go back to it I think. I am not very smart and forget stuff that I don’t use, but that book was jam packed with handy hints.

                      1. 4

                        (Author here) The key factor, for me, in improving my Vim ability was to create a dotfiles README where I documented everything I learned and referred back to it again and again until the commands were seared into my fingers.

                        1. 3

                          I think this applies to almost everyone; there’s actually a bunch of things in Vim that I know exist, but don’t really use much anyway.

                          For example to select an entire { .. } block you can use va{V, but generally I’ll just manually go to one of the brackets (e.g. with [[, [{, {, or just moving the cursor) and use V%. This is rarely faster, but I got in the habit of doing it like that years ago before I really knew about text objects and such, and now I’m kinda stuck with it 😅

                          I’m mostly okay with this, since I try to optimize for cognitive load rather than absolute speed, and manually moving the cursor tends to be a bit better for that since I don’t need to think so much about what it’s going to select.

                          1. 3

                            I believe one of the greatest aspects of the design of Vim is that you can choose your own tools and level of usage of more ‘advanced’ techniques.

                            Over the years I have slowly added more and more tools to my belt - progressively enhancing, but never needing to, only choosing to when it suited me.

                            For example, it was about ten years before I started using macros. Perhaps another ten before I started moving by searching. Maybe in another ten I’ll use buffers rather than opening an editor for one file, closing it - and opening another later. I use the shell to drive, but I know I can get more of an IDE experience if I ever want to.

                            I don’t think I’d seen the ‘args’ feature before today. When I’ve done multi file edits in the past I’ve passed the names on the command line and done a ‘:n’ to move to the next file at the end of my macro (apologies if that’s wrong - I don’t do it often enough to remember).

                      1. 15

                        Ctrl-Z and fg are shell features and have nothing to do with vim. Also the “-” as a sign to read from stdin is a very common pattern in unix and many tools understand this. The “**” thing looks like fzf and has also nothing to do with vim itself. I know that I am splitting hairs a bit here, but I feel we should attribute things to the right tools and not conflate shell features with vim.

                        1. 10

                          Also “nvim” is neovim - not Vim!

                          1. 2

                            I didn’t mean to mislead so much as impress VS-Code users / command-line-shy types with the might of what you can do with vim’s nimble CLI :)

                            1. 1

                              And here I thought I was a fool for installing fzf, but no - it looks like ** simply expands for the current directory

                            1. 3

                              also related: just use / to search or :tag to jump to ctags on vim’s own prompt without needing to exit vim first :P

                              Opening a file with colon and line number doesn’t work for me; i think some plugin is mentioned but i don’t get the name. I would love that feature, but much rather have this on the :e command within vim, to copy paste error locations. For that I use :make and the :clist command family instead.

                              I’m always thrilled to find out hidden vim gems i was ignorant about, but this one wasn’t it.

                              1. 1

                                Vim-fetch is what you’re looking for :)

                                BTW I’ve got show-notes for all the plugins etc. used over here: https://www.semicolonandsons.com/episode/Vim's-Versatile-CLI

                                1. 1

                                  found the file:line plugin that works from cmdline and :e, so i did get something from it after all <3

                                  https://www.vim.org/scripts/script.php?script_id=2184

                                1. 1

                                  I got my first ever programming job after being a bit tipsy and answering a question about meta-programming on a local Ruby mailing list (Ruby Ireland) and impressing someone who had freelance work on offer.

                                  1. 27

                                    Every time I learn a new vim keybind it feels like I have a new superpower, even after 20 years.

                                    In general, I’ve paid a lot of attention to “durability” as a metric. How long is the shelf life of the skill? Text editor shortcuts, shell shortcuts and scripting for one-liners, vim, awk, sed, zsh, fuzzy finding tools like sk that I have shell shortcuts for piping results into vim.

                                    The things that have helped me to be a better programmer are the things that have let me better understand the things I build. It doesn’t matter how fast I can churn out garbage. It doesn’t matter how pretty code is if it wakes somebody up at night, so tools related to learning about operational characteristics of my programs are the ones that have made me a better programmer because a program is the thing that ultimately runs somewhere:

                                    • quickcheck paired with fault injection for deterministically replayable machine-generated tests that can avoid some of the assumptions I make while writing the code. avoids the pesticide paradox, so my code never becomes fully immune to my tests. for distributed systems, I pair this with a simulator. I hate writing tests so I really prefer to let a machine generate millions of them for me while I lazily just specify what the program is supposed to do. without a doubt this is the main tool that has improved the quality of my software the most. most bugs are in your error handling code.
                                    • linux internals. kernels are virtual machines and we write far better systems when we’re aware of what the VM underneath our programs is going to do to translate our use of memory, files, sockets and threads into things that happen in hardware.
                                    • gdb, lldb, but especially rr with chaos mode for time traveling debugging of concurrent code. shell shortcuts like this for dumping the stacks of a running process:
                                      stacktrace () {
                                        gdb --batch -ex "t a a bt" -p `pgrep $1`
                                      }
                                      
                                    • massif visualized through massif visualizer for understanding memory characteristics of a program
                                    • dhat visualized with the built-in web viewer for finding short-lived allocations that could have happened on the stack
                                    • perf for getting stats about instructions, cache misses, branch mispredictions and correlating it back to lines of code, paired with perf report, flamegraph and hotspot for visualization. Be wary of metrics that don’t actually make your important metrics better. More instructions is sometimes still faster due to fewer branch mispredictions. Often the most interesting metrics are max latency per request, minimum throughput for some time period, peak memory usage etc… because they have significant capacity planning implications.
                                    • dstat for visualizing where my system’s high-level resources are being consumed by my programs

                                    The mindset that has helped me to improve my skills is that you should always just roll up your sleeves and look at the running process and its code rather than wasting more than 5 minutes searching for help or pouring over documentation that’s out of date and never correct. Put a short time cap on the time you spend doing that, and then just put on your explorer hat and pull out your scary toolkit to hunt that thing down.

                                    1. 4

                                      This is a great list of tools. I like your mindset as well.

                                      I’m still at the junior/intermediate stage and I think the confidence to be able to “roll up my sleeves” is what’s needed to move forward.

                                      1. 17

                                        I also lacked confidence to really dive in earlier on. Two things happened at roughly the same time:

                                        • I learned about the Feynman technique. Just write down how the thing you’re trying to learn works (a bug, a technique, anything) until you hit a wall in your understanding. This makes the boundary of your understanding explicit. Often, it’s surprising how much you actually do understand before hitting the wall, but before you build confidence, there’s a bias to assume the amount you don’t understand is greater than it really is. Make the boundary explicit, and then you have a specific point to push forward from, rather than kind of a cloud of doubt and anxiety. This made it clear that I knew more than I felt I knew, and at the very least helped me hone in on specific questions to ask.
                                        • I had a couple people around me at work who would pull me over when they were having a problem, before they found the answer, just to teach me about the debugging process. They would find the bugs in languages they didn’t know at all by just simple grepping through the repo for the codebase after observing some high-level systems metric that had gone out of whack. “we get bursts of connection errors every hour”. They would apply rough binary search to the space of the unknown to make rapid progress. They would keep a short log of observed clues and observations in a text file as they went. They would make a guess about how the program maybe worked, then guess words that variables and structures may be named that related to that guess, and with kind of shocking regularity they would find the cause of the misbehavior in like 4 or 5 guesses most of the time, even if they had cloned the repo for the first time just then and there. They taught me that looking for bugs is a pretty mechanical process that doesn’t really require anything more than making guesses and keeping your fingers moving as you search for your guess. “what happens every hour? what’s in the crontab on the machine initiating the connections that fail? how many concurrent connections does it spin up? what’s the TCP backlog for the server it’s connecting to? why isn’t the server it’s connecting to closing the connections after responding to a request? what’s this streaming mode config option that’s set on the server but not the client? ohh… that’s it.”

                                        Digging is mechanical and just foraging around in the dark guided by your best guesses. Your guesses are better than they feel like they should be. Your guesses get better and better over time, but that doesn’t matter so much, because even with bad guesses the process is still mechanical and tends to terminate after a short number of guesses. The biggest thing is knowing that bad guesses will still drive you forward. They illuminate the negative space. Keeping a log as you go lets you use your full attention on the current question being asked, rather than trying to remember all of the clues you’ve observed up to that point (gets bigger and bigger as you go along). Mechanically apply the Feynman technique with your log until you arrive at the answer. Knowing super high-level architecture of how things are wired up is helpful to know up-front, but if not, that’s just another mechanical thing to learn to guide your search into the cause.

                                      2. 2

                                        I just wanted to chime in that this was a truly inspiring post. I fully agree with your philosophy and you’ve given me a bucket list of tools to experiment with the next time I’m stuck. There’s nothing worse for me than retrying something ten times based on shoddy out-of-date docs – why not investigate the source?

                                      1. 4

                                        Podcast work. Doing one’s been on my bucket list for a while and considering the barrage of hellthreads here, I can focus on this instead of getting distracted so often. The commitment was made as soon as I bought the domain name.

                                        1. 3

                                          We have hellthreads? :) What have I been missing? :)

                                          Also yay new podcast! Good luck. It’s a lot of work but totally rewarding IMO.

                                          1. 2

                                            What’s the podcast?

                                            1. 3

                                              You know how people are probably tired of RPG podcasts where some band of misfits pillage and plunder? Except this is more me and my nasaly-ass voice doing the same thing, but single-player and how to play these things alone.

                                            2. 2

                                              Woo! I’m always looking for new podcasts, be sure to link it here when ready! On a sidenote, that “just bought the domain name” motivation is just about the strongest there is.

                                            1. 2

                                              Fun:

                                              Sit with my new ukulele and sing scales up and down while figuring out how different chords harmonize with the notes. E.g. In key of C major, the scale is C D E F G A B C. One possible harmonization is I V I IV V IV V I (C G C F G F G C). Another is IV V I IV I IV iii vi. etc.

                                              Work-ish:

                                              Record a screencast about browser integration testing pro-tips. Looking at things like removing brittleness by having a special unchanging test-interface, auto-failing upon all JS errors, being careful about clearing state between tests with various dependencies (e.g. redis, elasticsearch, mailbox, caching layer, etc.) Context: I do non-beginner programming screencasts about production code-bases here: https://www.semicolonandsons.com/series/Inside-The-Muse

                                              1. 2

                                                C G C F G F G C

                                                You should also try minor chords! For example Am Dm Em F G F G C.

                                              1. 1

                                                Combination of stuff:

                                                • building software for booking and subsequent transfer of tests results for a fleet of COVID-19 testing centers.
                                                • realizing that Pride and Prejudice is one of the wittiest pieces of writing ever created. My gateway drug was the 1995 mini-series and I immediately bought the Audible e-book afterwards.
                                                • recording part-II of my screencast episode about programmatic SEO (I’m an indie developer and manage to wrangle 200k page views). First one here https://www.youtube.com/channel/UC17mJJnvzAa_e9qQqLIfIeQ?view_as=subscriber