1. 3

    What’s the performance like compared to 1.16’s wasm output? I remember back in the early days of wasm that Gopher JS actually beat wasm in several benchmarks.

    1. 1

      Great question. I’m sure nobody has done benchmarks yet. I’d love to see them, too!

    1. 11

      Agile is a manifesto, a movement. The movement created/adopted/popularized a pile of tools from standups to storyboarding to kanban. You’re meant to pick and choose what fits your needs to help you get in sync with your clients/customers – and change your choices if they need change, when the goals change! That’s agility.

      Scrum is prescriptivist management that takes on the facade of Agile. It’s rigid, conformist, and designed for managers to have control over the processes and output. The absolute opposite of agility.

      1. 10

        You can’t have it both ways. Either Agile is a prescriptive system that can be implemented, described, analyzed and criticized.

        Or it’s a lofty goal, like “balance”, “correctness” or “wealth”, that by itself doesn’t give you any meaningful guidance on how to get there.

        Agile proponents seem to be perfectly fine talking about their pet system as a panacea that will Rid us of Bureaucracy and Bad Decisions, but when criticism comes, they quickly turn the tables and shout YOU WERE DOING IT WRONG.

        1. 4

          Scrum can be implemented well IMO, but it’s rare. Often it’s a top-down mandate from management seen as a silver bullet to solve their productivity problems (hint: it’s not a solution, it may even make things worse). I’m at a small startup and we’re using a very loose version of scrum (2 weeks sprints, story pointing, etc.). It wasn’t mandated, we decided it was a decent rough framework to keep track of our productivity and plan out milestones and we don’t follow it to a T if it’s not working for us.

          1. 7

            I tend to think of this as “Scrum can be agile, in spite of itself, but it’s rare.”

          2. 2

            There is even a church about Agile: http://www.thechurchofagile.org/

          1. 16

            I’m surprised you didn’t talk about learning to use interfaces effectively. To me, this is the most non-intuitive part of Go for most newcomers (including myself at that stage).

            But to comment on your actual examples… The first example you provide demonstrates a concious choice on the part of the Go authors. Of course, it’s anyone’s prerogative to agree or disagree with the merit of that objective. And that object is that Go values ease of readability over ease of writing.

            In that vein, Go is “easy”. It’s obvious, how v is removed, to any casual observer. The cyclomatic complexity is staring you in the face. That is easy, in a sense–the sense the Go creators intended.

            1. 3

              I don’t think that interfaces are really a good example of “it can be quite hard to combine simple concepts to do $useful_stuff”, which is the “deeper issue” here, and also not something that’s unique to Go, or even programming languages: microkernels (or indeed, microservices) kind of suffer from the same issue.

              I do agree interfaces can be tricky to use effectively; but it’s a bit of a different issue than what I wanted to explore.

            1. 1

              @jhall you list several things NOT to use debug logs for, but what would you actually put in debug logs? I have rarely found debug logs useful.

              1. 3

                Yes. My advice to folks who are just starting out with good practices for reliability is to make metrics, not logs, and definitely not log-based metrics. I have heard that comments are like apologies to the next programmer; similarly, logs are apologies to the next operator.

                1. 1

                  comments are like apologies to the next programmer

                  I love this! Any idea where it originated?

                  1. 1

                    I believe it originated with @washort and came out of discussions that we had had about how comments and docstrings should work in Monte.

                2. 3

                  I like them for explaining why a decision was made by business logic. Say, there’s a reasonably complicated set of rules, which results in a pass/fail decision. If you put some debug logging in each fork of the decision tree, it’ll be hidden normally, but you can change the level for that package/log zone/whatever when you’re getting a wrong answer and it just might give you enough information to fix it quickly without having to step through and see where it goes wrong.

                  1. 2

                    I actually pretty strongly dislike debug logs. I don’t think I ever use them. In my experience, they’re mostly used by inexperienced developers, who aren’t yet up to speed on writing properly modular code, and proper tests.

                    But I haven’t yet spent enough time looking into it to come out with a blanket statement that I think they’re an anti-pattern… although that is the direction I’m leaning.

                    1. 5

                      I think that’s very dismissive. Some people (like me) use languages that don’t have a very good debugger, and write complicated logic. What should I use to find what’s going on in my code, if not printf-debugging?

                      1. 2

                        I think the very assumption that only developers need debug logs is wrong. Users may want to see program’s decision flow and internal data when they encounter an unusual problem, or when the program is something highly configurable and complex. Not all users are familiar with debuggers, and not everyone can easily build from source, so debug logs can be quite a time saver for them. They can also attach those logs to forum posts/bug reports.

                        1. 1

                          @c-cube It’s not dismissive. I was expressly explaining my experience.

                          But expounding on my experience, debuggers are usually overrated. I think a good test suite can replace a debugger in the vast majority of cases. A good test suite also replaces the need for debug logs in most cases.

                          @peter I’ve thought about this a bit more, and there are clearly some times when debug logs are appropriate. I use debug logs in third-party services all the time (think: apache, nginx, etc).

                          But this is a different sense of “debug” than what I’m reeling against. This is helping the user of an application debug things, not a developer of the application.

                          Perhaps “dev debug” logs make sense in some context, too. I’m happy to see this has sparked an enlightening discussion…

                          1. 2

                            I think a good test suite can replace a debugger in the vast majority of cases.

                            Arguments like this read to me like “don’t ship bugs and you won’t need a debugger,” (1) which is fine but I think it assumes a certain relationship between the debugging human and the program that isn’t actually true in the “majority of cases”, at least in my experience.

                            When you need the debugger you often don’t have the test suite, or it is not comprehensive to cover the problem, and it is not often the case that it is low friction enough to debug by writing test cases, and this happens a lot (in the “vast majority of cases”) because the person dealing with the bug didn’t get to write the test suite or the program.

                            When I get to start from scratch “debug via tests” certainly can work quite well. But mostly I don’t.

                            1: I know this is a slightly unfair characterization, it is more like, use your test suite to avoid bugs, and if one is discovered write more tests to explore/describe the bug in the pursuit of fixing it.

                            1. 1

                              I mean it more in the sense of “A good test suite tells you almost exactly where a failure is occuring, so you don’t need to step through a debugger to get the same information.”

                              Of course all bets are off for legacy codebases with little or no test coverage…

                              1. 2

                                I mean it more in the sense of “A good test suite tells you almost exactly where a failure is occuring, so you don’t need to step through a debugger to get the same information.”

                                This still sounds like “the bug is prevented before shipping” rather than fixed/debugged to me.

                                Maybe we are talking about different bugs. If the bug only exists before publishing/is detected during development I don’t really even think of those as “bugs”, just “I’m not done yet,” and I would agree a debugger is not going to be what I reach for first.

                                I’m having difficulty imagining a passing test suite that didn’t catch it before release “giving enough information” after the bug was detected through other means, but maybe that’s a failure of imagination on my part.

                                1. 1

                                  Thanks for your thoughts. It’s clear I need to ruminate on this some more.

                                  I still think a good test suite helps a lot in discovering new bugs (I’ve experienced this), but clearly my thoughts on the matter (and thus my verbalization) are not complete… Maybe I’ll write about this some day.

                                  Thanks again.

                        2. 2

                          It’s a tool, just like anything else, and they can be useful for when unit tests or attaching a debugger aren’t. I just recently used some debugging statements to isolate a bug in one of our services. Why did the unit tests miss it? Because for a variety of reasons, there are no unit tests (written in C++ years ago with no clearly defined “units” to test [1]). A well placed debug log let me see how a particular piece of data was being parsed let me easily see a few hundred broken examples.

                          It also doesn’t help that our ops guys may not fully understand syslog at all. Last year they freaked out when I left a syslog debug message in a service I wrote, and they ask me how to disable debug level messages. I ask them why they didn’t filter out debug level messages with the syslog configuration. They didn’t know what I was talking about. Sigh.

                          [1] There are numerous functions only called from one spot with a ton of assumptions about how it’s called and they’ll all be inlined anyway by the compiler, right? Legacy code man, legacy code.

                      1. 3

                        Log output should be included in your unit tests (it is one of the affects of your function, right?).

                        I disagree with this. That seems like it will get very close to testing the implementation quickly, and generally seems too far on the painful side of the testing spectrum.

                        (I’m sure there’s exceptions where it does make sense because you have some kind of contract with log consumers, e.g. if you have alerting tied to specific log messages, though in that particular case I’d probably prefer metrics.)

                        The one thing I’ve found useful is selected log analysis in integration tests. Collect service logs during integration tests, and have a post-test check that has some kind of modifiable whitelist/blacklist. I.e., by default an error level log causes a test failure, but then you might allow certain specific errors. It can also be good trade-off to wait for certain info log lines to appear when trying to set up the correct pre-conditions.

                        1. 1

                          This was probably poorly phrased. IMO, you should test that the expected logging occurs, when expected. Not the format of the logs (the logger implementations I use for testing inevitably log a very different format than the loggers used in production).

                          Do you still disagree with this approach?

                          1. 1

                            Generally still disagree, yes. Outside particular circumstances, I don’t see logging as relevant to the purpose of the code, hence wouldn’t want to tie that part of the behaviour down by tests. Just like I wouldn’t want to test any number of other tangential side-effects. Don’t have particularly good analogies, but say I know some function will open a certain number of sockets; that’s observable behaviour, but I don’t want to ensure it doesn’t change.

                          2. 1

                            I agree, log output should not be included in unit tests, although like you, I can see a tiny bit of value in this. At $JOB, all our logs have a unique id associated with them [1], and they look like “BAR0025: host=db.example.com failure='Network dropped connection because of reset'”. A unit test of logging messages could check that it has a unique tag (or it hasn’t changed).

                            [1] Why? To make it easier to search for particular log messages. They don’t have to be full UUIDs, just something that is unique enough for the organization.

                          1. 10

                            The solution is to use an object instance for your logger, the same as you should with a database, or other external dependencies, then use dependency injection to provide the logger to the code that generates logs.

                            I know that’s the proper way to do things, but I hate having a logger instance past around to every single script and object. That’s what I used to do and I went back to a global instance, which can be initialised differently for test units or the app. Much simpler and good enough for my use. It makes the code a bit less modular since you have to make sure that this global logger has been initialised somewhere, but so much better than this extra dependency everywhere.

                            In fact, I wonder if an object that is passed absolutely everywhere via dependency injection doesn’t in the end count as a global object anyway, and could be made one to simplify the code.

                            1. 7

                              I don’t think there can ever be a single “right” way to do this. The key is to know the caveats. If you’re willing to accept the extra cognitive load that comes with handling a global safely, as a trade-off against the extra overhead of passing objects around, I think that’s fine.

                              I prefer the more explicit approach (especially on large projects, with many developers–many of whom will NOT be careful about global state), but of course YMMV.

                              1. 2

                                The way I like to look at it is that logging is an added behavior that we can separate from business logic. With this in mind, we can decorate the pieces of biz logic with logging at the entry point of the application. This way we can properly test biz logic without knowing anything about logging, and test logging without knowing anything about the biz logic. We can even inject faulty implementations to test logging when something goes wrong. And we can also decorate the dependencies of our dependencies… so turtles all the way down. An approach like this will inevitably push your codebase to be extremely modular. Too modular, to the point it can make people uncomfortable. I see this modularity as good thing, but not a lot of people share this same point of view… In any case, this is how I usually approach it. So +1 for dependency injection :-)

                                Here’s a pseudo-code example:

                                // ---
                                
                                interface Emailer {
                                  Send(EmailMessage)
                                }
                                
                                // ---
                                
                                interface Logger {
                                  Trace(string)
                                }
                                
                                // ---
                                
                                class StdoutLogger implements Logger {
                                  public Trace(message: string) {
                                    print(os.Stdout, message)
                                  }
                                }
                                
                                // ---
                                
                                class SendGridEmailer implements Emailer {
                                  constructor(client: SendGridClient) {…}
                                
                                  public Send(e: EmailMessage) {
                                    client.SendEmail(client.NewEmail(…))
                                  }
                                }
                                
                                // ---
                                
                                class VerboseEmailer implements Emailer {
                                  constructor(logger: Logger, emailer: Emailer)
                                
                                  public Send(e: EmailMessage) {
                                    logger.Trace("sending email")
                                    emailer.Send(e)
                                    logger.Trace("email sent")
                                  }
                                }
                                
                                // ---
                                
                                main() {
                                  new VerboseEmailer(
                                    new StdoutLogger(),
                                    new SendGridEmailer(…)
                                  ).Send(new EmailMessage(…))
                                }
                                
                              2. 2

                                So don’t give it to everything. That means not everything can log. Which is great! Logging should be a considered and deliberate action, like any other important thing in your application.

                                1. 1

                                  Sometimes there are good reasons for having a static variable in a program. This is one of them.

                                  1. 1

                                    Eh, nah. Few things are true process singletons, and loggers ain’t one of them.

                                  2. 1

                                    In fact, I wonder if an object that is passed absolutely everywhere via dependency injection doesn’t in the end count as a global object anyway, and could be made one to simplify the code.

                                    Keeping dependencies explicit ensures that each component is testable in isolation from others, and establishes a clear rule — the possible effects of a component are constrained by its inputs, i.e. no side effects — which helps enormously in the understanding (and therefore maintenance) of programs.

                                  1. 21

                                    The article never mentions the, in my humble opinion, most important part of good logging practices and that is structured logging. Without it you end up with weird regexes or other hacks trying to parse your log messages.

                                    1. 4

                                      As a sibling post notes, if you use structured logging you’re mostly throwing away the idea that the entries must be easily parsable by a human. If that’s the case, and we’ll need a custom method of displaying the structured logs in a human friendly way, I believe we should forego plain text all together and gain the benefits of logging directly to binary.

                                      1. 5

                                        You can do human readable structured logging if you use key="value" formats inside text messages. Some people still prefer json, but there is a middle ground.

                                        1. 2

                                          If you need just key=value, that’s not really structured in my opinion.

                                          1. 4

                                            Why not?

                                            1. 2

                                              Because the amount of information added by this format would be infinitesimal over a line based logger with manual tokenization. The reason why you’d want a structured logger is to allow proper context to a message. Unless you’re working with simple cases, the structure that would offer such context is more than one level deep.

                                              1. 3

                                                Hmm, definitely not.

                                                Structured logging is about decorating log events with just enough of a schema to make them machine parseable, so that searching, aggregating, filtering, etc. can more than a crapshoot. Deeply nested events significantly increase the complexity of that schema, and therefore the requirements of the consumer.

                                                By default, structured logs should be flat key/value pairs. It gets you the benefits of richer parseability, without giving up the ability to grep.

                                      2. 2

                                        Excellent point. That’s become such second nature to me by now, that I forgot to even mention it!

                                        1. 2

                                          I’m surprised it wasn’t mentioned, but the larger advantage of passing a logger around to constructors is the ability to then have nested named loggers, such as

                                          Battery.ChargingStatus.FileReader: Failed to open file { file: "/tmp/battery charge", error: ... }
                                          Battery.ChargingStatus: Failed to access status logs, skipping report
                                          
                                          1. 1

                                            On top of that, structured logger if implemented properly, can often be faster and be operated at granular levels (like the other comments pointed out, sometimes you do want to on-fly turn on some logs at some locations, not all logs at all locations).

                                            1. 1

                                              I love structured logging, with one caveat: the raw messages emitted (let’s assume JSON) are harder for me to scan when tailing directly (which I usually only do locally as we have better log querying tools in the cloud), in contrast to a semi-structured simple key-value format. Do you all use a different format than JSON? Or a tool that transforms structured logs to something more friendly to humans, eg. with different log levels displayed in different appropriate colors, eg. JSON syntax characters diminished, for local tailing?

                                              1. 5

                                                At Joyent, we used the Bunyan format. Each line in the file was a separate JSON object with standard properties, some mandatory and some optional, and freeform additional properties. We shipped a tool, bunyan, that was capable of acting as a filter that would render different human readable views of the JSON. For example, you would often run something like:

                                                tail -F $(svcs -L manatee) | bunyan -o short
                                                

                                                It also had some rudimentary filtering options. It also had a relatively novel mode that would, instead of reading from a file or standard input, use DTrace probes for different log levels to allow you to dynamically listen for DEBUG and TRACE events even when those were not ordinarily present in the log files. The DTrace mode could target a particular process, or even all processes on the system that emitted Bunyan logs.

                                                1. 1

                                                  Hi, what were the required fields? Was it just a unique request ID? Thanks for sharing about bunyan. Even though it’s been out for a while I was unaware of it.

                                                2. 5

                                                  Do you all use a different format than JSON? Or a tool that transforms structured logs to something more friendly to humans, eg. with different log levels displayed in different appropriate colors, eg. JSON syntax characters diminished, for local tailing?

                                                  We use JSON and the only tools I use are grep and jq. And although I am pretty much still a novice with these two, I found that with the power of shell piping I can do almost anything I want. Sometimes I reach for the Kibana web interface, get seriously confused and then go back to the command line to figure out how to do it there.

                                                  I wrote a simple tutorial for the process, just a couple of weeks ago.

                                                  1. 1

                                                    If you rely on external tools to be able to make sense of your logs, why not go all the way, gain the speed and size benefits that binary logs would bring, and write your own log pager? I feel like the systemd folks had the right idea even when everyone was making fun of them.

                                                    1. 3

                                                      I don’t think the average employer would be happy subsidizing an employee writing a log pager instead of implementing something that would bring a tangible result to the business. The potential money savings by using binary logs probably doesn’t outweigh the new subs/increased profits of churning out more features.

                                                      1. 1

                                                        To me that sounds like an excuse. The world is not made up of only software that is beholden to the all mighty shareholder.

                                                        1. 1

                                                          I mean, yes, if you’re developing something in your personal time, go bananas on what you implement.

                                                          But I also know my manager would look at me funny and ask why I’m not just shoving everything into CloudWatch/<cloud logging service>

                                                      2. 2

                                                        I’m sure most problems with systemd journals are fixable, but they’ve left a very bad taste in my mouth for two main reasons: if stuff gets deleted from under them they apparently never recover (my services continue to say something like “journal was rotated” until I restart them), and inspecting journals is incredibly slow. I’m talking magnitudes slower than log files. This is at its worst (I often have time to make a cup of tea) when piping the output into grep or, as journalctl already does by default, less, which means every byte has to be formatted by journalctl and copied only to be skipped over by its recipient. But it’s still pretty bad (I have time to complain on IRC about the wait) when giving journalctl filters that reduce the final output down to a few thousand lines, which makes me suspect that there are other less fundamental issues.

                                                        I should note that I’m using spinning disks and the logs I’m talking about are tens to hundreds of GB over a few months. I feel like that situation’s not abnormal.

                                                        1. 1

                                                          If you rely on external tools to be able to make sense of your logs, why not go all the way, gain the speed and size benefits that binary logs would bring, and write your own log pager?

                                                          It’s hard to imagine a case at work where I could justify writing my own log pager.
                                                          Here are some of the reasons I would avoid doing so:

                                                          • Logs are an incidental detail to the application.
                                                          • Logs are well understood; I can apply a logging library without issues.
                                                          • My application isn’t a beautiful and unique snowflake. I should use the same logging mechanisms and libraries as our other applications unless I can justify doing something different.
                                                          • JSON is boring, has a specification, substantial library support, tooling, etc.
                                                          • Specifying, documenting, and testing a custom format is a lot of work.
                                                          • Engineering time is limited; I try to focus my efforts on tasks that only I can complete.
                                                          1. 2

                                                            Logs are an incidental detail to the application.

                                                            I think this is trivially disproved by observing that if the logs stop working for your service, that is (hopefully!) a page-able event.

                                                            Logs are a cross-cutting concern, but as essential as any other piece of operational telemetry.

                                                            1. 1

                                                              Logs are a cross-cutting concern, but as essential as any other piece of operational telemetry.

                                                              I rely heavily on logging for the services I support but the applications I wrote for work have only error reporting. They are used by a small audience and problems are rare; I might get a crash report every 18 months or so.

                                                              1. 1

                                                                Ah, yeah, I presume the context here is services.

                                                        2. 1

                                                          Agreed. jq is a really nice tool. It made the decision to transition to using JSON for logging very easy.

                                                        3. 3

                                                          Don’t use JSON, use logfmt.

                                                          1. 1

                                                            Yes! Logfmt is the good stuff. But it’s only semi-structured. Why not use JSON and a tool to transform to logfmt (with nested data elided probably) when needing to scan as a human?

                                                            1. 1

                                                              Logfmt is fully structured, it just doesn’t support nesting, which is an important feature! Structured logs should be flat.

                                                      1. 26

                                                        Pro tip: this applies to you if you’re a business too. Kubernetes is a problem as much as it is a solution.

                                                        Uptime is achieved by having more understanding and control over the deployment environment but kubernetes takes that away. It attracts middle managers and CTOs because it seems like a silver bullet without getting your hands dirty but in reality it introduces so much chaos and indirections into your stack that you end up worse off than before, and all the while you’re emptying your pockets for this experience.

                                                        Just run your shit on a computer like normal, it’ll work fine.

                                                        1. 9

                                                          This is true, but let’s not forget that Kubernetes also has some benefits.

                                                          Self-healing. That’s what I miss the most with a pure NixOS deployment. If the VM goes down, it requires manual intervention to be restored. I haven’t seen good solutions proposed for that yet. Maybe uptimerobot triggering the CI when the host goes down is enough. Then the CI can run terraform apply or some other provisioning script.

                                                          Zero-downtime deployment. This is not super necessary for personal infrastructures but is quite important for production environments.

                                                          Per pod IP. It’s quite nice not to have to worry about port clashes between services. I think this can be solved by using IPv6 as each host automatically gets a range of IPs to play with.

                                                          Auto-scaling. Again not super necessary for personal infrastructure but it’s nice to be able to scale beyond one host, and not to have to worry on which host one service lives.

                                                          1. 6

                                                            Did anyone tried using Nomad for personal projects? It has self-healing and with the raw runner one can run executables directly on NixOS without needing any containers. I have not tried it myself (yet), but would be keen on hearing the experiences.

                                                            1. 3

                                                              I am experimenting with the Hashiscorp stack while off for the holidays. I just brought up a vagrant box (1GB ram) with Consul, Docker and Nomad runing (no jobs yet) and the overhead looks okay:

                                                                            total        used        free      shared  buff/cache   available
                                                              Mem:          981Mi       225Mi       132Mi       0.0Ki       622Mi       604Mi
                                                              Swap:         1.9Gi       7.0Mi       1.9Gi
                                                              

                                                              but probably too high to fit Postgres, Traefik or Fabio and a Rails app into it as well, but 2GB will probably be lots (I am kind of cheap so the less resources the better).

                                                              I have a side project running in ‘prod’ using Docker (for Postgres and my Rails app) along with Caddy running as a systemd service but it’s kind of a one off machine so I’d like to move towards something like Terraform (next up on the list to get running) for bring up and Nomad for the reasons you want something like that.

                                                              But… the question that does keep running through the back of my head, do I need even Nomad/Docker? For a prod env? Yes, it’s probably worth the extra complexity and overhead but for personal stuff? Probably not… Netlify, Heroku, etc are pretty easy and offer free tiers.

                                                              1. 1

                                                                I was thinking about doing this but I haven’t done due diligence on it yet. Mostly because I only have 2 droplets right now and nobody depends on what’s running on them.

                                                              2. 1

                                                                If you’re willing to go the Amazon route, EC2 has offered most of that for years. Rather than using the container as an abstraction, treat the VM as a container: run one main process per VM. And you then get autoscaling, zero downtime deploys, self-healing, and per-VM IPs.

                                                                TBH I think K8s is a step backwards for most orgs compared to just using cloud VMs, assuming you’re also running K8s in a cloud environment.

                                                                1. 2

                                                                  That’s a good point. And if you don’t care about uptime too much, autoscaling + spot instances is a pretty good fit.

                                                                  The main downside is that a load-balancer is already ~15.-/month if I remember correctly. And the cost can explode quite quickly on AWS. It takes quite a bit of planning and effort to keep the cost super low.

                                                              3. 5

                                                                IMO, Kubernetes’ main advantage isn’t in that it “manages services”. From that POV, everything you say is 100% spot-on. It simply moves complexity around, rather than reducing it.

                                                                The reason I like Kubernetes is something entirely different: It more or less forces a new, more robust application design.

                                                                Of course, many people try to shoe-horn their legacy applications into Kubernetes (the author running git in K8s appears to be one example), and this just adds more pain.

                                                                Use K8s for the right reasons, and for the right applications, and I think it’s appropriate. It gets a lot of negative press for people who try to use it for “everything”, and wonder why it’s not the panacea they were expecting.

                                                                1. 5

                                                                  I disagree that k8s forces more robust application design; fewer moving parts are usually a strong indicator of reliability.

                                                                  Additionally, I think k8s removes some of the pain of microservices–in the same way that a local anathestic makes it easier to keep your hand in boiling water–that would normally help people reconsider their use.

                                                                2. 5

                                                                  And overhead. Those monster yaml files are absurd in so many levels.

                                                                  1. 2

                                                                    Just run your shit on a computer like normal, it’ll work fine.

                                                                    I think that’s an over-simplification. @zimbatm’s comment makes good points about self-healing and zero-downtime deployment. True, Kubernetes isn’t necessary for those things; an EC2 auto-scaling group would be another option. But one does need something more than just running a service on a single, fixed computer.

                                                                    1. 3

                                                                      But one does need something more than just running a service on a single, fixed computer.

                                                                      I respectfully disagree…worked at a place which made millions over a few years with a single comically overloaded DO droplet.

                                                                      We eventually made it a little happier by moving to hosted services for Mongo and giving it a slightly beefier machine, but otherwise it was fine.

                                                                      The single machine design made things a lot easier to reason about, fix, and made CI/CD simpler to implement as well.

                                                                      Servers with the right provider can stay up pretty well.

                                                                      1. 2

                                                                        I don’t see how your situation/solution negates the statement.

                                                                        You’ve simply traded one “something” (Kubernetes) with another (“the right provider”, and all that entails–probably redundant power supplies, network connections, hot-swappable hard drives, etc, etc).

                                                                        The complexity still exists, just at a different layer of abstraction. I’ll grant you that it does make reasoning about the application simpler, but it makes reasoning about the hardware platform, and peripheral concerns, much more complex. Of course that can be appropriate, but it isn’t always.

                                                                        I’m also unsure how a company’s profit margin figures into a discussion about service architectures…

                                                                        1. 5

                                                                          I’m also unsure how a company’s profit margin figures into a discussion about service architectures…

                                                                          There is no engineering without dollar signs in the equation. The only reason we’re being paid to play with shiny computers is to deliver business value–and while I’m sure a lot of “engineers” are happy to ignore the profit-motive of their host, it is very unwise to do so.

                                                                          I’ll grant you that it does make reasoning about the application simpler, but it makes reasoning about the hardware platform, and peripheral concerns, much more complex.

                                                                          That engineering still has to be done, if you’re going to do it at all. If you decide to reason about it, do you want to be able to shell into a box and lay hands on it immediately, or hope that your k8s setup hasn’t lost its damn mind in addition to whatever could be wrong with the app?

                                                                          You’ve simply traded one “something” (Kubernetes) with another (“the right provider”, and all that entails–probably redundant power supplies, network connections, hot-swappable hard drives, etc, etc).

                                                                          The complexity of picking which hosting provider you want to use (ignoring colocation issues) is orders and order of magnitudes less than learning and handling k8s. Hosting is basically a commodity at this point, and barring the occasional amazingly stupid thing among the common names there’s a baseline of competency you can count on.

                                                                          People have been sold this idea that hosting a simple server means racking it and all the craziness of datacenters and whatnot, and it’s just a ten spot and an ssh key and you’re like 50% of the way there. It isn’t rocket surgery.

                                                                        2. 2

                                                                          Servers with the right provider can stay up pretty well.

                                                                          I was one of the victims of the DDOS that hit Linode on Christmas day (edit: in 2015; didn’t mean to omit that). DO and Vultr haven’t had perfect uptime either. So I’d rather not rely on single, static server deployments any more than I have to.

                                                                          1. 1

                                                                            can you share more details about this?

                                                                            I’ve always been impressed by teams/companies maintaining a very small fleet of servers but I’ve never heard of any successful company running a single VM.

                                                                            1. 4

                                                                              It was a boring little Ubuntu server if I recall correctly, I think like a 40USD general purpose instance. The second team had hacked together an impressive if somewhat janky system using the BEAM ecosystem, the first team had built the original platform in Meteor, both ran on the same box along with Mongo and supporting software. The system held under load (mostly, more about that in a second), and worked fine for its role in e-commerce stuff. S3 was used (as one does), and eventually as I said we moved to hosted options for database stuff…things that are worth paying for. Cloudflare for static assets, eventually.

                                                                              What was the business environment?

                                                                              Second CTO and fourth engineering team (when I was hired) had the mandate to ship some features and put out a bunch of fires. Third CTO and fifth engineering team (who were an amazing bunch and we’re still tight) shifted more to features and cleaning up technical debt. CEO (who grudgingly has my respect after other stupid things I’ve seen in other orgs) was very stingy about money, but also paid well. We were smart and well-compensated (well, basically) developers told to make do with little operational budget, and while the poor little server was pegged in the red for most of its brutish life, it wasn’t drowned in bullshit. CEO kept us super lean and focused on making the money funnel happy, and didn’t give a shit about technical features unless there was a dollar amount attached. This initially was vexing, but after a while the wisdom of the approach became apparent: we weathered changes in market conditions better without a bunch of outstanding bills, we had more independence from investors (for better or worse), and honestly the work was just a hell of a lot more interesting due in no small part to the limitations we worked under. This is key.

                                                                              What problems did we have?

                                                                              Support could be annoying, and I learned a lot about monitoring on that job during a week where the third CTO showed me how to setup Datadog and similar tooling to help figure out why we had intermittent outages–eventual solution was a cronjob to kill off a bloated process before it became too poorly behaved and brought down the box. The thing is, though, we had a good enough customer success team that I don’t think we even lost that much revenue, possibly none. That week did literally have a day or two of us watching graphs and manually kicking over stuff just in time, which was a bit stressful, but I’d take a month of that over sitting in meetings and fighting matrix management to get something deployed with Jenkins onto a half-baked k8s platform and fighting with Prometheus and Grafana and all that other bullshit…as a purely random example, of course. >:|

                                                                              The sore spots we had were basically just solved by moving particular resource-hungry things (database mainly) to hosting–the real value of which was having nice tooling around backups and monitoring, and which moving to k8s or similar wouldn’t have helped with. And again, it was only after a few years of profitable growth that it traffic hit a point where that migration even seemed reasonable.

                                                                              I think we eventually moved off of the droplet and onto an Amazon EC2 instance to make storage tweaks easier, but we weren’t using them in any way different than we’d use any other barebones hosting provider.

                                                                              1. 4

                                                                                Did that one instance ever go completely down (becoming unreachable due to a networking issue also counts), either due to an unforeseen problem or scheduled maintenance by the hosting provider? If so, did the company have a procedure for bringing a replacement online in a timely fashion? If not, then I’d say you all just got very lucky.

                                                                                1. 1

                                                                                  Yes, and yes–the restart procedure became a lot simpler once we’d switched over to EC2 and had a hot spare available…but again, nothing terribly complicated and we had runbooks for everything because of the team dynamics (notice the five generations of engineering teams over the course of about as many years?). As a bonus, in the final generation I was around for we were able to hire a bunch of juniors and actually teach them enough to level them up.

                                                                                  About this “got very lucky” part…

                                                                                  I’ve worked on systems that had to have all of the 9s (healthcare). I’ve worked on systems, like this, that frankly had a pretty normal (9-5, M-F) operating window. Most developers I know are a little too precious about downtime–nobody’s gonna die if they can’t get to their stupid online app, most customers–if you’re delivering value at a price point they need and you aren’t specifically competing on reliability–will put up with inconvenience if your customer success people treat them well.

                                                                                  Everybody is scared that their stupid Uber-for-birdwatching or whatever app might be down for a whole hour once a month. Who the fuck cares? Most of these apps aren’t even monetizing their users properly (notice I didn’t say customers), so the odd duck that gets left in the lurch gets a hug and a coupon and you know what–the world keeps turning!

                                                                                  Ours is meant to be a boring profession with simple tools and innovation tokens spent wisely on real business problems–and if there aren’t real business problems, they should be spent making developers’ lives easier and lowering business costs. I have yet to see k8s deliver on any of this for systems that don’t require lots of servers.

                                                                                  (Oh, and speaking of…is it cheaper to fuck around with k8s and all of that, or just to pay Heroku to do it all for you? People are positively baffling in what they decide to spend money on.)

                                                                                2. 1

                                                                                  eventual solution was a cronjob to kill off a bloated process before it became too poorly behaved and brought down the box … That week did literally have a day or two of us watching graphs and manually kicking over stuff just in time, which was a bit stressful,…

                                                                                  It sounds like you were acting like human OOM killers, or more generally speaking manual resource limiters of those badly-behaved processes. Would it be fair to say that sort of thing would be done today by systemd through its cgroups resource management functionality?

                                                                                  1. 1

                                                                                    We probably could’ve solved it through systemd with Limit* settings–we had that available at the time. For us, we had some other things (features on fire, some other stuff) that took priority, so just leaving a dashboard open and checking it every hour or two wasn’t too bad until somebody had the spare cycles to do the full fix.

                                                                        1. 4

                                                                          I went for yet another option and decided to colo my older intel i3 nuc with 16gb of memory and 128gb + 1tb of storage. I pay 70euro per 6months and get 2 ipv4 addresses and the standard /64 for ipv6. Transfer volume included is so high I don’t think I even managed to get to 5%.

                                                                          This was to be honest the cheapest solution I could find, and I can always request my hardware back, I think the fee for that is 20euro. I can also pay a one time fee of 25euro to get an access card to go visit my nuc myself and swap hardware/reboot it. The cost of the server is highly tied to the amount electricity my hardware is using, so I could lower the cost/gain more by upgrading the hardware to something more powerful that’s using the same amount of energy.

                                                                          You’d be surprised to see how many companies let you colo small devices like nuc’s or even a raspberry pi for very low cost. It’s a route not many people take, but to be honest it works great for me and might be worth to have a look.

                                                                          1. 1

                                                                            How did you find out about the collocation?

                                                                            1. 1

                                                                              I was talking to colleagues about 1.5year ago and complained about the lack of good quality vps providers for a low price. One of them said something like “it’s a shame you can’t put your nuc in a datacenter” and I remembered some people exactly do that. I looked for colo providers; mailed 2 that offered NAS hosting because it matched most, and both replied they had no issue with putting a nuc in and gave me a price.

                                                                              I could drop the nuc off personally, or send it to them via registered & insured mail. I choose the latter, and 2 days later it was installed and up and running. Went extremely well, and I had a 3minute downtime during the 1.5 years due to a planned maintenance when they upgraded the DDoS protection service.

                                                                            2. 1

                                                                              What’s the volume and where do you find them?

                                                                              1. 1

                                                                                I think transfer volume included is something like 5tb a month. Like I said that’s more than enough for me.

                                                                                I just did a search for colo providers and picked a couple that also offered NAS hosting. Since my nuc and a synology NAS are about the same volume (physically) I mailed them and both said it’s no problem and gave me the price. I picked the more expensive one actually, because they offered extra’s as the option to go and look at my hardware myself.

                                                                              2. 1

                                                                                How do you handle data backups?

                                                                                1. 1

                                                                                  I cross-transfer between my home nuc and my colo nuc via syncthing. It’s probably not the best or most glorious solution, but it works like a charm for me. I don’t generate a lot of data, to be honest, so I wasn’t looking for an advanced solution.

                                                                              1. 9

                                                                                Forgive my ignorance, but it seems like just a boring droplet with some nginx would solve most of the problems here. What’s the driver for all that power?

                                                                                EDIT:

                                                                                Like, I spend 20USD/month at prgmr and have been quite happy with it for Gittea plus all kinds of other weird things.

                                                                                1. 11

                                                                                  I needed to learn Kubernetes for work and I decided to do so by moving a bunch of my own infrastructure to it. That’s the whole reason really. It’s really expensive in retrospect and I’m really looking at going back to a more simple setup as a result.

                                                                                  1. 2

                                                                                    I remember when you started learning and I asked why not just Ansible?

                                                                                    Now I see you regretting your choice ~1.5 years later. So I ask again: why not just Ansible? :D

                                                                                    I’ve been using Ansible to manage my personal infra for the same amount of time (I started learning Ansible when you started K8s) and love it.

                                                                                    1. 5

                                                                                      TL;DR: Ansible describes how to get to your desired system state. NixOS describes your desired system state.

                                                                                      1. 1

                                                                                        AFAIK Ansible also described the desired system state. E.g.

                                                                                        service:
                                                                                          state: started
                                                                                        

                                                                                        Something like that.

                                                                                        1. 1

                                                                                          That describes the fact that you want the service started, not the fact that the service has a bunch of configuration for it. See here for an example: https://github.com/Xe/nixos-configs/blob/master/common/services/xesite.nix

                                                                                      2. 1

                                                                                        My personal answer to “why not ansible” would have been: it is slow, very tedious in some aspects (running locally, obeying your ~/.ssh/config or .netrc) and you need to write yaml. Personally I have moved to pyinfra which fixes all of these and more.

                                                                                      3. 2

                                                                                        I learned Kubernetes the same way. Although I hosted it on Google. I’m still paying for that cluster, far more than it’s worth, simply because I can’t be bothered to migrate to something else (a $5/mo droplet or equivalent would do the trick).

                                                                                        That said, using a hosted K8s solution I think makes a lot more sense for a small project–although it also potentially increases the cost significantly for a hobby project.

                                                                                        I guess IMO, K8s probably isn’t the right tool for most hobby projects (outside of learning K8s). For small teams, hosted K8s is often a good choice.

                                                                                        1. 1

                                                                                          That totally makes sense! I just didn’t know if there was some other technological forcing function, given the relative expense of some of the options in your table. Also, what’s the hacks column about?

                                                                                          1. 2

                                                                                            Hacks needed to install NixOS.

                                                                                            1. 1

                                                                                              Ah cool.

                                                                                              Prgmr actually has a NixOS image you can start with, though I think it’s a couple versions back.

                                                                                        2. 3

                                                                                          I just wanted to mention https://www.hetzner.com/cloud which is also quite cheap, and can boot on a NixOS ISO as well. <3 Hetzner

                                                                                          1. 1

                                                                                            Is there a way to install NixOS on a Hetzner cloud box using the ISO fully automated with Terraform or Ansible? Everything I’ve read about it involves manual steps :/

                                                                                            1. 1

                                                                                              Not with the ISO but the nixos-infect approach works quite nicely. Here is an example from my own Terraform:

                                                                                              resource "hcloud_server" "mon2" {
                                                                                                image       = "debian-10"
                                                                                                keep_disk   = true
                                                                                                name        = "mon2"
                                                                                                server_type = "cx21"
                                                                                                ssh_keys    = local.hcloud_keys
                                                                                                backups     = false
                                                                                              
                                                                                                user_data = <<EOF
                                                                                                #cloud-config
                                                                                                runcmd:
                                                                                                  - curl https://raw.githubusercontent.com/zimbatm/nixos-infect/3e9d452fa6060552a458879b66d8eea1334d93d2/nixos-infect | NIX_CHANNEL=nixos-20.09 bash 2>&1 | tee /tmp/infect.log
                                                                                                EOF
                                                                                              }
                                                                                              
                                                                                              1. 1

                                                                                                Thanks!!!

                                                                                                Do you then do provisioning with Ansible, or is the next step Terraform, too?

                                                                                                1. 1

                                                                                                  Still Terraform, using https://github.com/tweag/terraform-nixos/tree/master/deploy_nixos that I wrote a while ago.

                                                                                                  I generally try to keep everything in Terraform as much as possible.

                                                                                                  1. 1

                                                                                                    thanks, that looks great!

                                                                                        1. 3

                                                                                          “YAML is a HORRIBLE format for configuration”

                                                                                          I completely agree! Although I’d go one step further and say simply “YAML is a HORRIBLE format”.

                                                                                          1. 1

                                                                                            A colleague of mine made a good point about YAML: the many projects that abuse it probably should have never used it. On the other hand, it’s way better for static data like test fixtures. But in thase cases, you probably aren’t using nearly as much of the YAML spec as you would be for configuration.

                                                                                          1. 4

                                                                                            I read this before seeing it here, and came to the conclusion: It’s all irrelevant.

                                                                                            In SVN commits are diffs. From the end user’s standpoint, the difference is semantics. Snapshots vs diffs is technical esoterica that makes no actual difference to daily usage.

                                                                                            Now having said that, if thinking of commits (in either git or SVN) as snapshots instead of diffs makes it easier for someone to reason about, there’s no harm there. But if thinking about commits as colored dots on a piece of (digital) graph paper helps, do that, too. Use whatever model helps.

                                                                                            1. 2

                                                                                              Well, storing patches has some advantages over storing snapshots, Pijul explain it pretty well in article linked above. On the other hand some other actions are easier to do when working with snapshots. It is not just “technical esoterica that makes no difference to daily usage”.

                                                                                              1. 3

                                                                                                Those problems, as real as they are, are not ones that I’ve hit in more than a decade of heavy, daily git usage (and I used darcs before that)

                                                                                                1. 1

                                                                                                  “storing patches has some advantages over storing snapshots” – Yes, of course. These advantages just make no difference to the overwhelmingly vast majority of people using git.

                                                                                                  To your every day git user, it makes no difference whether git stores snapshots, diffs, or ice cream sandwiches. As long as it faithfully reproduces the code you store in it.

                                                                                              1. 2

                                                                                                I was half expecting this link to lead to a deleted Tweet…

                                                                                                1. 2

                                                                                                  Well, the first link in the post actually seems to do just that.

                                                                                                1. 2

                                                                                                  This looks interesting, but I don’t have 90 minutes right now. I got to the part where he talks about programs being dynamic + temporal rather than static + spatial – could someone summarize the argument against ‘if’?

                                                                                                  1. 4

                                                                                                    He mainly argues against ‘if’ statements that have been put in place to guard against a value that was entered into a function which caused the bug. If a function higher up in the call stack also has a condition that does a similar check, these two ‘if’ statements are now connected. This is a ‘wormhole if’.

                                                                                                    This means that if you are changing code, you might have an undesired effect somewhere else if you do not know that there is an ‘if’ somewhere else that checks similar conditions.

                                                                                                    1. 2

                                                                                                      IMO, the title is intentionally click-baity.

                                                                                                      My take is that he’s mainly arguing against a specific case of if: The repeated if.

                                                                                                      When you’re constantly checking for the same condition in different places, you’re asking for bugs. I think a common example (although not the only one by any means!) is feature flags.

                                                                                                      Rather than if featureX == true 50 times in your code, find some way to control that condition in fewer places (ideally exactly one). A constructor function can be a good place, depending on your language/architecture.

                                                                                                      Large “if trees” are also often cases where conditions are repeated (and what I think @kiwec is responding to in his comments), and refactoring those (i.e. with preconditions and early returns) can also reduce bugs.

                                                                                                    1. 8

                                                                                                      Still too complicated for me.

                                                                                                      Nine times out of ten, I define an interface for the external dependency (cache in this article) and then write two implementations: an in-memory version and the “real” version. The in-memory is as simple as possible, perhaps added with additional methods to retrieve stored data, or ways to trigger an error response.

                                                                                                      This implementation is almost always shorter, easier to use and occasionally can be used as actual functionality. As a bonus, it requires you to think more about that interface and whether you can simplify that, or break it up into multiple interfaces, making your code more idiomatic Go.

                                                                                                      The only downside is that it makes you allergic to generated mocks and you constantly get the urge to rewrite the existing tests in the code base.

                                                                                                      1. 4

                                                                                                        define an interface for the external dependency . . . and then write two implementations: an in-memory version and the “real” version.

                                                                                                        I do exactly this, and highly recommend others do the same. Mock generation is a (minor) code smell.

                                                                                                        Given an interface X, a useful default mock implementation is a struct MockX with an MFunc field for each interface method M. Let each test, or group of tests, create an instance of the mock, and set the fields to whatever behavior they need.

                                                                                                        1. 2

                                                                                                          Nine times out of ten, I define an interface for the external dependency (cache in this article) and then write two implementations: an in-memory version and the “real” version. The in-memory is as simple as possible, perhaps added with additional methods to retrieve stored data, or ways to trigger an error response.

                                                                                                          I found myself doing this recently, for a small personal project. Though I can’t imagine this practice applied on a company-wide level. It feels tedious to maintain multiple copies of code for each data source (since Go lacks generics). Even more frequently, I find myself relying on full on integration tests – either against a local datastore or AWS APIs, to have confidence that a change didn’t break things in subtle ways.

                                                                                                          For some shameless self promotion, I wrote up my experience comparing gomock and testify, as well as a “tip” on using gomock.

                                                                                                          https://www.jcheng.org/post/mocking-in-go-part-1/ https://www.jcheng.org/post/mocking-in-go-part-2/ https://www.jcheng.org/post/gomock/ (The Do function)

                                                                                                          1. 1

                                                                                                            I’ve done that, too. More times than I can count.

                                                                                                            But more than once, about the time I find myself adding a third or fourth special-case for triggering an error case, or some other corner-case, I wish I had a generic stub like the one described in the article.

                                                                                                            1. 2

                                                                                                              Yes, I can understand that. It is a nice middle ground between the simple method and generated bloat. I’ll keep it in mind for that one in ten case where a more complex solution is required. Thanks for writing it down!

                                                                                                            2. 1

                                                                                                              write two implementations: an in-memory version and the “real” version.

                                                                                                              I always do this, and very often wind up using it as an actual storage backend. The in-memory datastore I use to make it so that I can have a multiplayer server that I compile to a single static .exe that I can give to gamedevs. The in-memory representation also comes up in prod; in a few places, I use it as an in-process cache for some write-once data. And of course, the in-memory backend powers our tests. We get a lot of utility out of this approach.

                                                                                                            1. 2

                                                                                                              In my experience, at least for my tech stack of choice (Go primarily), the main tool I need for effective offline development is the habit of using TDD. When I follow TDD practices, I find that my code works pretty well in isolation, and can be tested and developed easily without access to any network.

                                                                                                              A couple of years ago I let myself get lazy, and built an OAuth flow without proper TDD. Then I had a trans-atlantic flight (back when those were still possible, aaah the good ol’ days….).

                                                                                                              I spent the first half of the flight writing an OAuth implementation that would work without a server, so that I could continue development.

                                                                                                              Obviously this isn’t a complete answer. It’s just the one thing I found that I kicked myself over when it bit me.

                                                                                                              1. 2

                                                                                                                Seems cool… but I might be more tempted to buy a personal computer, if it came pre-assembled…

                                                                                                                1. 2

                                                                                                                  Do you not have code review?

                                                                                                                  1. 8

                                                                                                                    Sure. But I believe that tools should do as much code review as possible. If a tool can tell me that I forgot to remove some code immediately, that’s better than waiting a few minutes/hours for a human to extend effort to find the same problem.

                                                                                                                    I like to optimize for shorter feedback loops, and less human toil.

                                                                                                                    1. 2

                                                                                                                      The tooling surely helps for the most common things (like console.log which is really detectable). Here are some cases that slipped through to review lately:

                                                                                                                      • copied a thing (1) to make a more general thing (2), made all callers use (2), forgot to delete (1)
                                                                                                                      • noticed a wrongly spelled safeword
                                                                                                                      • just using the wrong interfaces (hard to describe in a generic way, “used abstractions of the wrong level”?)
                                                                                                                      • more soft but still annoying: “linking” to the wrong issue ID with a magic issue tracker term

                                                                                                                      So, I agree with your sentiment and the tooling looks great if adapted to your team’s standards, but an almost magic bullet could be: read your own code review before publishing it. It’s so annoying to read code outside your editor (mostly Gitlab’s GUI for my part), so it kind of hampers you to read code like you’re used to, in a good way.

                                                                                                                      NB “you” are hopefully the reader of this comment, I’m sure @jhall already does this.

                                                                                                                      1. 1

                                                                                                                        (Not to mention the countless times when console.log("debug") has been missed in code review, and made it into production anyway…)

                                                                                                                        1. 1

                                                                                                                          If even an explicit console.log() isn’t caught, are you sure you have - effective - code review?

                                                                                                                          (Saving the reviewer time by having the machine do it makes sense, of course!)

                                                                                                                          1. 4

                                                                                                                            It doesn’t matter how effective a code review is. Its still being done by a human, who are fallible. Even a NASA style code review has been known to let bugs through to “production”.

                                                                                                                      2. 4

                                                                                                                        Code review? Can’t you just never write bugs?

                                                                                                                        1. 2

                                                                                                                          If you learn how to do that, I’ll buy the book! :D

                                                                                                                      1. 2

                                                                                                                        I usually find most of these already in the lint stage. Except instead of magic strings we just use TODO as most tools already look for this and prevent merge. Likewise, no-console or similar are mostly automatically there when you start new projects and shouldn’t be an issue.

                                                                                                                        Comments in the build file are somewhat problematic. Sometimes I want a comment there, so how you distinguishthe needed ones vs the forgotten ones?

                                                                                                                        Code Review is important for this stuff.

                                                                                                                        1. 2

                                                                                                                          Indeed. The usefulness of this technique I think depends an awful lot on which language/tooling you’re using. Some languages have amazingly powerful linters that can catch most or all of these types of things. Others, not so much.

                                                                                                                          1. 2

                                                                                                                            One quite workable approach is to have a linter that uncomments one comment block at a time, and then tries to parse the AST. If it parses, it’s likely code.

                                                                                                                            1. 1

                                                                                                                              I’d suggest a slight modification: instead of looking for all comments except those starting with #-#, only look for comments that start with # -, since those are more likely to be commented-out commands. I guess you’d still get false positive from lists, though.

                                                                                                                            1. 11

                                                                                                                              “Proper” use of git add and git commit also helps here: instead of blindly adding all changes you can add them either individually by file or even by patch (-p).

                                                                                                                              1. 3

                                                                                                                                This is how I avoid it as well. git add -p and then (s)plitting the chunks when possible so I know exactly what I’m staging, then a final git diff --staged to validate before committing.

                                                                                                                                1. 2

                                                                                                                                  fun little alias of mine: git staged

                                                                                                                                2. 2

                                                                                                                                  Yes, that definitely helps. Although there are times when I want to commit something, that shouldn’t get into mainline. Debugging of CI scripts is perhaps when I use this feature the most.