1. 4

    The bit about how it’s hard to tell what will close a Reader/WriterCloser underneath you is super valid. I’m not sure how you’d manage that without some kind of “reference count” or similar (i.e. only actually close until the count hits zero).

    Another awkward use case is how to write http middleware that hash-sums a post body (i.e. for slack webhook validation) and then also lets inner actions read from the request post body.

    1. 6

      It’s simple. If you pass a *Closer to something, you should assume that something is going to close it. Otherwise you’d just pass it a Reader or a Writer.

      1. 2

        Not everyone gets the memo that this is the way it’s supposed to work. Very often, folks will create large interfaces and pass them around, with no intent on using everything defined.

        1. 2

          Sure, but at a certain point, what can you do about people ignoring the docs and even the sort of high level guidance of the language? I mean, deep reading docs is hard, reading the go proverbs isn’t https://go-proverbs.github.io/ – and you only have to get to the 4th one to get to “The bigger the interface, the weaker the abstraction.”

          1. 5

            “The bigger the interface, the weaker the abstraction.” says very little to someone not engaged in deeply understanding it.

            Obviously, we can’t throw our hands up and say, “what can you do? People will be people!!!” What we can do is ensure that code we write and review takes these principles to heart, and shows the value of it.

            1. 2

              Absolutely, and after doing all that – people are still going to write terrible code that directly goes against all the norms and recommendations. I am all for doing our level best to try to guide people – but leading a horse to water and all that.

        2. 1

          I think this is true, but it basically means you should never pass a *Closer unless you really really have to. The callee should manage the IO lifecycle.

          I would even go so far as to say one of the heavily opinionated Go linters should warn (maybe they do, I have never checked because I don’t think highly opinionated linters are a good idea for anything but beginners).

          1. 1

            This makes sense, but two difficulties.

            1. Still requires careful analysis of docs. It’s very easy to pass a closereader off to a function taking a reader.

            2. You can’t just pass something like a gzip.reader to another layer. Even if that layer closes, it doesn’t close the bottom.

            1. 1

              When I read stuff like this I change my mind about go being a language that can be learned in a weekend.

              1. 1

                You can certainly pick it up and use it effectively in a weekend, but surely you couldn’t learn the ins and outs of anything substantial in just a weekend.

            2. 4

              Between io.TeeReader and io.Pipe I think you can probably wire something up. There’s a decent amount of “plumbing” included, although it took me a few passes through the docs to find it all.

              1. 4

                Yeah, its quite worth it to read through the whole std library docs, I seem to find a new thing each time I skim it again.

              2. 1

                how to write http middleware that hash-sums a post body (i.e. for slack webhook validation) and then also lets inner actions read from the request post body.

                I’ve had to do something like that and I ended up writing a small TeeReadCloser struct that wraps TeeReader but also has a Close method that closes both the reader and the writer. You can probably get by with a version that takes a WriteCloser like mine and one that just takes a Writer and combine them as needed, though I wonder why they couldn’t just put these in the standard library.

              1. 20

                I do agree with the theme of this post: at scale software is complex. Whether you use a monorepo or polyrepos, you’ll need a lot of complicated tooling to make things manageable for your developers.

                But I want to draw attention to sfink’s excellent rebuttal (full disclosure, he is a colleague of mine).

                Additionally, I’d like to address the VCS Scalablilty downside. The author’s monorepo experience seems to be with Git. Companies like Google, Facebook (who have two of the largest monorepos in the world) and to a lesser extent Mozilla all use Mercurial for a reason: it scales much better. While I’m not suggesting the path to get there was easy, the work is largely finished and contributed back upstream. So when the author points to Twitter’s perf issues or Microsoft’s need for a VFS, I think it is more a problem related to using the wrong tool for the job than it is something inherently wrong with monorepos.

                1. 5

                  I was under the impression (possibly mistaken) that Google still used perforce predominantly (or some piper wrapper thing), with a few teams using mercurial or git for various externally visible codebases (android, chrome, etc).

                  1. 10

                    Perforce has been gone for quite a while. Internal devs predominantly use Piper, though an increasing group is using Mercurial to interact with Piper instead of the native Piper tooling. The Mercurial install is a few minor internal things (eg custom auth), evolve and core Mercurial. We’ve been very wary of using things outside of that set, and are working hard to keep our workflow in line with the OSS Mercurial workflow. An example of something we’ve worked to send upstream is hg fix which helps you use a source code formatter (gofmt or clang-format) as you go, and another is the narrow extension which lets you clone only part of a repo instead of the whole thing.

                    Non-internal devs (Chrome, Android, Kubernetes, etc etc) that work outside of Piper are almost exclusively on Git, but in a variety of workflows. Chrome, AIUI is one giant git repo of doom (it’s huge), Android is some number of hundreds (over 700 last I knew?) of git repos, and most other tools are doing more orthodox polyrepo setups, some with Gerrit for review, some with GH Pull Requests, etc.

                    1. 3

                      Thanks for the clarification, sounds like Piper is (and will continue to be) the source of truth while the “rollout” Greg mentioned is in reference to client side tooling. To my original point, Google still seems to have ended up with the right tool for the job in Piper (given the timeline and alternatives when they needed it).

                      1. 2

                        But how does Mercurial interact with Piper? Is Mercurial a “layer” above Piper? Do you have a Mercurial extension that integrates with Piper?

                        1. 3

                          We have a custom server that speaks hg’s wire protocol. Pushing to piper exports to the code review system (to an approximation), pulling brings down the new changes that are relevant to your client.

                          (Handwaving because I’m assuming you don’t want gory narrow-hg details.)

                          1. 2

                            It’s a layer, yeah. My understanding is that when you send out a change, it makes Piper clients for you. It’s just a UX thing on top of Piper, not a technical thing built into it.

                        2. 2

                          I’m fuzzy on the details, but my understanding is that they’re in the middle of some sort of phased Mercurial rollout. So it’s possible only a sample population of their developers are using the Mercurial backend. What I do know is that they are still actively contributing to Mercurial and seem to be moving in that direction for the future.

                          1. 1

                            I wonder if they are using some custom mercurial backend to their internal thing (basically a VFS layer as the author outlined)? It would be interesting to get some first of second hand information on what is actually being used, as people tend to specifically call out Google and Facebook as paragons of monorepos.

                            My feeling is that google/facebook are both huge organizations with lots of custom tooling and systems. /Most/ companies are not google/facebook nor have google/facebook problems.

                            1. 6

                              This is largely my source (in addition to offline conversations): https://groups.google.com/forum/#!topic/mozilla.dev.version-control/hh8-l0I2b-0

                              The relevant part is:

                              Speaking of Google, their Mercurial rollout on the massive Google monorepo continues. Apparently their users are very pleased with Mercurial - so much so that they initially thought their user sentiment numbers were wrong because they were so high! Google’s contribution approach with Mercurial is to upstream as many of their modifications and custom extensions as possible: they seem to want to run a vanilla Mercurial out-of-the-box as possible. Their feature contributions so far have been very well received upstream and they’ve been contributing a number of performance improvements as well. Their contributions should translate to a better Mercurial experience for all.

                              So at the very least it seems they endeavour to avoid as much custom tooling on top of Mercurial as possible. But like you said, they have Google problems so I imagine they will have at least some.

                              1. 6

                                Whoa. This could be the point where Mercurial comes back after falling behind git for years.

                                Monorepo sounds sexy because Facebook and Google use that. If both use Mercurial and open source their modifications then Mercurial becomes very attractive suddenly.

                                In git, neither submodules nor LFS are well integrated and generate pain for lots of developers. If Mercurial promises to fix that many will consider to switch.

                                Sprinkling some Rust into the code base probably helps to seduce some developers as well.

                                1. 10

                                  Narrow cloning (authored by Google) has been OSS from the very start, and now ships in the hg tarball. If you’ve got need of it, it’s still maturing (and formats can change etc) but it’s already in use by at least 3 companies. I’d be happy to talk to anyone that might want to deploy hg at their company, and can offer at least some help on narrow functionality if that’s needed.

                                2. 1

                                  Thanks for digging!
                                  Pretty interesting for sure.

                            2. 0

                              I’m getting verification from someone at Google, but the quick version as I understood it:

                              Google hasn’t actually used Perforce for a long time. What they had was a Perforce workalike that was largely their own thing. They are now using normal Mercurial.

                              1. 12

                                This isn’t true, Google uses Piper (their perforce clone) internally. Devs have the option of using mercurial or git for their personal coding environments, but commits get converted to piper before they land in the canonical monorepo.

                                1. 2

                                  I’ll ping @durin42; I don’t think I’m misremembering the discussion, but I may have misunderstood either the current state or implementation details.

                            3. 3

                              What is it about git that makes it a poor choice for very large repos?

                              What does Mercurial and Perforce do differently?

                              1. 2

                                In addition to the article @arp242 linked, this post goes into a bit more technical detail. Tl;dr, it’s largely due to how data is stored in each. Ease of contribution is another reason (scaling Git shouldn’t be impossible, but for one reason or another no one has attempted it yet).

                                1. 1

                                  Microsoft has a 300GB git repo. They built a virtual file system to make it work.

                                  1. 1

                                    True, but in the scalability section of the article the author argues that the need for a VFS is proof that monorepos don’t scale. So I think most of this thread is centered around proving that monorepos can scale without the need for a VFS.

                                    I agree that a VFS is a perfectly valid solution if at the end of the day the developers using the system can’t tell the difference.

                                2. 2

                                  Facebook wrote about Scaling Mercurial at Facebook back in 2014:

                                  After much deliberation, we concluded that Git’s internals would be difficult to work with for an ambitious scaling project. [..] Importantly, it [mercurial] is written mostly in clean, modular Python (with some native code for hot paths), making it deeply extensible.

                                  It’s a great example of how applications in a slower language can be made better performing than applications in a faster language, just because it’s so much easier to understand and optimize.

                              1. 18

                                What a curious article. Let’s start with the style, such as calling some of the (perceived) advantages of a monorepo a “lie”. Welp, guess I’m a liar 🤷‍ Good way to have a conversation, buddy. Based on this article I’d say that working at Lyft will be as much fun as working at Uber.

                                Anyway, we take a deep breath and continue, and it seems that everything is just handwaved away.

                                Our organisation has about 25 Go applications, supported by about 20 common dependency packages. For example, we have packages log, database, cache, etc. Rolling out updates to a dependency organisation-wide is hard, even for compatible changes. I need to update 25 apps, make PRs for 25 apps. It’s doable, but a lot of work. I expect that we’ll have 50 Go applications before the year is out.

                                Monorepos exist exactly to solve problems like this. These problems are real, and can’t just be handwaved away. Yes, I can (and have) written tools to deal with this to some extent, but it’s hard to get this right, and in the end I’ve still got 25 PRs to juggle with. The author is correct that tooling for monorepos also needs to be written, but it seems to me that that tooling will be a lot simpler and easier to maintain (Go already does good caching of builds and tests out of the box, so we just have to deal with deploys). in particular, I find it’s very difficult to maintain any sense of “overview” of stuff because everything is scattered over 25 PRs.

                                Note that the total size of our codebase isn’t even that large. It’s just distributed over dozens of repos.

                                It’s still a difficult problem, and there is no “one size fits all” solution. If our organisation would still have just one product in Go (as we started out three years ago) then the current polyrepo approach would continue to suffice. It still worked mostly okay when we expanded to two and three products. But now that we’ve got five products (and probably more on the way in the future) it’s getting harder and harder to manage things. I can write increasingly more advanced tooling, but that’s not really something I’m looking forwards to.

                                I’m not sure how to solve it yet; for us, I think the best solution will be to consolidate our 20 dependency packages in to a single one and consolidate all services of different applications in their own repo, so we’ll end up having 6 repos.

                                Either way, the problems are real, and people who look towards monorepos aren’t all stupid or liars.

                                1. 4

                                  I would imagine that if all you use is Go, and nothing much else, then I would image that you are in the monorepo “sweet spot” (especially if your repo size isn’t enormous). From what I understand, Go was more or less designed around the google internal monorepo workflow. At least until Go 1.10/1.11 or so (6 years? after Go 1.0).

                                  It makes me wonder…

                                  • Are there other languages that seem to make monorepo style repos easier?
                                  • Are monorepos harder/worse if you have many apps written in multiple disparate languages?
                                  1. 7

                                    Main issue with monorepos (imo) is that lots of existing tools assume you are not using them (eg: github webhooks, CI providers, VCS (support for partial worktrees), etc). Not an issue at google scale where such tools are managed (or built) in-house.

                                    1. 3

                                      This point isn’t made enough in the monorepo debate. The cost of a monorepo isn’t just the size of the checkout, it’s also all of the tooling you loose by using something non-standard. TFA mentioned some of it, but even things like git log become problematic.

                                      1. 2

                                        Is there a middleground that scopes the tooling better? What I mean is, keep your web app and related backend services in their monorepo assuming they aren’t built on drastically different platforms and you desire standardisation and alignment. Then keep your mobile apps in separate repos, unless you are using some cross-platform framework which permits a mobile monorepo. You get the benefits of the monorepo for what is possibly a growing set of services that need to refactored together while not cluttering git log et al with completely unrelated changes.

                                        1. 2

                                          Sort of. What really matters is whether you end up with a set of tools that work effectively. For small organizations, that means polyrepos, since you don’t often have to deal with cross-cutting concerns and you don’t want to build / self-host tools.

                                          Once you grow to be a large organization, you start frequently making changes which require release coordination, and you have budget to setup tools to meet your needs.

                                    2. 4

                                      Interesting, Go in my experience is one of the places I have seen the most extreme polyrepo/microservice setups. I helped a small shop of 2 devs with 50+ repos. One of the devs was a new hire…

                                    3. 0

                                      Rolling out updates to a dependency organisation-wide is hard, even for compatible changes. I need to update 25 apps, make PRs for 25 apps.

                                      What exactly is the concern here? Project ownership within an org? I fail to see how monorepo is different from having commit access to all the repos for everyone. PRs to upstream externally? Doesn’t make a difference either.

                                      1. 3

                                        The concern is that it’s time-consuming and clumsy to push updates. If I update e.g. the database package I will need to update that for 25 individual apps, and them create and merge 25 individual PRs.

                                        1. 3

                                          The monorepo helps with this issue, but it can also be a bit insidious. The dependency is a real one and it’s one that any updates to need to be tested. It’s easier to push the update to all 25 apps in a monorepo, but it also can tend to allow developers to make updates without making sure the changes are safe everywhere.

                                          Explicit dependencies with a single line update to each module file can be a forcing function for testing.

                                          1. 2

                                            but it also can tend to allow developers to make updates without making sure the changes are safe everywhere

                                            The Google solution is by pushing the checking of the safety of a change onto the team consuming it, not the one creating it.

                                            Changes are created using Rosie, and small commits created with a review from a best guess as to who owns the code. Some Rosie changes wait for all people to accept. Some don’t, and in general I’ve been seeing more of that. Rosie changes generally assume that if your tests pass, the change is safe. If a change is made and something got broke in your product, your unit tests needed to be better. If that break made it to staging, your integration tests needed to be better. If something got to production, you really have bigger problems.

                                            I generally like this solution. I have a very strong belief that during a refactor, it is not the responsibility of the refactor author to prove to you that it works for you. It’s up to you to prove that it doesn’t via your own testing. I think this applies equally to tiny changes in your own team up to gigantic monorepo changes.

                                          2. 1

                                            Assuming the update doesn’t contain breaking changes, shouldn’t this just happen in your CI/CD pipeline? And if it does introduce breaking changes, aren’t you going to need to update 25 individual apps anyway?

                                            1. 4

                                              aren’t you going to need to update 25 individual apps anyway?

                                              The breaking change could be a rename, or the addition of a parameter, or something small that doesn’t require careful modifications to 25 different applications. It might even be scriptable. Compare the effort of making said changes in one repo vs 25 repos and making a PR for each such change.

                                              Now, maybe this just changes the threshold at which you make breaking changes, since the cost of fixing downstream is high. But there are trade offs there too.

                                              I truthfully don’t understand why we’re trying to wave away the difference in the effort required to make 25 PRs vs 1 PR. Frankly, in the way I conceptualize it, you’d be lucky if you even knew that 25 PRs were all you needed. Unless you have good tooling to tell you who all your downstream consumers are, that might not be the case at all!

                                              1. 1

                                                Here’s the thing: I shouldn’t need to know that there are 25PRs that have to be sent, or even 25 apps that need to be updated. That’s a dependency management problem, and that lives in my CI/CD pipeline. Each dependent should know which version(s) it can accept. If I make any breaking changes, I should make sure I alter the versioning in such a way that older dependents don’t try and use the new version. If I need them to use my new version, then I have to explicitly deprecate it.

                                                I’ve worked in monorepos with multiple dependents all linking back to a single dependency, and marshalling the requirements of each of those dependents with the lifecycle of the dependency was just hell on Earth. If I’m working on the dependency, I don’t want to be responsible for the dependents at the same time. I should be able to mutate each on totally independent cycles. Changes in one shouldn’t ever require changes in the other, unless I’m explicitly deprecating the version of the dependency one dependent needs.

                                                I don’t think VCS is the right place to do dependency management.

                                                1. 3

                                                  Round and round we go. You’ve just traded one problem for another. Instead of 25 repos needing to be updated, you now might have 25 repos using completely different versions of your internal libraries.

                                                  I don’t want to be responsible for the dependents at the same time.

                                                  I mean, this is exactly the benefit of monorepos. If that doesn’t help your workflow, then monorepos ain’t gunna fly. One example where I know this doesn’t work is in a very decentralized ecosystem, like FOSS.

                                                  If you aren’t responsible for your dependents, then someone else will be. Five breaking changes and six months later, I feel bad for the poor sap that needs to go through the code migration to address each of the five breaking changes that you’ve now completely forgotten about just to add a new feature to that dependency. I mean sure, if that’s what your organization requires (like FOSS does), then you have to suck it up and do it. Otherwise, no, I don’t actually want to apply dependency management to every little thing.

                                                  Your complaints about conflating VCS and dependency management ring hollow to me.

                                                  1. 1

                                                    I mean, again, this arises from personal experience: I’ve worked on a codebase where a dependency was linked via source control. It was an absolute nightmare, and based on that experience, I reached this conclusion: dependencies are their own product.

                                                    I don’t think this is adding “dependency management to every little thing”, because dependency management is like CI: it’s a thing you should be doing all the time! It’s not part of the individual products, it’s part of the process. Running a self-hosted dependency resolver is like running a self-hosted build server.

                                                    And yes, different products might be using different versions of your libraries. Ideally, nobody pinned to a specific minor release. That’s an anti-pattern. Ideally, you carefully version known breaking changes. Ideally, your CI suite is robust enough that regressions never make it into production. I just don’t see how different versions of your library being in use is a problem. Why on Earth would I want to go to every product that uses the library and update it, excepting show-stopping, production-critical bugs? If it’s just features and performance, there’s no point. Let them use the old version.

                                                    1. 2

                                                      You didn’t really respond to this point:

                                                      Five breaking changes and six months later, I feel bad for the poor sap that needs to go through the code migration to address each of the five breaking changes that you’ve now completely forgotten about just to add a new feature to that dependency.

                                                      You ask why it’s a problem to have a bunch of different copies of your internal libraries everywhere? Because it’s legacy code. At some point, someone will have to migrate its dependents when you add a new feature. But the point at which that happens can be delayed indefinitely until the very moment at which it is required to happen. But at that point, the library may have already gone through 3 refactorings and several breaking changes. Instead of front-loading the migration of dependents as that happens by the person making the changes, you now effectively have dependents using legacy code. Subsequent updates to those dependents now potentially fall on the shoulders of someone else, and it introduces surprise yak shaves. That someone else then needs to go through and apply a migration to their code if they want to use an updated version of the library that has seen several breaking changes. That person then needs to understand the breaking changes and apply them to their dependent. If all goes well, maybe this is a painless process. But what if the migration in the library resulted in reduced functionality? Or if the API made something impossible that you were relying on? It’s a classic example of someone not understanding all of the use cases of their library and accidentally removing functionality from users of their library. Happens all the time. Now that person who is trying to use your new code needs to go and talk to you to figure out whether the library can be modified to support original functionality. You stare at them blankly for several seconds as you try to recall what it is you did 6 months ago and what motivated it. But all of that would have been avoided if you were forced to go fix the dependent in the first place.

                                                      Like I said, your situation might require one to do this. As I said above, which you seem to have completely ignored, FOSS is one such example of this. It’s decentralized, so you can’t realistically fix all dependents. It’s not feasible. But in a closed ecosystem inside a monorepo, your build doesn’t pass unless all dependents are fixed. Everything moves forward, code migrations are front loaded and nobody needs to spend any time being surprised by a necessary code migration.

                                                      I experience both of these approaches to development. With a monorepo at work and lots of participation in FOSS. In the FOSS world, the above happens all the time exactly because we have a decentralized system of libraries that are each individually versioned, all supported by semver. It’s a great thing, but it’s super costly, yet necessary.

                                                      Dependency management with explicit versioning is a wonderful tool, but it is costly to assign versions to things. Sometimes it’s required. If so, then great, do it. But it is most certainly not something that you “just do” like you do CI. Versioning requires some judgment about the proper granularity at which you apply it. Do you apply it to every single module? Every package? Just third party dependencies? You must have varying answers to these and there must be some process you follow that says when something should be independently versioned. All I’m saying is that if you can get away with it, it’s cheaper to make that granularity as coarse as possible.

                                      1. 9

                                        I switched to Visual Studio Code with Neovim backend yesterday. Neovim provides all the Ext functionality so you can :s/foo/bar to your heart’s content. It’s finally Good Enough to operate as a Vim without having to spend months tuning your .vimrc. I have been using Vim for 5+ years and wrote all my Go code in it.

                                        I think this is what the future of Vim actually is for the majority of people: Neovim providing a backend for their preferred IDE. Interacting in a terminal is incredibly antiquated, even if it’s the sort of thing you are super used to. You can spend your time actually understanding and learning Vim, not trying to make Vim do what you think is reasonable/behaves like your previous editor of choice.

                                        1. 5

                                          Despite being somewhat of a diehard vim fan, 99% of my ‘vim’ usage these days is via emulators - either in VS, VSCode or Firefox.

                                          For me the true value of vim is modal editing (and the associated muscle memory); the plugin ecosystem etc is fine (and at one point I spent a lot of time honing my plugin config) but there’s very little I miss.

                                          1. 2

                                            My experience is the same. I don’t even have gVIm installed on my workstation anymore, but I love love working with the vim extensions in VS, Code, and Firefox.

                                          2. 3

                                            Maybe some day an interface to neovim will appear for Emacs, that would be a nice thing to happen. Perhaps I could start writing it, if I have a chance to learn elisp. Emacs as the extensible front end, with a proper modal backend. In fact the front end could be something better than Emacs, an scheme implementation would be amazing, in order to preserve separation of interests and provide users with a lightweight but infinitely extensible editing environment. If someone with adecuate skills for this (I don’t have them at the moment, so I will have to invest some time learning) is willing to start a with me such project, I would be more than honored to do so, if no interest is shown, eventually I would do it on my own.

                                            1. 3

                                              Check out Oni!

                                              1. 1

                                                Thanks for the recommendation, but I’m not interested in bringing the web to the desktop with the Electron framework, as exciting as it may be for many programmers I think it is still a bad idea. Personally, I think we don’t need tenths of MB in programs’ binaries in order to do text editing, and Javascript isn’t a coherent nor a well defined language to justify its expansion on servers and home computers, I think there are better alternatives to this. Nevertheless, if you like it and it solves your problems, then that’s all that matters in the end.

                                                1. 2

                                                  I don’t actually use it - I use plain neovim in my terminal. I agree with you on the criticisms of electron - it’s just the only program of its kind that I’ve found.

                                                  1. 2

                                                    Sorry If I assumed something incorrect. Some of the ideas in Oni seem interesting, and would be a worthwhile effort to have a look at the source code.

                                            2. 3

                                              Little off-topic, but what do you use to do that integration?

                                              1. 4

                                                The VSCode Vim plugin will do it out of the box, just check “Enable NeoVim”

                                            1. 3

                                              It’d be good in general for commands that have an effect on the state of your filesystem to have a way to declare their inputs and their outputs for a given set of options. This way you’d be able to analyze e.g. install scripts to review which files would be read and written before even running it, and check if you’re missing anything the script depends on.

                                              1. 4

                                                They are unfortunately very chatty though. Alias rm to rm -i and it’s just an obnoxious amount of y y y for any non-trivial removal. I wish someone would fix this to something more like a table output showing a reasonable summary of what’s going and let me confirm the once.

                                                1. 4

                                                  This is a powerful paradigm.

                                                  Incremental confirmation can result in a state where the system has a half-performed operation when the operator decides to abort.

                                                  Displaying a plan – and requiring confirmation of the entire plan – ensures that the operator intends to do everything, or intends to do nothing.

                                                  1. 3

                                                    The book Unix Power Tools, originally published in 1993, includes a recipe for the behavior to rm you’re describing. It is surprising this feature hasn’t made it in to coreutils sometime in the intervening 2 1/2 decades.

                                                    1. 2

                                                      I’ve worked on systems that enforce -i and it just made me develop a very bad -f habit.

                                                  1. 6

                                                    In the general case, I have developed a deep and long-lasting skepticism of DSLs. I was a very heavy proponent of them during my grad studies, and investigated pretty thoroughly using a rules engine for Space Invaders Enterprise Edition and a runtime monitor for Super Mario World.

                                                    I went a little further down this path before I abandoned it for reasons unrelated to the DSL skepticism. That happened later. I just wanted to give context that I was actually naturally predisposed to liking them.

                                                    What has happened in my time on this earth as a software engineer is the feeling that it is axiomatic that all DSLs eventually tend towards something Turing complete. New requirements appear, features are added, the DSL heads further towards Turing completeness. Except the DSL does not have the fundamental mechanics to express Turing completeness, it is by fundamental design supposed to not do that. What you end up with is something very complex, where users are performing all sorts of crazy contortions to get behavior they want, and you can never roll that back. I feel like DSLs are essentially doomed from the outset.

                                                    I am much, much more optimistic about opinionated libraries as the means to solve the problems DSLs do (Ruby on Rails being the most obvious one). That way any of the contortions can be performed in a familiar language that the developer is happy to use and won’t create crazy syntax, and the library then called to do whatever limited subset of things it wants to support. For basic users, they’ll interact with the library only and won’t see the programming language. As things progress, the base language can be brought in to handle more complex cases as pre/post-processing by the caller, without infringing on the design of the library.

                                                    At Google, we have a number of DSLs to perform many different tasks which I won’t go into here. Each one requires a certain learning curve and a certain topping-out where you can’t express what you want. I was much happier with an opinionated library approach in Python, where I could do a great deal of what I wanted without peering behind the curtain of what was going to be performed.

                                                    1. 6

                                                      sklogic on Hacker News had a different view: you start with a powerful, Turing-complete language that supports DSL’s with them taking the place of libraries. He said he’ll use DSL’s for stuff like XML querying, Prolog where logic approach makes more sense, Standard ML when he wants it type-safe in simple form, and, if all else fails or is too kludgy, drops back into LISP that hosts it all. He uses that approach to build really complicated tools like his mbase framework.

                                                      I saw no problem with the approach. The 4GL’s and DSL’s got messy because they had to be extended toward powerful. Starting with something powerful that you constrain where possible eliminates those concerns. Racket Scheme and REBOL/Red are probably best examples. Ivory language is an example for low-level programming done with Haskell DSL’s. I have less knowledge of what Haskell’s can do, though.

                                                      1. 3

                                                        I think it’s a good approach, but it’s still hard to make sure that the main language hosting all the DSLs can accomodate all of their quirks. Lisp does seem to be an obvious host language, but if it were that simple then this approach would have taken off years ago.

                                                        Why didn’t it? Probably because syntax matters and error messages matter. Towers of macros produce bad error messages. And programmers do want syntax.

                                                        And I agree that syntax isn’t just a detail; it’s an essential quality of the language. I think there are fundamental “information theory” reasons why certain syntaxes are better than others.

                                                        Anything involving s-expressions falls down – although I know that sklogic’s system does try to break free of s-expression by adding syntax.

                                                        Another problem is that ironically by making it too easy to implement a DSL, you get bad DSLs! DSLs have to be stable over time to be made “real” in people’s heads. If you just have a pile of Lisp code, there’s no real incentive for stability or documentation.

                                                        1. 4

                                                          “but if it were that simple then this approach would have taken off years ago.”

                                                          It did. The results were LISP machines, Common LISP, and Scheme. Their users do little DSL’s all the time to quickly solve their problems. LISP was largely killed off by AI Winter in a form of guilt by association. It was also really weird vs things like Python. At least two companies, Franz and LispWorks, are still in Common LISP business with plenty of success stories on complex problems. Clojure brought it to Java land. Racket is heavy on DSL’s backed by How to Design Programs and Beautiful Racket.

                                                          There was also a niche community around REBOL, making a comeback via Red, transformation languages like Rascal, META II follow-ups like Ometa, and Kay et al’s work in STEPS reports using “IS” as foundational language. Now, we have Haskell, Rust, Nim, and Julia programmers doing DSL-like stuff. Even some people in formal verification are doing metaprogramming in Coq etc.

                                                          I’d say the idea took off repeatedly with commercial success at one point.

                                                          “Probably because syntax matters and error messages matter. Towers of macros produce bad error messages. And programmers do want syntax.”

                                                          This is a good point. People also pointed out in other discussions with sklogic that each parsing method had its pro’s and con’s. He countered that they can just use more than one. I think a lot of people don’t realize that today’s computers are so fast and we have so many libraries that this is a decent option. Especially if we use or build tools that autogenerate parsers from grammars.

                                                          So, IIRC, he would use one for raw efficiency first. If it failed on something, that something would get run through a parser designed for making error detection and messages. That’s now my default recommendation to people looking at parsers.

                                                          “Anything involving s-expressions falls down – although I know that sklogic’s system does try to break free of s-expression by adding syntax.”

                                                          Things like Dylan, Nim, and Julia improve on that. There’s also just treating it like a tree with a tree-oriented language to manipulate it. A DSL for easily describing DSL operations.

                                                          “nother problem is that ironically by making it too easy to implement a DSL, you get bad DSLs!”

                                                          The fact that people can screw it up probably shouldn’t be an argument against it since they can screw anything up. The real risk of gibberish, though, led (per online commenters) a lot of teams using Common LISP to mandate just using a specific coding style with libraries and no macros for most of the apps. Then, they use macros just handling what makes sense like portability, knocking out boilerplate, and so on. And the experienced people wrote and/or reviewed them. :)

                                                          1. 2

                                                            Probably because syntax matters and error messages matter. Towers of macros produce bad error messages. And programmers do want syntax.

                                                            Another problem is that ironically by making it too easy to implement a DSL, you get bad DSLs! DSLs have to be stable over time to be made “real” in people’s heads. If you just have a pile of Lisp code, there’s no real incentive for stability or documentation.

                                                            I’m so glad to see this put into words. Although for me, I find it frustrating that this seem to be universally true. I was pretty surprised the first time around when I felt my debugger was telling me almost nothing because my syntax was so uniform, I couldn’t really tell where I was in the source anymore!

                                                            Some possibilities for this not to be true that I’m hoping for: maybe its like goto statements and if we restrict ourselves to make DSLs in a certain way, they won’t become bad (or at least won’t become bad too quickly). By restricting the kind of gotos we use (and presenting them differently), we managed to still keep the “alter control flow” aspect of goto.

                                                            Maybe there’s also something to be done for errors. Ideally, there’d be a way to spend time proportional to the size of the language to create meaningful error messages. Maybe by adding some extra information somewhere that currently implicit in the language design.

                                                            I don’t know what to do about stability though. I mean you could always “freeze” part of the language I guess.

                                                            For this particular project, I’m more afraid that they’ll go the SQL route where you need to know so much about how the internals work that it mostly defeats the purpose of having a declarative language in the first place. I’d rather see declarative languages with well-defined succinct transformations to some version of the code that correspond to the actual execution.

                                                            1. 1

                                                              (late reply) Someone shared this 2011 essay with me, which has apparently been discussed to death, but I hadn’t read it until now. It says pretty much exactly what I was getting at!


                                                              In this essay, I argue that Lisp’s expressive power is actually a cause of its lack of momentum.

                                                              I said:

                                                              Another problem is that ironically by making it too easy to implement a DSL, you get bad DSLs!

                                                              So that is the “curse of Lisp”. Although he clarifieds that they’re not just “bad” – there are too many of them.

                                                              He mentions documentation several times too.

                                                              Thus, they will have eighty percent of the features that most people need (a different eighty percent in each case). They will be poorly documented. They will not be portable across Lisp systems.

                                                              Domain knowledge is VERY hard to acquire, and the way you share that is by developing a stable and documented DSL. Like Awk. I wouldn’t have developed Awk on my own! It’s a nice little abstraction someone shared with me, and now I get it.

                                                              The “bipolar lisp programmer” essay that he quotes also says the same things… I had not really read that one either but now I get more what they’re saying.

                                                              1. 1

                                                                Thanks for sharing that link again! I don’t think I’ve seen it before, or at least have forgotten. (Some of the links from it seem to be broken unfortunately.)

                                                                One remark I have is that I think you could transmit information instead of code and programs to work around this curse. Implicit throughout the article is that collaboration is only possible if everyone uses the same language or dialect of it; indeed, this is how version controlled open-source projects are typically structured: around the source.

                                                                Instead, people could collaboratively share ideas and findings so everyone is able to (re)implemented it in their own DSL. I say a bit more on this in my comment here.

                                                                In my case, on top of documentation (or even instead of it), I’d like to have enough instructions for rebuilding the whole thing from scratch.

                                                                To answer your comment more directly

                                                                Domain knowledge is VERY hard to acquire, and the way you share that is by developing a stable and documented DSL

                                                                I totally agree that domain knowledge is hard to acquire but I’m saying that this only one way of sharing that knowledge once found. The other way is through written documents.

                                                        2. 4

                                                          Since I like giving things names, I think of this as the internal DSL vs external DSL argument [1]. This applies to your post and the reply by @nickpsecurity about sklogic’s system with Lisp at the foundation. If there is a better or more common name for it, I’d like to know.

                                                          I agree that internal DSLs (ones embedded in a full programming language) are preferable because of the problems you mention.

                                                          The external DSLs always evolve into crappy programming languages. It’s “failure by success” – they become popular (success) and the failure mode is that certain applications require more power, so they become a programming language.

                                                          Here are my examples with shell, awk, and make, which all started out non Turing-complete (even Awk) and then turned into programming languages.


                                                          Ilya Sher points out the same problems with newer cloud configuration languages.


                                                          I also worked at Google, and around the time I started, there were lots of Python-based internal DSLs (e.g. the build system that became Blaze/Bazel was literally a Python script, not a Java interpreter for a subset of Python).

                                                          This worked OK, but these systems eventually got rewritten because Python isn’t a great language for internal DSLs. The import system seems to be a pretty significant barrier. Another thing that is missing is Ruby-style blocks, which are used in configs like Vagrantfile and I think Puppet. Ruby is better, but not ideal either. (Off the top of my head: it’s large, starts up slowly, and has version stability issues.)

                                                          I’m trying to address some of this with Oil, although that’s still a bit far in the future :-/ Basically the goal is to design a language that’s a better host for internal DSLs than Python or Ruby.

                                                          [1] https://martinfowler.com/bliki/InternalDslStyle.html

                                                          1. 3

                                                            If a programming language is flexible enough, the difference between DSL and library practically disappears.

                                                            1. 1

                                                              DSL’s work great when the domain is small and stays small and is backed by corporal punishment. Business Software is an astronomically large domain.

                                                            1. 1

                                                              I use Google Books for a one-by-one search. Google Books is a digital search inside an analog book. I search for the phrase I think I want in the book I own, use the snippet to figure out which page I want, then find it in the book. Works very well.

                                                              I usually don’t have multiple books on the same topic, I don’t have that much space :)

                                                              1. 1

                                                                “the phrase I think I want in the book I own” -> this is interesting, why would you like something like that before you buy a book? I’m more like, curious to explore what the book gives me.

                                                                1. 1

                                                                  I meant I already own the book. So I have a book, and I think “I think there’s a word or phrase in here that will get me what I want, but the index in this book is garbage”. So I use Google Books.

                                                              1. 1

                                                                I wrote my thesis in LaTeX, which got converted into a book. The publisher, who is technical (not O’Reilly) and you can probably find the book if you doxx me hard enough, made me convert it into Word so the editor could leave comments. The editor would never actually do the edits, just leave comment after comment, which drove me nuts.

                                                                It was not a good experience. I wish I could have just gotten a LaTeX template or Pandoc or something.

                                                                1. 1

                                                                  I use paper a lot for kinda ephemeral stuff but by far the number one problem I have is that it prevents being able to always capture. If I’m away from my desk I now need a secondary capture system to get it onto that list.

                                                                  This is a bit inevitable of course, I think you can’t avoid having more than one and doing some cleanup work. The trick I have at the moment when I’m away from my primary todo-capture systems is to use my phone to set a reminder to remind myself to write it down later.

                                                                  For the “don’t keep a long time” thing, I tend to make daily to-do’s and review the previous day’s to-do’s to actively build the following one. This requires a lot of honesty about not doing something though (otherwise you end up aggregating dead to-do’s in your list). For some things that have been in the list too long I set a calendar event to come back to it in a week or two (time for me to accept failure/have a different perspective on the task).

                                                                  1. 2

                                                                    Yeah, for me it’s a bit different, I’m a kind of sitting-all-day guy, I do have a second list at home as well, the two lists for two kinds of tasks. So I guess, if you gonna need a list when you’re away from your desk, that list should have a different type of tasks from the one you have at your desk :D

                                                                    1. 2

                                                                      The Bullet Journal app on iOS is pretty good for this. It lets you add tasks… but it deletes them after 2 days. You have to put them in your book or they’re gone. So when you’re away from your desk you can enter stuff, but you have to jot it down if you care about it.

                                                                    1. 5

                                                                      I greatly enjoyed using Bullet Journaling. I tried Project Evo and bought 4 from the Kickstarter, and it’s kinda good, but I kinda regret it. I thought “Oh, I would use this monthly layout! Oh, I would use this weekly todo list” and I never do. I do like the gratitude/wellness prompts, but I could have come up with those myself. I get no particular value out of the app aspect.

                                                                      The forcing function of rewriting tasks is the clutch bit of Bullet Journalling. I used to put all my tasks in Inbox, snooze them, snooze them, snooze them, I got anxious and deluged. I think I will go back to Bullet Journals once I fill up my current Evo.

                                                                      1. 3

                                                                        Does anyone know any more about this? I’ve never heard of it and it seems very new, but there is already a BallerinaCon in July? Looks like it’s owned by WSO2 who I’ve never heard of before either.

                                                                        1. 3

                                                                          It has been about 3 years in development but we really started talking about it earlier this year. The origins indeed have been in WSO2’s efforts in the integration space (WSO2 is an open-source integration company and had a research project on code-first approach to integration). Ballerina is an open-source project - at this moment has 224 contributors.

                                                                          It is getting a lot of interest in the microservices and cloud-native (CNCF) space because it supports all the modern data formats and protocols (HTTP, WebSockets, gRPC, etc.), has native Docker and Kubernetes integration (build directly into a Docker image and K8S YAMLs), is type-safe, compiled, has parallel programming and distributed constructs baked in, etc.

                                                                          You can see lots of language examples in Ballerina by Example and Ballerina Guides.

                                                                          1. 2

                                                                            I actually posted this hoping someone would have more info. The language looks interesting and far along to be under the radar.

                                                                            1. 1

                                                                              The company seems to be based in Sri Lanka. It is nice to see cool tech coming from countries like that.

                                                                              1. 1

                                                                                The company seems to be based in Sri Lanka. It is nice to see cool tech coming from countries like that.

                                                                                The project has non-WSO2 contributors as well, and WSO2 has also offices in Mountain View, New York, Pao Paolo, London, and Sydney, but indeed Colombo (Sri Lanka) is the biggest office so at the moment my guess would be that Ballerina is 90% from Sri Lanka - which indeed is a fantastic place! :)

                                                                            1. 1

                                                                              I think standups are all doomed to devolve into misery without an incredibly strong and dedicated hand leading and cutting people off. That would be step 1 for me: find someone who is willing to say “No, you’re standing up so you are as uncomfortable as we are… No your time is up…”

                                                                              I only find the first two or three sentences of what anyone is doing to be useful. The other thing that is useful is to know what problem anyone is wrestling with at the time in case someone knows how to help.

                                                                              If I was running my Iron Fist standups, I think I could get it down to 30s per person ;)

                                                                              1. 5

                                                                                Examples of major changes:


                                                                                simplified, improved error handling?

                                                                                I am glad to see they are considering generics for Go2.

                                                                                1. 5

                                                                                  Russ has more background on this from his Gophercon talk: https://blog.golang.org/toward-go2

                                                                                  The TL;DR for generics is that Go 2 is either going to have generics or is going to make a strong case for why it doesn’t.

                                                                                  1. 1

                                                                                    As it should be…

                                                                                    1. 1

                                                                                      Glad to hear that generics are very likely on the way from someone on the Go team.

                                                                                      The impression I got was that generics were not likely to be added without a lot of community push in terms of “Experience Reports”, as mentioned in that article.

                                                                                      1. 1

                                                                                        They got those :)

                                                                                    2. 1

                                                                                      Wouldn’t generic types change Go’s error handling too? I mean that when you can build a function that returns a Result<Something, Error> type, won’t you use that instead of returning Go1 “tuples” ?

                                                                                      1. 5

                                                                                        For Result type, you either need boxing, or sum type (or union, with which you can emulate sum type), or paying memory cost of both value and error. It’s not automatic with generics.

                                                                                        1. 1

                                                                                          I see, thanks for clarifying! :)

                                                                                        2. 1

                                                                                          As I understand it Go has multiple return values and does not have a tuple type, so not sure how your example would work. There are some tickets open looking at improving the error handling though.

                                                                                      1. 5

                                                                                        Google contributes suprisingly little back to in terms of open source compared to the size of the company and the number of developers they have. (They do reciprocate a bit, but not nearly as much as they could.)

                                                                                        For example this is really visible in the area where they do some research and/or set a standard like with compression algorithms (zopfli, brotli), network protocols (HTTP/2, QUIC), the code and glue they release is minimal.

                                                                                        It’s my feeling that Google “consumes”/relies on a lot more open source code than they then contribute back to.

                                                                                        1. 10

                                                                                          Go? Kubernetes? Android? Chromium? Those four right there are gargantuan open source projects.

                                                                                          Or are you specifically restricting your horizon to projects that aren’t predominantly run by Google? If so, why?

                                                                                          1. 11

                                                                                            I’m restricting my horizon for projects that aren’t run by Google because it better showcases the difference between running and contributing to a project. Discussing how Google runs open source projects is another interesting topic though.

                                                                                            Edit: running a large open source project for a major company is in large part about control. Contributing to a project where the contributor is not the main player running the project is more about cooperation and being a nice player. It just seems to me that Google is much better at the former than the latter.

                                                                                            1. 2

                                                                                              It would be interesting to attempt to measure how much Google employees contribute back to open source projects. I would bet that it is more than you think. When you get PRs from people, they don’t start off with, “Hey so I’m an engineer at Google, here’s this change that we think you might like.” You’d need to go and check out their Github profile and rely on them listing their employer there. In other words, contributions from Google may not look like Contributions From Google, but might just look like contributions from some random person on the Internet.

                                                                                              1. 3

                                                                                                I don’t have the hat, but for the next two weeks (I’m moving teams) I am in Google’s Open Source office that released these docs.

                                                                                                We do keep a list of all Googlers who are on GitHub, and we used to have an email notification for patches that Googlers sent out before our new policy of “If it’s a license we approve, you don’t need to tell us.” We also gave blanket approval after the first three patches approved to a certain repo. It was ballpark 5 commits a day to non-Google code when we were monitoring, which would exclude those which had been given the 3+ approval. Obviously I can share these numbers because they’re all public anyway ;)

                                                                                                For reasons I can’t remember, we haven’t used the BigQuery datasets to track commits back to Googlers and get a good idea of where we are with upstream patches now. I know I tried myself, and it might be different now, but there was some blocker that prevented me doing it.

                                                                                                I do know that our policies about contributing upstream are less restrictive than other companies, and Googlers seem to be happy with what they have (particularly since the approved licenses change). So I disagree with the idea that Google the company doesn’t do enough to upstream. It’s on Googlers to upstream if they want to, and that’s no different to any other person/group/company.

                                                                                                1. 2

                                                                                                  So I disagree with the idea that Google the company doesn’t do enough to upstream.

                                                                                                  Yeah, I do too. I’ve worked with plenty of wonderful people out of Google on open source projects.

                                                                                                  More accurately, I don’t even agree with the framing of the discussion in the first place. I’m not a big fan of making assumptions about moral imperatives and trying to “judge” whether something is actually pulling its weight. (Mostly because I believe its unknowable.)

                                                                                                  But anyway, thanks for sharing those cool tidbits of info. Very interesting! :)

                                                                                                  1. 3

                                                                                                    Yeah, sorry I think I made it sound like I wasn’t agreeing with you! I was agreeing with you and trying to challenge the OP a bit :)

                                                                                                    Let me know if there’s any other tidbits you are interested in. As you can tell from the docs, we try to be as open as we can, so if there’s anything else that you can think of, just ping me on this thread or cflewis@google.com and I’ll try to help :D

                                                                                                    1. 1

                                                                                                      FWIW I appreciate the effort to shed some light on Google’s open source contributions.Do you think that contributions could be more systemic/coordinated within Google though, as opposed to left to individual devs?

                                                                                                      1. 1

                                                                                                        Do you think that contributions could be more systemic/coordinated within Google though, as opposed to left to individual devs?

                                                                                                        It really depends on whether a patch needs to be upstreamed or not, I suppose. My gut feeling (and I have no data for this) and entirely personal and not representative of my employer opinion, is that teams as a whole aren’t going to worry about it if they can avoid it… often the effort to convince the upstream maintainers to accept the patch can suck up a lot of time, and if the patch isn’t accepted then that time was wasted. It’s also wasted time if the project is going in a direction that’s different to yours, and no-one really ever wants to make a competitive fork. It’s far simpler and a 100% guarantee of things going your way if you just keep a copy of the upstream project and link that in as a library with whatever patches you want to do.

                                                                                                        The bureaucracy of upstreaming, of course, is working as intended. There does have to be guidance and care to accepting patches. Open source != cowboy programming. That’s no problem if you are, say, a hobbyist who is doing it in the evenings here and there, where timeframes and so forth are less pressing. But when you are a team with directives to get your product out as soon as you can, it generally isn’t something a team will do.

                                                                                                        I don’t think this is a solved problem by any company that really does want to commit back to open source like Google does. And I don’t think the issue changes whether you’re a giant enterprise or a small mature startup.

                                                                                                        This issue is also why you see so much more open source projects released by companies rather than working with existing software: you know your patches will be accepted (eventually) and you know it’ll go in your direction, It’s a big deal to move a project to community governance as you now lose that guarantee.

                                                                                            2. 0


                                                                                              Did you ever tried to compile it?

                                                                                              1. 2

                                                                                                Yeah, and?

                                                                                                1. 0

                                                                                                  How much time it took? On which hardware?

                                                                                                  1. 1

                                                                                                    90 minutes, on a mid-grade desktop from 2016.

                                                                                                    1. 1

                                                                                                      Cool! You should really explain to Google your build process!

                                                                                                      And to everybody else, actually.

                                                                                                      Because a convoluted and long build process, concretely reduce the freedom that an open source license gives you.

                                                                                                      1. 1

                                                                                                        Cool! You should really explain to Google your build process!

                                                                                                        Google explained it to me actually. https://chromium.googlesource.com/chromium/src/+/lkcr/docs/linux_build_instructions.md#faster-builds

                                                                                                        Because a convoluted and long build process, concretely reduce the freedom that an open source license gives you.

                                                                                                        Is the implication that Google intentionally makes the build for Chromium slow? Chromium is a massive project and uses the best tools for the job and has made massive strides in recent years to improve the speed, simplicity, and documentation around their builds. Their mailing lists are also some of the most helpful I’ve ever encountered in open source. I really don’t think this argument holds any water.

                                                                                            3. 5

                                                                                              The amount Google invests in securing open source software basically dwarfs everyone else’s investment, it’s vaguely frightening. For example:

                                                                                              • OSS-Fuzz
                                                                                              • Patch Rewards for OSS projects
                                                                                              • Their work on Clang’s Sanitizers and libFuzzer
                                                                                              • Work on the kerne’s self protection program and syzkaller
                                                                                              • Improvements to linux kernel sandboxing technologies, e.g. seccomp-bpf

                                                                                              I don’t think anyone else is close, either by number (and severity) of vulnerabilities reported or in proactive work to prevent and mitigate them.

                                                                                              1. 2

                                                                                                Google does care a lot about security and I know of plenty of positive contributions that they’ve made. We probably could spend days listing them all, but in addition to what you’ve mentioned project zero, pushing the PKI towards sanity, google summer of code (of which I was one recipient about a decade ago), etc all had a genuinely good impact.

                                                                                                OTOH Alphabet is the world’s second largest company by market capitalization, so there should be some expectation of activity based on that :)

                                                                                                Stepping out of the developer bubble, it is an interesting thought experiment to consider if it would be worth trading every open source contribution Google ever made for changing the YouTube recommendation algoritm to stop promoting extremism. (Currently I’m leaning towards yes.)

                                                                                            1. 5

                                                                                              It is kind of funny how companies refuse to use software with any freedom restrictions, it is almost as if they know it is a bad thing to have done to you.

                                                                                              1. 4

                                                                                                FWIW Google does not refuse to use GPL, it’s right there in the docs. Using GPL and other restrictive licenses at Google does have some legal overhead involved, and most teams understandably don’t want to have to jump those hoops. The non-restrictive licenses like MIT/BSD/Apache (Apache 2 being the license the vast number of our projects use because of the patent grant) just make staying compliant easier.

                                                                                                The Open Source team deeply cares about being compliant, not just because of the legal issues, but because it’s the right thing to do. It’s important to us as engineers who joined the team because we <3 open source in the first place that we do right by authors. I think it’s easy to think about these things as being created by faceless entities, and not realize that the people that staff these teams all had previous lives, and with our team, all of them released open source projects one way or another.

                                                                                                1. 4

                                                                                                  The GPL is not restrictive, given that it starts with copyright and gives freedoms from that.

                                                                                                  1. 1

                                                                                                    The only time I ever wanted to use GPL was to restrict competition to software I was working on.

                                                                                                2. 1

                                                                                                  Or default on the opposites of how they acquire 3rd party software for their own software that they build or license for their users with careful exceptions for open source:

                                                                                                  “Google gives you a personal, worldwide, royalty-free, non-assignable and non-exclusive license to use the software provided to you by Google as part of the Services. This license is for the sole purpose of enabling you to use and enjoy the benefit of the Services as provided by Google, in the manner permitted by these terms. You may not copy, modify, distribute, sell, or lease any part of our Services or included software, nor may you reverse engineer or attempt to extract the source code of that software, unless laws prohibit those restrictions or you have our written permission.

                                                                                                  Open source software is important to us. Some software used in our Services may be offered under an open source license that we will make available to you. There may be provisions in the open source license that expressly override some of these terms.”

                                                                                                  Right there Google tells you what kind of software is most valuable to them and funds all the FOSS they support. Maybe that should be more developers’ default, too, if they can find a business model to pull it off. ;)

                                                                                                1. 4

                                                                                                  If I move off of OS X, it will be to Windows. For what I use my machine for, the applications simply aren’t there on any Unix other than OS X.

                                                                                                  1. 4

                                                                                                    I’ve been using Linux as my main desktop since about 3 years and used all of the major desktop environments. KDE Plasma looks good but either its file indexer (baloo) is taking hostage of one CPU core or the desktop crashes if you type specific words or too fast in the launcher, in short a horrible experience. I used Gnome for about a year and it was not much better, the plugins/extensions are often buggy and especially under wayland it crashes often and can’t restart like on X11, i.e. you loose all of your session state. Additionally, it feels laggy even on a beefed out machine (6 cores, latest gen. AMD GPU) because the compositor is single-threaded. GDM, gnome’s display manager, is also sluggish, runs since gnome 3.26 a process for each settings menu and starts a pulseaudio session which breaks bluetooth headset connections. Also unsuable for a productive environment in my opinion. Eventually I switched back to the desktop environment with what I started my Linux journey, namely XFCE with lightdm as a display manager. With compton as compositor it looks quite okay, is rock solid (in relation to the other DE I used) and everything feels snappy. As a note, I run all of the DEs on Arch Linux and I haven’t even talked about display scaling and multi-monitor usage, still a horror story.

                                                                                                    TL;DR The year of the Linux desktop is still far away in the future.

                                                                                                    1. 5

                                                                                                      I wouldn’t really know where to go. I have an Arch desktop at home (quad Xeon, 24 GB RAM, 2 SSDs), while the machine is much faster than my MacBook Pro, I usually end up using the MacBook Pro at home (and always at work), simply because there are no equivalents for me for applications like OmniGraffle, Pixelmator/Acorn, Microsoft Office (project proposals are usually floated in Word/Excel format with track changes), Beamer, etc. Also, both at work and home, AirPlay is the standard way to get things on large screens, etc.

                                                                                                      Also, despite what people are saying. The Linux desktop is still very buggy. E.g. I use GNOME on Wayland with the open amdgpu drivers on Arch (on X I can’t drive two screens with different DPIs). And half of the time GNOME does not even recover from simple things like switching the screen on/off (the display server crashes, HiDPI applications become blurry, or application windows simply disappear).

                                                                                                      Windows would probably have more useful applications for me than Linux or BSD (since many open source applications run fine on WSL). But my brain is just fundamentally incompatible with any non-unix.

                                                                                                      1. 8

                                                                                                        Linux has been my main desktop for 20 years or so? Although I am a software developer and either do not need the applications you mentioned or use alternatives.

                                                                                                        Anyway, what I actually wanted to say: on the hardware side I’ve had little issues with Linux, certainly not more than with Windows or OS X and at least with Linux (if I put the time into it) the issues can generally be fixed. I’ve been running multiple monitors for years and hibernation used to be a pain in the ass in the early 2000’s but has been good for me on a wide array of hardware for years (definitely better than both Windows and OS X which run on supported hardware!). Granted, I can’t blindly grab hardware off the shelf and have to do some research up front on supported hardware. But that’s what you get if hardware vendors do not officially support your OS and it does come with many upsides as well.

                                                                                                        I run pretty bare systems though and try to avoid Window’isms that bring short-term convenience but also bring additional complexity, so no systemd, pulseaudio, desktop environments like Gnome for me. Still, I’m running Linux because I want to be able to run Dropbox (actually pCloud in my case), Steam, etc.

                                                                                                        1. 4

                                                                                                          Linux has been my main desktop for 20 years or so?

                                                                                                          Different people, different requirements. I have used Linux and BSD on the desktop from 1994-2007. I work in a group where almost everybody uses Macs. I work in a university where most of the paperwork is done in Word (or PDF for some forms). I have a fair teaching load, so I could mess around for two hours to get a figure right in TikZ (which I sometimes do if I think it is worth the investment and have the time) or I could do it in five minutes in OmniGraffle and have more time to do research.

                                                                                                          It’s a set of trade-offs. Using a Mac saves a lot of time and reduces friction in my environment. In addition, one can pretty run much the same open source applications as on Linux per Homebrew.

                                                                                                          I do use Linux remotely every day, for deep learning and data processing, since it’s not possible to get a reasonable Mac to do that work.

                                                                                                          Anyway, what I actually wanted to say: on the hardware side I’ve had little issues with Linux, certainly not more than with Windows or OS X and at least with Linux (if I put the time into it) the issues can generally be fixed.

                                                                                                          The following anecdote is not data, but as a lecturer I see a lot of student presentations. Relatively frequently, students who run Linux on their laptops have problems getting projectors working with their laptops, often ending up borrowing a laptop from one of their colleagues. Whereas the Mac-wielding students often forget their {Mini DisplayPort, USB-C} -> VGA connectors, but have no problems otherwise.

                                                                                                      2. 2

                                                                                                        Same. I don’t use them every day, but I do need Adobe CS. I also want (from my desktop) solid support for many, many pixels of display output. Across multiple panels. And for this, Windows tends to be better than Mac these days.

                                                                                                        1. 1

                                                                                                          The Windows Linux Subsystem is also surprisingly good. I would say that it offers just enough for most OS X users to be happy. Linux users, maybe not.

                                                                                                        2. 1

                                                                                                          One thing I’m finding is that a lot of Mac apps I rely on have increasingly capable iOS counterparts (Things, OmniOutliner, Reeder, etc.) so I could potentially get away with not having desktop versions of those. That gets me closer to cutting my dependency on macOS, though there’s still a few apps that keep me around (Sketch, Pixelmator) and the ever-present requirement of having access to Xcode for iOS development.

                                                                                                        1. 5

                                                                                                          I can’t speak for Go’s genesis within Google, but outside of Google, this underanalysed political stance dividing programmers into “trustworthy” and “not” underlies many arguments about the language.”

                                                                                                          Correct, but not unique. Java is probably as popular as it is precisely because encapsulation is so strong, allowing you to protect your code from others who you worry will do harm.

                                                                                                          On the unskilled programmers side, the language forbids features considered “too advanced.”… This is the world in which Go programmers live - one which is, if anything, even more constrained than Java 1.4 was.

                                                                                                          My feelings on Go is that it’s largely defined by what isn’t in there than what is. What isn’t in there allows me to read the code in the stdlib and understand it. I can look at any piece of code from another developer and not worry that there are multiple abstraction levels that I’m missing that alter the behavior. There isn’t enough in the language to support subset dialects like you’ll see with something like Perl or even Java (a Spring code base looks very different to a POJO code base looks very different to someone obsessed with interfaces looks very different to…)

                                                                                                          I think Go succeeds very admirably along the dimension of “everyone at the company should be able to read everyone else’s code and modify it safely.” That necessarily means some blunting of the scissors. If you want the bleeding edge tool, Rust is a great alternative, but there’s enough in it that I would be afraid letting a junior programmer at it without guidance, and that new people to Rust will have a long ramp up time to figure out how the code base works and be able to modify it.

                                                                                                          EDIT: This is not to say I don’t have my own axes to grind: I do not like not being able to easily differentiate between types of errors in particular. I don’t feel like interface{} is a good solution to anything, but I don’t necessarily have any desire for generics either. I don’t care for the way contexts work, although I understand why they exist (I like this proposal about this).

                                                                                                          1. 10

                                                                                                            I have found meaning by:

                                                                                                            1. Letting go of thinking that I, at 34, will do something that will meaningfully change the world. I am too late in my life to realistically expect me to have one of those Big Ideas, and I came too late to the party of realizing I found meaning in making the planet better (I am really ambivalent about humanity, whereas when I grew up I felt like technology would improve everything for everyone, which is not a belief I hold anymore). I think you and I have had very similar feelings, I felt very much like the world wasn’t benefitting from what I did, so why bother?
                                                                                                            2. Assuming I won’t have that impact I think I should make, my best bet is to act as an enabler for someone who can. So I work on foundational products like cloud systems, backend APIs, that sort of thing. The cheaper and more accessible foundational systems get, the more likely I can enable someone that will do something amazing. That way I don’t have to buy into the vision of a consumer product in order to get meaning. Consumer products come and go and very few are ever going to make the world a better place (maybe the last one was Facebook or even as far back as Google Maps). Two years ago I took a position to try and optimize for the day-to-day (working on smaller systems after burning out on a big one) that caused the enabling dimension to suffer, and I lost my sense of purpose and meaning. That has affected my excitement about my work much more negatively than the issues of what I left.

                                                                                                            I think of projects like Mozilla, cloud, even something like Tensorflow or Kubenetes, as being these kind of enablers.

                                                                                                            Sure, the most meaningful thing in my life is my family. The most meaningful things in your life are almost certainly going to be outside of work. I’m trying to do more meaningful things like Hour of Code and encourage girls and minorities into STEM (again following the enabling track). But it is important, I think, to find some sort of thing you can extract from the day-to-day as worth it to you, otherwise you’re just going to stop getting out of bed.

                                                                                                            1. 3

                                                                                                              I think when I posed the question, your second bullet point is what I secretly wanted to hear the most.

                                                                                                              1. 3

                                                                                                                Last night, I saw that @cflewis and @freddyb as two sides of same coin in terms of doing meaningful work in IT. One is making sure it’s easy to create things that benefit humanity using technology. One is making sure it’s easy to consume them. In each case, you want what you make to be designed for high uptake for one. Usability, marketing/branding, and cost trumps internal tech on either of these almost every time. Then, if dollars and/or code contributions are rolling in, you can use your influence to make sure whatever it is goes in public-benefiting rather than predatory directions.

                                                                                                                Firefox is actually a good example where they make money off ad-driven search but let you do private search easily. Always having an affordable, private version of anything ad driven is another example. On organizational side, you might charter or contract in basic protections/benefits for everyone from employees to users. On the technology stack, you might build on better foundations to shift more money or effort into quality tech that deserves it. Quick example from my field would be things like routers most half-ass with shoddy tech instead using OpenBSD, secure admin interface, and automatic updates. On web side, it might be those using tech like Erlang to be efficient and highly-reliable with big, success stories getting more people investing in its tooling.

                                                                                                                There’s a lot of possibilities that involve doing something that people want to use or buy that’s just more effective and/or less evil than the norm. Sadly, the norm has so much of those two that there’s plenty of ways to differentiate on those. The relief being that there’s plenty of ways to differentiate on those. :)

                                                                                                            1. 25

                                                                                                              I used to do the things listed in this article, but very recently I’ve changed my mind.

                                                                                                              The answer to reviewing code you don’t understand is you say “I don’t understand this” and you send it back until the author makes you understand in the code.

                                                                                                              I’ve experienced too much pain from essentially rubber-stamping with a “I don’t understand this. I guess you know what you’re doing.” And then again. And again. And then I have to go and maintain that code and, guess what, I don’t understand it. I can’t fix it. I either have to have the original author help me, or I have to throw it out. This is not how a software engineering team can work in the long-term.

                                                                                                              More succinctly: any software engineering team is upper-bound architecturally by the single strongest team member (you only need one person to get the design right) and upper-bound code-wise by the single weakest/least experience team member. If you can’t understand the code now, you can bet dollars to donuts that any new team member or new hire isn’t going to either (the whole team must be able to read the code because you don’t know what the team churn is going to be). And that’s poison to your development velocity. The big mistake people make in code review is to think the team is bound by the strongest team member code-wise too and defer to their experience, rather than digging in their heels and say “I don’t understand this.”

                                                                                                              The solution to “I don’t understand this” is plain old code health. More functions with better names. More tests. Smaller diffs to review. Comments about the edge cases and gotchas that are being worked around but you wouldn’t know about. Not thinking that the code review is the place to convince the reviewer to accept the commit because no-one will ever go back to the review if they don’t understand the code as an artifact that stands by itself. If you don’t understand it as a reviewer in less than 5 minutes, you punt it back and say “You gotta do this better.” And that’s hard. It’s a hard thing to say. I’m beginning to come into conflict about it with other team members who are used to getting their ungrokkable code rubber stamped.

                                                                                                              But code that isn’t understandable is a failure of the author, not the reviewer.

                                                                                                              1. 7

                                                                                                                More succinctly: any software engineering team is upper-bound architecturally by the single strongest team member (you only need one person to get the design right) and upper-bound code-wise by the single weakest/least experience team member.

                                                                                                                Well put – hearing you type that out loud makes it incredibly apparent.

                                                                                                                Anywhoo, I think your conclusion isn’t unreasonable (sometimes you gotta be the jerk) but the real problem is upstream. It’s a huge waste when bad code makes it all the way to review and then and then needs to be written again; much better would be to head it off at the pass. Pairing up the weaker / more junior software engineers with the more experienced works well, but is easier said than done.

                                                                                                                1. 4

                                                                                                                  hmm, you make a good point and I don’t disagree. Do you think the mandate on the author to write understandable code becomes weaker when the confusing part is the domain, and not the code itself? (Although I do acknowledge that expressive, well-structured and well-commented code should strive to bring complicated aspects of the problem domain into the picture, and not leave it up to assumed understanding.)

                                                                                                                  1. 3

                                                                                                                    I think your point is very much applicable. Sometimes it takes a very long time to fully understand the domain, and until you do, the code will suffer. But you have competing interests. For example, at some point, you need to ship something.

                                                                                                                    1. 2

                                                                                                                      Do you think the mandate on the author to write understandable code becomes weaker when the confusing part is the domain, and not the code itself?

                                                                                                                      That’s a good question.

                                                                                                                      In the very day-to-day, I don’t personally find that code reviews have a problem from the domain level. Usually I would expect/hope that there’s a design doc, or package doc, or something, that explains things. I don’t think we should expect software engineers to know how a carburetor works in order to create models for a car company, the onus is on the car company to provide the means to find out how the carburetor works.

                                                                                                                      I think it gets much tricker when the domain is actually computer science based, as we kind of just all resolved that there are people that know how networks work and they write networking code, and there’s people who know how kernels work and they write kernel code etc etc. We don’t take the time to do the training and assume if someone wants to know about it, they’ll learn themselves. But in that instance, I would hope the reviewer is also a domain expert, but on small teams that probably isn’t viable.

                                                                                                                      And like @burntsushi said, you gotta ship sometimes and trust people. But I think the pressure eases as the company grows.

                                                                                                                      1. 1

                                                                                                                        That makes sense. I think you’ve surfaced an assumption baked into the article which I wasn’t aware of, having only worked at small companies with lots of surface area. But I see how it comes across as particularly troublesome advice outside of that context

                                                                                                                    2. 4

                                                                                                                      I’m beginning to come into conflict about it with other team members

                                                                                                                      How do you resolve those conflicts? In my experience, everyone who opens a PR review finds their code to be obvious and self-documenting. It’s not uncommon to meet developers lacking the self-awareness required to improve their code along the lines of your objections. For those developers, I usually focus on quantifiable metrics like “it doesn’t break anything”, “it’s performant”, and “it does what it’s meant to do”. Submitting feedback about code quality often seems to regress to a debate over first principles. The result is that you burn social capital with the entire team, especially when working on teams without a junior-senior hierarchy, where no one is a clear authority.

                                                                                                                      1. 2

                                                                                                                        Not well. I don’t have a good answer for you. If someone knows, tell me how. If I knew how to simply resolve the conflicts I would. My hope is that after a while the entire team begins to internalize writing for the lowest common denominator, and it just happens and/or the team backs up the reviewer when there is further conflict.

                                                                                                                        But that’s a hope.

                                                                                                                        1. 2

                                                                                                                          t’s not uncommon to meet developers lacking the self-awareness required to improve their code along the lines of your objections. For those developers, I usually focus on quantifiable metrics like “it doesn’t break anything”, “it’s performant”, and “it does what it’s meant to do”. Submitting feedback about code quality often seems to regress to a debate over first principles.

                                                                                                                          Require sign-off from at least one other developer before they can merge, and don’t budge on it – readability and understandability are the most important issues. In 5 years people will give precisely no shits that it ran fast 5 years ago, and 100% care that the code can be read and modified by usually completely different authors to meet changing business needs. It requires a culture shift. You may well need to remove intransigent developers to establish a healthier culture.

                                                                                                                          The result is that you burn social capital with the entire team, especially when working on teams without a junior-senior hierarchy, where no one is a clear authority.

                                                                                                                          This is a bit beyond the topic at hand, but I’ve never had a good experience in that kind of environment. If the buck doesn’t stop somewhere, you end up burning a lot of time arguing and the end result is often very muddled code. Even if it’s completely arbitrary, for a given project somebody should have a final say.

                                                                                                                          1. 1

                                                                                                                            The result is that you burn social capital with the entire team, especially when working on teams without a junior-senior hierarchy, where no one is a clear authority.

                                                                                                                            This is a bit beyond the topic at hand, but I’ve never had a good experience in that kind of environment. If the buck doesn’t stop somewhere, you end up burning a lot of time arguing and the end result is often very muddled code. Even if it’s completely arbitrary, for a given project somebody should have a final say.

                                                                                                                            I’m not sure.

                                                                                                                            At very least, when no agreement is found, the authorities should document very carefully and clearly why they did take a certain decision. When this happens everything goes smooth.

                                                                                                                            In a few cases, I saw a really seasoned authority to change his mind while writing down this kind of document, and finally to choose the most junior dev proposal. And I’ve also seen a younger authority faking a LARGE project just because he took any objection as a personal attack. When the doom came (with literally hundreds of thousands euros wasted) he kindly left the company.

                                                                                                                            Also I’ve seen a team of 5 people working very well for a few years together despite daily debates. All the debates were respectful and technically rooted. I was junior back then, but my opinions were treated on pars with more senior colleagues. And we were always looking for syntheses, not compromises.

                                                                                                                        2. 2

                                                                                                                          I agree with the sentiment to an extent, but there’s something to be said for learning a language or domain’s idioms, and honestly some things just aren’t obvious at first sight.

                                                                                                                          There’s “ungrokkable” code as you put it (god knows i’ve written my share of that) but there’s also code you don’t understand because you have had less exposure to certain idioms, so at first glance it is ungrokkable, until it no longer is.

                                                                                                                          If the reviewer doesn’t know how to map over an array, no amount of them telling me they doesn’t understand will make me push to a new array inside a for-loop. I would rather spend the time sitting down with people and trying to level everyone up.

                                                                                                                          To give a concrete personal example, there are still plenty of usages of spreading and de-structuring in JavaScript that trip me up when i read them quickly. But i’ll build up a tolerance to it, and soon they won’t.

                                                                                                                        1. 12

                                                                                                                          It is very, very unlikely that he was rejected purely for a binary tree search issue. He says in the OP he had 7 interviews. When a hiring committee looks at the data from 7 interviews, a single bad interview isn’t enough to torpedo a whole application. Sometimes an interviewer and and interviewee just don’t jive. It might have been that he was just middling in the interviews, and middling is great for most companies.

                                                                                                                          From all I’ve ever read on this topic, it seems to me that Howell thinks his previous record should have counted in the process, and that’s a legitimate feeling, but that’s not the way Google interviews. A good record gets you through the door, but once you’re in the interview room, interviewers barely look at the resume past some warmup questions or a “Hey cool, you did Homebrew? That’s wild! I use that all the time!” and then move on to the questions they are used to asking and calibrating their recommendations for. Personally, I am pretty glad that external record isn’t counted, I like that a fresh university graduate or someone who has been stuck working in a secret at a company that doesn’t produce exciting software has the same starting potential as Howell. But I get how that can burn.

                                                                                                                          1. 3

                                                                                                                            We find experience can actually work against candidates; if the interviewer reads too deeply into a resume and well known projects on it they can set unrealistically high expectations on how they’ll do in the interview.

                                                                                                                            Starting from a more or less blank slate is really the only fair way to do interviews in volume with lots of different interviewers.