1. 55

  2. 30

    Most of my gigs have been small companies so the “if you or your friends made it it doesn’t count” rule knocks out a whole lot of helper scripts and infra.

    That said, I hope that the bash we left behind and the bash we’re writing now make our coworkers lives’ a little easier and frees them up to lead more fulfilling lives. :)

    1. 23

      This might mean that you work at a place that doesn’t have nice tooling, and that’s probably because nobody’s built anything good. That in turn might be because nobody builds anything, period.

      Or they’re just busy building a product for their customers. In a tiny company, the programmer(s) might be too busy with that to build internal tools. Besides, the more open source is available for everyone to use and build on, the less we need internal tools.

      1. 36

        I really have a different POV from the author. Most of the world is not a special snowflake and should not be building (and maintaining!) their own custom infrastructure. Like cars and trucks, there are millions of mechanics around the world but only a few companies building new vehicle platforms from scratch. Companies need software mechanics to customize, diagnose and fix “glue” issues but they really shouldn’t be building complex infrastructure from scratch (that’s literally not their business) when COTS or OSS will suit their needs just fine.

        1. 11

          Most of the world is not a special snowflake and should not be building (and maintaining!) their own custom infrastructure.

          I don’t really understand this viewpoint–why not? I think companies are hurt more by trying to customize overly-generalized solutions like SAP.

          1. 5

            Or for a more contemporary example, Kubernetes. This is why the Unix principle is still important – we should have many small tools (/ services), all doing a single thing very well but nothing else. That way people can pick out the correct combinations for each occasion.

            It seems this conflicts with capitalism though – fattening up your own product/service seems to be good for the bottom line, unfortunately. People dream about single products that fix all their problems even when it bogs them down.

            1. 1

              I’d be interested to hear more about your experiences with Kubernetes. What were the big mistakes in the implementations you’ve seen?

              1. 1

                I wasn’t implying that people deploy it badly, just that it’s a huge “fix everything” sort of thing, which is generally speaking not a good thing, but which is also a thing that many people seem to want.

            2. 5

              I think there’s a balance to be struck. In my personal experience, if you find yourself working to customise a third-party/open-source tool you’re in one of a few situations:

              • There’s a fix that can be contributed upstream, either because it’s open source or thanks to a responsive vendor. If your vendor isn’t responsive, it’s my own opinion that you’re not in this situation no matter how simple the fix, so carry on down the list
              • You are using the tool wrong and would benefit from adjusting your workflow to use the tool as it was intended
              • You chose a tool that doesn’t work for your use case

              The last two points can be difficult to distinguish and require some honesty during assessment. Sometimes the answer to “but our use case is special” is to stop being special, but sometimes we have good reasons.

              If you find yourself in the last situation, it’s time to look for a new tool or build something. But not everybody does, and that’s fine.

            3. 8

              Agreed. There are definitely tools that I’ve used at various companies that I’ve enjoyed working with and that I’ve been grateful to have been exposed to, but they were pretty much all open-source projects that some person or team at the company found and decided to use in some official or recommended workflow, rather than bespoke tooling written by people at the company. I view this as a good thing - it means that I don’t actually have to give up the use of a useful tool when I switch jobs, and also that if anyone at my company makes a fix or improvement to a tool, that change can get propagated to everyone who uses it.

              1. 6

                Far from all problems have a generic open source solution. So your viewpoint implies that the company should create such a solution rather than a bespoke, solving-just-the-problem-at-hand thing.

                That is a tall order. Creating something for a very specific task usually means a simpler implementation, which usually means lower maintenance, fewer bugs, better performance characteristics, and so on.

              2. 5

                I mostly agree with you. Most companies can do fine without custom tools. I don’t see how the lack of custom tools implies that the company sucks.

                The one aspect I can see: If your company needs no custom tools then the business is not revolutionary/world changing/unicorn.

                1. 2

                  This is an extremely revisionist point of view. Your not-custom architecture initially was someone’s custom architecture. Standard solutions we’ve converged on are/were either maintained by companies who built them to solve their problems or by non-profit organizations that couldn’t use stock solution because it didn’t exist.

                  And last, but not least, calling someone who worked primarily for large orgs that build custom solutions because they need to “a special snowflake” while not recognizing that you’re bashing something you’re doing yourself requires some creative thinking.

                2. 15

                  It’s night-and-day between working at a company that prioritizes infrastructure and internal tooling (one of companies Rachel worked for - we overlapped in time but worked on different things) and those that are on the opposite end of the buy vs. build spectrum. There’s no comparison between the coherent experience of using tools that are built to solve the real problems of the organization (probably experienced by the same people building the tools), to something that was designed externally by committee to address what some business team believes “the market” needs. I also worked at a place where the company had seemingly went on a buying spree tour of SoMa software vendors to patch together their internal infrastructure. I could not believe how much money was being spent on these crappy 80% solutions, and how leadership would nod their head about the value of tooling while letting this advice go in one ear and out the other. I did not last long there.

                  1. 10

                    Interesting. The chain of reasoning is that managing infrastructure requires good infrastructure tooling and that most good tooling is internal, and therefore if you don’t have internal tooling you’re going to have a hard time managing infrastructure.

                    That’s a big company mindset. You can just not manage infrastructure if you’re a startup. Current startup wisdom is that you focus on what differentiates you and buy the rest because time and agility are crucial.

                    1. 7

                      I miss the proprietary table libraries from inside two investment banks I worked at (for Python, which was the house language). A much simpler API than pandas and considerably better memory usage. Plus, because they were common dependencies in everything people didn’t avoid them as they sometimes do with heavyweight libraries like pandas - none of the “I’ll start with a list of dicts and migrate to a proper table library if we really need to”.

                      If you want to get a sense of them see here for a FOSS reimplementation: https://pythonhosted.org/eztable/examples.html#building-a-table

                      1. 1

                        I’m working at a bank… I get the use when you have a lot of data, but people sometimes try to fit small data that needs structured properties and it’s not the be all end all.

                        I wonder if we overlap or if there are three investment banks with these libraries.

                        1. 2

                          Yes quite possible we’re talking about the same thing! For the avoidance of doubt I’m talking about the libraries QzTable and xtds at BAML and JP Morgan respectively.

                          In practice for these table libraries there is no data too small because even a one row table can be useful in a semi join against another (larger) table. IMO a good table datastructure is nearly universally applicable replacement for list and dict but that is a drum to bang another time… :)

                          1. 1

                            Indeed, I’m using QzTable :)

                            Sadly it’s impossible to enforce a unique key constraint within these schema. A different table library in a different life would have probably had this.

                            Enforcing such schema in code only works if you trust your conditionals will be left alone.

                      2. 7

                        A very one-sided post. The premises are nice, but the conclusion is wrong.

                        Using all software-as-a-service? I wish more people would call on this; it’s just that most companies are not considered “infrastructure” shops; so, all these tools get delegated outside of the org.

                        I also think there’s a pretty big difference between using someone else’s software that’s closed and proprietary, and/or provided only as a service, versus using OSS, possibly with a support contract. I guess if you’ve got a support contract, you might not be justified to hack on it yourself, too, but this all depends.

                        Also, the solution would be to release more OSS from within the org, and to open up any and all proprietary software that you may have developed in-house, so that, if necessary, you could continue benefit from these tools even once you depart. However, this has to be done in a smart way: once you get the permission from the supervisor chain and the legal to open up a tool, you have to do it in one go without notifying all internal customers, with a public announcement loud enough to not make it possible to revert the decision — the announcement has to be wide, and there have to be many clones of the repo, such that if sudden corporate detractors are found that would want to undo the release, they wouldn’t have any practical way of doing so. E.g., if you’re part of a very big org, make sure to have a HN post and Twitter announcements ready BEFORE you change the GitHub permissions; otherwise, some random people from other teams of your big org, which might be mere non-developer consumers of your product, might ruin your show. Source: it did happen to a project I worked on.

                        1. 3

                          Lol, that plan sounds oddly specific. You did it before and got burned? Or you did it before and got it right.

                          1. 2

                            Yeah, we in the server team got the permission to open-source a server implementation, but the owner of the desktop client team/experience saw the new permissions on GitHub, and moved against it. Lesson learned: HN early, HN often.

                            And, it was a worry for Erlang in Ericsson, too; gladly, they did better than us:


                            Until Erlang was out, many did not believe it would happen. There was a fear that, at the last minute, Ericsson was going to pull the plug on the whole idea. Open Source, a term which had been coined a few months earlier, was a strange, scary new beast large corporations did not know how to handle. The concerns Ericsson had of sailing in uncharted territory, rightfully so, were many. To mitigate the risk of Erlang not being released, urban legend has it that our friend Richard O’Keefe, at the time working for the University of Otago in New Zealand, came to the rescue. Midnight comes earlier in the East, so as soon as the clocks struck midnight in New Zealand, the erlang.org website went online for a few minutes. Just long enough for an anonymous user to download the very first Erlang release, ensuring its escape. When the download was confirmed, the website went offline again, only to reappear twelve hours later, at midnight Swedish time. I was in Dallas, fast asleep, so I can neither confirm nor deny if any of this actually happened. But as with every legend, I am sure there is a little bit of truth behind it.

                        2. 5

                          Only big companies have R&D money for developer tooling. The important part is the product that you are trying to build.

                          I have been in companies that have a good setup, yet everything they use is off-the-shelf or open source tooling. This is a much better situation to be in, because all the knowledge gathered is transferable to the next company.

                          1. 2

                            I have been in companies that have a good setup, yet everything they use is off-the-shelf or open source tooling. This is a much better situation to be in, because all the knowledge gathered is transferable to the next company.

                            Exactly. If you’re using these proprietary tools, you’re gaining knowledge that has no applicability at the new place. It’s the same as programming in a proprietary language used by a single workplace, or using anything proprietary. It should be a big deal-breaker for anyone who’s interested in portable knowledge and workplace mobility.

                            Even if it’s OSS, if it’s not popular-enough, it could still interfere with the marketability of your skills, and if it’s full-on proprietary, then most people wouldn’t even know much about it in the first place, and would have not much way of knowing even if they kind of wanted to, leaving you with fewer tickboxes for applicable experience.

                            The more you think about it — the premise of the article is interesting, but the conclusion and title are just really couldn’t be more wrong.

                          2. 4

                            Which company told her to stop writing and stop helping random people in the company? (From the chopped off link in the OP.)

                            1. 9

                              Rachel used to be at Facebook.

                              1. 4

                                Ironically, a company whose service is connecting people, letting them write what’s on their mind, and share it with others.

                                1. 5

                                  Facebook’s service is surveilling people, sorting them into demographic buckets, and selling their attention to the highest bidder. Anything else they do is simply fertilising the field in which the targettable consumers grow.

                                  1. 2

                                    I was talking the user-facing side. You’re spot on about the business side. Love that last sentence, too.

                                2. 1


                              2. 4


                                Although now CloudWatch logs does the same thing :)

                                1. 3

                                  WinDbg and XPerf . Although these are both publicly available, moving employers would probably imply moving operating systems, so these tools can’t come along.

                                  For the smaller stuff, I’ve been progressively rewriting them specifically to allow me to use them at home/in future. It’s kind of sad that companies can build up a treasure trove of small, useful tools then keep them under lock and key.

                                  1. 1

                                    What’s the saying: grow senior engineers from within your own company, or hire them from places that do?

                                    I can’t really imagine my tools getting worse, to be honest.