1. 45

To be specific, articles that changed in some way how you program but also aren’t already recommended by everyone. For example, everybody already knows about Goto Considered Harmful, so that doesn’t count, nor does The Case Against Python 3 (because nobody found it influential). I’m looking for the lessons you took away that other programmers might not necessarily be exposed to.

Here are some of the ones that influenced me:

  • An introduction to property-based testing: Introduces “why quickcheck” in F#. I can’t say for certain that it made me start thinking in terms of adversarial programming, but it definitely helped.

  • Scalability! But at what COST?: Big data systems vs some guy’s laptop. For a few dozen gigabytes (the stuff it’s trendy to call “big data”) the laptop is orders-of-magnitude better. It’s an example I use at work for why we can’t just rely on “throw more servers at it” for scaling, as well as why it’s often better to first try algorithm improvements before infrastructure improvements.

  • Use of Formal Methods at AWS How AWS uses TLA+. Convinced me that formal methods might be practical in my day-to-day work, which lead to me learning TLA+, which lead to me obsessing about TLA+. TLA+ TLA+ TLA+.

  • Static vs. dynamic languages: a literature review: Argues that there’s no rigorous evidence that static typing makes your code less buggy. I’m sure everybody has thoughts about the conclusion, but my main takeaway is that we should do more controlled research on software engineering methods, vs arguing from experience or first principles. My other main takeaway was to buy a copy of Making Software, which I’d really recommend.

Anyway, I’m interesting in hearing what was important for you.


  2. 15

    JB Rainsberger’s http://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam-part-1

    This was referenced in a recent Lobster’s post https://lobste.rs/s/qq3c2r/boundaries_2012 which incidentally is an excellent solution to the problem posed by Rainsberger

    1. 14

      So, I can’t really claim that all of the following are “unkown” or “lesser-known”, but they’ve all been pretty influential for me. I’d suggest a lot of the “well-known” papers/talks/books are also mostly paid lip service–see also “A classic is that which everybody knows and nobody has read”.


      • Execution in the Kingdom of the Nouns: In his usual sprawling but fun way Yegge brings up the queer notions of OOP programming and hints a bit how a different worldview could be interesting.

      • Dynamic Storage Allocation: A Survey and Critical Review: When I’d read this at school years ago, it was probably my first encounter with rigorous academic computer engineering writing–at least the first that gave me the impression “Oh, shit, people can write stuff that isn’t just weird math and still have rigor”. It’s not the only paper on malloc, it might not even be the best paper on malloc, but I enjoyed it immensely.

      • Free Electron and Bored People Quit: Rands has some essays that are a bit hit or miss, but these two are representative of some of his thoughts on how to manage talented engineers. Pick some stuff at random from the archives on his blog, at least half are good.

      • The Rise of “Worse is Better”: Love it, hate it, miscontsrue it–all valid reactions to this essay. That said, it helped me really reflect on why a lot of my own views on systems development were wrong. Youngersock was very enamored with large, complex systems because they had so many intricate parts and did all the things and how can you have worked on something impressive if it’s simple and doesn’t solve every problem? Youngersock was, needless to say, wrong.

      • Microkernels are slow and Elvis didn’t do no drugs: A solid writeup from last year, but it’s a good reminder on how to just really eviscerate a point you disagree with, using quotes and reference and just generally building an argument properly.


      That’s it for now, but hopefully those’ll at least be some entertaining reads for you folks.

      1. 2

        darknedgy is a great blog. I loved their post on init systems.

        1. 2

          I think microkernels are slow for information theoretic reasons.

          When microkernels are not slow, it’s because they don’t use the standard microkernel module design. I like the QNX OS a lot but, as Wikipedia points out: “Later versions of QNX reduce the number of separate processes and integrate the network stack and other function blocks into single applications for performance reasons”

        2. 8
          1. 8

            slumming with basic programmers - it is very clear to me that basic did something (or several somethings) very right, and one of my background life goals is to figure out what and see if i can bring it to more modern systems.

            some theories i have:

            • near-frictionless access to input and output

            • the ability to just write code without ever having to think of code structure up-front

            • a relatively low set of basic concepts, so even badly-written spaghetti code could be worked through with a little patience

            • full access to the useful parts of the operating system

            of course, there are a lot of things basic did wrong too, but it seems as though in fixing them we have thrown the baby out with the bathwater.

            1. 4

              I agree with all of your points. I would also add that everyone’s standards were looser back then, as far as what constituted a polished program.

              1. 4

                It’s one of the curious points about Walter Bright of D language….

                One of his inspirations, believe or not, for D, was strings should be at least as easy as strings in Basic was.

                C and C++ strings were a gigantic leap backwards compared to Basic.

                D language has lots of purity …. but dollops of hard nose practicality.

                It’s a hard balance to achieve and I’m not sure they have it perfect yet…. but the impressive thing is they are so open to learning along the way.

                1. 3

                  Don’t forget

                  • a slick (for the time) baked-in IDE and interpreter
                2. 10
                  • Railway Oriented Programming really opened my eyes to a different way of error handling. It’s a great metaphor for explaining the concept.

                  • Replacing Python series by Thomas Leonard, about choosing a new language for his project 0install. Very in-depth and interesting considerations.

                  • Alternate Hard and Soft Layers was the first WikiWeb-style software pattern I was introduced to. Markedly different from the software patterns I learned in school! And yet still useful.

                  • Human Factors And Object Oriented Programming is a great example of WikiWeb culture, as programmers grapple with the OOP paradigm and how to most effectively wield it.

                  • nostrademons comment from 2008 about the lessons software engineers learn over a long career.

                  • Deep C a slideshow more than an article, but it taught me a lot about C (and how much I don’t know about it!)

                  • Magpie programming language again not truly an article, but it is a very interesting stab at proglang design. I discovered the language during my honeymoon with Ruby, and it was fascinating to see Bob’s take on a Ruby-like language with multimethods baked in. I can see why Google hired him to work on Dart.

                  1. 3

                    I can see why Google hired him to work on Dart.

                    For what it’s worth, I was hired before Dart existed. I got the job at Google doing JS front end programming.

                  2. 5

                    Peter Miller’s Recursive Make Considered Harmful taught me how to truly use make, and a lot about build systems.


                    1. 4

                      This is a really scattered list, and isn’t really papers and isn’t all engineering; anyhow here goes:

                      • Steve McConnell’s Classic Mistakes in Software Project Management got to me early. Another thing heard through him that’s stuck with me is to prefer to name things in your program after their relationship to the user/outside world, not just describing the code itself.

                      • Not technical, but I was exposed to Eugene Volokh’s Academic Legal Writing in school, and specifically something it argued (riffing on requirements to get a patent) that applies outside of its field: “Good legal scholarship should make (1) a claim that is (2) novel, (3) nonobvious, (4) useful, (5) sound, and (6) seen by the reader to be novel, nonobvious, useful, and sound.”

                      • Kyle Kingsbury’s blog definitely changed how I look at distributed systems and consistency. The Tail at Scale was an interesting intro to another way that things are really different in large deployments. There are lots of practical lessons scattered around the Web about controlling retries, avoiding unbounded queues, having circuit breakers and backpressure, doing more work in the easy case so you’re prepared for the hard one, and so on; Jeff Hodges' collection gives you some of the flavor.

                      • Yao Yue’s talk about caching at Twitter gets into very practical details on how important and difficult it is to get steady predictable performance. Also included the observation that separating services is not a magical way to isolate problems (e.g., a broken service can still request a single key over and over until the cache server holding it melts down). It suggests that some of the wins from service architectures are from the things we traditionally do at service boundaries (logging, ratelimiting, etc.). I found it a lot more candid than many tech talks that focus on how awesome a service is. :)

                      • There is an old talk from Google’s head of site reliability engineering about the managerial mechanisms they use to keep things working: services get “error budgets” and have to slow down deploys when they go over them (so they’re neither going for an impossible goal of 0 nor free to break whatever); devs do occasional shifts as SREs (nothing builds empathy like doing the work yourself); in extreme circumstances a service’s SRE team can choose to disband and be reassigned within the company (rarely-used nuclear option when a service is intolerable to support). Can’t find the video :(. Also under “what big orgs do,” was fascinating to hear Rachel Potvin talk about Google’s monorepo and maybe more importantly the related tools she touches on (like global checks and refactorings).

                      • Weirdly inclined to include, given its context, the long post in late 2013 that argued mobile JavaScript speed was never really going to improve much. Going through it with reference to the substantial changes that did happen since then is more interesting to me than reading it when it came out.

                      • Less is exponentially more. It’s (consciously) a manifesto, and I’d say best treated as one extreme in the debates in your head. There’s something there, though. It’s not just languages; often when a Google or Amazon paper comes out describing some key system running their business, the surprise is often what they decided to leave out or do in a very un-clever sloppy-looking way, and there turn out to be good reasons. (Following Go’s changelists, issues, reviews, etc. is interesting too.)

                      • Futzing with with OCaml didn’t go anywhere practically speaking (I worked some Project Euler exercises) but was helpful for seeing how something like compiler-checked exhaustive pattern matching can be useful (and fits with other things like sum types). Generally, it’s neat to at least read about languages/environments with wildly different goals, including tools you wouldn’t want to use or aspects that aren’t often emphasized when people discuss programming languages in the abstract. For example, Nim and D are big on compile-time metaprogramming, and Perl is something I wouldn’t use anymore but lives in its own world of linguistic ideas. (Ada too, obviously in a very different way.) Qt is something I got exposed to early and impressed me for trying to provide a human-friendly environment (lots of tools and a form of refcounting) in the often-unfriendly context of ‘90s C++.

                      • For folks running mostly OLTP-type database workloads, including me and many of us here, fast random access is where it’s at. However, there’s this whole other world of batch algorithms for huge datasets that are mostly bandwidth-constrained and can zoom given a big RAID array of slow-seeking HDDs to read from (and, depending on the algo, lots of RAM). It’s interesting to read about BLAST or Redshift or Sort Benchmark’s pennysort or basic algorithms for larger-than-RAM data. Scalyr has a post that’s kind of related to this.

                      • On a practical note, recently Camille Fournier published a post about how developers and other individual contributors get stuck that will probably ring a lot of bells for folks here.

                      1. 3

                        Symmetry and Symmetry Breaking in Software (PDF)

                        This one talks about the deep connection between symmetry and symmetry breaking in software and the real world. It got me to understand why when I close my eyes, I can feel that my software has a weight and a shape.

                        1. 1

                          Ok. I have honestly tried with this one.

                          But every time I have a go at it…. I get to the point where I yell “Bollocks!” or “Waffle!” and throw it at the wall.

                          But I have a sneaking suspicion they might , just might be on to something, but are saying it very very badly.

                          Would you care to pull out the gems from that one as, well, not a TL;DR; but a DFRATIATW (Didn’t Fully Read As Threw It At The Wall)?

                          1. 1

                            TL;DR; Symmetry and symmetry breaking is deeply related to physics and geometry. Software also has symmetry and symmetry breaking. Therefore their is a connection between software and the physical world.

                            If you close your eyes and imagine your software project, I bet you have a feeling of shape and weight to it and can “navigate” it with your minds eye. This is no accident, we are wired for symmetry and symmetry breaking.

                        2. 3

                          Should be better known: Parnas Software Jewels.

                          1. 3

                            If you read both of these papers, you get something profound:

                            Notes on Postmodern Programming : http://www.mcs.vuw.ac.nz/comp/Publications/CS-TR-02-9.abs.html

                            Design Beyond Human Abilities (by Richard Gabriel, author of Worse is Better): https://scholar.google.com/scholar?cluster=5397162041763930663&hl=en&as_sdt=0,5&sciodt=0,5

                            The theme here is to give up on thinking of software as a well-formed and engineered crystal, i.e. a “modern” structure. I notice this tendency among programmers to want to make a perfect world within a single model that you understand. Everything is in Java in my IDE; everything is modelled with strong types in Haskell, code and data in a single uniform Lisp syntax, single language operating systems, etc.

                            Through tremendous effort, you can make your software a crystal. But that betrays your small point of view more than anything. All you have to do is zoom out to the larger system and you’ll see that it’s wildly inconsistent – made of software in different languages, and of different ages.

                            For some reason I feel the need to defend the word “post-modern”. Certain types of programmers are allergic to this term.

                            Here is means something very concrete:

                            “Modern” – we have a model of things, we understand it, and and we can change it.

                            “Postmodern” – there are multiple models of things – local models that fit specific circumstances. We don’t understand everything, and we can’t change some things either.

                            Even though humans created everything in computing, no one person understands it all. For example, even when you make a single language OS like Mirage in OCaml, you still have an entire world of Xen and most likely the Linux kernel below you.

                            Another example: a type system is literally a model, and this point seems beyond obvious to me, but lots of people don’t recognize it: you need different type systems for different applicaitons. People seem to be searching for the ultimate type system, but it not only doesn’t exist, but is mathematically impossible.

                            1. 2

                              I find this to be a refreshing perspective on software engineering as it does have interesting points. I agree with the sentiment for large software systems. After a certain size, a single individual can only really have a mental model of a small portion of the overall system.

                              But from mid-size to small-sized system I think this is incorrect. You can definitely have a grand narrative on mid-sized systems. For example, using “post-modern” techniques like design patterns. Most payment processing systems at banks have an architecture team which will gather requirements and create high-level (architecture) diagrams and mid-level diagrams (object diagrams). Essentially the Software Requirements Specifications allows us to translate it to a grand narrative that can be understood visually. Additionally this gets easier as you go closer to bare metal.

                              1. 2

                                Maybe a typo, but to clarify design patterns are a modern technique and not post-modern. They are a model within OOP, which is itself a modern idea, if you think that “everything is an object”. In the post-modern perspective OOP is an appropriate language and abstraction for some systems.

                                I haven’t worked on payment processing systems, but they seem like large and long-lived systems. I’m sure there is decades-old code in almost all current production systems. I thought there were a lot of banks with systems in COBOL?

                                The point is that if you look from a large enough perspective, they will be composed of heterogeneous parts, each with different models.

                                I briefly worked at an insurance company when I was young, and that was definitely true. You had a data warehouse, probably on a mainframe, and they were connecting to it with Microsoft Access. If you think that Microsoft Access “was” the system, then you are sorely mistaken. There is decades of legacy code behind the scenes, and it absolutely affects the way you do your work. If you try to “wall off” or “abstract” the old system, you end up with poorly working and inefficient systems.

                                Based on my customer experience, payment systems don’t appear to be gracefully adapting to adversaries.

                                That’s not to say that modern techniques are bad. They work; they’re just limited to a domain. If you read the first article, they explicitly make the point that post-modern techniques encompass modern ones. They just pick and choose whatever is useful, rather than claiming to have the one solution and grand narrative.

                              2. 2

                                a type system is literally a model,

                                Eh? Yes, you may choose to use it to model things…. but I think you’re on to a pretty sticky wicket as soon as you try.

                                Type systems to me are purely axiomatic rules for a mathematical game.

                                Just symbols and declarations and definitions and references in a digraph with rules for what is allowed and what isn’t.

                                Model? Really?

                                Ye Olde 1980’s era Object Orient Analysis and Design texts were full of that ….

                                But I rapidly gave up on that as totally disconnected from reality.

                                You work out what outputs you want from the inputs…. and then you play the type game to produce the simplest thing that will do what you want.



                                Just a game with rules that you can pile up. But because the rules are consistent, you can make very big piles and know what they will do. (So long as you don’t cheat).

                                1. 2

                                  When I say model, I mean it’s a map that you can reason about at compile time for what the program does at runtime.

                                  But the map is not the territory. I wrote this comment a few years ago and it still reflects my position: https://news.ycombinator.com/item?id=7913684

                                  I don’t think the use of the word “model” is very controversial. On that page:


                                  “I think it’s most helpful to consider a modern static type system (like those in ML or Haskell) to be a lightweight formal modelling tool integrated with the compiler.”

                                  Models and maps are useful. But they can be applied inappropriately, and don’t apply in all situations. That’s all I’m saying, which I think should be relatively controversial. And that’s the “postmodern” point of view, although I admit that this is perhaps a bad term because it inflames the debate with other connotations.

                                  Wikipedia has a pretty good explanation:

                                  The map–territory relation describes the relationship between an object and a representation of that object, as in the relation between a geographical territory and a map of it. Polish-American scientist and philosopher Alfred Korzybski remarked that “the map is not the territory” and that “the word is not the thing”, encapsulating his view that an abstraction derived from something, or a reaction to it, is not the thing itself. Korzybski held that many people do confuse maps with territories, that is, confuse models of reality with reality itself.


                                  1. 2

                                    Ye Olde Object Oriented Design folk were very keen on modelling reality by types…..

                                    Aha! We have a Car in our problem domain, we’re going to have a Car type and a Wheel type and …. and these are going to be the symbols we draw on our maps to represent the territory.

                                    Over the decades of dealing with large systems I have completely divorced from that mindset.

                                    I’m only interested in invariants and state and constraints and control flow and dependencies.

                                    Models of reality? I don’t see that in the lines of code. I just see the digraph of dependencies, encapsulated state, invariants enforced, and ultimately inputs and outputs.

                                    Models of reality? If reality says X and class invariant says Y, I know all kinds of buggy things will happen if I violate the class invariant…

                                    But if I violate reality…. It’s an enhancement request, not a bug.

                                    Part of our difference is you’re talking about static typing, I’m talking about typing.

                                    Smalltalk / ruby / … are typed, just dynamically typed.

                                    When you invoke a method on a variable, you can’t know until run time which implementation of that method will actually get invoked.

                                    But you can know exactly the algorithm by which it decides which one will get invoked.

                                    With a static typed language like C, you can know which function at compile time, because you can evaluate it’s rules at compile time.

                                    ie. For both, its an axiomatic mathematical game with fixed rules.

                                    If you show me a class “Car”, I ignore completely the similarity of the word “Car” to the name of the thing I drive.

                                    I look at it’s state, it’s methods, it’s invariants, it’s dependencies.

                                    I might after awhile, as an afterthought, mutter that it’s badly named…

                                    To me we only map to some semblance of reality at a feature level, not a type.

                                    1. 2

                                      Yeah I know what you are saying – there was a this naive way of thinking of taking the nouns and verbs in your domain and trying to “model” reality with objects. That is not how I think either, but it’s not what I meant by the word “model”. I meant model in a more mathematical sense, like “formal modelling”.

                                      I think of it as naive top-down design. I tend to work from the bottom up… write the simplest thing that does what you need to do. Then does it have the structure of an object? Factor it out. I don’t try to start with the classes. Then you get a structure that doesn’t match what your code actually does.

                              3. 2


                                TL;DR; The people involved, and how thing are in their lives are on the day are more important in creating software than tools or processes.

                                  1. 2

                                    This one’s as old as the hills :) But it’s a goodie IMO.

                                    csh.whynot - I read this article super early in my career and as a result developed a rather strong dislike for csh in general, but especially for using it in any kind of scripting context.


                                      1. 1

                                        Ease At Work

                                        Ok, the following is not a paper, but a talk.

                                        But he is talking from a very human and personal perspective, and hence the medium is more suited to what he is saying.


                                        If you don’t feel up to wrangling infoq and want a youtube link, I don’t know if it is exactly the same thing.. But hey, that’s what you get for being human.


                                        And interesting technical counterpoint to that one is Rich Hickey’s Simple made Easy

                                        1. 1

                                          Justice Will Take Us Millions Of Intricate Moves, a talk on web service design and networked computing history by Leonard Richardson:


                                          1. 1

                                            Alvaro Videla has a great blog post titled “A Programmer’s Role”. In it, he goes over an article from 1967(?), titled “What a Progammer Does”.

                                            That original article is pure gold – shame that the author’s name was lost in archival.