1. 12

I was reading Flash Boys (book about HFT) and noticed an interesting excerpt:

Russians had a reputation for being the best programmers on Wall Street, and Serge thought he knew why: They had been forced to learn to program computers without the luxury of endless computer time. Many years later, when he had plenty of computer time, Serge still wrote out new programs on paper before typing them into the machine. “In Russia, time on the computer was measured in minutes,” he said. “When you write a program, you are given a tiny time slot to make it work. Consequently we learned to write the code in ways that minimized the amount of debugging. And so you had to think about it a lot before you committed it to paper. . . . The ready availability of computer time creates this mode of working where you just have an idea and type it and maybe erase it ten times. Good Russian programmers, they tend to have had that one experience at some time in the past—the experience of limited access to computer time.”

I was curious what your thoughts were about this - in this specific case (computer time) and in general (any other tooling that has made programming more accessible)? Is Serge right or wrong about this?

  1. 28

    I think the analysis is completely off.

    Alongside these tools, many things changed in programming. Complexity of systems exploded, the bar for quality dropped (mostly because software got applied first into mission-critical domains where failure was not acceptable) and expectactions of the value a programmer should provide in its worktime skyrocketed.

    This puts a lot of pressure on the single developer or on the team. Overall the goal of programming fundamentally changed. The mode of production also changed: bigger teams, in bigger organizations, working on bigger problems and being hired from a much bigger and more varied workforce, with different drives than perfectionism, that for sure won’t be enforced by the stakeholders. The more you grow the number of people working concurrently on a single system, the harder it is to progress.

    The way software was developed, let’s say, in the 60s survives only in very small niches and it has no resemblace with how software is developed overall so the comparison is imho unfair.

    On top of it there’s a huge misconception about the contribution of the individual and its skills to a piece of software. Software is written by teams situated in organizations that are in turn situated in networks of organizations. The informational flow is complex and rarely individual programmers are hubs in this flow. There are some that are more prominent and influential than others and retain some small degree of autonomy and creativity, but they are ultimately constrained by the incredible amount of complexity they operate in. Some exceptions exist, like Terry Davis, but clearly they are not representative of the category.

    This misconception arises, imho, from the extremely individualist and machist culture that shaped the IT since its early years, where the production of software is a kind of performative act much alike a sport. You’re judged by your peers and by yourself on the results of your performance, real or imagined. Only in this ideologica framework this rethoric of the degradation of the individual skills can make sense. It’s not by chance that it echoes the reactionary narratives of old people celebrating an imagined “past-time” to contrast with the “softness” of the present times.

    Then for sure the two ways of writing code can condition people to think differently and lead to different behaviors and “performance” on some metrics, but the whole problem is ill-concieved.

    I’m reading Evidence-based Software Engineering these days and I’m grown more and more skeptical of all the pseudo-science that is pushed as facts to defend the cultural status quo. Most of these claims are unsubstatiated and the low bar for accepting these kinds of naive and broad statements over matters of software development without the need for strong evidence should go out of fashion as soon as possible.

    1. 5

      If an individual’s role is ultimately so context dependent that we cannot assess skill, then what is the appropriate unit of critique? The end product? And how do we sell the fact that dev performance is so context-dependent to the people who hire and fire?

      1. 9

        Well, we speak about “software ecosystems”. Assessing their properties seems like a more meaningful endeavor. Most of them are not doing well.

        For the second question, I would say it’s political and it’s hard to discuss without going OT. Anyway salaries are always determined by a more complex network of factors than the simple expectation of performance, something that is going out of fashion anyway in favor of soft skills. I personally believe the only way out is to implement workplace democracy and let colleagues vote your salary. Or even better, give the same salary to everyone regardless of role, skill and seniority.

      2. 4

        …with different drives than perfectionism, that for sure won’t be enforced by the stakeholders.

        Programming is a job. Being good at your job means delivering what you’re asked to deliver. One can argue that you should also insert yourself into the conversation to some extent, and I agree with that, but at the end of the day it’s always going to be a compromise. Programmers who insist on any kind of “purity” probably just aren’t good at their actual jobs.

        Software is written by teams situated in organizations that are in turn situated in networks of organizations.

        I couldn’t agree more and, with respect to teams, I think John Wooden said it best:

        “A player who makes a team great is more valuable than a great player. Losing yourself in the group, for the good of the group, that’s teamwork.”

        Does a person help my team deliver software to stakeholders? If they stomp their feet and demand that we spend a week reducing the program’s memory footprint by 10% when the stakeholders don’t care, then the answer is “no”.

        1. 3

          Does a person help my team deliver software to stakeholders? If they stomp their feet and demand that we spend a week reducing the program’s memory footprint by 10% when the stakeholders don’t care, then the answer is “no”.

          What if the person is demanding you spend a week improving critical security when the stakeholders don’t care?

          1. 4

            Short answer: no, if the stakeholders don’t want security and we can’t convince them to want security then whining about it isn’t going to change anything. Quit or get back to work.

            Long answer: it’s really complicated. Is the team split on whether to care about security? If so, then we need some internal discussions to either get the rest of the team on board or to convince the teammate with concerns that the problem isn’t critical.

            Also, who’s liable? Is it me personally? Then the answer is pretty clear, I’m not going to take the fall for something like that. But what if the stakeholders hold (hehe) liability? Well, then maybe that teammate needs to decide whether they care enough to quit in protest. Obviously we can keep bringing up, keep forcing conversations, but at the end of the day if the people paying for the software don’t care about security, we can’t make them care.

            Another possibility is that we find time to fix it ourselves. I’m not going to kill myself working overtime to protect some corporate behemoth from an avalanche of lawsuits or embarrassment, but maybe the stakeholder is a non-profit with an extremely limited budget trying to solve some important social problem. Or maybe I feel solidarity with the corporate behemoth’s customers. It all depends.

            Ultimately, it’s not cut and dry. The world is messy and we just have to deal with this stuff case-by-case.

          2. 2

            Programming is a job. Being good at your job means delivering what you’re asked to deliver. One can argue that you should also insert yourself into the conversation to some extent, and I agree with that, but at the end of the day it’s always going to be a compromise. Programmers who insist on any kind of “purity” probably just aren’t good at their actual jobs.

            Lets flip this around a bit to demonstrate how your viewpoint applied to another job results in something you don’t want and ultimately shows how little power programmers truly have in decision making with this mindset.

            Civil engineering is a job. Being good at your job means delivering what you’re asked to deliver. One can argue that you should also insert yourself into the conversation to some extent, and I agree with that, but at the end of the day it’s always going to be a compromise. Civil engineers who insist on any kind of “purity” probably just aren’t good at their actual jobs.

            Lets imagine a civil engineer insisting on the “purity” of doing design constraints and getting a better idea of the actual loads and lifetime for a bridge. I’d consider this “purity” to be necessary, and civil engineers not doing it to be the ones that are bad at their jobs. Yet we as a profession in software seem to take every corner and cut it off not with a chisel, but a full blown guillotine.

            Delivering a bridge that will fail because we didn’t think things through to me isn’t really doing any stakeholders anything good. Doing that for software to me is no different, we’ve just deluded and infantalized ourselves into thinking the end result is the only thing that matters to the alter of stakeholders. If there isn’t a conversation about the shortcuts that have been taken, then how can they properly evaluate the deliverable? If that 10% memory footprint is the difference between success and failure, and your team knows it, you’re doing nobody any good by ignoring it.

            I agree with Jonathan Blow on a lot of modern programming, we are deluding ourselves if we think we can make robust software lately that uses the least resources and fits the domain well. If we keep saying “nobody does it cause nobody will pay for it or its not really needed”, do we really think we can do it at all if we never do it? All we’re ending up doing is building shitty bridges that get one car across in steady winds constantly, patching them up when they inevitably fell down, and called it “engineering” to fix it to be closer to a design that could stand a lifetime. Then we throw it away with the next fad and build a new bridge in the new fad just like the first bridge that fell down and re learn our mistakes and lack of foresight yet again.

            Most programming should be boring, and damn near bulletproof. What we have today is to me the equivalent to making obsidian arrowheads. We have no clue what we’re doing and seem to as a profession ignore things that other fields have done to improve their fields. We also have no real stakes in what we do and seem to lack the wherewithal to push back on stakeholders to say: I can give you something now, but it won’t last and the long term cost will be more than the short term cost of doing it right. We constantly get fat off of the seed corn for tomorrow in this profession and wonder why our time is starved later in our projects and we have so little corn to plant to make tomorrow great.

            Sorry for the rant, I’m seeing this everywhere and had to get my thoughts over the past two years into a more concrete form.

            1. 4

              If there isn’t a conversation about the shortcuts that have been taken, then how can they properly evaluate the deliverable?

              This is exactly what I meant when I said that there should be compromise. A conversation seldom ends in one side or the other getting exactly what they originally (said they) wanted. Even in Civil Engineering the engineers might come back and say “look, you’re asking for a two lane bridge, but the traffic patterns on this road will mean traffic jams within 10 years if we don’t bump it up to four lanes”. But if there’s only a budget for a two lane bridge, then that may be what gets built.

              If that 10% memory footprint is the difference between success and failure, and your team knows it, you’re doing nobody any good by ignoring it.

              When I said “purity” I was thinking of people who want to do things a certain way not because it will meaningfully improve the product, but because they believe that is the “right” way to do it. And I specifically said that the stakeholders don’t care about the 10% memory gain. That means that either the customers don’t care (because they won’t notice it) or the business won’t pay for it and is willing to suffer the consequences. Ideally, the team told the stakeholders what the performance would look like, and hopefully even presented benchmarks and demonstrations. But at the end of the day, it’s a conversation, and just because you have a conversation doesn’t mean you get your way. We shouldn’t infantilize ourselves, but we also shouldn’t infantilize the non-developers we work with by pretending that we’re always “right”.

              As an aside, I think it’s interesting that you brought up Civil Engineering because that’s one of the most regulated branches of engineering (at least in the US). I’m actually 100% fine with the government stepping in and laying down rules for Software Engineering and who is allowed to engage in it, particularly for areas where people might be injured or killed. But there’s a libertarian streak a mile wide in this industry and those folks want to keep it the “wild wild west”. If Civil Engineering didn’t have rules and the accompanying liability for breaking those rules there’s no way things would be as “pure” (to use the term the way you used it) as they are.

            2. 1

              Being good at your job means delivering what you’re asked to deliver. Debatable. Not all organizations are vertical authoritarian machines where orders are transfered downwards. It’s a very ineffective way of organizing things. Actually for programmers it is quite the opposite: our power as workers and our desire for autonomy forced a great deal of organizational science progress to make organizations horizontal, to some degree. But as you say, this autonomy should be paired with responsability and be driven by intent, not by edonism.

              Just because some (most?) programmers abuse this autonomy, it’s not a good argument to become sheeps and surrender to the will of the shepherd.

              1. 2

                I see your point. I guess what I had in mind was that you can be “asked” to deliver something by the marketplace, customers, your team through a group decision, etc. I agree wholeheartedly that a vertical command-and-control pattern is almost never the right way to run an organization.

          3. 6

            There’s a story that Knuth first wrote the source code of TeX on paper, typed it in, and a few hours later after some minor fixes it worked.

            On the other hand, we don’t know how long it took him to design it, and to hand-write many thousands of lines of code before he began to type. Maybe overall it took longer than if he’d started the process at a computer.

            I started out with limited access to computer time, so I wrote my programs on paper; but I was writing silly little BASIC programs. I’d never want to go back to that. I use the editor as a sketch pad and experiment with ideas and implementations in a way you can’t do on paper.

            1. 10
              1. 4

                Once you have the essential design nailed, you can usually argue you wrote it with a bug here and there and it just worked.

                However, IME, arriving there involves hours upon hours of evolving/discarding ideas and designs. Hand-waving that work away is disingenuous at best.

                1. 4

                  That’s the thing, isn’t it? Computers are just better at editing and revising text.

                  You’ll have the same experience when writing a letter the old-fashioned way (which I have done a few times when I was younger). You can’t expect a perfect letter the first time around, so either you use a lot of whiteout (or pencil and an eraser) or you write it, cross out stuff and then when you’re happy with the result, copy it over to a new piece of paper and hope you won’t make any mistakes transcribing the final version.

                  Of course, when you’re editing code on a computer, the temptation to just run whatever work-in-progress you have is rather big (and not necessarily wrong; it can save you a lot of headache later on if you let the compiler catch brainfarts early on). Taken to the extreme, you end up with shotgun debugging and cowboy coding where you go with the first thing that barely runs and ship that. Programmers tend to think in black and white, so given the fact that sometimes you might get tempted to do this means that programming in a (semi-)interactive way on a computer must be bad.

                  It makes sense from a stoic sort of perspective to try coding with pen and paper once in a while, for the same reason it might make sense to undergo voluntary hardships by camping out in the woods, to take cold showers or to go entirely offline for a period. And it may feel good to be “virtuous” in this way, but don’t mistake that feeling of superiority for efficiency :)

                  1. 2

                    Yup. The dangerous practice, I think, is living by “do the simplest thing that could possibly work” without the discipline of refactoring when things get messy. I always try to do both. My designs and codebases evolve a lot, and each individual line probably gets rewritten twice before I’m done, but it helps me a lot.

                  2. 3

                    I find that a REPL helps for certain types of programs as well. This is a tool that has existed in one form or another for quite some time but has gotten significantly more powerful in the last decade or so.

                    1. 1

                      Which types of programs?

                      1. 4

                        I use a REPL frequently when I’m implementing a tricky algorithm or trying to decide how best to do a particular thing and have questions about performance or ergonomics. For example:

                        • I might run a quick test against some real data and time how long different things actually take. O(?) doesn’t mean much if the performance still feels slow.
                        • I can try out a particular API to see how it feels qualitatively before committing to it. Does it actually feel natural to use classes or should I make it more functional? That sort of thing.
                        • I work in a lot of different languages so sometimes it’s nice to jump into a REPL and remind myself how various language features work and what’s available to me.
                  3. 6

                    The post from @chobeat is something I agree with in terms of the bigger picture that is being missed by a mile. Analogies like those presented are flawed and ultimately damage our industry.

                    They had been forced to learn to program computers without the luxury of endless computer time. Many years later, when he had plenty of computer time, Serge still wrote out new programs on paper before typing them into the machine …snip… Good Russian programmers, they tend to have had that one experience at some time in the past—the experience of limited access to computer time.

                    This is written like there is something special about being Russian. The truth is that many programmers learned to write code in this fashion. Distribution of programmers with this experience would vary globally and with time period. In the mid-90s, I was taught by a curmudgeon how to program with this method in the UK. I’d expect that the best programmers from the US, Asian, and Europe of different ages had a similar experience at a different time. This method is only compatible with some personality types, so it’s never going to remain the pervasive method when alternatives exist.

                    So, there is nothing special about being Russian in the general case when it comes to knowing this technique. That said, at certain times, Russian programmers may have been in demand in particular areas. For example, the wisdom at a certain time might have been that you could get “inexpensive programmers with solid mathematical foundations easily from Russia.” This is really inappropriate and biased.

                    The result however, is that you have concentrations of developers who recommend former colleagues further reinforcing the cycle. This pattern has happened with immigration in the US many times over the history. Sponge diving in Florida brought many Greeks to the US. Greeks were considered to be the best at that, but other than having experience, what makes a Greek person explicitly suited to the task? The common pattern is that there is a person that can be exploited for money, even though life in the US may be better than that in Russia, or Greece for that matter.

                    I was curious what your thoughts were about this - in this specific case (computer time) and in general (any other tooling that has made programming more accessible)? Is Serge right or wrong about this?

                    Every industry suffers from this same pattern because as soon as something is profitable it brings more people to the field. Competition is good, especially at the top level because it really drives a field forward. More and more so-called menial jobs are created as programmers move on to more prestigious roles and don’t want to do mundane work.

                    The experienced programmers realise that their work can be the basis for new software, so they create tools that empower those without experience. That’s the whole point of better tools.

                    We have some other forces at work which I won’t go into at length. We’ve got software becoming applicable in many more areas where it wasn’t feasible before. We don’t really have a surplus of programmers either. Our industry doesn’t really represent our society as a whole. Society will force the industry to change rather than the other way round.

                    1. 3

                      I do agree that the reasons given for why Russian programmers are better are probably not the actual reasons, but there is truth in the fact that a lot of Russian programmers have good theoretical knowledge. Russia has been dominating in programming competitions over the last couple of decades (14 wins in ICPC over the last 20 years) and have a strong competitive programming tradition in general. Looking into the courses of Russian universities, I can see that they place a lot more importance for theoretical topics than the western ones. I don’t think that a blanket statement “Russian programmers are better” is correct, but there might be some truth in “Russian programmers have better theoretical knowledge”.

                      1. 4

                        ICPC/Olympiad type competitions are national programmes in Russia and other post-Soviet states. Typically all urban high schools participate. School level winners are groomed for city competitions, city competition winners for local, then regional, then national, culminating in international competitions.

                        The teachers of winning students are paid a premium and the students have a chance of simplified enrolment into better universities, so incentives are in place. Hence comparing this industrialized effort to largely amateur act in the West is not entirely fair. The whole mother Russia was combed with a fine brush, repeatedly, for these 14 medals.

                        1. 1

                          I know all of this. But to win those competitions, you have to have good theoretical knowledge. So the teachers are more incentivized to tech them the theoretical knowledge. Hence, the “Russian programmers have better theoretical knowledge”. It’s not about how fair is it to compare the results of the competitions or such. It’s about how much they teach it there. And they teach it a lot more.

                          1. 3

                            There’s a huge difference in how they teach their competition potential (with region’s best teachers holding training seminars for the talent), and “ordinary” students. Hence “Russian programmers have better theoretical knowledge” is incorrect. “Russian programmers who were groomed for competition have better training” is more like it, and it’s only a subset of Russian programmers in general.

                    2. 3

                      Well, seems easy enough to test for yourself and any volunteers you can get - practice writing your code on paper and limiting yourself to some minutes to enter and compile the code.

                      If you do the experiment, let us know what you learn.

                      1. 3

                        Kind of, except I wouldn’t call it “changes in development practices” or “tooling”. I’d call it changes in teaching methodology and attitudes towards programming as a craft, both driven by the fact that 21st-century business demands way more programmers than we could have turned out the old way.

                        But yeah, when I took my first programming course we didn’t actually have access to computers for the first three or four weeks. We got a refresher on logic, we learned syntax, we were given assignments to write programs on paper, and we were taught how to “run” a program on paper by tracing through it, line by line, checking conditionals, writing the values of variables in the corner of the paper, crossing them out and writing new values when they changed, etc., and we were expected to be able to tell what value some small function would return, using this process. I’d say it developed valuable debugging skills, and a feel for “what the computer is doing” which (while less precise) is actually more useful than the gates-and-transistors view.

                        1. 2

                          Betteridge’s.

                          1. 4

                            Betteridge’s

                            Well, I am not a newspaper publisher so it isn’t clickbait, I am curious what people here think.

                            1. 9

                              Fair enough.

                              I see no evidence that programmer quality has declined. Coding 24 years professionally, and 5-6 years before that as hobbyist/student.

                              At my first job we had a few old schoolers from Soviet days. Some were great, some less so. I find that it still holds the same.