1. 6

    Does anyone know of cases where a big language with a big compiler code base goes through the kind of renewal that Stephen implies here? It seems like the way that usually happens is not incrementally but a step function when someone creates a new language derived from the old one.

    1. 4

      Maybe C++ going from C++98 to C++11?

      It’s not quite the same, though, because C++ has multiple compilers and a larger user base.

      1. 3

        a step function when someone creates a new language derived from the old one.

        Yes, historically that has been the case.

      1. 9

        This is a nice enumeration of language features and it hints at the larger truth. The real civilizational value of Haskell is being quite literally the only language laboratory in which genuinely novel ideas about programming can be tested, refined and deployed in the wild. Haskell will absolutely never have many users, but it has enough users to vet the good ideas and the bad ideas from both an academic and industrial perspective. The really good ones work their way downstream and its contributions enrich the entire software ecosystem.

        I’ve come to really appreciate the language extension model that Simon put in place decades ago, and its place as a great economic engine of progress in software.

        1. 7

          Haskell is being quite literally the only language laboratory in which genuinely novel ideas about programming can be tested, refined and deployed in the wild

          Nothing against Haskell, but it’s just not true. Most truly novel ideas are first tried in research languages/dialects, and far from all of them make it, or even can make it into Haskell.

          Haskell also doen’t have a decent module system after decades. ;)

          1. 7

            Nothing against Haskell, but it’s just not true. Most truly novel ideas are first tried in research languages/dialects, and far from all of them make it, or even can make it into Haskell.

            You’re missing the point:

            in which genuinely novel ideas about programming can be tested, refined and deployed in the wild

            No one is asserting that all interesting PLT ideas first show up Haskell exclusively, sdiehl’s point is that the extension system and culture allows for testing out more experimental ideas along with solid production-tested features in an existing language used in industry in a way that is unique in the current ecosystem.

            And stating that Haskell doesn’t have a decent module system glosses over the fact that typeclasses fulfill a lot of the same function in Haskell. And that’s not to mention that Backpack exists–it’s not like members of the Haskell community are unaware of the existence of modules or don’t see their value, but Haskell provides a different model with different tradeoffs.

            1. 5

              Yes precisely, there’s plenty of interesting work that is done in *ML dialects too. It’s just that in Haskell things get tossed into the language and out to industrial users a lot faster because it’s all opt-in extensions. There are pros and cons to this model but it is very much the Haskell Way and quite unique in the space of all languages.

          2. 7

            Haskell […] being quite literally the only language laboratory in which genuinely novel ideas

            That’s a bit too absolute, I think. There are many other languages that are exploring genuinely novel ideas, and I would argue that Lisps, in particular Racket, still make better “laboratories” because of the ease with which you can explore new semantics. Haskell is has done a great job at opened up a specific space of new possibilities in type systems and their relationship with category theory while remaining a very practical and high-performance programming language at the same time, and that is excellent, but there are still completely different directions worth exploring as well.

          1. 4

            Haven’t heard anything about the buttcoin circus in ages, I honestly kinda forgot about it. It’s kinda surreal that lots of people work with cryptocurrency as their day job, while I forget that it even exists.

            ERC-777 is an updated version of the ERC-20 standard

            LOL, did they just put a… casino-sounding number instead of a sequential one on a new spec?

            It’s an incorrect function name (optionIDs instead of optionsIDs). This function unlocks liquidity in expired contracts. If it doesn’t work, funds are just forever locked

            Wait, what? Does the compiler just proceed full steam ahead if you call a nonexistent function?? Is this PHP?

            Nah, that would be actually nuts, their official tweet is a bit wrong. Actually they used options.length instead of optionIDs.length while they had options defined in outer scope. Well, still, I’m not the only one who thought of PHP :D

            Why are people iterating over lists with indexes?! Is there no for..of / iterators type thing in Solidity?!?!

            Also now that I think about it, this reminds me even more of C++, where this-> is optional, so you can easily mistype a class member name instead of a similar sounding local variable of the same type.

            1. 8

              Putting millions of (pseudo?) dollars under the control of immutable code written in what is more or less JavaScript is utterly insane. There seems to be a large supply of people to whom that is not immediately obvious.

              1. 3

                Everyone who’s on the JS-hate-bandwagon latched on to this JS comparison, but it’s completely unfair.

                “Solidity was influenced by C++, Python and JavaScript”. All three have proper iteration: for (auto x : xs), for x in xs, for (let x of xs). The Solidity examples have stuff like for (uint p = 0; p < proposals.length; p++), which makes it most inspired by… well known super safe language C, haha.

                1. 8

                  I’m told (by early Ethereum guys I can’t name) that they actually wanted to use just JavaScript with added built-ins, but couldn’t make it work. So Solidity is definitely a JavaScript descendant, and more from JS than anything else. And they were absolutely targeting middling JavaScript coders with Solidity.

                  Ethereum and Solidity are IMO excellent examples of “worse is better” - if you’re going to do something as self-hobbling as smart contracts, then you’re going to want provable languages, preferably not Turing-complete, preferably functional.

                  But middling programmers don’t cope well with that stuff. So, give ’em something easy to use, and add free money as an incentive! And it totally worked … if inelegantly.

                  There are better languages for the Ethereum Virtual Machine - but Solidity is still the overwhelming favourite, because its popularity means you can apply standard state of the art software engineering principles to it: that is, cut’n’paste’n’bodge.

                  1. 4

                    As I occasionally say, you can tell the difference between a salesperson and an engineer this way: the salesperson solves the easiest problem first; the engineer solves the hardest problem first.

                  2. 4

                    Substitute C++ or Python in my comment and it still stands. I actually can’t think of any language I’d be comfortable doing this in, but it would at least have a theorem prover.

                    1. 4

                      Funnily enough, Solidity can use Why3 for verification. In 2017 the translation was “not even tested”, no idea what progress has been made.

                      Either way, the human problem is still the bigger one. You can’t force everyone to verify. Especially if currently some don’t even test.

                      1. 13

                        Some serious PL people looked at this problem years ago but the Solidity language has no defined semantics and is very unprincipled as a language. Basically all these formal verification tools are some 5% finished work that some grad student who got tasked to “do formal verification” purely as a means to pump the value of some token, and then abandoned it when they realised the problem was intractable.

                        From a larger perspective, effectively a 100% of these contracts and DeFi companies are either just gambling products or thinly veiled exit-scams so whether the code works or not isn’t really important so long as the right stakeholders get their winnings out before it blows up. David’s conclusion in the article is spot on.

                2. 1

                  Actually they used options.length instead of optionIDs.length while they had options defined in outer scope.

                  ah, thank you! I didn’t dive into the github to see if they’d described their “own” code correctly. Unsurprising, given they apparently just copied a pile of it. Correcting …

                1. 8

                  This won’t come as a surprise to anyone working in enterprise software. There is an absolutely enormous cottage industry of consultancies selling these “blockchain” solutions to large companies when all they needed was a simple database. It’s metastasised so much that we’re seeing these second order solutions just trying to solve the original prviacy problems induced by the first order solutions. This is really the last ditch effort of IBM trying to maintain some semblance of relevance in the modern software era by creating completely unmaintainable back office software which is a loss leader for their Bluemix Cloud solution and which they can parasite off of for consulting the next decade.

                  1. 1

                    I was actually surprised not to see IBM’s name on this one.

                    yeah, I fully expect there will be systems that sort of work better than not existing at all, with a gratuitous Hyperledger or private Ethereum somewhere in there that doesn’t actually do anything, and people like me will maintain these awful things for decades longer than is in any way reasonable. I think the ASX Blockchain project will end up one of these, for example.

                    1. 3

                      Microsoft and Ernst & Young are just playing the same game on top of this bizarre Ethereum thing they built to sell Azure products. It’s the same cargo-cultism masquerading as innovation to produce press releases. It’s all just pumped up by the public blockchain press because “any blockchain success makes number go up” and so the cycle of suffering continues for the poor programmers left to maintain this mess.

                  1. 31

                    A rebuttal, though from an OpenLibra contributor rather than Facebook.

                    1. 9

                      I suspect Facebook’s longer term plans, were they to actually launch Libra, are to move to a permissionless proof-of-stake model […]

                      some of the testnets we are running for other BFT protocols are operating at those performance levels or above

                      Is it actually sound? I don’t know, it’s brand new and looks unfinished to me […]

                      Right now Libra is missing many features of other proof-of-stake blockchains like governance, however these mechanisms can be used to implement things like payment reversals […]

                      Nothing in the rebuttal other than the cryptography section sounds particularly strong.

                      1. 8

                        The cryptography section isn’t wrong but it misses my point entirely. I claim that the more mission critical the cryptography is in a piece of software the more scrutiny it should be given and that these libraries are relatively new and not as tested as older ones. A company with resources like Facebook could have put a lot more resources into this and it would have been good for the Rust community for this to happen as it makes the whole ecosystem more robust.

                        When you’re going before congress to testify that this system can safely handle private user data, the QA should be much higher on your software because the public’s interest is involved. I thought this was a fairly uncontroversial opinion.

                      2. 8

                        the original: “these ideas are bad and incoherent, and this code cannot possibly implement reality consistently”

                        the riposte: “but look how shiny the pieces are! also, it has to suck cos it’s a blockchain,”

                        that’s really not a good rebuttal

                        1. 5

                          I agree, it isn’t a rebuttal of my original points. It’s more a statement of blind faith that the technology just needs to be invented and that the public should just trust that currently intractable problems will just be somehow solved before it goes live. I don’t have any faith in blockchain tech, I only look at what exists and is provable today.

                        2. 4

                          Thanks for finding this, always interesting to read a rebuttal on a similar level.

                          For context, I am a contributor to OpenLibra, a group independent of Facebook (or should I say “FACEBOOK”)

                          I struggle to understand the joke here, unless it’s a poke at cryptocurrency boosters to refer to conventional money as “FIAT” (in all caps).

                          1. 18

                            Probably related to the rebrand.

                            1. 7

                              Ah, I’ve missed that. Thanks. That is a fugly logo.

                        1. 4

                          I would love to see this book completed. I think it’d be a great service to the Haskell community.

                          1. 6

                            Having spent the last 2 years starting a (Haskell) company, my time for open source is not abundant. If things go well I’ll certainly pick it up again though.

                            1. 3

                              I’m with him on this, I’ve just read the first 4 or 5 chapters, but I’m planning to eventually sit down and follow it through, implementing everything. If there is anything we can do (as a community I mean) to help with it, say so! At the very least I can help reporting typos :]

                              1. 2

                                That’s completely understandable. I hope the company’s doing well.

                                1. 2

                                  No rush, we can wait :)

                                  I actually binge-read/reproduced write you a haskell for the last three days.

                                  I really enjoyed it, thanks for writing it!

                                  1. 2

                                    Wonderful, makes me happy to hear people get some value out of the first manuscript.

                                    1. 1

                                      Yep, its quite cool, I learnt a lot. Just being selfish in saying it would be great to have it completed. But I also understand you starting a business and having commitments. So no worries, thanks for all the fish so far!

                              1. 4

                                This is an amazing piece of engineering work. The language integrated ability to discharge proofs of the integrity of cryptographic protocols to SMT solvers is way ahead of its time.

                                1. 3

                                  Here’s an example of how such a tool is integrated in an overall flow for high-assurance security:

                                  http://www.ccs.neu.edu/home/pete/acl206/slides/hardin.pdf

                                  I chose this one since it takes it down to CPU microcode using AAMP7G: a CPU verified mathematically to be correct and enforce separation a la seL4. The other end at Rockwell is model-driven development with Simulink or Stateflow converted to provers for correctness, then SPARK Ada for correctness, and then AAMP7G for correct execution. Correctness is from requirements in human-inspected model down to microcode.

                                1. 4

                                  There’s only one sign that you’re in a feature factory. If the the pursuit of number of features outweighs the design of the software itself.

                                  If you have an environment where business people who will come to you with the set-theoretic union of every possible end-user item possible, and then turn that into a timeline you’ll end up with an incoherent wreck whose only ‘feature’ is being rewritten every few years and cycling through engineers who end up breaking out of the cycle of suffering and quiting.

                                  1. 4

                                    There’s a much more interesting story about having a global network of machines with the ability to deploy executable code whose inputs and outputs are kept in sync via a consensus protocol. That’s really the interesting idea here, detached from the cryptocurrency world.

                                    Unfortunately there are deep foundational problems with this ecosystem, and it’s mostly due to the fact the code being deployed on the network is brittle and not amenable to analysis. Nobody really can answer simple questions about state transitions in these contracts or whether they end up in a broken state with funds misrouted or drained. Until there’s a stronger claim than “trust me, this contract works as I claim it works” it’s really kind of the wild-west.

                                    1. 4

                                      This transaction model based on Javascript string comparison is completely and utterly insane. If you use this, you deserve whatever happens to your data.

                                      1. 7

                                        Looking for a “Haskell job” is the wrong mindset. Look for problems that are amenable to Haskell’s strengths and then find work on those problems with people who are open to using solutions that demonstrably exceed the projects needs.

                                        The problem I’ve personally had hiring Haskellers is that there are too many hobbyists who try to turn work projects into experiments in the latest type-level programming technique, and end up burning through massive amounts of time and capital trying to solve problems that don’t really add value to the business.

                                        1. 33

                                          No, you really shouldn’t target Rust unless you happen to be writing a language that is nearly semantically identical to Rust. Consider for instance how you’d bolt a non-standard calling convention (register pinning), like LLVM supports, onto Rust. You’re basically locked into the high-level semantics that Rust choose which are basically call/ret with it’s specific stack setup. LLVM is a much more malleable medium to target for languages of many different sorts because it makes much fewer assumptions, that’s why it’s such a great target.

                                          1. 25

                                            This is extremely true, but also a specific case of a more general problem that people often seem to overlook: to compile language X to intermediate representation Y, Y must be able to efficiently implement all the concepts in X. LLVM IR keeps getting more features over time, in part to represent concepts that exist in certain source languages but cannot be efficiently (or at all) represented in LLVM IR without changes.

                                            Languages don’t generally make good general-purpose IRs unless they’re designed for that (and get beaten on for years until they have the features people need). Sometimes people make new IRs for their own purposes because LLVM’s really isn’t a good mapping to what they need – but that new IR is likely only good for that purpose, because that’s what it was made to do. It’s not “better” than LLVM – just (usually) more specialized and different.

                                            (The inverse is true too: LLVM IR has a lot of implicit assumptions, e.g. in its memory model, arithmetic system, and so on, that roughly match a C-like language model. And if you don’t want those…. it can be hard to work around them!)

                                          1. 21

                                            Highly-skilled people that I respect just got really angry and irrational and basically refused to even try to learn Haskell, which resulted in a culture divide in the company.

                                            This is kind of a depressing story, by every account this team did everything right. They rationally evaluated their options, ended up building solid reliable software that added value to the company, and did internal training for the new tools to ensure it could be maintained. I really question the judgement of the management that divided the company over what was seemingly a solid solution, and I don’t blame the OP for leaving.

                                            Why bother calling our profession “software engineering” if weird superstitions and “feelings” about languages trump empiricism and measurement.

                                            1. 15

                                              Highly-skilled people that I respect just got really angry and irrational and basically refused to even try to learn Haskell, which resulted in a culture divide in the company.

                                              I’ve generally found most tech choices are driven by pure emotion.

                                              1. 10

                                                I’ve found a vanishingly small number of technical decisions derived from some rational set of principles in the ~25 years that I’ve been doing this work for money.

                                              2. 9

                                                One thing that I’ve found, unfortunately, with high-productivity languages is that there’s a sort of political Amdahl’s Law. Instead of it being the serial component of a program that holds up parallelism, the political component of your workflow can dominate your time once you make the coding aspect efficient. The negative side effect of making the work quicker, in other words, is that a higher proportion of your time is spent on political bullshit, such as justifying work. If the people you work for aren’t capable of valuing the technical excellence that such languages facilitate, it can just make your life a lot worse.

                                                The truth is that improving efficiency isn’t always a good thing. You have to be able to trust your employer. A good employer will give you more autonomy and let you take on more ambitious technical projects. An evil employer will say, “This is great, now I need three fewer of you.”

                                                Now that I’m more of a manager/architect who also codes, I’m less zealous about languages. If the other programmers want to use Haskell (which I also prefer) then it’s my job to make that possible. If they want to use Python, that’s more than fine with me. I’ll write Python, then. Even Java is OK if they have a damn good reason.

                                                1. 2

                                                  This is a good point. The only languages I won’t write are C++ and PHP.

                                                2. 9

                                                  People build their professional identities upon their tools and platforms, which is in all honesty an absolutely stupid way to approach things (but understandable, given the investment involved). So when you propose switching out your stack or your programming language, it’s perceived as a personal attack on the engineer’s competence or professional self-worth. (I am aware of such an actual situation at a certain large corporation involving certain developers and certain front-end stacks.)

                                                  1. 6

                                                    People build their professional identities upon their tools and platforms, which is in all honesty an absolutely stupid way to approach things (but understandable, given the investment involved).

                                                    This is absolutely true, and you’re utterly right to call it stupid.

                                                    I think that the short-termist and ageist culture of corporate programming is largely to blame. People no longer identify as computer scientists or problem solvers, but as X programmers. It has become this high-stakes game of choosing tools (a) where one can contribute to a corporate codebase right away, make a quick impression, and get pulled on to a leadership track in the first 3 months as opposed to never, and (b) that seem, at the time, to have a long-term future in the corporate world.

                                                    When companies invested in people, and when you didn’t need to make a strong, “10x”, impression in your first 3 months to have a career at a place, there was less of a need to brand oneself based on tooling choices or based on silly silos like “data science” (which is mostly watered-down machine learning).

                                                    I’m within a few days of turning 33, which is ancient by corporate programming standards, and I’ve worked with enough different tool sets to get a sense of the recurring themes. I find it a lot more useful and rewarding to think of myself as a computer scientist and problem-solver who can pick up any tool than as an XYZ developer. That, however, seems to be a luxury that comes with professional standing, insofar as I’d no longer take (and, because of my “advanced” age, probably not even be able to get, even if I needed a throwaway job to fill an income gap) the kind of job where I have to justify work in 2-week increments. If I was forced to play that horrid game, I’d do the same thing as everyone else and invest more energy into the tool-selection process than the work itself.

                                                    1. 2

                                                      I’m curious about the halcyon days of corporate culture you are referring to. When was that? For example, “The Soul of a New Machine” - a book about a team developing a minicomputer more than 35 years ago - shows a picture of the industry that’s remarkably similar to what we see today. My impression is that corporate culture hasn’t really changed in at least the last 40 years.

                                                      1. 1

                                                        The worst employers probably were the same now as then. The distributions are different, with a lot more bad examples and very few good ones.

                                                        People always complained about short-term outlooks and meddling management, but it used to be that they complained about 5-year focus as opposed to the next-quarter mentality, and that management could meddle but it was limited compared to now. You didn’t have to stick with a bad employer, in tech, back then. You still don’t, now, but the odds are that if you roll the dice, it’s not going to be better… and you’re going to have one more job hop to explain in the future.

                                                  2. 6

                                                    I worked at IMVU a little before this. It was a good team and definitely some strong personalities. Large swaths of the PHP code at that point were… well “the horror” is a good way to describe it. I too am curious what would have happened with Java.

                                                  1. 3

                                                    Not many people argue that society shouldn’t allocate a certain amount of capital to invest in research on the 10-100 year horizon. Nation states are the only entities that really can take that risk and history has shown that it pays off. The argument about the benefits of long-term fundamental research is not really something most rational people would debate, the only thing we do debate is how much capital nation states should allocate and who they should allocate it to.

                                                    Over the years, I’ve seen people evaluate research by how closely the paper translates into a startup idea. I’ve seen people evaluate research by how easily the work can get media attention.

                                                    This is where it argument looses people like me, who in industry, are constantly forced to justify our work to investors and shareholders because that’s how our system works. If you can’t justify your work to the funding agencies then you have to adjust your ideas to be more relevant to people who will allocate capital (like industry), that’s not ideal, but that’s life in a society based on capitalism.

                                                    In an ideal post-scarcity world, we’d be able to give every person enough resources to go off and build projects that benefit society on their own terms. But we don’t live in that world and that means tactical allocations of resources to people who can justify returns on investment, be they societal or economic.

                                                    1. 6

                                                      I don’t know if it’s fair claim this is due to capitalism. I’m not sure decision makers in a communist system would be any more likely to allocate funding for crazy, out in left field ideas with no clear real world application.

                                                      There’s a never ending supply of ideas that will never result in any practical real world benefit to anybody, or will never go anywhere no matter how much time and money get thrown at them.

                                                      People are free to do all the research they want, but if they’re depending on other people’s resources they’ll always have to justify why the resources should be spent on that project instead of another one.

                                                    1. 9

                                                      The article makes the big leap between “shippable products” and the much simpler goal of just having code that is available and compileable. Shippable product by industry standard means at the very minimum the code is on some public VCS, permissively licensed, documented, has a public bug tracker and is tested on major platforms. That’s not what’s expected of researchers.

                                                      For code distributed for research, it is very common to not even have it a) public b) a list of directions on how to build it, what’s needed is not a well-supported product; just the minimum viable effort to reproduce it. And in this day and age of free code hosting, free continuous integration and abundance of open source if academics aren’t giving minimal reproducible artifacts I have little sympathy for when their work gets judged as impractical and tossed out.

                                                      As I’ve moved away from academic work, I’m quite bitter about how many academics are readily willing to complain about the lack of industry uptake but are not willing to spend 30 minutes to learn Git or make a short list of install directions for a README file. There’s this vast asymmetry in the time it takes to reproduce bulids, 10 minutes on your end can literally save 10,000 community hours.

                                                      1. 4

                                                        Polyvariadic functions are pretty ugly without language support. The next “big” functional programming should have a better story for named arguments because while curried positional arguments are suitable for a large class of lawful functions, they’re really painful for dealing with anything unix functions or filesystem functions that are mostly large collections of variadic optional named arguments. Having first class records in the language would pretty much solve most of this.

                                                        1. 1

                                                          Do you have an example of what you mean by first class records?

                                                          1. 2

                                                            I stumbled upon this: https://nikita-volkov.github.io/record/

                                                            I assume by first class you mean records as a type level construct.

                                                            1. 1

                                                              Do you mean stuff like OCaml’s polymorphic variants? Polymorphic records would be nice.

                                                              EDIT: Actually, I suppose OCaml’s objects and row polymorphism are a kind of heavyweight first class records.

                                                              1. 1

                                                                I don’t think he means first class records. Or maybe he does. A record is something that supports named fields with no namespace clashing.

                                                                What I think he is specifically talking about, though, is SML style, where you pass a record into a function to get names associated to arguments.

                                                                makePerson { name=“Bill”; age=21 }

                                                                Would be an example usage. By passing a data structure with named fields you get names attached to arguments, gratis.

                                                            1. 9

                                                              This is a really great essay on the whole, but I have to call the OP out for this:

                                                              Haskell makes abstraction so cheap that people get out of hand. I know that I’m guilty of this.

                                                              A typical progression of a Haskell programmer goes something like this:

                                                              Week 1: “How r monad formed?” Week 2: “What is the difference between Applicative+Category and Arrow?” Week 3: “I’m beginning a PhD in polymorphic type family recursion constraints” Week 4: “I created my own language because Haskell’s not powerful enough”

                                                              That’s ridiculous. It’s statements like this that create the impression that Haskell is only for super geniuses. They’re also painful to read for the “silent majority” of smart programmers who have impostor syndrome. (Unfortunately, our industry tends, both in engineering and management, to have the Russellian problem of the best people being full of doubt and the worst being full of certainty.) It is not normal, nor should it be expected, to understand Applicative, Category, and Arrow in any meaningful way after two weeks. One should certainly not give up if it takes a while to understand these concepts and why they are important. Frequently, learning them requires approaching them from multiple different angles, and even the best tutorials and exercises seem to be single-approach-based, so it often requires finding your own resources and it’s very individual what works and what doesn’t.

                                                              At risk of sounding like a douchebag, Haskell was hard to learn even for me. (It’s probably easier now, because there are better resources out there.) Was it utterly worth it? Yes, absolutely, hundreds of times over. Did I “get” monads immediately? No. It took a lot of reading and work and playing with the language. I will give Haskell this, without reservation, though: the effort is worth it. That’s not true of a lot of the other stuff that people have to absorb to keep afloat in this industry.

                                                              1. 3

                                                                In defense of Gabriel I think the timeline he mentioned was just rhetorical, and the the gist of what he was saying was orthogonal: that it’s easy to go overboard on abstraction and that Haskell encourages this. Not to literally indicate that everyone should understand Category typeclass in “Week 3”.

                                                              1. 3

                                                                Basically it’s impossible to get anything into the browsers that doesn’t serve the economic interests of the browser vendors. It’s always going to be more cost effective to just route around the problems using the existing technologies.

                                                                1. 26

                                                                  At some point just before the turn of this century, the culture in Silicon Valley shifted from being what it was (a quirky, pay-it-forward culture) toward a resource-extraction culture. Most resource extraction cultures emerged around physical commodities (e.g. oil in the Gulf States) but this one taps a different resource: the earnestness of the American middle class. People like the OP go into the tech industry (and often move to California) because they’ve been told that “it’s different” over there– that hard work is rewarded, that smart people are venerated rather than marginalized, and that these tech companies have something different from the manage-or-be-managed culture of the typical corporate grind. They read Paul Graham, failing to realize that the man hasn’t accomplished much in the past 20 years other than to monetize the reputation that comes from a mid-90s success and a couple good Lisp books, and drink enough Kool-Aid to last through three or four jobs before they realize that they’ve been sold a lie.

                                                                  Then they hit middle age, and realize that the managers have all become executives, making high 6- and 7-figure salaries for jobs that involve no real work and minimal accountability, and they’ve all become… aging ticket jockeys, thanks to “Agile” and open-plan offices and the low-status, macho-subordinate culture that has infested programming. Sure, they know a lot about how to design software after 10, 20, or 30 years, but that doesn’t matter in a world where, if you didn’t get the title, you didn’t do it… and management titles are valued most of all because the people who make those decisions… (wait for it…) are managers.

                                                                  It’s hard to say what “the solution” is. The indicated solution is to push a bunch of highly intelligent people to compete for management jobs that they don’t really want. (I don’t think that the OP wanted to be a manager. He just wanted the credibility that nobody ever told him that only managers get.) That might be better for the individual, but it’s bad for society. We lose engineering talent, we increase political competition for the good jobs, and we move further toward being a second-rate society governed by bullshitting rainmakers rather than real people doing real work. In which event, China will actually kick the shit out of us and Donald Trump, Jr. will win the 2024 presidential election in a landslide.

                                                                  In that light, I think it’s incumbent on us as a generation to decide what we want software engineering to be. If we want it to be a stupid young man’s game [1] that people play for 5 years in exchange for lottery tickets called startup options, and to have a world in which skill is devalued and mediocrity in products is the norm, we can continue with the current path. It’s clearly making a lot of money for some people. On the other hand, if we want to be taken seriously as a profession, we have to get organized, we have to fight this nonsense (“Agile” and open-plan offices and the halfway-house culture for kids who are emotionally still attached to college life) and get serious about being treated like adults. I think the best model might be to institute something like the actuarial exams, because while I absolutely hate the idea of making software require formal education in the form of expensive institutional degrees, I do think we need to differentiate ourselves from the unqualified 19-year-olds who “write code” from their mothers' basements and the long-ago-checked-out perma-juniors (i.e. the people for whom Agile/Scrum is intended).

                                                                  [1] Note that both parses work. {Stupid young man}'s game and Stupid {young man's game}.

                                                                  1. 15

                                                                    I once had the opportunity to review a manager’s resume. Full of “built this, built that.” I thought, “cool, guy did some stuff but he’s grown tired of life in the trenches.” But then it turns out he was just the manager at the time. Didn’t actually build anything. Only watched people build things. I thought this was pretty deceptive, but then my manager explained that’s how all manager resumes look. Never really explain what they did, only focused on what the team did. It was disappointing.

                                                                    1. 4

                                                                      What you’re saying is pretty much the observation most people make after working in certain parts of the industry few years. There’s a lot of systemic corruption.

                                                                      You go out to the west coast and you see these companies packing engineers like battery chickens into open office plans, and offering them like 0.01 equity in some risky venture that will just toss them out like waste a week before their option cliff and replace them with another naive 20-something who will do the same and the cycle continues. It’s no wonder all the valley companies prefer younger people, they’re much easier to exploit. The Silicon Valley scene is really corrupt, and most people only realize this after they’ve been in the machine for a few years.

                                                                      If we’re going to move forward as a profession, software engineering really needs to become a viable career path that you can feel safe devoting 10-30 years of your life to without being marginalized by this kind of corruption.

                                                                      1. 2

                                                                        Most of what you write on this topic is spot on. How can I join the revolution?

                                                                        1. 7

                                                                          I’m writing a blog post (yeah, I decided to get back into that game, although with reduced time expenditure) on it right now. ETA: link to said post here.

                                                                          1. 0

                                                                            I’m very happy to hear you’re blogging again :)

                                                                        2. 1

                                                                          I agree with most of what you write, but you should leave out the nation-state stuff. China is no less, and in many cases quite a bit more, governed in both large and small by bullshitting rainmakers. Their engineering practices are no more evolved. Their technical people are, in fact, on the same side of the same boat as developers in the US.

                                                                          1. 2

                                                                            What you say is true of Asian corporate/management culture. That illness is worldwide. That said, it wouldn’t surprise if China (or, perhaps, Hong Kong or Taiwan or Singapore) managed to shuck this off faster than we do, if only because they’re hungrier and more willing to change.

                                                                            It is correct that, as of 2016, the U.S. is a better place to be a software engineer than China. I don’t see that as a static fact, necessarily.

                                                                        1. 3

                                                                          The article is a bit hyperbolic, but It’s hard to work in tech and not notice a really deep form of systemic corruption in the startup space. A lot of people have written about this topic, but it’s worth stating that a lot of current trends in building companies really exploit young programmers and leave them with very little to show for many lost years. It’s sad because it doesn’t have to be this way.