1. 9
  1.  

  2. 5

    In general, I’m actually very anti-“data science”, at least in terms of what it means in the corporate world. I think that it has an atrocious culture in addition to being mostly advertisement. For the most part, companies advertise themselves as needing more AI/ML firepower than they actually do, and this seems to please investors and clients but it makes everyone on the ground unhappy. The AI experts and mathematicians end up landing in jobs that are, in terms of intellectual rigor, 3 levels below what they were trained to do.

    I feel like “data science” has also largely emerged as a way of keeping smart people in professional programming, but that creates more problems than it solves. Let’s be honest. No one of high intelligence, whether a mathematician or a research-grade engineer, is going to put up with what “Agile” means in most organizations: extreme short-term focus, lots of micromanagement, no professional autonomy or interest in career growth. To a major degree, “data science” is something that Corporate America came up with in order to attract people who are smarter than “regular engineers”. The problem is that, instead of fixing engineering– because the fact is that engineering and data science are dramatically different skill sets and both can require high intelligence– it turns it into a ghetto. As for data science and the higher degree of autonomy and respect, I feel like that’s similar to what “business class” is on many airplanes. In the same way that “business class” is often what economy used to be, before air travel became shitty, “data science” is the R&D job that programming was before the business people decided to replace the highly-paid experts with cut-rate rent-a-coders straight out of boot camps. And so organizations feel more comfortable treating “regular” software engineers like garbage.

    Data science also has a terrible culture in my experience. Most academics, in my experience, don’t consider themselves intellectually superior. They recognize that they made certain decisions and accepted certain trade-offs and that others played life differently. But you have a lot of industry data scientists who, because they’re bitter ex-academics who feel a need to retrench, refuse to believe that someone without a PhD (or even without a PhD at a school comparable to theirs in the rankings) can be as smart as they are. They tend to hog whatever interesting work there is, and to argue (contrary to fact) that only ex-academics with PhDs could possibly do the work they do, and they become another force pushing down on software engineering… which tends to be both less credentialed in the aggregate and remarkably easy to push down upon.

    There’s use for machine learning, for sure. It’s a fascinating field. Moreover, applied data analysis can do a lot of good for the world, even when applied to relatively mundane business problems. However, I tend to think that the corporate animal called “data science”, which seems to be 10% watered-down machine learning and 90% line-of-business data analysis, has done a lot of harm. Most of these companies have no real interest in supporting genuine R&D and the pretense ends up hurting a lot of people.

    1. 3

      I love your posts. I usually agree with like 60-70%, but I always enjoy reading.

      Businesses used to do all their reporting with the office spreadsheet wizard. Some web businesses have a SQL wizard to do all their reporting when every website was a simple sql database + perl/php. These days running reports is more likely to involve a little bit of python and some json. It may even involve running a query in hive. That’s the sort of work most of the data scientists I know do.

      And here’s the thing. It’s incredibly valuable and (to some people) interesting. I think we need to figure out how to have room for these people, while still having some way to broadcast that there are differences between someone who wrote the linux kernel and someone who prepared marketing’s monthly report. I don’t care if we call them an engineer or not.

      1. 4

        It’s incredibly valuable and (to some people) interesting.

        I don’t disagree that it can be valuable work. The problem is that we aren’t treated like it is.

        I think that programmers suffer a Teacher/Executive Problem (“TEP”). We need teachers. We don’t really need corporate executives. (We need entrepreneurs, but corporate executives are people who use political skills to win control over existing businesses.) We need teachers so much that we need a lot of them. In fact, as a society we pay more (in total) for teachers than we do for corporate executives, but the latter make a lot more money (and get more prestige) because the denominator is so small. For the sake of argument, let’s say that we pay our 3.5 million teachers a total sum of $210 billion and our 10,000 big-company executives a sum of $20 billion. That’s $60,000 per year for each teacher and $2 million each for corporate executives that we don’t really need.

        A TEP means that the high demand that exists for something can actually reduce payoff for the people who can deliver it. That’s what we’re facing. Businesses do need us and they know that, and they’re miles ahead of us in terms of organizing to drive down our leverage.

        So, maybe we need (gasp!) to organize. Nothing else that programmers have tried has worked. Silicon Valley pseudo-libertarianism works for rich sociopaths, but that path ends in a bad place for everyone else, including the engineers who build the products and the data scientists who crunch the numbers.

        1. 6

          I look at the same problem, but I call it the sanitation worker problem. The sanitation system probably adds years to my life, it’s incredibly valuable and cheap. But sanitation workers aren’t compensated based on the value they deliver to me. They are compensated based on the approximate cost of replacing them. It turns out that sanitation workers cost less than doctors despite the fact that my GP might do less for my quality of life than my county sanitation department. I think you’ve mentioned elsewhere, doctors have done a pretty good job of erecting moats around their profession.

          The executive side of the problem is related to the fact that power concentrates. Most executives are from rich families and have had opportunities that make them better executives. There’s also a positive feedback loop. They basically get to pay themselves more, so they do. People don’t like giving other people money, but they love giving it to themselves.

          Most engineering jobs are probably closer to the sanitation worker dynamics because of how we capture or fail to capture the value we generate. There are a few companies where engineers and their work are more closely tied to the money, and where culturally this is understood. I think this is one positive thing about the Big 4.

      2. 1

        Data science also has a terrible culture in my experience. Most academics, in my experience, don’t consider themselves intellectually superior. They recognize that they made certain decisions and accepted certain trade-offs and that others played life differently. But you have a lot of industry data scientists who, because they’re bitter ex-academics who feel a need to retrench, refuse to believe that someone without a PhD (or even without a PhD at a school comparable to theirs in the rankings) can be as smart as they are. They tend to hog whatever interesting work there is, and to argue (contrary to fact) that only ex-academics with PhDs could possibly do the work they do, and they become another force pushing down on software engineering… which tends to be both less credentialed in the aggregate and remarkably easy to push down upon

        Interesting so your essentially your saying that Data Science is harmful due to the demand for profit.