This is a great article and covers many of the mistakes that “data science” teams make. A big part of the problem is that corporate “data science” is often far too watered down to qualify as machine learning, there’s a (very related) bubble in the field which means that it’s taking in a lot of people who are qualified for corporate data science but not real machine learning– i.e. people who don’t understand what overfitting is and why it’s bad– and many data scientists can’t code, either. It’s probably made even worse by the fact that “data science” culture has copied some of the negatives of finance’s quant culture, so you have people who refuse to get better at coding out of the fear that it’ll have them shoehorned into a cost-center IT role.
Visibility debt and undeclared (possibly uninvited) customers are a textbook example of Why No One Can Have Nice Things in the corporate world. I can’t even count the number of times I’ve seen good projects derailed by incoherent requirements. You know how people who do business in corrupt countries have bribe funds for every pissant bureaucrat who might have the power to slow things down, and the result is that entrepreneurs get nickel and dimed? You see the same thing in the enterprise: everyone whose sign-off (officially or unofficially) is important to the project gets a feature, even if the end result is a product that does 15 things poorly and nothing well.
Not to rag on the paper for using it, because I do so too, but technical “debt” is a terrible metaphor, I’d argue. Taking on debt is a valid business decision. If you know that you can turn $100 into $115 next year, then it makes sense to take on a 10% interest loan. The problem with technical “debt” is that its interest rate is not only obscene but unpredictable. The interest rate might be 20% per month, or it might be 500% per month. Unfortunately, the terminology of technical debt legitimizes short-term thinking (by management, rarely by engineers) and low-quality products to a degree that no one should be comfortable with.
Taking on technical debt can be a valid decision, as long as there’s an intent and ability to pay it back at some point in the future, which is going to come at the cost of features for a while. I agree that the “interest rate” is unpredictable, which means one needs to be very well prepared indeed. Very few managers who decide to take it on are in a position to pay it back, and it is generally a very bad idea.
But also, the paper is using it in precisely the sense you describe - as being much riskier than is usually appreciated. :)
This is a great article and covers many of the mistakes that “data science” teams make. A big part of the problem is that corporate “data science” is often far too watered down to qualify as machine learning, there’s a (very related) bubble in the field which means that it’s taking in a lot of people who are qualified for corporate data science but not real machine learning– i.e. people who don’t understand what overfitting is and why it’s bad– and many data scientists can’t code, either. It’s probably made even worse by the fact that “data science” culture has copied some of the negatives of finance’s quant culture, so you have people who refuse to get better at coding out of the fear that it’ll have them shoehorned into a cost-center IT role.
Visibility debt and undeclared (possibly uninvited) customers are a textbook example of Why No One Can Have Nice Things in the corporate world. I can’t even count the number of times I’ve seen good projects derailed by incoherent requirements. You know how people who do business in corrupt countries have bribe funds for every pissant bureaucrat who might have the power to slow things down, and the result is that entrepreneurs get nickel and dimed? You see the same thing in the enterprise: everyone whose sign-off (officially or unofficially) is important to the project gets a feature, even if the end result is a product that does 15 things poorly and nothing well.
Not to rag on the paper for using it, because I do so too, but technical “debt” is a terrible metaphor, I’d argue. Taking on debt is a valid business decision. If you know that you can turn $100 into $115 next year, then it makes sense to take on a 10% interest loan. The problem with technical “debt” is that its interest rate is not only obscene but unpredictable. The interest rate might be 20% per month, or it might be 500% per month. Unfortunately, the terminology of technical debt legitimizes short-term thinking (by management, rarely by engineers) and low-quality products to a degree that no one should be comfortable with.
Taking on technical debt can be a valid decision, as long as there’s an intent and ability to pay it back at some point in the future, which is going to come at the cost of features for a while. I agree that the “interest rate” is unpredictable, which means one needs to be very well prepared indeed. Very few managers who decide to take it on are in a position to pay it back, and it is generally a very bad idea.
But also, the paper is using it in precisely the sense you describe - as being much riskier than is usually appreciated. :)