1. 14

  2. 4

    The simple answer is that finance is hard. This article does a great job explaining why though.

    1. 2

      I’d put it somewhat differently. Machine learning, in finance, works. It just doesn’t work well enough, compared to explicit trading strategies, to be profitable enough to justify the risks, in most cases.

      If you try to attack image recognition with feature engineering and logistic regression, you’re going to have worse results than if you use a convolutional neural network (which comes with helpful biases in terms of what features it values). With millions of input dimensions, all related to each other in hard-to-define ways (e.g., edges, contrasts) you need something “better” than standard statistical modeling. On its own, the fact that pixel (173, 29) is #d43759 doesn’t mean anything, and that’s why you need the CNN.

      In finance, it’s relatively common that a gradient-boosting algorithm or neural net does better than standard techniques, but it rarely does enough better to justify the increased complexity and (in HFT) runtime. Often the difference is between 0.793 and 0.791 AUC– possibly statistically significant (if you have enough data) but not practically significant.