1. 50
  1. 10

    I really loved the spirit of this article, particularly this bit from the conclusion:

    No matter how much data you have you still have to ask the right questions. It’s painfully easy to have good intentions but ask the wrong question and find the wrong answer.

    And I especially loved the example given in the bonus section. I don’t think I would’ve ever thought that geography (or rather, download speed) was the cause of the increase in average page load time. While it’s such a simple answer, thinking about how to arrive there (or: find the right question) just feels so far out there to me. It reminds me of the Ultimate Answer/Question from The Hitchhiker’s Guide to the Galaxy. Deep Thought is able to arrive at the Answer (“42”), yet a much more powerful computer must be designed and run for even longer in order to find the Ultimate Question.

    1. 4

      Pearl’s critiques of Simpson’s Paradox from the point of view of causal reasoning are excellent.

      The core idea is that Simpson’s Paradox exists because without information pertaining to the causal structure you’re interrogating (a.k.a. assumptions which underline the external validity of your analysis) then there’s no way to distinguish between the two valid statistical interpretations. There are too many degrees of freedom in purely statistical reasoning. Therefore, we need causal reasoning to resolve the issue.

      https://ftp.cs.ucla.edu/pub/stat_ser/r414.pdf http://dagitty.net/learn/simpson/

      1. 1

        That’s a good point. Another possible critique is that the “simple” model that compared men and women admission rates was misspecified because it didn’t take into consideration the clustered nature of the data (applicants within departments). The same point also stands for the kidney stone example. Not modeling the clusters leads to biased estimates.

        Another possible critique is that there is an implicit causal claim in the original admission study (women are admitted at a lower rate because they are women). However, the claim was made using non-experimental data. In econometric-speak, the observation is probably the result of an endogenous factor (such as department preference of applicants). Results from most observational studies are difficult to interpret casually for this reason.

        1. 1

          The gendered applications example was heavily scrutinized during the lawsuit and the ultimate statistical analyses which prevailed all did so atop a causal analysis. This is of course critical from a legal perspective where statistical proof is insufficient to demonstrate court evidence and you need causal claims to be made anyway, but it’s also interesting that it’s the tool people turned to in order to bombproof their analysis anyway.

          Pearl goes into this exhaustively in his book “The Book of Why”.

      2. 2

        If you’re interested in things like this, you may want to check out the excellently-written Statistics Done Wrong by Alex Reinhart.