1. 15

  2. 6

    I think this piece does something I see a lot. It takes an argument about software that stands or falls on its own, and attaches it to a very broad theoretical claim.

    Are counterfactuals causes? In a past life, I studied such things just long enough to conclude it’s complicated and I don’t really know anything about the subject (https://plato.stanford.edu/entries/causation-counterfactual/#ConInv). Should the right way to develop software depend on such questions? I hope not!

    1. 10

      The more logic/philosophy version of the question is a bit mind-bending! It seems different specific applications might come up with different views here depending on what work you want the word “cause” to do. In law, for example, treating counterfactuals as causes is very common, maybe even the norm. There’s a concept of “but-for causation” that applies to some kinds of liability, in which to prevail, you need to show that a negative outcome would not have happened “but for” the behavior of the entity you’re suing. For example, I would not have been injured by an industrial machine if defendant had followed proper maintenance procedures, therefore the defendant’s negligence in following proper maintenance procedures is (legally) the cause of my injury.

      1. 1

        One helpful nuance is that we could define causality entirely in terms of standard physics: an event X causes an event Y precisely when all frames of reference observe X occurring before Y.

        It’s important to keep in mind that no counterfactual calculus can get off the ground without a foundational assumption of modality, and counterfactual conclusions can only be used as evidence in our world if one is a modal realist. As Wikipedia says, “possible worlds’ metaphysical status has been a subject of controversy in philosophy,”

        1. 1

          “and counterfactual conclusions can only be used as evidence in our world if one is a modal realist.”

          Modal realism is the view that non-actual worlds are as real as the actual world–that they are all physically existant (though not spacetime connected worlds). You can absolutely use counterfactual claims as evidence without being a modal realist. Perhaps you mean that modal claims are truth-evaluable and knowable? That’s “realism” in a sense (it’s not non-cognitivism), but it’s philosophical “common sense”, not the very controversial modal realism.

          1. 1

            Counterfactually, suppose that magic is real and somehow surpasses Turing’s results. Then we should be able to use magic to create a hypercomputer. Strangely, though, I can’t import this counterfactual reasoning into our world. The reason is that magic is not real in our world; this counterfactual claim cannot be used in our world. Indeed, the counterfactual claims which can be used in our world are precisely those for which our world already provides evidence.

            Generally, modal logic gives us statements like, “given some implausible context C, event X is possible.” It is typically the case that, unless X is a self-refuting event, there exists some C which makes X possible. So exploring the range of X, the space of counterfactual possibilities, is nearly worthless. Instead, we should note that C must be instantiated by whichever possible worlds host event X, and ask whether C can be satisfied in our world; this property of C is non-modal and less slippery.

        2. 1

          I studied such things long enough to conclude that it’s complicated, I don’t really know anything about the subject and those who had studied it much longer than me and was in charge of the philosophy seminars also concluded that it is complicated and they don’t really know anything about the subject. In spite of that I also had to provide some explanations to questions such as “what causes a deadlock” (see also: PDF@ systemic software debugging, Introduction chapter, subsection on ‘cause and/of panic’).

          But should the right way to develop software depend on such questions?

          Well it does, it is in the language - “if -> then -> else -> try -> except || panic”. Computing would be a whole lot simpler if we’d just leave them out – but it would also be much less useful.

          In day to day life, counterfactuals might perhaps only help your decision making if there is opportunity (non-fatal outcome) and some generality (aspects recurring, not a one-time event) to learning from mistakes “if I just hadn’t done I wouldn’t be in ” rather than directing the effort to damage control. That is the path I interpret the article as taking; overgeneralising to the point that forward chaining seem to be the best way out. Try improving your pinball skills without counterfactuals ;).

          For machine code however, there is a near guarantee of further generalities and that the very much incomplete causal model (program) you are building (programming), as part of an even grander system (making a living) will be executed in multiple “realities” (execution environments, customers) where your factuals sometimes are just incorrect and the answer is in one of the supposed counterfactuals. To get there is navigating forward and backward chaining through multiple conflicting causal models at several levels of detail and abstraction - debugging.

        3. 5

          This is a good post, but it’s somewhat missing in clarity of thought – I suspect because it tries to cast failure analysis in a framework of counterfactuals and causality.

          I prefer to instead view failure as a system that for some reason went outside the constraints placed on it. It can do this for primarily three reasons: a sub-system went out of constraints (disk full), a control mechanism was insufficient (disk space freed too slowly) or a control mechanism was missing in the first place (disk space not automatically freed.) Of course, the reasons for control mechanism failure is often failure of control mechanisms higher up in the system hierarchy.

          I’m not yet very good at describing this view, which is known as the System theoretic accident model. It also emphasises how we shouldn’t stop at a fixable cause, but rather explore the causal links as far as economy allows us, because the more leveraged fixes tend to be deeper. (As an example in my company, a simple hardware failure led us to realise we were missing an entire department in our organisation, that we would need to prevent failures like this one – and many, many others – in the future.)

          I strongly recommend the CAST handbook, which is available as a free PDF and was my intro to this way of looking at things. It presents many of the same things as this article, except in my opinion with a much more clear conceptual framework. It also goes much deeper.

          1. 3

            Hey thanks for that. Next time I’ll be sure to write a full-length monograph so I can hit every aspect of a topic.

            (OK, I’ll set aside the sarcasm. For now.)

            Look, when writing a blog, one always has to pick one thing to discuss in one post. Safety and system failure is a vast topic. Most readers, and from what I’ve seen most people in our industry, don’t know about any of it. Instead, they are shown examples of behavior in their companies. The typical company today still believes in human error (e.g. HBO Max throwing an intern under the bus last week), root cause analysis, and situational awareness. Absent external influences, people duplicate the practices they see.

            My post was written to address a single, very specific pattern of behavior I see as unhelpful.

          2. 4

            As a very general take on the relationship between counterfactual reasoning and causality (the linked article is specifically about diagnosing causes of tech failures), I found the 1999 psychology paper “When Possibility Informs Reality: Counterfactual Thinking as a Cue to Causality” interesting.

            Link is a PDF, but here’s the abstract:

            People often engage in counterfactual thinking, that is, imagining alternatives to the real world and mentally playing out the consequences. Yet the counterfactuals people tend to imagine are a small subset of those that could possibly be imagined. There is some debate as to the relation between counterfactual thinking and causal beliefs. Some researchers argue that counterfactual thinking is the key to causal judgments; current research suggests, however, that the relation is rather complex. When people think about counterfactuals, they focus on ways to prevent bad or uncommon outcomes; when people think about causes, they focus on things that covary with outcomes. Counterfactual thinking may affect causality judgments by changing beliefs about the probabilities of possible alternatives to what actually happened, thereby changing beliefs as to whether a cause and effect actually covary. The way in which counterfactual thinking affects causal attributions may have practical consequences for mental health and the legal system.

            1. 3

              Looks interesting, thank you!

              My thinking was influenced heavily by Douglas Hofstadter, who had some similar arguments in one of his books.

            2. 2

              So we are giving up our ability to run simulation with our minds now?

              “If the Earth hadn’t been habitable then we would not be here to care about our CI builds not finishing.”

              True. But you could also keep your imagination closer to the event and then they will be more relevant. Just because a search space is big doesn’t mean there are no good ways to zero in on relevant areas.

              this cloud of hypotheticals becomes a “blamestorm” looking to rain down on somebody. Defenses go up, and learning stops.

              The correct solution to this is to create a good culture of fixing problems so people are confident and high-trust. Not to stop thinking because then scary thoughts might make some people sad.

              1. 3

                I don’t think that’s what the author intends at all.

                The counterfactual leap indicates where people stop looking for causes and jump to thinking about solutions. Try to reformulate the counterfactual as a statement about future prevention

                As long as we remain clear that these counterfactuals are not the cause of the problem that already happened, but are changes to our reality that can prevent future occurrences, we can use them without inducing blamestorming

                Whether you agree with his point about using them as causes or not, it’s very clear here that the author’s not denying that counterfactuals are useful, or trying to stop you thinking of ways to improve things.

                1. 1

                  Suppose I brainstorm a solution to a problem, and I imagine three solutions. The first solution uses Looney Tunes logic, the second solution uses Harry Potter logic, and the third solution uses a modal logic of your choice. In all three situations, just because the logic makes sense in the world which hosted it does not mean that it makes sense in our world. Given this obstacle, how can we compare different possible worlds in order to determine which possible worlds have logical conclusions which can be used in our world to deduce facts?

                  So we are giving up our ability to run simulation with our minds now?

                  How do you show that your simulation accurately emulates reality and isn’t an imagined fantasy?

                  1. 1

                    How do you show that your simulation accurately emulates reality and isn’t an imagined fantasy?

                    Based on similar logic being used to great success in the past.

                    Given this obstacle, how can we compare different possible worlds in order to determine which possible worlds have logical conclusions which can be used in our world to deduce facts?

                    This is too abstract for me. What’s the relevant of this line of thought to the prior post?

                    1. 1

                      How do you show that your counterfactual reasoning is not based in e.g. Harry Potter logic, but in a logic which is valid in our world?

                      1. 1

                        Based on past events that describe our world?

                        1. 1

                          Reasoning based on past events is not counterfactual; the past really did happen in our world. It can be partial and provisional, since our knowledge of the past is imperfect and incomplete. But instead of counterfactual assumptions which lead to modal logic, our observations about the past give us analogies and narratives for considering and analyzing current events. When those analogies are coupled with a descriptive explanatory model, then we have a field of science.

                          All of this fails to account for black swans. Indeed, you’ve found one of the big gaps between counterfactual reasoning and science: Counterfactual reasoning is deductive, while science is inductive.