1. 31
  1.  

  2. 23

    Part I starts with a faulty premise. This means that our explanations might not fulfill the explainable-AI requirements. Why? Because the discovery of the Higgs boson was made by theories about why particles have mass having various implications, and those implications being followed through with formal logic. In this arena of reasoning, we are not doing statistical analysis on lots of pseudo-continuous data, but instead our inputs are proof statements.

    If I had to explain to a lay reporter why we expected the Higgs boson, I would start my attempt with something like, “Electricity and magnetism are two ways to look at the same single happenstance in physics, where negatively- and positively-charged objects interact with each other. When we look at particle physics, we see a way to combine the behaviors we see, with atoms splitting and joining, with electromagnetism. This helps us build our models. But imagine that we switched ‘positive’ and ‘negative’. Look up the history with Ben Franklin, it’s a great story. The point is that those labels could have been switched, and that gives us a sort of symmetry for particles. The Higgs boson is a particle that should exist according to what we know about particle symmetries, and we call it ‘Higgs’ after one of the people who first noticed that it should exist.”

    Crucially, this explanation uses a real example as an analogy to bootstrap intuition about the unknown. Rather than handwaving, poisoning the well, or appealing to authority; the explanation lays out a context, including specific symbols (keywords) which can link to further details. The explanation does not rely on a vague statistical argument made using many weak indicators, but uses one main concept, symmetry, as its focus.

    Now, having said all this, I strongly suspect that the author might reasonably reply that the question they wanted to ask was more like, “what pattern in experimental data prompted the theoretical work which led to the proposal of the Higgs mechanism?” This does sound like something that could be answered with a data-driven correlation. And, indeed, that is what happened; the models of that time were faulty and predicted that certain massive particles should be massless. But the actual statistical techniques that were used were the standard ones; the explanation could begin and end with a t-test.

    All of this context is necessary to understand what will happen to poor Steve. Historically, Steve’s last name might be the most important input to the algorithm, or possibly their ethnic background if the algorithm can get hold of it more directly. And Steve’s inability to participate in society is explained away by the reassurance that there are millions of parameters which might have influenced the decision. This is exactly the sort of situation that explainable-AI proponents are trying to avoid.

    But in both cases, the reasoning is not simple, there’s no single data point that is crucial, if even a few inputs were to change slightly the outcome might be completely different, but the input space is so fast it’s impossible to reason about all significant changes to it.

    I don’t agree with this. Specifically, I don’t think that those millions of parameters are actually used much. Instead, I think that NNAEPR and there are only a handful of parameters which account for almost all of the variance in loan amounts, and that the error of the remaining parameters is subject to roundoff. Similarly, only one measurement, mass, needed to be wrong to provoke the development of the Higgs mechanism in theory.

    The explanation in part III is not a valid proof, because correlation is not transitive. I do appreciate the exploration here into epistemology and the nature of justification. But I can’t ignore the fact that the maths are incorrect; if an AI can generate English which successfully bullshits people, then is it really explaining or just lying? In a hypothetical world where AIs have civil rights, we would expect AI explanations to be just as cross-examinable as human explanations, and thus to stand up under scrutiny. What difference is there between an opaque AI loan officer and an opaque human loan officer?

    As we have explored here before, we must be extremely skeptical of the argument that it is simply too hard to explain ourselves to others, in the context of the immense social harm which results from being judged by opaque corporations. Specifically, when they claim that they cannot be legible to outsiders, they are trying to find ways to be less responsible for their own actions; be assured that the corporation is legible to itself.

    1. 10

      we must be extremely skeptical of the argument that it is simply too hard to explain ourselves to others, in the context of the immense social harm which results from being judged by opaque corporations

      Just want to say that I think this is a really thoughtful and true thing, beyond the rest of your commentary. Ultimately the worth of these tools, surely, must be measured in how beneficial they are to society.

      If a neural net loan officer saves society a few tens of of thousands human-labor-hours a year, subsequently making loans slightly cheaper and more profitable, that’s good. But if they do that while also making it impossible to answer the question “why was this loan denied”, then well, the net effect is that you made the world worse and more open to exploitation and your approach should be curtailed.

      1. 5

        Back in 1972 John Kemeny (co-developer of BASIC) was warning about uninterrogable decision-making (in Man and the Computer):

        I have heard a story about the design of a new freeway in the City of Los Angeles. At an open hearing a number of voters complained bitterly that the freeway would go right through the midst of a part of the city heavily populated by blacks and would destroy the spirit of community that they had slowly and painfully built up. The voters’ arguments were defeated by the simple statement that, according to an excellent computer, the proposed route was the best possible one. Apparently none of them knew enough to ask how the computer had been instructed to evaluate the variety of possible routes. Was it asked only to consider costs of building and acquisition of property (in which case it would have found routing through a ghetto area highly advantageous), or was it asked to take into account the amount of human suffering that a given route would cause? Perhaps the voters would even have agreed that it is not possible to measure human suffering in terms of dollars. But if we omit considering of human suffering, then we are equating its cost to zero, which is certainly the worst of all procedures!

        (This message brought to you by the Campaign for the Rehabilitation of BASIC.)

        1. 3

          You raise an important point about model interpretability. All models that predict the future by training on historical data propagate historical bias. This is an effect, not a side-effect.

          A simple example can be found in natural language processing, where words become numbers to be usable as model features. With a model trained on a corpus of human-written documents, you’ll be able to “subtract” the word “woman” from the word “king” to get the result of “queen” and think yourself quite clever. Then, you’ll subtract the word “woman” from the word “doctor” and find yourself uncomfortable to discover the result is “nurse”.

          An additional example drawing from the above comment: if it is illegal and unethical to deny a loan on the basis of race, but you build an opaque model to predict loan outcome that (under the hood) incorporates e.g. census block as a feature, you will still have built a redlining AI that reinforces historical racial segregation.

        2. 1

          I don’t agree with this. Specifically, I don’t think that those millions of parameters are actually used much. Instead, I think that NNAEPR and there are only a handful of parameters which account for almost all of the variance in loan amounts, and that the error of the remaining parameters is subject to roundoff. Similarly, only one measurement, mass, needed to be wrong to provoke the development of the Higgs mechanism in theory.

          Okay, How do you explain why you believe the parameters necessary are smaller? If you want to counter his argument based on the maths being wrong you have to explain why you think the maths are wrong. And that in some sense is playing straight into his argument.

          1. 4

            I strongly suggest that you spend some time with the linked paper. From a feature-based POV, polynomial regression directly highlights the relatively few robust features which exist in a dataset. Neural nets don’t do anything desirable on top of it; indeed, they are predicted and shown to have a sort of collinearity which indicates redundancy in their reasoning and can highlight spurious features rather than the robust features which we presumably desire.

            Even leaving that alone, we can use the idea of entropy and surprise to double-check the argument. It would be extremely surprising if variance in Steve’s last name caused variance in Steve’s loan qualifications, given the expectation that loan officers do not discriminate based on name. Similarly, it would be extremely surprising if variance in Steve’s salary did not cause variance in Steve’s loan qualifications, given the expectation that salaries are correlated with ability to pay back loans. This gives us the ability to compare an AI loan officer with a human loan officer.

          2. 1

            This means that our explanations might not fulfill the explainable-AI requirements. Why? Because the discovery of the Higgs boson was made by theories about why particles have mass having various implications, and those implications being followed through with formal logic. In this arena of reasoning, we are not doing statistical analysis on lots of pseudo-continuous data, but instead our inputs are proof statements.

            If I had to explain to a lay reporter why we expected the Higgs boson, I would start my attempt with something like, “Electricity and magnetism are two ways to look at the same single happenstance in physics, where negatively- and positively-charged objects interact with each other. When we look at particle physics, we see a way to combine the behaviors we see, with atoms splitting and joining, with electromagnetism. This helps us build our models. But imagine that we switched ‘positive’ and ‘negative’. Look up the history with Ben Franklin, it’s a great story. The point is that those labels could have been switched, and that gives us a sort of symmetry for particles. The Higgs boson is a particle that should exist according to what we know about particle symmetries, and we call it ‘Higgs’ after one of the people who first noticed that it should exist.”

            I think we might have different intuitions for what “explanation” means here.

            The above is the kind of news-conference explanation that is not at all satisfying. It’s a just-so story, not something you’d want from, e.g., an emergency system controlling the launch of nuclear missiles… or even from an organ transplant model that decides who the most likely to benefit patients are.

            Maybe, if you actually know physics yourself, try to answer a question like:

            “Why is the mass of the higgs boson not 125.18 ± 0.15 GeV/c^2 instead of 125.18 ± 0.16 (as per Wikipedia) ?”

            or

            “What would it take for the mass to be 125.18 ± 0.15 ? How would the theory or experiments have to differ ?”

            Those are the kind of explanations that, I presume, a physicist working on the Higgs boson could give (not 100% sure how accessible they are, maybe anyone with a physics PhD could given a bit of digging). But the likelihood of me understand them is small, you can’t make a “just-so story” to explain those kinds of details.

            Yet ultimately it’s the details that matter, the metaphysics are important but explaining those does not give a full picture…. they are more like intuition pumps to begin learning from.

            I don’t agree with this. Specifically, I don’t think that those millions of parameters are actually used much. Instead, I think that NNAEPR and there are only a handful of parameters that account for almost all of the variance in loan amounts, and that the error of the remaining parameters is subject to roundoff. Similarly, only one measurement, mass, needed to be wrong to provoke the development of the Higgs mechanism in theory.

            This I actually agree with, and my last 4 months of leisure time research have basically been spent on reading up on parameter pruning and models that use equations that are quick to forward prop but take long to fit (quick to forward prop is often ~= easy to understand, but I prefer to focus on it since it has industry application).

            That being said it’s not all clear to me that this is the case, it could be the case in some scenarios but (see III.2 and IV in the article) it’s probably an NP-hard problem to distinguish those scenarios. And maybe my intuition that most models are over-parameterized and using backprop-optimized operations is just wishful thinking.

            Again, not to say our intuitions differ here, but I tried to go against my intuitions when writing this article, as state in it.

          3. 8

            This is philosophical bullshit. It’s not wrong, it’s just irrelevant. No one expects lossless knowledge transfer from explanations, except in simple cases where inherently interpretable models have decent predictive performance. Even then, I’d argue that’s not the goal.

            Lossy explanations are powerful.

            • Doctor explains to the patient why she thinks they have cancer.
            • Physicist explains to media why her discovery is important.
            • Boss asks you why the project is late.

            Lossless explanations are epitomized by the phrase TMI, too much information (I’ll leave out the bathroom jokes). We rarely, if ever, want lossless explanations. What we really want is some combination of several smaller questions:

            • Accuracy: are you usually right in cases like this?
            • Confidence: am I an outlier?
            • Blame: what feature led to the prediction?
            • Counterfactual: why was it X instead of Y?
            • Mental model: teach me to make decisions more like this high performing model.

            And plenty others. One problem is we don’t have good measures of explanation accuracy, mainly because psychologists don’t have a great objective methods yet. That’s not a blocker, but it makes it significantly more complex to evaluate the effectiveness an explanation than the predictive error of a model. For that reason, I see ML explanations as probably too early.

            The article is philosophical bullshit because no one wants explanations in the sense it describes. The problem as framed isn’t just impossible, it’s not interesting or useful if it were possible.

            1. 3

              His physics example is flawed because the Higgs mechanism has nothing to do with General Relativity.

              But other than that, he has a point :)

              1. 1

                Well, my misunderstanding of physics can hopefully count as a meta-example of the whole “physics is hard to get if you are not a physicist” thing. Granted, the relationship between gravity (and thus general relativity) and the Higgs boson, I’m pretty sure was just some trivia I heard at some point… so I could have researched that better.

                1. 1

                  The Higgs field makes certain particles massive that would otherwise be massless. Gravity is the way things with mass interact with each other.

                  So there is a connection between the two, but it’s not causal. The standard model of particle physics, of which the Higgs field is a well-established part, completely ignores gravity & always has done.

              2. 2

                I think lots of machine learning is not in the realm of explainability, but there is plenty of machine learning that is explainable.

                What is your definition of machine learning? Is linear regression machine learning? Are there more basic examples you can start with? Are you just talking about deep learning?

                The first example uses 900PB of data, which is one of the largest datasets you can think of. Start smaller and use first principles. I use plenty of ML that is explainable - for example LamdaMART decision trees are complicated, but you can explain the results after a pass through the model.

                1. 1

                  I… agree with this, I even mention it in my last points (IV and III.2)

                  The problem is that you have “easy” problems where explainable algorithms work well and achieve the best possible accuracy. But then you have “hard” problems where explainable algorithms might work, but with a sacrifice of accuracy.

                  Even worst, the “easy” vs “hard” distinction (i.e. known if the best possible generic solution can be achieved with a simple decision-tree booster/regression style algorithm) is very hard to generate, basically impossible in the case of datasets with millions of examples and requires a tremendous amount of compute even for play datasets (since you have to run a very exhaustive train + validate loop on many different folds for a large variety of algorithms, either until you find 100% accuracy OR have a high certainty you’ve reached the accuracy cap).

                  I.e. the problem of figuring out the ideal algorithm is NP-hard in all but a few edge cases.

                  So unless I know in advance my problem is “easy”, I have two choices:

                  1. Use an explainable algorithm, but lose accuracy
                  2. Use a model with more parameters, but make explainability harder

                  Even if you assume this is a false dichotomy and the most explainable algorithms are also the most accurate (highly unlikely, see point I and II as well as the last 20 years of history for machine learning) you’re still left with these problems, it just translates to a problem of investing research & compute into investigating accuracy versus explainability.

                  But I digress since I’m kind of re-writing the last 2 chapters from a different angle.

                  1. 2

                    Even if you assume this is a false dichotomy and the most explainable algorithms are also the most accurate (highly unlikely, see point I and II as well as the last 20 years of history for machine learning)

                    I challenge this assumption about history. I definitely think robust, explainable models at this moment in time are not SOTA, but that’s simply because a lot of the groundwork hadn’t been laid, and in the coming decade we may see explainable SOTA models. There are lots of individual innovations that “unlocked” DL to be where it is now, namely lowering graphics compute costs driven by gaming, and innovations in activation functions (such as ReLU) and even in programming language accessibility (with languages like Python and R prominent in ML, built atop tensor libraries) it’s a lot easier for a researcher to iterate on DL models.

                    Likewise were seeing a lot of these benefits translate to explainable models as well, such as Bayesian models or polynomial regression models (as has been discussed upthread). And as a sibling poster said, there’s nothing stopping you from choosing the explainability of your model around domain needs. Making a game AI? Explainability isn’t that important. Determining credit scores? Auditability is essential. As it is, SOTA often comes with the caveat of reproducing the exact circumstances the authors massaged in order to release their results, so models with slightly worse theoretical performance may not in reality perform that much worse.

                    1. 2

                      The problem is that you have “easy” problems where explainable algorithms work well and achieve the best possible accuracy. But then you have “hard” problems where explainable algorithms might work, but with a sacrifice of accuracy.

                      I don’t see why this is a problem, maybe I’m missing something. Making a choice between the two, then, just requires expanding your - philosophical - utility function to include whatever context the explaining becomes valuable in.

                      Are you doing machine vision in a driving setting, where the value of being able to say with certainty that there aren’t bonkers edge case behaviours is high? Then perhaps it’s worth losing precision: you might choose to have injuries rise by some well-known fraction than to have a huge unknown in the system (“turns out during solar eclipses the cars just run over pedestrians all day long and won’t stop”).

                      Or are you doing something where the value of auditing and comprehension is low? If so, by all means chose the more accurate model and lose auditability.

                      Maybe I’m seeing this naively though.

                      Edit: Maybe what I’m missing is that you don’t mean that “this is a problem” in the sense of like “this is a problem we need to solve”, but rather in the sense you have in the article of like “well, this is something that looks like it’s just a fundamental trade-off we have to make here”, which seems intuitively right. Of course, I’d then argue that that is a problem, because we would be able to use these algorithms in many, many more places if they were able to meet explainability criteria.

                      1. 1

                        Maybe what I’m missing is that you don’t mean that “this is a problem” in the sense of like “this is a problem we need to solve”, but rather in the sense you have in the article of like “well, this is something that looks like it’s just a fundamental trade-off we have to make here”, which seems intuitively right.

                        Yeap, that was what I was trying to convert there, that I see a tradeoff between the two (accuracy and explainability) often being the case/

                  2. 2

                    I didn’t find this convincing at all.

                    “It’s too complex” just isn’t a good argument and completely dodges the issue. If there are a million parameters then it’s reasonable to expect an explanation of where the parameters came from, why a particular parameter value was chosen (or what what other parameters it depended on), why other parameters weren’t used, etc. We know it’s complex, that’s why we need a computer to explain it to us.

                    At the end of the day, somebody wrote the program that made those choices, and they used far fewer parameters and should be able to justify the decisions they made, and how the algorithm made its choices.

                    “Too boring” is irrelevant. In that case the person running the algorithm doesn’t care, but it says nothing about whether the AI can be explained.

                    The third point doesn’t support the argument because it assumes the algorithm is explainable to a subset of people. That’s par for the course for explaining anything. Obviously a person needs to understand the domain before they can understand the explanation.

                    1. 2

                      I agree with the conclusion but disagree with the reasoning the author used to get there.

                      This situation is both recursive and ironic, as being able to explain why a conclusion was made is the entire thrust of the essay. I find myself unable to do so in a brief space.

                      (This comment was not meant as a troll, simply an observation. Being able to explain something and being able to explain something under a particular situation, particular conditions, and given a specific audience are two entirely different things)

                      1. 2

                        I’ve been convinced, over the past five years, that the reason Google is so tight lipped about why it bans people is because the employees themselves don’t know the answer—the computer(s) just do the ban and it’s up to the employees to “fix” things after the fact (if they are forced to by public pressure, probably by biasing one or two values they know works to fix things).

                        1. 2

                          probably by biasing one or two values

                          I wonder if they even know that much?

                          It wouldn’t surprise me if there’s an override field for those cases that prevents or reverses bans. Maybe a parameter that tells the AI it can’t ban a person, but has to raise the issue to a human who will make the decision.

                          1. 2

                            I had another thought—the system is told the account is not banned and then it’s run to relearn how not to ban that particular account at this time.

                        2. 1

                          I have often pondered that as machine learning becomes increasingly more complex and able, reasoning about why it has come to a conclusion will become equivalent to reasoning about why an animal made a decision.

                          To that end I don’t believe machine learning is fundamentally unexplainable, but that as it becomes more capable that reasoning will shift from analysing it as a machine towards analysing it as a mind.

                          1. 1

                            I like this article because it helps me see things in a new light. It approaches futurology in a way I find appealing but that actually relates to the science, as opposed to being sensationalist like most people being published. Nevertheless, I don’t think its premise is definitely correct. Interesting, yes. Possible, yes. But it should be evaluated like it suggests most things are, from various angles.

                            Random question, though: aren’t there ways to train a model while obscuring random parts of the data set, so it doesn’t over-rely on, e.g., a figure of $340?