1. 36

    1. 23

      one of the greatest risks is not that chatbots will become super-intelligent, but that … systems operating without evidence or logic could become our overlords by becoming superhumanly persuasive, imitating and supplanting the worst kinds of political leader.

      Well before that happens, such systems will support and empower the worst kinds of political leader.

      I’m expecting an onslaught of fully-automated, GPT-powered, X-amplified bullshit during the 2024 US elections that will make us nostalgic for the onslaught of merely human-generated bullshit in 2020.

      1. 14

        systems operating without evidence or logic could become our overlords

        Let’s be honest, we already have that. It’s called governments and large corporations. Through emergent complexity, these entities are as inscrutable as neural nets. These systems have developed a life of their own and are governed by rules that no human can understand or control, almost like an alien species that settled down on this planet. Thus, the fear is largely unfounded as we already find ourselves in this scenario and have been living like this for a long time.

      2. 1

        I’m not so worried about that. Those political leaders can already employ teams of propaganda specialists to craft their messages. Things like ChatGPT are also going to empower the most toxic kind of social media ‘influencer’ by giving them tools that are almost as good.

        1. 3

          The problem here is again that of scale. If you can outbullshit the other guy, people will never have a chance of seeing his truth, just more bullshit from everywhere.

          1. 3

            I agree that the problem is scale, I just don’t see it being led by politicians. Established politicians already employ teams of psychologists to identify bullshit that will resonate with their target audience. The amount of bullshit they can generate is a function of money and they have a lot of money. At the moment, there are few politicians as a proportion of the general population. Things like ChatGPT make this kind of thing accessible to the general public. Any random conspiracy theorist can easily flood the world with bullshit that supports their world view.

    2. 7

      Alan has recently had a book on this theme published. I agree with a lot of his underlying thesis: many of the things that ‘AI’ is being touted for are examples of usability failures and can be better addressed with classical approaches.

      1. 1

        many of the things that ‘AI’ is being touted for are examples of usability failures

        Can you give an example from the book?

        1. 10

          [ Disclaimer: I don’t remember which of these are in the book and which are from conversations with Alan ]

          A lot of the things where Copilot is used are ‘look we can generate this boilerplate automatically’, where Alan argues that the correct solution is languages with richer metaprogramming facilities where you simply don’t write that boilerplate in the first place. Similarly, the proposed uses in Excel are an indication that Excel, in spite of being the worlds most successful end-user-programming environment, is very bad at that task. There are better end-user-programming environments that, if they were integrated into mainstream apps, he believes would give a better experience than LLM-based tooling. There are two key problems that you need to solve:

          • Unambiguously state the requirements.
          • Execute a solution to that problem.

          LLMs help with taking an ad-hoc description of a problem and turning it into something that can be executed but they are just as ambiguous as any other natural-language description. Worse, they’re opaque blobs and so may do something different if you give them the same prompt twice. The thing that people actually need help with is expressing what they want unambiguously. If you can solve that via things like direct visual feedback and better design-space exploration tooling, then you end up with something where the outputs are reliable.

          1. 1

            Thank you

    3. 4

      Frankfurt’s On Bullshit is a fun read, but I don’t believe it has very much theoretical utility. He focuses entirely on the intentions of the bullshitters, and no time on the political economy of the bullshit.

      I think a far better text in this regard is Christian Thorne’s The Dialectic of Counter-Enlightenment. He begins by discussing the philosophy of the Pyrrhonists lead by Sextus Empiricus:

      Call it what you will—antilogy, aporia, the argument ad utramque partem—motivating even the most extreme flight of rhetorical fancy is the notion that the orator must be able to argue with equal facility on either side of a question, and it is a tenet generally cited as one of the key components of the classical rhetorical education. To argue both sides of an issue, any issue, is to hone your mind to purely rhetorical strategies; it is to devote your attention to the sheer language of argument regardless of the end to which that language is put, and hence to forestall our habitual urge to uncover the truth of the matter. (p. 22)

      Severing the connection between language and a theory of knowledge is very, very old it turns out!

      Pyrrhonism dances on the fringes of philosophical argument, employing any arguments it finds useful in undermining the claims of others, only to relinquish that argument as soon as it is challenged. The Pyrrhonist portrayed by Sextus is a rhetorical gadfly and gleeful obscurantist, militating against transparency like some philosophical smoke machine, piling on claim after claim until thought has more or less fogged over. (p. 27)

      His aim is to shut cognition down altogether, to transform argument into unknowing. (p. 28)

      The Pyrrhonists argued against all forms of knowledge in order to reach a state of mind they called ataraxia.

      …in Pyrrhonism, aporia is the point where confusion passes into calm. It is the very key to peace of mind—or ataraxia—that Sextus takes as the sole end of human endeavors. (p. 34)

      The value of Thorne’s analysis is that he is not content to take the Pyrrhonists at their word.

      Rhetoric, of course, is the business of pursuasion. But persuasion can only transpire if somebody, somewhere is acting non-rhetorically—reasoning or holding beliefs, willing one way or another to assent to some claim about the world … The effect of Pyrrhonism may be to turn its acolytes into rhetoricians, but by thus initiating them, it evangelically spreads its resolve to withdraw assent, innoculating its acolytes against rhetoric itself, and thus robbing the orators of their audience one member at a time. (p. 36)

      Pyrrhonism is inherently self-defeating, so why evangelize it so strongly? Because its adherents were living through a period of crisis in the Roman Empire—because direct confrontation was illegal, political enemies of the state turned to philosophy to air their grievances, and the state was beginning to catch wise.

      This, ultimately, is what grounds the Pyrrhonist rejection of philosophy and rhetoric alike: in the most hallowed sites of public life, philosophy and rhetoric come together in some horrible, insurgent pact, opening up power to epistemological review, and thus placing the locus of authority permanently, perilously in question. Indeed, it would seem to be the startling upshot of Pyrrhonism that authority itself is an impossible concept; it is literally unthinkable, because the attempt to derive power from some prior term—to legitimize it with reference to something that is not itself power—can only serve to unsettle the social order … Rhetoric is just sedition by another name, and orator a fancy word for rebel, “depraving the crowd by his doctrines, using flattering words, and setting them against the better class by his slanders.” … Pyrrhonism’s celebration of ataraxia—of peace of mind, of tranquility—is simply an analog of its commitment to existing political and social arrangements. When it asks us to suspend judgement, it means simply to lure us back into the confines of the common forms. (p. 42-43)

      “ ‘To be persuaded’ has different senses: on the one hand, it means not to resist but simply to follow without much proclivity or strong pro feeling, as the child is said to be persuaded by or obedient to his teacher; but sometimes it means to assent to something by choice and with a kind of sympathy due to strong desire, as when a profligate man is persuaded by one who approves of living extravagantly.” The distinction here is between persuasion as a form of obedience and persuasion as a form of choice, and Sextus, plainly, wishes us to sacrifice the latter to the former, to engineer a kind of non-philosophical way of life that is capable only of acquiescence, that can never be exhorted out of orthodoxy. The difference between fideism and Pyrrhonism again becomes apparent: Fideism dispenses with philosophy by substituting faith for knowledge, but Pyrrhonism goes fideism one better by making a less familiar switch: It seems to replace knowledge with faith, but then, at the last minute, replaces faith with compliance, which need not include faith of any kind. (p. 50)

      The true purpose of bullshit is compliance. Its purpose is to overload the interlocutor and halt the process of creating knowledge—either of a certain kind, or of any kind.

      The Pyrrhonists are only the most straightforward example of this. Thorne traces the history of this practice through Montaigne, Bacon, Hobbes, Swift, even Stanislaw Lem makes a surprise appearance. Very interesting read.

    4. 4

      I agree that the current state-of-the-art AI text-generation systems do not have the ability to generate text that people can have confidence is stating true facts about the world; I just don’t see this as a problem. It’s always been possible for human writers to write fiction, to write lies masquerading as the truth in order to mislead people, and to write complete bullshit in this article’s sense of disregard for truth value; and LLMs can produce text that fulfill all these functions, depending on what the human user asks of it. People certainly shouldn’t automatically believe that the text a generative model produces has any connection to the truth of the world; but this is true for every genre of text that has ever existed.

      But Frankfurt’s book is not the only classic text on bullshit. David Graeber’s famous analysis of Bullshit Jobs explains precisely why Elon Musk’s claim to the British PM is so revealing of the true nature of AI. Graeber revealed that over 30% of British workers believe their own job contributes nothing of any value to society. These are people who spend their lives writing pointless reports, relaying messages from one person to another, or listening to complaints they can do nothing about. Every part of their job could easily be done by ChatGPT.

      I’ve never been convinced by Graeber’s writing on bullshit jobs. Even if an employee dislikes their job, which all sorts of people do for all sorts of reasons, someone finds it worthwhile to pay them to do it. And when the institution that was previously employing a person realizes that they were wrong and actually it really isn’t worthwhile to pay someone to do that job, we typically call this a “layoff”, and most people subject to them aren’t pleased about it. If the reason they’re being laid off is because ChatGPT actually can do their job as well as a human could (or at least for a more favorable price/quality ratio), we typically call this “technological unemployment”, and this is also a thing the people subject to it are generally not very happy about.

      I’ve actually personally been in a position where I disliked a lot of aspects of my job, was considering quitting and doing something else, and then found myself laid off and being kinda happy about it. And even in that case, I would not say that my job was bullshit in any meaningful sense - if nothing else, the paycheck money hitting my bank account every two weeks was very much not bullshit to me.

      1. 7

        People certainly shouldn’t automatically believe that the text a generative model produces has any connection to the truth of the world; but this is true for every genre of text that has ever existed.

        But we needn’t automate the bullshit any more than we should create machines that overload our septic systems.

      2. 3

        It’s always been possible for human writers to write fiction, to write lies masquerading as the truth in order to mislead people, and to write complete bullshit in this article’s sense of disregard for truth value

        Have you ever tried to write convincing nonsense that leads people to a particular belief? It’s not easy. If it were, hiring an advertising company would be a lot cheaper. The problem is that it is increasingly as simple as providing a prompt to a tool stating what you want.

      3. 2

        Even if an employee dislikes their job, which all sorts of people do for all sorts of reasons, someone finds it worthwhile to pay them to do it.

        There is a theory of management which suggests that the main reason managers hire employees is to increase headcount. In Bullshit Jobs, this is referred to as “managerial feudalism.” According to this theory, hiring is not done because it is “worthwhile;” it is not done to save the business money by acquiring specialized labor, nor to free up managers from their current tasks so that the business can grow. Rather, hiring is done in order to flatter the egos of individual hiring managers and directors.

        And even in that case, I would not say that my job was bullshit in any meaningful sense - if nothing else, the paycheck money hitting my bank account every two weeks was very much not bullshit to me.

        That’s still a bullshit job, though. If one doesn’t get paid, then one wasn’t employed; a job implies compensation.

        1. 4

          It’s worth noting that a lot of this kind of behaviour depends on timescales. A company that hires people to work bullshit jobs will have higher costs than one that doesn’t and so the stable state is for the company employing bullshit workers to go out of business. It can take a very long time to reach that stable state. It’s taken two hundred years since the Industrial Revolution for the idea that managers should understand intrinsic and extrinsic motivation and work to get the best out of their employees to become mainstream. Even then, it isn’t universal.

          On empire building, there are also factors that work to propagate this. Someone who builds a big empire gets promoted and then moves to another company. The new company has no visibility into how many of the manager’s reports were working bullshit jobs and so that culture moves over to the new company. That manager then (subtly) encourages his direct reports to grow teams as a goal in and of itself. Folks I know at Google are complaining that Google has been hiring a load of ex-Microsoft people with exactly this mindset, but when you get enough of them you end up promoting people based on these criteria.

          Big companies take a long time to fail. IBM made terrible decisions for 20 years and is still around, though a fraction of its former power. This kind of change happens over a period of decades, not years.

    5. 5

      i’m sure if you picked some run-of-the-mill examples of AI generated text, and sent them to the author 5 years ago, they would confidently say “an AI could never write this”, so i’m not going to state too confidently what AI will or won’t do 5 years from now

      1. 4

        We have had an AI winter before. We are either at the start of a hockey-stick curve of AI development, or at a local maximum, before the start of another.

        1. 3

          There’s no exponential growth in a finite universe that both consumes any kind of resource and goes on forever, so there is going to be a plateau at some point.

          But I’ll readily admit that I’m one of the people who five years ago wouldn’t have guessed an AI would be able to generate text at GPT-4 quality today, so I’m not going to pretend I can confidently predict where that plateau is going to be.

          1. 7

            Markov chains could generate plausible text in the ‘90s. In the Cambridge computer lab, they ran a weekly happy hour (beer and snacks) and, after writing a load of announcements every week, one of the organisers got bored and wrote a tiny Python program that built Markov chains of all of the existing ones and wrote new ones. It worked well, right up until it announced a free one by accident.

            ChatGPT is not fundamentally different in functionality, only in scale. The model is a denser encoding of the probability space than a Markov chain and the training set is many orders of magnitude larger, but the output is of the same kind: it looks plausibly like something that could have been in the input set but contains clear examples of not being backed by any underlying understanding (ChatGPT will confidently assert things of the form ‘A, therefore not A’).

            1. 3

              I’ve written several Markov chain-based text generators myself - the most amusing one used the blog of a famously bullshit-prone local political commentator as its training corpus; the result ended up sometimes being very hard to discern from the real deal. But only on the snippet level: If I had it compose an entire essay, the result was so obviously nonsense that nobody was fooled for a second. That’s what having a “context window” of two words and a blog-sized corpus gets you. :)

              I suppose my own inability to predict the quality of current-day text generation was more about underestimating the current scale of datacenter compute (and the practicality of working with truly enormous data sets) than anything else. If anything, LLMs are as much an accomplishment of Big Data as one of AI.

              1. 9

                If anything, LLMs are as much an accomplishment of Big Data as one of AI.

                I completely agree with this. It’s also worth noticing which companies are pushing them. It’s not just that they’re companies that have senior leadership who are promoted on their ability to spout plausible bullshit and so think that’s what intelligence looks like, they’re also companies that make money selling large compute services.

                Back in the ‘80s, if you needed a database for payroll and so on, you wanted to buy something from Oracle and IBM, often with a big piece of hardware to run it. By the late ‘90s, you could do the same thing with PostgreSQL on a cheap PC. Maybe a slightly more expensive PC if you wanted RAID and tape backups. Now, things like payroll, accounting, inventory control, and so on are all handling such tiny amounts of data that you wouldn’t think or running them with anything other than commodity infrastructure (unless you’re operating at the scale of Amazon or Walmart).

                This is always the problem for folks selling big iron. Any workload that needs it today probably won’t in 10 years. As such, you need a continuing flow of new workloads. Video streaming was good for a while: it needs a lot of compute for transcoding, a lot of storage to hold the files, and a lot of bandwidth to stream them. As the set of CODECs that you need shrank and they improved, these requirements went down. With FTTP bringing gigabit connections to a lot of folks, there’s a risk that this will go away. With something like BitTorrent streaming, a single RPi on a fast home Internet can easily handle 100 parallel streams of HD video and so as long as 1% of your viewers are willing to keep seeding then you’re good and the demand for cloud video streaming goes away for anyone that isn’t a big streaming company (and they may build their own infrastructure).

                But then AI comes along. It needs big data sets (which you can only collect if you’ve got the infrastructure to trawl a large fraction of the web), loads of storage for them (you use the same data repeatedly in training so need to have enough storage for a large chunk of the web), loads of specialised hardware to train (too expensive to buy for a single model, you need to amortise the cost over a bunch of things, which favours people renting infrastructure to their customer) and then requires custom hardware for inference to be effective. It’s a perfect use case for cloud providers. The more hype that you can drive to it, the more cloud services you can sell. Whether they’re actually useful to your customers doesn’t matter: as long as they’re part of the hype cycle, they’re giving you money.

      2. 2

        I disagree. NLP researchers have been experimenting with text generation for decades. I first heard the term “hallucinated text” in 2014, which back then meant any text generated by a model because because it was a given that text models aren’t concerned with the truth. People in our department were convinced that more complex models with more data would generate more coherent text. Especially after the leaps and bounds we saw image generators making. The big surprise is that the architecture turned out to be quite simple, an even bigger surprise is how much money is spent on training the models.

        “We could never make an AI that can write this with our budget” more like.

        There’s not nearly such optimism in fact generating AI that I know of. People in expert systems research have been humble realists ever since the AI winter, and nobody serious is jumping on the LLM hype train. All I hear are wishes from business people.

    6. 2

      I’ve been wondering if part of the problem is that the world “feels” (subjectively) overwhelmingly complex to many people. If thinking about what’s true causes anxiety and stress, is it any wonder that people are happy to accept things that seem (again, subjectively), or feel right? Part of this might be that the world got really complicated, but I wonder if people also got less able to handle the complexity for some reason.

      1. 5

        What if the problem is that the world is in actually overwhelmingly complex, and the many people who feel that way about it are correct to feel that way? What if optimizing for truth above all other concerns has only ever been what a small minority of unusual people has done?

        1. 6

          An LLM can’t begin to optimize for truth because it has no model of the behavior of consciousness systems and therefore can’t evaluate the speaker’s statements within the context of their motivation, let alone broader contexts.

        2. 1

          That’s an interesting take on it. My gut reaction was to say “well no one, until recently, believed something as stupid as the government is putting microchips in the Covid vaccines”, but then I reflected a bit and realized that people have always believed some pretty dumb (based on my perspective) things. I do think, though, that even if you’re right, technology and population have magnified the collective choices people make, so now we’re stuck in a position where we need more people to be concerned with truth.

    7. 1

      The author might have missed this:

      But I think that I remember that one of the differentiating training factors of ChatGPT was actually the large corpus of training input from employees with a high level of education.

      Rather than only (but of course also) social media or people randomly chosen and hired for money.

    8. 1

      I ear and read that a lot but:

      • we are not only automating bs
      • automating bs has value

      The later point is lost to a lot of programmers because they tend to shy away from it, but a huge number of human interactions are total BS. You may want to be part of some activity where BS is an unfortunate requirement, and if you accept this premise, then automating BS is fantastic.

      1. 4

        Can you think of a case of BS that is not just a net private gain, but also a net benefit to society at large?

        It’s my observation that BS is a useful tool for individual goals at the expense of social welfare.

        1. 1

          I used to care about this, but I don’t: the fact is BS is everywhere.

          You can chose to fight it or go with it. You can chose to do both, depending of the context, but it’s there.