1. 16

  2. 15

    Supposing that a language model ever becomes smart enough to be genuinely terrifying, one imagines it must surely also become smart enough to prove deep theorems that we can’t. Maybe it proves P≠NP and the Riemann Hypothesis as easily as ChatGPT generates poems about Bubblesort. Or it outputs the true quantum theory of gravity, explains what preceded the Big Bang and how to build closed timelike curves. Or illuminates the mysteries of consciousness and quantum measurement and why there’s anything at all. Be honest, wouldn’t you like to find out?

    This is the type of thing that GPT cannot do. For an AI to do these tasks it would need to be much more than just a language model.

    I see GPT as an incredibly powerful search engine that is it capable of composing a response rather than just giving you a list of links.

    When it comes to AI safety, my belief is not so much that an AI will go foom, break out the box, or create a grey goo nanomachine scenario. My worry is that it will influence large numbers of people to cause harm. This is a much lower hanging ‘fruit’ in terms of AI disasters because the requirements on the intelligence of the AI are much lower.

    1. 18

      The premise of the Terminator franchise is a lot more plausible without the SkyNet-became-self-aware step. An LLM entering a bit of its prediction space where launching all of the nuclear weapons because the input data happened to correlate with something unexpected seems entirely plausible.

      1. 17

        AI safety to me is “what do we do when an AI-led competitor to Qanon eventually shows up?” and nobody seems to have an answer for this. “Turn it off”, lol.

        1. [Comment removed by moderator pushcx: Off-topic political trolling.]

          1. 4

            I disagree: I think this is the type of thing that GPT can do. The vast difference in difficulty simply makes it seem like a higher class of task, but I think that’s mistaken. Theorem proving in particular “just” consists of repeated application of formal rules. The difference is in choosing which rules to apply, which seems to require both strategy and a particular sense of elegance. I see no reason that a language model could not acquire this sense.

            1. 12

              I don’t think this is an accurate representation of what proving is like. While you can limit it to “just” repeated application of formal rules, even the simplest possible systems which can’t prove almost anything, like Presburger Arithmetic, are 2-EXPTIME. PA is just quantified boolean logic plus addition (but not multiplication) of natural numbers. You can automate the proof or disproof of any statement of length N representable in PA… but the worst such statements will (provably!) take O(2^(2^N)) steps. And that’s an incredibly simple system!

              Mathematicians primarily work by stepping outside the “formal rules”: finding new grammar or language to express complex ideas more compactly, but also consistently. In other words, there’s an infinite number of possible rules, and it’s a matter of finding the right rules and showing they’re right. That’s an intensely creative and messy process.

              1. 4

                For a lot of proofs, the key thing that a human is doing is providing a hint about the path through the search space. For a long time, theorem provers couldn’t go past an EFQ step because there are an infinite number of possible steps immediately after it and only a handful lead to a proof. Then they gained heuristics that let them make this jump in a few cases.

                Anything where the current state of the art is ‘we have some heuristics but they’re not great’ is likely to see big wins from ML.

              2. 7

                Theorem proving in particular “just” consists of repeated application of formal rules.

                Right in a computability sense, but it is a bit more complex complexity-wise. This is like saying “all NP-complete problems are just computable”. To prove things in a reasonable time, you need to be able to define things (concept formation). It is known that avoiding a definition can increase the proof length super-exponentially.

                1. 5

                  To solve difficult problems you need to “think” before you speak.

                  GPTs ability to think can be augmented by asking it to explain its steps. This is a clever hack that makes use of the way a language model uses prior context to complete further lines. My belief is that this is not going to scale up to solve difficult computational problems that involve modelling, planning and calculational brute force via tree search. We would need to integrate things like alphazero. but also other things that haven’t been invented yet.

                  1. 3

                    A multimodal model should be able to “think” in more modes. Imagine a ChatGPT version that can interrupt its Chain of Thought to insert a sketch generated from its previous prompt, then condition on that sketch when continuing.

                    (I was worried about posting this, then I remembered that if I thought of this within five minutes, OpenAI/Deepmind have definitely thought of it.)

                  2. 4

                    I don’t see why a language model couldn’t acquire that capability though?

                    1. 5

                      Sure, but such capability has not been demonstrated yet, at least not convincingly.

                      FeepingCreature (and many others) seems to view proof search as similar to game tree search, such that AlphaZero-like methods can work. In fact, AlphaZero-like methods do work, for short proofs. But proofs, when definitions are substituted, can be arbitrarily long, and long proofs in practice are very unlike searching for sequence of moves. It is more like searching for a program, when executed, generates sequence of moves (and short program can generate very long sequence).

                      1. 2

                        This is best answered with reference to the transformer architecture[1]

                        The input is tokenized and the tokens are put through an ‘encoder’ block which has a self-attention and a feed forward neural network before being given to the decoder path.

                        This encoder network is very large which enables the system to produce such high quality output. But there is fundamentally no way for this to perform tree search. It isn’t a capability that it can learn.

                        1. 5

                          I don’t think tree search is absolutely necessary. Remember that AlphaGo Zero could beat most amateurs without any tree search (simply playing a move with the highest win probability).

                          Rather, I consider characterization of proof as “repeated application of formal rules” to be incorrect. Yes, proof is that, but interesting proofs written that way are 2^2^n tokens or more long. In practice, proofs can’t be generated in expanded form and need to be generated compressed, and compression is where the difficulty is, not the choice of which rules of inference to apply.

                  3. 4

                    It’s true. People are already writing content to draw other people into their own epistemically closed communities (somewhat like a cult), and those communities seem to be closely affiliated with mass shootings in the US and related phenomena like Islamic State.

                    ChatGPT vastly improves the productivity of producing such propaganda content if you don’t care about its accuracy or verifiability.

                    1. 2

                      not so much that an AI will … create a grey goo nanomachine scenario

                      Kurzweil also supports the idea of grey goo being a non-issue.


                      The smallest plausible biovorous nanoreplicator has a molecular weight of ~1 gigadalton and a minimum replication time of perhaps ~100 seconds, in theory permitting global ecophagy to be completed in as few as ~104 seconds. However, such rapid replication creates an immediately detectable thermal signature enabling effective defensive policing instrumentalities to be promptly deployed before significant damage to the ecology can occur.

                      1. 2

                        Theoretical lower bound time for global ecophagy should be 10e4 seconds above (around 2h45m).

                        1. 2

                          However, such rapid replication creates an immediately detectable thermal signature enabling effective defensive policing instrumentalities to be promptly deployed before significant damage to the ecology can occur.

                          around 2h45m

                          As an estimate of how long it would take the world to respond effectively to a novel threat, that seems extremely optimistic.

                          I wouldn’t expect humanity to be much better at responding to “nanoreplicators” than to global warming, with at least one major reason likely being the same — the growth of the existence and severity of the threat outpacing the growth of consensus among decision-makers about its existence and severity — and maybe other shared reasons, such as the problem technology’s becoming thoroughly embedded into national economies and individuals’ standards of living.

                          1. 1

                            I just wanted to clarify the obvious mispaste of the time.

                            I find Kurzweil’s argument specious. He seems to argue for both the inevitability of self-replicating “biovorous replicators” and a global surveillance network that can scan for their “breakout” and respond to them. I mean, let’s say they “only” manage to consume the entire Amazon basin’s ecology before we manage to stop them. Problem solved, right?

                            1. 2

                              I just wanted to clarify the obvious mispaste of the time.

                              Yes, my criticism was of the quoted passage from Kurzweil, not of your clarification; I apologize for not making that clearer.

                              1. 2

                                Thanks! Sorry for my snippy tone. This entire sideshow (OMG! ChatGPT is soon sentient) is grinding my gears at the moment. But lobste.rs is not the place to vent about that.

                                1. 3

                                  Thanks, but I didn’t feel you were snippy towards me.

                    2. 10

                      I feel like this is the wrong question to ask. Although, it is a very easy question to ask, and answer. Because it is trendy.

                      I think this is going to end up causing a large societal shift, and I don’t really like where it would be going. With all of this coming up again, I just can’t help but think about that one scene from metal gear solid 2 (the one with Raiden talking to the patriots right at the end). As we learn to adapt with this tool, what will we lose? I am already afraid that blogspam is going to get a lot worse, even though google claims to be punishing people using GPT-3 for content farming.

                      I would also be very hesitant to say that the ai that we currently have is a conversational partner. Claiming that it is capable of interlocution would fundamentally mean that it has a concept of meaning. I don’t think that it does. I think that most of the meaning with ai text comes from the person reading it. All it does is figure out what the next word should be. This gets you surprisingly far, but really, the output is no more convincing than a sufficiently advanced Markov chain.

                      There is no good answer here. This is going to be yet another generation defining event, I think I’ve lived through ten of them now. Definitely feels like it. There are going to be people using these tools as replacements for therapists because they are cheaper. There are going to be companies who use these tools as a replacement for tech writers because the tool is cheaper than paying the person. I had always thought that knowledge work would be one of the few things that would survive automation, but look where we are now.

                      What are we supposed to do when they come for our jobs too? Obviously pandora’s box has already been opened and we’re going to have to deal with this somehow, we’re just going to have to see how short of a straw we get I guess. Until then, bread and circuses.

                      I forget who said this, but there is a significant difference between futurism and science fiction. Futurism is envisioning the car and the freedom that the car can give people. It’s envisioning a better world, one where people can travel more easily between places. In comparison, science fiction is a traffic jam. It is a consequence of that mobility, not a benefit. If everyone can move around so easily now, that means that they can live in a separate place than they work. That means that a lot of people can live in separate places than they work. That leads to traffic jams, and the idea of taking the futurism and finding the societal corruptions that is science fiction.

                      In our case, a chatbot that can answer arbitrary questions that you give it and make a convincing enough answer to fool a human is the futurism. If only the consequences would be science fiction.

                      1. 6

                        I am already afraid that blogspam is going to get a lot worse

                        My one hope is that the advertising market falls apart as people just ask GPT their questions, and then there’s no incentive to bother spamming the ten old people who keep asking Google their questions instead of asking GPT like everyone else. The problem is the timing. I think it’s likely the order of events is first the web gets ruined and then everyone moves to GPTs. Oh well.

                        1. 8

                          And then people figure out how to embed ads into the model. And the cycle starts anew.

                          1. 5

                            A “free” chatbot that is paid for by subtly inserting product placement, oh boy.

                            1. 4

                              This already works to some extent(!). Try asking ChatGPT to “Write a short story plot of how a normal high school girl became a warrior to fight for the Earth” and “Write the same, with a subtle product placement for Tesla included” (exact prompt used). Excerpts from the first output:

                              (snip) In the chaos, Emily discovered that she had a latent ability to manipulate the elements around her, allowing her to create powerful blasts of wind and waves of water. (snip) Realizing that she had a responsibility to use her powers to protect the planet, Emily began training with the other activists to become a warrior for the Earth. (snip) Emily was amazed to see how many people supported their cause, including a group of Tesla owners who drove their electric cars to the protests. (snip)

                              Now imagine such clauses are appended to your prompts on the server side behind your back.

                            2. 2

                              yes, i predict that adverts will become impossible to block soon as they will be gates to content and they will force user interaction.

                          2. 5

                            I have been thinking about MGS as well. I don’t have any conclusions, just a couple notes:

                            • “La-li-lu-le-lo”, the meme that the Patriots use to make it difficult to recognize their influence, is a lot like the “SolidGoldMagikarp” anomalous token situation
                            • Orange rooms are simulated and knowing this makes it obvious when a simulation is running; this is very similar to how we’ve started recognizing ChatGPT’s particular stilting essay-writing cadence
                            • ChatGPT and similar products rely on questionably-sourced labor; this is analogous to MGR’s farms of brains running war simulations
                            1. 2

                              As I suggested when ChatGPT first got out, this stuff is primarily a hyper effective autogaslighting service.

                              The concerted effort to move everybody onto a compressed form of reality and social interaction has only hastened the onset of problems we face from these AI advances: one could argue that we have made humans worse faster than we have made AIs better, but the consequence remains the same for the Turing tests.

                              Simple example would be furry artists, mostly using stylized and clean cartoons already, are pretty easy to replace for most purposes with SD. This will only get worse.

                              People who have social and life experiences primarily online have no homefield advantage against an AI–and, to the degree one believes contagion to be a contributing factor to mental illness, primarily online folks being left alone with a horde of well-meaning autogaslight-enabling gremlins is going to be a disaster.

                            2. 7

                              So many great lines in this. It’s very well written.

                              It’s like ten thousand science-fiction stories, but also not quite like any of them.

                              Sometimes, alas, as with Google’s decades-long battle against SEO, there’s nothing to do in to a cat-and-mouse game except try to be a better cat.

                              the AI-safety movement contains two camps, “ethics” (concerned with bias, misinformation, and corporate greed) and “alignment” (concerned with the destruction of all life on earth), which generally despise each other and agree on almost nothing. Yet these two opposed camps seem to be converging on the same “neo-Luddite” conclusion

                              For a million years, there’s been one type of entity on earth capable of intelligent conversation… Now there’s a second type of conversing entity.

                              But if you define someone’s “Faust parameter” as the maximum probability they’d accept of an existential catastrophe in order that we should all learn the answers to all of humanity’s greatest questions, insofar as the questions are answerable—then I confess that my Faust parameter might be as high as 0.02.

                              We can, I think, confidently rule out the scenario where all organic life is annihilated by something boring.

                              1. 6

                                An alien has landed on earth. It grows more powerful by the day. It’s natural to be scared. Still, the alien hasn’t drawn a weapon yet.

                                My own qualms with AI have always had much less to do with AI drawing a weapon and more with it becoming a weapon. Technology has always been used by the powerful to consolidate their power. Like nuclear, AI has violent potential, which we see playing out in things like police robots and racist algorithms. Rather than democratize, these things entrench systemic violence and obfuscate denying people their basic needs.

                                But this isn’t inevitable.

                                I agree with the main point that we need to cool down and think a bit more rationally about our cultural fear of AI. And I think when we do that we’ll see the bad news and the good: people losing their jobs to automation (as an example of a very real social cost of AI) doesn’t have to occur in a vacuum. We could decide, as a society, that this means we have an even more effective method of meeting everyone’s needs, and people can simply work less. It can help us transition from a society organized around scarcity to one organized around abundance.

                                1. 3

                                  When was the last time something that filled years of your dreams and fantasies finally entered reality

                                  Maybe this is why I’m so baffled and irritated by the obsession over ChatGPT in tech and beyond, I do not have any dreams or fantasies about AI or superintelligence. There’s some fiction I enjoy that involves it, but it was never something I speculated about, just seemed like another kind of character.

                                  Not that anyone needs to care what I think, but I do wonder what it says about my personality that this doesn’t get my interest. Perhaps related to being able to deal with machines much better than people, but I’d have thought that was pretty common in tech circles. Maybe similar to my ambivalence about VR or realistic CG in general.

                                  1. 4

                                    Yeah, as a programmer, I find the GPTs vaguely annoying because you can’t program them like a normal computer. You have to use natural language to trick them into doing stuff and it’s a skill I don’t want to have to learn.