It has been two years since this paper was published, and large language model’s have become even larger, without the issues identified in this paper being addressed. Trusting Stochastic Parrots might not be wise…
It has been two years since this paper was published, and large language model’s have become even larger, without the issues identified in this paper being addressed. Trusting Stochastic Parrots might not be wise…
I think it’s interesting how comparatively few words humans need to encounter in childhood to reach the level of intelligence they develop. Either the brain is far more powerful than any computer system we have, our learning algorithms are far below the level of efficiency they could achieve, other human sensory stimuli fill in the gap, or the brain is not a computer. The last one seems hard to believe but as seems popular to point out, humans love believing the technology du jour (the wheel, books, mechanical systems, now computers) to be the true nature of reality. Brains could be an entirely new category of technology beyond computers we have yet to develop or conceive.
LLMs have, to a close approximation, read every word ever written by humans. That isn’t close to true of even the greatest geniuses our species has produced.
If you want to continue the rabbit hole of brains being an entirely new kind of technology, you may be interested in this post:
(As well: this response)
Biology constantly shows us how much further we have to go in the depth of our understanding.
My understanding is that the process of training a LLM on a (very) large collection of text is not at all like a person reading. The current LLMs are super impressive and surprising but they have taken a different path from human cognition.
There is a relatively big community of AI researchers who explore techniques to make learning a lot more efficient. This book provides a nice CS-oriented summary [1].
In a nutshell, humans are very good at building abstractions using hierarchies and mixtures of concepts. These map quite nicely to the eponymous models in statistics.
[1] https://probmods.org
I suspect our brains come with a certain amount of implicit knowledge built in and just need to connect it to words. (How else could you communicate the concept of “thoughts” to a child?) This wouldn’t explain everything, but it would give us a head start.
There’s some pretty famous research that was done in remote parts of (at the time) the USSR with people who were not and never had been literate, and which established some fascinating things about the changes in our thinking literacy – even slight literacy – produces. One of which is the concept of words as things, and as things that can be reasoned about, have things predicated of them, and so on. You almost certainly take that and many other similar abstract-concepts-as-things for granted, but you were not born with them.
(if you want to dig more into this, read up on Walter Ong – his work, or even just a good summary or any good bibliography will turn up more things to read on the subject)
Literacy is another topic entirely, although I’m sure it does have dramatic effects on the brain. I’m talking about how the developing brain connects the concept of heavy with the spoken word “heavy”, or the idea of being helpful with the word “help”. Parents chattering at children (describing their behavior to them) and children being little sponges is critical, but I suspect there’s also a pre-built set of templates that are waiting for words to match up to.
Visual sensory input is absurdly high bandwidth compared to text. It’s not surprising that it takes a large amount of text to reach comprehension. If anything, it’s amazing how little input is needed. I suspect this is because text input is relatively high entropy (i.e. information density) compared to vision.
GPT4 is trained on both text and images, and shows considerably improved comprehension. Most likely, adding video and audio input would have a similar affect, although the computational demands would be much, much higher.
Any thoughts along these lines will have to contend with the existence of Hellen Keller, though.
Fair point. Touch, too, is much higher bandwidth than text. But also much lower information density. Smell and taste are less clear.
Edit: but also agree with others that there is some built-in bias (via evolution) on what humans can efficiently learn. Just as there is some built-in bias of what any given artificial neural network architecture can efficiently learn.
I’m referring to LLMs as Stochastic Parrots from now on
So many refer LLM as “Stochastic Parrots”. It’s an insult to parrots. It is known that parrots are very intelligent. And Bing agrees,
I understand some researchers want to feel superior in their “human level intelligence” compared with what an LLM exhibits. But I’m very confused as what “Stochastic Parrots” even mean here. Surely they don’t mean real parrots that speak as well as an LLM, do they? If so, I would call it an AGI. If not, please stop using this phrase. I don’t care what an LLM is, but Parrots are intelligent.
I think it refers to ability of some parrots to “parrot” human speech, rather than a reference to their intelligence which is far greater than LLM’s…