It’s obvious that, inasmuch as LLMs learn language, they do it in a fundamentally different way from humans. Humans learn language from a notably small corpus, surprisingly small and in need of explanation, which is what the Chomskian school of formal linguistics is/was trying to provide. LLMs learn language from an a corpus that has to be absolutely ginormous for them to function at all, and even then, they have defects on the semantic level that are unlike anything seen in humans, even in toddlers. I’d say that LLMs are at best orthogonal to Chomsky’s approach to language - they might, ideally, show a viable path to language acquisition that has nothing to do with human language acquisition.
Less charitably, the very size of the corpus required for an LLM to appear to have learned a language (with the obvious defects you mention, though if a model developed that made those more subtle or even eliminated them, the point would still stand, IMO…) actually serves to validate the mental model associated with Chomsky’s approach.
Saying that there’s another useful theory of language acquisition that applies to something other than humans does not seem to dent Chomsky’s approach in any way, let alone refute it. And I think the paper might be more interesting (but less clickbaity) if it took the tact of supplementing rather than claiming to supplant that approach.
It may also be the case that at worst LLMs are orthogonal and at best human language acquisition is some combination of Chomskian & probabilistic linguistics.
Although I am immediately inclined to disagree with Steven (I haven’t read the paper in full yet, but the beginning makes only observations that I don’t find convincing), this is not a reasonable response.
When looking at a new paper, there are two perspectives one can take:
One “inside” perspective would be to say that the standing of the author doesn’t matter, the arguments should be evaluated on their own merits.
A second, more “outside” perspective, would argue that you have to ask whether the critic has the standing to be taken seriously. It’s not absolutely misguided. When someone claims they’ve refuted a major theory or solved a major problem, you do a gut check about the presentation, the author, the theory in question, and sometimes you toss the paper in the crank pile without reading. Very rarely, a paper in the crank pile is worthwhile, but life is short.
I take it you’re taking this second perspective. Unfortunately, you’re still completely misguided. Steven is a professor at UC Berkeley, he got his Ph.D. in linguistics from MIT (where Chomsky teaches). We also weren’t friends, but went to the same undergrad, so I also know that everyone I knew in linguistics (including folks in grad school at UMass and MIT) thought highly of him.
As for the theory he’s targeting, while Chomsky is incredibly influential, his theory of universal grammar is far from universally agreed upon in linguistics. A huge number of articles must be written every year arguing against his theory. Virtually all of them are written by people who do not have Chomsky’s stature in the field. That does not mean that they are ignored.
None of this in any way means that Piantadosi is right. Hell, I had an immediate response that this paper is gonna be wrong. It’s just that I am confident your dismissal adds nothing to this conversation.
Do they really refute anything? It seems the main thing this paper is presenting is that Chomsky (and anyone studying formal languages) argues for the separation of syntax and semantics, whereas in LLMs they are not distinguished. Theories are points of view, not facts. They create a worldview, within which arguments can be formed based on rules.
Is the LLM approach not just a different theory, with different rules and a different world view?
From what I’m reading, they seem to prove the Chomsky worldview correct with the “colorless green ideas” example. The generated similar sentences to that fail to capture what’s really going on in that sentence semantically, they only copy the structure of the sentence.
I get that Chomsky has thrown plenty of stones the way of LLMs, but the whole language of this paper just comes off as defensive and trying to prove itself right in contrast to the entire field of formal language theory. I highly doubt that what we have built up about formal languages is “wrong.” It’s just another useful model.
Isn’t Chomsky’s whole thing about the Poverty of the Stimulus relevant here? LLMs require vast amounts of training data in order to produce grammatical sentences, while human children get by with comparatively very little. That suggests to me that there might be a different process of learning grammar in LLMs compared to humans.
It’s obvious that, inasmuch as LLMs learn language, they do it in a fundamentally different way from humans. Humans learn language from a notably small corpus, surprisingly small and in need of explanation, which is what the Chomskian school of formal linguistics is/was trying to provide. LLMs learn language from an a corpus that has to be absolutely ginormous for them to function at all, and even then, they have defects on the semantic level that are unlike anything seen in humans, even in toddlers. I’d say that LLMs are at best orthogonal to Chomsky’s approach to language - they might, ideally, show a viable path to language acquisition that has nothing to do with human language acquisition.
Less charitably, the very size of the corpus required for an LLM to appear to have learned a language (with the obvious defects you mention, though if a model developed that made those more subtle or even eliminated them, the point would still stand, IMO…) actually serves to validate the mental model associated with Chomsky’s approach.
Saying that there’s another useful theory of language acquisition that applies to something other than humans does not seem to dent Chomsky’s approach in any way, let alone refute it. And I think the paper might be more interesting (but less clickbaity) if it took the tact of supplementing rather than claiming to supplant that approach.
It may also be the case that at worst LLMs are orthogonal and at best human language acquisition is some combination of Chomskian & probabilistic linguistics.
Non peer reviewed article trying to demolish none other than Chomsky. Needs more than that.
Although I am immediately inclined to disagree with Steven (I haven’t read the paper in full yet, but the beginning makes only observations that I don’t find convincing), this is not a reasonable response.
When looking at a new paper, there are two perspectives one can take:
One “inside” perspective would be to say that the standing of the author doesn’t matter, the arguments should be evaluated on their own merits.
A second, more “outside” perspective, would argue that you have to ask whether the critic has the standing to be taken seriously. It’s not absolutely misguided. When someone claims they’ve refuted a major theory or solved a major problem, you do a gut check about the presentation, the author, the theory in question, and sometimes you toss the paper in the crank pile without reading. Very rarely, a paper in the crank pile is worthwhile, but life is short.
I take it you’re taking this second perspective. Unfortunately, you’re still completely misguided. Steven is a professor at UC Berkeley, he got his Ph.D. in linguistics from MIT (where Chomsky teaches). We also weren’t friends, but went to the same undergrad, so I also know that everyone I knew in linguistics (including folks in grad school at UMass and MIT) thought highly of him.
As for the theory he’s targeting, while Chomsky is incredibly influential, his theory of universal grammar is far from universally agreed upon in linguistics. A huge number of articles must be written every year arguing against his theory. Virtually all of them are written by people who do not have Chomsky’s stature in the field. That does not mean that they are ignored.
None of this in any way means that Piantadosi is right. Hell, I had an immediate response that this paper is gonna be wrong. It’s just that I am confident your dismissal adds nothing to this conversation.
Do they really refute anything? It seems the main thing this paper is presenting is that Chomsky (and anyone studying formal languages) argues for the separation of syntax and semantics, whereas in LLMs they are not distinguished. Theories are points of view, not facts. They create a worldview, within which arguments can be formed based on rules.
Is the LLM approach not just a different theory, with different rules and a different world view?
From what I’m reading, they seem to prove the Chomsky worldview correct with the “colorless green ideas” example. The generated similar sentences to that fail to capture what’s really going on in that sentence semantically, they only copy the structure of the sentence.
I get that Chomsky has thrown plenty of stones the way of LLMs, but the whole language of this paper just comes off as defensive and trying to prove itself right in contrast to the entire field of formal language theory. I highly doubt that what we have built up about formal languages is “wrong.” It’s just another useful model.
Isn’t Chomsky’s whole thing about the Poverty of the Stimulus relevant here? LLMs require vast amounts of training data in order to produce grammatical sentences, while human children get by with comparatively very little. That suggests to me that there might be a different process of learning grammar in LLMs compared to humans.
The “Chapter 1” bit seems very ominous.