However, it exposes a very interesting phenomenon in many people's thinking in computing: that many people have belief systems that are religious in nature. They do not care about evidence, rationality and logical reasoning: they just believe.
So, for instance, I've found that the Unix world in general is now so large, and so old, that many people have erroneous folk beliefs that are set by 50+ years of tradition. They're not faith: they just don't know anything else. Their little domain is the whole world because they've never known there was anything else.
That's human nature. It's sad -- there's so much more out there -- but it's how people work.
So, for instance, a few random erroneous beliefs from the Unix tradition are:
all OSes have "files" in "filesystems" with a hierarchical tree structure
programming languages come in two forms: interpreters, and compilers. There are profound important differences between these.
the basic programming language is C and everything else is implemented in C.
C is a low-level language, meaning that it's close to how the processor works.
None of them are even slightly close to general truths, but they are axioms in Unix.
But that's just ignorance.
The faith thing is much more frightening to me.
That LLMs are artificial, meaning that they were made by people. (They weren't, aren't, and can't be.)
That they are intelligent, meaning that they can think in any way at all. (They can't, but they fake it in ways some find compellingly persuasive.)
That they can program. (They can't, but they can make up text that looks like code.)
That they can analyse and improve themselves. (Nothing can.)
That bigger models will make them better at what they do. (This one just exposes basic ignorance about very simple statistics, such as scaling effects.)
That LLMs are in any way a step to reasoning, thinking software. (Pure magical thinking, based on lack of understanding.)
The thing is that I now continually encounter smart, educated people who believe in these magic beans.
There's two things I want to push back against in your comment.
1) The magic bean believer strawman
So much of the LLM discourse I see (especially here on Lobsters) is polarized to the point where you're either a believer or a denier. I see people who make very reasonable and tempered claims about the utility that LLMs provide them (sometimes with evidence!) that are blanket rejected because "ChatGPT can't count its own toes correctly" or whatever new chicanery people have devised to get it to spit out a wrong answer.
Yes, there is an LLM cargo-cult. Yes, there are CEOs aplenty who want to sell you magic beans. Yes, I am sick of it too. But can we please reserve our hate for the myriad people and ideas deserving of it and openly hear out the ones who aren't coming with this kind of agenda? The ones who honestly believe they have found something helpful?
2) LLMs ~ magic beans
It's not clear to me whether you're arguing in your comment that LLMs have no merit whatsoever, but since that's a common sentiment I see and want to rebut, you'll have to forgive me if I have inferred incorrectly.
The other thing that bothers me is the almost religious rejection of any LLM results and realities. Correct me if I'm wrong, because I only speak from vibes, but I feel the anti-LLM sentiment when copilot came out was "LLMs will never write code." Advent of Code this year, for example, has had sub-30 second submissions on the leaderboard – if this is not evidence that LLMs are capable of some kind of programming, I don't know what is, because humans surely cannot read the whole prompt and code in that much time. And now the sentiment I see has shifted to "well, LLMs can write some code but it's not complicated," seemingly in denial of these previous claims.
I want to remind/inform whoever's reading this that in the decades-old field of program synthesis, the poster child for the longest time was FlashFill, which generates (simple) Excel formulas from examples. There simply wasn't any usable general-purpose tool for program synthesis (regardless of hallucinations or syntactic inaccuracies or …). Now a large number of synthesis papers are (in a very simplistic and reductionist approximation) LLM + special domain.
You can debate whether LLMs in their current form have legitimate utility, but this debate becomes personal and I expect colored by your own perception of how they help you (placebo/nocebo). I think it's too reductionist to write them off entirely.
These are but a brief summary of my thoughts on LLMs and program synthesis, I hope to getting around to writing more in the new year…
But can we please reserve our hate for the myriad people and ideas deserving of it and openly hear out the ones who aren’t coming with this kind of agenda?
The problem with this framing is that you're only looking at the ends, when for many of us the means play a part in the resistance.
If you bring an LLM that was only trained on content whose authors actively assented to inclusion, and where no companies were seriously considering building their own nuclear reactors because it takes so much power to run, and where there's some hope of ownership and control of the LLM by end users instead of just large technology/surveillance companies, then sure! I'm all ears!
Alas, there is precious little of that in the space where Copilot and ChatGPT and Claude and so on are playing.
Completely agreed. I'm not interested in having discussions about trying to invent useful applications for this technology, because even the small-scale models that hobbyists can run on their home computers are produced at massive expense by extremely unethical actors out of datasets which were harvested without consent and then legally encumbered.
Supposedly innocent tinkering with LLMs furthers the goals and rhetoric of absolutely monstrous entities and helps them justify their abuses of the commons. Building hobby projects on LLaMa and writing effusive blog posts about cheating at code golf, automatically synthesizing a fart app for your phone, or accomplishing any other number of trivial tasks with help from a Claude-powered rube goldberg machine is doing free marketing for the generative "AI" merchants and cloud vendors.
LLMs are completely radioactive from many ethical angles even before you start digging into the harmful ways they're being applied today or (as this article focuses upon) their complete unsuitability as a building material for reliable, efficient, or generally trustworthy software.
LLMs are completely radioactive from many ethical angles even before you start digging into the harmful ways they’re being applied today or (as this article focuses upon) their complete unsuitability as a building material for reliable, efficient, or generally trustworthy software.
Then (as someone who believes these points should be made) I implore you to focus on the first half (not struck-through) part of what you're saying. I think it does your argument no good to incorporate claims that are deniable, especially if they apply the fallacious reasoning I discuss in my parent comment.
You don't even need to open the door to arguing whether in two years ChatGPT can write Minecraft if its existence today already constitutes as significant of a problem as you claim. I think it's good to have people thinking critically about these tools (ethically and technically), but thinking critically means not getting swept up in the LLM hype and the anti-LLM hype.
I think it is good that companies are building their own nuclear reactors to power datacenters for generative AI applications. We need way more of this kind of thing, cheap energy makes all of society more prosperous and we get cheap energy from having lots of nuclear power plants generate it.
Ownership and control of LLMs by end users is important and it's a genuine concern that we don't get locked in AI systems controlled by a small number of companies. But this is not really a different problem than a small number of companies controlling proproetary software platforms used by huge swaths of the population (i.e. Google and Meta and Twitter and Apple existing).
I think it is good that companies are building their own nuclear reactors to power datacenters for generative AI applications
How many are actually doing this? Nuclear power plants take many years (often a decade or more) to build. Most of the announcements I’ve seen have been ‘we aren’t a climate disaster, look we have this long-term strategy to be energy independent. We’re not actually funding it at the level required to be realistic, but let us burn vast amounts of coal now, we promise it’s a temporary thing!’.
Even if a SMR is built, it still needs huge volumes of clean water to operate and cool. It's never going to be a little container-sized cube that magically emits electricity, like a portable fossil fuel generator.
Then make it illegal to burn coal to incentivize building the nuclear power plants faster (and reduce the amount of regulation on them); I do not want to sacrifice humanity's ability to harness prodigious amounts of energy on the altar of the climate. This isn't even about LLM datacenters specifically, I want nuclear power plants powering aluminum smelters too. Or maybe photovoltaic solar is actually cheaper if we just pepper the earth with panels, in which case I support doing that, and the electricity can go into LLM datacenters as easily as it can go into anything else.
What I don't want is for any human endeavor that uses a lot of electrical energy to get labeled a "climate disaster" by people who want to shut it down - least of all because scrubbing CO2 from the atmosphere itself is something that's gonna require a lot of electrical energy.
But this is not really a different problem than a small number of companies controlling proproetary software platforms used by huge swaths of the population (i.e. Google and Meta and Twitter and Apple existing).
I'm glad you agree it's a problem! The difference, I think, is that I'm not constantly hearing about how I should learn to stop worrying and love Gmail.
You hang out in very different places than I do, it appears. That's interesting to realize... I don't even know what it would be like to be in a place where that view is low-status. It felt like the mainstream belief within the broader software community, when I was growing up. It still feels like a common belief to me. It's really interesting to hear a perception to the contrary; thank you.
As a bit of extra flavour… I, and the people I grew up with, shared the same belief you did: we believed that software like Gmail is unethical and that we were taking the moral high ground by believing that.
For me, though, there was a moment along the way where… I still love open source software and wish more of the world was made up of OSS, but also came to the conclusion that many of the people I grew up with who held those beliefs the strongest… weren’t really going anywhere. Many of them are still exceptionally talented programmers but they’ve lived a life where they’ve consistently struggled to make enough money to house and feed themselves. Whether they’re considered low status because of their beliefs or because of their relative poverty or due to other lives choices is hard to say but it’s pretty tough to argue that they’re doing well in life.
In my life now, from where I’m standing, it seems like the general broad perspective towards OSS is mostly indifference. Most people I know run their production software on Linux, some on Windows, and none of them really do it for OSS reasons but rather because Linux is a good platform. They don’t really care that it’s open source, just that it works. I’m actually feeling a bit sad writing this.
but also came to the conclusion that many of the people I grew up with who held those beliefs the strongest… weren’t really going anywhere
I don't think it makes sense to evaluate any idea based on perceived success* of those that hold it. Especially when that idea is likely to make you avoid chasing the riches of the tech industry.
To be blunt, reading that it sounds to me like you were willing to compromise those ideas for money and don't value the same things as the people "not doing well* in life."
* this is subjective as it depends on personal goals only those people can judge themselves
I agree most people are apathetic to things being FOSS, and that is quite sad but also why activism is needed. Not only for FOSS but all subjects; the status quo has strong inertia, if not players with incentive to maintain it.
To wrap back to the original subject, I believe Gmail is unethical mostly due to Google's data pillaging and federation busting. Gmail not being FOSS is part of it, but far from the main reason, and mostly orthogonal: I believe VS Code is unethical even if it is mostly OSS and VS Codium exists.
To be blunt, reading that it sounds to me like you were willing to compromise those ideas for money and don’t value the same things as the people “not doing well* in life.”
100% will agree that I did compromise on those ideals in part for money but in the bigger picture for happiness as well.
The real tragedy with respect to the “not doing well in life” part is the magnitude of that. I agree it’s relative but it makes me so sad to see the some of the brilliant people I knew in school posting on Facebook (there’s a certain irony there…) about how their night-shift gas station job sucks and that they have to kick out another roommate for stealing or being abusive. It’s not just that they’re “not doing well” on my personal scale but that they also seem genuinely unhappy themselves.
But… this is all just one small community. I’m sure it’s not a representative sample.
The problem with this framing is that you’re only looking at the ends, when for many of us the means play a part in the resistance.
That's fine to articulate, but my reply is to ~lproven's comment which makes no mention of the ethical considerations for using LLMs.
To be clear: I'm not saying that discussions about the ethics of LLMs should be silenced (I am inclined to believe otherwise). But I am saying that even if you think you have the moral high ground for not wanting to use LLMs, this doesn't entitle you to misrepresent what they're capable of. (not accusing you specifically, but I hope you get what I'm saying)
Put differently, I don't like the simultaneous standpoints I see people take of "LLMs are bad for authors and the environment" and "LLMs can't produce code, LLMs are stupid, etc." The first point is much more damning and — perhaps more importantly — easier to verify facts for. I don't see any good reason for denying evidence or inventing strawmans to support the latter point.
They do not care about evidence, rationality and logical reasoning: they just believe.
I feel the same but like... the opposite lol
That LLMs are artificial, meaning that they were made by people. (They weren’t, aren’t, and can’t be.)
I don't even understand this. Is it even a question? Of course they are artificial.
That they are intelligent, meaning that they can think in any way at all. (They can’t, but they fake it in ways some find compellingly persuasive.)
You're gonna have to define "intelligent". Depending on the context, LLMs are obviously intelligent, but under other definitions it clearly is not - in fact you'll find that many people have wildly divergent definitions. A panpsychist is not likely to say they're intelligence, a functionalist may. Even by some pretty rigorous definitions. This isn't a religious question, it's a metaphysics question. Is intelligence the ability to reason? Is it a sufficient set of justified beliefs/ knowledge? Does it imply abstract properties? Is it an abstract property? Is it physical? Emergent? Seriously, it's incredibly reductive to say "LLMs are not really any kind of intelligence".
That bigger models will make them better at what they do. (This one just exposes basic ignorance about very simple statistics, such as scaling effects.)
Also confusing since that has been the case. The question is really just a matter of the limits of this scaling. It's sort of like saying "gzip isn't going to benefit from a larger disk because at some point compression has information theoretic limits" well yeah, but the bigger disk still helps. I'm not really sure what you're trying to get at here though, maybe you can be more specific.
That they can program. (They can’t, but they can make up text that looks like code.)
I mean... this is just a weird definition to me. They use statistical reasoning to generate valid programs. I guess you think that's not programming?
That they can analyse and improve themselves. (Nothing can.)
WOW lol sorry but I'm out. You're seriously going to accuse people of religious thinking and then make these kinds of assertions? Like there are entire metaphysical theories devoted to these questions that you're so glibly dismissing. Nothing can analyse and improve itself? That is such a wild claim to just assert like this.
That LLMs are in any way a step to reasoning, thinking software. (Pure magical thinking, based on lack of understanding.)
If we define reasoning as using evidentiary inputs to produce conclusions, LLMs reason. Statistical reasoning is reasoning.
These arguments (read: unjustified assertions) are so weak and you seem to not even realize the metaphysical commitments you're making by pushing this worldview.
If you want to have a reasonable discussion, by all means please do. I see so few discussions that look reasoned on this site and it's a bummer. Let's talk about knowledge, let's talk about reasoning. Let's do it! Let's ditch these assertions, let's ditch the question begging, let's have a little epistemic humility, okay?
Yeah, it doesn't really matter whether LLMs have intelligence and rationality. They can make our lives easier by solving some problems, just like cars and computers, and that's enough. That being said, I also agree with the article that LLMs need to be improved in terms of reliability and explain-ability.
Yeah, it doesn’t really matter whether LLMs have intelligence and rationality.
This black-and-white language leads to confusion.
I recommend the following definition: Intelligence is the ability for an agent to solve a task. The degree to which such abilities span different areas is the degree to which we call it general intelligence.
Intelligence is the ability for an agent to solve a task.
Nope.
Place a blind invertebrate into a box with a particle of smelly food. Wait. It will move around in a semi-random walk but it will find the food. Problem: solved. Was intelligence used? No. The algorithm is so trivially easy, I've solved it:
Move in long straight lines until you smell food, then move shorter distances and turn more.
That's it. That is the algorithm. It works.
Pools of water connected by pipes: find the lowest level. It works.
In any case, researchers Shane Legg and Marcus Hutter have made the case that intelligence includes the following features:
Intelligence is a property of some entity or agent that interacts with some form of environment
Intelligence is generally indicative of an entity’s ability to succeed (by a given set of criteria) at a particular task or achieving a stated goal
When speaking of an “authentic” intelligence, there is an emphasis on learning, adaptation, and flexibility within a wide range of environments and scenarios
The only thing you did with your comment here is do an appeal to authority.
How about this: nobody is an authority on what is an intelligence and what even is reasoning. People get a PhD and start pretending to know, that’s the crux of it, let's not sugarcoat it please. That applies to every single non-practical discipline, philosophy and many others included.
Also your comment came across as disingenuous because you first offered a blurry definition of intelligence and then, when called out, retreated into the appeal to authority and that a need for nuance is needed.
I don’t see how the latter follows from anything at all. No need for nuance, whatever that means in this situation even; we need good working definitions of the sort “what constitutes a fusion reactor?” for example.
All the discussion about what passes for “AI” these days is just sad to watch and that includes your comments. It resembles theological discussion first and foremost.
Surely technically-inclined people can do better? Inputs, expected outputs, process, all that? You know, engineering?
Personally, your comment comes across as unkind. Maybe just a bad day? Are you willing to try a more constructive and charitable direction?
Some responses:
Remember the thread context; I was criticizing some flaws in a definition someone else offered. My definition addressed the flaws I pointed out.
I “retreated” above? I’m happy to admit if I made a mistake, but where are you getting this?
It isn’t fair, accurate, or charitable to say that “appeal to authority” is all I did.
I pointed to what I think are some good jumping off points. Did you read any of them (Russell, Hutter, etc)?
Do you know a better definition of intelligence? … and why do you think it’s better? (You can see in my other comments the need for a definition that isn’t human centric.)
I’ve seen the anti-PhD venom before. I used to say the same kind of thing decades ago. It was ignorance on my part. This lessened as I interacted with more people with Ph.D. experience and as read more machine learning and CompSci literature.
No experts? There are people with more and less expertise. Acting like there are no experts is an unhelpful exaggeration.
If you read the comments I’ve written, you’ll see it is difficult to place me into any simple categories regarding AI.
I find it bizarre that you think my comments are anything like theology. Offering a working and operational definition of intelligence does not theology make.
I’ve also seen the anti-philosophy trope before. It is unfortunate and self-limiting. The classic response here is: many other sciences and disciplines were birthed from philosophy, but philosophy rarely gets credit. Yes, some philosophy is painful to read. One usually has to put in a lot of effort searching, thinking, writing, and discussing to reap the biggest benefits. Asking for reading recommendations from people you respect is a good start.
Well, it might be cultural background on my part because I am not used to dance around disagreements, and as I get older this is less and less likely to ever change. Not my intention to offend, mostly to pull you away from what I perceive is a comfortable and maybe even complacent position.
I did simplify your comments, that much is true, and 99% of the reason is that the "AI" discussions inevitably devolve into "But how do we know what intelligence is? We might already be witnessing it but are disparaging it!" which, my turn to say it, I view as extremely unproductive, not helpful for advancing any discussion, and basically serving no other purpose than to congratulate ourselves how smart we are, and to offer a plethora of very out-there "what if"-s.
If you believe you are doing something more than that then I'd love to see more concrete claims and proofs.
I have not read philosophy on the mind and intelligence. I tried. I found it unproductive and very rarely did I stumble upon a lone article (not a book) where somebody actually attempted to invent / classify a framework in which we should be thinking about what mind / intelligence are. Everything else I attempted to read did read like empty hand-waving to me.
If you are willing, I'd like to get back to some semblance of a topic: do you believe LLMs can "reason" or are "intelligent"? Do you believe we are mistakenly putting them down while they, the poor things, are the next-stage-of-evolution, marginalized and oppressed artificial life forms that are just misunderstood? If not, what, apropo, was your point?
Do you believe we are mistakenly putting them down while they, the poor things, are the next-stage-of-evolution, marginalized and oppressed artificial life forms that are just misunderstood? If not, what, apropo, was your point?
You’re being a bit of a dick here on multiple fronts, and misspelling the italicised apropos in the final sentence doesn't shake the look.
Misspelling can happen to anyone. Shame that the glorious and almighty "AI" never improved autocorrect in phone keyboards, right? I'm sure that requires godlike intelligence though, so it's excused. But hey, it actually "understands" what it does and it's obviously intelligent. Surely.
And I'm not interested in philosophical discussions unlike a lot of people who can't help themselves every time the "AI" is mentioned.
I'm interested in seeing proof that their downright religion-like beliefs have any rational foundation.
Alas, that kind of expectation is misguided. Faith doesn't require proof to exist, as we all know historically.
If challenging belief makes me a dick then I'm okay with it. I was still never answered in a satisfying manner.
My conclusion is that this is a cozy topic for many. Mention "AI" and they all pull up the cigar and the 20-year old whiskey, and they're all suddenly decorated philosophers who are perfectly qualified to speculate and to present conjectures as facts while they want us to believe that their entirely unrelated Ph.D. makes them educated on a topic that absolutely nobody has ever solved.
So yeah. Believe what you will. But recognize it's only that: a belief.
Well, I think it matters a lot if LLMs have intelligence because the implications are pretty huge. Kind of like, "are there objective moral goods" - if we could answer that question we could rule out all sorts of metaphysical theories about the universe, and that seems valuable to me. Practically, and I think to your point, whether it's intelligence or a facsimile of it (assuming this distinction even makes sense, which is a HUGE assumption!), as long as the results are the same it isn't important (in terms of how it's used).
I also agree with the article. I thought it was well written and makes sense - I thought the idea of breaking down models into testable components was particularly interesting. The comment I responded to doesn't even seem related to the contents, which I also thought was ironic since it was a plea for rationality and informed commenting.
Well, I think it matters a lot if LLMs have intelligence because the implications are pretty huge.
There’s an even further metaphysical question that goes with that… how do we even define intelligence? What’s the threshold? Is a tree intelligent because it grows its roots towards water? Are bacteria intelligent? Fish? Cats? Dolphins? Horses?
Are current ML models like any of these? Or are they more like https://en.m.wikipedia.org/wiki/Clever_Hans? Or was Clever Hans actually intelligent, just not in the way that it was claimed? (He couldn’t do arithmetic but he could very accurately read subtle human signals)
All really interesting things to ponder on holidays :)
Yes/no questions fall flat quite often. For many interesting subjects, including intelligence, consciousness, justice, fairness, etc., there are better framings.
Kind of like, “are there objective moral goods” - if we could answer that question we could rule out all sorts of metaphysical theories about the universe, and that seems valuable to me.
We already have human-created machines that try to answer these questions. They're called religions. Adding another one with a LLM dash of paint will probably not resolve anything.
We already have human-created machines that try to answer these questions. They’re called religions.
No, religions don't inherently try to answer these questions. These are questions that fall into the domain of philosophy. Whether moral goods exist, their nature, etc, is not a religious question but a metaphysical one.
My statements stands; the ability to answer questions about the mind would indeed hold value, it would resolve many open questions, rule out some metaphysical theories, make others more or less unlikely, etc. It would potentially have very direct impact on human life - theory of mind is essential to theory of personhood, for example.
OK. I read your original comment as something akin to Platonism - there are eternal truths that are hidden from us. How a machine trained on the sum of humanity's writing on these questions would be able to reveal them was unclear to me.
I see. No, I'm not expecting an LLM to reveal its truths. I'm saying that our investigation into knowledge, perhaps through exploration of technologies of LLMs, will reveal the truth.
Again, going back to Descartes, one of his main arguments for human exceptionalism as well as mind body dualism was that humans can speak. He suggested that machines could move, look like animals, look like humans, etc. He suggested that one could not differentiate a monkey from a robot monkey. But he maintained that language was the indicator of a true mind - so a machine could never master language.
LLMs at least provide evidence against that. You can argue about whether they're a true counter example, but they have are evidence against a Descartes theory of mind. As we continue to push forward on technology we may have stronger evidence for or against other theories of mind.
Let me try to understand here: because LLMs disproved an extremely flawed hypothesis by a scientist who is a product of his time, this means... what exactly? Can you help me here? That LLMs possess mind, or is it something else?
. because LLMs disproved an extremely flawed hypothesis by a scientist who is a product of his time,
No, not at all. I was giving an example of how our ability to create machines has historically changed our understanding of our theory of mind. It was an example of how producing new technologies can help us evaluate existing theories about things like intelligence. Applying it ot a very old, well understood (and unpopular, therefor not contentious to apply to) theory was my way of giving such an example.
As I said, I think that as we generate technologies like LLMs we may be able to generate evidence for or against theories.
Maybe it is an example of how producing new tech can help us evaluate existing theories, yes... if those theories are not as old and almost laughable. Because comparing to them is a classic "tearing down a straw man" debate.
There is no straw man here, I don't think you know what that term means. It is chosen explicitly because it is not contentious to say that dualism is rejected, and to show how one could use LLMs as evidence. That it is already rejected (largely) is the benefit of using this example.
Maybe it is an example of how producing new tech can help us evaluate existing theories, yes
Literally all I was doing. Soooo we're good? Same page.
Artificial, meaning, built by artifice, created using the skills of an artificer.
LLMs are not built by humans. LLMs are built by software built by humans, running in large-scale clusters in datacentres. LLMs are a multi-dimensional grid of statistical weights of relatedness of words, with millions of dimensions.
The point being that humans didn't construct those databases, can't read them, can't analyse them, and cannot modify them.
The humans built the software that built the models. Humans did not build the models, and can't.
The tools that built the models are artificial. The models are not.
You’re gonna have to define “intelligent”.
Cambridge dictionary: showing intelligence, or able to learn and understand things easily
Intelligence: the ability to learn, understand, and make judgments or have opinions that are based on reason
Merriam-Webster: having or indicating a high or satisfactory degree of intelligence and mental capacity
Intelligence: the ability to learn or understand or to deal with new or trying situations; the ability to apply knowledge to manipulate one's environment or to think abstractly as measured by objective criteria (such as tests)
These are not abstruse technical terms.
LLMs cannot reason, understand, deal with new situations, etc. ALL they can do is generate text in response to prompts.
They cannot count. They cannot reason. They cannot deduce. But they can produce a modified version of text which does those things.
It is a fallacy of thinking to leap from "it produces text which sounds smart" to "it is smart".
I guess you think that’s not programming?
It isn't. Something that can't tell you how many Ms there are in the word "programming" can't program.
You’re seriously going to accuse people of religious thinking and then make these kinds of assertions?
Yep, 100% am. Refute me: prove me wrong. Falsify my statements.
Nothing you've said falsifies my points. All you are doing is mocking and trying to redefine terms.
LLMs are not built by humans. LLMs are built by software built by humans, running in large-scale clusters in datacentres. LLMs are a multi-dimensional grid of statistical weights of relatedness of words, with millions of dimensions.
This is just a really weird standard. If I build a chair from Ikea parts, did I not build a chair because someone else built the parts? What's the point of this definition?
A good simple explainer is the Financial Times’s one here and a more in-depth one is Stephen Wolfram’s one here. Just in case you were about to accuse me of not knowing what I am talking about.
It's not that you don't understand the technology, it's that you're applying ridiculously strict terms to it and acting like anyone who doesn't take those as gospel truth must be experiencing irrational, religious thinking. Nothing in those pages is going to justify a flat assertion that an LLM is not "artificial" tbh but even if they justified it I don't think it matters - I mean the stakes on this couldn't be lower.
The humans built the software that built the models. Humans did not build the models, and can’t.
Shrug. I don't think this matters, it's just pedantic. "A human built a machine that built the thing" okay. I've never programmed because actually someone else built the keyboard and the compiler that produced the assembly, so no, not programming. I'm a fraud, I suppose.
These are not abstruse technical terms.
Dictionary definitions are fine but I think if you follow the citations there you'll find they're often cyclic. They're also not some hard truth. These are metaphysical concepts with entire papers dedicated to breaking them down. Pointing to the dictionary as an authority isn't a strong argument.
They cannot reason.
I disagree and I've already justified why.
It is a fallacy of thinking to leap from “it produces text which sounds smart” to “it is smart”.
It's not a fallacy it's the foundation of functionalism, which is a metaphysical theory worth taking seriously.
Something that can’t tell you how many Ms there are in the word “programming” can’t program.
Baseless. Why should we connect these two things? I can't factor massive primes in my head but I can program. Why should I connect these two things together? Again, LLMs can produce valid programs, so I think it's on you to justify why that isn't programming since it's intuitively the case that it is.
Yep, 100% am. Refute me: prove me wrong. Falsify my statements.
I provided theories counter to your assertions. I mean, should I break down functionalism for you? I brought it up, you can learn about it if you'd like. I brought up panpsychism, abstract properties, etc. I think there's plenty of breadcrumbs if you want to learn the metaphysical foundations of what I've described.
If you really want to provide some sort of formal syllogistic argument instead of just baseless assertions I could probably provide really strong, rational, well researched arguments from people who study these things explaining why someone could rationally reject your premises or conclusion. Like, my entire point is that what you think are "rational" premises are totally justifiably rejected by people.
All you are doing is mocking and trying to redefine terms.
I'm not redefining terms lol these terms all have varied definitions! That's my point. A functionalist and a panpsychist will have radically different definitions of intelligence. If you want to plainly assert things like "nothing can ever analyse itself" well jesus christ dude that's a massive metaphysical commitment that you are making. I'm not the one making positive assertions here, I'm not the one accusing others are being irrational.
Nothing you’ve said falsifies my points. All you are doing is mocking and trying to redefine terms.
TBH you're the one who started off with the mocking "everyone else is irrational and 'religious thinking', now here's a list of baseless assertions" sooooo idk, I feel really fine with my response. I don't think your list of assertions requires much "refutation", I can just show that they're incompatible with very reasonable metaphysical theories and so anyone who subscribes to those theories is perfectly justified in rejecting your unjustified assertions.
That's absolutely fine and you are of course completely free to say that.
I have been a skeptic (with a K, which is not the normal spelling in my country) for about a quarter of a century now. But that's when I learned the label: I've had the mindset since roughly when I reached puberty.
There is a whole tonne of stuff I don't believe in that billions of people passionately, fervently believe. Many of them would kill me for it. I don't care.
I have seen no evidence that LLMs are in any way intelligent, that they can reason, or learn, or think, or deduce, or in any way reproduce any of the elements of intelligent behaviour.
Metaphysical or philosophical arguments along the lines of "we don't know what 'intelligence' means' or "what does it mean to 'think' anyway?" are just pointless word games. Have fun with them if you want but I'm not interested.
So, I can happily state my assertions:
LLMs are not "AI". They fail to fulfil both the "A" and the "I".
LLMs are not a pathway to AGI or anything else. They're a linguistic trick with few useful real-world applications, and the construction of ever-more-complex prompts in efforts to constrain them into producing useful output is futile: it's just a vastly inefficient new form of programming, one which can never work because the best it can ever do is a small statistical reduction in the amount of noise in the output.
There are useful real-world jobs for neural networks, for machine learning, for evolutionary programming, and so on. LLMs are not part of it.
We are in a new tech bubble driven by people constitutionally unable to confront hard facts and uncomfortable truths, such as "this does not work, can not work, and will never work."
I think you're wrong, but I don't think there is any evidence I or anyone can come up with to convince you.
That's OK. It's just a passing annoyance. The bubble will pop, there'll be another AI winter just like circa 1970 (+~5) when they worked out the limitations of single-layer neural networks, and then again circa 1985 (+~5) when Unix got cheaper and faster than Lisp.
Personally I am looking forward to it. It is very irritating.
I also don't believe that there's any practical useful application of blockchains, anywhere or for anything, and that all cryptocurrencies, NFTs, Web3 and the entire field is an empty bubble that will implode.
Other things I am happy to tell billions of people they are wrong about...
Supplementary, Complementary and Alternative Medicine. It's all 100% fake. Medicine is what can be proved to work; if you can't, it's not medicine. That's why I use the term: SCAM. It's a multi-billion dollar industry and it's entirely and completely fake from top to bottom.
This one is easy: show it works, and it immediately ceases to be SCAM, it becomes conventional medicine.
Religions. All of them. There are no gods, no soul, no afterlife, none of it. Every believer in every religion alive and who has ever lived: all totally wrong.
I am perfectly happy to recant, if I am provided with objective, verifiable, reproducible evidence of the supernatural.
But until then, I repeat, it is all completely fake, and so is the entire field of LLM-driven "AI."
There is a whole tonne of stuff I don’t believe in that billions of people passionately, fervently believe. Many of them would kill me for it. I don’t care.
None of my arguments are "lots of people believe X so X is true". What I'm saying is that there are very rational, viable, reasonable metaphysical theories where your assertions are either outright rejected or are unlikely, and you saying that anyone who doesn't follow your line of thought is irrational or thinking "religiously" is itself an irrational statement, unaware of the dialectic.
Metaphysical or philosophical arguments along the lines of “we don’t know what ‘intelligence’ means’ or “what does it mean to ‘think’ anyway?”
That is not what I'm saying. Different metaphysical theories will define these terms differently. Again, it's just ignorance of the dialectic to say things like "AI isn't intelligence" - not only is it an unjustified assertion, it doesn't even make sense without defining the term, and it would be great to align whatever your definition is with some sort of established metaphysical theory so that we can know your commitments.
It's very common for people who aren't familiar with actual logic, reasoning, and metaphysics to think that it's just "word games". You're the one saying that people should be rational. You're just not meeting the standard you've set, from my view.
So, I can happily state my assertions:
I'll be honest. Your assertions are lacking. They're informal, lack coherent premises, bring in tons of terminology that you've failed to define, etc. For such bold claims I'd really expect more.
Again, you are the one making the positive claims here, it's kinda on you to do better if you want to accuse everyone else of being irrational.
I also don’t believe that there’s any practical useful application of blockchains, anywhere or for anything, and that all cryptocurrencies, NFTs, Web3 and the entire field is an empty bubble that will implode.
These are just incredibly bold claims. To say that there are literally no useful applications of a technology is extraordinarily profligate.
Religions. All of them. There are no gods, no soul, no afterlife, none of it. Every believer in every religion alive and who has ever lived: all totally wrong.
Yeah, I can't stand this form of atheist. As an atheist myself, I find it quite annoying how modern atheism treat religion as totally irrational. I'm what modern atheists would call a "strong atheist" (garbage terminology but I'm going to speak in the terms that I suspect are more familiar to you) but I think anyone tho thinks that you can't be rational and religious at the same time is just extremely ignorant of the actual papers and research on the topic. I'm not compelled at all by that research but I'm aware of it and understand its legitimacy.
I'll be honest with you, I think you should take some time to learn basic philosophy and metaphysics so that you can understand how much you're oversimplifying things and how bad of a job you've done at presenting anything like an argument. I don't mean this as an insult, I say this as a peer and someone whose views almost certainly align to a degree with yours. You should learn what a basic syllogism is - a good exercise would be tot ake your assertions and break them down into distinct syllogistic arguments that you then defend in terms of evidence and justifications for your premises and a justification for the conclusion following from them.
That said, here's my response to your assertions. It's a bit tricky since you haven't presented any kind of theory or aligned with any existing theory, you haven't defined terms, you haven't justified any beliefs, you haven't put your arguments into any kind of formal logical form like a syllogism or a bayesian inference, etc. This is very "internet argument" level, not what I'd want from someone telling everyone else how irrational they are for disagreeing.
LLMs are not “AI”. They fail to fulfil both the “A” and the “I”.
I rejected this already. I reject your definition of A, you've failed to define I entirely. I've already pointed to functionalism, so if you want to reject functionalism by all means you can feel free to do so but I think it would be quite ridiculous to call all functionalists irrational even if you disagree with them. Functionalism is well defended, even if there are competing theories and great arguments against it - I brought up panpsychism, although I suspect you would reject panpsychism as, despite being compatible with atheism, it does strike me as being less likely under atheism. But that's a whole separate issue.
LLMs are not a pathway to AGI or anything else. They’re a linguistic trick with few useful real-world applications, and the construction of ever-more-complex prompts in efforts to constrain them into producing useful output is futile: it’s just a vastly inefficient new form of programming, one which can never work because the best it can ever do is a small statistical reduction in the amount of noise in the output.
I'm going to break this down because this is actually numerous assertions.
2a. LLMs are not a pathway to AGI or anything else.
Unjustified, lacks proper definition of terms.
2b. with few useful real-world applications
I reject this trivially. I find them useful. Trivial for me to reject this.
2c. and the construction of ever-more-complex prompts in efforts to constrain them into producing useful output is futile:
Possibly justifiable, though you didn't do so. But I would probably grant this if "useful" meant something along the lines of deterministic or "highly reliable" like the article mentions. I mean, I would grant it because the stakes are low and I don't care to argue about it, I don't actually think it's true.
2d. it’s just a vastly inefficient new form of programming
I wouldn't care to reject this because "inefficient" is so subjective and ill defined.
2e. one which can never work because the best it can ever do is a small statistical reduction in the amount of noise in the output.
"Work" isn't defined, "small" is confusing, unclear what your point here is.
There are useful real-world jobs for neural networks, for machine learning, for evolutionary programming, and so on. LLMs are not part of it.
I think this is a really odd take since machine learning algorithms are statistical models. To help you out, what you're looking for here would be called a "symmetry breaker". If you grant that those algorithms are useful but you reject that LLMs are you need to show why.
We are in a new tech bubble driven by people constitutionally unable to confront hard facts and uncomfortable truths, such as “this does not work, can not work, and will never work.”
Honestly, I find this deeply ironic. I find your post highly guilty of what you're accusing others of. I think you should deeply investigate these areas and the rational defenses one can put forward before throwing stones at others like this. You're clearly unaware of the academic discussions around these topics, how arguments are formed, what different types of reasoning are, what evidence is, what rationality and what rational beliefs are, etc etc etc. I think I've provided plenty of information for you to learn about these topics if you'd like to raise the bar for discourse, something I'd personally love to see.
Again, I'm not trying to be insulting. I'm a bit too lazy to be less glib here, I hope that the mere fact that I've taken the time to try to express myself to you shows that I'm willing to engage constructively. I can see that you want to elevate the discourse, that you're sick of irrational beliefs, and I'm extremely sympathetic if not empathetic to that worldview. It just seems that you would benefit greatly from learning about what that has looked like over the course of the last few thousand year and to learn about how many metaphysical theories exist that are arguably very rational.
If you want to just make baseless assertions, go for it. That's how most people talk online. But since you went out of your way to point out how you think others do that, I thought it worth pointing this out to you.
If I were to really dig into your assertions I'd have to do a ton of steel manning of them and expand on them to make them something I could actually try to refute or justify competing theories against. You'll have to forgive me, as much as I'd like to and believe it's worth doing, I don't have the time right now. Maybe I'll write something up at some point. Until then, I'll suggest we just agree to disagree. Cheers.
Unbelievable that this is flagged as "troll" lol this site is so fucking ridiculous sometimes. @pushcx can we please get some kind of system for handling people who erroneously flag content? Disagreeing with me is fine but it is nuts how often I put in significant effort, provide justifications, even reference research and papers, and get flagged because someone doesn't like my post. This has happened way too many times to me.
I not only explained clearly what an argument is, what the current academic conversation looks like, etc, I even went out of my way to go beyond what I think was even reasonable to expect and responded to these assertions despite the fact that they are so poorly expressed - I was unreasonably charitable in my response, especially given the context of this conversation starting through someone stating that everyone else is irrational and won't engage in logic.
You provide exactly nothing except long philosophical essays in technical discussions. It's quite telling that you don't see why you got flagged (for the record, I have no ability to do so on this website but I absolutely would if I could).
You have "rejected" and "proven" exactly nothing as well. All you do is hide behind metaphysical clouds and empty philosophy while pretending to give objective evidence.
Yeah, many of us see through that and do perceive it as trolling. Until you come up with something concrete then you will not convince anyone that LLMs / "AI" are not snake oil and that their next winter is coming up soon and it's going to hit hard and last decades.
I actually can believe that you are fully believing what you say, which would be even sadder. In conclusion of these fruitless and never-going-anywhere "discussions" (because I am 100% convinced you will double down; prove me wrong!) I can only tell you that every now and then it's good to revisit your own bias and bubble. Now THAT would constitute true intelligence in my eyes. ;)
By the way, I used f.ex. Claude with success. It helped me fill the gaps in my knowledge in a niche area that I was not having the patience to invest learning from scratch. I was impressed and I loved the experience.
...But it also demonstrated that to me that people can very quickly just spiral into "no, that's subtly wrong, I need X and you are giving me 0.85*X -- let's keep refining this until I get what I want". I actually viewed the daily prompt limit (free tier) as a very good thing: it forced me stop and revisit some assumptions that I subconsciously did along the session, and for good reason -- turned out I was going in the wrong direction and was given wrong input to test an algorithm with (which really did make me laugh because Claude did not validate that input either).
And in conclusion to that topic: LLMs will at best just remain very useful tools for a bunch of tasks. They are not intelligent by absolutely any meaning of the word except those that are super loose and just love hiding behind clouds of philosophical ambiguity... like yourself.
No need to reply as reading your comments here has convinced me you are not capable of productive replies that evoke true practical observable evidence. But you do you.
You provide exactly nothing except long philosophical essays in technical discussions. It’s quite telling that you don’t see why you got flagged (for the record, I have no ability to do so on this website but I absolutely would if I could).
You can dislike my posts but to say it's trolling is just silly. You can think "wow this guy is so dumb!" but that's not against the rules, it's not trolling.
You have “rejected” and “proven” exactly nothing as well. All you do is hide behind metaphysical clouds and empty philosophy while pretending to give objective evidence.
I've never pretended to give objective evidence? What are you referring to? As for "hiding", what?
Let's remember how this thread started - someone stated that anyone who thinks LLMs aren't a scam is guilty of being irrational and "religious thinking". That's the first comment. I justified why I reject that by pointing to many live, well thought out, rational metaphysical theories that would reject this. This isn't hiding or "empty philosophy", it's a direct contradiction to the idea that anyone disagreeing is irrational or religious thinking.
Until you come up with something concrete then you will not convince anyone that LLMs / “AI” are not snake oil and that their next winter is coming up soon and it’s going to hit hard and last decades.
I have zero requirement to do this. The positive assertion made was that anyone who believes that LLMs / AI are not snake oil are irrational. All I have to do is show that that's not true by showing live models that are rational.
I can only tell you that every now and then it’s good to revisit your own bias and bubble
I wonder what it is you even think that I believe? I've made almost no commitments in this discussion because, and I've said this a few times, I'm not the one making the positive assertions.
And in conclusion to that topic: LLMs will at best just remain very useful tools for a bunch of tasks. They are not intelligent by absolutely any meaning of the word except those that are super loose and just love hiding behind clouds of philosophical ambiguity… like yourself.
I mean, lol. Congrats but according to the person I was responding to you are irrational and guilty of religious thinking because you think that LLMs have any use at all. So... what is it you disagree with, exactly?
Anyway you've just done what the other poster has done. You say "They are not intelligent" okay well that's a fine opinion? You're not justifying it. I guess you think justifying anything would be "hiding behind clouds of philosophical ambiguity" idk it's super irrelevant because the premise asserted by the author was that anyone who disagrees with that statement is irrational and religiously thinking, but you think LLMs have a use so they think that about you too!
As predicted, you doubled down. There's zero substance to your "I mean, lol" bit because I am pretty convinced this person would recognize where an LLM can save you a few minutes (if not then they are indeed too irrational).
The crux of what people like myself say is "take the tool, use it, stop believing it's something more than it is because it absolutely it is not".
To have any semblance on topical discussion: I claim that we have zero intelligent machines today. No intelligence. No reasoning.
I owe no proof for refusing to believe something I cannot find even if I tried very hard to find it (I mean, who wouldn't want their job being done by agents they pay $20 - $100 a month for?). You are the one owing proof to me if you say LLMs are "intelligent" and that they do "reasoning". The burden of proof is on the one making the claim that something exists.
So go on. Prove it. And no "according to this scientist X and the other Y who, like all of us, have zero clue what intelligence and reasoning actually are but have some credentials so they love to pretend that they know". No -- nothing of that, please. Hard facts. Let's see them.
An easy prediction to make since... I still think I'm right. I mean, why wouldn't I?
because I am pretty convinced this person would recognize where an LLM can save you a few minutes
We must have a pretty different reading of their posts. They make some extreme assertions about LLMs being useless and a scam.
The crux of what people like myself say is “take the tool, use it, stop believing it’s something more than it is because it absolutely it is not”
But I have no problem with that? I have a problem with saying that if someone thinks LLMs are useful then they are irrational and guilty of religious thinking.
I owe no proof for refusing to believe something
I literally don't care. I've argued only against the idea that anyone who disagrees is irrational or religious thinking.
You are the one owing proof to me if you say LLMs are “intelligent” and that they do “reasoning”. The burden of proof is on the one making the claim that something exists.
They are the ones who made positive claims. I have never made a positive claim. I owe nothing other than a rational response to their claims. The fact that you seem to not understand this leads to my next point.
So go on. Prove it. And no “according to this scientist X and the other Y who, like all of us, have zero clue what intelligence and reasoning actually are but have some credentials so they love to pretend that they know”. No – nothing of that, please. Hard facts. Let’s see them.
Look, I'm genuinely sorry but I don't think you or the other poster know what words like "fact", "evidence", or "reasoning" even mean. I've realized how fruitless it is to try to talk to people about things like this because I just don't think an internet forum is the right place to teach someone what these words mean.
I see. Well, if your only goal was to make your parent poster less extreme then cool. I kind of thought that you were going into the other extreme: "LLMs are the future" and "LLMs are AGI" etc. bullcrap.
Nope. I never said anything about LLMs other than that one can rationally disagree with the parent poster's view on them without being irrational and "religious thinking".
My thoughts on LLMs are not reflected by any of these posts, or are barely reflected. They made positive assertions that were extreme and, frankly, ridiculous. They failed to justify them whatsoever. I presented rational theories that would reject them, which is all that is necessary to refute the idea that anyone disagreeing is irrational.
The details of Unix, like say stat() and terminals, are not fundamental, but the general architecture is. It could have been Plan 9, which is basically a "cleaned up" Unix in my mind ("everything is a file", etc.)
It's a mathematical argument with respect to the amount of software that has to be written. Narrow waists are a way of mitigating combinatorial explosion / enabling interoperability.
This argument goes back to a thread from a few years ago:
The blog posts were basically a big elaboration on my reply to that comment
(As an aside, I think that @matklad has come around to the viewpoint that Emacs/Vim are also fundamental -- i.e. they are centered around the narrow waist of text, or attributed text. As opposed to having different programs that edit each different type of data. Again, this is a mathematical issue with the amount of software that has to be written by us, and learned by users.)
With regard to C -- I will again claim that if you look at semantics / expressible programs, then C++, Zig, and Rust are a form of C. They all use LLVM, or started out using it.
People argued with Bjarne for years about this -- why does C++ use the C machine model? Isn't there something better?
And a design principle of C++ is that "it does not leave room for any language below it". This has actually been true!
The main new language I know of that does anything different is Mojo. Mojo is built on MLIR, which is inherently more parallel. It has a different machine model.
All of the LLVM languages are "C-ish" in my mind, just like Plan 9 and even the web are Unix-ish. The addition of types and metaprogramming is absolutely a big deal, but they are additions rather than fundamental changes, in my mind.
It's easy to tear down what exists -- it's harder to say what the alternative is. Whenever I read these rants, I welcome the author to build an alternative and show us :-) Or even just share a link to something relevant
Are you saying LLMs are pushing Unix or something?
The structure of the comment was: LLMs show a disturbing fact that some people in our field have religious levels of faith in things regardless of evidence. We see something that seems similar with Unix where people assume that it is inevitable, but that's not the same, it's just ignorance.
The bullet points they provide about Unix aren't what you're talking about at all. They're not talking about narrow waists and text as a universal exchange. Consider files in a hierarchical filesystem (and I'd add: a distinction between file and data in memory). That is certainly not fundamental. The AS/400 is the clear, commercially successful counterexample for the points they brought up.
OK, I guess I'll just repeat: show me what you're talking about
Maybe write a blog post :-) Or just drop a link
As it is, it reads like "I have secret, fundamental knowledge that other people don't have, but I am only going to write about it in the negative, not in the positive. Also, I am not necessarily going to do anything about it."
I still don't see any connection to LLMs. I guess the connection is "other people are wrong" ? Who are these people? I think there are different motivations at play, like talking your book, or simply liking things that help you get work done
Unix was just an example. People tend to believe all kinds of strange things in technology and treating such things as deity-ordained facts, but this is due to ignorance.
LLMs are so good at bullshitting humans into believing they are sapient, thinking machines; similarly the success of Unix has led people to believe all OSes are based on files. When I say believe, I do not mean in the assume sense, but in the fundamental, faith sense.
Note that the comment you just replied to is by @madhadron, who is different from the author of that first comment, @lproven. I think you meant your repeat of “show me what you’re talking about” to be directed at @lproven.
I took it to be so, but I do not know what I am supposed to do.
Personally I think the BCS article that we're discussing makes the case pretty well.
LLMs are not "AI" and they are not a pathway to "AI". They are a dead end: a clever linguistic trick that may have uses in translations and things, but nothing much more. The current hype is a pure unsupported bubble, it will collapse soon and we'll probably have another AI winter. Which one, I lose count; the 3rd?
You got way too defensive here, as if being mandated to defend... something. Maybe the UNIX paradigm?
Others people already told you but the comment was basically using analogies that people get entangled in certain technical stacks or get invested in tech in a certain way so much that they can't see past it and start evangelizing it... like the actual religious faith. That was all really.
I gave you the AS/400 as a counterexample. I'm not willing to write a blog post on it because 1) I am not an expert on the platform and 2) there is already a large amount out there about it.
The connection is a contrast: the relationship of many people with LLMs is different than what we often see in technology, such as Unix, because it's not just parochial ignorance, it's irrational faith.
The structure of the comment was: LLMs show a disturbing fact that some people in our field have religious levels of faith in things regardless of evidence. We see something that seems similar with Unix where people assume that it is inevitable, but that’s not the same, it’s just ignorance.
Correct. Thank you.
It is very interesting to me to find that while I usually can't understand what @andyc says in his writing, he also can't understand me.
I don't know what to do about it, but it's fascinating.
Again I claim that "Unix" [1] having won is fundamental, not accidental, because the design of the other ecosystems doesn't enable enough end-user functionality to be created. You have to write too much software.
[1] Windows being essentially a superset of Unix at this point; there isn't significant software that runs on Unix but not Windows
C is just a slightly higher level zero-cost abstraction over Von Neumann CPU and I would argue there isn't really any other practical/good alternative abstraction to come up with. In that sense all of: C, C++, Rust, Zig build/build on the same abstraction.
I think you’re missing a lot there. Flat memory is a pretty poor abstraction these days, when even a phone is a NUMA device. That’s a C abstraction that is not present in the hardware. There’s a lot of interesting recent research (and some commercial products from the 1960s and ‘70s) on object-based addressing. These ideas are hard to adopt in C though and will probably never see mainstream deployment as a result.
Similarly, hardware is message-passing all the way down. It uses this to build cache coherency protocols, which it then uses to build a shared-memory abstraction, because that’s what the C abstract machine demands. This costs a lot of power.
On a modern CPU, the amount of area dedicated to executing instructions is surprisingly low. The biggest power draw comes from the register rename engine, which (along with the instruction scheduler and all of the speculative execution machinery) exists solely to allow a massively parallel system to pretend to be a sequential one. This is required to provide a C-like abstract machine.
So, yes, C is a thin abstraction layer over a PDP-11, which is a Von Neumann CPU that made a few choices about memory, but modern CPUs look almost nothing like Von Neumann machines. They can emulate one, but they burn vast amounts of power doing so.
Good question. Something with no shared mutable state would be easier to scale. The caches for shared immutable data and exclusive mutable data are easy to build, it’s the shared and mutable that’s difficult. Similarly, something that has high degree of parallelism is easy. If you don’t do speculative execution, you need a lot of runnable threads to keep execution units full. Some of this might be easier to do with hardware support for creating thread-like things. For example, map-like functionality might be easy to implement, where you could do something not quite SIMT, which would help with instruction-cache usage.
Unfortunately, this hasn’t had nearly as much research as it deserves because you can’t run existing code fast on such a system and co-designing languages and hardware is really expensive. The early CHERI research cost around $10m, the total has cost $250m or so. And our changes he abstract machine were very small: we intentionally made something that was easy to map a C abstract machine onto.
Doing a proper co-design project to build a good language for high-performance hardware would probably cost around a billion.
We’ve started to explore some of these ideas in Verona. The really hard thing is to build something that can run well on current hardware but an order of magnitude faster on hypothetical more efficient future hardware, to then enable the development of that hardware.
I am not a hardware guy but I know I would work hard and passionately about making OS-es and software for hardware systems like this! I always felt shared + mutable was the easy way out and then everybody started preaching because "X million people can't be wrong".
(late reply) The big data frameworks derived from MapReduce and Spark are all "functional programming in the large" -- there are at least 10 or 20 of them, and many research systems
A cluster of PDP-11's connected by networks doesn't look like a PDP-11 - it's a different machine model (in particular, it's not a state machine)
A multi-core PDP-11 doesn't look like a PDP-11 - it's a different machine model
The functional properties like idempotence, commutativity, and associativity enable fault tolerance and parallelism. It has worked in practice
You can also look at research like Sequoia: Programming the Memory Hierarchy
(IIRC it is a functional language; I don't remember the use cases being that diverse)
Importantly, this observation doesn't make imperative programming obsolete. The low level is still imperative, but the high level is functional / declarative / graph-based.
I don't find it relevant. C just does not abstract over memory organization, and that's OK. On 8bit computers they had bank switching, on x86 memory segmenation, all bigger modern CPUs have virtual memory etc. You could and can still use any of these languages to write software for machines like that. In embedded you're often responsible for page tables, TLB mgmt, coherence, flushing, etc. The CPU/asm/C-like-language only cares about memory being addressable.
Well, nowadays there's "Provenance" in Rust (and a bit of if in C), so there's some stuff going on there in between the levels of abstraction, but that's about it.
Smalltalk always seemed like the great thing so ahead of its time that it couldn't be recognized for how great it was, even though it probably wasn't as great as it was made out to be. Still, I feel like I missed out on something special.
Years ago I got to have lunch with an old Smalltalker who held up four fingers and said something along the lines of, "FOUR KEYWORDS. Smalltalk had FOUR keywords and you could even do things like mess with the stack if you wanted." Wikipedia claims six, but it's still fascinating how much power they crammed into that language. I keep forgetting to make time to try out Pharo.
single system image
My understanding of Smalltalk was the image was "live", but it always felt like Docker was a ghostly shadow of that language that we try to emulate.
Smalltalk was originally created as a bet that you can fully specify a useful language on a single piece of US letter paper.
According to Wikipedia, the six keywords are:
true, false, nil, self, super, and thisContext
I’m not sure that any of these are actually keywords. True and False are class names. These classes are used as singletons and they implement methods like ifTrue: for conditionals. Nil is also a class, which implements a fallback method that does nothing and returns self, so any message sent to Nil does nothing. You can seriously mess up a Smalltalk system by redefining some of these methods, but you can also do fun things, such as add instrumentation to ifTrue: to trace program behaviour, or do this in a subclass and return a tracing True to see what things use the result of a Boolean.
Both self and thisContext are local variable names, the first is an implicit argument the same is the name of the current context. They’re reserved identifiers, not keywords. They don’t introduce new semantics, they’re just the names that the language gives for a couple of local variables that aren’t explicitly named. As I recall, you can assign to both of them (I’ve no idea what happens if you assign to thisContext, probably explosions).
I think super might be something I’d count as a real keyword. It is kind-of an alias for self, but uses a different dispatch mechanism for message sending (start search at a specific class, rather than the dynamic type of the receiver).
True, False and UndefinedObject are singleton classes; their instances true, false and nil are special cases in the VM and the bytecode for efficiency but otherwise they could be implemented as pretty vanilla global variables. Preventing them from being assigned to could be done outside the VM, within the SystemDictionary class for example, so they're not keywords in my opinion. The fact that they're treated as keywords is an implementation detail of the compiler.
On the other hand you can't assign to either self or thisContext, as those are not normal local variables. I would say that thisContext and super are definitely keywords, and self is too important to not be considered one.
On the other hand you can’t assign to either self or thisContext, as those are not normal local variables
Huh, in Objective-C, it’s quite common to assign to self, I forgot you couldn’t do that in Smalltalk. It’s something you often do with factory instances, but Smalltalk typically doesn’t separate allocation and initialisation, so I guess it’s less relevant there. The most common Objective-C idiom that assigns to self is for +alloc to return a singleton placeholder that you then call some -init-family method on. Depending on the arguments, it will then do self = [SomeConcreteSubclass alloc] and then the rest of the initialisation.
I can see treating thisContext as a special case, but to me it’s just a predefined name for the local variable that contains a reference to the current activation record. I thought some of the coroutine / green thread things worked by assigning to thisContext (and capturing the old one), but possibly they use some other primitive methods.
If you include implementation details, most Smalltalk VMs have a bunch of primitive methods (I don’t remember if yours did?) that will always be direct-dispatched to real things. Some bits of the standard library (byte arrays and so on) can’t be implemented efficiently in pure Smalltalk, so you can either provide these as classes implemented elsewhere, or provide some core parts that can be wrapped in mostly Smalltalk classes. You could regard the primitive method names as keywords, but they’re not part of the language, just implementation details.
Hmm, now I want to implement Smalltalk again. We have a tiny JavaScript engine ported to CHERIoT that uses a fairly Smalltalk-like bytecode, I bet something Blue Book-like could fit in a similar amount of code.
Edit: I just checked the Blue Book and the primitive methods are specified there, though in the implementation part.
Local variables are effectively "thisContext at: n" (with the stack above the local variables). Maybe I am tainted by actually implementing the VM but I see thisContext more as a VM "register" than as a local variables—even more so than self.
It's been a while and I don't remember exactly how you did coroutines in GNU Smalltalk. I think there was some kind of Continuation class that was a wrapper for the context and had a (non-Blue Book) call-cc primitive. [found: https://git.savannah.gnu.org/cgit/smalltalk.git/commit/?id=44c6e4445cbb04466 was the commit where the thisContext-based implementation of Continuation was turned into a primitive for speed. The primitive code can be written in Smalltalk but it wouldn't assign to thisContext; rather it would assign to the parent of thisContext, which does switch to a different context but without assignments. There were no high level coroutines; only generators as in Python but without a way to send a value back to the generator]
Green threads are in the blue book and they are different; they are cooperative for CPU bound code but also preemptive. Priority lists are in a class known to the VM and preemption could be triggered either by hand ("Processor yield") or by the VM signaling a semaphore (via the signal method on Semaphores of course, but also on a timeout expiring; GNU Smalltalk added SIGIO and single stepping). Each Process class has its own context which becomes thisContext when switching to that process.
I don't see a way to downvote it, so I'm leaving this comment to register my strong disagreement about the LLM comments above. Most of what the comment says about LLMs strikes me as wrong, overconfident, and poorly framed.
It is an intentional feature such that people don't content themselves with unconstructively attacking others and instead contribute to discussions meaningfully. Maybe you should try it by explaining what it is you disagree with. If you don't care to do that it is fine, but you should then take the advice of the site guidelines and just let the person you disagree with be wrong. There is no need to register your discontent, nobody is keeping track.
It is an intentional feature such that people don’t content themselves with unconstructively attacking others and instead contribute to discussions meaningfully.
Was this the intention behind the feature? How do you know?
In any case, I will grant this sounds like a good intention. But the road to Hackers News is paved with good intentions.
Let's flip the argument on its head. If one suggests downvoting enables people to not engage meaningfully, the same must be said for upvoting, does it not? Clicking an arrow, whether it be up or down, seems about the same w.r.t. "meaningful discussion".
There are many other designs available out in the wild. I encourage people to be dissatisfied with the status quo here. Most forums are terrible. Lobsters is, luckily, just a little bit less terrible. But it hardly lives up to its potential. If you go over to LessWrong, you can get some ideas of what is possible, which include more nuanced voting. You can, for instance, independently answer:
"How much do you like this, overall?"
"How much do you agree with this, separate from whether you think it is a good comment?"
This distinction is key. It allows the community to register sentiments such as "This is written clearly and in good faith, but I still disagree."
Users can flag stories and comments when there's a serious problem that needs moderator attention; two flags are needed to add the story or comment to the moderation queue. Users must reach 50 karma to flag. To guide usage and head off distracting meta conversations ("Why was this flagged!?", etc), flagging requires selecting from a preset list of reasons.
Was this the intention behind the feature? How do you know?
I believe it's been discussed in some meta threads through the years.
Slashdot/Reddit style upvotes / downvotes systems have multiple issues that people have observed through the years. Specifically, a downvote is just one bit - it's impossible to convey whether it's because the downvotes believes the comment is incorrect, a troll, or just doesn't like the commenter.
But the road to Hackers News is paved with good intentions.
This site is a off-shoot of HN, created to address the many issues the creator @jcs observed there. HN also has downvotes. So this is a flippant comment that I'm surprised a site member since 4 years would make.
If you want a LessWrong style multi-axis voting system, the source for this site is public, and a pull request to implement it would be welcomed.
Yes, you correctly identify many of the problems with one-bit downvote systems. My reply, in short, was saying: much of the same kind of problem applies to one-bit upvotes.
I understand if my joke wasn’t funny, but it wasn’t flippant. Designing good communities requires a lot more than intentions.
My comment was a little bit of a test balloon. I think it would take significant leadership and sustained persuasion to make it happen. I’ll think about some ways to give it a try.
Thanks for the conversation. Why do you guess I'm not humble, if you do? Or is it more about my style?
Some people conflate / confuse vigor and advocacy with a lack of humility. I wonder if you agree with this: A person can push back directly and persistently while still being humble. Asking a lot of questions and having strong opinions doesn't imply a lack of humility.
Here are three things I know about myself (as do people who know me). First, I don't think {community building, communication norms, interaction design} are easy. Second, I know I'm not capable of making broad changes by myself. Third, I am open to feedback.
I think a possible undercurrent is differences in communication styles, culture, and experiences. I've lived in many places across North America, and to a lesser extent, spent time in different countries in the world. I've seen variations in directness, passive aggression, confidence, overconfidence, humility, false humility.*
Another possible factor is that some people conflate intellectual pushback with a lack of humility. I'll tell you this, If I dish it out, I better be able to take it. My goal is not to tear people down. My goal is to tear down weak reasoning and poor framings and replace them with stronger stuff. Some people can't handle this, but I intentionally try to err on the side of optimism.
In my experience the topics of {collective action, cultural design, communication norms, consensus building} are widely misunderstood in technical circles. I voice my point of view as clearly as I can, and I welcome disagreement and clarification. Sometimes people take this in unintended ways. Some people are inclined to give me advice. All good. Communication is hard.
* I think many people can benefit by thinking more about the topics raised in The Proper Use of Humility. Have you thought about the questions raised there?
It would've been easier for me to notice that if you didn't make a half dozen separate comments in response to the same single comment, the first of which being you just vaguely complaining you disagreed.
Using granular discussion, point by point, works better for complex questions. Taking it slow and methodically is better for truth seeking. The criteria isn’t what one person says it easier. A better criteria is what results in high quality discussion from many people.
On the writing side, it is easier for one person to write one rambling message. It is harder to write compact, separable points which invite detailed and substantive discussion. Think about it? Try it? Let me know what you think.
I welcome specific and focused comments to any/all. I split the comments into granular chunks intentionally. I explained my reasoning in these two comments: 12.
I registered general disagreement because I found the comment to be egregiously misguided. When I see otherwise informed and capable people led astray, I say something, because I care. I've been civil about it. The comment was well-written in the sense that it persuaded a lot of people, as measured by upvotes. (I couldn't see the net upvotes-versus-downvotes because of the information architecture here.) It used a lot of rhetoric; it sounded confident. And it was quite misleading (at best) or wrong (at worse) about LLMs.
Why do I care? AI -- actually AI Safety in particular -- is a big deal to me, and I'm trying to do what I can to root out misunderstandings. I assure you I take this topic and its implications very seriously, possibly as seriously as anyone here.
That they are intelligent, meaning that they can think in any way at all. (They can’t, but they fake it in ways some find compellingly persuasive.)
First, using the words intelligent and think without definitions is asking for misunderstandings.
Second, the quote implies some sort of dividing line between "intelligent" and "not intelligent". That's misguided.
Third, like many, I find value in the definition of intelligence as the ability of an agent to accomplish some task. This makes it situated and contextual. Defining it this way helps clarify discussion.
I find value in the definition of intelligence as the ability of an agent to accomplish some task
I like your definition quite a bit because I think it captures a lot of nuance in a practical way. That said, I have a pair of potential counter-examples to consider from my day-to-day work that I’m struggling to classify as intelligent or not using that definition. For context, I work with autonomous aircraft and in a previous life worked in a biology lab that was studying insect flight characteristics.
The first example is the flight control software itself: given a series of waypoints, fly the aircraft along that trajectory as best as you can. I’m going to reveal a dirty little secret here: most of it is just nested PID loops. For quadrotors, they’re typically:
position is controlled by…
velocity is controlled by…
acceleration…
attitude…
attitude rate…
motor thrust (controls attitude rate and acceleration in the local Up direction)
This accomplishes a task and to a human observer actually looks quite intelligent. If there’s a gust of wind that knocks you off-course then the position PID loop will have an increased error term that ripples down through the stack to turn into motor commands that get it back on course. Is this intelligence? It accomplishes a task even in the face of adverse inputs. But under the hood it’s basically just a bunch of integrators trying to push the error terms to 0.
The second example is closely related: Kalman filters for processing noisy sensor inputs. Over time these are building up a statistical model of how a sensor is behaving and then making decisions on every sensor reading on how much that reading aught to be trusted. It’s an iterative model; each sensor reading gets assessed for trustworthiness and is also used for updating the internal statistical model of how the sensor is performing. It’s pretty much just a rote linear algebra equation each iteration though. Is that intelligent? It almost looks magical when it works well!
The last part… why I mentioned the insect research back in undergrad. One of the experiments I helped with was building a flight simulator for locusts (bizarre, right?). Forward velocity and turning through the simulator was accomplished by inserting electrodes into the insect’s wing muscles and measuring the timing of the muscle activations. If they activated simultaneously the insect should fly straight, if the right wing fired before the left then it would induce a turn. Once we’d established that the simulation was accurate enough that the insect could fly around in our crude world without colliding with objects (what?!?), the biologists hooked up additional electrodes to a single specific neuron called the Descending Contralateral Motion Detector, which was connected pretty much directly from the insect’s eyes to the muscle control system (I’m not a biologist…). What we observed was a series of electrical pulses from this neuron that were directly correlated with the wing EMG asymmetry: if the DCMD was reporting that a collision was imminent, the muscles would fire to turn away from the obstacle.
Is that intelligent? It enables amazing collision-free swarming behaviour. But in some ways it’s even simpler than the drone FCS… and I’m not sure I’m comfortable calling PID loops intelligent?
Yes, for certain environments, the right sensors and actuators in a PID loop will get the job done reliably. If that setup serves an agent well, I would call that intelligent behavior.
The word intelligence is a fascinating mess. For example, many people look at a world-class chess player with awe and suggest they are outliers in intelligence. By the definition above, if they win at chess given their environmental constraints, they are intelligent at chess. But from another point of view, much of what chess players do is rooted in pattern matching and tree search. When you frame it this way, it doesn't seem that "intelligent" does it?
It seems to me you are talking about more general forms of intelligence. The degree to which an agent's intelligence transfers to other contexts reflects its generality or adaptivity.
LLMs can indeed output language that is consistent with logical reasoning.
I figure this is the claim - LLMs can output language that is consistent with logical reasoning. LLMs can simulate reasoning better than any prior reasoning-simulation software. However, they do not actually perform reasoning; their reasoning is performative, because the output is algorithmic (and algorithmic in a way that human reasoning is not).
If a system outputs text that consistently matches logical reasoning to a high standard, isn’t the simplest explanation that it is indeed reasoning?
(I’m not asking questions of consciousness or personhood or any of that.)
Perhaps you would say these systems are not using maximally simple, purely logical, circuits / structures / mechanisms that only do logical reasoning, such as forward-chaining or RETE or whatever?
If so, how are humans different? At best, humans use their handwritten symbols to make logical proofs according to set rules. But we still use complex and often flawed lower-level mechanisms (brains) to do this logic. In this sense, I think your claims about performative reasoning are moot.
If a system outputs text that consistently matches logical reasoning to a high standard, isn’t the simplest explanation that it is indeed reasoning?
no, because the Chinese Room isn't reasoning either.
LLMs are collections of things that humans have said before, boiled down into statistics. The sole job of an LLM is to output words based on statistical locational frequency, that is not, and cannot be, reasoning. It's incredibly provable that LLMs cannot actually reason, too, and there are multiple papers on it — see https://arxiv.org/abs/2406.02061 and also https://link.springer.com/article/10.1007/s10676-024-09775-5
I think the comment above misrepresents or misunderstands the Chinese Room argument. Per Wikipedia:
The Chinese room argument holds that a computer executing a program cannot have a mind, understanding, or consciousness
Searle’s point about the Chinese Room is not about if a machine can turn the crank and do modus ponens or other reasoning rules. (This has been settled for a long time.)
The C.R. highlights the question of qualia, of inner experience. I’m not talking about consciousness, “thinking” or a “mind”. I’m talking about something measurable: can an AI generate correct conclusions according to the rules of logic? (For example, they do LSAT level reasoning quite well, and they are getting better.)
Ah. Your claim boils down to the fact that the low level mechanism of token generation is not identical to reasoning. Am I representing your point of view fairly?
There was also a time when I fixated on this definition. I get it.
I hope you realize that when some people say LLMs “can reason”, they aren’t talking about the mechanics of token generation process. They are talking about higher level behavior.
I think it is also important to realize that by your definition, one could argue that humans don’t reason either. We’re just neural impulses firing. I hope you take my point.
LLM bots cannot analyse and improve themselves, because they cannot analyse -- anything ever -- and they cannot improve on what was in their training corpus.
I can see why you’d think that LLM bots cannot analyse and improve themselves. What really confuses me is that you say “nothing can”.
Are you saying that humans cannot analyse and improve themselves? Because I think plenty of humans do that after, for example, failing at something many times in a row, or reading a self-help book.
Or by “nothing”, do you mean only things that aren’t alive? If you do, consider the hypothetical case that someone wrote a (non-LLM-based) program that perfectly simulated a human’s behavior (human-like words, leg movements, etc.). The program was even observed to change its outputs after interacting with a simulation of a self-help book. If you agreed that humans can analyse and improve themselves and you had this definition of “nothing”, I think you would have to be making one of these three assertions:
That program would not be analysing and improving itself, despite acting exactly like a human, which does those things.
Creating such a program is impossible – computers can never match human capabilities.
Such a program would be alive, and thus not part of your definition of “nothing”.
LLMs are, by definition, vast databases of multi-dimensional weights of relationships between symbols.
An LLM can't analyse an LLM, including itself, and nothing else can analyse an LLM either.
Claims of redesigning LLM bots to hallucinate less, for instance, are lies, because the databases are not amenable to study or analysis. And, of course, hallucination is what they do. There is no distinction between making good and desirable stuff up and making bad stuff up.
They are lies, just as software companies claiming they have rewritten huge codebases for security. In most cases, they've just done automated scans, maybe a few people looked at a few especially critical routines.
Real rewrites are very rare and usually disastrous, as Joel Spolsky observed about 15 or 20 years ago.
My comment was about your claim that “nothing can” “analyse and improve themselves”, a claim I disagree with. What you say about LLM analysis may be true, but that’s irrelevant, because I was not arguing that LLMs are an example of a thing that can analyse and improve themselves.
With my comment, I was hoping to pinpoint where, if anywhere, we disagree about the statement “nothing can analyse and improve themselves”. You can do that by answering these questions:
Do you believe that humans can analyse and improve themselves?
(I believe that humans can.)
If so, then do you still believe that nothing can analyse and improve themselves?
(I would interpret my above belief as disproving this, but maybe you’re using a different definition of “nothing” from me.)
If so, then which assertions are you making out of the three I listed in my previous comment?
I just thought of a possible explanation for your response being weirdly focused on LLMs instead of on the questions I had posed. Looking at your original wording again:
That they can analyse and improve themselves. (Nothing can.)
Did you intend that last sentence to be interpreted like this?
If that statement does not capture your meaning, then it is unfortunate that your replies to both insanitybit and rele’s comments, which contained that rephrasing, did not correct the misunderstanding.
Regarding exaggerations and lies about what LLMs are capable of in the near-term... Yes. There is hucksterism in AI. I know the arguments; I've read Kapoor and Narayanan, which I generally recommend. I've also read Gary Marcus, who makes some good points but gets too ranty for my tastes.
However, regarding the comment here above, there are claims here about what is and is not possible, such as:
An LLM can’t analyse an LLM, including itself, and nothing else can analyse an LLM either.
If we define analysis as "detailed examination of the elements or structure of something", then, yes, an LLM can analyze its own weights and structure.
If you say "but LLM X can't analyze itself", that is a different claim -- a claim about implementation not about what is possible.
If you say "but it can't analyze itself up to some quality standard", that is a different claim, and it invites discussion about the bar and what progress is being made towards it. This kind of discussion might be interesting; it invites us here to dig into the details.
nothing else can analyse an LLM either
This is a tall claim. To me, it is so bold that I wonder if it is designed to attract attention. I'll bite: I ask the author to (a) clarify their terms and either (b) prove this claim or (c) make a testable prediction, on the record.
That bigger models will make them better at what they do. (This one just exposes basic ignorance about very simple statistics, such as scaling effects.)
What do you mean by "what they do"?
What papers on LLM scaling laws have you read? I suggest we ground our conversation there. The papers show improvement across many metrics. Bigger models perform better in almost all ways*.
If you say "but Improvement X isn't what matters", then you are making a different argument.
If you say "but the improvement isn't worth the cost", then you are making a different argument.
If you say "but most people can't notice the improvement", then you are making a different argument.
Do you want to refine your claim?
* Bigger models can be more deceptive, which is unfortunate if you care about alignment (which you should). But increasing deceptive ability is most certainly an improvement in capability.
I am not interested in radical new methods for estimating with unprecedented accuracy the numbers of angels dancing on entire fabric-fastener manufacturing units. I don't care, because there are not such things as angels.
I find it highly amusing that saying "this type of software is not intelligent" provokes responses which attempt to nail down a possibly negotiable definition of intelligence. Again: this is not a rebuttal, in my view.
You try to split hairs, perhaps so that you can cut each one, but I'm not here for a haircut. I am not here to persuade anyone. I don't see any potential gain in even trying.
LLMs are a fascinating technology which may in time enable applications such as simultaneous translators, perhaps not only between human languages but between computer languages as well, which could have amazing consequences. I speculated on some of those here but I remain extremely sceptical about the entire current industry, as I said in the earlier article that the above is a follow-on to.
But they are not artificial by any reasonably strict definition, they are not intelligent by any definition that isn't a fantasy, and because they are not in any way what they are depicted as being, they will never lead to anything that is.
It is empty hype, nothing more, and I sincerely hope it collapses soon and that it destroys the careers of the hucksters selling these lies about thinking computers.
Here’s what I notice. First, you are obviously bothered by hucksterism in AI. (So am I, by the way.)
Second, this is a bit of a guess, but you seem to take my pushback personally. And then you channel a lot of negativity towards me. I’m not the problem. The key problem is these topics are complicated, communication is hard, and we all have egos. Very few people want to be wrong, much less shown to be wrong.
Third, you write at length, but haven’t answered the specific questions I ask. You sometimes wave them off. Sometimes you ignore them altogether. You also use a lot of rhetorical devices.
On the substance: Are you claiming LLMs are not getting better as they get bigger? If so, by what metric? (I ask for it to be clearly defined and measurable.)
Please respond to what I’m writing, not about what some other person says.
Are you claiming LLMs are not getting better as they get bigger?
Again with the demands for metrics. No, I cannot give you metrics, because as I have said already, I do not care. You are claiming the numbers of ratio of dancing angels to pins is important: I'm saying there are no angels. No angels means no counting means no numbers.
Yes, they have, so far, provider better results -- more plausible-looking textual or image based output -- with greater size; however, it is not just foolish but downright stupid to blindly assume this will continue indefinitely forever.
Secondly, the bigger the model, the bigger the power and storage use of this technology which is already environmentally catastrophic. The boosters who keep calling for bigger models are guilty of ecologically criminal acts. You are entirely as culpable as the cryptocurrency promoters.
Please respond to what I’m writing, not about what some other person says.
... it is not just foolish but downright stupid to blindly assume this will continue indefinitely forever.
I have not said anything about such improvements continuing forever (neither in this thread or sibling threads, nor anywhere I can recall.) I strive to not blindly assume anything.
You are entirely as culpable as the cryptocurrency promoters.
Please stop attacking me. Such attacks are not welcome here.
And, to be clear, I have not called for bigger models. (And if I had, would such attacks be justified and appropriate here? It is hard to listen and learn when you are attacking and accusing.)
Again with the demands for metrics.
You may perceive them as demands, but this does not make them demands. I have asked questions. Repeating a question doesn't make it a demand. We have both attempted to redirect the conversation, such is the nature of conversation. You call my attempted redirection a "demand". To my eye, this is probably either (a) the result of you being upset for other reasons and/or (b) a rhetorical device to make me seem unreasonable.
You are claiming the numbers of ratio of dancing angels to pins is important
This is an uncharitable description of what I'm asking.
When I said "rhetoric" before, this is the kind of thing to which I was referring. It raises the temperature but doesn't promote mutual understanding.
Overall, this conversation has been unnecessarily fraught and combative. I have strived to disagree with you without being disagreeable. I'm sorry if you think my point-by-point commentary is mean or in bad faith. I do not intend any of these things, and I'm sorry if I've offended you somehow.
It would seem you have labeled me in some negative way, such as being an AI booster or ecologically irresponsible. I don't see the logic in why you think these labels apply to me. And again, even if they did, being unkind about it isn't necessary or welcome.
Whatever you feel, I would ask you to stop taking it out on me. It seems you are making assumptions about my point of view or my impact on the world. I have no doubt that your point of view is valuable, but I would ask that you be more civil about it.
I am not sure what you mean. Based on your comment, you sound angry about certain mannerisms and/or beliefs? Such as?
If you want to offer some specifics and are willing to discuss charitably rather than dismissively, I’ll put in some additional effort. No need to be mean about it, please.
I think it’s fair to say the context here is about artificial intelligence and some common misunderstandings. Unless I’ve lost track of the parent comment, I think this all started when one person made a list of common UNIX misconceptions and then offered various AI misconceptions. My comments, taken together, point out how many of these claimed AI misconceptions are actually misconceptions themselves.
Many people were far from calm. The conversation has the usual signs of people overreacting to each other. Even relatively neutral people got attacked, because one side or the other accused them of being on the wrong side. It has been a case study of polarization and poor community discussion norms. Truly pathetic, as in I have pity for how we all got here.
These discussions need to get clear and as specific as possible if we want to make sense of the world. I hope you can appreciate this.
Are clarity and precision and rigor tedious to you? If true, this discussion might not be worth your time? Truth seeking isn’t “sexy”, nor does it provide the dopamine hit of leaving a nasty comment. It doesn’t make it easy to form a tribe of “us versus them” because anything is subject to scrutiny and revision. These are the values of critical thinking and intellectual honesty.
I flagged this as unkind, it's probably trolling too? I don't know. I figured I'd let you know. Calling "effective altruism" a cult is definitely hyperbolic and clearly intended to be insulting.
I know that some online communities dislike some members or something like that but it's not a cult, it's just an ethical framework, and the user doesn't even seem to be advocating for it. I don't know what mannerisms you're even referring to, personally, but this seems like a silly and overly negative interpretation of a post that's basically "please stop insulting me and engage with my points, I am willing to engage with yours".
your explanation has been given a consideration it deserves, and found wanting. specifically, the following statements are wrong: “calling ‘effective altruism’ a cult is definitely hyperbolic” (it isn't); “it's not a cult” (effective altruism movement, especially its californian variety, has many characteristics of a cult), “it's just an ethical framework” (no, it's not just an ethical framework).
if you don't know “what mannerisms i'm even referring to”, you have no basis to claim that “it seems like a silly and overly negative interpretation of a post” (and my reply wasn't referring to a single post).
to spare your precious time, i will have tagged my reply as unkind myself.
Look at adversarial inputs for image recognition. I can show you two pictures, which you can't tell apart. One the AI says "giraffe: 99%". The other has undergone adversarial input modifications. The AI say "dog: 99%".
Whatever you want to believe about AI, that example alone shows that it's not doing anything like what you or I are doing. There's definitely some statistical magic behind it, but there's no possible way that it's "AI".
Add to that the fact that "AI" needs 1000's to 10's of 1000's of pictures to "learn" what a horse is. But the average 2 year-old can learn the same fact in 1-2 viewings.
The current crop of AI is doing super-optimization of techniques which are appealing, but which are nothing like what people actually do. Until such time as the researchers take a step back, we're just optimizing ourselves into a deep, dark, and unproductive corner.
that example alone shows that it’s not doing anything like what you or I are doing
I don't think that LLMs are doing what our brains do but I don't think your example demonstrates that at all. An obvious defeator here is that humans are entirely subject to adversarial illusions.
I don’t think your example demonstrates that at all. An obvious defeator here is that humans are entirely subject to adversarial illusions.
Let me re-phrase the final bit of that argument: "humans are entirely subject to different adversarial illusions."
The adversarial illusions are a window into the underlying model. See the common illusion of a rotating, but inverse face mask. The inverted face rotates left to right. Your brain interprets it as a normal face, rotating right to left.
Why?
Because for the past 10 million years, every face seen by every brain has been normal. When your brain sees something that looks vaguely like a face, it interprets that thing as a normal face. The "illusion" here is a breakdown of the model. The output of the model no longer matches reality.
For AI, I can take a picture of a giraffe, change 1000 random pixels, and then convince the AI that it's a dog. However, your brain isn't subject to that adversarial illusion. The only possible conclusion here is that the models used by AI and by your brain are completely different.
I'm not sure what's different here. Why is changing 1000 pixels (not random fwiw) different from showing me something that "looks" like it's rotating, or 3d, or multiple colors, etc etc etc. I don't see the difference honestly.
TBH I think it's moot because I don't think anyone argues that LLMs and human brains are the same, only that they may share some features. If you're arguing that they share no features, I don't think this argument based on illusions works well for that.
Whatever you want to believe about AI, that example alone shows that it’s not doing anything like what you or I are doing. There’s definitely some statistical magic behind it, but there’s no possible way that it’s “AI”.
I don't think adversarial modification is good to generalize intelligence on though I agree it's an good story to illustrate that "AI" is statistical games at this point.
To make a parallel, say I bring a blind person some horse hair, and they say "99% horse" based on the smell. Then I give them horse hair that's been sprayed with dog sent and they say "99% dog." It doesn't mean they're not intelligent.
Or otherwise, encrypted data should not be differentiable from random data. Thus encryption could be seen as an unbeatable adversarial modification.
I think that if you instead have the recognition system report features (extremely long neck, four legs, brownish), LLM will readily reason that it's likely a giraffe.
Having poor architecture doesn't mean that the components are unusable.
That recognition system isn't used by the bulk of the AI tools, so far as I can see. Instead, the method used is statistical magic on RGB pixels.
As best I can tell, the human brain does essentially what you say here. It forms a model of "cat" based on recognizing the 3d look & feel of the cat. And even that is based on modifications of previously learned animals. When you see a new cat, your brain uses those pre-existing models to find the "best fit" to a particular animal.
The current crop of AI tools might eventually get to that point. But the method of "analyze 10,000 pictures to discover what's a cat" isn't doing what we do.
Are you also saying that “what people actually do” is “better”? In what sense? Can you give an example of what you mean?
It's blatantly obvious that what people do is "better" for many, many, things. If AI takes 10,000 pictures to learn "cat", and people take 1, that's "better". If AI is fooled by changing 1000 random pixels in a picture and you're not, that's "better". If AI reads 10,000 books and then hallucinates answers to questions, humans are "better".
As for the rest of your comments, you're asking a bunch of fairly hostile questions which demand that I do a bunch of things. You're rather missing the point.
My point is that AI proponents make all kinds of claims about AI. Yet anyone honestly looking at AI can find trivial, obvious, counter-examples which disprove those claims.
I'm not saying that the current crop of AI isn't useful. It's tremendous for synthesizing statistical knowledge out of reams of data. It's amazingly good at looking like it understands language, using only statistics. But there is no possible way it's "thinking", or that it is "reasoning". It's statistical magic, and is doing nothing like what people are doing.
There is simply no question that people are better than AI at a huge list of tasks. Until such time as AI starts to use models similar to those used by our brains, AI won't be nearly as good at them as we are.
Ah. There was a misunderstanding. I wasn't talking about relative performance. (Sure, humans are better at some things, AIs others.)
So, I'll write it a different way, to see if this helps:
Are you saying that "how humans think" is "better"?
In context, I was responding to:
The current crop of AI is doing super-optimization of techniques which are appealing, but which are nothing like what people actually do. Until such time as the researchers take a step back, we’re just optimizing ourselves into a deep, dark, and unproductive corner.
This is a big claim about what research directions are more and less productive. It implies that AIs should be more like humans, does it not? This is why I asked: "Do you know the stock arguments for/against silicon relative to carbon computation?" These are an essential part of diving into the discussion of human-vs-machine intelligence.
Here is something to consider: one can value and promote human thriving without locking-in current modes of human thought. Modern statistical reasoning only became common among professionals around the 1950s. My point: we should be open to better reasoning, even if it is "unnatural" or "non-existent" among a given human population.
I notice many people claim hostility. In my case, I think you are confusing hostility with requests for clarification. Instead of making assumptions about your claims, I’m asking. Generally speaking, claims around AI are fraught and getting more polarized. Slowing down and explaining helps.
Edit (hours later): in retrospect, this was an obvious miscommunication -- see sibling comment. It is notable that you gravitated towards the word hostility. Many people do this, myself included, and we can do better by remembering "there can be many explanations for what I'm seeing; I don't have to assume malice."
The "requests for clarification" came across as hostile. The questions weren't "What do you mean by that?" or "Can you explain more?" But instead you asked a bunch of leading questions, which insinuated that I know nothing. That comes across as hostile.
I thought my comments were based on pretty trivial observations. An AI can be fooled by adversarial inputs. Yet you can't tell the difference between the two pictures which fooled the AI. There is no possible way that the AI is doing the same thing that people do. This is a trivial conclusion to make.
When you look at a picture of a giraffe and say "giraffe", but an AI looks at the same picture and says "dog", it is also trivially obvious that a human is better at that task than an AI is.
Many of the rest of the arguments for AI are just "the emperor has no clothes". I don't need to read thousands of papers on how beautiful his garments are. I can see his dangly bits. He's naked.
I'm amazed at how good AI is, considering it's just statistical magic. But there is no possible way in gods green earth that it's doing anything like what people do. And there is no possible way that it's doing any kind of reasoning.
I think a significant number of people have a tendency to think “I see hostility” before they think “Let me think about a charitable interpretation”. I see this in your comments. Fair?
I asked some pointed questions. Call them leading if you want. That doesn’t make them inappropriate. I think if you step back, and it wasn’t you involved, you would agree.
I also think you know I’m not asking you to read 1000 papers. Not 100. Not 10. I am asking if you know what you zero, one, or few shot learning is.
I know that people don’t want to admit they don’t know something. But there are more important things, like learning and being thankful if someone corrects your understanding. Why? Once your ego recovers, more knowledge makes you more powerful.
It is fascinating that some people bristle at the idea of learning. Dawkins forbid we encourage each other to deepen our knowledge! What are the chances that in a pairwise interaction that one person has deeper knowledge in some area? Close to 100% probably in most cases.
I’m trying to share some things I know with you, in case you don’t know them. You have a choice here. Do you reach for the “you are being condescending” line of thinking? Or maybe instead grant that, yes, XYZ was a reasonable next question.
Overall, I value promoting civil, frank, truth-seeking and open discussion while assuming charitable intent. I’m not perfect. I know from experience my standards are higher than average. I’ve been accused of being hostile, not humble, etc., etc. I can handle it, because I know who I am and what I stand for. I sometimes spend time with people whose curiosity peaked long ago. It is sad.
Like many others who study AI Safety, I estimate there is perhaps 10% to 30% chance over the next 40 years that artificial intelligence technology will decimate human civilization or worse. With this in mind, I will be civil, but I’m going to reserve the right to push back against incorrect or overconfident or too-narrow claims. When I think there is a better framing, I’m going to say it.
I don’t think any of us have the luxury to be ignorant or wrong about AI. I don’t think any of us have the time or luxury to not push ourselves to be better thinkers.
My views don’t reflect my employer or investments I hold. They reflect a stubborn and persistent drive towards better understanding the world despite humanity’s tendency to get in the way of long term thriving.
P.S. Someone accused me of talking like an effective altruist, as if that is some horrible thing to be. Many people conflate one bad apple (SBF) with a philosophy that varies in its application.
Many people, whether intentionally or accidentally, take it as a given that the goal of AI is to be human-like, and as such mimicing humans more accurately is "better" by definition.
One example of this is FSD, where the goal is explicitly for autonomously driven cars to coexist with human driven cars and replace human drivers without changing the surrounding expectations much, and as such it's desirable for them to understand human signals, for them to know a wave of the hand like so in one country means "I'm giving you right of way, please turn", but in another country the same wave means "wait a second", and if a police officer does it means "stop right now."
It also feels more comforting if we can fool ourselves into thinking we understand how an LLM reasons, and the closer it's output is to what a human (or super-human I guess) would output, the easier it is for us to think we understand its capabilities and limits, whereas the above example of an adversarial modified image throws this into sharp relief.. so it makes people feel uncomfortable, which also is a fair enough argument on its own, since "uncomfortable" is bad I guess
I also want to distinguish between (a) how humans "think" and (b) what we value. I'll make three claims about these:
Like I mentioned in a recent comment, I would argue that humans should be open to better ways of thinking, by which I mean analysis, planning, and so on.
Better ways of "thinking" can help us get more of what we value.
As argued by Willam MacAskill, I think we should be vary of value lock-in. One of the markers of societal progress is the changing and improvement of our values.
Now, how these points interact with AI Safety is complex, so I won't say much here other than: beware powerful optimizers. People such as Stuart Russell and Robert Miles, explain this quite well.
My comment above got labeled as "spam" which doesn't match what is written at https://lobste.rs/about :
For comments, these are: "Off-topic" for drifting into meta or topics that aren't related to the story; "Me-too" when a comment doesn't add new information, typically a single sentence of appreciation, agreement, or humor; "Troll" for derailing conversations into classic arguments, well-intentioned or not; "Unkind" when uncharitable, insulting, or dismissive; and "Spam" for promoting commercial services.
Add to that the fact that “AI” needs 1000’s to 10’s of 1000’s of pictures to “learn” what a horse is. But the average 2 year-old can learn the same fact in 1-2 viewings.
Have you read some papers and/or implemented {zero-, one-, few-}shot learning?
I recommend these techniques because it is not true to claim that all AI/ML techniques require 10's to 1000's of pictures to learn new categories. ML research has made extensive progress on this. In practice, many systems lag behind, but it is often a failing of human choices of where to spend their efforts. Generally speaking, it is not a failure of what is possible or even feasible!
People often lack key information and skills that would otherwise allow them to reach for better alternatives. Often the "first" failure I see in people is a lack of curiosity and imagination, followed by overconfidence. In particular, a large failure mode I see is humans overgeneralizing about what AI/ML systems can and cannot do. This is why I respond (asking questions, clarifying, pushing back) when I see ignorance and/or overgeneralization.
On what basis is my comment above flagged as spam? It directly addresses the point. I’m more than happy to hear criticism and error correct. Along with others, I am seeing a lot of incorrect flagging, possibly vindictive. Who can check on this and/or design countermeasures?
Could you please use a single reply instead of multiple replies? It makes everything very confusing to read. Just split your post into multiple quote reply sections if you must.
Replying in a granular fashion, point-by-point, allows comments to be more targeted. As a result, it allows conversation to continue to be productive even at the deeper levels. It might be unconventional, sure. Have you considered the current convention might not very good?
About me: I've spent thousands of hours studying (and experimenting with) various approaches for dialogue, deliberation, collaboration, and interaction. This is one of my top interest areas, and I care about it deeply. The current state of online discussion and the degree to which people fall into certain suboptimal patterns is a drag upon our collaboration potential.
One important aspect of collaboration is to tailor your response to the audience. You're trying to apply conventions from one forum (I'm guessing LessWrong, based on your previous comments) to this one, where they don't apply.
Yes, tailoring responses to the audience is often offered as advice. Sometimes this is all that is said, leaving a key follow-up question unspoken: what is actually good for the audience? To different degrees, we have both been outspoken as to what we think it is good for "the audience".*
To what degree is the following true, in your mind?... You know better than me, because you've participated more here, and for a longer duration? I'm newer and bring unusual points of view. To put it bluntly, I'm the other, the outsider.
If you were in my shoes, might you see part of this as a push for a certain kind of conformity? I'm more comfortable than many with non-conformity, because I aim for different goals. Perhaps you can appreciate this.
You’re trying to apply conventions from one forum ... where they don’t apply.
How do you know they don't apply? I'm pretty sure I disagree with that claim.
Preference aggregation is tricky. Do you claim to have know what people want?
Do you claim to represent the group? I often see outspoken people that claim to represent a group's interests.
What people want varies over time, often for fickle reasons. When we ask people what they want and compare it to what they do, it can be a confusing mess. (Just compare how people wish they would have spent their time versus what they compulsively click on and how much they doom-scroll.)
My point: I claim that subdividing points into separate threads serves people better. I am also claiming that "what people want" is probably not the right metric. I am suggesting that a better target is "what is good for people"; in particular, it is more important to aim for good practices and habits in service of truth-seeking. Not all of these practices are currently appreciated; some feel foreign. That does not make them wrong, of course. Blood-letting was once a popular medical "strategy". And so on.
I see a good chance you may be past the point of neutral conversation here. Are you? Can you see why I would say this?
* Is it fair to say that I have accentuated what I think is good for the audience if we were to stop and think about it? Is it fair to say that you've accentuated what you think is good for the audience based on their current tendencies and habits? Is there a synthesis here where both of our perspectives get factored in?
Hi! I don't often post comments, but I do read just about every comment posted on Lobste.rs as I enjoy learning different points of view. I wanted to give some feedback that, as a neutral party in this discussion, I also found your use of multiple different replies to the same comment confusing and hard to follow.
The reason is we go from a threaded conversation where both sides are replying to each other, to a multi-threaded conversation where you have to keep track of which part of the tree you are in. i.e. we go from this:
a
|-b
|-a
|-b
To this:
a
|-b
|-b
| |-a
|-b
|-a
Replying in a granular fashion, point-by-point, allows comments to be more targeted.
I don't disagree with the above statement in general, I do disagree with the implementation. Similar to programming something that is multi-threaded, things get a lot more complicated quickly when you split this thread up. I suspect if you take the different comments you made, put them into the same comment box and separated them with lines or just different quotes, it would be a lot easier for people to
Follow your argument
Respond appropriately to your overall argument, not get lost in individual bits.
It adds a lot of additional empty space due to forum layouts, makes the tree more chaotic, makes it more difficult to respond (because people may want to respond to points that were spread across comments), etc. I do think it would be helpful for you to stick to one reply.
Did... anyone even read the article before commenting? This article doesn't say anything about AI's capability, i.e. it isn't saying that current AIs will not keep improving in terms of what they are able to do. Yet it seems like that's what all the comments here are talking about. (Of course, I could be misunderstanding the points people are trying to make.)
This article is pointing out current AIs' lack of interpretability - their lack of human understandable internal structures. Therefore they can't be verified and trusted, and is a dead end in that sense.
And the article itself doesn't even have insight into evolution of LLMs, but is instead arguing this should have been done differently, with nothing but "hope" to back up such approach.
They are only a dead end from the perspective of those seeking AGI. To everybody else, the question might better be phrased as "Does current AI represent useful tools." I would argue from that perspective, large language models are not at that end. They are in fact incredibly useful for maybe not replacing people, but definitely for augmenting what people can do in a 24-hour period.
The article isn’t talking about AGI, it’s about reliability. The point, as I read it, is that these systems are black boxes that can’t be decomposed, that are not predictable or reliable, whose behavior we can’t understand, and that have no accountability. I tend to agree.
The main mentions of accountability and responsibility I see in the article are:
By ‘software engineering’, I mean developing software to align with the principle that impactful software systems need to be trustworthy, which implies their development needs to be managed, transparent and accountable.
When I last gave talks about AI ethics, around 2018, my sense was that AI development was taking place alongside the abandonment of responsibility in two dimensions. Firstly, and following on from what was already happening in ‘big data’, the world stopped caring about where AI got its data — fitting in nicely with ‘surveillance capitalism. And secondly, contrary to what professional organisations like BCS and ACM had been preaching for years, the outcomes of AI algorithms were no longer viewed as the responsibility of their designers — or anybody, really.
The point, as I read it, is that these systems are black boxes that can’t be decomposed,
If you don't know the weights, sure, neural networks are by definition black boxes. Many algorithms also look like black boxes if you can't see the relevant details.
But if you know the weights and the architecture of a NN, there are ways to interpret them. There is interesting work being done here. It is a very different kind of interpretation than understanding a program based on its source code.
I've learned a lot from the interpretability research from the AI safety community. I recommend it.
The best I gather is that the state of the art in investigating regions of connectiveness or looking at the second or penultimate layers and inward, etc, is that it is all still very noisy and not generalizable to anything except another can of worms of 98% accuracy and a chaos monkey of the remaining 2% cascading into further error, just like how they behave interactively.
I learnt some explanation methods about neural network before LLM appeared. But I think large LM it too large to be explained, there are too much parameters, which means informations about how LLM works.
In my opinion, reliability is the last huge challenge in the road from current LLM to AGI. If current LLM is reliable, we will know when we can trust it 100%, when we cannot trust LLM and why, how we can improve it, after that, we always can achieve AGI.
Human intelligence is far from reliable or trustworthy; why would you expect AGI to be better? Especially when we’re building it using architectures inspired by our own brain structure.
In my opinion, Human intelligence is reliable, as least for himself / herself.
The content I output are based on what I knew, there is a reasonable process from what I knew to what I output.
Maybe sometimes what I output are not right externally, but it it for me right now, This is how I define "reliability".
How much psychology have you read? There are any number of experiments showing that our minds are much, much less reliable than we think.
To the extent people are reliable it’s because we have checks and balances, rewards and punishment. In the small groups we're evolved for, if you lie a lot or cheat, the people around you will learn not to trust you and you’ll lose social connection, which translates to emotional pain. Humans have significant amounts of cerebral cortex to track this. In larger societies we can’t scale up to keep doing this, but the smaller subgroups like offices and hierarchies help.
But when you add intrinsically unreliable AI agents with no social connections, all this breaks down. RHEL helps, but post-training there’s no reinforcement; after ChatGPT makes up an answer or acts according to biases encoded into it we have no way to punish it. You can’t shun it or complain to its boss and get it fired or take it out for drinks and change its mind.
I only knew a little psychology, but I agree with your meaning about "our minds are much less reliable than we think".
Even when people are lying, there is still a reasonable thought process involved. This is something LLMs often lack.
I know what you mean, but there isn't absolute ground truth for reliability, sorry for the misunderstanding, I think reasonability is a better word to express my idea.
I agree. Humans can be objectively wrong, but they're very often consistently wrong. There's an "internal state" that seems to be lacking from LLM output.
I agree that framing AI from the utility POV is more tractable.
To your first sentence: Anyone who makes a claim that a technology is a “dead end” is making a prediction about the future. Very few of such people take such predictions seriously. Even fewer make testable predictions on the record. Even fewer will be held accountable if wrong.
People like to speculate. I tend to heavily discount claims that aren't (a) logically derivable or (b) clear and testable, where the author is accountable.
I think we're just getting away from the idea that algorithms are unequivocally good things after sites like Twitter, Meta (and friends) really screwed the pooch with them. At this point any large site with a closed algorithm is a public/democratic safety hazard and should be banned (is my opinion).
AI is effectively a black box algorithm on steroids. It's useful for a bunch of things but I agree with this assessment that either we use LLMs as a generative component of a more rigid framework (something which I feel is already happening) or we always keep a human in the loop.
I think it’s clear that absent massive advances in electric power generation (fusion?) LLMs are a dead end because they’re going to need to get so much bigger to do anything much more useful.
I think they’re not a dead end in the sense that they are already capable enough that as part of some kind of ensemble of computing approaches it’s plausible that something very new and cool could happen.
I don't think they need to get bigger and I think big leaps are likely to lower power consumption. For example, let's imagine that an LLM could "learn math" if it got big enough. Well, we could just not do that and instead have a smaller, more efficient LLM write "math programs" and offload their execution to a calculator. That'd be cheaper and more accurate.
I think it’s clear that absent massive advances in electric power generation (fusion?) LLMs are a dead end because they’re going to need to get so much bigger to do anything much more useful.
Please define "dead end". Can you convert this into a testable prediction?
Otherwise, down the road, how will you know you were right or wrong?
Given the relative efforts of unsupervised training versus human error correction and feedback learning, there can never be confidence in correctness, arguing from scale alone.
I really hope the author isn't setting the bar at 100% certainty of 100% correctness.
The sentence needs rephrasing. I suggest this: "It is important to get statistical bounds on correctness of the models we use."
There is a large body of work on various metrics for LLMs. They include various measures of quality, such as correctness, alignment, helpfulness, and more.
Mechanisms for fine-tuning LLMs are important and under active development. There is RLHF, Constitutional AI, and more. This work is fascinating on many levels, and there is more to do. It is hard to know the details of what various frontier labs are using for their training. It is reasonable that we would want more transparency about the quality and reliability of our models.
I didn't get much value from the article. To my eye, it didn't teach nor clarify.
If you want to learn more, I suggest finding resources on AI Safety. Here are two I recommend. First, Dan Hendrycks has a free online book. Second, BlueDot Impact has its AI Safety Fundamentals curriculum online.
You're not wrong that there is a ton of interesting research, but people have so many unconscious cues for correctness, and we can't reliably predict the correctness of the next token with any more total probability, reliably, than the models themselves can. Really interesting things are happening in cross discipline research in tandem with NLP expert guidance, but, for general use, there's an information hazard that is not well understood and is having unobservable knock on effects, and this has been the case for quite some time with no meaningful signs of improving.
I totally get the point of the author. But I think that with proper use LLMs can favor experimentation, which is necessary to foster innovation. It is very easy now to build a POC of a non trivial application in a few hours. And one can learn a lot in the process.
What I see a lot these days is though that using LLMs is amplifying the Dunning-Krueger effect and making it more difficult to build software that is well designed and easy to maintain. In fact while providing code snippets, the LLM does not a good job in systemical thinking and the if you go beyond the POC with a LLM the chances of ending with spagetti code are high.
Yes.
However, it exposes a very interesting phenomenon in many people's thinking in computing: that many people have belief systems that are religious in nature. They do not care about evidence, rationality and logical reasoning: they just believe.
So, for instance, I've found that the Unix world in general is now so large, and so old, that many people have erroneous folk beliefs that are set by 50+ years of tradition. They're not faith: they just don't know anything else. Their little domain is the whole world because they've never known there was anything else.
That's human nature. It's sad -- there's so much more out there -- but it's how people work.
So, for instance, a few random erroneous beliefs from the Unix tradition are:
None of them are even slightly close to general truths, but they are axioms in Unix.
But that's just ignorance.
The faith thing is much more frightening to me.
That LLMs are artificial, meaning that they were made by people. (They weren't, aren't, and can't be.)
That they are intelligent, meaning that they can think in any way at all. (They can't, but they fake it in ways some find compellingly persuasive.)
That they can program. (They can't, but they can make up text that looks like code.)
That they can analyse and improve themselves. (Nothing can.)
That bigger models will make them better at what they do. (This one just exposes basic ignorance about very simple statistics, such as scaling effects.)
That LLMs are in any way a step to reasoning, thinking software. (Pure magical thinking, based on lack of understanding.)
The thing is that I now continually encounter smart, educated people who believe in these magic beans.
There's two things I want to push back against in your comment.
1) The magic bean believer strawmanSo much of the LLM discourse I see (especially here on Lobsters) is polarized to the point where you're either a believer or a denier. I see people who make very reasonable and tempered claims about the utility that LLMs provide them (sometimes with evidence!) that are blanket rejected because "ChatGPT can't count its own toes correctly" or whatever new chicanery people have devised to get it to spit out a wrong answer.
Yes, there is an LLM cargo-cult. Yes, there are CEOs aplenty who want to sell you magic beans. Yes, I am sick of it too. But can we please reserve our hate for the myriad people and ideas deserving of it and openly hear out the ones who aren't coming with this kind of agenda? The ones who honestly believe they have found something helpful?
2) LLMs ~ magic beansIt's not clear to me whether you're arguing in your comment that LLMs have no merit whatsoever, but since that's a common sentiment I see and want to rebut, you'll have to forgive me if I have inferred incorrectly.
The other thing that bothers me is the almost religious rejection of any LLM results and realities. Correct me if I'm wrong, because I only speak from vibes, but I feel the anti-LLM sentiment when copilot came out was "LLMs will never write code." Advent of Code this year, for example, has had sub-30 second submissions on the leaderboard – if this is not evidence that LLMs are capable of some kind of programming, I don't know what is, because humans surely cannot read the whole prompt and code in that much time. And now the sentiment I see has shifted to "well, LLMs can write some code but it's not complicated," seemingly in denial of these previous claims.
I want to remind/inform whoever's reading this that in the decades-old field of program synthesis, the poster child for the longest time was FlashFill, which generates (simple) Excel formulas from examples. There simply wasn't any usable general-purpose tool for program synthesis (regardless of hallucinations or syntactic inaccuracies or …). Now a large number of synthesis papers are (in a very simplistic and reductionist approximation) LLM + special domain.
You can debate whether LLMs in their current form have legitimate utility, but this debate becomes personal and I expect colored by your own perception of how they help you (placebo/nocebo). I think it's too reductionist to write them off entirely.
These are but a brief summary of my thoughts on LLMs and program synthesis, I hope to getting around to writing more in the new year…
The problem with this framing is that you're only looking at the ends, when for many of us the means play a part in the resistance.
If you bring an LLM that was only trained on content whose authors actively assented to inclusion, and where no companies were seriously considering building their own nuclear reactors because it takes so much power to run, and where there's some hope of ownership and control of the LLM by end users instead of just large technology/surveillance companies, then sure! I'm all ears!
Alas, there is precious little of that in the space where Copilot and ChatGPT and Claude and so on are playing.
Completely agreed. I'm not interested in having discussions about trying to invent useful applications for this technology, because even the small-scale models that hobbyists can run on their home computers are produced at massive expense by extremely unethical actors out of datasets which were harvested without consent and then legally encumbered.
Supposedly innocent tinkering with LLMs furthers the goals and rhetoric of absolutely monstrous entities and helps them justify their abuses of the commons. Building hobby projects on LLaMa and writing effusive blog posts about cheating at code golf, automatically synthesizing a fart app for your phone, or accomplishing any other number of trivial tasks with help from a Claude-powered rube goldberg machine is doing free marketing for the generative "AI" merchants and cloud vendors.
LLMs are completely radioactive from many ethical angles even before you start digging into the harmful ways they're being applied today or (as this article focuses upon) their complete unsuitability as a building material for reliable, efficient, or generally trustworthy software.
Then (as someone who believes these points should be made) I implore you to focus on the first half (not struck-through) part of what you're saying. I think it does your argument no good to incorporate claims that are deniable, especially if they apply the fallacious reasoning I discuss in my parent comment.
You don't even need to open the door to arguing whether in two years ChatGPT can write Minecraft if its existence today already constitutes as significant of a problem as you claim. I think it's good to have people thinking critically about these tools (ethically and technically), but thinking critically means not getting swept up in the LLM hype and the anti-LLM hype.
I think it is good that companies are building their own nuclear reactors to power datacenters for generative AI applications. We need way more of this kind of thing, cheap energy makes all of society more prosperous and we get cheap energy from having lots of nuclear power plants generate it.
Ownership and control of LLMs by end users is important and it's a genuine concern that we don't get locked in AI systems controlled by a small number of companies. But this is not really a different problem than a small number of companies controlling proproetary software platforms used by huge swaths of the population (i.e. Google and Meta and Twitter and Apple existing).
How many are actually doing this? Nuclear power plants take many years (often a decade or more) to build. Most of the announcements I’ve seen have been ‘we aren’t a climate disaster, look we have this long-term strategy to be energy independent. We’re not actually funding it at the level required to be realistic, but let us burn vast amounts of coal now, we promise it’s a temporary thing!’.
@dgerard has a critical look at SMRs here
https://pivot-to-ai.com/2024/10/17/google-amazon-buy-nonexistent-mini-nuclear-reactors-for-ai-data-centers/
Even if a SMR is built, it still needs huge volumes of clean water to operate and cool. It's never going to be a little container-sized cube that magically emits electricity, like a portable fossil fuel generator.
Then make it illegal to burn coal to incentivize building the nuclear power plants faster (and reduce the amount of regulation on them); I do not want to sacrifice humanity's ability to harness prodigious amounts of energy on the altar of the climate. This isn't even about LLM datacenters specifically, I want nuclear power plants powering aluminum smelters too. Or maybe photovoltaic solar is actually cheaper if we just pepper the earth with panels, in which case I support doing that, and the electricity can go into LLM datacenters as easily as it can go into anything else.
What I don't want is for any human endeavor that uses a lot of electrical energy to get labeled a "climate disaster" by people who want to shut it down - least of all because scrubbing CO2 from the atmosphere itself is something that's gonna require a lot of electrical energy.
That's the thing; no matter how clean or plentiful your energy source is, there are so many better uses of that energy than LLMs!
I'm glad you agree it's a problem! The difference, I think, is that I'm not constantly hearing about how I should learn to stop worrying and love Gmail.
That's because the hardcore RMS-ish view that Gmail is unethical software because it is proprietary is low status.
You hang out in very different places than I do, it appears. That's interesting to realize... I don't even know what it would be like to be in a place where that view is low-status. It felt like the mainstream belief within the broader software community, when I was growing up. It still feels like a common belief to me. It's really interesting to hear a perception to the contrary; thank you.
As a bit of extra flavour… I, and the people I grew up with, shared the same belief you did: we believed that software like Gmail is unethical and that we were taking the moral high ground by believing that.
For me, though, there was a moment along the way where… I still love open source software and wish more of the world was made up of OSS, but also came to the conclusion that many of the people I grew up with who held those beliefs the strongest… weren’t really going anywhere. Many of them are still exceptionally talented programmers but they’ve lived a life where they’ve consistently struggled to make enough money to house and feed themselves. Whether they’re considered low status because of their beliefs or because of their relative poverty or due to other lives choices is hard to say but it’s pretty tough to argue that they’re doing well in life.
In my life now, from where I’m standing, it seems like the general broad perspective towards OSS is mostly indifference. Most people I know run their production software on Linux, some on Windows, and none of them really do it for OSS reasons but rather because Linux is a good platform. They don’t really care that it’s open source, just that it works. I’m actually feeling a bit sad writing this.
I don't think it makes sense to evaluate any idea based on perceived success* of those that hold it. Especially when that idea is likely to make you avoid chasing the riches of the tech industry.
To be blunt, reading that it sounds to me like you were willing to compromise those ideas for money and don't value the same things as the people "not doing well* in life."
* this is subjective as it depends on personal goals only those people can judge themselves
I agree most people are apathetic to things being FOSS, and that is quite sad but also why activism is needed. Not only for FOSS but all subjects; the status quo has strong inertia, if not players with incentive to maintain it.
To wrap back to the original subject, I believe Gmail is unethical mostly due to Google's data pillaging and federation busting. Gmail not being FOSS is part of it, but far from the main reason, and mostly orthogonal: I believe VS Code is unethical even if it is mostly OSS and VS Codium exists.
100% will agree that I did compromise on those ideals in part for money but in the bigger picture for happiness as well.
The real tragedy with respect to the “not doing well in life” part is the magnitude of that. I agree it’s relative but it makes me so sad to see the some of the brilliant people I knew in school posting on Facebook (there’s a certain irony there…) about how their night-shift gas station job sucks and that they have to kick out another roommate for stealing or being abusive. It’s not just that they’re “not doing well” on my personal scale but that they also seem genuinely unhappy themselves.
But… this is all just one small community. I’m sure it’s not a representative sample.
That's fine to articulate, but my reply is to ~lproven's comment which makes no mention of the ethical considerations for using LLMs.
To be clear: I'm not saying that discussions about the ethics of LLMs should be silenced (I am inclined to believe otherwise). But I am saying that even if you think you have the moral high ground for not wanting to use LLMs, this doesn't entitle you to misrepresent what they're capable of. (not accusing you specifically, but I hope you get what I'm saying)
Put differently, I don't like the simultaneous standpoints I see people take of "LLMs are bad for authors and the environment" and "LLMs can't produce code, LLMs are stupid, etc." The first point is much more damning and — perhaps more importantly — easier to verify facts for. I don't see any good reason for denying evidence or inventing strawmans to support the latter point.
I feel the same but like... the opposite lol
I don't even understand this. Is it even a question? Of course they are artificial.
You're gonna have to define "intelligent". Depending on the context, LLMs are obviously intelligent, but under other definitions it clearly is not - in fact you'll find that many people have wildly divergent definitions. A panpsychist is not likely to say they're intelligence, a functionalist may. Even by some pretty rigorous definitions. This isn't a religious question, it's a metaphysics question. Is intelligence the ability to reason? Is it a sufficient set of justified beliefs/ knowledge? Does it imply abstract properties? Is it an abstract property? Is it physical? Emergent? Seriously, it's incredibly reductive to say "LLMs are not really any kind of intelligence".
Also confusing since that has been the case. The question is really just a matter of the limits of this scaling. It's sort of like saying "gzip isn't going to benefit from a larger disk because at some point compression has information theoretic limits" well yeah, but the bigger disk still helps. I'm not really sure what you're trying to get at here though, maybe you can be more specific.
I mean... this is just a weird definition to me. They use statistical reasoning to generate valid programs. I guess you think that's not programming?
WOW lol sorry but I'm out. You're seriously going to accuse people of religious thinking and then make these kinds of assertions? Like there are entire metaphysical theories devoted to these questions that you're so glibly dismissing. Nothing can analyse and improve itself? That is such a wild claim to just assert like this.
If we define reasoning as using evidentiary inputs to produce conclusions, LLMs reason. Statistical reasoning is reasoning.
These arguments (read: unjustified assertions) are so weak and you seem to not even realize the metaphysical commitments you're making by pushing this worldview.
If you want to have a reasonable discussion, by all means please do. I see so few discussions that look reasoned on this site and it's a bummer. Let's talk about knowledge, let's talk about reasoning. Let's do it! Let's ditch these assertions, let's ditch the question begging, let's have a little epistemic humility, okay?
Yeah, it doesn't really matter whether LLMs have intelligence and rationality. They can make our lives easier by solving some problems, just like cars and computers, and that's enough. That being said, I also agree with the article that LLMs need to be improved in terms of reliability and explain-ability.
This black-and-white language leads to confusion.
I recommend the following definition: Intelligence is the ability for an agent to solve a task. The degree to which such abilities span different areas is the degree to which we call it general intelligence.
Nope.
Place a blind invertebrate into a box with a particle of smelly food. Wait. It will move around in a semi-random walk but it will find the food. Problem: solved. Was intelligence used? No. The algorithm is so trivially easy, I've solved it:
Move in long straight lines until you smell food, then move shorter distances and turn more.
That's it. That is the algorithm. It works.
Pools of water connected by pipes: find the lowest level. It works.
https://en.wikipedia.org/wiki/Water_integrator
Yes, I am aware of this criticism.
According to my definition (which matches that of Stuart Russell, more or less), there is a wide range of intelligent behavior.
Do you want a definition that works across a wide range of situations? Or do you want a narrow definition? Why?
Can the above commenter offer a non-binary definition that is useful across a broad range of intelligent behavior? I would like to hear it.
Those who have studied artificial intelligence and animal behavior for decades often go with something quite like my definition. See e.g.
https://emerj.com/what-is-artificial-intelligence-an-informed-definition/
[Comment removed by author]
The only thing you did with your comment here is do an appeal to authority.
How about this: nobody is an authority on what is an intelligence and what even is reasoning. People get a PhD and start pretending to know, that’s the crux of it, let's not sugarcoat it please. That applies to every single non-practical discipline, philosophy and many others included.
Also your comment came across as disingenuous because you first offered a blurry definition of intelligence and then, when called out, retreated into the appeal to authority and that a need for nuance is needed.
I don’t see how the latter follows from anything at all. No need for nuance, whatever that means in this situation even; we need good working definitions of the sort “what constitutes a fusion reactor?” for example.
All the discussion about what passes for “AI” these days is just sad to watch and that includes your comments. It resembles theological discussion first and foremost.
Surely technically-inclined people can do better? Inputs, expected outputs, process, all that? You know, engineering?
Personally, your comment comes across as unkind. Maybe just a bad day? Are you willing to try a more constructive and charitable direction?
Some responses:
Remember the thread context; I was criticizing some flaws in a definition someone else offered. My definition addressed the flaws I pointed out.
I “retreated” above? I’m happy to admit if I made a mistake, but where are you getting this?
It isn’t fair, accurate, or charitable to say that “appeal to authority” is all I did.
I pointed to what I think are some good jumping off points. Did you read any of them (Russell, Hutter, etc)?
Do you know a better definition of intelligence? … and why do you think it’s better? (You can see in my other comments the need for a definition that isn’t human centric.)
I’ve seen the anti-PhD venom before. I used to say the same kind of thing decades ago. It was ignorance on my part. This lessened as I interacted with more people with Ph.D. experience and as read more machine learning and CompSci literature.
No experts? There are people with more and less expertise. Acting like there are no experts is an unhelpful exaggeration.
If you read the comments I’ve written, you’ll see it is difficult to place me into any simple categories regarding AI.
I find it bizarre that you think my comments are anything like theology. Offering a working and operational definition of intelligence does not theology make.
I’ve also seen the anti-philosophy trope before. It is unfortunate and self-limiting. The classic response here is: many other sciences and disciplines were birthed from philosophy, but philosophy rarely gets credit. Yes, some philosophy is painful to read. One usually has to put in a lot of effort searching, thinking, writing, and discussing to reap the biggest benefits. Asking for reading recommendations from people you respect is a good start.
Well, it might be cultural background on my part because I am not used to dance around disagreements, and as I get older this is less and less likely to ever change. Not my intention to offend, mostly to pull you away from what I perceive is a comfortable and maybe even complacent position.
I did simplify your comments, that much is true, and 99% of the reason is that the "AI" discussions inevitably devolve into "But how do we know what intelligence is? We might already be witnessing it but are disparaging it!" which, my turn to say it, I view as extremely unproductive, not helpful for advancing any discussion, and basically serving no other purpose than to congratulate ourselves how smart we are, and to offer a plethora of very out-there "what if"-s.
If you believe you are doing something more than that then I'd love to see more concrete claims and proofs.
I have not read philosophy on the mind and intelligence. I tried. I found it unproductive and very rarely did I stumble upon a lone article (not a book) where somebody actually attempted to invent / classify a framework in which we should be thinking about what mind / intelligence are. Everything else I attempted to read did read like empty hand-waving to me.
If you are willing, I'd like to get back to some semblance of a topic: do you believe LLMs can "reason" or are "intelligent"? Do you believe we are mistakenly putting them down while they, the poor things, are the next-stage-of-evolution, marginalized and oppressed artificial life forms that are just misunderstood? If not, what, apropo, was your point?
You’re being a bit of a dick here on multiple fronts, and misspelling the italicised apropos in the final sentence doesn't shake the look.
Misspelling can happen to anyone. Shame that the glorious and almighty "AI" never improved autocorrect in phone keyboards, right? I'm sure that requires godlike intelligence though, so it's excused. But hey, it actually "understands" what it does and it's obviously intelligent. Surely.
And I'm not interested in philosophical discussions unlike a lot of people who can't help themselves every time the "AI" is mentioned.
I'm interested in seeing proof that their downright religion-like beliefs have any rational foundation.
Alas, that kind of expectation is misguided. Faith doesn't require proof to exist, as we all know historically.
If challenging belief makes me a dick then I'm okay with it. I was still never answered in a satisfying manner.
My conclusion is that this is a cozy topic for many. Mention "AI" and they all pull up the cigar and the 20-year old whiskey, and they're all suddenly decorated philosophers who are perfectly qualified to speculate and to present conjectures as facts while they want us to believe that their entirely unrelated Ph.D. makes them educated on a topic that absolutely nobody has ever solved.
So yeah. Believe what you will. But recognize it's only that: a belief.
Well, I think it matters a lot if LLMs have intelligence because the implications are pretty huge. Kind of like, "are there objective moral goods" - if we could answer that question we could rule out all sorts of metaphysical theories about the universe, and that seems valuable to me. Practically, and I think to your point, whether it's intelligence or a facsimile of it (assuming this distinction even makes sense, which is a HUGE assumption!), as long as the results are the same it isn't important (in terms of how it's used).
I also agree with the article. I thought it was well written and makes sense - I thought the idea of breaking down models into testable components was particularly interesting. The comment I responded to doesn't even seem related to the contents, which I also thought was ironic since it was a plea for rationality and informed commenting.
There’s an even further metaphysical question that goes with that… how do we even define intelligence? What’s the threshold? Is a tree intelligent because it grows its roots towards water? Are bacteria intelligent? Fish? Cats? Dolphins? Horses?
Are current ML models like any of these? Or are they more like https://en.m.wikipedia.org/wiki/Clever_Hans? Or was Clever Hans actually intelligent, just not in the way that it was claimed? (He couldn’t do arithmetic but he could very accurately read subtle human signals)
All really interesting things to ponder on holidays :)
Well, OpenAI recently defined "artificial general intelligence" as "whatever enables OpenAI to realize $100B in profits".
Yes/no questions fall flat quite often. For many interesting subjects, including intelligence, consciousness, justice, fairness, etc., there are better framings.
We already have human-created machines that try to answer these questions. They're called religions. Adding another one with a LLM dash of paint will probably not resolve anything.
No, religions don't inherently try to answer these questions. These are questions that fall into the domain of philosophy. Whether moral goods exist, their nature, etc, is not a religious question but a metaphysical one.
My statements stands; the ability to answer questions about the mind would indeed hold value, it would resolve many open questions, rule out some metaphysical theories, make others more or less unlikely, etc. It would potentially have very direct impact on human life - theory of mind is essential to theory of personhood, for example.
OK. I read your original comment as something akin to Platonism - there are eternal truths that are hidden from us. How a machine trained on the sum of humanity's writing on these questions would be able to reveal them was unclear to me.
I see. No, I'm not expecting an LLM to reveal its truths. I'm saying that our investigation into knowledge, perhaps through exploration of technologies of LLMs, will reveal the truth.
Again, going back to Descartes, one of his main arguments for human exceptionalism as well as mind body dualism was that humans can speak. He suggested that machines could move, look like animals, look like humans, etc. He suggested that one could not differentiate a monkey from a robot monkey. But he maintained that language was the indicator of a true mind - so a machine could never master language.
LLMs at least provide evidence against that. You can argue about whether they're a true counter example, but they have are evidence against a Descartes theory of mind. As we continue to push forward on technology we may have stronger evidence for or against other theories of mind.
Let me try to understand here: because LLMs disproved an extremely flawed hypothesis by a scientist who is a product of his time, this means... what exactly? Can you help me here? That LLMs possess mind, or is it something else?
No, not at all. I was giving an example of how our ability to create machines has historically changed our understanding of our theory of mind. It was an example of how producing new technologies can help us evaluate existing theories about things like intelligence. Applying it ot a very old, well understood (and unpopular, therefor not contentious to apply to) theory was my way of giving such an example.
As I said, I think that as we generate technologies like LLMs we may be able to generate evidence for or against theories.
Definitely not that LLMs possess a mind.
Maybe it is an example of how producing new tech can help us evaluate existing theories, yes... if those theories are not as old and almost laughable. Because comparing to them is a classic "tearing down a straw man" debate.
There is no straw man here, I don't think you know what that term means. It is chosen explicitly because it is not contentious to say that dualism is rejected, and to show how one could use LLMs as evidence. That it is already rejected (largely) is the benefit of using this example.
Literally all I was doing. Soooo we're good? Same page.
Sure, I don't know what it means. OK, lol.
Whatever helps you, man. :)
No, they are not.
Artificial, meaning, built by artifice, created using the skills of an artificer.
LLMs are not built by humans. LLMs are built by software built by humans, running in large-scale clusters in datacentres. LLMs are a multi-dimensional grid of statistical weights of relatedness of words, with millions of dimensions.
A good simple explainer is the Financial Times's one here and a more in-depth one is Stephen Wolfram's one here. Just in case you were about to accuse me of not knowing what I am talking about.
The point being that humans didn't construct those databases, can't read them, can't analyse them, and cannot modify them.
The humans built the software that built the models. Humans did not build the models, and can't.
The tools that built the models are artificial. The models are not.
Cambridge dictionary: showing intelligence, or able to learn and understand things easily
Intelligence: the ability to learn, understand, and make judgments or have opinions that are based on reason
Merriam-Webster: having or indicating a high or satisfactory degree of intelligence and mental capacity
Intelligence: the ability to learn or understand or to deal with new or trying situations; the ability to apply knowledge to manipulate one's environment or to think abstractly as measured by objective criteria (such as tests)
These are not abstruse technical terms.
LLMs cannot reason, understand, deal with new situations, etc. ALL they can do is generate text in response to prompts.
They cannot count. They cannot reason. They cannot deduce. But they can produce a modified version of text which does those things.
It is a fallacy of thinking to leap from "it produces text which sounds smart" to "it is smart".
It isn't. Something that can't tell you how many Ms there are in the word "programming" can't program.
Yep, 100% am. Refute me: prove me wrong. Falsify my statements.
Nothing you've said falsifies my points. All you are doing is mocking and trying to redefine terms.
This is just a really weird standard. If I build a chair from Ikea parts, did I not build a chair because someone else built the parts? What's the point of this definition?
It's not that you don't understand the technology, it's that you're applying ridiculously strict terms to it and acting like anyone who doesn't take those as gospel truth must be experiencing irrational, religious thinking. Nothing in those pages is going to justify a flat assertion that an LLM is not "artificial" tbh but even if they justified it I don't think it matters - I mean the stakes on this couldn't be lower.
Shrug. I don't think this matters, it's just pedantic. "A human built a machine that built the thing" okay. I've never programmed because actually someone else built the keyboard and the compiler that produced the assembly, so no, not programming. I'm a fraud, I suppose.
Dictionary definitions are fine but I think if you follow the citations there you'll find they're often cyclic. They're also not some hard truth. These are metaphysical concepts with entire papers dedicated to breaking them down. Pointing to the dictionary as an authority isn't a strong argument.
I disagree and I've already justified why.
It's not a fallacy it's the foundation of functionalism, which is a metaphysical theory worth taking seriously.
Baseless. Why should we connect these two things? I can't factor massive primes in my head but I can program. Why should I connect these two things together? Again, LLMs can produce valid programs, so I think it's on you to justify why that isn't programming since it's intuitively the case that it is.
I provided theories counter to your assertions. I mean, should I break down functionalism for you? I brought it up, you can learn about it if you'd like. I brought up panpsychism, abstract properties, etc. I think there's plenty of breadcrumbs if you want to learn the metaphysical foundations of what I've described.
If you really want to provide some sort of formal syllogistic argument instead of just baseless assertions I could probably provide really strong, rational, well researched arguments from people who study these things explaining why someone could rationally reject your premises or conclusion. Like, my entire point is that what you think are "rational" premises are totally justifiably rejected by people.
I'm not redefining terms lol these terms all have varied definitions! That's my point. A functionalist and a panpsychist will have radically different definitions of intelligence. If you want to plainly assert things like "nothing can ever analyse itself" well jesus christ dude that's a massive metaphysical commitment that you are making. I'm not the one making positive assertions here, I'm not the one accusing others are being irrational.
TBH you're the one who started off with the mocking "everyone else is irrational and 'religious thinking', now here's a list of baseless assertions" sooooo idk, I feel really fine with my response. I don't think your list of assertions requires much "refutation", I can just show that they're incompatible with very reasonable metaphysical theories and so anyone who subscribes to those theories is perfectly justified in rejecting your unjustified assertions.
That's absolutely fine and you are of course completely free to say that.
I have been a skeptic (with a K, which is not the normal spelling in my country) for about a quarter of a century now. But that's when I learned the label: I've had the mindset since roughly when I reached puberty.
There is a whole tonne of stuff I don't believe in that billions of people passionately, fervently believe. Many of them would kill me for it. I don't care.
I have seen no evidence that LLMs are in any way intelligent, that they can reason, or learn, or think, or deduce, or in any way reproduce any of the elements of intelligent behaviour.
Metaphysical or philosophical arguments along the lines of "we don't know what 'intelligence' means' or "what does it mean to 'think' anyway?" are just pointless word games. Have fun with them if you want but I'm not interested.
So, I can happily state my assertions:
This is discussed in depth in this Nautilus article: https://nautil.us/ai-is-the-black-mirror-1169121/
It's the silicon valley tech-bro mindset.
I think you're wrong, but I don't think there is any evidence I or anyone can come up with to convince you.
That's OK. It's just a passing annoyance. The bubble will pop, there'll be another AI winter just like circa 1970 (+~5) when they worked out the limitations of single-layer neural networks, and then again circa 1985 (+~5) when Unix got cheaper and faster than Lisp.
Personally I am looking forward to it. It is very irritating.
I also don't believe that there's any practical useful application of blockchains, anywhere or for anything, and that all cryptocurrencies, NFTs, Web3 and the entire field is an empty bubble that will implode.
Other things I am happy to tell billions of people they are wrong about...
Supplementary, Complementary and Alternative Medicine. It's all 100% fake. Medicine is what can be proved to work; if you can't, it's not medicine. That's why I use the term: SCAM. It's a multi-billion dollar industry and it's entirely and completely fake from top to bottom.
This one is easy: show it works, and it immediately ceases to be SCAM, it becomes conventional medicine.
Religions. All of them. There are no gods, no soul, no afterlife, none of it. Every believer in every religion alive and who has ever lived: all totally wrong.
I am perfectly happy to recant, if I am provided with objective, verifiable, reproducible evidence of the supernatural.
But until then, I repeat, it is all completely fake, and so is the entire field of LLM-driven "AI."
None of my arguments are "lots of people believe X so X is true". What I'm saying is that there are very rational, viable, reasonable metaphysical theories where your assertions are either outright rejected or are unlikely, and you saying that anyone who doesn't follow your line of thought is irrational or thinking "religiously" is itself an irrational statement, unaware of the dialectic.
That is not what I'm saying. Different metaphysical theories will define these terms differently. Again, it's just ignorance of the dialectic to say things like "AI isn't intelligence" - not only is it an unjustified assertion, it doesn't even make sense without defining the term, and it would be great to align whatever your definition is with some sort of established metaphysical theory so that we can know your commitments.
It's very common for people who aren't familiar with actual logic, reasoning, and metaphysics to think that it's just "word games". You're the one saying that people should be rational. You're just not meeting the standard you've set, from my view.
I'll be honest. Your assertions are lacking. They're informal, lack coherent premises, bring in tons of terminology that you've failed to define, etc. For such bold claims I'd really expect more.
Again, you are the one making the positive claims here, it's kinda on you to do better if you want to accuse everyone else of being irrational.
These are just incredibly bold claims. To say that there are literally no useful applications of a technology is extraordinarily profligate.
Yeah, I can't stand this form of atheist. As an atheist myself, I find it quite annoying how modern atheism treat religion as totally irrational. I'm what modern atheists would call a "strong atheist" (garbage terminology but I'm going to speak in the terms that I suspect are more familiar to you) but I think anyone tho thinks that you can't be rational and religious at the same time is just extremely ignorant of the actual papers and research on the topic. I'm not compelled at all by that research but I'm aware of it and understand its legitimacy.
I'll be honest with you, I think you should take some time to learn basic philosophy and metaphysics so that you can understand how much you're oversimplifying things and how bad of a job you've done at presenting anything like an argument. I don't mean this as an insult, I say this as a peer and someone whose views almost certainly align to a degree with yours. You should learn what a basic syllogism is - a good exercise would be tot ake your assertions and break them down into distinct syllogistic arguments that you then defend in terms of evidence and justifications for your premises and a justification for the conclusion following from them.
That said, here's my response to your assertions. It's a bit tricky since you haven't presented any kind of theory or aligned with any existing theory, you haven't defined terms, you haven't justified any beliefs, you haven't put your arguments into any kind of formal logical form like a syllogism or a bayesian inference, etc. This is very "internet argument" level, not what I'd want from someone telling everyone else how irrational they are for disagreeing.
I rejected this already. I reject your definition of A, you've failed to define I entirely. I've already pointed to functionalism, so if you want to reject functionalism by all means you can feel free to do so but I think it would be quite ridiculous to call all functionalists irrational even if you disagree with them. Functionalism is well defended, even if there are competing theories and great arguments against it - I brought up panpsychism, although I suspect you would reject panpsychism as, despite being compatible with atheism, it does strike me as being less likely under atheism. But that's a whole separate issue.
I'm going to break this down because this is actually numerous assertions.
2a. LLMs are not a pathway to AGI or anything else.
Unjustified, lacks proper definition of terms.
2b. with few useful real-world applications
I reject this trivially. I find them useful. Trivial for me to reject this.
2c. and the construction of ever-more-complex prompts in efforts to constrain them into producing useful output is futile:
Possibly justifiable, though you didn't do so. But I would probably grant this if "useful" meant something along the lines of deterministic or "highly reliable" like the article mentions. I mean, I would grant it because the stakes are low and I don't care to argue about it, I don't actually think it's true.
2d. it’s just a vastly inefficient new form of programming
I wouldn't care to reject this because "inefficient" is so subjective and ill defined.
2e. one which can never work because the best it can ever do is a small statistical reduction in the amount of noise in the output.
"Work" isn't defined, "small" is confusing, unclear what your point here is.
I think this is a really odd take since machine learning algorithms are statistical models. To help you out, what you're looking for here would be called a "symmetry breaker". If you grant that those algorithms are useful but you reject that LLMs are you need to show why.
Honestly, I find this deeply ironic. I find your post highly guilty of what you're accusing others of. I think you should deeply investigate these areas and the rational defenses one can put forward before throwing stones at others like this. You're clearly unaware of the academic discussions around these topics, how arguments are formed, what different types of reasoning are, what evidence is, what rationality and what rational beliefs are, etc etc etc. I think I've provided plenty of information for you to learn about these topics if you'd like to raise the bar for discourse, something I'd personally love to see.
Again, I'm not trying to be insulting. I'm a bit too lazy to be less glib here, I hope that the mere fact that I've taken the time to try to express myself to you shows that I'm willing to engage constructively. I can see that you want to elevate the discourse, that you're sick of irrational beliefs, and I'm extremely sympathetic if not empathetic to that worldview. It just seems that you would benefit greatly from learning about what that has looked like over the course of the last few thousand year and to learn about how many metaphysical theories exist that are arguably very rational.
If you want to just make baseless assertions, go for it. That's how most people talk online. But since you went out of your way to point out how you think others do that, I thought it worth pointing this out to you.
If I were to really dig into your assertions I'd have to do a ton of steel manning of them and expand on them to make them something I could actually try to refute or justify competing theories against. You'll have to forgive me, as much as I'd like to and believe it's worth doing, I don't have the time right now. Maybe I'll write something up at some point. Until then, I'll suggest we just agree to disagree. Cheers.
Unbelievable that this is flagged as "troll" lol this site is so fucking ridiculous sometimes. @pushcx can we please get some kind of system for handling people who erroneously flag content? Disagreeing with me is fine but it is nuts how often I put in significant effort, provide justifications, even reference research and papers, and get flagged because someone doesn't like my post. This has happened way too many times to me.
I not only explained clearly what an argument is, what the current academic conversation looks like, etc, I even went out of my way to go beyond what I think was even reasonable to expect and responded to these assertions despite the fact that they are so poorly expressed - I was unreasonably charitable in my response, especially given the context of this conversation starting through someone stating that everyone else is irrational and won't engage in logic.
Here's another absolutely ridiculous example of me being flagged as a "troll": https://lobste.rs/s/4czo0b/dropping_hyper#c_fsyjel
I could find more but idk how to scroll further into my comment history.
You provide exactly nothing except long philosophical essays in technical discussions. It's quite telling that you don't see why you got flagged (for the record, I have no ability to do so on this website but I absolutely would if I could).
You have "rejected" and "proven" exactly nothing as well. All you do is hide behind metaphysical clouds and empty philosophy while pretending to give objective evidence.
Yeah, many of us see through that and do perceive it as trolling. Until you come up with something concrete then you will not convince anyone that LLMs / "AI" are not snake oil and that their next winter is coming up soon and it's going to hit hard and last decades.
I actually can believe that you are fully believing what you say, which would be even sadder. In conclusion of these fruitless and never-going-anywhere "discussions" (because I am 100% convinced you will double down; prove me wrong!) I can only tell you that every now and then it's good to revisit your own bias and bubble. Now THAT would constitute true intelligence in my eyes. ;)
By the way, I used f.ex. Claude with success. It helped me fill the gaps in my knowledge in a niche area that I was not having the patience to invest learning from scratch. I was impressed and I loved the experience.
...But it also demonstrated that to me that people can very quickly just spiral into "no, that's subtly wrong, I need X and you are giving me 0.85*X -- let's keep refining this until I get what I want". I actually viewed the daily prompt limit (free tier) as a very good thing: it forced me stop and revisit some assumptions that I subconsciously did along the session, and for good reason -- turned out I was going in the wrong direction and was given wrong input to test an algorithm with (which really did make me laugh because Claude did not validate that input either).
And in conclusion to that topic: LLMs will at best just remain very useful tools for a bunch of tasks. They are not intelligent by absolutely any meaning of the word except those that are super loose and just love hiding behind clouds of philosophical ambiguity... like yourself.
No need to reply as reading your comments here has convinced me you are not capable of productive replies that evoke true practical observable evidence. But you do you.
You can dislike my posts but to say it's trolling is just silly. You can think "wow this guy is so dumb!" but that's not against the rules, it's not trolling.
I've never pretended to give objective evidence? What are you referring to? As for "hiding", what?
Let's remember how this thread started - someone stated that anyone who thinks LLMs aren't a scam is guilty of being irrational and "religious thinking". That's the first comment. I justified why I reject that by pointing to many live, well thought out, rational metaphysical theories that would reject this. This isn't hiding or "empty philosophy", it's a direct contradiction to the idea that anyone disagreeing is irrational or religious thinking.
I have zero requirement to do this. The positive assertion made was that anyone who believes that LLMs / AI are not snake oil are irrational. All I have to do is show that that's not true by showing live models that are rational.
I wonder what it is you even think that I believe? I've made almost no commitments in this discussion because, and I've said this a few times, I'm not the one making the positive assertions.
I mean, lol. Congrats but according to the person I was responding to you are irrational and guilty of religious thinking because you think that LLMs have any use at all. So... what is it you disagree with, exactly?
Anyway you've just done what the other poster has done. You say "They are not intelligent" okay well that's a fine opinion? You're not justifying it. I guess you think justifying anything would be "hiding behind clouds of philosophical ambiguity" idk it's super irrelevant because the premise asserted by the author was that anyone who disagrees with that statement is irrational and religiously thinking, but you think LLMs have a use so they think that about you too!
As predicted, you doubled down. There's zero substance to your "I mean, lol" bit because I am pretty convinced this person would recognize where an LLM can save you a few minutes (if not then they are indeed too irrational).
The crux of what people like myself say is "take the tool, use it, stop believing it's something more than it is because it absolutely it is not".
To have any semblance on topical discussion: I claim that we have zero intelligent machines today. No intelligence. No reasoning.
I owe no proof for refusing to believe something I cannot find even if I tried very hard to find it (I mean, who wouldn't want their job being done by agents they pay $20 - $100 a month for?). You are the one owing proof to me if you say LLMs are "intelligent" and that they do "reasoning". The burden of proof is on the one making the claim that something exists.
So go on. Prove it. And no "according to this scientist X and the other Y who, like all of us, have zero clue what intelligence and reasoning actually are but have some credentials so they love to pretend that they know". No -- nothing of that, please. Hard facts. Let's see them.
An easy prediction to make since... I still think I'm right. I mean, why wouldn't I?
We must have a pretty different reading of their posts. They make some extreme assertions about LLMs being useless and a scam.
But I have no problem with that? I have a problem with saying that if someone thinks LLMs are useful then they are irrational and guilty of religious thinking.
I literally don't care. I've argued only against the idea that anyone who disagrees is irrational or religious thinking.
They are the ones who made positive claims. I have never made a positive claim. I owe nothing other than a rational response to their claims. The fact that you seem to not understand this leads to my next point.
Look, I'm genuinely sorry but I don't think you or the other poster know what words like "fact", "evidence", or "reasoning" even mean. I've realized how fruitless it is to try to talk to people about things like this because I just don't think an internet forum is the right place to teach someone what these words mean.
[Comment removed by author]
I see. Well, if your only goal was to make your parent poster less extreme then cool. I kind of thought that you were going into the other extreme: "LLMs are the future" and "LLMs are AGI" etc. bullcrap.
Nope. I never said anything about LLMs other than that one can rationally disagree with the parent poster's view on them without being irrational and "religious thinking".
My thoughts on LLMs are not reflected by any of these posts, or are barely reflected. They made positive assertions that were extreme and, frankly, ridiculous. They failed to justify them whatsoever. I presented rational theories that would reject them, which is all that is necessary to refute the idea that anyone disagreeing is irrational.
For this, I am apparently trolling.
I'm not sure I get the jump from LLMs to Unix
Are you saying LLMs are pushing Unix or something?
Anyway, I claim that the general Unix / Plan 9 / Web / "REST" architecture is fundamental, not accidental. I wrote many words on that:
https://www.oilshell.org/blog/tags.html?tag=software-architecture#software-architecture
The details of Unix, like say
stat()and terminals, are not fundamental, but the general architecture is. It could have been Plan 9, which is basically a "cleaned up" Unix in my mind ("everything is a file", etc.)It's a mathematical argument with respect to the amount of software that has to be written. Narrow waists are a way of mitigating combinatorial explosion / enabling interoperability.
This argument goes back to a thread from a few years ago:
https://lobste.rs/s/vl9o4z/case_against_text_protocols
The blog posts were basically a big elaboration on my reply to that comment
(As an aside, I think that @matklad has come around to the viewpoint that Emacs/Vim are also fundamental -- i.e. they are centered around the narrow waist of text, or attributed text. As opposed to having different programs that edit each different type of data. Again, this is a mathematical issue with the amount of software that has to be written by us, and learned by users.)
With regard to C -- I will again claim that if you look at semantics / expressible programs, then C++, Zig, and Rust are a form of C. They all use LLVM, or started out using it.
People argued with Bjarne for years about this -- why does C++ use the C machine model? Isn't there something better?
And a design principle of C++ is that "it does not leave room for any language below it". This has actually been true!
The main new language I know of that does anything different is Mojo. Mojo is built on MLIR, which is inherently more parallel. It has a different machine model.
All of the LLVM languages are "C-ish" in my mind, just like Plan 9 and even the web are Unix-ish. The addition of types and metaprogramming is absolutely a big deal, but they are additions rather than fundamental changes, in my mind.
It's easy to tear down what exists -- it's harder to say what the alternative is. Whenever I read these rants, I welcome the author to build an alternative and show us :-) Or even just share a link to something relevant
The structure of the comment was: LLMs show a disturbing fact that some people in our field have religious levels of faith in things regardless of evidence. We see something that seems similar with Unix where people assume that it is inevitable, but that's not the same, it's just ignorance.
The bullet points they provide about Unix aren't what you're talking about at all. They're not talking about narrow waists and text as a universal exchange. Consider files in a hierarchical filesystem (and I'd add: a distinction between file and data in memory). That is certainly not fundamental. The AS/400 is the clear, commercially successful counterexample for the points they brought up.
OK, I guess I'll just repeat: show me what you're talking about
Maybe write a blog post :-) Or just drop a link
As it is, it reads like "I have secret, fundamental knowledge that other people don't have, but I am only going to write about it in the negative, not in the positive. Also, I am not necessarily going to do anything about it."
I still don't see any connection to LLMs. I guess the connection is "other people are wrong" ? Who are these people? I think there are different motivations at play, like talking your book, or simply liking things that help you get work done
Unix was just an example. People tend to believe all kinds of strange things in technology and treating such things as deity-ordained facts, but this is due to ignorance.
LLMs are so good at bullshitting humans into believing they are sapient, thinking machines; similarly the success of Unix has led people to believe all OSes are based on files. When I say believe, I do not mean in the assume sense, but in the fundamental, faith sense.
Note that the comment you just replied to is by @madhadron, who is different from the author of that first comment, @lproven. I think you meant your repeat of “show me what you’re talking about” to be directed at @lproven.
I took it to be so, but I do not know what I am supposed to do.
Personally I think the BCS article that we're discussing makes the case pretty well.
LLMs are not "AI" and they are not a pathway to "AI". They are a dead end: a clever linguistic trick that may have uses in translations and things, but nothing much more. The current hype is a pure unsupported bubble, it will collapse soon and we'll probably have another AI winter. Which one, I lose count; the 3rd?
You got way too defensive here, as if being mandated to defend... something. Maybe the UNIX paradigm?
Others people already told you but the comment was basically using analogies that people get entangled in certain technical stacks or get invested in tech in a certain way so much that they can't see past it and start evangelizing it... like the actual religious faith. That was all really.
I gave you the AS/400 as a counterexample. I'm not willing to write a blog post on it because 1) I am not an expert on the platform and 2) there is already a large amount out there about it.
The connection is a contrast: the relationship of many people with LLMs is different than what we often see in technology, such as Unix, because it's not just parochial ignorance, it's irrational faith.
What's good about AS/400 ?
Correct. Thank you.
It is very interesting to me to find that while I usually can't understand what @andyc says in his writing, he also can't understand me.
I don't know what to do about it, but it's fascinating.
Some serious alternatives that "didn't make it":
Again I claim that "Unix" [1] having won is fundamental, not accidental, because the design of the other ecosystems doesn't enable enough end-user functionality to be created. You have to write too much software.
[1] Windows being essentially a superset of Unix at this point; there isn't significant software that runs on Unix but not Windows
C is just a slightly higher level zero-cost abstraction over Von Neumann CPU and I would argue there isn't really any other practical/good alternative abstraction to come up with. In that sense all of: C, C++, Rust, Zig build/build on the same abstraction.
I think you’re missing a lot there. Flat memory is a pretty poor abstraction these days, when even a phone is a NUMA device. That’s a C abstraction that is not present in the hardware. There’s a lot of interesting recent research (and some commercial products from the 1960s and ‘70s) on object-based addressing. These ideas are hard to adopt in C though and will probably never see mainstream deployment as a result.
Similarly, hardware is message-passing all the way down. It uses this to build cache coherency protocols, which it then uses to build a shared-memory abstraction, because that’s what the C abstract machine demands. This costs a lot of power.
On a modern CPU, the amount of area dedicated to executing instructions is surprisingly low. The biggest power draw comes from the register rename engine, which (along with the instruction scheduler and all of the speculative execution machinery) exists solely to allow a massively parallel system to pretend to be a sequential one. This is required to provide a C-like abstract machine.
So, yes, C is a thin abstraction layer over a PDP-11, which is a Von Neumann CPU that made a few choices about memory, but modern CPUs look almost nothing like Von Neumann machines. They can emulate one, but they burn vast amounts of power doing so.
What programming model would map more naturally to a modern cpu not emulating a pdp 11?
Good question. Something with no shared mutable state would be easier to scale. The caches for shared immutable data and exclusive mutable data are easy to build, it’s the shared and mutable that’s difficult. Similarly, something that has high degree of parallelism is easy. If you don’t do speculative execution, you need a lot of runnable threads to keep execution units full. Some of this might be easier to do with hardware support for creating thread-like things. For example, map-like functionality might be easy to implement, where you could do something not quite SIMT, which would help with instruction-cache usage.
Unfortunately, this hasn’t had nearly as much research as it deserves because you can’t run existing code fast on such a system and co-designing languages and hardware is really expensive. The early CHERI research cost around $10m, the total has cost $250m or so. And our changes he abstract machine were very small: we intentionally made something that was easy to map a C abstract machine onto.
Doing a proper co-design project to build a good language for high-performance hardware would probably cost around a billion.
We’ve started to explore some of these ideas in Verona. The really hard thing is to build something that can run well on current hardware but an order of magnitude faster on hypothetical more efficient future hardware, to then enable the development of that hardware.
I am not a hardware guy but I know I would work hard and passionately about making OS-es and software for hardware systems like this! I always felt shared + mutable was the easy way out and then everybody started preaching because "X million people can't be wrong".
(late reply) The big data frameworks derived from MapReduce and Spark are all "functional programming in the large" -- there are at least 10 or 20 of them, and many research systems
A cluster of PDP-11's connected by networks doesn't look like a PDP-11 - it's a different machine model (in particular, it's not a state machine)
A multi-core PDP-11 doesn't look like a PDP-11 - it's a different machine model
The functional properties like idempotence, commutativity, and associativity enable fault tolerance and parallelism. It has worked in practice
You can also look at research like Sequoia: Programming the Memory Hierarchy
https://ieeexplore.ieee.org/document/4090178
(IIRC it is a functional language; I don't remember the use cases being that diverse)
Importantly, this observation doesn't make imperative programming obsolete. The low level is still imperative, but the high level is functional / declarative / graph-based.
That's basically the idea behind Hay - https://www.oilshell.org/release/latest/doc/hay.html
I don't find it relevant. C just does not abstract over memory organization, and that's OK. On 8bit computers they had bank switching, on x86 memory segmenation, all bigger modern CPUs have virtual memory etc. You could and can still use any of these languages to write software for machines like that. In embedded you're often responsible for page tables, TLB mgmt, coherence, flushing, etc. The CPU/asm/C-like-language only cares about memory being addressable.
Well, nowadays there's "Provenance" in Rust (and a bit of if in C), so there's some stuff going on there in between the levels of abstraction, but that's about it.
Smalltalk always seemed like the great thing so ahead of its time that it couldn't be recognized for how great it was, even though it probably wasn't as great as it was made out to be. Still, I feel like I missed out on something special.
Years ago I got to have lunch with an old Smalltalker who held up four fingers and said something along the lines of, "FOUR KEYWORDS. Smalltalk had FOUR keywords and you could even do things like mess with the stack if you wanted." Wikipedia claims six, but it's still fascinating how much power they crammed into that language. I keep forgetting to make time to try out Pharo.
My understanding of Smalltalk was the image was "live", but it always felt like Docker was a ghostly shadow of that language that we try to emulate.
Smalltalk was originally created as a bet that you can fully specify a useful language on a single piece of US letter paper.
According to Wikipedia, the six keywords are:
I’m not sure that any of these are actually keywords. True and False are class names. These classes are used as singletons and they implement methods like
ifTrue:for conditionals. Nil is also a class, which implements a fallback method that does nothing and returns self, so any message sent to Nil does nothing. You can seriously mess up a Smalltalk system by redefining some of these methods, but you can also do fun things, such as add instrumentation toifTrue:to trace program behaviour, or do this in a subclass and return a tracing True to see what things use the result of a Boolean.Both self and thisContext are local variable names, the first is an implicit argument the same is the name of the current context. They’re reserved identifiers, not keywords. They don’t introduce new semantics, they’re just the names that the language gives for a couple of local variables that aren’t explicitly named. As I recall, you can assign to both of them (I’ve no idea what happens if you assign to thisContext, probably explosions).
I think
supermight be something I’d count as a real keyword. It is kind-of an alias forself, but uses a different dispatch mechanism for message sending (start search at a specific class, rather than the dynamic type of the receiver).True, False and UndefinedObject are singleton classes; their instances true, false and nil are special cases in the VM and the bytecode for efficiency but otherwise they could be implemented as pretty vanilla global variables. Preventing them from being assigned to could be done outside the VM, within the SystemDictionary class for example, so they're not keywords in my opinion. The fact that they're treated as keywords is an implementation detail of the compiler.
On the other hand you can't assign to either self or thisContext, as those are not normal local variables. I would say that thisContext and super are definitely keywords, and self is too important to not be considered one.
Welcome! It’s been a while!
Huh, in Objective-C, it’s quite common to assign to self, I forgot you couldn’t do that in Smalltalk. It’s something you often do with factory instances, but Smalltalk typically doesn’t separate allocation and initialisation, so I guess it’s less relevant there. The most common Objective-C idiom that assigns to self is for +alloc to return a singleton placeholder that you then call some -init-family method on. Depending on the arguments, it will then do
self = [SomeConcreteSubclass alloc]and then the rest of the initialisation.I can see treating thisContext as a special case, but to me it’s just a predefined name for the local variable that contains a reference to the current activation record. I thought some of the coroutine / green thread things worked by assigning to thisContext (and capturing the old one), but possibly they use some other primitive methods.
If you include implementation details, most Smalltalk VMs have a bunch of primitive methods (I don’t remember if yours did?) that will always be direct-dispatched to real things. Some bits of the standard library (byte arrays and so on) can’t be implemented efficiently in pure Smalltalk, so you can either provide these as classes implemented elsewhere, or provide some core parts that can be wrapped in mostly Smalltalk classes. You could regard the primitive method names as keywords, but they’re not part of the language, just implementation details.
Hmm, now I want to implement Smalltalk again. We have a tiny JavaScript engine ported to CHERIoT that uses a fairly Smalltalk-like bytecode, I bet something Blue Book-like could fit in a similar amount of code.
Edit: I just checked the Blue Book and the primitive methods are specified there, though in the implementation part.
Thanks :)
Local variables are effectively "thisContext at: n" (with the stack above the local variables). Maybe I am tainted by actually implementing the VM but I see thisContext more as a VM "register" than as a local variables—even more so than self.
It's been a while and I don't remember exactly how you did coroutines in GNU Smalltalk. I think there was some kind of Continuation class that was a wrapper for the context and had a (non-Blue Book) call-cc primitive. [found: https://git.savannah.gnu.org/cgit/smalltalk.git/commit/?id=44c6e4445cbb04466 was the commit where the thisContext-based implementation of Continuation was turned into a primitive for speed. The primitive code can be written in Smalltalk but it wouldn't assign to thisContext; rather it would assign to the parent of thisContext, which does switch to a different context but without assignments. There were no high level coroutines; only generators as in Python but without a way to send a value back to the generator]
Green threads are in the blue book and they are different; they are cooperative for CPU bound code but also preemptive. Priority lists are in a class known to the VM and preemption could be triggered either by hand ("Processor yield") or by the VM signaling a semaphore (via the signal method on Semaphores of course, but also on a timeout expiring; GNU Smalltalk added SIGIO and single stepping). Each Process class has its own context which becomes thisContext when switching to that process.
I don't see a way to downvote it, so I'm leaving this comment to register my strong disagreement about the LLM comments above. Most of what the comment says about LLMs strikes me as wrong, overconfident, and poorly framed.
It is an intentional feature such that people don't content themselves with unconstructively attacking others and instead contribute to discussions meaningfully. Maybe you should try it by explaining what it is you disagree with. If you don't care to do that it is fine, but you should then take the advice of the site guidelines and just let the person you disagree with be wrong. There is no need to register your discontent, nobody is keeping track.
Was this the intention behind the feature? How do you know?
In any case, I will grant this sounds like a good intention. But the road to Hackers News is paved with good intentions.
Let's flip the argument on its head. If one suggests downvoting enables people to not engage meaningfully, the same must be said for upvoting, does it not? Clicking an arrow, whether it be up or down, seems about the same w.r.t. "meaningful discussion".
There are many other designs available out in the wild. I encourage people to be dissatisfied with the status quo here. Most forums are terrible. Lobsters is, luckily, just a little bit less terrible. But it hardly lives up to its potential. If you go over to LessWrong, you can get some ideas of what is possible, which include more nuanced voting. You can, for instance, independently answer:
This distinction is key. It allows the community to register sentiments such as "This is written clearly and in good faith, but I still disagree."
Because, I took part in the bikeshedding of this 10 years ago.
The following is over a decade of discussions on the subject, if you'd like to see what people were thinking and saying about this at different times:
Truly, there is nothing new under the sun.
Hrm, I mean the About page is right there …
I believe it's been discussed in some meta threads through the years.
Slashdot/Reddit style upvotes / downvotes systems have multiple issues that people have observed through the years. Specifically, a downvote is just one bit - it's impossible to convey whether it's because the downvotes believes the comment is incorrect, a troll, or just doesn't like the commenter.
This site is a off-shoot of HN, created to address the many issues the creator @jcs observed there. HN also has downvotes. So this is a flippant comment that I'm surprised a site member since 4 years would make.
If you want a LessWrong style multi-axis voting system, the source for this site is public, and a pull request to implement it would be welcomed.
Yes, you correctly identify many of the problems with one-bit downvote systems. My reply, in short, was saying: much of the same kind of problem applies to one-bit upvotes.
I understand if my joke wasn’t funny, but it wasn’t flippant. Designing good communities requires a lot more than intentions.
My comment was a little bit of a test balloon. I think it would take significant leadership and sustained persuasion to make it happen. I’ll think about some ways to give it a try.
I'm just surprised you've been a member for 4 years and are only now discovering the mechanics of how voting and flagging works.
This community has evolved in large part due to the feedback from its members. If you wish to change it, maybe a bit more humility is in order.
Thanks for the conversation. Why do you guess I'm not humble, if you do? Or is it more about my style?
Some people conflate / confuse vigor and advocacy with a lack of humility. I wonder if you agree with this: A person can push back directly and persistently while still being humble. Asking a lot of questions and having strong opinions doesn't imply a lack of humility.
Here are three things I know about myself (as do people who know me). First, I don't think {community building, communication norms, interaction design} are easy. Second, I know I'm not capable of making broad changes by myself. Third, I am open to feedback.
I think a possible undercurrent is differences in communication styles, culture, and experiences. I've lived in many places across North America, and to a lesser extent, spent time in different countries in the world. I've seen variations in directness, passive aggression, confidence, overconfidence, humility, false humility.*
Another possible factor is that some people conflate intellectual pushback with a lack of humility. I'll tell you this, If I dish it out, I better be able to take it. My goal is not to tear people down. My goal is to tear down weak reasoning and poor framings and replace them with stronger stuff. Some people can't handle this, but I intentionally try to err on the side of optimism.
In my experience the topics of {collective action, cultural design, communication norms, consensus building} are widely misunderstood in technical circles. I voice my point of view as clearly as I can, and I welcome disagreement and clarification. Sometimes people take this in unintended ways. Some people are inclined to give me advice. All good. Communication is hard.
* I think many people can benefit by thinking more about the topics raised in The Proper Use of Humility. Have you thought about the questions raised there?
I’ve replied at length in multiple granular comments. Did you look first?
I think truth matters. I hope some people are keeping track.
It would've been easier for me to notice that if you didn't make a half dozen separate comments in response to the same single comment, the first of which being you just vaguely complaining you disagreed.
[Comment removed by author]
Using granular discussion, point by point, works better for complex questions. Taking it slow and methodically is better for truth seeking. The criteria isn’t what one person says it easier. A better criteria is what results in high quality discussion from many people.
On the writing side, it is easier for one person to write one rambling message. It is harder to write compact, separable points which invite detailed and substantive discussion. Think about it? Try it? Let me know what you think.
I accept this criticism as fair. I'll go back and hyperlink each of my specific criticisms to that comment, if I can.
There's no downvoting on lobsters, you can only "flag" something as basically breaking the rules or upvote it.
Lest people think I did not also engage substantively, here are four of my specific criticisms:
I welcome specific and focused comments to any/all. I split the comments into granular chunks intentionally. I explained my reasoning in these two comments: 1 2.
I registered general disagreement because I found the comment to be egregiously misguided. When I see otherwise informed and capable people led astray, I say something, because I care. I've been civil about it. The comment was well-written in the sense that it persuaded a lot of people, as measured by upvotes. (I couldn't see the net upvotes-versus-downvotes because of the information architecture here.) It used a lot of rhetoric; it sounded confident. And it was quite misleading (at best) or wrong (at worse) about LLMs.
Why do I care? AI -- actually AI Safety in particular -- is a big deal to me, and I'm trying to do what I can to root out misunderstandings. I assure you I take this topic and its implications very seriously, possibly as seriously as anyone here.
Everything we do here is folklore.
First, using the words intelligent and think without definitions is asking for misunderstandings.
Second, the quote implies some sort of dividing line between "intelligent" and "not intelligent". That's misguided.
Third, like many, I find value in the definition of intelligence as the ability of an agent to accomplish some task. This makes it situated and contextual. Defining it this way helps clarify discussion.
I like your definition quite a bit because I think it captures a lot of nuance in a practical way. That said, I have a pair of potential counter-examples to consider from my day-to-day work that I’m struggling to classify as intelligent or not using that definition. For context, I work with autonomous aircraft and in a previous life worked in a biology lab that was studying insect flight characteristics.
The first example is the flight control software itself: given a series of waypoints, fly the aircraft along that trajectory as best as you can. I’m going to reveal a dirty little secret here: most of it is just nested PID loops. For quadrotors, they’re typically:
This accomplishes a task and to a human observer actually looks quite intelligent. If there’s a gust of wind that knocks you off-course then the position PID loop will have an increased error term that ripples down through the stack to turn into motor commands that get it back on course. Is this intelligence? It accomplishes a task even in the face of adverse inputs. But under the hood it’s basically just a bunch of integrators trying to push the error terms to 0.
The second example is closely related: Kalman filters for processing noisy sensor inputs. Over time these are building up a statistical model of how a sensor is behaving and then making decisions on every sensor reading on how much that reading aught to be trusted. It’s an iterative model; each sensor reading gets assessed for trustworthiness and is also used for updating the internal statistical model of how the sensor is performing. It’s pretty much just a rote linear algebra equation each iteration though. Is that intelligent? It almost looks magical when it works well!
The last part… why I mentioned the insect research back in undergrad. One of the experiments I helped with was building a flight simulator for locusts (bizarre, right?). Forward velocity and turning through the simulator was accomplished by inserting electrodes into the insect’s wing muscles and measuring the timing of the muscle activations. If they activated simultaneously the insect should fly straight, if the right wing fired before the left then it would induce a turn. Once we’d established that the simulation was accurate enough that the insect could fly around in our crude world without colliding with objects (what?!?), the biologists hooked up additional electrodes to a single specific neuron called the Descending Contralateral Motion Detector, which was connected pretty much directly from the insect’s eyes to the muscle control system (I’m not a biologist…). What we observed was a series of electrical pulses from this neuron that were directly correlated with the wing EMG asymmetry: if the DCMD was reporting that a collision was imminent, the muscles would fire to turn away from the obstacle.
Is that intelligent? It enables amazing collision-free swarming behaviour. But in some ways it’s even simpler than the drone FCS… and I’m not sure I’m comfortable calling PID loops intelligent?
Fun examples!
Yes, for certain environments, the right sensors and actuators in a PID loop will get the job done reliably. If that setup serves an agent well, I would call that intelligent behavior.
The word intelligence is a fascinating mess. For example, many people look at a world-class chess player with awe and suggest they are outliers in intelligence. By the definition above, if they win at chess given their environmental constraints, they are intelligent at chess. But from another point of view, much of what chess players do is rooted in pattern matching and tree search. When you frame it this way, it doesn't seem that "intelligent" does it?
It seems to me you are talking about more general forms of intelligence. The degree to which an agent's intelligence transfers to other contexts reflects its generality or adaptivity.
"in any way"? This is hyperbole, right? I cannot take it seriously.
LLMs can indeed output language that is consistent with logical reasoning. The trend is upward.
What claim are you making, then?
I figure this is the claim - LLMs can output language that is consistent with logical reasoning. LLMs can simulate reasoning better than any prior reasoning-simulation software. However, they do not actually perform reasoning; their reasoning is performative, because the output is algorithmic (and algorithmic in a way that human reasoning is not).
If a system outputs text that consistently matches logical reasoning to a high standard, isn’t the simplest explanation that it is indeed reasoning?
(I’m not asking questions of consciousness or personhood or any of that.)
Perhaps you would say these systems are not using maximally simple, purely logical, circuits / structures / mechanisms that only do logical reasoning, such as forward-chaining or RETE or whatever?
If so, how are humans different? At best, humans use their handwritten symbols to make logical proofs according to set rules. But we still use complex and often flawed lower-level mechanisms (brains) to do this logic. In this sense, I think your claims about performative reasoning are moot.
no, because the Chinese Room isn't reasoning either.
LLMs are collections of things that humans have said before, boiled down into statistics. The sole job of an LLM is to output words based on statistical locational frequency, that is not, and cannot be, reasoning. It's incredibly provable that LLMs cannot actually reason, too, and there are multiple papers on it — see https://arxiv.org/abs/2406.02061 and also https://link.springer.com/article/10.1007/s10676-024-09775-5
I think the comment above misrepresents or misunderstands the Chinese Room argument. Per Wikipedia:
Searle’s point about the Chinese Room is not about if a machine can turn the crank and do modus ponens or other reasoning rules. (This has been settled for a long time.)
The C.R. highlights the question of qualia, of inner experience. I’m not talking about consciousness, “thinking” or a “mind”. I’m talking about something measurable: can an AI generate correct conclusions according to the rules of logic? (For example, they do LSAT level reasoning quite well, and they are getting better.)
Ah. Your claim boils down to the fact that the low level mechanism of token generation is not identical to reasoning. Am I representing your point of view fairly?
There was also a time when I fixated on this definition. I get it.
I hope you realize that when some people say LLMs “can reason”, they aren’t talking about the mechanics of token generation process. They are talking about higher level behavior.
I think it is also important to realize that by your definition, one could argue that humans don’t reason either. We’re just neural impulses firing. I hope you take my point.
Nothing can analyze and improve itself? What? Am I misunderstanding something here?
LLM bots cannot analyse and improve themselves, because they cannot analyse -- anything ever -- and they cannot improve on what was in their training corpus.
All claims to the contrary are snake oil.
I can see why you’d think that LLM bots cannot analyse and improve themselves. What really confuses me is that you say “nothing can”.
Are you saying that humans cannot analyse and improve themselves? Because I think plenty of humans do that after, for example, failing at something many times in a row, or reading a self-help book.
Or by “nothing”, do you mean only things that aren’t alive? If you do, consider the hypothetical case that someone wrote a (non-LLM-based) program that perfectly simulated a human’s behavior (human-like words, leg movements, etc.). The program was even observed to change its outputs after interacting with a simulation of a self-help book. If you agreed that humans can analyse and improve themselves and you had this definition of “nothing”, I think you would have to be making one of these three assertions:
LLMs are, by definition, vast databases of multi-dimensional weights of relationships between symbols.
An LLM can't analyse an LLM, including itself, and nothing else can analyse an LLM either.
Claims of redesigning LLM bots to hallucinate less, for instance, are lies, because the databases are not amenable to study or analysis. And, of course, hallucination is what they do. There is no distinction between making good and desirable stuff up and making bad stuff up.
They are lies, just as software companies claiming they have rewritten huge codebases for security. In most cases, they've just done automated scans, maybe a few people looked at a few especially critical routines.
Real rewrites are very rare and usually disastrous, as Joel Spolsky observed about 15 or 20 years ago.
My comment was about your claim that “nothing can” “analyse and improve themselves”, a claim I disagree with. What you say about LLM analysis may be true, but that’s irrelevant, because I was not arguing that LLMs are an example of a thing that can analyse and improve themselves.
With my comment, I was hoping to pinpoint where, if anywhere, we disagree about the statement “nothing can analyse and improve themselves”. You can do that by answering these questions:
I just thought of a possible explanation for your response being weirdly focused on LLMs instead of on the questions I had posed. Looking at your original wording again:
Did you intend that last sentence to be interpreted like this?
If so, then that’s very different from the interpretation that @insanitybit, @rele, and I had. insanitybit’s comment and rele’s comment both rephrased your statement as follows:
If that statement does not capture your meaning, then it is unfortunate that your replies to both insanitybit and rele’s comments, which contained that rephrasing, did not correct the misunderstanding.
Correct.
Regarding exaggerations and lies about what LLMs are capable of in the near-term... Yes. There is hucksterism in AI. I know the arguments; I've read Kapoor and Narayanan, which I generally recommend. I've also read Gary Marcus, who makes some good points but gets too ranty for my tastes.
However, regarding the comment here above, there are claims here about what is and is not possible, such as:
If we define analysis as "detailed examination of the elements or structure of something", then, yes, an LLM can analyze its own weights and structure.
If you say "but LLM X can't analyze itself", that is a different claim -- a claim about implementation not about what is possible.
If you say "but it can't analyze itself up to some quality standard", that is a different claim, and it invites discussion about the bar and what progress is being made towards it. This kind of discussion might be interesting; it invites us here to dig into the details.
This is a tall claim. To me, it is so bold that I wonder if it is designed to attract attention. I'll bite: I ask the author to (a) clarify their terms and either (b) prove this claim or (c) make a testable prediction, on the record.
[Comment removed by author]
An LLM cannot analyze? Are we having just misunderstanding each other -- i.e. having semantic confusion?
Maybe these questions will help:
What do you mean by "what they do"?
What papers on LLM scaling laws have you read? I suggest we ground our conversation there. The papers show improvement across many metrics. Bigger models perform better in almost all ways*.
Do you want to refine your claim?
* Bigger models can be more deceptive, which is unfortunate if you care about alignment (which you should). But increasing deceptive ability is most certainly an improvement in capability.
I am not interested in radical new methods for estimating with unprecedented accuracy the numbers of angels dancing on entire fabric-fastener manufacturing units. I don't care, because there are not such things as angels.
I find it highly amusing that saying "this type of software is not intelligent" provokes responses which attempt to nail down a possibly negotiable definition of intelligence. Again: this is not a rebuttal, in my view.
You try to split hairs, perhaps so that you can cut each one, but I'm not here for a haircut. I am not here to persuade anyone. I don't see any potential gain in even trying.
LLMs are a fascinating technology which may in time enable applications such as simultaneous translators, perhaps not only between human languages but between computer languages as well, which could have amazing consequences. I speculated on some of those here but I remain extremely sceptical about the entire current industry, as I said in the earlier article that the above is a follow-on to.
But they are not artificial by any reasonably strict definition, they are not intelligent by any definition that isn't a fantasy, and because they are not in any way what they are depicted as being, they will never lead to anything that is.
It is empty hype, nothing more, and I sincerely hope it collapses soon and that it destroys the careers of the hucksters selling these lies about thinking computers.
Here’s what I notice. First, you are obviously bothered by hucksterism in AI. (So am I, by the way.)
Second, this is a bit of a guess, but you seem to take my pushback personally. And then you channel a lot of negativity towards me. I’m not the problem. The key problem is these topics are complicated, communication is hard, and we all have egos. Very few people want to be wrong, much less shown to be wrong.
Third, you write at length, but haven’t answered the specific questions I ask. You sometimes wave them off. Sometimes you ignore them altogether. You also use a lot of rhetorical devices.
On the substance: Are you claiming LLMs are not getting better as they get bigger? If so, by what metric? (I ask for it to be clearly defined and measurable.)
Please respond to what I’m writing, not about what some other person says.
Again with the demands for metrics. No, I cannot give you metrics, because as I have said already, I do not care. You are claiming the numbers of ratio of dancing angels to pins is important: I'm saying there are no angels. No angels means no counting means no numbers.
Yes, they have, so far, provider better results -- more plausible-looking textual or image based output -- with greater size; however, it is not just foolish but downright stupid to blindly assume this will continue indefinitely forever.
Secondly, the bigger the model, the bigger the power and storage use of this technology which is already environmentally catastrophic. The boosters who keep calling for bigger models are guilty of ecologically criminal acts. You are entirely as culpable as the cryptocurrency promoters.
I have not said anything about such improvements continuing forever (neither in this thread or sibling threads, nor anywhere I can recall.) I strive to not blindly assume anything.
Please stop attacking me. Such attacks are not welcome here.
And, to be clear, I have not called for bigger models. (And if I had, would such attacks be justified and appropriate here? It is hard to listen and learn when you are attacking and accusing.)
You may perceive them as demands, but this does not make them demands. I have asked questions. Repeating a question doesn't make it a demand. We have both attempted to redirect the conversation, such is the nature of conversation. You call my attempted redirection a "demand". To my eye, this is probably either (a) the result of you being upset for other reasons and/or (b) a rhetorical device to make me seem unreasonable.
This is an uncharitable description of what I'm asking.
When I said "rhetoric" before, this is the kind of thing to which I was referring. It raises the temperature but doesn't promote mutual understanding.
Overall, this conversation has been unnecessarily fraught and combative. I have strived to disagree with you without being disagreeable. I'm sorry if you think my point-by-point commentary is mean or in bad faith. I do not intend any of these things, and I'm sorry if I've offended you somehow.
It would seem you have labeled me in some negative way, such as being an AI booster or ecologically irresponsible. I don't see the logic in why you think these labels apply to me. And again, even if they did, being unkind about it isn't necessary or welcome.
Whatever you feel, I would ask you to stop taking it out on me. It seems you are making assumptions about my point of view or my impact on the world. I have no doubt that your point of view is valuable, but I would ask that you be more civil about it.
you do realise that the effective altruism cult mannerisms are incredibly tedious to anyone not involved in the cult's way of framing reality?
I am not sure what you mean. Based on your comment, you sound angry about certain mannerisms and/or beliefs? Such as?
If you want to offer some specifics and are willing to discuss charitably rather than dismissively, I’ll put in some additional effort. No need to be mean about it, please.
I think it’s fair to say the context here is about artificial intelligence and some common misunderstandings. Unless I’ve lost track of the parent comment, I think this all started when one person made a list of common UNIX misconceptions and then offered various AI misconceptions. My comments, taken together, point out how many of these claimed AI misconceptions are actually misconceptions themselves.
Many people were far from calm. The conversation has the usual signs of people overreacting to each other. Even relatively neutral people got attacked, because one side or the other accused them of being on the wrong side. It has been a case study of polarization and poor community discussion norms. Truly pathetic, as in I have pity for how we all got here.
These discussions need to get clear and as specific as possible if we want to make sense of the world. I hope you can appreciate this.
Are clarity and precision and rigor tedious to you? If true, this discussion might not be worth your time? Truth seeking isn’t “sexy”, nor does it provide the dopamine hit of leaving a nasty comment. It doesn’t make it easy to form a tribe of “us versus them” because anything is subject to scrutiny and revision. These are the values of critical thinking and intellectual honesty.
I flagged this as unkind, it's probably trolling too? I don't know. I figured I'd let you know. Calling "effective altruism" a cult is definitely hyperbolic and clearly intended to be insulting.
I know that some online communities dislike some members or something like that but it's not a cult, it's just an ethical framework, and the user doesn't even seem to be advocating for it. I don't know what mannerisms you're even referring to, personally, but this seems like a silly and overly negative interpretation of a post that's basically "please stop insulting me and engage with my points, I am willing to engage with yours".
your consistent appetite for tone policing was noted, thanks.
Nothing to do with tone, I just think your comment was bad. As I explained.
if you insist, we can continue.
your explanation has been given a consideration it deserves, and found wanting. specifically, the following statements are wrong: “calling ‘effective altruism’ a cult is definitely hyperbolic” (it isn't); “it's not a cult” (effective altruism movement, especially its californian variety, has many characteristics of a cult), “it's just an ethical framework” (no, it's not just an ethical framework).
if you don't know “what mannerisms i'm even referring to”, you have no basis to claim that “it seems like a silly and overly negative interpretation of a post” (and my reply wasn't referring to a single post).
to spare your precious time, i will have tagged my reply as unkind myself.
Yes.
Look at adversarial inputs for image recognition. I can show you two pictures, which you can't tell apart. One the AI says "giraffe: 99%". The other has undergone adversarial input modifications. The AI say "dog: 99%".
Whatever you want to believe about AI, that example alone shows that it's not doing anything like what you or I are doing. There's definitely some statistical magic behind it, but there's no possible way that it's "AI".
Add to that the fact that "AI" needs 1000's to 10's of 1000's of pictures to "learn" what a horse is. But the average 2 year-old can learn the same fact in 1-2 viewings.
The current crop of AI is doing super-optimization of techniques which are appealing, but which are nothing like what people actually do. Until such time as the researchers take a step back, we're just optimizing ourselves into a deep, dark, and unproductive corner.
I don't think that LLMs are doing what our brains do but I don't think your example demonstrates that at all. An obvious defeator here is that humans are entirely subject to adversarial illusions.
Let me re-phrase the final bit of that argument: "humans are entirely subject to different adversarial illusions."
The adversarial illusions are a window into the underlying model. See the common illusion of a rotating, but inverse face mask. The inverted face rotates left to right. Your brain interprets it as a normal face, rotating right to left.
Why?
Because for the past 10 million years, every face seen by every brain has been normal. When your brain sees something that looks vaguely like a face, it interprets that thing as a normal face. The "illusion" here is a breakdown of the model. The output of the model no longer matches reality.
For AI, I can take a picture of a giraffe, change 1000 random pixels, and then convince the AI that it's a dog. However, your brain isn't subject to that adversarial illusion. The only possible conclusion here is that the models used by AI and by your brain are completely different.
I'm not sure what's different here. Why is changing 1000 pixels (not random fwiw) different from showing me something that "looks" like it's rotating, or 3d, or multiple colors, etc etc etc. I don't see the difference honestly.
TBH I think it's moot because I don't think anyone argues that LLMs and human brains are the same, only that they may share some features. If you're arguing that they share no features, I don't think this argument based on illusions works well for that.
I don't think adversarial modification is good to generalize intelligence on though I agree it's an good story to illustrate that "AI" is statistical games at this point.
To make a parallel, say I bring a blind person some horse hair, and they say "99% horse" based on the smell. Then I give them horse hair that's been sprayed with dog sent and they say "99% dog." It doesn't mean they're not intelligent.
Or otherwise, encrypted data should not be differentiable from random data. Thus encryption could be seen as an unbeatable adversarial modification.
I think that if you instead have the recognition system report features (extremely long neck, four legs, brownish), LLM will readily reason that it's likely a giraffe.
Having poor architecture doesn't mean that the components are unusable.
That recognition system isn't used by the bulk of the AI tools, so far as I can see. Instead, the method used is statistical magic on RGB pixels.
As best I can tell, the human brain does essentially what you say here. It forms a model of "cat" based on recognizing the 3d look & feel of the cat. And even that is based on modifications of previously learned animals. When you see a new cat, your brain uses those pre-existing models to find the "best fit" to a particular animal.
The current crop of AI tools might eventually get to that point. But the method of "analyze 10,000 pictures to discover what's a cat" isn't doing what we do.
Are you also saying that "what people actually do" is "better"? In what sense? Can you give an example of what you mean?
Do you know the stock arguments for/against silicon relative to carbon computation?
It's blatantly obvious that what people do is "better" for many, many, things. If AI takes 10,000 pictures to learn "cat", and people take 1, that's "better". If AI is fooled by changing 1000 random pixels in a picture and you're not, that's "better". If AI reads 10,000 books and then hallucinates answers to questions, humans are "better".
As for the rest of your comments, you're asking a bunch of fairly hostile questions which demand that I do a bunch of things. You're rather missing the point.
My point is that AI proponents make all kinds of claims about AI. Yet anyone honestly looking at AI can find trivial, obvious, counter-examples which disprove those claims.
I'm not saying that the current crop of AI isn't useful. It's tremendous for synthesizing statistical knowledge out of reams of data. It's amazingly good at looking like it understands language, using only statistics. But there is no possible way it's "thinking", or that it is "reasoning". It's statistical magic, and is doing nothing like what people are doing.
There is simply no question that people are better than AI at a huge list of tasks. Until such time as AI starts to use models similar to those used by our brains, AI won't be nearly as good at them as we are.
Ah. There was a misunderstanding. I wasn't talking about relative performance. (Sure, humans are better at some things, AIs others.)
So, I'll write it a different way, to see if this helps:
Are you saying that "how humans think" is "better"?
In context, I was responding to:
This is a big claim about what research directions are more and less productive. It implies that AIs should be more like humans, does it not? This is why I asked: "Do you know the stock arguments for/against silicon relative to carbon computation?" These are an essential part of diving into the discussion of human-vs-machine intelligence.
Here is something to consider: one can value and promote human thriving without locking-in current modes of human thought. Modern statistical reasoning only became common among professionals around the 1950s. My point: we should be open to better reasoning, even if it is "unnatural" or "non-existent" among a given human population.
I notice many people claim hostility. In my case, I think you are confusing hostility with requests for clarification. Instead of making assumptions about your claims, I’m asking. Generally speaking, claims around AI are fraught and getting more polarized. Slowing down and explaining helps.
Edit (hours later): in retrospect, this was an obvious miscommunication -- see sibling comment. It is notable that you gravitated towards the word hostility. Many people do this, myself included, and we can do better by remembering "there can be many explanations for what I'm seeing; I don't have to assume malice."
The "requests for clarification" came across as hostile. The questions weren't "What do you mean by that?" or "Can you explain more?" But instead you asked a bunch of leading questions, which insinuated that I know nothing. That comes across as hostile.
I thought my comments were based on pretty trivial observations. An AI can be fooled by adversarial inputs. Yet you can't tell the difference between the two pictures which fooled the AI. There is no possible way that the AI is doing the same thing that people do. This is a trivial conclusion to make.
When you look at a picture of a giraffe and say "giraffe", but an AI looks at the same picture and says "dog", it is also trivially obvious that a human is better at that task than an AI is.
Many of the rest of the arguments for AI are just "the emperor has no clothes". I don't need to read thousands of papers on how beautiful his garments are. I can see his dangly bits. He's naked.
I'm amazed at how good AI is, considering it's just statistical magic. But there is no possible way in gods green earth that it's doing anything like what people do. And there is no possible way that it's doing any kind of reasoning.
I think a significant number of people have a tendency to think “I see hostility” before they think “Let me think about a charitable interpretation”. I see this in your comments. Fair?
I asked some pointed questions. Call them leading if you want. That doesn’t make them inappropriate. I think if you step back, and it wasn’t you involved, you would agree.
I also think you know I’m not asking you to read 1000 papers. Not 100. Not 10. I am asking if you know what you zero, one, or few shot learning is.
I know that people don’t want to admit they don’t know something. But there are more important things, like learning and being thankful if someone corrects your understanding. Why? Once your ego recovers, more knowledge makes you more powerful.
It is fascinating that some people bristle at the idea of learning. Dawkins forbid we encourage each other to deepen our knowledge! What are the chances that in a pairwise interaction that one person has deeper knowledge in some area? Close to 100% probably in most cases.
I’m trying to share some things I know with you, in case you don’t know them. You have a choice here. Do you reach for the “you are being condescending” line of thinking? Or maybe instead grant that, yes, XYZ was a reasonable next question.
Overall, I value promoting civil, frank, truth-seeking and open discussion while assuming charitable intent. I’m not perfect. I know from experience my standards are higher than average. I’ve been accused of being hostile, not humble, etc., etc. I can handle it, because I know who I am and what I stand for. I sometimes spend time with people whose curiosity peaked long ago. It is sad.
Like many others who study AI Safety, I estimate there is perhaps 10% to 30% chance over the next 40 years that artificial intelligence technology will decimate human civilization or worse. With this in mind, I will be civil, but I’m going to reserve the right to push back against incorrect or overconfident or too-narrow claims. When I think there is a better framing, I’m going to say it.
I don’t think any of us have the luxury to be ignorant or wrong about AI. I don’t think any of us have the time or luxury to not push ourselves to be better thinkers.
My views don’t reflect my employer or investments I hold. They reflect a stubborn and persistent drive towards better understanding the world despite humanity’s tendency to get in the way of long term thriving.
P.S. Someone accused me of talking like an effective altruist, as if that is some horrible thing to be. Many people conflate one bad apple (SBF) with a philosophy that varies in its application.
Many people, whether intentionally or accidentally, take it as a given that the goal of AI is to be human-like, and as such mimicing humans more accurately is "better" by definition.
One example of this is FSD, where the goal is explicitly for autonomously driven cars to coexist with human driven cars and replace human drivers without changing the surrounding expectations much, and as such it's desirable for them to understand human signals, for them to know a wave of the hand like so in one country means "I'm giving you right of way, please turn", but in another country the same wave means "wait a second", and if a police officer does it means "stop right now."
It also feels more comforting if we can fool ourselves into thinking we understand how an LLM reasons, and the closer it's output is to what a human (or super-human I guess) would output, the easier it is for us to think we understand its capabilities and limits, whereas the above example of an adversarial modified image throws this into sharp relief.. so it makes people feel uncomfortable, which also is a fair enough argument on its own, since "uncomfortable" is bad I guess
I also want to distinguish between (a) how humans "think" and (b) what we value. I'll make three claims about these:
Like I mentioned in a recent comment, I would argue that humans should be open to better ways of thinking, by which I mean analysis, planning, and so on.
Better ways of "thinking" can help us get more of what we value.
As argued by Willam MacAskill, I think we should be vary of value lock-in. One of the markers of societal progress is the changing and improvement of our values.
Now, how these points interact with AI Safety is complex, so I won't say much here other than: beware powerful optimizers. People such as Stuart Russell and Robert Miles, explain this quite well.
My comment above got labeled as "spam" which doesn't match what is written at https://lobste.rs/about :
Have you read some papers and/or implemented {zero-, one-, few-}shot learning?
I recommend these techniques because it is not true to claim that all AI/ML techniques require 10's to 1000's of pictures to learn new categories. ML research has made extensive progress on this. In practice, many systems lag behind, but it is often a failing of human choices of where to spend their efforts. Generally speaking, it is not a failure of what is possible or even feasible!
People often lack key information and skills that would otherwise allow them to reach for better alternatives. Often the "first" failure I see in people is a lack of curiosity and imagination, followed by overconfidence. In particular, a large failure mode I see is humans overgeneralizing about what AI/ML systems can and cannot do. This is why I respond (asking questions, clarifying, pushing back) when I see ignorance and/or overgeneralization.
On what basis is my comment above flagged as spam? It directly addresses the point. I’m more than happy to hear criticism and error correct. Along with others, I am seeing a lot of incorrect flagging, possibly vindictive. Who can check on this and/or design countermeasures?
What do you mean by "step back"? Do you have an alternative research direction in mind? What is it?
Care to convert your opinion into a testable prediction? How confident are you?
Could you please use a single reply instead of multiple replies? It makes everything very confusing to read. Just split your post into multiple quote reply sections if you must.
Replying in a granular fashion, point-by-point, allows comments to be more targeted. As a result, it allows conversation to continue to be productive even at the deeper levels. It might be unconventional, sure. Have you considered the current convention might not very good?
About me: I've spent thousands of hours studying (and experimenting with) various approaches for dialogue, deliberation, collaboration, and interaction. This is one of my top interest areas, and I care about it deeply. The current state of online discussion and the degree to which people fall into certain suboptimal patterns is a drag upon our collaboration potential.
One important aspect of collaboration is to tailor your response to the audience. You're trying to apply conventions from one forum (I'm guessing LessWrong, based on your previous comments) to this one, where they don't apply.
Yes, tailoring responses to the audience is often offered as advice. Sometimes this is all that is said, leaving a key follow-up question unspoken: what is actually good for the audience? To different degrees, we have both been outspoken as to what we think it is good for "the audience".*
To what degree is the following true, in your mind?... You know better than me, because you've participated more here, and for a longer duration? I'm newer and bring unusual points of view. To put it bluntly, I'm the other, the outsider.
If you were in my shoes, might you see part of this as a push for a certain kind of conformity? I'm more comfortable than many with non-conformity, because I aim for different goals. Perhaps you can appreciate this.
How do you know they don't apply? I'm pretty sure I disagree with that claim.
My point: I claim that subdividing points into separate threads serves people better. I am also claiming that "what people want" is probably not the right metric. I am suggesting that a better target is "what is good for people"; in particular, it is more important to aim for good practices and habits in service of truth-seeking. Not all of these practices are currently appreciated; some feel foreign. That does not make them wrong, of course. Blood-letting was once a popular medical "strategy". And so on.
I see a good chance you may be past the point of neutral conversation here. Are you? Can you see why I would say this?
* Is it fair to say that I have accentuated what I think is good for the audience if we were to stop and think about it? Is it fair to say that you've accentuated what you think is good for the audience based on their current tendencies and habits? Is there a synthesis here where both of our perspectives get factored in?
“Lobsters is more of a garden party than a debate club.”
You’re very much treating it as the latter, and people enjoying the former are reacting predictably!
Hi! I don't often post comments, but I do read just about every comment posted on Lobste.rs as I enjoy learning different points of view. I wanted to give some feedback that, as a neutral party in this discussion, I also found your use of multiple different replies to the same comment confusing and hard to follow.
The reason is we go from a threaded conversation where both sides are replying to each other, to a multi-threaded conversation where you have to keep track of which part of the tree you are in. i.e. we go from this:
To this:
I don't disagree with the above statement in general, I do disagree with the implementation. Similar to programming something that is multi-threaded, things get a lot more complicated quickly when you split this thread up. I suspect if you take the different comments you made, put them into the same comment box and separated them with lines or just different quotes, it would be a lot easier for people to
Just my two cents, have a great day!
Thanks for explaining your take and being kind about it.
It adds a lot of additional empty space due to forum layouts, makes the tree more chaotic, makes it more difficult to respond (because people may want to respond to points that were spread across comments), etc. I do think it would be helpful for you to stick to one reply.
Did... anyone even read the article before commenting? This article doesn't say anything about AI's capability, i.e. it isn't saying that current AIs will not keep improving in terms of what they are able to do. Yet it seems like that's what all the comments here are talking about. (Of course, I could be misunderstanding the points people are trying to make.)
This article is pointing out current AIs' lack of interpretability - their lack of human understandable internal structures. Therefore they can't be verified and trusted, and is a dead end in that sense.
And the article itself doesn't even have insight into evolution of LLMs, but is instead arguing this should have been done differently, with nothing but "hope" to back up such approach.
They are only a dead end from the perspective of those seeking AGI. To everybody else, the question might better be phrased as "Does current AI represent useful tools." I would argue from that perspective, large language models are not at that end. They are in fact incredibly useful for maybe not replacing people, but definitely for augmenting what people can do in a 24-hour period.
The article isn’t talking about AGI, it’s about reliability. The point, as I read it, is that these systems are black boxes that can’t be decomposed, that are not predictable or reliable, whose behavior we can’t understand, and that have no accountability. I tend to agree.
The main mentions of accountability and responsibility I see in the article are:
If you don't know the weights, sure, neural networks are by definition black boxes. Many algorithms also look like black boxes if you can't see the relevant details.
But if you know the weights and the architecture of a NN, there are ways to interpret them. There is interesting work being done here. It is a very different kind of interpretation than understanding a program based on its source code.
I've learned a lot from the interpretability research from the AI safety community. I recommend it.
The best I gather is that the state of the art in investigating regions of connectiveness or looking at the second or penultimate layers and inward, etc, is that it is all still very noisy and not generalizable to anything except another can of worms of 98% accuracy and a chaos monkey of the remaining 2% cascading into further error, just like how they behave interactively.
I learnt some explanation methods about neural network before LLM appeared. But I think large LM it too large to be explained, there are too much parameters, which means informations about how LLM works.
Can you tighten up what you mean? Make it precise? Test it? Framing the questions well is essential.
I recommend reading about modern techniques.
In my opinion, reliability is the last huge challenge in the road from current LLM to AGI. If current LLM is reliable, we will know when we can trust it 100%, when we cannot trust LLM and why, how we can improve it, after that, we always can achieve AGI.
Human intelligence is far from reliable or trustworthy; why would you expect AGI to be better? Especially when we’re building it using architectures inspired by our own brain structure.
In my opinion, Human intelligence is reliable, as least for himself / herself. The content I output are based on what I knew, there is a reasonable process from what I knew to what I output. Maybe sometimes what I output are not right externally, but it it for me right now, This is how I define "reliability".
How much psychology have you read? There are any number of experiments showing that our minds are much, much less reliable than we think.
To the extent people are reliable it’s because we have checks and balances, rewards and punishment. In the small groups we're evolved for, if you lie a lot or cheat, the people around you will learn not to trust you and you’ll lose social connection, which translates to emotional pain. Humans have significant amounts of cerebral cortex to track this. In larger societies we can’t scale up to keep doing this, but the smaller subgroups like offices and hierarchies help.
But when you add intrinsically unreliable AI agents with no social connections, all this breaks down. RHEL helps, but post-training there’s no reinforcement; after ChatGPT makes up an answer or acts according to biases encoded into it we have no way to punish it. You can’t shun it or complain to its boss and get it fired or take it out for drinks and change its mind.
I only knew a little psychology, but I agree with your meaning about "our minds are much less reliable than we think".
Even when people are lying, there is still a reasonable thought process involved. This is something LLMs often lack.
I know what you mean, but there isn't absolute ground truth for reliability, sorry for the misunderstanding, I think reasonability is a better word to express my idea.
I agree. Humans can be objectively wrong, but they're very often consistently wrong. There's an "internal state" that seems to be lacking from LLM output.
I agree that framing AI from the utility POV is more tractable.
To your first sentence: Anyone who makes a claim that a technology is a “dead end” is making a prediction about the future. Very few of such people take such predictions seriously. Even fewer make testable predictions on the record. Even fewer will be held accountable if wrong.
People like to speculate. I tend to heavily discount claims that aren't (a) logically derivable or (b) clear and testable, where the author is accountable.
So, you claim that LLM won't augment a human neither in a 16-hour period, nor in 37-hour one, nor in 82-hour one? Only in 24-hour period
I think we're just getting away from the idea that algorithms are unequivocally good things after sites like Twitter, Meta (and friends) really screwed the pooch with them. At this point any large site with a closed algorithm is a public/democratic safety hazard and should be banned (is my opinion).
AI is effectively a black box algorithm on steroids. It's useful for a bunch of things but I agree with this assessment that either we use LLMs as a generative component of a more rigid framework (something which I feel is already happening) or we always keep a human in the loop.
I think it’s clear that absent massive advances in electric power generation (fusion?) LLMs are a dead end because they’re going to need to get so much bigger to do anything much more useful.
I think they’re not a dead end in the sense that they are already capable enough that as part of some kind of ensemble of computing approaches it’s plausible that something very new and cool could happen.
I don't think they need to get bigger and I think big leaps are likely to lower power consumption. For example, let's imagine that an LLM could "learn math" if it got big enough. Well, we could just not do that and instead have a smaller, more efficient LLM write "math programs" and offload their execution to a calculator. That'd be cheaper and more accurate.
I think you’re describing an instance of the ensemble approaches I mention in my second paragraph :)
Ah I was thinking you were referring to the "Mixture of Experts" approach.
Please define "dead end". Can you convert this into a testable prediction?
Otherwise, down the road, how will you know you were right or wrong?
From the article in the "Faults" section:
I really hope the author isn't setting the bar at 100% certainty of 100% correctness.
The sentence needs rephrasing. I suggest this: "It is important to get statistical bounds on correctness of the models we use."
There is a large body of work on various metrics for LLMs. They include various measures of quality, such as correctness, alignment, helpfulness, and more.
Mechanisms for fine-tuning LLMs are important and under active development. There is RLHF, Constitutional AI, and more. This work is fascinating on many levels, and there is more to do. It is hard to know the details of what various frontier labs are using for their training. It is reasonable that we would want more transparency about the quality and reliability of our models.
I didn't get much value from the article. To my eye, it didn't teach nor clarify.
If you want to learn more, I suggest finding resources on AI Safety. Here are two I recommend. First, Dan Hendrycks has a free online book. Second, BlueDot Impact has its AI Safety Fundamentals curriculum online.
You're not wrong that there is a ton of interesting research, but people have so many unconscious cues for correctness, and we can't reliably predict the correctness of the next token with any more total probability, reliably, than the models themselves can. Really interesting things are happening in cross discipline research in tandem with NLP expert guidance, but, for general use, there's an information hazard that is not well understood and is having unobservable knock on effects, and this has been the case for quite some time with no meaningful signs of improving.
I totally get the point of the author. But I think that with proper use LLMs can favor experimentation, which is necessary to foster innovation. It is very easy now to build a POC of a non trivial application in a few hours. And one can learn a lot in the process. What I see a lot these days is though that using LLMs is amplifying the Dunning-Krueger effect and making it more difficult to build software that is well designed and easy to maintain. In fact while providing code snippets, the LLM does not a good job in systemical thinking and the if you go beyond the POC with a LLM the chances of ending with spagetti code are high.
Yes.