From a consequentialist point of view, one person’s differential impact on the factors listed in the article is statistically minimal (energy, job replacement, etc). You can read this in many ways, some at odds with each other, such as: (1) don’t worry about LLM usage, focus on higher impact areas instead: use them to research and help your favorite causes for example; (2) the diffusion of responsibility can undermine many sensible notions of ethics; (3) “traditional” consequentialism is an insufficient ethical compass; lots more.
right, I mean, I am far more sympathetic to consequentialism than to deontology, but decisions that have aggregated effects are just something it’s bad at. my personal belief is that with questions like this, we have to ask ourselves what would happen if everybody followed the same reasoning we do and reached the same decision……… because that’s pretty much exactly what happens.
I am sympathetic to the ethical argument of acting as if your actions formed a guide for everyone to follow. This has merit as one criteria, but not as a unitary guiding principle for at least two reasons.
First, it is complex to figure out what conditions are getting baked in.
To explain — the ethics of lying can be illustrative. Lying is not categorically wrong according to most sensible moralities (or at least not the worst option in all cases). For example, lying to a visibly violent person to protect an innocent person is generally viewed as the better choice.
Second, there is another problem of the “what if everyone acted this way?” analysis: done too an extreme it can ignore probabilistic assessments and repeated-game interactions. For example, we don’t want everyone to go out and burn a lot of resources on a new startup idea, but it is beneficial to society that some tiny fraction of people do, largely because they are more risk averse and the upside is large.
Both the first and second points emphasize the contextual factors, which make the practice of discussing “general ethical behavior” very difficult. It raises questions of consequences, intent, and what is known.
I respect the way that careful AI researchers and statisticians are digging into ethical questions … they are making them rigorous and applied in ways that philosophy alone cannot.
As an example, how unethical is it to be ignorant and/or clueless? Ignorance can undermine even the best of intentions. With simulations, we can play out what it means and get a better understanding of our ethical intuitions.
But in any case, stepping outside ourselves and giving some emotional distance is usually a good starting point to thinking about morality. Our egos are so pervasive that they tend to taint even our perceptions!
No harm in continuing a good conversation. :) Having just reread your comment, I personally think a nice response to the question of “what if everyone behaved this way?” is to ask “what is the probability of that happening?”.
There are indeed dynamics where people are largely reasoning in the same way. These are relatively easier to reason about.
It takes a lot more “theory of mind” to develop meta-ethical systems that work well when agents have differing ethical systems. (This is why I highly recommend the philosopher Robert Kane; he has framework for thinking about such situations called the moral sphere.)
hi, i’m daniel. i’m a 15-year-old high school junior. in my free time, i hack billion dollar companies and build cool stuff.
Best intro ever.
Signal instantly dismissed my report, saying it wasn’t their responsibility and it was up to users to hide their identity: “Signal has never attempted to fully replicate the set of network-layer anonymity features that projects like Wireguard, Tor, and other open-source VPN software can provide”.
However, I did a quick test using two Signal accounts and it appears that images from unknown numbers are not auto-downloaded; you have to accept the message request first. This heavily mitigates the issue and honestly I understand Signal’s stance given the context.
Additionally if you’re concerned about this you can disable media auto-download, even from contacts, in Settings -> Data and storage (though that doesn’t help for progile pictures, if it turns out the caching service used for those is also vulnerable).
No matter what the weakest point of Signal is its cloud infrastructure and usage of phone numbers. If you’re worried about these kinds of attacks it’s in your best interest not to use Signal or any other chat provider with a complex cloud deployment.
it’s in your best interest not to use Signal or any other chat provider with a complex cloud deployment.
Okay, so what is it? What are you using instead? Manually writing ciphertext via physical mail yourself? Afterall, your computer used “a complex cloud deployment” to get to the state it’s in, so it cannot be trusted.
yeah, this should not be understood as a Signal-specific attack. geo location is a topic where privacy defenses are just woefully inadequate across the board. as a privacy person I can attest that using a VPN won’t necessarily help, though it will help against this specific type of attack.
Kind of surprised by this response. Signal already generates link previews through a proxy specifically to avoid this kind of thing.
If I am understanding their article correctly, they are mostly talking about downloading assets from the original source through the anonymizing proxy system they have setup, not their own content which is the core issue here. I wonder if their fix for this would be to just use that same proxy to access their own cdn’ed content as well.
Signal instantly dismissed my report, saying it wasn’t their responsibility and it was up to users to hide their identity: “Signal has never attempted to fully replicate the set of network-layer anonymity features that projects like Wireguard, Tor, and other open-source VPN software can provide”.
Not only have they never attempted it, they also seem to have no interest in doing so in the future either. I offered to start work on integrating tor routing into the app for text and data flows. Never heard anything back:
Yeah I was also thinking about their past blogs mentioning the proxy-ing. They also do the same thing for GIFs (which is mentioned in the article you linked but worth also pointing out here for visibility).
Just tried it again but using a custom profile picture on the other account, and it appears not to download the profile picture until you tap on it. But I’m now realizing that the placeholders for the supposedly-not-downloaded images and profile picture are blurred versions of the originals. So if the blurred version is also cached, that would mean that this in indeed a zero-click (or one-click if the caching for profile pictures isn’t vulnerable) on Signal. I might check it out later.
Often the blurred version in apps like this are constructed using techniques like BlurHash, where a tiny blurred version is delivered inline as part of the backend response to provide an immediate fallback. This issue for Signal Desktop seems to imply this is how this is done. In that case there’s no publicly cached response which can be used to side channel attack in the way described in the article.
huh. well, I don’t agree with it, but it’s a well-structured argument.
I just… don’t find any of these proposed downsides to be particularly arduous, and when I look at the constant inbound traffic that’s always trying to brute-force something or other, I see no reason to take my chances. the article does a pretty good job of enumerating many of the ways I could be inadvertently exposed; I just disagree that those things aren’t a problem, in part because I don’t trust myself to be hypervigilant about every single one of them at all times.
I’ve had password-based ssh login turned off entirely on all my machines for something like twenty years now, for both internet and LAN traffic. it’s fine. you learn pretty quickly to not lose your ssh key - which you should be making sure to learn no matter what, because if you leave a copy of the file somewhere and forget it’s there, you have unexpected attack surface.
The general consensus in my professional network, even reaching up to EU SME’s (!), is to just geoblock the UK given the costs to reach compliance are usually much higher than any business brought in by possible UK customers. This is also in light of other impeding UK legislation.
I would also do the same here in this case, if I were you @pushcx. This is a community of tech-literate people who know how to get a VPN. At the same time, the (slight) inconvenience is a constant reminder of this overbearing legislation for all UK visitors, and it might have a stronger effect on the situation than we could imagine.
It shocks me time and time again how much of a postmodern cyberpunk surveillance- and police-state the UK has become. The UK’s people seems to be content with this development given they voted for it, so who I am to judge? Fortunately I don’t have to live there.
Because they actually believe it is a good thing, and would have passed such an act themselves if it didn’t already exist.
It is very easy to repeal an Act which has not yet come in to force. The coalition gov quickly repealed the ID card legislation despite it already being in force, with its active database, and cards already issued.
So by their deeds (and/or inaction’s) you can determine their beliefs.
I suspect it’s more that they simply don’t understand it. My MP is one of them and he is, uh, not the brightest. I’ve never had an interaction with him where I felt he understood the legislation that he was voting on at more than a superficial level. His level of reasoning is probably ‘child abuse bad, law says child abuse bad, law must be good’ rather than having any actual understanding of the impact of the law.
Not sure. There’s a fair overlap between Labour today and Labour 17 years ago, and you may recall the Terrorism Act 2006, which had similarly vague formulations about what exactly constituted “glorification of terrorism” (which in turn provoked a bunch of SF authors to release a book called “Glorifying Terrorism” in order to see what would happen). It’s not unlikely they view having a law that can be inconsistently applied (“oh but we don’t mean you, you’re one of the good ones”, “we know it when we see it”) as an actual feature - after all, they don’t bear the costs and neither does anyone they care about.
To be fair, irrespective of the belief / intelligence of individual MPs, most are backbenchers with very little power within the party and just follow the whip (especially considering how many are new MPs)
In this particular case they probably agree with it but also it’s a general fact (I haven’t actually checked this - it’s just from my experience - so somebody please correct me if I’m wrong - https://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham's_Law%22 !) that governments repeal almost nothing. They almost universally prefer to spend their limited energy on passing new laws. I’m not sure why that is - maybe something to do with Pareto optimality.
the primary goal of most politicians is to get re-elected. due to sample bias, this is especially true with ones who’ve been around longer and thus have the seniority to influence agendas at the party level.
passing a new law is something that it’s easy to brag about and take credit for. it presents the public with an achievement that one or more involved politicians can attach their names to, or over-state their role in if they’re so inclined.
repealing a new law is not nearly as easy to convince the public to congratulate people on, since the net result is that there’s nothing to look at, no new results to point to. it’s certainly the case that there are parties that build their brand on repealing laws as their primary activity, but there is an inherent disadvantage to doing so.
Indeed: it is most probably on the new government’s radar, but they might choose to allow shit to hit the fan before “saving the day”.
We can draw a parallel with hero culture in many dysfunctional companies: there’s no glory to gain by avoiding an incident while the employee who solves an incident get credit, even if they caused it in the first place.
The intentional invocation of Cunningham’s law went right over my head. Shame on me for reading before finishing the first coffee.
I wonder if intentional invocation of Cunningham’s law is like intentional invocation of Murphy’s law… such an act might well turn the incorrect statement into a question and therefore place it on the wrong side of the law.
I’m now a bit unhappy that I did actually read the documents last time this came up, but didn’t write down my off-the-cuffs assessment. For context, I’ve done some GDPR-Compliance related stuff before.
The first thing to look for in those documents is the thresholds and IIRC, lobste.rs is below all thresholds. Below those, a lot of things become relatively tame or already exist. E.g. you need to have a content policy (lobste.rs has one) and you need to present that you can remove illegal content on need. All of those exist.
It’s similar to the GDPR: a lot of the things look daunting until you figure out that you’re actually out of scope.
All that being said, no one is helped with 5 40 page PDFs for running an international service, so I would totally understanding if you just blocked the UK.
Note that “recommended” does not mean all of them need to be implemented. However, at a quick glance, lobste.rs is neither a large service, nor a multi-risk service (none of the content deemed a risk is discussed here), so a lot of the heavy-hitters do not apply. We do have a content policy, and we do have a legally responsible person and a path to report illegal content. There’s no need for documenting all this at a higher level.
I thought so as well, having worked for a MEP. Well, I’ve started to read the actual law and OH MY GOD, what a mess! Are all UK laws written in this impenetrable style? With multiple levels of outlining and gotos that criss-cross the document?
GDPR can be read and grokked in a day. But this?
Like this for instance:
(1) Subsections (2) to (4) apply to determine which of the duties set out in this Chapter must be complied with by providers of regulated search services.
(2) All providers of regulated search services must comply with the following duties in relation to each such service which they provide—
Why the hell do you include (1) if you scope (2) to (4) appropriately anyway? Was it an actual intention to make this into as long as possible read? Was author paid by word?
(3) Additional duties must be complied with by providers of particular kinds of regulated search services, as follows.
Oh god, kill me now. Yes, captain obvious, that’s what legislation is supposed to do. I am holding my breath, I am literally reading the law here, the suspense is killing me, do tell, what are the duties of various kinds of search service providers?
(4) All providers of regulated search services that are likely to be accessed by children must comply with the following duties in relation to each such service which they provide—
(a) the duties about children’s risk assessments set out in section 28, and
(b) the duties to protect children’s online safety set out in section 29(2) to (8).
Oh yeah, strong agreement from me. The ethics module in my CS undergrad had us reading both British and EU legislation. The British was no doubt the most confusing — worse than the ancient French texts we were reading (in French).
I think that a lot of what the Online Safety Act implements is overzealous and there’s a big mismatch between the stated aims, and what the legislation will achieve. I’m not sure the quality of the drafting is really the issue.
GDPR can be read and grokked in a day.
I think this is a misconception and possibly a case of Dunning Kruger – just because you read a legal text and think it means X doesn’t necessarily mean that other people will, or (worse) that a judge will believe your interpretation. Further, these acts tend to be pinned down in case law, which is important to understand and can materially change the nature of what certain acts mean. For example, EC261 provides for compensation for cancellations (and doesn’t specify that it applies to delays in the act, iirc), but EU case law has clarified that it also applies to delayed flights.
Why the hell do you include (1)
Why do papers have abstracts? Clarity is imperative (even if it feels like stating the obvious), and prevents further issues from cropping up down the line. It’s like how “experts write baby code” (rather than big soups of OOP). I also think as a general writing principle that providing an outline of the structure of your content is pretty important. It’s like how a technical book will say “in this chapter, we look at how to solve X problem” so that you know what it’s trying to do, rather than just having to guess.
Another GOTO? Are you kidding me?
It’s not that surprising that legislation is drafted in clauses; you can point to the clause, instead of typing things out lots of times. It also allows people to verify that the clauses are correct by looking in one place, rather than having to search all over the document.
I am sorry, but this specific piece of legislation sucks. Normally sets are defined first, then clauses that apply to those sets are introduced, grouped topically. This law used sprawling if-else and repetitive vacuous clauses. It is objectively hard to read, not easier.
Imagine being reprimanded for braking the law. What did I do? You are breaching paragraph X, section Y, clause Z. Oh, does that apply to me? Yes, section C says so. Nope. Just nope.
Sane law can be read from the bottom, looking up if you fall in the set from the consequences. Good luck here.
I have to say I don’t really understand your issue with the chapter? As far as I can tell, chapter 3 is set out roughly along the lines of
an explanation as to who has which duties of care
an explanation of what the scope of each of the duties of care is
an explanation of what each duty of care entails
Seems pretty straightforward to me. My objection is more to the actual aim of the bill rather than whether it was drafted for a mathematically-minded person.
Thanks for trying to help run down the OSA’s scope and effects. With the reminder that I don’t get much confidence from non-lawyers interpreting the law, I’ll post some more of my open questions.
I’m pretty sure Lobsters is categorized as a multi-risk service (section 5.6 on page 78 of that PDF) but can’t assemble a coherent definition between the vague terms, foreign legalisms, and bottomless well of cross-references. My primary open question is whether the OSA considers these categories to apply because all of them are possible in a textarea, only if a service doesn’t write policies opposing them and variously track its compliance (ICU A1-A7), or only if a service demonstrates a consistent pattern of being used for them. A couple relevant facts: Our search engine doesn’t search multiple sites, but we do have direct messaging and image uploads (and do not currently have access to databases of CSAM URLs or image hashes). Moderation is reactive rather than proactive screening. We do not attempt to identify individuals (except in cases of specific abuse like sockpuppeting) and do not attempt demographic measurement like age or location. This vagueness informs where we land on the Risk Level Tables (pages 58-68), which cautions in bold “we expect providers to err on the side of caution and select the higher risk level”). With the endless ambiguities in these documents and that warning, the safe thing to do is geoblock the UK at least until Ofcom makes a clarifying statement or demonstrates for a couple years that it’s not going to demolish hobbyist forums.
Because software has permeated every aspect of life, including relationships, we have regular mentions of listed topics like gambling, sex, pornography, suicide, drugs, crimes, and various kinds of abuse. I’m reasonably sure Ofcom is capable of use-mention distinction here, but it’s worth being aware of the topics if the OSA doesn’t make that distinction somewhere (especially likely around the four horsemen) or if there’s room for a political appointee to decide to make an example. I’d really like to see an explicit distinction on this in the OSA.
Also I swear I saw a page on Ofcom acting as a table of contents and listing all their current OSA PDFs but I can’t find it again. Does anyone have a link like that? I am more than a bit lost in a mazy of twisty PDFs, all alike, as I try to cross-reference these documents and run down the many vague terms.
As I noted in another comment, none of this addresses jurisdiction. While the UK and USA share legal history and culture, I also have to look down the road to when much less compatible countries (or administrations) attempt to enforce similar laws.
From spending a few days this week researching the act, I now believe it has been intentionally made broad and vague so as to allow Ofcom the flexibility to reinterpret it as technology changes. This is so that they don’t have a similar cat and mouse game that they do with legal highs.
Interestingly, under section 3.3 (pg.30) they propose to “exempt providers whose UK referable revenue is less than £10m in a qualifying period” from paying fees. I do wonder if this includes penalties.
Ofcoms enforcement of the act must be entirely self funded via fees and penalties, their own rules dictate that penalties must be considered reasonable and one can consider fining an individual many times the amount they could earn in a lifetime to be unreasonable. Due to this, and the fact that they are to be self funded I suspect that they are going to focus entirely on entities that earn more than £250m revenue in a year because it does not make economic sense to spend hundreds of thousands of pounds going after an individual where you might at best collect a few thousand in a “reasonable” fine.
It is my hope that in the coming months we see the published result of this consulation and confirmation that they will be excempting small communities such as Lobsters much in the same way as GDPR does.
This gives me hope, thanks for sharing. I was worried I’d have to block my own country for my own product (a forum SaaS thing) and I clearly need to do some research. Similarly, hobby-adjacent projects like mine, while revenue generating aren’t anywhere near the numbers matching these lawyers and penalties. We’ll see… as someone who grew up on forums and they are the sole purpose I even have a job designing/coding this whole thing disappoints me. But it flew under the radar for me and I regret not reaching out to my representatives when I had the chance.
While my plans to leave this country are largely due to things like this, it does feel like a “frying-pan-fire” situation as the sentiment reflected in the OSA seems to be cropping up in a lot of countries, including where I’m heading… I do miss the old internet. (grumpy old man moment)
speaking as someone who immigrated to Canada a few years ago, good luck! it’s a country with many challenges but it’s a functioning democracy, and remains a very good place to live.
thanks! I’m looking forward to it, and looking at the options a few years ago, everywhere has its issues so its a tradeoff, canada struck a nice balance for myself and my partner as a similar-to-the-us-but-not-as-insane kind of vibe, plus natural beauty ticks a lot of boxes!
Thanks for trying to help run down the OSA’s scope and effects. With the reminder that I don’t get much confidence from non-lawyers interpreting the law, I’ll post some more of my open questions.
I agree on this. However, as service providers and people that do need to create compliance, I also find interaction with peers useful.
To be honest, I have drafted and killed a few replies to this. I think your assessment of lobste.rs as a multi-risk service is overly negative, but given the poor communication of Ofcom particularly towards lay-people, I would just engage in trying to nail pudding at the wall together with you and reading tea leaves. So I’ll leave it at the gut-feeling and be honest about that being the only thing I can address.
This is the Ofcom page I was thinking of as a table of contents. It’s a little confusing because some early links are repeated, but Ofcom’s guidance totals 2,426 pages, with a further 7+ guides forthcoming. Plus, of course, the 350 page act itself.
They added 295 pages of material this morning. It is not linked from the page I thought was a table of contents, so I guess that is not intended to be complete. It seems likely there is more than the 3,071 pages I’m aware of that Ofcom expects sites to comply with.
The first document says “From today, these services have three months (by 16 April 2025 at the latest) to complete their children’s access assessment.” and then another doc says “To establish whether a service, or part of a service, is likely to be accessed by children, service providers must first complete a children’s access assessment.” which is this doc which depends on section 37 of the OSA which defines the “Meaning of ‘likely to be accessed by children’” as “possible for children to access the service or a part of it” and “the child user condition is met”, where (to skip several dereferencing steps) that is defined in the OSA chapter 4, section 35, subsection (3), clause (b) which says “the service, or that part of it, is of a kind likely to attract a significant number of users who are children” which is all the definition available and also hinges on the infamously undefined term “significant number”.
I know learned to program as a child and that most children are capable of reading, and a quick search shows the UK has promoted many programs encouraging children to learn programming. So perhaps Ofcom also demands Lobsters perform intrusive checks of visitors’ private personal information (“highly effective age assurance”) like demanding a state-issued identification document and live selfie. And of course to document the hell out of that decision and implementation. Or perhaps not. There’s really no way of knowing.
You’re correct, they made a new tool available today.
As long as I’m writing, I’ll include a small update I mentioned on yesterday’s office hours: I’ve started an email conversation with Ofcom and gotten a human response about routing my inquiry to the right office. I recently talked with the owner of a US small, noncommercial forum who asked Ofcom about the OSA and received a demand that all services, no matter how small, undertake the enormous task of evaluating against their thousands of pages of vague guidance. So I didn’t bother re-asking that and only asked them to explain their legal basis for claiming global jurisdiction.
I’m also reaching out to the US Embassy of the UK to see if they can help with the jurisdictional overreach.
The US similarly asserts extraterritorial jurisdiction over communications received or receivable in the US. I don’t think you’re going to get far with contesting jurisdiction to effectively build walls around countries.
Well, my optimism in that other thread is beginning to look pretty foolish.
FWIW, which I concede will likely turn out to be very little, I did write to my MP. One of their caseworkers wrote back a couple of days ago to say that they had raised my concerns with the Culture Secretary for “a formal response in writing”. I look forward to receiving it.
I finally received a reply today from Baroness Maggie Jones, the Minister for the Future Digital Economy and Online Safety. Alas, it appears to be a copy & pasted screed about “small or medium-sized enterprises” and does not address any of the concerns I raised about volunteer-run community forums.
As you probably know, a letter “from” a Minister is actually a letter from a minion (I was one, briefly) and that minion is paid to avoid committing anyone to anything. So its content is worthless … the fact that they felt compelled to pay someone to write it might be a very good thing though :-)
I’m afraid I’m pretty sure they just clicked the “Mail Merge” button in Microsoft Word. I suppose they must have received enough complaints and queries about the OSA to make it worth writing a template reply… but they don’t seem to feel compelled to do anything more than that.
I’m afraid I’m pretty sure they just clicked the “Mail Merge” button in Microsoft Word.
Right, but when they do that they send a report up their management chain saying how many addresses got merged in, so at least you’ve added one to that :-)
Yep writing to your mp is the same thing. They pick their pre written blurb that they believe pertains to the general topic of your letter. As such it’s usually totally unresponsive to what you say to them.
Speaking of the GDPR, there’s surprisingly little online furor over the OSA compared to the frankly almost hysterical coverage of the GDPR back in the day.
For instance, if lobste.rs is in scope, than HN is, and I don’t think they’re gonna take any action blocking UK users. But I might be wrong.
The fediverse also seems to be shrugging off the OSA, even if many instances probably are in scope.
TBH, I’m a fan of the GDPR. Having run affected web services for years, GDPR was mainly a slap on the wrist e.g. to providers that (already illegally) didn’t practice double-opt-in for mailing list. GDPR was a massive “we’ve had it, there’s fines now, and we note down the best practice here”. Like, a lot of the stuff the GDPR bans were already illegal, but all the large services ignored it, because the laws were toothless.
So people who were already quite conscious around privacy, which most self-hosters are, were pretty fine. Also, there are clear indicators in the GDPR should be easy to comply for small services and the first reaction is a stern warning.
The OSA documents are hard to track and even finding the definitions of what a large service is always takes me 5 minutes, because it’s hidden somewhere.
To be honest I think a lot of the furor was from online marketers who saw a credible dent appearing in their cash flow and who tried to astroturf some opposition in the US. Even now the maliciously compliant “accept all cookie” banners are being blamed on the EU, not on the advertisers who implement them.
I disagree. It introduces tons of regulations with often marginal effect. Big service companies can be mostly happy about the regulatory capture effect.
But basically most interesting services are in non-compliance with a strict reading. E.g. from the text of law (I am not a lawyer), I cannot see how it is allowed that I can check my emails from a non-compliant country.
It also applies to non-online services like doctor’s practices. Theoretically requiring doctors to call people by something other than the names that is not PII.
Rather than imposing all this churn on so many, one should focus on penalizing big abuses.
My comment included quite different points, though.
One was that it applies to a lot of small businesses – yes, I did here about concrete absysmal effects there about creating extra costs but I did not hear about persecutions of small businesses for violations.
The second one was about the hypothetical cases with the email.
If I read your comment right, you focus on violations for which there have been actual fines. I don’t think that this is the full picture.
Law, in the best case, should make clear what is allowed and what is not. If it fails to do that, the ambiguity creates unnecessary work. In the case of small businesses, a lot. And I am not even talking about primarily online businesses.
I think that I am connected to a bubble of small business owners that actually want to comply with the law (in Germany), and it is very annoying and creates extra costs. Try to get a lawyer sanction your GDPR compliance and tell me what the bill is… And that is only a fraction of the costs.
Also please don’t mistake this as me being in favor of keeping all of this unregulated. It can’t be. Many companies would just do things to their own advantage. E.g. disallowing shady double opt-outs is not a problem for me. But the GDPR does so much more…
I do beleive that the GDPR is not a problem for many best-practice online services. But that is only a small fraction of business in total to which the GDPR applies.
I can’t speak for the situation in Germany. Trying to get a framework that works in all of the EU with the very differing views on privacy is a big job. And if there’s a balance between indivdual’s privacy and business’ costs, then maybe the privacy is worth more.
I would expect trade associations for the businesses involved to have at least some guidance in place by now, since the law has been enforced for so long, so that an individual business owner can ask “is this software GDPR compliant”, “what is the best practice for this situation”, etc.
it’s worth understanding that the GDPR challenged power, whereas the OSA’s primary impact is on already-marginalized groups and local, community-centric undertakings. complying with the OSA will be significantly easier for the big companies than complying with GDPR was and remains, because it simply establishes rules for already-existing enforcement mechanisms to follow.
the topics and themes the media chooses to cover are not immune to influence from capital. :/
It’s similar to the GDPR: a lot of the things look daunting until you figure out that you’re actually out of scope.
I disagree with this sentiment. This is simply not the right kind of comparison.
For GDPR your scope is: If you don’t invade people’s privacy there is close to nothing to do. Afterwards it scales with how much you want to invade people’s privacy and even there it feels soft: Tell people whom you give data to. Keep track of whom you give data to. Ask for permission before you do. Also explain what you use it for?
For the Online Safety act you need to take action based on your users’ actions, also you have to make essentially legal judgement, which means you wanna be strict, because the risk of not being is yours. That’s just for allowing users to communicate/interact, which basically affects every forum. So unlike in GDPR where if you just have some website collecting nothing you can just say “Not collecting anything here” in a privacy policy the mere fact that content is being made available in any form means you need to have a mechanism to be scanned BEFORE it even is published. That means that website owners, even for hobby things, interests groups, etc. there need to be measures in place, which as is seen has the potential to kill projects, simply by costs involved. For GDPR if you don’t give user data away the only cost is becoming aware of it.
On top of that the “harmful content” part isn’t defined, while in the GDPR private data is pretty well defined. So the OSA is also pretty subjective, which is never good for a law (but I understand it might be necessary sometimes).
So while I agree with the sentiment to remain calm and objective I don’t think that comparing OSA with GDPR in anything but attention and the scope of who is the target audience (everyone running a website) is really justified. GDPR is what YOU do with other people’s data. OSA is (also) about what OTHERS do (ab)using your platform.
It’s similar to the GDPR: a lot of the things look daunting until you
figure out that you’re actually out of scope.
I disagree with this sentiment. This is simply not the right kind of
comparison.
For GDPR your scope is:
The word “scope” was used here as a legal terminus technicus which
differs from the ordinary meaning. Law acts often come with an
explicit definition of their “scope of application”, usually at the
beginning. For the GDPR, this is laid down in Art. 2 and Art. 3 GDPR.
If you do not fall within the stipulations of these articles, the
entire GDPR and all it has to say about privacy rules does not apply
to you. We call this “to be out of scope” for brevity, because nobody
really wants to cite the precise articles all the time.
I cannot say whether the OSA works like that, because I’m only
familiar with German/European law, but I would highly suspect that
this is what was meant.
I cannot say whether the OSA works like that, because I’m only familiar with German/European law, but I would highly suspect that this is what was meant.
OSA is neither German nor European. (Don’t think it’s relevant here but in fact German and European law are a completely different system from UK, civil law vs common law).
However, it exposes a very interesting phenomenon in many people’s thinking in computing: that many people have belief systems that are religious in nature. They do not care about evidence, rationality and logical reasoning: they just believe.
So, for instance, I’ve found that the Unix world in general is now so large, and so old, that many people have erroneous folk beliefs that are set by 50+ years of tradition. They’re not faith: they just don’t know anything else. Their little domain is the whole world because they’ve never known there was anything else.
That’s human nature. It’s sad – there’s so much more out there – but it’s how people work.
So, for instance, a few random erroneous beliefs from the Unix tradition are:
all OSes have “files” in “filesystems” with a hierarchical tree structure
programming languages come in two forms: interpreters, and compilers. There are profound important differences between these.
the basic programming language is C and everything else is implemented in C.
C is a low-level language, meaning that it’s close to how the processor works.
None of them are even slightly close to general truths, but they are axioms in Unix.
But that’s just ignorance.
The faith thing is much more frightening to me.
That LLMs are artificial, meaning that they were made by people. (They weren’t, aren’t, and can’t be.)
That they are intelligent, meaning that they can think in any way at all. (They can’t, but they fake it in ways some find compellingly persuasive.)
That they can program. (They can’t, but they can make up text that looks like code.)
That they can analyse and improve themselves. (Nothing can.)
That bigger models will make them better at what they do. (This one just exposes basic ignorance about very simple statistics, such as scaling effects.)
That LLMs are in any way a step to reasoning, thinking software. (Pure magical thinking, based on lack of understanding.)
The thing is that I now continually encounter smart, educated people who believe in these magic beans.
There’s two things I want to push back against in your comment.
1) The magic bean believer strawman
So much of the LLM discourse I see (especially here on Lobsters) is polarized to the point where you’re either a believer or a denier. I see people who make very reasonable and tempered claims about the utility that LLMs provide them (sometimes with evidence!) that are blanket rejected because “ChatGPT can’t count its own toes correctly” or whatever new chicanery people have devised to get it to spit out a wrong answer.
Yes, there is an LLM cargo-cult. Yes, there are CEOs aplenty who want to sell you magic beans. Yes, I am sick of it too. But can we please reserve our hate for the myriad people and ideas deserving of it and openly hear out the ones who aren’t coming with this kind of agenda? The ones who honestly believe they have found something helpful?
2) LLMs ~ magic beans
It’s not clear to me whether you’re arguing in your comment that LLMs have no merit whatsoever, but since that’s a common sentiment I see and want to rebut, you’ll have to forgive me if I have inferred incorrectly.
The other thing that bothers me is the almost religious rejection of any LLM results and realities. Correct me if I’m wrong, because I only speak from vibes, but I feel the anti-LLM sentiment when copilot came out was “LLMs will never write code.” Advent of Code this year, for example, has had sub-30 second submissions on the leaderboard – if this is not evidence that LLMs are capable of some kind of programming, I don’t know what is, because humans surely cannot read the whole prompt and code in that much time. And now the sentiment I see has shifted to “well, LLMs can write some code but it’s not complicated,” seemingly in denial of these previous claims.
I want to remind/inform whoever’s reading this that in the decades-old field of program synthesis, the poster child for the longest time was FlashFill, which generates (simple) Excel formulas from examples. There simply wasn’t any usable general-purpose tool for program synthesis (regardless of hallucinations or syntactic inaccuracies or …). Now a large number of synthesis papers are (in a very simplistic and reductionist approximation) LLM + special domain.
You can debate whether LLMs in their current form have legitimate utility, but this debate becomes personal and I expect colored by your own perception of how they help you (placebo/nocebo). I think it’s too reductionist to write them off entirely.
These are but a brief summary of my thoughts on LLMs and program synthesis, I hope to getting around to writing more in the new year…
But can we please reserve our hate for the myriad people and ideas deserving of it and openly hear out the ones who aren’t coming with this kind of agenda?
The problem with this framing is that you’re only looking at the ends, when for many of us the means play a part in the resistance.
If you bring an LLM that was only trained on content whose authors actively assented to inclusion, and where no companies were seriously considering building their own nuclear reactors because it takes so much power to run, and where there’s some hope of ownership and control of the LLM by end users instead of just large technology/surveillance companies, then sure! I’m all ears!
Alas, there is precious little of that in the space where Copilot and ChatGPT and Claude and so on are playing.
Completely agreed. I’m not interested in having discussions about trying to invent useful applications for this technology, because even the small-scale models that hobbyists can run on their home computers are produced at massive expense by extremely unethical actors out of datasets which were harvested without consent and then legally encumbered.
Supposedly innocent tinkering with LLMs furthers the goals and rhetoric of absolutely monstrous entities and helps them justify their abuses of the commons. Building hobby projects on LLaMa and writing effusive blog posts about cheating at code golf, automatically synthesizing a fart app for your phone, or accomplishing any other number of trivial tasks with help from a Claude-powered rube goldberg machine is doing free marketing for the generative “AI” merchants and cloud vendors.
LLMs are completely radioactive from many ethical angles even before you start digging into the harmful ways they’re being applied today or (as this article focuses upon) their complete unsuitability as a building material for reliable, efficient, or generally trustworthy software.
LLMs are completely radioactive from many ethical angles even before you start digging into the harmful ways they’re being applied today or (as this article focuses upon) their complete unsuitability as a building material for reliable, efficient, or generally trustworthy software.
Then (as someone who believes these points should be made) I implore you to focus on the first half (not struck-through) part of what you’re saying. I think it does your argument no good to incorporate claims that are deniable, especially if they apply the fallacious reasoning I discuss in my parent comment.
You don’t even need to open the door to arguing whether in two years ChatGPT can write Minecraft if its existence today already constitutes as significant of a problem as you claim. I think it’s good to have people thinking critically about these tools (ethically and technically), but thinking critically means not getting swept up in the LLM hype and the anti-LLM hype.
I think it is good that companies are building their own nuclear reactors to power datacenters for generative AI applications. We need way more of this kind of thing, cheap energy makes all of society more prosperous and we get cheap energy from having lots of nuclear power plants generate it.
Ownership and control of LLMs by end users is important and it’s a genuine concern that we don’t get locked in AI systems controlled by a small number of companies. But this is not really a different problem than a small number of companies controlling proproetary software platforms used by huge swaths of the population (i.e. Google and Meta and Twitter and Apple existing).
I think it is good that companies are building their own nuclear reactors to power datacenters for generative AI applications
How many are actually doing this? Nuclear power plants take many years (often a decade or more) to build. Most of the announcements I’ve seen have been ‘we aren’t a climate disaster, look we have this long-term strategy to be energy independent. We’re not actually funding it at the level required to be realistic, but let us burn vast amounts of coal now, we promise it’s a temporary thing!’.
Even if a SMR is built, it still needs huge volumes of clean water to operate and cool. It’s never going to be a little container-sized cube that magically emits electricity, like a portable fossil fuel generator.
Then make it illegal to burn coal to incentivize building the nuclear power plants faster (and reduce the amount of regulation on them); I do not want to sacrifice humanity’s ability to harness prodigious amounts of energy on the altar of the climate. This isn’t even about LLM datacenters specifically, I want nuclear power plants powering aluminum smelters too. Or maybe photovoltaic solar is actually cheaper if we just pepper the earth with panels, in which case I support doing that, and the electricity can go into LLM datacenters as easily as it can go into anything else.
What I don’t want is for any human endeavor that uses a lot of electrical energy to get labeled a “climate disaster” by people who want to shut it down - least of all because scrubbing CO2 from the atmosphere itself is something that’s gonna require a lot of electrical energy.
But this is not really a different problem than a small number of companies controlling proproetary software platforms used by huge swaths of the population (i.e. Google and Meta and Twitter and Apple existing).
I’m glad you agree it’s a problem! The difference, I think, is that I’m not constantly hearing about how I should learn to stop worrying and love Gmail.
You hang out in very different places than I do, it appears. That’s interesting to realize… I don’t even know what it would be like to be in a place where that view is low-status. It felt like the mainstream belief within the broader software community, when I was growing up. It still feels like a common belief to me. It’s really interesting to hear a perception to the contrary; thank you.
As a bit of extra flavour… I, and the people I grew up with, shared the same belief you did: we believed that software like Gmail is unethical and that we were taking the moral high ground by believing that.
For me, though, there was a moment along the way where… I still love open source software and wish more of the world was made up of OSS, but also came to the conclusion that many of the people I grew up with who held those beliefs the strongest… weren’t really going anywhere. Many of them are still exceptionally talented programmers but they’ve lived a life where they’ve consistently struggled to make enough money to house and feed themselves. Whether they’re considered low status because of their beliefs or because of their relative poverty or due to other lives choices is hard to say but it’s pretty tough to argue that they’re doing well in life.
In my life now, from where I’m standing, it seems like the general broad perspective towards OSS is mostly indifference. Most people I know run their production software on Linux, some on Windows, and none of them really do it for OSS reasons but rather because Linux is a good platform. They don’t really care that it’s open source, just that it works. I’m actually feeling a bit sad writing this.
but also came to the conclusion that many of the people I grew up with who held those beliefs the strongest… weren’t really going anywhere
I don’t think it makes sense to evaluate any idea based on perceived success* of those that hold it. Especially when that idea is likely to make you avoid chasing the riches of the tech industry.
To be blunt, reading that it sounds to me like you were willing to compromise those ideas for money and don’t value the same things as the people “not doing well* in life.”
* this is subjective as it depends on personal goals only those people can judge themselves
I agree most people are apathetic to things being FOSS, and that is quite sad but also why activism is needed. Not only for FOSS but all subjects; the status quo has strong inertia, if not players with incentive to maintain it.
To wrap back to the original subject, I believe Gmail is unethical mostly due to Google’s data pillaging and federation busting. Gmail not being FOSS is part of it, but far from the main reason, and mostly orthogonal: I believe VS Code is unethical even if it is mostly OSS and VS Codium exists.
To be blunt, reading that it sounds to me like you were willing to compromise those ideas for money and don’t value the same things as the people “not doing well* in life.”
100% will agree that I did compromise on those ideals in part for money but in the bigger picture for happiness as well.
The real tragedy with respect to the “not doing well in life” part is the magnitude of that. I agree it’s relative but it makes me so sad to see the some of the brilliant people I knew in school posting on Facebook (there’s a certain irony there…) about how their night-shift gas station job sucks and that they have to kick out another roommate for stealing or being abusive. It’s not just that they’re “not doing well” on my personal scale but that they also seem genuinely unhappy themselves.
But… this is all just one small community. I’m sure it’s not a representative sample.
The problem with this framing is that you’re only looking at the ends, when for many of us the means play a part in the resistance.
That’s fine to articulate, but my reply is to ~lproven’s comment which makes no mention of the ethical considerations for using LLMs.
To be clear: I’m not saying that discussions about the ethics of LLMs should be silenced (I am inclined to believe otherwise). But I am saying that even if you think you have the moral high ground for not wanting to use LLMs, this doesn’t entitle you to misrepresent what they’re capable of. (not accusing you specifically, but I hope you get what I’m saying)
Put differently, I don’t like the simultaneous standpoints I see people take of “LLMs are bad for authors and the environment” and “LLMs can’t produce code, LLMs are stupid, etc.” The first point is much more damning and — perhaps more importantly — easier to verify facts for. I don’t see any good reason for denying evidence or inventing strawmans to support the latter point.
They do not care about evidence, rationality and logical reasoning: they just believe.
I feel the same but like… the opposite lol
That LLMs are artificial, meaning that they were made by people. (They weren’t, aren’t, and can’t be.)
I don’t even understand this. Is it even a question? Of course they are artificial.
That they are intelligent, meaning that they can think in any way at all. (They can’t, but they fake it in ways some find compellingly persuasive.)
You’re gonna have to define “intelligent”. Depending on the context, LLMs are obviously intelligent, but under other definitions it clearly is not - in fact you’ll find that many people have wildly divergent definitions. A panpsychist is not likely to say they’re intelligence, a functionalist may. Even by some pretty rigorous definitions. This isn’t a religious question, it’s a metaphysics question. Is intelligence the ability to reason? Is it a sufficient set of justified beliefs/ knowledge? Does it imply abstract properties? Is it an abstract property? Is it physical? Emergent? Seriously, it’s incredibly reductive to say “LLMs are not really any kind of intelligence”.
That bigger models will make them better at what they do. (This one just exposes basic ignorance about very simple statistics, such as scaling effects.)
Also confusing since that has been the case. The question is really just a matter of the limits of this scaling. It’s sort of like saying “gzip isn’t going to benefit from a larger disk because at some point compression has information theoretic limits” well yeah, but the bigger disk still helps. I’m not really sure what you’re trying to get at here though, maybe you can be more specific.
That they can program. (They can’t, but they can make up text that looks like code.)
I mean… this is just a weird definition to me. They use statistical reasoning to generate valid programs. I guess you think that’s not programming?
That they can analyse and improve themselves. (Nothing can.)
WOW lol sorry but I’m out. You’re seriously going to accuse people of religious thinking and then make these kinds of assertions? Like there are entire metaphysical theories devoted to these questions that you’re so glibly dismissing. Nothing can analyse and improve itself? That is such a wild claim to just assert like this.
That LLMs are in any way a step to reasoning, thinking software. (Pure magical thinking, based on lack of understanding.)
If we define reasoning as using evidentiary inputs to produce conclusions, LLMs reason. Statistical reasoning is reasoning.
These arguments (read: unjustified assertions) are so weak and you seem to not even realize the metaphysical commitments you’re making by pushing this worldview.
If you want to have a reasonable discussion, by all means please do. I see so few discussions that look reasoned on this site and it’s a bummer. Let’s talk about knowledge, let’s talk about reasoning. Let’s do it! Let’s ditch these assertions, let’s ditch the question begging, let’s have a little epistemic humility, okay?
Yeah, it doesn’t really matter whether LLMs have intelligence and rationality. They can make our lives easier by solving some problems, just like cars and computers, and that’s enough. That being said, I also agree with the article that LLMs need to be improved in terms of reliability and explain-ability.
Yeah, it doesn’t really matter whether LLMs have intelligence and rationality.
This black-and-white language leads to confusion.
I recommend the following definition: Intelligence is the ability for an agent to solve a task. The degree to which such abilities span different areas is the degree to which we call it general intelligence.
Intelligence is the ability for an agent to solve a task.
Nope.
Place a blind invertebrate into a box with a particle of smelly food. Wait. It will move around in a semi-random walk but it will find the food. Problem: solved. Was intelligence used? No. The algorithm is so trivially easy, I’ve solved it:
Move in long straight lines until you smell food, then move shorter distances and turn more.
That’s it. That is the algorithm. It works.
Pools of water connected by pipes: find the lowest level. It works.
In any case, researchers Shane Legg and Marcus Hutter have made the case that intelligence includes the following features:
Intelligence is a property of some entity or agent that interacts with some form of environment
Intelligence is generally indicative of an entity’s ability to succeed (by a given set of criteria) at a particular task or achieving a stated goal
When speaking of an “authentic” intelligence, there is an emphasis on learning, adaptation, and flexibility within a wide range of environments and scenarios
The only thing you did with your comment here is do an appeal to authority.
How about this: nobody is an authority on what is an intelligence and what even is reasoning. People get a PhD and start pretending to know, that’s the crux of it, let’s not sugarcoat it please. That applies to every single non-practical discipline, philosophy and many others included.
Also your comment came across as disingenuous because you first offered a blurry definition of intelligence and then, when called out, retreated into the appeal to authority and that a need for nuance is needed.
I don’t see how the latter follows from anything at all. No need for nuance, whatever that means in this situation even; we need good working definitions of the sort “what constitutes a fusion reactor?” for example.
All the discussion about what passes for “AI” these days is just sad to watch and that includes your comments. It resembles theological discussion first and foremost.
Surely technically-inclined people can do better? Inputs, expected outputs, process, all that? You know, engineering?
Personally, your comment comes across as unkind. Maybe just a bad day? Are you willing to try a more constructive and charitable direction?
Some responses:
Remember the thread context; I was criticizing some flaws in a definition someone else offered. My definition addressed the flaws I pointed out.
I “retreated” above? I’m happy to admit if I made a mistake, but where are you getting this?
It isn’t fair, accurate, or charitable to say that “appeal to authority” is all I did.
I pointed to what I think are some good jumping off points. Did you read any of them (Russell, Hutter, etc)?
Do you know a better definition of intelligence? … and why do you think it’s better? (You can see in my other comments the need for a definition that isn’t human centric.)
I’ve seen the anti-PhD venom before. I used to say the same kind of thing decades ago. It was ignorance on my part. This lessened as I interacted with more people with Ph.D. experience and as read more machine learning and CompSci literature.
No experts? There are people with more and less expertise. Acting like there are no experts is an unhelpful exaggeration.
If you read the comments I’ve written, you’ll see it is difficult to place me into any simple categories regarding AI.
I find it bizarre that you think my comments are anything like theology. Offering a working and operational definition of intelligence does not theology make.
I’ve also seen the anti-philosophy trope before. It is unfortunate and self-limiting. The classic response here is: many other sciences and disciplines were birthed from philosophy, but philosophy rarely gets credit. Yes, some philosophy is painful to read. One usually has to put in a lot of effort searching, thinking, writing, and discussing to reap the biggest benefits. Asking for reading recommendations from people you respect is a good start.
Well, it might be cultural background on my part because I am not used to dance around disagreements, and as I get older this is less and less likely to ever change. Not my intention to offend, mostly to pull you away from what I perceive is a comfortable and maybe even complacent position.
I did simplify your comments, that much is true, and 99% of the reason is that the “AI” discussions inevitably devolve into “But how do we know what intelligence is? We might already be witnessing it but are disparaging it!” which, my turn to say it, I view as extremely unproductive, not helpful for advancing any discussion, and basically serving no other purpose than to congratulate ourselves how smart we are, and to offer a plethora of very out-there “what if”-s.
If you believe you are doing something more than that then I’d love to see more concrete claims and proofs.
I have not read philosophy on the mind and intelligence. I tried. I found it unproductive and very rarely did I stumble upon a lone article (not a book) where somebody actually attempted to invent / classify a framework in which we should be thinking about what mind / intelligence are. Everything else I attempted to read did read like empty hand-waving to me.
If you are willing, I’d like to get back to some semblance of a topic: do you believe LLMs can “reason” or are “intelligent”? Do you believe we are mistakenly putting them down while they, the poor things, are the next-stage-of-evolution, marginalized and oppressed artificial life forms that are just misunderstood? If not, what, apropo, was your point?
Do you believe we are mistakenly putting them down while they, the poor things, are the next-stage-of-evolution, marginalized and oppressed artificial life forms that are just misunderstood? If not, what, apropo, was your point?
You’re being a bit of a dick here on multiple fronts, and misspelling the italicised apropos in the final sentence doesn’t shake the look.
Misspelling can happen to anyone. Shame that the glorious and almighty “AI” never improved autocorrect in phone keyboards, right? I’m sure that requires godlike intelligence though, so it’s excused. But hey, it actually “understands” what it does and it’s obviously intelligent. Surely.
And I’m not interested in philosophical discussions unlike a lot of people who can’t help themselves every time the “AI” is mentioned.
I’m interested in seeing proof that their downright religion-like beliefs have any rational foundation.
Alas, that kind of expectation is misguided. Faith doesn’t require proof to exist, as we all know historically.
If challenging belief makes me a dick then I’m okay with it. I was still never answered in a satisfying manner.
My conclusion is that this is a cozy topic for many. Mention “AI” and they all pull up the cigar and the 20-year old whiskey, and they’re all suddenly decorated philosophers who are perfectly qualified to speculate and to present conjectures as facts while they want us to believe that their entirely unrelated Ph.D. makes them educated on a topic that absolutely nobody has ever solved.
So yeah. Believe what you will. But recognize it’s only that: a belief.
Well, I think it matters a lot if LLMs have intelligence because the implications are pretty huge. Kind of like, “are there objective moral goods” - if we could answer that question we could rule out all sorts of metaphysical theories about the universe, and that seems valuable to me. Practically, and I think to your point, whether it’s intelligence or a facsimile of it (assuming this distinction even makes sense, which is a HUGE assumption!), as long as the results are the same it isn’t important (in terms of how it’s used).
I also agree with the article. I thought it was well written and makes sense - I thought the idea of breaking down models into testable components was particularly interesting. The comment I responded to doesn’t even seem related to the contents, which I also thought was ironic since it was a plea for rationality and informed commenting.
Well, I think it matters a lot if LLMs have intelligence because the implications are pretty huge.
There’s an even further metaphysical question that goes with that… how do we even define intelligence? What’s the threshold? Is a tree intelligent because it grows its roots towards water? Are bacteria intelligent? Fish? Cats? Dolphins? Horses?
Are current ML models like any of these? Or are they more like https://en.m.wikipedia.org/wiki/Clever_Hans? Or was Clever Hans actually intelligent, just not in the way that it was claimed? (He couldn’t do arithmetic but he could very accurately read subtle human signals)
All really interesting things to ponder on holidays :)
Yes/no questions fall flat quite often. For many interesting subjects, including intelligence, consciousness, justice, fairness, etc., there are better framings.
Kind of like, “are there objective moral goods” - if we could answer that question we could rule out all sorts of metaphysical theories about the universe, and that seems valuable to me.
We already have human-created machines that try to answer these questions. They’re called religions. Adding another one with a LLM dash of paint will probably not resolve anything.
We already have human-created machines that try to answer these questions. They’re called religions.
No, religions don’t inherently try to answer these questions. These are questions that fall into the domain of philosophy. Whether moral goods exist, their nature, etc, is not a religious question but a metaphysical one.
My statements stands; the ability to answer questions about the mind would indeed hold value, it would resolve many open questions, rule out some metaphysical theories, make others more or less unlikely, etc. It would potentially have very direct impact on human life - theory of mind is essential to theory of personhood, for example.
OK. I read your original comment as something akin to Platonism - there are eternal truths that are hidden from us. How a machine trained on the sum of humanity’s writing on these questions would be able to reveal them was unclear to me.
I see. No, I’m not expecting an LLM to reveal its truths. I’m saying that our investigation into knowledge, perhaps through exploration of technologies of LLMs, will reveal the truth.
Again, going back to Descartes, one of his main arguments for human exceptionalism as well as mind body dualism was that humans can speak. He suggested that machines could move, look like animals, look like humans, etc. He suggested that one could not differentiate a monkey from a robot monkey. But he maintained that language was the indicator of a true mind - so a machine could never master language.
LLMs at least provide evidence against that. You can argue about whether they’re a true counter example, but they have are evidence against a Descartes theory of mind. As we continue to push forward on technology we may have stronger evidence for or against other theories of mind.
Let me try to understand here: because LLMs disproved an extremely flawed hypothesis by a scientist who is a product of his time, this means… what exactly? Can you help me here? That LLMs possess mind, or is it something else?
. because LLMs disproved an extremely flawed hypothesis by a scientist who is a product of his time,
No, not at all. I was giving an example of how our ability to create machines has historically changed our understanding of our theory of mind. It was an example of how producing new technologies can help us evaluate existing theories about things like intelligence. Applying it ot a very old, well understood (and unpopular, therefor not contentious to apply to) theory was my way of giving such an example.
As I said, I think that as we generate technologies like LLMs we may be able to generate evidence for or against theories.
Maybe it is an example of how producing new tech can help us evaluate existing theories, yes… if those theories are not as old and almost laughable. Because comparing to them is a classic “tearing down a straw man” debate.
There is no straw man here, I don’t think you know what that term means. It is chosen explicitly because it is not contentious to say that dualism is rejected, and to show how one could use LLMs as evidence. That it is already rejected (largely) is the benefit of using this example.
Maybe it is an example of how producing new tech can help us evaluate existing theories, yes
Literally all I was doing. Soooo we’re good? Same page.
Artificial, meaning, built by artifice, created using the skills of an artificer.
LLMs are not built by humans. LLMs are built by software built by humans, running in large-scale clusters in datacentres. LLMs are a multi-dimensional grid of statistical weights of relatedness of words, with millions of dimensions.
The point being that humans didn’t construct those databases, can’t read them, can’t analyse them, and cannot modify them.
The humans built the software that built the models. Humans did not build the models, and can’t.
The tools that built the models are artificial. The models are not.
You’re gonna have to define “intelligent”.
Cambridge dictionary: showing intelligence, or able to learn and understand things easily
Intelligence: the ability to learn, understand, and make judgments or have opinions that are based on reason
Merriam-Webster: having or indicating a high or satisfactory degree of intelligence and mental capacity
Intelligence: the ability to learn or understand or to deal with new or trying situations; the ability to apply knowledge to manipulate one’s environment or to think abstractly as measured by objective criteria (such as tests)
These are not abstruse technical terms.
LLMs cannot reason, understand, deal with new situations, etc. ALL they can do is generate text in response to prompts.
They cannot count. They cannot reason. They cannot deduce. But they can produce a modified version of text which does those things.
It is a fallacy of thinking to leap from “it produces text which sounds smart” to “it is smart”.
I guess you think that’s not programming?
It isn’t. Something that can’t tell you how many Ms there are in the word “programming” can’t program.
You’re seriously going to accuse people of religious thinking and then make these kinds of assertions?
Yep, 100% am. Refute me: prove me wrong. Falsify my statements.
Nothing you’ve said falsifies my points. All you are doing is mocking and trying to redefine terms.
LLMs are not built by humans. LLMs are built by software built by humans, running in large-scale clusters in datacentres. LLMs are a multi-dimensional grid of statistical weights of relatedness of words, with millions of dimensions.
This is just a really weird standard. If I build a chair from Ikea parts, did I not build a chair because someone else built the parts? What’s the point of this definition?
A good simple explainer is the Financial Times’s one here and a more in-depth one is Stephen Wolfram’s one here. Just in case you were about to accuse me of not knowing what I am talking about.
It’s not that you don’t understand the technology, it’s that you’re applying ridiculously strict terms to it and acting like anyone who doesn’t take those as gospel truth must be experiencing irrational, religious thinking. Nothing in those pages is going to justify a flat assertion that an LLM is not “artificial” tbh but even if they justified it I don’t think it matters - I mean the stakes on this couldn’t be lower.
The humans built the software that built the models. Humans did not build the models, and can’t.
Shrug. I don’t think this matters, it’s just pedantic. “A human built a machine that built the thing” okay. I’ve never programmed because actually someone else built the keyboard and the compiler that produced the assembly, so no, not programming. I’m a fraud, I suppose.
These are not abstruse technical terms.
Dictionary definitions are fine but I think if you follow the citations there you’ll find they’re often cyclic. They’re also not some hard truth. These are metaphysical concepts with entire papers dedicated to breaking them down. Pointing to the dictionary as an authority isn’t a strong argument.
They cannot reason.
I disagree and I’ve already justified why.
It is a fallacy of thinking to leap from “it produces text which sounds smart” to “it is smart”.
It’s not a fallacy it’s the foundation of functionalism, which is a metaphysical theory worth taking seriously.
Something that can’t tell you how many Ms there are in the word “programming” can’t program.
Baseless. Why should we connect these two things? I can’t factor massive primes in my head but I can program. Why should I connect these two things together? Again, LLMs can produce valid programs, so I think it’s on you to justify why that isn’t programming since it’s intuitively the case that it is.
Yep, 100% am. Refute me: prove me wrong. Falsify my statements.
I provided theories counter to your assertions. I mean, should I break down functionalism for you? I brought it up, you can learn about it if you’d like. I brought up panpsychism, abstract properties, etc. I think there’s plenty of breadcrumbs if you want to learn the metaphysical foundations of what I’ve described.
If you really want to provide some sort of formal syllogistic argument instead of just baseless assertions I could probably provide really strong, rational, well researched arguments from people who study these things explaining why someone could rationally reject your premises or conclusion. Like, my entire point is that what you think are “rational” premises are totally justifiably rejected by people.
All you are doing is mocking and trying to redefine terms.
I’m not redefining terms lol these terms all have varied definitions! That’s my point. A functionalist and a panpsychist will have radically different definitions of intelligence. If you want to plainly assert things like “nothing can ever analyse itself” well jesus christ dude that’s a massive metaphysical commitment that you are making. I’m not the one making positive assertions here, I’m not the one accusing others are being irrational.
Nothing you’ve said falsifies my points. All you are doing is mocking and trying to redefine terms.
TBH you’re the one who started off with the mocking “everyone else is irrational and ‘religious thinking’, now here’s a list of baseless assertions” sooooo idk, I feel really fine with my response. I don’t think your list of assertions requires much “refutation”, I can just show that they’re incompatible with very reasonable metaphysical theories and so anyone who subscribes to those theories is perfectly justified in rejecting your unjustified assertions.
That’s absolutely fine and you are of course completely free to say that.
I have been a skeptic (with a K, which is not the normal spelling in my country) for about a quarter of a century now. But that’s when I learned the label: I’ve had the mindset since roughly when I reached puberty.
There is a whole tonne of stuff I don’t believe in that billions of people passionately, fervently believe. Many of them would kill me for it. I don’t care.
I have seen no evidence that LLMs are in any way intelligent, that they can reason, or learn, or think, or deduce, or in any way reproduce any of the elements of intelligent behaviour.
Metaphysical or philosophical arguments along the lines of “we don’t know what ‘intelligence’ means’ or “what does it mean to ‘think’ anyway?” are just pointless word games. Have fun with them if you want but I’m not interested.
So, I can happily state my assertions:
LLMs are not “AI”. They fail to fulfil both the “A” and the “I”.
LLMs are not a pathway to AGI or anything else. They’re a linguistic trick with few useful real-world applications, and the construction of ever-more-complex prompts in efforts to constrain them into producing useful output is futile: it’s just a vastly inefficient new form of programming, one which can never work because the best it can ever do is a small statistical reduction in the amount of noise in the output.
There are useful real-world jobs for neural networks, for machine learning, for evolutionary programming, and so on. LLMs are not part of it.
We are in a new tech bubble driven by people constitutionally unable to confront hard facts and uncomfortable truths, such as “this does not work, can not work, and will never work.”
I think you’re wrong, but I don’t think there is any evidence I or anyone can come up with to convince you.
That’s OK. It’s just a passing annoyance. The bubble will pop, there’ll be another AI winter just like circa 1970 (+~5) when they worked out the limitations of single-layer neural networks, and then again circa 1985 (+~5) when Unix got cheaper and faster than Lisp.
Personally I am looking forward to it. It is very irritating.
I also don’t believe that there’s any practical useful application of blockchains, anywhere or for anything, and that all cryptocurrencies, NFTs, Web3 and the entire field is an empty bubble that will implode.
Other things I am happy to tell billions of people they are wrong about…
Supplementary, Complementary and Alternative Medicine. It’s all 100% fake. Medicine is what can be proved to work; if you can’t, it’s not medicine. That’s why I use the term: SCAM. It’s a multi-billion dollar industry and it’s entirely and completely fake from top to bottom.
This one is easy: show it works, and it immediately ceases to be SCAM, it becomes conventional medicine.
Religions. All of them. There are no gods, no soul, no afterlife, none of it. Every believer in every religion alive and who has ever lived: all totally wrong.
I am perfectly happy to recant, if I am provided with objective, verifiable, reproducible evidence of the supernatural.
But until then, I repeat, it is all completely fake, and so is the entire field of LLM-driven “AI.”
There is a whole tonne of stuff I don’t believe in that billions of people passionately, fervently believe. Many of them would kill me for it. I don’t care.
None of my arguments are “lots of people believe X so X is true”. What I’m saying is that there are very rational, viable, reasonable metaphysical theories where your assertions are either outright rejected or are unlikely, and you saying that anyone who doesn’t follow your line of thought is irrational or thinking “religiously” is itself an irrational statement, unaware of the dialectic.
Metaphysical or philosophical arguments along the lines of “we don’t know what ‘intelligence’ means’ or “what does it mean to ‘think’ anyway?”
That is not what I’m saying. Different metaphysical theories will define these terms differently. Again, it’s just ignorance of the dialectic to say things like “AI isn’t intelligence” - not only is it an unjustified assertion, it doesn’t even make sense without defining the term, and it would be great to align whatever your definition is with some sort of established metaphysical theory so that we can know your commitments.
It’s very common for people who aren’t familiar with actual logic, reasoning, and metaphysics to think that it’s just “word games”. You’re the one saying that people should be rational. You’re just not meeting the standard you’ve set, from my view.
So, I can happily state my assertions:
I’ll be honest. Your assertions are lacking. They’re informal, lack coherent premises, bring in tons of terminology that you’ve failed to define, etc. For such bold claims I’d really expect more.
Again, you are the one making the positive claims here, it’s kinda on you to do better if you want to accuse everyone else of being irrational.
I also don’t believe that there’s any practical useful application of blockchains, anywhere or for anything, and that all cryptocurrencies, NFTs, Web3 and the entire field is an empty bubble that will implode.
These are just incredibly bold claims. To say that there are literally no useful applications of a technology is extraordinarily profligate.
Religions. All of them. There are no gods, no soul, no afterlife, none of it. Every believer in every religion alive and who has ever lived: all totally wrong.
Yeah, I can’t stand this form of atheist. As an atheist myself, I find it quite annoying how modern atheism treat religion as totally irrational. I’m what modern atheists would call a “strong atheist” (garbage terminology but I’m going to speak in the terms that I suspect are more familiar to you) but I think anyone tho thinks that you can’t be rational and religious at the same time is just extremely ignorant of the actual papers and research on the topic. I’m not compelled at all by that research but I’m aware of it and understand its legitimacy.
I’ll be honest with you, I think you should take some time to learn basic philosophy and metaphysics so that you can understand how much you’re oversimplifying things and how bad of a job you’ve done at presenting anything like an argument. I don’t mean this as an insult, I say this as a peer and someone whose views almost certainly align to a degree with yours. You should learn what a basic syllogism is - a good exercise would be tot ake your assertions and break them down into distinct syllogistic arguments that you then defend in terms of evidence and justifications for your premises and a justification for the conclusion following from them.
That said, here’s my response to your assertions. It’s a bit tricky since you haven’t presented any kind of theory or aligned with any existing theory, you haven’t defined terms, you haven’t justified any beliefs, you haven’t put your arguments into any kind of formal logical form like a syllogism or a bayesian inference, etc. This is very “internet argument” level, not what I’d want from someone telling everyone else how irrational they are for disagreeing.
LLMs are not “AI”. They fail to fulfil both the “A” and the “I”.
I rejected this already. I reject your definition of A, you’ve failed to define I entirely. I’ve already pointed to functionalism, so if you want to reject functionalism by all means you can feel free to do so but I think it would be quite ridiculous to call all functionalists irrational even if you disagree with them. Functionalism is well defended, even if there are competing theories and great arguments against it - I brought up panpsychism, although I suspect you would reject panpsychism as, despite being compatible with atheism, it does strike me as being less likely under atheism. But that’s a whole separate issue.
LLMs are not a pathway to AGI or anything else. They’re a linguistic trick with few useful real-world applications, and the construction of ever-more-complex prompts in efforts to constrain them into producing useful output is futile: it’s just a vastly inefficient new form of programming, one which can never work because the best it can ever do is a small statistical reduction in the amount of noise in the output.
I’m going to break this down because this is actually numerous assertions.
2a. LLMs are not a pathway to AGI or anything else.
Unjustified, lacks proper definition of terms.
2b. with few useful real-world applications
I reject this trivially. I find them useful. Trivial for me to reject this.
2c. and the construction of ever-more-complex prompts in efforts to constrain them into producing useful output is futile:
Possibly justifiable, though you didn’t do so. But I would probably grant this if “useful” meant something along the lines of deterministic or “highly reliable” like the article mentions. I mean, I would grant it because the stakes are low and I don’t care to argue about it, I don’t actually think it’s true.
2d. it’s just a vastly inefficient new form of programming
I wouldn’t care to reject this because “inefficient” is so subjective and ill defined.
2e. one which can never work because the best it can ever do is a small statistical reduction in the amount of noise in the output.
“Work” isn’t defined, “small” is confusing, unclear what your point here is.
There are useful real-world jobs for neural networks, for machine learning, for evolutionary programming, and so on. LLMs are not part of it.
I think this is a really odd take since machine learning algorithms are statistical models. To help you out, what you’re looking for here would be called a “symmetry breaker”. If you grant that those algorithms are useful but you reject that LLMs are you need to show why.
We are in a new tech bubble driven by people constitutionally unable to confront hard facts and uncomfortable truths, such as “this does not work, can not work, and will never work.”
Honestly, I find this deeply ironic. I find your post highly guilty of what you’re accusing others of. I think you should deeply investigate these areas and the rational defenses one can put forward before throwing stones at others like this. You’re clearly unaware of the academic discussions around these topics, how arguments are formed, what different types of reasoning are, what evidence is, what rationality and what rational beliefs are, etc etc etc. I think I’ve provided plenty of information for you to learn about these topics if you’d like to raise the bar for discourse, something I’d personally love to see.
Again, I’m not trying to be insulting. I’m a bit too lazy to be less glib here, I hope that the mere fact that I’ve taken the time to try to express myself to you shows that I’m willing to engage constructively. I can see that you want to elevate the discourse, that you’re sick of irrational beliefs, and I’m extremely sympathetic if not empathetic to that worldview. It just seems that you would benefit greatly from learning about what that has looked like over the course of the last few thousand year and to learn about how many metaphysical theories exist that are arguably very rational.
If you want to just make baseless assertions, go for it. That’s how most people talk online. But since you went out of your way to point out how you think others do that, I thought it worth pointing this out to you.
If I were to really dig into your assertions I’d have to do a ton of steel manning of them and expand on them to make them something I could actually try to refute or justify competing theories against. You’ll have to forgive me, as much as I’d like to and believe it’s worth doing, I don’t have the time right now. Maybe I’ll write something up at some point. Until then, I’ll suggest we just agree to disagree. Cheers.
Unbelievable that this is flagged as “troll” lol this site is so fucking ridiculous sometimes. @pushcx can we please get some kind of system for handling people who erroneously flag content? Disagreeing with me is fine but it is nuts how often I put in significant effort, provide justifications, even reference research and papers, and get flagged because someone doesn’t like my post. This has happened way too many times to me.
I not only explained clearly what an argument is, what the current academic conversation looks like, etc, I even went out of my way to go beyond what I think was even reasonable to expect and responded to these assertions despite the fact that they are so poorly expressed - I was unreasonably charitable in my response, especially given the context of this conversation starting through someone stating that everyone else is irrational and won’t engage in logic.
You provide exactly nothing except long philosophical essays in technical discussions. It’s quite telling that you don’t see why you got flagged (for the record, I have no ability to do so on this website but I absolutely would if I could).
You have “rejected” and “proven” exactly nothing as well. All you do is hide behind metaphysical clouds and empty philosophy while pretending to give objective evidence.
Yeah, many of us see through that and do perceive it as trolling. Until you come up with something concrete then you will not convince anyone that LLMs / “AI” are not snake oil and that their next winter is coming up soon and it’s going to hit hard and last decades.
I actually can believe that you are fully believing what you say, which would be even sadder. In conclusion of these fruitless and never-going-anywhere “discussions” (because I am 100% convinced you will double down; prove me wrong!) I can only tell you that every now and then it’s good to revisit your own bias and bubble. Now THAT would constitute true intelligence in my eyes. ;)
By the way, I used f.ex. Claude with success. It helped me fill the gaps in my knowledge in a niche area that I was not having the patience to invest learning from scratch. I was impressed and I loved the experience.
…But it also demonstrated that to me that people can very quickly just spiral into “no, that’s subtly wrong, I need X and you are giving me 0.85*X – let’s keep refining this until I get what I want”. I actually viewed the daily prompt limit (free tier) as a very good thing: it forced me stop and revisit some assumptions that I subconsciously did along the session, and for good reason – turned out I was going in the wrong direction and was given wrong input to test an algorithm with (which really did make me laugh because Claude did not validate that input either).
And in conclusion to that topic: LLMs will at best just remain very useful tools for a bunch of tasks. They are not intelligent by absolutely any meaning of the word except those that are super loose and just love hiding behind clouds of philosophical ambiguity… like yourself.
No need to reply as reading your comments here has convinced me you are not capable of productive replies that evoke true practical observable evidence. But you do you.
You provide exactly nothing except long philosophical essays in technical discussions. It’s quite telling that you don’t see why you got flagged (for the record, I have no ability to do so on this website but I absolutely would if I could).
You can dislike my posts but to say it’s trolling is just silly. You can think “wow this guy is so dumb!” but that’s not against the rules, it’s not trolling.
You have “rejected” and “proven” exactly nothing as well. All you do is hide behind metaphysical clouds and empty philosophy while pretending to give objective evidence.
I’ve never pretended to give objective evidence? What are you referring to? As for “hiding”, what?
Let’s remember how this thread started - someone stated that anyone who thinks LLMs aren’t a scam is guilty of being irrational and “religious thinking”. That’s the first comment. I justified why I reject that by pointing to many live, well thought out, rational metaphysical theories that would reject this. This isn’t hiding or “empty philosophy”, it’s a direct contradiction to the idea that anyone disagreeing is irrational or religious thinking.
Until you come up with something concrete then you will not convince anyone that LLMs / “AI” are not snake oil and that their next winter is coming up soon and it’s going to hit hard and last decades.
I have zero requirement to do this. The positive assertion made was that anyone who believes that LLMs / AI are not snake oil are irrational. All I have to do is show that that’s not true by showing live models that are rational.
I can only tell you that every now and then it’s good to revisit your own bias and bubble
I wonder what it is you even think that I believe? I’ve made almost no commitments in this discussion because, and I’ve said this a few times, I’m not the one making the positive assertions.
And in conclusion to that topic: LLMs will at best just remain very useful tools for a bunch of tasks. They are not intelligent by absolutely any meaning of the word except those that are super loose and just love hiding behind clouds of philosophical ambiguity… like yourself.
I mean, lol. Congrats but according to the person I was responding to you are irrational and guilty of religious thinking because you think that LLMs have any use at all. So… what is it you disagree with, exactly?
Anyway you’ve just done what the other poster has done. You say “They are not intelligent” okay well that’s a fine opinion? You’re not justifying it. I guess you think justifying anything would be “hiding behind clouds of philosophical ambiguity” idk it’s super irrelevant because the premise asserted by the author was that anyone who disagrees with that statement is irrational and religiously thinking, but you think LLMs have a use so they think that about you too!
As predicted, you doubled down. There’s zero substance to your “I mean, lol” bit because I am pretty convinced this person would recognize where an LLM can save you a few minutes (if not then they are indeed too irrational).
The crux of what people like myself say is “take the tool, use it, stop believing it’s something more than it is because it absolutely it is not”.
To have any semblance on topical discussion: I claim that we have zero intelligent machines today. No intelligence. No reasoning.
I owe no proof for refusing to believe something I cannot find even if I tried very hard to find it (I mean, who wouldn’t want their job being done by agents they pay $20 - $100 a month for?). You are the one owing proof to me if you say LLMs are “intelligent” and that they do “reasoning”. The burden of proof is on the one making the claim that something exists.
So go on. Prove it. And no “according to this scientist X and the other Y who, like all of us, have zero clue what intelligence and reasoning actually are but have some credentials so they love to pretend that they know”. No – nothing of that, please. Hard facts. Let’s see them.
An easy prediction to make since… I still think I’m right. I mean, why wouldn’t I?
because I am pretty convinced this person would recognize where an LLM can save you a few minutes
We must have a pretty different reading of their posts. They make some extreme assertions about LLMs being useless and a scam.
The crux of what people like myself say is “take the tool, use it, stop believing it’s something more than it is because it absolutely it is not”
But I have no problem with that? I have a problem with saying that if someone thinks LLMs are useful then they are irrational and guilty of religious thinking.
I owe no proof for refusing to believe something
I literally don’t care. I’ve argued only against the idea that anyone who disagrees is irrational or religious thinking.
You are the one owing proof to me if you say LLMs are “intelligent” and that they do “reasoning”. The burden of proof is on the one making the claim that something exists.
They are the ones who made positive claims. I have never made a positive claim. I owe nothing other than a rational response to their claims. The fact that you seem to not understand this leads to my next point.
So go on. Prove it. And no “according to this scientist X and the other Y who, like all of us, have zero clue what intelligence and reasoning actually are but have some credentials so they love to pretend that they know”. No – nothing of that, please. Hard facts. Let’s see them.
Look, I’m genuinely sorry but I don’t think you or the other poster know what words like “fact”, “evidence”, or “reasoning” even mean. I’ve realized how fruitless it is to try to talk to people about things like this because I just don’t think an internet forum is the right place to teach someone what these words mean.
I see. Well, if your only goal was to make your parent poster less extreme then cool. I kind of thought that you were going into the other extreme: “LLMs are the future” and “LLMs are AGI” etc. bullcrap.
Nope. I never said anything about LLMs other than that one can rationally disagree with the parent poster’s view on them without being irrational and “religious thinking”.
My thoughts on LLMs are not reflected by any of these posts, or are barely reflected. They made positive assertions that were extreme and, frankly, ridiculous. They failed to justify them whatsoever. I presented rational theories that would reject them, which is all that is necessary to refute the idea that anyone disagreeing is irrational.
The details of Unix, like say stat() and terminals, are not fundamental, but the general architecture is. It could have been Plan 9, which is basically a “cleaned up” Unix in my mind (“everything is a file”, etc.)
It’s a mathematical argument with respect to the amount of software that has to be written. Narrow waists are a way of mitigating combinatorial explosion / enabling interoperability.
This argument goes back to a thread from a few years ago:
The blog posts were basically a big elaboration on my reply to that comment
(As an aside, I think that @matklad has come around to the viewpoint that Emacs/Vim are also fundamental – i.e. they are centered around the narrow waist of text, or attributed text. As opposed to having different programs that edit each different type of data. Again, this is a mathematical issue with the amount of software that has to be written by us, and learned by users.)
With regard to C – I will again claim that if you look at semantics / expressible programs, then C++, Zig, and Rust are a form of C. They all use LLVM, or started out using it.
People argued with Bjarne for years about this – why does C++ use the C machine model? Isn’t there something better?
And a design principle of C++ is that “it does not leave room for any language below it”. This has actually been true!
The main new language I know of that does anything different is Mojo. Mojo is built on MLIR, which is inherently more parallel. It has a different machine model.
All of the LLVM languages are “C-ish” in my mind, just like Plan 9 and even the web are Unix-ish. The addition of types and metaprogramming is absolutely a big deal, but they are additions rather than fundamental changes, in my mind.
It’s easy to tear down what exists – it’s harder to say what the alternative is. Whenever I read these rants, I welcome the author to build an alternative and show us :-) Or even just share a link to something relevant
Are you saying LLMs are pushing Unix or something?
The structure of the comment was: LLMs show a disturbing fact that some people in our field have religious levels of faith in things regardless of evidence. We see something that seems similar with Unix where people assume that it is inevitable, but that’s not the same, it’s just ignorance.
The bullet points they provide about Unix aren’t what you’re talking about at all. They’re not talking about narrow waists and text as a universal exchange. Consider files in a hierarchical filesystem (and I’d add: a distinction between file and data in memory). That is certainly not fundamental. The AS/400 is the clear, commercially successful counterexample for the points they brought up.
OK, I guess I’ll just repeat: show me what you’re talking about
Maybe write a blog post :-) Or just drop a link
As it is, it reads like “I have secret, fundamental knowledge that other people don’t have, but I am only going to write about it in the negative, not in the positive. Also, I am not necessarily going to do anything about it.”
I still don’t see any connection to LLMs. I guess the connection is “other people are wrong” ? Who are these people? I think there are different motivations at play, like talking your book, or simply liking things that help you get work done
Unix was just an example. People tend to believe all kinds of strange things in technology and treating such things as deity-ordained facts, but this is due to ignorance.
LLMs are so good at bullshitting humans into believing they are sapient, thinking machines; similarly the success of Unix has led people to believe all OSes are based on files. When I say believe, I do not mean in the assume sense, but in the fundamental, faith sense.
Note that the comment you just replied to is by @madhadron, who is different from the author of that first comment, @lproven. I think you meant your repeat of “show me what you’re talking about” to be directed at @lproven.
I took it to be so, but I do not know what I am supposed to do.
Personally I think the BCS article that we’re discussing makes the case pretty well.
LLMs are not “AI” and they are not a pathway to “AI”. They are a dead end: a clever linguistic trick that may have uses in translations and things, but nothing much more. The current hype is a pure unsupported bubble, it will collapse soon and we’ll probably have another AI winter. Which one, I lose count; the 3rd?
You got way too defensive here, as if being mandated to defend… something. Maybe the UNIX paradigm?
Others people already told you but the comment was basically using analogies that people get entangled in certain technical stacks or get invested in tech in a certain way so much that they can’t see past it and start evangelizing it… like the actual religious faith. That was all really.
I gave you the AS/400 as a counterexample. I’m not willing to write a blog post on it because 1) I am not an expert on the platform and 2) there is already a large amount out there about it.
The connection is a contrast: the relationship of many people with LLMs is different than what we often see in technology, such as Unix, because it’s not just parochial ignorance, it’s irrational faith.
The structure of the comment was: LLMs show a disturbing fact that some people in our field have religious levels of faith in things regardless of evidence. We see something that seems similar with Unix where people assume that it is inevitable, but that’s not the same, it’s just ignorance.
Correct. Thank you.
It is very interesting to me to find that while I usually can’t understand what @andyc says in his writing, he also can’t understand me.
I don’t know what to do about it, but it’s fascinating.
Again I claim that “Unix” [1] having won is fundamental, not accidental, because the design of the other ecosystems doesn’t enable enough end-user functionality to be created. You have to write too much software.
[1] Windows being essentially a superset of Unix at this point; there isn’t significant software that runs on Unix but not Windows
C is just a slightly higher level zero-cost abstraction over Von Neumann CPU and I would argue there isn’t really any other practical/good alternative abstraction to come up with. In that sense all of: C, C++, Rust, Zig build/build on the same abstraction.
I think you’re missing a lot there. Flat memory is a pretty poor abstraction these days, when even a phone is a NUMA device. That’s a C abstraction that is not present in the hardware. There’s a lot of interesting recent research (and some commercial products from the 1960s and ‘70s) on object-based addressing. These ideas are hard to adopt in C though and will probably never see mainstream deployment as a result.
Similarly, hardware is message-passing all the way down. It uses this to build cache coherency protocols, which it then uses to build a shared-memory abstraction, because that’s what the C abstract machine demands. This costs a lot of power.
On a modern CPU, the amount of area dedicated to executing instructions is surprisingly low. The biggest power draw comes from the register rename engine, which (along with the instruction scheduler and all of the speculative execution machinery) exists solely to allow a massively parallel system to pretend to be a sequential one. This is required to provide a C-like abstract machine.
So, yes, C is a thin abstraction layer over a PDP-11, which is a Von Neumann CPU that made a few choices about memory, but modern CPUs look almost nothing like Von Neumann machines. They can emulate one, but they burn vast amounts of power doing so.
Good question. Something with no shared mutable state would be easier to scale. The caches for shared immutable data and exclusive mutable data are easy to build, it’s the shared and mutable that’s difficult. Similarly, something that has high degree of parallelism is easy. If you don’t do speculative execution, you need a lot of runnable threads to keep execution units full. Some of this might be easier to do with hardware support for creating thread-like things. For example, map-like functionality might be easy to implement, where you could do something not quite SIMT, which would help with instruction-cache usage.
Unfortunately, this hasn’t had nearly as much research as it deserves because you can’t run existing code fast on such a system and co-designing languages and hardware is really expensive. The early CHERI research cost around $10m, the total has cost $250m or so. And our changes he abstract machine were very small: we intentionally made something that was easy to map a C abstract machine onto.
Doing a proper co-design project to build a good language for high-performance hardware would probably cost around a billion.
We’ve started to explore some of these ideas in Verona. The really hard thing is to build something that can run well on current hardware but an order of magnitude faster on hypothetical more efficient future hardware, to then enable the development of that hardware.
I am not a hardware guy but I know I would work hard and passionately about making OS-es and software for hardware systems like this! I always felt shared + mutable was the easy way out and then everybody started preaching because “X million people can’t be wrong”.
(late reply) The big data frameworks derived from MapReduce and Spark are all “functional programming in the large” – there are at least 10 or 20 of them, and many research systems
A cluster of PDP-11’s connected by networks doesn’t look like a PDP-11 - it’s a different machine model (in particular, it’s not a state machine)
A multi-core PDP-11 doesn’t look like a PDP-11 - it’s a different machine model
The functional properties like idempotence, commutativity, and associativity enable fault tolerance and parallelism. It has worked in practice
You can also look at research like Sequoia: Programming the Memory Hierarchy
(IIRC it is a functional language; I don’t remember the use cases being that diverse)
Importantly, this observation doesn’t make imperative programming obsolete. The low level is still imperative, but the high level is functional / declarative / graph-based.
I don’t find it relevant. C just does not abstract over memory organization, and that’s OK. On 8bit computers they had bank switching, on x86 memory segmenation, all bigger modern CPUs have virtual memory etc. You could and can still use any of these languages to write software for machines like that. In embedded you’re often responsible for page tables, TLB mgmt, coherence, flushing, etc. The CPU/asm/C-like-language only cares about memory being addressable.
Well, nowadays there’s “Provenance” in Rust (and a bit of if in C), so there’s some stuff going on there in between the levels of abstraction, but that’s about it.
Smalltalk always seemed like the great thing so ahead of its time that it couldn’t be recognized for how great it was, even though it probably wasn’t as great as it was made out to be. Still, I feel like I missed out on something special.
Years ago I got to have lunch with an old Smalltalker who held up four fingers and said something along the lines of, “FOUR KEYWORDS. Smalltalk had FOUR keywords and you could even do things like mess with the stack if you wanted.” Wikipedia claims six, but it’s still fascinating how much power they crammed into that language. I keep forgetting to make time to try out Pharo.
single system image
My understanding of Smalltalk was the image was “live”, but it always felt like Docker was a ghostly shadow of that language that we try to emulate.
Smalltalk was originally created as a bet that you can fully specify a useful language on a single piece of US letter paper.
According to Wikipedia, the six keywords are:
true, false, nil, self, super, and thisContext
I’m not sure that any of these are actually keywords. True and False are class names. These classes are used as singletons and they implement methods like ifTrue: for conditionals. Nil is also a class, which implements a fallback method that does nothing and returns self, so any message sent to Nil does nothing. You can seriously mess up a Smalltalk system by redefining some of these methods, but you can also do fun things, such as add instrumentation to ifTrue: to trace program behaviour, or do this in a subclass and return a tracing True to see what things use the result of a Boolean.
Both self and thisContext are local variable names, the first is an implicit argument the same is the name of the current context. They’re reserved identifiers, not keywords. They don’t introduce new semantics, they’re just the names that the language gives for a couple of local variables that aren’t explicitly named. As I recall, you can assign to both of them (I’ve no idea what happens if you assign to thisContext, probably explosions).
I think super might be something I’d count as a real keyword. It is kind-of an alias for self, but uses a different dispatch mechanism for message sending (start search at a specific class, rather than the dynamic type of the receiver).
True, False and UndefinedObject are singleton classes; their instances true, false and nil are special cases in the VM and the bytecode for efficiency but otherwise they could be implemented as pretty vanilla global variables. Preventing them from being assigned to could be done outside the VM, within the SystemDictionary class for example, so they’re not keywords in my opinion. The fact that they’re treated as keywords is an implementation detail of the compiler.
On the other hand you can’t assign to either self or thisContext, as those are not normal local variables. I would say that thisContext and super are definitely keywords, and self is too important to not be considered one.
On the other hand you can’t assign to either self or thisContext, as those are not normal local variables
Huh, in Objective-C, it’s quite common to assign to self, I forgot you couldn’t do that in Smalltalk. It’s something you often do with factory instances, but Smalltalk typically doesn’t separate allocation and initialisation, so I guess it’s less relevant there. The most common Objective-C idiom that assigns to self is for +alloc to return a singleton placeholder that you then call some -init-family method on. Depending on the arguments, it will then do self = [SomeConcreteSubclass alloc] and then the rest of the initialisation.
I can see treating thisContext as a special case, but to me it’s just a predefined name for the local variable that contains a reference to the current activation record. I thought some of the coroutine / green thread things worked by assigning to thisContext (and capturing the old one), but possibly they use some other primitive methods.
If you include implementation details, most Smalltalk VMs have a bunch of primitive methods (I don’t remember if yours did?) that will always be direct-dispatched to real things. Some bits of the standard library (byte arrays and so on) can’t be implemented efficiently in pure Smalltalk, so you can either provide these as classes implemented elsewhere, or provide some core parts that can be wrapped in mostly Smalltalk classes. You could regard the primitive method names as keywords, but they’re not part of the language, just implementation details.
Hmm, now I want to implement Smalltalk again. We have a tiny JavaScript engine ported to CHERIoT that uses a fairly Smalltalk-like bytecode, I bet something Blue Book-like could fit in a similar amount of code.
Edit: I just checked the Blue Book and the primitive methods are specified there, though in the implementation part.
Local variables are effectively “thisContext at: n” (with the stack above the local variables). Maybe I am tainted by actually implementing the VM but I see thisContext more as a VM “register” than as a local variables—even more so than self.
It’s been a while and I don’t remember exactly how you did coroutines in GNU Smalltalk. I think there was some kind of Continuation class that was a wrapper for the context and had a (non-Blue Book) call-cc primitive. [found: https://git.savannah.gnu.org/cgit/smalltalk.git/commit/?id=44c6e4445cbb04466 was the commit where the thisContext-based implementation of Continuation was turned into a primitive for speed. The primitive code can be written in Smalltalk but it wouldn’t assign to thisContext; rather it would assign to the parent of thisContext, which does switch to a different context but without assignments. There were no high level coroutines; only generators as in Python but without a way to send a value back to the generator]
Green threads are in the blue book and they are different; they are cooperative for CPU bound code but also preemptive. Priority lists are in a class known to the VM and preemption could be triggered either by hand (“Processor yield”) or by the VM signaling a semaphore (via the signal method on Semaphores of course, but also on a timeout expiring; GNU Smalltalk added SIGIO and single stepping). Each Process class has its own context which becomes thisContext when switching to that process.
I don’t see a way to downvote it, so I’m leaving this comment to register my strong disagreement about the LLM comments above. Most of what the comment says about LLMs strikes me as wrong, overconfident, and poorly framed.
It is an intentional feature such that people don’t content themselves with unconstructively attacking others and instead contribute to discussions meaningfully. Maybe you should try it by explaining what it is you disagree with. If you don’t care to do that it is fine, but you should then take the advice of the site guidelines and just let the person you disagree with be wrong. There is no need to register your discontent, nobody is keeping track.
It is an intentional feature such that people don’t content themselves with unconstructively attacking others and instead contribute to discussions meaningfully.
Was this the intention behind the feature? How do you know?
In any case, I will grant this sounds like a good intention. But the road to Hackers News is paved with good intentions.
Let’s flip the argument on its head. If one suggests downvoting enables people to not engage meaningfully, the same must be said for upvoting, does it not? Clicking an arrow, whether it be up or down, seems about the same w.r.t. “meaningful discussion”.
There are many other designs available out in the wild. I encourage people to be dissatisfied with the status quo here. Most forums are terrible. Lobsters is, luckily, just a little bit less terrible. But it hardly lives up to its potential. If you go over to LessWrong, you can get some ideas of what is possible, which include more nuanced voting. You can, for instance, independently answer:
“How much do you like this, overall?”
“How much do you agree with this, separate from whether you think it is a good comment?”
This distinction is key. It allows the community to register sentiments such as “This is written clearly and in good faith, but I still disagree.”
Users can flag stories and comments when there’s a serious problem that needs moderator attention; two flags are needed to add the story or comment to the moderation queue. Users must reach 50 karma to flag. To guide usage and head off distracting meta conversations (“Why was this flagged!?”, etc), flagging requires selecting from a preset list of reasons.
Was this the intention behind the feature? How do you know?
I believe it’s been discussed in some meta threads through the years.
Slashdot/Reddit style upvotes / downvotes systems have multiple issues that people have observed through the years. Specifically, a downvote is just one bit - it’s impossible to convey whether it’s because the downvotes believes the comment is incorrect, a troll, or just doesn’t like the commenter.
But the road to Hackers News is paved with good intentions.
This site is a off-shoot of HN, created to address the many issues the creator @jcs observed there. HN also has downvotes. So this is a flippant comment that I’m surprised a site member since 4 years would make.
If you want a LessWrong style multi-axis voting system, the source for this site is public, and a pull request to implement it would be welcomed.
Yes, you correctly identify many of the problems with one-bit downvote systems. My reply, in short, was saying: much of the same kind of problem applies to one-bit upvotes.
I understand if my joke wasn’t funny, but it wasn’t flippant. Designing good communities requires a lot more than intentions.
My comment was a little bit of a test balloon. I think it would take significant leadership and sustained persuasion to make it happen. I’ll think about some ways to give it a try.
Thanks for the conversation. Why do you guess I’m not humble, if you do? Or is it more about my style?
Some people conflate / confuse vigor and advocacy with a lack of humility. I wonder if you agree with this: A person can push back directly and persistently while still being humble. Asking a lot of questions and having strong opinions doesn’t imply a lack of humility.
Here are three things I know about myself (as do people who know me). First, I don’t think {community building, communication norms, interaction design} are easy. Second, I know I’m not capable of making broad changes by myself. Third, I am open to feedback.
I think a possible undercurrent is differences in communication styles, culture, and experiences. I’ve lived in many places across North America, and to a lesser extent, spent time in different countries in the world. I’ve seen variations in directness, passive aggression, confidence, overconfidence, humility, false humility.*
Another possible factor is that some people conflate intellectual pushback with a lack of humility. I’ll tell you this, If I dish it out, I better be able to take it. My goal is not to tear people down. My goal is to tear down weak reasoning and poor framings and replace them with stronger stuff. Some people can’t handle this, but I intentionally try to err on the side of optimism.
In my experience the topics of {collective action, cultural design, communication norms, consensus building} are widely misunderstood in technical circles. I voice my point of view as clearly as I can, and I welcome disagreement and clarification. Sometimes people take this in unintended ways. Some people are inclined to give me advice. All good. Communication is hard.
* I think many people can benefit by thinking more about the topics raised in The Proper Use of Humility. Have you thought about the questions raised there?
It would’ve been easier for me to notice that if you didn’t make a half dozen separate comments in response to the same single comment, the first of which being you just vaguely complaining you disagreed.
Using granular discussion, point by point, works better for complex questions. Taking it slow and methodically is better for truth seeking. The criteria isn’t what one person says it easier. A better criteria is what results in high quality discussion from many people.
On the writing side, it is easier for one person to write one rambling message. It is harder to write compact, separable points which invite detailed and substantive discussion. Think about it? Try it? Let me know what you think.
I welcome specific and focused comments to any/all. I split the comments into granular chunks intentionally. I explained my reasoning in these two comments: 12.
I registered general disagreement because I found the comment to be egregiously misguided. When I see otherwise informed and capable people led astray, I say something, because I care. I’ve been civil about it. The comment was well-written in the sense that it persuaded a lot of people, as measured by upvotes. (I couldn’t see the net upvotes-versus-downvotes because of the information architecture here.) It used a lot of rhetoric; it sounded confident. And it was quite misleading (at best) or wrong (at worse) about LLMs.
Why do I care? AI – actually AI Safety in particular – is a big deal to me, and I’m trying to do what I can to root out misunderstandings. I assure you I take this topic and its implications very seriously, possibly as seriously as anyone here.
That they are intelligent, meaning that they can think in any way at all. (They can’t, but they fake it in ways some find compellingly persuasive.)
First, using the words intelligent and think without definitions is asking for misunderstandings.
Second, the quote implies some sort of dividing line between “intelligent” and “not intelligent”. That’s misguided.
Third, like many, I find value in the definition of intelligence as the ability of an agent to accomplish some task. This makes it situated and contextual. Defining it this way helps clarify discussion.
I find value in the definition of intelligence as the ability of an agent to accomplish some task
I like your definition quite a bit because I think it captures a lot of nuance in a practical way. That said, I have a pair of potential counter-examples to consider from my day-to-day work that I’m struggling to classify as intelligent or not using that definition. For context, I work with autonomous aircraft and in a previous life worked in a biology lab that was studying insect flight characteristics.
The first example is the flight control software itself: given a series of waypoints, fly the aircraft along that trajectory as best as you can. I’m going to reveal a dirty little secret here: most of it is just nested PID loops. For quadrotors, they’re typically:
position is controlled by…
velocity is controlled by…
acceleration…
attitude…
attitude rate…
motor thrust (controls attitude rate and acceleration in the local Up direction)
This accomplishes a task and to a human observer actually looks quite intelligent. If there’s a gust of wind that knocks you off-course then the position PID loop will have an increased error term that ripples down through the stack to turn into motor commands that get it back on course. Is this intelligence? It accomplishes a task even in the face of adverse inputs. But under the hood it’s basically just a bunch of integrators trying to push the error terms to 0.
The second example is closely related: Kalman filters for processing noisy sensor inputs. Over time these are building up a statistical model of how a sensor is behaving and then making decisions on every sensor reading on how much that reading aught to be trusted. It’s an iterative model; each sensor reading gets assessed for trustworthiness and is also used for updating the internal statistical model of how the sensor is performing. It’s pretty much just a rote linear algebra equation each iteration though. Is that intelligent? It almost looks magical when it works well!
The last part… why I mentioned the insect research back in undergrad. One of the experiments I helped with was building a flight simulator for locusts (bizarre, right?). Forward velocity and turning through the simulator was accomplished by inserting electrodes into the insect’s wing muscles and measuring the timing of the muscle activations. If they activated simultaneously the insect should fly straight, if the right wing fired before the left then it would induce a turn. Once we’d established that the simulation was accurate enough that the insect could fly around in our crude world without colliding with objects (what?!?), the biologists hooked up additional electrodes to a single specific neuron called the Descending Contralateral Motion Detector, which was connected pretty much directly from the insect’s eyes to the muscle control system (I’m not a biologist…). What we observed was a series of electrical pulses from this neuron that were directly correlated with the wing EMG asymmetry: if the DCMD was reporting that a collision was imminent, the muscles would fire to turn away from the obstacle.
Is that intelligent? It enables amazing collision-free swarming behaviour. But in some ways it’s even simpler than the drone FCS… and I’m not sure I’m comfortable calling PID loops intelligent?
Yes, for certain environments, the right sensors and actuators in a PID loop will get the job done reliably. If that setup serves an agent well, I would call that intelligent behavior.
The word intelligence is a fascinating mess. For example, many people look at a world-class chess player with awe and suggest they are outliers in intelligence. By the definition above, if they win at chess given their environmental constraints, they are intelligent at chess. But from another point of view, much of what chess players do is rooted in pattern matching and tree search. When you frame it this way, it doesn’t seem that “intelligent” does it?
It seems to me you are talking about more general forms of intelligence. The degree to which an agent’s intelligence transfers to other contexts reflects its generality or adaptivity.
LLMs can indeed output language that is consistent with logical reasoning.
I figure this is the claim - LLMs can output language that is consistent with logical reasoning. LLMs can simulate reasoning better than any prior reasoning-simulation software. However, they do not actually perform reasoning; their reasoning is performative, because the output is algorithmic (and algorithmic in a way that human reasoning is not).
If a system outputs text that consistently matches logical reasoning to a high standard, isn’t the simplest explanation that it is indeed reasoning?
(I’m not asking questions of consciousness or personhood or any of that.)
Perhaps you would say these systems are not using maximally simple, purely logical, circuits / structures / mechanisms that only do logical reasoning, such as forward-chaining or RETE or whatever?
If so, how are humans different? At best, humans use their handwritten symbols to make logical proofs according to set rules. But we still use complex and often flawed lower-level mechanisms (brains) to do this logic. In this sense, I think your claims about performative reasoning are moot.
If a system outputs text that consistently matches logical reasoning to a high standard, isn’t the simplest explanation that it is indeed reasoning?
no, because the Chinese Room isn’t reasoning either.
LLMs are collections of things that humans have said before, boiled down into statistics. The sole job of an LLM is to output words based on statistical locational frequency, that is not, and cannot be, reasoning. It’s incredibly provable that LLMs cannot actually reason, too, and there are multiple papers on it — see https://arxiv.org/abs/2406.02061 and also https://link.springer.com/article/10.1007/s10676-024-09775-5
I think the comment above misrepresents or misunderstands the Chinese Room argument. Per Wikipedia:
The Chinese room argument holds that a computer executing a program cannot have a mind, understanding, or consciousness
Searle’s point about the Chinese Room is not about if a machine can turn the crank and do modus ponens or other reasoning rules. (This has been settled for a long time.)
The C.R. highlights the question of qualia, of inner experience. I’m not talking about consciousness, “thinking” or a “mind”. I’m talking about something measurable: can an AI generate correct conclusions according to the rules of logic? (For example, they do LSAT level reasoning quite well, and they are getting better.)
Ah. Your claim boils down to the fact that the low level mechanism of token generation is not identical to reasoning. Am I representing your point of view fairly?
There was also a time when I fixated on this definition. I get it.
I hope you realize that when some people say LLMs “can reason”, they aren’t talking about the mechanics of token generation process. They are talking about higher level behavior.
I think it is also important to realize that by your definition, one could argue that humans don’t reason either. We’re just neural impulses firing. I hope you take my point.
LLM bots cannot analyse and improve themselves, because they cannot analyse – anything ever – and they cannot improve on what was in their training corpus.
I can see why you’d think that LLM bots cannot analyse and improve themselves. What really confuses me is that you say “nothing can”.
Are you saying that humans cannot analyse and improve themselves? Because I think plenty of humans do that after, for example, failing at something many times in a row, or reading a self-help book.
Or by “nothing”, do you mean only things that aren’t alive? If you do, consider the hypothetical case that someone wrote a (non-LLM-based) program that perfectly simulated a human’s behavior (human-like words, leg movements, etc.). The program was even observed to change its outputs after interacting with a simulation of a self-help book. If you agreed that humans can analyse and improve themselves and you had this definition of “nothing”, I think you would have to be making one of these three assertions:
That program would not be analysing and improving itself, despite acting exactly like a human, which does those things.
Creating such a program is impossible – computers can never match human capabilities.
Such a program would be alive, and thus not part of your definition of “nothing”.
LLMs are, by definition, vast databases of multi-dimensional weights of relationships between symbols.
An LLM can’t analyse an LLM, including itself, and nothing else can analyse an LLM either.
Claims of redesigning LLM bots to hallucinate less, for instance, are lies, because the databases are not amenable to study or analysis. And, of course, hallucination is what they do. There is no distinction between making good and desirable stuff up and making bad stuff up.
They are lies, just as software companies claiming they have rewritten huge codebases for security. In most cases, they’ve just done automated scans, maybe a few people looked at a few especially critical routines.
Real rewrites are very rare and usually disastrous, as Joel Spolsky observed about 15 or 20 years ago.
My comment was about your claim that “nothing can” “analyse and improve themselves”, a claim I disagree with. What you say about LLM analysis may be true, but that’s irrelevant, because I was not arguing that LLMs are an example of a thing that can analyse and improve themselves.
With my comment, I was hoping to pinpoint where, if anywhere, we disagree about the statement “nothing can analyse and improve themselves”. You can do that by answering these questions:
Do you believe that humans can analyse and improve themselves?
(I believe that humans can.)
If so, then do you still believe that nothing can analyse and improve themselves?
(I would interpret my above belief as disproving this, but maybe you’re using a different definition of “nothing” from me.)
If so, then which assertions are you making out of the three I listed in my previous comment?
I just thought of a possible explanation for your response being weirdly focused on LLMs instead of on the questions I had posed. Looking at your original wording again:
That they can analyse and improve themselves. (Nothing can.)
Did you intend that last sentence to be interpreted like this?
If that statement does not capture your meaning, then it is unfortunate that your replies to both insanitybit and rele’s comments, which contained that rephrasing, did not correct the misunderstanding.
Regarding exaggerations and lies about what LLMs are capable of in the near-term… Yes. There is hucksterism in AI. I know the arguments; I’ve read Kapoor and Narayanan, which I generally recommend. I’ve also read Gary Marcus, who makes some good points but gets too ranty for my tastes.
However, regarding the comment here above, there are claims here about what is and is not possible, such as:
An LLM can’t analyse an LLM, including itself, and nothing else can analyse an LLM either.
If we define analysis as “detailed examination of the elements or structure of something”, then, yes, an LLM can analyze its own weights and structure.
If you say “but LLM X can’t analyze itself”, that is a different claim – a claim about implementation not about what is possible.
If you say “but it can’t analyze itself up to some quality standard”, that is a different claim, and it invites discussion about the bar and what progress is being made towards it. This kind of discussion might be interesting; it invites us here to dig into the details.
nothing else can analyse an LLM either
This is a tall claim. To me, it is so bold that I wonder if it is designed to attract attention. I’ll bite: I ask the author to (a) clarify their terms and either (b) prove this claim or (c) make a testable prediction, on the record.
That bigger models will make them better at what they do. (This one just exposes basic ignorance about very simple statistics, such as scaling effects.)
What do you mean by “what they do”?
What papers on LLM scaling laws have you read? I suggest we ground our conversation there. The papers show improvement across many metrics. Bigger models perform better in almost all ways*.
If you say “but Improvement X isn’t what matters”, then you are making a different argument.
If you say “but the improvement isn’t worth the cost”, then you are making a different argument.
If you say “but most people can’t notice the improvement”, then you are making a different argument.
Do you want to refine your claim?
* Bigger models can be more deceptive, which is unfortunate if you care about alignment (which you should). But increasing deceptive ability is most certainly an improvement in capability.
I am not interested in radical new methods for estimating with unprecedented accuracy the numbers of angels dancing on entire fabric-fastener manufacturing units. I don’t care, because there are not such things as angels.
I find it highly amusing that saying “this type of software is not intelligent” provokes responses which attempt to nail down a possibly negotiable definition of intelligence. Again: this is not a rebuttal, in my view.
You try to split hairs, perhaps so that you can cut each one, but I’m not here for a haircut. I am not here to persuade anyone. I don’t see any potential gain in even trying.
LLMs are a fascinating technology which may in time enable applications such as simultaneous translators, perhaps not only between human languages but between computer languages as well, which could have amazing consequences. I speculated on some of those here but I remain extremely sceptical about the entire current industry, as I said in the earlier article that the above is a follow-on to.
But they are not artificial by any reasonably strict definition, they are not intelligent by any definition that isn’t a fantasy, and because they are not in any way what they are depicted as being, they will never lead to anything that is.
It is empty hype, nothing more, and I sincerely hope it collapses soon and that it destroys the careers of the hucksters selling these lies about thinking computers.
Here’s what I notice. First, you are obviously bothered by hucksterism in AI. (So am I, by the way.)
Second, this is a bit of a guess, but you seem to take my pushback personally. And then you channel a lot of negativity towards me. I’m not the problem. The key problem is these topics are complicated, communication is hard, and we all have egos. Very few people want to be wrong, much less shown to be wrong.
Third, you write at length, but haven’t answered the specific questions I ask. You sometimes wave them off. Sometimes you ignore them altogether. You also use a lot of rhetorical devices.
On the substance: Are you claiming LLMs are not getting better as they get bigger? If so, by what metric? (I ask for it to be clearly defined and measurable.)
Please respond to what I’m writing, not about what some other person says.
Are you claiming LLMs are not getting better as they get bigger?
Again with the demands for metrics. No, I cannot give you metrics, because as I have said already, I do not care. You are claiming the numbers of ratio of dancing angels to pins is important: I’m saying there are no angels. No angels means no counting means no numbers.
Yes, they have, so far, provider better results – more plausible-looking textual or image based output – with greater size; however, it is not just foolish but downright stupid to blindly assume this will continue indefinitely forever.
Secondly, the bigger the model, the bigger the power and storage use of this technology which is already environmentally catastrophic. The boosters who keep calling for bigger models are guilty of ecologically criminal acts. You are entirely as culpable as the cryptocurrency promoters.
Please respond to what I’m writing, not about what some other person says.
… it is not just foolish but downright stupid to blindly assume this will continue indefinitely forever.
I have not said anything about such improvements continuing forever (neither in this thread or sibling threads, nor anywhere I can recall.) I strive to not blindly assume anything.
You are entirely as culpable as the cryptocurrency promoters.
Please stop attacking me. Such attacks are not welcome here.
And, to be clear, I have not called for bigger models. (And if I had, would such attacks be justified and appropriate here? It is hard to listen and learn when you are attacking and accusing.)
Again with the demands for metrics.
You may perceive them as demands, but this does not make them demands. I have asked questions. Repeating a question doesn’t make it a demand. We have both attempted to redirect the conversation, such is the nature of conversation. You call my attempted redirection a “demand”. To my eye, this is probably either (a) the result of you being upset for other reasons and/or (b) a rhetorical device to make me seem unreasonable.
You are claiming the numbers of ratio of dancing angels to pins is important
This is an uncharitable description of what I’m asking.
When I said “rhetoric” before, this is the kind of thing to which I was referring. It raises the temperature but doesn’t promote mutual understanding.
Overall, this conversation has been unnecessarily fraught and combative. I have strived to disagree with you without being disagreeable. I’m sorry if you think my point-by-point commentary is mean or in bad faith. I do not intend any of these things, and I’m sorry if I’ve offended you somehow.
It would seem you have labeled me in some negative way, such as being an AI booster or ecologically irresponsible. I don’t see the logic in why you think these labels apply to me. And again, even if they did, being unkind about it isn’t necessary or welcome.
Whatever you feel, I would ask you to stop taking it out on me. It seems you are making assumptions about my point of view or my impact on the world. I have no doubt that your point of view is valuable, but I would ask that you be more civil about it.
I am not sure what you mean. Based on your comment, you sound angry about certain mannerisms and/or beliefs? Such as?
If you want to offer some specifics and are willing to discuss charitably rather than dismissively, I’ll put in some additional effort. No need to be mean about it, please.
I think it’s fair to say the context here is about artificial intelligence and some common misunderstandings. Unless I’ve lost track of the parent comment, I think this all started when one person made a list of common UNIX misconceptions and then offered various AI misconceptions. My comments, taken together, point out how many of these claimed AI misconceptions are actually misconceptions themselves.
Many people were far from calm. The conversation has the usual signs of people overreacting to each other. Even relatively neutral people got attacked, because one side or the other accused them of being on the wrong side. It has been a case study of polarization and poor community discussion norms. Truly pathetic, as in I have pity for how we all got here.
These discussions need to get clear and as specific as possible if we want to make sense of the world. I hope you can appreciate this.
Are clarity and precision and rigor tedious to you? If true, this discussion might not be worth your time? Truth seeking isn’t “sexy”, nor does it provide the dopamine hit of leaving a nasty comment. It doesn’t make it easy to form a tribe of “us versus them” because anything is subject to scrutiny and revision. These are the values of critical thinking and intellectual honesty.
I flagged this as unkind, it’s probably trolling too? I don’t know. I figured I’d let you know. Calling “effective altruism” a cult is definitely hyperbolic and clearly intended to be insulting.
I know that some online communities dislike some members or something like that but it’s not a cult, it’s just an ethical framework, and the user doesn’t even seem to be advocating for it. I don’t know what mannerisms you’re even referring to, personally, but this seems like a silly and overly negative interpretation of a post that’s basically “please stop insulting me and engage with my points, I am willing to engage with yours”.
your explanation has been given a consideration it deserves, and found wanting. specifically, the following statements are wrong: “calling ‘effective altruism’ a cult is definitely hyperbolic” (it isn’t); “it’s not a cult” (effective altruism movement, especially its californian variety, has many characteristics of a cult), “it’s just an ethical framework” (no, it’s not just an ethical framework).
if you don’t know “what mannerisms i’m even referring to”, you have no basis to claim that “it seems like a silly and overly negative interpretation of a post” (and my reply wasn’t referring to a single post).
to spare your precious time, i will have tagged my reply as unkind myself.
Oh that is an AMAZING research question. Thanks so much for linking this, there’s no way I’d have seen it otherwise and it’s highly relevant to my interests.
You are welcome. It is a very interesting topic. I love Prolog, but I think the execution model is a bit problematic sometimes. I think alternative techniques such as answer-set programming or constraint logic programming don’t get sufficient exposure.
The author of this article has a reading comprehension problem. The linked story about .yu domain does not remotely corroborate his summary. As someone who studied at University of Ljubljana at that time right across the street and never heard anything so wild, which I’m sure I would, I have difficulty believing that it happened as described.
Which parts are you saying it doesn’t corroborate? There are a lot of assertions of fact in it, and also a bunch of subjective stuff.. from a quick read of both pieces, it does look to me like the most important details of the alleged heist are mentioned in the cited source, but I may be failing to understand the significance of something that seems minor enough that it doesn’t occur to me to doubt it, or something like that…?
It would be insanely crazy for something like that to happen during ongoing Balkan war and also completely unnecessary since .yu domain was administered from Ljubljana since start in 1989. The only “heist” was some colleagues allegedly breaking into her office, copying software and disconnecting her network. All of them worked in the same building and no computers were moved.
On a Sunday in July 1992, Jerman-Blažič told me that ARNES, which included some of her former colleagues, broke into her lab, copied the domain software and data from the server, and cut off the line that connected it to the network. “Both ARNES directors had no knowledge of internet networking and did not know how to run the domain server,” she said. Though they only used the network for email, ARNES secretly kept .yu running for the next two years, ignoring requests from a rival group of scientists in Serbia who needed the domain for their work.
I agree that this doesn’t say anything as to whether the former colleagues thought of what they did as a “heist”, and it quotes Jerman-Blažič’s opinion that the people who took the software didn’t know how to use it without help, but she wouldn’t have been in a position to see what happened afterwards, since, as the piece also says, the next part was secret. However, the last sentence is clear-cut that the domain was in fact run by ARNES, at least according to this publication. That’s the substance of the “heist” claim, surely?
I’m unable to find anything in the Dial piece that speaks to whether these people worked in the same building at the time of the alleged theft, as you suggest. I may not be looking closely enough. You’re right, that does bear on the claims in the Every piece, which says the alleged thieves traveled there in connection with the alleged theft.
Your point that no computers were moved doesn’t seem relevant to me; the Every piece says “On arrival, they broke into the university and stole all the hosting software and domain records for the .yu top-level domain—everything they needed to seize control”, which notably makes no claim about hardware, only software and data. In general, in any sort of computer-related theft, real or fictional, the software is the important payload… I don’t really see what difference the hardware would make to anyone.
I think these authors are describing the same factual claims, while differing substantially on who they view as the “protagonist”. That may account for why it feels like they’re talking about something different? I do think it would be entirely reasonable to point out that there’s very little information about how the participants really felt about the events at the time, or how they understood the purpose of their actions.
I do think the author of the Every piece goes perhaps a little too far in inferring people’s intentions, and leaves out the important caveat that the copy of the domain that ARNES ran was kept secret. I think perhaps the author doesn’t have a deep understanding of DNS, and incorrectly believes that running one copy of a domain somewhere, privately, implies that nobody else could be running it at the same time. That does seem like a material error, and quite unfortunate, though it’s not about the so-called heist per se.
So… I think I see why you object, although I’d stop short of saying the story was invented from whole cloth. Does that help?
I don’t have an issue with The Dial piece. I definitely made a mistake about servers; no idea why. I guess reading comprehension issues on my part as well.
My comment was exclusively about the Every piece that has a truly awful summary of The Dial piece. I disagree that they are the same factual claims because to me there’s a huge difference between Slovenian academics going to a different country to steal domain and software or essentially an internal dispute around who will continue running the .yu domain, which since its beginning until mid 90s has always been administered from Ljubljana in current Slovenia.
That makes sense, now that I understand the distinction you’re making. It does seem like an important one. Thanks for talking it through and helping me understand.
This is surprising to me because I have arrived at almost the opposite conclusion, being a developer for almost 15 years.
My years spent programming, although satisfying for reasons depicted in the article, were largely bereft of varied human interaction, or contemplation on deeper topics that may touch one’s inner life - things that might be influenced by the arts, philosophy, literature, etc.
I was a student of the humanities (both academically and recreationally) before embarking on my software development career more than a decade ago. In recent years, I have looked back and noticed a stark contrast in the richness of inner life and flexibility in socializing after a decade spent solving problems with code on engineering teams. Some if not most of these teams were excellent, and I look back on my experiences fondly. However today, I do feel somewhat robbed of my time.
there are definitely years that I have regrets about, because of how isolated I was
today I actively try to focus on projects that I will be able to talk to my friends about, and also I’ve cultivated a social group who enjoy discussing each others’ projects
I find software much more enjoyable when it’s a social undertaking. that isn’t automatic, and the article doesn’t really address it. sometimes software is social, sometimes it isn’t.
Maybe it’s just not what you love anymore, or even ever.
Human interaction is often great (when dealing with right kind of people), but programming and SWE is still the thing I love doing, and will never feel like time wasted. And quite the opposite - it allowed me to meet with awesome people, gave me a great life and introduced me to all sorts of adjacent areas of life and study.
From my perspective software is a deeply human thing and programming languages are languages (each with their own sociocultural background as well). I feel robbed of my time with the countless hours spent in meetings or “agile cerimonies”, doing things because of company politics, etc. But programming in itself feels like a very fulfilling and relaxing activity where I can really express myself. Sure not all software is good or interesting, but the same can be said for some human interactions. Moreover I’ve been doing pair programming lately and although I was hesitant at the beginning I’ve started to appreciate it, it really turns software development into a collaborative art
I feel this strongly also. But at the same time, the right people can bring the richness into the technical space. It just requires a subculture that’s fascinated and interested in mixing that richness and contemplation with the technical work. Once I encountered that kind of culture (that’s interested in melding the humanities and the technical), I couldn’t go back to purely technical spaces.
bereft of varied human interaction, or contemplation on deeper topics that may touch one’s inner life
I’m a bit shocked. For me, software is deepest and most human field, because it expands your scope of responsibility. Where 2 centuries ago, a letter would have to be sent by a series of horsemen (or at best, by signalling stations), we’ve abstracted that away. So to many hours of other tasks, which free us to think about what we’re doing in a more profound way and create real solutions. We can identity uniquely human faculties and automate away other parts, creating complex industrial processes to service many customers and improve their lives. Sure, many people and organizations don’t, but we are free to! If you have a problem, you can often modify your workflow or create tools to overcome it. Symbolic and logical/constraint programing reduce a lot of stress planning. Then you can track friends’ birthdays, stories, schedule fun and work tasks and so much more. (If someone says it’s insincere to send an automated birthday message, why do they choose to ignore the love and care for your loved ones which made you craft the thing?)
I feel I’m in a similar situation to you — just shy of 15 years’ professional software development; studied (human) language before embarking on that — worked on some really neat teams, but I spent a lot of my time outside work developing those other areas of life. (Transitioning will mostly force that!) These days I have the flexibility to work part-time, so I do, and it means I have the mental space in my life to actually pursue a hobby. It turns out that’s programming, with a side-order of more language study.
The ikea effect hits hard. I usually feel more at home In a objectively worse product I wrote myself than a more polished one. Knowing exactly how it works is just an awesome feeling. But throwing away bad code after having learned is sometimes the right move as well.
That’s the main downside. Sure, you learn a lot if you build it yourself. But it is very hard to compromise once a mental model has been formed that something “needs” to be done a certain way.
Of course that does not affect all the “build it to learn about it” projects, but it’s probably a also a source of the “Not Invented Here” approach that some folks apply.
“Programming is time well-spent and joyful (when you’re not dealing with bugs)”
So less than 10% of the time spent programming is well-spent and joyful? Or is it just me and everyone else is spending most of their time producing and using working code?
(1) Fixing bugs in my own code is definitely time well spent – it’s a lesson for how to improve. How you can improve without reflecting on your mistakes? Bugs are mistakes.
(2) What about bugs in gaps between “your code” and “other code”? e.g. invoking a library the wrong way, because you didn’t understand what it was supposed to do
I’d also treat this as an opportunity to learn about your dependencies / specify their behavior
I frequently write tests for my depenendencies, to try to avoid this kind of bug
(3) Fixing bugs in “other people’s code” often feels the worst, and takes the most time – definitely as a less experienced programmer
Well I’d say, if the bug can be reproduced in an automated way, and you have a working build system … that’s usually 90% of the battle
I wouldn’t necessarily call this time not well spent, but depending on the situation it may not be joyful. But it could be joyful – if the code is otherwise good, and you learn something. (This is often not the case “at work” – in open source there is more code written to a higher standard, although lots of bad code too)
And actually I wouldn’t say I spent 90% of the time dealing with bugs … I think I spent more time testing my own code, and testing my dependencies, so that I don’t have to do that … I do find testing/specification more enjoyable than fixing bugs in hastily written code
(3) Fixing bugs in “other people’s code” often feels the worst, and takes the most time – definitely as a less experienced programmer
Not really a counter-point: I generally enjoy finding and fixing other peoples bugs. Far more than my own. I also know quite a few people who feel the same.
I also like working on old code-bases more than new ones.
Yeah, if I can swoop in and find their bug myself (which is rare), I feel super smart and amazing. What sucks is when other people’s code is buggy and I can’t figure it out, which is quite frequent.
This is definitely a problem with a lot of programming discourse. Instead of saying “X is hard for Y reasons” we tend to hear a lot of “don’t build X yourself”. The argument is often brought up to prevent people from making mistakes that established solutions had to learn the hard way.
While I very much agree with being aware of the risks associated with things, but I’ve noticed that people are liable to make basic mistakes using established higher level abstractions since they don’t understand how the abstraction works fundamentally.
A typical example is authn/authz. While it’s easy to throw in some middleware to “handle” it, without understanding the concepts behind it it’s often easy to introduce trivial insecurities.
I would go even further to say that sometimes it’s worth your time to build a thing yourself for no other reason than to think through the problem domain, and understand the solution concretely, then throw it away and use a production-ready implementation.
Although not “mistakes,” per se, I feel like people do write about footguns and gotchas quite a bit. Whenever I’m learning a new technology/language/etc, I spend some early time searching around for gotchas and reading through things to look out for. I recommend it.
the short answer to this is no. (edit to add: that is, no, there are no automated tools for it)
to reliably detect LLM output would require having more training data for the detector than the company that made the LLM had. it’s more profitable to sell LLMs than LLM detectors.
the fact that Google is drowning in spam search results (in my subjective opinion based solely on public information) should be a strong indicator that nobody on the defense side has a clear advantage.
I’m not entirely convinced that “LLM detectors” based on existing techniques could be made reliable even with vast quantities of training data. If you had one, you could use it to adversarially train an LLM to smooth out whatever statistical/syntactic wrinkles the detector was keying upon. If anything could identify the semantic flaws in LLM output, you’d have something resembling the mythical “strong AI” that would make LLMs obsolete, and nobody credibly knows how to do that (not for lack of snake oil sales pitches for it, of course!)
The situation is also muddled by the fact that SEO sites can directly make money for Google, so they have perverse incentives at play across the organization to do a poor job filtering out spam, even if they had the technology. They don’t even bother to give users tools to filter spam domains out of their personal results!
In places valued for the quality of their non-bullshit, at least ones which are run and frequented by people who care about that, bullshit should be curated out.
In a few years, places like this which actively check if submissions are sufficiently coherent to be authored by humans will be even juicer targets for LLM ingestion operations. That is if there are any submissions left that are authored by humans.
very nice write-up. I have a Framework that’s a year older than this author’s, and have had many of the same experiences. I did try the heavy-hinge update, and I like it; I’ve also replaced a few other parts for minor quality-of-life and cosmetic reasons.
I did at one point over-tighten a screw, and break off a bit of plastic that held the threaded insert it was supposed to mate with. oops. I was happy to be able to easily buy a first-party replacement, but it was pricier than I’d hoped since that piece is sold only as part of a larger assembly. don’t over-tighten the screws.
deeply tragic. I think everyone who was paying attention to linguistics research or to corpora saw this day coming, but that doesn’t make it any less sad now that it has.
But isn’t great that there’s a public, open, non-owned application platform without any vendors acting as gatekeepers?
…like most of the readily available free/open-source operating systems?
Yeah, they’re (regrettably) not as ubiquitously deployed, but all else being equal I’ll take a native application over webtech (whether in a browser or a warmed-over repackaged one) 100.0% of the time.
I am glad that the web exists and that it is useful for the many interactions I have with governments and capitalism. I think we are better off with the web than we were before the web. I enjoy building stuff for the web, up to a point.
I also think that any platform which is used for commercial purposes will, over time, be developed in directions which enable and encourage site owners to consume as much as possible of their visitors’ computational resources.
The unfortunate truth is that if, for example, a social media site gets even 1% better subscriber retention by consuming those gigabytes, it’s worth it to them, since the cost in the terms the company cares about is much smaller than that.
This is frustrating to me. I’d like to simply advocate that corporations be encouraged to develop free software against, say, some minimal common-denominator set of Linux functionality (I don’t regard containers as a solution to the bloat problem). However, that would result in Linux taking on the same characteristics that I dislike about the web.
I don’t yet know what to do about this. I hope that stating the problem as I see it will inspire somebody, someday down the line…
oh excellent, I hadn’t heard of this and very much want to try it. Thunderbird works well for me and solves my actual needs, but I have been wanting something terminal-based for a few years. alas, alpine doesn’t have an option to save oauth credentials across boots, and elm’s descendants are no longer maintained…
huh. what a really cool write-up. I’m convinced. now if only web1913 were packaged for nixos’s dictd setup…. wiktionary is, and seems to have built on it, but I’d like to have both. maybe a nice weekend project…
From a consequentialist point of view, one person’s differential impact on the factors listed in the article is statistically minimal (energy, job replacement, etc). You can read this in many ways, some at odds with each other, such as: (1) don’t worry about LLM usage, focus on higher impact areas instead: use them to research and help your favorite causes for example; (2) the diffusion of responsibility can undermine many sensible notions of ethics; (3) “traditional” consequentialism is an insufficient ethical compass; lots more.
right, I mean, I am far more sympathetic to consequentialism than to deontology, but decisions that have aggregated effects are just something it’s bad at. my personal belief is that with questions like this, we have to ask ourselves what would happen if everybody followed the same reasoning we do and reached the same decision……… because that’s pretty much exactly what happens.
I am sympathetic to the ethical argument of acting as if your actions formed a guide for everyone to follow. This has merit as one criteria, but not as a unitary guiding principle for at least two reasons. First, it is complex to figure out what conditions are getting baked in.
To explain — the ethics of lying can be illustrative. Lying is not categorically wrong according to most sensible moralities (or at least not the worst option in all cases). For example, lying to a visibly violent person to protect an innocent person is generally viewed as the better choice.
Second, there is another problem of the “what if everyone acted this way?” analysis: done too an extreme it can ignore probabilistic assessments and repeated-game interactions. For example, we don’t want everyone to go out and burn a lot of resources on a new startup idea, but it is beneficial to society that some tiny fraction of people do, largely because they are more risk averse and the upside is large.
Both the first and second points emphasize the contextual factors, which make the practice of discussing “general ethical behavior” very difficult. It raises questions of consequences, intent, and what is known.
I respect the way that careful AI researchers and statisticians are digging into ethical questions … they are making them rigorous and applied in ways that philosophy alone cannot.
As an example, how unethical is it to be ignorant and/or clueless? Ignorance can undermine even the best of intentions. With simulations, we can play out what it means and get a better understanding of our ethical intuitions.
But in any case, stepping outside ourselves and giving some emotional distance is usually a good starting point to thinking about morality. Our egos are so pervasive that they tend to taint even our perceptions!
I only just noticed this (oops) - thanks for it. Good thoughts, which I do appreciate. Given the length of time that’s passed, I’ll leave it at that.
No harm in continuing a good conversation. :) Having just reread your comment, I personally think a nice response to the question of “what if everyone behaved this way?” is to ask “what is the probability of that happening?”.
There are indeed dynamics where people are largely reasoning in the same way. These are relatively easier to reason about.
It takes a lot more “theory of mind” to develop meta-ethical systems that work well when agents have differing ethical systems. (This is why I highly recommend the philosopher Robert Kane; he has framework for thinking about such situations called the moral sphere.)
Best intro ever.
Kind of surprised by this response. Signal already generates link previews through a proxy specifically to avoid this kind of thing.
However, I did a quick test using two Signal accounts and it appears that images from unknown numbers are not auto-downloaded; you have to accept the message request first. This heavily mitigates the issue and honestly I understand Signal’s stance given the context.
Additionally if you’re concerned about this you can disable media auto-download, even from contacts, in Settings -> Data and storage (though that doesn’t help for progile pictures, if it turns out the caching service used for those is also vulnerable).
Anyway, great writeup, fun to read.
No matter what the weakest point of Signal is its cloud infrastructure and usage of phone numbers. If you’re worried about these kinds of attacks it’s in your best interest not to use Signal or any other chat provider with a complex cloud deployment.
Okay, so what is it? What are you using instead? Manually writing ciphertext via physical mail yourself? Afterall, your computer used “a complex cloud deployment” to get to the state it’s in, so it cannot be trusted.
You don’t need a phone number to sign up any longer.
You don’t need to share your phone number with other people anymore.
Phone number still needed to sign up.
Ah, then I was misinformed. It’s a bummer…
I just downloaded the app and couldn’t get past the phone number screen. So this appears to be untrue.
ah, thanks for your testing
yeah, this should not be understood as a Signal-specific attack. geo location is a topic where privacy defenses are just woefully inadequate across the board. as a privacy person I can attest that using a VPN won’t necessarily help, though it will help against this specific type of attack.
If I am understanding their article correctly, they are mostly talking about downloading assets from the original source through the anonymizing proxy system they have setup, not their own content which is the core issue here. I wonder if their fix for this would be to just use that same proxy to access their own cdn’ed content as well.
Not only have they never attempted it, they also seem to have no interest in doing so in the future either. I offered to start work on integrating tor routing into the app for text and data flows. Never heard anything back:
https://community.signalusers.org/t/use-an-anonymizing-overlay-network/62670/1
Yeah I was also thinking about their past blogs mentioning the proxy-ing. They also do the same thing for GIFs (which is mentioned in the article you linked but worth also pointing out here for visibility).
Signal’s security FAQ shows a profile picture in the message request screen. Is that not the case for you? Is it shown in the conversation list screen?
IIRC it was shown before, maybe the docs are outdated.
Just tried it again but using a custom profile picture on the other account, and it appears not to download the profile picture until you tap on it. But I’m now realizing that the placeholders for the supposedly-not-downloaded images and profile picture are blurred versions of the originals. So if the blurred version is also cached, that would mean that this in indeed a zero-click (or one-click if the caching for profile pictures isn’t vulnerable) on Signal. I might check it out later.
Often the blurred version in apps like this are constructed using techniques like BlurHash, where a tiny blurred version is delivered inline as part of the backend response to provide an immediate fallback. This issue for Signal Desktop seems to imply this is how this is done. In that case there’s no publicly cached response which can be used to side channel attack in the way described in the article.
I love how he wrote only that first paragraph in all lowercase.
huh. well, I don’t agree with it, but it’s a well-structured argument.
I just… don’t find any of these proposed downsides to be particularly arduous, and when I look at the constant inbound traffic that’s always trying to brute-force something or other, I see no reason to take my chances. the article does a pretty good job of enumerating many of the ways I could be inadvertently exposed; I just disagree that those things aren’t a problem, in part because I don’t trust myself to be hypervigilant about every single one of them at all times.
I’ve had password-based ssh login turned off entirely on all my machines for something like twenty years now, for both internet and LAN traffic. it’s fine. you learn pretty quickly to not lose your ssh key - which you should be making sure to learn no matter what, because if you leave a copy of the file somewhere and forget it’s there, you have unexpected attack surface.
The general consensus in my professional network, even reaching up to EU SME’s (!), is to just geoblock the UK given the costs to reach compliance are usually much higher than any business brought in by possible UK customers. This is also in light of other impeding UK legislation.
I would also do the same here in this case, if I were you @pushcx. This is a community of tech-literate people who know how to get a VPN. At the same time, the (slight) inconvenience is a constant reminder of this overbearing legislation for all UK visitors, and it might have a stronger effect on the situation than we could imagine.
It shocks me time and time again how much of a postmodern cyberpunk surveillance- and police-state the UK has become. The UK’s people seems to be content with this development given they voted for it, so who I am to judge? Fortunately I don’t have to live there.
We voted out the government that passed this stupid law. Unfortunately, the new government has not repealed it.
Just out of curiosity, why haven’t they?
Because they actually believe it is a good thing, and would have passed such an act themselves if it didn’t already exist.
It is very easy to repeal an Act which has not yet come in to force. The coalition gov quickly repealed the ID card legislation despite it already being in force, with its active database, and cards already issued.
So by their deeds (and/or inaction’s) you can determine their beliefs.
I suspect it’s more that they simply don’t understand it. My MP is one of them and he is, uh, not the brightest. I’ve never had an interaction with him where I felt he understood the legislation that he was voting on at more than a superficial level. His level of reasoning is probably ‘child abuse bad, law says child abuse bad, law must be good’ rather than having any actual understanding of the impact of the law.
Not sure. There’s a fair overlap between Labour today and Labour 17 years ago, and you may recall the Terrorism Act 2006, which had similarly vague formulations about what exactly constituted “glorification of terrorism” (which in turn provoked a bunch of SF authors to release a book called “Glorifying Terrorism” in order to see what would happen). It’s not unlikely they view having a law that can be inconsistently applied (“oh but we don’t mean you, you’re one of the good ones”, “we know it when we see it”) as an actual feature - after all, they don’t bear the costs and neither does anyone they care about.
To be fair, irrespective of the belief / intelligence of individual MPs, most are backbenchers with very little power within the party and just follow the whip (especially considering how many are new MPs)
In this particular case they probably agree with it but also it’s a general fact (I haven’t actually checked this - it’s just from my experience - so somebody please correct me if I’m wrong - https://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham's_Law%22 !) that governments repeal almost nothing. They almost universally prefer to spend their limited energy on passing new laws. I’m not sure why that is - maybe something to do with Pareto optimality.
the primary goal of most politicians is to get re-elected. due to sample bias, this is especially true with ones who’ve been around longer and thus have the seniority to influence agendas at the party level.
passing a new law is something that it’s easy to brag about and take credit for. it presents the public with an achievement that one or more involved politicians can attach their names to, or over-state their role in if they’re so inclined.
repealing a new law is not nearly as easy to convince the public to congratulate people on, since the net result is that there’s nothing to look at, no new results to point to. it’s certainly the case that there are parties that build their brand on repealing laws as their primary activity, but there is an inherent disadvantage to doing so.
Indeed: it is most probably on the new government’s radar, but they might choose to allow shit to hit the fan before “saving the day”.
We can draw a parallel with hero culture in many dysfunctional companies: there’s no glory to gain by avoiding an incident while the employee who solves an incident get credit, even if they caused it in the first place.
oh good parallel, indeed
(edit: typo)
You linked
https://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham's_Law%22
which talks about how Ward Cunningham, a computer programmer, is credited with the idea that
Is that actually what you meant to link? If so, I’m not quite seeing the relationship to what governments do or don’t repeal.
Parenthesis: it applies to the statement before the link. Let me highlight it:
So it applies to the fact they didn’t check, not to what governments do.
The intentional invocation of Cunningham’s law went right over my head. Shame on me for reading before finishing the first coffee.
I wonder if intentional invocation of Cunningham’s law is like intentional invocation of Murphy’s law… such an act might well turn the incorrect statement into a question and therefore place it on the wrong side of the law.
Haha, no problem, happy to be of help. It took me a close reading as well.
I believe they are saying “I’m saying this and if I’m incorrect someone will surely correct me”, that is - utilizing Cunningham’s law themselves
You are, of course, correct. That went right over my head.
The media ignores it so it’s hard to blame the general populace.
I probably wouldn’t bother.
I’m now a bit unhappy that I did actually read the documents last time this came up, but didn’t write down my off-the-cuffs assessment. For context, I’ve done some GDPR-Compliance related stuff before.
The first thing to look for in those documents is the thresholds and IIRC, lobste.rs is below all thresholds. Below those, a lot of things become relatively tame or already exist. E.g. you need to have a content policy (lobste.rs has one) and you need to present that you can remove illegal content on need. All of those exist.
It’s similar to the GDPR: a lot of the things look daunting until you figure out that you’re actually out of scope.
All that being said, no one is helped with 5 40 page PDFs for running an international service, so I would totally understanding if you just blocked the UK.
I did a quick review as a refresher. This is the most important document for you: https://www.ofcom.org.uk/siteassets/resources/documents/online-safety/information-for-industry/illegal-harms/illegal-content-codes-of-practice-for-user-to-user-services.pdf?v=387711
Note that “recommended” does not mean all of them need to be implemented. However, at a quick glance, lobste.rs is neither a large service, nor a multi-risk service (none of the content deemed a risk is discussed here), so a lot of the heavy-hitters do not apply. We do have a content policy, and we do have a legally responsible person and a path to report illegal content. There’s no need for documenting all this at a higher level.
I thought so as well, having worked for a MEP. Well, I’ve started to read the actual law and OH MY GOD, what a mess! Are all UK laws written in this impenetrable style? With multiple levels of outlining and gotos that criss-cross the document?
GDPR can be read and grokked in a day. But this?
Like this for instance:
Why the hell do you include (1) if you scope (2) to (4) appropriately anyway? Was it an actual intention to make this into as long as possible read? Was author paid by word?
Oh god, kill me now. Yes, captain obvious, that’s what legislation is supposed to do. I am holding my breath, I am literally reading the law here, the suspense is killing me, do tell, what are the duties of various kinds of search service providers?
Another GOTO? Are you kidding me?
I give up, just block UK.
Writing laws like this is a post-Brexit cottage industry, and ensures the perpetual employment of both civil servants and lawyers.
Ah, because they are not spending their time harmonizing, they must do something. I understand. Sigh.
Oh yeah, strong agreement from me. The ethics module in my CS undergrad had us reading both British and EU legislation. The British was no doubt the most confusing — worse than the ancient French texts we were reading (in French).
I think that a lot of what the Online Safety Act implements is overzealous and there’s a big mismatch between the stated aims, and what the legislation will achieve. I’m not sure the quality of the drafting is really the issue.
I think this is a misconception and possibly a case of Dunning Kruger – just because you read a legal text and think it means X doesn’t necessarily mean that other people will, or (worse) that a judge will believe your interpretation. Further, these acts tend to be pinned down in case law, which is important to understand and can materially change the nature of what certain acts mean. For example, EC261 provides for compensation for cancellations (and doesn’t specify that it applies to delays in the act, iirc), but EU case law has clarified that it also applies to delayed flights.
Why do papers have abstracts? Clarity is imperative (even if it feels like stating the obvious), and prevents further issues from cropping up down the line. It’s like how “experts write baby code” (rather than big soups of OOP). I also think as a general writing principle that providing an outline of the structure of your content is pretty important. It’s like how a technical book will say “in this chapter, we look at how to solve X problem” so that you know what it’s trying to do, rather than just having to guess.
It’s not that surprising that legislation is drafted in clauses; you can point to the clause, instead of typing things out lots of times. It also allows people to verify that the clauses are correct by looking in one place, rather than having to search all over the document.
I am sorry, but this specific piece of legislation sucks. Normally sets are defined first, then clauses that apply to those sets are introduced, grouped topically. This law used sprawling if-else and repetitive vacuous clauses. It is objectively hard to read, not easier.
Imagine being reprimanded for braking the law. What did I do? You are breaching paragraph X, section Y, clause Z. Oh, does that apply to me? Yes, section C says so. Nope. Just nope.
Sane law can be read from the bottom, looking up if you fall in the set from the consequences. Good luck here.
I have to say I don’t really understand your issue with the chapter? As far as I can tell, chapter 3 is set out roughly along the lines of
Seems pretty straightforward to me. My objection is more to the actual aim of the bill rather than whether it was drafted for a mathematically-minded person.
The Dunning-Kruger effect has been debunked for a long time.
Thanks for trying to help run down the OSA’s scope and effects. With the reminder that I don’t get much confidence from non-lawyers interpreting the law, I’ll post some more of my open questions.
I’m pretty sure Lobsters is categorized as a multi-risk service (section 5.6 on page 78 of that PDF) but can’t assemble a coherent definition between the vague terms, foreign legalisms, and bottomless well of cross-references. My primary open question is whether the OSA considers these categories to apply because all of them are possible in a
textarea, only if a service doesn’t write policies opposing them and variously track its compliance (ICU A1-A7), or only if a service demonstrates a consistent pattern of being used for them. A couple relevant facts: Our search engine doesn’t search multiple sites, but we do have direct messaging and image uploads (and do not currently have access to databases of CSAM URLs or image hashes). Moderation is reactive rather than proactive screening. We do not attempt to identify individuals (except in cases of specific abuse like sockpuppeting) and do not attempt demographic measurement like age or location. This vagueness informs where we land on the Risk Level Tables (pages 58-68), which cautions in bold “we expect providers to err on the side of caution and select the higher risk level”). With the endless ambiguities in these documents and that warning, the safe thing to do is geoblock the UK at least until Ofcom makes a clarifying statement or demonstrates for a couple years that it’s not going to demolish hobbyist forums.Because software has permeated every aspect of life, including relationships, we have regular mentions of listed topics like gambling, sex, pornography, suicide, drugs, crimes, and various kinds of abuse. I’m reasonably sure Ofcom is capable of use-mention distinction here, but it’s worth being aware of the topics if the OSA doesn’t make that distinction somewhere (especially likely around the four horsemen) or if there’s room for a political appointee to decide to make an example. I’d really like to see an explicit distinction on this in the OSA.
Also I swear I saw a page on Ofcom acting as a table of contents and listing all their current OSA PDFs but I can’t find it again. Does anyone have a link like that? I am more than a bit lost in a mazy of twisty PDFs, all alike, as I try to cross-reference these documents and run down the many vague terms.
As I noted in another comment, none of this addresses jurisdiction. While the UK and USA share legal history and culture, I also have to look down the road to when much less compatible countries (or administrations) attempt to enforce similar laws.
From spending a few days this week researching the act, I now believe it has been intentionally made broad and vague so as to allow Ofcom the flexibility to reinterpret it as technology changes. This is so that they don’t have a similar cat and mouse game that they do with legal highs.
Ofcom for their part are currently in consultation on implementing fees and penalties under the Online Safety Act 2023, their report on the progress can be found here https://www.ofcom.org.uk/siteassets/resources/documents/consultations/category-1-10-weeks/consultation-online-safety---fees-and-penalties/main-documents/consultation-online-safety---fees-and-penalties.pdf?v=383849
Interestingly, under section 3.3 (pg.30) they propose to “exempt providers whose UK referable revenue is less than £10m in a qualifying period” from paying fees. I do wonder if this includes penalties.
Ofcoms enforcement of the act must be entirely self funded via fees and penalties, their own rules dictate that penalties must be considered reasonable and one can consider fining an individual many times the amount they could earn in a lifetime to be unreasonable. Due to this, and the fact that they are to be self funded I suspect that they are going to focus entirely on entities that earn more than £250m revenue in a year because it does not make economic sense to spend hundreds of thousands of pounds going after an individual where you might at best collect a few thousand in a “reasonable” fine.
It is my hope that in the coming months we see the published result of this consulation and confirmation that they will be excempting small communities such as Lobsters much in the same way as GDPR does.
This gives me hope, thanks for sharing. I was worried I’d have to block my own country for my own product (a forum SaaS thing) and I clearly need to do some research. Similarly, hobby-adjacent projects like mine, while revenue generating aren’t anywhere near the numbers matching these lawyers and penalties. We’ll see… as someone who grew up on forums and they are the sole purpose I even have a job designing/coding this whole thing disappoints me. But it flew under the radar for me and I regret not reaching out to my representatives when I had the chance.
While my plans to leave this country are largely due to things like this, it does feel like a “frying-pan-fire” situation as the sentiment reflected in the OSA seems to be cropping up in a lot of countries, including where I’m heading… I do miss the old internet. (grumpy old man moment)
Where are you moving to?
Canada, for better or worse!
speaking as someone who immigrated to Canada a few years ago, good luck! it’s a country with many challenges but it’s a functioning democracy, and remains a very good place to live.
thanks! I’m looking forward to it, and looking at the options a few years ago, everywhere has its issues so its a tradeoff, canada struck a nice balance for myself and my partner as a similar-to-the-us-but-not-as-insane kind of vibe, plus natural beauty ticks a lot of boxes!
it truly is gorgeous and full of nature
I agree on this. However, as service providers and people that do need to create compliance, I also find interaction with peers useful.
To be honest, I have drafted and killed a few replies to this. I think your assessment of lobste.rs as a multi-risk service is overly negative, but given the poor communication of Ofcom particularly towards lay-people, I would just engage in trying to nail pudding at the wall together with you and reading tea leaves. So I’ll leave it at the gut-feeling and be honest about that being the only thing I can address.
This is the Ofcom page I was thinking of as a table of contents. It’s a little confusing because some early links are repeated, but Ofcom’s guidance totals 2,426 pages, with a further 7+ guides forthcoming. Plus, of course, the 350 page act itself.
They added 295 pages of material this morning. It is not linked from the page I thought was a table of contents, so I guess that is not intended to be complete. It seems likely there is more than the 3,071 pages I’m aware of that Ofcom expects sites to comply with.
The first document says “From today, these services have three months (by 16 April 2025 at the latest) to complete their children’s access assessment.” and then another doc says “To establish whether a service, or part of a service, is likely to be accessed by children, service providers must first complete a children’s access assessment.” which is this doc which depends on section 37 of the OSA which defines the “Meaning of ‘likely to be accessed by children’” as “possible for children to access the service or a part of it” and “the child user condition is met”, where (to skip several dereferencing steps) that is defined in the OSA chapter 4, section 35, subsection (3), clause (b) which says “the service, or that part of it, is of a kind likely to attract a significant number of users who are children” which is all the definition available and also hinges on the infamously undefined term “significant number”.
I know learned to program as a child and that most children are capable of reading, and a quick search shows the UK has promoted many programs encouraging children to learn programming. So perhaps Ofcom also demands Lobsters perform intrusive checks of visitors’ private personal information (“highly effective age assurance”) like demanding a state-issued identification document and live selfie. And of course to document the hell out of that decision and implementation. Or perhaps not. There’s really no way of knowing.
Just in case you’ve not seen it, Ofcom are publishing a compliance checker https://www.ofcom.org.uk/Online-Safety/check-how-to-comply/ ‘in early 2025’.
I believe that’s the tool discussed in these comments, though it’s difficult to be sure given the structure and organization of the site.
They look different to me?
You’re correct, they made a new tool available today.
As long as I’m writing, I’ll include a small update I mentioned on yesterday’s office hours: I’ve started an email conversation with Ofcom and gotten a human response about routing my inquiry to the right office. I recently talked with the owner of a US small, noncommercial forum who asked Ofcom about the OSA and received a demand that all services, no matter how small, undertake the enormous task of evaluating against their thousands of pages of vague guidance. So I didn’t bother re-asking that and only asked them to explain their legal basis for claiming global jurisdiction.
I’m also reaching out to the US Embassy of the UK to see if they can help with the jurisdictional overreach.
The US similarly asserts extraterritorial jurisdiction over communications received or receivable in the US. I don’t think you’re going to get far with contesting jurisdiction to effectively build walls around countries.
Well, my optimism in that other thread is beginning to look pretty foolish.
FWIW, which I concede will likely turn out to be very little, I did write to my MP. One of their caseworkers wrote back a couple of days ago to say that they had raised my concerns with the Culture Secretary for “a formal response in writing”. I look forward to receiving it.
I finally received a reply today from Baroness Maggie Jones, the Minister for the Future Digital Economy and Online Safety. Alas, it appears to be a copy & pasted screed about “small or medium-sized enterprises” and does not address any of the concerns I raised about volunteer-run community forums.
As you probably know, a letter “from” a Minister is actually a letter from a minion (I was one, briefly) and that minion is paid to avoid committing anyone to anything. So its content is worthless … the fact that they felt compelled to pay someone to write it might be a very good thing though :-)
I’m afraid I’m pretty sure they just clicked the “Mail Merge” button in Microsoft Word. I suppose they must have received enough complaints and queries about the OSA to make it worth writing a template reply… but they don’t seem to feel compelled to do anything more than that.
Right, but when they do that they send a report up their management chain saying how many addresses got merged in, so at least you’ve added one to that :-)
Yep writing to your mp is the same thing. They pick their pre written blurb that they believe pertains to the general topic of your letter. As such it’s usually totally unresponsive to what you say to them.
Turn up at a ‘Surgery’ and put them on the spot. That will (possibly) get a bit more notice.
<3 sorry for the pain. Good luck with the Embassy!
Speaking of the GDPR, there’s surprisingly little online furor over the OSA compared to the frankly almost hysterical coverage of the GDPR back in the day.
For instance, if lobste.rs is in scope, than HN is, and I don’t think they’re gonna take any action blocking UK users. But I might be wrong.
The fediverse also seems to be shrugging off the OSA, even if many instances probably are in scope.
TBH, I’m a fan of the GDPR. Having run affected web services for years, GDPR was mainly a slap on the wrist e.g. to providers that (already illegally) didn’t practice double-opt-in for mailing list. GDPR was a massive “we’ve had it, there’s fines now, and we note down the best practice here”. Like, a lot of the stuff the GDPR bans were already illegal, but all the large services ignored it, because the laws were toothless.
So people who were already quite conscious around privacy, which most self-hosters are, were pretty fine. Also, there are clear indicators in the GDPR should be easy to comply for small services and the first reaction is a stern warning.
The OSA documents are hard to track and even finding the definitions of what a large service is always takes me 5 minutes, because it’s hidden somewhere.
To be honest I think a lot of the furor was from online marketers who saw a credible dent appearing in their cash flow and who tried to astroturf some opposition in the US. Even now the maliciously compliant “accept all cookie” banners are being blamed on the EU, not on the advertisers who implement them.
I rather think those banners are maliciously noncompliant.
I disagree. It introduces tons of regulations with often marginal effect. Big service companies can be mostly happy about the regulatory capture effect.
But basically most interesting services are in non-compliance with a strict reading. E.g. from the text of law (I am not a lawyer), I cannot see how it is allowed that I can check my emails from a non-compliant country.
It also applies to non-online services like doctor’s practices. Theoretically requiring doctors to call people by something other than the names that is not PII.
Rather than imposing all this churn on so many, one should focus on penalizing big abuses.
The GDPR has been effect for years now. If the weird edge cases you mention were actually an issue, we’d have heard about it.
You can see the fines imposed by member countries’ data inspection authorities here:
https://www.enforcementtracker.com/
I did hear about them :)
My comment included quite different points, though.
One was that it applies to a lot of small businesses – yes, I did here about concrete absysmal effects there about creating extra costs but I did not hear about persecutions of small businesses for violations.
The second one was about the hypothetical cases with the email.
If I read your comment right, you focus on violations for which there have been actual fines. I don’t think that this is the full picture.
Law, in the best case, should make clear what is allowed and what is not. If it fails to do that, the ambiguity creates unnecessary work. In the case of small businesses, a lot. And I am not even talking about primarily online businesses.
I think that I am connected to a bubble of small business owners that actually want to comply with the law (in Germany), and it is very annoying and creates extra costs. Try to get a lawyer sanction your GDPR compliance and tell me what the bill is… And that is only a fraction of the costs.
Also please don’t mistake this as me being in favor of keeping all of this unregulated. It can’t be. Many companies would just do things to their own advantage. E.g. disallowing shady double opt-outs is not a problem for me. But the GDPR does so much more…
I do beleive that the GDPR is not a problem for many best-practice online services. But that is only a small fraction of business in total to which the GDPR applies.
I can’t speak for the situation in Germany. Trying to get a framework that works in all of the EU with the very differing views on privacy is a big job. And if there’s a balance between indivdual’s privacy and business’ costs, then maybe the privacy is worth more.
I would expect trade associations for the businesses involved to have at least some guidance in place by now, since the law has been enforced for so long, so that an individual business owner can ask “is this software GDPR compliant”, “what is the best practice for this situation”, etc.
BTW, nice web page!
it’s worth understanding that the GDPR challenged power, whereas the OSA’s primary impact is on already-marginalized groups and local, community-centric undertakings. complying with the OSA will be significantly easier for the big companies than complying with GDPR was and remains, because it simply establishes rules for already-existing enforcement mechanisms to follow.
the topics and themes the media chooses to cover are not immune to influence from capital. :/
I disagree with this sentiment. This is simply not the right kind of comparison.
For GDPR your scope is: If you don’t invade people’s privacy there is close to nothing to do. Afterwards it scales with how much you want to invade people’s privacy and even there it feels soft: Tell people whom you give data to. Keep track of whom you give data to. Ask for permission before you do. Also explain what you use it for?
For the Online Safety act you need to take action based on your users’ actions, also you have to make essentially legal judgement, which means you wanna be strict, because the risk of not being is yours. That’s just for allowing users to communicate/interact, which basically affects every forum. So unlike in GDPR where if you just have some website collecting nothing you can just say “Not collecting anything here” in a privacy policy the mere fact that content is being made available in any form means you need to have a mechanism to be scanned BEFORE it even is published. That means that website owners, even for hobby things, interests groups, etc. there need to be measures in place, which as is seen has the potential to kill projects, simply by costs involved. For GDPR if you don’t give user data away the only cost is becoming aware of it.
On top of that the “harmful content” part isn’t defined, while in the GDPR private data is pretty well defined. So the OSA is also pretty subjective, which is never good for a law (but I understand it might be necessary sometimes).
So while I agree with the sentiment to remain calm and objective I don’t think that comparing OSA with GDPR in anything but attention and the scope of who is the target audience (everyone running a website) is really justified. GDPR is what YOU do with other people’s data. OSA is (also) about what OTHERS do (ab)using your platform.
The word “scope” was used here as a legal terminus technicus which differs from the ordinary meaning. Law acts often come with an explicit definition of their “scope of application”, usually at the beginning. For the GDPR, this is laid down in Art. 2 and Art. 3 GDPR. If you do not fall within the stipulations of these articles, the entire GDPR and all it has to say about privacy rules does not apply to you. We call this “to be out of scope” for brevity, because nobody really wants to cite the precise articles all the time.
I cannot say whether the OSA works like that, because I’m only familiar with German/European law, but I would highly suspect that this is what was meant.
OSA is neither German nor European. (Don’t think it’s relevant here but in fact German and European law are a completely different system from UK, civil law vs common law).
Did you mean GDPR?
Yes.
However, it exposes a very interesting phenomenon in many people’s thinking in computing: that many people have belief systems that are religious in nature. They do not care about evidence, rationality and logical reasoning: they just believe.
So, for instance, I’ve found that the Unix world in general is now so large, and so old, that many people have erroneous folk beliefs that are set by 50+ years of tradition. They’re not faith: they just don’t know anything else. Their little domain is the whole world because they’ve never known there was anything else.
That’s human nature. It’s sad – there’s so much more out there – but it’s how people work.
So, for instance, a few random erroneous beliefs from the Unix tradition are:
None of them are even slightly close to general truths, but they are axioms in Unix.
But that’s just ignorance.
The faith thing is much more frightening to me.
That LLMs are artificial, meaning that they were made by people. (They weren’t, aren’t, and can’t be.)
That they are intelligent, meaning that they can think in any way at all. (They can’t, but they fake it in ways some find compellingly persuasive.)
That they can program. (They can’t, but they can make up text that looks like code.)
That they can analyse and improve themselves. (Nothing can.)
That bigger models will make them better at what they do. (This one just exposes basic ignorance about very simple statistics, such as scaling effects.)
That LLMs are in any way a step to reasoning, thinking software. (Pure magical thinking, based on lack of understanding.)
The thing is that I now continually encounter smart, educated people who believe in these magic beans.
There’s two things I want to push back against in your comment.
1) The magic bean believer strawmanSo much of the LLM discourse I see (especially here on Lobsters) is polarized to the point where you’re either a believer or a denier. I see people who make very reasonable and tempered claims about the utility that LLMs provide them (sometimes with evidence!) that are blanket rejected because “ChatGPT can’t count its own toes correctly” or whatever new chicanery people have devised to get it to spit out a wrong answer.
Yes, there is an LLM cargo-cult. Yes, there are CEOs aplenty who want to sell you magic beans. Yes, I am sick of it too. But can we please reserve our hate for the myriad people and ideas deserving of it and openly hear out the ones who aren’t coming with this kind of agenda? The ones who honestly believe they have found something helpful?
2) LLMs ~ magic beansIt’s not clear to me whether you’re arguing in your comment that LLMs have no merit whatsoever, but since that’s a common sentiment I see and want to rebut, you’ll have to forgive me if I have inferred incorrectly.
The other thing that bothers me is the almost religious rejection of any LLM results and realities. Correct me if I’m wrong, because I only speak from vibes, but I feel the anti-LLM sentiment when copilot came out was “LLMs will never write code.” Advent of Code this year, for example, has had sub-30 second submissions on the leaderboard – if this is not evidence that LLMs are capable of some kind of programming, I don’t know what is, because humans surely cannot read the whole prompt and code in that much time. And now the sentiment I see has shifted to “well, LLMs can write some code but it’s not complicated,” seemingly in denial of these previous claims.
I want to remind/inform whoever’s reading this that in the decades-old field of program synthesis, the poster child for the longest time was FlashFill, which generates (simple) Excel formulas from examples. There simply wasn’t any usable general-purpose tool for program synthesis (regardless of hallucinations or syntactic inaccuracies or …). Now a large number of synthesis papers are (in a very simplistic and reductionist approximation) LLM + special domain.
You can debate whether LLMs in their current form have legitimate utility, but this debate becomes personal and I expect colored by your own perception of how they help you (placebo/nocebo). I think it’s too reductionist to write them off entirely.
These are but a brief summary of my thoughts on LLMs and program synthesis, I hope to getting around to writing more in the new year…
The problem with this framing is that you’re only looking at the ends, when for many of us the means play a part in the resistance.
If you bring an LLM that was only trained on content whose authors actively assented to inclusion, and where no companies were seriously considering building their own nuclear reactors because it takes so much power to run, and where there’s some hope of ownership and control of the LLM by end users instead of just large technology/surveillance companies, then sure! I’m all ears!
Alas, there is precious little of that in the space where Copilot and ChatGPT and Claude and so on are playing.
Completely agreed. I’m not interested in having discussions about trying to invent useful applications for this technology, because even the small-scale models that hobbyists can run on their home computers are produced at massive expense by extremely unethical actors out of datasets which were harvested without consent and then legally encumbered.
Supposedly innocent tinkering with LLMs furthers the goals and rhetoric of absolutely monstrous entities and helps them justify their abuses of the commons. Building hobby projects on LLaMa and writing effusive blog posts about cheating at code golf, automatically synthesizing a fart app for your phone, or accomplishing any other number of trivial tasks with help from a Claude-powered rube goldberg machine is doing free marketing for the generative “AI” merchants and cloud vendors.
LLMs are completely radioactive from many ethical angles even before you start digging into the harmful ways they’re being applied today or (as this article focuses upon) their complete unsuitability as a building material for reliable, efficient, or generally trustworthy software.
Then (as someone who believes these points should be made) I implore you to focus on the first half (not struck-through) part of what you’re saying. I think it does your argument no good to incorporate claims that are deniable, especially if they apply the fallacious reasoning I discuss in my parent comment.
You don’t even need to open the door to arguing whether in two years ChatGPT can write Minecraft if its existence today already constitutes as significant of a problem as you claim. I think it’s good to have people thinking critically about these tools (ethically and technically), but thinking critically means not getting swept up in the LLM hype and the anti-LLM hype.
I think it is good that companies are building their own nuclear reactors to power datacenters for generative AI applications. We need way more of this kind of thing, cheap energy makes all of society more prosperous and we get cheap energy from having lots of nuclear power plants generate it.
Ownership and control of LLMs by end users is important and it’s a genuine concern that we don’t get locked in AI systems controlled by a small number of companies. But this is not really a different problem than a small number of companies controlling proproetary software platforms used by huge swaths of the population (i.e. Google and Meta and Twitter and Apple existing).
How many are actually doing this? Nuclear power plants take many years (often a decade or more) to build. Most of the announcements I’ve seen have been ‘we aren’t a climate disaster, look we have this long-term strategy to be energy independent. We’re not actually funding it at the level required to be realistic, but let us burn vast amounts of coal now, we promise it’s a temporary thing!’.
@dgerard has a critical look at SMRs here
https://pivot-to-ai.com/2024/10/17/google-amazon-buy-nonexistent-mini-nuclear-reactors-for-ai-data-centers/
Even if a SMR is built, it still needs huge volumes of clean water to operate and cool. It’s never going to be a little container-sized cube that magically emits electricity, like a portable fossil fuel generator.
Then make it illegal to burn coal to incentivize building the nuclear power plants faster (and reduce the amount of regulation on them); I do not want to sacrifice humanity’s ability to harness prodigious amounts of energy on the altar of the climate. This isn’t even about LLM datacenters specifically, I want nuclear power plants powering aluminum smelters too. Or maybe photovoltaic solar is actually cheaper if we just pepper the earth with panels, in which case I support doing that, and the electricity can go into LLM datacenters as easily as it can go into anything else.
What I don’t want is for any human endeavor that uses a lot of electrical energy to get labeled a “climate disaster” by people who want to shut it down - least of all because scrubbing CO2 from the atmosphere itself is something that’s gonna require a lot of electrical energy.
That’s the thing; no matter how clean or plentiful your energy source is, there are so many better uses of that energy than LLMs!
I’m glad you agree it’s a problem! The difference, I think, is that I’m not constantly hearing about how I should learn to stop worrying and love Gmail.
That’s because the hardcore RMS-ish view that Gmail is unethical software because it is proprietary is low status.
You hang out in very different places than I do, it appears. That’s interesting to realize… I don’t even know what it would be like to be in a place where that view is low-status. It felt like the mainstream belief within the broader software community, when I was growing up. It still feels like a common belief to me. It’s really interesting to hear a perception to the contrary; thank you.
As a bit of extra flavour… I, and the people I grew up with, shared the same belief you did: we believed that software like Gmail is unethical and that we were taking the moral high ground by believing that.
For me, though, there was a moment along the way where… I still love open source software and wish more of the world was made up of OSS, but also came to the conclusion that many of the people I grew up with who held those beliefs the strongest… weren’t really going anywhere. Many of them are still exceptionally talented programmers but they’ve lived a life where they’ve consistently struggled to make enough money to house and feed themselves. Whether they’re considered low status because of their beliefs or because of their relative poverty or due to other lives choices is hard to say but it’s pretty tough to argue that they’re doing well in life.
In my life now, from where I’m standing, it seems like the general broad perspective towards OSS is mostly indifference. Most people I know run their production software on Linux, some on Windows, and none of them really do it for OSS reasons but rather because Linux is a good platform. They don’t really care that it’s open source, just that it works. I’m actually feeling a bit sad writing this.
I don’t think it makes sense to evaluate any idea based on perceived success* of those that hold it. Especially when that idea is likely to make you avoid chasing the riches of the tech industry.
To be blunt, reading that it sounds to me like you were willing to compromise those ideas for money and don’t value the same things as the people “not doing well* in life.”
* this is subjective as it depends on personal goals only those people can judge themselves
I agree most people are apathetic to things being FOSS, and that is quite sad but also why activism is needed. Not only for FOSS but all subjects; the status quo has strong inertia, if not players with incentive to maintain it.
To wrap back to the original subject, I believe Gmail is unethical mostly due to Google’s data pillaging and federation busting. Gmail not being FOSS is part of it, but far from the main reason, and mostly orthogonal: I believe VS Code is unethical even if it is mostly OSS and VS Codium exists.
100% will agree that I did compromise on those ideals in part for money but in the bigger picture for happiness as well.
The real tragedy with respect to the “not doing well in life” part is the magnitude of that. I agree it’s relative but it makes me so sad to see the some of the brilliant people I knew in school posting on Facebook (there’s a certain irony there…) about how their night-shift gas station job sucks and that they have to kick out another roommate for stealing or being abusive. It’s not just that they’re “not doing well” on my personal scale but that they also seem genuinely unhappy themselves.
But… this is all just one small community. I’m sure it’s not a representative sample.
That’s fine to articulate, but my reply is to ~lproven’s comment which makes no mention of the ethical considerations for using LLMs.
To be clear: I’m not saying that discussions about the ethics of LLMs should be silenced (I am inclined to believe otherwise). But I am saying that even if you think you have the moral high ground for not wanting to use LLMs, this doesn’t entitle you to misrepresent what they’re capable of. (not accusing you specifically, but I hope you get what I’m saying)
Put differently, I don’t like the simultaneous standpoints I see people take of “LLMs are bad for authors and the environment” and “LLMs can’t produce code, LLMs are stupid, etc.” The first point is much more damning and — perhaps more importantly — easier to verify facts for. I don’t see any good reason for denying evidence or inventing strawmans to support the latter point.
I feel the same but like… the opposite lol
I don’t even understand this. Is it even a question? Of course they are artificial.
You’re gonna have to define “intelligent”. Depending on the context, LLMs are obviously intelligent, but under other definitions it clearly is not - in fact you’ll find that many people have wildly divergent definitions. A panpsychist is not likely to say they’re intelligence, a functionalist may. Even by some pretty rigorous definitions. This isn’t a religious question, it’s a metaphysics question. Is intelligence the ability to reason? Is it a sufficient set of justified beliefs/ knowledge? Does it imply abstract properties? Is it an abstract property? Is it physical? Emergent? Seriously, it’s incredibly reductive to say “LLMs are not really any kind of intelligence”.
Also confusing since that has been the case. The question is really just a matter of the limits of this scaling. It’s sort of like saying “gzip isn’t going to benefit from a larger disk because at some point compression has information theoretic limits” well yeah, but the bigger disk still helps. I’m not really sure what you’re trying to get at here though, maybe you can be more specific.
I mean… this is just a weird definition to me. They use statistical reasoning to generate valid programs. I guess you think that’s not programming?
WOW lol sorry but I’m out. You’re seriously going to accuse people of religious thinking and then make these kinds of assertions? Like there are entire metaphysical theories devoted to these questions that you’re so glibly dismissing. Nothing can analyse and improve itself? That is such a wild claim to just assert like this.
If we define reasoning as using evidentiary inputs to produce conclusions, LLMs reason. Statistical reasoning is reasoning.
These arguments (read: unjustified assertions) are so weak and you seem to not even realize the metaphysical commitments you’re making by pushing this worldview.
If you want to have a reasonable discussion, by all means please do. I see so few discussions that look reasoned on this site and it’s a bummer. Let’s talk about knowledge, let’s talk about reasoning. Let’s do it! Let’s ditch these assertions, let’s ditch the question begging, let’s have a little epistemic humility, okay?
Yeah, it doesn’t really matter whether LLMs have intelligence and rationality. They can make our lives easier by solving some problems, just like cars and computers, and that’s enough. That being said, I also agree with the article that LLMs need to be improved in terms of reliability and explain-ability.
This black-and-white language leads to confusion.
I recommend the following definition: Intelligence is the ability for an agent to solve a task. The degree to which such abilities span different areas is the degree to which we call it general intelligence.
Nope.
Place a blind invertebrate into a box with a particle of smelly food. Wait. It will move around in a semi-random walk but it will find the food. Problem: solved. Was intelligence used? No. The algorithm is so trivially easy, I’ve solved it:
Move in long straight lines until you smell food, then move shorter distances and turn more.
That’s it. That is the algorithm. It works.
Pools of water connected by pipes: find the lowest level. It works.
https://en.wikipedia.org/wiki/Water_integrator
Yes, I am aware of this criticism.
According to my definition (which matches that of Stuart Russell, more or less), there is a wide range of intelligent behavior.
Do you want a definition that works across a wide range of situations? Or do you want a narrow definition? Why?
Can the above commenter offer a non-binary definition that is useful across a broad range of intelligent behavior? I would like to hear it.
Those who have studied artificial intelligence and animal behavior for decades often go with something quite like my definition. See e.g.
https://emerj.com/what-is-artificial-intelligence-an-informed-definition/
The only thing you did with your comment here is do an appeal to authority.
How about this: nobody is an authority on what is an intelligence and what even is reasoning. People get a PhD and start pretending to know, that’s the crux of it, let’s not sugarcoat it please. That applies to every single non-practical discipline, philosophy and many others included.
Also your comment came across as disingenuous because you first offered a blurry definition of intelligence and then, when called out, retreated into the appeal to authority and that a need for nuance is needed.
I don’t see how the latter follows from anything at all. No need for nuance, whatever that means in this situation even; we need good working definitions of the sort “what constitutes a fusion reactor?” for example.
All the discussion about what passes for “AI” these days is just sad to watch and that includes your comments. It resembles theological discussion first and foremost.
Surely technically-inclined people can do better? Inputs, expected outputs, process, all that? You know, engineering?
Personally, your comment comes across as unkind. Maybe just a bad day? Are you willing to try a more constructive and charitable direction?
Some responses:
Remember the thread context; I was criticizing some flaws in a definition someone else offered. My definition addressed the flaws I pointed out.
I “retreated” above? I’m happy to admit if I made a mistake, but where are you getting this?
It isn’t fair, accurate, or charitable to say that “appeal to authority” is all I did.
I pointed to what I think are some good jumping off points. Did you read any of them (Russell, Hutter, etc)?
Do you know a better definition of intelligence? … and why do you think it’s better? (You can see in my other comments the need for a definition that isn’t human centric.)
I’ve seen the anti-PhD venom before. I used to say the same kind of thing decades ago. It was ignorance on my part. This lessened as I interacted with more people with Ph.D. experience and as read more machine learning and CompSci literature.
No experts? There are people with more and less expertise. Acting like there are no experts is an unhelpful exaggeration.
If you read the comments I’ve written, you’ll see it is difficult to place me into any simple categories regarding AI.
I find it bizarre that you think my comments are anything like theology. Offering a working and operational definition of intelligence does not theology make.
I’ve also seen the anti-philosophy trope before. It is unfortunate and self-limiting. The classic response here is: many other sciences and disciplines were birthed from philosophy, but philosophy rarely gets credit. Yes, some philosophy is painful to read. One usually has to put in a lot of effort searching, thinking, writing, and discussing to reap the biggest benefits. Asking for reading recommendations from people you respect is a good start.
Well, it might be cultural background on my part because I am not used to dance around disagreements, and as I get older this is less and less likely to ever change. Not my intention to offend, mostly to pull you away from what I perceive is a comfortable and maybe even complacent position.
I did simplify your comments, that much is true, and 99% of the reason is that the “AI” discussions inevitably devolve into “But how do we know what intelligence is? We might already be witnessing it but are disparaging it!” which, my turn to say it, I view as extremely unproductive, not helpful for advancing any discussion, and basically serving no other purpose than to congratulate ourselves how smart we are, and to offer a plethora of very out-there “what if”-s.
If you believe you are doing something more than that then I’d love to see more concrete claims and proofs.
I have not read philosophy on the mind and intelligence. I tried. I found it unproductive and very rarely did I stumble upon a lone article (not a book) where somebody actually attempted to invent / classify a framework in which we should be thinking about what mind / intelligence are. Everything else I attempted to read did read like empty hand-waving to me.
If you are willing, I’d like to get back to some semblance of a topic: do you believe LLMs can “reason” or are “intelligent”? Do you believe we are mistakenly putting them down while they, the poor things, are the next-stage-of-evolution, marginalized and oppressed artificial life forms that are just misunderstood? If not, what, apropo, was your point?
You’re being a bit of a dick here on multiple fronts, and misspelling the italicised apropos in the final sentence doesn’t shake the look.
Misspelling can happen to anyone. Shame that the glorious and almighty “AI” never improved autocorrect in phone keyboards, right? I’m sure that requires godlike intelligence though, so it’s excused. But hey, it actually “understands” what it does and it’s obviously intelligent. Surely.
And I’m not interested in philosophical discussions unlike a lot of people who can’t help themselves every time the “AI” is mentioned.
I’m interested in seeing proof that their downright religion-like beliefs have any rational foundation.
Alas, that kind of expectation is misguided. Faith doesn’t require proof to exist, as we all know historically.
If challenging belief makes me a dick then I’m okay with it. I was still never answered in a satisfying manner.
My conclusion is that this is a cozy topic for many. Mention “AI” and they all pull up the cigar and the 20-year old whiskey, and they’re all suddenly decorated philosophers who are perfectly qualified to speculate and to present conjectures as facts while they want us to believe that their entirely unrelated Ph.D. makes them educated on a topic that absolutely nobody has ever solved.
So yeah. Believe what you will. But recognize it’s only that: a belief.
Well, I think it matters a lot if LLMs have intelligence because the implications are pretty huge. Kind of like, “are there objective moral goods” - if we could answer that question we could rule out all sorts of metaphysical theories about the universe, and that seems valuable to me. Practically, and I think to your point, whether it’s intelligence or a facsimile of it (assuming this distinction even makes sense, which is a HUGE assumption!), as long as the results are the same it isn’t important (in terms of how it’s used).
I also agree with the article. I thought it was well written and makes sense - I thought the idea of breaking down models into testable components was particularly interesting. The comment I responded to doesn’t even seem related to the contents, which I also thought was ironic since it was a plea for rationality and informed commenting.
There’s an even further metaphysical question that goes with that… how do we even define intelligence? What’s the threshold? Is a tree intelligent because it grows its roots towards water? Are bacteria intelligent? Fish? Cats? Dolphins? Horses?
Are current ML models like any of these? Or are they more like https://en.m.wikipedia.org/wiki/Clever_Hans? Or was Clever Hans actually intelligent, just not in the way that it was claimed? (He couldn’t do arithmetic but he could very accurately read subtle human signals)
All really interesting things to ponder on holidays :)
Well, OpenAI recently defined “artificial general intelligence” as “whatever enables OpenAI to realize $100B in profits”.
Yes/no questions fall flat quite often. For many interesting subjects, including intelligence, consciousness, justice, fairness, etc., there are better framings.
We already have human-created machines that try to answer these questions. They’re called religions. Adding another one with a LLM dash of paint will probably not resolve anything.
No, religions don’t inherently try to answer these questions. These are questions that fall into the domain of philosophy. Whether moral goods exist, their nature, etc, is not a religious question but a metaphysical one.
My statements stands; the ability to answer questions about the mind would indeed hold value, it would resolve many open questions, rule out some metaphysical theories, make others more or less unlikely, etc. It would potentially have very direct impact on human life - theory of mind is essential to theory of personhood, for example.
OK. I read your original comment as something akin to Platonism - there are eternal truths that are hidden from us. How a machine trained on the sum of humanity’s writing on these questions would be able to reveal them was unclear to me.
I see. No, I’m not expecting an LLM to reveal its truths. I’m saying that our investigation into knowledge, perhaps through exploration of technologies of LLMs, will reveal the truth.
Again, going back to Descartes, one of his main arguments for human exceptionalism as well as mind body dualism was that humans can speak. He suggested that machines could move, look like animals, look like humans, etc. He suggested that one could not differentiate a monkey from a robot monkey. But he maintained that language was the indicator of a true mind - so a machine could never master language.
LLMs at least provide evidence against that. You can argue about whether they’re a true counter example, but they have are evidence against a Descartes theory of mind. As we continue to push forward on technology we may have stronger evidence for or against other theories of mind.
Let me try to understand here: because LLMs disproved an extremely flawed hypothesis by a scientist who is a product of his time, this means… what exactly? Can you help me here? That LLMs possess mind, or is it something else?
No, not at all. I was giving an example of how our ability to create machines has historically changed our understanding of our theory of mind. It was an example of how producing new technologies can help us evaluate existing theories about things like intelligence. Applying it ot a very old, well understood (and unpopular, therefor not contentious to apply to) theory was my way of giving such an example.
As I said, I think that as we generate technologies like LLMs we may be able to generate evidence for or against theories.
Definitely not that LLMs possess a mind.
Maybe it is an example of how producing new tech can help us evaluate existing theories, yes… if those theories are not as old and almost laughable. Because comparing to them is a classic “tearing down a straw man” debate.
There is no straw man here, I don’t think you know what that term means. It is chosen explicitly because it is not contentious to say that dualism is rejected, and to show how one could use LLMs as evidence. That it is already rejected (largely) is the benefit of using this example.
Literally all I was doing. Soooo we’re good? Same page.
Sure, I don’t know what it means. OK, lol.
Whatever helps you, man. :)
No, they are not.
Artificial, meaning, built by artifice, created using the skills of an artificer.
LLMs are not built by humans. LLMs are built by software built by humans, running in large-scale clusters in datacentres. LLMs are a multi-dimensional grid of statistical weights of relatedness of words, with millions of dimensions.
A good simple explainer is the Financial Times’s one here and a more in-depth one is Stephen Wolfram’s one here. Just in case you were about to accuse me of not knowing what I am talking about.
The point being that humans didn’t construct those databases, can’t read them, can’t analyse them, and cannot modify them.
The humans built the software that built the models. Humans did not build the models, and can’t.
The tools that built the models are artificial. The models are not.
Cambridge dictionary: showing intelligence, or able to learn and understand things easily
Intelligence: the ability to learn, understand, and make judgments or have opinions that are based on reason
Merriam-Webster: having or indicating a high or satisfactory degree of intelligence and mental capacity
Intelligence: the ability to learn or understand or to deal with new or trying situations; the ability to apply knowledge to manipulate one’s environment or to think abstractly as measured by objective criteria (such as tests)
These are not abstruse technical terms.
LLMs cannot reason, understand, deal with new situations, etc. ALL they can do is generate text in response to prompts.
They cannot count. They cannot reason. They cannot deduce. But they can produce a modified version of text which does those things.
It is a fallacy of thinking to leap from “it produces text which sounds smart” to “it is smart”.
It isn’t. Something that can’t tell you how many Ms there are in the word “programming” can’t program.
Yep, 100% am. Refute me: prove me wrong. Falsify my statements.
Nothing you’ve said falsifies my points. All you are doing is mocking and trying to redefine terms.
This is just a really weird standard. If I build a chair from Ikea parts, did I not build a chair because someone else built the parts? What’s the point of this definition?
It’s not that you don’t understand the technology, it’s that you’re applying ridiculously strict terms to it and acting like anyone who doesn’t take those as gospel truth must be experiencing irrational, religious thinking. Nothing in those pages is going to justify a flat assertion that an LLM is not “artificial” tbh but even if they justified it I don’t think it matters - I mean the stakes on this couldn’t be lower.
Shrug. I don’t think this matters, it’s just pedantic. “A human built a machine that built the thing” okay. I’ve never programmed because actually someone else built the keyboard and the compiler that produced the assembly, so no, not programming. I’m a fraud, I suppose.
Dictionary definitions are fine but I think if you follow the citations there you’ll find they’re often cyclic. They’re also not some hard truth. These are metaphysical concepts with entire papers dedicated to breaking them down. Pointing to the dictionary as an authority isn’t a strong argument.
I disagree and I’ve already justified why.
It’s not a fallacy it’s the foundation of functionalism, which is a metaphysical theory worth taking seriously.
Baseless. Why should we connect these two things? I can’t factor massive primes in my head but I can program. Why should I connect these two things together? Again, LLMs can produce valid programs, so I think it’s on you to justify why that isn’t programming since it’s intuitively the case that it is.
I provided theories counter to your assertions. I mean, should I break down functionalism for you? I brought it up, you can learn about it if you’d like. I brought up panpsychism, abstract properties, etc. I think there’s plenty of breadcrumbs if you want to learn the metaphysical foundations of what I’ve described.
If you really want to provide some sort of formal syllogistic argument instead of just baseless assertions I could probably provide really strong, rational, well researched arguments from people who study these things explaining why someone could rationally reject your premises or conclusion. Like, my entire point is that what you think are “rational” premises are totally justifiably rejected by people.
I’m not redefining terms lol these terms all have varied definitions! That’s my point. A functionalist and a panpsychist will have radically different definitions of intelligence. If you want to plainly assert things like “nothing can ever analyse itself” well jesus christ dude that’s a massive metaphysical commitment that you are making. I’m not the one making positive assertions here, I’m not the one accusing others are being irrational.
TBH you’re the one who started off with the mocking “everyone else is irrational and ‘religious thinking’, now here’s a list of baseless assertions” sooooo idk, I feel really fine with my response. I don’t think your list of assertions requires much “refutation”, I can just show that they’re incompatible with very reasonable metaphysical theories and so anyone who subscribes to those theories is perfectly justified in rejecting your unjustified assertions.
That’s absolutely fine and you are of course completely free to say that.
I have been a skeptic (with a K, which is not the normal spelling in my country) for about a quarter of a century now. But that’s when I learned the label: I’ve had the mindset since roughly when I reached puberty.
There is a whole tonne of stuff I don’t believe in that billions of people passionately, fervently believe. Many of them would kill me for it. I don’t care.
I have seen no evidence that LLMs are in any way intelligent, that they can reason, or learn, or think, or deduce, or in any way reproduce any of the elements of intelligent behaviour.
Metaphysical or philosophical arguments along the lines of “we don’t know what ‘intelligence’ means’ or “what does it mean to ‘think’ anyway?” are just pointless word games. Have fun with them if you want but I’m not interested.
So, I can happily state my assertions:
This is discussed in depth in this Nautilus article: https://nautil.us/ai-is-the-black-mirror-1169121/
It’s the silicon valley tech-bro mindset.
I think you’re wrong, but I don’t think there is any evidence I or anyone can come up with to convince you.
That’s OK. It’s just a passing annoyance. The bubble will pop, there’ll be another AI winter just like circa 1970 (+~5) when they worked out the limitations of single-layer neural networks, and then again circa 1985 (+~5) when Unix got cheaper and faster than Lisp.
Personally I am looking forward to it. It is very irritating.
I also don’t believe that there’s any practical useful application of blockchains, anywhere or for anything, and that all cryptocurrencies, NFTs, Web3 and the entire field is an empty bubble that will implode.
Other things I am happy to tell billions of people they are wrong about…
Supplementary, Complementary and Alternative Medicine. It’s all 100% fake. Medicine is what can be proved to work; if you can’t, it’s not medicine. That’s why I use the term: SCAM. It’s a multi-billion dollar industry and it’s entirely and completely fake from top to bottom.
This one is easy: show it works, and it immediately ceases to be SCAM, it becomes conventional medicine.
Religions. All of them. There are no gods, no soul, no afterlife, none of it. Every believer in every religion alive and who has ever lived: all totally wrong.
I am perfectly happy to recant, if I am provided with objective, verifiable, reproducible evidence of the supernatural.
But until then, I repeat, it is all completely fake, and so is the entire field of LLM-driven “AI.”
None of my arguments are “lots of people believe X so X is true”. What I’m saying is that there are very rational, viable, reasonable metaphysical theories where your assertions are either outright rejected or are unlikely, and you saying that anyone who doesn’t follow your line of thought is irrational or thinking “religiously” is itself an irrational statement, unaware of the dialectic.
That is not what I’m saying. Different metaphysical theories will define these terms differently. Again, it’s just ignorance of the dialectic to say things like “AI isn’t intelligence” - not only is it an unjustified assertion, it doesn’t even make sense without defining the term, and it would be great to align whatever your definition is with some sort of established metaphysical theory so that we can know your commitments.
It’s very common for people who aren’t familiar with actual logic, reasoning, and metaphysics to think that it’s just “word games”. You’re the one saying that people should be rational. You’re just not meeting the standard you’ve set, from my view.
I’ll be honest. Your assertions are lacking. They’re informal, lack coherent premises, bring in tons of terminology that you’ve failed to define, etc. For such bold claims I’d really expect more.
Again, you are the one making the positive claims here, it’s kinda on you to do better if you want to accuse everyone else of being irrational.
These are just incredibly bold claims. To say that there are literally no useful applications of a technology is extraordinarily profligate.
Yeah, I can’t stand this form of atheist. As an atheist myself, I find it quite annoying how modern atheism treat religion as totally irrational. I’m what modern atheists would call a “strong atheist” (garbage terminology but I’m going to speak in the terms that I suspect are more familiar to you) but I think anyone tho thinks that you can’t be rational and religious at the same time is just extremely ignorant of the actual papers and research on the topic. I’m not compelled at all by that research but I’m aware of it and understand its legitimacy.
I’ll be honest with you, I think you should take some time to learn basic philosophy and metaphysics so that you can understand how much you’re oversimplifying things and how bad of a job you’ve done at presenting anything like an argument. I don’t mean this as an insult, I say this as a peer and someone whose views almost certainly align to a degree with yours. You should learn what a basic syllogism is - a good exercise would be tot ake your assertions and break them down into distinct syllogistic arguments that you then defend in terms of evidence and justifications for your premises and a justification for the conclusion following from them.
That said, here’s my response to your assertions. It’s a bit tricky since you haven’t presented any kind of theory or aligned with any existing theory, you haven’t defined terms, you haven’t justified any beliefs, you haven’t put your arguments into any kind of formal logical form like a syllogism or a bayesian inference, etc. This is very “internet argument” level, not what I’d want from someone telling everyone else how irrational they are for disagreeing.
I rejected this already. I reject your definition of A, you’ve failed to define I entirely. I’ve already pointed to functionalism, so if you want to reject functionalism by all means you can feel free to do so but I think it would be quite ridiculous to call all functionalists irrational even if you disagree with them. Functionalism is well defended, even if there are competing theories and great arguments against it - I brought up panpsychism, although I suspect you would reject panpsychism as, despite being compatible with atheism, it does strike me as being less likely under atheism. But that’s a whole separate issue.
I’m going to break this down because this is actually numerous assertions.
2a. LLMs are not a pathway to AGI or anything else.
Unjustified, lacks proper definition of terms.
2b. with few useful real-world applications
I reject this trivially. I find them useful. Trivial for me to reject this.
2c. and the construction of ever-more-complex prompts in efforts to constrain them into producing useful output is futile:
Possibly justifiable, though you didn’t do so. But I would probably grant this if “useful” meant something along the lines of deterministic or “highly reliable” like the article mentions. I mean, I would grant it because the stakes are low and I don’t care to argue about it, I don’t actually think it’s true.
2d. it’s just a vastly inefficient new form of programming
I wouldn’t care to reject this because “inefficient” is so subjective and ill defined.
2e. one which can never work because the best it can ever do is a small statistical reduction in the amount of noise in the output.
“Work” isn’t defined, “small” is confusing, unclear what your point here is.
I think this is a really odd take since machine learning algorithms are statistical models. To help you out, what you’re looking for here would be called a “symmetry breaker”. If you grant that those algorithms are useful but you reject that LLMs are you need to show why.
Honestly, I find this deeply ironic. I find your post highly guilty of what you’re accusing others of. I think you should deeply investigate these areas and the rational defenses one can put forward before throwing stones at others like this. You’re clearly unaware of the academic discussions around these topics, how arguments are formed, what different types of reasoning are, what evidence is, what rationality and what rational beliefs are, etc etc etc. I think I’ve provided plenty of information for you to learn about these topics if you’d like to raise the bar for discourse, something I’d personally love to see.
Again, I’m not trying to be insulting. I’m a bit too lazy to be less glib here, I hope that the mere fact that I’ve taken the time to try to express myself to you shows that I’m willing to engage constructively. I can see that you want to elevate the discourse, that you’re sick of irrational beliefs, and I’m extremely sympathetic if not empathetic to that worldview. It just seems that you would benefit greatly from learning about what that has looked like over the course of the last few thousand year and to learn about how many metaphysical theories exist that are arguably very rational.
If you want to just make baseless assertions, go for it. That’s how most people talk online. But since you went out of your way to point out how you think others do that, I thought it worth pointing this out to you.
If I were to really dig into your assertions I’d have to do a ton of steel manning of them and expand on them to make them something I could actually try to refute or justify competing theories against. You’ll have to forgive me, as much as I’d like to and believe it’s worth doing, I don’t have the time right now. Maybe I’ll write something up at some point. Until then, I’ll suggest we just agree to disagree. Cheers.
Unbelievable that this is flagged as “troll” lol this site is so fucking ridiculous sometimes. @pushcx can we please get some kind of system for handling people who erroneously flag content? Disagreeing with me is fine but it is nuts how often I put in significant effort, provide justifications, even reference research and papers, and get flagged because someone doesn’t like my post. This has happened way too many times to me.
I not only explained clearly what an argument is, what the current academic conversation looks like, etc, I even went out of my way to go beyond what I think was even reasonable to expect and responded to these assertions despite the fact that they are so poorly expressed - I was unreasonably charitable in my response, especially given the context of this conversation starting through someone stating that everyone else is irrational and won’t engage in logic.
Here’s another absolutely ridiculous example of me being flagged as a “troll”: https://lobste.rs/s/4czo0b/dropping_hyper#c_fsyjel
I could find more but idk how to scroll further into my comment history.
You provide exactly nothing except long philosophical essays in technical discussions. It’s quite telling that you don’t see why you got flagged (for the record, I have no ability to do so on this website but I absolutely would if I could).
You have “rejected” and “proven” exactly nothing as well. All you do is hide behind metaphysical clouds and empty philosophy while pretending to give objective evidence.
Yeah, many of us see through that and do perceive it as trolling. Until you come up with something concrete then you will not convince anyone that LLMs / “AI” are not snake oil and that their next winter is coming up soon and it’s going to hit hard and last decades.
I actually can believe that you are fully believing what you say, which would be even sadder. In conclusion of these fruitless and never-going-anywhere “discussions” (because I am 100% convinced you will double down; prove me wrong!) I can only tell you that every now and then it’s good to revisit your own bias and bubble. Now THAT would constitute true intelligence in my eyes. ;)
By the way, I used f.ex. Claude with success. It helped me fill the gaps in my knowledge in a niche area that I was not having the patience to invest learning from scratch. I was impressed and I loved the experience.
…But it also demonstrated that to me that people can very quickly just spiral into “no, that’s subtly wrong, I need X and you are giving me 0.85*X – let’s keep refining this until I get what I want”. I actually viewed the daily prompt limit (free tier) as a very good thing: it forced me stop and revisit some assumptions that I subconsciously did along the session, and for good reason – turned out I was going in the wrong direction and was given wrong input to test an algorithm with (which really did make me laugh because Claude did not validate that input either).
And in conclusion to that topic: LLMs will at best just remain very useful tools for a bunch of tasks. They are not intelligent by absolutely any meaning of the word except those that are super loose and just love hiding behind clouds of philosophical ambiguity… like yourself.
No need to reply as reading your comments here has convinced me you are not capable of productive replies that evoke true practical observable evidence. But you do you.
You can dislike my posts but to say it’s trolling is just silly. You can think “wow this guy is so dumb!” but that’s not against the rules, it’s not trolling.
I’ve never pretended to give objective evidence? What are you referring to? As for “hiding”, what?
Let’s remember how this thread started - someone stated that anyone who thinks LLMs aren’t a scam is guilty of being irrational and “religious thinking”. That’s the first comment. I justified why I reject that by pointing to many live, well thought out, rational metaphysical theories that would reject this. This isn’t hiding or “empty philosophy”, it’s a direct contradiction to the idea that anyone disagreeing is irrational or religious thinking.
I have zero requirement to do this. The positive assertion made was that anyone who believes that LLMs / AI are not snake oil are irrational. All I have to do is show that that’s not true by showing live models that are rational.
I wonder what it is you even think that I believe? I’ve made almost no commitments in this discussion because, and I’ve said this a few times, I’m not the one making the positive assertions.
I mean, lol. Congrats but according to the person I was responding to you are irrational and guilty of religious thinking because you think that LLMs have any use at all. So… what is it you disagree with, exactly?
Anyway you’ve just done what the other poster has done. You say “They are not intelligent” okay well that’s a fine opinion? You’re not justifying it. I guess you think justifying anything would be “hiding behind clouds of philosophical ambiguity” idk it’s super irrelevant because the premise asserted by the author was that anyone who disagrees with that statement is irrational and religiously thinking, but you think LLMs have a use so they think that about you too!
As predicted, you doubled down. There’s zero substance to your “I mean, lol” bit because I am pretty convinced this person would recognize where an LLM can save you a few minutes (if not then they are indeed too irrational).
The crux of what people like myself say is “take the tool, use it, stop believing it’s something more than it is because it absolutely it is not”.
To have any semblance on topical discussion: I claim that we have zero intelligent machines today. No intelligence. No reasoning.
I owe no proof for refusing to believe something I cannot find even if I tried very hard to find it (I mean, who wouldn’t want their job being done by agents they pay $20 - $100 a month for?). You are the one owing proof to me if you say LLMs are “intelligent” and that they do “reasoning”. The burden of proof is on the one making the claim that something exists.
So go on. Prove it. And no “according to this scientist X and the other Y who, like all of us, have zero clue what intelligence and reasoning actually are but have some credentials so they love to pretend that they know”. No – nothing of that, please. Hard facts. Let’s see them.
An easy prediction to make since… I still think I’m right. I mean, why wouldn’t I?
We must have a pretty different reading of their posts. They make some extreme assertions about LLMs being useless and a scam.
But I have no problem with that? I have a problem with saying that if someone thinks LLMs are useful then they are irrational and guilty of religious thinking.
I literally don’t care. I’ve argued only against the idea that anyone who disagrees is irrational or religious thinking.
They are the ones who made positive claims. I have never made a positive claim. I owe nothing other than a rational response to their claims. The fact that you seem to not understand this leads to my next point.
Look, I’m genuinely sorry but I don’t think you or the other poster know what words like “fact”, “evidence”, or “reasoning” even mean. I’ve realized how fruitless it is to try to talk to people about things like this because I just don’t think an internet forum is the right place to teach someone what these words mean.
I see. Well, if your only goal was to make your parent poster less extreme then cool. I kind of thought that you were going into the other extreme: “LLMs are the future” and “LLMs are AGI” etc. bullcrap.
Nope. I never said anything about LLMs other than that one can rationally disagree with the parent poster’s view on them without being irrational and “religious thinking”.
My thoughts on LLMs are not reflected by any of these posts, or are barely reflected. They made positive assertions that were extreme and, frankly, ridiculous. They failed to justify them whatsoever. I presented rational theories that would reject them, which is all that is necessary to refute the idea that anyone disagreeing is irrational.
For this, I am apparently trolling.
I’m not sure I get the jump from LLMs to Unix
Are you saying LLMs are pushing Unix or something?
Anyway, I claim that the general Unix / Plan 9 / Web / “REST” architecture is fundamental, not accidental. I wrote many words on that:
https://www.oilshell.org/blog/tags.html?tag=software-architecture#software-architecture
The details of Unix, like say
stat()and terminals, are not fundamental, but the general architecture is. It could have been Plan 9, which is basically a “cleaned up” Unix in my mind (“everything is a file”, etc.)It’s a mathematical argument with respect to the amount of software that has to be written. Narrow waists are a way of mitigating combinatorial explosion / enabling interoperability.
This argument goes back to a thread from a few years ago:
https://lobste.rs/s/vl9o4z/case_against_text_protocols
The blog posts were basically a big elaboration on my reply to that comment
(As an aside, I think that @matklad has come around to the viewpoint that Emacs/Vim are also fundamental – i.e. they are centered around the narrow waist of text, or attributed text. As opposed to having different programs that edit each different type of data. Again, this is a mathematical issue with the amount of software that has to be written by us, and learned by users.)
With regard to C – I will again claim that if you look at semantics / expressible programs, then C++, Zig, and Rust are a form of C. They all use LLVM, or started out using it.
People argued with Bjarne for years about this – why does C++ use the C machine model? Isn’t there something better?
And a design principle of C++ is that “it does not leave room for any language below it”. This has actually been true!
The main new language I know of that does anything different is Mojo. Mojo is built on MLIR, which is inherently more parallel. It has a different machine model.
All of the LLVM languages are “C-ish” in my mind, just like Plan 9 and even the web are Unix-ish. The addition of types and metaprogramming is absolutely a big deal, but they are additions rather than fundamental changes, in my mind.
It’s easy to tear down what exists – it’s harder to say what the alternative is. Whenever I read these rants, I welcome the author to build an alternative and show us :-) Or even just share a link to something relevant
The structure of the comment was: LLMs show a disturbing fact that some people in our field have religious levels of faith in things regardless of evidence. We see something that seems similar with Unix where people assume that it is inevitable, but that’s not the same, it’s just ignorance.
The bullet points they provide about Unix aren’t what you’re talking about at all. They’re not talking about narrow waists and text as a universal exchange. Consider files in a hierarchical filesystem (and I’d add: a distinction between file and data in memory). That is certainly not fundamental. The AS/400 is the clear, commercially successful counterexample for the points they brought up.
OK, I guess I’ll just repeat: show me what you’re talking about
Maybe write a blog post :-) Or just drop a link
As it is, it reads like “I have secret, fundamental knowledge that other people don’t have, but I am only going to write about it in the negative, not in the positive. Also, I am not necessarily going to do anything about it.”
I still don’t see any connection to LLMs. I guess the connection is “other people are wrong” ? Who are these people? I think there are different motivations at play, like talking your book, or simply liking things that help you get work done
Unix was just an example. People tend to believe all kinds of strange things in technology and treating such things as deity-ordained facts, but this is due to ignorance.
LLMs are so good at bullshitting humans into believing they are sapient, thinking machines; similarly the success of Unix has led people to believe all OSes are based on files. When I say believe, I do not mean in the assume sense, but in the fundamental, faith sense.
Note that the comment you just replied to is by @madhadron, who is different from the author of that first comment, @lproven. I think you meant your repeat of “show me what you’re talking about” to be directed at @lproven.
I took it to be so, but I do not know what I am supposed to do.
Personally I think the BCS article that we’re discussing makes the case pretty well.
LLMs are not “AI” and they are not a pathway to “AI”. They are a dead end: a clever linguistic trick that may have uses in translations and things, but nothing much more. The current hype is a pure unsupported bubble, it will collapse soon and we’ll probably have another AI winter. Which one, I lose count; the 3rd?
You got way too defensive here, as if being mandated to defend… something. Maybe the UNIX paradigm?
Others people already told you but the comment was basically using analogies that people get entangled in certain technical stacks or get invested in tech in a certain way so much that they can’t see past it and start evangelizing it… like the actual religious faith. That was all really.
I gave you the AS/400 as a counterexample. I’m not willing to write a blog post on it because 1) I am not an expert on the platform and 2) there is already a large amount out there about it.
The connection is a contrast: the relationship of many people with LLMs is different than what we often see in technology, such as Unix, because it’s not just parochial ignorance, it’s irrational faith.
What’s good about AS/400 ?
Correct. Thank you.
It is very interesting to me to find that while I usually can’t understand what @andyc says in his writing, he also can’t understand me.
I don’t know what to do about it, but it’s fascinating.
Some serious alternatives that “didn’t make it”:
Again I claim that “Unix” [1] having won is fundamental, not accidental, because the design of the other ecosystems doesn’t enable enough end-user functionality to be created. You have to write too much software.
[1] Windows being essentially a superset of Unix at this point; there isn’t significant software that runs on Unix but not Windows
C is just a slightly higher level zero-cost abstraction over Von Neumann CPU and I would argue there isn’t really any other practical/good alternative abstraction to come up with. In that sense all of: C, C++, Rust, Zig build/build on the same abstraction.
I think you’re missing a lot there. Flat memory is a pretty poor abstraction these days, when even a phone is a NUMA device. That’s a C abstraction that is not present in the hardware. There’s a lot of interesting recent research (and some commercial products from the 1960s and ‘70s) on object-based addressing. These ideas are hard to adopt in C though and will probably never see mainstream deployment as a result.
Similarly, hardware is message-passing all the way down. It uses this to build cache coherency protocols, which it then uses to build a shared-memory abstraction, because that’s what the C abstract machine demands. This costs a lot of power.
On a modern CPU, the amount of area dedicated to executing instructions is surprisingly low. The biggest power draw comes from the register rename engine, which (along with the instruction scheduler and all of the speculative execution machinery) exists solely to allow a massively parallel system to pretend to be a sequential one. This is required to provide a C-like abstract machine.
So, yes, C is a thin abstraction layer over a PDP-11, which is a Von Neumann CPU that made a few choices about memory, but modern CPUs look almost nothing like Von Neumann machines. They can emulate one, but they burn vast amounts of power doing so.
What programming model would map more naturally to a modern cpu not emulating a pdp 11?
Good question. Something with no shared mutable state would be easier to scale. The caches for shared immutable data and exclusive mutable data are easy to build, it’s the shared and mutable that’s difficult. Similarly, something that has high degree of parallelism is easy. If you don’t do speculative execution, you need a lot of runnable threads to keep execution units full. Some of this might be easier to do with hardware support for creating thread-like things. For example, map-like functionality might be easy to implement, where you could do something not quite SIMT, which would help with instruction-cache usage.
Unfortunately, this hasn’t had nearly as much research as it deserves because you can’t run existing code fast on such a system and co-designing languages and hardware is really expensive. The early CHERI research cost around $10m, the total has cost $250m or so. And our changes he abstract machine were very small: we intentionally made something that was easy to map a C abstract machine onto.
Doing a proper co-design project to build a good language for high-performance hardware would probably cost around a billion.
We’ve started to explore some of these ideas in Verona. The really hard thing is to build something that can run well on current hardware but an order of magnitude faster on hypothetical more efficient future hardware, to then enable the development of that hardware.
I am not a hardware guy but I know I would work hard and passionately about making OS-es and software for hardware systems like this! I always felt shared + mutable was the easy way out and then everybody started preaching because “X million people can’t be wrong”.
(late reply) The big data frameworks derived from MapReduce and Spark are all “functional programming in the large” – there are at least 10 or 20 of them, and many research systems
A cluster of PDP-11’s connected by networks doesn’t look like a PDP-11 - it’s a different machine model (in particular, it’s not a state machine)
A multi-core PDP-11 doesn’t look like a PDP-11 - it’s a different machine model
The functional properties like idempotence, commutativity, and associativity enable fault tolerance and parallelism. It has worked in practice
You can also look at research like Sequoia: Programming the Memory Hierarchy
https://ieeexplore.ieee.org/document/4090178
(IIRC it is a functional language; I don’t remember the use cases being that diverse)
Importantly, this observation doesn’t make imperative programming obsolete. The low level is still imperative, but the high level is functional / declarative / graph-based.
That’s basically the idea behind Hay - https://www.oilshell.org/release/latest/doc/hay.html
I don’t find it relevant. C just does not abstract over memory organization, and that’s OK. On 8bit computers they had bank switching, on x86 memory segmenation, all bigger modern CPUs have virtual memory etc. You could and can still use any of these languages to write software for machines like that. In embedded you’re often responsible for page tables, TLB mgmt, coherence, flushing, etc. The CPU/asm/C-like-language only cares about memory being addressable.
Well, nowadays there’s “Provenance” in Rust (and a bit of if in C), so there’s some stuff going on there in between the levels of abstraction, but that’s about it.
Smalltalk always seemed like the great thing so ahead of its time that it couldn’t be recognized for how great it was, even though it probably wasn’t as great as it was made out to be. Still, I feel like I missed out on something special.
Years ago I got to have lunch with an old Smalltalker who held up four fingers and said something along the lines of, “FOUR KEYWORDS. Smalltalk had FOUR keywords and you could even do things like mess with the stack if you wanted.” Wikipedia claims six, but it’s still fascinating how much power they crammed into that language. I keep forgetting to make time to try out Pharo.
My understanding of Smalltalk was the image was “live”, but it always felt like Docker was a ghostly shadow of that language that we try to emulate.
Smalltalk was originally created as a bet that you can fully specify a useful language on a single piece of US letter paper.
According to Wikipedia, the six keywords are:
I’m not sure that any of these are actually keywords. True and False are class names. These classes are used as singletons and they implement methods like
ifTrue:for conditionals. Nil is also a class, which implements a fallback method that does nothing and returns self, so any message sent to Nil does nothing. You can seriously mess up a Smalltalk system by redefining some of these methods, but you can also do fun things, such as add instrumentation toifTrue:to trace program behaviour, or do this in a subclass and return a tracing True to see what things use the result of a Boolean.Both self and thisContext are local variable names, the first is an implicit argument the same is the name of the current context. They’re reserved identifiers, not keywords. They don’t introduce new semantics, they’re just the names that the language gives for a couple of local variables that aren’t explicitly named. As I recall, you can assign to both of them (I’ve no idea what happens if you assign to thisContext, probably explosions).
I think
supermight be something I’d count as a real keyword. It is kind-of an alias forself, but uses a different dispatch mechanism for message sending (start search at a specific class, rather than the dynamic type of the receiver).True, False and UndefinedObject are singleton classes; their instances true, false and nil are special cases in the VM and the bytecode for efficiency but otherwise they could be implemented as pretty vanilla global variables. Preventing them from being assigned to could be done outside the VM, within the SystemDictionary class for example, so they’re not keywords in my opinion. The fact that they’re treated as keywords is an implementation detail of the compiler.
On the other hand you can’t assign to either self or thisContext, as those are not normal local variables. I would say that thisContext and super are definitely keywords, and self is too important to not be considered one.
Welcome! It’s been a while!
Huh, in Objective-C, it’s quite common to assign to self, I forgot you couldn’t do that in Smalltalk. It’s something you often do with factory instances, but Smalltalk typically doesn’t separate allocation and initialisation, so I guess it’s less relevant there. The most common Objective-C idiom that assigns to self is for +alloc to return a singleton placeholder that you then call some -init-family method on. Depending on the arguments, it will then do
self = [SomeConcreteSubclass alloc]and then the rest of the initialisation.I can see treating thisContext as a special case, but to me it’s just a predefined name for the local variable that contains a reference to the current activation record. I thought some of the coroutine / green thread things worked by assigning to thisContext (and capturing the old one), but possibly they use some other primitive methods.
If you include implementation details, most Smalltalk VMs have a bunch of primitive methods (I don’t remember if yours did?) that will always be direct-dispatched to real things. Some bits of the standard library (byte arrays and so on) can’t be implemented efficiently in pure Smalltalk, so you can either provide these as classes implemented elsewhere, or provide some core parts that can be wrapped in mostly Smalltalk classes. You could regard the primitive method names as keywords, but they’re not part of the language, just implementation details.
Hmm, now I want to implement Smalltalk again. We have a tiny JavaScript engine ported to CHERIoT that uses a fairly Smalltalk-like bytecode, I bet something Blue Book-like could fit in a similar amount of code.
Edit: I just checked the Blue Book and the primitive methods are specified there, though in the implementation part.
Thanks :)
Local variables are effectively “thisContext at: n” (with the stack above the local variables). Maybe I am tainted by actually implementing the VM but I see thisContext more as a VM “register” than as a local variables—even more so than self.
It’s been a while and I don’t remember exactly how you did coroutines in GNU Smalltalk. I think there was some kind of Continuation class that was a wrapper for the context and had a (non-Blue Book) call-cc primitive. [found: https://git.savannah.gnu.org/cgit/smalltalk.git/commit/?id=44c6e4445cbb04466 was the commit where the thisContext-based implementation of Continuation was turned into a primitive for speed. The primitive code can be written in Smalltalk but it wouldn’t assign to thisContext; rather it would assign to the parent of thisContext, which does switch to a different context but without assignments. There were no high level coroutines; only generators as in Python but without a way to send a value back to the generator]
Green threads are in the blue book and they are different; they are cooperative for CPU bound code but also preemptive. Priority lists are in a class known to the VM and preemption could be triggered either by hand (“Processor yield”) or by the VM signaling a semaphore (via the signal method on Semaphores of course, but also on a timeout expiring; GNU Smalltalk added SIGIO and single stepping). Each Process class has its own context which becomes thisContext when switching to that process.
I don’t see a way to downvote it, so I’m leaving this comment to register my strong disagreement about the LLM comments above. Most of what the comment says about LLMs strikes me as wrong, overconfident, and poorly framed.
It is an intentional feature such that people don’t content themselves with unconstructively attacking others and instead contribute to discussions meaningfully. Maybe you should try it by explaining what it is you disagree with. If you don’t care to do that it is fine, but you should then take the advice of the site guidelines and just let the person you disagree with be wrong. There is no need to register your discontent, nobody is keeping track.
Was this the intention behind the feature? How do you know?
In any case, I will grant this sounds like a good intention. But the road to Hackers News is paved with good intentions.
Let’s flip the argument on its head. If one suggests downvoting enables people to not engage meaningfully, the same must be said for upvoting, does it not? Clicking an arrow, whether it be up or down, seems about the same w.r.t. “meaningful discussion”.
There are many other designs available out in the wild. I encourage people to be dissatisfied with the status quo here. Most forums are terrible. Lobsters is, luckily, just a little bit less terrible. But it hardly lives up to its potential. If you go over to LessWrong, you can get some ideas of what is possible, which include more nuanced voting. You can, for instance, independently answer:
This distinction is key. It allows the community to register sentiments such as “This is written clearly and in good faith, but I still disagree.”
Because, I took part in the bikeshedding of this 10 years ago.
The following is over a decade of discussions on the subject, if you’d like to see what people were thinking and saying about this at different times:
Truly, there is nothing new under the sun.
Hrm, I mean the About page is right there …
I believe it’s been discussed in some meta threads through the years.
Slashdot/Reddit style upvotes / downvotes systems have multiple issues that people have observed through the years. Specifically, a downvote is just one bit - it’s impossible to convey whether it’s because the downvotes believes the comment is incorrect, a troll, or just doesn’t like the commenter.
This site is a off-shoot of HN, created to address the many issues the creator @jcs observed there. HN also has downvotes. So this is a flippant comment that I’m surprised a site member since 4 years would make.
If you want a LessWrong style multi-axis voting system, the source for this site is public, and a pull request to implement it would be welcomed.
Yes, you correctly identify many of the problems with one-bit downvote systems. My reply, in short, was saying: much of the same kind of problem applies to one-bit upvotes.
I understand if my joke wasn’t funny, but it wasn’t flippant. Designing good communities requires a lot more than intentions.
My comment was a little bit of a test balloon. I think it would take significant leadership and sustained persuasion to make it happen. I’ll think about some ways to give it a try.
I’m just surprised you’ve been a member for 4 years and are only now discovering the mechanics of how voting and flagging works.
This community has evolved in large part due to the feedback from its members. If you wish to change it, maybe a bit more humility is in order.
Thanks for the conversation. Why do you guess I’m not humble, if you do? Or is it more about my style?
Some people conflate / confuse vigor and advocacy with a lack of humility. I wonder if you agree with this: A person can push back directly and persistently while still being humble. Asking a lot of questions and having strong opinions doesn’t imply a lack of humility.
Here are three things I know about myself (as do people who know me). First, I don’t think {community building, communication norms, interaction design} are easy. Second, I know I’m not capable of making broad changes by myself. Third, I am open to feedback.
I think a possible undercurrent is differences in communication styles, culture, and experiences. I’ve lived in many places across North America, and to a lesser extent, spent time in different countries in the world. I’ve seen variations in directness, passive aggression, confidence, overconfidence, humility, false humility.*
Another possible factor is that some people conflate intellectual pushback with a lack of humility. I’ll tell you this, If I dish it out, I better be able to take it. My goal is not to tear people down. My goal is to tear down weak reasoning and poor framings and replace them with stronger stuff. Some people can’t handle this, but I intentionally try to err on the side of optimism.
In my experience the topics of {collective action, cultural design, communication norms, consensus building} are widely misunderstood in technical circles. I voice my point of view as clearly as I can, and I welcome disagreement and clarification. Sometimes people take this in unintended ways. Some people are inclined to give me advice. All good. Communication is hard.
* I think many people can benefit by thinking more about the topics raised in The Proper Use of Humility. Have you thought about the questions raised there?
I’ve replied at length in multiple granular comments. Did you look first?
I think truth matters. I hope some people are keeping track.
It would’ve been easier for me to notice that if you didn’t make a half dozen separate comments in response to the same single comment, the first of which being you just vaguely complaining you disagreed.
Using granular discussion, point by point, works better for complex questions. Taking it slow and methodically is better for truth seeking. The criteria isn’t what one person says it easier. A better criteria is what results in high quality discussion from many people.
On the writing side, it is easier for one person to write one rambling message. It is harder to write compact, separable points which invite detailed and substantive discussion. Think about it? Try it? Let me know what you think.
I accept this criticism as fair. I’ll go back and hyperlink each of my specific criticisms to that comment, if I can.
There’s no downvoting on lobsters, you can only “flag” something as basically breaking the rules or upvote it.
Lest people think I did not also engage substantively, here are four of my specific criticisms:
I welcome specific and focused comments to any/all. I split the comments into granular chunks intentionally. I explained my reasoning in these two comments: 1 2.
I registered general disagreement because I found the comment to be egregiously misguided. When I see otherwise informed and capable people led astray, I say something, because I care. I’ve been civil about it. The comment was well-written in the sense that it persuaded a lot of people, as measured by upvotes. (I couldn’t see the net upvotes-versus-downvotes because of the information architecture here.) It used a lot of rhetoric; it sounded confident. And it was quite misleading (at best) or wrong (at worse) about LLMs.
Why do I care? AI – actually AI Safety in particular – is a big deal to me, and I’m trying to do what I can to root out misunderstandings. I assure you I take this topic and its implications very seriously, possibly as seriously as anyone here.
Everything we do here is folklore.
First, using the words intelligent and think without definitions is asking for misunderstandings.
Second, the quote implies some sort of dividing line between “intelligent” and “not intelligent”. That’s misguided.
Third, like many, I find value in the definition of intelligence as the ability of an agent to accomplish some task. This makes it situated and contextual. Defining it this way helps clarify discussion.
I like your definition quite a bit because I think it captures a lot of nuance in a practical way. That said, I have a pair of potential counter-examples to consider from my day-to-day work that I’m struggling to classify as intelligent or not using that definition. For context, I work with autonomous aircraft and in a previous life worked in a biology lab that was studying insect flight characteristics.
The first example is the flight control software itself: given a series of waypoints, fly the aircraft along that trajectory as best as you can. I’m going to reveal a dirty little secret here: most of it is just nested PID loops. For quadrotors, they’re typically:
This accomplishes a task and to a human observer actually looks quite intelligent. If there’s a gust of wind that knocks you off-course then the position PID loop will have an increased error term that ripples down through the stack to turn into motor commands that get it back on course. Is this intelligence? It accomplishes a task even in the face of adverse inputs. But under the hood it’s basically just a bunch of integrators trying to push the error terms to 0.
The second example is closely related: Kalman filters for processing noisy sensor inputs. Over time these are building up a statistical model of how a sensor is behaving and then making decisions on every sensor reading on how much that reading aught to be trusted. It’s an iterative model; each sensor reading gets assessed for trustworthiness and is also used for updating the internal statistical model of how the sensor is performing. It’s pretty much just a rote linear algebra equation each iteration though. Is that intelligent? It almost looks magical when it works well!
The last part… why I mentioned the insect research back in undergrad. One of the experiments I helped with was building a flight simulator for locusts (bizarre, right?). Forward velocity and turning through the simulator was accomplished by inserting electrodes into the insect’s wing muscles and measuring the timing of the muscle activations. If they activated simultaneously the insect should fly straight, if the right wing fired before the left then it would induce a turn. Once we’d established that the simulation was accurate enough that the insect could fly around in our crude world without colliding with objects (what?!?), the biologists hooked up additional electrodes to a single specific neuron called the Descending Contralateral Motion Detector, which was connected pretty much directly from the insect’s eyes to the muscle control system (I’m not a biologist…). What we observed was a series of electrical pulses from this neuron that were directly correlated with the wing EMG asymmetry: if the DCMD was reporting that a collision was imminent, the muscles would fire to turn away from the obstacle.
Is that intelligent? It enables amazing collision-free swarming behaviour. But in some ways it’s even simpler than the drone FCS… and I’m not sure I’m comfortable calling PID loops intelligent?
Fun examples!
Yes, for certain environments, the right sensors and actuators in a PID loop will get the job done reliably. If that setup serves an agent well, I would call that intelligent behavior.
The word intelligence is a fascinating mess. For example, many people look at a world-class chess player with awe and suggest they are outliers in intelligence. By the definition above, if they win at chess given their environmental constraints, they are intelligent at chess. But from another point of view, much of what chess players do is rooted in pattern matching and tree search. When you frame it this way, it doesn’t seem that “intelligent” does it?
It seems to me you are talking about more general forms of intelligence. The degree to which an agent’s intelligence transfers to other contexts reflects its generality or adaptivity.
“in any way”? This is hyperbole, right? I cannot take it seriously.
LLMs can indeed output language that is consistent with logical reasoning. The trend is upward.
What claim are you making, then?
I figure this is the claim - LLMs can output language that is consistent with logical reasoning. LLMs can simulate reasoning better than any prior reasoning-simulation software. However, they do not actually perform reasoning; their reasoning is performative, because the output is algorithmic (and algorithmic in a way that human reasoning is not).
If a system outputs text that consistently matches logical reasoning to a high standard, isn’t the simplest explanation that it is indeed reasoning?
(I’m not asking questions of consciousness or personhood or any of that.)
Perhaps you would say these systems are not using maximally simple, purely logical, circuits / structures / mechanisms that only do logical reasoning, such as forward-chaining or RETE or whatever?
If so, how are humans different? At best, humans use their handwritten symbols to make logical proofs according to set rules. But we still use complex and often flawed lower-level mechanisms (brains) to do this logic. In this sense, I think your claims about performative reasoning are moot.
no, because the Chinese Room isn’t reasoning either.
LLMs are collections of things that humans have said before, boiled down into statistics. The sole job of an LLM is to output words based on statistical locational frequency, that is not, and cannot be, reasoning. It’s incredibly provable that LLMs cannot actually reason, too, and there are multiple papers on it — see https://arxiv.org/abs/2406.02061 and also https://link.springer.com/article/10.1007/s10676-024-09775-5
I think the comment above misrepresents or misunderstands the Chinese Room argument. Per Wikipedia:
Searle’s point about the Chinese Room is not about if a machine can turn the crank and do modus ponens or other reasoning rules. (This has been settled for a long time.)
The C.R. highlights the question of qualia, of inner experience. I’m not talking about consciousness, “thinking” or a “mind”. I’m talking about something measurable: can an AI generate correct conclusions according to the rules of logic? (For example, they do LSAT level reasoning quite well, and they are getting better.)
Ah. Your claim boils down to the fact that the low level mechanism of token generation is not identical to reasoning. Am I representing your point of view fairly?
There was also a time when I fixated on this definition. I get it.
I hope you realize that when some people say LLMs “can reason”, they aren’t talking about the mechanics of token generation process. They are talking about higher level behavior.
I think it is also important to realize that by your definition, one could argue that humans don’t reason either. We’re just neural impulses firing. I hope you take my point.
Nothing can analyze and improve itself? What? Am I misunderstanding something here?
LLM bots cannot analyse and improve themselves, because they cannot analyse – anything ever – and they cannot improve on what was in their training corpus.
All claims to the contrary are snake oil.
I can see why you’d think that LLM bots cannot analyse and improve themselves. What really confuses me is that you say “nothing can”.
Are you saying that humans cannot analyse and improve themselves? Because I think plenty of humans do that after, for example, failing at something many times in a row, or reading a self-help book.
Or by “nothing”, do you mean only things that aren’t alive? If you do, consider the hypothetical case that someone wrote a (non-LLM-based) program that perfectly simulated a human’s behavior (human-like words, leg movements, etc.). The program was even observed to change its outputs after interacting with a simulation of a self-help book. If you agreed that humans can analyse and improve themselves and you had this definition of “nothing”, I think you would have to be making one of these three assertions:
LLMs are, by definition, vast databases of multi-dimensional weights of relationships between symbols.
An LLM can’t analyse an LLM, including itself, and nothing else can analyse an LLM either.
Claims of redesigning LLM bots to hallucinate less, for instance, are lies, because the databases are not amenable to study or analysis. And, of course, hallucination is what they do. There is no distinction between making good and desirable stuff up and making bad stuff up.
They are lies, just as software companies claiming they have rewritten huge codebases for security. In most cases, they’ve just done automated scans, maybe a few people looked at a few especially critical routines.
Real rewrites are very rare and usually disastrous, as Joel Spolsky observed about 15 or 20 years ago.
My comment was about your claim that “nothing can” “analyse and improve themselves”, a claim I disagree with. What you say about LLM analysis may be true, but that’s irrelevant, because I was not arguing that LLMs are an example of a thing that can analyse and improve themselves.
With my comment, I was hoping to pinpoint where, if anywhere, we disagree about the statement “nothing can analyse and improve themselves”. You can do that by answering these questions:
I just thought of a possible explanation for your response being weirdly focused on LLMs instead of on the questions I had posed. Looking at your original wording again:
Did you intend that last sentence to be interpreted like this?
If so, then that’s very different from the interpretation that @insanitybit, @rele, and I had. insanitybit’s comment and rele’s comment both rephrased your statement as follows:
If that statement does not capture your meaning, then it is unfortunate that your replies to both insanitybit and rele’s comments, which contained that rephrasing, did not correct the misunderstanding.
Correct.
Regarding exaggerations and lies about what LLMs are capable of in the near-term… Yes. There is hucksterism in AI. I know the arguments; I’ve read Kapoor and Narayanan, which I generally recommend. I’ve also read Gary Marcus, who makes some good points but gets too ranty for my tastes.
However, regarding the comment here above, there are claims here about what is and is not possible, such as:
If we define analysis as “detailed examination of the elements or structure of something”, then, yes, an LLM can analyze its own weights and structure.
If you say “but LLM X can’t analyze itself”, that is a different claim – a claim about implementation not about what is possible.
If you say “but it can’t analyze itself up to some quality standard”, that is a different claim, and it invites discussion about the bar and what progress is being made towards it. This kind of discussion might be interesting; it invites us here to dig into the details.
This is a tall claim. To me, it is so bold that I wonder if it is designed to attract attention. I’ll bite: I ask the author to (a) clarify their terms and either (b) prove this claim or (c) make a testable prediction, on the record.
An LLM cannot analyze? Are we having just misunderstanding each other – i.e. having semantic confusion?
Maybe these questions will help:
What do you mean by “what they do”?
What papers on LLM scaling laws have you read? I suggest we ground our conversation there. The papers show improvement across many metrics. Bigger models perform better in almost all ways*.
Do you want to refine your claim?
* Bigger models can be more deceptive, which is unfortunate if you care about alignment (which you should). But increasing deceptive ability is most certainly an improvement in capability.
I am not interested in radical new methods for estimating with unprecedented accuracy the numbers of angels dancing on entire fabric-fastener manufacturing units. I don’t care, because there are not such things as angels.
I find it highly amusing that saying “this type of software is not intelligent” provokes responses which attempt to nail down a possibly negotiable definition of intelligence. Again: this is not a rebuttal, in my view.
You try to split hairs, perhaps so that you can cut each one, but I’m not here for a haircut. I am not here to persuade anyone. I don’t see any potential gain in even trying.
LLMs are a fascinating technology which may in time enable applications such as simultaneous translators, perhaps not only between human languages but between computer languages as well, which could have amazing consequences. I speculated on some of those here but I remain extremely sceptical about the entire current industry, as I said in the earlier article that the above is a follow-on to.
But they are not artificial by any reasonably strict definition, they are not intelligent by any definition that isn’t a fantasy, and because they are not in any way what they are depicted as being, they will never lead to anything that is.
It is empty hype, nothing more, and I sincerely hope it collapses soon and that it destroys the careers of the hucksters selling these lies about thinking computers.
Here’s what I notice. First, you are obviously bothered by hucksterism in AI. (So am I, by the way.)
Second, this is a bit of a guess, but you seem to take my pushback personally. And then you channel a lot of negativity towards me. I’m not the problem. The key problem is these topics are complicated, communication is hard, and we all have egos. Very few people want to be wrong, much less shown to be wrong.
Third, you write at length, but haven’t answered the specific questions I ask. You sometimes wave them off. Sometimes you ignore them altogether. You also use a lot of rhetorical devices.
On the substance: Are you claiming LLMs are not getting better as they get bigger? If so, by what metric? (I ask for it to be clearly defined and measurable.)
Please respond to what I’m writing, not about what some other person says.
Again with the demands for metrics. No, I cannot give you metrics, because as I have said already, I do not care. You are claiming the numbers of ratio of dancing angels to pins is important: I’m saying there are no angels. No angels means no counting means no numbers.
Yes, they have, so far, provider better results – more plausible-looking textual or image based output – with greater size; however, it is not just foolish but downright stupid to blindly assume this will continue indefinitely forever.
Secondly, the bigger the model, the bigger the power and storage use of this technology which is already environmentally catastrophic. The boosters who keep calling for bigger models are guilty of ecologically criminal acts. You are entirely as culpable as the cryptocurrency promoters.
I have not said anything about such improvements continuing forever (neither in this thread or sibling threads, nor anywhere I can recall.) I strive to not blindly assume anything.
Please stop attacking me. Such attacks are not welcome here.
And, to be clear, I have not called for bigger models. (And if I had, would such attacks be justified and appropriate here? It is hard to listen and learn when you are attacking and accusing.)
You may perceive them as demands, but this does not make them demands. I have asked questions. Repeating a question doesn’t make it a demand. We have both attempted to redirect the conversation, such is the nature of conversation. You call my attempted redirection a “demand”. To my eye, this is probably either (a) the result of you being upset for other reasons and/or (b) a rhetorical device to make me seem unreasonable.
This is an uncharitable description of what I’m asking.
When I said “rhetoric” before, this is the kind of thing to which I was referring. It raises the temperature but doesn’t promote mutual understanding.
Overall, this conversation has been unnecessarily fraught and combative. I have strived to disagree with you without being disagreeable. I’m sorry if you think my point-by-point commentary is mean or in bad faith. I do not intend any of these things, and I’m sorry if I’ve offended you somehow.
It would seem you have labeled me in some negative way, such as being an AI booster or ecologically irresponsible. I don’t see the logic in why you think these labels apply to me. And again, even if they did, being unkind about it isn’t necessary or welcome.
Whatever you feel, I would ask you to stop taking it out on me. It seems you are making assumptions about my point of view or my impact on the world. I have no doubt that your point of view is valuable, but I would ask that you be more civil about it.
you do realise that the effective altruism cult mannerisms are incredibly tedious to anyone not involved in the cult’s way of framing reality?
I am not sure what you mean. Based on your comment, you sound angry about certain mannerisms and/or beliefs? Such as?
If you want to offer some specifics and are willing to discuss charitably rather than dismissively, I’ll put in some additional effort. No need to be mean about it, please.
I think it’s fair to say the context here is about artificial intelligence and some common misunderstandings. Unless I’ve lost track of the parent comment, I think this all started when one person made a list of common UNIX misconceptions and then offered various AI misconceptions. My comments, taken together, point out how many of these claimed AI misconceptions are actually misconceptions themselves.
Many people were far from calm. The conversation has the usual signs of people overreacting to each other. Even relatively neutral people got attacked, because one side or the other accused them of being on the wrong side. It has been a case study of polarization and poor community discussion norms. Truly pathetic, as in I have pity for how we all got here.
These discussions need to get clear and as specific as possible if we want to make sense of the world. I hope you can appreciate this.
Are clarity and precision and rigor tedious to you? If true, this discussion might not be worth your time? Truth seeking isn’t “sexy”, nor does it provide the dopamine hit of leaving a nasty comment. It doesn’t make it easy to form a tribe of “us versus them” because anything is subject to scrutiny and revision. These are the values of critical thinking and intellectual honesty.
I flagged this as unkind, it’s probably trolling too? I don’t know. I figured I’d let you know. Calling “effective altruism” a cult is definitely hyperbolic and clearly intended to be insulting.
I know that some online communities dislike some members or something like that but it’s not a cult, it’s just an ethical framework, and the user doesn’t even seem to be advocating for it. I don’t know what mannerisms you’re even referring to, personally, but this seems like a silly and overly negative interpretation of a post that’s basically “please stop insulting me and engage with my points, I am willing to engage with yours”.
your consistent appetite for tone policing was noted, thanks.
Nothing to do with tone, I just think your comment was bad. As I explained.
if you insist, we can continue.
your explanation has been given a consideration it deserves, and found wanting. specifically, the following statements are wrong: “calling ‘effective altruism’ a cult is definitely hyperbolic” (it isn’t); “it’s not a cult” (effective altruism movement, especially its californian variety, has many characteristics of a cult), “it’s just an ethical framework” (no, it’s not just an ethical framework).
if you don’t know “what mannerisms i’m even referring to”, you have no basis to claim that “it seems like a silly and overly negative interpretation of a post” (and my reply wasn’t referring to a single post).
to spare your precious time, i will have tagged my reply as unkind myself.
Oh that is an AMAZING research question. Thanks so much for linking this, there’s no way I’d have seen it otherwise and it’s highly relevant to my interests.
You are welcome. It is a very interesting topic. I love Prolog, but I think the execution model is a bit problematic sometimes. I think alternative techniques such as answer-set programming or constraint logic programming don’t get sufficient exposure.
The author of this article has a reading comprehension problem. The linked story about .yu domain does not remotely corroborate his summary. As someone who studied at University of Ljubljana at that time right across the street and never heard anything so wild, which I’m sure I would, I have difficulty believing that it happened as described.
Which parts are you saying it doesn’t corroborate? There are a lot of assertions of fact in it, and also a bunch of subjective stuff.. from a quick read of both pieces, it does look to me like the most important details of the alleged heist are mentioned in the cited source, but I may be failing to understand the significance of something that seems minor enough that it doesn’t occur to me to doubt it, or something like that…?
Where is the Belgrade heist mentioned?
It would be insanely crazy for something like that to happen during ongoing Balkan war and also completely unnecessary since .yu domain was administered from Ljubljana since start in 1989. The only “heist” was some colleagues allegedly breaking into her office, copying software and disconnecting her network. All of them worked in the same building and no computers were moved.
The Dial piece, cited as the source, says:
I agree that this doesn’t say anything as to whether the former colleagues thought of what they did as a “heist”, and it quotes Jerman-Blažič’s opinion that the people who took the software didn’t know how to use it without help, but she wouldn’t have been in a position to see what happened afterwards, since, as the piece also says, the next part was secret. However, the last sentence is clear-cut that the domain was in fact run by ARNES, at least according to this publication. That’s the substance of the “heist” claim, surely?
I’m unable to find anything in the Dial piece that speaks to whether these people worked in the same building at the time of the alleged theft, as you suggest. I may not be looking closely enough. You’re right, that does bear on the claims in the Every piece, which says the alleged thieves traveled there in connection with the alleged theft.
Your point that no computers were moved doesn’t seem relevant to me; the Every piece says “On arrival, they broke into the university and stole all the hosting software and domain records for the .yu top-level domain—everything they needed to seize control”, which notably makes no claim about hardware, only software and data. In general, in any sort of computer-related theft, real or fictional, the software is the important payload… I don’t really see what difference the hardware would make to anyone.
I think these authors are describing the same factual claims, while differing substantially on who they view as the “protagonist”. That may account for why it feels like they’re talking about something different? I do think it would be entirely reasonable to point out that there’s very little information about how the participants really felt about the events at the time, or how they understood the purpose of their actions.
I do think the author of the Every piece goes perhaps a little too far in inferring people’s intentions, and leaves out the important caveat that the copy of the domain that ARNES ran was kept secret. I think perhaps the author doesn’t have a deep understanding of DNS, and incorrectly believes that running one copy of a domain somewhere, privately, implies that nobody else could be running it at the same time. That does seem like a material error, and quite unfortunate, though it’s not about the so-called heist per se.
So… I think I see why you object, although I’d stop short of saying the story was invented from whole cloth. Does that help?
I don’t have an issue with The Dial piece. I definitely made a mistake about servers; no idea why. I guess reading comprehension issues on my part as well.
My comment was exclusively about the Every piece that has a truly awful summary of The Dial piece. I disagree that they are the same factual claims because to me there’s a huge difference between Slovenian academics going to a different country to steal domain and software or essentially an internal dispute around who will continue running the .yu domain, which since its beginning until mid 90s has always been administered from Ljubljana in current Slovenia.
That makes sense, now that I understand the distinction you’re making. It does seem like an important one. Thanks for talking it through and helping me understand.
hmmmm, thanks, interesting read
This is surprising to me because I have arrived at almost the opposite conclusion, being a developer for almost 15 years.
My years spent programming, although satisfying for reasons depicted in the article, were largely bereft of varied human interaction, or contemplation on deeper topics that may touch one’s inner life - things that might be influenced by the arts, philosophy, literature, etc.
I was a student of the humanities (both academically and recreationally) before embarking on my software development career more than a decade ago. In recent years, I have looked back and noticed a stark contrast in the richness of inner life and flexibility in socializing after a decade spent solving problems with code on engineering teams. Some if not most of these teams were excellent, and I look back on my experiences fondly. However today, I do feel somewhat robbed of my time.
there are definitely years that I have regrets about, because of how isolated I was
today I actively try to focus on projects that I will be able to talk to my friends about, and also I’ve cultivated a social group who enjoy discussing each others’ projects
I find software much more enjoyable when it’s a social undertaking. that isn’t automatic, and the article doesn’t really address it. sometimes software is social, sometimes it isn’t.
Maybe it’s just not what you love anymore, or even ever.
Human interaction is often great (when dealing with right kind of people), but programming and SWE is still the thing I love doing, and will never feel like time wasted. And quite the opposite - it allowed me to meet with awesome people, gave me a great life and introduced me to all sorts of adjacent areas of life and study.
From my perspective software is a deeply human thing and programming languages are languages (each with their own sociocultural background as well). I feel robbed of my time with the countless hours spent in meetings or “agile cerimonies”, doing things because of company politics, etc. But programming in itself feels like a very fulfilling and relaxing activity where I can really express myself. Sure not all software is good or interesting, but the same can be said for some human interactions. Moreover I’ve been doing pair programming lately and although I was hesitant at the beginning I’ve started to appreciate it, it really turns software development into a collaborative art
I feel this strongly also. But at the same time, the right people can bring the richness into the technical space. It just requires a subculture that’s fascinated and interested in mixing that richness and contemplation with the technical work. Once I encountered that kind of culture (that’s interested in melding the humanities and the technical), I couldn’t go back to purely technical spaces.
I’m a bit shocked. For me, software is deepest and most human field, because it expands your scope of responsibility. Where 2 centuries ago, a letter would have to be sent by a series of horsemen (or at best, by signalling stations), we’ve abstracted that away. So to many hours of other tasks, which free us to think about what we’re doing in a more profound way and create real solutions. We can identity uniquely human faculties and automate away other parts, creating complex industrial processes to service many customers and improve their lives. Sure, many people and organizations don’t, but we are free to! If you have a problem, you can often modify your workflow or create tools to overcome it. Symbolic and logical/constraint programing reduce a lot of stress planning. Then you can track friends’ birthdays, stories, schedule fun and work tasks and so much more. (If someone says it’s insincere to send an automated birthday message, why do they choose to ignore the love and care for your loved ones which made you craft the thing?)
I’ll write a blog post.
I feel I’m in a similar situation to you — just shy of 15 years’ professional software development; studied (human) language before embarking on that — worked on some really neat teams, but I spent a lot of my time outside work developing those other areas of life. (Transitioning will mostly force that!) These days I have the flexibility to work part-time, so I do, and it means I have the mental space in my life to actually pursue a hobby. It turns out that’s programming, with a side-order of more language study.
The ikea effect hits hard. I usually feel more at home In a objectively worse product I wrote myself than a more polished one. Knowing exactly how it works is just an awesome feeling. But throwing away bad code after having learned is sometimes the right move as well.
for SURE, on all counts
That’s the main downside. Sure, you learn a lot if you build it yourself. But it is very hard to compromise once a mental model has been formed that something “needs” to be done a certain way.
Of course that does not affect all the “build it to learn about it” projects, but it’s probably a also a source of the “Not Invented Here” approach that some folks apply.
“Programming is time well-spent and joyful (when you’re not dealing with bugs)”
So less than 10% of the time spent programming is well-spent and joyful? Or is it just me and everyone else is spending most of their time producing and using working code?
Hm interesting quote – I would disagree
And I’d break it down into 3 situations
(1) Fixing bugs in my own code is definitely time well spent – it’s a lesson for how to improve. How you can improve without reflecting on your mistakes? Bugs are mistakes.
(2) What about bugs in gaps between “your code” and “other code”? e.g. invoking a library the wrong way, because you didn’t understand what it was supposed to do
I’d also treat this as an opportunity to learn about your dependencies / specify their behavior
I frequently write tests for my depenendencies, to try to avoid this kind of bug
(3) Fixing bugs in “other people’s code” often feels the worst, and takes the most time – definitely as a less experienced programmer
Well I’d say, if the bug can be reproduced in an automated way, and you have a working build system … that’s usually 90% of the battle
I wouldn’t necessarily call this time not well spent, but depending on the situation it may not be joyful. But it could be joyful – if the code is otherwise good, and you learn something. (This is often not the case “at work” – in open source there is more code written to a higher standard, although lots of bad code too)
And actually I wouldn’t say I spent 90% of the time dealing with bugs … I think I spent more time testing my own code, and testing my dependencies, so that I don’t have to do that … I do find testing/specification more enjoyable than fixing bugs in hastily written code
Not really a counter-point: I generally enjoy finding and fixing other peoples bugs. Far more than my own. I also know quite a few people who feel the same.
I also like working on old code-bases more than new ones.
Yeah, if I can swoop in and find their bug myself (which is rare), I feel super smart and amazing. What sucks is when other people’s code is buggy and I can’t figure it out, which is quite frequent.
I enjoy debugging………
This is definitely a problem with a lot of programming discourse. Instead of saying “X is hard for Y reasons” we tend to hear a lot of “don’t build X yourself”. The argument is often brought up to prevent people from making mistakes that established solutions had to learn the hard way.
While I very much agree with being aware of the risks associated with things, but I’ve noticed that people are liable to make basic mistakes using established higher level abstractions since they don’t understand how the abstraction works fundamentally.
A typical example is authn/authz. While it’s easy to throw in some middleware to “handle” it, without understanding the concepts behind it it’s often easy to introduce trivial insecurities.
I would go even further to say that sometimes it’s worth your time to build a thing yourself for no other reason than to think through the problem domain, and understand the solution concretely, then throw it away and use a production-ready implementation.
very much agreed
Why don’t people just document mistakes? A book of mistakes that you can make in a programming language would be so helpful.
Yet all we ever hear are success stories.
Although not “mistakes,” per se, I feel like people do write about footguns and gotchas quite a bit. Whenever I’m learning a new technology/language/etc, I spend some early time searching around for gotchas and reading through things to look out for. I recommend it.
I’ve seen one one or two stories posted to lobsters that I suspected were written by a LLM.
Apart from humans, are there any good tools to inspect content and make a judgement as to whether it was generated or not?
Should content generated in whole or part by LLMs be banned from places like lobsters?
the short answer to this is no. (edit to add: that is, no, there are no automated tools for it)
to reliably detect LLM output would require having more training data for the detector than the company that made the LLM had. it’s more profitable to sell LLMs than LLM detectors.
the fact that Google is drowning in spam search results (in my subjective opinion based solely on public information) should be a strong indicator that nobody on the defense side has a clear advantage.
I’m not entirely convinced that “LLM detectors” based on existing techniques could be made reliable even with vast quantities of training data. If you had one, you could use it to adversarially train an LLM to smooth out whatever statistical/syntactic wrinkles the detector was keying upon. If anything could identify the semantic flaws in LLM output, you’d have something resembling the mythical “strong AI” that would make LLMs obsolete, and nobody credibly knows how to do that (not for lack of snake oil sales pitches for it, of course!)
The situation is also muddled by the fact that SEO sites can directly make money for Google, so they have perverse incentives at play across the organization to do a poor job filtering out spam, even if they had the technology. They don’t even bother to give users tools to filter spam domains out of their personal results!
Should bullshit be banned?
In places valued for the quality of their non-bullshit, at least ones which are run and frequented by people who care about that, bullshit should be curated out.
If you check the moderation logs, you’ll see @pushcx regularly deletes articles for being LLM slop and bans sites that repeatedly post them.
In a few years, places like this which actively check if submissions are sufficiently coherent to be authored by humans will be even juicer targets for LLM ingestion operations. That is if there are any submissions left that are authored by humans.
If the content has value I see no reason to care if it’s human or LLM generated with regards to it being submitted to lobsters.
very nice write-up. I have a Framework that’s a year older than this author’s, and have had many of the same experiences. I did try the heavy-hinge update, and I like it; I’ve also replaced a few other parts for minor quality-of-life and cosmetic reasons.
I did at one point over-tighten a screw, and break off a bit of plastic that held the threaded insert it was supposed to mate with. oops. I was happy to be able to easily buy a first-party replacement, but it was pricier than I’d hoped since that piece is sold only as part of a larger assembly. don’t over-tighten the screws.
oh, neat!!!!
I run Signal without a phone (no, this is not a supported configuration), so I don’t hit this problem, but I’m sure this will help other people
How do you do that? signal-cli or the signal in an android container/emulator or what?
the second one. initial setup was challenging and a working phone number was still required.
I would only recommend this to people who are highly confident in their ability to 1) not lose it; 2) not have it stolen.
deeply tragic. I think everyone who was paying attention to linguistics research or to corpora saw this day coming, but that doesn’t make it any less sad now that it has.
Not pertinent to the actual article, but I’m kinda sad that “opening modern web pages” now falls under “excursions to maximum CPU load”.
You could wrote a web page that took 100% in 1998.
But isn’t great that there’s a public, open, non-owned application platform without any vendors acting as gatekeepers? I think it is.
…like most of the readily available free/open-source operating systems?
Yeah, they’re (regrettably) not as ubiquitously deployed, but all else being equal I’ll take a native application over webtech (whether in a browser or a warmed-over repackaged one) 100.0% of the time.
I am glad that the web exists and that it is useful for the many interactions I have with governments and capitalism. I think we are better off with the web than we were before the web. I enjoy building stuff for the web, up to a point.
I also think that any platform which is used for commercial purposes will, over time, be developed in directions which enable and encourage site owners to consume as much as possible of their visitors’ computational resources.
The unfortunate truth is that if, for example, a social media site gets even 1% better subscriber retention by consuming those gigabytes, it’s worth it to them, since the cost in the terms the company cares about is much smaller than that.
This is frustrating to me. I’d like to simply advocate that corporations be encouraged to develop free software against, say, some minimal common-denominator set of Linux functionality (I don’t regard containers as a solution to the bloat problem). However, that would result in Linux taking on the same characteristics that I dislike about the web.
I don’t yet know what to do about this. I hope that stating the problem as I see it will inspire somebody, someday down the line…
oh excellent, I hadn’t heard of this and very much want to try it. Thunderbird works well for me and solves my actual needs, but I have been wanting something terminal-based for a few years. alas, alpine doesn’t have an option to save oauth credentials across boots, and elm’s descendants are no longer maintained…
huh. what a really cool write-up. I’m convinced. now if only web1913 were packaged for nixos’s dictd setup…. wiktionary is, and seems to have built on it, but I’d like to have both. maybe a nice weekend project…