I definitely agree but I would much rather have people be straightforward and label the AI generated bit so I can scroll past if I want than try to subtly integrate it into the article. Obviously all human content would be better, but this feels like the lesser of two evils.
When someone quotes ChatGPT, I know they have not learned enough about the topic to be able to put it into their own words or check if the quoted verbiage is correct.
In this case the author then went on to quote the wrong parts of the disassembly, confirming that my prejudice was right. Saddening, because I like seeing people learning a new thing and writing it up well. But I think this author is learning and I hope that continues.
My experience is that when people quote others they are more actively engaging their brain. Quoting ChatGPT reminds me of the irritatingly sophomoric habit americans have of quoting the dictionary, as if their audience are barely literate. It shows that the author has not read around the subject.
fanf already gave a lot of good points, but I wanna give a couple more.
ChatGPT is always going to generate descriptions that are at least slightly incorrect when they’re that long, and either the author doesn’t know better, or is willfully jettisoning misinformation out into the world. If I, as the reader, don’t know what branch prediction is and I care enough to learn about it, I’m just going to open a new tab and go learn about it for a minute.
Instead of spending the time going to ChatGPT and generating some unverifiable stream-of-un-consciousness, they could have just found an actual knowledgable description written by a human, and they could’ve linked to it, or quoted some small descriptive section of it with a citation.
These AI insertions are not neutral or harmless. They deprive us of actual research.
Ok, I see… If you don’t mind, I would just like to provide a couple of counterpoints:
While it’s true that ChatGPT often makes mistakes, it’s also true that in most cases it does describe things pretty well (as in this case), if not perfectly. Sometimes even better than some experts do (although that’s more rare), if only because of its linguistic skills.
In many cases, especially when you’re just superficially approaching a topic, ChatGPT is much more efficient to get at the gist of things than doing actual research, since it has superhuman-level knowledge, it adapts the answers to the specific scenario you’re interested in, and doesn’t spit out loads and loads of spam, including blog spam, like search engines do. It also doesn’t require you to buy and read a 400-page book or a PhD thesis to get at some interesting idea or description of something.
In this case, the second point doesn’t apply as I’m sure there are many good descriptions of branch prediction and they’re very easy to find, so you have a very good point there.
And even when it does apply, that doesn’t mean you can’t still go look at a more trustworthy source, so I definitely see your point there as well.
In any case, I really appreciate your comment (and @fanf’s) as I was definitely very curious as to why there was so much ChatGPT backlash and I hadn’t considered your points of view!
I don’t want to be abrasive, but I want to make this super clear, because it’s the main thing that bugs me about all the current AI talk: This wording represents a fundamental misunderstanding of how this current generation of LLMs work. They do not have knowledge. They’re just really good at repeating back patterns they have been shown before in a form that resembles natural human writing. They cannot explain anything better than an expert, because their responses are an average of their training data, and nothing more. It can simulate domain familiarity through tokens and probabilities, but it does not have knowledge, and it is by no measure “expert”.
I think spam/annoyingly floating blog posts/search engines being bad are all systemic problems that very much exist, and that are serious and deserving of attention, but that can be resolved.
that doesn’t mean you can’t still go look at a more trustworthy source
Yes, but people are lazy, myself included, and a ChatGPT explanation is clearly enough to hit the bar for a lot of people. By including these AI responses, you’re not forbidding people from it, but you are still nearly guaranteeing that they won’t do their own research.
…and as a final point, I just like interacting with humans. I like that this website exists, and we can have conversations amongst ourselves like this. I like reading what people write and share here. I like when I go to read someone’s blog and I recognize the author/site from previous reading. and I can start to form opinions about whether their posts are worth continuing to read in the future. I’m never going to “bond” with ChatGPT. With remote work, the lack of third-places in suburbanized countries like the US, the increasingly busy adult lives of all my friends who I barely get to see anymore…I really dread people trying to popularize yet another way to isolate us from each other.
They cannot explain anything better than an expert, because their responses are an average of their training data, and nothing more.
I’m pretty sure this claim you just made is demonstrably false, as I’ve seen many experts make really, really confusing or convoluted explanations – and not just industry experts, but also Ivy League professors (not to mention all the other ones) who you’d think would be good at communicating – and that’s when they don’t make mistakes or downright false claims (either due to ignorance or due to other reasons).
Not to say that ChatGPT isn’t susceptible to that too, or even in greater amounts, but my point is that ChatGPT can indeed give better explanations than some experts, which is what I had said in my previous post.
Also, perhaps I am completely wrong, but I would argue that you also appear to have a too simplistic view of what the LLMs are actually doing and not really fully grasping what’s really going on. But I really don’t want to go into this debate, as it would probably devolve into a much deeper philosophical and/or technical discussion which I am probably not equipped to argue, since I’m not really an expert at either philosophy or LLMs…
I think spam/annoyingly floating blog posts/search engines being bad are all systemic problems that very much exist, and that are serious and deserving of attention, but that can be resolved.
Perhaps I’m wrong, but my subjective experience is that this seems to be worsening (i.e. the signal/noise ratio seems to have decreased sharply), so I hope you are right and that can be solved, as currently it takes me a lot longer to find reliable information about some topics by searching the web than by just asking ChatGPT.
Of course, whenever I find it important, I always try to confirm with more trustworthy sources. But in some cases there’s just no need, because sometimes it’s obvious whether the information is true or not (e.g. just try doing something and see if it works). Other times it’s obvious that it’s true just because of how the information provided by ChatGPT perfectly clicks with everything else you already know.
But yes, I don’t consider ChatGPT as reliable as most experts. But I’ve also seen ChatGPT give much more accurate and useful information than actual experts (like a lawyer that I’ve consulted, with decades of experience), so there’s also that…
…and as a final point, I just like interacting with humans. I like that this website exists, and we can have conversations amongst ourselves like this.
I really dread people trying to popularize yet another way to isolate us from each other.
I both completely understand and agree with this larger point you’ve made, but am also incredibly grateful for the great tool that ChatGPT v4 is right now, not to mention the future potential that it has.
In the few months it has been available, it has saved me hundreds and hundreds of hours and helped me come up with ideas and solve problems that I simply could not have come up with or solved myself in its absence, period. I have absolutely no question about it and have quite a few examples of that.
Still, right now ChatGPT is not comparable to a human in almost any way, and therefore cannot replace that experience. I simply view it as an incredibly useful tool right now, and a great one at that, that complements the human experience.
That’s really not made clear in the article but it actually causes the replacement of a conditional move by a branch. Because the test data set is invoked with increasing values, the branch is always taken, and thus perfectly predicted, which is faster than a cmov (/ csel).
Furthermore the cmov generates a data dependency between iteration while the predicted branch will just set the current value as the new max on every iteration and abort on failure.
Wow, thank you. Because of how tightly the assembly is clipped, I had to stare at the code for awhile before I convinced myself how it could even work without the branch instruction. I’d have enjoyed the post a lot more if I’d seen your note first.
Yeah I actually got this info from the orange site thread, where people went on to plug the code into godbolt and look at the assembly to suss out what was happening. TFA cut out the entire bit necessary to understand the jump to “branch prediction”.
It’s not an extra branch, it’s rather that the compiler decides to generate a branch rather than a conditional move (if only because you can’t cmov io).
Outputting a conditional move is an optimization, that in this case would actually slow down the code. Adding a print statement makes that optimization ineligible, leaving the program in its original branched form.
Conditional move can be faster when the branch is unpredictable, because waiting for a definitely-correct answer can be faster than rolling back an incorrect speculative execution the 50% the time the branch prediction was wrong. The test uses strictly increasing values which makes the branch totally predictable, as masklinn mentions in another comment. So if the data were random the non-lol version might be faster!
I came across a paper 10-15 years ago that suggested that the only static branch predictor hint that’s useful is ‘unpredictable’. If a branch is predictable, a static hint might save you a bit the first time you hit it but probably not too much. It might save you some predictor state, but not a huge amount (and predictor state is aliased in fun ways so it’s less than you might think on modern predictors). If a branch is completely unpredictable, learning that is quite hard and you may accidentally de-train other branches and so having a static hint that says ‘don’t store any state associated with this in the branch predictor’ and, optionally ‘and don’t bother executing it speculatively, you’ll get it wrong’ can improve performance.
I’d love to see someone try to reproduce their result with a modern branch predictor. It seems totally plausible but modern branch predictors are neural networks and I’ve no idea if they have the same characteristics.
Not per se a hint, but I would like to see ‘optional branches’. In effect: if the cpu predicts taken, and it turns out the condition of the branch was false, the cpu is allowed to nondeterministically take the branch anyway. But if the cpu predicts not taken, and the condition was true, then it does have to go back and take the branch. The idea is to work for cases when you have a fast path (not taken) and a slow path (taken), where the fast path is worth having, but the difference in performance between it and the slow path is less than the cost of a mispredict; and the fast path handles a subset of cases, whereas the slow path handles all of them.
I wonder whether “stall waiting on the flags here” would be a useful behavior for the branch predictor to be able to learn or use when its prediction is low-confidence, as well as a potential instruction variation/hint. The right move may depend on more than the prediction confidence, like how long we expect to have to wait for the flags to be calculated.
Versions of common instructions with better-predictable latencies would be interesting for writing code to resist timing attacks as well, but that involves much more than getting the branch predictor out of the picture (caching, etc.).
Flags are interesting because they provide a static hint. Branch on overflow is usually predicted not taken because integer overflow is rare, for example.
Really enjoyed this, other than that half of it was a chatgippity explanation of branch prediction
I truly loathe that this is becoming normal
I definitely agree but I would much rather have people be straightforward and label the AI generated bit so I can scroll past if I want than try to subtly integrate it into the article. Obviously all human content would be better, but this feels like the lesser of two evils.
I mean, wouldn’t even just linking (or quoting, if license compatible) Wikipedia be fine here too?
Can I ask why? I’m genuinely curious.
The ChatGPT explanation didn’t seem worse than a human one to me.
I have to say, I am surprised by the increasing anti-AI sentiment in tech communities.
When someone quotes ChatGPT, I know they have not learned enough about the topic to be able to put it into their own words or check if the quoted verbiage is correct.
In this case the author then went on to quote the wrong parts of the disassembly, confirming that my prejudice was right. Saddening, because I like seeing people learning a new thing and writing it up well. But I think this author is learning and I hope that continues.
Would you say the same if the author was quoting a person?
My experience is that when people quote others they are more actively engaging their brain. Quoting ChatGPT reminds me of the irritatingly sophomoric habit americans have of quoting the dictionary, as if their audience are barely literate. It shows that the author has not read around the subject.
Interesting. That’s not something I had even thought about.
I’ll keep that in mind for similar situations in the future, to confirm your theory. Thanks!
fanf already gave a lot of good points, but I wanna give a couple more.
ChatGPT is always going to generate descriptions that are at least slightly incorrect when they’re that long, and either the author doesn’t know better, or is willfully jettisoning misinformation out into the world. If I, as the reader, don’t know what branch prediction is and I care enough to learn about it, I’m just going to open a new tab and go learn about it for a minute.
Instead of spending the time going to ChatGPT and generating some unverifiable stream-of-un-consciousness, they could have just found an actual knowledgable description written by a human, and they could’ve linked to it, or quoted some small descriptive section of it with a citation.
These AI insertions are not neutral or harmless. They deprive us of actual research.
Ok, I see… If you don’t mind, I would just like to provide a couple of counterpoints:
In this case, the second point doesn’t apply as I’m sure there are many good descriptions of branch prediction and they’re very easy to find, so you have a very good point there.
And even when it does apply, that doesn’t mean you can’t still go look at a more trustworthy source, so I definitely see your point there as well.
In any case, I really appreciate your comment (and @fanf’s) as I was definitely very curious as to why there was so much ChatGPT backlash and I hadn’t considered your points of view!
I don’t want to be abrasive, but I want to make this super clear, because it’s the main thing that bugs me about all the current AI talk: This wording represents a fundamental misunderstanding of how this current generation of LLMs work. They do not have knowledge. They’re just really good at repeating back patterns they have been shown before in a form that resembles natural human writing. They cannot explain anything better than an expert, because their responses are an average of their training data, and nothing more. It can simulate domain familiarity through tokens and probabilities, but it does not have knowledge, and it is by no measure “expert”.
I think spam/annoyingly floating blog posts/search engines being bad are all systemic problems that very much exist, and that are serious and deserving of attention, but that can be resolved.
Yes, but people are lazy, myself included, and a ChatGPT explanation is clearly enough to hit the bar for a lot of people. By including these AI responses, you’re not forbidding people from it, but you are still nearly guaranteeing that they won’t do their own research.
…and as a final point, I just like interacting with humans. I like that this website exists, and we can have conversations amongst ourselves like this. I like reading what people write and share here. I like when I go to read someone’s blog and I recognize the author/site from previous reading. and I can start to form opinions about whether their posts are worth continuing to read in the future. I’m never going to “bond” with ChatGPT. With remote work, the lack of third-places in suburbanized countries like the US, the increasingly busy adult lives of all my friends who I barely get to see anymore…I really dread people trying to popularize yet another way to isolate us from each other.
You’re not, and I hope I am not either…
I’m pretty sure this claim you just made is demonstrably false, as I’ve seen many experts make really, really confusing or convoluted explanations – and not just industry experts, but also Ivy League professors (not to mention all the other ones) who you’d think would be good at communicating – and that’s when they don’t make mistakes or downright false claims (either due to ignorance or due to other reasons).
Not to say that ChatGPT isn’t susceptible to that too, or even in greater amounts, but my point is that ChatGPT can indeed give better explanations than some experts, which is what I had said in my previous post.
Also, perhaps I am completely wrong, but I would argue that you also appear to have a too simplistic view of what the LLMs are actually doing and not really fully grasping what’s really going on. But I really don’t want to go into this debate, as it would probably devolve into a much deeper philosophical and/or technical discussion which I am probably not equipped to argue, since I’m not really an expert at either philosophy or LLMs…
Perhaps I’m wrong, but my subjective experience is that this seems to be worsening (i.e. the signal/noise ratio seems to have decreased sharply), so I hope you are right and that can be solved, as currently it takes me a lot longer to find reliable information about some topics by searching the web than by just asking ChatGPT.
Of course, whenever I find it important, I always try to confirm with more trustworthy sources. But in some cases there’s just no need, because sometimes it’s obvious whether the information is true or not (e.g. just try doing something and see if it works). Other times it’s obvious that it’s true just because of how the information provided by ChatGPT perfectly clicks with everything else you already know.
But yes, I don’t consider ChatGPT as reliable as most experts. But I’ve also seen ChatGPT give much more accurate and useful information than actual experts (like a lawyer that I’ve consulted, with decades of experience), so there’s also that…
I both completely understand and agree with this larger point you’ve made, but am also incredibly grateful for the great tool that ChatGPT v4 is right now, not to mention the future potential that it has.
In the few months it has been available, it has saved me hundreds and hundreds of hours and helped me come up with ideas and solve problems that I simply could not have come up with or solved myself in its absence, period. I have absolutely no question about it and have quite a few examples of that.
Still, right now ChatGPT is not comparable to a human in almost any way, and therefore cannot replace that experience. I simply view it as an incredibly useful tool right now, and a great one at that, that complements the human experience.
Edit: removed superfluous point (I think).
So am I, but pleasantly so.
To be clear, the print statement is not called, but it forces an extra branch to be added to a loop.
That’s really not made clear in the article but it actually causes the replacement of a conditional move by a branch. Because the test data set is invoked with increasing values, the branch is always taken, and thus perfectly predicted, which is faster than a cmov (/ csel).
Furthermore the cmov generates a data dependency between iteration while the predicted branch will just set the current value as the new max on every iteration and abort on failure.
Wow, thank you. Because of how tightly the assembly is clipped, I had to stare at the code for awhile before I convinced myself how it could even work without the branch instruction. I’d have enjoyed the post a lot more if I’d seen your note first.
Yeah I actually got this info from the orange site thread, where people went on to plug the code into godbolt and look at the assembly to suss out what was happening. TFA cut out the entire bit necessary to understand the jump to “branch prediction”.
Why does it happen though? Why does an extra print statement force an extra branch? I can’t see the connection between the two things.
It’s not an extra branch, it’s rather that the compiler decides to generate a branch rather than a conditional move (if only because you can’t cmov io).
Outputting a conditional move is an optimization, that in this case would actually slow down the code. Adding a print statement makes that optimization ineligible, leaving the program in its original branched form.
Conditional move can be faster when the branch is unpredictable, because waiting for a definitely-correct answer can be faster than rolling back an incorrect speculative execution the 50% the time the branch prediction was wrong. The test uses strictly increasing values which makes the branch totally predictable, as masklinn mentions in another comment. So if the data were random the non-lol version might be faster!
I came across a paper 10-15 years ago that suggested that the only static branch predictor hint that’s useful is ‘unpredictable’. If a branch is predictable, a static hint might save you a bit the first time you hit it but probably not too much. It might save you some predictor state, but not a huge amount (and predictor state is aliased in fun ways so it’s less than you might think on modern predictors). If a branch is completely unpredictable, learning that is quite hard and you may accidentally de-train other branches and so having a static hint that says ‘don’t store any state associated with this in the branch predictor’ and, optionally ‘and don’t bother executing it speculatively, you’ll get it wrong’ can improve performance.
I’d love to see someone try to reproduce their result with a modern branch predictor. It seems totally plausible but modern branch predictors are neural networks and I’ve no idea if they have the same characteristics.
Not per se a hint, but I would like to see ‘optional branches’. In effect: if the cpu predicts taken, and it turns out the condition of the branch was false, the cpu is allowed to nondeterministically take the branch anyway. But if the cpu predicts not taken, and the condition was true, then it does have to go back and take the branch. The idea is to work for cases when you have a fast path (not taken) and a slow path (taken), where the fast path is worth having, but the difference in performance between it and the slow path is less than the cost of a mispredict; and the fast path handles a subset of cases, whereas the slow path handles all of them.
GPUs already have something like this.
That’s a neat idea. Thanks for sharing.
I wonder whether “stall waiting on the flags here” would be a useful behavior for the branch predictor to be able to learn or use when its prediction is low-confidence, as well as a potential instruction variation/hint. The right move may depend on more than the prediction confidence, like how long we expect to have to wait for the flags to be calculated.
Versions of common instructions with better-predictable latencies would be interesting for writing code to resist timing attacks as well, but that involves much more than getting the branch predictor out of the picture (caching, etc.).
Flags are interesting because they provide a static hint. Branch on overflow is usually predicted not taken because integer overflow is rare, for example.