The standard pre-publication peer review process is only intended to reject work that is not notable or is fairly obviously bullshit. It’s not intended to catch all errors.
Post-publication review (in the form of other scholars writing review papers, discussing the work, trying to replicate it, challenging it, etc) is where the academic process slowly sifts truth from fiction.
Expanding: reviewers certainly can and do catch minor errors — but it’s not their primary job, and they generally don’t have the time to be very thorough about it.
I am definitely a summer child. Academically I wasn’t fortunate enough. I have worked through a few papers, but this was the first time I encountered a publication error.
When I shared this story with a few, they were surprised that I didn’t know paper could have errors. I simply didn’t know because no one had told me till now!
We’ve all been sweet summer children at different times in different walks of life.
Before I’ve had any insight into the details through friends and colleagues, I, too, had illusions of academic publishing being this extremely rigorous process, triple- and quadruple-checked and reproduced before publishing. Discovering the fallibility of authors, peer reviewers and publishers was both mildly heartbreaking and extremely relieving. :)
It really isn’t something they teach you in school — or at least they haven’t in any of the ones I’ve attended.
Pre-publication peer review is a process where papers are filtered via a set of biases. Papers are as likely to be rejected for not being the right style for a venue as for technical content. At this stage, o one tries to reproduce results and often will not check maths (my favourite example here is the Marching Cubes algorithm, which was both patented and published at the top graphics conference, and is fairly obviously wrong if you spend half an hour working through it. The fix is simple but the reviewers didn’t notice it).
After publication, 90% of papers are then ignored. The remaining 10% will have people read them and use them as inspiration for something. An even smaller fraction will have people try to reproduce the result ps to try to build on them. Often, they discover that there was a bug. When we tried to use the lowFAT Pointers work, for example, we discovered that the compression scheme was described in prose, maths, and a circuit in the original paper and these did not quite agree (I think two of them were correct). For a tiny subset of papers, a lot of things will be built on the result and you can have confidence in them.
The key is to think of academic papers as pretentious blogs. They might have some good ideas but until multiple other people have reproduced them that’s the most that you can guarantee.
It was sobering for me to attend ISCA back in 2015, when there was a panel on the role of simulation in computer architecture papers. I expected the panel to say ‘don’t rely on simulators, build a prototype’. The panel actually went the other way and said some abstract model was probably fine. This was supported by the industry representative (I think he was from NVIDIA), who said that they look at papers for ideas, they skip the results section entirely because they don’t trust them: the error margins are often 50% for a 20% speed up, so unless they’ve internally reproduced the results on their own simulation infrastructure they assume the results are probably nonsense.
the error margins are often 50% for a 20% speed up, so unless they’ve internally reproduced the results on their own simulation infrastructure they assume the results are probably nonsense.
To be fair to simulators, I suspect industry has to ignore the results sections even on papers which do have implementations. So experimenting on a simulator is reasonable because it’s cheaper.
Say I have an idea for making arithmetic faster. I implement an ALU with my idea (the experiment) and an ALU without it (the control) and compare performance. If my control has a mistake in it which tanks its performance, the experiment will look great by comparison.
I have seen people ranting about precisely this problem with literature on data structures and algorithms here on lobsters. That’s largely solved by making sure you benchmark against an industrially relevant open source competitor, but those largely aren’t available in the hardware space?
Indeed. This was just a typo, too. Wait until the author gets deep into the literature and starts finding stuff that is just outright wrong. By the end of grad school, I assumed that any biology paper in Nature, Science, or Cell was probably flawed to the point of being unusable until proven otherwise.
Or, just maybe, the system of peer review that was instituted when there were less researchers has been overwhelmed by the massive increase in researchers who also have a direct economic interest in publishing.
Anyway, peer review is just a basic step to tell you a paper isn’t entirely worthless. It states conclusions and presents evidence that’s not totally unbelievable. The real science starts when (like in the linked post) someone tries to reproduce the findings.
Or if not wrong then incomplete, vague, or low-key deceptive. Frankly, the format rather sucks and we should do better. The failures are often a lot more interesting and useful, but the successes are what get reported. So anyone who wants to actually reproduce the work has to re-tread all the failures from scratch, again.
The researchers that I respect often have fascinating blogs. I am aware, for example, that @ltratt publishes some papers to keep the REF and his funding bodies happy. Occasionally I might even read one. I’ll read everything that he writes on his blog though. If the REF actually measured impact, they’d weigh the blog far more heavily than a lot of publications.
“Is it possible for a peer-reviewed paper to have errors?”
My sweet, sweet summer child. You are vastly overestimating the competence of this species.
The standard pre-publication peer review process is only intended to reject work that is not notable or is fairly obviously bullshit. It’s not intended to catch all errors.
Post-publication review (in the form of other scholars writing review papers, discussing the work, trying to replicate it, challenging it, etc) is where the academic process slowly sifts truth from fiction.
Expanding: reviewers certainly can and do catch minor errors — but it’s not their primary job, and they generally don’t have the time to be very thorough about it.
I am definitely a summer child. Academically I wasn’t fortunate enough. I have worked through a few papers, but this was the first time I encountered a publication error.
When I shared this story with a few, they were surprised that I didn’t know paper could have errors. I simply didn’t know because no one had told me till now!
We’ve all been sweet summer children at different times in different walks of life.
Before I’ve had any insight into the details through friends and colleagues, I, too, had illusions of academic publishing being this extremely rigorous process, triple- and quadruple-checked and reproduced before publishing. Discovering the fallibility of authors, peer reviewers and publishers was both mildly heartbreaking and extremely relieving. :)
It really isn’t something they teach you in school — or at least they haven’t in any of the ones I’ve attended.
Pre-publication peer review is a process where papers are filtered via a set of biases. Papers are as likely to be rejected for not being the right style for a venue as for technical content. At this stage, o one tries to reproduce results and often will not check maths (my favourite example here is the Marching Cubes algorithm, which was both patented and published at the top graphics conference, and is fairly obviously wrong if you spend half an hour working through it. The fix is simple but the reviewers didn’t notice it).
After publication, 90% of papers are then ignored. The remaining 10% will have people read them and use them as inspiration for something. An even smaller fraction will have people try to reproduce the result ps to try to build on them. Often, they discover that there was a bug. When we tried to use the lowFAT Pointers work, for example, we discovered that the compression scheme was described in prose, maths, and a circuit in the original paper and these did not quite agree (I think two of them were correct). For a tiny subset of papers, a lot of things will be built on the result and you can have confidence in them.
The key is to think of academic papers as pretentious blogs. They might have some good ideas but until multiple other people have reproduced them that’s the most that you can guarantee.
It was sobering for me to attend ISCA back in 2015, when there was a panel on the role of simulation in computer architecture papers. I expected the panel to say ‘don’t rely on simulators, build a prototype’. The panel actually went the other way and said some abstract model was probably fine. This was supported by the industry representative (I think he was from NVIDIA), who said that they look at papers for ideas, they skip the results section entirely because they don’t trust them: the error margins are often 50% for a 20% speed up, so unless they’ve internally reproduced the results on their own simulation infrastructure they assume the results are probably nonsense.
To be fair to simulators, I suspect industry has to ignore the results sections even on papers which do have implementations. So experimenting on a simulator is reasonable because it’s cheaper.
Say I have an idea for making arithmetic faster. I implement an ALU with my idea (the experiment) and an ALU without it (the control) and compare performance. If my control has a mistake in it which tanks its performance, the experiment will look great by comparison.
I have seen people ranting about precisely this problem with literature on data structures and algorithms here on lobsters. That’s largely solved by making sure you benchmark against an industrially relevant open source competitor, but those largely aren’t available in the hardware space?
Indeed. This was just a typo, too. Wait until the author gets deep into the literature and starts finding stuff that is just outright wrong. By the end of grad school, I assumed that any biology paper in Nature, Science, or Cell was probably flawed to the point of being unusable until proven otherwise.
At the end of high school, you believe you know everything.
At the end of college, you believe you know nothing.
At the end of a PhD, you believe nobody knows anything.
(No clue, where I got it from)
After five years in industry, you start assuming active malice…
(edit: Software developer, for context. I’m not talking about papers so much as the garbage that makes up the modern ecosystem.)
We each choose whether to be that kind of person.
Or, just maybe, the system of peer review that was instituted when there were less researchers has been overwhelmed by the massive increase in researchers who also have a direct economic interest in publishing.
Anyway, peer review is just a basic step to tell you a paper isn’t entirely worthless. It states conclusions and presents evidence that’s not totally unbelievable. The real science starts when (like in the linked post) someone tries to reproduce the findings.
Or if not wrong then incomplete, vague, or low-key deceptive. Frankly, the format rather sucks and we should do better. The failures are often a lot more interesting and useful, but the successes are what get reported. So anyone who wants to actually reproduce the work has to re-tread all the failures from scratch, again.
The researchers that I respect often have fascinating blogs. I am aware, for example, that @ltratt publishes some papers to keep the REF and his funding bodies happy. Occasionally I might even read one. I’ll read everything that he writes on his blog though. If the REF actually measured impact, they’d weigh the blog far more heavily than a lot of publications.
Lesson hopefully learned: “three sleepless nights” before reaching out to someone else is too long :)