Besides the probable-bug-in-the-code angle, an interesting aspect of this to me is that the whole dispute takes place in pretty non-exotic material: water cooled to between -20 and -40 C (albeit very pure water). I guess it’s not surprising that there are plenty of unanswered questions in that kind of material too, I’m just more used to these popular writeups of physics disputes being about material that you find only in distant galaxies or supercolliders.
The trick is that it’s extremely pure water with no nucleation sites. That’s more of a theoretical thing. In all real water samples a little bit of bacteria or something causes the water to freeze before the interesting transition happens.
I don’t think there are excuses for not publishing the code your results were obtained with.
In experimental science, there’s always a risk of accidentally, or even unknowingly omitting a crucial detail of the experiment setup (e.g. unknown impurity working as a catalyst), but I don’t know anyone who would withhold details of an experiment setup from peers on purpose. When experiment setup is a physical device, people realize that to have anyone repeat and confirm their findings they need every detail.
With code, repeating findings and reviewing the setup is much easier, and yet some people purposely make it harder.
Well, the science they are doing is genuine. A methodology mistake doesn’t equal cargo cult science.
It’s the standard of communicating that science that I have a problem with.
Sure, I guess the point in that address I wanted to zoom in on was the experimental protocol Young discovered and published – had he not done that (or left out parts of his setup), there would have been no standard to hold other studies against (even if Feynman claims it was ignored). I didn’t mean to disparage the scientists involved, I just wanted to bolster your point that withholding experimental setups (knowingly or not) can lead to bad effects down the line.
This is why I religiously publish all code of my scientific publications with the publication itself. If someone wants to reproduce the results 20-30 years later, it would be unlikely I have the code still at hand. If it is published with the publication, it is one unit.
This isn’t just one paper, of course. Science code is much poorer quality and much less tested than engineering code. We expect it to have many, many more bugs. Any paper that relies on science code should be treated as suspect unless they share the code.
So much anger and sadness over something so relatively unimportant :(
Don’t stake alliances, friendships, your future or your life on being right, unless you are certain it’s really that important. Your code never is. Nor are most other things.
Besides the probable-bug-in-the-code angle, an interesting aspect of this to me is that the whole dispute takes place in pretty non-exotic material: water cooled to between -20 and -40 C (albeit very pure water). I guess it’s not surprising that there are plenty of unanswered questions in that kind of material too, I’m just more used to these popular writeups of physics disputes being about material that you find only in distant galaxies or supercolliders.
The trick is that it’s extremely pure water with no nucleation sites. That’s more of a theoretical thing. In all real water samples a little bit of bacteria or something causes the water to freeze before the interesting transition happens.
Below the homogeneous nucleation temperature, it crystalizes almost instantly, even if it’s perfectly pure.
I don’t think there are excuses for not publishing the code your results were obtained with.
In experimental science, there’s always a risk of accidentally, or even unknowingly omitting a crucial detail of the experiment setup (e.g. unknown impurity working as a catalyst), but I don’t know anyone who would withhold details of an experiment setup from peers on purpose. When experiment setup is a physical device, people realize that to have anyone repeat and confirm their findings they need every detail.
With code, repeating findings and reviewing the setup is much easier, and yet some people purposely make it harder.
Reminds me of that great address made by Feynman.
Well, the science they are doing is genuine. A methodology mistake doesn’t equal cargo cult science. It’s the standard of communicating that science that I have a problem with.
Sure, I guess the point in that address I wanted to zoom in on was the experimental protocol Young discovered and published – had he not done that (or left out parts of his setup), there would have been no standard to hold other studies against (even if Feynman claims it was ignored). I didn’t mean to disparage the scientists involved, I just wanted to bolster your point that withholding experimental setups (knowingly or not) can lead to bad effects down the line.
This is why I religiously publish all code of my scientific publications with the publication itself. If someone wants to reproduce the results 20-30 years later, it would be unlikely I have the code still at hand. If it is published with the publication, it is one unit.
“Truth never triumphs—its opponents just die out.”
This isn’t just one paper, of course. Science code is much poorer quality and much less tested than engineering code. We expect it to have many, many more bugs. Any paper that relies on science code should be treated as suspect unless they share the code.
So much anger and sadness over something so relatively unimportant :(
Don’t stake alliances, friendships, your future or your life on being right, unless you are certain it’s really that important. Your code never is. Nor are most other things.