When I was an undergraduate, my advisor brought in a software engineer to look at the software that our research group was using. We knew that the code was a dumpster fire of Perl and custom C modules and wanted advice on how it could be improved. The consultant spent a month reading our code and then came in to give us his suggestions.
The software engineer spent the entire hour telling us that our code was worthless because we were doing floating point math. He gave the standard intro to floating point rounding and said that all of our calculations were gibberish and always would be unless we switched to a fixed point number library. We asked if he could point to a specific line of code that was causing rounding issues and his was “every line of code”. You could only get accurate math from fixed point.
My advisor pulled out where we were using Kahan’s summation algorithm to total an array and asked if he had found a mistake in our implementation. He said the mistake was believing that you could ever accurately sum floats.
My advisor asked about the fixed point implementation that he suggested, but we quickly discovered that our research regularly used values that could not be expressed in his preferred library. He said that we would just need to write a series of shim wrappers to handle the different orders of magnitude.
We thanked him for his time, but said that we would not be needing his work any further. He promised to write every journal we had ever published in and tell them to retract our research because the math was garbage.
I’m now in the position of being the software guy helping out scientists and I spend every day remembering not to be him (or a dozen other developers I’ve worked with who tried to “fix” scientific code without understanding the science or the code).
That’s an interesting interaction. Did the person go through with their threat?
There’s a spectrum of wrongness of code, of course. There’s “My software converted gene names to dates.” wrongness and then there’s rounding errors that the data don’t hit or which stays orders of magnitude smaller than other systemic or random errors in the data.
From experience, scientific programmers know about numerical stability and it goes into their algorithm designs. In fact, “algorithm computing X, but more numerically stable” will get you a paper in the field. In contrast this is something ordinary software engineers essentially never grapple with.
Many years ago I got to teach a 2nd year computer architecture course to Comp Sci students. When we were discussing floating point math in class we went over a few examples of multiplication and addition and I explicitly put an example of adding a large number and a small number and showing how the mantissa bits of the small number didn’t end up anywhere in the final sum. And then had a homework assignment where they had to write a routine to sum an array of N floating point numbers. I provided a file of test cases and they had to write a routine that would pass all of the test cases. There was at least one case where it wouldn’t pass unless you were careful about the order you summed. And I was very pleased to see that they all got it right, with a bunch of different approaches.
So at least the 20 students in CMPT215 summer session have seen the problem before :)
Nothing shook my beliefs in the foundations of computers more than a course in numerical programming I took in the second year of university. Since then I’ve had a healthy respect for where the dragons reside among the floating point numbers, and where I should go to seek help navigating them.
Unfortunately no, that seemed to diverge too far from the general topic of the class. I think a few of them did implement it that way though! Lots of others did less efficient approaches like a bubble sort first to sum from smallest to largest or things like that. One or two did a clever thing where they would do a loop that I identified the two smallest absolute values (skipping zeros), add those, store the result back into one of the array spots and zero out the other. Terminate when there’s only one non-zero value left.
I was overall pretty impressed with their creativity! One of the really interesting things I found teaching a Spring & Summer session course was that the students generally fell into two bins: people who were super keen and were trying to get ahead in their degree and people who were behind and retaking the course after failing it the first time. What ended up happening though is that both groups did quite well… it’s been pretty well documented that in most comp sci programs there ends up being a bimodal distribution in grades. I did still see a weak two-hump distribution but it was nowhere as extreme as I’d seen in courses taught by others, and it was only weakly correlated to whether it was a student’s first or second time taking the course. Maybe it was beginner’s luck on my part (first time I was ever directly responsible for teaching a whole course) but I like to think I made the assignments so that it was relatively easy to get a 70 by demonstrating rote understanding of the course and achievable to get a 90+ by demonstrating understanding + some creative synthesis to solve practical issues that arise from the theory.
I wish I did. That would have probably been in 2010 or so and I’m pretty sure that’s all lost in the winds of time. The base of the course would have been based on Hennessy and Patterson’s Computer Architecture (unknown edition) and then biased towards the sections that seemed most relevant based on my couple years of industry experience before coming back to grad school.
One of my favourite lectures was the second last one. I had slightly misjudged my pace and got through the material I’d planned out one lecture early. I’d always planned the last one to be a review/Q&A session but had a gap to fill. The book is all based on MIPS assembler; more than once people had asked the relevance of MIPS over Arm or x86 (spoiler: it’s because MIPS is a nice clean orthogonal ISA that doesn’t require too much handwaving). So anyway, what did I fill the last 2h lecture with? JVM bytecode! We took everything we’d learned about MIPS and took some super simple Java code (which they’d learned in 1st year), compiled it, disassembled it, and walked through how the JVM was doing fundamentally the same thing as MIPS assembler was but with a couple extra instructions for handling the OO side of it. I was honestly a little shocked by how engaged everyone was, it ended up being a really fun way to wrap things up and a few students years later pointed out to me how memorable it was for them for demystifying how all of these complex high-level languages just break things down to simple steps under the hood.
Eeeeeh… the good ones do. Some scientists make a serious effort at understanding numerical methods and tend to have a good grasp on the mathematics, and come up with novel algorithms and know how to make sure existing ones are correct. The other kind of scientists then ask them to double-check their own work in exchange for a nice dinner.
I’m sure there are some, but probably not many. In a lot of cases doing it wrong will give obviously incorrect results that won’t even pass a “smell test”.
There are entire books about doing it the right way, though.
hah. True, but also, few observations from my own time in The Numpy Mines:
Most scientific code is write-only, or at least, only ever gets read by one person. Very often it’s written, debugged, run, and then thrown away.
The plethora of globals and function parameters tend to make it really hard to figure out what the hell is going on after the fact, such as when you’ve inherited some code with a subtle math bug. Reusing variables is especially painful to figure out.
jfc please write comments.
On the flip side, I also have intimate memories of sitting down with an electrical engineer to try to dissect a device driver written in exactly the over-clever sort of way described and being unable to explain why anyone would think it was a good idea to write it that way. Or reviewing some of same EE’s code and him saying “this is very stupid and bad and you’ll want to rewrite it”, and then me saying “actually reading this serial input a chunk at a time, decoding it, and shoving it into a state machine is basically ideal, so you’re doing it right”. He was a little incredulous.
The two times I recall software engineering really helping people out is when they were dealing with complicated problems, or ones that were non-obvious if you hadn’t seen them solved before: one person was making a UI with back buttons and stateful menus and stuff and was like “how the hell do I juggle where the user is and what the back button should do?” and I was like “state machine and stack maybe?” And another was doing hairy input processing stuff interleaved with computation and I was like “have you tried an event loop on its own input thread?” Most programs are small, and organizing a small program really isn’t that difficult. I think that it’s just that everything in programming is so difficult and abstract that it’s really hard to know in advance which problems are complex or will grow into crawling monstrosities, and which are the equivalent of a scuff that doesn’t affect functionality at all.
This sounds exactly like the sort of gripe anyone would have after dealing with one or two under-trained developers being told to do something wooly like “Improve this code” or “Build me a system worthy of Mordor enterprise!” The software engineering sins* mentioned are all from the architecture astronomy section of bad practice, and are rare these days.
But more importantly, how do you avoid these issues? I think the following could work:
Hire at least one developer with plenty of real-world experience and some nice things to show for it (like having contributed readable code to big open source projects). Make it clear to them in the job description that their primary job is to upskill all the developers, whether they have a science or software engineering background. And for the love of all that’s good, don’t give anyone a piece of horrible software and just tell them to “improve” it! By that point most of the potential wins are lost, the original author isn’t going to learn anything from the process (compared to involving the software engineer from the start), and the software engineer has an almost impossible and definitely unbounded job ahead of them.
As a corollary, the vast majority of academia (at least in the western world) tries to hire developers at far below market rate. What sort of developers do you think you’re going to get with that?
* More like smells, since in rare cases most of the abstractions mentioned can be useful in moderation.
Wow. My experience writing Matlab for neuroscience during my time as an academic was the exact opposite.
E.g., scientists I knew wrote:
Tons of one-off code that discouraged replicability, or even reuse later within the same lab.
Memory-inefficient designs that made algorithms slow down by orders of magnitude once they exceeded the size of main memory.
Non-existent verification of correct code behavior, combined with bugs that invisibly reduced or inflated power. Nobody knows how many experimental results world-wide are incorrect due to software analysis bugs.
And this wasn’t just in my lab. I saw it with collaborators’ code. I also looked at the code of the dominant FOSS packages at the time (SPM and FieldTrip, for fMRI and EEG/MEG, respectively), and they were only slightly better.
IME, scientists are generally smart enough to learn how to code, but lack the study and the incentives to do it well. For their careers, papers >>>> code.
Not so with software engineers, whose sins fall into entirely different categories:
Multiple/virtual/high-on-crack inheritance
7 to 14 stack frames composed principally of thin wrappers, some of them function pointers/virtual functions, possibly inside interrupt handlers or what-not
Files spread in umpteen directories
Lookup using dynamic structures from hell – dictionaries of names where the names are concatenated from various pieces at runtime, etc.
Dynamic loading and other grep-defeating techniques
A forest of near-identical names along the lines of DriverController, ControllerManager, DriverManager, ManagerController, controlDriver ad infinitum – all calling each other
Templates calling overloaded functions with declarations hopefully visible where the template is defined, maybe not
Decorators, metaclasses, code generation, etc. etc.
Ok but how can the enterprise scale without these things? /s
I’m pretty sure everyone who likes to get things done instead of being busy hates the above. Some things really need to be big and generic (paging @david_chisnall for some war story) like UI frameworks, but generally no.
Yeah, over time as I’ve gone through my career I’ve definitely gone through the junior/mid/senior meme where my code started out simple, got ridiculously complex, and then has trended back towards simple. The nuance, for me, is probably what I’d call “clever plumbing, simple business logic”. An example from the current system I work on:
It’s a high-performance soft realtime system that takes about 500MB/s worth of pixel data from a “special” camera, forks the image feed to processors, disk writers, video encoders / gstreamer / RTSP, an ML pipeline, etc. It also has to process data from a variety of sensors (accelerometers, gyros, GPS receivers, aircraft telemetry data, etc) and incorporate the cooked sensor data into metadata associated with the images
We set up a somewhat clever event bus that allows “active objects” to subscribe to event feeds and react to them as well as emitting events for other modules to consume. We’ve had to add a lot of cleverness to the event bus over the years to carefully squeeze as much performance out of it as possible without invalidating correctness. “Performance” here primarily means latency but we have to delicately balance latency and throughput.
Some of the more complex AOs are modelled as state machines, some of them are relatively dumb and just take raw input and produce cooked output events. But all of them are implemented as basically a while(keep_running()) { auto evt = pop_event(); switch(evt->type)... } kind of loop. Dead simple to inspect each one and figure out precisely what it’s doing.
There’s some template magic involved in the event bus. There’s a kind of interesting subclass hierarchy involved too. But we very rarely have to dig into any of that because we hid the complexity behind a nice and simple interface which allows our day-to-day development to mostly ignore the complexity and focus on implementing the very linear straightforward easy-to-follow logic that lives inside those classes. We’ve been very frequently surprised at how easy it’s been to add new functionality that wasn’t even a little bit in our imaginations when we started without having to do any real heavy lifting in the system.
When I was an undergraduate, my advisor brought in a software engineer to look at the software that our research group was using. We knew that the code was a dumpster fire of Perl and custom C modules and wanted advice on how it could be improved. The consultant spent a month reading our code and then came in to give us his suggestions.
The software engineer spent the entire hour telling us that our code was worthless because we were doing floating point math. He gave the standard intro to floating point rounding and said that all of our calculations were gibberish and always would be unless we switched to a fixed point number library. We asked if he could point to a specific line of code that was causing rounding issues and his was “every line of code”. You could only get accurate math from fixed point.
My advisor pulled out where we were using Kahan’s summation algorithm to total an array and asked if he had found a mistake in our implementation. He said the mistake was believing that you could ever accurately sum floats.
My advisor asked about the fixed point implementation that he suggested, but we quickly discovered that our research regularly used values that could not be expressed in his preferred library. He said that we would just need to write a series of shim wrappers to handle the different orders of magnitude.
We thanked him for his time, but said that we would not be needing his work any further. He promised to write every journal we had ever published in and tell them to retract our research because the math was garbage.
I’m now in the position of being the software guy helping out scientists and I spend every day remembering not to be him (or a dozen other developers I’ve worked with who tried to “fix” scientific code without understanding the science or the code).
That’s an interesting interaction. Did the person go through with their threat?
There’s a spectrum of wrongness of code, of course. There’s “My software converted gene names to dates.” wrongness and then there’s rounding errors that the data don’t hit or which stays orders of magnitude smaller than other systemic or random errors in the data.
I wonder how many experimental results out there are compromised because of floating point issues.
Great story, btw.
From experience, scientific programmers know about numerical stability and it goes into their algorithm designs. In fact, “algorithm computing X, but more numerically stable” will get you a paper in the field. In contrast this is something ordinary software engineers essentially never grapple with.
Many years ago I got to teach a 2nd year computer architecture course to Comp Sci students. When we were discussing floating point math in class we went over a few examples of multiplication and addition and I explicitly put an example of adding a large number and a small number and showing how the mantissa bits of the small number didn’t end up anywhere in the final sum. And then had a homework assignment where they had to write a routine to sum an array of N floating point numbers. I provided a file of test cases and they had to write a routine that would pass all of the test cases. There was at least one case where it wouldn’t pass unless you were careful about the order you summed. And I was very pleased to see that they all got it right, with a bunch of different approaches.
So at least the 20 students in CMPT215 summer session have seen the problem before :)
Nothing shook my beliefs in the foundations of computers more than a course in numerical programming I took in the second year of university. Since then I’ve had a healthy respect for where the dragons reside among the floating point numbers, and where I should go to seek help navigating them.
Did they learn about Kahan summation as well? It’s useful to know the name of the o.g. version of compensated summation.
Unfortunately no, that seemed to diverge too far from the general topic of the class. I think a few of them did implement it that way though! Lots of others did less efficient approaches like a bubble sort first to sum from smallest to largest or things like that. One or two did a clever thing where they would do a loop that I identified the two smallest absolute values (skipping zeros), add those, store the result back into one of the array spots and zero out the other. Terminate when there’s only one non-zero value left.
I was overall pretty impressed with their creativity! One of the really interesting things I found teaching a Spring & Summer session course was that the students generally fell into two bins: people who were super keen and were trying to get ahead in their degree and people who were behind and retaking the course after failing it the first time. What ended up happening though is that both groups did quite well… it’s been pretty well documented that in most comp sci programs there ends up being a bimodal distribution in grades. I did still see a weak two-hump distribution but it was nowhere as extreme as I’d seen in courses taught by others, and it was only weakly correlated to whether it was a student’s first or second time taking the course. Maybe it was beginner’s luck on my part (first time I was ever directly responsible for teaching a whole course) but I like to think I made the assignments so that it was relatively easy to get a 70 by demonstrating rote understanding of the course and achievable to get a 90+ by demonstrating understanding + some creative synthesis to solve practical issues that arise from the theory.
That’s cool. Do you have any course notes or handouts for that course available anywhere?
I wish I did. That would have probably been in 2010 or so and I’m pretty sure that’s all lost in the winds of time. The base of the course would have been based on Hennessy and Patterson’s Computer Architecture (unknown edition) and then biased towards the sections that seemed most relevant based on my couple years of industry experience before coming back to grad school.
One of my favourite lectures was the second last one. I had slightly misjudged my pace and got through the material I’d planned out one lecture early. I’d always planned the last one to be a review/Q&A session but had a gap to fill. The book is all based on MIPS assembler; more than once people had asked the relevance of MIPS over Arm or x86 (spoiler: it’s because MIPS is a nice clean orthogonal ISA that doesn’t require too much handwaving). So anyway, what did I fill the last 2h lecture with? JVM bytecode! We took everything we’d learned about MIPS and took some super simple Java code (which they’d learned in 1st year), compiled it, disassembled it, and walked through how the JVM was doing fundamentally the same thing as MIPS assembler was but with a couple extra instructions for handling the OO side of it. I was honestly a little shocked by how engaged everyone was, it ended up being a really fun way to wrap things up and a few students years later pointed out to me how memorable it was for them for demystifying how all of these complex high-level languages just break things down to simple steps under the hood.
Eeeeeh… the good ones do. Some scientists make a serious effort at understanding numerical methods and tend to have a good grasp on the mathematics, and come up with novel algorithms and know how to make sure existing ones are correct. The other kind of scientists then ask them to double-check their own work in exchange for a nice dinner.
Works out okay, honestly.
I’m sure there are some, but probably not many. In a lot of cases doing it wrong will give obviously incorrect results that won’t even pass a “smell test”.
There are entire books about doing it the right way, though.
hah. True, but also, few observations from my own time in The Numpy Mines:
On the flip side, I also have intimate memories of sitting down with an electrical engineer to try to dissect a device driver written in exactly the over-clever sort of way described and being unable to explain why anyone would think it was a good idea to write it that way. Or reviewing some of same EE’s code and him saying “this is very stupid and bad and you’ll want to rewrite it”, and then me saying “actually reading this serial input a chunk at a time, decoding it, and shoving it into a state machine is basically ideal, so you’re doing it right”. He was a little incredulous.
The two times I recall software engineering really helping people out is when they were dealing with complicated problems, or ones that were non-obvious if you hadn’t seen them solved before: one person was making a UI with back buttons and stateful menus and stuff and was like “how the hell do I juggle where the user is and what the back button should do?” and I was like “state machine and stack maybe?” And another was doing hairy input processing stuff interleaved with computation and I was like “have you tried an event loop on its own input thread?” Most programs are small, and organizing a small program really isn’t that difficult. I think that it’s just that everything in programming is so difficult and abstract that it’s really hard to know in advance which problems are complex or will grow into crawling monstrosities, and which are the equivalent of a scuff that doesn’t affect functionality at all.
This sounds exactly like the sort of gripe anyone would have after dealing with one or two under-trained developers being told to do something wooly like “Improve this code” or “Build me a system worthy of
Mordorenterprise!” The software engineering sins* mentioned are all from the architecture astronomy section of bad practice, and are rare these days.But more importantly, how do you avoid these issues? I think the following could work:
Hire at least one developer with plenty of real-world experience and some nice things to show for it (like having contributed readable code to big open source projects). Make it clear to them in the job description that their primary job is to upskill all the developers, whether they have a science or software engineering background. And for the love of all that’s good, don’t give anyone a piece of horrible software and just tell them to “improve” it! By that point most of the potential wins are lost, the original author isn’t going to learn anything from the process (compared to involving the software engineer from the start), and the software engineer has an almost impossible and definitely unbounded job ahead of them.
As a corollary, the vast majority of academia (at least in the western world) tries to hire developers at far below market rate. What sort of developers do you think you’re going to get with that?
* More like smells, since in rare cases most of the abstractions mentioned can be useful in moderation.
Wow. My experience writing Matlab for neuroscience during my time as an academic was the exact opposite.
E.g., scientists I knew wrote:
And this wasn’t just in my lab. I saw it with collaborators’ code. I also looked at the code of the dominant FOSS packages at the time (SPM and FieldTrip, for fMRI and EEG/MEG, respectively), and they were only slightly better.
IME, scientists are generally smart enough to learn how to code, but lack the study and the incentives to do it well. For their careers, papers >>>> code.
Ok but how can the enterprise scale without these things? /s
I’m pretty sure everyone who likes to get things done instead of being busy hates the above. Some things really need to be big and generic (paging @david_chisnall for some war story) like UI frameworks, but generally no.
Yeah, over time as I’ve gone through my career I’ve definitely gone through the junior/mid/senior meme where my code started out simple, got ridiculously complex, and then has trended back towards simple. The nuance, for me, is probably what I’d call “clever plumbing, simple business logic”. An example from the current system I work on:
It’s a high-performance soft realtime system that takes about 500MB/s worth of pixel data from a “special” camera, forks the image feed to processors, disk writers, video encoders / gstreamer / RTSP, an ML pipeline, etc. It also has to process data from a variety of sensors (accelerometers, gyros, GPS receivers, aircraft telemetry data, etc) and incorporate the cooked sensor data into metadata associated with the images
We set up a somewhat clever event bus that allows “active objects” to subscribe to event feeds and react to them as well as emitting events for other modules to consume. We’ve had to add a lot of cleverness to the event bus over the years to carefully squeeze as much performance out of it as possible without invalidating correctness. “Performance” here primarily means latency but we have to delicately balance latency and throughput.
Some of the more complex AOs are modelled as state machines, some of them are relatively dumb and just take raw input and produce cooked output events. But all of them are implemented as basically a
while(keep_running()) { auto evt = pop_event(); switch(evt->type)... }kind of loop. Dead simple to inspect each one and figure out precisely what it’s doing.There’s some template magic involved in the event bus. There’s a kind of interesting subclass hierarchy involved too. But we very rarely have to dig into any of that because we hid the complexity behind a nice and simple interface which allows our day-to-day development to mostly ignore the complexity and focus on implementing the very linear straightforward easy-to-follow logic that lives inside those classes. We’ve been very frequently surprised at how easy it’s been to add new functionality that wasn’t even a little bit in our imaginations when we started without having to do any real heavy lifting in the system.