I used to work on a product that my employer had lost the source code to. When we needed to make changes, we just patched the object code directly. New functions were written in assembly, appended to the binary, and hooked in by overwriting some other instruction to branch to the new code.
Some of our customers were not permitted to connect their systems to the internet, or to any network at all. Some of them received new versions of the program via tapes in the mail, but others opted for a cheaper and “faster” option; for them, we would print out and fax hex dumps of the changes to the binary.
I don’t have all day so I just picked one at the bottom, this one. I didn’t even know that RDMA was a thing and now I’m sitting in the corner hugging my knees and rocking slowly.
Honestly, fork is more the problem, here, than RDMA is. (And I see that that’s also the name of the text that links there on the iceberg.) forkonly works well for private memory mappings in single-threaded processes. Every other process configuration / OS resource is problematic. This issue we encountered is much less exotic, but a natural consequence of the trouble with mixing “everything is a file” behaviors with “fork”.
(Also, I’m amused to see that that link describes a bug very much like one we encountered. Our local upshot was that we couldn’t make our system compatible with rsocket, we had to fall back to using libibverbs directly, specifically so that we could control which memory got the MADV_DONTNEED treatment.)
I must pedantically point out that just because it wasn’t installed for the reasons the story describes, doesn’t mean it didn’t behave as the story describes. XD
I recently worked on debugging what turned out to be a hardware issue. A microcontroller on an I/O board occasionally crashed and took other components with it, and the best way to trigger it intentionally was to touch the system’s power button. Not press, just touch. I think it was eventually found to be a process flaw: something in the way that the PCB and the parts on it were manufactured, soldered, cleaned with solvent, and assembled was causing a shielding layer of epoxy or something on the board to be damaged just enough to occasionally let current leak through to places it shouldn’t be going. The big capacitor that is your body touching the (metal) power button made enough electrons move around to crash the microcontroller. Apparently sometimes you didn’t even have to touch it, just get your finger near by. As far as I know, the fix was to use a different solvent during board assembly.
This reminds me of one of my favourite things about electronics. Sometimes an electronic engineer will design a circuit, test it, find out that it fails, try to debug it with an oscilloscope, find that it works reliably but only when the oscilloscope probes are in, and then finally give up and fix it permanently by adding a 22pF capacitor at each spot where the oscilloscope probes were connected, in order to simulate an oscilloscope being attached and thereby make the circuit behave how it did with the scope plugged in. :)
Yep, that sort of debugging happens everywhere. Building physical devices often has a “fit-and-finish” step, where you put everything together, try to make it work, and then by hand file down or smooth out any bits that don’t quite mesh together right. Getting your design and manufacturing process to the point where you don’t need this sort of manual fiddling all the time takes either a lot of hands-on experience or a lot of iteration or both. Sound familiar, software people designing complex systems?
But that sort of iteration is also necessary for automation and interchangable parts to work, which lets you make majillions of those devices, and so the high up-front cost pays itself off by letting you scale up. The thing about software is that you can do that sort of iteration very fast and in very small pieces, and now with the internet you get the benefits almost instantly, and so the cost-benefit scale is very different.
But that sort of iteration is also necessary for automation and interchangable parts to work, which lets you make majillions of those devices, and so the high up-front cost pays itself off by letting you scale up
This is why I think of engineering as the process of paying a very expensive person to make something else cheaper. :)
Oh, certainly! I found these new details a few weeks ago while looking up the story for our junior engineers, as this was always the classic lesson of “seemingly-unconnected devices can still influence each other at the electrical level”.
Or to put it another way, “next time, politely tell the internal helpdesk customer to turn off their desktop plasma globe when trying to use their company-issued yubikey”.
I used to work on a product that my employer had lost the source code to. When we needed to make changes, we just patched the object code directly. New functions were written in assembly, appended to the binary, and hooked in by overwriting some other instruction to branch to the new code.
Some of our customers were not permitted to connect their systems to the internet, or to any network at all. Some of them received new versions of the program via tapes in the mail, but others opted for a cheaper and “faster” option; for them, we would print out and fax hex dumps of the changes to the binary.
Wow, that’s so cute! How long ago was that?
Roughly 20 years ago, I’m pretty sure they still sell the product to the same set of customers.
I hate this. Great job!
Janet Jackson Rhythm Nation would be a good addition.
I got almost to the end before I realized that the titles are links.
I got to this comment before realizing it…
I don’t have all day so I just picked one at the bottom, this one. I didn’t even know that RDMA was a thing and now I’m sitting in the corner hugging my knees and rocking slowly.
Honestly,
forkis more the problem, here, than RDMA is. (And I see that that’s also the name of the text that links there on the iceberg.)forkonly works well for private memory mappings in single-threaded processes. Every other process configuration / OS resource is problematic. This issue we encountered is much less exotic, but a natural consequence of the trouble with mixing “everything is a file” behaviors with “fork”.(Also, I’m amused to see that that link describes a bug very much like one we encountered. Our local upshot was that we couldn’t make our system compatible with
rsocket, we had to fall back to usinglibibverbsdirectly, specifically so that we could control which memory got theMADV_DONTNEEDtreatment.)RDMA scares me from a security standpoint but it doesn’t feel horrifying from a software engineering standpoint.
Of course it’s really only meant to be enabled and used with physically co-located clusters all locally networked together.
Sadly, the “more magic” story seems to have been debunked by Tom Knight himself in 1996.
That is, the part about crashing the computer and not the actual switch itself.
I must pedantically point out that just because it wasn’t installed for the reasons the story describes, doesn’t mean it didn’t behave as the story describes. XD
I recently worked on debugging what turned out to be a hardware issue. A microcontroller on an I/O board occasionally crashed and took other components with it, and the best way to trigger it intentionally was to touch the system’s power button. Not press, just touch. I think it was eventually found to be a process flaw: something in the way that the PCB and the parts on it were manufactured, soldered, cleaned with solvent, and assembled was causing a shielding layer of epoxy or something on the board to be damaged just enough to occasionally let current leak through to places it shouldn’t be going. The big capacitor that is your body touching the (metal) power button made enough electrons move around to crash the microcontroller. Apparently sometimes you didn’t even have to touch it, just get your finger near by. As far as I know, the fix was to use a different solvent during board assembly.
This reminds me of one of my favourite things about electronics. Sometimes an electronic engineer will design a circuit, test it, find out that it fails, try to debug it with an oscilloscope, find that it works reliably but only when the oscilloscope probes are in, and then finally give up and fix it permanently by adding a 22pF capacitor at each spot where the oscilloscope probes were connected, in order to simulate an oscilloscope being attached and thereby make the circuit behave how it did with the scope plugged in. :)
Yep, that sort of debugging happens everywhere. Building physical devices often has a “fit-and-finish” step, where you put everything together, try to make it work, and then by hand file down or smooth out any bits that don’t quite mesh together right. Getting your design and manufacturing process to the point where you don’t need this sort of manual fiddling all the time takes either a lot of hands-on experience or a lot of iteration or both. Sound familiar, software people designing complex systems?
But that sort of iteration is also necessary for automation and interchangable parts to work, which lets you make majillions of those devices, and so the high up-front cost pays itself off by letting you scale up. The thing about software is that you can do that sort of iteration very fast and in very small pieces, and now with the internet you get the benefits almost instantly, and so the cost-benefit scale is very different.
This is why I think of engineering as the process of paying a very expensive person to make something else cheaper. :)
Oh, certainly! I found these new details a few weeks ago while looking up the story for our junior engineers, as this was always the classic lesson of “seemingly-unconnected devices can still influence each other at the electrical level”.
Or to put it another way, “next time, politely tell the internal helpdesk customer to turn off their desktop plasma globe when trying to use their company-issued yubikey”.
That’s so cool that we can see the switch!
Is it bad to treat this as a to-do list?
FIXME:More like “commit this crime too and see how it goes” XD
:D
I love this and it looks like the list now has more entries. I ended up making two podcast episodes just discussing some items from the original list.
This is an excellent resource! I clicked on https://folk.idi.ntnu.no/agnar/it272/pekere/local/newsci/primordial.html. That looks to be an enjoyable rabbit hole.