1. 50
    1. 10

      This is pretty cool, gets me excited about computing again. I’ve come from a more web/api oriented background and for some time I’ve been curious about hardware/firmware. I’m pretty excited about where we might go with compute in more and more places but most my experience is running cloud services so firmware seems uncharacteristically unattainable compared to web/app stuff for me.

      Does anyone know a meaningful way to get involved that isn’t necessarily working at one of the few large bureaucracies funding hardware, and isn’t collecting anymore raspberry pis?

      1. 5

        I found retro-computing hobby to be a great stepping stone to getting to work on hardware-software interface full time. I started by needing a terminal board with serial interface on one end and VGA plus keyboard on the other. At first I built an existing design by Geoff Graham and then gradually improved its hardware and software. It was good experience learning microcontroller programming and the never ending intricacies of historic VT terminals.

        1. 2

          This sounds so interesting, specifically seeing the interest in retro-computing on circles like sdf.org, gopher and gemini places…

          For me retro computing is not really nostalgia so… if somebody was curious about retro computing, where do you think I’ll be a good place to start? Commodore? Atari?

          1. 3

            I can only reflect on my personal story of building simple logical circuits out of what was series 74 analogue in the USSR when I was in my early teens. As well as reading amateur radio magazine of the time explaining how to build home computer from Soviet version of 8080. All this led to recent me building primitive CPU from series 74 (to which I finally have unfettered access!) and trying to interact with it via serial terminal. This CPU was closer to Apollo guidance computer than to microcomputers of the 70s, allowing for exciting clock by clock debugging experience.

    2. 5

      Slightly disappointed that this isn’t the talk where Oxide actually releases Hubris, but still an interesting talk.

      About the dubious distinction between firmware and software: I imagine there are many applications that run on single-use single-board computers running an OS like Linux, that could instead run on a microcontroller (especially a higher-end one like the ESP32), if only the task of developing for such a resource-constrained environment were more approachable. And it sounds like Rust does make that type of development more approachable. So it’ll be interesting to watch what people do with that.

      1. 12

        In my experience the thing that makes working with ESP32 and other microcontrollers so unapproachable is the proprietary software, complete with broken web sign-ups forms, required to get the pin configuration and initialization code for your specific board(s). If I took over as CEO of one of these microcontroller companies the first thing I would do is dump all that code on github and spin up a discord server for interactive free first level dev support. I’d then re-orient my sales strategy around those venues. Wrapping the initialization code in your proprietary IDE is stopping potential customers from becoming customers and then repeat customers. It’s definitely not stopping the competition from learning anything about your products.

        1. 7

          I wonder if we’re looking at it wrong; I suspect open-source-everything indiviual types are a small minority, and the vast majority of their customers are corporate 9-5 people who want an IDE (because they don’t have xterms with GCC) for developing whatever washing machine firmware they’re doing today.

          1. 5

            I think that is probably how the microcontroller execs see it, probably because that is what they hear from their sales force. It has no doubt served them well this far. However, in line with the theme of the talk, I think the potential size of their industry is bigger than they do. I think there is a lack of density in the matrix of skills x products that is preventing a smart phone-esque development in the way we deploy embedded computers.

            1. 3

              The open source elite few are small in numbers as much as we like to talk about ourselves. Most people aren’t like Oxide, designing cutting edge firmware. They want to make the widget work, and don’t care about contributing upstream fixes nor a custom toolchain. There’s a lot more blub programmers than you think. Those IDEs and toolchains they provide are a hell of a lot more accessible to the clock punchers than rawdogging gcc, and I dread to imagine what a Discord for it would look like.

              1. 3

                I don’t mean to say the code should be on GitHub as an avenue for the companies to benefit from community contributions. The code is already open source in the sense that it is freely distributed and open to inspection. I’m suggesting GitHub as a way to completely lower the barrier to entry in a way that lets you retain a view into the identity and further future needs of the people using it. The embedded world and the ubiquitous compute future it could enable has a talent pipeline problem, which demands recucing all potential barriers to experimentation IMO.

      2. 8

        This is a little more complicated than it looks, not the least because the distinction between firmware and software is indeed pretty arbitrary.

        At least in my experience, in terms of volume, the competition for “this is running on Raspberry Pis but a Cortex M3 would suffice” is mostly Python and JavaScript, not C. You mostly take the price hit because you either need a lot of user interaction so you need something that will put an AngularJS app on a touch screen, or because you need things like clown cloud connectivity and the like and you’d much rather just run the same MQTT and web stack everyone on StackOverflow runs. (In commercial scenarios at least, hobbyist and low-volume devices for personal use are a whole other story). I can’t remember ever doing a Linux BSP for consumer hardware that didn’t fall under one of these two.

        There are exceptions but nowadays, especially on the consumer hardware side, these are mostly confined to things like devices that have to run on battery. Otherwise, especially if you have some user interaction going on, hardware that can put an Android app, or a QML or Angular UI on a touch screen is so cheap that it’s getting hard to convince users that the price difference is worth it.

        There’s definitely a good case to be made for Rust in spaces where the real competition really is C and C++. Things like BMCs, modems, EFI firmware, absolutely. As in, it’s more than worth paying the initial overhead of getting Rust programs running on said devices.

        Thing is, changing the language that you use to develop for platforms like the ESP32 (which, FWIW, is really above average :P) doesn’t generally make these things more approachable. Usually it makes it worse. Some of the bad parts of the stack are still there – you’re still stuck with quirky GDB interfaces or proprietary IDEs, expensive debuggers, proprietary flashing tools and so on. And the bad parts that you’ve replaced (e.g. the language) come with their own headaches. You’re now using a language that the vendor doesn’t support (which quickly gets your support tickets closed) and doesn’t have too many other users on that platform. That’s nasty to deal with on these gizmos, where project boilerplate is almost never written from scratch but copy-pasted from vendor examples, not the least because of things like replacing .bss with .BSS in the linker script ends up producing images that crash the flashing tool. (I wish I were kidding).

        1. 7

          not the least because of things like replacing .bss with .BSS in the linker script ends up producing images that crash the flashing tool. (I wish I were kidding).

          There’s the flashbacks from the war, thank you for that. I came back a scarred and broken husk of a man, knowing that the one true God is the build system and its holy spirit the bootloader and that memory is never, ever, safe. Not once, but thrice, have I had to wrie IAR linkmap parsers that shuffles things around because-it-is-better-that-way(TM). It was a long time ago when I last was surprised to see .xls and netcat as part of embedded tool-chains.

          In that context … cargo will have several uphill battles.

          1. 3

            Oh yeah. Sometimes I think people in this field are so cynical about things like Rust partly because there are lots of other, simpler things that would have a massively greater impact, like, say, a debugger that doesn’t suck. (Also because now we could’ve been writing Ada but I digress). A compiler for a more advanced language sounds like a good idea, I mean sure, it’s great, but aren’t we skipping a few steps here, I mean we’re regularly running out of goats to sacrifice just to keep the flashing tool working…

            You know, mostly by accident, I’ve actually not done a major embedded project (I mean for money, not fun) in about two years now and I am now seriously starting to doubt I have the nerves to ever go back at it again :-D

            1. 1

              There has been a steady supply of memory safe languages well within resource constraints for a long time, Rust is far from unique or even interesting in that regard. Being able to recruit engineerings willing to work in the space, with sufficient skills across the range, is a tall order even without a niche language constraint. Safety and quality measures with higher chance to succeed (model checkers, fuzzing, sanitisers…) are hard sells as is. There is, indeed, so many more relevant targets for improvement within build chains, flashing tools, mastering processes, post-mortem report generators, .. Even just not having to see gcc 4.x and linux 2.6.x in something produced this year would be a pleasant change. Speaking of debuggers, were you ever exposed to Lauterbach and the venerable Trace32? I am only ‘indirectly’ involved in embedded development these days, and only in relatively unproductive ways – it turns out that just breaking the things and pointing out the mistakes of others with no intent of fixing it yourself is more fun.

              1. 2

                I’ve heard good things about the Trace32 but I haven’t used one. In fact – and this is just another symptom of the universal care for quality on account of which we’re still dealing with gcc 4.x and Linux 2.6.32 in 2021 – back when I wasn’t a contractor, and I was still doing a lot of embedded work, my experience with debuggers was 20% using them and 80% explaining people what they are and why we need one. There are Linux consulting shops out there that unironically have a single JTAG debugger in a single department of 100+ people, only one or two of which know how to use it.

                One of the things that I found out the hard way is that there are many continents in the world of embedded development. That explains a lot of this disparity and it’s not immediately obvious from the outside.

                On one end you got people doing avionics or medical devices or whatever, who generally work, if not with good tools, at least with the best tools their vendor can supply. I don’t have too many good things to say about VisualDSP++ & friends, for example, but back when I was working in this field, it was pretty much the best there was, and in general, there was never any question of whether we want to use a better tool or not. Within time constraints (indeed, working in this space is a little scary, so we were perpetually understaffed), if anyone came up with an idea that seemed good – fuzzing, model checking, better logging, better automated testing – we always got the green light and the budget to get extra hardware as long as it was clear we were improving things, not playing with expensive toys.

                At the other end you got the people doing things like touch panels for HVAC systems and Bluetooth speakers and whatever, who are the overworked victims of cost slashing. I was fortunately exposed to this only by proxy but this is the main audience of Linux 2.6 BSPs, mostly because it’s the cheapest there is and you can get out of most bad places with Stack Overflow. These are the places where product managers cross out firmware upgrade off the feature list because they’re saving money on flash storage, plus their SDK is an ancient Yocto-based blob with 400 patches, none of which apply anymore if you unfreeze a single package. Selling fuzzers (or debuggers) to these guys is a non-starter. What are they going to do with them, it’s not like they can get any of the fixes they make on the device.

        2. 1

          This is one of the reasons I really liked AVR chips when I was doing small projects. AVRGCC “just works”. It lagged a little bit for the latest chips if I remember correctly, but for almost every one you could avr-gcc your app and flash it via ISP. No surprises. Way better than keil’s toolchain for c51 from way back.

          1. 1

            The AVR port is a little older and it’s really cool but I have to say, at least back in 2012 when I last used it on a reasonably non-mainstream architecture, Theo de Raadt’s old rant about GCC held surprisingly true.

            At one point, the MSP430 port, which was also very “just works”, would generate (under specific but IIRC not terribly difficult to hit conditions) some extremely efficient code for reading volatile variables, specifically, in the form of a few nop instructions. Debugging that was about as fun as chewing nails, minus the bragging points.

        3. 1

          Things like BMCs, modems, EFI firmware, absolutely.

          I’ve spent the last year porting OpenBMC to several semi-custom x86 servers I’ve had a hand in designing. OpenBMC was originally python, but that was too slow on a relatively underpowered BMC so they switched to C++. I suspect this would be a really good project to rewrite-in-Rust, but current maintainers are happy with C++ and a re-write would be a lot of work for limited functional benefit. OpenBMC’s design makes it possible to re-write one piece at a time if someone was interested though.

          Up next year is looking at replacing the BIOS/UEFI on the same servers. The obvious choice there is coreboot which I believe is mostly C with a smattering of Assembly. I’d imagine this is a lot harder to rewrite-in-Rust.

          So yeah, Rust sounds like a great idea, but in practice it seems fairly limited for a lot of this stuff. This might just be inertia and with Rust support finally landing in upstream Yocto just a few months ago this may help move the needle.

          1. 2

            Up next year is looking at replacing the BIOS/UEFI on the same servers. The obvious choice there is coreboot which I believe is mostly C with a smattering of Assembly. I’d imagine this is a lot harder to rewrite-in-Rust.

            I don’t know offhand how it compares with Coreboot in terms of maturity and hardware support, but Oreboot is pretty much exactly that…