The happy state of 2D painting and 3D modelling open source software.

• MyPaint and Krita are, together, great software for 2D painting. Both have great default brushes and engine.
• Blender 2.8… is a beast. It’s a whole new level when compared with old Blender, new rendering engine, old stuff removed, interface revamped. I can see Blender becoming industry standard in the following years.

C compilers.

10 years ago, there was only one C compiler that anybody used. Spurred by pressure (and ideas!) from clang, gcc has been improved enormously since then. We now have two excellent-quality and nearly-compatible C implementations with better-than-ever performance, optimizations, and diagnostics. Tools like asan have worked miracles for correct code.

Since nobody else has written about Rust yet, and you mentioned it, I’ll bite.

I guess one of the problems of waxing poetic about Rust is that it has kind of already been beaten to death. Oodles of people are excited about Rust. Since it’s still growing, and since there is no zeal like the zeal of the newly converted, there are a lot of people singing its praises. To such an extent that an entire culture war has evolved around it, including right here on Lobsters. I’m a biased observer, and I get my hands dirty in it, but if I try to pop up a level, it really is an interesting phenomenon. But that’s a topic for a different day…

I am not new to Rust, so I think if I was ever touched by the zeal of the newly converted, it has assuredly worn off by now. With that in mind, I’ll try to avoid just regurgitating the stuff other people say about Rust, but that’s hard.

I think the Happy State of Rust is its entire package. I’ve written code in a lot of programming languages and ecosystems (with Windows and the APL lineage being at least two of my major blind spots), and I feel reasonably safe in the conclusion that Rust’s complete package is fairly unique. My main shtick is being relentless about being upfront about trade offs, and there are few ecosystems out there that let you express the range of trade offs offered by Rust. On the one hand, you can dip down deep into high performing generic data structures that benefit from the amazing work of optimizing compilers like LLVM, but on the other hand, you can also stay in the relative comfort of a high level application oriented language when you want to. The former might require arcane knowledge of what exactly is and isn’t undefined behavior while the latter might cause you to say, “UB? What’s that?”

If you pop up a level, that’s a really impressive range to encapsulate in one ecosystem. It’s not like that range hasn’t been done before. Look at Python for example. The typical way you might see that range being expressed is with modules written in C with a nice, comfy and safe Pythonic API layered on top of it. But there’s just all sorts of trade offs there. First, there’s the friction of dealing with Python’s C module boundary. Second, there’s the inherent cost in providing that boundary in the frist place. And thirdly, there’s the cost of Python itself.

I am perhaps running the risk of putting up a straw man, but that’s OK. The fact that “write low level code in C and wrap it up in a Python module” is a very popular approach to expressing the range of trade offs I’m referring to is I think enough. An example might help.

Consider something as simple as counting the number of fields in a CSV file. The Python code is deliciously simple:

import csv
import sys

count = 0
for record in csv.reader(sys.stdin):
    count += len(record)
print(count)

The Rust code is a smidge bigger, but not materially so:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut rdr = csv::ReaderBuilder::new()
        .has_headers(false)
        .from_reader(std::io::stdin());
    let mut count = 0;
    for result in rdr.byte_records() {
        let record = result?;
        count += record.len();
    }
    println!("{}", count);
    Ok(())
}

Running the program on a 445MB CSV file from the Open Policing project with the Python program:

$ time python happy-state.py < /m/sets/csv/openpolicing/MA-clean.csv
78620877

real    5.844
user    5.777
sys     0.060
maxmem  8 MB
faults  0

And now Rust:

$ time ./target/release/happy-state < /m/sets/csv/openpolicing/MA-clean.csv
78620877

real    0.915
user    0.873
sys     0.040
maxmem  6 MB
faults  0

Despite the fact that Python’s CSV parser is written in C, it just can’t compete. Rust’s CSV parser (disclaimer: I wrote it, I’m using an example I know well) is itself a very complex beast. But as in the Python ecosystem, all of this is well hidden from typical users of the library. It’s got some gnarly internals, but users are presented with an API that is at least close to the simplicity of Python’s API, except it’s also 6 times faster.

And this is just the beginning. Since Rust’s range of trade offs can really be cashed in almost everywhere, you can make small tweaks to your program to make it run even faster, if you’re willing to put up with a bit more complexity:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut rdr = csv::ReaderBuilder::new()
        .has_headers(false)
        .from_reader(std::io::stdin());
    let mut record = csv::ByteRecord::new();
    let mut count = 0;
    while rdr.read_byte_record(&mut record)? {
        count += record.len();
    }
    println!("{}", count);
    Ok(())
}

And running it:

$ time ./target/release/happy-state < /m/sets/csv/openpolicing/MA-clean.csv
78620877

real    0.769
user    0.729
sys     0.037
maxmem  6 MB
faults  0

The speed boost is small, but things like this can add up as your data set grows. And keeping in mind this is just an example. The point is that you just can’t do stuff like this with Python because Python doesn’t really let you easily amortize allocation. You’d likely need to drop all the way back down into C to get these kinds of optimizations. That’s a lot of friction to pay for.

OK, so why am I picking on Python? Surely, my point is not to say, “See, look, Rust is faster than Python!” That’s silly. My point is more holistic than that. It’s more like, “See, look, you can be in a comfy, safe high level language, but are also free to trade code complexity for faster programs without a lot of friction.”

Other ecosystems like modern C++ start to approach this. But obviously there are some critical distinctions there. The main one being that C++ is “unsafe by default everywhere.” You don’t get that same assurance of a safe and comfy high level language that you do in Rust. IMO, anyway.

And then there are all the other little things that Rust does well that makes me as a pragmatic programmer very happy. The built in unit testing and API documentation tooling is just lovely. It’s one of those things that Rust didn’t invent, of course, but it feels like I’m constantly missing it (in some form, to varying degrees) from almost every other language or ecosystem I use. There’s always some detail that’s missing or not done well enough or not ubiquitous.

I think I’ll stop it here. I wrote this comment with rose tinted glasses. Rust is not a panacea and there are lots of things about it that are inferior to other languages. For example, I presented one particular example where Rust and Python code have similar levels of complexity, but obviously, that does not necessarily generalize. There are other things that make Rust inferior in certain cases, such as compile times, platform support and certification. Language complexity is hard to fix, but I largely see the other things as at least having the potential to improve significantly. And if they don’t—or don’t improve as much as folks hope—then that’s OK. Rust doesn’t have to be All Things to All People at All Times. The rosy picture I painted above can still be true for men(and hopefully many others) while not being true for others. Trade offs aren’t just purely technical endeavors; costs that may not matter as much to me might matter a lot more to others.

Nix and NixOS is what comes to mind, because they kind of rescued my whole interest in computers back in 2015, not by being perfect, indeed being quite rough, but by showing that it’s at least possible to have a packaging and deployment mechanism that makes sense, works reliably by design, and makes previously hard things quite nice.

In a sense it’s a single project (a “pearl in the mud”) but at the same time it’s a big octopus that embraces everything else, almost kind of like an Internet Archive for the whole interconnected free software universe. And it makes it possible, indeed fun, to make projects that span multiple ecosystems.

I recently made (for a client) a small web frontend app whose Nix-based build can run—reliably, on any CI server—a smoke test against a headless Chromium via WebDriver and hitting a fresh local Ethereum testnet with several contracts deployed, all recorded with ffmpeg, etc. Nix easily ties all this together in such a way that I’m confident it will work the same way years from now.

NixOS seems to me also like an optimal base for a potential flourishing ecosystem of custom distributions. You can generate your own virtual machine images, AWS images, live USB ISOs, etc. I’m looking forward to doing fun stuff with this. A couple of years ago just for fun I made a custom NixOS ISO that boots into a configured X system optimized for airgapped blockchain transactions, and I have a few ideas for kiosk-type projects and specialized systems for which NixOS would be perfect.

Computer data interchange in general. It’s far from perfect, but we have good, universal solutions for many very fundamental things that various companies, governments and research institutions fought over basically up until the 1980’s:

• Computers use 8-bit bytes, word sizes are powers of 2
• Computers use twos-complement integers
• Computers are mostly byte-addressed with little-endian words, and the places that are big-endian or word-addressed are well-defined and relatively rare
• We have a, not perfect, but very good standard for floating point and it’s used nearly everywhere
• We have a, not perfect, but very good encoding and representation for text and it’s used in… well, probably most places

These are all problems that I look forward to not having to re-solve in the next 50 or 100 years.

The wonderful world of ShaderToy! A community of graphics programming enthusiasts from beginners to seasoned pros. All code is open, the results are immediate, and everyone is in it for the fun.

The happy state of Haiku OS. It’s a completely open source operating system that is a joy to use, fast, simple, and elegant.

More importantly, the devs have managed to push the state of the art forward (packagefs is the most interesting package management strategy in years); and they’ve managed to build some of the best open source desktop hardware support anywhere.

3D Printing. It’s a glorious golden age for 3D printing. It’s all even opensource. I have an opensource printer, that I control with opensource software, on a libre operating system, printing designs entirely created in open source software. And the whole thing has gotten so reliable it’s almost unbelievable.

I’m not sure what to attribute all that success to exactly, but I think it’s probably some combination of that fact that the basic tech is fairly basic and very well understood (since the main thing stopping it taking off was patents before they expired), the fact that it overlaps with the general maker community who are used to rolling up their sleeves and getting things done and then giving that knowledge away for free, and the fact that there’s bespoke hardware to be sold & therefore companies that benefit from the sounding software ecosystem being good so they can sell more hardware.

Disclaimer: opinions are my own, and I no longer do IT or software development for a living.

Anyways, I work for an ISP (cable TV) doing installation and troubleshooting. One of our internal tools for telemetry (mostly modem signal level/quality statistics over time, used for provisioning service and diagnosing if there is a problem with someone’s service) is wonderful and checks all boxes.

It makes our jobs in the field infinitely easier. A customer with chronic issues with their internet or telephone service will call in, and have a technician out. When one of us is out at their home or business, we find nothing wrong with any of their cabling from the telephone pole to their modem. At this point, most technicians can only throw their hands up and say that it isn’t a problem here, have a nice day.

Our telemetry allows us to conveniently (from a smartphone) change orders for obscure provisioning issues, and most importantly, view statistics for an entire network (generally bound geographically) and say that yes, we are having issues in the area and that you, dear customer, are not the only person having issues with your service, look and let me explain. Now, I can use the proof I have from our telemetry to present this to our “outside plant” maintenance crew or my boss, and hopefully get things fixed.

Whether or not anything ever gets fixed in a timely matter is a different topic. :)

The development team is always looking for input and for ways to make our telemetry more useful. The user interface is intuitive, and the important information it is forthright with. From the outside looking in, that team is having one hell of a fun time developing a useful product, mostly because their objective is helping out our other departments, not making a profit or gaining internet cred.

• Wayland (and KMS/DRM/GBM/EGL/evdev)
  • we still have to deal with grumpy internet commenters who don’t think it’s good or important or there yet ;)
  • but we have a pretty clean, maintainable, understandable, hackable free desktop ecosystem now! Still with wobbly windows like in the “good” old days, but now our software stack is actually mostly sane and good.
• Mesa (sorry, single project, but I have to put it there)
  • isn’t it awesome that AMD and Intel employees constantly work on a common open repo with all the graphics drivers and their shared components?
  • the story of RADV is amazing: some German students decided not to wait for AMD’s open Vulkan driver and just wrote their own to learn about GPUs and Vulkan, and this ended up in Mesa too, and recently Valve sponsored an alternative compiler backend for this!
  • and there’s a ton of excellent reverse-engineered drivers for embedded GPUs
    • and some of them even became supported by vendors (most famously, Broadcom hiring Eric Anholt of VC4 fame to make V3D for the next hardware)
• ARM Servers
  • not fully mature yet, but AWS coming out with a second (and very fast) revision of the Graviton basically secured the growth
  • finally, x86’s monopoly days are coming to an end, and the winning alternative has a nice clean ISA and a good software ecosystem based on open standards (SBSA/SBBR)
• Open Source Firmware
  • EDK2, Coreboot, Slim Bootloader, TF-A — lots of active projects with vendor support! Of course we’re not there yet, there’s always blobs somewhere, there’s still annoying protections against custom firmware (Boot Guard) in consumer devices, but open firmware is taking off for real.
  • OCF is going to require open system firmware support in a year

The happy state of hacker-friendly hardware! Some examples:

• RISC-V ( https://riscv.org/
• Pine64 ( https://www.pine64.org/ )

It is possible to program FPGAs using entirely open source tools!

A project called SymbiFlow aims to combine a synthesis tool with multiple bitstream backends, amounting to what it titles itself as the “GCC of FPGAs”.

Of course, open source FPGA work isn’t really taken seriously by the FPGA industry. There are big support contracts involved and so on, but like the prevalence of open source software in industry these days compared to ten or twenty years ago, I suspect it’s a matter of time before this mental model of open source software catches on in the industry.

Here is a nice presentation about what you can use FPGAs for in your own projects.

WebAssembly. While it’s still a nascent technology that has yet to win major mindshare, the work that is being done in this area will help bring us faster, safer and more reliable web applications, as well as making a bevy of existing software more useful and portable.

In my opinion, Berkeley Packet Filter (BPF) was a very elegant and successful project. It introduced a safe way of “scripting” the in-kernel packet filter and formalized an approach on how to do such things in general, which would beforehand be unthinkable: Uploading user code to run in the kernel? You must be mad!

The BPF approach was adopted by Linux, which later added a JIT-compiler to make it run even faster and extended it to be used in the context of defining security policies for applications with seccomp-bpf.

UPDATE: Also, thank you for this. We need more positive notes, especially in times like these.

I want to digress (but see the last two paragraphs).

These “sad state of” things do not, in my opinion, reflect the state of the subject under discussion so much as the expectations of the writer.

In the most recent case, the writer is sad about GUI things, and writes stuff about Qt and others that’s astonishingly similar to what similar people wrote twenty years ago, when I worked at Trolltech, with hardly a change. He’s sad (why do I feel sure it’s a he?) that… something. The something could have been more specific, but I did get the gist of it.

The software he writes about either has a business model that produces income, or doesn’t. As long as I paid attention to this, both kinds behaved as one might expect. The maintainers with a business model paid attention to whatever was compatible with their income, and were reluctant to take on responsibility for anything incompatible. The ones without a business model did whatever interested them, and were nice people who’d spend a bit of time helping people, too.

This is what you should expect. Neither class tended to merge patches that have nothing to do with their business model or interests (for separate, different reasons, and in both cases the reasons make sense). Neither class tended to implement stuff that interests you but not them, or do it in the way you want but they/their customers don’t.

IMO this is fine. It’s what should be expected. People with customers should serve their customers, people without should write some interesting code and then go for a lovely picknick with their beloved. IMO that’s fine, and people who consider it a sad state should adjust their expectations…

Therefore, I’d like to mention the happy state of Qt, which has had better documentation than whatever you work on, and has kept that up for for more than twenty years. That’s praiseworthy longevity. The team has also always been careful about maintaining compatibility — the changes necessary in order to go to a two-years-later version of Qt have always been admirably small.

I could say something similarly positive about most of the items on that list. I choose Qt because it was first on the naysayer’s list, and because I know it better than most other GUI libraries.

The scientific Python community is amazing. Astronomy, image processing (usually biology), earth sciences, just learned about magnetoencephalography and electroencephalography, bioinformatics, chemistry… list goes on and on.

• Jupyter notebooks help introduce people to programming and data analyses.
• All communications software that helps people talk to each other during the different lockdowns

the go tooling (compiler, modules, fmt, guru, etc.)

So, I wanted to get back into programming. I was re-learning Python by just using it for several, personal projects. Where I work, we were getting slammed in all areas with horrible technology and processes holding us back. This was causing people misery in and outside the company in large numbers. I figured out a way to speed things up several times with an out-of-band system that was simpler. I got the prototype together, manually entered all the data since I can’t connect it to the network, and our productivity improved several fold. After some practice iterations, I wrote that quick and dirty prototype during about 20 min of a lunch break. It may or may not get adoption on top but coworkers thought it was best day we had. Yay!

The reason for the comment isn’t my program, though. Its interface was horrible even if efficient. My memory is too broken to even program right. So, I was mostly doing Print-Oriented Programming & Debugging on the side projects until they worked. I had to DuckDuckGo all kinds of problems since I was coding before learning the language in the subset I used in all of them. I eventually got Visual Studio Code, pylint, etc. I built up a set of functions to do things like I/O and requests I kept having to deal with. Eventually, it was just easy to write that one program within the IDE. All the problems it immediately found in the bigger, earlier project I did in Gedit made me wish I used it then, too.

What I saw in all this reminds me of a Mr Rogers quote I saw recently. At each point, there were people there helping me. They built the interpreter, made a better IDE, made its error-finding extensions, made pretty good docs, answered folks questions on forums (not just SO), wrote prototypes of code to give me headstarts (big help with Flask), wrote great error messages, made me libraries, etc. Unlike others I tried, the Python ecosystem had all of this in such a way I could keep making progress while barely knowing what I was doing. As I learned for real, I made progress even faster with more code I could keep.

Although I learned its weaknesses, the empowering effect this huge ecosystem had keeps me doing Python with less time on the better language. Gave me a good idea for what other language I need to be using to address those deficiencies. Might learn it, might not. Meanwhile, I cranked out some more code today for my custom, bookmarking system. I used Python knowing others’ got my code’s failings covered with their tools, docs, and helpful write-ups. It’s pretty awesome compared to when I first started programming.

I’m a fan of Prometheus. It’s operationally simple and rock solid. It’s equally easy to set up Prometheus for a traditional bare metal server farm and containerized microservices. Compared to Graphite/Carbon, which despite its merits necessitates the operator to learn how to operate 38 different Python daemons.

We have some big Epyc 512GB RAM regional primary pollers scraping tons of metrics, as well as customized/mini containerized Prometheus + Thanos shipping to Ceph. Versatile.

silverblue has been a really interesting ride for someone looking for a home between conventional linux installs and nix. The benefits of the mature Fedora+rpmfusion repositories with ostree have made me feel like I’ve finally found a combination of stable/rolling that matches up to the lifestyle I want to have with my tooling.

Installing stuff. Ok, when I started using computers, there was no “installing stuff”, but then if your stuff wasn’t on a ROM cartridge you had to know at least one different command for each model of computer to get it to listen to the tape.

Then when home computers started to get hard disks, you had to know whether to copy the stuff from the source media into the hard disk, or whether to run a thing, and what virtual drives you had to map where to get it to work. If your thing came from the internet or a BBS, you had to have extractors for whichever crazy lha/hqx/sit/lhz algorithm the person on the other end had used. Of course your program might not work if it didn’t have the right file in L: or /lib or wherever. Of course, another program might not work if it did.

And that was after you’d got your operating system into the disk, having had to know things like which IDE device the disk was on and what partition layout you needed. Some of the bad ones asked you how big a partition you wanted for swap, or whether to “bless” a certain file.

These days, almost all operating systems have reasonable installers, and almost all have simple systems for adding software. The good ones give each additional application its own environment so it doesn’t mess up the libraries for everyone else.

The happy state of developers spectrum. I like how if you are a developer you can choose to be a preacher of a new shiny thing (see neighboring comment about rust for example) or a dark matter developer (https://www.hanselman.com/blog/DarkMatterDevelopersTheUnseen99.aspx). You can also easily switch from preacher to dark and back. Freedom and diversity is everything!

For me in recent years:

• The Rust ecosystem. The honeymoon is over (I have been using Rust for a few years). But I still like the language, the package manager, and several high-profile crates a lot (ndarray, petgraph, etc.). For me Rust has changed programming a lot in that it’s made it possible to make high-level abstractions which are still performant.

• Nix. I am probably still in the overzealous honeymoon phase. But for me there is clearly a tech life before and after Nix. It’s certainly not perfect, but it has made many aspects of software engineering easier for me: defining development environments (with direnv/nix-shell), making CI builds more robust (since you can trivially pull in all kinds of dependencies), defining systems declaratively, having access to a huge package system, and the community is very friendly and makes it easy to start contributing. My only qualm with Nix is that it is probably has too much of a learning curve and is too different to see a wide adoption.

• The Linux ecosystem. For a long time, I have been very critical of Linux, despite loving it and having used Linux in some form since 1994. In recent years, things have really been started to get together. System management has become easier, better documented, more uniform thanks to systemd. The graphics stack has really improved with Wayland, Mesa, and desktops that move to Wayland. Flatpaks + portals + pipewire are slowly bringing truly sandboxed applications.