Today’s interview is with Lobsters user Ted Unangst. His weblog is at tedunangst.com/flak/.
What do you do for work? How long have you been working at it?
Around these parts, I’m perhaps best known as an OpenBSD developer, releases 3.4 and up. At the time of writing, my account has been active for 3993 days. I stick to the src tree mostly. For a while I was something of a filesystems and VM guy, but by feigning ignorance I have tricked other people into taking over.
I used to work at Fog Creek Software on the FogBugz Bug Tracker, releases 7.x and 8.x. I worked on pretty much literally everything except the actual tracking of bugs.
Before that, I worked at Coverity on C and C++ static analysis, releases 2.x and 3.x. There I did work on every part of the product, and also did some customer facing stuff like support, training, technical sales, etc.
How did you get started with OpenBSD? I feel like it’s a pretty intimidating project for new people to contribute to.
I started using OpenBSD around 2000. At the time I was using all of the operating systems; FreeBSD, OpenBSD, Linux, QNX, etc. I didn’t care too much about open source, just so long as the price was right. Then slowly the other system started to fall by the wayside. Whatever advantages they had, I didn’t notice; but the simplicity of configuring and setting up OpenBSD led to it becoming the default choice whenever I needed to experiment with a new service. (My Debian installation bit the dust because I wanted ssh2, which at the time was in unstable. Upgrading to ssh2 meant upgrading the dependency chain that went something like ssh2 -> openssl -> apache -> mod_perl -> perl -> ifconfig. Turns out that version of ifconfig was totally broken and without network to downgrade I now had a space heater under my desk.) Since then, I’ve gotten a lot better at sysadminning a variety of systems, but at the time OpenBSD best withstood the wild thrashings of a novice without imploding.
Eventually I got better at programming, and started trying to port code from FreeBSD and NetBSD. This wasn’t exactly warmly received, because in many cases there was a reason such code had not been ported, but the developers pointed me in the right direction. I started spending more time reviewing the existing code and reading the bugs@ mailing list and trying to solve real problems. Eventually that led to me getting an account. One of the first things I worked on (with a few other developers) was ridding the src tree of strcpy calls, replacing them all with strlcpy.
What’s the toughest bug you ever tracked down? I imagine you would have a few good stories from Coverity.
There are a lot of crazy stories from Coverity, though I can’t tell too many of them here. One thing I did gain was a better calibration for crazy. We had to integrate with the customer’s environment, and they always did something crazy, so we’d start with the typical five whys investigation, which barely scratched the surface. You needed to ask fifty whys. It sounds crazy that the Windows build machine would, mid build, ping a cgi script running on a Linux box. It sounds crazier to learn that the cgi script would assemble a disk image, launch dosbox and run the compiler from autoexec.bat. But eventually you learn they’re building code for a microcontroller using an obsolete 16-bit compiler that doesn’t run on Win64. Many of our customers were in the medical/aerospace/military fields and had strict validation requirements. They couldn’t upgrade the microcontroller or firmware just because it would have been convenient to do so. All of these insane bureaucratic externalities manifested in code. Why are you wasting effort building a space pen when a pencil is better?
What is your work/computing environment like?
I like ThinkPads, currently using a T430s + 24" 1080p monitor. I also have an X200s and T60, and some HP laptops I don’t talk about. About a year ago, I built a midrange gaming PC with a 27" 1440p monitor, but I never turn it on. iPhone 5, iPad Air, Kindle Paperwhite, plus all the regular networking paraphernalia.
If I’m typing for a long time, I’ll switch to dvorak, but it’s a lot easier to use qwerty.
I’m sitting on an imitation Eames Aluminum Group chair.
What software are you most often using?
I’m pretty happy with the Windows desktop environment for everything that’s not software development. There are a few exceptions, where I’ll use Visual Studio and C# to build desktop apps, but otherwise everything is developed on OpenBSD. dwm, xterm, ksh, vim, cvs (OpenBSD) and mercurial (everything else).
OpenBSD development is pretty much all C, all the time. For fun projects, I mostly work in Lua, specifically luajit. As needed, I fall back to python to use a library (pygments, lxml, feedparser) and communicate over a socket. My next project will be in go. Or maybe the one after that.
What’s an interesting project you’ve been working on recently?
Currently working on a scheme to cryptographically sign OpenBSD releases. Earlier mention of signify. This has been a fun exercise in reinventing the wheel. It started off extremely simple, but then little by little features keep creeping back in, and next thing you know, you’ve turned into the thing you hate. Well, not quite, I’m still happy with it. A more complete writeup can be expected after we ship.
For fun, I usually reinvent other wheels in Lua, like RSS readers. Internet web tube stuff. Unfortunately, dealing with real world inputs quickly saps the fun out of these projects. With a tool like signify, I design all the inputs, all the use cases, all the installer integration. My world, my rules. In contrast, trying to extract the one true original link out of a feedburner RSS would stump the brightest minds at the NSA.
A while back I wrote my own email client in ruby, because that too had never been done before. It went really well until it went really not well at all. I still use it, but it’s been on life support for more than a year. Lots of reasons why, but I’ve sworn off ruby for pretty much forever. Inevitable rewrite in Lua coming soon.
What is something new you’ve used or integrated into your work that has made a positive impact?
A software configuration change and work habit all in one! I use the default, or nearly the default, config for as much software as possible. My .vimrc file is slightly different on every machine, because I start with the example. Then when something really starts annoying me, I change it. A lot of the time the thing that annoyed me is particular to the task I was doing on that specific machine at that specific time. The annoyance doesn’t recur, even without syncing configs. I used to have a pretty complex .vimrc file with magic macros and whatnot, but then I’d use another computer and be helpless. I miss the syntax highlighting and the slightly more intelligent indenting, but otherwise I’m about 98% efficient using plain vi.
Even better is software like dwm that I can’t configure. It eliminates all the mental friction from wondering if I could make it work better. Reformed Window Maker user here. I was constantly trying to decide which of the seven mouse focus modes I wanted to use. Then I tried xfce, but every upgrade changed the icons and I’d have to try to find a way to get the old icons back, which is a task just this side of impossible. If it had truly been impossible, I would have settled for the new icons without a second thought.
I think of it as optimizing the pessimistic case, not the optimistic case. Squeezing another 1% productivity gain out of your tools is nice, but eliminating the possibility of a once per year crisis time 50% productivity hit is nicer.
Something I think was omitted as just a personal remark, but it’s probably worth adding: kb disagreed with me about my approach to tooling, which is totally fair, to which I replied:
I don’t think I’d actually recommend the minimal tool approach. I started down this road because I like to tinker too much, and the more time I spend on my tools the less time I spend using them. I can only claim it makes me more productive than otherwise.
This interview was brought to you by @kb. Thanks, Kevin.
[Comment removed by author]
T430s - Mostly always runs Windows, but can dual boot into OpenBSD installed on an msata card. The main things are that 2 of 3 USB ports are USB 3 and don’t work in OpenBSD and the battery life is substantially worse. It is otherwise perfectly usable; in fact the iwn driver seems more reliable than the Windows equivalent, which I need to toggle on and off frequently.
T60 - ran OpenBSD for years. Now runs Windows. X200s - Always ran OpenBSD, still does.
I know you said “Lots of reasons why”, but could you expand on why you’ve sworn off ruby? I’m not a huge fan of ruby myself, and it’s interesting to see why other people don’t like it either.
Short version: all the gems I used were broken. I’ll fully admit I may have had a bad experience, some bad luck, etc., but when everything goes wrong, you’re not inclined to keep trying until it goes right. It was mostly about the gems, not the language, but if I’m going to write everything from scratch, I prefer Lua.
For the database, I started with dbi but certain usage patterns didn’t work with the sqlite driver. File bugs, and it’s like dbi says sqlite is wrong, sqlite says dbi is wrong, it used to work with this version, that project’s abandoned but for the past three years we’ve been rewriting it all for 2.0… Giant runaround. There was, maybe at some time, a version of dbi and a version of sqlite that worked together, but it was like searching for the holy grail. I wrote a dead simple ffi binding in less time than I spent wrestling with the above. This was the kind of wheel reinvention I would actually prefer to avoid. Talk to ruby people. “Oh, everybody knows to use ActiveRecord.” I didn’t want ActiveRecord; it’s a lot more than just a database connection.
I’m writing a mail client, so I have a ton of encoding issues. Poorly or incorrectly encoded mail still has to be processed. I can’t leave it in my maildir. I started with 1.8.7 (not unicode aware). This worked decently apart from a few edge cases. The important thing is you try to deal with everything in UTF-8, but failing that, at least dump the bytes into the database so I can read them back later. Unfortunately, ruby 1.9 switches to unicode awareness, which is kind of good, but not. You can’t have bad encodings running around anymore, but they already are running around. I went through the work of converting to 1.9 only to discover it crashed constantly. Too much work to maintain fallbacks, though in the end I think I had my code at least working.
Relatedly, I use the mail gem for parsing. It has bugs. They get fixed, I upgrade, it has new bugs that are harder to workaround. Supposedly it works really well with 1.9 and encodings. In practice, it was a disaster. Way too many uncaught exceptions for bad encodings. I dug in a bit fixing it, but it was overwhelming. I’m stuck at 1.8.7 until the end of time.
Another example. I would expect this to work:
Instead it would throw exceptions about invalid characters in decode. Meaning json.encode was creating crap. Not being able to roundtrip data through an encoding library is pretty bad form.
In the beginning, I really believed in the future of ruby, and tried to be a good sport. I tested all my code in 1.8, 1.9, rubinius, and jruby (on windows, mac, and openbsd). I discovered far more bugs in other people’s code than in my own that way. jruby support in particular, which I need because I wrote the GUI in swing (for serious), is terrible. There will be a gem with the same name, but it’s actually a reimplementation, and I guarantee you it will handle (e.g. nil vs “” function parameters) differently than the mri gem.
Long story short, I went off the rails reservation and discovered that anything that’s not rails doesn’t work as advertised. (I have no experience with how well rails itself works.) And then to top it off, googling for solutions leads to blogs and stackoverflow answers where everybody assumes you’re using rails and have all the monkey patched methods attached to all your objects, but there’s no way I know of to discover which gem I could use in isolation to get that function.