This is the weekly thread to discuss what you have done recently and are working on this week.
Please be descriptive and don’t hesitate to champion your accomplishments or ask for help, advice or other guidance.
This is the weekly thread to discuss what you have done recently and are working on this week.
Please be descriptive and don’t hesitate to champion your accomplishments or ask for help, advice or other guidance.
I just happened by Lobste.rs after many months, and will take a couple of paragraphs to explain my history before the past week, if anyone can bear with me..
For the past couple of years I’ve been researching ways to write software that make it easier for newcomers to understand rather than for insiders to maintain. In this quest I built a toy OS and Basic-like language called Mu which tries to “rhyme with” the design process by which Unix was coevolved with C. The big additional design constraint compared to Unix+C is to make the OS primitives testable. I want to be able to pretend in tests that we run out of memory or disk, that a context switch happens between these two instructions, and so on. My hypothesis is that having the ability to easily write such tests from day 1 would radically impact the culture of an eco-system in a way that no bolted-on tool or service at higher levels can replicate: it would enable new kinds of tests to be written and make it easier to be confident that an app is free from regression if all automated tests pass. This would make the stack easy to rewrite and simplify by dropping features, without fear that a subset of targeted apps might break. As a result people might fork projects more easily, and also exchange code between disparate forks more easily (copy the tests over, then try copying code over and making tests pass, rewriting and polishing where necessary). The community would have in effect a diversified portfolio of forks, a “wavefront” of possible combinations of features and alternative implementations of features instead of the single trunk with monotonically growing complexity that we get today. Application writers who wrote thorough tests for their apps (something they just can’t do today) would be able to bounce around between forks more easily without getting locked in to a single one as currently happens.
There’s more details on my site, perhaps starting from my mission statement. I’ll also add below a couple of hopefully tantalizing bullet lists.
A. The zen of Mu:
B. Mu’s vision of utopia:
(I know this is all extremely unconventional and risky. I might well be wasting my life barking up the wrong tree. But it seems promising enough to be worth one lifetime. And if someone can persuade me that it’s a bad idea I’ll gratefully take what’s left of my life back.)
Over 2 years I’ve at least failed to disprove some aspects of the hypothesis in Mu, and I’ve had some success teaching with it and getting feedback on the design in the process. I’ve managed to convince one person – smalina – to join me in my quest[1], and the two of us have been investigating ways to bring the lessons out of Mu and into a ‘real’ stack, one with a real compiler and libraries that would permit more ambitious programs to be written, and with an eco-system of apps that we can try to port to our approach. The challenge here is a chicken-and-egg problem: we can’t make radical changes to a real-world OS without (at least) years of understanding all the things it’s trying to do. After much soul-searching we’ve decided to experiment with OpenBSD, not least because of tedu’s writings which show that at least a few people understand this platform from end to end. I have zero confidence that that’s true of Linux anymore[2][3][4][5], at least past the kernel.
Anyway, after that lengthy prologue, here’s what we’ve been doing in recent weeks:
Any critique of all this would be appreciated. Also help or pointers from people knowledgeable about OpenBSD. (Heh, I’d started to write about another issue in OpenBSD that had me stumped, but figured it out as I was writing. So this comment has already paid for itself.)
[1] He’s the one who reminded me of lobste.rs a couple of days ago, and here I am.
[2] http://queue.acm.org/detail.cfm?id=2349257
[3] http://web.archive.org/web/20131205090841/http://deadmemes.net/2010/10/19/fear-and-loathing-in-debianubuntu-or-who-needs-etcmotd
[4] http://landley.net/aboriginal/history.html
[5] http://harmful.cat-v.org/cat-v; http://harmful.cat-v.org/software; etc.
Your project is interesting. I’d say check out Oberon System and Smalltalk-80 as they were both designed for doing a number of these goals. They were successful, too, where Oberon got extended and rewritten by many students in small time while Smalltalk did more amazing stuff in terms of extensions or modifications. A safe language with good abstractions and generic interfaces is a start. Concurrency bugs also hurt users so maybe integrate something like Concurrent Pascal or Eiffel’s SCOOP to make it all race-free by design. The only thing here that really sets off a red flag is how you’re going into testing philosophy so much that you appear to be eliminating interfaces or a focus on them. Evidence going back decades suggest that’s a mistake.
Let me illustrate with a simple example: a function only works if the input intN is between 20 and 80 stored in a 32-bit integer since that’s language default. Testing to make sure your function is behaving as expected requires testing every single input. That’s around 4 billion tests. Alternative is using specifications or Eiffel-style Design by Contract to force it to be correct without the test. Here’s the specification approach: On entry, IF (N >= 20) and (N <= 80) THEN perform function ELSE raise error. A simple, interface check makes the problem impossible.
This effect scales up as program size increases. Number of states, interactions, and compounds of states go up. So, testing has to basically run through them all without missing one to recreate all execution traces that could be a problem. The result is a huge pile of tests that probably still will miss something. Whereas, Eiffel, Ada/SPARK, Frama-C, and even smart asserts show very few statements in each function enforcing pre or post conditions can prevent or catch whole classes of errors. Plus errors you don’t see coming because you didn’t know series of steps that would lead to them. Amazon’s use of TLA+ to verify its distributed code gave example of an error that would’ve required a sequence of 30+ specific events to trigger. They said such things happen in production but that code passed their extensive testing. Model-checking caught the problem in no time then proved nothing like that could happen again in modified version in all executions.
So, I suggest taking an alternate approach where you figure out a better way to handle interfaces. Make sure invariants are encoded in them to prevent propagation of errors, preferrably at compile-time. These also serve as straight-forward documentation of exactly what the function expects and will do. Make it easy to change them in a plug-and-play fashion reducing how many interface checks or implementation files need to be changes. Also, write a bunch of tests and documentation. Overall result of a safe language with interface checks, docs, and tests will get you closer to your goal.
Thanks! Yes, I am aware of those drawbacks of tests. Certainly there are some things that we can and should avoid by design, like memory corruption bugs, the races that you alluded to, and so on. Mu disallows out-of-bounds access by always having arrays know their length; eliminates memory corruption bugs by always refcounting allocations and only freeing memory when its refcount drops to 0; and avoids data races by transparently deep-copying any data passed through a channel between threads. These are things I definitely don’t want to rely on tests for. Seeing the word ‘security’ in your username reminds me that in encryption code in particular tests are of little use, at least the kinds of manually-created tests I’ve been talking about. For security you do want to be much more rigorous. The kinds of tests that can help there are Haskell’s QuickCheck tests and fuzzing (which I have very little experience with). But even those aren’t sufficient. I get the sense it just takes a lot more intensive effort per line of code to get good encryption libraries, and I don’t claim to be an expert in that area.
Invariants and Design by Contract are extremely useful ideas, absolutely. I’m aware of them, and I’m not trying to replace them. The mechanisms I’m working on are meant to complement them.
I wasn’t trying to say that we don’t need interfaces at all. Instead, my claim is that there’s a long history among programmers of thinking we’re managing complexity merely by dividing things into parts, without taking into account where we draw the dividing lines, the quality of the interfaces. There’s a huge bias to enshrine existing interfaces in the name of backward compatibility, and to force readers to navigate code based entirely on those interfaces. This leads in turn to a combinatorial explosion in the amount of code that runs on our devices, most of it just to support some internal interface or other that the end-user is utterly oblivious to. One of my favorite examples is this story which complains about Unix
cathaving grown to 800 LoC back in 2002. When I checked in 2013,catin GNU coreutils had grown to 36 thousand lines of code! How much of that could we prune for people who don’t care about ancient platforms or arcane compatibility constraints like#ifdef _XOPEN_SOURCE? tedu has many more examples in his writings.I don’t want to prevent people from making interfaces. I want to provide “escape hatches” that allow them to change those interfaces when a better interface becomes apparent, and to allow readers to understand the behavior of a library without having to understand its internal interfaces.
Makes sense. I’ll add to your 2nd article that the high-assurance field solved part of your territory problem along time ago. What they did was create abstract, formal specifications of the design or the code. These told you what it was doing more than how. The constraints were encoded in that. The code was done in a way + tested to correspond to that. You could read an English description of it with the formal specs to know exactly what any part of your system would do under various circumstances. Similar effect is done with model-checkers like Amazon’s TLA work and provers like seL4 where they pulled specs right out of the code.
Of course, that’s pretty heavy. So, I mentioned lightweight version like Design-by-Contract where you encode the constraints in your code. The basic documentation + annotated code should by themselves tell you what you can or can’t do with a module. That applies to all execution traces, too. Then, tests can be derived from them + written for other stuff you couldn’t specify. The combo of these made for strong correctness early on and during modifications. So, something to keep in mind as you experiment with your testing and other assurance practices.
re cat That’s pretty crazy. I didn’t know about that one. My reference for how ridiculous stuff is was the PHK essay below. Enjoy if you didn’t have it. :)
https://queue.acm.org/detail.cfm?id=2349257
Note: There’s actually a nice, horrific illustration of where the test-only methods can go in this one. Of course, you still intend on good interfaces where possible. I doubt you’ll ever make such a monstrosity.
I’m hacking on watchexec, a pure Rust (aka native) file watcher that runs commands when it detects modifications.
Currently I’m working on two things: creating a Homebrew tap, and figuring out the command-line UI.
Right now you can run it like this:
$ watchexec src "make test"Wondering if it’s worth expanding that to handle multiple paths, like so:
$ watchexec app lib test --exec rspecOr if I should just relegate such functionality to something like a
Watchfile.Join me in the bikeshed, together we’ll decide on a color! No, seriously, opinions very much welcome here.
Neat, you might want to check out
justrun- github.com/jmhodges/justrun - which has a discussion of some of the design decisions and tradeoffs inherent in such a tool.Good spot. Definitely a few features worth borrowing, such as the ability to kill the running process if more events come in before it terminates.
Fun! I wrote an inotify-based version in Go at one point, it’s quite a useful tool to have around. What are you using for file system notifications?
Edit:
This only builds when I update the org-mode sources for my site.
The notify crate is doing all the hard work of efficient platform-specific fs notifications. I need to put some time in and contribute a
kqueue()-based monitor (BSD users are people too!).I like the simplicity of filtering to sidestep multiple watched dirs. Will take a look at adding that.
I’ll look into this. Rust has been on my list for a while, so it’ll be nice to look at code doing something I generally understand.
The inotify-only but was because I had a need to learn that for something else. Much prefer kqueue!
Yeah since this is a shell over the notify crate there isn’t much code at all. This is my first real Rust project :)
In my honest, it would have been better named
watchcmd. Just sounds more authoritative or something. :)To be fair, it was hacked together in a few hours, and “cmd^I” is better than “watchc^I”…
This week I started poking at illumos. I’ve got a server running SmartOS, which has been good for learning the system administration bits again (it’s been a while since I ran a Solaris system), and this weekend I started digging through the kernel.
It’s the first time in a long while that I spent time on the weekends doing anything technical or really was even excited about doing anything technical, so there’s that.
What are the main advantages of Illumos/SmartOS? What are the pitfalls? I’d be interested to hear the opinion of someone with some experience.
Keep in mind that I’ve only just started using illumos, though I was a Solaris sysadmin years ago and ran OpenSolaris on my laptop in school for a while.
illumos advantages:
illumos Pitfalls:
SmartOS advantages:
SmartOS disadvantages:
Expect a longer writeup once I’ve used it in anger more.
I’m trying to figure out a reasonable API for a parsing library that provides all parse trees when the grammar is ambiguous. In C. The lack of garbage collection makes seemingly-obvious approaches (e.g. persistent data structures) tricky.
I’m making a small dent in the big stack of papers I heard about during Papers We Love Conf & Strange Loop.
I just got a partial laser-cut key tray prototype for a custom mechanical keyboard in the mail – I’m figuring out whether I want to adjust any positioning before I get the full key tray cut and start laying out the PCB.
Work: Bringing up a bootloader on new dev board, particularly its interactions with a crypto-authentication chip. After that, initial USB support in the firmware.
I’d be interested in hearing more about this C parsing library API design; both at a high-level and also diving into why lack of GC is making your design hard (I can relate).
Happy to take it to messages. I’m working on more organized writing (blog posts, documentation), but it’s probably a ways off, and thinking out loud may help.
Just thinking of giving mezzano (operating system implemented with Common Lisp) a try, has anyone played with it?
I think I gave it a spin last year. It took a little bit of work to get running in VirtualBox, and I seem to remember there wasn’t much to do besides hack on lisp and hang out on IRC inside the OS.
I put up a landing page for a project I’ve been noodling on off-and-on for close to 20 years: http://www.browsethesource.com/
I plan to work on it this fall and release a beta.
What’s this gonna be, like Google Grok?
The barebones version will be a single page app with an explorer tree on the left, a search bar on top and the main workspace for viewing code. Each symbol in the code viewer, when clicked, opens a search result under the line it on which defaults to the current file and can be expanded to any part of the tree with a breadcrumb path interface. The search bar would show results for the search in the main workspace.
The beta will be “demo mode” only meaning I will pick a bunch of popular git repos and the system will only run against them. If it gets traction I’ll open it up to user defined public and private git repos using the same pricing model as GitHub - free for public, pay for private. Beyond that, I have lots of other ideas for business models to sustain development and lots of ideas of where to go after version 0.1 - mostly around it customizing its UI or search behavior based on the project type and also how to annotate and publish explorations of code.
I actually built a Windows app very much like it in 1996 that generated static html files but never released it (I called it “HyperCross” because in 1996 “Hyper” was cool) and a few years ago I built an online version called sherlockcode.com but its interface was too kludgy. I’m hoping the 3rd time is the charm or maybe I’m just proving the definition of insanity.
I’m in Vancouver currently, on my way back from Alaska and at the 7,000 mile mark of the long road trip. It has been quite an experience, even got roared at by a bear next to our tent at the Icefields in Jasper!
The rest of the week will include spending a day in Vancouver, a day in Seattle, and then taking time to get back to the Bay Area. A few days of rest then I’ll be excited to start working and hacking again. It has been so nice to sit in the car and read various PDFs posted here. Thank you all for helping keep my mind off the long, long road.
I was sick last week so I only had time for a single Ansible role: ansible-role-syncthing. I’m feeling better but I’m no longer unemployed so I won’t have time to finish my go file browser.
I still need to find an lightweight encrypted remote backup software.
Is tarsnap an option? It’s a paid service, but it’s cheap. Very very very cheap.
For backups, I’m using duplicity + rsync to push backups. I couldn’t find anything lighter.
Ooh thanks, that looks like it’ll solve a problem I currently have.
Is your go file browser source available somewhere?
Camlistore? Or is that too heavy?
Does camlistore support files? Last I used it, I don’t recall it supporting file backups.
Gradually putting the pieces in place for a transient session API for triton. Intending on writing the server in ruby initially rather than Node.js (the rest of the APIs provided by triton are written in JS, but I know ruby better and I’ll never be motivated to do it if I write it in JS.)
So far I’ve got a bare bones JSON logger (port of node-bunyan), and am now implementing a library to handle HTTP Signature Authentication using SSH private keys to sign the payload. (See the spec for that here). In practice the triton API is really neat using your ssh key to authenticate with - basically means to add new users to the system you just create them a user with their ssh fingerprint attached and they have access with cli tools. Perfect.
Other projects have stagnated recently, gave up messing with the car and handed it to my lovely mechanic to replace the clutch & investigate the cooling system not working properly. Diagnosed that as the fan not working properly - having replaced that it appears (touch wood) to be working so far.
Also need to get over to the boat, having replaced the block of wood the mast steps into I increased the height of the step by about 5mm not realising there was a pin that needs to locate into the mast higher up. Which is of course now 5mm out of alignment so I can’t finish stepping the mast currently. Need to take some shims of the right size over to relocate that pin higher off the deck by 5mm so the mast is properly secured. I will get her in the water this year, honest!
Working on the next release of the EventQL database this week. The release will include little new features but mainly performance improvements for what’s already there – we’re aiming to support up to 10TB new data/day (3PB/yr) with the next version.
If anybody is interested in contributing to EventQL or discussing some internals/architecture, please hit us up!
I’ve got a wireframe concept version of my microphone driven game. Working title is “You Noisy”. I ended up going with matter.js for physics (instead of box2d.js) since it was purpose built for web and mobile, so has a friendlier API. Now that I have some experience with the lib, this week I will continue development of the game. Once the mechanics are a bit tighter I will throw together some basic art and get out of wireframe land.
Here is the thread from last week providing more info: https://lobste.rs/s/x3rzze/what_are_you_working_on_this_week/comments/rk9a5t#c_rk9a5t
That sounds fun! On the converse idea of using audio output as a primary source of feedback, there was a Gameboy Advance game called SoundVoyager. With only stereo and volume to give relative position and proximity, the games were all pretty elementary: collect instruments to build up a multi-layered track; shoot at monsters you can hear but not see; drive on a highway and avoid collisions by sound alone. It was really something different and I found it enjoyable.
Ooh that game looks great, thanks for the link!
Indeed, sound is an interface that is vastly underused, besides music and sound effects. And when mentioned most people only jump to language as the conclusive interface. I can’t wait to watch my friends screaming like maniacs at their phone trying to move an object around the screen.
My brother had made a Ludum Dare game that was going to be microphone driven a few years back in javascript as well. He didn’t have time to get a complete mechanic out of it, so instead he switched to controls via whistling volume. If you’d like, I can reach out to him and see if that code is still available.
Thinking of porting Clojure hiccup to Elixir. Otherwise, I’ll be looking into how to add an
updateoperation to BedquiltDB.I’ve been working on a blog post using interactive visualizations to explain why there is no quintic formula. My goal is to have it be convincing with the reader just having some familiarity with basic group theory and complex functions; this is possible because the proof I’m basing this on is topological and sidesteps all the hard Galois theory stuff.
As part of this, I also ported my blog to use KaTeX instead of MathJax, which makes it much faster to render math. :) The only downside is that KaTeX doesn’t have quite the full range of available commands that MathJax has, so I had to find a bunch of workarounds.
(This is my first lobsters post!)
OEC Monday, Wednesday, and Saturday. Also drag link parts for my Jeep should be arriving sometime this week, so I’ll be trying to install those sometime this week as well.
Attempting to get a release candidate ready by Wednesday. Might end up slipping to Friday but I’ll try my best.
Just finished moving into my new appartment, so I’ll mostly be working on undoing my boxes and tidying up the place! At work, besides fixing the usual bugs, I’m still trying to get better test coverage.
Trying to figure out what the hell is wrong with my server, hardware-wise. I think I’ve isolated it down to the CPU or motherboard. (And today, it seems more stable than it was before - sometimes it reboots in minutes, sometimes half a day…)
Programming wise, polishing my app up, and that generally means workflow wise, I come up with a feature, implement a backend, implement it in Windows Forms, then I implement it again in GTK#. GTK# isn’t terrible to work with, but it can be clumsy in some aspects.
edit: Filed an RMA with MSI. Wish me luck…
How are you debugging the server issues?
Process of elimination with each part. Swapped PSU, no change. Removed HD, no change. Tried various combinations of RAM, no consistent change. It’ll die at firmware, in ESXi, and randomly (not any specific test) in Memtest86.
I will be spending most of my time looking for an interesting internship where I can use Ruby. Replying to emails and phone calls about that. And on the side, starting this little startup thing about driving instructors on demand that I’m trying to make into a “any kind of service on demand”.
Good luck with the search!
I’ll be testing out and debugging a specific format output that I finished adding to an HLA logger last week. Capturing all the messages sent between components in a small scale distributed simulation and saving them in a truly archaic format, what fun!
For funsies, I started ES6 / browser / canvas stuff to try to recreate “Differential Line” in a different platform, but with spatial indexing (quadtree) to make it go faster. As I left it yesterday, I was visualizing points and neighborhoods of points and building quadtrees successfully and thinking about how I wanted to update them (or simply rebuild them) as points move. I might finish the recreation this week.
Startup Week! TC
Working on building a Sails based RSS example app that uses our personalization API for the machine learning side of things.
Not necessarily attending, but enjoying the side effects (bike races eg crossvegas, film screenings, friends in town) of Interbike coming to my hometown ?
Learning to do sales having completed an MVP of projectlemma.com. Advice and feedback welcome!
I might try something like MLT again, but I haven’t had a whole lot of luck with docs/finding working examples. If anyone knows of good video processing libs for C (maybe C++ if it’s not too crazy), Lua, Ruby or Scheme, I’d be grateful to learn about them.
If nothing crops up, I’ll probably just bite the bullet and write some Blender plugins to help me, since that seems to be the only scriptable video editor out there.
I finished my internship in Munich last Friday, and moved to London on Saturday. Now I’m spending the week at my parents house in Northern England for a holiday :)
I’m planning on looking at two things this week:
Building a replacement for MDP (terminal presentation tool) in rust. I have code blocks with syntax highlighting and most of the element types sorted, now working on parsing and auto margins.
Trying to decide if Tables, Footnotes, etc should be in first release, open to opinions.
My team at work was disbanded (I joined 6 months ago), and I’ve landed on one of the teams I was working with. Still trying to figure out whether I’ll stay at the company and in what capacity (they’ve all been great about trying to make it work for me). Wasn’t happy with the direction of certain things anyway, but now I’m rethinking everything (i.e., what do I want to do?). :)