This last week in Open Dylan land has been unexpected.

Last year, I wrote a post Thoughts on a New Dylan Ecosystem. My idea there was that since we have an old and relatively mature language, but a small userbase, this is a good time and opportunity to make changes to the core libraries and encourage a high standard for what we accept (in terms of documentation, testing, etc).

In Dylan, we have a pretty traditional streams library that is largely used for I/O. There are some situations where having a more advanced stream library would be useful. Especially a functional, compositional one that is similar to scalaz-stream. This presents a number of interesting challenges:

• Understanding how scalaz-stream works in the first place.
• Having a good understanding of what we need in Dylan.
• Thinking about how to make this work with the weaker type system in Dylan, yet still provide a good test case for improving our type system as we move forward with strengthening the type system.
• How to avoid a lot of the complexity present in using the stream system to try to have a nice and elegant API without a lot of incidental complexity.

One of the first places we’ll be using this new streams library is in our HTTP server.

I’m writing a series of blog posts about this topic and what I’m coming across as I work through the above, but plan to publish them once they’re all written rather than as I write them.

If the above sounds interesting, I’d be more than happy to have some collaborators.

For work, I am just about to submit a large pull request to emscripten that I will be very happy to have done.

I’m putting the finishing touches on the initial release of Manifold, which is meant to sit at the intersection of the n-many synchronous and asynchronous JVM stream representations. Anyone who has an opinion on Clojure or stream abstractions is welcome to give feedback.

More Rust docs, as always. The Guide is really coming along: http://doc.rust-lang.org/guide.html I’m also really pumped about the new Pointer Guide I started Friday: https://github.com/rust-lang/rust/pull/15789

I’m hoping to launch a new blog this week. Here’s the URL: http://metaphysics.io/ . I just have a little mini-essay on ‘what is this’ up, but I’m hoping to write an actual first essay tonight or tomorrow. You heard it here first!

I’m also going to be speaking about Rust at the Manhattan JavaScript meetup, still gotta do slides.

At work, I’m adding per process resource tracking (CPU, memory, disk, network) for all of our application’s processes to our internal logging/recording infrastructure.

For fun, I’m writing/wrote a markov text generator to practice coding in Common Lisp. It’s just about done, so I’ll probably start on another small CL project later in the week.

I’ll be working on the Perl driver at MongoDB this week.

Outside of work I’m hoping to finally learn Haskell. Any good resources for these two topics would be much appreciated.

Outside of work, I just released Turbine, a port of LMAX-Disruptor to Rust. This week I’ll be working on performance tuning and spending a lot of time getting cozy with cachegrind. Performance is already pretty good (on my Macbook Air, ~30m messages/s and ~512ns latency) but I’m sure there are some silly pieces of code that can be tuned. Also working on randomized tests to catch rarer bugs.

I’m continuing to work on Scow[1], my implementation of Raft in Ocaml. I’m making good progress, just filling in the various states of the FSM. I’m hoping to have these fleshed out by the end of the week. After that I need to figure out how to test it. Once I’ve got that down I want to add snapshotting and member changes.

[1] https://github.com/orbitz/scow

Outside of work, I’m working on my Rust kernel project, Rustic - porting it to the Rasberry Pi and BeagleBoard, and turning it into more of a framework to build embedded applications for these kind of targets rather than a raw “operating system” project. Enabling people to implement their crazy ideas and run them on the raw hardware, without needing to write stacks of assembly and boring low-level support code, essentially :)

Most of this week’s work on that will probably center around figuring out how to provide Rust’s concurrency primitives (eg, green threads, locking, etc) in code that can’t use libstd.

This week in an effort to make the bindings generated by Melange more idiomatic, I hacked on Dylan’s c-ffi library.

Many C libraries use the function return value to signal an error occurred, but in Dylan this is unnecessary because we have support for error conditions and exception handling.

Using libgit2 as an example, before my changes we had to either:

1. Return the error along the possible function return values and let the library user check it
2. Declare the return values as a C-mapped-subtype, and in the import-function check its value, eventually throwing an exception
3. Write a wrapper around each function to throw an exception if the function failed

At first we went with solution #2, the problem is that, by how the C-ffi module is designed, there was no way to call the mapping function and then discard the return value.

So my change to C-ffi involved defining a new way to declare the return value in a C-function definition, I called it error-result. The PR is not yet merged because I need to update the release notes and then have someone review it. The hard part writing the patch was understanding C-function-definer’s control flow, when I had that clear it was actually straightforward. C-ffi uses Dylan’s procedural-macros that are only available inside the compiler, so I had to understand them too. Compared to Dylan “standard” macros, they are more powerful, like Common Lisp macros.

Probably this week or the next I’m going to write a blog post explaining how it works in more detail.

After that, I changed Melange to allow a new map-error-result option to function declarations. After these changes I can get back hacking on libgit2 Dylan bindings!

Whole bunch of UI improvements to my product, yellerapp.com (an exception analytics SaaS). I’m focussing on three areas:

1. Speed - Yeller’s real good at server side speed (average page render is 20ms or so), but less good at client side perf. This update is working towards fixing that.
2. Density - I want to fit as much on the screen as I can, so you’re less likely to miss crucial information
3. Understanding - Whilst I care about density a bunch, a big focus is making it so you understand your exceptions, so that you can fix them faster.

I’m thinking about writing a tool that will make debugging embedded systems drivers easier. Basically, it will be a shell which allows you to mmap in IO memory and read and write values to it. This will make the process of figuring out the right memory mapped IO patterns a lot faster. I might try writing it in Rust as a learning exercise.

Working on some C# REST Services at work.

As a side project, outside of work, I’m going to start working on a ROR website to get better with Ruby/Rails.

$work - Finished SAML investigation, sort of, more work looms.

!$work - My poor plant’s, I cleaned up the infestation, but the damage was done, half of the Zinnias had irreparable damage to their stems, most of my cilantro (which was already not doing great, because the pot was too small, I was trying to bolster it up before this infestation started) is dead, only my beanplant made it out more or less unscathed, only a couple leaves got killed. The soap/water combo cleared up everything pretty quick, but the little bastards work fast.

I’ve learned a lot about what not to do with plants from all this, I’ll tell you. First, plant in a bigger pot than you think will suffice, you’re always wrong. It’s like Hofstadter’s law, but for plants. Second, I think the next round of green things will get the ‘spray with soap/water pest repellent’ from day zero, ounce of prevention and all that, but they won’t get it every day. Finally, I think that I’m going to shy away from vegetable/herb growing while I’m limited to the small space I’m in (just my windowsill). Herbs especially seem to act more like weeds, and have a habit of choking out other plants, and themselves, whenever they get the chance. Similarly, I think I need to do better research about species. The Zinnia are lovely, but they’re a bit too tall, and that, combined with the too-small pot, means their stems didn’t grow very strong, leading to increased damage from the infestation.

Finally back to working on things!

• I am going to start analyzing/writing about usability problems and solutions presented by Oculus VR and the software people are writing for it. I pre-ordered an Oculus, while I am waiting for that, I registered the domain vrusability.com and will set up a website this week/try to start writing about that topic. If you live in SF and have an Oculus I can borrow for a month, please get in touch. Would be happy to trade time/energy/consulting time for a dev kit for a month.

• I’ll be doing some more work with the Jenkins UI. If you’ve tried 1.572 or newer, you should see newer buttons, friendlier forms, Helvetica instead of Verdana, and larger font sizes in general. I may try pushing for more aggressive changes this week, though it would be good to see the feedback from the changes so far.

• I went yak shaving while trying to get vacation photos up online, and noticed iPhoto randomly decided to believe that I have zero photos, even though all of the photos are there. This prompted me to set up Glacier/git backups for iPhoto, and I’ll try to write about it this week. If you know a better photo management tool that should be around for longer than iPhoto, please let me know. Also, if you know anything about iPhoto internals: http://apple.stackexchange.com/questions/139431/which-metadata-files-should-i-back-up-to-ensure-my-iphoto-library-is-consistent, please respond! Trying to find out more.

I’m working on Scala-related ideas. I’m investigating ways of abstracting “collection” operations from concrete execution semantics (e. g. if, in which order and when elements will be evaluated in code like someData.map(…).filter(…).groupBy(…)…).

Current state of the art: Collections, database queries, reactive event streams, non-relational databases, etc. don’t have any common interface, despite providing more or less the same operations

Idea: It should be possible to have one API which allows composing those operations without being chained to one specific implementation.

Not working on anything! On vacation playing on my tan. Learned doing work is for suckers and that playing on tans is so much better.

Transit. https://github.com/cognitect/transit-format

To learn Haskell, I’ve been working on a Python 3 interpreter called hython. I’m seeing how much Haskell (and FP) can help with an ambitious project. It’s been a lot of fun so far, and /r/haskell has been quite helpful and patient.

I’m still working through implementing all the basic control-flow keywords and testing them. Once I get those done, I’ll probably move on to supporting all of the basic data types and operations supported on them. I don’t have any idea how I will do more advanced constructs such as generators or exception handling at this time.

This week I’ll be continuing porting my MVP to our new stack (Elixir and Scala).

This week, I’ll be finishing up our RethinkDB driver, ID generator and Service Registry. Once they’ve been completed and open sourced with decent documentation, work will start on the application specific portions of the service, firstly by building out our core API and making sure we’re meeting our REST API specification.

I hope I’ll have some finished libraries and services to show you all this time next week!

Just did a presentation at General Assembly which was meant to be on ‘programming for non-programmers’ but kind of mutated into ‘how to not burn all your startup cash on a deathmarch’. Now I’m trying to work out how to improve it based on audience feedback.

Also just scored a gig developing an HTML5 app which interfaces to SAP using oData, which is both awe-inspiring and awful, and is yet more fuel on the fire under Tranquil APIs …

This week, having finally gotten fed-up with seeing what our Java teams have in the form of Nexus, I’ve been hacking valiantly on making a turnkey package hosting solution for Python. Like existing tools such as devpi, it will allow automatically mirroring the official PyPI servers and transparently mixing in private Python packages. Unlike devpi, it’s designed to be as stateless as possible: it uses Azure Blob Storage or S3 as the backing layer for the packages; supports OpenID, OAuth2, and other SSO solutions out-of-the-box so that you can trivially integrate it with Google Apps; and has a pretty (in my opinion) ReactJS-based UI. Knewton should begin rolling over to it in production next week, and I’ll be announcing and open-sourcing it once we’ve ironed out any kinks that show up in that process.

I’m also working on some much less sexy stuff at the moment (de-engineering a hilariously over-engineered Java codebase not to use every single possible concurrency primitive to accomplish what amounts to a pretty straightforward XML transform), but I’m mostly excited about the package hosting.

On the side, I’m working on a category theory based generic programming experiment/library in Scala that’s a port of a haskell code. https://github.com/gclaramunt/scala-reggen

Would love some feedback :)

At work I’ve started playing with React.js which seems like good stuff so far.

Not at work I’m getting ready to open my side project for beta testers. What I’m making is a tool that makes it easy for people to correct and comment on your writings. Check out Redpen.it and feel free to sign up if you’d like to help me test it.

Nothing really, just a learning experience. I’m working on setting up a dovecot/postfix mail server using my RasPI.

$work: get trading strategy in Java properly evaluated on training data and figure out how to get it making simulated money. Once that gets working (which I would love to get done this week but am not having any success in estimating) then we can turn it loose on the real market.

On that project, I just spent some time on a tag-based logging library, which strikes me as the kind of timewasting infrastructural work I should probably be getting from an existing library. It uses hashtags instead of debug levels, but still has .d, .e, etc., methods that add #d etc. hashtags to whatever other tags you supply.

Also thinking about adopting Guava for this project (I already have about half a page of code duplicating things that are in Guava) and moving to Java 8, since we unsurprisingly fondle datetimes a lot. Also we still need a time-series database for a few gigabytes of historical price data, which is a thing I wish I could get off the shelf but I think I may have to NIH also. (Maybe I should censor and publish the notes I have so far on why the existing TSDBs that I tried are useless shit.)

In related learning-remedial-Java news, I learned that I should really use Hamcrest for my JUnit tests instead of implementing .equals(); I learned how to eliminate most of the line noise associated with Java generics by using generic static factory methods that are overloaded on their return types; I learned that Java 8 verifies that all of the pieces of a package are signed by the same authority (beware mixed classpaths!); why it’s a bad idea to make Iterators Iterable like in Python; and generally am getting a lot more comfortable with Eclipse’s refactoring keystrokes.

In non-$work, I urgently need to answer reviewers' comments on my 500lines chapter, a thing I’ve been putting off forever, and get my personal website back online so I can share the cool things I’ve done or partly done in the last few months, which include:

• modeling 3-D shapes in Python with STL output (on GitHub);
• ragequitting Twitter again;
• generating 3-D printable trees in JavaScript using implicit functions and interval arithmetic;
• an offline DHTML notepad app for smartphones using application cache manifests and localStorage (fuck you, iOS 5);
• an archival virtual machine, like the UVC, that you can implement in half an hour in any programming language, like Brainfuck, that is designed to minimize the possibilities for accidental incompatibility and that is actually practical to write compiler backends for (unlike the UVC and BF), so we can preserve existing software in a useful way into the deep future;
• “opacity holograms” to store megabytes of text on a sheet of regular archival Mylar using a regular inkjet printer (with archival inks) with optical decoding, thus needing no microscope or computer to access the archived material.
• some other 3-D-printing stuff;
• a new calendar based on the Sierpiński triangle.

Beyond that, I need to blog about the explosion in new software that seems like it is starting to fulfill my vision for the equivalent of free software for online services, things like Irmin and the rest of OpenMirage, Sandstorm.io, MaidSafe, IPFS (talk), content-centric networking, and ownCloud, plus basic enabling technologies like Git, NaCl, unhosted apps, BaaS, Meteor, OCap security in current JS, and of course Tor. Some of these are flawed, some of them I don’t know enough about to know if they’re flawed, and some are super interesting.

I guess it’s a day late, but at work I’m back in C, doing engineering work on a crypto project.

Outside of work, reading through the Nature of Code and Cities and Complexity. I also just grabbed a Pico Computing E-102 FPGA module that I have some ideas for.

My office recently switched to Slack, and we’ve been writing silly little bots for it. I’ve been copying and pasting code enough that I decided to sit down and write a library once and for all, so here it is: https://github.com/hlian/linklater (Haskell)

Playing around with elance to see if I can cheaply and successfully outsource any of the projects on my TODO list that I haven’t been able to get to in a while. I’ve had mixed results with elance in the past so I’m not quite sure what to expect.

I have been working on my interactive WWII timeline. Be sure to use the scroll wheel or arrow keys to scroll the timeline as I haven’t incorporated arrow buttons on the timeline itself yet.

EDIT

The buttons are now added.