1. 22

It’s Monday, so it is time for our weekly “What are you working on?” thread. Please share links and tell us about your current project. Do you need feedback, proofreading, collaborators?


  2. 15

    This last week in Open Dylan land has been unexpected.

    Last year, I wrote a post Thoughts on a New Dylan Ecosystem. My idea there was that since we have an old and relatively mature language, but a small userbase, this is a good time and opportunity to make changes to the core libraries and encourage a high standard for what we accept (in terms of documentation, testing, etc).

    In Dylan, we have a pretty traditional streams library that is largely used for I/O. There are some situations where having a more advanced stream library would be useful. Especially a functional, compositional one that is similar to scalaz-stream. This presents a number of interesting challenges:

    • Understanding how scalaz-stream works in the first place.
    • Having a good understanding of what we need in Dylan.
    • Thinking about how to make this work with the weaker type system in Dylan, yet still provide a good test case for improving our type system as we move forward with strengthening the type system.
    • How to avoid a lot of the complexity present in using the stream system to try to have a nice and elegant API without a lot of incidental complexity.

    One of the first places we’ll be using this new streams library is in our HTTP server.

    I’m writing a series of blog posts about this topic and what I’m coming across as I work through the above, but plan to publish them once they’re all written rather than as I write them.

    If the above sounds interesting, I’d be more than happy to have some collaborators.

    For work, I am just about to submit a large pull request to emscripten that I will be very happy to have done.

    1. 8

      You noted on Twitter the huge amount of work these weekly updates represent, and asked what makes it interesting. I think this is a better venue for answering that (I think I’m already over 140 characters ;)). So, here’s my take on why your weekly updates are so interesting for me (and I anticipate, for many others). Up front, I’m exhausted today so this may be less coherent than I’d like – apologies in advance.

      First off, what’s most interesting is the scale of your work on Open Dylan. From the first comments you’ve made about it, it’s obvious the passion and energy you’ve put into it, and watching you make your way through many of the challenges is encouraging. It’s refreshing to see someone working on big problems and solving them, especially as the problem is a long-term project, and to see you succeeding at it.

      As a Lisper, Dylan is an interesting language; I wish I had time to dig into it, but there are so many projects in the queue that I just don’t have time to give it, and you touch on a lot of aspects of working on a language runtime. I hadn’t realised Dylan had been developed by Harlequin, who did a lot of interesting things with Lisp too, including LispWorks (which flew on Deep Space 1 as RAX).

      From this week’s discussion of streams, to hacking on LLDB, to your work on type systems, to build systems, to editor integration, you cover an impressive breadth of topics, all the more so considering the scale of the undertaking. A lot of the reward of reading these is to see your thought processes in tackling the problems you’re facing. You also make for a good model for getting involved with opensource projects; for example, with LLDB, making spelling and grammar corrections while reading through docs (I started doing the same when learning Lisp-Flavoured Erlang).

      LLDB and some of the earlier work with SIGFPE and runtime work were really interesting to read about; my professional career has been mostly C, with a mix of Go as well lately and a smattering of assembly thrown in during my time on embedded systems, so the low level details are always fun to read about.

      To wrap this up, you give us a window into an interesting, large-scale project with a rich history. You seem to be carrying the flag well.

      1. 3

        Thanks for this feedback. I really do enjoy what I’ve been doing and realize how lucky I am that I’ve been able to do it.

        Part of what I really enjoy is the broad scope of things that need to be done and that I have to continually learn lots of new stuff.

        Your mention of the SIGFPE changes reminds me that that work isn’t complete. I ran into some issues that weren’t readily solvable at the time. I often wish that I had even more time than I already do.

        This is also something that I try to convey when I talk to people about contributing. We have things to do that don’t involve Dylan at all, but code in C++, Python, JS, etc. Some things would benefit a number of other projects, but I’d be happy to help provide guidance, mentoring, tutoring. Examples of this are changes to LLDB to support additional languages, changes to Sphinx to improve ePub support, and so on.

    2. 10

      I’m putting the finishing touches on the initial release of Manifold, which is meant to sit at the intersection of the n-many synchronous and asynchronous JVM stream representations. Anyone who has an opinion on Clojure or stream abstractions is welcome to give feedback.

      1. 2

        This looks like a great way to deal with asynchronous operations in Clojure! Will definitely be using this in upcoming projects.

      2. 9

        More Rust docs, as always. The Guide is really coming along: http://doc.rust-lang.org/guide.html I’m also really pumped about the new Pointer Guide I started Friday: https://github.com/rust-lang/rust/pull/15789

        I’m hoping to launch a new blog this week. Here’s the URL: http://metaphysics.io/ . I just have a little mini-essay on ‘what is this’ up, but I’m hoping to write an actual first essay tonight or tomorrow. You heard it here first!

        I’m also going to be speaking about Rust at the Manhattan JavaScript meetup, still gotta do slides.

        1. 8

          At work, I’m adding per process resource tracking (CPU, memory, disk, network) for all of our application’s processes to our internal logging/recording infrastructure.

          For fun, I’m writing/wrote a markov text generator to practice coding in Common Lisp. It’s just about done, so I’ll probably start on another small CL project later in the week.

          1. 1

            Very cool. Resource tracking is one thing I need to work on too. My goal is to implement a sort of “adaptive scaling” feature in a resource heavy application I’m working on (back-end data processing). I’d like to be able to dynamically scale the number of threads in the application based on resource availability. Not only would it be a cool feature to add, but it would allow me to deploy my project in different environments without having to tune them individually to each system’s unique capacity.

          2. 5

            I’ll be working on the Perl driver at MongoDB this week.

            Outside of work I’m hoping to finally learn Haskell. Any good resources for these two topics would be much appreciated.

            1. 5

              I really enjoy Learn You a Haskell, but some people are turned off by the levity.

              Read http://homepages.inf.ed.ac.uk/wadler/papers/marktoberdorf/baastad.pdf before you read anything else about Monads ;)

              1. 4

                Learn You a Haskell was a great intro; another useful resource for me was What I Wish I Knew When Learning Haskell.

              2. 2

                This is a good resource for learning Haskell: https://github.com/bitemyapp/learnhaskell

                1. 1

                  I’ve been enjoying Brent Yorgey’s Haskell course http://www.cis.upenn.edu/~cis194/lectures.html

                2. 5

                  Outside of work, I just released Turbine, a port of LMAX-Disruptor to Rust. This week I’ll be working on performance tuning and spending a lot of time getting cozy with cachegrind. Performance is already pretty good (on my Macbook Air, ~30m messages/s and ~512ns latency) but I’m sure there are some silly pieces of code that can be tuned. Also working on randomized tests to catch rarer bugs.

                  1. 2

                    That’s really cool! The disruptor is a sweet piece of technology.

                    1. 2

                      Thanks! Disruptor is indeed a very cool piece of tech, and I learned a ton while working on Turbine. Got to take a deep dive into atomics, memory barriers, cache coherency, lock-free programming, etc etc. Was a really great learning project

                  2. 5

                    I’m continuing to work on Scow[1], my implementation of Raft in Ocaml. I’m making good progress, just filling in the various states of the FSM. I’m hoping to have these fleshed out by the end of the week. After that I need to figure out how to test it. Once I’ve got that down I want to add snapshotting and member changes.

                    [1] https://github.com/orbitz/scow

                    1. 1

                      Are you aware of Heidi Howard’s implementation? https://github.com/heidi-ann/ocaml-raft. A draft of what should become a UCAM tech report on her implementation is available here: http://hh360.user.srcf.net/blog/2014/07/seeking-feedback-on-arc-analysis-of-raft-consensus/

                      1. 1

                        Yep, it’s API doesn’t really suit my aesthetic choices though. And mostly mine for learning.

                    2. 4

                      Outside of work, I’m working on my Rust kernel project, Rustic - porting it to the Rasberry Pi and BeagleBoard, and turning it into more of a framework to build embedded applications for these kind of targets rather than a raw “operating system” project. Enabling people to implement their crazy ideas and run them on the raw hardware, without needing to write stacks of assembly and boring low-level support code, essentially :)

                      Most of this week’s work on that will probably center around figuring out how to provide Rust’s concurrency primitives (eg, green threads, locking, etc) in code that can’t use libstd.

                      1. 2

                        Sounds awesome! I looked at Rustic before but skipped over it because I was trying to find something with ARM support. I’d like to help you with the porting if possible. I have a Cortex A9 board I could run it on. I’m a complete Rust noob, though, so I might take me a while to ramp up.

                        1. 2

                          Yup, it wasn’t very exciting as just a pure x86 kernel that didn’t do much :-) I wrote Rustic to learn Rust.

                          Once the new code goes into the public repo, I’m definitely happy to accept PRs to add support for other boards and architectures.

                          In the least, once the public repo is updated, having other people try and use the build system helps find chances to improve the build documentation and the build system itself :-)

                      2. 4

                        This week in an effort to make the bindings generated by Melange more idiomatic, I hacked on Dylan’s c-ffi library.

                        Many C libraries use the function return value to signal an error occurred, but in Dylan this is unnecessary because we have support for error conditions and exception handling.

                        Using libgit2 as an example, before my changes we had to either:

                        1. Return the error along the possible function return values and let the library user check it
                        2. Declare the return values as a C-mapped-subtype, and in the import-function check its value, eventually throwing an exception
                        3. Write a wrapper around each function to throw an exception if the function failed

                        At first we went with solution #2, the problem is that, by how the C-ffi module is designed, there was no way to call the mapping function and then discard the return value.

                        So my change to C-ffi involved defining a new way to declare the return value in a C-function definition, I called it error-result. The PR is not yet merged because I need to update the release notes and then have someone review it. The hard part writing the patch was understanding C-function-definer’s control flow, when I had that clear it was actually straightforward. C-ffi uses Dylan’s procedural-macros that are only available inside the compiler, so I had to understand them too. Compared to Dylan “standard” macros, they are more powerful, like Common Lisp macros.

                        Probably this week or the next I’m going to write a blog post explaining how it works in more detail.

                        After that, I changed Melange to allow a new map-error-result option to function declarations. After these changes I can get back hacking on libgit2 Dylan bindings!

                        1. 3

                          This has been some really nice work and something I’m very happy to see happening. I’m even more happy to see someone other than myself being able to hack on the C-FFI and some of the related compiler internals. :)

                        2. 3

                          Whole bunch of UI improvements to my product, yellerapp.com (an exception analytics SaaS). I’m focussing on three areas:

                          1. Speed - Yeller’s real good at server side speed (average page render is 20ms or so), but less good at client side perf. This update is working towards fixing that.
                          2. Density - I want to fit as much on the screen as I can, so you’re less likely to miss crucial information
                          3. Understanding - Whilst I care about density a bunch, a big focus is making it so you understand your exceptions, so that you can fix them faster.
                          1. 2

                            Density - I want to fit as much on the screen as I can, so you’re less likely to miss crucial information

                            This may not be the right solution to “making information visible” problem, though you probably know more about the problem space/problems with the existing UI, than I do.

                            1. 1

                              Yeah, it’s tricky. My description here wasn’t really right on by what I mean by density - the real thing I’m aiming for is

                              “everything you need, at a glance”

                              at least on the individual exception page

                          2. 3

                            I’m thinking about writing a tool that will make debugging embedded systems drivers easier. Basically, it will be a shell which allows you to mmap in IO memory and read and write values to it. This will make the process of figuring out the right memory mapped IO patterns a lot faster. I might try writing it in Rust as a learning exercise.

                            1. 2

                              Lots of people are interested in embedded Rust, and http://zinc.rs/ is basically the only thing that exists. If you do end up doing it, please write a blog post or something, that’d be sweet.

                              1. 1

                                Not really embedded in the sense of running on bare metal. The board I’m using has a Cortex A9 processor, so it runs Linux. But I often have to code custom memory-mapped IO drivers because I’m interfacing with custom FPGA peripherals.

                                1. 1

                                  Ahh, gotcha. Still cool sounding though. :)

                            2. 3

                              Working on some C# REST Services at work.

                              As a side project, outside of work, I’m going to start working on a ROR website to get better with Ruby/Rails.

                              1. 3

                                $work - Finished SAML investigation, sort of, more work looms.

                                !$work - My poor plant’s, I cleaned up the infestation, but the damage was done, half of the Zinnias had irreparable damage to their stems, most of my cilantro (which was already not doing great, because the pot was too small, I was trying to bolster it up before this infestation started) is dead, only my beanplant made it out more or less unscathed, only a couple leaves got killed. The soap/water combo cleared up everything pretty quick, but the little bastards work fast.

                                I’ve learned a lot about what not to do with plants from all this, I’ll tell you. First, plant in a bigger pot than you think will suffice, you’re always wrong. It’s like Hofstadter’s law, but for plants. Second, I think the next round of green things will get the ‘spray with soap/water pest repellent’ from day zero, ounce of prevention and all that, but they won’t get it every day. Finally, I think that I’m going to shy away from vegetable/herb growing while I’m limited to the small space I’m in (just my windowsill). Herbs especially seem to act more like weeds, and have a habit of choking out other plants, and themselves, whenever they get the chance. Similarly, I think I need to do better research about species. The Zinnia are lovely, but they’re a bit too tall, and that, combined with the too-small pot, means their stems didn’t grow very strong, leading to increased damage from the infestation.

                                1. 3

                                  Finally back to working on things!

                                  • I am going to start analyzing/writing about usability problems and solutions presented by Oculus VR and the software people are writing for it. I pre-ordered an Oculus, while I am waiting for that, I registered the domain vrusability.com and will set up a website this week/try to start writing about that topic. If you live in SF and have an Oculus I can borrow for a month, please get in touch. Would be happy to trade time/energy/consulting time for a dev kit for a month.

                                  • I’ll be doing some more work with the Jenkins UI. If you’ve tried 1.572 or newer, you should see newer buttons, friendlier forms, Helvetica instead of Verdana, and larger font sizes in general. I may try pushing for more aggressive changes this week, though it would be good to see the feedback from the changes so far.

                                  • I went yak shaving while trying to get vacation photos up online, and noticed iPhoto randomly decided to believe that I have zero photos, even though all of the photos are there. This prompted me to set up Glacier/git backups for iPhoto, and I’ll try to write about it this week. If you know a better photo management tool that should be around for longer than iPhoto, please let me know. Also, if you know anything about iPhoto internals: http://apple.stackexchange.com/questions/139431/which-metadata-files-should-i-back-up-to-ensure-my-iphoto-library-is-consistent, please respond! Trying to find out more.

                                  1. 3

                                    I’m working on Scala-related ideas. I’m investigating ways of abstracting “collection” operations from concrete execution semantics (e. g. if, in which order and when elements will be evaluated in code like someData.map(…).filter(…).groupBy(…)…).

                                    Current state of the art: Collections, database queries, reactive event streams, non-relational databases, etc. don’t have any common interface, despite providing more or less the same operations

                                    Idea: It should be possible to have one API which allows composing those operations without being chained to one specific implementation.

                                    1. 3

                                      Not working on anything! On vacation playing on my tan. Learned doing work is for suckers and that playing on tans is so much better.

                                        1. 3

                                          To learn Haskell, I’ve been working on a Python 3 interpreter called hython. I’m seeing how much Haskell (and FP) can help with an ambitious project. It’s been a lot of fun so far, and /r/haskell has been quite helpful and patient.

                                          I’m still working through implementing all the basic control-flow keywords and testing them. Once I get those done, I’ll probably move on to supporting all of the basic data types and operations supported on them. I don’t have any idea how I will do more advanced constructs such as generators or exception handling at this time.

                                          1. 2

                                            This week I’ll be continuing porting my MVP to our new stack (Elixir and Scala).

                                            This week, I’ll be finishing up our RethinkDB driver, ID generator and Service Registry. Once they’ve been completed and open sourced with decent documentation, work will start on the application specific portions of the service, firstly by building out our core API and making sure we’re meeting our REST API specification.

                                            I hope I’ll have some finished libraries and services to show you all this time next week!

                                            1. 2

                                              Just did a presentation at General Assembly which was meant to be on ‘programming for non-programmers’ but kind of mutated into ‘how to not burn all your startup cash on a deathmarch’. Now I’m trying to work out how to improve it based on audience feedback.

                                              Also just scored a gig developing an HTML5 app which interfaces to SAP using oData, which is both awe-inspiring and awful, and is yet more fuel on the fire under Tranquil APIs

                                              1. 2

                                                This week, having finally gotten fed-up with seeing what our Java teams have in the form of Nexus, I’ve been hacking valiantly on making a turnkey package hosting solution for Python. Like existing tools such as devpi, it will allow automatically mirroring the official PyPI servers and transparently mixing in private Python packages. Unlike devpi, it’s designed to be as stateless as possible: it uses Azure Blob Storage or S3 as the backing layer for the packages; supports OpenID, OAuth2, and other SSO solutions out-of-the-box so that you can trivially integrate it with Google Apps; and has a pretty (in my opinion) ReactJS-based UI. Knewton should begin rolling over to it in production next week, and I’ll be announcing and open-sourcing it once we’ve ironed out any kinks that show up in that process.

                                                I’m also working on some much less sexy stuff at the moment (de-engineering a hilariously over-engineered Java codebase not to use every single possible concurrency primitive to accomplish what amounts to a pretty straightforward XML transform), but I’m mostly excited about the package hosting.

                                                1. 2

                                                  On the side, I’m working on a category theory based generic programming experiment/library in Scala that’s a port of a haskell code. https://github.com/gclaramunt/scala-reggen

                                                  Would love some feedback :)

                                                  1. 2

                                                    At work I’ve started playing with React.js which seems like good stuff so far.

                                                    Not at work I’m getting ready to open my side project for beta testers. What I’m making is a tool that makes it easy for people to correct and comment on your writings. Check out Redpen.it and feel free to sign up if you’d like to help me test it.

                                                    1. 2

                                                      Nothing really, just a learning experience. I’m working on setting up a dovecot/postfix mail server using my RasPI.

                                                      1. 2

                                                        $work: get trading strategy in Java properly evaluated on training data and figure out how to get it making simulated money. Once that gets working (which I would love to get done this week but am not having any success in estimating) then we can turn it loose on the real market.

                                                        On that project, I just spent some time on a tag-based logging library, which strikes me as the kind of timewasting infrastructural work I should probably be getting from an existing library. It uses hashtags instead of debug levels, but still has .d, .e, etc., methods that add #d etc. hashtags to whatever other tags you supply.

                                                        Also thinking about adopting Guava for this project (I already have about half a page of code duplicating things that are in Guava) and moving to Java 8, since we unsurprisingly fondle datetimes a lot. Also we still need a time-series database for a few gigabytes of historical price data, which is a thing I wish I could get off the shelf but I think I may have to NIH also. (Maybe I should censor and publish the notes I have so far on why the existing TSDBs that I tried are useless shit.)

                                                        In related learning-remedial-Java news, I learned that I should really use Hamcrest for my JUnit tests instead of implementing .equals(); I learned how to eliminate most of the line noise associated with Java generics by using generic static factory methods that are overloaded on their return types; I learned that Java 8 verifies that all of the pieces of a package are signed by the same authority (beware mixed classpaths!); why it’s a bad idea to make Iterators Iterable like in Python; and generally am getting a lot more comfortable with Eclipse’s refactoring keystrokes.

                                                        In non-$work, I urgently need to answer reviewers' comments on my 500lines chapter, a thing I’ve been putting off forever, and get my personal website back online so I can share the cool things I’ve done or partly done in the last few months, which include:

                                                        • modeling 3-D shapes in Python with STL output (on GitHub);
                                                        • ragequitting Twitter again;
                                                        • generating 3-D printable trees in JavaScript using implicit functions and interval arithmetic;
                                                        • an offline DHTML notepad app for smartphones using application cache manifests and localStorage (fuck you, iOS 5);
                                                        • an archival virtual machine, like the UVC, that you can implement in half an hour in any programming language, like Brainfuck, that is designed to minimize the possibilities for accidental incompatibility and that is actually practical to write compiler backends for (unlike the UVC and BF), so we can preserve existing software in a useful way into the deep future;
                                                        • “opacity holograms” to store megabytes of text on a sheet of regular archival Mylar using a regular inkjet printer (with archival inks) with optical decoding, thus needing no microscope or computer to access the archived material.
                                                        • some other 3-D-printing stuff;
                                                        • a new calendar based on the Sierpiński triangle.

                                                        Beyond that, I need to blog about the explosion in new software that seems like it is starting to fulfill my vision for the equivalent of free software for online services, things like Irmin and the rest of OpenMirage, Sandstorm.io, MaidSafe, IPFS (talk), content-centric networking, and ownCloud, plus basic enabling technologies like Git, NaCl, unhosted apps, BaaS, Meteor, OCap security in current JS, and of course Tor. Some of these are flawed, some of them I don’t know enough about to know if they’re flawed, and some are super interesting.

                                                        1. 2

                                                          I guess it’s a day late, but at work I’m back in C, doing engineering work on a crypto project.

                                                          Outside of work, reading through the Nature of Code and Cities and Complexity. I also just grabbed a Pico Computing E-102 FPGA module that I have some ideas for.

                                                          1. 2

                                                            My office recently switched to Slack, and we’ve been writing silly little bots for it. I’ve been copying and pasting code enough that I decided to sit down and write a library once and for all, so here it is: https://github.com/hlian/linklater (Haskell)

                                                            1. 1

                                                              Playing around with elance to see if I can cheaply and successfully outsource any of the projects on my TODO list that I haven’t been able to get to in a while. I’ve had mixed results with elance in the past so I’m not quite sure what to expect.

                                                              1. 1

                                                                I have been working on my interactive WWII timeline. Be sure to use the scroll wheel or arrow keys to scroll the timeline as I haven’t incorporated arrow buttons on the timeline itself yet.


                                                                The buttons are now added.