1. 14

Welcome to the first WAYWO thread for February. These threads are a chance for us to discuss our plans for the week and make a commitment to getting things done. Links to projects and self promotion are encouraged.

This week, let’s also reflect on the first month of the year. Did you accomplish what you set out to do? Reach any important milestones? Tell us about in the comments.

  1.  

  2. 8

    I’m wrapping up my LMDB port to Go this week. I just need to get deletion, tree rebalancing and page reclamation working. And a whole bunch of documentation. The library is pre-alpha but here’s the repo for anyone interested in poking around:

    https://github.com/boltdb/bolt

    I started using the testing/quick standard library which is a QuickCheck-style library for black box testing. It’s a cool library but I never hear anyone talking about it.

    1. 2

      Very cool. I haven’t used LMDB before, how would you compare it in practice with LevelDB?

      1. 3

        I used LevelDB before as a backing store in the past but I had some issues with it when using it from multiple threads using the C API. It worked pretty well but it’s approach is somewhat complicated. It has multiple levels of storage files that have to be compacted periodically so performance can be variable.

        LMDB uses a mmap’d B+tree that is updated in-place so there’s no compaction required. The mmap is read-only so data structures can be mapped directly to the underlying data so there’s no memcpy() required. (Writes occur using vectorized I/O on a regular file descriptor)

        Ultimately I like the LMDB approach because it’s simple and grokable. My implementation omits the niche features in LMDB and it’s currently 1500 LOC.

        1. 2

          Thanks, that’s helpful. I’m using LevelDB in a project (on the topic of things we’re working on this week) but I’ve only read about LMDB. In the reading it’s hard to sort out its real merits from the author’s ranting but being able to use values directly from mapped memory sounds useful. One caveat to the LMDB benchmarks for anyone who’s following this and skims them is that they disable compression in LevelDB. This probably speeds it up for the memory workloads but will hurt if you’re using slower storage like AWS EBS volumes.

          1. 3

            Also, take a look at BangDB.

            http://highscalability.com/blog/2012/11/29/performance-data-for-leveldb-berkley-db-and-bangdb-for-rando.html

            I’m currently trying to fight through integrating it into a C project (with an autotools flag to switch between LevelDB, BDB, and BangDB).

            1. 2

              Thanks for that! BangDB looks pretty awesome, and I’ll probably give it a try later. I’ve been looking at cross-platform kv store as a persistence layer for a lua project. My initial thought was leveldb, and while it’s supposed to support windows I couldn’t get it to compile at all. I ended up using UnQLite (and wrote luajit bindings for it), but I’m still open to anything with a decent license.

        2. 2

          https://symas.com/is-lmdb-a-leveldb-killer

          We state quite clearly that LMDB is read-optimized, not write-optimized. I wrote this for the OpenLDAP Project; LDAP workloads are traditionally 80-90% reads. Write performance was not the goal of this design, read performance is. We make no claims that LMDB is a silver bullet, good for every situation. It’s not meant to be – but it is still far better at many things than all of the other DBs out there that do claim to be good for everything.

          Disclaimer: I have no experience with LMDB. I maintain one of the ruby LevelDB wrappers: https://github.com/vjoel/ruby-leveldb-native.

      2. 5

        Wrapping up a couple of last minute customer projects.

        One of the nice things about being a freelancer is that I get to take a lot of vacations with my family. Friday, we’re heading to Peru (Lima, Cusco, Machu Piccu and the Amazon). Hopefully, I won’t have good internet connectivity :)

        1. 5

          Last week I got it into my head to start playing with building an Apple //c emulator, so I started writing a 6502 CPU emulator first.

          1. 3

            What are you implementing the emulator in?

            1. 2

              C++ – it’ll eventually end up on an Arduino, and this will make the porting process easier, I hope.

            2. 1

              Also, forgot I submitted a pull request to the Go standard library this weekend, so I’ll probably be working on getting that cleaned up and fixed as per the feedback I get on it.

            3. 4

              I’m thinking about implementing an 8-bit CPU in Chisel (https://chisel.eecs.berkeley.edu/) to put on my FPGA. I decided to pick the 14-bit PIC architecture as my target. The first step is to implement a PIC emulator in order to get familiar with the instruction encoding and to get a reference implementation against which I can test the CPU.

              1. 4

                Working on a new Lobsters interview with a surprise guest!

                Also the train trip has mixed feelings; probably the easiest is to ride Caltrain back and forth all day, but you have to get off the train and wait for it to turn around at each end. Another option is to go overnight to somewhere like San Luis Obispo.

                I submitted this pull request for retry support in urllib3, will be working on getting that shipped. https://github.com/shazow/urllib3/pull/326

                1. 3

                  I’m beginning my trek into making a Clojurescript game about time travel. I’m loosely following Chris Granger’s Conj talk about Component Entity System.

                  1. 3

                    At work I fulfil change request after change request for our SAP PPM based propositions management system. They are all pretty small but occasionally I encounter something new and I have to ask a colleague. The first 2 months of the year are usually very slow.
                    At home I started to learn C. I want to be able to dig into lower level stuff and the sourcecode of useful tools. To keep myself engaged I decided to blog about my progress.

                    1. 3

                      Actually, now I am getting something visible to show on my C++ game-engine project!

                      http://i.imgur.com/snrcTtA.png

                      WASD + mouse movements and camera rotation around the player. Also note the awesome developer art.

                      The project is progressing really well and most of the low level stuff has been implemented.

                      1. 1

                        That is a good looking IDE!

                        1. 1

                          Visual Studio 2013 with the Dark skin. And yes, I have to agree, it does look pretty damn awesome :P

                          With a VsVim plugin its just about prefect for what I want from IDE while coding C++/C#.

                      2. 3

                        This week, for work, I’ve started a new contract and will be doing some work to upstream some changes from the client to the main emscripten repo (among other tasks).

                        For Open Dylan, this week will bring more compiler-side support for doing Objective C integration. This week will probably also bring some further evolution to the objc-dylan library now that things are mostly working.

                        I will also add some new content to the website to try to answer the question “Why Dylan?” in a satisfying way.

                        As for last month, things went pretty well. Unfortunately, the SIGFPE changes haven’t yet landed as they proved to be unsafe and made the compiler bootstrap fail which requires further investigation. But other tasks went well, for example, a big set of changes to emscripten landed over the course of the month, including changes to how exceptions are handled.

                        1. 3

                          I just got Erlang and Android interop working in preparation for my ErlangFactorySF talk.

                          https://github.com/knewter/erlanglauncher

                          Wasn’t sure I’d be able to do distribution from android, but turned out not to be so bad. Very proud of figuring it out tbh…

                          1. 2

                            On the business front, I’m working on adding support for Unity3D to StackMachine. I’ve had users asking for it since the site launched.

                            Since I’ve been on a Magic: The Gathering stint the last few weeks I decided to build an API for cards. The first version is live at api.deckbrew.com/cards. It’s still in flux and missing features, but the infrastructure is all in place. I’m using Go and martini for the API, Postgres as the backing store, Varnish in front of the service for caching, and Packer and Chef to build complete AMIs. Why so much infrastructure? I want to be able to over unlimited API access, so I want to be able to use autoscaling groups on EC2. This is also been an excuse to try out all these new technologies.

                            1. 3

                              This week, it’s a mix of a million things.

                              On the job front, it’s Ansible for a while, and then on to some pairing with a Java team making the transition to Ruby. We’ll also be digging into some architecture ideas to move a bit monolithic app to a finer grained, service oriented approach. Layered on top of that, I’ve got some prototypes to build for an idea I had about a sort of Reactor-style SOA architecture, akin to a quite old idea of the “Enterprise Data Bus.” Near as I can tell, not many of the new-wave SOA proponents have explored the concept to much, but I think it has some potential, and it gives me an excuse to play w/ ZeroMQ.

                              On the not-job front, I’m drilling Linear Algebra all week, trying to finish relearning all the stuff I feel like I need to relearn before I take the Subject GREs. It’s been nearly 4 years since I left school, and I’m finally feeling like I’ve got all the high level stuff under my foot, now it’s just a matter of drill-and-kill on the lower level stuff and actually taking the GREs.

                              I’m also going to be setting up my fancy new laptop this week, which will be fun.

                              EDIT: Naturally I click reply to the wrong thing in the email, and it ends up in the wrong place. sigh – Someday I’ll understand technology.

                              On Mon, Feb 3, 2014 at 2:59 PM, conroy conroy@lobste.rs wrote:

                              On the business front, I’m working on adding support for Unity3D to StackMachine. I’ve had users asking for it since the site launched.

                              Since I’ve been on a Magic: The Gathering stint the last few weeks I decided to build an API for cards. The first version is live at api.deckbrew.com/cards. It’s still in flux and missing features, but the infrastructure is all in place. I’m using Go and martini for the API, Postgres as the backing store, Varnish in front of the service for caching, and Packer and Chef to build complete AMIs. Why so much infrastructure? I want to be able to over unlimited API access, so I want to be able to use autoscaling groups on EC2. This is also been an excuse to try out all these new technologies.

                              – Vote: https://lobste.rs/s/5qwro8/_/comments/ttduw1

                            2. 2

                              Adding super-classes to PureScript.

                              The idea is that class can imply, rather than just use super-classes. Here’s a little example:

                              class Functor f where
                                (<$>) :: forall a b. (a -> b) -> f a -> f b
                              
                              class Bind f -> (Functor f) where
                                (>>=) :: forall a b. f a -> (a -> f b) -> f b
                              
                              class Monad m -> (Bind m) where
                                (<$>) f fa = (>>=) fa (\a -> ret (f a))
                                ret :: forall a. a -> m a
                              
                              instance Prelude.Monad Maybe where
                                  (<$>) f = maybe Nothing (\j -> Just (f j))
                                  ret = Just
                                  (>>=) m f = maybe Nothing f m
                              
                              1. 1

                                wrapping up my user-powered baseball research archive, saber archive. aiming to get a beta online soon. also going to be adding brewery leaderboards to my beer analytics site, beergraphs.

                                1. 1

                                  Getting over a really nasty cold, and later in the week planning to start work on replacing ${dayjob}’s darner based queueing stack with NSQ.

                                  1. 1

                                    This week, I’m trying to finish up a chapter on search in Riak for my book. It’ll be eventually published through Packt Pub. It’s been a trying experience, but also a learning one.

                                    After that, I’m trying to ram our product through the alpha stage and get it in shape for demos.

                                    1. 1

                                      Shooting for rolling out a new feature every day on my new analytics app Rangefinder. Managed to easily do that last week and I’ve got tons of ideas/plans, so it shouldn’t be to hard to keep the streak going.