It’s Monday, and I wanted to post in this thread, but it wasn’t up yet, so I decided to make it so.
Please share links and tell us about your current project. Do you need feedback, proofreading, collaborators?
I’m getting closer to having a releasable Haskell article for http://howistart.org/.
Progress on the book I’m writing for learning Haskell is slow, but still ongoing.
Some discussions going on about possibly making a “permanent” free and online version of cis194 (This is the first course I recommend in my guide) - not yet sure how much time to devote to that when I still have the book.
I fixed Hackage to compile with GHC 7.8 which should free up some other changes. Got a lot of help from @carter on that. Annoyingly, TravisCI is breaking for unrelated reasons, not my PR’s fault AFAICT.
I’m also considering making a video this week of showing how I do bottom-top and top-bottom development in Haskell with the REPL.
Last week was crazy with various scholarship/fellowship applications for grad school. In the end I went in for 2 of them, and abandoned my Fulbright application because I was struggling with it. This was basically all I got done, I had some very very long days.
I’ve not made much progress on my Idris -> Erlang compiler. I’ll get back onto that on Wednesday, once I’ve finished an assignment due tuesday (a toy compiler, so I don’t want to get confused between them both). Edwin and I have worked out a good solution to the Primitives problem I had last week, based on a paper he knew of, so that’s first on the list.
Sadly it looks as if the radio show/podcast is on hiatus this week, due to us not having a studio. This has been annoying because we wanted to interview someone, but thankfully he can make it in a fortnight, so we’ll definitely have that interview.
So, it looks like a quieter week, which is much needed after last week.
Have you written anything or published any source code for your idris -> erlang project? It sounds really interesting and I’d love to learn more.
https://github.com/lenary/Idris-dev/pull/1 Contains the source code. It’s incomplete, of course, but is capable of a hello world (with my small changes to idris, also in that repo). https://gist.github.com/lenary/7abb7ad5d533bf376804 is an example of hello world.
I’ve not really written anything yet, but it’s dissertation work, so I will be writing something eventually, at length. It’s been fun to do so far, and more docs are needed for that part of the Idris codebase.
I’m trying to find if there are good cache simulators for things that aren’t CPUs. I spent a chunk of last week experimenting with CPU cache eviction policies, and I’d like to be able to see how the results generalize to other domains without having to actually convince someone to switch production machines over.
I also want to get back to debugging a couple bugs found by a fuzzer I wrote. One has been unreproducible (a base library function randomly segfaults, but replaying the fuzzer with the same seed doesn’t hit the segfault), and I haven’t been able to get a test case less than about 5 minutes long for another (an exception is randomly not caught by a try/catch block that should catch all exceptions, but modifying the test case at all makes the bug go away), which is too long for me to really want to dig into the compiler to figure out what’s wrong.
I probably need to write some kind of test case reducer to really attack those bugs.
I find cache eviction policies really interesting. I find it interesting how they’re somewhat like generational GC, and whether some of the algorithms from caches can be applied to GC (thinking SRRIP and DRRIP). I guess the only difference is you can evict something you’ll use again, this is not the case with GC.
Your eviction policy post was interesting!
I’m not much of an expert here, but the first things I thought of were evaluations of disk buffer caches and network processor ASICs.
I don’t have any actual direct recommendations, but all the things I found in my quick googling just now were posted in SIGMETRICS, so that’s where I’d start.
There’s also multiprocessor cache coherence directories, although replacement policies there get complicated. ( I liked this paper about SCD (Scalable Coherence Directory) at HPCA ‘12 from Daniel Sanchez and Christos Kozyrakis - has some interesting info about how design constraints of coherence directories impacts replacement, noting that their preferred cache design ends up selecting replacement candidates in a de-facto random fashion. I haven’t thought about coherence directories in a few years and that paper was pretty easy to follow - they did a nice job on the background (section 2).
$work: Success, ported giant legacy Java app with arcane oracle database configuration from bespoke windows setup designed for a configure-once-and-forget system, to a pair of repeatable docker images running ubuntu 14.04, simplifying deployment and generally improving things (even if the only presently felt improvement is no more windows). The next step is to build up the deployment infrastructure so we can start spinning up environments on the fly for the voracious QA team to consume with abandon.
tl;dr We have met the enemy, and they are ours.
!$work: Started working on a new recipe for Boeuf Daube for the missus. She loves potroast and anything cooked low-and-slow, and has been bugging me to learn something special for her (my Marsala recipe being relatively famously tailored to my Mother’s tastes over the course of the last 10 years). The first test case was good, but needs tweaking, after a few more iterations I’m going to introduce the not-really-all-that-secret variation, replacing the usual egg-noodles the stew is served over with Pomme Frites (her love of potatoes being second to none). I think a thick French Beef Stew and Fried Potatoes is a pretty winning combo.
Also on the cooking front, I’ve started compiling some of my family’s recipes into a cookbook. Both the original versions and versions with a modernized (but authentic) approach. Some of my Memére’s recipes in particular are tuned for a Surveyor’s budget, and an Army of 11 (She had 9 kids). Which means the proportions are all large and there is quite a bit of margarine (it was the 60s, after all). My (extended) family doesn’t have much in the way of culinary types, and we have a collective love of food, so I really want to preserve that legacy. Also it gives me an excuse to make lots of tasty food, so it’s not without selfish intent.
This week I’ll continue working on my sat solver with opendylan; you might remember my request for links here.
I’m also seriously thinking of restarting my adventure on Software Foundations, maybe work on the first chapters before this one before I start it. If anyone is interested in exchanging contacts for future reflections, PM me.
Last week, I was on vacation, and too relaxed to think in a structured way much of the time. I did some work on Hython that I’m a bit suspicious of.
This week, I need to admit I’m a bit stuck, actually (this is hard for me to do). I’m trying to improve the design of the Interpreter and make it easier for me to work with, but it’s quite coupled at the moment. I posed a question to /r/haskellquestions that has all the gory details. Would appreciate any pointers, as this is primarily about improving my design sense going forward.
Last week I made a prototype of the async background task queue I mentioned a few weeks ago.
The prototype we’ve created combines a job queue and worker management. Queues are local to machines they run on. Queues are declared in a crontab-like file, along with the maximum concurrent workers for that queue and the command the workers will run.
image 5 /usr/local/bin/convert_image
audio 6 /usr/local/bin/process_audio
Jobs are just an ASCII formatted string of the queue name and args to pass directly to the command.
echo 'image --file foo.jpg --format png --size=1024x768' | nc worker.foo.com
Upon receipt of a job, either a new worker will be spawned or, if the maximum number of workers are already running, the job will be placed in the queue until a new worker can be spawned.
It’s written in standard C with no external dependencies. Memory usage is moderate, ~1MB/10k jobs. We’re still exploring redundancy patterns and have a few nice-to-haves in the todo list (reload conf on SIGHUP, graceful shutdown both with and without queue persistance, per queue uid/guid).
One of our applications accepts user-supplied audio, video, and images. Jobs are put onto a task queue to be processed. Our current configuration uses Gearman as a queue, the main web app as the producer, and assorted scripts as the consumers. While it mostly works fine, we’re of the opinion that it’s a little heavy to maintain several consumer processes over numerous machines to just run gm or ffmpeg.
This sounds like a pretty lightweight queue implementation, I’d be interested to hear more about how you handle redundancy in the system as you explore it.
Presumably you’re securing this with firewalling, rather than an authentication protocol in the workers themselves?
Correct. In our case, servers are on a VLAN which has no direct routes from the outside world (in addition to firewall policies, just in case.)
The more I think about the redundancy, the more I don’t think it’ll take anything special. It can mostly be handled like anything else: load balance or have the client send jobs to all queues and ensure the commands can synchronize state. In our case, the latter is very easy since we’re reading media file from temp storage anyhow. If it’s there, continue, else exit.
The more interesting problem we’ve found is how you handle failed jobs and lost queues. Is it up to the producer or consumer to resubmit failed jobs? How do you track fails? What happens when you’re just constantly failing and resubmitting? In a one way system (producer -> consumer), how do you synchronize state?
I figure the worst case scenario is that you wind up with something like, say, a Python script that runs the job and updates a table in MySQL (or some other shared state) with the status of the job. Ultimately we’re still ahead because we still don’t need to manage processes and we don’t need any extra hardware for our queue and it’s redundant spare.
I’m all ears for thoughts and opinions! Like I said, it’s just a prototype to explore the space and see if we can improve on the current architecture.
Regarding fail loops, exponential back off with a max number of retries.
Using a relational database to manage jobs isn’t bad. You need a pretty high volume of jobs for the DB to become the problem there. Presumably you are doing async jobs because each job is expensive, which implies a lot about how much load will really be put on the DB: not a lot. You can also prevent long-term degradation of DB performance on the active jobs table by moving completed jobs to a separate table. Then you get a full job history as a side effect, which can be really useful in the long run, or at least fun to query from time to time. =)
This week I spent a lot of time working on a bifunctor library for idris. This should hopefully be useful for both the timing attack resistant type systems project I’ve been working on for a while now (bifunctors make things with tuples super easy) and also a couple different possibilities for an independent study I’m trying to take next semester. If anyone who knows a bit more category theory than I do could take a look at it, especially the bit about bimonads as I couldn’t find a really good reference for those, I’d appreciate it.
Apart from that, college applications are still heavily on my mind. I should be doing the first Minerva “assessment” this week, whatever that entails, and I’m still deciding on who to pick for my recommendation letters and writing/revising all kinds of essays.
Other than that, it’s been a quieter week, mostly filled with working on boring but necessary bits of my life (which, contrasted with certain non-quiet weeks of note, I can certainly appreciate).
I was intrigued enough after watching Duality and the End of Reactive (video) by Erik Meijer that I stole some of the concepts to create an asynchronous transducers library, transduce-async.
This week I plan on using it as a basis, along with underscore-transducer, to dust off and rewrite underarm, a toy library I wrote a couple years ago to try to understand Reactive Cocoa, Rx*. I also would like to look more into FRP concepts (arrowized, sampling rates, etc).
I’ve decided it’s past due to learn me some Haskell, so I ordered “Real World Haskell” to get started. If anybody has any other suggestions to get started, please let me know!
@bitemyapp has some great stuff up on github
Personally, I learned from Learn you a haskell for great good, but I’ve heard mixed reviews, so YMMV.
This week, I’m working on implementing password resets on https://hex.pm. I also working on improving the tests, a new tarball metadata format and better S3 log parsing.
If anyone is interested on working on Hex with some Elixir/Erlang, you can find Hex on GitHub here.
I spent an inordinate amount of time writing a parser Propositional Logic in Scala. probably because I decided to write it using patten matching, instead of using Scala’s built in parsing stuff. It’s nearly finished now (I’ve parsed the input into a tree structure), I just need to implement better error handling (read detection) and then writing a simple SAT solver shouldn’t be too hard.
I also started writing an applet in C to compile source code on a remote server. I’ve currently been using bash to do it, but wanted something more generic and robust. Unfortunately, the University systems doesn’t have libssh installed, which kind of defeats the object of the project (since I wanted it for compiling LaTeX on systems that don’t have it installed or have few packages installed, of which uni is a prime example).
In my University studies, I came across quickselect, which is a nice variant of quicksort for finding the nth largest item in a list that works in O(n) time.
For the last four months I’ve been mostly not-developing as I’ve been traveling around, seeing family, friends and a few other things. I’ve stopped that for now and found a place to live for the next month at least, so it’s time to get back into a normal type of life. And it also means I have something to post here.
I spent last week on a few things:
Firstly, trying to find some work, I have an interesting lead now so I’ll see where that goes.
A side project with a designer friend of mine, it’s a simple web app in the Buffer vein of things, built with Clojure and plain React.js (no Clojurescript / Om in this project)
Working through Understanding Computation, I read it a while ago but I don’t feel like I really absorbed it. I’m going back through, writing out and experimenting myself with all of the code in the book.
At work: Investigating TypeScript, particularly how the TS internal module system plays with Angular. Also attempting to break JS and CSS dependencies between some of our (quite old) web apps so that changes in one app won’t require changes in another.
Personal non-tech: Experimenting with adopting an agile/scrum approach to household management. So far it seems to be going well, but we’re only in the second week.
Finally putting some spit and shoe shine on Grimoire, 0.4.0 draws nearer. Yes I’m skipping the 0.3.10 listed on the bugtracker. I got carried away and mixed 0.3.10 items with 0.4.0 items so it’s all just gonna be 0.4.0.
Dropped a first release of lein-grim, the documentation dataset generator.
Dropped a first release of lib-grimoire, a library that abstracts away the backing datastore and allows for code sharing between lein-grim and Grimoire itself.
Resumed work on porting Grimoire from doing inline filesystem manipulation to using lib-grimoire throughout.
The 0.4.0 release should be a lot of fun, because it will open the doors of the Grimoire project to arbitrary documentation submissions from the community. This means that adding documentation of say core.async or core.typed or core.logic to Grimoire will become trivial when it wasn’t before. Lots of polishing to do on the website in terms of CSS cleanups, adding content, adding previews and soforth. I also need to figure out how I’m gonna handle “notes” (community documentation), contributed documentation and extending my existing data representation with contributed information such as type signatures that are not (currently) official metadata.
I did not do too much, but I did manage to create a Mac build of Fire★. I did not publish it officially yet on the website because I need people to test it. If anyone wants to download it and tell me if the thing runs, you can get it here.
I really really will work on more documentation for the project, I promise. I think I will work on the API reference, and then if I get around to it, some tutorials on writing Apps.
I downloaded the app and tried to open, got a crash due to libbotan not being found:
Library not loaded: /usr/local/lib/libbotan-1.10.dylib.0
Hope that helps! (MBP running 10.10 if that helps.)
I updated the dmg, now it should work.
However I discovered another problem, being a mac n00b. Mac has a feature called App Nap, and apparently changing the app’s Info.plist isn’t enough to turn it off. App Nap will put important background threads to sleep causing Fire★ to disconnect if you don’t touch the app.
I am going to write a bit of code to turn off App Nap using the ‘defaults’ cmdline tool on startup by adding Fire★ to the global blacklist.
If anyone knows a better way to turn off App Nap, I would love to hear about it.
I’m not a Fire★ user, but the app does open now!
Regarding the app nap, I thought you could write NSAppSleepDisabled -bool YES into your app’s preference file to disable it. There’s a toggle in the Info pane for the .app in Finder too, although I’m not sure what that changes. Modifying the global blacklist would scare me as a developer to be honest, I’m surprised Apple don’t have anything in the docs about turning it off per-app. I guess they want people to be forced into using it, and opt-out for certain tasks using the APIs available.
NSAppSleepDisabled -bool YES
caius, thanks for trying! Glad it runs.
I found this thread which has a hackish solution, but should work. I am going to try this tonight and upload a new version. If I can get App Nap to turn off then I will finally be able to publish a Mac build officially, which would make me happy.
Why on earth did apple decide to create a feature that would break old applications and destroy POSIX compatibility is beyond me. No way in hell am I going to bend over backwards to support a feature only Macs have. I understand the motivation to save battery life, but really it needs to be opt-in. They already provide a way for users to figure out which app is the biggest energy drain.
Have a new build with App Nap turned off here, just in case anyone cares.
It is a Qt App and I used macdeployqt utility to package all the dependencies. Looks like it missed one! not sure why… I’ll post an updated one in the evening.
Seriously enjoyed my half week holiday last week, feeling much more relaxed about life once more. Tinkered a little bit with my spare raspberry pi, and I think have the required pieces in place to have it automagically detect my sports watch being on & nearby to it, and upload the latest activities from the watch to dropbox. Just need to hook a couple of scripts together and have it run frequently. (Yes, I am utterly lazy when a computer can solve a problem for me. The darn USB dongle for my watch is tiny and £45 to replace which is justification enough to have it in one machine permanently.)
This week it’s back to work and continuing on with the feature work I started a couple of weeks ago. I do love working on a product with designers & product people, makes developing pretty things that are useful for users so much easier than trying to do it on my own.
Outside of work I need to get my bike(s) serviced and back into working order so I don’t “accidentally on purpose” take the winter season off cycling as I did last year. Still haven’t finished Dishonoured either, so will probably continue gaming instead of computering in my evenings until it’s finished. (Such a fantastic game. Reading up on it I found that one of the designers of it was involved with Deus Ex years ago, which would explain why I love it so much. Deus Ex was amazing.)
Still working on the hardware-accelerated key-value store. So far, the main accelerator part is working. It can perform lookups and serve data out of its local SRAM. Now we have to get it hooked up to the NIC and CPU, which is of course the hard part.
Work this week has not been so fun, with two projects I’m not very excited about. My main priority is nailing down the requirements of a project for our intern. My next priority is documenting the Python module I wrote for accessing our logging database, so that the rest of the development team, QA/QE, and the automation teams can use it. I’m also hoping to squeeze in some coding, so that the whole week doesn’t feel unproductive.
Outside of work I’m still playing with Common Lisp, but I really need to find a bigger project to work on, as I feel I’m kind of wandering aimlessly. This past week I was looking at some web libraries (hunchentoot, cl-who) and a PostgreSQL library (pg). This coming week I’d like to use those to build something, but I’m not sure what.
My favourite postgres library for Common Lisp has to be
postmodern, in case you haven’t
Thanks! I had seen it, but somehow got the impression it was more of an ORM. On second look, it appears Submarine is an ORM built on Postmodern, but Postmodern itself is just another (better) interface to Postgres.
In fact, the Cliki page for PG has a plug for Postmodern, so I guess that says all I need to know about which one to use…
This guide to Postmodern popped up on my Twitter feed, and was one of the most useful resources I found for working with Postmodern. You might find it useful, too.
I’ve been working on a simple battery status bar for X. It runs on OpenBSD and Linux at the moment. The bar is 2 pixels thick by default and has a bottom placement.