@work I talked my company into letting me do a TLA+ workshop for the other engineers, so working out the logistics for that. Also doing a postmortem on a month long fire. I’m pessimistic we’ll make take good lessons away but hey, we’ll see how it develops.

@home I asked Twitter what I should learn and Twitter said haskell so now I gotta learn haskell. I have Twitter my word and I only sometimes go back on my word.

Made big progress on my solitaire solver thanks to help from another Recurse Center attendee. It now solves most games, though sometimes it can be very slow. It’s doing a naive, exhaustive depth-first search and sometimes can fall into deep branches of the state space where it is uselessly shuffling cards around. This was mitigated by canonicalization of equivalent layouts. This week I plan to do a lot of the fun tinkering items that have been hanging out in the readme.

I wrote a short blog post about becoming the sysop of Lobsters. This week I’ll be doing some Rails work on the Lobsters codebase. I aim to review issues and PRs at least weekly, and I’d like to start on tools for moderator awareness I mentioned here.

And this weekend I’m taking advantage of the density of the Eastern seaboard to visit some high school and college friends living in neighboring cities, which is going to be some really satisfying catching-up.

In the last update I said I would run abuild, the build script for Alpine Linux under OSH [1]. Well I was able to run abuild -h, so that’s good.

Also good is that I was able to parse a million lines of shell scripts from dozens of different projects very successfully. The remaining 3 or 4 things that need to be parsed are extremely rare (e.g. case terminators like ;& and ;;& only appear in one out of dozens of projects – I suspect few people even know what those are.).

But the parser is way too slow. I wrote it in a very abstract style because I wanted to not end up with a mess of ad hoc if statements. But now I am grappling with the fact it’s crazy slow, like more than 100x too slow.

I worked on some benchmarking scripts, and found three (!) different bottlenecks:

1. Traversal of the AST while parsing to determine which here docs need to be read [2]. The algorithm is correct but inefficient.
2. The Lexer needs to be converted to re2c [3], not use a series of interpreted regexes. This is mechanical as I designed it originally to be converted to re2c (which in turn converts it to a series of C goto statements, where each jump location is a DFA state.)
3. ASDL [4] classes also need code gen. Python generates C code too but I probably won’t use that implementation.

For productivity/abstractness, I always chose to use the most dynamic form of metaprogramming possible (e.g. with the lexer and ASDL). But now I need to harden it and actually generate code.

This is related to the ideas here: https://lobste.rs/s/aqdixr/gentle_introduction_compile_time

I think those 3 improvements will result in 2 - 10x speedup. The remaining 10x is probably Python interpreter overhead, so I need a different solution that.

[1] http://www.oilshell.org/blog/2017/10/06.html

[2] http://www.oilshell.org/blog/2016/10/17.html

[3] http://www.oilshell.org/cross-ref.html#re2c

[4] http://www.oilshell.org/blog/tags.html?tag=ASDL#ASDL

Finalizing backpressure in the Pony runtime.

https://github.com/ponylang/ponyc/pull/2264

Last week

I didn’t end up having time last week to work on my Insteon controller library, I’m planning on continuing work on that this week. Last time I promise something in this thread. :)

This week I’m also working on putting together talk proposals for upcoming conferences: one Go talk and one JavaScript talk. My conference acceptance rate is currently 0%; fill free to poke me if you’re good at these things and have useful advice.

Work has a release in customer beta and we’ve found a few bugs, so mainly working on squashing those mixed with triaging other incoming reports, and hopefully squeezing in some cleanup stuff for a later release.

I recently found out that https://webscript.io is shutting down, which I think is a really big shame, when I went to go re-enable my account. So despite it probably being a bad plan, I’m working on a “clone”. Not yet sure if I’m going to try to get money out of it, or just run it for myself and release it as opensource. I had already started working on something like this before I found webscript, so I’m a bit familiar with the idea. I’ve already got a skeleton of the script runner stuff setup with basic sandboxing.

Just added stats that link frontend and backend performance. This is going to be huge for my companies black Friday situational awareness.

Can’t wait to plug it into https://github.com/nghialv/promviz

Also deploying my new container friendly load balancer into an prod. It’s container friendly because it sniffs http/sni and passes the FD to the appropriate backend process rather than proxy.

[edit]

forgive my spelling/typos for a couple weeks. I’m 1 week post bicep re-attachment so I’m one handed and doped up.

@work After a surprise restructuring two weeks ago and helping a friend with a college football photoshoot this weekend, this week is trying to identify what work should be now. Looking through Matt Mitchell’s Strange Loop keynote for ideas, along with revisiting my professional network. Finishing a SANS course/cert in the meantime, since it was already paid for.

@home Sorting through projects to decide which one to spend time on: ongoing research into distributed 3d/content creation, experimenting with Inferno on embedded/phone platforms, or scratching an itch and building a PIM server in MirageOS for use as a cloud/AWS image.

I’m adding (what will hopefully be) a smooth history-diving workflow to vgit, to help track down when sheet joins were broken in VisiData. If anyone wants to play with an experimental git TUI (or help design), I’d love to get some feedback!

Work: we launched so the integration work has finally come to end. Going to be moving on to optimising things. One of the problems we have now is our computations use tons of memory, and AWS only sells VMs with normal CPU/memory ratios so the CPUs are underutilised. We are going to be working on eventually reducing memory usage, and in the short term I’m going to be working on making individual computations multithreaded.

The first step though is writing a profiler that we can keep enabled all the time. Perf generates way too much data to run constantly, so I’m writing a sampling profiler that works in a constant amount of space. Basically it’s a fixed size hashtable, and when you can’t find an empty bucket you overwrite the bucket with the lowest hitcount from the set you looked at while reprobing. It’s probably not the best strategy but it was easy and in quick tests it seems to work ok.

Home: slow game progress, haven’t been programming much. Geometry clipmaps apparently are not a silver bullet. If I draw tiles I have to deal with seams, if I draw a massive mesh I lose frustum culling. Also struggling to get my head around the details of geomorphing.

I’m working on helmspoint, a way to deploy ML apps. I’ll be working on the docker client connection to create images. Before, I was just using popen for running docker on the command line. I’ll then need to do authentication for kubernetes and docker REST APIs.

I finished a blog post about how Norway got Japan to eat salmon sushi after a decade long campaign. https://medium.com/torodex/salmon-sushi-is-not-a-japanese-invention-9189d9cd78b7

I’m making it a point to read more lately. I’m currently finishing up “An Introduction to Error Analysis,” and I’m excited to move on to “Music By Computer,” which I picked up at a used bookstore the other day. It’s a collection of papers published in 1969, covering some very early work in programmatic music generation. It even came with a set of “flexi-disc” records, which I want to transfer to computer at some point.

Other than that I’m working on a graph library in Common Lisp. It’s reinventing the wheel, but I couldn’t find an existing library that I liked, and it’s fun to work through the algorithms.

$work: After massive, unbridled failures two weeks ago, and a release last week, and then a couple more bugfix releases, I think we’re finally out of the woods. I’m doing paperwork, trying to get back in the groove; and trying to direct whatever resources I can muster towards the project to get us off our old hardware and onto the new environments we have which are much better built and should make this sort of thing not so much an issue anymore (and might get us to a nicer deployment model to boot).

!$work: Turned another bowl, ambrosia maple, going to be a Christmas present for Mom. Came up with a shop plan, scrapped it, came up with another, scrapped it; this week is more planning. At some point I’m just going to buy a bunch of material and then start doing stuff. I am in an analysis loop; I don’t think I’ll make it to spring.

Put the Plane restoration to the side, until I get the shop set up, it’s really hard to do that kind of work efficiently. It was more likely for a father’s day present anyway, so it’s not the end of the world.

Started a brand new project involving crypto-currencies. Always fun to work on new domains.

I’m been working on a Django app for Dramatiq and making small improvements to it in the process.