1. 95

As I prepare to go on vacation, was thinking about where I am with my side-projects. I know I have a predisposition to yak shaving, but just now realized I’m in six levels deep. My current stack is:

  1. Make a color picker that supports gradients, therefore
  2. Make a library of UI widgets to allow you, among other things, to make these kinds of controls (X/Y pad, sliders with many handles, etc.), but then
  3. Make a color-handling library because with the existing libraries I can’t process user input the way I need. Oh, this turns out to be a nice library which needs equally nice documentation, so
  4. Build a static website generator because clearly none exist at the moment. It’s going to use Nunjucks templates because I have used, and like, Twig. But Nunjucks does not support the really good parts of Twig, therefore
  5. Write a templating language in the vein of Jinja/Liquid/Twig; but a full blown parser seems… excessive, so I wonder whether I could use a JavaScript parser instead, therefore:
  6. Extend the Acorn JS parser to support evaluating expressions you’d find in a template language.

So, get ready for the best color picker 2025 has ever seen?

But, seriously, I wonder: fellow lobsters who are prone to yak shaving, how many levels deep are you, and what’s your current stack?

  1. 53

    All I wanted to do was help a company ship its domain-specific software faster so I looked into Docker.

    Years later I’m working in infrastructure for banks trying to set up a microservices platform.

    I can’t remember what business logic looks like anymore.

    1. 0


    2. 24

      I want to write a machine code development tool.

      I’ll write it in Common Lisp.

      I need to be able to exit the program (nonportable) and send terminal control codes.

      I spend a day researching all of the different exiting S-expressions before collecting them into a library.

      I don’t like Ncurses or binding with C at all, so I’ll write my own terminal library.

      Using this terminal library isn’t all that pleasant, so I’ll write an abstract terminal library on top of that.

      That underlying terminal library doesn’t depend on ASCII, but does detect it and optimize for it, I could turn character-set detection into its own library.

      This tool should be customizable, but I don’t care for arbitrary customization languages, so I’ll make it customizable in a machine code.

      The target, CHIP-8, is entirely unsuited to this, so I’ll create a Meta-CHIP-8.

      My Meta-CHIP-8 virtual machine needs to be able to access the primitive routines of the system in a well-designed fashion, so I must spend time designing each one where all behavior makes enough sense.

      I don’t have a Meta-CHIP-8 development environment, so I’ll just use a hex editor.

      This Common Lisp program, with its own machine code customization language, really isn’t that pleasant to read through, even though it’s efficient and works, sans a few minor flaws in some more advanced functionality I’ve simply yet to correct. It also consumes megabytes of memory, since it runs in a Common Lisp environment. I could rewrite it in chunks, again, and start removing Meta-CHIP-8, but I’m so exhausted by it at this point.

      I’ll write a new version in Ada, which will use far less memory.

      I’ll start learning Ada.

      Now that I’ve largely learned Ada, I really should start implementing some of my Common Lisp libraries in Ada so I can enjoy some of the same abstractions.

      That’s roughly where I’m at, currently. Some of this isn’t strictly hierarchical, but you get the idea. Something pleasant to do is occasionally take a break and work on something else, such as how I’ll be working on some server software in Common Lisp and Ada, to compare the two. However, I need my own networking library in both, since I’m dissatisfied with what’s available, and I dislike the Common Lisp multi-threading options, so I’ll need to write my own there. You understand the general trend.

      1. 2

        That’s great. The CL and Ada thing especially. I once suggested on Schneier blog that mixing the most powerful and most constrained languages together might make for interesting experience. Maybe mock up Ada in CL with extraction-to-Ada tool. Then, you get CL’s rapid prototyping and metaprogramming with Ada’s safety checks and compiler performance. Never tried it since they were both pretty complicated. ZL guy is closest doing C/C++ in Scheme.

      2. 21

        Well, my goal was to have food and safe lodging for my family, and yadda yadda yadda now I’ve learned how to cast magic spells using my fingers on this weird contraption made out of a specialized form of sand.

        1. 18

          One day, I read a book called Daemon (by Daniel Suarez). Its year 2010 and I begin to think about the infrastructure and architecture of the Daemon.

          I like personal wiki software, and I am trying to reconcile something like three to five places where I write my notes into one place in computer shared across all devices I own at that time. I have this beautiful idea; lets make decentralized infrastructure that will allow you to work with structured data. It should also be a personal wiki. Tool which you can use to write your personal notes, use it to generate your homepage, but also to interact with all feeds you use and generate (everything from jabber, irc to last.fm feed and catalog of your movie ratings on imdb), index them and work with them with all of your programs. Websites? Pfffff. Just write an API for them and mirror them in this tool, so you can build custom views on the comment streams and whatnot! It will be glorious.

          So I start to learn Rebol, because it seems to be really ideal for this kind of tool. I read everything there is about Rebol. I don’t mean it figuratively, but literally. I’ve read all books, all articles I could find and most of the discussions in systematic manner during a period of year or two. Then I decide that whole tooling is kinda immature and strange and broken and forgotten. But I learn that Rebol is like a lisp with different syntax. Hmm, lisp you say?

          So I learn lisp. I even write my own interpreter in D language. I really like metacircular evaluators and macros and all kind of strange stuff you can get with lisp. I read about Genera and other live environments, which leads me to Smalltalk.

          Smalltalk! This seems fine. And I really like the environment. So I order a bunch of books about Smalltalk and start using Pharo. But after a year or so, I realize that class based programming is really not the best solution for my problem.

          In 2015/9, I find Self. So I order a PDF manual printed on demand, and I find that it contains a lot of bugs. So I fix them, and order it again. It still contains a lot of bugs, so I fix them again and order another copy. Bonus: I now know Self, because I’ve read a handbook three times in a row. I have a whole personal wiki dedicated to just Self, what I don’t like, what I do, tips and tricks. I read almost all the papers about Self, and all articles (I have a page in wiki just dedicated to articles and I read them one by one), and then whole mail conference.

          When I try to write my first programs, I find out that Self is really broken. Unicode doesn’t work, environment is fragile and it is written in C++. I don’t like C++. How hard it would be to rewrite it from scratch? I really don’t want to do that, but how hard would it be to just implement frontend layer and use some existing backend virtual machine, like JVM, or Lua?

          One day I read about rpython. So I naturally try to write Self parser in it. And then AST representation. And then compiler. And VM. And suddenly, I am really doing my own reimplementation. And I am writing articles about Self, and about writing your own language, and having this strange conversations with people about structured operating systems and nature of the true object orientation and religious flame wars about languages and their object models.

          Suddenly its 2019, my language almost works, and there is this publicly accessible course about GraalVM in the city where I live. So I take some time from work and go there, and the teacher really likes my articles about programming your own language and connects me with some people from Oracle, who worked on the original Self. Sadly, I have no time to push this towards something, because I am trying to bootstrap hybrid of the first version of my wiki in PyQT and tinySelf (thats how I call my little language).

          Meanwhile, I still use CherryTree and also notion.so and my distributed wiki is still mostly idea and bunch concept images and pages and mindmaps, more than anything working. But I have dreams. Sometimes they are so vivid, that I am really mad that they are just dreams. And I also still have a lot of frustration from the technology around me.

          So I keep working, and shaving the Yak. He doesn’t know it yet, but I am determined to shave him and collapse him back to depth 1. He is restless and kicks around itself and fights me, but I’ll keep shaving it, with the upmost rigor and determination. I will do so, until it is bald and naked. I will conquer all his yakiness, and use it for my own purposes, or die trying.

          1. 1

            I definitely had a different, less-positive experience reading Daemon. I did not enjoy that book.

            1. 1

              I guess that depends on the type of the person you are (I was still a teenager back then) and also on time. In 2010, this was really visionary, specifically the idea that you can control physical items via virtual interfaces you see in AR. Of course, there were people who talked about this, but this was the first time I’ve read about it in some consistent and well thought manner. This was before Oculus and google glass and magic leaps, hololens and all kind of this technology we know today.

              BTW: I think that most interesting description of the Daemon technology was in the sequel “Freedom”.

          2. 17

            This isn’t that current but it’s been going on for a long time:

            Also featuring a related sidetrack:

            1. 8

              I need to implement a Forth so I can write firmware for my keyboard so I can write actual code.

              this thread, summarized. thanks for sharing the journeys!

              1. 2

                I would think that most Forths would fit in 2.5 KB of RAM if you implemented them in assembly, right? It’s just the C implementation that makes it too big, I would think.

                1. 2

                  Right, well I mean technically my Forth implementation did fit in 2.5KB; it’s just that most of the RAM got used up by the Forth itself, so when I went to load my application code, it would quickly run out of memory.

                  It turns out that even for seasoned Forth pros, it’s a challenge to get an implementation that’s practical on the ATmega32u4. The only one I know of is this: http://michaelalynmiller.com/blog/2017/10/04/enforth/ (btw the quote near the top from the #forth IRC channel was me) It’s also the only example I know of a programming environment for that chip where the compiler runs on the device.

                  1. 2

                    Oh, I see. That’s a lot better than I would have thought. Thank you for the link! The rabbit hole goes deeper….

              2. 13

                One of my stacks is:

                1. I don’t like my window manager
                2. I should write one in OCaml!
                3. I’ll write my own X11 bindings because there’s no existing OCaml library
                4. Ooh X11 has a protocol spec, guess I’ll generate the bindings from that instead of binding to libxcb
                5. The Python script that does that for C sucks, I’ll just write my own

                Recently I switched to XMonad which is very similar to what I want, except in Haskell, but I haven’t quite given up on my own.

                1. 16

                  This is exactly what happened to me a few years ago. Following my trail:

                  1. I don’t like my window manager.
                  2. But Openbox was pretty close.
                  3. I should write one in Go!
                  4. Go had some basic bindings, but they were incomplete and suffered from various issues, so I built my own.
                  5. Indeed, the Python script sucks, so I wrote my own.
                  6. The X protocol itself is pretty low level, so you tend to implement a lot of helpers to deal with things like EWMH, ICCCM, painting, event loops, input and so on.
                  7. Several years later, voila! I still use it every day on all my machines. I’ve even inflicted it on my wife, and she likes it.

                  I am not looking forward to the day when I’m “forced” to use Wayland. I can only hope whatever exists at that point fits my desired workflow.

                  1. 3

                    thanks for xgbutil, I literally just used it yesterday to build an app switcher in <150 lines of go

                    1. 1

                      Sweet! I’ve used your library as reference for a few things, not sure why I didn’t include it in the README for mine.

                      I don’t really know what to do about Wayland either. I suppose (if I ever finish shaving this yak) I’ll have to bind to an existing compositor and try to reuse as much logic as possible from the X11 version, because I really don’t see anybody ever writing a whole Wayland compositor in OCaml, or anything that is not C and maybe Rust for that matter.

                      1. 3

                        Yeah, using a different language—even Rust—looks like it will be a lot of work: http://way-cooler.org/blog/2019/04/29/rewriting-way-cooler-in-c.html

                        I just hope whatever Wayland compositors exist by that time will be good enough. As long as they have good multi-monitor support (being able to switch workspaces independently on each monitor), the ability to add/remove named workspaces on demand, and good support for both floating and tiling windows, then I can probably be happy. These seem like simple requirements, but they were hard to find when I started my Yak shave.

                        1. 1

                          Wayfire is a thing :)

                          being able to switch workspaces independently on each monitor

                          has always been the case in the current (wlroots) incarnation

                          good support for both floating and tiling windows

                          some cool tiling is coming!

                          add/remove named workspaces on demand

                          not yet, it’s a fixed grid for now..

                          but we have much more important features like wobbly windows and fire!

                          1. 1

                            Neat! I’ll keep an eye out. But yeah, being able to manage workspaces dynamically with names is a critical thing for me.

                    2. 5

                      This is basically how I got started programming, years and years ago:

                      1. The touchpad on my (Windows) laptop broke, rendering it useless
                      2. I read up about this “Linux” thing and specifically XMonad, which has full keyboard-only support
                      3. Install Arch
                      4. Reinstall Arch several times until I manage to get all the vital components including sudo, etc
                      5. Install XMonad
                      6. Attempt to configure XMonad, but the configuration language is baffling
                      7. Install a real editor to try to not mess up the indentation in XMonad configuration file
                      8. Can’t really grok Haskell, so try learning Python

                      … (a decade+ later) …

                      1. Switching back to Vim from Emacs because the VSCode Vim mode is vastly superior to the Emacs Keymap I found, and my team uses VSCode for golang development.
                      2. Guess I’ll switch my personal editor to vim. I’ve been meaning to try out this neovim thing anyway…

                      … :)

                      (Just Now) Notice that something funky in the lobste.rs css occurs when you have a numbered list in the hundreds or thousands….

                      1. 1

                        and it’s all for naught since Wayland and wlroot will replace all that X11 stuff … you know … any day now…

                        1. 2

                          I switched to sway a few weeks after the 1.0 release, and now most all of the applications I run are native wayland apps and not using xwayland.

                          1. 2

                            I have xwayland completely disabled in the main compositor (Wayfire), for the super rare occasion I need X apps (some legacy java or gtk2 junk) I run a nested compositor (cage) with xwayland enabled :D

                            1. 1

                              TIL about cage. That looks pretty cool, thanks for sharing it!

                      2. 9

                        I love this question!

                        1. Supposed to be writing a book proposal
                        2. Why do I need a publisher? Don’t care about the money, just put it online! I need a static site generator.
                        3. Well, if I write in the R7RS scheme, I can enter it in that little contest that’s going on. I should use Chibi for this
                        4. I guess I should learn the internals of Chibi, and how to get the most out of it. Wonder if there’s an openbsd feature so I can write an FFI for unveil(2)
                        5. Better make a PR to add an openbsd feature Done
                        6. The default scribble docs aren’t what I want. Better go figure out how to hack on Chibi’s Scribble.
                        7. Oh! There’s a bug in the issue tracker that looks easy to fix dealing with POSIX compat. clearly, I need to spend time understanding how chibi-ffi works to generate stub C code, to fix this bug.
                        8. Ooohhh, mailing list questions! They are asking about something I haven’t looked into yet, but will surely be important to me, soon.

                        1. 1

                          At least you were able to check off something.

                          1. 1
                            1. 2

                              I am well aware of pollen—it’s nice! But, not mine. :)

                              1. 1

                                I looked at Pollen and it seems interesting, but the ultimate decision to just write my own was motivated by lack of a community. With Jekyll or Hugo, there are lots of people writing howtos and making their sites open source—if it’s doable at all, there’s usually an example to learn from already. Matthew Butterick and the few third party users I could find all keep the source of their books to themselves, so there’s nowhere to look how they did this or that.

                                Which is why I’ve added an annotated config and a readme to my own site source right away—there’s no community for it either, but at least there’s on live example to learn from.

                                1. 1

                                  Matthew Butterick and the few third party users I could find all keep the source of their books to themselves, so there’s nowhere to look how they did this or that.

                                  Maybe they are not easy to find, but some examples are linked at the bottom of this documentation page (16.4 & 16.5), including one of Matthew Butterick’s own books.

                                  1. 1

                                    Thanks! I would likely have never found it without a pointer.

                          2. 8

                            That’s actually a really good question. I’ll be more than happy to answer you once I finish working on a replacement of this site which is clearly not meant for posting complex essays. Web sucks anyway, so I might have to invent a nice alternative and build a browser or some other tools to make it possible. That actually sounds like a lot of work and I wouldn’t want to waste my time developing such complex software for a substandard operating system, so I better finally get myself to write the right one. I obviously wouldn’t want to put in such huge effort for nothing and that would soon happen, because these days you can’t even buy a decent laptop. Better build it! But hey, who says x86 is the right choice? I’ll give a look to the other platforms, but I feel like designing my own is the way to go anyway – as always. I have yet to decide on the right language for the software part, but there’s something wrong with most of the options, so… yeah.

                            (This may sound a lot like irony. Wish it was.)

                            1. 12

                              Right now? 0. The last time I ended up shaving a yak? 64 levels deep. It was not a good day.

                              1. 4

                                …and this is why we have Linux distributions :)

                                1. 5

                                  I understand and share your sentiment, but I’m afraid calling the list “64 levels deep” is unfair to other “contestants” here. Quoting just a random fragment from the linked article:

                                  50. Realize I have to recompile git to use the updated openssl.
                                  51. Clean out previous object files in my local copy of git.
                                  52. Make git.
                                  53. Install git.
                                  54. Done!
                                  55. No, wait … there was something … what was it?

                                  I’m afraid those 6 bullet points are more like “just” one level of depth: “rebuild git” or something alike (not that it’s trivial, but compared to the OP’s list, they listed “make a color picker”, which is much more coarse-grained).

                                  1. 3

                                    Okay, I was a bit lazy in trying to determine the actual levels of yak shaving.

                                    1. 1

                                      We should write a standard for that. We’d need a website for that standard. So I guess we’re gonna need a web site generator… «yodawg.gif»

                                2. 7

                                  I keep a small herd of yaks. Whenever I want to do something productive, I always end up out in the herd, shaving a bunch of them until I get tired. Then the feeling passes and I’m back to watching Love Boat.

                                  What I want to do is write a static code analysis tool. The language doesn’t matter, so I’ll probably pick F#. I have an idea for a “cognitive load” metric that should be a new and useful way of looking at applications.

                                  What I’m actually doing is writing an essay on learning: learning how to make stuff people want, learning how to be a better programmer, and learning as a generic process, whether it’s a startup, large corp, or machine learning.

                                  The static code analysis would be easier and much more fun. Dive down into the bits, maybe do some cool compiler object coding, end up with a number or smiley face that we can all argue about. Fun times. The essay, however, is more important, as it frames up the books I’m writing. Being a really good learner is the key part of learning to be a better programmer, and without that framework there’s really no criteria for judging whether a static code analysis tool is any good or not.

                                  So for this week, at least, I’ve left my normal herd of yaks and am working with Meta-Yak, the Mother of all Yaks. Leave it to an old OO guy to always be looking for virtual base class.

                                  1. 5

                                    13 years ago, I, a poor college student, couldn’t afford $40 entry for a LAN party so I volunteered as staff…

                                    …in two weeks I’ll hold my ninth tech conference as an organizer.

                                    Eventually, I’ll be able to just pay for (read: have time to go) a freaking LAN party.

                                    1. 3

                                      My yaks are deep, but it’s not so much depth of recursion as depth of time.

                                      I wanted to make a single specific music widget (a little like WireWorld, where select cells play notes when activated) for SuperCollider, but I’m not a fan of how sluggish the language feels. So I started making my own language. This was probably 6 or 7 years ago.

                                      Along the way I managed to convince myself that my other project ideas (games, tools, whatever) were worth consolidating into this one project so I can eventually(!) shave them all at once. So instead of dozens of regular yaks, half-shaven, I’ve ended up with one larger, shaggier mega-yak. Maybe one day I’ll get back to that music widget…

                                      1. 3
                                        1. Make bunch of dev screencasts for side hustle
                                        2. Needs static backend to withstand billions of page views
                                        3. Setup jekyll with usual secret library of includes
                                        4. Gotta make this library open source
                                        5. Ofcourse needs a command line parser to install includes on demand
                                        6. jeez this command line parsing library argparse is basic
                                        7. Fine, I’ll make it more powerful

                                        Sooo a while later and now I have this — action heroes

                                        1. 3

                                          I’m pretty sure I’m about 2 steps away from actually saving a yak. My stack overfloweth.

                                          1. 3
                                            1. I wanted to give a presentation at the Emacs meetup I run.
                                            2. Can I outline it in Emacs, in org mode? Sure!
                                            3. Can I present that outline in Emacs? Let’s look at the existing Emacs presentation libraries! Ugh, those all suck. I’ll write my own!
                                            4. I just need a parser for org-mode files! Ugh, I don’t like any of them.
                                            5. I’ll write my own!
                                            6. You know, org mode’s design is kind of…hacky. I’ll just write my own org replacement! NO WAIT THIS IS A BAD IDEA BAIL OUT.
                                            1. 3

                                              I’m building a fully automated laser hair removal system, so that my yaks may never need to be shaved again.

                                              1. 3
                                                1. Want to make something on an embedded device.
                                                2. Realize there’s an extra undocumented processor on the SoC, and it’s faster than the main CPU!
                                                3. Try to offload some work to that processor.
                                                4. Make a compiler & toolchain for that processor, because there are none, eg. there isn’t a GCC target for it.
                                                5. Make an assembler for that processor, to get at least some code running before having to port binutils+gcc.
                                                6. Try to figure out what the instruction set of that processor is, because that’s — guess what — undocumented as well
                                                7. Try to turn that processor on in the first place. Did you really expect this to be documented? Well have I got some bad news for you.
                                                1. 2


                                                  1. Update $COMPANY (OSS) tools to better handle shared dependency versions, allowing:
                                                  2. Update $CLIENT server-setup scripts, allowing:
                                                  3. Provision new redundant web/app server VMs

                                                  It’s slowly working down, it was at 4 (updating the shared library the tools in point 1 are built on) a couple of weeks ago.

                                                  1. 2

                                                    I manage to avoid yak-stacking at work (better than most of my colleagues) just by being fairly lazy & not arguing with management’s demands about tech choice. But, basically every one of my personal projects (and some semi-personal projects) have incredible yak-depth.

                                                    When I worked at Xanadu, I was given the task of fixing & finishing the XanaduSpace demo, so:

                                                    1. the actual source code wasn’t available, so I asked around & got a couple different incomplete sets of source from different commits & knit them together,
                                                    2. but the code was poorly structured, so I restructured it (which took a good month),
                                                    3. revealing bad development habits stemming from the developer not using revision control, so I simplified the flow & removed duplicate & near-duplicate code,
                                                    4. revealing that there was missing functionality, which I filled in,
                                                    5. revealing that premature optimization in the code that functionality depended on actually made things worse,
                                                    6. so I rewrote those parts (and eventually the whole thing) in python,
                                                    7. but this revealed limits in the amount of text that could be displayed, so I embarked on a rewrite of the rendering mechanism to swap text in and out of video memory while also compressing the text representation in video memory,
                                                    8. but since this broke rendering altogether for a while, I ported the whole thing to a fully 2d frontend in TK,
                                                    9. but since TK didn’t have alpha, I ported the TK frontend to use plain SDL,
                                                    10. but there were some text-flow glitches, so I wrote a brand new graphics toolkit against SDL to prototype the behavior I wanted.

                                                    I also briefly worked on OpenXanadu, wherein:

                                                    1. I was expected to write an editor,
                                                    2. but the editor ran up against same-origin policy for third party sites, so I wrote a proxy for fetching,
                                                    3. but sometimes the proxy would run up against rate limiters, so I made it a caching proxy,
                                                    4. but even then, rate limiters for sites like project gutenberg were over-rigorous & didn’t follow w3c rules for how to indicate to spiders that a page isn’t available, so I downloaded all of project gutenberg in bulk & preloaded my cache with it

                                                    For my upcoming book:

                                                    1. I wanted to write a book about different user interfaces & how they influence user behavior, with the intent of opening up people’s minds to the possibility of things not already built into their favorite gui toolkits,
                                                    2. but first I would need to dive deep into the history of computing, in order to dig up interfaces that weren’t familiar (even to people who are more hard core than I am),
                                                    3. but the most interesting types of interfaces are the ones that integrate full programming languages,
                                                    4. so in order to progress on this book, I need to learn scheme, common lisp, oberon, ansi forth, colorforth, APL, and smalltalk. (I also have a stack of about 30 books to read for research purposes)

                                                    When I did Mycroft:

                                                    1. I wanted to write an expert system,
                                                    2. but expert systems in prolog tend to be a bit slow (unless optimized after the fact), so I wanted to make it parallel & memoizing so even naive code would be pretty fast, & I had some qualms about the closed-world assumption made by prolog-style planners & provers,
                                                    3. so I dug into the literature about probablistic logic networks & about attempts to parallelize prolog,
                                                    4. and wrote a totally new language with a prolog-like execution style but that had a composite fuzzy logic,
                                                    5. but I had trouble with parsing in the C implementation, so I rewrote it totally in lua,
                                                    6. but the lua implementation (because of lua’s non-standard regex) had a weird bug in parsing that affected the way truth values worked, so I started a total rewrite in python

                                                    Meanwhile, within the lua implementation:

                                                    6a) I thought it would be cool to have a handheld node, so I made sure all the code was compatible with the ESP version of lua (though it probably is too big to fit in an ESP’s memory), which involved writing code to set up a wifi mesh network and detect peers

                                                    6b) I liked how Julia had syntax highlighting even in the REPL, so I wrote a duplicate version of the syntax code for an ANSI-colored pretty-printing facility,

                                                    6c) and then I rewrote the inline help facility to work with the pretty-printing facility,

                                                    6d) and then I hooked the inline help facility into the code embedding / extension facility,

                                                    6e) but then I thought I should make something that prints an equivalent to the original implementation for any defined predicate, which meant putting hints into my implementation about compiler-created anonymous functions,

                                                    Another project:

                                                    1. I wrote an essay about how a UI framework might be set up so that it encouraged users to become programmers,
                                                    2. but people weren’t understanding it, so I started working on an implementation,
                                                    3. and the language that fit my requirements best was IO, so I started on an implementation in IO,
                                                    4. but IO is no longer maintained, the third party libraries I wanted to use for drawing wouldn’t build on my system, and my prototype intermittently broke on the last released version because of a known bug in coroutine speculative execution,
                                                    5. so I decided to write my own dialect of IO (and while I was at it, implement proper concurrency & embed some other features at the language level),
                                                    6. but the most appropriate language to implement it in was clearly go, which I don’t know, so I’m learning go while implementing a language in it,
                                                    7. meanwhile, I wanted to add the rollback/versioning feature at the language level, so I dug into the literature & previous implementation work on OSMIC,
                                                    8. but the format for these implementations was too bloated – a record for every operation was liable to fill the disk quickly, and records were too big to be safe as a journal – so I worked out a new format that didn’t have these problems,
                                                    9. but since I’m not an expert on this kind of concerns, I wrote up a description of the changes in order to have it checked by other people,
                                                    10. but no single document gives sufficient background on OSMIC for somebody to understand what I’m changing & why, so I started working on an overview of OSMIC itself

                                                    Some of the worst have been nanogenmo entries:

                                                    • My ‘asemic writing’ demo built a language from the ground up – producing glyphs based on a small set of actually-distinguishable marks that differentiate characters in real alphabets, combined based on heuristics about how the effort of pen movements affects glyph shapes over time, given actual pronunciations, with a vocabulary automatically produced where the frequency of 1-grams, 2-grams, and 3-grams actually matches the zipf distribution, because I produced a corpus based on random aggultination & then produced a markov model from traversing that corpus, and built the actual document from that markov model.
                                                    • My ‘comic generator’ used a bunch of heuristics to lay out arbitrary images into a comic-style page with gutters and an obvious reading order without drastically changing aspect ratio, & had different rules for how to filter images based on how many pages had been generated (based on the tendency in high-profile manga for the first few pages of a volume to be full-color, followed by the brunt being smooth-shaded greyscale, with some line/pencil sketches in the back). This produced a comic from arbitrary input text & an arbitrary set of images.
                                                    • My ‘mythology generator’ produced stuff that actually resembles ETCSL translations of sumerian myths by producing gods whose attributes are determined by the actual meanings & associations of the sumerian root words that their names are composed of, producing sets of ritual epithets associated with those attributes, and slotting them into templates based on actual sumerian myths. Being a giant nerd, I didn’t need to do a lot of research into sumerian myths or the language to do this at the time, but still.
                                                    • My ‘heist generator’ involved writing a planner & turning the planner’s debug information into a first-person narration of planning and execution. My efforts to make the planner’s narration seem sufficiently varied took away a lot of time from actually producing an interesting ‘world’ to plan through.
                                                    • My ‘movie script generator’ actually literally implements most of the rules from Save the Cat, but I ultimately got stuck on writing a bunch of dialogue templates for different character types in different kinds of scenes & how they’re allowed to interact.
                                                    • For some reason, I decided that in order to write a generator that produced supernatural stories structurally similar to those of M. R. James and H. P. Lovecraft (i.e., an academic, in the course of research, encounters an item with hidden supernatural significance and is traumatized), I had to read all of James’ and Lovecraft’s stories first. I did, but I’m no further on the generator.

                                                    I’ve gotten better over the years (I’m no longer resorting to literally writing operating systems, because I have a day job & no longer have the time), but having strong opinions about how tooling ought to behave drastically increases the amount of time one spends on yak-shaving. (Luckily, yak-shaving skills are transferrable: were it not for all this time I spent at the bottom of the stack, I would not have confidence in my ability to, say, bitbang valid msgpack or write a planner.)

                                                    Also luckily: when I actually need to, I know how to knock out a quick and dirty implementation in a few minutes. So, some of the complex things mentioned above were only conceptually complex, & took less time to code than to explain.

                                                    1. 2

                                                      Probably n + 1 unfortunately

                                                      1. 2

                                                        I wish I had the focus and drive to go this deep. My problem is usually the opposite: I get all excited about a new project but then I tend to burn out / lose interest before I get to finish anything.

                                                        1. 3

                                                          I feel like it’s related. Deep-yak-shavers get all exited and then lose interest in the original project too, so their next step is to dive headfirst into something tangentially related to avoid working on the main project. In both cases, the first project doesn’t get done but lots of other little things get started.

                                                        2. 2

                                                          I don’t know if it’s yak shaving as such because I’m not doing it, but rather I’m trying my best to prevent myself from addressing my programming annoyances. I’ve already spent enough time thinking about it to work out that it would require a new kind of IDE based on a custom UI framework, as well as a compiler and a VCS, so it would be a lot of levels.

                                                          1. 2

                                                            Been on this for the better part of a decade. I’m not very fast.

                                                            1. People should be able to do whatever they like with their computers.
                                                            2. Secure distributed computing should not be hard.
                                                            3. There should be a language for expressing secure distributed computations.
                                                            4. There should be a more modern flavor of E.
                                                            5. There should be Monte, a fast, featureful, modern flavor of E.
                                                            6. Monte is about as fast as CPython at best; it should be about as fast as PyPy at worst.
                                                            7. To speed up a language implementation, remove interpretative overhead. Monte’s toolchain should remove more interpretative overhead.
                                                            8. Kernel-Monte is shaped roughly like a lambda calculus. I should apply state-of-the-art compilation techniques for lambda calculus to Kernel-Monte in order to design a better Kernel-Monte toolchain.
                                                            9. There should be a reduction from Kernel-Monte to efficient low-level code.
                                                            10. There should be a reduction from Kernel-Monte to a gradually-typed lambda calculus with single-use delimited continuations and list patterns.
                                                            11. There should be a toolchain toolchain, to ease the writing of compiler-shaped modules in Monte.
                                                            12. The Monte toolchain should be able to compile certain non-Monte source files into Monte modules, given an appropriate compiler helper.

                                                            Given all of that, my current stack frame is a little surprising, but the tip of the stack is always a wild place. Currently: There is an ∞-category Tomb whose arrows are transformations from one language to another, hosted in a third language. The 3-arrows of Tomb are specializers, in the Futamura sense. There should be a Monte module which composes arrows of Tomb in a type-safe manner and can run selected arrows in order to compile arbitrary artifacts.

                                                            1. 2

                                                              I started by writing a basic ASCII terminal game and now have an entire tile game engine mostly written and no idea what the original ASCII game way about.

                                                              1. 2
                                                                1. The non-profit I’m volunteering with needs websites for their local chapters! I’ll set up a web server to host them.
                                                                2. Server setup+maintenance would be easier in the future if we use Ansible for configuration management, so I’d better learn Ansible.
                                                                3. Actually it’s terrible if I’m the only person who understands how things work, so I’d better teach Ansible to the other sysadmin volunteers.
                                                                4. Actually it sucks if I have to teach new volunteers one-on-one in real-time (volunteers come and go), we’d better write documentation for how we’re using Ansible etc. to set up the server.
                                                                5. I’m realizing I don’t really know how to write good documentation, maybe we should recruit a technical writer who we know to help write it…

                                                                In case it’s not obvious from a post about yak-shaving, the web server has not really been set up yet and has yet to serve up any web sites.

                                                                1. 2

                                                                  The progression from 2009 to the current day:

                                                                  1. I want to get back into 3D animation and video production, but Autodesk just bought Softimage and killed my XSI license. Blender’s come a long way since 2.1. But UI-wise all they’ve done is add clicky widgets over the 1.8 interface.
                                                                  2. Wait, what’s this “Verse” collaboration server and new UI proposal from Eskil Steenberg? It looks like it got stonewalled.
                                                                  3. Well, the Verse protocol has a lot of potential for building tools. I should learn some more about networked 3D systems(OpenCroquet, Second Life) and distributed systems in general(plan 9).
                                                                  4. Analyzing the Verse protocol, there’s a lot of attention paid to a custom transport just to get around TCP limitations. I should research network protocols(ZMQ, multicast).
                                                                  5. There doesn’t seem to be a way to get people to agree about binary formats across a network of wildly diverse environments. I should research message-passing(Erlang, OpenSoundControl, 9P, protobufs).
                                                                  6. It’s more apparent that it’s not about just sending bits around as fast as possible, the data model has to be flexible enough to work without subscriptions or interruption. I should research these distributed system ideas(CALM/Bloom, CRDTs, Google Wave / operational transforms).
                                                                  7. Users still like the metaphor of a filesystem to manage changes. I should research internals of source control(Perforce, git, mercurial)
                                                                  8. Hey, these Go and Clojure languages seems to have thought a lot about async and concurrency. I should research them. Along with these languages with neat ideas about topological/temporal nodes(netkernel, Orc, CMS pipelines).
                                                                  9. Graphics-wise however, it’s still hard to plug into display systems in a safe way that doesn’t involve C++. I should research these languages that make systems programming more tractable(OCaml, Rust).
                                                                  10. I’m currently out of work, so I should take some time to sit down and do some deep thinking/spikes on these concepts. Hm, I have time to read this 2006 paper on Applescript, and this message-passing reminds me a bit of Armstrong’s thoughts about Erlang..

                                                                  So far, this has been more about driving personal research and professional development. Sometimes I think “I have made no real demos on this thing itself, just whitepapers and notes, what’s the point in continuing?”

                                                                  Then I see a tweet like this or this, and that fire ignites again. My pinboard tag is about to hit 3000 entries…

                                                                  1. 2
                                                                    1. I was building a parametric optimization framework.
                                                                    2. Needed to run on multiple machines so created script to generate jobs for a batch processing system.
                                                                    3. Wasn’t satisfied with using json to describe the optimization operations. So decided to write a DSL.
                                                                    4. Which language to write the DSL in? Oh, let’s learn Perl6
                                                                    5. Wrote DSL using perl6 grammars.
                                                                    6. Wrote a compiler from DSL to batch processing framework.

                                                                    So also 6 I guess.

                                                                    1. 1

                                                                      don’t ask 🙉🙈

                                                                      1. 1

                                                                        I’m guessing my yak-shaving tree isn’t deep, but wide, since I indulge in many smaller projects, rather than recursively working on one.

                                                                        1. 1

                                                                          My current stack is 5 levels deep.

                                                                          1. Supposed to be writing firmware for some kind of microcontroller.
                                                                          2. Get the build and upload toolchain in order.
                                                                          3. Begin with writing the program for the firmware.
                                                                          4. Discover that you really need to write a custom PID-controller variant because bolting on an unntested 3rd party library is a bad idea.
                                                                          5. Write code to automatically estimate the variables that the PID-controller uses.

                                                                          However: Yak shaving is one of the primary reasons why I try to stick to the idea that “everything should run on the latest version of Debian stable and if that’s not possible, I immediately switch to a custom VM-environment in which everything does work”. If possible, I try to pull everything I can from their repositories and if that is not the case, I’ll include the required libraries into the project and build everything with the project.

                                                                          If this is also not possible, I switch to a customized environment in a VM and just pass that image around. It takes less time and it’s generally less work to set up. When I have to do this, I have a yak-shaving depth of 1 step, which is to boot up that VM in the future.

                                                                          1. 1


                                                                            1. Receive feedback in my review on the need to improve organizational skills;
                                                                            2. open .emacs.d/init.el;
                                                                            3. notice that I don’t use certain non-org related keybindings;
                                                                            4. uh oh I’m editing init.el … ?. halp

                                                                            All my yak-shaving is emacs related. Emacs is perhaps the greatest medium for yak-shaving ever invented.

                                                                            1. 1

                                                                              Surprisingly low:

                                                                              • Preparing documentation and website for a release next week.
                                                                              1. 1

                                                                                I don’t have a stack of yaks. I have a whole tree of them. I want to manage and generate configuration files as code so:

                                                                                1. Clearly I need to create a DSL for describing and generating Config file formats. But this means I probably actually want a programming language so.
                                                                                2. I should write a special purpose language for this
                                                                                  1. which means I need a parser for that and I really like parser combinators but I want to do robust error reporting.
                                                                                    1. So I should write my own parser comibinator library.
                                                                                  2. Tree walking interpreters are easy for a prototype but really I need an opcode virtual machine for this longer term.
                                                                                    1. But existing VM’s don’t really suite my needs so I should build a custom set of opcodes.
                                                                                    2. And a stack based VM to execute them.
                                                                                    3. And a compiler from the syntax to the opcodes
                                                                                1. 1

                                                                                  I made the transition to management a little while ago, and I’d say yak shaving is the definition of the work I do now.

                                                                                  • At the top I have company goals that need to be achieved
                                                                                  • There are various initiatives that need to be completed in order to reach those goals
                                                                                  • In order to complete an initiative I need to create projects to focus the team on
                                                                                  • Stories need to be created in order to have bite sized chunks of work for the project
                                                                                  • in order to complete the stories we need to have meetings to refine the work and get the development team everything they need

                                                                                  That is just one possible path that can be drilled down. Every possible piece of work that I’m doing can be traced back to some goal set by the company, usually pretty deep. The less obvious how something achieves a goal the deeper my yak-shaving stack goes.

                                                                                  1. 1
                                                                                    1. Build a spaced repetition flashcard app and website with usable minimalist UI.
                                                                                    2. Build a simple static site for every domain name I own, but have otherwise not used.
                                                                                    3. Build that forum software I’ve always wanted to create.
                                                                                    4. Build that twitter bot I keep day dreaming about.
                                                                                    5. Build that check-list app that meets my belief about what a check-list really is.
                                                                                    1. 1

                                                                                      At work, I’m moving a service to a new datacenter. I’ve switched the DNS over and would like to shut down the old stack…

                                                                                      • But first, I need to make sure any remaining traffic isn’t hitting the old stack.
                                                                                      • Which it is, and it turns out there’s something that (indirectly) hardcoded the old address.
                                                                                      • Oh look, I can’t actually change it unless I have a way to configure this property differently for qa vs. staging vs. production
                                                                                      • How even does Java synchronization and lazy init work? (Need a global object, but it can’t be initialized until the config is loaded, which is at runtime.)
                                                                                      1. 1

                                                                                        Ha, reminds me of “Hal fixes a lightbulb”


                                                                                        1. 1
                                                                                          1. Make an integrated version control and build system.
                                                                                          2. Make a requirements management tool because eventually there should be a server and multiple clients (Linux, Windows, Android, etc) implemented probably in different languages.
                                                                                          3. Design XML format and HTML export for that. (Is that a separate step?)