This is the opposite of the “side projects” ask. Share projects you haven’t been able to start on or complete, as a side project or at work, what’s stopped you, how it could be improved etc.
In 2014 I submitted a conference proposal to talk about how to extend the editor-agnostic and language-agnostic nrepl protocol with more operations similar to how the LSP protocol is used today but also a lot broader: http://p.hagelb.org/clojurewest-2014.org.html I had hacked together a prototype using Emacs and the Clojure nrepl server.
The talk didn’t get accepted, and my attempts to convince the tooling maintainers that this was a useful direction didn’t really go anywhere. Now seeing LSP take off across a huge number of languages with a much more limited vocabulary of built-in operations (no eval operation? no test operation? come on!) makes me wish I had pushed harder on that.
That hits close to home, my diploma thesis was about getting IDEs better access to CI results, with an example of an eclipse plugin to map errors in Jenkins XML output back to code fragments (think PMD and stuff, less linters and LSP).
The department at the university didn’t find this idea great, I also tried some conferences (not interesting) and years later we see all kinds of nice integration of this stuff in IDEs. Maybe developer experience does matter after all :P
This is interesting to me, as I have been thinking about doing something like this for Scheme since last year. Part of the research would require looking into other approaches, such as LSP or nrepl. I would be very interested in hearing your criticisms of LSP, the protocol, and what should have been done better.
The protocol itself is OK-ish; the main problems lie around the set of default operations being very anemic and uninspired. Everything is centered around static analysis when you could get so much more out of the protocol if it allowed actually interacting with running code. My understanding is that hypothetically you could add this as an extension of the protocol, but you would lose the benefits of standardization.
The big two protocol-level benefits of nrepl over lsp are A) it’s full duplex instead of request/response; you can send a single command that results in a series of replies over time. Without this you have to resort to some pretty hideous hacks to get a code evaluation command which could print something, wait for a bit, print something, wait, then terminate. The other advantage is that nrepl uses bencode over sockets. The bencode conversion is trivial to implement in ~two pages of code, (66 lines of Lua) vs lsp which uses json+http which is a much more heavyweight stack. Granted in most cases you have a json parser and http client already available, but I still prefer the simpler approach when there aren’t really any technical downsides.
I actually wrote an nrepl server for Racket a few years ago but I didn’t end up using it much so it might be kinda janky: https://gitlab.com/technomancy/ogion
Some time ago I got bitten with the roguelike craze and was heavily into different roguelikes (Dwarf Fortress, Stone Soup, Nethack). I decided I’d take a stab at building my own (in Rust, of course, because it’s the best language on earth).
Unfortunately, I got a bit carried away with designing the game. Instead of trimming the specification down to what I could handle, I stupidly went ahead and tried to implement that monstrosity (the mapgen code was, what, 3k lines?). I had all kinds of fancy plans for keeping track of each tile’s temperature, atmospheric makeup, air density, etc. and even for modeling each monster’s thoughts (similar to Dwarf Fortress). I can’t even bear to look at that code anymore, so it’s unlikely I’ll pick it up.
Oh God, I haven’t finished so many things that I’ve long lost count of them. I have a long series of projects in various states of unfinished. I don’t really regret it, most of them are unfinished because there was something I really wanted to do, and I did it, and the rest of the project was just an excuse to do that particular thing. Others, especially those that I did in a professional context, were cut short by budget and/or time constraints. But it’s fun to reminisce about them. In no particular order, some of the things I started and never finished in the last 15 years or so include:
Single-board computers, based on various processors (6809, Z80, 8086, we-don’t-need-no-stinking-microprocessor-I’ll-just-wire-my-own – the Z80-based one is that sorta made it to the breadboard stage). But I did build/play with various parts that I didn’t understand well enough – like clock generators or the most expensive programmable interrupt controller in history, an 8259-ish clone that “ran” on a Zynq-7000 development board because that’s what I had lying around at work. Honestly, the biggest reason why none of these got finished is that I didn’t really want to build a whole computer, I just wanted something with front-panel switches. I have some killer front panel designs, I just don’t have a computer to plug them into :-D.
Sort of in the same vein, an emulator. I must have started dozens of them but never finished one. It’s one of those goals I never accomplished but one day it’s gonna happen.
A debugger/monitor for small (e.g. MSP430) systems (context: I was working on an operating system for that kind of devices at the time – that one was actually finished, put in boxes and sold and all – and I wanted to test/prototype various pieces of peripheral code/drivers without the whole OS behind me, but I also wanted to be able to poke at things in memory in an interactive manner and so on, and debugger support at the time was really bad on some of the platforms we needed, including the MSP430). It sort of happened but I never used it enough to polish the rough edges. It was actually useful and interesting – at the expense of a little flash space, you got an interactive debugger of sorts over a serial port that allowed you to “load” (eh) run and edit small programs off of a primitive filesystem. Realistically, it was mostly a waste of time: this wasn’t a microcomputer, it “ran” on MCUs inside various gadgets. The time it took to “port” it to a new one vs. what you got in return just wasn’t worth it.
A SDR-based radiotelescope. I had a pair of Ettus Research SDR boxes more or less all to myself for a few months and I could play with them more or less at will as long as I didn’t break them, but the company I was at went under before I got to try anything (my knowledge of antennae was, uh, I’d say rudimentary but that would probably be overselling it). I did get to write some antenna positioning code that I later integrated into some real-life firmware at $work so it wasn’t all wasted.
A Star Trek meets Rogue, uh, I’d say rogue-like? Unfortunately implementing all the cool things (random story generators! random races with political intrigue and all! Gandalf-like figures roaming the galaxy!) was way more fun than implementing the actual game so I ended up with 40,000 lines of Java that spit galaxy news in a log file and nothing else. I learned a lot about sparse matrices though – that was actually the whole reason why I wanted to get into it in the first place (tl;dr I wanted to model something that supported tens of thousands of star systems with millions of ships and so on) – and of all the projects in this list, it’s the one that would’ve probably been easier to make into something cool. I tried to restart it at some point, then I learned about Dward Fortress and I honestly couldn’t see the point anymore :-).
A software synthesiser that tried to use some pretty advanced physical models to generate wind instrument sounds. Unfortunately I got so bogged down into the modelling side of things that by the time I had some basic prototypes, integrating them into a program worth using wasn’t really fun anymore, and I also couldn’t (still can’t…) really play any wind instrument so my understanding of these things was limited. I later tried to do a more ambitious synthesiser for a harp (tl;dr also software-driven but it used lasers instead of strings) for my final year university project but that never happened, and while I have a crude hardware prototype tucked in a closet somewhere, I never got around to writing any of the hard parts of the software. The biggest problem I had, and the main reason why this didn’t get anywhere, is that I just didn’t understand enough about real-time audio processing to get something useful. I still don’t.
An Amiga Workbench clone for Wayland. By the time enough of the cool features got implemented (e.g. multiple screens) I got so fed up with Wayland and Linux in general that I never wanted to finish it. Various bits and pieces, like an Amidock clone, got to a usable(-ish) state. This is the only project in this list that I didn’t really enjoy. I was already fed up with these things when I started it, I just didn’t really want to admit it. I don’t want to say anything about how this one could be improved and why I failed at it because I’m quite bitter over these things, but tl;dr I’d rather have all my teeth pulled out and swallow them than touch any of that stuff again.
There were others, much smaller, these are the cool ones.
All in all I think I finished very few of the side projects I started but I learned a lot out of all of them and many of them came in handy when doing stuff I actually got paid for. I have zero regrets for not finishing them. It’s important to finish some things but not all of them.
Reading a long list of projects that failed sounds a bit like a long list of failures but really, they weren’t. I achieved most of my goals. If I had an infinite supply of free time I could probably finish the ones that were never finished because they were too ambitious for my level of knowledge at the time (e.g. the wind instruments thingie) but there are so many cool things that I don’t know how to make that it kindda feels pointless to use my free time doing the ones that I now know how to make.
(Edit: I guess the point I’m trying to make is that no time spent hacking on something cool is truly lost, no matter what comes out of it in the end, and no matter how modest or grand the ambitions behind them. There’s a whole side project hustle mill going on these days and this whole “don’t send us a resume show us your Github profile” thing and I think it’s a con, and all it’s doing is making people afraid on doing things in their spare time, because they treat these things the way they treat projects they do at work. Computing was my hobby long before it became my profession, and it still is; finishing something “successfully” is besides the point when it comes to these things – their function is fulfilled as soon as a line of code is written or a piece of schematic is drawn, that brings me joy all by itself. Don’t fall into the trap of taking these things more seriously than you ought to. Most of us spend at least 8 hours/day agonising over whether something will be finished successfully or not – unless you enjoy that part of the job, there’s no reason to take it home with you.)
I, too, have looked at Dwarf Fortress and concluded I couldn’t possibly top it. Perhaps the way to approach something like that is to bite off a chunk of DF and trying to make it better, more realistic, more complex, or more fun. Of course, a lot of the magic of DF is the interconnectedness of the complex systems. But I can imagine one person making a very complex 2 person battle/dueling system, or a complex home decorator, or a terrain generator that goes higher into the sky or deeper into the earth, or a DF with birds instead of dwarves.
An Amiga Workbench clone for Wayland. By the time enough of the cool features got implemented (e.g. multiple screens) I got so fed up with Wayland and Linux in general that I never wanted to finish it. Various bits and pieces, like an Amidock clone, got to a usable(-ish) state. This is the only project in this list that I didn’t really enjoy. I was already fed up with these things when I started it, I just didn’t really want to admit it. I don’t want to say anything about how this one could be improved and why I failed at it because I’m quite bitter over these things, but tl;dr I’d rather have all my teeth pulled out and swallow them than touch any of that stuff again.
Holy cow this sounds cool. I’m trying to envision what this even would look like.
Specifically because as you’re well aware I’m sure an Amiga “screen” was kind of a different animal from anything that exists in a modern desktop context, and I don’t know how you’d enforce that kind of sliding behavior with modern windowing systems.
I just recently saw this project which bundles a fully emulated Amiga system into a Visual Studio Code package so you can compile, debug and run your Amiga code from a modern environment.
Specifically because as you’re well aware I’m sure an Amiga “screen” was kind of a different animal from anything that exists in a modern desktop context, and I don’t know how you’d enforce that kind of sliding behavior with modern windowing systems.
It’s been done before (to some degree) on X11 as well, see e.g. AmiWM (I think? I might be misremembering it, but I think AmiWM supported sliding screens. e16 had support for something like this a long time ago but I don’t recall if it could do split-screen). I only implemented a very rudimentary prototype which worked sort of like the famous spinning cube thing, except instead of mapping each desktop surface on a spinning cube, I just mapped it on different screen sections. I wasn’t really planning on adding it so It was more of a hack I cobbled together, it only worked under some basic scenarios, but I’m sure it can be done with a little patience.
Time tracker that helps me fill in timesheets based on what I did yesterday, when I forgot to track my time, and the time tracker wasn’t even running!
I’ve implemented a working proof of concept that scanned my disk for all kinds of timestamps and changes anywhere, like e-mails, browser history, filesystem, git, backups, etc.
What killed it:
The barely-working proof of concept was enough to work for me, but not good enough to release
I left the job that required timesheets
In the meantime operating systems improved security, so scanning all this scary private info doesn’t work any more
I’ve realized it’s a dream for stalkers and abusive workplaces, so it’s better that it doesn’t exist
Oh, time trackers! Mine only parsed git logs, but was good for initial versions of timesheets that I’d edit manually before invoicing.
Aside from the that I think fun things were the kernel (a teeny tiny one) and hacking on Circle MUDb codebase. Well, and the snake. And the… No wait, this list will keep growing.
I have found arbtt which seems like interesting solution to something you want. This will silently record your activity constantly and then you can extract data out of that. It still needs to run, but it can be configured to auto start on sign-in and then do it’s job always.
I solved part of this problem by just writing a log when the screensaver/lockscreen activates and deactivates. More for actual time spent working and breaks than what project, but if you only work on like 1-3 tickets per day it’s good enough.
I’ve been dreaming of making a “semantic history” browser add-on for ages, but never started.
(As in something that would collect all the RDFa/microdata/microformats/JSON-LD/… objects on web pages you’ve seen, and give you an interface to browse not just pages, but these various objects like Organization, Person, Article, etc.)
Wonder even if you haven’t started it, do you have any prior art to share? Would be very interesting!
Also have you seen https://www.geoffreylitt.com/wildcard ?
To my lasting sorrow, above all others, my long-dreamed-of attempt to build a great research writing environment. Maybe I’ll get back to it someday. But at the current phase of life, the sheer scope of it—something which can be basically an Integrated Research Environment, which can robustly handle reference management, authoring, publishing, etc.—is just waaaaaay beyond what I can do in my limited side time.
I tried building the Rust compiler dev-lang/rust on Gentoo but cancelled it after many hours and installed the binary package dev-lang/rust-bin instead.
A modern display server in the style of NeWS. NeWS had (display) server-side widgets, but they were written in PostScript which was very painful (PostScript is not the most user-friendly programming language). I’d love to have time to build something where (using X11 terminology for client and server):
All state lives on the client and so an application can disconnect from the server and reconnect to another or can reconnect to the server after the server crashes.
State can be cached on the server (e.g. geometry, text, pictures, fonts).
All rendering happens on the server.
Code running on the server is sandboxed (probably WebAssembly) and can respond to events without a network round trip (e.g. for animating a button when it’s pressed).
There’s a widget toolkit that explicitly builds asynchronous communication between the client and server in at the API level.
Synchronizing audio and video happens on the server and is exposed.
There’s a browser-based version of the server so that any application can be run in a web browser.
On the plus side, I have been sketching out the design of a programming language for modern hardware that allows unsafe-language code to be used safely (I mostly got sidetracked in recent years by working on the CPU features needed for the language to be efficient, but that’s mostly done now). When I started at MSR, I met some folks who had a much better solution to one of the key problems and so we’re now working on a language that addresses the problems I was trying to solve, in a much better way than I was planning on solving them.
Just like the USA, I never loose a war, I just “strategically retreat”. Therefore, I have a lot of projects that are suspended, and probably more I have forgotten.
Here are few off the top of my head:
One of my hardest failures was in high-school, when given the task to schedule classes for a particular problem. I didn’t know about NP, SAT, linear programming, hard/soft constraints, reduction, etc. So I tried to hard-code it, and it didn’t work out that well. The effort I invested vs the result was very depressing, and almost kept me from studying CS. The story turns out well in the end, as I managed to solve it as my bachelors thesis.
Around the same time I wrote an android game involving regular expressions. The code was horrible, and the puzzles were easy to solve. Had I known language theory, I might have figured out that I could construct a random regular expression, and then generate words to match and not to match. I can do that part now, but I’m lost when it comes to Android programming :/
I had the idea of using receipt paper and a little printer to generate a little summary every morning, of all the information I needed: When my buses arrive, what appointments I have, my todo-list, weather, etc. The idea was to replace my smartphone with a little piece of paper. But instead, I just have a dozen rolls of receipt paper lying under my desk, as I never found the time to look into how to hook up all the parts.
I also want to add that this kind of thread is very refreshing. People usually just talk about what the have managed. Only hearing that part of the story makes one feel very incompetent. These kinds of discussions help fix that impression.
I had the idea of using receipt paper and a little printer to generate a little summary every morning, of all the information I needed: When my buses arrive, what appointments I have, my todo-list, weather, etc. The idea was to replace my smartphone with a little piece of paper. But instead, I just have a dozen rolls of receipt paper lying under my desk, as I never found the time to look into how to hook up all the parts.
Last night I plugged in a cheap receipt printer (where GS V, the command for cutting, is a no op) and catted to it. I could realistically log to it from my shell now, and this is a superpower for me. I encourage you to find a USB interface receipt printer and just plug it in to a linux machine, it Just Works!
I tried to build a little “IoT” device: a physical knob that would just POST its value (0 to 1) to a little web endpoint whenever you turned the knob. Then anyone could view the value of the knob by going to that website. Mostly for the juvenile wordplay that this enabled (“knob” being a mild vulgarism, at least in the US).
I didn’t know anything about making physical things, so I got something that I thought would make it as “easy” as possible: a Spark Core (since discontinued, google tells me), because I assumed it would be really hard to get Wi-Fi working by myself.
But I didn’t realize, before I bought it, that the only way to load code onto the Spark Core was to use their weird online compiler thing, and the device would actually download the compiled image over Wi-Fi and like flash itself (??). There was no way to just flash from my computer, even though it had a USB thing. That sounds insane, and my memory might be tricking me, but I definitely remember having to use a little web-based “IDE” thing and I could not find a local alternative.
Anyway, if you were writing your own program for this thing – and you didn’t know what you were doing – it was very easy to like lock the device by doing your own I/O in a tight loop, which would prevent you from loading new code onto it – a bug that would prevent you from fixing the bug. So you had to do a hard reset, and for some reason that I can’t remember anymore, that was extremely painful…
Saying this out loud, this doesn’t sound that bad, but I remember being so frustrated by the experience that I haven’t tried to do anything in the physical world since. This was probably… six years ago now? I know that it was really just the fault of the device I was using, and googling this now tells me that its successor does have a way to load code directly onto it (using… Node.js??) so maybe this isn’t so hard anymore.
I’ve always wanted to go back and finish this, because I still giggle at the domain I had registered, but I got so discouraged that I gave away my breadboard/accessories in a recent move. Any pointers, if I were going to pick this back up? (Alternatives that don’t require me to think about Wi-Fi or use JavaScript?)
I haven’t been able to complete a userland TCP/IP stack on raw sockets. Writing C or C++ is the easiest in one sense because for interacting with Linux APIs and for doing integer/bit twiddling more succinctly than it would be in Go or Rust. But I got caught 1) not writing unit tests for basic message transformations 2) basic dynamic memory management bugs and then 3) couldn’t figure out how to do state management for TCP.
When I get back to it, unit testing all marshaling/unmarshaling will be an imperative and I’ll probably use boehm gc in
C++ to make memory management easier on myself.
I have a common pattern for failing with side projects wherein a friend(s) will propose an idea and I spend a huge amount of time creating a PoC, but then fail to create a decent feedback loop with them to push the project forward. Some examples of this are:
goftpd - an ftpd that tried to replace an ancient closed source ftpd called glftpd.
mx.ax - an email forwarding service that lets you to create mailbox aliases for your domain that can be forwarded on to other email addresses. You can also send outbound emails with it.
whois.bi - a domain monitoring service
The only benefit to these projects now is that I have a playground for trying out new technologies without having to create something from scratch.
Opposite of the topic (nice topic btw) but make sure to celebrate your successes too. I keep a project log of ideas, designs and plans (not just software) and looking back, I just assumed they were all unfinished but many of them were done. Some of them were simply spikes and learning something is done. Flipping through, I felt silly beating myself up because I had chunked this book as a list of project ideas that never materialized. It wasn’t a book of bad.
Alright, I’m coming back to this finally. What’s the current status on the project? It sounds like the thing I have been dreaming of for a long time, especially the differences from SSB.
Also: how do you do serialization? Do you have some kind of vector clock thing?
Greetings! Thanks for looking into the project deeper.
To answer your questions:
What’s the current status: I will say “hibernation”. I’m basically waiting for two things: A cleaner schedule on my end, and an increase in outside interest. If you want to dig deeper into the codebase, feel free to email me. My email is publicly available on Github. If not, PM me for it on Reddit (same user name as on Lobste.rs).
How do I do serialization: I unapologetically re-invented the wheel for this. The serialization format is a custom solution. I don’t know how deep you dived into SSB, but if you have ever tried to sign a message by hand, you will understand why JSON was not appropriate for the job. I needed a serialization format that was human-readable but which also did not leave room for interpretation or ambiguous whitespace (quite a big deal when you need to sign things and verify signatures).
Vector clocks: I copied SSB’s way of doing it. Every user in Pigeon keeps their own personal clock. Each message increments the feed’s depth by 1. This prevents hash collisions, also.
It’s been a while since I’ve touched the project but please do keep in touch if you are as interested in it as I am.
One final thing I will say: I did a lot of experiments to get the project to where it is. In doing so, I learned a lot and uncovered a lot of flawed thinking. If you decide to help on Pigeon or decide to spin off your own project, I’d be happy to share these lessons learned.
feef was initially going to be a full TUI feed reader, pioneering my incoherent ideas about “reducing noise” - it would have a pane with new unreads, all unreads and one with noisy feeds such as commits. A sort of plan is still to be found here. I ended up collaborating with 2 people on it - one of which was very new to programming, who disappeared shortly. Me and the other collaborator wrote some useless stubs - a random RSS feed generator for testing, and some UI bits. After that, I continued on a months-long path of making exploring various Go crap but not producing anything useful - that’s why there were so many branches. The other collaborator esentially left, but I rudely still tried to get his attention. It got close to 600 lines before I decided to scrap it, and I rewrote it as a CLI tool, which I still use quite often today for downloading podcasts and piping into FZF to make a decent feed reader. I honestly don’t care about telling people about it anymore. I don’t want to improve it because I can’t write Go now without getting enraged at slightly non-idiomatic libraries and consider feef’s design jagged and incoherent. But it’s extremely useful to me, so is it a failiure?
During my Go phase I also tried writing an activity monitor after becoming sick of the stupid “modular” crap that is ActivityWatch. Funny that I scratched an itch for simplicity by writing a bunch of bad, overthought DB abstractions (let’s make a KV DB structured because SQL bad!). I also messed with haphazard Go automation and keylogging libraries, and Neovim RPC. I loved writing it because I could use “nice” libraries but I never actually used it. Again, I ended up re-writing this from the bottom up, as a shell script then as a small Go program. All this stuff has mostly been superseded by a shell script piping some command into a CSV file.
I’ve also tried writing FUSE/9P filesystems. First a FUSE FS for clone-less GitHub browsing (like github1s but FUSE). I wrote way too much code before running it, with full, hard-to-debug laziness and all. I learnt how painful debugging FUSE filesystems is - I/O errors by default, having to use a separate tool to abort a FUSE filesystem. It was fun to write and think about and completely fruitless. After that, I thought Plan 9’s 9P might be easier to use, so I tried reimplementing playlistfs (a playlist file server, disappeared after Plan 9 1st Edition). Didn’t go that far really, got confused with the pain of actually mounting a 9P filesystem on Linux.
I could go on for ages, I’ll save you the reading though. I like to think I’m more sensible and patient than this but who knows! But despite nothing really being produced, I had quite a bit of fun exploring Go libraries and overthinking all this stuff.
I’ve been trying mightily and failing to build a mouse driver for my bootloader project. I think I’m missing a lot of fundamentals here: initializing the interrupt vector table, using Qemu and Bochs, debugging with Qemu and Bochs. And after all that it’s exhausting to know that there’ll be more struggles in getting it to work on real hardware rather than a simulator. Perhaps programming seems as difficult to a non-programmer.
I have an idea that I am constantly restarting, but never finishing in any useable state:
Local development reverse proxy that works with *.localhost TLD and allows dynamic registration of services with or without HTTPS support. I need to finally make it work one day in somewhat useable state.
The general concept is simple - provide HTTP Reverse Proxy on port 80 and TCP Reverse Proxy (with potential TLS termination) on port 443, so you can test your applications against “real” domain. Right now I am simulating such workflow via Haproxy, but it is very limited in capabilities (for example no dynamic service registration and needlessly complicated configuration). Maybe next week (I am between two jobs) I will finally try to smash something together.
That’s a cool project! In case you hadn’t heard of it, I wanted to mention https://ngrok.com/ – it sounds like an implementation of what you’re describing. It’s not open source anymore – although an early version is? – but it’s free (no signup or anything required) to use in modest capacity. Apparently it rate limits you to 40 connections/minute? I’ve used it for years as a way to share dumb side projects with friends. Super handy.
Ngrok serves different purpose, it is meant to be reverse tunnel for exposing locally running services externally, while my idea is to work fully locally.
I had tried building out something ngrok-y at one point, and during market research found it. It’s such a good idea, and deserves all the success it can get.
Back in the day, I used to use http://pow.cx/manual.html for this when I was working on Rails projects. It’s not exactly what you’re describing, but there might be some inspiration to be had.
Caddy server does the exact thing you have described as of version 2 - generates & registers a CA to sign certs locally for *.localhost domains, and supports dynamic service registration.
I was looking at Caddy 2, but I haven’t found a way to do dynamic registration/deregistration of services. Additionally Caddy cannot work as TCP proxy and will always work as a TLS terminating proxy, which I want to avoid if possible as I want my application to be fully self-contained (so it manages TLS termination and certificate generation via ACME on its own). That is why I am using Haproxy with manual config instead of Caddy right now.
Sorry if I’m wrong, but wouldn’t this be as simple as adding your chosen domain name to your /etc/hosts file? I already do this myself, naming my server IP’s so I can easily access them without remembering the whole address.
However, I don’t know if that would work with a *.localhost TLD…
on systemd-enabled systems *.localhost is already resolved to loopback
it fails when you want to test custom sub domains for example for multi tenant applications (or you need to create new entry for each domain)
you still need to run some additions proxy/web server to be able to listen on 80/443 instead of always specifying new port (or remembering which port is for which application)
this still requires me to somehow handle creating and maintaining TLS certificates which I wanted to handle “automatically” within the tool itself
For servers I am running DNS in my lab, so I can access all of them without remembering IPs as well.
The top several are all parts of a grand plan for a web-based IDE so there’s some stuff about syntax tree editors and some stuff about WebUSB and what even is a programming language anyway? I’ve been messing around with these ideas for decades but keep getting sidetracked.
Also I started writing a Lo-res video game for Apple 2, to celebrate my approximate 40th year of programming (which more-or-less started on the Apple 2) It’s really fun to play with but also takes a long time to make any progress since, well, 6502.
I started on a thing called Zombie which attempts to get rid of the entire web programming thing by having the server take remote control of the browser and pass it bite-sized pieces of javascript as necessary. The core idea is tiny and I think it’d fit in well with Django but I need to actually spend some time on it to develop useful components.
Oh and like a whole fertile mulch of smaller stuff …
My longest running failed project, is writing my own implementation of Prolog. Nothing fancy, nothing optimized, just a simple implementation. One of the reasons this happens, is that I am a procrastinator. To that add some level of financial insecurity which makes the “extra” time one could devote to a side project rerouted to a project for extra income and you’re there.
So a combination of procrastination, financial insecurity, new ideas worth learning about that come from interesting people around me, and basically any simple thing that can break the pace (and this includes other interesting things).
At the start of the pandemic I built a site that ripped live data from Google Maps on how busy a store was and displayed it with color coded icons on an embedded map. The idea was that you could enter the place where you wanted to get some groceries and that the five nearest supermarkets would show up and you could easily decide which one was the least busy, to lower the risk of infection.
I got it basically working, got a domain and all, but when I showed it to others they could not figure out how it worked. There was a map with an input field and some info. That was all. Apparently it is really far fetched to ask users to enter the street of city they want to visit.
Then I remembered two thing: I suck at design (UX or otherwise) and I loathe frontend development. So I didn’t want to spend more time on it. I used it myself for a while but then I switched to online ordering.
I observed that users want POI name, category, public transport station to search for more than street name. Still doable with Google maps/places if I remember correctly for the last time I used their API (even catching the same category shops in the same neighborhood).
In term of UX, something like the map and a big modal in front of it with the form to fill is usually easier to grasp for newcomers eg. “I fill that and it go away and the map shows me what I want” with a header that allows to launch new research. I realized that when a map takes most part of the screen people would want to click/drag everywhere on it and do not realize that maybe a input form somewhere need to be filled.
two-man project. We tried to build a social network around event attendance: Who’s going / who’s bringing what?
E.g., find a designated driver, organize a potluck. Go to a music festival as a group and make sure got everything your village needs etc.
This was before Facebook had an events feature, but it failed due to lack of steam and network effects preventing us from getting new users. was fun though. :)
Quite a few. Programming languages are never complete, but PISC ended well before it realized what I would have liked to have finished with it, at least with my original vision. It’s also the project that burnt me out on Go as a side project language. Learned a lot with it, though.
I have an unfinished web-based outliner project.
In general, I have found that involved GUI work tends to take a lot more time than I’d like it to, especially on the Web.
There are a lot of other projects that I have started and not finished well. Ideas are cheaper than execution, by a long, long way.
Back in 2013, I wrote a USPS address validation service (I had built a similar custom service for internal use a few years prior in the mobile telecom industry, and thought “hey I could build something like this again and make money off it this time!”).
I got the whole thing working - Stripe integration, API rate limits, user dashboard, fuzzy match for suggestions, batch loading of USPS address files with an Elasticsearch backend. Thought I could compete with Smartystreets on price. Even had a designer co-worker do some professional looking branding for it.
Thought to myself, “hey this new Docker stuff seems cool - I’ll deploy it with that on CoreOS and learn something from the experience!”
That, along with me learning that the USPS has a service that does the same thing effectively for free (as long as you’re using them for shipping) effectively killed it. :(
Every once in a while I think about resurrecting it, but I’ve also come to learn that “competing on price” is often a fools errand, and there are other players with more functionality. I also learned that there’s a very good reason why people start out with Heroku - I didn’t want to start with Heroku because as part of my career I’ve done literally three migrations off from it, but I was scaling before I even had a product. For my next project I still didn’t use Heroku, but I stuck w/ DigitalOcean and the capistrano scripts that I already knew.
I’ve been working on a game idea in TIC-80 off and on for a while.
You’re the ‘keeper’ of the eggs for an alien race, and the object of the game is to prevent aliens from a rival race from stealing the eggs and ‘turning’ them.
I’ve got the basic sprite animation working great (including the keeper which is a 3 piece sprite. Getting the tentacles to animate properly when he moves was fun).
However, I got stuck trying to properly implement the timers that would cause the un -molested eggs to safely hatch.
Currently have the source on my private Gitea but I could put it up on Github easily enough.
I’ve not given up on this. I definitely intend to get back to it and finish it as the idea keeps popping back into my mind.
(so far): animatronic butterfly jewelry. Covid really threw a wrench in my plans:
Supply chain screwed: can’t get my specialist tiny motors with high torque (only available from a few places in China)
Hackerspaces closed: nowhere to actually work on the project (just after I’d found somewhere with the PERFECT CNC machine ffs)
I had prototype that flapped wings for about 3 hours, running off a tiny LiPo battery. But I’ve been stuck there for ages, and now with my busy job I would barely have time even if the above issues weren’t in play :(
Chronicle allows you to form reliable memories. It keeps track of a tree of memories that are, ultimately, linked to the factual records recorded at the very moment they happened.
I was working on this years ago as part of learning Elm. The idea is still relevant, so I might revisit it - in the modern context of “thought streams” (as popularized by microblogging like Twitter/Mastodon, but done privately).
A 3D fighting simulator that uses human-like characters, physics simulation and evolution to find the best moves. Like an ultimate martial art style creator. This was far too ambitious.
A program for translating the SHA256 algorithm to a circuit. Even a compact description of the circuit is about 5 GB. It just became to heavy too work with, as a hobby project.
CPU-based ray-tracing that is fast enough to render not only a couple of spheres in 320x200 (which I made) but making something that could be used for an actual game. This still might be fast enough to render the stones of a Go-board with light, shadow and secular shading, though. The project is stranded, for now. Fast ray-tracing of texture mapped polygons in software was apparently not as easy as spheres. Who would have thought.
A Linux distro where there are no packages, only source code stored in a database that is compiled both on a need-to-run basis and also in the background until all the code is compiled. Inspired by TCCBOOT by Fabrice Bellard: https://bellard.org/tcc/tccboot.html
A Linux installer where the installation process is divided into: per-user configuration, per-use configuration and per-machine configuration. The per-user configuration (like keyboard layout, timezone etc) should be possible for users to provide exactly one time, then just refer to a code or an URL when installing any new operating system. The per-machine configuration should be detected automatically. The per-use configuration (like “music workstation”) should be specified at install-time and trigger the installation of a relevant selection of packages + default configuration. This stranded because I tried to solve too much at once. I should focus on just a per-user configuration-string generator web-page.
There are plenty more where there those come from. Luckily I also have a long list of completed projects under my belt, both on a hobby-basis and at work, so my self-confidence isn’t entirely crushed. :)
I wanted and still want to implement a pseudo-modern line editor (like ed). I have a few plan, mock-ups and ideas that I wrote down here and there but never took the time to actually do it.
During my master, I designed a full overhaul of the information system for the administration of the university on geospatial base. Merging outside and inside of building with a interface for the different units working on maintenance, security and etc. I got hired after my master to develop a prototype and delivered. The project never got further and I also wanted to push further but due to the complexity of the system, it will take half a lifetime to make it a viable product and the staff interested in it has moved to other projects.
A homework/examen/test generator that will generate as much as different version of questions and their solutions that students following the course. A prototype was developed and (almost) used when I was a TA in Linear algebra. I was banging my head on the code generation part of the solutions of the assignement (in Matlab thanks to codegen capabilities of Sympy). I wanted to push and generalize the idea but got struck on over-architecting/designing a full fledge prototype that go furter than a simple command-line tool. I will come back to the idea and principles one day. It was my answer due to the pandemic situation and the overused of platform that spy on students when you could simply make the communication between students the most complicated and meaningless possible :).
A recurring theme with failed projects related to work is that most of the time the prototype is GoodEnough(TM) and by the time it delivers, I have to move to other things and nobody need a generalization of the tools. Today again, i was reviewing a old code to submit as a code sample for an interview and looking at my TODO/NOTE command to expand it everywhere. On a personal side, those projects get lost on social life, others stuffs and never get a high enough priority. To be honest with myself, if it is not a tool that a need and can not already find, I don’t code that much on my free time.
A web browser. There’s a decent community of hobbyists doing/writing operating systems but no such similar community around browser development. Yes, it’s not possible for one person to compete with Firefox or Chrome, but certainly there should be more hobbyists browsers that can render some limited HTML (with minimal CSS).
Some other projects “on hold”:
A cross platform C GUI toolkit alternative to GTK.
Reverse engineering a DOS game from 1990 (this one I will eventually finish).
My ambitions have receded dramatically, as both my own limitations and the general badness of software as an undertaking have become clearer. Still, there’re a handful of things that come to mind.
Somewhere around maybe 2006-07, I started sketching out ideas with some friends in the social sciences for a platform built around structured historical datasets. We were inspired by a few experiments with things like volunteer-transcribed US Census data. I’m not sure I regret not going down that path, but it might have redirected me to a history PhD and a research career and a very different life overall, so it’s interesting to reflect on what might have been.
These days, the project that I have perpetually backburnered and occasionally pick at the edges of is a mechanism for unifying shell history, my written notes, bookmarks, and a tagging system. I want a way to slice and correlate things by time and subject matter across:
Documents I read and write
Commands I run
Things on the filesystem
Version control
Mail, photos, etc.
Financial transactions
External data sources - weather records and so forth, production logs from work
I’d also like this all to feed into a process for gradually accumulating scripts and possible menus of actions by topic or path.
Long before the AI, Alexa/Siri, and Home automation craze I wanted to build a sort of “personal agent” in chat bot form, where I could give it tasks via plugins/agents/workers for stuff I want to automate. Small examples would be weather/dictionary/interesting news articles.
In the end there were several problems. How to interface? At the time I guess Jabber would’ve been most convenient, IRC was too limited in being plaintext. Also “do what I mean” is surprisingly hard, that’s why IRC bots usually only respond to very fixed commands or, in the best case RegEx and praying for few false negatives/positives. Also I noticed that while I liked the idea I would NEVER recoup the time invested even if I looked up stuff manually for 10mins every day :)
Crypto For Science is a crowdfunding platform that uses the spirit of cryptocurrency in order to fund scientific research: no fees and no intermediaries - all the funds go directly to the researchers.
Not used for any actual fundraise because researchers from countries where this kind of funding would be needed were not able to understand cryptocurrency.
The first was a recreation of a game I really liked with a vibrant community but lacking involvement from the developers. I didn’t get very far at all; it turns out to write even “trivial” physics for a video game, you’re expected to know a whole bunch of I’m sure very basic stuff I never learned in my youth. That was frustrating and the first time I realised my lack of a formal education beyond my early teens could be a limiting factor, at least without significant extra effort. It’s not like learning another web technology or language where you can pretty much just pick it up based upon what you already know.
The second, more recent, is my desire to write my own programming language as a learning exercise. I’ve always had an affinity for mathematics which has unintentionally led me to topics like pure functional programming and type theory. Regardless of that, if you’re not familiar with concepts and syntax from certain branches of formal mathematics it’s tremendously hard to break into. I’m currently very slowly working my way through PLFA which introduces programming language theory using Agda. Even here, in this detour, I’m struggling because there’s terminology I have to go and pick up in order to progress. Honestly, this feels like the intellectual equivalent of picking up Minecraft for the first time without a guide. I’m sure given time and perseverance it’s possible without a backing formal education, but it’s rather challenging.
I made a gossiping membership and failure detection library/application written in Rust. This was async/await keyword so some of the combinators were pretty nasty. I spent a lot of time reading the serf source code, as well as papers related to gossip. I got a it working pretty decently with some tests even. I stopped working on it when I realized I had no use for it and wouldn’t realistically be able to test it at a scale that matters.
If I started again it would need to be upgraded to the latest Futures version, unsure how much work that would be. More importantly I would need to improve the error handling and do a general hardening of the code to avoid e.g. DDOSing the network.
Oh my god this is a very tough one for me. As someone who always starts a projects but somehow ends up leaving in the middle or unfinished.
So I started a project where I was going to build a news webscraper that would scrape news articles and run those articles through machine learning model that would tag those articles with different defined categories and then output those articles in a form of pdf report with tags and summary.
I managed to finish the scraping part and summarizing the news articles using gensim but never got a chance to implement the tagging feature as I lost the motivation and never finished the project. Now i feel like I should get back and try to finish that project.
Foursquare/Swarm clone based on OpenStreetMap data.
I got hooked up on the “check in” concept, but Foursquare pivoted from gathering venue metadata to straight up tracking people 24/7 and selling that, so I wanted a non-creepy version of that.
What killed it:
I’ve been overambitious and wanted web-based mobile UI, but with fancy animations, incrementally updating UI, and this was when smartphones could barely render anything, and React didn’t exist yet.
Venue info was spotty on OSM. Tags were a mess.
Missing data made me think I should redesign to focus on adding new businesses instead, but that wasn’t the app I wanted, so I lost motivation
I started it when I was on university, but later lost interest as I started “real work”. Since then a lot of other projects like that popped out and I never got back to it. However as const generics are now more useable I will try to revive it a little.
I started a side project called SimpleBlog. The idea was to auto publish Google Docs as blog posts with one click. I figured there were enough people who didn’t care at all about dealing with arcane web editors with too much configuration and not enough features in their word editor. For example, Substack is all the rage right now, and it still doesn’t have inline code blocks.
I made about a third of an MVP and got lazy and convinced myself no one would use it.
Hah, I had the same idea, and also abandoned it after building an MVP. I never had intentions of monetizing / publicizing, so for me the motivation was having a reasonable offline mobile editor (the Google Docs app) available to write trip journal entries while out in the wilderness.
I dropped the project after I realized I don’t write this kind of content generally, and it’s not due to a lack of software.
If anyone’s interested in a more complete implementation of this idea, Blot may fit the description.
I think at work I have my name on at least two “major” features that I sunk loads of time in, and both basically got frozen cuz we weren’t finding good product market fit. Though at one level it’s kind of unreasonable to expect basically one person to figure out a feature and like coordinate market research and user interviews etc and pull of the technical aspect (our team was very small at the time and everything ended up like this), on another level I know that if I had made just a couple small differences to the plan early on it would have gone out the door.
Huge sunk cost fallacies at play of course, but at one point you are throwing a part of your soul into a project and seeing it not get completed sucks.
(OTOH I tried like 5 different times to get Bazel working somehow with our setup for reduced CI times and last year I finally figured it out, that was really my white whale for a long time)
Nothing fancy but tried to build a web-based platform to evaluate programming assignments in colleges (especially in India where we still create hard-paper practical files).
Was in the process to build features like:
Hackerank style test case evaluation
Plagiarism detection (using MOSS ofc)
Code execution using Judge0 online IDE
Failed due to lack of planning (from my side) and team-effort since it was a major project :(
What I learned:
Working with a team is a very hard & important thing to do
In 2014 I submitted a conference proposal to talk about how to extend the editor-agnostic and language-agnostic nrepl protocol with more operations similar to how the LSP protocol is used today but also a lot broader: http://p.hagelb.org/clojurewest-2014.org.html I had hacked together a prototype using Emacs and the Clojure nrepl server.
The talk didn’t get accepted, and my attempts to convince the tooling maintainers that this was a useful direction didn’t really go anywhere. Now seeing LSP take off across a huge number of languages with a much more limited vocabulary of built-in operations (no eval operation? no test operation? come on!) makes me wish I had pushed harder on that.
That hits close to home, my diploma thesis was about getting IDEs better access to CI results, with an example of an eclipse plugin to map errors in Jenkins XML output back to code fragments (think PMD and stuff, less linters and LSP).
The department at the university didn’t find this idea great, I also tried some conferences (not interesting) and years later we see all kinds of nice integration of this stuff in IDEs. Maybe developer experience does matter after all :P
This is interesting to me, as I have been thinking about doing something like this for Scheme since last year. Part of the research would require looking into other approaches, such as LSP or nrepl. I would be very interested in hearing your criticisms of LSP, the protocol, and what should have been done better.
The protocol itself is OK-ish; the main problems lie around the set of default operations being very anemic and uninspired. Everything is centered around static analysis when you could get so much more out of the protocol if it allowed actually interacting with running code. My understanding is that hypothetically you could add this as an extension of the protocol, but you would lose the benefits of standardization.
The big two protocol-level benefits of nrepl over lsp are A) it’s full duplex instead of request/response; you can send a single command that results in a series of replies over time. Without this you have to resort to some pretty hideous hacks to get a code evaluation command which could print something, wait for a bit, print something, wait, then terminate. The other advantage is that nrepl uses bencode over sockets. The bencode conversion is trivial to implement in ~two pages of code, (66 lines of Lua) vs lsp which uses json+http which is a much more heavyweight stack. Granted in most cases you have a json parser and http client already available, but I still prefer the simpler approach when there aren’t really any technical downsides.
I actually wrote an nrepl server for Racket a few years ago but I didn’t end up using it much so it might be kinda janky: https://gitlab.com/technomancy/ogion
Sorry to hear this. this is a valuable lesson for me in the future
Some time ago I got bitten with the roguelike craze and was heavily into different roguelikes (Dwarf Fortress, Stone Soup, Nethack). I decided I’d take a stab at building my own (in Rust, of course, because it’s the best language on earth).
Unfortunately, I got a bit carried away with designing the game. Instead of trimming the specification down to what I could handle, I stupidly went ahead and tried to implement that monstrosity (the mapgen code was, what, 3k lines?). I had all kinds of fancy plans for keeping track of each tile’s temperature, atmospheric makeup, air density, etc. and even for modeling each monster’s thoughts (similar to Dwarf Fortress). I can’t even bear to look at that code anymore, so it’s unlikely I’ll pick it up.
been there, done that (on purpose). I stopped when I had a system that could allow to mod the game in natural language
Wait what. Can you give me an example?
Programming in natural language isn’t really that new. Here’s an example: https://osmosianplainenglishprogramming.blog/
Oh God, I haven’t finished so many things that I’ve long lost count of them. I have a long series of projects in various states of unfinished. I don’t really regret it, most of them are unfinished because there was something I really wanted to do, and I did it, and the rest of the project was just an excuse to do that particular thing. Others, especially those that I did in a professional context, were cut short by budget and/or time constraints. But it’s fun to reminisce about them. In no particular order, some of the things I started and never finished in the last 15 years or so include:
There were others, much smaller, these are the cool ones.
All in all I think I finished very few of the side projects I started but I learned a lot out of all of them and many of them came in handy when doing stuff I actually got paid for. I have zero regrets for not finishing them. It’s important to finish some things but not all of them.
Reading a long list of projects that failed sounds a bit like a long list of failures but really, they weren’t. I achieved most of my goals. If I had an infinite supply of free time I could probably finish the ones that were never finished because they were too ambitious for my level of knowledge at the time (e.g. the wind instruments thingie) but there are so many cool things that I don’t know how to make that it kindda feels pointless to use my free time doing the ones that I now know how to make.
(Edit: I guess the point I’m trying to make is that no time spent hacking on something cool is truly lost, no matter what comes out of it in the end, and no matter how modest or grand the ambitions behind them. There’s a whole side project hustle mill going on these days and this whole “don’t send us a resume show us your Github profile” thing and I think it’s a con, and all it’s doing is making people afraid on doing things in their spare time, because they treat these things the way they treat projects they do at work. Computing was my hobby long before it became my profession, and it still is; finishing something “successfully” is besides the point when it comes to these things – their function is fulfilled as soon as a line of code is written or a piece of schematic is drawn, that brings me joy all by itself. Don’t fall into the trap of taking these things more seriously than you ought to. Most of us spend at least 8 hours/day agonising over whether something will be finished successfully or not – unless you enjoy that part of the job, there’s no reason to take it home with you.)
I, too, have looked at Dwarf Fortress and concluded I couldn’t possibly top it. Perhaps the way to approach something like that is to bite off a chunk of DF and trying to make it better, more realistic, more complex, or more fun. Of course, a lot of the magic of DF is the interconnectedness of the complex systems. But I can imagine one person making a very complex 2 person battle/dueling system, or a complex home decorator, or a terrain generator that goes higher into the sky or deeper into the earth, or a DF with birds instead of dwarves.
Holy cow this sounds cool. I’m trying to envision what this even would look like.
Specifically because as you’re well aware I’m sure an Amiga “screen” was kind of a different animal from anything that exists in a modern desktop context, and I don’t know how you’d enforce that kind of sliding behavior with modern windowing systems.
I just recently saw this project which bundles a fully emulated Amiga system into a Visual Studio Code package so you can compile, debug and run your Amiga code from a modern environment.
It’s been done before (to some degree) on X11 as well, see e.g. AmiWM (I think? I might be misremembering it, but I think AmiWM supported sliding screens. e16 had support for something like this a long time ago but I don’t recall if it could do split-screen). I only implemented a very rudimentary prototype which worked sort of like the famous spinning cube thing, except instead of mapping each desktop surface on a spinning cube, I just mapped it on different screen sections. I wasn’t really planning on adding it so It was more of a hack I cobbled together, it only worked under some basic scenarios, but I’m sure it can be done with a little patience.
Time tracker that helps me fill in timesheets based on what I did yesterday, when I forgot to track my time, and the time tracker wasn’t even running!
I’ve implemented a working proof of concept that scanned my disk for all kinds of timestamps and changes anywhere, like e-mails, browser history, filesystem, git, backups, etc.
What killed it:
Oh, time trackers! Mine only parsed git logs, but was good for initial versions of timesheets that I’d edit manually before invoicing.
Aside from the that I think fun things were the kernel (a teeny tiny one) and hacking on Circle MUDb codebase. Well, and the snake. And the… No wait, this list will keep growing.
I have found
arbtt
which seems like interesting solution to something you want. This will silently record your activity constantly and then you can extract data out of that. It still needs to run, but it can be configured to auto start on sign-in and then do it’s job always.I solved part of this problem by just writing a log when the screensaver/lockscreen activates and deactivates. More for actual time spent working and breaks than what project, but if you only work on like 1-3 tickets per day it’s good enough.
I’ve been dreaming of making a “semantic history” browser add-on for ages, but never started.
(As in something that would collect all the RDFa/microdata/microformats/JSON-LD/… objects on web pages you’ve seen, and give you an interface to browse not just pages, but these various objects like Organization, Person, Article, etc.)
Wonder even if you haven’t started it, do you have any prior art to share? Would be very interesting! Also have you seen https://www.geoffreylitt.com/wildcard ?
To my lasting sorrow, above all others, my long-dreamed-of attempt to build a great research writing environment. Maybe I’ll get back to it someday. But at the current phase of life, the sheer scope of it—something which can be basically an Integrated Research Environment, which can robustly handle reference management, authoring, publishing, etc.—is just waaaaaay beyond what I can do in my limited side time.
I tried building the Rust compiler dev-lang/rust on Gentoo but cancelled it after many hours and installed the binary package dev-lang/rust-bin instead.
A modern display server in the style of NeWS. NeWS had (display) server-side widgets, but they were written in PostScript which was very painful (PostScript is not the most user-friendly programming language). I’d love to have time to build something where (using X11 terminology for client and server):
On the plus side, I have been sketching out the design of a programming language for modern hardware that allows unsafe-language code to be used safely (I mostly got sidetracked in recent years by working on the CPU features needed for the language to be efficient, but that’s mostly done now). When I started at MSR, I met some folks who had a much better solution to one of the key problems and so we’re now working on a language that addresses the problems I was trying to solve, in a much better way than I was planning on solving them.
Just like the USA, I never loose a war, I just “strategically retreat”. Therefore, I have a lot of projects that are suspended, and probably more I have forgotten.
Here are few off the top of my head:
I also want to add that this kind of thread is very refreshing. People usually just talk about what the have managed. Only hearing that part of the story makes one feel very incompetent. These kinds of discussions help fix that impression.
Last night I plugged in a cheap receipt printer (where GS V, the command for cutting, is a no op) and catted to it. I could realistically log to it from my shell now, and this is a superpower for me. I encourage you to find a USB interface receipt printer and just plug it in to a linux machine, it Just Works!
I tried to build a little “IoT” device: a physical knob that would just POST its value (0 to 1) to a little web endpoint whenever you turned the knob. Then anyone could view the value of the knob by going to that website. Mostly for the juvenile wordplay that this enabled (“knob” being a mild vulgarism, at least in the US).
I didn’t know anything about making physical things, so I got something that I thought would make it as “easy” as possible: a Spark Core (since discontinued, google tells me), because I assumed it would be really hard to get Wi-Fi working by myself.
But I didn’t realize, before I bought it, that the only way to load code onto the Spark Core was to use their weird online compiler thing, and the device would actually download the compiled image over Wi-Fi and like flash itself (??). There was no way to just flash from my computer, even though it had a USB thing. That sounds insane, and my memory might be tricking me, but I definitely remember having to use a little web-based “IDE” thing and I could not find a local alternative.
Anyway, if you were writing your own program for this thing – and you didn’t know what you were doing – it was very easy to like lock the device by doing your own I/O in a tight loop, which would prevent you from loading new code onto it – a bug that would prevent you from fixing the bug. So you had to do a hard reset, and for some reason that I can’t remember anymore, that was extremely painful…
Saying this out loud, this doesn’t sound that bad, but I remember being so frustrated by the experience that I haven’t tried to do anything in the physical world since. This was probably… six years ago now? I know that it was really just the fault of the device I was using, and googling this now tells me that its successor does have a way to load code directly onto it (using… Node.js??) so maybe this isn’t so hard anymore.
I’ve always wanted to go back and finish this, because I still giggle at the domain I had registered, but I got so discouraged that I gave away my breadboard/accessories in a recent move. Any pointers, if I were going to pick this back up? (Alternatives that don’t require me to think about Wi-Fi or use JavaScript?)
The esp8266 is a reasonable little bugger with just enough connectivity to do this
MicroPython might be good match, or Espressif’s SDK is pretty easy to work with if you like C.
I haven’t been able to complete a userland TCP/IP stack on raw sockets. Writing C or C++ is the easiest in one sense because for interacting with Linux APIs and for doing integer/bit twiddling more succinctly than it would be in Go or Rust. But I got caught 1) not writing unit tests for basic message transformations 2) basic dynamic memory management bugs and then 3) couldn’t figure out how to do state management for TCP.
When I get back to it, unit testing all marshaling/unmarshaling will be an imperative and I’ll probably use boehm gc in C++ to make memory management easier on myself.
I have a common pattern for failing with side projects wherein a friend(s) will propose an idea and I spend a huge amount of time creating a PoC, but then fail to create a decent feedback loop with them to push the project forward. Some examples of this are:
The only benefit to these projects now is that I have a playground for trying out new technologies without having to create something from scratch.
Opposite of the topic (nice topic btw) but make sure to celebrate your successes too. I keep a project log of ideas, designs and plans (not just software) and looking back, I just assumed they were all unfinished but many of them were done. Some of them were simply spikes and learning something is done. Flipping through, I felt silly beating myself up because I had chunked this book as a list of project ideas that never materialized. It wasn’t a book of bad.
I tried to build an offline-only peer-to-peer protocol but realized it will take more work/collaboration than I had available. Might come back later.
I will get back to it at some point. I learned a lot in the process and I still view the project as a worthy endeavor.
This looks really neat. I’ll have to look into it more later.
Glad you like it! Feel free to raise an issue or contact me privately if you have questions.
Alright, I’m coming back to this finally. What’s the current status on the project? It sounds like the thing I have been dreaming of for a long time, especially the differences from SSB.
Also: how do you do serialization? Do you have some kind of vector clock thing?
Greetings! Thanks for looking into the project deeper.
To answer your questions:
depth
by1
. This prevents hash collisions, also.It’s been a while since I’ve touched the project but please do keep in touch if you are as interested in it as I am.
One final thing I will say: I did a lot of experiments to get the project to where it is. In doing so, I learned a lot and uncovered a lot of flawed thinking. If you decide to help on Pigeon or decide to spin off your own project, I’d be happy to share these lessons learned.
Many, many things:
feef was initially going to be a full TUI feed reader, pioneering my incoherent ideas about “reducing noise” - it would have a pane with new unreads, all unreads and one with noisy feeds such as commits. A sort of plan is still to be found here. I ended up collaborating with 2 people on it - one of which was very new to programming, who disappeared shortly. Me and the other collaborator wrote some useless stubs - a random RSS feed generator for testing, and some UI bits. After that, I continued on a months-long path of making exploring various Go crap but not producing anything useful - that’s why there were so many branches. The other collaborator esentially left, but I rudely still tried to get his attention. It got close to 600 lines before I decided to scrap it, and I rewrote it as a CLI tool, which I still use quite often today for downloading podcasts and piping into FZF to make a decent feed reader. I honestly don’t care about telling people about it anymore. I don’t want to improve it because I can’t write Go now without getting enraged at slightly non-idiomatic libraries and consider feef’s design jagged and incoherent. But it’s extremely useful to me, so is it a failiure?
During my Go phase I also tried writing an activity monitor after becoming sick of the stupid “modular” crap that is ActivityWatch. Funny that I scratched an itch for simplicity by writing a bunch of bad, overthought DB abstractions (let’s make a KV DB structured because SQL bad!). I also messed with haphazard Go automation and keylogging libraries, and Neovim RPC. I loved writing it because I could use “nice” libraries but I never actually used it. Again, I ended up re-writing this from the bottom up, as a shell script then as a small Go program. All this stuff has mostly been superseded by a shell script piping some command into a CSV file.
I’ve also tried writing FUSE/9P filesystems. First a FUSE FS for clone-less GitHub browsing (like github1s but FUSE). I wrote way too much code before running it, with full, hard-to-debug laziness and all. I learnt how painful debugging FUSE filesystems is - I/O errors by default, having to use a separate tool to abort a FUSE filesystem. It was fun to write and think about and completely fruitless. After that, I thought Plan 9’s 9P might be easier to use, so I tried reimplementing playlistfs (a playlist file server, disappeared after Plan 9 1st Edition). Didn’t go that far really, got confused with the pain of actually mounting a 9P filesystem on Linux.
I could go on for ages, I’ll save you the reading though. I like to think I’m more sensible and patient than this but who knows! But despite nothing really being produced, I had quite a bit of fun exploring Go libraries and overthinking all this stuff.
I’ve been trying mightily and failing to build a mouse driver for my bootloader project. I think I’m missing a lot of fundamentals here: initializing the interrupt vector table, using Qemu and Bochs, debugging with Qemu and Bochs. And after all that it’s exhausting to know that there’ll be more struggles in getting it to work on real hardware rather than a simulator. Perhaps programming seems as difficult to a non-programmer.
I have an idea that I am constantly restarting, but never finishing in any useable state:
Local development reverse proxy that works with
*.localhost
TLD and allows dynamic registration of services with or without HTTPS support. I need to finally make it work one day in somewhat useable state.The general concept is simple - provide HTTP Reverse Proxy on port 80 and TCP Reverse Proxy (with potential TLS termination) on port 443, so you can test your applications against “real” domain. Right now I am simulating such workflow via Haproxy, but it is very limited in capabilities (for example no dynamic service registration and needlessly complicated configuration). Maybe next week (I am between two jobs) I will finally try to smash something together.
That’s a cool project! In case you hadn’t heard of it, I wanted to mention https://ngrok.com/ – it sounds like an implementation of what you’re describing. It’s not open source anymore – although an early version is? – but it’s free (no signup or anything required) to use in modest capacity. Apparently it rate limits you to 40 connections/minute? I’ve used it for years as a way to share dumb side projects with friends. Super handy.
Ngrok serves different purpose, it is meant to be reverse tunnel for exposing locally running services externally, while my idea is to work fully locally.
I had tried building out something ngrok-y at one point, and during market research found it. It’s such a good idea, and deserves all the success it can get.
Back in the day, I used to use http://pow.cx/manual.html for this when I was working on Rails projects. It’s not exactly what you’re describing, but there might be some inspiration to be had.
It is my inspiration as well. I just want to extend it with TLS and use proper domain instead of
*.dev
.Caddy server does the exact thing you have described as of version 2 - generates & registers a CA to sign certs locally for *.localhost domains, and supports dynamic service registration.
I was looking at Caddy 2, but I haven’t found a way to do dynamic registration/deregistration of services. Additionally Caddy cannot work as TCP proxy and will always work as a TLS terminating proxy, which I want to avoid if possible as I want my application to be fully self-contained (so it manages TLS termination and certificate generation via ACME on its own). That is why I am using Haproxy with manual config instead of Caddy right now.
Sorry if I’m wrong, but wouldn’t this be as simple as adding your chosen domain name to your /etc/hosts file? I already do this myself, naming my server IP’s so I can easily access them without remembering the whole address.
However, I don’t know if that would work with a *.localhost TLD…
There are few points:
*.localhost
is already resolved to loopbackFor servers I am running DNS in my lab, so I can access all of them without remembering IPs as well.
Alright, a bit more difficult than I imagined.
Ha ha ha how long have you got.
The top several are all parts of a grand plan for a web-based IDE so there’s some stuff about syntax tree editors and some stuff about WebUSB and what even is a programming language anyway? I’ve been messing around with these ideas for decades but keep getting sidetracked.
Also I started writing a Lo-res video game for Apple 2, to celebrate my approximate 40th year of programming (which more-or-less started on the Apple 2) It’s really fun to play with but also takes a long time to make any progress since, well, 6502.
I started on a thing called Zombie which attempts to get rid of the entire web programming thing by having the server take remote control of the browser and pass it bite-sized pieces of javascript as necessary. The core idea is tiny and I think it’d fit in well with Django but I need to actually spend some time on it to develop useful components.
Oh and like a whole fertile mulch of smaller stuff …
My longest running failed project, is writing my own implementation of Prolog. Nothing fancy, nothing optimized, just a simple implementation. One of the reasons this happens, is that I am a procrastinator. To that add some level of financial insecurity which makes the “extra” time one could devote to a side project rerouted to a project for extra income and you’re there.
So a combination of procrastination, financial insecurity, new ideas worth learning about that come from interesting people around me, and basically any simple thing that can break the pace (and this includes other interesting things).
At the start of the pandemic I built a site that ripped live data from Google Maps on how busy a store was and displayed it with color coded icons on an embedded map. The idea was that you could enter the place where you wanted to get some groceries and that the five nearest supermarkets would show up and you could easily decide which one was the least busy, to lower the risk of infection.
I got it basically working, got a domain and all, but when I showed it to others they could not figure out how it worked. There was a map with an input field and some info. That was all. Apparently it is really far fetched to ask users to enter the street of city they want to visit.
Then I remembered two thing: I suck at design (UX or otherwise) and I loathe frontend development. So I didn’t want to spend more time on it. I used it myself for a while but then I switched to online ordering.
I observed that users want POI name, category, public transport station to search for more than street name. Still doable with Google maps/places if I remember correctly for the last time I used their API (even catching the same category shops in the same neighborhood).
In term of UX, something like the map and a big modal in front of it with the form to fill is usually easier to grasp for newcomers eg. “I fill that and it go away and the map shows me what I want” with a header that allows to launch new research. I realized that when a map takes most part of the screen people would want to click/drag everywhere on it and do not realize that maybe a input form somewhere need to be filled.
two-man project. We tried to build a social network around event attendance: Who’s going / who’s bringing what? E.g., find a designated driver, organize a potluck. Go to a music festival as a group and make sure got everything your village needs etc.
This was before Facebook had an events feature, but it failed due to lack of steam and network effects preventing us from getting new users. was fun though. :)
Side projects I haven’t been able to complete:
Quite a few. Programming languages are never complete, but PISC ended well before it realized what I would have liked to have finished with it, at least with my original vision. It’s also the project that burnt me out on Go as a side project language. Learned a lot with it, though.
I have an unfinished web-based outliner project.
In general, I have found that involved GUI work tends to take a lot more time than I’d like it to, especially on the Web.
There are a lot of other projects that I have started and not finished well. Ideas are cheaper than execution, by a long, long way.
Back in 2013, I wrote a USPS address validation service (I had built a similar custom service for internal use a few years prior in the mobile telecom industry, and thought “hey I could build something like this again and make money off it this time!”).
I got the whole thing working - Stripe integration, API rate limits, user dashboard, fuzzy match for suggestions, batch loading of USPS address files with an Elasticsearch backend. Thought I could compete with Smartystreets on price. Even had a designer co-worker do some professional looking branding for it.
Thought to myself, “hey this new Docker stuff seems cool - I’ll deploy it with that on CoreOS and learn something from the experience!”
That, along with me learning that the USPS has a service that does the same thing effectively for free (as long as you’re using them for shipping) effectively killed it. :(
Every once in a while I think about resurrecting it, but I’ve also come to learn that “competing on price” is often a fools errand, and there are other players with more functionality. I also learned that there’s a very good reason why people start out with Heroku - I didn’t want to start with Heroku because as part of my career I’ve done literally three migrations off from it, but I was scaling before I even had a product. For my next project I still didn’t use Heroku, but I stuck w/ DigitalOcean and the capistrano scripts that I already knew.
I’ve been working on a game idea in TIC-80 off and on for a while.
You’re the ‘keeper’ of the eggs for an alien race, and the object of the game is to prevent aliens from a rival race from stealing the eggs and ‘turning’ them.
I’ve got the basic sprite animation working great (including the keeper which is a 3 piece sprite. Getting the tentacles to animate properly when he moves was fun).
However, I got stuck trying to properly implement the timers that would cause the un -molested eggs to safely hatch.
Currently have the source on my private Gitea but I could put it up on Github easily enough.
I’ve not given up on this. I definitely intend to get back to it and finish it as the idea keeps popping back into my mind.
(so far): animatronic butterfly jewelry. Covid really threw a wrench in my plans:
I had prototype that flapped wings for about 3 hours, running off a tiny LiPo battery. But I’ve been stuck there for ages, and now with my busy job I would barely have time even if the above issues weren’t in play :(
Chronicle:
I was working on this years ago as part of learning Elm. The idea is still relevant, so I might revisit it - in the modern context of “thought streams” (as popularized by microblogging like Twitter/Mastodon, but done privately).
Failed / postponed:
There are plenty more where there those come from. Luckily I also have a long list of completed projects under my belt, both on a hobby-basis and at work, so my self-confidence isn’t entirely crushed. :)
[Comment removed by author]
So much. A few on top of my head:
A recurring theme with failed projects related to work is that most of the time the prototype is GoodEnough(TM) and by the time it delivers, I have to move to other things and nobody need a generalization of the tools. Today again, i was reviewing a old code to submit as a code sample for an interview and looking at my TODO/NOTE command to expand it everywhere. On a personal side, those projects get lost on social life, others stuffs and never get a high enough priority. To be honest with myself, if it is not a tool that a need and can not already find, I don’t code that much on my free time.
A web browser. There’s a decent community of hobbyists doing/writing operating systems but no such similar community around browser development. Yes, it’s not possible for one person to compete with Firefox or Chrome, but certainly there should be more hobbyists browsers that can render some limited HTML (with minimal CSS).
Some other projects “on hold”:
There is a community around creating a browser for Serenity OS. Have you seen the videos by Andreas Kling, ie. https://www.youtube.com/watch?v=b3a5V45LLss ?
My ambitions have receded dramatically, as both my own limitations and the general badness of software as an undertaking have become clearer. Still, there’re a handful of things that come to mind.
Somewhere around maybe 2006-07, I started sketching out ideas with some friends in the social sciences for a platform built around structured historical datasets. We were inspired by a few experiments with things like volunteer-transcribed US Census data. I’m not sure I regret not going down that path, but it might have redirected me to a history PhD and a research career and a very different life overall, so it’s interesting to reflect on what might have been.
These days, the project that I have perpetually backburnered and occasionally pick at the edges of is a mechanism for unifying shell history, my written notes, bookmarks, and a tagging system. I want a way to slice and correlate things by time and subject matter across:
I’d also like this all to feed into a process for gradually accumulating scripts and possible menus of actions by topic or path.
Long before the AI, Alexa/Siri, and Home automation craze I wanted to build a sort of “personal agent” in chat bot form, where I could give it tasks via plugins/agents/workers for stuff I want to automate. Small examples would be weather/dictionary/interesting news articles.
Think a combo of huginn/Zapier/Slack Workflows.
In the end there were several problems. How to interface? At the time I guess Jabber would’ve been most convenient, IRC was too limited in being plaintext. Also “do what I mean” is surprisingly hard, that’s why IRC bots usually only respond to very fixed commands or, in the best case RegEx and praying for few false negatives/positives. Also I noticed that while I liked the idea I would NEVER recoup the time invested even if I looked up stuff manually for 10mins every day :)
A version control system for huge monorepos.
https://github.com/For-Science/Crypto-For-Science
Not used for any actual fundraise because researchers from countries where this kind of funding would be needed were not able to understand cryptocurrency.
Two things spring to mind.
The first was a recreation of a game I really liked with a vibrant community but lacking involvement from the developers. I didn’t get very far at all; it turns out to write even “trivial” physics for a video game, you’re expected to know a whole bunch of I’m sure very basic stuff I never learned in my youth. That was frustrating and the first time I realised my lack of a formal education beyond my early teens could be a limiting factor, at least without significant extra effort. It’s not like learning another web technology or language where you can pretty much just pick it up based upon what you already know.
The second, more recent, is my desire to write my own programming language as a learning exercise. I’ve always had an affinity for mathematics which has unintentionally led me to topics like pure functional programming and type theory. Regardless of that, if you’re not familiar with concepts and syntax from certain branches of formal mathematics it’s tremendously hard to break into. I’m currently very slowly working my way through PLFA which introduces programming language theory using Agda. Even here, in this detour, I’m struggling because there’s terminology I have to go and pick up in order to progress. Honestly, this feels like the intellectual equivalent of picking up Minecraft for the first time without a guide. I’m sure given time and perseverance it’s possible without a backing formal education, but it’s rather challenging.
I made a gossiping membership and failure detection library/application written in Rust. This was async/await keyword so some of the combinators were pretty nasty. I spent a lot of time reading the serf source code, as well as papers related to gossip. I got a it working pretty decently with some tests even. I stopped working on it when I realized I had no use for it and wouldn’t realistically be able to test it at a scale that matters.
If I started again it would need to be upgraded to the latest Futures version, unsure how much work that would be. More importantly I would need to improve the error handling and do a general hardening of the code to avoid e.g. DDOSing the network.
Oh my god this is a very tough one for me. As someone who always starts a projects but somehow ends up leaving in the middle or unfinished.
So I started a project where I was going to build a news webscraper that would scrape news articles and run those articles through machine learning model that would tag those articles with different defined categories and then output those articles in a form of pdf report with tags and summary.
I managed to finish the scraping part and summarizing the news articles using gensim but never got a chance to implement the tagging feature as I lost the motivation and never finished the project. Now i feel like I should get back and try to finish that project.
Foursquare/Swarm clone based on OpenStreetMap data.
I got hooked up on the “check in” concept, but Foursquare pivoted from gathering venue metadata to straight up tracking people 24/7 and selling that, so I wanted a non-creepy version of that.
What killed it:
Cryptography library written in Rust - https://github.com/libOctavo/octavo
I started it when I was on university, but later lost interest as I started “real work”. Since then a lot of other projects like that popped out and I never got back to it. However as const generics are now more useable I will try to revive it a little.
I started a side project called SimpleBlog. The idea was to auto publish Google Docs as blog posts with one click. I figured there were enough people who didn’t care at all about dealing with arcane web editors with too much configuration and not enough features in their word editor. For example, Substack is all the rage right now, and it still doesn’t have inline code blocks.
I made about a third of an MVP and got lazy and convinced myself no one would use it.
Hah, I had the same idea, and also abandoned it after building an MVP. I never had intentions of monetizing / publicizing, so for me the motivation was having a reasonable offline mobile editor (the Google Docs app) available to write trip journal entries while out in the wilderness.
I dropped the project after I realized I don’t write this kind of content generally, and it’s not due to a lack of software.
If anyone’s interested in a more complete implementation of this idea, Blot may fit the description.
I think at work I have my name on at least two “major” features that I sunk loads of time in, and both basically got frozen cuz we weren’t finding good product market fit. Though at one level it’s kind of unreasonable to expect basically one person to figure out a feature and like coordinate market research and user interviews etc and pull of the technical aspect (our team was very small at the time and everything ended up like this), on another level I know that if I had made just a couple small differences to the plan early on it would have gone out the door.
Huge sunk cost fallacies at play of course, but at one point you are throwing a part of your soul into a project and seeing it not get completed sucks.
(OTOH I tried like 5 different times to get Bazel working somehow with our setup for reduced CI times and last year I finally figured it out, that was really my white whale for a long time)
After years of slacking, I finally set up redundant backups today. I’m not great at starting side projects. ☹️
Nothing fancy but tried to build a web-based platform to evaluate programming assignments in colleges (especially in India where we still create hard-paper practical files). Was in the process to build features like:
Failed due to lack of planning (from my side) and team-effort since it was a major project :(
What I learned: