sled is the first project I’ve created that I’ve put more than 2 years into. Being a database, there are basically infinite ways of scratching different itches that I’m curious to learn more about, as someone interested in distributed systems, concurrency, correctness testing, optimization, caching techniques, etc… But I have actually done a lot of things to protect its role in my own life, which causes the motivation to be there over time.

• I have refused to take any action that would reduce its therapeutic role in my life - I often find comfort in working on it because it is my own, I don’t have many contributors to shepherd. I don’t allow my employers to take any control over it at all. I demand that any contracts I sign in no way impinge on my personal projects.
• I’m absolutely militant about shutting out assholes and preventing them from making any of the work feel like I’m giving them anything.
• I am uncompromising in my messaging. I don’t try to act like some corporate narc, desperate for users. I make jokes that I like, and this pushes away people who I don’t want to interact with.
• I give myself space when there is not motivation. I’m intrinsically motivated by a ton of topics that will lead me back to the project. I don’t force it.
• I extensively use a TUI mind-mapper that I wrote to take notes on papers I read, and visualize how various interests of mine overlap. I’ll often get excited about how much overlap there is between different things I’m interested in.
• I start other projects that can be combined with the main one, but also let me feel like I’m mixing things up and possibly reaching new users.
• I prioritize people in my life over spending time coding. I have a lot of friends who are also super psyched about many of the same subjects as me, and their excitement rubs off on me. Interactions with humans is extremely refreshing and causes my motivation to really increase, and I can channel it into my project.
• I created a discord channel where people are often asking me what the current direction is, which sometimes I don’t like thinking about, but usually it’s good to think about because it maintains some amount of focus on “the big picture”. The channel is more about talking about the technical things I’m interested in with like-minded folks than the specific project, and I get a lot of the same motivation benefits as when I spend time with friends, although it’s absolutely not a substitute.

Write down a one-sentence mantra that you’d use to describe your current work to a technical friend who is in a hurry and doesn’t have time to receive a full download of your vision for the project. Chisel it down until it can be pronounced in a single breath, then recite it a few times a day, or whenever uncertainty creeps in.

For example, here’s my mantra for one of the projects I’m hacking on:

I’m building a self-hosted API & mobile app for virtualizing a remotely operated cell phone.

This helps me leave by the wayside any features or yak shaving excursions which don’t directly contribute to the technical bottom line.

Viciously limit scope until it’s something you can actually implement. Then dogfood relentlessly. I find that if I can’t have something at least marginally interesting written in a month, I’m not going to stay motivated in a project, let alone design it well.

Develop the minimally useful implemitation (“MVP”) and go from there.

Don’t worry too much about keeping things “extensible” or “oh, we might want to add this…”. In theory, that sounds all great, but in practice you’ll find it’s very hard to actually get things done. Don’t be stupid of course, and do follow general practices which make things easier to edit afterwards when possible, but in general I don’t worry much about it and just do what I want to do now. Otherwise it’s very east to end up in this “technical rabbithole” you seem to have ended at.

Remember that a lot of the times you don’t really know what a good solution is anyway, so all this work might be for naught anyway. Often times you need to actually implement it to really understand the problem (one of the reasons I’m not a big fan of detailed design docs upfront).

If the project is so large that you can’t keep its major features in-mind for the time it will take to bring it to fruition, nor even write it down satisfactorily in case you forget, I’d say it’s prudent to scale down.

Briefly note down everything you want to do. Take the best (or figure out the smallest subset possible that you’d be happy with), work towards that, and forget about the rest. Once you’re happy with it, check your notes again and see if anything brings back the same enthusiasm you have now. If so, great! Pick the next feature and carry on like that. If not, that’s okay - you’ll have new ideas with a solid base to work from and the valuable experience of getting there. No point forcing future you to follow a vision you can’t be certain you’ll still believe.

Of course, this may not work for you or your ideas, but I say this as someone with several similar projects. All with a few false starts, most abandoned, and the current survivor at a great divergence from my plans from years ago. Every so often, I look at old notes for possible features - some of them I still think are great, and the rest I no longer care about, or won’t work well with what I’ve already done, or are otherwise unnecessary, and I’m glad I didn’t waste time thinking about them.

When you’re doing a long term project, things will change over time, for one. So, if your mind changes on something, it should be ok to run with that change, IMO, as long as it doesn’t churn too hard.

My only successful (in that they stayed interesting and coherent to me over time) longer-term projects have been fairly open-ended, like PISC. So I may not be the best person to give advice on this.

But, the bigger an idea is, the harder it is to flesh out, in both construction and consideration.

Perhaps you could start with a prototype that captures the end results you’re looking for, or captures something like them? One example of that that stands out is. @andyc and oil shell, with the python prototyping, but an end result in C++ that he is working towards.

Long-term projects are emotionally exhausting. Especially if they’re at the edge of your abilities. Just look at the burnout rate of PhDs. At the same time, they’re one of the best ways to explore and flesh out one’s interests.

My project is about 2 years old. Not much to show other than a lot of prototypes and half-baked ideas. I continue to clarify the vision about what I’m building but I’m unsure when it will emerge. I’ve given up assuming it will be important and focus on making something beautiful to me. I’m also not a fan of talking about it right now; it’s still a bit fragile as the vision evolves.

I try to work on small components out in the open. This gives me the freedom to rabbit hole for awhile if necessary. It is hard to be exclusively big or little picture on these projects, so you have to inject some way to move between them to let part of your mind rest.

Ultimately it’s a marathon, not a sprint, so half the work feels like figuring out processes that let you work well while getting you closer to your goal.

The following is with the assumption that this is some side-project with potential commercial ambitions.

I think going “deep in a technical rabbit hole” is a problem that almost all side-project programmers face, and I think it’s because fiddling with some technical challenge is within our comfort zone, whereas picking up the phone and asking a stranger to hand over their dollars to use your project (which you probably think is terrible and not worthy of demanding fees) is daunting, and can potentially be a huge blow to your ego.

When a compiler says “whoops, I can’t compile this without the locks of hair from a few dozen yaks”, it makes you grumble.

When a potential customer says “this looks nice, but I don’t care about it enough to pay you for it”, it is truly gut-wrenching.

This was (and is) my challenge too, and I don’t have any handy secret to overcome it. You just need to push through that pain. Keep picking up the phone and hustling until it sucks less, and people say “yeah, that sounds great. Let’s do this.”

Having external accountability (especially from people waiting to send me money) is a huge motivator for me to just ship stuff, regardless of how gross the code is.

By starting simple and staying on top of changing requirements over the years. I learned this largely with my blogging software over the past twenty years. As others have already said, start simple and dogfood the hell out of it.

Another aspect (especially if other people start submitting feature ideas) is to learn to say “No.” It’s your project, so don’t be afraid of rejecting ideas if they don’t match what you want out of it. Give preference to working code over mere ideas.

I have very similar problems, but I have a couple of ideas.

1. Coherence of vision and process of breaking it into sub visions and goals and tasks:

I feel like I know when a vision is coherent and clear and when it’s not. I think it’s the clarity and coherence of the initial vision that allows it to be broken down into also coherent sub-visions, goals and tasks. Goals and tasks must be connected to a vision, even if it’s a sub-vision of the main one.

If you picture that like a tree with your main vision at the top, at any level in that tree you should be able to trace up to reach a clear explanation of why you’re doing something and you should have a clear picture of everything you currentl know that’s required to achieve the vision at the top.

2. Maintaining vision while working, distractions

For goals and tasks you must have a feel for what is the least work you can do now in order to achieve that thing well. If you’re losing coherence at some point such that you go down a rabbit hole it means either

• For the task or goal you’re working on you don’t have a related coherent vision (the vision is not defined well, you have a task/goal that’s not connected to a vision)
• The thing you’re doing is very hard and you’re not recognising how hard it is - this is disconcerting and it can be difficult to keep working on something which explodes in complexity (your vision/goal/task tree suddenly becomes huge around one node - it expands faster than you can maintain a coherent tree)
• You’re doing something else, like enjoying NIH engineering or finding interesting problems which aren’t actually critical to your goal/vision - (your goals/tasks have become disconnected from vision as you’re doing them because of enjoyment/procrastination/etc)

Capturing these things for me is 95% just making and refining the initial vision. I feel like if I can’t express the initial vision clearly such that anyone can read it, it’s not coherent to me such that I could start work on it. If something reaches a threshhold of coherence + motivation to start on it, then I will do the next level of breakdown, but often I start work/coding before I have completed breaking down the vision and goals to a reasonable level and therefore there’s high risk coherence and therefore motivation can be lost while I’m working.

Hi! I too keep one long term project at hand. I’ve had several that have run their course.

I’ve tried different things and the one I like currently is to keep a use cases document in markdown. Here I write out user flows describing what I want the software to do and how I interact with it.

I then have a design document where I jot down potential problems and solutions and any resource estimates.

I have another document where I jot down all the things I learned while doing the project. My current project is in C++ and that’s what I’ve been jotting down.

I tend to end up with document fragmentation: I start a lot of documents, forget about them, repeat ideas and notes and end up with a horrible clutter. So I restrict myself to large documents. I use a VS Code plugin called Markdown TOC which automatically generates inline TOCs for each document.

My issue is largely motivation. I just took a long walk after I woke up today to see if that will jog something within me. I spend most of my time indoors and am rather sedentary (work from home). I’m relocating to the Bay Area soon and will have a real-life job (non-startup) so that may spark some motivation within me.

Back on topic, I take notes of all my ideas and have them grouped into their own folders and it certainly helps when I look back on a stagnant project and see future features and whatnot.

Like others have said, develop an MVP and go from there. You could also split up the long-term project into sections or phases. Each item in an phase then becomes a checklist item to complete.