G’day Lobsters :)
For a while I’ve been thinking through and exploring concepts for a fairly large and long-term project. At the moment it’s just me, and I’m planning to start it out as a hobby project and see where it goes from there.
So far though I’ve had a few false starts. What tends to happen is that I develop a vision for where I want to take the project, but soon after I start coding I end up deep in a technical rabbit hole and lose sight of the vision that was so clear.
I’ve tried collecting these thoughts in mind-maps, detailed roadmaps, and by writing docs that outline (as articulately as I can) the problem this project will be solving and how it will go about doing that. But none of these feel like they capture that initial goal with the same clarity, and often I get restless feeling like putting time into these documents would be better spent elsewhere.
My ask is then, how do you capture your vision especially for large and long-term projects? And how do you make sure to stay focused and on track moving towards the same (or an adapted) vision and goal?
sled is the first project I’ve created that I’ve put more than 2 years into. Being a database, there are basically infinite ways of scratching different itches that I’m curious to learn more about, as someone interested in distributed systems, concurrency, correctness testing, optimization, caching techniques, etc… But I have actually done a lot of things to protect its role in my own life, which causes the motivation to be there over time.
Write down a one-sentence mantra that you’d use to describe your current work to a technical friend who is in a hurry and doesn’t have time to receive a full download of your vision for the project. Chisel it down until it can be pronounced in a single breath, then recite it a few times a day, or whenever uncertainty creeps in.
For example, here’s my mantra for one of the projects I’m hacking on:
This helps me leave by the wayside any features or yak shaving excursions which don’t directly contribute to the technical bottom line.
This is a great advice. A mantra is gold worth and seems like a great motivation utility.
Viciously limit scope until it’s something you can actually implement. Then dogfood relentlessly. I find that if I can’t have something at least marginally interesting written in a month, I’m not going to stay motivated in a project, let alone design it well.
Develop the minimally useful implemitation (“MVP”) and go from there.
Don’t worry too much about keeping things “extensible” or “oh, we might want to add this…”. In theory, that sounds all great, but in practice you’ll find it’s very hard to actually get things done. Don’t be stupid of course, and do follow general practices which make things easier to edit afterwards when possible, but in general I don’t worry much about it and just do what I want to do now. Otherwise it’s very east to end up in this “technical rabbithole” you seem to have ended at.
Remember that a lot of the times you don’t really know what a good solution is anyway, so all this work might be for naught anyway. Often times you need to actually implement it to really understand the problem (one of the reasons I’m not a big fan of detailed design docs upfront).
I don’t think I’ve ever even halfway completed any project that even had a vision or overarching goal, so this is like 90% of my answer right there.
If you have certain goals I think it does make sense though to reevaluate them every time you make a bigger design decision and think a little if your MVP will make your long-term goal harder to achieve.
If the project is so large that you can’t keep its major features in-mind for the time it will take to bring it to fruition, nor even write it down satisfactorily in case you forget, I’d say it’s prudent to scale down.
Briefly note down everything you want to do. Take the best (or figure out the smallest subset possible that you’d be happy with), work towards that, and forget about the rest. Once you’re happy with it, check your notes again and see if anything brings back the same enthusiasm you have now. If so, great! Pick the next feature and carry on like that. If not, that’s okay - you’ll have new ideas with a solid base to work from and the valuable experience of getting there. No point forcing future you to follow a vision you can’t be certain you’ll still believe.
Of course, this may not work for you or your ideas, but I say this as someone with several similar projects. All with a few false starts, most abandoned, and the current survivor at a great divergence from my plans from years ago. Every so often, I look at old notes for possible features - some of them I still think are great, and the rest I no longer care about, or won’t work well with what I’ve already done, or are otherwise unnecessary, and I’m glad I didn’t waste time thinking about them.
When you’re doing a long term project, things will change over time, for one. So, if your mind changes on something, it should be ok to run with that change, IMO, as long as it doesn’t churn too hard.
My only successful (in that they stayed interesting and coherent to me over time) longer-term projects have been fairly open-ended, like PISC. So I may not be the best person to give advice on this.
But, the bigger an idea is, the harder it is to flesh out, in both construction and consideration.
Perhaps you could start with a prototype that captures the end results you’re looking for, or captures something like them? One example of that that stands out is. @andyc and oil shell, with the python prototyping, but an end result in C++ that he is working towards.
Long-term projects are emotionally exhausting. Especially if they’re at the edge of your abilities. Just look at the burnout rate of PhDs. At the same time, they’re one of the best ways to explore and flesh out one’s interests.
My project is about 2 years old. Not much to show other than a lot of prototypes and half-baked ideas. I continue to clarify the vision about what I’m building but I’m unsure when it will emerge. I’ve given up assuming it will be important and focus on making something beautiful to me. I’m also not a fan of talking about it right now; it’s still a bit fragile as the vision evolves.
I try to work on small components out in the open. This gives me the freedom to rabbit hole for awhile if necessary. It is hard to be exclusively big or little picture on these projects, so you have to inject some way to move between them to let part of your mind rest.
Ultimately it’s a marathon, not a sprint, so half the work feels like figuring out processes that let you work well while getting you closer to your goal.
The following is with the assumption that this is some side-project with potential commercial ambitions.
I think going “deep in a technical rabbit hole” is a problem that almost all side-project programmers face, and I think it’s because fiddling with some technical challenge is within our comfort zone, whereas picking up the phone and asking a stranger to hand over their dollars to use your project (which you probably think is terrible and not worthy of demanding fees) is daunting, and can potentially be a huge blow to your ego.
When a compiler says “whoops, I can’t compile this without the locks of hair from a few dozen yaks”, it makes you grumble.
When a potential customer says “this looks nice, but I don’t care about it enough to pay you for it”, it is truly gut-wrenching.
This was (and is) my challenge too, and I don’t have any handy secret to overcome it. You just need to push through that pain. Keep picking up the phone and hustling until it sucks less, and people say “yeah, that sounds great. Let’s do this.”
Having external accountability (especially from people waiting to send me money) is a huge motivator for me to just ship stuff, regardless of how gross the code is.
By starting simple and staying on top of changing requirements over the years. I learned this largely with my blogging software over the past twenty years. As others have already said, start simple and dogfood the hell out of it.
Another aspect (especially if other people start submitting feature ideas) is to learn to say “No.” It’s your project, so don’t be afraid of rejecting ideas if they don’t match what you want out of it. Give preference to working code over mere ideas.
I have very similar problems, but I have a couple of ideas.
I feel like I know when a vision is coherent and clear and when it’s not. I think it’s the clarity and coherence of the initial vision that allows it to be broken down into also coherent sub-visions, goals and tasks. Goals and tasks must be connected to a vision, even if it’s a sub-vision of the main one.
If you picture that like a tree with your main vision at the top, at any level in that tree you should be able to trace up to reach a clear explanation of why you’re doing something and you should have a clear picture of everything you currentl know that’s required to achieve the vision at the top.
For goals and tasks you must have a feel for what is the least work you can do now in order to achieve that thing well. If you’re losing coherence at some point such that you go down a rabbit hole it means either
Capturing these things for me is 95% just making and refining the initial vision. I feel like if I can’t express the initial vision clearly such that anyone can read it, it’s not coherent to me such that I could start work on it. If something reaches a threshhold of coherence + motivation to start on it, then I will do the next level of breakdown, but often I start work/coding before I have completed breaking down the vision and goals to a reasonable level and therefore there’s high risk coherence and therefore motivation can be lost while I’m working.
Hi! I too keep one long term project at hand. I’ve had several that have run their course.
I’ve tried different things and the one I like currently is to keep a use cases document in markdown. Here I write out user flows describing what I want the software to do and how I interact with it.
I then have a design document where I jot down potential problems and solutions and any resource estimates.
I have another document where I jot down all the things I learned while doing the project. My current project is in C++ and that’s what I’ve been jotting down.
I tend to end up with document fragmentation: I start a lot of documents, forget about them, repeat ideas and notes and end up with a horrible clutter. So I restrict myself to large documents. I use a VS Code plugin called Markdown TOC which automatically generates inline TOCs for each document.
My issue is largely motivation. I just took a long walk after I woke up today to see if that will jog something within me. I spend most of my time indoors and am rather sedentary (work from home). I’m relocating to the Bay Area soon and will have a real-life job (non-startup) so that may spark some motivation within me.
Back on topic, I take notes of all my ideas and have them grouped into their own folders and it certainly helps when I look back on a stagnant project and see future features and whatnot.
Like others have said, develop an MVP and go from there. You could also split up the long-term project into sections or phases. Each item in an phase then becomes a checklist item to complete.