1. 30

I’m running into the problem where I’m working on my own sizeable project while also working a day job. Both require large amounts of programming and keeping a decent amount of data in my head regarding state, rules, other systems, reasons for doing new stuff, what needs to be done and why components of the system are like they are in the first place, etc.

The problem is that when I push one set of data into my head and work with it I also pushes varying amounts of the other set of data out. I assume this is by necessity since maybe my brain space or working memory or whatever is too small to accommodate all of the information at once. This results in me having to repeatedly go over work, comments, tests, design decisions and everything else which as you could probably guess is tiring. I’m writing this now because I had completely forgotten something I’d done 3-4 weeks ago and was about to do redo it until I pulled the code and checked why there was a conflict.

This is the first time I’ve ever really noticed a limit on the capabilities of my brain. Hopefully someone else has come up with a solution.

TLDR: My brain is limited, What tips do you have for holding more stuff in it and stop having to re-digest information. Or at least reduce having to do it so often. It’s happening weekly and it almost feels like pain at this point.

  1.  

  2. 36

    I think that developers (well most people actually but developers are the loudest about this) spend way too much time trying to think and do things at the same time.

    What I’ve done for the past couple years of my life to great effect is to go really deep into Getting Things Done. There’s a lot of components to it, but the one most relevant here is to separate thinking from doing.

    When you think about a project, you should write out all the tasks you have to do on this project. And the tasks should be detailed!

    Not “improve the app’s billing page” as a single task, because that’s too vague and requires thinking. It should probably end up down to something of very high granularity (the rule of thumb is “someone vaguely aware of your work should know what you mean”). So “make sure that the billing modal shows an error message on payment failure by checking Stripe response code”. Or “research how to know if we have a payment failure”.

    Brains aren’t for holding info, they’re for thinking. So if you can empty your brain out by writing stuff onto a todo list, onto your calendar, etc, your brain will have more space for the high level stuff that’s a bit harder to get on paper. And if you wrote down “do thing” on the todo-list… well you just have to do the thing afterwards. There’s nothing to think about or redigest if you write out the things in more detail

    (I’d really recommend reading all of Getting Things Done because there’s a lot more to all this process, but getting stuff out of your brain and splitting thinking and doing has been super helpful for me)

    1. 7

      I really liked this quote:

      Brains aren’t for holding info, they’re for thinking.

      I feel like we generally undervalue doing regular “braindumps” of thought state about a given topic.

      1. 2

        It’s hard to do much thinking without holding some info. But there’s much to be said for cache eviction. GTD FTW!

      2. 6

        Agreed. I started doing this while I had a day job / side project as soon as I started noticing those side project tasks accumulating in my head.

        I set up a Phabricator instance for myself and started writing detailed specs for each task, as if I wasn’t the only one working on the project. In the past I’d always go straight from idea to code. But writing things out in plain language, especially when I’m designing an entire system from scratch (not just implementing prescribed features), helped in many ways I never anticipated.

        1. 2

          I also highly recommend GTD. I read the book and started using the process about 6 months ago, and it was really great for productivity and keeping track of projects. It took some discipline to make sure I wrote everything down, but it was great knowing all the information I needed was in one place and really helped to keep track of loose ends and one-off TODO items.

          I was using org-mode to keep track of everything, with the setup from this article.

          Unfortunately when I switched jobs I got out of the habit of using it, but this might be the reminder/motivation I need to get back to using it.

          1. 2

            This is great advice. I used to try to figure out a problem in my head, then code it. I find it easier if I just start typing my thoughts out. Oftentimes my thoughts will need “editing” before a cohesive plan is formed, and this is much easier to do in an editor than in your head. After that it’s just matter of coding up the plan.

            1. 1

              Pretty much everyone has said to write my thoughts down which makes sense so I’ve started a big markdown file for my projects I can just look through for explicit thoughts, notes and as a todo list. I’ll strive to write down the tasks as detailed bite sized chunks but this definitely does not feel natural. Just getting what I need to do into words seems difficult.

              I don’t know if this analogy is correct but for me, programming feels like painting. I have an idea of what I want to make and start with a rough sketch that slowly fills as other ideas bubble up. Once one feature is finished or maybe part way through I usually realize that I can or should do something in a safer/more efficient manner which requires making a function/method somewhere else. Then after I’m done with that I touch up a different part and the part that I was working on previously so it all comes together while making it cleaner and actually work. It feels like painting in a canvas, so to speak; I jump around to where my thoughts direct me.

              Writing these things down feels rigid, like painting from left to right. It is systematic which means that I can easily grok it at future points in time which will solve my problem but being 100% honest I don’t like the idea for my workflow…yet I guess? I’m going to just suck it up and do it because going as I have will probably burn me out.

              Thanks, to everyone in the thread.

            2. 9

              I could write a lot on this, but my approach is:

              • Be able to make progress on projects without keeping “state” in you head. I also second “Getting Things Done”. A concept I like from that book is simply writing things down to get them out of your head. And then figure out “the next action”. Instead of keeping the whole project state in your head, you just have to load “the next action”, and if you do that, you should be confident that you’re making progress over all.
              • I write detailed notes in a Wiki and hyperlink them in order to get things out of my head. The wiki has thousands of pages accumulated over more than a decade.
              • I write comprehensive automated tests. I try not to keep too many “reasons this code works” in my head. The project should be structured in a way so that if I break it, I will know without thinking about it. Once you have confidence in that, you can move forward a lot more aggressively on your projects. You won’t be constantly “loading up state” to convince yourself that a certain change is correct.
              • I put almost every shell command I run in a shell script. I don’t want to recall how to invoke a certain tool or test. Instead I just need to recall where I put the shell snippet that lets me make progress on the current task.

              Here’s a pretty hacky shell script for my current task. I clean these up over time, but they start out raw.

              https://github.com/oilshell/oil/blob/master/mycpp/run.sh

              IMO “automation” is an important part of getting things done, and shell is the best tool for programmers to automate things. It takes very little effort so there’s really no excuse not to do it. You literally save what you typed in a text file, and that’s a shell script :)

              1. 1

                I use Makefiles for that. That also handles the dependencies like build before running a benchmark.

                1. 1

                  Yeah I’ve seen many people use Make as well. The addition of dependencies is an upside, but it seems to cause me to try to model everything too “nicely”, and Make has a lot of deficiencies which make it impossible to be really accurate. So I just stick with the quick and dirty shell scripts, which are easily refactorable and explicit about dependencies (if redundant).

                  Also, I find that the shell language is a lot nicer than Make, e.g. for loops, conditionals, and functions. Make calls shell so you can of course mix them, but that tends to involve a lot of \ at the end of lines, and $$foo, and another annoying things. And Makefiles are harder to trace than shell scripts.

                  I have experimented with Makefiles calling shell actions in a different file, which works OK for complex things, but it’s unwieldy for simple things.

                  I do recognize the value of dependencies but I’d like to do it in a different way than Make. Hopefully someday I will address that in Oil …

              2. 6

                This is a tough question. A lot of creative activities are like this - you think on it a lot and then it forms in your mind and if you are lucky, you can put it into a physical form (like code) before it collapses. This becomes especially challenging if you are switching contexts a lot.

                I write up comments in the code, I write up stuff as documentation or tutorials, I make blog posts. I find writing stuff down in whatever form is beneficial in the long term. It acts as a memory aid and consolidates thoughts. It’s annoying because it reduces the time available for “real work” in the short run but is invaluable in the longer run.

                In response to one particular point you bring up:

                I’m writing this now because I had completely forgotten something I’d done 3-4 weeks ago and was about to do redo it until I pulled the code and checked why there was a conflict.

                I keep a features checklist in a text file and check them off.

                There have been occasions when I’ve created a new features checklist document by mistake.

                I believe the problem you outline is a tough one.

                1. 6

                  My only advice mirrors that of @kghose.

                  Whenever I work on projects nowadays, one of the first things I do is write my roadmap and start filing issues, and then each issue gets a feature branch that is always synced home whenever I’m done working on it for the day. If something is complicated enough to need it (usually because I need to explain design constraints or decisions), block comments or wiki entries cover it.

                  The plus side of all of this is that I’m practicing like I play: at work, that is my baseline for the way I build and ship products, and that discipline pays off.

                  Trying to keep it all in my head is just a recipe for disappointment, since my head is small and frankly I’d rather spend that space on sunny days with friends, walks with my dog, and dating.

                  1. 4

                    I mean you can always not keep entire projects in your head. I work on like a few million plus line projects and I definitely do NOT keep this stuff in my head. I occasionally will look at my work history in a given project to jog my memory.

                    Brains are limited, use more tools for organization xmind for mind maps for example, use your version control to its full potential and communicate with your peers regularly. It sounds like you need to leverage general organization of like what’s in your head. Paper don’t forget. Document stuff. Do project management for your project. I think even if you could, keeping it all in your head is bad for mental health.

                    1. 4

                      The obvious thing everybody suggests is to externalize your knowledge. If you want to keep it in your head, you could try spaced repetition (e.g. Anki).

                      1. 2

                        The art of memory forums and wiki has a lot of techniques as well. I think it’s useful to improve your memory but I still think its a bad idea to shove it all in your brains even if you can.

                        https://artofmemory.com/

                      2. 4

                        I ran into this exact problem when switching between clients while freelancing. My solution was to swap to disk :)

                        https://orgmode.org/

                        http://www.orgzly.com/

                        https://syncthing.net/

                        Orgmode to organise everything - project notes, debugging processes, presentations (even does the slides with org-present), time tracking.

                        Orgzly to give me Orgmode on my Android phone.

                        Syncthing to seamlessly synchronise my Orgmode files across Android, Linux, FreeBSD, OSX, and MS Windows.

                        1. 3

                          I’ll echo all of the advice here. Basically, operating systems work by having a memory manager page in and out memory pages when needed. You can page out thoughts (write them down) and then page in (literally read the notes). It’s expensive, yes, but once you get good at it, it becomes second nature, and frees up an incredible amount of mental cycles for other tasks.

                          1. 2

                            The resounding advice in this thread is write things down. Some do it in notepad-like things (literally notepad, org-mode, general text editor, etc.) but I think treating your project like any other work project is useful, too. That is, establish an work tracker and make tickets for everything that comes to mind that needs to be done. Get yourself into the habit of searching that work tracker when you have an idea, both for your personal project and your work project. The tooling may differ but the workflow should not:

                            Idea --> Search --> Create new ticket
                                           \--> Annotate existing ticket with "I had this idea again"
                            

                            Another important task is to leave breadcrumbs: that means noting what you were working on, leaving tests is a broken state that can be easily fixed and have some notes on what you were looking at when you had to stop. This is a noted inter-day strategy for long-running work projects, so apply it to your personal project, too. This also means that you should write READMEs and other instructions. You’re not necessarily writing these for someone else, rather, but for that someone else that is you six months from now.

                            At a previous job, we had 20 repositories that comprised the product we were working on. While some of them shared codebases – literally forks that merited a separate release process – others were “rarely touched” parts of the under-development codebase. We could go months without touching it. That README that we spent 45 minutes pair-writing was indispensable to folks who needed to make changes, including those who were its original authors. Onboarding should be continuous: people who have never seen the codebase before are at an advantage because nothing has changed since the last time they looked at it.

                            1. 2

                              I use Anki for this. I started a new job around Thanksgiving and there was a lot to learn. I currently have 119 notes, 900 cards in the work deck. I have cards for systems, tools, teams, processes, layers, and coworkers. I have stopped having the “I know I learned how to A/why we B/who works on C like two months ago…” feeling.

                              I also have a log, just a text file I keep open in vim all day. I’ll jot done meeting notes, identifiers, topics, etc. or paste terminal sessions when I puzzle something out. Very handy to be able to grep.

                              1. 1

                                I have an action item in my bullet journal for this, I just need to spend the time to do it.

                              2. 2

                                In addition to the planning and tracking mentioned by others, I would keep a tagged (like hashtags) journal, and review it on a daily and weekly basis.

                                I find that externalizing my thought process helps me.pick up where I left off a lot easier, and it gives you a record of what you’ve done so far.

                                Also, make sure that you’re taking care of yourself physically (if you aren’t already doing that) Exercise and good food can help a lot there.

                                1. 1

                                  I’ve recently started trying use topic pages in a bullet journal with a detailed list of tasks. As I figure out that one task needs to be broken down further, I add more tasks. There are a few things I need to tune, like how to group tasks together under one banner, but for my own projects, this is working pretty well so far.

                                  1. 1

                                    There’s no point in trying to hold more and more information in your head. You can’t.

                                    I completely agree with andyc’s points. Write stuff down.

                                    Sit down, list the things that need to get done for your projects, ruthlessly prioritise them, and track things as you go.

                                    What I like to do (both for personal projects as well as my day job) is to keep a work log. It’s just a simple text file where I list the stuff I worked on, any blockers, etc.

                                    When you have a work log, you can quickly scan over the most recent entries to recall what you were working on.

                                    1. 1

                                      It sounds like you need some project management there. I have no idea if you’re using any kind of project management system at day job or just building some huge thing by yourself. If you already have one at work, make sure you’re using it well, put more detail into stories/cards/whatever it uses. If not, maybe you need to add one. Maybe there’s already an org that has something you could use, or maybe you create your own. This stuff often doesn’t get a lot of love from programmers, but if you have a large multi-person project without it, you’re going to run into stuff like that.

                                      Same thing on your personal project. But on personal, you don’t need to fire up Jira or Rally or anything like that. Just a simple spreadsheet with a detailed description of tasks to do, status, and date entered/done can help a ton. That’s something you can flip through to see if you’ve already thought of something or already done it, and a place to put an order of importance on things to do.

                                      1. 1

                                        I would ask for a 4 days work week, then you have 2-3 days for your project, depends on where your priorities are but would make working on both much easier IMHO

                                        1. 1

                                          Yes, I’m experiencing the same, and it’s tiring. I also tend to forget quite a few day-to-day things outside of development, when focusing on a project for a while.

                                          1. 1

                                            If you have an Android phone I highly recommend Habits for day-to-day things / developing good habits. You get tight control over the intervals for each habit (do dishes every 2nd day, make dinner 6 out of 7 days, etc) and visibility on how you’re doing over time.

                                          2. 0

                                            Have you tried nootropic drugs like caffeine or modafinil? I hear conflicting things about whether or not they actually work, and I suspect it’s different for different people, but it might be worth a shot if you feel like there’s a biological limitation of your brain preventing you from achieving your goals. You might try looking at: https://www.gwern.net/Nootropics for instance to learn more about them.

                                            1. 4

                                              I guess that would be a solution but I’m not entirely comfortable relying on drugs to get work done. I run into a tolerance issue with caffeine and operate better without it. I have not taken modafinil but I’ll look into gwern. Thanks.

                                              1. 2

                                                Just so you know this isn’t the kind of advice a parent would approve of, and depending on your brains could ruin them.

                                                1. 1

                                                  Came here to say this. I decided long ago that while I might be able to do more with certain chemicals, I was not willing to go any more extreme than caffeine and alcohol, and even those in careful moderation.

                                                  1. 1

                                                    Yes I have personally found that it’s easy to go to far even with caffeine and have mostly eliminated it from my diet. I’m more reliably productive and in my humble opinion reliable productivity provides greater yields in the long term than unreliable spikes of productivity.

                                                2. 1

                                                  Other nootropics of note include: eating well, exercising, and meditating! Gwern’s site is very interesting though if you’ve got time for some reading.

                                                3. 2

                                                  This is one of those bits of advice that might solve the stated problem but doesn’t address the underlying problem of a herculean effort invested in a bad strategy. Putting more effort in does nothing to address the bad strategy :P.

                                                  1. 1

                                                    Modafinil is not very useful. It mostly makes you not want to sleep, and I use it while driving very tired as a safety measure. Short-term though, it might not be a bad idea to try out.

                                                    Caffeine is similar in this regard, although in general it doesn’t have any strong effects on me, which seems to be a YMMV thing–others report differently.

                                                    I have no idea what to chalk up my periods of improved mood and/or efficiency to, but they don’t have a clear relation to chemicals like these.