1. 39

I expanded on a comment I left on this entry into a blog post, and I’m hoping some of the experienced and wise lobsters can provide some insight.

  1.  

  2. 21

    In my experience, the developers who make it past the “intermediate” level take the time to learn “deeper” things instead of the latest trending library, API, or “practice”. I don’t have a good definition of “deeper” but it can include algorithms, architecture/systems, distribution, focusing on a niche like security or performance, etc.

    I see a lot of developers with several years of experience tasked with similar things as me (<2 years). The ones I know that aren’t and have improved their career all have the above in common.

    1. 24

      In a way, tech news sites are actively harmful to those pursuing mastery, because they continually push beginner-intermediate stuff at the expense of deeper, harder material. They emphasize quantity over quality, and provide a facade of ‘technical relevance,’ which is self-referentially defined as, “what everyone else is talking about,” rather than more durable skills.

      “But, HN sometimes links deeply technical stuff!” you contend. Of course it does. It’s also buried amongst the sea of largely-skippable-marketing-disguised-as-technical content. Is your time worth digging through all that?

      1. 13

        Ha, I am in complete agreement with you. I quit HN, Twitter, and reddit last month for this very reason. Lobsters (for now) is more than enough :-)

        Not wasting my time with poorly written Medium articles anymore feels nice.

        1. 9

          Medium is one of the worst offenders. Lots of articles relevant to tech/programming are basically the person patting themselves on the back: “How I made X into a success with over 1,000 GitHub stars.”

          Such, erm, self-aggrandizement (if you get my gist) is tedious to read for everyone involved. Instead of writing material to help the reader, they’re writing the article to help themselves. Worse, people lap up this brand PR as if it’s genuinely helpful.

          1. 4

            The great thing about Medium is that it’s an effective signal that I shouldn’t waste my time.

            If all the content was hosted individually I’d lose that filter.

        2. 4

          If we keep downvoting those who try to keep this site on-topic, we will soon have the same problem.

          1. 4

            I don’t really agree with that. If a person jumps onto every technology and tool bandwagon they see mentioned, then yes, it will be a problem. But who does that?

            I read HN and Lobsters a few times every day, and 90% of the articles (on both sites, FWIW) I just ignore because I can tell from the headline they’re about some technology I have no interest in or some event that I don’t care about.

            On the contrary, I’d say reading the sites makes me a better developer because while I might not dump everything and move to Swift or Go or Haskell tomorrow, seeing what’s going on with them can make me more aware of the general software development landscape.

            1. 1

              Mindshare is precious. It is a choice to regard the things that people discuss as ‘relevant.’ I don’t; I’m advocating that position.

            2. 5

              HN erases scores on comments, which is something they introduced a few years ago so they could “rank ban” contributors deemed controversial without it being obvious, and so moderators could tweak rankings of content favorable/disfavorable to YC startups to the same effect. This actually encourages people to game the system more, insofar as prestige comes from (a) having top comment, and (b) having submissions that get a lot of points. For reasons that you described, this pushes people toward non-technical news and bike-shed topics that drive out the sort of things that drive out the harder, denser material.

              The problem with Hacker News isn’t just community decay, one should note. When you consider the salaries of people like Paul Graham, who used to spend several hours per day on it, it’s probably the most expensively moderated web forum in history. HN moderation takes an active role in pursuing the marketing and business interests of Y Combinator and its companies. It’s certainly unethical, and I’m disgusted that such a cool concept (I’m related to the guy who came up with it) is used as the name of that scummy business operation.

              I don’t see this danger for Lobsters, because it only appeals to technical people. Lobsters fame has no business value and there’s no economic incentive to game it. I don’t think the points really matter. I’d rather get +4 for a post on a technical topic than +25 for a throwaway joke.

              I agree with your larger point, too. Tech news aggregators feed the monkey, and it’s hard to develop expertise without focus. The difference is that Lobsters has less intellectual junk food, and it also dries up whereas Hacker News and Reddit serve up a buffet of distractions.

            3. 8

              One way to go about this is to put emphasis on durable skills, so that they can be built upon at all before being made obsolete by a trend. Lower level stuff tends to change far slower, and can often be squeezed for more value over time per unit of time spent learning.

              1. 9

                IMO, software design skills are universally applicable and don’t change very quickly.

                You may realize them differently in different languages and paradigms, but the basis of them forms a foundational aesthetic that you can trust to guide you to writing your best code as much as possible, further improving your practice.

                1. 4

                  I agree, but software design skills are also hard to demonstrate. So, while those skills are durable, the game of getting credit for having them is not always straightforward.

                  Front-end designers have it easier insofar as average people actually know what a good UI looks like. They might not be able to create them (someone is creating bad UIs, after all) but they can tell beautiful work from ugly, sloppy, crappy work. They can build portfolios, rather than just hoping to find someone of similar intelligence and tastes who can vouch for them.

                  With software, you have subjectivity but it’s also nearly impossible for a non-programmer to tell good design from bad. Ultimately, shots are often called by non-technical managers who aren’t qualified to be making them, and this is where the constantly shifting fads come in.

                  1. 6

                    Ultimately, shots are often called by non-technical managers who aren’t qualified to be making them

                    There are firms that value engineering expertise, it is just up to the developer to find them.

            4. 14

              I’d add that the most important thing is to try things you’re not comfortable with.

              There’s this pit of comfort that engineers tend to fall into where they don’t really care about the technologies any more, and they get stuck in old technologies because they are afraid to try new things that may feel a bit uncomfortable at first.

              1. 2

                Risk and failure are important modes for learning - but they can have costs - fear of failure is the enemy of the life long learner.

                In BMX coaching my riders take lots of risks - but as the coach I need to ensure that those risks don’t end in injury - as the negative effects of injury defeat the benefits of practice. So I try to ensure that risks taken are pushing the development of the riders without putting them at unneccessary risk.

              2. 13

                Although I definitely don’t consider myself wise or an intermediate programmer (~2 years of experience in industry qualifies me as a novice in my opinion), I think about deliberate practice’s relationship to programming often. When I read Peak a few months ago, I noticed that Ericsson felt that most professionals hadn’t figured this out. I remember him distinguishing between brain surgeons and radiologists for example - surgeons improved over their careers whereas radiologists (used as a representative for most other types of doctors) actually got worse [1]. This also matches my observations of the lawyers I know. Each hit a certain level of competency a few years into their career and plateaued and potentially even degraded. This means we have to be discerning when looking for examples from other professions.

                In my own search for people who think about this in a programming-relevant way, I’ve found a few helpful or at least thought-provoking resources:
                - Cal Newport, a tenured Computer Science professor at Georgetown writes about how he applies deliberate practice on his blog and applied Ericsson’s theoretical framework to modern knowledge work in his book Deep Work. More discerning readers will notice he mentions rubber-duck proving with his dog. From Cal, I’ve adopted the practice of blocking out time where I’m working on one difficult but focused task and ignoring distractions. In an ideal world, this includes all communications and the internet as much as possible. In reality, the open office environment doesn’t always lend itself to hours of distraction-less work, but I do my best.
                - This response to a Quora question about tips for advanced writers feels relevant to programming as well. In particular, this quote [2] resonated with me. Thinking about the programming equivalent of rewriting, (refactoring?) makes me wonder whether those who focus on algorithms (interview-like) problems or even coding “katas” as “deliberate practice” are missing out on the practice required to master programming in the large. Clearly, algorithms and other types of clever programming require deliberate practice and skill, but I question whether these skills transfer to architecting a large system, designing an API, or innovating on a legacy codebase. As I’ve gone from noob to novice (first I knew nothing, now I know enough to know how much I don’t know), I’ve become much less interested in excelling at programming in the small and much more interested in excelling at programming in the large, but deliberate strategies for the latter have eluded me thus far. I suspect that strategies for the latter set of skills will leverage or at least acknowledge the iterative nature of these activities.
                - Mastery by Robert Greene provides a set of case studies and an (opinionated) framework for how one goes about become a master in a discipline. Worth it for the historical anecdotes alone, this book outlines how to, over the course of an entire career, internalize the practice of a discipline until it becomes a part of you. I’ve included some choice quotes below [3]. I recommend ignoring the teleological stuff and focusing on how well Greene understands Ericsson’s notion of mental representations and single-tasking deep focus being the key to mastery. From Mastery, I gained a somewhat corny but useful perspective on my work. When I’m not cursing at my keyboard, lamenting the mess we’re in, I view programming as a long-term, fulfilling activity which I’d like to pursue for a long time. Mastery also led me to eschew jumping from one trendy framework to another in favor of trying to build an understanding of what’s come before in our discipline.

                A few questions that I’m left with even after reading through the above materials and comments on this post:
                - To what degree is separating research from programming useful? I’ve recently experimented with writing down I things I would’ve looked up on StackOverflow previously and continuing programming. I later go back and look up questions on StackOverflow in batch and retroactively apply what I find to my work. This feels like a good way to further narrow the scope of my practice, but I’m curious to hear others' thoughts.
                - Where do non-programming activities fit into deliberate practice for programmers? For example, reviewing code and writing design docs both tax my intellect and my working memory when done with full attention. On the other hand, I often question whether it’s possible to build new mental representations when doing design or whether designing draws purely from already developed knowledge representations. Similarly, I question whether reviewing code as typically done purely consists of applying existing pattern recognition brain modules to someone else’s code.

                [1] I just discovered this Harvard Business Review article which confirms that he drew the distinction between typical radiologists and brain surgeons.

                [2]

                The HUGE difference between everyday writing that everybody does and serious writing is the proportion that is re-writing. I’d estimate that for non-writers, rewriting accounts for maybe 10-20% of their writing.

                For serious writers, it accounts for anywhere between 50-90% depending on how critical the particular piece is. This Quora answer is not very critical for me, so I’d say it’ll hit 50% by the time I am done. There are single paragraphs in my book though that took 5 minutes to write down initially, and then cost me hours to whip into shape, so that’s like 99% rewriting. For my for-pay work, I probably average about 75%. For my own blog, I am erratic. Some pieces hit book-like 99% levels. Other pieces are at 70%. I don’t think I’ve ever hit more than 65% on Quora.

                People often ask me how I am so prolific. To be blunt, that’s so easy for me, it is not much harder than just breathing. But rewriting is hard. It is torture. Since one measure of rewriting progress is words eliminated, I often joke that I write for free but charge for eliminating words.

                But rewriting is the only kind of writing that counts. If you aren’t rewriting, you aren’t developing as a writer.

                When you hit 10,000 hours of rewriting, you’ll be a skilled writer or a skilled thinker with the written word. If you want to be both, it’ll take you 20,000 hours.

                [3]

                All of us have access to a higher form of intelligence, one that can allow us to see more of the world, to anticipate trends, to respond with speed and accuracy to any circumstance. This intelligence is cultivated by deply immersing ourselves in a field of study and staying true to our inclinations, no matter how unconventional our approach might seem to other. Through such intense immersion over many years we come to internalize and gain an intuitive feel with the rational processes, we expand our minds to the outer limits of our potential and are able to see into the secret core of life itself. We then come to have powers that approximate the instinctive force and speed of animals, but with the added reach that our human consciousness brings us. This power is what our brains are designed to attain, and we will naturally led to this type of intelligence if we follow our inclinations to their ultimate ends.

                &

                You must avoid at all cost the idea that you can manage learning several skills at a time. You need to develop your powers of concentration, and understand that trying to multitask will be the death of the process.

                1. 3

                  I’d say that the analogy to rewriting is adding features and refactoring. Get some practice with building a dead-simple prototype of something, then add features to it until parts of the architecture start feeling getting creaky, then refactor appropriately. All of the little decisions involved in that are important to practice - how to get an initial prototype of something small into production as fast as possible by not over-designing it, how to decide when the “creakiness” and future plans of an app justify refactoring, how to create an improved architecture for the new size of the problem and move the app to it smoothly. Not to mention designing test suites that help instead of hindering refactoring and knowing the realistic scale of the app and what issues are and aren’t a concern at that scale.

                2. [Comment removed by author]

                  1. 4

                    Thank you for your comment. I’ll be sure to read The Pragmatic Programmer as my next programming book. The idea of reading classic SF as a way to explore problem solving processes is outside-the-box and something I’m going to try.

                    I’d love to take you up on the virtual mentor offer! I didn’t find any contact information in your profile, but you can reach me at eric@ericdykstra.me

                  2. 9

                    I think the musician analogy is (mostly) a red herring. Much of the difference between an intermediate and master player is mechanical. It’s about muscles, nerves, muscle memory, hearing, maintaining the instrument, etc. There is certainly a very large mental component to it, but novel thought and creative problem solving is not a big part.

                    If an activity that require a lot of physical and mental discipline but relatively little creativity, then I think you can learn a lot from the techniques used by musicians, athletes, and dancers.

                    But I put programming in the same bucket as writing, architecture, and composing music. There is virtually no physical component to these tasks. The mental components require discipline, but critically, they also require coming up with novel solutions to complex problems. I don’t think there’s an equivalent to four hours in a batting cage to improve at doing that.

                    What I do think helps are things like:

                    • Work on your self-discipline. A big part of solving a hard problem is simply not giving up. Often, you just have to keep grinding, and anything that increases your ability to push through something when it’s frustrating is good practice. Ideas:

                      • Long distance running or other endurance sports.
                      • Mastering a musical instrument. It won’t make you a great problem solver, but the discpline required to practice scales for two hours will help you focus.
                      • Reading long, complex books.
                      • Crafts that require dedication to finish projects.
                      • Finishing stuff in general.
                    • Work on your self confidence. This ties in to the above point. A complex problem that doesn’t already have a ready-made solution is scary! It means no one has solved it yet, and there might not be anyone you can rely on. Maybe it’s too hard for you. When you’re hip deep in it, you need to have faith that you are smart enough to get through it. Ideas:

                      • There’s a whole world of material around improving your confidence.
                    • Get better at breaking problems down. The way to solve a big problem is to turn it into a bunch of little problems and solve them. This is, I think, one of the most fundamental skills of creative work. I don’t know much about how to improve it, but I definitely see that novices struggle with it.

                    • Grow your solution toolbox. Solving a complex problem often involves combining several pieces of other solutions into a whole. The more pieces and parts you’re familiar with, the better you’re able to fit them together. Ideas:

                      • This is another one where experience really comes into play. Every program you write is a collection of solutions you’ll be able to pull from later.
                      • Programming challenges and contests. They are artificially small, so they don’t help with discipline, but they do give you small solutions you can combine into bigger ones.
                      • Read other people’s code.
                      • Programming books, especially algorithms, data structures, and other fundamentals like that.
                    • Get better at analogies. I find problem solving very often involves translating your problem into something analogous, and then using an existing solution for that problem. This is why you so often hear stuff like, “Ah, yes, if we think of X as a graph, then we can use algorithm Y.” Sort of a lateral thinking thing.

                      My impression is that this aspect is underappreciated by people who say they “want to be a better programmer”. They tend to want to just beeline straight to “awesome programmer” and aren’t comfortable with things that feel fuzzy or indirect. Even thinking of programming as a linear skill that you can just level up belies an overly straightline mindset.

                      Often, the best way to get from problem to solution is not a straight line between the two. You may need to recast your problem in a different light, find a solution to that, and then remap the solution back to your original problem space.

                      Doing this well gives you those moments of inspiration where all of a sudden a hard problem becomes trivial. But it can be frustrating since you can’t predict when those happen. Self-confidence helps. Self-discipline can hinder this. Often, for this to work, you need to stop grinding, take a step back, and let your subconscious stumble on to some association.

                      Ideas:

                      • Generalize yourself. Take in interests outside of your domain, and outside of programming. Spend time with people who don’t code. Stuff as much variety into your brain as you can.
                      • Read fiction and poetry. Metaphor is a huge component of creative writing and it trains your brain to see X as if it were a Y, which is exactly the skill you’re working on. Poets are masters of this.
                      • Watch/read comedy. A lot of humor turns on taking a scene and then realizing you were misinterpreting as X when it’s really a Y. Again, that kind of lateral perspective shift is what you’re going for.
                      • Try writing fiction, comedy, or poetry.
                      • Turn down the distractions. This skill tends to be a bottom-up mental process. You can’t force yourself to do it. Instead, it sort of arises spontaneously when your brain isn’t otherwise occupied. But now that we have smartphones, your brain is always occupied.

                        Tone that down. Stare at your phone less. Spend more time walking, staring off into space, or doing physical activities that don’t require mental effort. Mow the lawn. Do some knitting. Clean the house. Whatever frees up CPU cycles for all of your background mental threads.

                    1. 2

                      This is excellent insight into problem solving. Thank you for writing this.

                    2. 5

                      The only thing that enables true improvement is learning new concepts and putting them into practice. Reading papers and books, watching talks, and so on, as well as working on side projects and programming challenges, is my method for improvement. I recommend doing Project Euler, kattis, exercism, and so on as your time allows. Do them in many languages. Make sure you use multiple paradigms and levels of abstraction (I like using Haskell, Python, C, and Rust).

                      1. 5

                        “Peak” has been a book on my to-read list since I listened to the author speak on “The Art of Manliness” podcast. I am also intrigued by the concept of deliberate practice. While I agree that implementing something is a wonderful way to really learn something, it is easy for me to get lost in the weeds and it feels somewhat inefficient. One thing I have attempted to do since listening to the podcast is more focused practice. So I have started to apply the kata methodology to my practice. The flow works something like this:

                        1. Study and learn a new algorithm or design pattern.
                        2. Find some examples for implementation.
                        3. Implement that pattern or algorithm every day for a week using the examples.

                        This is a bit repetitious but I think that is the idea that is proposed in “Peak”. Since doing this for the last 4 months, I have noticed some improvement where I more readily recognize places where to apply these patterns and variations of those patterns. But when I set out on this, I realized this would likely be an endeavor of several years before it bore real fruit.

                        As some background - I have been developing software for about 15 years now professionally and am a bit of a productivity nerd.

                        1. 4

                          I like to find concepts (monads, applicatives, maybe some patterns) and try to apply them in a few test programs as a way to learn. I see it as expanding my toolbox.

                          1. 3

                            I’ve been reading Peak https://smile.amazon.com/Peak-Secrets-New-Science-Expertise/dp/1531864880

                            The book talks a lot about the difference between casual, purposeful, and deliberate practice, and the neurological changes that occur with their application.

                            Seems like this could apply in a really big way here.

                            1. 5

                              I consider myself upper-division intermediate and think that it’s extremely hard to get to the next level where one can credibly call oneself an expert. The term is overused in the corporate world and I’m a “subject matter expert” by the low standard out there, but I don’t really consider myself that good. The fact that I’m better than 9x percent of professional programmers isn’t something to congratulate me on; it’s a sad statement about the reality of this industry. Most programmers never get a chance to develop.

                              The first problem is that programming is very specialized. A BA from a solid college or a couple years of professional experience can get you to what I’d call lower-division intermediate (top 15%). A few more solid years (and a lot of extracurricular learning) can get you to upper-division intermediate (top 3%). What then? If you read a lot of AI books, you still might not understand graphics and security and practical database design and so on. You’ll have to chart your own course, and there will always be a lot of cool stuff that you don’t know, and more to learn than you have time to learn.

                              Every field has this branching-out in the upper levels. The difference is that, with professional structure, people who get to that “What now?” point have so much autonomy and respect that they’re trusted to learn on the fly. A math professor who specialized in algebra might not know anything about differential topology, but he has enough of (a) core knowlege, (b) control over his time, and (c ) access to smart people that he can get started, and will be qualified to teach a class on it in a couple of months. Same with lawyers and doctors who decide to pick up another specialty. They have the resources necessary to hit the books. We don’t really have that in programming. In the business world, most people are pushed into management by the time they’re even lower-division intermediate.

                              The truth is that there aren’t a lot of jobs that will provide the kind of work experience that you need if you want to become an expert. You get to a point where, if you want to progress, you have to be very selective in the jobs you take, and that will often mean some combination of (a) willingness to move across the country, (b) taking a pay cut, and (c ) strongly considering getting at least an MS and probably a PhD. The truth about the giant “FANG” tech companies is that, while they pay well, they’re paying in part for depreciation because most people get assigned to regular corporate work that you don’t learn much from after the first year.

                              Plenty of people will counter-argue that self-study is the way to go, and that it doesn’t require institutional support to do that. That’s true. Once you’re an expert, self-study is your default mode and finding a lecture or class that’s useful is somewhat of a lucky break. However, there’s an order of magnitude difference in what you can achieve via self-study if you’re allowed to do it on the job versus what you can achieve if you have 40 hours per week of shoveling Jira tickets from one category to another.

                              So, the short answer is that you’ll probably have to change careers and reinvent yourself as a researcher.

                              1. 4

                                Find a guy that’s better than you. Watch what he commits. Send him code reviews. You might learn things like:

                                Don’t just fix a problem. Try to make the problem impossible. If a function isn’t supposed to be passed NULL because it will crash, put the NULL check in the function so you can delete all the checks in the callers. If a data structure isn’t supposed to be inspected, put its declaration in a .c file, not in a .h file.

                                Audit the source tree proactively. How does the thing you’re calling work? Jump into the function and see.

                                Don’t write a lot of comments. A lot of comments that people write are bad.