1. 20

I posed this question on Hacker News and wanted to see if the response was any different here.

Today software development looks pretty fragmented and complex depending on what angle you’re coming at it from. I’ve spent the past decade building web apps, distributed systems, infrastructure and all sorts. My feeling is that as a developer there is still so much that stands in my way getting from some local piece of software to something that runs and scales in “production”. I’m starting to rethink this from first principles and curious to know what others deem as the ideal developer experience for 2020.

Share your thoughts

  1.  

  2. 24
    1. All modern conveniences work in major IDEs: syntax highlighting, code completion, jump-to-symbol and debugger.
    2. I can run the entire thing I’m working on locally, or in a non-production environment without affecting others.
    3. I can attach performance tools and debuggers manually (e.g. run with gdb attached) with configurations at least optionally loaded from file or command line.
    4. Tools (editors, etc.) take a reasonable amount of time to load–no more than 15, definitely no more than 30 seconds.
    1. 9

      I would add a CI setup that gives you feedback reasonably fast to that list. Other than that 100% agree.

      1. 4

        I’d add a qualification to that: a CI setup should only run integration tests. If unit testing (or worse: compilation) is dependent on a central environment, it lengthens the feedback loop, instead of shortening it.

        1. 1

          Preach it.

          Ideally unit tests should just be precommit hooks–letting us make progress by skipping them if need be. If the unit tests fail but the integration/acceptance tests pass, that’s okay in the short run.

          1. 2

            I’m interested in the sorts of unit tests you’d be okay with skipping. To me unit tests are for more mathematical properties of functions and units of code where if they failed I’d be really concerned about correctness. Perhaps you’re unit testing too much? Or maybe I’ve misunderstood.

            1. 2

              So, say a big customer comes in and specifically asks for 1000 units for an order. The pricing code is complicated, and because reasons (tight deadline, excited salespeople, whatever) the order is given to circumvent the code for orders of 1000 and just always return a certain value. This change breaks unit tests, but the thing’s on fire right now so it’s worth it to punt it along until the crisis is past.

              Another example would be some midway refactor point where the code works and needs to go live to fix some other bug but the unit tests haven’t been fixed yet to reflect the new dependencies that need to be injected or the new APIs or whatever.

              Tests are important, but they’re usually not deliverables to the business and it is important to be wise when deciding how much to let them guide us.

              I’ve worked on a codebase where somebody diligently made hundreds of unit tests that a) failed to catch major, basic regressions in the money funnel (checkout) and b) only meant we had things to fight against during refactors.

              In my own work, especially numerical stuff I’ve done, I still do 100% test coverage before shipping 1.0, and view it as negligent to ship libraries intended for external use with anything less than that. But, for a business application, needs must when the devil drives.

              1. 2

                Ah that’s interesting, yeah I wasn’t thinking about it in terms of unit testing for business requirements at all. I was thinking entirely in terms of numerical/relational properties.

                1. 1

                  breaks unit tests, but the thing’s on fire right now

                  So remove/disable the unit test, instead of skipping it? It reflects historical requirements, not current ones.

                  1. 1

                    The test isn’t wrong, it’s just in the way. Committing while skipping precommit hooks is less of a sin than removing the test and forgetting to put it back later.

                    Having lots of disabled flappy tests, or even worse removing tests which coworkers assume are present, is an anti-pattern to me.

              2. 1

                What? No. This is wild.

                1. 1

                  Sorry, I don’t follow. Say again?

                  1. 2

                    If the unit tests fail but the integration/acceptance tests pass, that’s okay in the short run.

                    This is bonkers. If your system is built in such a way where this is possible, or even (as your later example kind of suggests) even sometimes necessary, your system is built incorrectly.

                    Or,

                    The test isn’t wrong, it’s just in the way.

                    This is a category error: there is no such thing as a failing unit test which isn’t wrong but merely in the way. Unit tests and the code they cover are inextricably interwoven. And there is no such thing as a change so urgent that there isn’t enough time to update failing unit tests. If your code base and processes have you believing either of these things are sometimes or even frequently true, you’ve dug yourself into a really bad hole.

                    In my own work, especially numerical stuff I’ve done, I still do 100% test coverage before shipping 1.0, and view it as negligent to ship libraries intended for external use with anything less than that.

                    This is also bonkers.

                2. 1

                  unit tests should just be precommit hooks–letting us make progress by skipping them if need be

                  I’d go the other direction: unit tests are an integral part of every class (insert unit of choice here) and run on every build without any additional overhead. Failing test means failing build.

                  Yes, this means tests must be fast.

                  I just wrote about this a few weeks ago: MPWTest: Reducing Test Friction by Going Beyond the xUnit Model

                  I’d also add:

                  Contrary to what you may have read in the Agile literature, the key to agility is the ability to change code quickly and safely. And the key to that is the ability to re-test code quickly and effectively. Fast-running automated tests (“unit tests”) are the key to agility.

                  Jason Gorman

            2. 3

              Fully agree with your list.

              I’ll just add automatic code formatting to the list of modern IDE conveniences.

              After having used Prettier, gofmt and similar there is no going back.

              1. 1

                That’s interesting to me because autoformatting code is really nothing new, it’s been around for decades. indent is 44 years old and GNU indent is 31 years old.

                Is it autoformatting you like or is it more about having a fixed set of rules that applies to all code written in a language?

                1. 1

                  What I love about autoformatting, specifically when integrated in the editor and adopted by the majority of the community of that language, is that it removes the need to have discussions around where to put brackets, spaces, etc.

                  All you do is type code the way you like, hit save and have the tool ensure your code respects a consistent style. I don’t have to waste time reading a style guide or think about where I need to put spaces inside the brackets or not.

                  Even better, my code looks just like the code written by somebody else, which makes it easier to read.

            3. 10

              I’d say low compilation/synthesis time. Waiting minutes at a time for each run will ruin you.

              1. 1

                True since the beginning of software development!

              2. 7

                Simplicity is the most impactful improvement that can be made to my developer experience. Being able to dive into a codebase and rapidly follow a thread of data flow from start to end, through the various interacting systems is a huge part of my time, and the fewer and smaller those systems are, the easier it is for me to get my work done.

                1. 4

                  After I came across Elm, my development experience for writing frontend apps skyrocketed. The FRP model in conjunction with a type-safe language is just so predictable and pleasant to write code in (compared to say callbacks or React). I don’t use Elm anymore, but use GHCJS primarily for writing web apps. Obelisk takes this one notch up in terms of development experience of full-stack apps, especially as it integrates with Nix .

                  In the environment / deployment side of things, likewise, when I came across Nix and NixOS - things took a pleasant momentum. I now manage a dedicated server without sweating, all using declarative configuration, running various services. Nix is used to setup my development environment as well.

                  From the IDE side of Haskell, things are not perfect yet, but ghcide is promising.

                  EDIT: Here are some testimonials of delight from users discovering the use of Nix to set development environments for an open source project they are about to contribute to.

                  1. 4

                    Basically Visual Studio but faster and without crashes.

                    1. 4

                      If, by “ideal” you mean “the tools that we should have (given our current level of technical knowledge e.g. no neural lace allowing us to program by thinking)”:

                      • Code structure editors. Code is not text, it’s a tree, and text-based tooling comes with massive disadvantages and inefficiencies relative to structure-based tooling.

                      • REPLs. No language community to claim to have decent tooling without having a REPL. The productivity difference is significant, especially when it comes to debugging.

                      • Hotpatching. There’s no reason why I should have to restart my entire program to update the definition of a single function, even for compiled code.

                      • SLIME-like “inspectors” - tooling for editing data in memory. We have oodles of programs that edit source code on disk, but very few that allow us to (ergonomically) edit data structures in a running program’s memory, which makes debugging and development much easier.

                      • Low-latency tools. It’s perfectly fine for a tool to take a significant amount of time to start - as long as you only have to start it once (which excludes all batch-processing-like (non-interactive) tools that exit when they’re done with a specific task (e.g. most modern compilers)). Most people don’t complain about their phone taking 30 seconds to cold boot, because most people only cold boot their phone once every few weeks/months, and the rest of the time just put it in sleep mode, which takes only a second to resume from (most of the time). Similarly, it’s fine (even though suboptimal) for tools to consume relatively large amounts of memory, as long as they’re fast after they start up - poorly-written programs, like Slack and Chrome, not only consume large amounts of memory, but also are noticeably laggy and take up tons of CPU just sitting there. Similarly, compilers in “development” mode, where you want code fast, not fast code, should take no perceptible amount of time for small programs.

                      • Flexibility and introspection. I want my tools to be written in a general-purpose programming language, to be scripted in the same language, and to allow me to hotpatch and improve them while they run.

                      EDIT: Now that I think of it, I’m just rehashing Steve Yegge’s essay, so go read that, instead: https://steve-yegge.blogspot.com/2007/01/pinocchio-problem.html

                      1. 3

                        Basically where I can do what I think, and don’t have to context switch between struggling with the problem and struggling with the environment. I think most things mentioned here (tool integration, speed, introspection, etc.) all boil down to this.

                        1. 3
                          1. A quiet place to work without visual and aural distractions.
                          2. Linux laptop or desktop.
                          3. nixOS or nix as a packaging system so you can be sure binaries in production are the same as the binaries you tested.
                          4. a language server for your choice of programming language.

                          (yeah, also source control and continuous integration in there somewhere)

                          1. 4

                            There is no ideal developer experience, full stop, end of thread. Anyone who argues otherwise is ignorant of the sheer diversity of individual human perception, experience, and skills.

                            There’s only what works for each individual. Even if you took a small group of developers who work on the same narrow product and have had remarkably similar career paths, you are going to find that one person uses a graphical IDE while another is just fine with Vim and a terminal. Or one might have put in a lot of effort into memorizing the various stages of a complicated product build while another wrote a bunch of scripts to manage everything.

                            And this is a good thing. The diversity of different perspectives generally leads to a team that is more flexible and able to come up with better solutions than teams where everyone is forced to work in the same environment and use the same tools in the same way.

                            My feeling is that as a developer there is still so much that stands in my way getting from some local piece of software to something that runs and scales in “production”

                            I wish you all the best but if the process of creating complex software going from an idea all the way to deployment was in any way easy, we wouldn’t need developers. The same way that we wouldn’t need carpenters if building a house was easy or mechanics if maintaining equipment was easy. Mastering a craft is almost always more about developing skill with the tools than the ability to do the actual work. Anyone who does woodworking as a hobby, for example, will tell you that a surprising amount of their time is spent solely on building, buying, organizing, or maintaining their tools. I don’t think it’s that much different for proficient developers.

                            1. 6

                              The diversity of different perspectives generally leads to a team that is more flexible and able to come up with better solutions than teams where everyone is forced to work in the same environment and use the same tools in the same way.

                              I have never seen this be the case, whereas in teams with shared tooling and environments I’ve seen massive increases in the ability to help each other solve common problems and improve the shared tooling.

                              1. 3

                                There is no ideal developer experience, full stop, end of thread. Anyone who argues otherwise is ignorant of the sheer diversity of individual human perception, experience, and skills.

                                That assumes that tools have to be designed in such a way that they fit only some peoples’ workflows and not another - which is false.

                                There are two kinds of features - those which preclude or limit other features, and those which don’t. The former, which are in the minority, can be exposed via configuration and flexibility of the underlying tool. The latter, which are in the majority, can just be bundled in an application with no (inherent) drawbacks.

                                If I build a tool with a REPL, the presence of that feature is beneficial for the majority of developers, and detrimental for exactly none of them.

                                What feature are you thinking of that is beneficial for some people and, if included in a tool, would necessarily be detrimental for others?

                                you are going to find that one person uses a graphical IDE while another is just fine with Vim and a terminal

                                This has everything to do with preference, and nothing to do with “ideal developer experience” - which is necessarily concerned with efficiency and not perception. Lots of programmers don’t learn to touch-type for the first few years because they think it isn’t necessary or are just procrastinating. This isn’t “diversity” - this is just some people flat-out lacking a particular skill that will objectively make them more efficient.

                              2. 2
                                • Solid terminal + editor with appropriate customization for language and individual. This is well worth investing time in, anyone that I know who has done so has reaped huge benefit for weeks/months/years after.
                                • Docker (or other containers) based development, with a well defined and low/0-effort path to production. At work we’ve built an in-house tool (kubetools) to enable local dev via docker-compose with one command pushes to K8s clusters. Any process/flow like this is ideal IMO.

                                Ultimately there are just some things that only a production environment needs that. Someone has to handle this - be it the programmer of the features or a dedicated operations team. I suppose (never used) services like Heruko and Elasticbeanstalk abstract these elements away, which is cool.

                                1. 7

                                  Docker absolutely does not belong in your edit/build/test loop. It’s a tool for integration testing, not your laptop (or prod, for that matter). If this seems unrealistic to you, in my opinion, you haven’t balkanized your architecture in an appropriate way for delivering business value, and that deserves fixing.

                                  1. 3

                                    Curious, but why do you say not in production?

                                    1. 3

                                      I spent enough time in the sausage factory to become a vegetarian, so to speak. I was in the core Docker ecosystem for several years at the outset, and the code quality I observed convinced me that it has no place running important workloads.

                                    2. 1

                                      It’s a tool for integration testing, not your laptop (or prod, for that matter)

                                      Assuming your objection is to “on laptop integration testing,” I get the idealism here. But, we live in the real world where pushing to a “proper integration testing environment” takes a lot of time, and removes us from the flow of development. Having the ability to do some local integration testing, on my laptop, is pretty huge. It allows me to test that my assumptions in regards to an API / protocol / etc aren’t completely misguided, long before I submit it for a full test run in a more realistic setup.

                                      (Note: I am not a docker apologist.)

                                      1. 2

                                        we live in the real world where pushing to a “proper integration testing environment” takes a lot of time, and removes us from the flow of development. Having the ability to do some local integration testing, on my laptop, is pretty huge

                                        I guess I don’t agree that integration testing with real dependencies should be part of your development cycle, because I believe (quite strongly) that creating business value at the service level with mock dependencies should be an explicit and enforced design goal of any architecture. Said another way, if you need to integration test to be confident in your changes, you’re too tightly coupled.

                                        1. 1

                                          Said another way, if you need to integration test to be confident in your changes, you’re too tightly coupled.

                                          Something, somewhere has to actually touch the database.

                                          1. 1

                                            Sure, but I don’t need a real DB when I’m developing. In fact I explicitly don’t want a real DB when I’m developing.

                                            Of course I’m speaking from a specific context: business applications where DB access can be easily modeled with a simple contract, and not services that put a lot of logic at the DB layer.

                                            1. 1

                                              not services that put a lot of logic at the DB layer.

                                              I know of few “business applications” that don’t use the whole power of the database. but if you are just talking “Bob’s rails app,” fine. That’s a pretty generous generalization though.

                                              1. 1

                                                I know of few “business applications” that don’t use the whole power of the database

                                                Almost all business applications I’m exposed to have simple CRUD-ish contracts with their persistence layer. Not much logic there. But I’m sure we work in different contexts.

                                      2. 1

                                        I’ll bite, I am curious as I am working on a re-write of an open source platform for sharing electronics projects. I am re-using the Gitea open source git hosting software (a self-hostable Github clone) as a back-end. I have everything dockerized and use docker-compose so that any dev that wants to help out can spin up the whole application. I currently also deploy it to staging.kitspace.org using the same setup (with some production overrides in a docker-compose.production.yml). How would you suggest I balkanize this architecture correctly and stop using docker during dev?

                                    3. 2
                                      • Error messages that point you to the root cause of the problem, not irrelevant language implementation details
                                      • Seamless reloads with a repl that’s easy to integrate into your editor
                                      • Dependency fetching which is 100% consistent everywhere (ideally in a content-addressed way)
                                      • Tracing should be easy to enable for quick sanity checks, especially in distributed systems

                                      That last one is something I’ve only seen on Erlang and Erlang-derived systems. You need a way to trace a given function across all invocations on your entire cluster, ideally with the ability to specify a pattern match to narrow it down to just the calls you care about. And you need a built-in killswitch so that after a maximum of N messages it disables itself, otherwise you risk taking the whole cluster down with a flood of tracing.

                                      1. 1

                                        I’ve been thinking about this a lot in the context of building cloud-based software, as I’m working on a product specifically to drastically improve developer productivity in backend development.

                                        There’s an almost endless amount of improvements we could make to the developer experience. I think the real question is: why haven’t we? I think the answer comes down to this: our tools are too general-purpose. The big players are cloud providers, that want to support any application regardless how it’s built. As a result we end up with innovations like containers that further push us towards the “applications are black boxes” end of the spectrum.

                                        I think we could 10x the developer experience if we started from the opposite end: if we built our developer experience around purpose-built tooling (backend development in my case), and we added constraints to how you write your application, we could infer so much more about the application than what is possible today.

                                        For example, all of these things you should get for free, with no work needed other than writing your business logic:

                                        • Build & deployment orchestration, with your whole app running “serverless”
                                        • Setting up databases, managing connections, passwords, backups, and DB migrations
                                        • Automatic API documentation based on static analysis of your API function declarations
                                        • Generating type-safe client code for calling your API for any language
                                        • Expressing API calls between backend services as function calls (that get compiled into real API calls), and getting compile-time validation
                                        • Automatic distributed tracing of your whole application
                                        • Automatic management of production & test environments, and preview environments (for each pull request)
                                        • Run everything locally with no code changes needed
                                        • Cross-service debugging
                                        • Error monitoring, graphing & alerting (observability)

                                        I could go on :)

                                        1. 1

                                          I’m with you on that one :)

                                          https://m3o.com

                                          1. 1

                                            Very cool, I hadn’t seen that one! Will have a look, thanks for sharing!

                                        2. 1

                                          My ideal coding experience is having the path completely free of obstacles and detours between my thought of what I want to do and the actual making of that mental intangible into a reality. Anything and everything that either removes such obstacles (or doesn’t put them there in the first place), or makes that path as straight and direct as possible – I hold onto them, and defend them.

                                          In the context of this idea, here a few things that I like: Ruby (note: not Rails), Vue, Cypress, (certain) webpack dev servers, in-browser dev tools.

                                          And, in contrast, a(n incomplete) list of things I dislike: Angular, extreme linting rulesets, Docker (crafting, not usage), most cloud devops-y stuff (AWS, etc.), bitfields, [overuse of] state machines, slow app standup time, extremely slow page load times [in development], extremely long-running test suites, gargantuan monolithic classes/modules/whatever, tight coupling, silent coercion, silent failures, unhelpful errors, misleading errors, invisible state (e.g. yarn link), unclear mapping between UI element and source code file, MySQL CLI, poor or spartan documentation, having documentation instead of making something so easy to use it doesn’t need [as much] documentation, git rebase messing up long-running github PR code reviews, PRs that are [too] large, long-lived branches, insufficient test coverage, unrealistic deadlines, poor management of technical debt.

                                          1. 1

                                            I’d say a REPL is super useful…

                                            1. 1

                                              The Elm ecosystem is now what I hold as the gold standard for dev UX. The compiler alone includes:

                                              • fastest build times I’ve ever experienced in web dev by far
                                              • error messages that both point to the exact cause as well possible solutions
                                              • repl for those that want it
                                              • dev server (for Elm only)
                                              • semver is enforced in public packages

                                              If you want more tooling there’s elm-format which follows in the footsteps of gofmt. There’s elm-language-server which supports a handful of IDEs and provides a lot of autocompletion and other niceties. If you need a slightly better dev server there’s elm-live, and soon Snowpack will support as well.

                                              For most hobby projects I rarely have to use anything beyond the cli and elm-language-server, and one or two 3rd party dependencies!

                                              1. 1

                                                I spent a couple of decades programming for a living, before going post-technical and stepping into engineering management a few years ago. I’ve worked professionally on Windows, Linux, FreeBSD, MacOS, Solaris, in C, Perl, Ruby, JavaScript, C#, Python, Java, Clojure, VBA, Bash. Editors and environments include vi*, Emacs, Visual Studio (pre- and post-.NET), IntelliJ, NetBeans, Excel.

                                                With that context, now that I program purely recreationally, I use Common Lisp, SLIME, and Emacs wherever humanly possible. My reasons are as follows:

                                                1. Common Lisp is the most powerful language I’ve used. Multi-paradigm, with macros and reader macros, and an exceptional (heh) error handling mechanism with conditions.

                                                2. REPL-based development is far more productive than non-REPL based - faster, more iterative, far more amenable to TDD (in fact is IMO a distillation of same), more focused, and more supportive of Flow. Having commercial experience with a number of REPLs, I’ve found the Common Lisp REPL the best.

                                                3. On top of the Common Lisp REPL, SLIME adds an additional layer of power, speed, and flexibility. I can program exploratively, write tests, run tests, profile code, trace code, handle conditions (errors), etc. all within the same environment. Previously I’d considered Visual Studio’s debugging facilities my favourite; SLIME now holds that position.

                                                4. In certain circumstances (especially, on long-running servers I own for personal projects) the ability to attach SLIME to a running instance of Lisp and inspect / debug / correct problems is amazingly powerful.

                                                5. Alongside SLIME, Emacs offers exceptional scripting support and plugins, including Magit for git, terminals, countless modes with previews (e.g. PlantUML mode for diagramming).

                                                6. GitLab for project management, and super-easy containerised CI.

                                                I’m busy experimenting to see if I can bring the power of the above stack to native mobile development on the PinePhone platform, but it’s very early days.

                                                (Also worth mentioning that the above has a very heavy ‘in my opinion’ flavor to it :) )

                                                1. 1

                                                  It might be helpful to approach the question from another direction as well: what can hurt the developer experience, break the flow and sap motivation?

                                                  Uncertainty, complexity and long waiting times are the three categories which come to mind.

                                                  Uncertainty - imagine you implemented a new feature, but aren’t really sure everything else still works. Having tests can help to reduce uncertainty.

                                                  Complexity - having to think about the exact commands and steps you have to take while developing every single time. Something like a Makefile to simplify interactions is a good first step to reduce workflow complexity. Writing readable and not-too-smart code is another example.

                                                  Waiting times - unwanted context switching is an issue. Taking a voluntary break from coding to fetch a tea or go for a walk can be great. Being forced to wait for 20 more seconds than necessary on every change or for 10 minutes for your Docker images to rebuild will just drag your momentum down. A countermeasure can be to deliver parts of the feedback loop as soon as possible so the average feedback time goes down. Running a linting step to detect bad patterns before the tests are triggered for example.

                                                  Is there something else I have missed?

                                                  1. 2

                                                    Taking a voluntary break from coding to fetch a tea or go for a walk can be great. Being forced to wait for 20 more seconds than necessary on every change or for 10 minutes for your Docker images to rebuild will just drag your momentum down.

                                                    When I was a teenager, part of the appeal of becoming a software developer was xkcd 303. How wrong I was…