1. 90
  1.  

  2. 29

    I agree with the points raised in the article. One thing to add is that many tools built in academia are there to support the paper, not to take a life of their own. Given the short life of a thesis, there is not much one could do to make these tools gain much traction (all examples in the article show this).

    There is another weird thing in our industry — I’ve seen many companies embracing the latest shiny tools and frameworks (k8s, react, containers, …), yet when it comes to thinking and implementing ways to speed up developer work by improving build times or reorganize dependencies, that is always a task for the intern, the least suitable person to handle such a thing.

    1. 10

      yet when it comes to thinking and implementing ways to speed up developer work by improving build times or reorganize dependencies, that is always a task for the intern, the least suitable person to handle such a thing.

      That has not been my experience. I’ve been tasked with that sort of work many times, and given permission to pursue it many more. Currently a senior UI engineer at Microsoft, but previously worked at a 500 person EdTech company for several years, and had a brief, but enjoyable stint at Todoist. All three were very happy to give that work to an experienced developer.

      Have I had managers who were averse to that? Yes. But it has been the exception.

      1. 3

        Agreed with this. My experience at both large and small tech firms has been that when a senior person wanted to work on developer productivity, it was almost always welcomed. Maybe it’s different at companies whose product is something other than what the developers are building, though.

      2. 7

        The problem in academia is that it is controlled by where the professor can get money from. There is a lot more grant money for making shiny new tools than for maintaining and adapting old tools, and your academic life depends really strongly on how much grant money you can bring in (determines how many students you can support, how many conferences you can send your students to, and what equipment you can give your students). (New papers are also indirectly about grant money. You need new papers to bring in grant money, and you can’t write new papers simply by maintaining or updating existing tools).

        1. 3

          I agree, and it’s also about promotion criteria; it would be hard to get tenure if all you ever worked on was maintaining old tools. I think the solution is to create a new tenure-track academic role which is evaluated, promoted, and funded based on the maintenance and creation of useful open-source artifacts (rather than the traditional goal of the discovery of novel knowledge).

      3. 8

        Other fields of computer science don’t seem to have such a giant rift between the accomplishments of researchers and practitioners.

        I don’t think this is true. Academia is very very bad at transferring their learning to practice. I think there are a lot of reasons: the standard for conference papers seems to be “I wrote program and it definitely does based on no actual testing outside of the people who wrote it”, a format that is extraordinarily useless to other researchers except as a way of learning of someone who might be working on that kind of thing, and completely useless to practitioners.

        All of the tools mentioned here have wild claims attached, but they boil down to being good for the one use they were designed to be tested on - with the exception of the excel case study.

        Specific to this area of endeavor there is also the problem that industry is intensely bad at making use of tools, ideas, skills or professions that they have not encountered before. This comes from both software engineering arrogance (“we’re so smart we know everything”) and businesses being pennywise (“we already pay so much for engineers we can’t afford to also make them more efficient”).

        1. 7

          It varies a lot by subfield – I’d say that “systems” is sort of adjacent to software engineering tools, and a lot of those technologies have been commercialized. (e.g. think USENIX papers)

          VMWare came out of Stanford (there is a foundational paper about x86 virtualization). Spark and Mesos came out of Berkeley.

          Margo Seltzer started companies around BerkeleyDB. Stonebraker started a whole bunch of RDBMS companies, which led to Postgres being open sourced. I think NFS came from academic a long time ago.

          Anything regarding performance tends to get commercialized too.

          I believe my former boss wrote the original paper on Control Flow Integrity and that landed in Clang a few years ago. Although there was a big time gap, and he “funded” one of his reports implement it in Clang. Maybe it wouldn’t have been implemented without that, but it looks like there are other non open source implementations too:

          https://en.wikipedia.org/wiki/Control-flow_integrity

          Software engineering tools tend to be commercialized less, but they’re by no means the only subfield of academia that doesn’t get put into practice.

          I guess I would say “putting CS into practice” requires economic motivations. All the examples above have economic motivations. Software engineering tools not so much unfortunately.

          JetBrains, Atlassian, and Github are exceptions to the rule that developer tools don’t make money. But I would say that JetBrains and Github are concentrated on UI. I’d say that a lot of tools require 90% of the work on UI and 10% of the work on algorithms, which may be another thing that explains it.

          1. 4

            I think that’s a Texas sharp shooter fallacy- just because some things made it out into the world it does not mean that a larger proportion of good ideas in systems work make it out into the world.

            1. 4

              I’m specifically arguing that there has to be an economic incentive, and there’s less in dev tools, and more in systems.

              Even though you didn’t provide any evidence for your claims, or specific experiences, here are dozens and dozens of papers that have been deployed in Linux:

              https://github.com/oilshell/blog-code/blob/master/grep-for-papers/linux.txt

              and LLVM:

              https://github.com/oilshell/blog-code/blob/master/grep-for-papers/llvm.txt

              1. 3

                Yes, and that’s the fallacy. Just because those papers made it into production code, it doesn’t mean that systems research in general gets into production more than other research.

                1. 4

                  I’m not claiming the evidence is airtight, just that it’s well supported, matches my experience, and you didn’t provide any evidence for your claim.

                  I don’t see how you can possibly claim that all fields of CS are equally bad at commercializing their work. On its face it seems silly. Some fields are closer to practice than others – if you’ve ever worked in an industrial research setting, that would be obvious.

                  1. 2

                    You could also consider these two examples as an approach to avoid the problem: Get the research tool under the umbrella of a bigger FOSS project like Linux or LLVM here.

          2. 7

            I did my grad academic work in dev tools (2008-2012) - specifically, working in building debugging for parallel embedded systems. I took a road… less directly useful… and wandered through a great deal of time learning about the historical background of debuggers and programming tools.

            It is profoundly depressing. We have reinvented programming tools and toolchains constantly since about 1970. Industrial work gets gobbled by IP law when the company fails or puts the advanced tool into the shed, academic work goes obsolete. FLOSS work has a chance, but often fails to compile if its not maintained. Introspection tools (like the cited ones), are deeply coupled with the emitted code, f’ex.

            One lesson I took from my studies was to really get good at a long lived technology. Another was to know a better world was possible if reformist & incrementalist approaches were properly prioritized where they belong. Other takeaways are doubtless plausible.

            FWIW, the starting line about programming tools is an interesting one. It may have been “grep” and “formats” where the author was existing and in their communities. But even Visual C++ 5 was more advanced than that in 1999.

            1. 6

              Another solution to this problem would be to create a new kind of academic role whose function is to create and maintain open-source tools/content. Economically, the academic system is fundamentally well-suited to solve any societal tragedy-of-the-commons regarding intellectual public goods. However, the current system is more narrowly focused on the goal of discovering novel knowledge, which is only one special case of intellectual public goods.

              We should create a role of “open source professor” whose job would be to create and/or maintain open artifacts which are useful to society, rather than to make new discoveries. This would have applications beyond just developer tools, too.

              1. 4

                Excellent title for an excellent article! I’m glad you’ve made this point. Best of luck to you as an author.

                1. 3

                  I started working on advanced developer tools 9 years ago. Back when I started, “programming tools” meant file format viewers, editors, and maybe variants of grep.

                  … and no, this is not taken out of context. This is the whole first paragraph.

                  At least this post starts with wild hyperbole. Eclipse existed 9 years ago (as did a number of other tools), abd it goes a long way beyond this.

                  1. 4

                    … and no, this is not taken out of context. This is the whole first paragraph.

                    Is Eclipse not an editor?

                    Coverity also existed, but that’s not what people would think of when I said “dev tools.”

                    1. 5

                      I wouldn’t be as hard on you as the GP, but I feel like just dismissing Eclipse as an editor obscures more than it reveals. Aside from the whole IDE/editor terminology tarpit, Eclipse (and Intellij, which I’ve used more recently) include lots of tools for exploring, refactoring, visualizing and debugging code.

                      I think all three tools you describe in your post sound like the kind of things you could conceivably consume as plugins in one of those IDEs (that could either be via some connection to an external tool, or by actually embedding it as plugin with no external dependencies).

                      That quibble aside, I agree with your post. There’s a lot of interesting ideas coming out of academia that aren’t maintained, or even worse, never become usable. This may be a separate point, but even without an academic background, it’s pretty easy to find limitations of the existing tools as an industry programmer (e.g. IntelliJ structural search and replace is wonderful, but once you’re past what it can do, it gets painful quite quickly).

                      1. 2

                        Cool. I don’t think we have any disagreement, other than that you and GP think of a different cluster of tools when I gesture at “format viewers and editors.”

                      2. 4

                        It is/contains an editor, all IDEs do.

                        But it is not just “file format viewers, editors, and maybe variants of grep”:

                        There was intelligent refactoring, dependencies, debugging etc.

                    2. 2

                      my day job is writing python developer tools, and i found this article really inspiring. in particular i would love to explore the ideas behind reflexion models more - has anyone done related work in a dynamic language like python?

                      1. 2

                        It seems more difficult to do in a dynamically typed language or a statically typed language with access to something like Java reflection because you don’t have all of the class information before runtime.

                        1. 2

                          i have already written a tool that uses type information inferred by pytype to index and cross reference code, so i’m optimistic that that can be extended to reflexion models once i understand more about the latter.

                          1. 2

                            I just mean that you will never get full coverage due to things like eval and exec.

                            1. 2

                              ah, right you are. yeah, that might be insurmountable for the general problem, but most code doesn’t use those constructs so there is still value in static analysis tools for “well-behaved” python code.

                          2. 2

                            Doop does pretty well with reflection. http://yanniss.github.io/reflection-aplas15.pdf

                        2. 2

                          Seems like the author cherry picked a few examples that didn’t pan out, but I don’t think the generalization works. IME, “developer tools” have improved tremendously in the last twenty years that I’ve been working as a developer. Even the languages themselves have been moving in directions that make dev lives easier.

                          The biggest problem, IMO, is that by the time a language and its ecosystem has been around long enough to develop mature tooling, the trend followers have moved on to the next new thing. C++, Java, Lisp, and even Python have a ton of dev tools. The newest, most bleeding edge Javascript dialect probably not so much.

                          1. 5

                            Hmm, I’m not sure I agree.

                            The author picked tools from one lineage (Java and the community of researchers who attempted to improve it).

                            You could look at tools from another lineage (e.g. Smalltalk, Lisp) and note that big, useful ideas (e.g. the Smalltalk browser, Gemstone version control system, Common Lisp condition/restart exception handling, Interface Builder) tend to go by the wayside or emerge into popular practice hamstrung in crucial ways (Interface Builder’s interactivity subjugated to the edit-compile-run cycle).

                            I’m more familiar with the Lisp stuff than the Smalltalk stuff so I’ve provided links for reference for those.

                            1. 4

                              What is the best reference to read about the Gemstone system you mention?

                              1. 2

                                I don’t have a reference handy, I’m sorry :(

                                I learned about it from someone I met at bangbangcon a few years ago - they worked (many years ago) in finance on an all-Smalltalk system and we talked about it at length over lunch.

                                1. 2

                                  I used ENVY for version control, but only knew of Gemstone as an object database. I think they later morphed into a distributed object system with persistence ability.

                                  1. 2

                                    NP. I will dig around. Smalltalk inspired quite a few neat projects.

                                    1. 1

                                      I likely conflated Gemstone with ENVY, see mtnygard’s sibling comment!

                                      I also found this collection of descriptions of ENVY and how people used it - from this description it doesn’t seem obvious to me what specific technical feature was awesome about it.

                                      Several of those interviewed talk about method-at-a-time versioning reducing conflicts and the subjective experience of the “liveness” of the whole codebase.

                                      https://paulhammant.com/2017/09/01/smalltalk-envy/

                              2. 1

                                C++, Java, Lisp, and even Python have a ton of dev tools. The newest, most bleeding edge Javascript dialect probably not so much.

                                Let’s not ignore the advances that have been made relative to those languages, with regard to community, debugging, legibility, safety, ergonomics… etc. I admit, many people’s function for what to use is basically use?(thing, threshold) = (thing.hotness > threshold). But there are some legitimate reasons to use newer languages.

                                If using your newfangled tool required using a language whose inception took place at least 25 years ago, wouldn’t you consider that a problem for adoption? Would the kind of person to use such a well-established tool really be interested in trying out your new, shiny invention?

                              3. 1

                                For purposes of a broader and deeper conversation, I’d like to ask “what recent development tooling changes go against this trend?”

                                1. 1

                                  I think that the Language Server Protocol is a great example of recent progress in development tools. Language servers provide the language analysis metadata needed for what we think of as tolerable or good support for a language-aware editor, language clients consume LSP to make a better development experience. vscode and its extensions are one way that someone can start benefiting from LSP with very little installation effort.

                                  LSP is a solution to the problem that if M implementors of IDE tools each want to support N languages, then with no shared infrastructure they’ll need to write M*N modules- one for each IDE and each language. Instead the language analysis and the enhanced editing or analysis are decoupled.

                                  IDEs have long provided the sorts of features that LSP clients and servers enable but the standardization of LSP is less than 5 years old, which does say something about the slow progress of broadly available or broadly usable development tooling.

                                  https://langserver.org/

                                  https://microsoft.github.io/language-server-protocol/

                                  https://microsoft.github.io/language-server-protocol/implementors/servers/

                                  https://microsoft.github.io/language-server-protocol/implementors/tools/