1. 61

  2. 9

    This is a really thorough and well-informed post about IDEs, I enjoyed reading it!

    There is one big theme which is missing though: the post talks a lot about incrementality (reducing the amount of work after modification, by applying a diff to the current state), but it skips laziness (ability of an IDE to skip most of the work altogether, by just not looking into the irrelevant code). In my experience, laziness is a relatively more important aspect of an IDE.

    Type checking the body of a single function is fast. If you can write completion such that it only looks at one function, you don‘t need to make type checking incremental.

    Similarly, I feel there’s often(not in this post though) an over-emphasis on incremental parsing, while in reality it’s not that important. For a long time, rust-analyzer didn’t support incremental text synchronization, and sending (and parsing) the whole file on every keystroke wasn’t a bottleneck. Similarly, Dart analyzer integration with IntelliJ used whole-file sync.

    1. 9

      So IDEs are merging with compilers? Just 2 days ago I wrote about similar stuff, except in context of compilers and build systems. Olle Fredriksson wrote about query based compilers before that.

      1. 3

        Seems queryable AST idea become a trend recently. There is another article the other day: https://nshipster.com/as-we-may-code/ Curious what drives the explosion of these thoughts.

      2. 9

        This seems like it could be really powerful for complex refactorings a la semgrep.

        There’s a nice feeling of principled-ness with this Datalog structure, I hope you can figure out a nice way of doing error reporting through this.

        1. 6

          The ??? syntax reminds me of Idris’s holes, which are used to represent unfinished parts of a program to the typechecker, and much tooling exists in Idris to interact with those holes at editing time, including some pretty magic seeming stuff. If you haven’t checked out “Type-Driven Development With Idris” and this seems interesting to you, I’d definitely check it out.

          The language’s creator (and author of that book) Edwin Brady did a very informative interview on the CoRecursive podcast which is how I was introduced to this stuff.

          1. 6

            Why don’t we just edit the structured AST instead? This could make querying easier, avoid syntax errors, and let compilers skip a stage.

            1. 4

              I haven’t worked on structural editors, so my perspective is biased towards text, but I believe these are some of the important reasons:

              • With lossless syntax trees and error tolerant parsers, you can implement most of user-visible structured editing features anyway. Extend selection, context-aware syntax completion, smart indentation, on typing formatting and code assists, as found in JetBrains‘ IDEs, together provide very „structured“ editing experience.
              • For the end user, ability to edit text is an extra power. Extend selection in combination with multiple cursors is a surprisingly useful and powerful combo.
              • What you save in implementation effort by not writing the parser (which is big, but flat, boring and simple piece of code) you pay for by developing a custom editor for AST.
            2. 4

              The sibling of this problem is that the language server protocol or some successor could provide more data to the IDE https://perplexinglyemma.blogspot.com/2017/06/language-servers-and-ides.html

              I believe Kythe was supposed to do this, haven’t used it though. https://kythe.io/

              1. 3

                Very nice post. Its worth to mention https://kythe.io/ (by Google) which predate LSF and LSIF.

                Essentially Kythe is built into Bazel on the compiler level and generate a universal Syntax Tree that can be serialized for later use. One of the supported storage is Cayley https://kythe.io/examples/#using-cayley-to-explore-a-graphstore (also by google) or Standard MySQL.

                This is very useful of code intelligence at scale where you cannot simply load everything at once (like LSIF) but might want to update things incrementally instead.

                1. 3

                  Its worth to mention https://kythe.io/ (by Google) which predate LSF and LSIF.

                  This is mostly tangential, but while LSP and kythe seems similar, they differ in a very fundamental aspect. Kythe is semantic-level API, it talks about symbols and usages of the symbols. LSP is, by design, a UI-level API, it doesn’t have a concept of “symbol” per se. Rather, it talks about “when clicking on this offset in this file, the editor should jump to that offset in that other file”. So, LSP does not try to build a universal semantic model of a language, and this is one of the reasons why it is successful.

                  A Google technology which is much closer to LSP (and which also predates it) would be Dart analysis server protocol: https://htmlpreview.github.io/?https://github.com/dart-lang/sdk/blob/master/pkg/analysis_server/doc/api.html

                2. 2

                  I love how far the author was able to take this idea into the land of working tools, just from the simple idea of keeping a semantic database of the code.

                  This is also the idea behind the Rust Analyzer; there’s a slightly out of date but still very informative guide. There’s a database full of entities and their relationships and meanings, and facts get updated and queried as you go. The project even describes itself as an “IDE-first compiler front-end”.

                  1. 1

                    This was really interesting, and well put together. I’ve thought too for a while that we still have very rudimentary tools as software engineers.

                    1. 1

                      The idea of generating IDEs and a universal representation for programs is quite old - here is a paper from 1986 and I’ve seen once a demo of that system: