1. 16
  • How are the parsing libraries in your language?
  • Have you had to resort to different languages than the target language to make tools?
  • How do your language’s existing tools accomodate your tools?
  1. 15

    I’ve been working on a formatter for Fennel. I use the language’s own parser, which worked really well except for a few places where it uncovered shortcomings in the parser. (These were fixed once identified.) Most of the issues were around how it represented comments, because the parser omits comments by default and only gives you them if you ask specifically. A few were around missing filename/line data in the AST for certain constructs.

    I also created antifennel which compiles Fennel back into Lua code by walking the results of an existing Lua parser and constructing a Fennel AST to send to the formatter. I blogged about the process; the cool part about this is that we used it to make the compiler self-hosting.

    I’m not sure that my experience can be generalized from as I’m also the lead developer of the compiler itself. That said, the idea that you would need a 3rd-party parser in order to just parse code for a language seems really broken to me! If the compiler doesn’t allow you to use its own parser I would consider that a bug.

    1. 6

      If the compiler doesn’t allow you to use its own parser I would consider that a bug.

      I think the idea that compilers should also be consumable as libraries has only surfaced recently, and even then not all compilers are self-hosting and not all compiler parsers might be usable for your specific use-case (maybe you wanna keep track of stuff that is discarded by the compiler like whitespace/comments).

      Still, I definitely agree that compiler should be exposing APIs.

      1. 2

        Those are good observations. As a Janet user, I have a similar view.

      2. 4

        I’ve tried writing a Language Server for Scheme. (It’s not on GitHub yet, it’s barely even usable right now.)

        The main thing I learned from it is that there aren’t good debugging tools for the Language Server Protocol itself. I tried using LanguageClient-neovim (my usual dev environment) and VS Code’s extension development system, and both made it difficult to tell when the langserver crashed or hung. And when it did crash, even if it reported errors in the standard JSON-RPC way, those errors were only reported in a log file, if at all.

        It makes me want to write a simple text editor for the sole purpose of interactively debugging new langservers.

        The one useful feature I did get out of it was missing paren detection, which, for a Lisp, is extremely helpful. I wrote an incremental, error-tolerant parser that uses indentation to infer missing parentheses, then keeps parsing with the inferred parens but reports them as errors.

        1. 3

          That came up in the Haskell Language Server IRC channel. I suggested a stub lsp client as a test harness, but I would happily let someone else build it!

        2. 2

          I’ve used acorn to parse JavaScript and do some static analysis of a large AngularJS project that is being migrated to React. There’s an AST walker included which is really nice. Overall it’s been a great experience! There is also @babel/parser which appears to support JSX and TypeScript but I haven’t had the need to try it.

          1. 2

            Funny you should ask, I was just doing some work to update https://github.com/glesica/dcdg.dart to the latest version of the Dart toolchain.

            Historically, the Dart parsing situation has been pretty grim. The documentation was terrible and you had to use a lot of unsupported APIs to get an AST or a compiled tree (more useful) and third-party consumers of the compiler / static analyzer weren’t officially supported.

            More recently (I assume because of the popularity of Flutter) they appear to have improved that situation substantially. I just bumped up to a release version of the analyzer and the documentation is actually helpful now! That being said, Dart is still kind of a niche language, so there aren’t a lot of alternatives if the official libraries don’t do what you need.