1. 37

  2. 11

    This was an enjoyable read. I agree that a lot of the Perl6 intros fail to make much headway into it, while this makes headway into at least some corners of the (large) language.

    On one minor point:

    The problem with Perl 6’s grammar engine — and perhaps this shortcoming could be addressed in a future version of the language — is that if parsing fails for any reason, the parse method simply returns Nil. There is no indication of any kind about where the failure occurred; so for any kind of non-trivial input, grammars are a real pain to debug.

    For debugging purposes there’s a module Grammar::Tracer that will dump a trace of the parsing process (whether successful or failed). The Rakudo debugger is also regex and grammar-aware so you can do the usual single-stepping, breakpoints, etc. into them (even single-stepping construct by construct within a regex). Not an out of the box solution for generating nice error messages like the better parser generators can do, but purely for debugging they’re not totally opaque.

    1. 3

      It’s a shame to hear that Perl 6’s self-hosted parser is kinda hacky, and that the grammar engine has some deficiencies in error handling.

      It reminds me of this other link – Rebol’s parse dialect. As far as I understand, Rebol has a fixed lexer, but the parser can be reprogrammed using the parse dialect.


      This seems similar to Python’s Grammar file, which if you look carefully, uses the same lexical syntax as Python: https://docs.python.org/3/reference/grammar.html

      I noticed this while hacking on OPy for Oil. The grammar compiler pgen reuses the Python lexer (tokenize.c) for its metalanguage.

      Of course, Python doesn’t expose this parsing engine to the user – it’s an implementation detail.

      But I also want a self-hosting parser for Oil (or at least one written in a high level language), and have been looking systems doing similar things, like Perl 6 and Rebol.

      Are there any other languages which have built-in “grammars”?

      1. 4

        Are there any other languages which have built-in “grammars”?

        You might be interested in Racket. There is also a book that shows how much of the lexer and parser is under user control - Beautiful Racket.

        1. 2

          Yes I’ve read a few papers on Racket, and I know they advertise themselves as the “programmable programming language”. Or a language for writing langauges.

          That’s what I’m going for, although I think there is a difference. I don’t see anybody writing languages that look like Awk or Make [1] in Racket. They would probably just do it in something closer to s-expressions?

          I know Racket has a little syntax, but I think the philosophy is that of Lisp – to mostly avoid syntax, in favor of easier metaprogramming.

          Also, the Shill shell language [2] was apparently written in Racket, but I think they moved away from it because of some problems with the runtime.

          It’s very difficult to make a “generic language platform”. There are always some assumptions that creep in which are not valid for certain languages you didn’t think of. I think the WebAssembly guys are learning that and that is why “JavaScript should have been a VM” is sort of a facile statement.

          [1] http://www.oilshell.org/blog/2016/11/14.html

          [2] http://shill.seas.harvard.edu/

          1. 2

            It’s probably most popular to implement Lispy languages in Racket, but it does go to greater lengths than typical Lisp metaprogramming to support arbitrary syntax in a Racket #lang. The Racket authors wrote an implementation of Algol 60 to show off that support. It’s not really documented, but you can view it here.

            1. 1

              I don’t see anybody writing languages that look like Awk or Make [1] in Racket. They would probably just do it in something closer to s-expressions?

              You can find at least two such languages in the book I’ve linked to - brainfuck and basic.