Speaking of lexer modes (or stateful lexers), I took some inspiration from Pygments/Chroma (and also Oil) and added support for stateful lexing to Participle, my Go parser library.
Oh interesting, I hadn’t seen that post. I would definitely suggest adding Pygments lexers to that list. They are effectively state machines where each state is a distinct lexer, and so very much aligned with the idea of modal lexers.
I must admit that even though I wrote Chroma in 2017 (based on Pygments) I’ve had an issue open in Participle since 2018 to figure out how to support stateful lexing. It wasn’t until I read your article on modal lexers that it dawned on me that I could use the same approach from Chroma/Pygments for building general purpose lexers in Participle. Great set of articles Andy, thanks.
I’d be interested some analysis / comparison of all these different mechanisms!
Yes, that would be a really useful resource. Also missing, I’ve found, is best practice on where to draw the line between lexing and parsing, as it can often be quite blurry.
Speaking of lexer modes (or stateful lexers), I took some inspiration from Pygments/Chroma (and also Oil) and added support for stateful lexing to Participle, my Go parser library.
This allows lexing of distinct states, even recursively, for example in this string interpolation parser.
Hm interesting. I made a list of what other parser generators do here:
http://www.oilshell.org/blog/2020/07/ideas-questions.html#ad-hoc-context-in-common-lexing-tools-recursion
I’d be interested some analysis / comparison of all these different mechanisms!
Oh interesting, I hadn’t seen that post. I would definitely suggest adding Pygments lexers to that list. They are effectively state machines where each state is a distinct lexer, and so very much aligned with the idea of modal lexers.
I must admit that even though I wrote Chroma in 2017 (based on Pygments) I’ve had an issue open in Participle since 2018 to figure out how to support stateful lexing. It wasn’t until I read your article on modal lexers that it dawned on me that I could use the same approach from Chroma/Pygments for building general purpose lexers in Participle. Great set of articles Andy, thanks.
Yes, that would be a really useful resource. Also missing, I’ve found, is best practice on where to draw the line between lexing and parsing, as it can often be quite blurry.