No one looked at Common Lisp’s macros and said “What if these macros aren’t dangerous enough? What if we could make them even harder to write correctly, in order to marginally increase their power and expressiveness?”
if you had left and right of every node up the ancestor chain, instead of only the current node, you’d have a zipper over the syntax tree.
then, you could edit the entire syntax tree, and also change the “focus” element of what to macroexpand next. in case you wanted something more powerful/dangrous
This was my original idea! When I was thinking about more powerful macros I wanted to write. I think it would totally work, and make some transformations a lot easier, but I was very excited about the simplicity of the first-class macro approach – the only useful macros I’ve come up with involve very local changes where the slightly weird interface doesn’t matter too much. But a zipper interface would make “distant” transformations much easier.
It occurred to me while reading this that equality saturation (https://egraphs-good.github.io/) might be the missing piece that allows generalized macros to compose. A macro implemented with equality saturation could see every expansion step of its neighbors and rewrite based on the specific one(s) it’s looking for.
Whoa, that’s a really interesting idea! I’m not really sure how you’d decide on the “optimal” rewrite – I guess macros would include, in their expansions, how “specific” that expansion is? Or something like that? Definitely something to think about.
When writing macros within Scheme’s syntax-case model, they’re expressed as case-style pattern-matchers over syntax-objects (which themselves appear ideal for translation into e-nodes). In that context, I would posit that optimal extractions from saturated graphs are those which fulfill the earliest possible matches. Hence a match on a left-most set literal would 1) take precedence over the no-match case, 2) can be applied after the test macro expands, and 3) could maybe even propogate transformations of sub-nodes into equivalent expansions where those literals have been eliminated (or not yet expanded into being).
Implementing such a system would be difficult (let alone in Janet without an existing syntax-case to fork), and although I think it addresses the settable example as-given, it’s a rough model. There are still ambiguous cases where one would presumably fall-back into depth-first expansion.
The biggest problem is with that 3rd part, which is kinda out-there. Macarons are effectively expansions of their parent expressions, so they can’t actually contribute transformations of themselves or their sibling arguments that are seperable from those parent expansions. Putting that aside (maybe by annotating with source syntax objects / equiv. e-nodes when preserving the transformation would be valid), it would feel kinda cursed to allow a match on a literal which might only exist in superposition (don’t let reason stop you :p).
I guess 2/3 with the fall-back caveat ain’t too bad, but disclaimer: this is way over my head, i hardly grok nondeterminism and look at this this with the same awestruck unfamiliarity as µKanren, which i also don’t know nothin about
Fan of the post and ideas presented for sure. I’ve come up with ideas in the past to make automatic “infix” things, but it was never generalized. The transformation was (x .+ y) always would translate to (+ x y) – this is similar to the classic pair shorthand (a . d) -> (cons a d). So the rule was if you saw a .foo drop the dot, and swap. I only ever thought of this as for functions, but if you consider identifier macros you now have the potential for a generalized “infix-transformer” (that might also be hygienic-able!). plus if you create a precedence rule, they could compose….
(1 + 2 + 3 + 4) ->
(+ '(1) '(2 + 3 + 4)) ->
+ is responsible for further expanding the left and right, and in that regard, it’s more like an F-Expression, really, but always left-associative, kind of preserving Lisp’s head of list is operator rules.
Also: I’m still so bothered by unquote-splicing being ; in Janet. I can’t look past it not being a comment and it throws me every time. It’s single-handedly the thing that prevents me from trying Janet… which yes, is kind of dumb, I know.
This is super cool. I read the source and I’m a little confused how it works. I don’t know much Janet but it seems like there’s nothing that passively captures the surrounding context. How does it work?
I’m not really sure what you mean by “passively captures” so I might be responding to a different question than you’re asking.
The surrounding context is really only “lefts” and “rights.” macaronex1 iterates over each element in a form (where “form” just means “things surrounded by parentheses”), and expands that element. Then it checks if it expanded into a first-class macro: if it does, then it calls that macro with a list of all of the previously expanded elements as the “lefts” and everything that it hasn’t gotten to yet as the “rights,” and returns whatever that macro returned. If it didn’t expand into a first-class macro, then it adds the expanded form into the “lefts” that it’s accumulating.
If it reaches the end of the form without encountering a first-class macro, then the “accumulated lefts” becomes the new result of the expansion.
Once macaronex1 completes, macaronex checks if the expanded form is equal to the original form. If the expansion was a no-op, then you’re done – that form is fully expanded. If the expansion produced a different form, then it repeats the process, and keeps calling macaronex1 until there’s nothing left to expand.
Does that… does that make sense? Did that answer your question?
Sorry, “passively captures” wasn’t a great phrase.
I mean that while a normal macro has access to all of the forms within its parentheses, macaroni have access to the forms to the left and right as well, and those presumably have to be made available by some other macro or function call. I don’t see a mechanism that changes the janet reader to provide the enclosing form (aka given (some-func a b (some-macaron …) c d) some-macaron gets the whole thing), to the macaron invocation.
There’s the defmacro macaroni at the end of init.janet but I couldn’t find it being called anywhere.
So in order to actually use this as your default module system you’d have to write a custom module loader in Janet that provides macaronex as a custom macro expander (for example). But there is no such module loader in my experiment repo – I just wrote the expander.
Ah okay, that makes a lot more sense. Custom macro expanders are wild. How does run-context work? Never mind, I found it in the janet docs. My initial search must have been incorrect somehow.
I’m asking because I want to implement shenanigans like this in Clojure which is my preferred lisp and need all the help I can get ;)
I love this already
if you had left and right of every node up the ancestor chain, instead of only the current node, you’d have a zipper over the syntax tree.
then, you could edit the entire syntax tree, and also change the “focus” element of what to macroexpand next. in case you wanted something more powerful/dangrous
This was my original idea! When I was thinking about more powerful macros I wanted to write. I think it would totally work, and make some transformations a lot easier, but I was very excited about the simplicity of the first-class macro approach – the only useful macros I’ve come up with involve very local changes where the slightly weird interface doesn’t matter too much. But a zipper interface would make “distant” transformations much easier.
It occurred to me while reading this that equality saturation (https://egraphs-good.github.io/) might be the missing piece that allows generalized macros to compose. A macro implemented with equality saturation could see every expansion step of its neighbors and rewrite based on the specific one(s) it’s looking for.
Whoa, that’s a really interesting idea! I’m not really sure how you’d decide on the “optimal” rewrite – I guess macros would include, in their expansions, how “specific” that expansion is? Or something like that? Definitely something to think about.
When writing macros within Scheme’s
syntax-casemodel, they’re expressed as case-style pattern-matchers over syntax-objects (which themselves appear ideal for translation into e-nodes). In that context, I would posit that optimal extractions from saturated graphs are those which fulfill the earliest possible matches. Hence a match on a left-mostsetliteral would 1) take precedence over the no-match case, 2) can be applied after the test macro expands, and 3) could maybe even propogate transformations of sub-nodes into equivalent expansions where those literals have been eliminated (or not yet expanded into being).Implementing such a system would be difficult (let alone in Janet without an existing
syntax-caseto fork), and although I think it addresses the settable example as-given, it’s a rough model. There are still ambiguous cases where one would presumably fall-back into depth-first expansion.The biggest problem is with that 3rd part, which is kinda out-there. Macarons are effectively expansions of their parent expressions, so they can’t actually contribute transformations of themselves or their sibling arguments that are seperable from those parent expansions. Putting that aside (maybe by annotating with source syntax objects / equiv. e-nodes when preserving the transformation would be valid), it would feel kinda cursed to allow a match on a literal which might only exist in superposition (don’t let reason stop you :p).
I guess 2/3 with the fall-back caveat ain’t too bad, but disclaimer: this is way over my head, i hardly grok nondeterminism and look at this this with the same awestruck unfamiliarity as µKanren, which i also don’t know nothin about
Fan of the post and ideas presented for sure. I’ve come up with ideas in the past to make automatic “infix” things, but it was never generalized. The transformation was
(x .+ y)always would translate to(+ x y)– this is similar to the classic pair shorthand(a . d)->(cons a d). So the rule was if you saw a.foodrop the dot, and swap. I only ever thought of this as for functions, but if you consider identifier macros you now have the potential for a generalized “infix-transformer” (that might also be hygienic-able!). plus if you create a precedence rule, they could compose….+is responsible for further expanding the left and right, and in that regard, it’s more like an F-Expression, really, but always left-associative, kind of preserving Lisp’s head of list is operator rules.Also: I’m still so bothered by unquote-splicing being
;in Janet. I can’t look past it not being a comment and it throws me every time. It’s single-handedly the thing that prevents me from trying Janet… which yes, is kind of dumb, I know.This is super cool. I read the source and I’m a little confused how it works. I don’t know much Janet but it seems like there’s nothing that passively captures the surrounding context. How does it work?
I’m not really sure what you mean by “passively captures” so I might be responding to a different question than you’re asking.
The surrounding context is really only “lefts” and “rights.”
macaronex1iterates over each element in a form (where “form” just means “things surrounded by parentheses”), and expands that element. Then it checks if it expanded into a first-class macro: if it does, then it calls that macro with a list of all of the previously expanded elements as the “lefts” and everything that it hasn’t gotten to yet as the “rights,” and returns whatever that macro returned. If it didn’t expand into a first-class macro, then it adds the expanded form into the “lefts” that it’s accumulating.If it reaches the end of the form without encountering a first-class macro, then the “accumulated lefts” becomes the new result of the expansion.
Once
macaronex1completes,macaronexchecks if the expanded form is equal to the original form. If the expansion was a no-op, then you’re done – that form is fully expanded. If the expansion produced a different form, then it repeats the process, and keeps callingmacaronex1until there’s nothing left to expand.Does that… does that make sense? Did that answer your question?
Sorry, “passively captures” wasn’t a great phrase.
I mean that while a normal macro has access to all of the forms within its parentheses, macaroni have access to the forms to the left and right as well, and those presumably have to be made available by some other macro or function call. I don’t see a mechanism that changes the janet reader to provide the enclosing form (aka given
(some-func a b (some-macaron …) c d)some-macaron gets the whole thing), to the macaron invocation.There’s the defmacro macaroni at the end of init.janet but I couldn’t find it being called anywhere.
Oh gotcha! Yeah totally misunderstood.
So in order to actually use this as your default module system you’d have to write a custom module loader in Janet that provides
macaronexas a custom macro expander (for example). But there is no such module loader in my experiment repo – I just wrote the expander.Ah okay, that makes a lot more sense. Custom macro expanders are wild.
How doesNever mind, I found it in the janet docs. My initial search must have been incorrect somehow.run-contextwork?I’m asking because I want to implement shenanigans like this in Clojure which is my preferred lisp and need all the help I can get ;)