1. 41

  2. 14

    Disclaimer, I’m the least important Python user of all time, by my own estimation. However, I have read the PEP in its entirety before participating in discussion, something users at the orange site might want to take a page from my book about.

    I think this is a great starting place for syntactic macros, and I do agree with the author that there does exist a place in the Python universe for implementing things with macros.

    I dont think using Lisp as inspiration is a very wise idea, as it only serves to confound and intimidate. Python macros seem to be tools for a slightly different level of sophistication. They’re not interested in adjusting the syntax of Python per se, but merely adjoining statements in Python source code that would’ve been previously impossible. This is something that Lisp doesn’t do very well. Lisp users are often quick to create their own little universes, where there’s new core logical constructs and expressions to build programs with.

    638 macros are seemingly designed for a smaller purpose, and I think that so far, that’s their biggest strength.

    I’m not in love with the syntax. I think postfix operators kinda add noise since there’s already colons (:) everywhere in python, I think exclamation points are a bit too loud.

    I know it’d never get pushed through, but I think a leading squiggle (~) would be more readable.

    from! ast_matcher import match
    def calculate(node):
        if isinstance(node, Num):
            return node.n
        match! node:
            case! a + b:
                return calculate(a) + calculate(b)
            case! a - b:
                return calculate(a) - calculate(b)
            case! a * b:
                return calculate(a) * calculate(b)
            case! a / b:
                return calculate(a) / calculate(b)

    would be:

    ~from ast_matcher import match
    def calculate(node):
        if isinstance(node, Num):
            return node.n
        ~match node:
            ~case a + b:
                return calculate(a) + calculate(b)
            ~case a - b:
                return calculate(a) - calculate(b)
            ~case a * b:
                return calculate(a) * calculate(b)
            ~case a / b:
                return calculate(a) / calculate(b)

    which looks a lot nicer to me, and syntax highlights and ligatures could make these really pleasing to look at, I think.

    All in all, I think some of the points could use refining. One thing that goes unaddressed here is startup performance. I think Python’s startup time is pretty important, it is for the way that I use Python anyways, and in some cases it’s even more important than the overall performance characteristics of a program. If I can get bits in and out in few ms than a perl program then it’s often okay if the perl program completes quicker because I might have a long pipeline of programs, and the savings from starting up fast can outweigh performance issues of individual components. Maybe it’s just me. Either way, if I pull in simple libraries and I have to start paying time for macros to compile before my first line of python gets executed, that’s going to affect my bottom line in terms of where I can afford to use python. The pep just mentions that source->bytecode is negligible in most applications. And that’s true, but if we get to a position where macros are being expanded to get the core interpreter up in a console, that’s where it becomes non-negligible. I don’t want to pay those ms, I need that first line to run much more than I need fancy statements.

    Still, I hope this PEP gets a lot more attention and thoughtful discussion. The thought of having an elegant pattern matching macro alone is nearly worth the effort, much less trialing new features for free without disturbing those who work on the bytecode and compiler technologies.

    1. 8

      I think this proposal is also influenced by Julia. It don’t think bikeshedding on the syntax is as important as asking if this is a feature users want at all.

      Due to the dynamic nature of python you can already do a lot of this stuff by introspecting on function objects. But culturally it is not something popular in the community.

      For example here are two different ways to make what is effectively a macro right now:

      import inspect
      import ast
      class BindWrapper(ast.NodeTransformer):
          """Change bind to assignment"""
          def visit_Expr(self, node):
              if isinstance(node.value, ast.Compare):
                  comp = node.value
                  if isinstance(comp.ops[0], ast.Lt):
                      arg = comp.comparators[0]
                      if isinstance(arg.op, ast.Invert):
                          name = ast.Name(comp.left.id, ctx=ast.Store())
                          a = ast.Assign(targets=[name],
                          return a
              return node
      def transform_with_ast(f):
          s = inspect.getsource(f)
          s = "\n".join(s.split('\n')[1:])
          expr = ast.parse(s)
          expr = BindWrapper().visit(expr)
          namespace = {}
          exec(compile(expr, "<ast>", 'exec'), namespace)
          return namespace[f.__name__]
      def transform_with_str(f):
          s = inspect.getsource(f)
          s = "\n".join(s.split('\n')[1:])
          s = s.replace("<~", "=")
          namespace = {}
          exec(compile(s, "<ast>", 'exec'), namespace)
          return namespace[f.__name__]
      def foo(x):
          y <~ 3
          return x + y
      def bar(x):
          y <~ 3
          return x + y
      1. 3

        I think the PEP makes it clear enough that this isn’t necessarily a feature of Python that is meant to be front-and-center of a lot of applications.

        Macros should be treated like a loaded gun, carefully and respectfully, and with consideration of others nearby. In that same vein, I think Python could benefit from these capabilities, and it’s not that these things couldn’t be done before, but rather that it’s a lot easier for the macro practitioners to use them now.

        I would hate to see this pep go through and then see a lot of users trying to squeeze macros into their programs all over the place.

        This feature has a particular niche demand in writing highly scientific code where the domain is very very rich, and it’d be nicer to write code at a level more abstracted than regular python statements. That’d be huge for productivity.

        Also, FWIW, Julia macros are basically identical to Lisp macros, since Julia syntax is homoiconic. I can appreciate the parallel since Julia syntax is somewhat reminiscent of Python’s, but the implementation of each macro system couldn’t be any more different, really.

        1. 4

          Those are fair points. Let me try to rephrase my main point. I think there is no reason Python shouldn’t have a good macro system and there have definitely been APIs I think which would have been made cleaner with the facility like panda’s query function or numpy’s einsum

          My impression though is the Python community doesn’t seem particularly keen to have this feature. I hope I’m wrong since judiciously applied it could make for some less ugly code in a few places.

          1. 4

            I haven’t have time to respond justly, but I enjoyed your clarification. I think you’re absolutely correct.

      2. 4

        Either syntax looks fine to me. The exclamation point always has something directly after it so it feels more infix-y then postfix-y when I look at it. Tilde might be ruled out by the existing user of it for bitwise negation. No problem for statements but a biggie for expressions.

        I like that the definition of MACRO_NAME says they the exclamation mark has to be part of the identifier, never separate from it by whitespace or anything. That should cut down on parsing issues a lot, I think? cuz you can distinguish a MACRO_NAME token as early as the lexer, way before parsing proper.

        On the trialling new features part of the rationale, I do wonder if maybe Python would’ve had a much nicer/easier to explain semantics for things like async if the Twisted / Tornado / etc groups had been able to try several iterations of it in library code first before it got into the interpreter. Alas.

        Slightly interesting note about things like try/finally having had a not-strictly-necessary bytecode added for it when it was introduced. A lot of the time the speed at which Python runs has historically been dictated by one single giant switch statement in ceval.c. At I’ve point I saw updating gcc version double the rate at which my python code ran. It seemee that older versions of gcc had trouble generating good code for such s large and demanding function.

        The only aspect that I’m not 110% enthusiastic about is that macro evaluation may make python programs’ start up time even worse because of the metaprogramming. But Python already has this problem: AIUI code like class Foo(object): def Bar(): ... currently has to execute bytecode at import time. I suspect that solving it for things like class, method and function definitions would probably solve it for macros too.

        1. 3

          Small note, using ~ for a prefix notation is impossible as it is already used for unary inversion (binary NOT).

          1. 3

            yeah, I had forgotten about that. I suppose the postfix notation they’ve chosen is just ugly to me. I’m not attached to the squiggle, nor would the PEP team find my opinions worthwhile anyhow.

            You’re totally right, it couldn’t be done the way I sketched it.

            1. 2

              It is theoretically possible for the parser to assume that a macro is being used before assuming unary inversion, though.

              1. 1

                To clarify, are you meaning by the means of this PEP, via the compiler step of handling macros? Or is there another way? I’m not very educated about the ast module and what it can be forced to do.

          2. 3

            Thank you for posting this and so soon. How are convo starts is how it ends. I noticed almost everyone in the thread on the orange site didn’t even read the article and just had an axe to grind against macros in general w/o looking at the problems they are trying to solve.

            I have a case where I would like something like a Ruby block, maybe I can use a try/finally block, but Ruby block would be cleaner.

            One thing that would be interesting if with this macro system existing, how many other Python features could be implemented in-band vs as a special form?

          3. 8

            This is a decently written proposal, but to be frank, I don’t like it.

            First off the examples seem sort of contrived to me, and generally obfuscate the intent of the code. If try/finally were a macro rather than a language construct, I no longer know that some library I’ve imported hasn’t mucked with it. I think the bijection! Example is strictly worse than:

            Color_by_num = {…} Num_by_color = {v: k for k,v in color_by_num.items()}

            Because I don’t have to read the source/docs to a macro (and think of it in terms of ast nodes, which is a context switch). I’ve seen too many functions that don’t do what they’re called or have empty documentation to just trust that “bijection!” does what I mean. Sure, maybe I’m interacting with low quality code, but at least it’s readable. I don’t like go for unrelated reasons, so I’m not an expert but I think it does very well in this respect: It’s difficult to hide mistakes or laziness behind deep abstraction and fancy syntax, and up until reasonably recently, I felt the same way about python.

            The ast module provides the ability to compile whole files using python syntax and whatever semantics one would like (I’m not particularly proud of this, but I have done it) I’m not familiar with numba but using inspect to do roughly the same thing is a bit squicky imo, I don’t like it when languages suddenly become other languages in the middle of a file, and if that’s what one really wants, inspect, ast, and decorators can do it already without a ton of hassle.

            I’ve been a primarily python dev for almost a decade now and I’ve never thought “I wish I had a macro system”. I have frequently thought “thank heaven whoever wrote this had to abide by the rather stringent and regular rules of python, otherwise there’s no way I could understand it”. I’ve been trying to learn rust recently, and in my head, ! translates to roughly ‘give up trying to understand this expression, it’s magic and it ain’t gonna explain sh*t’ Rust is statically typed and dispatched though, so the things you can do with ast and inspect in python are by necessity impossible, so it needs another way to generate code.

            1. 3

              I don’t think the PEP makes clear enough that you are not the target user for 638 macros. It seems that they’re meant for quality-of-life improvements for specific niches of Python, particularly scientific programming. By your mention of not being familiar with Numba, I made the assumption that you’d never use 638 macros for your work in Python, and that’s a good thing.

              The examples in the PEP are illustrative not necessarily to the areas in which macros should be reached for, but for the mechanics of problems that they can solve. I think that wasn’t made very clear either in the PEP, tbh.

              Your frustration with Rust is exactly why I think this PEP needs more work to clarify it’s purpose and the role it should play in the Python ecosystem, and in the core ecosystem. I don’t think the majority of users of Python are going to interact with 638 macros directly if they’re used appropriately.

              I like your insight, because it’s so true that ast has been around for a long time, and that this isn’t a new idea to add it to python. Still, I think doing it this new way would be a safe way to find out if the benefits are worth it, without disrupting the majority of users who won’t ever notice them.

              1. 1

                True but in my experience, “am I the arget audience for this thing” isn’t related to “does this thing show up in code I have to use/maintain”. People love shiny things, and will often use them just because they can and it’s fun. Part of why I like python is that it lacks shiny things, not because I don’t want them, but because others can’t have them, which makes dependencies easier to audit and use.

                I don’t trust my fellow man to not use it when it’s not necessary. Mucking with the meaning of syntax is dark magic that is currently only possible using ast and inspect, but is not simple, and how to do it doesn’t show up in any introductory materials. therefore it tends not to be done unless absolutely necessary, and sometimes is worked around by people who don’t know it’s possible with boilerplate. I think this is a language feature in itself, and that making it more accessible is just going to make my job harder.

                1. 1

                  Well, that’s a very fair point. I agree that I don’t particularly trust the average user, and I stress again that this feature isn’t intended for average users. I think the capability is a needed feature in the SciPy community and in other niches, but I don’t expect a large number of people to ever actually interact with developing a macro. Hopefully. I resonate with your concern.

                  However, I think that with a distinct enough syntax, it would also be very obvious to ANY user that a macro-statement is different in nature, and hopefully that would prompt them to search for it.

            2. 6

              Been trying to wrap my head around this PEP for a bit. It feels like yet another step down the slippery slope of robbing my favorite programming language of the un-clever simplicity that got me excited about it to begin with.

              1. 8

                Python is not simple, it looks simple.

                The community is what drives a language in a specific direction. You can code Python in any style imaginable, same as Java or C. But each one has an idiomatic (should be called norms not idioms, but here we are) way of doing things that is acceptable to the community.

                It is already possible to circuit bend Python in wonderful wonerful ways

                1. 4

                  This is right. I’d argue the culture of simplicity has been slipping now away slowly for awhile now. Thinking of some of the controversial 2-3 changes here, and the optional gradual typing.

                  Syntactically the grammar is simple-ish for a mainstream, syntax heavy language. It’s admirable they tried to hew to LL. But underneath is not quite so simple.

                  1. 2

                    Isn’t coconut just a language which happens to compile to python? (Granted, as a result of that, it is able to take advantage of python’s library ecosystem; but that seems no different to any JVM language.)

                    1. 2

                      I take your point. The language is extremely flexible and has any number of pointy bits you can cerate yourself on.

                      BUT as a new Python programmer it used to present a simple straight forward mental model and I’m concerned that’s slipping away.

                  2. 5

                    The idea in the Motivations section seems reasonable: add one big extension point to avoid a thousand domain-specific syntax changes. The rest mostly just illustrates a proposed syntax, which isn’t enough to flesh out that idea.

                    It’d be interesting to drill down into whether the set of hacks for quickly evaluating numerical Python here (Cython, numba, and pandas.eval) could be made better to use, better to implement, or more smoothly interoperable with normally-evaluated Python with a proposal like this. Looks like the author worked on a JIT project (HotPy) so probably has some ideas in mind, but they aren’t spelled out in the PEP.

                    It’d also be interesting to compare it to another way people build DSLs in Python: lots of runtime metaprogramming. (A lot of tricky stuff is going on as a large Django app compiles!) It’s not totally clear it’d work out this way, but if you could take a task like describing a schema and show how you could replace a pile of __magical_methods__ and other dynamic arcana with more straightforward code generation, that would be cool.

                    I might be wrong but this seems like a walrus-operator-level difficult sell to the Python community, in that macros aren’t obvious in the fashion Python typically prides itself on. Doesn’t mean it’s not worth proposing, just that you really have to flesh out the case for it if you’re going to get much traction. I don’t think this draft has done that quite yet.