Not a huge win, true, but if you can make the top-100 Python libraries all run 30-50% faster the impact would be significant overall, given the scale of Python deployments.
I’d like to see some do the @benhoyt benchmark challenge with this. I feel like that’s the only microbenchmark I’ve seen where the unit of work is realistic enough to be worth comparing.
Mypyc is, of course, used or mypy and, I believe now, black.
In 2021 I had tried to compile some stuff with mypyc and hit several nasty bugs with it, though. It was application code, not library code, but it was still a bit of an indicator to me that I need to be careful.
I think it’s a very good, pragmatic tool. Ultimately I think that the “real” thing that would make sense is … for libraries to be written in systems languages when reasonable. Think about how much stuff goes into an attribute access in python… and then imagine your editor running your Python buffers through that all the time. But this is a great stopgap, and it will only get better IMO.
for libraries to be written in systems languages when reasonable
Agreed. But we need something richer than C for defining memory-safe APIs with a stable ABI and auto-generated language bindings. I think Microsoft was onto something with COM, but of course it wasn’t perfect, and when they tried to improve on it with WinRT, the industry at large mostly didn’t pay attention.
I’m particularly interested in what the Rust community is starting to do in this area. Rust’s metaprogramming is powerful enough that, at least in theory, one can define a rich API in some subset of Rust, perhaps with the addition of some macros and/or attributes, then auto-generate language bindings from that.
Via Twitter, someone tried a more realistic bit of code, got a nice speedup from 1.2s to 0.8s: https://gist.github.com/llimllib/11d4707e70738db05ee5843a2b81a35c
Not a huge win, true, but if you can make the top-100 Python libraries all run 30-50% faster the impact would be significant overall, given the scale of Python deployments.
I’d like to see some do the @benhoyt benchmark challenge with this. I feel like that’s the only microbenchmark I’ve seen where the unit of work is realistic enough to be worth comparing.
How well does mypyc work? I admit I’m put off by the docs page that say “Mypyc is currently alpha software.” and also “© Copyright 2020”.
Seems reasonably active. The copyright was probably when that file was first written.
Mypyc is, of course, used or mypy and, I believe now, black.
In 2021 I had tried to compile some stuff with mypyc and hit several nasty bugs with it, though. It was application code, not library code, but it was still a bit of an indicator to me that I need to be careful.
I think it’s a very good, pragmatic tool. Ultimately I think that the “real” thing that would make sense is … for libraries to be written in systems languages when reasonable. Think about how much stuff goes into an attribute access in python… and then imagine your editor running your Python buffers through that all the time. But this is a great stopgap, and it will only get better IMO.
Agreed. But we need something richer than C for defining memory-safe APIs with a stable ABI and auto-generated language bindings. I think Microsoft was onto something with COM, but of course it wasn’t perfect, and when they tried to improve on it with WinRT, the industry at large mostly didn’t pay attention.
I’m particularly interested in what the Rust community is starting to do in this area. Rust’s metaprogramming is powerful enough that, at least in theory, one can define a rich API in some subset of Rust, perhaps with the addition of some macros and/or attributes, then auto-generate language bindings from that.