Incidentally, hoisting the lambdas to be normal functions also massively speeds up PyPy and it becomes faster than Skybison.
This is almost certainly because they were hoisted and not because of the use of the def keyword instead of lambda. PyPy still compiles to bytecode prior to execution, just like CPython, and the dis module can be used to confirm that def and lambda forms generate identical bytecode for single-expression functions. A nice consequence is that the hoisting should therefore also help CPython slightly.
Another easy enough fix is to add __slots__ to the Value class.
Yes. This should be automatically inferred with today’s PyPy, although it doesn’t hurt to be explicit about it and should help on CPython.
Would PyPy make an effective JIT out of the tree-walking interpreter… even if it allocated all the AST nodes on the fly?
Yes, the RPython toolchain does great with AST walkers and the default garbage collector does well with GC-able ASTs. One nuance is that the ASTs – like all code objects in RPython – must be green; they must be annotated as immutable, and the RPython translator will complain if your annotations are wrong.
What if you generated Python code or bytecode? … Could PyPy compile this effectively?
Yes, although the normal caveats about eval and exec still apply, and Python’s overhead can’t be completely erased, particularly around dynamic dispatch.
This is almost certainly because they were hoisted and not because of the use of the def keyword instead of lambda.
Yes, you’re absolutely right. My writing is really not clear here (fixed). lambda and def should be identical code-wise, but creating either of them is expensive.
Thanks for the thought-provoking article! I’ve asked a question on PLDI Stack Exchange about what RPython can and can’t do. Hopefully we’ll get a good answer from somebody who knows more about this than me.
brilliant article, thank you for writing it :) I especially enjoyed the combination of how it introduces new ideas and concepts with examples that made them “click” - this post made a lot of things that I had vaguely understood fall into better places, and there were a lot of new things and ideas (writing c-code from python and then loading it in as a library! making drawings with excalidraw! a refresher on MLP, and the chain rule!) that serve as inspiration.
This was a fantastic article. I finally understood some aspects of ML that i’ve never taken the time to make sense of before. The way this was presented made it very easy for me to understand with my bg. I think this kind of programming is some of the most rewarding of all, getting that incredible speedup with such a simple technique using clear concise code is amazing.
This was so fun to read, and right up my alley as a ML n00b slash compilers enthusiast. I’ve also worked my way through Andrej Karpathy’s YT lectures recently and they are excellent.
It would be interesting to extend the compiler to support networks that aren’t static computation graphs, like RNNs. I guess the standard control flow tricks would apply, like emitting a placeholder after a condition test to patch up after the rest of the code was generated.
That’s the fun part – you probably don’t have to change the compiler. Apparently a lot of RNNs are “loop unrolled”, meaning they don’t actually involve any looping constructs. Like they just copy/paste the NN
This is almost certainly because they were hoisted and not because of the use of the
defkeyword instead oflambda. PyPy still compiles to bytecode prior to execution, just like CPython, and the dis module can be used to confirm thatdefandlambdaforms generate identical bytecode for single-expression functions. A nice consequence is that the hoisting should therefore also help CPython slightly.Yes. This should be automatically inferred with today’s PyPy, although it doesn’t hurt to be explicit about it and should help on CPython.
Yes, the RPython toolchain does great with AST walkers and the default garbage collector does well with GC-able ASTs. One nuance is that the ASTs – like all code objects in RPython – must be green; they must be annotated as immutable, and the RPython translator will complain if your annotations are wrong.
Yes, although the normal caveats about
evalandexecstill apply, and Python’s overhead can’t be completely erased, particularly around dynamic dispatch.Yes, you’re absolutely right. My writing is really not clear here (fixed).
lambdaanddefshould be identical code-wise, but creating either of them is expensive.Thanks for the comments!
Thanks for the thought-provoking article! I’ve asked a question on PLDI Stack Exchange about what RPython can and can’t do. Hopefully we’ll get a good answer from somebody who knows more about this than me.
Nice. I sent it to some folks I know and maybe they will answer.
Oh, thanks for posting :D Curious what you all think about it!
brilliant article, thank you for writing it :) I especially enjoyed the combination of how it introduces new ideas and concepts with examples that made them “click” - this post made a lot of things that I had vaguely understood fall into better places, and there were a lot of new things and ideas (writing c-code from python and then loading it in as a library! making drawings with excalidraw! a refresher on MLP, and the chain rule!) that serve as inspiration.
I’m glad!! Great to hear
This was a fantastic article. I finally understood some aspects of ML that i’ve never taken the time to make sense of before. The way this was presented made it very easy for me to understand with my bg. I think this kind of programming is some of the most rewarding of all, getting that incredible speedup with such a simple technique using clear concise code is amazing.
This is such high praise. Thank you.
This was so fun to read, and right up my alley as a ML n00b slash compilers enthusiast. I’ve also worked my way through Andrej Karpathy’s YT lectures recently and they are excellent.
It would be interesting to extend the compiler to support networks that aren’t static computation graphs, like RNNs. I guess the standard control flow tricks would apply, like emitting a placeholder after a condition test to patch up after the rest of the code was generated.
That’s the fun part – you probably don’t have to change the compiler. Apparently a lot of RNNs are “loop unrolled”, meaning they don’t actually involve any looping constructs. Like they just copy/paste the NN
Oops, you mentioned that in your footnotes, missed that - interesting, thanks!
Goddammit I thought this was going to be about ML as in O/Ca/ML.
Same, TBH. Still want to read it though.