YJIT’s ability to improve performance by adapting to run-time behavior can increase the speed of our code in a way that dropping down to C can’t.
This certainly can be true in general, but I don’t think they’ve demonstrated it here. All they’ve demonstrated is that, due to impedance mismatch, a program partly in c and partly in ruby runs slower than one written all in ruby. (Which, to be clear, is still a worthwhile and useful conclusion, and it’s all thanks to the great work that’s been done recently on ruby compilers.)
We can’t load the pure Ruby implementation of Racc and the C implementation at the same time, so we’ll have to compare this in different runs.
I have a not-so-hidden feature in derailed benchmarks for this exact case. Beyond getting the raw numbers it also gives a Kolmogorov Smirnov Test result that tells you if your measurement is likely the result of noise or not as well as ascii histograms to help gut check the results and distribution.
You can use it to compare different git shas in a project or with different libraries or use the internal (but stable) APIs to bench just about anything that can be spawn-d.
we achieve 45 iterations per second, 10% faster than the C implementation of Racc!
Without a bunch of runs and some good solid statistical analysis 10% could easily be noise (or the best of one and the worst of another). Aaron is a reputable guy and I don’t think he would make a claim if he didn’t believe in it. I like statistical checks because they (help) take belief out of the equation.
I hope that this post was able to give folks a closer look into how a JIT compiler can outperform a C extension, and more importantly, encourage people to write more Ruby
I’m all for that. I like the idea of getting the JIT so good that it could be on by default and rewriting stdlib from C to Ruby.
I think it’s a bit different, it’s ‘my language’s FFI is sufficiently slow that the performance gain from implementing something in a different language is eaten completely unless you carefully design interfaces to minimise domain transitions’.
Even worse: “the non-standard JIT for my language interacts poorly with the FFI for my language, because the FFI and the C code written against it assumes the interpreter”.
The first time I saw this was in a special issue of the IBM Systems Journal on Java Technology. The whole issue was fully of “how our super duper JIT is faster than C”. On specialised benchmarks when you do lots of iterations and don’t count the warmup time.
And then they had an experience report from the San Francisco Project. Which told a slightly different and more, er, “majestic” story.
This certainly can be true in general, but I don’t think they’ve demonstrated it here. All they’ve demonstrated is that, due to impedance mismatch, a program partly in c and partly in ruby runs slower than one written all in ruby. (Which, to be clear, is still a worthwhile and useful conclusion, and it’s all thanks to the great work that’s been done recently on ruby compilers.)
Having the stdlib written in ruby would do wonders for having new contributors.
If you’ve never heard of Rubinius, it’s worth checking out for exactly this reason.
You rang?
🙌
I have a not-so-hidden feature in derailed benchmarks for this exact case. Beyond getting the raw numbers it also gives a Kolmogorov Smirnov Test result that tells you if your measurement is likely the result of noise or not as well as ascii histograms to help gut check the results and distribution.
You can use it to compare different git shas in a project or with different libraries or use the internal (but stable) APIs to bench just about anything that can be spawn-d.
Without a bunch of runs and some good solid statistical analysis 10% could easily be noise (or the best of one and the worst of another). Aaron is a reputable guy and I don’t think he would make a claim if he didn’t believe in it. I like statistical checks because they (help) take belief out of the equation.
I’m all for that. I like the idea of getting the JIT so good that it could be on by default and rewriting stdlib from C to Ruby.
Great article as usual.
Feels like I’ve seen variants of this post a number of times since my Java days in latest 90s. It’s some version of:
“Sure my most beloved language is a tad slow, but with JIT and if you squint there might be some very specific cases where it’s faster”
I think it’s a bit different, it’s ‘my language’s FFI is sufficiently slow that the performance gain from implementing something in a different language is eaten completely unless you carefully design interfaces to minimise domain transitions’.
Even worse: “the non-standard JIT for my language interacts poorly with the FFI for my language, because the FFI and the C code written against it assumes the interpreter”.
YJIT has been merged into Ruby proper.
The first time I saw this was in a special issue of the IBM Systems Journal on Java Technology. The whole issue was fully of “how our super duper JIT is faster than C”. On specialised benchmarks when you do lots of iterations and don’t count the warmup time.
And then they had an experience report from the San Francisco Project. Which told a slightly different and more, er, “majestic” story.
See also: Jitterdämmerung
I found that part much more concerning.