You did not reverse the list, and for some reason the first measurement of timer.tc can often be off. Plus, it might be that garbage collection triggered while benchmarking map_body. Benchee measures multiple runs and runs garbage collection in between. Might also be something else, though - as the erlang page mentions architecture can also have an impact.
In the shell it indeed seems to behave like mapbody is slower on the first run (at least with the input list you used, 1000000 elements, I ran the benchmark with 10000 elements)
That might be it. Benchee runs a warmup of 2 seconds where it doesn’t measure (to simulate a warm/runnign system) and then takes measurements for 5 seconds so that we have lots of data points.
Might also still be something with elixir and/or hardware :) Maybe I should retry it with Erlang, but not this morning :)
Not sure what exactly you mean, the original benchmark was run compiled not in the shell. This here was just done for comparison with the reported erlang benchmarks - my erlang_fu (which is not very existent at this point in time) I currently fail to do that and don’t have the time to look it up atm :)
Ok, not like an elixir script executable like I’d want to but I wrote a benchmark function that benchmarks it and then called that in the shell - good enough for now I guess. There it is all much faster, and map_body seems to be about as fast as the non reversed tco version or even faster. I’d still need a proper benchmark to determine it all though.
Ok, I ran your code in erlang and I also get consistently faster results for the TCO version. I don’t get it, it is the same function I wrote in Elixir. The interesting thing for me, comparing Erlang and Elixir is, that with the same list size and what I think are equivalent implementations map_body seems to be much slower in Erlang. E,g, compare the numbers here to the other post where I do the same in Elixir and iex. In Elixir map_body settles at around 490k microseconds, the erlang version is between 904k microseconds and 1500k microseconds.
I’m probably just really dense, but it isn’t obvious to me why the reverse lists stuff is being used. Is it just to deal with the fact that the TCO versions create a swapped list because of the way they’re written?
I’m also curious if there are any good ways of comparing the compiled reductions/bytecode from the different methods–in C you’d just look at the assembly and clearly see what the compiler thought it was doing.
Thank you for introducing me to the benchmarking tool! I’ll be using that in my refresh of my math library.
It is always possible. Benchee or something Benchee uses might have a bug. I don’t think so, Benchee is well tested and rather simple. Plus you can reproduce the same with running :timer.tc a few times in a shell. Before I wrote benchee I also used Benchfella - results are similar much identical:
## MyMapBench
[06:38:12] 1/3: map with TCO reverse
[06:38:15] 2/3: map with TCO and ++
[06:38:17] 3/3: map simple without TCO
Finished in 6.26 seconds
## MyMapBench
map simple without TCO 10000 164.55 µs/op
map with TCO reverse 10000 211.55 µs/op
map with TCO and ++ 10 188430.40 µs/op
Did the same double check with flat_map back then. Plus I rather blog about unusual findings, other benchmarks are pretty much what you’d expect them to be (recursion is fastest for repeating something n times, sorting 100k elements is ~15 times slower than sorting 10k elements etc.) There are some more samples in benchees samples folder :)
Plus especially for this I sat down and read a lot (the Erlang performance myths etc.) as I also found it highly unlikely and needed to see that this discussion has been had and explained before to be courageous enough to blog about it in an environment that I’m not super familiar with (yet)
When I tested this myself the tail recursive version was substantially faster.
code
in the erlang shell
You did not reverse the list, and for some reason the first measurement of timer.tc can often be off. Plus, it might be that garbage collection triggered while benchmarking map_body. Benchee measures multiple runs and runs garbage collection in between. Might also be something else, though - as the erlang page mentions architecture can also have an impact.
Here it is with also reversing it after and its still faster. There is a consistent 2 second difference, this is not random fluctuations.
In the shell it indeed seems to behave like mapbody is slower on the first run (at least with the input list you used, 1000000 elements, I ran the benchmark with 10000 elements)
However, running it more often in the same iex session map_body gets faster:
That might be it. Benchee runs a warmup of 2 seconds where it doesn’t measure (to simulate a warm/runnign system) and then takes measurements for 5 seconds so that we have lots of data points.
Might also still be something with elixir and/or hardware :) Maybe I should retry it with Erlang, but not this morning :)
Never benchmark in the shell, it’s an interpreter. Compile the module with the benchmarker included and run that.
Not sure what exactly you mean, the original benchmark was run compiled not in the shell. This here was just done for comparison with the reported erlang benchmarks - my erlang_fu (which is not very existent at this point in time) I currently fail to do that and don’t have the time to look it up atm :)
Ok, not like an elixir script executable like I’d want to but I wrote a benchmark function that benchmarks it and then called that in the shell - good enough for now I guess. There it is all much faster, and map_body seems to be about as fast as the non reversed tco version or even faster. I’d still need a proper benchmark to determine it all though.
code
Ok, I ran your code in erlang and I also get consistently faster results for the TCO version. I don’t get it, it is the same function I wrote in Elixir. The interesting thing for me, comparing Erlang and Elixir is, that with the same list size and what I think are equivalent implementations map_body seems to be much slower in Erlang. E,g, compare the numbers here to the other post where I do the same in Elixir and iex. In Elixir map_body settles at around 490k microseconds, the erlang version is between 904k microseconds and 1500k microseconds.
I reran your benchmark and got similar results (consistently over 10x runs) that is around 35-40% faster:
Does it have something to do with the fact he used Elixir?
I’m probably just really dense, but it isn’t obvious to me why the reverse lists stuff is being used. Is it just to deal with the fact that the TCO versions create a swapped list because of the way they’re written?
I’m also curious if there are any good ways of comparing the compiled reductions/bytecode from the different methods–in C you’d just look at the assembly and clearly see what the compiler thought it was doing.
Thank you for introducing me to the benchmarking tool! I’ll be using that in my refresh of my math library.
No you are not dense :) The list is reversed because the first element in the list ends up being the last as the first element is the first you ad to the new list and you always append at the start (as the list is a linked list). If you look for a concrete example, the version of MyMap in my elixir_playground repo has doctests showing how it behaves without reverse. For the bytecode thing, there is a gist from sasa juric showcasing just that. I omitted it from the blog post, maybe should have included it :) Cool that you’ll use benchee, if you want to know more about it there is also an introduction blog post.
So, given two datapoints (TCO slower than non-TCO, and flat_map slower than map |> flatten)…
…is it possible at all that Benchee might have a bug?
It is always possible. Benchee or something Benchee uses might have a bug. I don’t think so, Benchee is well tested and rather simple. Plus you can reproduce the same with running
:timer.tca few times in a shell. Before I wrote benchee I also used Benchfella - results are similar much identical:Did the same double check with flat_map back then. Plus I rather blog about unusual findings, other benchmarks are pretty much what you’d expect them to be (recursion is fastest for repeating something n times, sorting 100k elements is ~15 times slower than sorting 10k elements etc.) There are some more samples in benchees samples folder :)
Plus especially for this I sat down and read a lot (the Erlang performance myths etc.) as I also found it highly unlikely and needed to see that this discussion has been had and explained before to be courageous enough to blog about it in an environment that I’m not super familiar with (yet)
[Comment removed by author]