I believe the Elixir CPU usage is due to the way the Erlang VM manages schedulers. A scheduler will literally busy-wait if it thinks there might be more work to do soon, in an effort to improve the responsiveness of the system. See the +sbwt* family of emulator flags at http://erlang.org/doc/man/erl.html.
The consequence of this is that you must use the tooling provided with Erlang to get a good idea of actual CPU usage of the application. Top and friends make the world look a bit strange.
Edit: clarified “actual CPU usage” a bit.
I’m curious about the CPU Usage shown for Elixir. Maybe since it wasn’t otherwise fully utilized, that was BEAM’s busy-waits spinning for a bit?
Yes, busy waits are the very likely cause of CPU saturation since BEAM was very responsive. I’d need to run the test with microstate accounting enabled VM to confirm this.
You could do an easy check by running the emulator with “+sbwt none” and see if the saturation disappears.
(Also, darn, I didn’t see meredith’s question before I posted my other comment.)
Edit: maybe you would also want to include “+sbwtdcpu none” and “+sbwtdio none”? I’m not sure.
I’m curious why the cluster module wasn’t used for Node.js? I’m not saying it’s as good as Elixir/Go lang, I just think perhaps the comparison would have been more fair.
I wasn’t aware of cluster module and agree it should be included—something for the next round of tests!
Wow, node performed amazingly! A single thread mostly keeping up with multi-threaded alternatives! Just run $(ncpus) per host and the throughput would soar.
But who runs benchmarks on VMs anyway? Seems like you’d want to run it on real hardware to get numbers you could count on.
Yes. The next test will include multiple node processes!
There were several runs each with new target EC2 instance to see if there would be notable difference in the results from getting “noisy neighbors”. All runs were very consistent.
Is node still single threaded?
Also, some of the graphs are confusing. The range bars dip below what should be the minimum limit. Sometimes the latency is below the 90ms (100 - 10%) floor. In one graph, go receives a request in negative microseconds.
Is node still single threaded?
Yes, Node is still essentially single threaded. (Essentially because you can, as of 11.7, create Worker threads, but it’s nothing like goroutines or BEAM processes)
Agree. Bars are based on standard error and I added them as both positive and negative, yet they’re likely to reflect mostly positive error from the graphed mean line. What would be the right way to do this?
A lot of natural processes actually follow a lognormal distribution, which nicely handles the impossibility of values below zero.
Surprising the three are very similar in these tests (other than node at 100k connections). If scalabilty characteristics are so similar, that removes that entire concern from choosing one of the three stacks. Developer familiarity and deployment ceremony become more prominent considerations.
Would you see a difference if you ran these on Dedicated hosts on AWS?
Interesting test would be to run shared vs dedicated with everything else being the same. My guess is the larger is the instance the less would be the advantage of dedicated host.
For most real world use cases you’ll see a huge difference between Go and the other 2. We tested elixir extensively for activity feeds (vs go) and there is a 8-10x difference in performance. It’s not close.
It depends on the use case of course.
This pretty much fed into my pre-existing notion that Node.JS sucks.