1. 2

See also discussion at http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/428620?428438-428631

  1.  

  2. 8

    I appreciate the thoroughness of the survey here, but this experiment is so methodologically flawed as to be useless.

    While I recorded all the numbers for both Apache Bench and wrk, all the request per second numbers you see in this report are for Apache Bench.

    Apache Bench is an out-dated tool which can’t saturate modern systems, as well as having serious coordinated omission problems.

    I ran into fewer errors and more consistent reporting using Apache Bench than I did with wrk […]

    wrk uses a thread pool to initiate connections, so these errors are almost assuredly the result of Ruby dropping connections under load. Apache Bench uses a single thread, which means it (unlike the real world) will never push these servers out of their comfort zones. Refusing to count the misses here heavily biases the results.

    All tests are run on a 2013 Macbook Air 1.3 GHz Intel Core i5 with 8 GB of DDR3 RAM and a 250 GB SSD.

    These benchmarks weren’t done on a server-class computer or operating system, nor were they even done on a real network.

    Finally, there is also no mention of latency in the entire article, which is a bummer, since it leaves us unable to do any handy Little’s Law modeling.

    At the very least, good experimental design here requires the following, in no particular order:

    • Use a modern load-testing tool capable of saturating the system under test.
    • Statistically compensate for coordinated omission in the load-testing tools.
    • Tests varying levels of concurrency, ideally using a empirically-validated model to interpret the results.
    • Measure and model latency as well as throughput.
    • Run the load-testing tool on a different machine to eliminate confounding effects of shared resources (e.g., CPU, memory bus, thread scheduler, memory allocator, etc.).
    • Perform measurements of a meaningful system under test:
      • Run on a representative operating system (i.e., use Ubuntu or RedHat, not OS X).
      • Run on a representative machine (i.e., use a few EC2 instances, not your tiny laptop).
      • Run on a representative machine (i.e., use a real packet-switching network, not loopback).

    I don’t like kicking someone’s puppy like this, but the entire point of a benchmark is to augment decision-making and this benchmark is incredibly misleading.

    1. 4

      With the right combination of server, runtime, and framework Ruby can be as fast as a language like Go or Scala.

      I don’t know much about Ruby so most of this was unclear to me where in the stack these things were happening. But is this true? As far as I could understand it looks like the result mostly depend on minimizing the amount of time spent in Ruby code. The actual application was just hello world, and TorqueBox is written in Java from what I gather so the claim that Ruby can be as fast as Go and Scala is not quite accurate since it looks like the fastest solutions just minimize how much time is spent in Ruby.

      Am I understanding this correctly?