1. -1
  1. 7

    It seems very weird to even have a “CPU MHz” graph, as if the hertz mean anything at all - especially since it’s comparing AMD CPUs and Intel CPUs?

    I have no idea what “Events per second” means. What kind of event? It doesn’t seem like that’s explained anywhere?

    I don’t know what’s going on with the memory read and write measurements. Obviously the average and minimum speed for a single memory read or write operation is going to be 0ms? Isn’t milliseconds way too coarse grained for that kind of measurement? What is counted as a “memory read operation” or “memory write operation” anyways? Does it measure a read from cache or does the benchmark make sure to actually read from main memory? Wouldn’t memory throughput and memory latency (with separate measurements for read and write) make more sense than “memory operations per second” and “milliseconds per memory operation”?

    Same with “File I/O”; isn’t latency and throughput more interesting than just ops per second? Is the “operations” the same as what’s measured when we measure IOPS or is it something else? What is the “minimum/maximum/average”? Is the “minimum time for a read operation” just measuring the time it takes to read a page from the page cache (aka just a benchmark of the memory system) or does it make sure the files aren’t in the page cache? And again, clearly milliseconds is way too coarse grained for these measurements given that they’re all at 0?

    Am I missing something or do most of these benchmarks seem underexplained and not that well thought through? I like the concept, seeing a wide variety of benchmarks on the various VPSes could be interesting, but I don’t really feel like I can conclude anything from the numbers here. Maybe running the Phoronix benchmark suite on the different $5/month VPSes could provide some more useful results.

    1. 5

      I’ve flagged this as spam because it’s some vague hand waving to get you to click on the referrer links at the bottom of the page. It looks as if it’s really just there to get referrer kick-backs.

      1. 3

        Am I missing something or do most of these benchmarks seem underexplained and not that well thought through?

        Aren’t you talking about most benchmarks that do the rounds? Benchmark blogs never seem to learn from the earlier criticism. They are often:

        • Have simplistic or naïve design. At most they test something the author was interested in.
        • Unreproducible even when they include scripts because they are missing other key information.
        • Designed around supporting a specific narrative (e.g. slamming Python for having the GIL).
        • Are purely designed to drive clicks for a company that’s not too close to any of the products or services being benchmarked, that they’ll complain a lot.

        A better test of VPS usage, especially when it’s a single node, might be to see how many requests per second you can get out of a WordPress instance on it. It’s far from perfect, but that’s a big reason they exist. Ideally, you’d add in some problematic code and see how well that performs. That was actually an idea that Ian Bicking had suggested at PyCon long ago for Python performance comparisons because that’s what is happening when most people need to investigate performance.

        1. 3

          Am I missing something or do most of these benchmarks seem underexplained and not that well thought through? I like the concept, seeing a wide variety of benchmarks on the various VPSes could be interesting, but I don’t really feel like I can conclude anything from the numbers here.

          You’re not missing anything.

          Another factor is that VPSs can have pretty variable performance, which is why he used three instances and “averaged the results where applicable”. A provider giving consistent performance vs. a provider with large differences seems like an interesting data point. Also n=3 seems pretty low to me.

          And things like “Maximum (ms)” for “CPU” (maximum what? The time an “event” took?) could be a single event that’s an outlier, and the mean average for these kind of statistics isn’t necessarily all that useful. You really want a distribution of timings.

          I did find the scripts he uses on his GitHub though; basically it just runs sysbench.

          I agree something like this could be useful; but this is not it. Quite frankly I’d say it’s borderline blogspam