1. 10

    1. 5

      Don’t editorialize in the title. If you want to complain about Phoronix’s methodology, the comments are the place to do so.

      1. [Comment removed by author]

    2. 2

      Here Phoronix acknowledges its bcachefs benchmark fiasco: they didn’t disable debug option

      1. 10

        Testing the functionality as shipped seems like an eminently reasonable option, IMO

        1. 11

          It depends on how it’s meant to be used. For a new filesystem, I’d expect people to feature test it like this but not deploy it in production yet. Once it’s tested enough for deployment, the debug features will be off in release builds and that’s the realistic configuration.

          Phoronix has history of this. FreeBSD trunk has WITNESS (lock order checking) an INVARIANTS (assertions, often very expensive) set because you’re not meant to run trunk in production (unless, like Netflix, you are building in fault tolerance at a higher level and are reporting and fixing bugs). It’s used for testing. Phoronix, for years, benchmarked trunk build against releases and other operating systems and produced sensational headlines showing that the next release would be much slower. Their early clang benchmarks also failed to notice that clang defaulted to O0, so compared clang without optimisation to gcc with.

          I tried using their benchmark suite for snmalloc and it doesn’t even try to do any sensible statistics (SPEC does it badly, but at least it tries). I would strongly recommend ignoring any Phoronix benchmarking articles.

      2. 4

        Shouldn’t debug have been disabled when merged anyway? Seems like yes, since it is now disabled by default.