I noticed that this library is performance tested, with assert statements that make sure that a given function is executed within X ms.
Are these kinds of tests helpful? Does it not make a difference what machine the tests are running on? Or if the test runner instance is responsible for running other test suites too and happened to be overloaded at the time these performance tests ran?
Or is there a way to isolate machine resources so that such tests yield predictable/consistent results?
I’d say they are useful for preventing performance regression. They are likely fairly consistent, but it is true that there’s potential signal pollution that can occur.
Exactly. I’ve done this type of assertion before and you wind up having to put such a wide margin of error in it (like order of magnitude with modern, cloud CI boxes) that it doesn’t catch anything but the most egregious of regressions.
That said, you can still assert relative comparisons (assert my time < “other gem” time) with some confidence.