Is it possible to have too many tests? (Full disclosure: I have a life threatening allergy to tests.)
At some point I imagine you end up with 10 tests for CapitalizeNumber, but then it turns out there are zero calls to that function. Normally dead code is irrelevant, but now there’s a cost associated.
Or is it truly necessary to run every test on every checkin? It seems with a little dependency analysis, one could run only the tests for changed code and its downstream. Is that harder than building the system described here?
Stripe was written in Ruby (may have written portions in other langs by now). There are no non-trivial Ruby tools for dependency analysis and the language has no effective tools for compartmentalizing code, just some conventions. Standard Ruby idioms involve passing around lots of mutable effectful objects via ad-hoc interfaces, so most code is interdependent in unpredictable ways.
Running all the tests for every commit is the only sane thing to do.
(I work at Stripe, and have spent a pretty significant time working on tests)
That sums it up pretty nicely - there are things you can do to get to a failure faster - run tests related to the change under test first (here’s a cool & short related paper that ties into benchmarks: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.59.7380); but ultimately that only optimizes for the (relatively speaking) more rare case of people pushing trivially broken code. In the end, you have to run all of them to fully know that it’s safe to deploy your change; running them all and speeding them up as much as possible is your best hope.
That said, we do often go through the tests to see which ones take too much time, or which ones are unneeded. I really wish there was a decent tree shaker for Ruby to detect dead code. Sadly, developer intuition and decent test coverage (that old chestnut) are the only tools that even remotely help here.
Surely if you modify main.rb you don’t need to run the separate unit tests for banana.rb. If you modify banana.rb, then you need to rerun main tests as well, but hopefully the lowest layers of code will gel at some point. I mean, just gripping out require lines has to get you pretty far.
I wasn’t kidding about interdependencies. Ruby programs infrequently have a “top”, a main point of entry with a deliberate control flow. They’re big buckets of ravioli code. Code with explicit layers is often derided as “overdesigned” or confusing.
The ActiveSupport gem that ships with Rails (and is commonly used without Rails) extends core types and functionality globally if any dependency requires it at runtime (and only recently was it possible to import only part of it, eg. get its extensions to Array without its extensions to Hash). One of the things it does is “auto-loading”. It hooks the builtin Ruby code that handles the exception for failed constant lookups (eg the name of a class), discards that exception, and attempts to find that constant by loading similar filenames from any directory in its autoload_paths (again, a runtime config setting). It will re-raise if it fails, and its message when foo.rb has invalid syntax or otherwise doesn’t define Foo is particularly opaque.
The directories searched by require lines change based on libraries loaded. As they get loaded at runtime, of course.
So yes, you could extract require lines and constant references to try to duplicate ActiveSupport’s auto-loading with good guesses at the directories that will be autoloaded, but that’ll be a lot of finicky code for little insight. E.g., it is increasingly common to construct constant references at runtime to find collaborators. This is a current practice advocated for by the leading OO designer in Ruby (search for const_get) - and I think it’s a good idea, or at least less bad than what it replaces.
Ruby as designed and written is very resistant to static analysis. (I kind of ranted about this recently.)
Is it possible to have too many tests? (Full disclosure: I have a life threatening allergy to tests.)
At some point I imagine you end up with 10 tests for CapitalizeNumber, but then it turns out there are zero calls to that function. Normally dead code is irrelevant, but now there’s a cost associated.
Or is it truly necessary to run every test on every checkin? It seems with a little dependency analysis, one could run only the tests for changed code and its downstream. Is that harder than building the system described here?
Stripe was written in Ruby (may have written portions in other langs by now). There are no non-trivial Ruby tools for dependency analysis and the language has no effective tools for compartmentalizing code, just some conventions. Standard Ruby idioms involve passing around lots of mutable effectful objects via ad-hoc interfaces, so most code is interdependent in unpredictable ways.
Running all the tests for every commit is the only sane thing to do.
(I work at Stripe, and have spent a pretty significant time working on tests)
That sums it up pretty nicely - there are things you can do to get to a failure faster - run tests related to the change under test first (here’s a cool & short related paper that ties into benchmarks: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.59.7380); but ultimately that only optimizes for the (relatively speaking) more rare case of people pushing trivially broken code. In the end, you have to run all of them to fully know that it’s safe to deploy your change; running them all and speeding them up as much as possible is your best hope.
That said, we do often go through the tests to see which ones take too much time, or which ones are unneeded. I really wish there was a decent tree shaker for Ruby to detect dead code. Sadly, developer intuition and decent test coverage (that old chestnut) are the only tools that even remotely help here.
Surely if you modify main.rb you don’t need to run the separate unit tests for banana.rb. If you modify banana.rb, then you need to rerun main tests as well, but hopefully the lowest layers of code will gel at some point. I mean, just gripping out require lines has to get you pretty far.
I wasn’t kidding about interdependencies. Ruby programs infrequently have a “top”, a main point of entry with a deliberate control flow. They’re big buckets of ravioli code. Code with explicit layers is often derided as “overdesigned” or confusing.
The ActiveSupport gem that ships with Rails (and is commonly used without Rails) extends core types and functionality globally if any dependency requires it at runtime (and only recently was it possible to import only part of it, eg. get its extensions to Array without its extensions to Hash). One of the things it does is “auto-loading”. It hooks the builtin Ruby code that handles the exception for failed constant lookups (eg the name of a class), discards that exception, and attempts to find that constant by loading similar filenames from any directory in its
autoload_paths(again, a runtime config setting). It will re-raise if it fails, and its message whenfoo.rbhas invalid syntax or otherwise doesn’t defineFoois particularly opaque.The directories searched by require lines change based on libraries loaded. As they get loaded at runtime, of course.
So yes, you could extract require lines and constant references to try to duplicate ActiveSupport’s auto-loading with good guesses at the directories that will be autoloaded, but that’ll be a lot of finicky code for little insight. E.g., it is increasingly common to construct constant references at runtime to find collaborators. This is a current practice advocated for by the leading OO designer in Ruby (search for
const_get) - and I think it’s a good idea, or at least less bad than what it replaces.Ruby as designed and written is very resistant to static analysis. (I kind of ranted about this recently.)