1. 4

  2. 3

    Wow, you were at NYC.rb too? Too bad I didn’t know. It would have been cool to meet you in person.

    Here are the notes I took at the talk. The italicized bullets below summarize information in the slides; the other bullets add detail beyond the slides:

    • Test suites have fidelity (no false negatives for bugs), resilience (no false positives for bugs), and precision (failing tests identify the specific bad code).
    • The speaker, Michael West, doesn’t know of any tools that measure the precision of your tests – he would love to be told of any.
    • Don’t use overlapping tests as a way to validate your tests. It’s not worth it.
    • Code coverage is a weak measure of fidelity, since it is possible for tests with no assertions to still have 100% code coverage (by just running the code without asserting anything).
    • The premier mutation testing gem is mutant – github.com/mbj/mutant.
    • Mutant has very little built-in documentation – users have to look at third-party blog posts and videos. Since Mutant is actively developed and the codebase is very clean, the speaker thinks contributing documentation to Mutant would be a good way to get into open-source.
    • Mutation testing is very slow – hours for a large codebase, 15 minutes for one module or class. So you should only run it manually on a part of the code.
    • The software can’t automatically detect “equivalent mutants”, which are exactly equivalent to the original code – the user has to identify them manually. The mutant gem in particular doesn’t even let you enter the number of equivalent mutants you found – you have to account for them yourself when reading the reported numbers.
    1. 2

      I was not, no, I live in Chicago (and am happy to meet more Lobsters folks).

      I know mutant’s author Markus Schirp from the research that went into my talk. I noticed he wrote several libraries I used or liked the style of (anima, vanguard, concord) and I have been attempting to draft him into my prospiracy (it’s like a conspiracy, but for public good) to improve the state of Ruby. He retweeted a link to the slides.

      Thanks for sharing your notes, those are great

    2. 2

      Seems like slides.com would warrant the slides tag. ;)

      1. 1

        Don’t tell @jcs or I’ll have to look at the story guidelines for another five days.

      2. 1

        Does Mutant support Minitest yet? Looks like 330 is still open and README says “The minitest integration is still in the works.” :/

        1. 1

          I think the biggest barrier to minitest support is lack of people willing to help test that pull request against their own projects. It’s not as easy in minitest as rspec to associate a test with the class or method it is intended to cover. A cover declaration needs to be added to a test so mutant knows what tests to re-run after it mutates some code.