1. 4

My idea what we can estimate errors with unit tests as well.

  1.  

  2. 4

    There is lots of smoke and mirrors nonsense out there.

    There are a few good papers, which required a lot of work to produce their numbers.

    There are papers that look at how many faults have been discovered, and then divide by the number of lines. This is after the fact estimating.

    There are processes that claim to reduce the number of faults per lines of code. These work by making sure people write and run tests, and they put lots of effort into code reviews.

    Estimating before anybody runs the code, I could sell you a bridge if you have money to spend.

    1. 1

      Estimating before anybody runs the code, I could sell you a bridge if you have money to spend.

      Mills’s Cleanroom method supposedly was able to do this. I read one book on the topic and thumbed through a few more and it seemed interesting but unusable by a team size of one (which is what it would have to be in my case).

      The APL/j/k folks have argued that there cannot be more errors than there are characters in your code, so minimizing the amount of code is an effective way of reducing the upper bound of possible errors (“it’s easier to find bugs in 10 lines of code than 10,000 lines”). There’s some truth to that, but it leads to (often humorously) dense code. The largest k program I’ve ever seen is perhaps only 400 lines (the entire k interpreter itself was supposedly only ~2000 lines of C, though I think Shakti k must be larger).

      (It’s also worth noting that “2000 lines of C” is a lot more when written in the k style than when written in idiomatic C style.)

      1. 1

        “…there cannot be more errors than there are characters in your code,”

        x=y;

        Can contain four errors, because it should have been written as: x=y*a+b[i]/3.14159-a_func(w+v);

        Unless you count the missing stuff as one error.

      2. 1

        I once had the pleasure to explain to a non-technical person (judicial profession) that it’s not possible to develop bug-free software. That person was truly embarassed as it genuinely believed that software developed by a professional software company does not normally have errors. I’ve since then become careful when talking to this kind of people, and since my word as a law student does not count for them, I’m always looking for “real” research I can point this kind of people to. Your link looks interesting, I’m going to read through it and see.

        1. 1

          Beware of people claiming to have verified the correctness of software, it’s all snake-oil.

      3. 1

        There are studies you can find with a tiny bit of googling, but you’d have to dig deep to find if methodology is any good.

        My personal anecdata: I translated algorithm I knew well from Python into math (not formal math, just enough for non-programmer scientist to read it with me helping with explanations). I asked someone who understood the code to review the math, they found no problems.

        In fact, IIRC I had 2 major transcription errors, in just 10 lines of math.

        1. 1

          My idea what we can estimate errors with unit tests as well.

          Can you clarify what you mean? Do you mean to estimate the number of errors currently in the program? If so, mutation analysis can help. The score is essentially a proxy for the possible errors that your test suites will catch, and 1-Mu would be a proxy for the errors that remain.

          1. 1

            My idea was, if probably for both tests and code is the same. I can calc combined probability with tests.