1. 5

    I’m curious why your goal is “100% coverage” and not “great tests for tricky parts”?

    My position on code coverage has always been that, unless you have a team of extremely disciplined developers, your code is going to have a lot of less than ideal tests to sustain that number. Rather than give into this, I’ve always chosen to use the red lines in coverage output as a way to spot check for complexity and make a decision on test value for that spot. I define complexity as an estimate of cyclomatic complexity, or “gut feel” on importance–it’s kind of arbitrary.

    Curious to hear your thoughts on this.

    1. 4

      Not OP, but this has been my experience as well. High coverage targets lead to (a) tests that don’t actually test anything useful but needed to be written to boost the coverage percentage, and worse, (b) less defensive programming in the code under test because it’s easier to leave out a “this should never happen” sanity check than to figure out how to simulate the theoretically impossible state you’re guarding against.

      100% condition coverage doesn’t imply a high-quality test:

      def add(a, b):
          if a > b:
              return a + b
          else:
              return a - b
      
      def test_add():
          assertEqual(add(0, 0), 0)
          assertEqual(add(2, 1), 3)
      
      1. 2

        I used this exact example in a talk on property testing! The “proper” test would be something like

        @given(int(), int())
        def test_add_is_commutative(a, b):
            assert add(a, b) == add(b, a)
        

        (Though that also lets through multiplication… you also want something like a+(b+1) = (a+1)+b)

        1. 2

          Being in a regulated field, 100% coverage is usually required. Not just statement but even branch coverage. In the extreme, we had test cases which did not check anything. They just executed code and increased coverage.

        2. 1

          I love code complexity! (Well, this sounds weird, but I truly love to mess with code complexity metrics. I have even invented one!)

          I try not mix these two concepts together. We always have 100% on out open-source projects. So, this is not a decision I make every time. It is a rule. We also try to have 100% on our closed-source projects. But, sometimes it is not worth it in budget / time terms. And we can go as low as 95% of coverage.

          You might like some of the things I do about code complexity:

          1. 1

            It is true that 100% coverage tells you nothing at all about whether you can trust the test suite to catch your mistakes.

            However, 90% coverage tells me something very important - specifically, that you definitely cannot trust the test suite to catch your mistakes.

          1. 2

            Great stuff. I have also enjoyed your https://bytes.yingw787.com/posts/2019/12/06/monads/ article.

            By the way, we have a typed implementation of several monads in Python: https://github.com/dry-python/returns Check it out.

            1. 2

              Looks like you’re at wemake. I haven’t had a chance to play [yet], so thought I’d ask: how does the flake8 configuration get along with automated formatters? It’d be nice if it got along easily with yapf (or black, or …).

              1. 2

                I found a page that seems to have the information you’re looking for: Auto-formatters – wemake-python-styleguide. To summarize, wemake-python-styleguide is compatible with autopep8, isort, and add-trailing-comma, but it is not compatible with yapf or black.

                1. 2

                  Thanks for finding that link, good reading. While they explain why black isn’t going to work,

                  black itself is actually not compatible with PEP8 and flake8

                  they only explain that they weren’t able to come up with a yapf configuration that matched their choices

                  If you have a working configuration for both yapf and wemake-python-styleguide, please, let us know!

                2. 1

                  Sorry, I have missed your question.

                  We don’t support black or any other auto-formatters, but we are compatible with autopep8 and autoflake8 by design.

                  Why don’t we support auto-formatters? Because we believe that developers should write code and pay maximum attention to the process. And a good linter eliminates the necessity of auto-formatters. Docs: https://wemake-python-stylegui.de/en/latest/pages/usage/integrations/auto-formatters.html

                1. 1

                  The “Pay other people to audit your code” link https://wemake.services/meta/rsdp/auditions/ is broken. Is there really such a service? I’m mostly looking for “code review trades” though.

                  1. 4

                    Sorry about that. Here’s the correct link: https://wemake.services/meta/rsdp/audits/

                    1. 1

                      Thanks!

                    2. 1

                      Pretty confident the url is supposed to end in /audits/ instead of /auditions/

                    1. 1

                      /bin/bash

                      nooooo bash doesn’t belong in /bin >_<

                      1. 4

                        Agreed,

                        /nix/store/hww7q7rlhj6xhngdwgmdkyhl10cqi6nm-bash-4.4-p5/bin/bash

                        or bust

                        1. 1

                          There was no image with /usr/bin/env bash available …

                          1. 1

                            I highly doubt that given /usr/bin/env is basically required for any posix compliant unix. Even linux distros fit that bill.

                            $ docker run --user root --rm -it alpine:latest sh
                            / # apk update
                            fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
                            apk add fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
                            v3.6.2-25-g69c0672374 [http://dl-cdn.alpinelinux.org/alpine/v3.6/main]
                            v3.6.2-26-gc0c3f19f19 [http://dl-cdn.alpinelinux.org/alpine/v3.6/community]
                            OK: 8436 distinct packages available
                            / # apk add bash
                            (1/5) Installing ncurses-terminfo-base (6.0-r7)
                            (2/5) Installing ncurses-terminfo (6.0-r7)
                            (3/5) Installing ncurses-libs (6.0-r7)
                            (4/5) Installing readline (6.3.008-r5)
                            (5/5) Installing bash (4.3.48-r1)
                            Executing bash-4.3.48-r1.post-install
                            Executing busybox-1.26.2-r5.trigger
                            OK: 12 MiB in 16 packages
                            / # which bash
                            /bin/bash
                            / # which env
                            /usr/bin/env
                            # /usr/bin/env bash
                            bash-4.3#
                            
                            1. 1

                              That was about image as in picture. The illustration in the article.