1. 24
  1. 8

    My professional life has been so much better since I started using primarily acceptance tests. Yes they take a while to run but they test the actual user journey. Unit tests with their numerous mocks are like pouring concrete on code and make even simple refactors arduous.

    I don’t even really know where an integration test fits in, like an acceptance test with a mock database?

    From my position at the moment I don’t see why people do it to themselves, acceptance tests FTW.

    1. 11

      I’ve had the reverse experience.

      A code base with mostly acceptance tests, that took an hour to run in the CI. Absolute velocity killer. Then some of the tests start flaking, forcing multiple CI runs, and merging basic changes takes half a day.

      Unit tests with their numerous mocks are like pouring concrete on code and make even simple refactors arduous.

      If most of your unit tests are like this, it’s usually a signal that the code needs to be designed differently. Unit tests shine for testing pure functions. Yes, for testing high-level “service coordination” code that has non-trivial business rules, the kind of tests with mocks for each service make sense. But even there, if the code is designed well, the tests themselves can be simple.

      Code design is the elephant in the room whenever testing is discussed. It’s impossible to talk about testing strategies without talking about code design, though.

      Michael Feathers - the deep synergy between testability and good design

      Finally, there is the connection between testing and the minute-by-minute experience of building software, which an acceptance-only approach performs poorly on:

      Gary Bernhardt’s Fast Test, Slow Test

      1. 2

        What type of code are you writing? I ask, because I often find talks about testing are very hard to apply to the work I do. I work on a legacy C base, using a proprietary library/device driver for about half of it [0]. My team might get four deployments per year, and our customer (the Oligarchic Cell Phone Company) has sprints that last years [1]. So an hour to run tests isn’t that bad [2].

        Another question I keep asking, but never get a satisfactory answer to, is, what is a unit? Our “program” actually consists of several programs—two accepting network requests [3], which then forward it to our business logic unit, which in turn makes two concurrent requests to outside databases [4] [5] which are then sliced, diced and merged into a reply. And because of the way the code is written (C code), in my opinion, the “units” are the individual programs. Could the “business logic” be a pure function? Maybe? At this point, it would require a major restructuring of the code and given current management, I don’t see that happening at all.

        I started listening to the Michael Feathers talk you linked to, but the assumption he makes is that you are writing in an object oriented programming language, so a lot of his advice doesn’t apply, or is hard to apply. Not all of us are using OOP.

        [0] A proprietary network stack, which we are only licensed to run on Solaris 5.10/5.11 on SPARC. The less said about The Protocol Stack From Hell, the better (the thing falls over if you breath on it hard, and we’re not paid to debug third party code, which itself is in legacy mode …)

        [1] For instance, it took five years for our customer to add one (1) piece of information to one of the SIP headers we receive. Five years. I don’t think they practice continuous integration over there.

        [2] Due to an overly-specific test plan, our first test platform (which wasn’t entirely automatic) took over five hours to run 120 tests. If only we didn’t specify the syslog output …

        [3] Via SS7 and SIP

        [4] Via DNS. We make NAPTR requests.

        [5] We are also under a real time deadline, with only two seconds to return a result, thus why we make the requests concurrently.

      2. 3

        Acceptance tests are important because they exercise the project from the perspective of the user, which is, ultimately, the thing that matters. But they model the project as a black box, a single unit whole thing, which means they’re totally agnostic to the internals. For nontrivial projects, this is a problem. Tests assert correctness and prevent regressions and all that good stuff, but they’re also the best tool we have to motivate and enforce coherence, encapsulation, and all the other virtues of software engineering. Each meaningful layer of abstraction in your project needs tests, if it’s gonna survive the test of time.

        1. 5

          Unit tests also from the perspective of a user, where the user is another dev or team in the project or company that will need to use or rely on this API.

      3. 3

        Funnily enough I would turn the pyramid upside down. I will not review code unless it is already formatted properly, ideally with an automated tool. Then I will check if a linter catches anything substantial. If not I will look at and run tests. After that I read the documentation if mandated. Then and only the I will look at the architecture, API, etc. Of course there are exceptions to the rule. This approach minimizes manual labor on the side of the reviewer and gives them more time to focus on the important things (which are similiar to the describe in the article)

        1. 10

          I think that’s what the pyramid is meant to say: you should spend the least time on formatting-related things. I think we agree on the extra constraint: you should spend no time on things that can be automated.

          My code-review experiences have become a lot better since adding clang-format, clang-tidy (which can catch some things in variable naming), and the test suite to CI, along with -Werror. When I start reviewing code, I know:

          • The formatting is correct.
          • There are no obvious machine-catchable bugs.
          • Everything that is tested works as intended.

          My first step is then to look at what the tests don’t check. I can then focus my review on code paths that aren’t likely to be triggered by the tests (even if CI did profiling, just because a particular line is hit doesn’t mean that a particular combination of conditions is).

          1. 1

            I think that’s what the pyramid is meant to say: you should spend the least time on formatting-related things

            Exactly that.

        2. 3

          Good framing, but I’m not sure about the specific decisions. Tests, for example, are surely more important than documentation. But perhaps the idea is that the upper parts should be table stakes and enforced by CI?

          1. 2

            Tests should be at the bottom for me. For multiple reasons, but one I’d like to highlight is they handle the API semantics anyway.

            1. 1

              For most of my career there was no such thing as “code reviews”. Then, their introduction in large public codebases was warranted, given code submission from unknown people and sources. That seems to have now been extended to effectively all development, particularly for a new generation of developers that know no different.

              Internal professional developers (both employees and contractors) used to be considered competent to write and introduce code for both features and bugs. These days that just doesn’t seem to be the case? Are developers less competent, less trustworthy or something else these days?

              1. 5

                Code reviews are not just about finding bugs. Or even mostly about that! Their primary purpose is education: to make sure more than 1 person understands how a given bit of code works, and is capable of maintaining it.

                edit: or, what edudobay said ;)

                1. 3

                  Though there is criticism about code review in professional software development (was it blindly taken from open source, without considering that professional teams interact differently?), and in some environments it might also be seen as mitigation for lack of trust or competence, code review (in any form it might take) is a valid way for communicating and sharing knowledge in a team.

                  1. 1

                    Code that I put out for review is better than code I checkin without review, even before the code review happens.