1. 9

After having recently implemented a very, very, very simple shell script based functional testing setup for a CLI tool, that output TAP to use with prove(1), I’ve become super obsessed with the simplicity of TAP, so I wrote a simple output translator for use with Go’s builtin testing framework. Not sure exactly how useful this actually is for most people, but you never know.

  1.  

  2. 1

    This is very cool! Does patter deal correctly with parallel tests?

    1. 2

      Not intentionally! But, I haven’t tried it, yet. This is the result of a whim, and about an hour of work.

      I think I’ll make this work with go test, too, so I’ll test when I do that work.

      1. 2

        I just tried executing a giant test suite with -p 10 (but no actual calls to t.Parallel). This is supposed to result in tests from other packages running in Parallel. Since the output is largely the same --- PASS: I think it just works.

        As for subtests, what currently happens is that if the parent passes, it gets counted as 1 test, and the sub tests are added as comments. I guess the easiest thing to do would be to strip leading spaces, but then I’d over count…

    2. 1

      TAP is great, the v13 of the spec makes it much easier to convert other types of output into TAP, glad to see it done for Go!

      I know so little of perl though, I didn’t know prove existed! Seems like I’ve been missing out. Thanks!

      1. 3

        I find it somewhat annoying that the status is the first thing that you have to print, followed by the message; it makes it clunky to write scripts, since the output is usually what you want as the message, and the status is not known until the end of the test.

        I’d far rather have:

        test name {
           diagnostic output
        } fail reason
        

        as the structure. This is actually what I settled on for the mbld test protocol: http://myrlang.org/mbld/subtests

        1. 2

          Here’s what I generally do, I just copy it around:

          out=$(mktemp $0.$$.XXXXXX)
          (
              echo this is a failed
              echo super bad test
              exit 5
          ) > $out &
          pid=$!
          wait $pid
          res=$?
          echo $res
          test $res -ne 0 && echo not ok -- $0 && (cat $out | sed 's/^/# /') && rm $out && exit $res
          echo ok -- $0 && rm $out && exit $res
          

          Now mind you I usually write it in rc, so it’s a little more compact, but otherwise there ya go! Don’t use a fifo unless you know you’re output will always be <16k. But that’s beside the point.

          I think test output ideally is something like Markdown. Sure there are warts and dialects for text based written docs, but most people agree for text-based entry, markdown is a decent way to do it. Then tools like pandoc can get you anywhere else.

          TAP (or MTEST) are close enough approximations for me. I’ll probably stick with TAP as I already have written up a few more tools now, but I can probably write a small tool to convert TAP output to MTEST output easily anyways :) Thanks for the link!

          1. 2

            Yeah, I’ve considered going to TAP; the marginally cleaner code that I get from writing my own may not be worth the extra work from using my own format.