1. 26

I wanted to find out how nice of a unit testing “DSL” I could make by abusing the C preprocessor, and I think the result isn’t too bad.

One notable feature are the Go-inspired defer functionality, which I think makes tearing down tests really nice, and makes teardown basically “just work” even when a test case fails. I also used the C11 _Generic stuff to make asserteq(a, b) and assertneq(a, b) automatically choose whether to compare strings or scalar values.

So, what does people think? Constructive criticism is welcome.

  1. 4

    At a glance it seems okay, but I guarantee those colour choices will look like crap on a light background. For most (all?) testing frameworks I’ve used that output in colour, I always have to find the “no colour” option or else I can’t read it.

    1. 2

      I support I sort of consider it the user’s responsiblility to have configured a color scheme where most colors are readable. However, it would be both easy and a good idea to make it possible to configure the color scheme (at least from the source code), and I should probably add an option to output without colors (and enable that option by default when the output is not a TTY).

      1. 3

        I use solarized light, and nearly everyone these days uses some form of dark colour scheme. The output from the Catch2 testing framework, for example, is mostly unreadable with the colour choices. In other cases, I’ve run across similar problems.

        If you’re going to offer colour output, I think you need to have an option to turn it off. (And I see that you’ve added #ifndefs around them.) If/when this ever gets a main function to manage test suites (most serious ones do), don’t forget the --color=no option.

        1. 2

          I added support for theming first because that was very easy to add.

          I just pushed a commit to add support for –no-color (and which disables color when stdout is not a TTY and such): https://github.com/mortie/snow/commit/c41d869c613a3a587279c6f833f74c609cb3bbf5

          The commit after that adds support for the NO_COLOR environment variable mentioned by @mulander.

        2. 3

          @jcs created http://no-color.org/ to propagate a consistent option to disable colors.

          1. 2

            Looks like I get to be the first software to support NO_COLOR on that list :)

        3. 1

          I always wanted a terminal which would automatically corrected colors based on contrast. At least a separate color scheme for default background color.

          It should not be that hard, maybe I could add PoC using suckless’s st to my overly long TODO list…

          1. 1

            It’s actually quite readable in black on white. Though I agree with the general sentiment, and it’s probably quite a bit worse on a yellowish background.

          2. 4

            Nice work, some thoughts:

            • Print line number where assertion failed
            • Way to compare doubles, possible with an optional precision
            • Way to compare blocks of memory
            • Consider renaming to snow.h
            1. 3

              I really would have liked to print the line number where the assertion fails, but I’m not sure if that’s possible. Because of the use of macros, everything ends up on the same line after the preprocessor, so __LINE__ will be the same for everything. If you know of a way to fix that, I’d love to hear it. (The "in example.c:files" message was originally supposed to be "in example.c:<line number>")

              More different asserts is a good idea, and so is renaming the header - the thing was under the temporary name “testfw” until right before I made this post.

              1. 2

                Looks neat! I feel that the line number of the end-of-block would still be useful, but don’t quite see how to word that without seeming incorrect.

                1. 2

                  It’s not just at the end of the it block which the error occurs in; it’s the end of the entire invokation of the describe macro. In the example.c file for example, __LINE__ inside of any of those it blocks will, as of the linked commit, be 62.

            2. 3

              I’m still looking for a test harness that doesn’t need me to explicitly call each test/suite in main. My current approach is a simple-minded code-generation. Is there a way to do this that avoids autogenerating files and whatnot?

              1. 3

                There’s a couple of ways I can imagine that would be possible. Currently, each top-level describe generates a function; I could have a global array of function pointers, and use the __COUNTER__ macro to automatically insert describe‘s functions into that array. However, that would mean that the length of the array would have to be static. It probably wouldn’t be too bad though if it was configurable by defining a macro before including the library, and defaulting the length to something like 1024, though.

                Another solution would be to not have these top-level describes, and instead have a macro called testsuite or something, which generates a main function. This would mean that, if your test suite is in multiple files, you’d have to be very careful what you have in those files, because they would be included from a function body, but it would be doable.

                I think the first approach would be the best. You could then also have a runtests() macro which loops from 0 through __COUNTER__ - 2 and runs all the tests.

                1. 1

                  That’s a great idea. Thanks!

                  1. 2

                    An update: the first solution will be much harder than I expected, because you can’t in C do things like foo[0] = bar outside of a function. That means you can’t assign the function pointer to the array in the describe macro. If you could append to a macro frow within a macro, you could have a macro which describe appends to which, when invoked, just calls all the functions created by describe, but there doesn’t seem to be any way to append to a macro from within a macro (though we can get close; using push_macro and pop_macro in _Pragma, it would be possible to append to a macro, but not from within another macro).

                    It would still be possible to call the functions something deterministic (say test_##__COUNTER__), and then, in the main function, use dlopen on argv[0], and then loop from i=0 to i=__COUNTER__-2 and use dlsym to find the symbol named "_test_$i" and call it… but that’s not something I want to do in Snow, because that sounds a little too crazy :P

                    1. 1

                      I appreciate the update. Yes, that would be too crazy for my taste as well. (As is your second idea above.)

                      1. 1

                        FWIW, you can do this by placing the function pointer in a custom linker section with linker-inserted begin/end symbols; unfortunately, that requires your user to use a custom linker script, which will be annoying for them.

                2. 2

                  Very interesting.

                  On an unrelated note, what’s the window manager in the screenshot?

                  1. 4

                    That’s i3-gaps, the terminal is termite, and the shell is zsh.

                  2. 1

                    defer() is basically independent of the rest of the library, isn’t it? Might want to extract that.

                    1. 2

                      It’s not entirely independent, as it depends on the "it" macro to create a bunch of variables and run the deferred statements. A stand-alone implementation would require a macro you call at the beginning of a block which will contain deferred expressions, and a macro you call before every return, so it’s not as nice to use. It also relies on GNU extensions, which imo is okay for a test suite, but I’d be careful relying on them in regular code.

                      Anyways, I did the work to extract it into its own small library: https://gist.github.com/mortie/0696f1cf717d192a33b7d842144dcf4a

                      Example usage:

                      #include "defer.h"
                      #include <stdio.h>
                      int main() {
                          defer_init();
                          defer(printf("world\n"));
                          defer(printf("hello "));
                          defer_return(0);
                      }
                      

                      If you want to do anything interesting with it, feel free to.