1. 13

  2. 3

    Is there more info available other than that single page? I’m for example interested in what the motivations are why this tool exists, what makes it stand out from other tools, what tech it’s using etc etc.

    1. 6

      Not quite yet. I would like to get a more detailed blog post out soon. I’m painfully close to something that I can release to a few brave souls to try out, so I’ve been focusing on wrapping it up. If you’re interested, please sign up (or follow me on Twitter if that’s your thing), and I’ll make sure to get some more details out there soon.

      Super quick summary: This tool is meant to marry property-based testing with models, temporal logic specifications (much like in TLA+), and browser testing with webdriver. I want to make it easy to use this on any web page, without loads of boilerplating test setup. Basically write a spec, point at your website, off you go.

      It takes a rather black-box approach to the SUT, although it needs to know about CSS selectors and other DOM attributes. But a web page can be React, server-side rendered, a mix, whatever. You can test multiple sites with one spec. It detects changes to relevant DOM elements so your state changes can be synchronous or asynchronous (like changing something after an HTTP request is completed).

      It is different from PBT with models because you don’t write the simplified/abstract model. You write a specification (which is a bit different), and you can gradually make that specification more detailed. With models and PBT, in my experience, you need to capture all of the essential complexity of the SUT in your model.

      It’s different from model checking as seen in TLA+, because this is testing the implementation. It’s also using finite traces which are much smaller. In TLA+ you can check huge state spaces in seconds. Here you might test a few hundred or thousand different cases with, say, 50 actions in each, and it’s currently in the order of minutes. But even if failing traces can be somewhat large, WebCheck tries to shrink it down (like QuickCheck does).

      Regarding implementation, the current version is built in Haskell and the logic DSL is an adaption of PureScript. I’m using the PureScript compiler as a library, but interpreting the CoreFn representation and adding the temporal logic operators next and always, along with queryAll. The specification must be a pure expression, it doesn’t support Effect from PureScript. But it supports PureScript packages! You can use monad transformers in your specs, if you’d be so inclined.

    2. 2

      What’s your plan for making sure the tool can discover actions even for, let’s say, non-semantic SPAs? ie. it might have onClick handlers on arbitrary divs, instead of as or buttons.

      I guess another way to ask is: when in the example you say:

      WebCheck generates click actions for all clickable elements.

      What is the definition of all clickable elements?

      I guess it could be a nice way to force folks to make their SPAs more accessible by requiring either correct elements or some ARIA / role=“button”-like attributes… But to help adoption, it should be possible for users to say their own rules on what’s clickable?

      1. 2

        Yeah, that’s a good point. Currently, WebCheck has its rather strict rules about “clickability”, like it has to be an anchor, button, or submit input, not be disabled or out-of-viewport, and so on. Maybe there is need for a custom “forced” click or whatever to support such JS solutions.

        1. 1

          Most frontend frameworks bind a document click handler and then dispatch events based on the current target.

          Figuring out which elements have event handlers attached statically is essentially impossible in these cases, other than by instrumenting all common frameworks.

          1. 1

            Yeah. Ultimately, it’s up to the user to define a set of possible actions (in the spec) that provide good enough coverage. WebCheck can be somewhat smart, but the specification writer might need to narrow down the search and be more specific than just “all possible clicks”.

            1. 1

              Yeah - that was roughly where I got to when I looked at implementing a similar tool (targeting screenshot testing).

              I’d actually be quite interested in a SAAS which would explore the state space of a page and show me screenshots of all the unique states it finds (particularly if it does so for a few browsers / screen sizes). Being able to attach an image to a pull request showing what the page looked like before / after is awesome but often overlooked.

              1. 1

                Definitely! I’ve thought about attaching screenshots in WebCheck to introspect failed state transitions. I don’t really have any plans yet for non-failing behaviors, but I guess it could be an option!

                1. 1

                  Selling any kind of developer tooling is super hard - here’s hoping for your success!

                  1. 1

                    Ugh, yeah it’s likely a steep uphill battle.Maybe if I can win the hearts of developers, but it’s hard with test tools. I think it’s not enough to just “find weird bugs”. It has to do that without adding a bunch of effort, because in many jobs you aren’t encouraged to do advanced testing or formal specification. It’s by your own choice. If WebCheck can help without introducing tons of extra work, I think it might be very appealing.