I’m thinking more and more that automated testing needs to become a part of standards generation. I recently implemented the FIDO U2F spec. On top of generally poor writing quality, there was a lot of confusion that would’ve been transparently obvious with an official test suite. Instead I spent a lot of time trying to read the semi-official reference implentations (which of course were not perfectly consistent) and, in one memorable 5-hour debugging session, running a debugger on Chrome to see what it was thinking.
The CSS acid tests were very useful for providing an objective measure of whether a browser could be said to “support” CSS. More comprehensive suites would also be an aid to implementors, not just as a checklist but in resolving ambiguities (especially with a process for adding new testcases).
I’m thinking more and more that automated testing needs to become a part of standards generation. I recently implemented the FIDO U2F spec. On top of generally poor writing quality, there was a lot of confusion that would’ve been transparently obvious with an official test suite. Instead I spent a lot of time trying to read the semi-official reference implentations (which of course were not perfectly consistent) and, in one memorable 5-hour debugging session, running a debugger on Chrome to see what it was thinking.
The CSS acid tests were very useful for providing an objective measure of whether a browser could be said to “support” CSS. More comprehensive suites would also be an aid to implementors, not just as a checklist but in resolving ambiguities (especially with a process for adding new testcases).