The final challenge is arguably a strength. MSVC cannot ingest a serialized form of the compiler’s IR, nor can it apply exactly one optimization while skipping others. Therefore, our test input is always C++ code which passes through multiple compiler phases before reaching the phase that is the desired test subject.
GCC does this too. The vast majority of their tests are C/C++ code with directives in comments for the test framework.
I’ve had to write a bunch of tests for GCC. It’s nice to have this integration approach for testing the compiler overall, but, well, I’m less convinced by the spin put on this next part.
Although this limitation initially appears to be a weakness, it has the advantage that our tests are realistic because the IR that ultimately gets tested is less sanitized.
While true, the weakness is that it can be very hard to write C/C++ source to trigger a certain behaviour in the code you want to test. If you add a pass, getting the test suite to cover all the nooks and crannies of that logic might involve a lot of tinkering to construct a source code example that may have to “fool” other passes so that your pass gets decent input.
GCC 7 added the ability to get around this with GIMPLE and RTL tests. They’re monsterously ugly and awful to write, but it can be better than writing the source code to trigger the condition.
GCC does this too. The vast majority of their tests are C/C++ code with directives in comments for the test framework.
I’ve had to write a bunch of tests for GCC. It’s nice to have this integration approach for testing the compiler overall, but, well, I’m less convinced by the spin put on this next part.
While true, the weakness is that it can be very hard to write C/C++ source to trigger a certain behaviour in the code you want to test. If you add a pass, getting the test suite to cover all the nooks and crannies of that logic might involve a lot of tinkering to construct a source code example that may have to “fool” other passes so that your pass gets decent input.
GCC 7 added the ability to get around this with GIMPLE and RTL tests. They’re monsterously ugly and awful to write, but it can be better than writing the source code to trigger the condition.