This work found over 100 bugs in SQLite, the database system humans trust and rely on more than any other, with over a trillion estimated deployments. I always enjoy skimming the SQLite testing page for inspiration with my own database projects. Note that even though they found more bugs in SQLite than the other systems, this was because they prioritized testing efforts by how responsive the project was. SQLite fixed many issues within hours of reporting, so they got rewarded with more testing and more bugs discovered.
This works by comparing two queries that SHOULD result in identical output, one being easy to optimize and another being more or less optimal already, and asserting their results are the same. A similar approach has been applied with spectacular results to compilers.
“Our core in-sight is that a given, potentially randomly-generatedoptimizedquerycan be rewritten to one that the DBMS cannot optimize.”
Clever. Reminds me of how tools like scrypt try to make password hashing that can’t be optimized with parallelism (esp FPGA’s or ASICs). Another team might also want to try to solve the original problem of a DB fork with optimizations turned off for verification and testing purposes.