Some of you may have run the mysql-test-run tool which is the MySQL test suite. But did you know there are actually multiple suites? If you just run the tool, you don’t get everything!
Check out the mysql-test/suites subdirectory. That’s all the stuff you don’t get when just running the tool normally. If you take a peek at the Makefiles, you will find a target test-bt (build team) which shows the extra calls and parameters for the additional suites.
OurDelta has had some interesting cases where a build that’s otherwise ok would fail when users tried the test suite on their installation. We reckon such a test should definitely pass, and thus we had some more homework to do. So now OurDelta builds with as many tests as exist enabled, on all platforms and architectures. Slow yes, but that’s not an argument to not test something, right? Failing tests are often indicative of other issues, so at the very least they merit some attention.
For instance… we found that on some platforms, the default distro packages are actually broken in fairly interesting ways. The testsuite in particular falls victim to this, making one wonder whether the distros actually test what they build, and which tests they do.