Hidden tests of the MySQL test suite

Some of you may have run the mysql-test-run tool which is the MySQL test suite. But did you know there are actually multiple suites? If you just run the tool, you don’t get everything!

Check out the mysql-test/suites subdirectory. That’s all the stuff you don’t get when just running the tool normally. If you take a peek at the Makefiles, you will find a target test-bt (build team) which shows the extra calls and parameters for the additional suites.

OurDelta has had some interesting cases where a build that’s otherwise ok would fail when users tried the test suite on their installation. We reckon such a test should definitely pass, and thus we had some more homework to do. So now OurDelta builds with as many tests as exist enabled, on all platforms and architectures. Slow yes, but that’s not an argument to not test something, right? Failing tests are often indicative of other issues, so at the very least they merit some attention.

For instance… we found that on some platforms, the default distro packages are actually broken in fairly interesting ways. The testsuite in particular falls victim to this, making one wonder whether the distros actually test what they build, and which tests they do.

Tags: , , ,

5 Responses to “Hidden tests of the MySQL test suite”

  1. Giuseppe Maxia Says:

    Have you filed a bug for the tests that fail in some architectures?
    That would help


  2. Ronald Bradford Says:

    Not really hidden if you have ever taken the time to look at mysql-test or read the docs, but still good to bring to the attention of readers.

    The benefit of suites is an organization can create their own verification tests using the mysql-test infrastructure.

  3. Sheeri Says:

    Can you give some concrete examples? Otherwise it just looks like a teaser post at best (“look at the suites directory”) and FUD at its worst (“default distro packages are actually broken in fairly interesting ways.”)

    (which distros? how are they broken? Is it actually broken, or do the tests not just pass? Tests were written with MySQL’s distro in mind, so failing the test doesn’t necessarily mean anything is wrong with the package, just that it didn’t pass MySQL’s tests).

  4. arjen Says:

    Hi Giuseppe – yes we have filed some bugs. However, we have a fairly dismal experience with how bugs are handled. The assessment is quite often quite borked, the people looking at it don’t appear to have an understanding of the subject matter, and thus it takes considerable extra effort to make something happen. That makes it a time drain.

  5. arjen Says:

    Hi Sheeri!

    Ronald is absolutely right, sometimes it’s an incorrect result file rather than an actual incorrect result – but like compiler warnings, if you have lots of those it might obscure a real error.

    There’s a myriad of things wrong with various distro packages, ranging from settings to missing or incorrectly located files, to some packaging internals that go wrong.

    We come across these things as we go, and do file bug reports where possible. We have submitted bugreports to Sun/MySQL, and Debian has included some fixes from OurDelta in their build system.

    Sometimes things are wrong but cannot be fixed without breaking existing installations.

    One example is that Red Hat’s RPM build (also CentOS, of course) do not set the mysqld binary path, so it shows up as /usr/libexec which is actually the default patch configure uses for a source build.
    As we know, mysqld on Linux Standards Base (LSB) directory layout should reside in /usr/sbin