X
  • MySQL |
    January 26, 2013

My way to verify MySQL bug reports

Guest Author
I promised to write this blog post long time ago at one of conferences in Russia. Don't know why I delayed this, but finally I did.

We, members of MySQL bugs verification group, have to verify bugs in all currently supported versions. We use not only version reported, but test in development source tree for each of supported major versions and identify recent regressions.

You can imagine that even if I would do so for simple bug report about wrong results with perfect test case, which requires me simply run few queries I would have to start 4 or more MySQL servers: one for each of currently supported versions 5.0, 5.1, 5.5 plus one for current development. And unknown number of servers if I could not repeat or if I want to check if this is regression.

Even if I have all these basic 4 servers running I still should type all these queries at least 4 times. How much time it would take to verify single bug report if I did so?

I know some members of my group preferred this way, because typing queries manually is same action which our customers do. Again, some bugs are repeatable only if you type queries manually.

But I prefer to test manually erroneous exceptions only and don't make it my routine job.

So how do I test bug reports?


Every version of MySQL server comes with regression test suite: a program, called mtr (mysql-test-run.pl), its libraries, mysqltest program (you should not call it directly, though) and set of tests. Good thing with MySQL test suite is that you can create your own test cases. So do I.

I write my tests in MTR format, then run MTR with record option and examine result. Actually this is kind of hack, because users expected to create result file first, then compare output of running test with that result file. But my purpose is to repeat bug report, not to create proper test case for it, so I can be lazy.

But simply running MTR manually still takes time. And I found a way to automate this process as well.

I created a BASH script, called do_test.sh, which run through all my MySQL trees and runs tests for me automatically, then prints result.

Let me explain it a little bit.

$ cat ~/scripts/do_test.sh
#!/bin/bash
# runs MySQL tests in all source directories
# prints usage information
usage ()
{
    echo "$VERSION"
    echo "
do_test copies MySQL test files from any place
to each of source directory, then runs them
Usage: `basename $0` [option]... [testfile ...]
    or `basename $0` [option]... -d dirname [test ...]
    or `basename $0` [option]... [-b build [build option]... ]...
Options:
    -d --testdir    directory, contains test files

I have a directory, there I store test files. It has subdirectory t where tests to be run are stored, subdirectory r, where results, sorted by MySQL server version number, are stored, and directory named archive, there test are stored for archiving purpose.

    -s --srcdir     directory, contains sources directories

This is path to the directory where MySQL package is located. I called it srcdir, but this is actually not so strict: program will work with binary packages as well.

    -b --build      mysql source directory

Name of MySQL source directory. You can specify any package name. For example, to run tests in 5.6.9 package in my current dir I call the program as do_test -s . -b mysql-5.6.9-rc-linux-glibc2.5-x86_64

    -c --clean      remove tests from src directory after execution
    -t --suite      suite where to put test

MTR can have test suites with their own rules of how to run test case. If you want to run your tests in specific suite, specify this option. You can also have directory for your own suite, but in this case you need to create directories your_suite, your_suite/t and your_suite/r in mysql-test/suite directory of your MySQL installation prior doing this.

As I told I am lazy, so I run tests in main test suite mostly. This can be not good idea if you use MySQL installation not only for tests of its own bugs, but for some other tests.

Rest of the code speaks for itself, so I would not explain it. What you need to do to run this program is simply call it: do_test.sh and pass paths to your test, src dir and MySQL installation.

    -v --version    print version number, then exit
    -h --help       print this help, then exit
You can also pass any option to mysqltest program.
    "
}
# error exit
error()
{
    printf "$@" >&2
    exit $E_CDERROR
}
# creates defaults values
initialize()
{

This probably not very obvious. These are my default paths and, most importantly, default set of servers I test

    TESTDIR=/home/sveta/src/tests
    SRCDIR=/home/sveta/src
    BUILDS="mysql-5.0 mysql-5.1 mysql-5.5 mysql-trunk"
    CLEAN=0 #false
    MYSQLTEST_OPTIONS="--record --force"
    TESTS_TO_PASS=""
    TESTS=""
    SUITE=""
    SUITEDIR=""
    OLD_PWD=`pwd`
    VERSION="do_test v0.2 (May 28 2010)"
}
# parses arguments/sets values to defaults
parse()
{
    TEMP_BUILDS=""
    
    while getopts "cvhd:s:b:t:" Option
    do
        case $Option in
            c) CLEAN=1;;
            v) echo "$VERSION";;
            h) usage; exit 0;;
            d) TESTDIR="$OPTARG";;
            s) SRCDIR="$OPTARG";;
            b) TEMP_BUILDS="$TEMP_BUILDS $OPTARG";;
            t) SUITE="$OPTARG"; SUITEDIR="/suite/$SUITE"; MYSQLTEST_OPTIONS="$MYSQLTEST_OPTIONS --suite=$SUITE";;
            *) usage; exit 0; ;;
        esac
    done
    if [[ $TEMP_BUILDS ]]
    then
        BUILDS="$TEMP_BUILDS"
    fi
}
# copies test to source directories
copy()
{
    cd "$TESTDIR/t"
    TESTS_TO_PASS=`ls *.test 2>/dev/null | sed s/.test$//`
    cd $OLD_PWD
    for build in $BUILDS
    do
        #cp -i for reject silent overload
        cp "$TESTDIR"/t/*.{test,opt,init,sql,cnf} "$SRCDIR/$build/mysql-test$SUITEDIR/t" 2>/dev/null
    done
}
# runs tests
run()
{
    for build in $BUILDS
    do
        cd "$SRCDIR/$build/mysql-test"
        ./mysql-test-run.pl $MYSQLTEST_OPTIONS $TESTS_TO_PASS
    done
    cd $OLD_PWD
}
# copies result and log files to the main directory
get_result()
{
    for build in $BUILDS
    do
        ls "$TESTDIR/r/$build" 2>/dev/null
        if [[ 0 -ne $? ]]
        then
            mkdir "$TESTDIR/r/$build"
        fi
        for test in $TESTS_TO_PASS
        do
            cp "$SRCDIR/$build/mysql-test$SUITEDIR/r/$test".{log,result} "$TESTDIR/r/$build" 2>/dev/null
        done
    done
}
# removes tests and results from MySQL sources directories
cleanup()
{
    if [[ 1 -eq $CLEAN ]]
    then
        for build in $BUILDS
        do
            for test in $TESTS_TO_PASS
            do
                rm "$SRCDIR/$build/mysql-test$SUITEDIR/r/$test".{log,result} 2>/dev/null
                rm "$SRCDIR/$build/mysql-test$SUITEDIR/t/$test.test"
            done
        done
    fi
}
# shows results
show()
{
    for build in $BUILDS
    do
        echo "=====$build====="
        for test in $TESTS_TO_PASS
        do
            echo "=====$test====="
            cat "$TESTDIR/r/$build/$test".{log,result} 2>/dev/null
            echo
        done
        echo
    done
}
E_CDERROR=65
#usage
initialize
parse $@
copy
run
get_result
cleanup
show
exit 0

After I finished with test I copy it to archive directory, again, with a script, named ar_test.sh:

$ cat ~/scripts/ar_test.sh
#!/bin/bash
# moves MySQL tests from t to archive directory and clean ups r directories
# prints usage information
usage ()
{
    echo "$VERSION"
    echo "
ar_test copies MySQL test files from t to archive folder
Usage: `basename $0` [-v] [-d dirname] [test ...]
Options:
    -d    directory, contains test files
    -v    print version
    -h    print this help
    "
}
# error exit
error()
{
    printf "$@" >&2
    exit $E_CDERROR
}
# creates defaults values
initialize()
{
    TESTDIR=/home/sveta/src/tests
    TESTS_TO_MOVE=""
    OLD_PWD=`pwd`
    VERSION="ar_test v0.2 (Dec 01 2011)"
}
# parses arguments/sets values to defaults
parse()
{    
    while getopts "vhd:" Option
    do
        case $Option in
            v) echo "$VERSION"; shift;;
            h) usage; exit 0;;
            d) TESTDIR="$OPTARG"; shift;;
            *) usage; exit 0;;
        esac
    done
    
    TESTS_TO_MOVE="$@"
}
# copies test to source directories
copy()
{
    if [[ "xx" = x"$TESTS_TO_MOVE"x ]]
    then
        cp "$TESTDIR"/t/* "$TESTDIR"/archive 2>/dev/null
    else
        for test in $TESTS_TO_MOVE
        do
            cp "$TESTDIR/t/$test".{test,opt,init,sql} "$TESTDIR"/archive 2>/dev/null
        done
    fi
}
# removes tests and results from r directories
cleanup()
{
    if [[ "xx" = x"$TESTS_TO_MOVE"x ]]
    then
        rm "$TESTDIR"/t/* 2>/dev/null
        rm "$TESTDIR"/r/*/* 2>/dev/null
    else
        for test in $TESTS_TO_MOVE
        do
            rm "$TESTDIR/t/$test".{test,opt,init,sql} 2>/dev/null
            rm "$TESTDIR/r/"*"/$test".{test,opt,init,sql} 2>/dev/null
        done
    fi
}
E_CDERROR=65
initialize
parse $@
copy
cleanup
exit 0

But most important part: what to do if I want to test on some specific machine which is not available at home? Fortunately, we have shared machines to run tests on, so I can simply move them to my network homedir, then choose appropriate machine and run. Since this is BASH script and test cases in MTR format this would work on any operating system.

$ cat ~/scripts/scp_test.sh
#!/bin/bash
# copies MySQL tests to remote box
# prints usage information
usage ()
{
    echo "$VERSION"
    echo "
scp_test copies MySQL test files from t directory on local box to MySQL's XXX
    
Usage: `basename $0` [-v] [-d dirname] [-r user@host:path] [test ...]
Options:
    -d    directory, contains test files
    -r    path to test directory on remote server, default: USERNAME@MACHINE_ADDRESS:~/PATH/src/tests/t
    -v    print version
    -h    print this help
    "
}
# error exit
error()
{
    printf "$@" >&2
    exit $E_CDERROR
}
# creates defaults values
initialize()
{
    TESTDIR=/home/sveta/src/tests
    MOVETO='USERNAME@MACHINE_ADDRESS:~/PATH/src/tests/t'
    TESTS_TO_MOVE=""
    OLD_PWD=`pwd`
    VERSION="scp_test v0.2 (Dec 1 2011)"
}
# parses arguments/sets values to defaults
parse()
{    
    while getopts "vhd:" Option
    do
        case $Option in
            v) echo "$VERSION"; shift;;
            h) usage; exit 0;;
            d) TESTDIR="$OPTARG"; shift;;
            r) MOVETO="$OPTARG"; shift;;
            *) usage; exit 0;;
        esac
    done
    
    TESTS_TO_MOVE="$@"
}
# copies test to source directories
copy()
{
    if [[ "xx" = x"$TESTS_TO_MOVE"x ]]
    then
        scp "$TESTDIR"/t/* "$MOVETO"
    else
        for test in $TESTS_TO_MOVE
        do
            scp "$TESTDIR/t/$test".{test,opt,init,sql} "$MOVETO"
        done
    fi
}
E_CDERROR=65
initialize
parse $@
copy
exit 0

Wanted to put them to Launchpad, but stack with name for this package. Does anybody have ideas?

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
Oracle

Integrated Cloud Applications & Platform Services