My way to verify MySQL bug reports

I promised to write this blog post long time ago at one of conferences in Russia. Don't know why I delayed this, but finally I did.

We, members of MySQL bugs verification group, have to verify bugs in all currently supported versions. We use not only version reported, but test in development source tree for each of supported major versions and identify recent regressions.

You can imagine that even if I would do so for simple bug report about wrong results with perfect test case, which requires me simply run few queries I would have to start 4 or more MySQL servers: one for each of currently supported versions 5.0, 5.1, 5.5 plus one for current development. And unknown number of servers if I could not repeat or if I want to check if this is regression.

Even if I have all these basic 4 servers running I still should type all these queries at least 4 times. How much time it would take to verify single bug report if I did so?

I know some members of my group preferred this way, because typing queries manually is same action which our customers do. Again, some bugs are repeatable only if you type queries manually.

But I prefer to test manually erroneous exceptions only and don't make it my routine job.


So how do I test bug reports?


Every version of MySQL server comes with regression test suite: a program, called mtr (mysql-test-run.pl), its libraries, mysqltest program (you should not call it directly, though) and set of tests. Good thing with MySQL test suite is that you can create your own test cases. So do I.

I write my tests in MTR format, then run MTR with record option and examine result. Actually this is kind of hack, because users expected to create result file first, then compare output of running test with that result file. But my purpose is to repeat bug report, not to create proper test case for it, so I can be lazy.

But simply running MTR manually still takes time. And I found a way to automate this process as well.

I created a BASH script, called do_test.sh, which run through all my MySQL trees and runs tests for me automatically, then prints result.

Let me explain it a little bit.


$ cat ~/scripts/do_test.sh

#!/bin/bash

# runs MySQL tests in all source directories

# prints usage information

usage ()

{

    echo "$VERSION"

    echo "

do_test copies MySQL test files from any place

to each of source directory, then runs them

Usage: `basename $0` [option]... [testfile ...]

    or `basename $0` [option]... -d dirname [test ...]

    or `basename $0` [option]... [-b build [build option]... ]...

Options:

    -d --testdir    directory, contains test files

I have a directory, there I store test files. It has subdirectory t where tests to be run are stored, subdirectory r, where results, sorted by MySQL server version number, are stored, and directory named archive, there test are stored for archiving purpose.


    -s --srcdir     directory, contains sources directories

This is path to the directory where MySQL package is located. I called it srcdir, but this is actually not so strict: program will work with binary packages as well.


    -b --build      mysql source directory

Name of MySQL source directory. You can specify any package name. For example, to run tests in 5.6.9 package in my current dir I call the program as do_test -s . -b mysql-5.6.9-rc-linux-glibc2.5-x86_64


    -c --clean      remove tests from src directory after execution

    -t --suite      suite where to put test

MTR can have test suites with their own rules of how to run test case. If you want to run your tests in specific suite, specify this option. You can also have directory for your own suite, but in this case you need to create directories your_suite, your_suite/t and your_suite/r in mysql-test/suite directory of your MySQL installation prior doing this.

As I told I am lazy, so I run tests in main test suite mostly. This can be not good idea if you use MySQL installation not only for tests of its own bugs, but for some other tests.

Rest of the code speaks for itself, so I would not explain it. What you need to do to run this program is simply call it: do_test.sh and pass paths to your test, src dir and MySQL installation.


    -v --version    print version number, then exit

    -h --help       print this help, then exit

You can also pass any option to mysqltest program.

    "

}

# error exit

error()

{

    printf "$@" >&2

    exit $E_CDERROR

}

# creates defaults values

initialize()

{

This probably not very obvious. These are my default paths and, most importantly, default set of servers I test


    TESTDIR=/home/sveta/src/tests

    SRCDIR=/home/sveta/src

    BUILDS="mysql-5.0 mysql-5.1 mysql-5.5 mysql-trunk"

    CLEAN=0 #false

    MYSQLTEST_OPTIONS="--record --force"

    TESTS_TO_PASS=""

    TESTS=""

    SUITE=""

    SUITEDIR=""

    OLD_PWD=`pwd`

    VERSION="do_test v0.2 (May 28 2010)"

}

# parses arguments/sets values to defaults

parse()

{

    TEMP_BUILDS=""

    

    while getopts "cvhd:s:b:t:" Option

    do

        case $Option in

            c) CLEAN=1;;

            v) echo "$VERSION";;

            h) usage; exit 0;;

            d) TESTDIR="$OPTARG";;

            s) SRCDIR="$OPTARG";;

            b) TEMP_BUILDS="$TEMP_BUILDS $OPTARG";;

            t) SUITE="$OPTARG"; SUITEDIR="/suite/$SUITE"; MYSQLTEST_OPTIONS="$MYSQLTEST_OPTIONS --suite=$SUITE";;

            *) usage; exit 0; ;;

        esac

    done

    if [[ $TEMP_BUILDS ]]

    then

        BUILDS="$TEMP_BUILDS"

    fi

}

# copies test to source directories

copy()

{

    cd "$TESTDIR/t"

    TESTS_TO_PASS=`ls *.test 2>/dev/null | sed s/.test$//`

    cd $OLD_PWD

    for build in $BUILDS

    do

        #cp -i for reject silent overload

        cp "$TESTDIR"/t/*.{test,opt,init,sql,cnf} "$SRCDIR/$build/mysql-test$SUITEDIR/t" 2>/dev/null

    done

}

# runs tests

run()

{

    for build in $BUILDS

    do

        cd "$SRCDIR/$build/mysql-test"

        ./mysql-test-run.pl $MYSQLTEST_OPTIONS $TESTS_TO_PASS

    done

    cd $OLD_PWD

}

# copies result and log files to the main directory

get_result()

{

    for build in $BUILDS

    do

        ls "$TESTDIR/r/$build" 2>/dev/null

        if [[ 0 -ne $? ]]

        then

            mkdir "$TESTDIR/r/$build"

        fi

        for test in $TESTS_TO_PASS

        do

            cp "$SRCDIR/$build/mysql-test$SUITEDIR/r/$test".{log,result} "$TESTDIR/r/$build" 2>/dev/null

        done

    done

}

# removes tests and results from MySQL sources directories

cleanup()

{

    if [[ 1 -eq $CLEAN ]]

    then

        for build in $BUILDS

        do

            for test in $TESTS_TO_PASS

            do

                rm "$SRCDIR/$build/mysql-test$SUITEDIR/r/$test".{log,result} 2>/dev/null

                rm "$SRCDIR/$build/mysql-test$SUITEDIR/t/$test.test"

            done

        done

    fi

}

# shows results

show()

{

    for build in $BUILDS

    do

        echo "=====$build====="

        for test in $TESTS_TO_PASS

        do

            echo "=====$test====="

            cat "$TESTDIR/r/$build/$test".{log,result} 2>/dev/null

            echo

        done

        echo

    done

}

E_CDERROR=65

#usage

initialize

parse $@

copy

run

get_result

cleanup

show

exit 0

After I finished with test I copy it to archive directory, again, with a script, named ar_test.sh:


$ cat ~/scripts/ar_test.sh

#!/bin/bash

# moves MySQL tests from t to archive directory and clean ups r directories

# prints usage information

usage ()

{

    echo "$VERSION"

    echo "

ar_test copies MySQL test files from t to archive folder

Usage: `basename $0` [-v] [-d dirname] [test ...]

Options:

    -d    directory, contains test files

    -v    print version

    -h    print this help

    "

}

# error exit

error()

{

    printf "$@" >&2

    exit $E_CDERROR

}

# creates defaults values

initialize()

{

    TESTDIR=/home/sveta/src/tests

    TESTS_TO_MOVE=""

    OLD_PWD=`pwd`

    VERSION="ar_test v0.2 (Dec 01 2011)"

}

# parses arguments/sets values to defaults

parse()

{    

    while getopts "vhd:" Option

    do

        case $Option in

            v) echo "$VERSION"; shift;;

            h) usage; exit 0;;

            d) TESTDIR="$OPTARG"; shift;;

            *) usage; exit 0;;

        esac

    done

    

    TESTS_TO_MOVE="$@"

}

# copies test to source directories

copy()

{

    if [[ "xx" = x"$TESTS_TO_MOVE"x ]]

    then

        cp "$TESTDIR"/t/* "$TESTDIR"/archive 2>/dev/null

    else

        for test in $TESTS_TO_MOVE

        do

            cp "$TESTDIR/t/$test".{test,opt,init,sql} "$TESTDIR"/archive 2>/dev/null

        done

    fi

}

# removes tests and results from r directories

cleanup()

{

    if [[ "xx" = x"$TESTS_TO_MOVE"x ]]

    then

        rm "$TESTDIR"/t/* 2>/dev/null

        rm "$TESTDIR"/r/*/* 2>/dev/null

    else

        for test in $TESTS_TO_MOVE

        do

            rm "$TESTDIR/t/$test".{test,opt,init,sql} 2>/dev/null

            rm "$TESTDIR/r/"*"/$test".{test,opt,init,sql} 2>/dev/null

        done

    fi

}

E_CDERROR=65

initialize

parse $@

copy

cleanup

exit 0

But most important part: what to do if I want to test on some specific machine which is not available at home? Fortunately, we have shared machines to run tests on, so I can simply move them to my network homedir, then choose appropriate machine and run. Since this is BASH script and test cases in MTR format this would work on any operating system.


$ cat ~/scripts/scp_test.sh

#!/bin/bash

# copies MySQL tests to remote box

# prints usage information

usage ()

{

    echo "$VERSION"

    echo "

scp_test copies MySQL test files from t directory on local box to MySQL's XXX

    

Usage: `basename $0` [-v] [-d dirname] [-r user@host:path] [test ...]

Options:

    -d    directory, contains test files

    -r    path to test directory on remote server, default: USERNAME@MACHINE_ADDRESS:~/PATH/src/tests/t

    -v    print version

    -h    print this help

    "

}

# error exit

error()

{

    printf "$@" >&2

    exit $E_CDERROR

}

# creates defaults values

initialize()

{

    TESTDIR=/home/sveta/src/tests

    MOVETO='USERNAME@MACHINE_ADDRESS:~/PATH/src/tests/t'

    TESTS_TO_MOVE=""

    OLD_PWD=`pwd`

    VERSION="scp_test v0.2 (Dec 1 2011)"

}

# parses arguments/sets values to defaults

parse()

{    

    while getopts "vhd:" Option

    do

        case $Option in

            v) echo "$VERSION"; shift;;

            h) usage; exit 0;;

            d) TESTDIR="$OPTARG"; shift;;

            r) MOVETO="$OPTARG"; shift;;

            *) usage; exit 0;;

        esac

    done

    

    TESTS_TO_MOVE="$@"

}

# copies test to source directories

copy()

{

    if [[ "xx" = x"$TESTS_TO_MOVE"x ]]

    then

        scp "$TESTDIR"/t/* "$MOVETO"

    else

        for test in $TESTS_TO_MOVE

        do

            scp "$TESTDIR/t/$test".{test,opt,init,sql} "$MOVETO"

        done

    fi

}

E_CDERROR=65

initialize

parse $@

copy

exit 0

Wanted to put them to Launchpad, but stack with name for this package. Does anybody have ideas?

Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

Working blog of Sveta Smirnova - MySQL Senior Principal Support Engineer working in Bugs Analysis Support Group

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today