The content of this blog was provided by Olaf van der Geest, Senior Director, TimesTen QA/Release Management at Oracle.
Our team develops, executes and evaluates tests for TimesTen, the world's fastest OLTP database. Testing a relational database is hard. The surface area to test is very large and requires a wide range of expertise and different programming skills. Over the years we have developed a very robust and extensive set of tests and completely streamlined our test environment.
Our standard test run includes more than 100,000 individual tests and this number is growing daily. What defines a test completely depends on the test developer. Some tests are small and straightforward, like making sure 'select * from dual' returns exactly one row. Another test may stress our replication feature by executing one million DML operations and makes sure all transactions were replicated properly.
We execute multiple of these test runs per night as we run them on different platforms, product versions and configurations. We use more than 1000 machines to execute these tests. The amount of test data to process every day is enormous, every single night we generate at least a quarter of a million test results. How do we handle this? One word: automation.
The test harness automatically schedules test runs and distributes tests over the available hardware. Complete test run results are collected automatically. Tests are designed to be precise in their failure reporting, and tell us what failed and why it is considered a failure. The test framework recognizes known failures automatically. New failures stand out and are quickly recognized. Core files are analyzed automatically. Relevant details of failures are packaged up and kept for analysis. Details of the failure are automatically entered into the defects database, and assigned to the responsible component owner.
It takes a considerable upfront investment to write tests suitable for automation. But the return on this investment is phenomenal, and a win-win both for the company and employees as it directly leads to greater product quality and much higher job satisfaction. Team members can now also focus all their time and energy on coding new tests instead of manually executing, triaging and recording test results. Automating this entire process and constantly looking for ways to improve it is a lot of fun and very rewarding. And remember, we are always looking for smart engineers with a passion for testing and automation to join our team!
Disclaimer: these are my personal thoughts and do not represent Oracle's official viewpoint in any way, shape, or form.