News, tips, partners, and perspectives for the Oracle Linux operating system and upstream Linux kernel work

Oracle's new Kernel Test Framework for Linux

Kernel developers Knut Omang and Alan Maguire collaborate on the new project KTF. Available on github, KTF is an extensible framework for doing better testing and validation for Linux developers. As Knut likes to remind us, test-driven-development can work even for writing new device drivers -- because he did just that on his previous project!

Kernel Test Framework (KTF) is a unit test framework we have developed to make it easier to test internal and external programming interfaces of the kernel. Unit testing and test driven development (TDD) is a recognized and much used development methodology for user space projects. Elements of the kernel are currently tested via user land unit test libraries or with standard applications and script logic and more ad-hoc test frameworks from user land. User land testing is limited in that it can only directly exercise the interfaces and code paths that can be made reachable from user space code. This limits potential coverage - error paths and other paths that are not easy to trigger from user space code cannot be easily tested. The tests that can be made is also often too high level, in that they execute a lot of functionality. Debugging intricate kernel issues can be immensely complicated. Having a way to make targeted tests that only exercises a small piece of code deep within the kernel can aid in narrowing down such issues. KTF allows tests to be written, in separate kernel modules, to test for instance internal, non-exported interfaces of kernel modules or other kernel components.

User land tests with broad coverage can be good for verification of use cases but not as good as developer tools for analysing and debugging identified issues. Also an important use of unit tests is for developers themselves to be able to make executable assumptions about how existing or new code works. Coupled with a continuous integration system, where tests are added incrementally as they are created or issues are found, this form of test driven, or assertion driven development becomes powerful as it allows individual developers to put 'guards' around the assumptions they have made about the semantics of the interfaces they use. If a kernel change later breaks any of these assumptions, it will be detected by the tests, avoiding a possible regression in the dependent module.

KTF is available on github, and we're keen to work with others who value TDD and want to bring it to the LInux kernel! It is encouraging to see continuous integration being adopted by kernel contributors like Intel.

KTF was conceived with a clear realization that for unit testing and more generic, systematic testing to gain traction, it really needs to be simple and easy to use, and individual developers must feel that they gain something for their own by adding tests and by making them available for others or even facilitate automated test runs. With KTF we have tried to take these considerations into the design.

With KTF we have tried to facilitate test-driven-development by making it

  1. Easy to run tests. Once the ktf module has been loaded, kernel test sets are added via dedicated modules.  Once loaded, all available tests can be run via the "ktfrun" command, a user level test program. This user program does not contain any test code by itself, instead it uses netlink to communicate with the kernel ktf module to query for available tests. It then uses the features of gtest, an established user land unit test framework, to select tests to run, and to generate nicely formatted reports of results.
    Test developers are encouraged to write zero-configuration tests, or at least tests that run with a sensible default configuration.  For example a network test could use loopback by default, if possible.
  2. Easy to debug/diagnose test failures. Tests are written using a set of assertion templates, i..e. what we expect to be true during a test. If an assertion fails the error is logged along with the line number/file in which it was triggered, helping to diagnose failures.  The gtest output emphasises pass/fail and shows the details of any failures. Results of the previous test run are always available via debugfs via

       # cat /sys/kernel/debug/ktf/results/<testset>

In the worst case if a test panics the kernel, postmortem debugging of crash dumps is possible. We'll describe how that is done later. Individual tests should be independent, i.e. test B should not rely on setup done in test A, that helps ensure a more reproducible environment where passes and fails are more consistent. 

3. Easy to write tests.   Tests are defined like this:  

              TEST(examples, hello_ok)

The test case here is "examples", the test "hello_ok". Tests consist of a set of ASSERT_*() or EXPECT_*() statements combined with any setup/teardown needed to make the assertions.  EXPECT_*() differs from ASSERT_*() in that the latter is considered fatal to the test and execution of the test is terminated.   KTF also supports loop tests, where a test is added with a start and end index and the index is implicitly passed into the test for use.  See examples/h2.c for an example of loop test usage.   Finally, KTF also supports the concept of a fixture - a way of defining a set of tests with common setup/teardown. See examples/h3.c for a fixture example.

4. Easy to carry out analysis around testing, i.e. measure test coverage, memory utilization during testing.   It's always valuable to measure how extensively tests cover the codebase, and this is particularly true for cases where tests are being written for an existing codebase.  The Linux kernel has code coverage mechanismns via gcc, but these are gcc-version dependent and cannot be applied dynamically to an already-built kernel.  For KTF we therefore developed simple code coverage support.

For now we have made KTF an independent out-of-tree module and made it easy to have multiple build trees from the same source tree towards different kernel versions, but we also recognize that having KTF in the kernel repository itself in the long run might make it easier to maintain it and to ensure a growing set of tests, and that these tests routinely get run. We do believe that one does not exclude the other and have ideas for how this can be arranged in practice, with minimal extra maintenance burden. Follow this blog for further development!

Join the discussion

Comments ( 1 )
  • sou_Lina Tuesday, November 20, 2018
    Could we use KTF as integration test framework since it supports combined tests ?
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.