Thursday Jun 17, 2010

Unit tests and checked exceptions

In the past, when I've seen test methods written by others that use throws Exception in the message signature, I've been irritated. Using throws Exception looks like a lazy attempt to circumvent the requirement to explicitly declare all thrown exceptions. But I think I've changed my mind on this, at least when writing test cases.

The main reason is one of maintainability. If you use some method throughout your test suite, and it is one day updated to throw a new exception, you now have to update the method signature of every single test case that calls that method. And, as far as I can figure, you gain exactly nothing for your efforts.

But, you might object, the requirement to explicitly declare all exceptions is a necessary evil, because it allows any calling code to be written defensively. You can't prepare for exceptional conditions that you don't know about. And that's true, and in most cases, you should continue to explicitly declare all exceptions. But I'd wager that with most test cases, you don't care about exceptions that might occur, except insofar as they cause the test to pass or fail, and you probably don't intend for any other code to call your test case method.

It will be interesting to see what other apparent stupidities eventually reveal their wisdom to me.

Thursday May 06, 2010

Words to live by

If I could give only one piece of advice to anybody who writes test code for a living, it would be this: Don't sleep, poll.

Friday Mar 19, 2010

Readable failures with theories and parameterized tests

The original was unfortunately lost while cleaning out spam, but a commenter recently asked a good question about parameterized tests and theories, which I'll paraphrase: How can failures be made more readable?

This is one of the initial disappointments of using parameterized tests and theories. If a failure occurs, JUnit will report the index of your data set where the failure occurred, but not the actual value. If your data set is huge, the failure can be hard to troubleshoot. For example:

org.junit.experimental.theories.internal.ParameterizedAssertionError: theoryOligarchsHaveYachts(data[2]) at org.junit.experimental.theories.Theories$TheoryAnchor.reportParameterizedError(

The solution is actually quite simple. Every JUnit assert method accepts a String message as its first argument, so you can use this to report the current data point, your actual result, and, if available, the expected result. Here's an example from a parameterized test class:

    1 @Test
    2 public void testWhatever()
    3 {
    4     Whatever w = new Whatever();
    5     String actualResult = w.doWhatever(this.datum);
    6     String errorMsg = "Given " this.datum + 
    7         ", expected " + this.expectedResult + 
    8         " but got " + actualResult;
    9     assertThat(errorMsg, actualResult, is(this.expectedResult));
   10 }

Thursday Mar 04, 2010

JUnit theories

So, theories. Whereas parameterized tests are generally built around a known set of inputs and expected outputs, a theories-based test focuses on the generalized relationship between inputs and outputs. It might help you to understand the difference if you try to recall learning about mathematical functions for the first time in junior high or high school.

Structure of a theory class

Structurally, a theory-based class is simpler than a parameterized test class. The class declaration should be annotated with @RunWith(Theories.class), and it must provide two entities:

  1. A data method that generates and returns test data, and
  2. A theory.

The data method must be annotated with @DataPoints, and each theory must be annotated with @Theory. As with an ordinary unit test, each theory should contain at least one assertion. I recommend limiting each theory to a single assertion; this constrains the scope of any given theory and makes failures more meaningful.

But what exactly is a theory?

Functionally, a theory is a just a kind of test — specifically, an alternative to JUnit's parameterized tests. Semantically, a theory encapsulates the tester's understanding of an object's universal behavior. That is, whatever it is that a theory asserts, it is expected to be true for all data. In theory, theories should be especially useful for finding bugs in edge cases.

Contrast this with a typical unit test, which asserts that a specific data point will have a specific outcome, and only asserts that. (For this reason, typical unit tests are sometimes called example-based tests to contrast them with theories.)

Theory example

    1 import static org.junit.Assert.assertThat;
    2 import static org.junit.matchers.JUnitMatchers.hasItem;
    4 import org.junit.experimental.theories.DataPoints;
    5 import org.junit.experimental.theories.Theories;
    6 import org.junit.experimental.theories.Theory;
    7 import org.junit.runner.RunWith;
    9 import com.yoyodyne.employee.Oligarch;
   11 @RunWith(Theories.class)
   12 public class TheoriesExample 
   13 {
   14     /\*
   15      \* Our test data are stored in an array of Oligarch objects.
   16      \* Since this is a method like any other, you can generate data 
   17      \* dynamically or fetch it from an external source, if needed.
   18      \*/
   19     @DataPoints
   20     public static Oligarch[] data() {
   21         return new Oligarch[] {
   22             new Oligarch("Monty Burns"),
   23             new Oligarch("Don Geiss"),
   24             new Oligarch("Arthur Jensen")
   25         }
   26     }
   28     /\*
   29      \* This theory confirms that all oligarchs have yachts.
   30      \*/
   31     @Theory
   32     public void theoryOligarchsHaveYachts(Oligarch suit) {
   33         assertThat(suit.getVehicles(), hasItem("yacht"));
   34     }
   36     /\*
   37      \* This theory confirms that all oligarchs have hearts of darkest black.
   38      \*/
   39     @Theory
   40     public void theoryOligarchsAreEvil(Oligarch suit) {
   41         assertThat(suit.getSoul().getOwner().getName(), is("The Devil"));
   42     }
   43 }

When to use theories and when to use parameterized tests

As the example code hopefully shows, a theory class is generally easier to read than a parameterized test class; note how this class doesn't need fields or a constructor. And theories are intended to be more expressive of the tester's goals; the original paper that proposed theories called them "specifications that catch bugs."

You should have noticed, however, that there is no means of pairing a specific result with a specific data point. You should use theories when you can express in the form of an assertion the general relationship between a data point and an expected result, and when that relationship will hold true for all data.

In cases where you have a large set of inputs with varying results, then, you will still need to write parameterized tests. Parameterized tests give the you greater flexibility as an author, but the semantics of the test are usually implicit. (If you want to be picky, you could actually write a test in the parameterized test style that acted like a theory, but nobody except a tester likes a pedant.)

Wednesday Mar 03, 2010

JUnit paramaterized tests vs. theories

So what about JUnit Theories? As I understand it, Theories are functionally similar to parameterized tests, but are expressively richer. A Theory expresses the tester's understanding of how a piece of code is expected to work, moving the focus away from input/output sets.

The differences between the two are admittedly subtle. A couple differences that stand out to me are:

  1. Parameterized test classes may be more difficult to read, and
  2. Theories better express the tester's intent. Parameterized tests require a reader to infer the relationship between inputs and outputs.

At the same time, parameterized tests can often be written more quickly, as the tester's only challenge is to list a set of proper inputs and outputs; the tester does not need to abstract the relationship between the inputs and outputs into a Theory. A Theory needs to be true for all test data, and this may be tricky to express in some situations.

Parameterized unit tests with JUnit 4

I'm going to share a few non-proprietary posts that I wrote for the benefit of my coworkers. I'll start with a post on JUnit. The JUnit 4 series introduced a range of features that make writing tests both easier and more expressive, but finding documentation for even simple features can be difficult.

Parameterized tests

If you've ever found yourself writing a series of tests which differ only in their inputs and expected results, you've probably realized that the sensible thing to do would be to abstract your tests into a single test that can be run against a varying set of data. JUnit 4 allows you to do this with either theories or parameterized tests; here, I'll discuss the latter.

Structure of a parameterized test class

To mark a test class as a parameterized test, you must first annotate it with @RunWith(Parameterized.class). The class must then provide at least three entities:

  1. A static method that generates and returns test data,
  2. A single constructor that stores the test data, and
  3. A test.

The method that generates test data must be annotated with @Parameters, and it must return a Collection of Arrays. Each array represents the data to be used in a particular test run. The number of elements in each array must correspond to the number of parameters in the class's constructor, because each array element will be passed to the constructor, one at a time as the class is instantiated over and over.

The constructor is simply expected to store each data set in the class's fields, where they can be accessed by the test methods. Note that only a single constructor may be provided. This means that each array provided by the data-generating method must be the same size, and you might have to pad your data sets with nulls if you don't always need a particular value.

Let's put this together. When the test runner is invoked, the data-generating method will be executed, and it will return a Collection of Arrays, where each array is a set of test data. The test runner will then instantiate the class and pass the first set of test data to the constructor. The constructor will store the data in its fields. Then each test method will be executed, and each test method will have access to that first set of test data. After each test method has executed, the object will be instantiated again, this time using the second element in the Collection of Arrays, and so on.

Parameterized test example

A sample test class with comments is below. In this class, each data set consists of a single test input and an expected result, but you can put any data that you need in there.

    1 import java.util.Arrays;
    2 import java.util.Collection;
    4 import org.junit.Test;
    5 import org.junit.runners.Parameterized;
    6 import org.junit.runners.Parameterized.Parameters;
    7 import org.junit.runner.RunWith;
    9 import static org.hamcrest.CoreMatchers.\*;
   10 import static org.junit.Assert.\*;
   12 import com.yoyodyne.something.Whatever;
   14 @RunWith(Parameterized.class)
   15 public class ParameterizedTestExample
   16 {
   17    // Fields
   18    private String datum;
   19    private String expectedResult;
   21    /\*
   22     \* Constructor.
   23     \* The JUnit test runner will instantiate this class once for every
   24     \* element in the Collection returned by the method annotated with
   25     \* @Parameters.
   26     \*/
   27    public ParameterizedTestExample(String datum, String expected)
   28    {
   29       this.datum = datum;
   30       this.expected = expected;
   31    }
   33    /\*
   34     \* Test data generator.
   35     \* This method is called the the JUnit parameterized test runner and
   36     \* returns a Collection of Arrays.  For each Array in the Collection,
   37     \* each array element corresponds to a parameter in the constructor.
   38     \*/
   39    @Parameters
   40    public static Collection<Object[]> generateData()
   41    {
   42       // In this example, the parameter generator returns a List of
   43       // arrays.  Each array has two elements: { datum, expected }.
   44       // These data are hard-coded into the class, but they could be
   45       // generated or loaded in any way you like.
   46       return Arrays.asList(new Object[][]) {
   47          { "AGCCG", "AGTTA" },
   48          { "AGTTA", "GATCA" },
   49          { "GGGAT", "AGCCA" }
   50       }
   51    }
   53    /\*
   54     \* The test.
   55     \* This test method is run once for each element in the Collection returned
   56     \* by the test data generator -- that is, every time this class is
   57     \* instantiated. Each time this class is instantiated, it will have a
   58     \* different data set, which is available to the test method through the
   59     \* instance's fields.
   60     \*/
   61    @Test
   62    public void testWhatever()
   63    {
   64       Whatever w = new Whatever();
   65       String actualResult = w.doWhatever(this.datum);
   66       assertThat(actualResult, is(this.expectedResult));
   67    }
   68 }

A weblog about identity management and testing. See here.


« April 2014