Thursday Jun 17, 2010

Unit tests and checked exceptions

In the past, when I've seen test methods written by others that use throws Exception in the message signature, I've been irritated. Using throws Exception looks like a lazy attempt to circumvent the requirement to explicitly declare all thrown exceptions. But I think I've changed my mind on this, at least when writing test cases.

The main reason is one of maintainability. If you use some method throughout your test suite, and it is one day updated to throw a new exception, you now have to update the method signature of every single test case that calls that method. And, as far as I can figure, you gain exactly nothing for your efforts.

But, you might object, the requirement to explicitly declare all exceptions is a necessary evil, because it allows any calling code to be written defensively. You can't prepare for exceptional conditions that you don't know about. And that's true, and in most cases, you should continue to explicitly declare all exceptions. But I'd wager that with most test cases, you don't care about exceptions that might occur, except insofar as they cause the test to pass or fail, and you probably don't intend for any other code to call your test case method.

It will be interesting to see what other apparent stupidities eventually reveal their wisdom to me.

Figuring out how to script WebSphere app server with Jython

A single, comprehensive reference document for WebSphere application server's Jython API is doubtless hidden away in the dark, tangled thicket that is IBM's web site. I am as certain of this fact as I am certain that I will never find it. When, long ago, I wrote a script to automate deployments to a WebSphere app server, I cobbled it together using both the odd blog post here and there as well as the vague samples scattered through IBM's documentation -- and some of those samples were written in JACL, not Jython. It almost goes without saying that there has always been an easier way to find this information.

It helps, first of all, to know that four objects are available as global variables to any Jython process that has been created by wsadmin. Those objects are:

  • AdminConfig
  • AdminControl
  • AdminApp
  • AdminTask

You can look at these wsadmin objects if you fire up the interactive Jython interpreter: $WASHOME/bin/ -lang jython, but if you're familiar with Python's built-in introspection commands, like dir() and help(), you'll quickly see that they're of no use to you. Fortunately, the four wsadmin objects implement their own help() methods. To use them, print the output of For example, print prints out the following:

WASX7095I: The AdminApp object allows application objects to be manipulated -- this includes installing, uninstalling, editing, and listing. Most of the commands supported by AdminApp operate in two modes: the default mode is one in which AdminApp communicates with theWebSphere server to accomplish its tasks. A local mode is also possible, in which no server communication takes place. The local mode of operation is invoked by bringing up the scripting client with no server connected using the command line "-conntype NONE" option or setting the "" property in the

deleteUserAndGroupEntries    Deletes all the user/group information for all the roles and all the user name/password information for RunAs roles for a given application.
edit    Edit the properties of an application
editInteractive    Edit the properties of an application interactively...

Calling help() with the name of a command provides details on that command. So print"edit") prints out:

WASX7104I: Method: edit

Arguments: application name, options

Description: Modifies the application specified by "application name" using the options specified by "options". The user is not prompted for any information.

This is absolutely basic stuff, but it took me forever to find it. Later, I'll cover the basics of scripting a deployment, which will exhaust my knowledge of this subject.

Thursday May 06, 2010

Words to live by

If I could give only one piece of advice to anybody who writes test code for a living, it would be this: Don't sleep, poll.

Friday Mar 19, 2010

Readable failures with theories and parameterized tests

The original was unfortunately lost while cleaning out spam, but a commenter recently asked a good question about parameterized tests and theories, which I'll paraphrase: How can failures be made more readable?

This is one of the initial disappointments of using parameterized tests and theories. If a failure occurs, JUnit will report the index of your data set where the failure occurred, but not the actual value. If your data set is huge, the failure can be hard to troubleshoot. For example:

org.junit.experimental.theories.internal.ParameterizedAssertionError: theoryOligarchsHaveYachts(data[2]) at org.junit.experimental.theories.Theories$TheoryAnchor.reportParameterizedError(

The solution is actually quite simple. Every JUnit assert method accepts a String message as its first argument, so you can use this to report the current data point, your actual result, and, if available, the expected result. Here's an example from a parameterized test class:

    1 @Test
    2 public void testWhatever()
    3 {
    4     Whatever w = new Whatever();
    5     String actualResult = w.doWhatever(this.datum);
    6     String errorMsg = "Given " this.datum + 
    7         ", expected " + this.expectedResult + 
    8         " but got " + actualResult;
    9     assertThat(errorMsg, actualResult, is(this.expectedResult));
   10 }

Thursday Mar 04, 2010

JUnit theories

So, theories. Whereas parameterized tests are generally built around a known set of inputs and expected outputs, a theories-based test focuses on the generalized relationship between inputs and outputs. It might help you to understand the difference if you try to recall learning about mathematical functions for the first time in junior high or high school.

Structure of a theory class

Structurally, a theory-based class is simpler than a parameterized test class. The class declaration should be annotated with @RunWith(Theories.class), and it must provide two entities:

  1. A data method that generates and returns test data, and
  2. A theory.

The data method must be annotated with @DataPoints, and each theory must be annotated with @Theory. As with an ordinary unit test, each theory should contain at least one assertion. I recommend limiting each theory to a single assertion; this constrains the scope of any given theory and makes failures more meaningful.

But what exactly is a theory?

Functionally, a theory is a just a kind of test — specifically, an alternative to JUnit's parameterized tests. Semantically, a theory encapsulates the tester's understanding of an object's universal behavior. That is, whatever it is that a theory asserts, it is expected to be true for all data. In theory, theories should be especially useful for finding bugs in edge cases.

Contrast this with a typical unit test, which asserts that a specific data point will have a specific outcome, and only asserts that. (For this reason, typical unit tests are sometimes called example-based tests to contrast them with theories.)

Theory example

    1 import static org.junit.Assert.assertThat;
    2 import static org.junit.matchers.JUnitMatchers.hasItem;
    4 import org.junit.experimental.theories.DataPoints;
    5 import org.junit.experimental.theories.Theories;
    6 import org.junit.experimental.theories.Theory;
    7 import org.junit.runner.RunWith;
    9 import com.yoyodyne.employee.Oligarch;
   11 @RunWith(Theories.class)
   12 public class TheoriesExample 
   13 {
   14     /\*
   15      \* Our test data are stored in an array of Oligarch objects.
   16      \* Since this is a method like any other, you can generate data 
   17      \* dynamically or fetch it from an external source, if needed.
   18      \*/
   19     @DataPoints
   20     public static Oligarch[] data() {
   21         return new Oligarch[] {
   22             new Oligarch("Monty Burns"),
   23             new Oligarch("Don Geiss"),
   24             new Oligarch("Arthur Jensen")
   25         }
   26     }
   28     /\*
   29      \* This theory confirms that all oligarchs have yachts.
   30      \*/
   31     @Theory
   32     public void theoryOligarchsHaveYachts(Oligarch suit) {
   33         assertThat(suit.getVehicles(), hasItem("yacht"));
   34     }
   36     /\*
   37      \* This theory confirms that all oligarchs have hearts of darkest black.
   38      \*/
   39     @Theory
   40     public void theoryOligarchsAreEvil(Oligarch suit) {
   41         assertThat(suit.getSoul().getOwner().getName(), is("The Devil"));
   42     }
   43 }

When to use theories and when to use parameterized tests

As the example code hopefully shows, a theory class is generally easier to read than a parameterized test class; note how this class doesn't need fields or a constructor. And theories are intended to be more expressive of the tester's goals; the original paper that proposed theories called them "specifications that catch bugs."

You should have noticed, however, that there is no means of pairing a specific result with a specific data point. You should use theories when you can express in the form of an assertion the general relationship between a data point and an expected result, and when that relationship will hold true for all data.

In cases where you have a large set of inputs with varying results, then, you will still need to write parameterized tests. Parameterized tests give the you greater flexibility as an author, but the semantics of the test are usually implicit. (If you want to be picky, you could actually write a test in the parameterized test style that acted like a theory, but nobody except a tester likes a pedant.)

Wednesday Mar 03, 2010

JUnit paramaterized tests vs. theories

So what about JUnit Theories? As I understand it, Theories are functionally similar to parameterized tests, but are expressively richer. A Theory expresses the tester's understanding of how a piece of code is expected to work, moving the focus away from input/output sets.

The differences between the two are admittedly subtle. A couple differences that stand out to me are:

  1. Parameterized test classes may be more difficult to read, and
  2. Theories better express the tester's intent. Parameterized tests require a reader to infer the relationship between inputs and outputs.

At the same time, parameterized tests can often be written more quickly, as the tester's only challenge is to list a set of proper inputs and outputs; the tester does not need to abstract the relationship between the inputs and outputs into a Theory. A Theory needs to be true for all test data, and this may be tricky to express in some situations.

Parameterized unit tests with JUnit 4

I'm going to share a few non-proprietary posts that I wrote for the benefit of my coworkers. I'll start with a post on JUnit. The JUnit 4 series introduced a range of features that make writing tests both easier and more expressive, but finding documentation for even simple features can be difficult.

Parameterized tests

If you've ever found yourself writing a series of tests which differ only in their inputs and expected results, you've probably realized that the sensible thing to do would be to abstract your tests into a single test that can be run against a varying set of data. JUnit 4 allows you to do this with either theories or parameterized tests; here, I'll discuss the latter.

Structure of a parameterized test class

To mark a test class as a parameterized test, you must first annotate it with @RunWith(Parameterized.class). The class must then provide at least three entities:

  1. A static method that generates and returns test data,
  2. A single constructor that stores the test data, and
  3. A test.

The method that generates test data must be annotated with @Parameters, and it must return a Collection of Arrays. Each array represents the data to be used in a particular test run. The number of elements in each array must correspond to the number of parameters in the class's constructor, because each array element will be passed to the constructor, one at a time as the class is instantiated over and over.

The constructor is simply expected to store each data set in the class's fields, where they can be accessed by the test methods. Note that only a single constructor may be provided. This means that each array provided by the data-generating method must be the same size, and you might have to pad your data sets with nulls if you don't always need a particular value.

Let's put this together. When the test runner is invoked, the data-generating method will be executed, and it will return a Collection of Arrays, where each array is a set of test data. The test runner will then instantiate the class and pass the first set of test data to the constructor. The constructor will store the data in its fields. Then each test method will be executed, and each test method will have access to that first set of test data. After each test method has executed, the object will be instantiated again, this time using the second element in the Collection of Arrays, and so on.

Parameterized test example

A sample test class with comments is below. In this class, each data set consists of a single test input and an expected result, but you can put any data that you need in there.

    1 import java.util.Arrays;
    2 import java.util.Collection;
    4 import org.junit.Test;
    5 import org.junit.runners.Parameterized;
    6 import org.junit.runners.Parameterized.Parameters;
    7 import org.junit.runner.RunWith;
    9 import static org.hamcrest.CoreMatchers.\*;
   10 import static org.junit.Assert.\*;
   12 import com.yoyodyne.something.Whatever;
   14 @RunWith(Parameterized.class)
   15 public class ParameterizedTestExample
   16 {
   17    // Fields
   18    private String datum;
   19    private String expectedResult;
   21    /\*
   22     \* Constructor.
   23     \* The JUnit test runner will instantiate this class once for every
   24     \* element in the Collection returned by the method annotated with
   25     \* @Parameters.
   26     \*/
   27    public ParameterizedTestExample(String datum, String expected)
   28    {
   29       this.datum = datum;
   30       this.expected = expected;
   31    }
   33    /\*
   34     \* Test data generator.
   35     \* This method is called the the JUnit parameterized test runner and
   36     \* returns a Collection of Arrays.  For each Array in the Collection,
   37     \* each array element corresponds to a parameter in the constructor.
   38     \*/
   39    @Parameters
   40    public static Collection<Object[]> generateData()
   41    {
   42       // In this example, the parameter generator returns a List of
   43       // arrays.  Each array has two elements: { datum, expected }.
   44       // These data are hard-coded into the class, but they could be
   45       // generated or loaded in any way you like.
   46       return Arrays.asList(new Object[][]) {
   47          { "AGCCG", "AGTTA" },
   48          { "AGTTA", "GATCA" },
   49          { "GGGAT", "AGCCA" }
   50       }
   51    }
   53    /\*
   54     \* The test.
   55     \* This test method is run once for each element in the Collection returned
   56     \* by the test data generator -- that is, every time this class is
   57     \* instantiated. Each time this class is instantiated, it will have a
   58     \* different data set, which is available to the test method through the
   59     \* instance's fields.
   60     \*/
   61    @Test
   62    public void testWhatever()
   63    {
   64       Whatever w = new Whatever();
   65       String actualResult = w.doWhatever(this.datum);
   66       assertThat(actualResult, is(this.expectedResult));
   67    }
   68 }

Friday Jan 22, 2010

Identity: A Correction

I have been informed that this weblog will in fact not discuss the philosophy of identity, but will instead have something to do with software. I sincerely regret the error.


Who am I? What are the necessary and sufficient conditions for personhood? In what way am I myself, and not somebody else? Do I have some kernel of selfhood independent of contingent factors, such as environment and parentage? Does my selfhood persist over time? Am I the same person now as my childhood self? Is the continuity of self constituted by consciousness or memory? What if I am an amnesiac? What if I were to fall into a persistent vegetative state? If a surgeon were to swap my brain with another person, which person could be said to be me? Can God cook a breakfast so big that even He can't finish it?

This weblog is about identity, the philosophical question of how we determine that a thing is itself and not another thing. Over the coming months, we hope to discuss -- and maybe even answer -- the above questions, and many more.

(Actually, we won't answer the last question, though please see my weblog about scale.)


A weblog about identity management and testing. See here.


« April 2014