Beyond the simple: An in-depth look at JUnit 5’s nested tests, dynamic tests, parameterized tests, and extensions

With the new JUnit Jupiter test engine, the popular framework is much more flexible than ever before and can be customized at will to fit your testing needs.

August 28, 2020

Download a PDF of this article

Legend has it that the original version of JUnit was pair-programmed in an airplane in 1997. Kent Beck and Erich Gamma have since moved on from this, but JUnit is still very much alive. Time has passed since then, and a number of JUnit versions have appeared. The latest one is JUnit 5, which brings not only a completely new architecture but also a host of new features for test authors.

JUnit 5’s revamped architecture introduces the notion of a JUnit platform. At a high level, this new platform includes APIs for IDEs and build tools to integrate with testing frameworks, as well as an abstraction to implement test engines.

Several test engines can be run on the platform in a single test run, most notably the old JUnit Vintage test engine—for executing JUnit 3 and JUnit 4 tests—and the brand-new JUnit Jupiter engine, which implements the JUnit 5 programming model. This article focuses on some of the more advanced features related to writing and running JUnit Jupiter tests, namely nested, dynamic, and parameterized tests.

This article is aimed at programmers using some form of code testing in their daily work, for example, tests based on JUnit 4 or TestNG. Prior exposure to JUnit 5 is helpful but not strictly necessary.

For a general introduction, see the Java Magazine special issue on JUnit 5. Some recent advances in JUnit 5 are detailed in an article by Mert Çalişkan.

Nested tests

When the JUnit 5 team first considered nested tests in the early planning phases for the JUnit Jupiter test engine in 2015, not everyone was immediately convinced. These tests rely on nested classes, which would lead to a considerably more complex implementation than tests defined by standard nonnested classes. The team finally decided to include nested tests in the framework, because they provide a special kind of expressiveness that’s not easy to achieve by other means.

Nested tests are well suited for all use cases where a hierarchical split of features gives a better overview than a linear split. This particularly includes tests that check protocols of implementations or state automata, as shown in the upcoming example. JUnit Jupiter directly supports the definition of nested tests with the annotation @Nested. This ability is roughly comparable to that of the HierarchicalContextRunner, an extension to JUnit 4.

While the nesting of tests via inner classes facilitates hierarchical thinking about the test structure, it is first and foremost a layout tool. Because inner tests depend logically on outer tests, there are, in principle, two ways in which inner tests can reuse preconditions achieved by outer tests.

The simplest possibility is to reuse the setups from outer lifecycle methods in inner tests. Such lifecycle methods annotated with parameters such as @BeforeEach may be present at every level of the containment hierarchy and are always executed for inner tests. In this context, note that class-level @BeforeAll methods are not supported for nested tests, since Java does not allow static methods within inner classes.

Here is a nested test with hierarchical setup methods:


class NestedTestWithHierarchicalSetupMethods {

    String state = "";

    @BeforeEach
    void outerSetup() {
        state = state + "outer";
    }

    @Nested
    class InnerClass {

        @BeforeEach
        void innerSetup() {
            state = state + "-inner";
        }

        @Test
        void checkSetup() {
            assertEquals("outer-inner",  state);
        }

    }

}

The standard instantiation lifecycle used above implies that the test classes are instantiated from scratch for the execution of every test method. This is certainly good practice for normal tests and has been the default in all versions of JUnit. As an alternative, you can instantiate the whole test class only once by adjusting the lifecycle of the test instance with @TestInstance(TestInstance.Lifecycle.PER_CLASS). This enables the contained tests to rely on the state set by the containing tests. In the following example, the inner test can see the state set by the outer test and assert its value.


@TestInstance(TestInstance.Lifecycle.PER_CLASS)
class NestedTestWithInstantiationPerClass {

    String outerState = null;

    @Test
    void setState() {
        this.outerState = "outer";
    }

    @Nested
    class InnerClass {

        @Test
        void checkState() {
            assertEquals("outer", outerState);
        }

    }

}

In the above example, the per-class lifecycle is employed. This causes the Jupiter test engine to instantiate the whole nested test only once, not over and over again for every test method. While this is not recommended for normal tests where the ordering of test methods should be irrelevant, it can sometimes be useful for nested tests, allowing for protocol-like call sequences spanning the hierarchy without setup code. Note that the order is fixed only by the containment hierarchy, not within one nested class. Both techniques can be employed for a simple and natural modeling of protocol-like invocation sequences.

After these theoretical and rather contrived examples, it is time to move on to a more realistic use case. The following example demonstrates the possibility to use preconditions from outer tests in inner tests by using hierarchical lifecycle methods for the setup code. This example features a data access object (DAO) that offers CRUD (create, read, update, delete) methods for customer objects. For the sake of simplicity, it assigns the ID 1 to the first object stored.


@DisplayName("A customer object")
class DAOTest {

    CustomerDAO dao = new CustomerDAO();

    @Test @DisplayName("can be created with the dao")
    void canBeCreated() { dao.create("Alice"); }

    @Nested @DisplayName("when created")
    class WhenCreated {

        Customer customer;

        @BeforeEach
        void setup() { customer = dao.create("Alice"); }

        @Test @DisplayName("it must be saved to the dao")
        void mustBeSaved() { assertEquals(1L, dao.save(customer)); }

        @Nested @DisplayName("after saving a customer")
        class AfterSaving {

            @BeforeEach
            void setup() { dao.save(customer); }

            @Test @DisplayName("it can be fetched from the dao")
            void canBeFetched() { assertTrue(dao.fetch(1L).isPresent()); }

            @Test @DisplayName("it cannot be deleted by wrong id")
            void cannotBeDeletedByWrongId() {
                assertThrows(IllegalArgumentException.class, () -> dao.delete(-77L));
            }
        }
    }
}

The above example shows that setup code from outer tests is run before inner tests are executed. While this might to lead to some duplication of setup code in an inner test with functionality already present in the outer test, it gives you the ability to run all tests independently. You can even run inner tests alone without outer tests, because the setup code from the outer tests is always executed. If you prefer to use the state set by outer test methods in inner tests, you can avoid the setup code, but you have to use the per-class lifecycle semantics mentioned previously.

This example makes heavy use of the @DisplayName annotation, a generic tool in JUnit 5 for naming tests more flexibly than the strict Java naming rules for methods allow. The execution of nested tests yields visually appealing results in an IDE (see Figure 1). When combined with suitable display names, this hierarchical structure rather closely expresses the specification and is reminiscent of the expressiveness of behavior-driven development frameworks.

Execution of nested tests in an IDE

Figure 1. Execution of nested tests in an IDE

Although nested tests support arranging several related tests hierarchically, there are two other types of mechanisms for creating several related tests without a hierarchical structure, namely dynamic tests and parameterized tests. Both alternatives are discussed in the following sections.

Dynamic tests and TestFactory

Parameterized and nested tests were available in JUnit 4 in one form or another, but the concept of dynamic tests is completely new to JUnit 5. This novelty comes in the form of the @TestFactory annotation.

Methods denoted by that annotation are not tests in and of themselves. Rather, they produce a stream or collection of test cases; hence, the name TestFactory. This stream is generated only at runtime, because its length is unknown during authoring and compilation times. However, through the listeners exposed by the JUnit platform, an IDE is still able to display one entry per test case within the stream in its execution tree.

The JUnit platform event listeners are part of the new JUnit 5 API for tool vendors and facilitate the integration of the platform with IDEs and other tools. Dynamic tests can also use dynamic names, leading to a flexible and expressive display in the IDE. Because the IDE has no way of knowing the number of dynamic tests—let alone their dynamic names—beforehand, it has to generate the execution display tree entries at runtime from the starting event for each individual dynamic test.

This flexibility to receive events at runtime is enabled by the JUnit platform’s notion of distinct discovery and execution phases. Although only the test factory is known at discovery time, the individual tests appear at a certain time during the execution phase.

Here’s a very simple example first. The code in the next listing shows a test factory with two products, the first one representing a successful test and the second one a failure. In this case, the list of two DynamicTest instances is fixed and known at test writing time. This deliberately short example serves to show the basics of dynamic test construction. For real use cases with a fixed and small number of tests, it would usually be simpler just to use standard (@Test) tests.


@TestFactory
List<DynamicTest> simple() {
    return Arrays.asList(
            dynamicTest("first", () -> assertTrue(true)),
            dynamicTest("second", Assertions::fail)
    );
}

The next example is still somewhat contrived but shows the ability to use a stream of unknown length for generating the list of test cases. A stream of random numbers is generated and numbers from this stream are taken as long as the numbers do not exceed 1 billion. The actual test then consists of a simple check for positivity.


Random random = new Random();

@TestFactory
Stream<DynamicTest> withStream() {
    return Stream.generate(random::nextInt)
        .takeWhile(n -> n < 1000000000)
        .map(n -> dynamicTest("is positive? " + n, () -> isPositive(n)));
}

private void isPositive(Integer n) {
    assertTrue(n > 0);
}

The result of one particular execution of this test factory is shown in Figure 2. Because this test factory was constructed to be purely (and somewhat arbitrary) dynamic, the next execution may yield more test results—or none at all.

Execution of dynamic tests in the IDE

Figure 2. Execution of dynamic tests in the IDE

After these introductory examples of a purely technical nature, let’s consider a use case from the financial domain. Dynamic tests often lend themselves to integration testing, where some interaction with external resources is validated. In the following example, imagine you are given the task to check the validity of payments. First fetch a number of Customer objects from a potentially remote service using List<Customer> getCustomers(). For each of these customers, then retrieve a stream of their individual payments from a database repository using Stream<Payment> findAllPayments(Customer customer).

Here’s the code that shows a test factory combining two streams in a dynamic manner:


CustomerService customerService = new CustomerService();
PaymentRepository paymentRepository = new PaymentRepository();

@TestFactory
Stream<DynamicTest> withCombinedStreams() {
    return customerService.getCustomers()
            .stream()
            .flatMap(c -> paymentRepository.findAllPayments(c))
            .map(p -> dynamicTest("valid payment? " + p, () -> validate(p)));
}

void validate(Payment payment) {
    //some asserts
}

The test factory method combines the customers and payments streams using flatMap(). Because you might not know either the number of customers received from the remote service or the number of each customer’s payments, you certainly do not know anything about the size of the combined stream. Thus, this example demonstrates an integration test use case where the resulting number of tests is unknown at authoring, compilation, or test instantiation times.

Of course, such a test can be approximated by just iterating over the asserting method, but that collapses the series of individual logical tests to a single physical test. Such a single test—with many repeated asserts—hides its true nature in the result display and is much harder to debug in case of failures.

These examples all show that such functionality can be implemented with lambda expressions elegantly and concisely. In the early planning phase for JUnit 5, the expressiveness for this and other features was one of the reasons to drop backwards compatibility with Java 5. As a matter of fact, the original project that led to the new JUnit version was even called “JUnit Lambda.”

All in all, dynamic tests are a lightweight tool for use cases with runtime behavior and an unknown number of tests combined with flexible name generation. The individual test cases within the stream as products of the test factory were never intended to be full-blown standard tests and, hence, do not support fine-grained lifecycle methods, which are applied only to the test factory methods. If you need a sequence of multiple tests with individual lifecycle methods, you might be better off with parameterized tests, covered next.

Parameterized tests

Parameterized tests constitute a frequently used and appreciated feature, regardless of the testing framework. Here’s some good news: The JUnit Jupiter test engine has brought tremendous progress in this area. These tests are now less verbose than in JUnit 4, and they are much easier to write and read. There are also new ways to create such tests, which make the overall feature really thorough and powerful.

If you want to try parameterized tests, the first step is to add the junit-jupiter-params dependency. Even better, you can use the junit-jupiter aggregate available since version 5.4, which includes both the junit-jupiter-api and junit-jupiter-params artifacts.

To repeat: New parameterized tests are much less verbose than before. There are no more runners and no more annotated fields. All you need is @ParameterizedTest. As for the parameters, just pass them as method arguments. Here’s an example of the smallest parameterized test you can think of:


@ParameterizedTest
@ValueSource(strings = {"Java", "JUnit"})
void a_very_concise_test(String name) {
    assertTrue(name.startsWith("J"));
}

Notice that values are provided through the @ValueSource annotation, which is useful when you have a single primitive parameter. If a test should use null as a value, annotate it with @NullSource. Additionally, @EmptySource or @NullAndEmptySource can be used to test empty values (which are "" in the case of strings).

What about tests with more than one parameter? Don’t worry: @CsvSource is here for you. With this annotation, you can provide as many parameters as needed in a CSV-like format, as shown in the example below. If the data is located in a real file, it can even be injected via @CsvFileSource.


@ParameterizedTest
@CsvSource({
        "Java, 4",
        "JUnit, 5"
})
void a_slightly_more_complex_test(String name, int expectedLength) {
    assertEquals(expectedLength, name.length());
}

What’s interesting here is the type of the expectedLength argument: That’s an int, even though you can provide only String parameters through @CsvSource. Some internal JUnit magic is at play here. This mechanism is based on implicit converters that can interpret a String into another type, such as int. This particular example may not seem that impressive, but it works with a lot of very useful common types, such as File, Path, BigDecimal, and URL. This mechanism also supports pretty much all of the Java Data and Time API, and it even works with enums. You can find the list of all implicit converters in the user guide.


@ParameterizedTest
@CsvSource({
        "2020-06-01, JUNE",
        "2019-04-15, APRIL"
})
void putting_those_converters_to_good_use(LocalDate date,
                                          Month expectedMonth) {
    assertEquals(expectedMonth, date.getMonth());
}

Overall, less boilerplate code is needed thanks to converters, and tests are more focused on the actual testing logic. But if those are implicit converters, there must be explicit ones, right?

You can indeed create your own converter by extending SimpleArgumentConverter with @ConvertWith on arguments, as shown below.

Similarly, you can write aggregators to merge several parameters into one argument by implementing ArgumentsAggregator and using @AggregateWith.


@ParameterizedTest
@ValueSource(strings = {"JANUARY", "FEBRUARY", "MARCH"})
void a_test_with_explicit_conversion(
        @ConvertWith(MonthToNumberConverter.class) int month) {
    assertTrue(month <= 3);
}


class MonthToNumberConverter extends SimpleArgumentConverter {

    @Override
    protected Object convert(Object source, Class<?> targetType) {
        Month month = Month.valueOf((String) source);
        return month.getValue();
    }

}

Coming back to parameter sources, there’s still a way to provide them through a method if you need to build more-complex objects: Use @MethodSource to point towards a method returning Stream<Arguments>. Arguments is a Jupiter wrapper for parameters. You can use this source to dynamically generate test executions as was done in the previous section. The main difference is that unlike dynamic tests, each parameterized test execution will act as a standalone test in regard to the lifecycle methods.

You can customize the display names of parameterized test executions as well as the test name. The name attribute of @ParameterizedTest is available for this purpose. The parameters can even be inserted in the name by putting their position between curly braces, as shown below. Also notice the use of @EnumSource, yet another useful way of building parameterized tests.


@DisplayName("There are 12 months in a year")
@ParameterizedTest(name = "Month #{index} --> {0}")
@EnumSource(Month.class)
void all_the_months(Month month) {
    // ...
}


There are 12 months in a year
├─Month #1 --> JANUARY
├─Month #2 --> FEBRUARY
...
├─Month #12 --> DECEMBER

Extensions

Unlike its predecessor, JUnit Jupiter now has a unique extension point available to both frameworks and test writers in the Extension API. That API is very developer-friendly and consists of several interfaces that you can directly implement. Then, all there is to do is register them with @ExtendWith on test classes. Under the hood, the implementation of this model is obviously very complex, but here is how it can be leveraged when you write tests.

Imagine having an integration test where you need to launch an in-memory database or mock an external server. You will probably do these things inside a @BeforeAll or @BeforeEach method. But what if you have dozens of tests that need the same setup? That’s where lifecycle callbacks come in handy, giving you one extension for every step of a test lifecycle, such as BeforeEachCallback or AfterTestExecutionCallback. The example below uses the ones that hook before and after all tests.


@ExtendWith(MyCoolExtension.class)
class MyIntegrationTests {

    @Test
    void test() {
        // Everything will already be up and running here
    }

}


class MyCoolExtension implements BeforeAllCallback, AfterAllCallback {

    @Override
    public void beforeAll(ExtensionContext context) {
        // Set up some useful stuff
    }

    @Override
    public void afterAll(ExtensionContext context) {
        // Then shut everything down
    }

}

Another really useful extension is ParameterResolver, which injects objects into tests or lifecycle methods as arguments.


class MyObjectResolver implements ParameterResolver {

    @Override
    public boolean supportsParameter(ParameterContext parameterContext,
                                     ExtensionContext extensionContext) {
        return parameterContext.getParameter().getType() == MyObject.class;
    }

    @Override
    public Object resolveParameter(ParameterContext parameterContext,
                                   ExtensionContext extensionContext) {
        return new MyObject();
    }

}

@ExtendWith(MyObjectResolver.class)
class MyTests {

    @Test
    void test(MyObject myObject) {
        assertNotNull(myObject);
    }

}

Unlike runners, several extensions can be registered on a class. You can, of course, use your custom extensions along with third-party ones implemented by other frameworks. Since JUnit now supports meta-annotations, you can also create custom annotations that regroup all of them, as shown below.


@ExtendWith({
        MyCoolExtension.class,
        MyObjectResolver.class,
        SpringExtension.class
})
@Retention(RetentionPolicy.RUNTIME)
@interface CustomTest {

}

Conclusion

This article barely scratched the surface of all the things you can do with extensions in JUnit. If this brief overview sparked your curiosity, you might want to look at the user guide where all extensions are fully documented.

This article explored three special kinds of tests in JUnit 5: nested, dynamic, and parameterized. All are used to group or organize tests and are perfectly integrated with JUnit Jupiter’s new features such as display names.

JUnit 5 has introduced and renovated a great number of important testing features. The JUnit 5 team also made a point of opening the framework to the outside, while keeping the APIs discoverable and easy to use. As a result, the framework is much more flexible than before and can be customized at will to fit your testing needs.

If you are still using JUnit 4 and want to migrate to JUnit 5, you might want to consult Brian McGlauflin’s recent migration article from an earlier Java Magazine issue. Happy testing!

The authors wish to thank Johannes Link, who kindly helped to review this article.

Dig deeper

Juliette de Rancourt

Juliette de Rancourt is a software engineer at Carbon IT in Paris, France. She is curious about everything and tries to learn new things every day. She is a member of the JUnit 5 team, which combines things she likes: Java, tests, open source, and great people!

Matthias Merdes

Matthias Merdes is a runaway physicist based in Heidelberg, Germany. He is a senior software architect with the payment provider heidelpay and has been a member of the core JUnit 5 team for five years. Merdes has worked with server-side Java for over 20 years and focuses on the test-centered development of web and streaming systems. When he is programming, he enjoys source code simplicity and readability most.

Share this Page