Getting Coverage For Integration Tests

Unit testing in my world is basically defined by being able to run the tests wherever and whenever; on the train, on the plane, at work, or at home. If you also have integration tests, chances are that they require some external data source or something that simply makes it impossible to run them everywhere. If that's the case, they should probably not be included in the unit test suite.

Having code coverage of your unit tests is important, because it'll give you an indication of where you should focus your efforts next. The open source coverage tool Cobertura gives you not only code coverage, it also shows you the code complexity level of packages and classes. This directs you towards the untested parts that are the most complex and most likely also the most error-prone.

The Problem

Rumor has it that Maven2 in an upcoming release will support having integration tests in a source folder called src/it/java, and run those in the integration-test phase. That would be a welcome improvement. Currently, however, Maven2 lacks good support for integration tests, and so does Cobertura. It's really a pain to get coverage on both unit tests and integration tests.

Let's say that you want to keep your integration tests in the same module as the unit tests. In fact, this might be a wise decision, since it's even harder to get Cobertura to provide coverage on two separate modules. Still, you do want to separate them somehow, and for now you have named the integration tests IntTestSomething and the unit tests TestSomething.

src/test/java/
`-- org
    `-- springframework
        `-- ldap
            `-- samples
                `-- article
                    |-- dao
                    |   |-- AbstractIntTestPersonDao.java
                    |   |-- IntTestPersonDaoImpl.java
                    |   `-- IntTestTraditionalPersonDaoImpl.java
                    `-- domain
                        `-- PersonTest.java

Cobertura will run the tests through Surefire, which has a default pattern that picks up the unit tests, but not the integration tests. We don't want coverage on the integration tests all the time, so that's fine. But how do we include the integration tests on-demand?

The Solution

I will keep the unit tests in the src/test/java folder, but move the integration tests to a newly created src/it/java folder. Then I will use some magic to incorporate that folder into the Maven build path, but only if I specify a certain profile. I will be able to do this:

% mvn test
// runs unit tests only

% mvn cobertura:cobertura
// provides coverage on unit tests only

% mvn test -Pitcov
// runs unit tests and integration tests

% mvn cobertura:cobertura -Pitcov
// provides coverage on unit tests and integration tests

Create the folder src/it/java and move your integration tests there. Rename them to match the Surefire default pattern, like SomethingIntegrationTest. That will enable Surefire to pick them up without any extra configuration.

src/it/java/
`-- org
    `-- springframework
        `-- ldap
            `-- samples
                `-- article
                    `-- dao
                        |-- AbstractPersonDaoIntegrationTest.java
                        |-- PersonDaoImplIntegrationTest.java
                        `-- TraditionalPersonDaoImplIntegrationTest.java

The unit test source folder now looks like this:

src/test/java/
`-- org
    `-- springframework
        `-- ldap
            `-- samples
                `-- article
                    `-- domain
                        `-- PersonTest.java

Now add the following profile to your Maven pom:

 
<profiles>
<profile>
      <id>itcov</id>
      <build>
<plugins>
<plugin>
               <groupId>org.codehaus.mojo</groupId>
               <artifactId>build-helper-maven-plugin</artifactId>
               <version>1.1</version>
               <executions>
                  <execution>
                     <id>add-test-source</id>
<phase>generate-sources</phase>
                     <goals>
                        <goal>add-test-source</goal>
                     </goals>
                     <configuration>
                        <sources>
                          <source>src/it/java</source>
                        </sources>
                     </configuration>
                  </execution>
               </executions>
            </plugin>
         </plugins>
      </build>
   </profile>
</profiles>
 

Now we're ready to run coverage:

mvn cobertura:cobertura
...
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.springframework.ldap.samples.article.domain.PersonTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
...

Here we can see that only one test was run. The coverage report reveals that we only get coverage on the unit test of the Person domain object, giving us a total coverage of 7%:

We clean the target folder and run again, this time with the 'itcov' profile:

mvn clean
mvn cobertura:cobertura -Pitcov
...
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.springframework.ldap.samples.article.dao.TraditionalPersonDaoImplIntegrationTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.124 sec
Running org.springframework.ldap.samples.article.dao.PersonDaoImplIntegrationTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.12 sec
Running org.springframework.ldap.samples.article.domain.PersonTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec

Results :

Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
...

This time we get both the unit tests and the integration tests, and the coverage report paints quite a different picture, with 75% coverage instead of 7%:

What we immediately see here is that we should focus our efforts on testing our web layer. But that's another story...

Ulrik Sandberg
Consultant at Jayway