Testing in Android can be complex, especially when your component is not isolated from the Android framework. One example of this is when your component is performing file system tasks using Context. E.g, the Activity class has the following methods (inherited from Context):

openOrCreateDatabase();
deleteDatabase();
getDatabasePath();
openFileInput();
openFileOutput();
getFileStreamPath();
deleteFile();
getCacheDir();

When calling any of the methods above, Context directs the calls to parts of the file system dedicated to your specific application, usually located under /data/data/<your.applications.package.name>/. To mock away these type of operations, there is a component called RenamingDelegatingContext. It basically allows you to replace the context used when calling file system methods, re-directing the call to a mocked data file instead of the real file. The source code for RenamingDelegatingContext can be looked at here:

RenamingDelegatingContext

There are a number of tutorials on how to use RenamingDelegatingContext to mock away your files or databases;

• http://dtmilano.blogspot.com/2009/11/android-testing-mock-contexts.html
• http://www.devdaily.com/java/jwarehouse/android/test-runner/src/android/test/ProviderTestCase.java.shtml

Advanced use of RenamingDelegatingContext

In addition to open resources directly using Context, one common task is to open a resoure using a ContentResolver:

Uri uri = Uri.parse("content://authority/path");
InputStream is = getContext().getContentResolver().openInputStream(uri);

The getContext() call can be omitted if you're making the call from within an Activity or Service, where the component itself is a Context.

One might want the returned InputStream to point to a test resource instead of the real resource. This can be achieved with some research. When opening the InputStream, the ContentResolver finds the proper provider for the given authority (see ContentProvider section for more detail). The source code reveals more on how this is done:

ContentResolver.openInputStream

public final InputStream openInputStream(Uri uri) throws FileNotFoundException {
   ...
   } else {
       AssetFileDescriptor fd = openAssetFileDescriptor(uri, "r");
       try {
           return fd != null ? fd.createInputStream() : null;
       } catch (IOException e) {
           throw new FileNotFoundException("Unable to create stream");
       }
   }
}

ContentResolver.openAssetFileDescriptor

public final AssetFileDescriptor openAssetFileDescriptor(Uri uri, String mode) throws FileNotFoundException {
    ...
    } else {
        IContentProvider provider = acquireProvider(uri);
        ...
        try {
            AssetFileDescriptor fd = provider.openAssetFile(uri, mode);
            ...
            ParcelFileDescriptor pfd = new ParcelFileDescriptorInner(fd.getParcelFileDescriptor(), provider);
            return new AssetFileDescriptor(pfd, fd.getStartOffset(), fd.getDeclaredLength());
        } catch ( ...

As seen in the extracts above, the ContentResolver acquires the correct provider based on the given Uri (authority). In order for us to mock away this behavior, we need to do four things:

1. Use a RenamingDelegatingContext to mock away the real Context (target Context) in the test.
2. Override the getContentResolver in RenamingDelegatingContext to return a MockContentResolver.
3. Create a ContentProvider to return a AssetFileDescriptor pointing to the test resource.
4. Register the ContentProvider with the MockContentResolver to handle the authority we want to override.

Mocking ContentResolver

public class MockContentResolverTestCase extends AssetManagerAndroidTestCase {

    public static final String URI_PREFIX = "content://";
    public static final String URI_AUTHORITY = "com.my.authority";
    public static final String URI_PATH = "/additional_path/";
    public static final String URI_FULL = URI_PREFIX + URI_AUTHORITY + URI_PATH;

    private RenamingSystemSettingsContext mockContext;
    private LinkedList assetFileQueue;

    public void createMockContext(Context targetContext) {
        mockContext = new RenamingSystemSettingsContext(targetContext);
        assetFileQueue = new LinkedList();
    }

    public Context getMockContext() {
        return mockContext;
    }

    public void pushAssetFile(String fileName) {
        assetFileQueue.offer(fileName);
    }

    public class RenamingSystemSettingsContext extends RenamingDelegatingContext {

        private static final String PREFIX = "test.";   // Not actually used, but needed by RenamingDelegatingContext

        public RenamingSystemSettingsContext(Context targetContext) {
            super(targetContext, PREFIX);
            super.makeExistingFilesAndDbsAccessible();
        }

        @Override
        public ContentResolver getContentResolver() {
            MockContentProvider provider = new MockContentProvider() {
                AssetFileDescriptor ret = null;

                @Override
                public AssetFileDescriptor openAssetFile(Uri uri, String mode) throws FileNotFoundException {
                    try {
                        ret = openTestResourceAsAssetFileDescriptor(assetFileQueue.poll());
                    } catch (Exception e) {
                        e.printStackTrace();
                        throw new FileNotFoundException();
                    }
                    return ret;
                }
            };
            MockContentResolver resolver = new MockContentResolver();
            resolver.addProvider(URI_AUTHORITY, provider);
            return resolver;
        }
    }
}

AssetManagerAndroidTestCase

public class AssetManagerAndroidTestCase extends AndroidTestCase {
    /*
      * For unknown reason, the 'getTestContext' method is hidden in source. Use
      * reflection to access it.
      */
    private AssetManager getAssetManagerForTestContext() throws Exception {
        Method m = AndroidTestCase.class.getMethod("getTestContext",
                new Class[]{});
        mContext = (Context) m.invoke(this, (Object[]) null);
        return mContext.getAssets();
    }
    protected InputStream openTestResourceAsStream(String fileName) throws Exception {
        AssetManager testAssets = getAssetManagerForTestContext();
        return testAssets.open(fileName, AssetManager.ACCESS_STREAMING);
    }

    protected AssetFileDescriptor openTestResourceAsAssetFileDescriptor(String fileName) throws Exception {
        AssetManager testAsset = getAssetManagerForTestContext();
        return testAsset.openFd(fileName);
    }
}

Creating an example test

In the following example, the test_file.midi is a text file stored in the res/asset/ directory. The .midi extension is a workaround to prevent the Android Asset Packaging Tool (aapt) from compressing the file. If the file is compressed, it can only be opened as an InputStream and not as a FileDescriptor, which is needed for mocking away the ContentProvider. See this bug for more info.

public class ExampleTests extends MockContentResolverTestCase {

    public void testUseCustomSettingsFile() throws IOException {
        String assetFile = "test_file.midi";
        createMockContext(getContext());
        pushAssetFile(assetFile);
        Uri uri = Uri.parse(URI_FULL);

        InputStream is = getMockContext().getContentResolver().openInputStream(uri);

        assertEquals("Text in test file", getStringFromStream(is));
    }

    public static String getStringFromStream(InputStream is) throws IOException {
        int size = is.available();
        byte[] buf = new byte[size];
        is.read(buf);
        return new String(buf);
    }
}

That's it!

I can understand the joys of making sure that every potential bug is removed, and the bragging rights of 100% automated test coverage as well. But I would argue that it is equally fraudulent to claim 100% automated test coverage to a client as a virtue. I am not arguing that automated tests should be abandoned, far from it. I am arguing that automated tests should be applied with the same care as optimizations.

Premature Automated Testing is Bad

We have all learned that premature optimizations are bad. Guessing what could fail is hard, the only safe route would be to write a check for everything! There is even a coding practice encouraging this, test driven development or TDD for short. Which should really be called ADD for assertion driven development. Since what is done is not testing, but checking by asserting conditions against a specification.

Take a moment and answer honestly; for any project with good automated test coverage what is the ratio between time spent writing the checks versus the time these checks have saved you? In my experience the more zealously automated test coverage is applied, the more time is spent maintaining the tests. For any project where automated tests have been written first, and code later, I have yet never experienced that the checks have saved more time than the time they cost to develop and maintain.

When optimizing it is hard to find the bottlenecks. Gut feeling is good many times, but often even your gut feeling is wrong and the performance bottle neck is somewhere you would never expect. It is the same with defects in code, your gut feeling is good for pin-pointing many defects, but often the cause is something you never expected. So instead of wasting time guessing what the problems may be down the road you should inspect and measure where the issues are. And when an issue is found then you write a check for it to ensure it does not break again.

Change the Algorithm

The best optimization is often not to optimize at all, but to change to a better algorithm. Writing a bubble sort algorithm in assembly will never yield a better result than a sloppy merge sort written in C anyway, so do not waste your time.

Same can be said about code quality. If you have many issues, and your code breaks often, adding automated tests will only be bandaid until it breaks the next time. Change the code into something more stable. If needed, write checks to ensure the new implementation conforms to the same public API when you refactor it. Take notice to write checks with as good coverage as needed when you are about to refactor your code not before.

If you architect your application with many small independent units, each with a clearly defined public API, then refactoring is not a big problem.

Perceived Quality

All green lights and a 100% automated test coverage looks good on a report. But it can never catch what is most important of all; is the application valuable to the user? In that regard the 100% automated test coverage is a false safety, it gives the illusion of quality. A dangerous illusion that can be used to blind managers, customers and other stakeholders from seeing the real issues with the application.

There is yet no testing framework made that is as efficient as a human being actually using the application. Not just people blindly following a test protocol, that is even worse. But stake holders using the application as intended. Not just short burst before committing code to source control, but also regularly on longer sessions. So that you get a feel for how the app is used, what works, or just what grinds your gears. Usability issues are also issues and should be dealt with.

Not every issue is created equal. With the limited time every project have fixing, or predicting, each and every one of them is never feasible. So do not waste time fixing the one in a million issue, or writing checks for the bread and butter code. But do make sure the application fails gracefully. Fail as gracefully as you can for the end-user, they are very forgiving if you fail with style and make sure no data is lost. But also fail gracefully for yourself, be generous with logging when you do fail. And remember to add an automated check so that the reason for the failure may never raise again, once you fixed it.

Model View Controller

All application logic can be divided into three categories. Model for the business logic of your product, View for the display and input to and from the user (or other application), and Controller that is the mediator between the two. Any application of any significant size will have smaller MVC patterns within their units as well. Well defined boundaries between what is a Model, View and Controller is not only good for re-usability, maintainability, and replacing parts. It is also great for testing. Have well defined and testable public APIs for the parts, and the private implementations tend to be simpler and of higher quality. Do not let the public API be defined by how something is implemented, let the implementation be defined by how you want to access it from the outside.

Writing unit tests for the Model of your application is seldom a waste of time. Often the Model is what you begin implementing, so using unit tests as the incubator until you have enough logic in place to implement the first draft of your application is only rational; code that is unused is seldom of any quality at all.

Writing unit tests for the View yield much less returns on investment. It is also highly probably that you will have to write these tests over and over again. I would go so far as call them harmful, too many automated tests on the Views may in fact discourage you from doing drastic but needed usability changes. And no automated test in the world can ever flag if a View is usable, looks good, and feels right to the end-user.

Writing unit tests for the Controller should be a waste of time, otherwise your Controller is doing too much. A Controller should only be the mediator between the View and Model, if it does more then it should be split up and the proper parts moved into Model and View layers. So writing unit tests for a good Controller is basically just verifying that your are using the APIs of the Model and Views correctly.

Conclusions

We have learned that premature optimization is bad, often counter productive or wastes our time for minimal gains. Let us all grow up and also learn that premature automated testing is equally bad. The time we have is to precious to waste on overzealous automated testing instead of adding real value to the product by applying tests where it makes real difference to the products quality.

1. You are not allowed to write any production code unless it is to make a failing unit test pass.
2. You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.
3. You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

That means that test cases are written before any production code. Not all tests are written up front, it's rather that a small test is written, then a small piece of production code is written that only allows that test to pass. This is repeated in many small iterations: test, fail, code, pass, test, fail, code, pass...

Many people consider TDD to encourage clean code, simple architectures and a stable system that's actually testable. Plus, it's also fun! We've previously written about various aspects of TDD, but in this tutorial we'll focus on how it works for XCode projects, where you write apps for Mac and iPhone. We will create a simple XCode project, do some special configuration steps and then demonstrate how TDD can be used to write your app. We're going to use OCUnit and its framework SenTestingKit, which nowadays is included with Apple's XCode tools.

Creating the Calculator project

Let's start by creating a new project in XCode. You can pick any template, since we aren't going to touch the generated code. I picked a Window-based iPhone project. Name the project Calculator. Select New Target from the Project menu. Under Cocoa select Unit test Bundle. Call it UnitTests (I've had problems with space in the target name so avoid that) and add it to the project. We now need to change some small settings for this target. Select the UnitTests target and hit Command-I for Info. Select the Build tab and locate Other Linker Flags. Replace Cocoa with Foundation. Also remove the entry for GCC_PREFIX_HEADER.

Now it's time to create your test suite. Create a new group in your project and call it "Test Classes". Add a new File to this group and make sure it's a Cocoa->Objective-C test case class. Name it CalculatorTest.m. Uncheck "Also create CalculatorTest.h" since we don't need a header file. Don't add it to the Calculator target, but check it to be included in UnitTests. Open CalculatorTest.m file and above the @implementation declaration add:

#import <SenTestingKit/SenTestingKit.h>
#import "Calculator.h"
@interface CalculatorTest : SenTestCase
{

}
@end

This is just because we don't want a rather empty header file. All your test classes will look like this.

Following TDD, we should build right away. First make sure that the active target is "UnitTets" and then press the "Build" button. This will not only do a normal build, but also perform some shell script magic and actually execute the unit tests. Now the build fails of course, since Calculator.h can't be found.

So let's add a file which represents production code. Create a new empty Cocoa Touch Objective-C class called Calculator.m (also create the .h file), this is the class that we want to test. Don't forget to check that they should be included in both your targets. Notice that we mix Cocoa and Cocoa Touch, that's is fine - all code is executed on your Mac. Build again, and this time you'll see in the build log that everything built and the test suite was executed, but contained 0 test cases. Let's remedy that.

Writing test cases

Next we want to write our first test case. Open up CalculatorTest.m and add the following method:

-(void)testAdd
{
        Calculator* calculator = [[Calculator alloc] init];
        int result = [calculator add:5 to:6];
}

All methods that start with "test" will be recognized as being a test case, and automatically run when building the target. Try and build and you'll unsurprisingly get the erro "unrecognized selector sent to instance...". That's all good. Let's add the add method to our Calculator class. In Calculator.h add:

-(int)add:(int)a to:(int)b;

Then in Calculator.m add:

-(int)add:(int)a to:(int)b
{
        return 0;
}

That should be enough to let it compile and run, right? Try it out! You'll see that it indeed build and executes the tests.

Now we change the test so that it actually checks that we get the expected value back. We want to "assert" that the returned value is what we expected. Change the test case to look like this:

-(void)testAdd
{
        Calculator* calculator = [[Calculator alloc] init];
        int expected = 11;
        int result = [calculator add:5 to:6];
        STAssertEquals(expected, result,
                @"We expected %d, but it was %d",expected,result);
}

Build and you see that the test runs and fails since our calculator always returns 0 regardless of the input. Change the production code so that it returns a+b and rerun the test. Nice, we have a successful build and our test works! Notice the fast cycle of test, code, fix, rerun. The point of TDD is to write tests and production code in small iterations so that you always have something stable (= the tests pass). If you follow the rules of TDD you'll have a growing amount of test cases and you'll right away know when something breaks.

Let's write one more test. Now we want to add the functionality to allow one number to be divided by another. Start up like before with a new test case like this:

-(void)testDivide
{
        Calculator* calculator = [[Calculator alloc] init];
        float expected = 2.5;
        int result = [calculator divide:5 by:2];
        STAssertEquals(expected, result,
                @"We expected %d, but it was %d",expected,result);
}

You see, this time we expect to get back a float. Add the corresponding divide method to Calculator.h and Calculator.m. If you need help see the full code listing at the end of this article. You might notice that just returning a/b gives you 2.0. Type cast the return value to (float)a/b and you'll be fine. Go ahead when you got it working.

What happens if you try to divide something with zero? Well, why not add a test case for this scenario? If you're into maths you know that the result of this operation is undefined. How do you expect something to be undefined? This is tricky. Start off by just expecting any number and see what you get back. As you see, the float value "inf" is returned. It seems like Objective-C treats the zero as "a value approaching zero" and that will indeed result in infinity in a division. But, hey, that might not be what we want our calculator to do. Let's change the test case to expect an exception to be raised instead:

-(void)testDivideByZero
{
        Calculator* calculator = [[Calculator alloc] init];
        STAssertThrows([calculator divide:5 by:0],
                @"We expected an exception to be raised when dividing by 0");
}

Build and notice the error message when no exception was raised. Now change the production code to something like this:

-(float)divide:(int)a by:(int)b
{
        float result =  (float)a/b;
        if (result==INFINITY) {
                [NSException raise:@"Cannot divide by zero!"
                            format:@"Not possible to divide %d with %d", a,b];
        }
        return result;
}

Rerun and smile as the test passes! There are many different variants of asserts you can do with SenTestingKit. Compare objects with STAssertEqualObjects. STAssertTrue to check that a returned boolean is true. Open up SentTestCase_Macros.h if you want to see what you can play around with. By the way. You might have tried using NSLog in your test cases (just to experiment). This is nothing you would do in real life, as you want all necessary information in the fail message and output nothing if the test passes. Anyway, since the tests are actually run using a separate shell script for the UnitTest target you won't see the log in your console as usual. Instead check the build log and click the "text" icon to the right for the "Run test suite" step.

Wrapping up

Finally, if you look at your test class you might notice that we're allocating a new Calculator for every test case, and we never release them. That's no good. Luckily there are setUp and tearDown methods that will be launched automatically before and after each test case. Change your implementation to look like this (this is the final listing):

#import <SenTestingKit/SenTestingKit.h>
#import "Calculator.h"
@interface CalculatorTest : SenTestCase
{
        Calculator* calculator;
}
@end

@implementation CalculatorTest

- (void) setUp
{
        calculator = [[Calculator alloc] init];
}

-(void)tearDown
{
        [calculator release];
}

-(void)testAdd
{
        int expected = 11;
        int result = [calculator add:5 to:6];
        STAssertEquals(expected, result,
                @"We expected %d, but it was %d",expected,result);
}

-(void)testDivide
{
        float expected = 2.5;
        float result = [calculator divide:5 by:2];
        STAssertEquals(expected, result,
                @"We expected %f, but it was %f",expected,result);
}

-(void)testDivideByZero
{
        STAssertThrows([calculator divide:5 by:0],
                @"We expected an exception to be raised when dividing by 0");
}
@end

For completeness, here's the listing for Calculator.h:

#import 

@interface Calculator : NSObject {

}

-(int)add:(int)a to:(int)b;
-(float)divide:(int)a by:(int)b;

@end

and for Calculator.m:

#import "Calculator.h"

@implementation Calculator

-(int)add:(int)a to:(int)b
{
        return a+b;
}

-(float)divide:(int)a by:(int)b
{
        float result =  (float)a/b;
        if (result==INFINITY) {
                [NSException raise:@"Cannot divide by zero!"
                            format:@"Not possible to divide %d with %d", a,b];
        }
        return result;
}

@end

Next blog post we'll see how to automate the running of test cases on a build server called Hudson!

 
public class Greeter {
	public static String getGreeting(String name) {
		return "Greetings " + string;
	}
}
 

And the test:

 
@RunWith(PowerMockRunner.class)
@PrepareForTest( Greeter.class )
public class MockStaticExampleTest {
	@Test
	public void mockStaticExample() throws Exception {
		String expectedGreeting = "greeting";
		String nameToGreet = "name";
 
		mockStatic(Greeter.class);
		expect(Greeting.getGreeting(nameToGreet)).andReturn(expectedGreeting);
		replayAll();
 
		String actualGreeting = Greeter.getGreeting(nameToGreet);
 
		verifyAll();
		assertEquals("Expected and actual greeting did not match",
                                  expectedGreeting, actualGreeting);
	}
}
 

The "power" behind PowerMock lies in its ability to modify the byte-code of the classes specified in the PrepareForTest annotation to make them testable. But because of restrictions enforced by the Java run-time PowerMock is simply not allowed to byte-code manipulate certain classes dynamically. These classes are those that are loaded by Java's bootstrap class-loader such as for example java.net.URLEncoder, java.lang.System and other "system classes" located in the java.lang or java.net package etc. While is this generally not a problem it may be so if you need to mock static methods in these classes because PowerMock cannot prepare them for test. So how do we get around this problem?

How does PowerMock work?

To start of with we probably should describe how PowerMock actually works under hood. What happens when PowerMock makes a class testable is that the byte-code is changed so that each method call, constructor call, field call etc are first routed to something that we refer to as the MockGateway. Simply put the MockGateway decides if it's OK for the call to be delegated to the original method/constructor/field or if a mock object should be returned instead. The MockGateway communicates with a MockRepository which stores all mock objects that has been setup for the current test method. Since PowerMock is just a layer on top of other mock frameworks we leave it for the underlying framework to do the actual mock creation, recording and verification of the mocks. The created mocks are put into the MockRepository by the PowerMock mock extension API. Basically all standard mock frameworks use CGLib to create a mock object which means that they're based on a hierarchical model (CGLib creates a sub class of the class to test at run-time which is the actual mock object) instead of a delegation model which PowerMock uses through it's byte-code manipulation by delegating to the MockGateway. This means that e.g. if a class is final CGLib cannot create a sub-class of it at run-time and therefore we can't mock it. So this is why most mock frameworks have the limitations that PowerMock doesn't.

So how do we mock static methods in a system class?

Since there's no way for PowerMock to modify the byte-code of a system class how do we route all calls to the MockGateway? The answer is quite simple, don't modify the system class itself but modify the class calling the system class! Thus outgoing calls to methods in the system class can be routed to the MockGateway instead of the system class intercepting incoming calls. That sounds simple enough but there's one more catch. As we saw earlier you set up expectations in the EasyMock extension API by reusing the standard expect method, e.g.

 
expect(Greeting.getGreeting(nameToGreet)).andReturn(expectedGreeting);
 

and this usually works fine because Greeting has been prepared for test and the call to method getGreeting is routed to the MockGateway. But in cases of system classes a call to a static method is NOT intercepted by the system class itself as described above and thus the call is never routed to the MockGatway. The solution is to prepare the actual test class for test as well so that the outgoing method call to getGreeting in the expect method is routed to the MockGateway as well! So let's look at a very simple example to demonstrate how to mock a call to URLEncoder.encode(..):

 
public class MyEncoder
	public String performEncode() throws UnsupportedEncodingException {
		return URLEncoder.encode("something", "encoding");
	}
}
 

and the test:

 
@RunWith(PowerMockRunner.class)
@PrepareForTest( { MyEncoder.class, MyEncoderTest.class })
public class MyEncoderTest {
 
	@Test
	public void demoOfSystemClassMocking() throws Exception {
		mockStatic(URLEncoder.class);
 
		expect(URLEncoder.encode("something", "encoding"))
                         .andReturn("something else");
		replayAll();
 
		assertEquals("something else", new MyEncoder().performEncode());
 
		verifyAll();
	}
}
 

But wait, will this work for final system classes?

That's a really good question because as we said earlier PowerMock delegates the mock creation process to the underlying mock framework (such as EasyMock) which uses CGLib. So even when mocking static methods the underlying mock framework is still used to create the CGLib mock of the class where the static methods are located. PowerMock then make use of the underlying framework's functionality to hold state for all recorded method invocations etc so that we don't have to deal with that in PowerMock as well. So usually what PowerMock does when it encounters a final class that should be prepared for test is simply to remove the final modifier. This means that CGLib can be used to create the mock object of this class without any problems. But what about a final system class? There's no way for PowerMock to remove the final modifier of a system class so what to do? What PowerMock does in these cases is to create a completely new class at run-time with the exact same structure as the original final system class. I.e. all method names and their corresponding signature are copied to a this new replica class. To allow for partial mocking all static methods of the replica class delegates to the original method in the final system class. It's also the replica class that is being mocked by the underlying mock framework instead of the original system class. The MockGateway then figures out that all methods bound for this particular system class should be routed to the replica mock instead. Thus mocking of static methods in final system classes such as java.lang.System or java.lang.String works as well. As a side note it would actually be possible to use this technique to implement duck-typing in Java as well. Anyway, here's an example to demonstrate what we've just said:

 
public class SystemPropertyMockDemo {
 
	public String getSystemProperty() throws IOException {
		return System.getProperty("property");
	}
}
 

and the test:

 
@RunWith(PowerMockRunner.class)
@PrepareForTest( { SystemPropertyMockDemo.class, SystemPropertyMockDemoTest.class })
public class SystemPropertyMockDemoTest
 
	@Test
	public void demoOfFinalSystemClassMocking() throws Exception {
		mockStatic(System.class);
 
		expect(System.getProperty("property")).andReturn("my property");
 
		replayAll();
 
		assertEquals("my property",
                                  new SystemPropertyMockDemo().getSystemProperty());
 
		verifyAll();
	}
}
 

Conclusion

As we've seen PowerMock is capable of achieving things not normally possible with standard mock frameworks because of technical limitations. Using PowerMock you can virtually choose any design you like without having to worry about the testability (or mockability to be more precise) aspects. Please visit our website for more information and downloads.

Prerequisites

In order to keep this reasonably brief I'll have to refer to the getting started guide for information on how to get going with Amazon EC2.

A FactoryBean to Launch EC2 Instances

What we want to achieve here is to launch an EC2 instance transparently and independently of the actual test code. Ideally, the test code should be oblivious of the target server and we want to externalize those details to external configuration. We will use Spring's excellent JUnit test support to automatically wire the integration test setup and make sure that the target server is up and running before the actual test code is running.

A powerful way to transparently perform complex initialization logic when using Spring is the FactoryBean concept. We will create a FactoryBean implementation to launch the EC2 image and set up our target resource:

AbstractEc2InstanceLaunchingFactoryBean.java
public abstract class AbstractEc2InstanceLaunchingFactoryBean extends AbstractFactoryBean {
  private static final int INSTANCE_START_SLEEP_TIME = 1000;
  private static final long DEFAULT_PREPARATION_SLEEP_TIME = 30000;
  private static final Log log = LogFactory.getLog(AbstractEc2InstanceLaunchingFactoryBean.class);
  private String imageName;
  private String awsKey;
  private String awsSecretKey;
  private String keypairName;
  private String groupName;
  private Instance instance;
  private long preparationSleepTime = DEFAULT_PREPARATION_SLEEP_TIME;
  public void setImageName(String imageName) {
      this.imageName = imageName;
  }
  public void setAwsKey(String awsKey) {
      this.awsKey = awsKey;
  }
  public void setAwsSecretKey(String awsSecretKey) {
      this.awsSecretKey = awsSecretKey;
  }
  public void setKeypairName(String keypairName) {
      this.keypairName = keypairName;
  }
  public void setGroupName(String groupName) {
      this.groupName = groupName;
  }
  public void setPreparationSleepTime(long preparationSleepTime) {
    this.preparationSleepTime = preparationSleepTime;
  }
  @Override
  protected final Object createInstance() throws Exception {
      Assert.hasLength(imageName, "ImageName must be set");
      Assert.hasLength(awsKey, "AwsKey must be set");
      Assert.hasLength(awsSecretKey, "AwsSecretKey must be set");
      Assert.hasLength(keypairName, "KeyName must be set");
      Assert.hasLength(groupName, "GroupName must be set");
 
      log.info("Launching EC2 instance for image: " + imageName);
 
      Jec2 jec2 = new Jec2(awsKey, awsSecretKey);
      LaunchConfiguration launchConfiguration = new LaunchConfiguration(imageName);
      launchConfiguration.setKeyName(keypairName);
      launchConfiguration.setSecurityGroup(Collections.singletonList(groupName));
 
      ReservationDescription reservationDescription = jec2.runInstances(launchConfiguration);
      instance = reservationDescription.getInstances().get(0);
      while (!instance.isRunning() && !instance.isTerminated()) {
          log.info("Instance still starting up; sleeping " + INSTANCE_START_SLEEP_TIME + "ms");
          Thread.sleep(INSTANCE_START_SLEEP_TIME);
          reservationDescription = jec2.describeInstances(Collections.singletonList(instance.getInstanceId())).get(0);
          instance = reservationDescription.getInstances().get(0);
      }
 
      if (instance.isRunning()) {
          log.info("EC2 instance is now running");
          if (preparationSleepTime > 0) {
              log.info("Sleeping " + preparationSleepTime + "ms allowing instance services to start up properly.");
              Thread.sleep(preparationSleepTime);
              log.info("Instance prepared - proceeding");
          }
          return doCreateInstance(instance.getDnsName());
      } else {
          throw new IllegalStateException("Failed to start a new instance");
      }
   }
 
  protected abstract Object doCreateInstance(String ip) throws Exception;
 
  @Override
  protected void destroyInstance(Object ignored) throws Exception {
    if (this.instance != null) {
      log.info("Shutting down instance");
      Jec2 jec2 = new Jec2(awsKey, awsSecretKey);
      jec2.terminateInstances(Collections.singletonList(this.instance.getInstanceId()));
    }
  }
}
 

This superclass makes use of the typica library (also available in maven) to launch the EC2 instance and delegates to doCreateInstance for creating the target resource that we will use in our test case. In our setup we want to create a ContextSource, which is the Spring LDAP equivalent of a DataSource:

ContextSourceEc2InstanceLaunchingFactoryBean.java
public class ContextSourceEc2InstanceLaunchingFactoryBean extends AbstractEc2InstanceLaunchingFactoryBean {
  private String base;
  private String userDn;
  private String password;
  @Override
  public final Class getObjectType() {
    return ContextSource.class;
  }
  @Override
  protected final Object doCreateInstance(final String dnsName) throws Exception {
    Assert.hasText(userDn);
    LdapContextSource instance = new LdapContextSource();
    instance.setUrl("ldap://" + dnsName);
    instance.setUserDn(userDn);
    instance.setPassword(password);
    instance.setBase(base);
 
    instance.afterPropertiesSet();
    return instance;
  }
  public void setBase(String base) {
    this.base = base;
  }
  public void setUserDn(String userDn) {
    this.userDn = userDn;
  }
  public void setPassword(String password) {
    this.password = password;
  }
}
 

Note that the AbstractEc2InstanceLaunchingFactoryBean will wait for the instance to start up properly. It will also wait an additional period of time once the instance is up and running to allow for all relevant services to start up properly. Finally it will automatically close down the launched instance once it is properly shut down (which it will be when executing using the Spring automated test support).

Test Case Configuration

We now have the infrastructure to have the EC2 instance launched transparently from a Spring Application Context. All we need to do is configure it:

testContext.xml
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="ldap.properties" />
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
</bean>
 
<bean id="contextSource" class="org.springframework.ldap.ContextSourceEc2InstanceLaunchingFactoryBean">
<property name="awsKey" value="${aws.key}" />
<property name="awsSecretKey" value="${aws.secret.key}" />
<property name="imageName" value="${aws.ami}" />
<property name="groupName" value="${aws.security.group}" />
<property name="keypairName" value="${aws.keypair}" />
<property name="base" value="dc=jayway,dc=se" />
<property name="userDn" value="${userDn}" />
<property name="password" value="${password}" />
</bean>
 
<bean id="ldapTemplate"
  class="org.springframework.ldap.core.LdapTemplate">
  <constructor-arg ref="contextSource" />
</bean>
 

We are referring to ldap.properties for the actual settings:

userDn=cn=admin,dc=jayway,dc=se
password=secret
base=dc=jayway,dc=se
aws.ami=ami-56ec083f
aws.keypair=spring-ldap-keypair
aws.security.group=spring-ldap

The referenced AMI is a public image specifically set up for our test cases. It has OpenLDAP installed and has been pre-populated with the test data that our test cases expect.

Note that we are not specifying any property value for aws.key and aws.secret.key, as this is the Amazon account access information. That will be the account settings of the individual developer and should be supplied at runtime using system properties (e.g. mvn -Daws.key=xxxxxxx -Daws.secret.key=xxxxxx test).

Performing the Test

Now all we need to do is use Spring's automated test support to start our test:

SomeLdapITest.java
@ContextConfiguration(locations = { "testContext.xml" })
public class SomeLdapITest extends AbstractJUnit4SpringContextTests {
  @Autowired
  private LdapTemplate tested;
 
  @Test
  public void testSomething() {
    // Use the automatically injected LdapTemplate instance to perform a test.
  }
}
 

Set to Go

With the above you should be all set to go to start using this powerful approach to integration testing. The actual code is available with the 1.3.0.RC1 release of Spring LDAP; links to downloads and documentation available here.

In the perfect world, each developer would at all times have access to a server park with all of these different server versions installed. Reality is however almost always a different question: Being an Open Source project with limited funding and developers located throughout the world this has been impossible for us, up until now. You're probably already guessing: The answer is in the Clouds.

Having used Amazon EC2 in a couple of customer projects, we suddenly realized this is the answer to our problems. For those of you not completely familiar with the concept, the general idea of Amazon EC2 is that you configure one or several machine images, installing whatever software and data you might need on each image. You will then start an instance from one of these pre-configured images, which will in effect start up a virtual computer somewhere in the amazon cloud, a computer that will then be yours to play around with in any way you might like until you're finished with it and just shut the instance down. All of this is available at a very reasonable pricing - you pay only (a very modest amount) for the time that your instances have been running.

The concept suits us perfectly: we want a number of isolated environments, each with a known start state where we can run our test suite against a particular server setup. So that's what we've started to do: we have now created an EC2 image with an Open LDAP installation, pre-loaded with the data that our test cases expect. We then devised support classes to have the JUnit test cases automatically start an appropriate instance (using the excellent typica library), run the tests against the started virtual machines, and shut down the instance after the tests are finished (regardless of the outcome - we don't want any stray instances running around costing us money ). We still have some more work to do with setting up more images for other server setups, but we're hoping to get that work going as we move on...

I'm planning to dig a little bit deeper into the details of the setup and the code involved in a later post. In the meantime: if you're really interested, feel free to check the code out from the svn repo and have a look at what we've done so far. The relevant code for this is in the test/integration-tests-openldap directory.

The main goal of Behaviour Driven Development, a TDD based methodology, is to identify behaviours in both the analysis and the design phases.
In BDD [1] the test code looks more like a specification of behaviours than a test of the production code.

Test Driven Development

TDD can be summarized in one rule:
“Writing code is only allowed if you have a failing unit test”.

The TDD development cycle:
1. Write a unit test
2. Run all tests and make sure the new one fails
3. Implement the simplest possible solution in order to make the test pass
4. Refactor the code. All tests must pass after the refactoring has been done.
5. Repeat - write a new unit test

Problems:

• Unit tests suggest a strong relationship between test class and implementation
  class. Each class or method usually has a corresponding test class or test meth-
  od. This causes a high coupling between the class under test and the test case.
  It also means that we focus too early on the implementation details, instead of
  doing the design first.
• Unclear definition of granularity of units. A common mistake is to mix up unit test with integration tests - which may require complex setup of several com-
  ponents (like a database). Having a high coupling between test code and the
  implementation is a serious problem when later on refactoring is needed. In-
  stead one should use mock objects to explore interactions between objects.
• Limitations on mock objects may have too big an impact on the design. We
  often have to stop using the private, static and final keywords. This problem was
  discussed in the previous issue of JayView - Mocking Static Methods.

The Marriage of Component
Specification with Unit Test

BDD has a similar development cycle, but instead of writing tests before coding you
write specifications before coding.
A specification consists of a description of behaviours which are expressed within a
context. Behaviours become self-verifying by writing (coding) expectations in the
form of examples. The specification is written in an executable form so that we can
generate a readable specification and verify that its expectations are valid.

Example

For example let’s write a specification for a stack. We start by identifing its contexts,
which are: “empty”, “neither empty nor full”, “full”. Example of behaviours for the
empty context are: “should be empty”, “should not be empty after a push”. For each
of those one continue to write expectations. Like TDD you specify a small set of
behaviours at a time then implement it - short iterations.
Let’s show how this can be done using standard JUnit since it is a well known
tool. JUnit is not a good tool for doing BDD since it was only designed for doing
testing. A more suitable tool is for example JDave [4] or RSpec [2] which let you
create much more readable specification. (The analysis part of RSpec is shown in
the next section).

// The enclosed annotation makes JUnit run all inner classes as tests
@RunWith(org.junit.runners.Enclosed.class)
public class StackTest {
    public static class EmptyStack {
        private Stack stack;
        @Before
        public void before() {
            stack = new Stack();
        }
        @Test
        public void shouldBeEmpty() {
            assertTrue(stack.isEmpty());
        }
        @Test
        public void shouldNotBeEmptyAfterAPush() {
            stack.push(“Some value”);
            assertFalse(stack.isEmpty());
        }
        // ...
    }
    public static class NeitherEmptyNorFullStack {
        private Stack stack;
        @Before
        public void before() {
            stack = new Stack();
            stack.push(“x”);
        }
        @Test
        public void shouldAddToTheTopWhenSentPush() {
            stack.push(“y”);
            assertEquals(“y”, stack.peek());
        }
        // ...
    }
}

By replacing the word “test” with “should” and organizing tests around behaviours it
becomes clear that we are using BDD to drive the design.
This format also allows us to generate an even more readable specification by using
for example the junitreport ant task with a custom style sheet that extracts the bold
words. A proper BDD tool would have this ability built in. I believe this specification
is much better than the (class) javadoc that are often missing or incorrect.
As shown above we keep specification and test together in an executable unit. This
approach can also be used at a system level.

The Marriage of Application
Requirements with Acceptance Tests

Requirements and acceptance tests are captured using a template containing a story
and one or more scenarios.
As a Role, I want Feature, So that Benefit
Scenario1: Given ... When ... Then
Scenario2 ...
A story’s behaviour is its acceptance criteria - the scenarios. By using the template
above we can collaborate on stories with customers. This is nothing new, it is just
one way of doing analysis.
What is new is that in BDD the specification is written down in an executable
unit in order to have self-verifying tests. The Ruby RSpec tool has a slightly different
approach. It can treat specification written in plain english as an executable unit.

An example of a story:

Story: transfer to cash account
As a savings account holder
I want to transfer money from my savings account
So that I can get cash easily from an ATM
Scenario: savings account is in credit
Given my savings account balance is 100
And my cash account balance is 10
When I transfer 20 from my savings account to my cash account
Then my savings account balance should be 80
And my cash account balance should be 20
Scenario: savings account is overdrawn
Given my savings account balance is -20
And my cash account balance is 10
When I transfer 20 from my savings account to my cash account
Then my savings account balance should be -20
And my cash account balance should be 10

To perform verification on this specification we create an interpreter where we de-
fine reusable test steps.

class AccountSteps < Spec::Story::StepGroup
  steps do |define|
    define.given("my $account account balance is $balance") do |account,
balance|
      @accounts ||= {} # creates a new hash if @accounts is nil
      @accounts[account] = Account.new(balance.to_f)
    end
    define.then("my $account account balance should be $amount") do |account,
amount|
      @accounts[account].balance.should == amount.to_f
    end
    define.when("I transfer $amount from my $from account to my $to account")
do |amount, from, to|
       @accounts[from].transfere_to(@accounts[to], amount.to_f)
    end
  end
end

To run it we need to tell which steps are using what story:

runner = Spec::Story::Runner::PlainTextStoryRunner.new(filename-of-the-story)
runner.steps << AccountSteps.new
runner.run 

RSpec matches each line in the story with the defined test steps. The code inside the
define block will be executed if a match is found.
For example, when the story runner reads the line “Given my savings account bal-
ance is 100” it will execute

@accounts['savings'] = Account.new("100".to_f).

That means a new account object is created and stored in the accounts hash.

Still stuck with Java ? Psst, you can use JRuby [5] to test and mock Java
objects. For example, try the JtestR [6] tool which also integrates nicely
with ant, maven and Buildr [3].

Conclusions

There is not a big difference between TDD and BDD if TDD is used correctly. TDD
often causes brittle test code - a small change in the production code may cause a
major change in the test code. BDD gives the possibility of structuring tests to make
them “refactoring friendly”. This is accomplished by organizing tests in small focused
examples of behaviours.
People are not always aware that TDD is able to drive the design. By not thinking
in terms of tests but instead describing the what and why of behaviours, it becomes
clear that BDD is a methodology for analysis and design.

References

[1] BDD - http://behaviour-driven.org/
[2] RSpec - http://rspec.rubyforge.org/
[3] Buildr - http://buildr.rubyforge.org/
[4] JDave - http://www.jdave.org/
[5] JRuby - http://jruby.codehaus.org/
[6] JtestR - http://jtestr.codehaus.org

Originally published in JayView.