Testing Among the Clouds

One of the major challenges we’ve been facing in the Spring LDAP project is to make certain that the library works together with different LDAP servers. Different servers behave differently in certain situations; some functionality might only be supported on select servers, etc. In the ideal situation we would run our automated test suite against a multitude of target platforms for each change made to the code. Each developer would be able to run the test suite locally against each of these platforms, and the continuous integration server would automatically run them on each commit. This is the key element to automated testing: the automated tests must be relevant, and they must be quickly and easily run at any time, by any member of the team (or – in this case, being an open source project – by anyone that might have checked the source out from subversion). These conditions are imperative for any type of automated tests – if the conditions are not fulfilled the tests will not be run at all and consequently they will die.

In the perfect world, each developer would at all times have access to a server park with all of these different server versions installed. Reality is however almost always a different question: Being an Open Source project with limited funding and developers located throughout the world this has been impossible for us, up until now. You’re probably already guessing: The answer is in the Clouds.

Having used Amazon EC2 in a couple of customer projects, we suddenly realized this is the answer to our problems. For those of you not completely familiar with the concept, the general idea of Amazon EC2 is that you configure one or several machine images, installing whatever software and data you might need on each image. You will then start an instance from one of these pre-configured images, which will in effect start up a virtual computer somewhere in the amazon cloud, a computer that will then be yours to play around with in any way you might like until you’re finished with it and just shut the instance down. All of this is available at a very reasonable pricing – you pay only (a very modest amount) for the time that your instances have been running.

The concept suits us perfectly: we want a number of isolated environments, each with a known start state where we can run our test suite against a particular server setup. So that’s what we’ve started to do: we have now created an EC2 image with an Open LDAP installation, pre-loaded with the data that our test cases expect. We then devised support classes to have the JUnit test cases automatically start an appropriate instance (using the excellent typica library), run the tests against the started virtual machines, and shut down the instance after the tests are finished (regardless of the outcome – we don’t want any stray instances running around costing us money :)). We still have some more work to do with setting up more images for other server setups, but we’re hoping to get that work going as we move on…

I’m planning to dig a little bit deeper into the details of the setup and the code involved in a later post. In the meantime: if you’re really interested, feel free to check the code out from the svn repo and have a look at what we’ve done so far. The relevant code for this is in the test/integration-tests-openldap directory.

Leave a Reply