You'll be self-signing with makecert from the Windows 7.1 SDK. The certificate tool is a single executable, specifically at %PROGRAMFILES%\Microsoft SDKs\Windows\v7.1\Bin\makecert.exe, per default. You'll need it - copy it to the server unless you already have the SDK installed on it.

Secondly, you've got a domain name, mydomain.com. Otherwise, just substitute the IP in the following instructions.

Without further ado; on the SERVER:

1. First, create the self-signed certificate with the regular cmd.exe:

   makecert -sk "mydomain.com" -ss My -sr localMachine -r -n "CN=mydomain.com, O=Jayway Stockholm AB, L=Drottninggatan 108 Stockholm" -a sha1 -eku "1.3.6.1.5.5.7.3.1"
   (flags ref)

   Substituting your own values.

   This command will create a certificate for Server Authentication and save it in the certificate store for the local machine.

   With powershell, get the thumbprint is for the generated certificate

   ls cert:\LocalMachine\my

   It'll look like this: 87B1C1818F1828958524A598B4131757EBAF4D35

2. Now, configure winrm (PowerShell needs escaping, so use cmd.exe):

   winrm create winrm/config/listener?Address=IP:8.8.8.8+Transport=HTTPS @{Hostname="mydomain.com";CertificateThumbprint="87B1C1818F1828958524A598B4131757EBAF4D35";Port="5986"}

   Again, substitute your variables.

3. Open port 5986, inbound TCP for all programs (and potentially restricted to your CI-server by IP), by going to Start > Administrative Tools > Windows Firewall with Advanced Security

4. Now, from the CLIENT, PowerShell:

   $so = New-PsSessionOption -SkipCACheck
   etsn -cn mydomain.com -Credential MyMachineName\YourUserName -UseSSL -SessionOption $so

   Answer the password dialog box, and you're in! Try running "whoami" to see the user name

That's it. Congratulations. If you stay at this for a small-scale deployment you have OK security on your communication with WinRM! However, do check out some more details:

More details

On makecert and how to switch certificate for one signed by a trusted certificate authority.

makecert switches

1. -sk says what the Subject Key name should be. This is like the 'user name for keys, i.e. a uniquely identifiable name for your key in the key store.

2. -ss is the store's name. It's most often either My or Root depending on what you're using your certificate for.

3. -sr is really just if it's a user-local store or the machine store that you are saving the key to. Has either value currentUser or localMachine.

4. -r is for self-signing which is what we want.

5. -sn stands for Subject Name and in this case our subject (what we want to authenticate and encrypt access to) is our domain.

6. -a stands for Algorithm - default is md5 which is not much more secure than a 4-digit pin code on a post-it taped to an old man's wallet. Ref: Attacking MD5, ref 2

Changing Certificates

Let's discuss what you do when you get a trusted-CA-signed certificate

Justification

So this works nicely for a while, until someone captures you laptop because you forgot it on the tube. Suddenly you can't revoke the certificate and you're in a world of pain of managing your keys (to be honest, this has never happened to me and this is already 1000x better than no authentication or basic unencrypted authentication!). Also, you grew a bit and have a few more servers to manage. Fine, let's upgrade WinRM's HTTPS certificate.

First, get the new one. I am personally involved in StartSSL.org, since I like the idea of free, community-driven crypto over the default server-taxation style type of certificates - this will also integrate finely with browser-based X.509 certificates if you're planning on adding an STS to your infrastructure and authenticating users with certificates rather than username/password combos

However, the world is a harsh place and we want real humans in front of bots to verify our identity for ultra security. We've used geotrust previously and have no real complaints.

Too long, didn't read

Use certreq.exe to generate the request, and install it on the server.

Once installed and visible through ls cert://localmachine/my in PowerShell, copy its thumbprint.

Then, you update the listener's binding (SERVER):

winrm set winrm/config/listener?Address=IP:8.8.8.8+Transport=HTTPS @{Hostname="mydomain.com";CertificateThumbprint="CB7434F35E3EA11D49D209F41BB16E96B472D30E";Port="5986"}

The set operation idempotently changes the listener's configuration to the hash presented inside the @{} hash syntax.

Now try it out (CLIENT):

etsn -cn mydomain.com -Credential MyMachineName\YourUserName -UseSSL

What on earth is etsn?

See for yourself! Get-Alias etsn in PowerShell

All examples below assume that we have already instantiated CloudBlobClient cloudBlobClient. It can be done using the code.

var account = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
var cloudBlobClient = account.CreateCloudBlobClient();

Upload a Blob from a string

First example, upload a string as a text file. We, for example, use this to upload crash logs to blob storage and then mail a link to the support team.

public Uri UploadBlob(string path, string fileName, string content)
{
    var cloudBlobContainer = cloudBlobClient.GetContainerReference(path);
    cloudBlobContainer.CreateIfNotExist();

    var blob = cloudBlobContainer.GetBlobReference(fileName);
    blob.UploadText(content);

    return blob.Uri;
}

So if we would call this method using the following parameters.

var uri = UploadBlob("folder", "file.txt", "text in file");

We would get a link pointing at http://yourstorageaccount.blob.core.windows.net/folder/file.txt containing the text – “text in file”. Please note that this call is made synchronously.

Upload a Blob from a stream

You will recognize this from the previous example. The difference is that we are using a stream instead of a string and that the call is made asynchronously.

public void UploadBlob(string path, string fileName, Stream stream)
{
    var cloudBlobContainer = cloudBlobClient.GetContainerReference(path);
    cloudBlobContainer.CreateIfNotExist();

    var blob = cloudBlobContainer.GetBlobReference(fileName);
    blob.BeginUploadFromStream(stream, UploadFinished, blob.Uri);
}

private void UploadFinished(IAsyncResult asyncResult)
{
    // do something
}

List all Blobs in a container

This will give a list of links to all blobs stored in a container.

public IEnumerable<Uri> ListAllBlobs(string folder)
{
    var cloudBlobContainer = cloudBlobClient.GetContainerReference(folder);
    var listBlobItems = cloudBlobContainer.ListBlobs();

    return listBlobItems.Select(b => b.Uri);
}

Simple enough. Once you got the setup right, Azure Blob Storage is actually quite easy to use. If you would like to list blobs in subfolders as well then use the overloaded method for ListBlobs and send in BlobRequestOptions { UseFlatBlobListing = true }.

Download content from a Blob

Very similar to uploading data, it works the same way with other data types than string.

public string DownloadBlob(string folder, string fileName)
{
    var cloudBlobContainer = cloudBlobClient.GetContainerReference(folder);
    var blob = cloudBlobContainer.GetBlobReference(fileName);

    return blob.DownloadText();
}

But, before downloading a blob, you might want to see if it exists…

See if a Blob exists

The only way that I have found for doing this (and please correct me if I’m wrong here) is to do a little peek and catch the resulting exception.

private bool BlobExists(CloudBlob blob)
{
    try
    {
        blob.FetchAttributes();
        return true;
    }
    catch (StorageClientException e)
    {
        if (e.ErrorCode == StorageErrorCode.ResourceNotFound)
        {
            return false;
        }
        throw;
    }
}

Thanks to Steve Marx for the solution. FetchAttributes() is used simply because it will be a small amount of data transferred or a 404. So even if I call BlobExists() on a 4Mb block blob it won’t be any different from doing the same call on a 2Kb blob.

Some tips and tricks

Well, first of all. Azure Blob storage can be used for much more than just file storage. Scalability is built in so if you, for example, have a static html page, you can easily upload it to Azure blob storage and then link to it. It is a good way to take away load from your WebRole.

All methods that I showed you have a Begin/End method as well. So you should be able to do most of the work asynchronously, as I did when I uploaded a blob from a stream.

If an Azure Queue message risk becoming bigger than 8Kb the recommended solution is to upload the message as a blob and then store the URI as the message on the queue. You won’t get charged for the extra data traffic as long as it’s done within the same data center.

Happy blobbing.

This annoys me. Mostly because I tend to forget about it and then get reminded by an exception. So here’s what I did in order to never ever run into the same issue again (right…).

Reflection?

Yes, reflection! As I usually organize my Azure projects I have a folder called Entities where I have all… well entities that I can read and write from/to Azure Table Storage. Illustrated below.

To be honest, I would probably put the entities in a separate project, but this will do as an example.

So, the idea is, if I know the namespace where I always put my entities, then I should also be able to find all my entities by looking into that namespace using reflection.

But reflection is quite bad for performance…

Yes… yes, it is. Therefore I prefer only using it when I need it and when my application won’t suffer from it. For example – at startup.

Now, before proceeding I recommend that you take a look at a previous blog post – Making the most out of your WorkerRoles. What I did there was basically to create worker that implemented IWorker and then using an IoC container I looped over all implementations in the Run method. We will do the same thing here, but instead of an IWorker we will create an IStartupTask.

public interface IStartupTask
{
    void Start();
}

Now we can get started. What we’ll need to do is:

• Find a way to programmatically find all Entities
• Implement IStartupTask
• Make sure that all (one at the moment) startup tasks are executed in the OnStart method of the WorkerRole

Finding the entities

So reflection is not one of my strongest .NET skills. Luckily, others such as Fredrik Mörk are more competent in the area. So I used his solution.

Implement IStartupTask

Here we go.

public class InitiateAzureStorage : IStartupTask
{
    public void Start()
    {
        var entityNames = GetTypesInNamespace("Awesome.WorkerRole.Entities");
        entityNames.ForEach(CreateTable);
    }

    private List<string> GetTypesInNamespace(string nameSpace)
    {
        return Assembly.GetCallingAssembly()
                       .GetTypes()
                       .Where(t => String.Equals(t.Namespace, nameSpace, StringComparison.Ordinal))
                       .Select(t => t.Name)
                       .ToList();
    }

    private void CreateTable(string tableName)
    {
        var storageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
        var cloudTableClient = storageAccount.CreateCloudTableClient();
        cloudTableClient.CreateTableIfNotExist(tableName);
    }
}

Neat.

Execute the task in the OnStart method

Same procedure as in the previous blog post I mentioned above. We will just loop through all instances of IStartupTask, that is registered in our IoC container, and then start them.

public override bool OnStart()
{
    SetConfigurationListener();
    IoC.Initialize(); //Register instances in the IoC container

    var startupTasks = ObjectFactory.GetAllInstances<IStartupTask>();
    foreach (var startupTask in startupTasks)
    {
        startupTask.Start();
    }

    return base.OnStart();
}

That’s it really.

What happens now?

Basically once our WorkerRoles starts we will make sure that all of our Azure tables exists. If not, we will create them. No more exceptions telling us that the table does not exist. Ever.

Each and every time we add a new entity it will automatically get sucked in by our startup task and then create the table. You could use a similar approach to Azure Queues if you for example serialize objects onto the queue.

But now when you have a nice infrastructure for startup task you can use them for so much more. We (that is Jayway), for example, use them to set up a socket that another server use to push in data. The sky's is the limit.

How to backup WordPress

There are numerous plugins for WordPress when it comes to backup. I have learnt to appreciate the plugin BackUpWordPress as it not only backs up your database but also all the files within the installation.

http://hmn.md/backupwordpress/

There are a few number (though quite complete) of configuration options like

- manual or daily scheduled backups
- number of backups to keep
- whether to include files or database or both

The backups are stored on disk on the path of your choice and could also be attempted to be emailed. Having this kind of backup data makes it very comfortable to restore your WordPress installation.

But what if your complete machine/instance is failing? On, for example Amazon EC2 one should take snapshots of instances when the setup of these change but this does not apply to content changes. Restoring from a snapshot does not bring back newly created or updated content from, for example WordPress. That is, moving those WordPress backup files to a safer place becomes vital. So, s3cmd to the rescue.

What is s3cmd?

Extracted from its' homepage:

"S3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3. It is best suited for power users who don't fear command line. It is also ideal for scripts, automated backups triggered from cron, etc."

http://s3tools.org/s3cmd

At the time of writing s3cmd requires Python 2.4 or newer and some pretty common Python modules. Installing it is merely a matter of running:

sudo apt-get install s3cmd

To configure it run

s3cmd --configure

You will be asked for the two keys of your Amazon account - copy and paste them from your confirmation email or from your Amazon account page.

To list all your buckets run

s3cmd ls

and to list the content of a bucket run

s3cmd ls s3://[bucket name]

Now, you could use 's3cmd put' to upload and 's3cmd remove' to remotely remove specific files but then you would have to do some scripting to select these specific files and to keep the bucket contents to an acceptable size. There is another option that simply syncs the content of your local directory to the remote bucket, doing the removal automatically, namely 's3cmd sync'.

s3cmd sync --delete-removed [source directory] s3://[bucket name]

The source directory is naturally replaced with the directory of where you put your WordPress backups.

Scheduling it

To make it a little easier to schedule via crontab you preferrably put the sync command in a shell script, for example 'sync-wp-backups.sh'

#!/bin/bash

/usr/bin/s3cmd sync --delete-removed [source directory] s3://[bucket name]

To schedule this script run

crontab -e

which brings you into editing the cron job with the editor of your choice. For example you could edit the crontab file to schedule it to run daily at 23:30 (11:30 PM)

30 23 * * * /home/ubuntu/sync-wp-backups.sh >> /home/ubuntu/s3-backup.log

which logs the output to file (s3-backup.log).

That's it! Now I have a number of days of backup automatically synced from the Amazon instance running WordPress to a more fail safe S3 bucket. At least I sleep a lot better with this in place.

While coding for this post I used StructureMap (can be downloaded from NuGet) and Windows Azure SDK 1.5. Should work just fine with an older version of the SDK though.

Wrapping up TableServiceContext

As you know you’ll need to extend TableServiceContext in order to save our entities in Azure Table Storage. Each and every example that I’ve seen is creating a specific ServiceContext for every entity type that needs to be saved. We’ll, I’m not going to do that.

We start with our interface.

public interface IServiceContext<TEntity>
{
    IQueryable<TEntity> QueryableEntities { get; }

    void Create(TEntity entity);

    void Update(TEntity entity);

    void Delete(TEntity entity);
}

Easy enough, right? A basic interface exposing CRUD capabilities. The implementation is not much harder.

public class ServiceContext<TEntity> : TableServiceContext, IServiceContext<TEntity>
    where TEntity : TableServiceEntity
{
    private readonly string tableName;

    public ServiceContext()
        : base(AzureProvider.TableEndpoint, AzureProvider.Credentials)
    {
        tableName = typeof(TEntity).Name;
    }

    public IQueryable<TEntity> QueryableEntities
    {
        get
        {
             return CreateQuery<TEntity>(tableName);
        }
    }

    public void Create(TEntity entity)
    {
        AddObject(tableName, entity);
        SaveChanges();
    }

    public void Update(TEntity entity)
    {
        UpdateObject(entity);
        SaveChanges();
    }

    public void Delete(TEntity entity)
    {
        DeleteObject(entity);
        SaveChanges();
    }
}

So we create our own ServiceContext, extending TableServiceContext as well as implementing our very own IServiceContext<TEntity>. As you can see, in the base constructor, we use a static helper class AzureProvider which basically just exposes values from CloudStorageAccount. This is just for readability and really not necessary.

Also worth mentioning is that we use whatever type name TEntity has as the table name. So if we create a ServiceContext<Dog>, our table would be “Dog”. This makes life easier and also solves a nasty performance issue.

Due to a known performance issue with the ADO.NET Data Services client library, it is recommended that you use the table name for the class definition or define the ResolveType delegate on the DataServiceContext. If this is not used and when the entity class name is not the same as table name, query performance degrades with number of entities returned in the result.

- WindowsClient.net

The ServiceContext class is basically the only code that will be untestable, but honestly… there are no actual logic or business rules inside that class. I would never strive for 100% coverage. As far as I’m concerned that’s a waste of time.

Now what?

First of all, in order to actually use our shiny new interface in a decent manner we need to register it with our IoC container of choice. As usual, I prefer StructureMap.

ObjectFactory.Initialize(x =>
{
    x.Scan(scanner =>
    {
        scanner.TheCallingAssembly();
        scanner.ConnectImplementationsToTypesClosing(typeof(IServiceContext<>));
    });
    x.For(typeof (IServiceContext<>)).Use(typeof (ServiceContext<>));
});

This little piece of code allows to instantiate a new class such as:

public class Foo : IFoo
{
    private readonly IServiceContext<Person> personContext;
    private readonly IServiceContext<Dog> dogContext;

    public Foo(IServiceContext<Person> personContext, IServiceContext<Dog> dogContext)
    {
        this.personContext = personContext;
        this.dogContext = dogContext;
    }
}

Our IoC container will be able to create it without any problems. This class will be completely ignorant what’s behind IServiceContext<TEntity>, it could just as well be a database. Since we inject all dependencies via the constructor this class is quite easy to test.

So?

Really, the point that I’m trying to make is that it’s not hard to wrap the not-very-testable parts of Windows Azure SDK. Just do it from start and you’ll be a happy (unit test writing) camper.

Happy testing!

- "Clojure combines the expressiveness of Lisp, the agility of a dynamic language, the performance of a compiled language, and the wide applicability of the JVM in a robust, production-ready package. Clojure is a practical language designed to support high-performance, concurrent applications which efficiently interoperate with other software in the JVM ecosystem. All of this combines to make it an ideal tool for the programmer to quickly build robust programs."

And motivate why they chose to support Clojure:

- "Clojure covers a new use case on the Heroku platform: components which demand correctness, performance, composability; and optionally, access to the Java ecosystem."

Ladies and gentlemen, it's time to consider moving the parenthesis from the end of the method name to the beginning of the function call. Try it, you'll like it.

Clojure maps

A map is a data structure that associates keys to values. Maps are very important in Clojure, because they are used to store entity data, much like custom classes with fields and setters/getters are used in traditional object-oriented languages like Java. In the Java world, we have java.util.Map. They are somewhat cumbersome to work with, mostly due to the static, generic typing, but also because there is no literal map syntax, plus the fact that you can't initialize it with elements. You must first create it, then mutate it:

import java.util.Map;
import java.util.HashMap;

Map<String,Object> map = new HashMap<String,Object>();
map.put("somekey", "somevalue");

Clojure maps, on the other hand, have a very concise literal syntax. Zero or more key-value pairs between curly braces. That's it.

(def map {:somekey "somevalue"})

The key :somekey in the map above is a Clojure keyword. Keywords are symbolic identifiers that always evaluate to themselves, and they have a very fast equality check. For those reasons (and others, as we will see), keywords are often used, rather than strings, as keys in maps.

A Clojure map is not only a data structure, it is also a function. When a map is called with a key, it will return the corresponding value (or nil if no key was found). It can look like this when testing it from the REPL:

user> (def animal {:species :lion, :name "Leo", :age 3})
user> (animal :name)
"Leo"

In fact, keywords are functions too. When a keyword is called with a map as argument, it will look itself up in the map and return the matching value:

user> (:name animal)
"Leo"

This form is more common in idiomatic Clojure.

The Ring library

Ring is a Clojure web applications library inspired by Ruby's Rack. Ring abstracts the HTTP request and response as maps, and expects handlers to be functions that take a map as argument and returns a map. The returned map is transformed by Ring into a HTTP response, which is sent back to the caller. This is a simple, but powerful abstraction. It means that web handlers can be written as regular functions, which can be tested in isolation. Web libraries like Compojure build on top of the Ring abstraction and provide more high-level constructs, like routing. In order to not complicate this example, however, we will stick to plain Ring.

Let's say that we have a HTTP request that looks like this:

GET /welcome HTTP/1.1
HOST: www.sayhello.com

Ring transforms this into a Clojure map that looks like this:

{:protocol        :http
 :request-method  :get
 :uri             "/welcome"
 :server-name     "www.sayhello.com"}

Let's further assume that we want to generate a corresponding HTTP response:

HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 11

Hello world

In order to produce that HTTP response, Ring expects this map:

{:status 200
 :headers
    {"Content-Type" "text/plain"
     "Content-Length" 11}
 :body "Hello world"}

So all we need to do is to provide a function that returns the above map, like this:

(defn app [req]
  {:status 200
   :headers
      {"Content-Type" "text/plain"
       "Content-Length" 11}
   :body "Hello world"})

Ring provides a response convenience function that returns a skeletal response map with status 200, no headers, and the given argument as body:

(use 'ring.util.response)
(defn app [req]
  (response "Hello world"))

We'll use the response function in our example later.

Leiningen

In order to manage our dependencies, we will use the Leiningen build tool. The simplest way to install it is to download the stable release of the script and place it somewhere in the PATH, like ~/bin. Then run lein self-install:

$ lein self-install
Downloading Leiningen now...
...

$ lein version
Leiningen 1.5.2 on Java 1.6.0_24 Java HotSpot(TM) 64-Bit Server VM

Building the web application

We create a new project and go there:

$ lein new cljheroku
$ cd cljheroku

The newly created directory contains the following files:

cljheroku/
├── README
├── project.clj
├── src
│   └── cljheroku
│       └── core.clj
└── test
    └── cljheroku
        └── test
            └── core.clj

First we need to create a Procfile for Heroku:

$ echo "web: lein run -m cljheroku.core" > Procfile

This will have Leiningen start the web application by running the function called -main in the namespace cljheroku.core, ie the file src/cljheroku/core.clj. We'll change that file to contain this:

(ns cljheroku.core
  (:use ring.util.response
        ring.adapter.jetty))

(defn app [req]
  (response "Hello World"))

(defn -main []
  (let [port (Integer/parseInt (get (System/getenv) "PORT" "8080"))]
    (run-jetty app {:port port})))

Let's look at this code in more detail. We begin with the first chunk:

(ns cljheroku.core
  (:use ring.util.response
        ring.adapter.jetty))

The ns macro will create a namespace cljheroku.core and load all functions in the namespaces ring.util.response and ring.adapter.jetty. Since we used :use and not :require, these functions can be referenced by name, without any prefix or namespace qualifier. This enables us to write (response ...) instead of (ring.util.response/response ...).

Then we define our Ring handler called app.

(defn app [req]
  (response "Hello World"))

As you can see, it's just a regular function that takes one argument: a HTTP request map, and returns the result of the response function: a HTTP response map. If we wanted to set the Content-Type header, we could use the content-type function, another function from ring.util.response. Instead of (response "Hello world"), we would pass the response through the content-type function: (content-type (response "Hello world") "text/plain"). But here we will just use response.

Finally we specify a -main function.

(defn -main []
  (let [port (Integer/parseInt (get (System/getenv) "PORT" "8080"))]
    (run-jetty app {:port port})))

It will serve the same purpose as the Java main method, namely provide an entry point from the outside world. It gets the value of the PORT environment variable, or uses 8080 as default. It can be enlightening to analyze that part in detail. (System/getenv) is simply Clojure's way of calling Java's static method System.getenv(), which returns a java.util.Map. The get function takes a map (yes, it works with Java maps too), a key and a default value, and returns the matching value. So, (get (System/getenv) "PORT" "8080") will get the PORT variable from the environment, and if there is no PORT set there, it will return "8080". The resulting string value of the port is parsed into an integer, and is stored as the local variable port. This value is then passed in to the Jetty adapter as part of a configuration map. All this in two lines. If you're not used to Clojure's lack of ceremony, you might find it overwhelming. I personally find it refreshingly to-the-point.

OK, that was the source code for our handler. We now need to adjust the project.clj file slightly, and provide a dependency to Ring. We'll also add an exclusion of a duplicate dependency that Jetty has. Here is the resulting project.clj:

(defproject cljheroku "1.0.0-SNAPSHOT"
  :description "Example Ring app running on Heroku"
  :dependencies [[org.clojure/clojure "1.2.1"]
                 [ring/ring-core "0.3.8"]
                 [ring/ring-jetty-adapter "0.3.8"]]
  :exclusions [org.mortbay.jetty/servlet-api])

Before we do anything else, though, we should add this baby to Git:

$ git init
Initialized empty Git repository in /Users/john/Source/cljheroku/.git/

$ git add .

$ git commit -m "Initial commit"

We should probably push this to a maintained repository somewhere, but for our purposes, we have now stored this in Git.

Testing locally

Before we test it, we will ask Leiningen to retrieve the dependencies:

$ lein deps

The lib directory now contains a list of jars that we need. Leiningen will make sure they all get on the classpath without you ever having to see it. You can ask Leiningen for the classpath, though, should you really like to see it:

$ lein classpath

You can also have Leiningen create a pom file for you, which can be handy if you need to look at the dependency:tree, import the project into an IDE that only knows Maven; things like that:

$ lein pom

Anyway, we can now start this web application using the same command as we placed in the Procfile for Heroku. It will start Clojure with the correct classpath, find and make available the given namespace (cljheroku.core), and call its -main function with any given arguments (none, in our case):

$ lein run -m cljheroku.core
2011-06-12 18:41:24.927:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
2011-06-12 18:41:24.928:INFO::jetty-6.1.26
2011-06-12 18:41:24.959:INFO::Started SocketConnector@0.0.0.0:8080

Browsing to localhost:8080 shows a page with "Hello world" on it. Good. It works.

We now change the handler function to instead respond with, say, "Hello everyone" and save it. When we reload the page, the new message appears. Excellent. It reloads automatically.

OK. We're happy with our web application and want to deploy it into the cloud. What do we do?

Deploying on Heroku

Heroku is, as I mentioned at the beginning, mainly a Ruby application platform. The command line tool for working with Heroku is a Ruby Gem. We first need to install Ruby and RubyGems. Assuming that has been done, we need to get the heroku gem:

$ sudo gem install heroku

Now we can create an app on the Cedar stack:

$ heroku create --stack cedar
Enter your Heroku credentials.
Email: john.doe@example.com
Password:
Found existing public key: /Users/john/.ssh/id_rsa.pub
Would you like to associate it with your Heroku account? [Yn]
Uploading ssh public key /Users/john/.ssh/id_rsa.pub
Creating furious-sword-794... done, stack is cedar
http://furious-sword-794.herokuapp.com/ | git@heroku.com:furious-sword-794.git
Git remote heroku added

We can list the remote repositories to see what the last sentence above meant:

$ git remote -v
heroku	git@heroku.com:furious-sword-794.git (fetch)
heroku	git@heroku.com:furious-sword-794.git (push)

And finally, in order to deploy our first verison of the web app, we push everything to heroku:

$ git push heroku master
The authenticity of host 'heroku.com (50.19.85.156)' can't be established.
RSA key fingerprint is 8b:48:5e:67:0e:c9:16:47:32:f2:87:0c:1f:c8:60:ad.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'heroku.com,50.19.85.156' (RSA) to the list of known hosts.
Counting objects: 13, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (13/13), 1.27 KiB, done.
Total 13 (delta 0), reused 0 (delta 0)

-----> Heroku receiving push
-----> Clojure app detected
-----> Installing Leiningen
       Downloading: leiningen-1.5.2-standalone.jar
       Writing: lein script
-----> Installing dependencies with Leiningen
       Running: lein deps :skip-dev
       Downloading: org/clojure/clojure/1.2.1/clojure-1.2.1.pom from central
       Downloading: ring/ring-core/0.3.8/ring-core-0.3.8.pom from clojars
       ...
       Copying 10 files to /tmp/build_19juox87ngduz/lib
-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size is 11.0MB
-----> Launching... done, v4
       http://furious-sword-794.herokuapp.com deployed to Heroku

To git@heroku.com:furious-sword-794.git
 * [new branch]      master -> master

We browse to http://furious-sword-794.herokuapp.com, and we see the text "Hello world" (if you remembered to change back from "Hello everyone"). Now we make a small change:

-  (response "Hello World"))
+  (response "Hello World, I tell you!"))

All we need to do to deploy the change is to commit it, and then push to heroku:

$ git ci -a
$ git push heroku master

When we reload the page, the new text is there.

Roll back to previous release

We can list the releases that have been pushed:

$ heroku releases
Rel   Change                          By                    When
----  ----------------------          ----------            ----------
v5    Deploy 34516b0                  john.doe@examp..  3 minutes ago
v4    Deploy e471ca1                  john.doe@examp..  5 minutes ago

Let's say there was a problem with the release that was just deployed, and we want to roll back. With Heroku, it's trivial to roll back to the previous release:

$ heroku rollback
Rolled back to v4

When we're done with this web app completely, we can destroy it like this:

$ heroku apps:destroy

It's even possible to roll back to earlier releases, but I won't show that here. This concludes the Clojure-on-Heroku guide. Thank you for your time.

References

1. Heroku: http://www.heroku.com/
2. Clojure: http://clojure.org/
3. Ring library: https://github.com/mmcgrana/ring
4. Leiningen: https://github.com/technomancy/leiningen
5. Compojure: https://github.com/weavejester/compojure/wiki
6. FlightCaster uses Clojure and Heroku: http://www.infoq.com/articles/flightcaster-clojure-rails
7. I got the idea for this blog from this Gist: https://gist.github.com/1001206

Now, there are samples and tutorials on the Microsofts Azure website that shows how to build similair systems. So I could just take one of those samples, modify it a little and be done. Piece of cake, right? NOT SO!

There are several pits I fell into during the development.

Pitfall 1
I wrote the system on my development machine, deployed it to Azure and tested it. Works fine! Great! Lets go to the customer and try it!

The client application crashed because the ServiceBus assemblies was not installed on the client machine, thus the "netEventRelayBinding" could not be found. Solution? Add it to the config file:

<system.serviceModel>
  <extensions>
    <bindingExtensions>
      <add name="netEventRelayBinding" type="Microsoft.ServiceBus.Configuration.NetEventRelayBindingCollectionElement, Microsoft.ServiceBus, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    </bindingExtensions>
  </extensions>
</system.serviceModel>

Pitfall2
The client application crashed again with a 'Access Denied' error. Cause? The customer firewall didn't allow connections on the port the service bus uses.

Solution? Use http/https on port 80/443 instead. Add this to the code:

ServiceBusEnvironment.SystemConnectivity.Mode = ConnectivityMode.Http;

Pitfall 3
The application crashed with a "HTTP 407 Proxy Authentication Required" because the customer has an internet proxy that requires authentication.

Add this to the config-file:

<system.net>
  <defaultProxy enable="true" useDefaultCredentials="true"/>
</system.net>

Pitfall 4
The application crashed again with a cryptic error: "Chunked encoding upload is not supported on the HTTP/1.0 protocol".
I worked together with a network technichian to try to find the problem. After a lot of log reading and head scratching, we have come to the conclusion the proxy thinks that servicebus.windows.net is an unapproperiate site and blocks it, just like facebook and other sites the customer employees shouldn't access during work.

Solution: Make sure the proxy allows outgoing connections to Azure AppFabric sites. Here's a list of IP-addresses:
http://blogs.msdn.com/b/windowsazureappfabricannounce/archive/2010/01/28/additional-data-centers-for-windows-azure-platform-appfabric.aspx

Conclusion
If you are planning on using Windows Azure AppFabric from within a company network, beware of proxies. They may cause a lot of headaches.

Also, I found this to be quite helpful:
http://social.technet.microsoft.com/wiki/contents/articles/troubleshooting-connectivity-issues-in-the-windows-azure-appfabric-service-bus.aspx

Before I get started I would like to extend a big thanks to Scott Densmore. More specifically thank you for this great blog post! Surely would have been real hard without it.

In order to reach our goal we need to:

1. Have our build server configured to build Azure projects, preferably without installing Visual Studio.
2. Create an MSBuild script that builds an Azure package ready to be uploaded to Windows Azure.
3. Replace the connection strings in our ServiceConfiguration.csfg production connection strings.
4. Create a certificate so that we can deploy using Windows Azure Service Management API.
5. Use PowerShell to deploy our Azure package and configuration file.
6. Tie it all together with TeamCity.

Let’s get started!

Make our TeamCity server build an Azure project

As it is, our build server can build basic projects without any problems. But, as we soon found out, it could not build Azure projects. So my goal here is to describe all the issues that we ran into and how we solved them. Hopefully this will help some of you trying to do the same thing.

As I mentioned before our goal was to avoid installing Visual Studio on the build machine. We decided this for several reasons. I guess that the main reason had something to do with developer pride… Simply put, it should not be necessary to install a full development environment on a build server just to build an Azure project.

Anyway, we started out with installing Windows Azure SDK on the build server triggered our first build.

First problem

The type or namespace name 'WindowsAzure' does not exist in the namespace 'Microsoft'

First solution

Well this isn’t good is it? We installed all Azure assemblies but it still refused to find them. Well, the solution for this problem is apparently to copy Microsoft.CloudService.targets from your development machine to the build machine. The file is located at C:\Program Files\MSBuild\Microsoft\Cloud Service\v1.0\.

Second problem

WAT050 : The version of the Windows Azure SDK currently installed is not compatible with this version of Windows Azure Tools for Microsoft Visual Studio.  Please download and install the latest version of both the Windows Azure SDK and the Windows Azure Tools for Microsoft Visual Studio

Second solution

Wait, what? We have already installed that, right? We’ll for some reason this wasn’t quite enough. I order to solve this problem you first need open up the Registry Editor and navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\ServiceHosting\v1.3. There you have an entry called ToolsVersionSupportLevel. Note the number, in our case is was 7.

Now navigate to Microsoft.CloudService.targets that you copied in the previous step and open it. Now find the element <ServiceHostingVSSupport> and change the value to whatever number you found in the registry. Some of you might call this a hack, I like to call it pragmatism.

Third problem

MSB4057: The target "PipelineTransformPhase" does not exist in the project.

Third solution

It seems that this informative message tells us that it misses the file Microsoft.WebApplication.targets. In order to get rid of this error you need to copy C:\Program Files\MSBuild\Microsoft\VisualStudio\v10.0\Web to your build server. This wasn’t easy to figure out… you’re welcome.

Fourth problem

Need to specify the physical directory for the virtual path 'Web/' of role MyProject.Web.

Fourth solution

Ah, this error actually made some sense. Or well, I don’t exactly know why we got this error, but the solution was at least not that hard to figure out. Since one of our Azure roles is a WebRole, we need to specify its physical directory in ServiceDefinition.csdf. So find the element <Site> and add the attribute physicalDirectory=”..\MyProject.Web” (assuming that your WebRole’s name is MyProject.Web… which it's probably not).

UPDATE

Fifth problem

error MSB4019: The imported project "C:\Program Files\MSBuild\Microsoft\VisualStudio\v10.0\Windows Azure Tools\1.5\Microsoft.WindowsAzure.targets" was not found.

Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

Fith solution

Noted by Andreas Hallberg, just copy copy the "Windows Azure Tools" dir to buildserver C:\Program Files\MSBuild\Microsoft\VisualStudio\v10.0

Building

That's it! Now our build server built our Azure project, ran all the test with coverage and doing this without complaining. First step done!

Build our Azure package file

The next step was to build our package file. You know, the same file that gets generated when you select publish in Visual Studio. This is basically an encrypted zip file containing all your assemblies needed to run your Azure roles. In order to have our build server build the very same package, we wrote a simple MSBuild script that invoked cspack. It looked something like this.

<?xml version="1.0" encoding="utf-8" ?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <PropertyGroup>
        <PackageCommand>
            "C:\Program Files\Windows Azure SDK\v1.3\bin\cspack.exe" MyProject.Azure\ServiceDefinition.csdef /out:Publish\Package.cspkg /role:MyProject.Web;MyProject.Web /rolePropertiesFile:MyProject.Web;roleProperties.txt /role:MyProject.Worker;MyProject.Worker\bin\Release;MyProject.Worker.dll /rolePropertiesFile:MyProject.Worker;roleProperties.txt
        </PackageCommand>
    </PropertyGroup>
    <Target Name="Build">
        <MakeDir Directories="Publish"></MakeDir>
        <Message Text="Using cspack create a package for deployment..."/>
        <Exec Command="$(PackageCommand)"/>
    </Target>
</Project>

As you can see we simply invoke cspack and pass arguments so that it creates a neat little package containing a WebRole (MyProject.Web) and a WorkerRole (MyProject.Worker).

Worth noting here is the option /rolePropertiesFile. This is absolutely necessary if you have an application compiled for .NET 4.0. The roleProperties.txt file referenced only contains a single line: TargetFrameWorkVersion=v4.0. If you don’t add this option your role(s) won't start. I know this from personal experience…

Replace our development configuration with production configuration

Now we have all the assemblies neatly packed in a file named Package.cspkg. But in order to successfully deploy we also need to update our ServiceConfiguration.cscfg. As it is, it looks like this.

<ServiceConfiguration serviceName="Azure" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">
  <Role name="AzureRole.Worker">
    <Instances count="1" />
    <ConfigurationSettings>
      <Setting name="DataConnectionString" value="UseDevelopmentStorage=true" />
      <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true" />
    </ConfigurationSettings>
  </Role>
  <Role name="AzureRole.Web">
    <Instances count="1" />
    <ConfigurationSettings>
      <Setting name="DataConnectionString" value="UseDevelopmentStorage=true" />
      <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true" />
    </ConfigurationSettings>
  </Role>
</ServiceConfiguration>

As you can see we have a configuration that uses our development storage. If we would deploy using these settings our deployment would simply not start. We could add our connection strings to our production storage, but that would make testing a somewhat delicate process.

What we don’t want to do is to change all connection strings before each commit so that our deployment works. This is exactly the type of manual work that we want to avoid. In other words, this is exactly the type of work we would like to delegate to our build server. So, we create another MSBuild script.

<Project ToolsVersion="3.5" DefaultTargets="Deploy" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup Condition="'$(BuildType)' == ''">
    <BuildType>Release</BuildType>
  </PropertyGroup>

  <!-- Azure Information -->
  <PropertyGroup>
    <HostedServiceName>azureservice</HostedServiceName>
    <StorageAccountName>azurestorage</StorageAccountName>
    <StorageAccountKey>****************</StorageAccountKey>
  </PropertyGroup>

  <!-- Project/Solution Information -->
  <PropertyGroup>
    <PackageLocation>Artifacts\</PackageLocation>
    <PackageName>Package.cspkg</PackageName>
    <ServiceConfigName>ServiceConfiguration.cscfg</ServiceConfigName>
  </PropertyGroup>

  <UsingTask AssemblyFile="lib\BuildTasks\Microsoft.Practices.WindowsAzure.MSBuildTasks.dll" TaskName="Microsoft.Practices.WindowsAzure.MSBuildTasks.RegexReplace"/>

  <Target Name="SetConnectionStrings">
    <RegexReplace
        Pattern='Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true"'
        Replacement='Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="DefaultEndpointsProtocol=https;AccountName=$(StorageAccountName);AccountKey=$(StorageAccountKey)"'
        Files='$(PackageLocation)\$(ServiceConfigName)'/>
    <RegexReplace
        Pattern='Setting name="DataConnectionString" value="UseDevelopmentStorage=true"'
        Replacement='Setting name="DataConnectionString" value="DefaultEndpointsProtocol=https;AccountName=$(StorageAccountName);AccountKey=$(StorageAccountKey)"'
        Files='$(PackageLocation)\$(ServiceConfigName)'/>
  </Target>
</Project>

I’m guessing that this might need some explaining. Starting with the Azure Information we have HostedServiceName which is whatever name you gave your service in the Azure Web Portal. Same goes for StorageAccountName. StorageAccountKey is the secret key used to access your storage.

Project/Solution Information starts with our PackageLocation. In our case we store our package in a directory called Artifacts. PackageName is simply the name you gave your package when executing cspack and the ServiceConfigName is the name of your configuration file (will most likely always be ServiceConfiguration.cscfg).

As you see, we are also using a task in order to do our RegEx replace. You can either use MSBuild Tasks or download the dll directly from Scott Densmore’s blog.

Then we basically do our replacement of our development configuration strings to our production connection strings. Magic!

Create and upload certificates to Azure

As it is you cannot deploy to Azure without a certificate that you first upload to your Azure account. You can actually do this quite easily with IIS 7, the Gu shows you how. You can safely ignore everything except on how to create the Self Signed Certificate.

Once you have your certificate you need move it to your personal certificates and export it. You can do this by opening Microsoft Management Console (by typing launching Run and type mmc). There you can press Ctrl + M and then choose Certificates and press “Add >” and then “Finish”.

First navigate to Trusted Root Certification Authorities\Certificates and find your newly created certificate. Then drag and drop it in the Personal\Certificates folder. Now right click and choose Export.

Important, make sure that the user that runs the TeamCity BuildAgent have the certificate in its Personal folder as well. It’s not going to work otherwise.

Now you only need to upload your certificate. Browse to the Management Portal and then on the left side find Management Certificates. There you can choose to upload your newly created certificate.

Use PowerShell and Azure Service CmdLets to deploy to Azure

Stay strong, only one more step before we can tie it all together. Now we have a package ready for deployment, we have a configuration file with the correct configuration strings and we have a certificate that allows us to deploy all of this to Windows Azure. The final step is to actually deploy everything that we fought so hard to build and configure.

We’ll be using PowerShell and Windows Azure Service Management CmdLets.

First you need to download the setup file from their web page and then execute it on the build server. Then it’s time to start writing the script (almost an exact copy of Scott Densmore’s script, thanks again!).

$cert = Get-Item cert:\CurrentUser\My\***********

$sub = "********-****-****-****-************"
$buildPath = $args[0]
$packagename = $args[1]
$serviceconfig = $args[2]
$servicename = $args[3]
$storageservicename = $args[4]
$package = join-path $buildPath $packageName
$config = join-path $buildPath $serviceconfig
$a = Get-Date
$buildLabel = $a.ToShortDateString() + "-" + $a.ToShortTimeString()

if ((Get-PSSnapin | ?{$_.Name -eq "AzureManagementToolsSnapIn"}) -eq $null)
{
  Add-PSSnapin AzureManagementToolsSnapIn
}

write-host Getting service...
$hostedService = Get-HostedService $servicename -Certificate $cert -SubscriptionId $sub | Get-Deployment -Slot Staging

if ($hostedService.Status -ne $null)
{
    write-host Service found.

    write-host Suspending service...
    $hostedService |
      Set-DeploymentStatus 'Suspended' |
      Get-OperationStatus -WaitToComplete

    write-host Removing deployment...
    $hostedService |
      Remove-Deployment |
      Get-OperationStatus -WaitToComplete
}

write-host Creating staging deployment...
Get-HostedService $servicename -Certificate $cert -SubscriptionId $sub |
    New-Deployment Staging -package $package -configuration $config -label $buildLabel -StorageServiceName $storageservicename |
    Get-OperationStatus -WaitToComplete

write-host Starting staging deployment...
Get-HostedService $servicename -Certificate $cert -SubscriptionId $sub |
    Get-Deployment -Slot Staging |
    Set-DeploymentStatus 'Running' |
    Get-OperationStatus -WaitToComplete

write-host Done!

If you never seen a PowerShell script before this might seem a little intimidating. I’ll try my best to explain it from the top. First we fetch our certificate and store it in our $cert variable. All the stars that you see is the Thumbprint for the certificate. Then we declare our subscription Id in $sub. We continue to declare a bunch of variables that we need to send in as parameters to the script.

When all variable declaring is done, we make sure that our AzureManagementToolsSnapIn is installed, if not we install it.

The next step is to find out if we already have a running service deployed in our staging environment. If so, we stop and delete it. Then we upload and start our new deployment.

So in order to actually use this script we need to extend our MSBuild script that we used when we replaced our development connection strings with production connection strings. At the end of the file we simply add:

<Target Name="Deploy" DependsOnTargets="SetConnectionStrings;Build">
  <MSBuild
    Projects="$(AzureProjectPath)\$(AzureProjectName)"
    Targets="CorePublish"
    Properties="Configuration=$(BuildType)"/>

  <Exec WorkingDirectory="$(MSBuildProjectDirectory)"
    Command=" $(windir)\system32\WindowsPowerShell\v1.0\powershell.exe -f deploy.ps1 $(PackageLocation) $(PackageName) $(ServiceConfigName) $(HostedServiceName)" />

</Target>

This will be executed after we updated the configuration file.

Tie it all together

So now we need to tie this all together in TeamCity. For our project we have two build configurations that look like this.

The first configuration, Build, have three build steps.

We first build the entire project. Then we run all our tests with coverage. Finally, we invoke our BuildPackage.msbuild (that’s the script creating the package file using cspack.exe). We then save our output from the build script as an Artifact.

In our second build configuration, Deploy to Staging, we only have one build step.

We simply execute the build script that replaces all the connection strings and deploys the package and configuration file to Windows Azure. Worth noting is that this build configuration have a dependency on the artifact created by Build.

The artifact in this case is the Package.cspkg and ServiceConfiguration.cscfg.

That’s it! Now you can either have Deploy to Staging to trigger for each successful build of Build or you could, for example, schedule it to run each morning.

Conclusion

First of all I would like to say that getting all of this up and running wasn’t easy. My hope is that by writing all of this down in one place I can make it easier for anyone else that wants an automatic deployment for their Azure project.

If you do run into any problems with your deployment (for example that it’s not starting) please have a look here. It’s a list of common errors and how to resolve them.

I would also like to recommend taking this example a little further. Now that you can deploy from your build server, there really is nothing stopping you from scheduled tear downs every evening and then redeployment every morning. This will seriously decrease the time you have your application deployed during development, which in turn will decrease the money you have to spend in order to use Azure. Use the cloud as it was intended, only pay for what you use!

Content

Dag König, Microsoft, kicked off the day, delivering two great presentations. The first session was Introducing Windows Azure followed by Demonstration of Windows Azure. Together, both of the sessions delivered excellent content for anyone new to Azure. You can download Dag’s slides here.

Next up was Robert Folkesson, Microsoft. Robert delivered a more advanced session about Azure Best Practices. We learned about poisonous message handling, dynamically adding functionality to a worker role, idempotency and much more. You can find Robert’s slides here.

After lunch Henrik Feldt, Jayway, continued with a session on Azure AppFabric ServiceBus. Henrik gave a deep technical presentation where he demonstrated the ServiceBus and encouraged discussion. Henrik’s slides can be downloaded here.

Then it was my turn. I gave a session titled Examples from real life. I gave some examples on how to best utilize your workers and how we design them so that we easily can add more functionality without adding to the cost. I also spoke about continuous integration and lessons learned. You can download my presentation here.

Thank you

Thanks to all of you that attended and helped making this a great event! I hope that we can do this again sometime in the future.

This post will show you how to make the most of a single WorkerRole instance. I’ll use StructureMap but you can just as well use any IoC container.

Why?

Right, so the scenario that we try to avoid is the one where you create a new WorkerRole instance for each specific task.

The thing is, if you only need to generate reports once every night you’ll end up with a WorkerRole that sleeps most of the time. The other two instances will probably have something to do every now and then, but not nearly enough to occupy an entire dedicated WorkerRole instance. The primary reason for avoiding this is of course the cost (since you need to pay for every instance), but also the amount of work when creating a new work task. You’ll basically have to create a new project every single time you need some a new type of worker.

What then?

What you want is something like this.

Instead of having three instances we have three workers on a single instance. This is both cost effective and we can easily add more workers as we need them.

How?

Well to start with I have an IWorker that looks like this.

public interface IWorker
{
    int Priority { get; }
    bool HasWork { get; }
    void DoWork();
}

It’s a simple interface that gives us two properties and one method to implement. Priority will be used when sorting the workers. HasWork will tell us if the current worker has any job that needs doing. Finally, DoWork is where the Worker will do his task.

So, take for example the worker that sends e-mails, let’s call him SendMailWorker.

public class SendMailWorker : IWorker
{
    private readonly IMailQueue _mailQueue;
    private readonly IMailSender _mailSender;

    public SendMailWorker(IMailQueue mailQueue, IMailSender mailSender)
    {
        _mailQueue = mailQueue;
        _mailSender = mailSender;
    }

    public int Priority { get { return 1; } }

    public bool HasWork { get { return _mailQueue.ContainsAnyMessages(); } }

    public void DoWork()
    {
        var message = _mailQueue.GetMessage();
        _mailSender.SendMail(message.AsString);
        _mailQueue.DeleteMessage(message);
    }
}

As you can see we take two dependencies in our constructor, IMailQueue and IMailSender. IMailQueue is basically a wrapper for an Azure Queue where we put our e-mails that we want to send and IMailSender actually sends our mail for us. If we would like to have another type of worker we would simply inject other dependencies in the constructor.

If we take a look at the HasWork property we see that we only check if the Queue contains any messages, actually we do a PeekMessage() on the queue.

The other two workers will look quite similar so I’ll skip posting their implementation, I’m guessing that you understand my point by now.

Now, in the StartUp method for the WorkerRole I’ll bootstrap StructureMap.

public override bool OnStart()
{
    ObjectFactory.Configure(x => x.Scan(scanner =>
                    {
                        scanner.WithDefaultConventions();
                        scanner.TheCallingAssembly();
                        scanner.AssemblyContainingType<IMailQueue>();
                        scanner.AddAllTypesOf<IWorker>();
                    }));

    return base.OnStart();
}

WithDefaultConventions tells StructureMap to register all instances that follow the pattern IFoo with the implementation Foo. TheCallingAssembly will look for instances to register in the executing assembly. By specifying AssemblyContainingType<IMailQueue> I make sure that we register both the IMailSender and IMailQueue, both located in the same assembly. Finally, AddAllTypesOf<IWorker> will register any class implementing IWorker.

Now we are ready to add some code to our Run method.

public override void Run()
{
    var workersInPriorityOrder = ObjectFactory.GetAllInstances<IWorker>()
                                              .OrderBy(w => w.Priority)
                                              .ToList();
    while (true)
    {
        foreach (var worker in workersInPriorityOrder)
        {
            if(worker.HasWork)
            {
                worker.DoWork();
            }
        }
        Thread.Sleep(10000);
    }
}

So, the first thing that we do is to fetch all registered instances of IWorker and sort them in priority order. Then we enter our infinite loop and start to process them one by one.

Result

If we would like to add another worker, all we have to do is to have implement IWorker. Then our IoC container will take care of the rest. Also, by injecting any external dependencies in the constructor we can easily implement our worker to do any kind of job. So we have both simplified the task of adding a new Worker and we also made it “free”. Success!

In this project we have a REST WCF service that will be called by devices such as WP7, iPhone and Android phones. This service is then used to create users and then persist them into Azure Table Storage. In order to fulfill my “done right” criteria I want to be able to inject dependencies into the constructors of the WCF services to make them easier to test. I also would like to delegate the instantiation work to StructureMap. To set all of this up we’ll have to

1. Implement our own IInstanceProvider
2. Implement our own IServiceBehaviour
3. Override WebServiceHost
4. Override WebServiceHostFactory
5. Make sure that our WCF service use our own WebServiceHostFactory
6. Register our dependencies in StructureMap
7. Taste the glory

I realize that it looks to be a lot of work, but the implementations really are straight forward and easy to follow. Basically, our end goal is to have our WCF service look something like this.

public class UserService : IUserService
{
    private readonly IUserRepository _userRepository;

    public UserService(IUserRepository userRepository)
    {
        _userRepository = userRepository;
    }

    public bool CreateUser(string name, string address, string mail)
    {
        var user = new User(name, address, mail);
        if(user.IsValid())
        {
            _userRepository.CreateNewUser(user);
            return true;
        }
        return false;
    }
}

By injecting an IUserRepository directly in the constructor we can easily test this class using mocking tools such as Rhino Mocks. Now, to be able to write our services like this we need to do two things. First we need replace the factories instantiating our UserService so that we can create them using constructors taking parameters. Secondly, we need to make sure that our implementation of IUserRepository is registered in StructureMap.

Before we start I would like to mention that I had a lot of help from both Jimmy Bogard’s post and PlanB Software’s post (thanks for you help!).

Create our own WCF instances

In order to create our own WCF instances we need to implement our own IInstanceProvider. This will allow us to create our instances using StructureMap.

public class StructureMapInstanceProvider : IInstanceProvider
{
    private readonly Type _serviceType;

    public StructureMapInstanceProvider(Type serviceType)
    {
        _serviceType = serviceType;
    }

    public object GetInstance(InstanceContext instanceContext)
    {
        return GetInstance(instanceContext, null);
    }

    public object GetInstance(InstanceContext instanceContext, Message message)
    {
        return ObjectFactory.GetInstance(_serviceType);
    }

    public void ReleaseInstance(InstanceContext instanceContext, object instance)
    { }
}

As you can see, in GetInstance(…), we call StructureMap’s ObjectFactory to create the instance for us. Now we have to tell WCF to actually use our own InstanceProvider instead of the default. In order to do that we need to implement IServiceBehaviour.

public class StructureMapServiceBehavior : IServiceBehavior
{
    public void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
    {
        var endpointDispatchers = serviceHostBase.ChannelDispatchers
                                                 .OfType<ChannelDispatcher>()
                                                 .SelectMany(cd => cd.Endpoints);

        foreach (var endpointDispatcher in endpointDispatchers)
        {
            endpointDispatcher.DispatchRuntime.InstanceProvider =
                new StructureMapInstanceProvider(serviceDescription.ServiceType);
        }
    }

    public void AddBindingParameters(ServiceDescription serviceDescription,
                                     ServiceHostBase serviceHostBase,
                                     Collection<ServiceEndpoint> endpoints,
                                     BindingParameterCollection bindingParameters)
    { }

    public void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
    { }
}

Here we make sure that we use our own StructureMapInstanceProvider for each endpoint. Now it’s time to add this behavior to our WCF service. First, we will override the default WebServiceHost.

public class StructureMapWebServiceHost : WebServiceHost
{
    public StructureMapWebServiceHost(object singletonInstance, params Uri[] baseAddresses)
        : base(singletonInstance, baseAddresses)
    { }

    public StructureMapWebServiceHost(Type serviceType, params Uri[] baseAddresses)
        : base(serviceType, baseAddresses)
    { }

    protected override void OnOpening()
    {
        Description.Behaviors.Add(new StructureMapServiceBehavior());
        base.OnOpening();
    }
}

In OnOpening() we make sure that we add our own StructureMapServiceBehavior. In order to have WCF use our StructureMapWebServiceHost we also need to override WebServiceHostFactory.

public class StructureMapWebServiceHostFactory : WebServiceHostFactory
{
    protected override ServiceHost CreateServiceHost(Type serviceType, Uri[] baseAddresses)
    {
        return new StructureMapWebServiceHost(serviceType, baseAddresses);
    }
}

Finally, since this is a REST WCF Service, we add in our Global.asax.cs. As you can see we also specifies that it should use our StructureMapWebServiceHostFactory.

public class Global : HttpApplication
{
    void Application_Start(object sender, EventArgs e)
    {
        RegisterRoutes();
    }        

    private void RegisterRoutes()
    {
        RouteTable.Routes.Add(new ServiceRoute("UserService", new StructureMapWebServiceHostFactory(), typeof(UserService)));
    }
}

That’s it! We have replaced the default WCF instantiation with our own. Now to the next part, register our IUserRepository in StructureMap.

Register types in StructureMap

First of all, let’s take a look at the implementation of IUserRepostory.

public class UserRepository : IUserRepository
{
    private readonly UserServiceContext _userServiceContext;

    public UserRepository(UserServiceContext userServiceContext)
    {
        _userServiceContext = userServiceContext;
    }

    public void CreateNewUser(User newUser)
    {
        _userServiceContext.AddObject(_userServiceContext.TableName, newUser);
        _userServiceContext.SaveChangesWithRetries();
    }
}

As you can see we take a UserServiceContext as a parameter in the constructor. This is our way into Azure Table Storage. Since we don’t inject a interface in the constructor, this class will be quite hard to test. So we’ll try to keep this class as simple and without logic as much as possible.

Since the WCF Service runs in an Azure WebRole, my first thought was to initialize the ObjectFactory in the OnStart() method of the RoleEntryPoint. From personal experience I can tell you that it won’t work, at least not in Azure SDK 1.3. Instead I put the initialization in Global.asax.cs Application_Start(…) method.

void Application_Start(object sender, EventArgs e)
{
    Bootstrap();
    RegisterRoutes();
}

public void Bootstrap()
{
    var storageCredentials = CloudStorageAccount.FromConfigurationSetting("DataConnectionString").Credentials;
    var cloudTableClient = CloudStorageAccount.FromConfigurationSetting("DataConnectionString").CreateCloudTableClient();
    var tableEndpoint = cloudTableClient.BaseUri;

    ObjectFactory.Initialize(x =>
    {
        x.Scan(scanner =>
        {
            scanner.AssemblyContainingType<IUserRepository>();
            scanner.TheCallingAssembly();
            scanner.LookForRegistries();
            scanner.WithDefaultConventions();
        });
        x.For<UserServiceContext>().Use(() => new UserServiceContext(tableEndpoint, storageCredentials));
    });
}

There is really no magic here (except the awesomeness of StructureMap). We let StructureMap scan our assemblies for type to register and we specifically tell it how to create the UserServiceContext so that it can be injected into our UserRepository.

Summary

With all this in place we basically have our infrastructure completed and hopefully we only need to make few, if any changes, while developing. We only use StructureMap directly in two places so we don’t need take any special care depending on which IoC container we use. It’s simply enough to create new types and inject them into the constructors and StructureMap will find and register them as we go.

Thankfully, this can be done using a small ruby script. First, install the needed ruby gems:

>sudo gem install right_aws

Then, paste this into your text editor, and edit before you run:

 
#!/usr/bin/env ruby -w
 
require 'rubygems'
require 'right_aws'
 
s3 = RightAws::S3.new(aws_access_key='your-access-key-here', aws_secret_access_key='your-secret-access-key-here')
 
s3.buckets.each do |bucket|
 
  keys = bucket.keys('max-keys'=>1000 )
 
  while keys.length > 0 do
    keys.each do |key|
      puts key.full_name
      key.delete
    end
    keys = bucket.keys('max-keys'=>1000 )
  end
 
  bucket.delete
 
end
 

Note that I only get a 1000 keys at a time. Having a lot of files in a bucket will break most GUI tools, because they try to do to much - like listing all of my files...

Happy deleting!

Tip: Run this for an instance on Amazon, not from your own computer. Running inside Amazon is much faster.

UPDATE: There is now a live CodePlex project for the realization of this concept; A .NET Client Library for Neo4j over HTTP using REST and JSON; http://neo4jrestsharp.codeplex.com/

Intro

Neo Technologies have come out with a Neo4j REST API for their popular Neo4j graph database. Since I am employed at Jayway and a couple of colleagues working with Neo are too a little bird by the name of Peter Neubauer tweeted in my ear that there was a free T-Shirt involved in making a working sample for .NET that communicated with the Neo4j server. Not only was that a handsome offer – I wanted to check out what the Architecture of such a client library for a graph database would look like.

I’ve recently been involved in client library abstractions and good architecture with the RESTful interface to Windows Azure Storage. (More on Azure and storage on my blog here, here, here, here and here.) Enough with the side tracking (sorry)!

Said and done; We set out to code up a working sample for .NET. I hacked the client code. The Neo-guys cheered me on and offered very helpful assistance when I ran into trouble or was just plane stumped on how to proceed with the API. A very nice team effort, including a remote team debugging session, later and we have a small and running POC

Tweeting on this effort it is notable how a bunch of people answer back with an interest in this idea for potential applications in the .NET world. I’ve been approached both on twitter (@noopman) and on LinkedIn with questions on this.

This blog post is to submit our findings to the world and hope that someone will be inspired to get going with a real .NET client library for Neo4j.

Architecture

First let’s look at the architecture for creating a client library for a RESTful service. This is a more general discussion that leads to our client library POC below. If your main reason for reading is the actual sample; scroll down.

When doing a client library abstractions and domain thinking comes into play. What we want is for the forgotten users (developers) of your application to work with their normal domain objects in the business code and for them to use an API that makes sense to their domain while talking to a highly specific graph database backend store. This means you have to keep all the RESTfulness, HTTP and json away from the domain code. Let me show you a sketch and see if I can make this explanation a bit less wordy:

At the other side of the Cloud from our .NET code is a Java based graph database. It is friendly enough to expose a RESTful API so that we may speak to it.

Doing so we want to create a client library that is capable of a couple of things:1) Taking care of serializing objects to json (and deserializing them back again). 2) Knowing about HTTP requests and how to handle them. In our sample we employed the power of the HttpWebRequest class to handle things like HTTP Headers, Content Types, URIs etc. 3) Finally this library must understand to expose an interface to your code that you can use from a specialized Data Access Component.

To protect the business layer from knowing anything about the specifics of the Neo4j API you really should use a specialized Data Access Component. There is now law being broken if you don’t do that. However a good rule of thumb is to add abstraction layers in your system to shield your domain code in your business components from any specifics of an underlying storage structure. The details of serialization of objects to json and making requests over HTTP are not a natural fit to the business problem you are solving. The code that does this separation is code you write yourself in your application. Only you know how to adapt a foreign library and technology to your domain. The sole purpose of this code, in other words, is to abstract away from your business logic the underlying technologies used to persist and query data used in the application. This component translates between the two worlds of your business logic and the techniques used to store data in a graph database. Most of the specific technical details of storage are wrapped up in the Client Lib mind you but the Data Access Layer DAL knows about this problem domain. It understands that the persistence indeed is a REST based service over HTTP with all that this entails. This component in your code can handle synchonisity and take care of specific errors or faults that may occur when using the storage. All of this is done in a language that is fairly close to the API that Neo4j exposes. In the end this component translates into terms useful to the domain in which your business operates. Instead of saying .PostNode() in your code perhaps you instead expose a .StorePerson() action.

What we achieve is a good and sound (and classic) layering model: Business Logic – DAL – Client Lib.

There is one more observation to make in the picture and that is the matter of IoC - Inversion of Control. In order to keep your code testable, maintainable and untangled from the implementations at lower layers you should always employ the method of Inversion of Control – making your business code for instance not reference the DAL direct but instead use a common contract to do so. The Depencency Inversion Principle states briefly that a higher layer should never depend direct on a lower layer but instead depend on an abstraction. If this is true you are able to test your business logic without using the actual underlying storage. This is for instance key in a TDD – Test Driven Development development style.

Code that depends on abstractions can be tested very quickly and does not depend on specific data in a data store to be correct in order to test the intended behavior. The Business Logic uses a contract for the DAL and the DAL implements it. Then you use something called an IoC container (read on at Wikipedia in the link above) to create the DAL instances and hand them to the Business Logic employing the technique of DI -Dependency Injection. This separation enables you to modify, re-deploy or even exchange DAL components without having to touch your Business Layer at all. If separating the Business Logic from the DAL using contracts is important it is almost event more important to do so in the lower layer that in run time speaks direct to the data store! You never want to end up in the situation where in order to test your DAL code you have to have access to a specific data store with specific data in it. Not only is that an inconvenience; it also makes executing your tests run as slow as transferring data over the Internet compared to as fast as your CPU can execute code. From experience it is an obvious conclusion that tests that are slow to run are not in fact run. The conclusion from all of these IoC discussions can only be that it is imperative that your Client Library comes with an abstraction layer.

What did we do with the Architecture in our POC?

All of these architectural considerations are of course premature in a POC. This is why in our sample we did not really separate the DAL code from the Client Lib. However to prove the point of IoC and DI I did however separate out the Business Logic from the Data Access logic using a simple contract and an equally simple Factory Method Pattern. Instead of new:ing up a DAL component in my code I called a factory method to access an abstraction of the DAL.

Our POC sample

Our code sample creates a small node graph with a few relationships. Then it makes a traversal of the graph to find all who “knows” someone according to the relationships. Finally it deletes the graph again.

Here is the graph we create in our sample: http://wiki.neo4j.org/content/The_Matrix We only create names on nodes and types on relationships in our simple sample.

And here is also the code so that you may download and follow along with our explanation your self:

First, in accordance with architectural considerations (above) we get hold of our DAL using a factory method. We also create a new Person object and store him (in this case Thomas Anderson – Neo):

INeo4jStorageClient neo4JStorageClient =
Neo4jStorageFactory.GetNodeStorageClient(neo4jServerUri);

neo = new Person("Thomas Anderson");
neo4JStorageClient.StoreNode(neo);

As you can see in our business code this is just standard .NET classes being instantiated and a “Neo” Person is stored in a very normal way. In the business logic code there is no real details on how this storage happens. There is of course a “node” concept and you can consider for your self if that should be hidden away from business code too.

Since this is a POC the implementation of .StoreNode(neo) is very very trivial. If this were a real client library we would have included things like a library to transform (serialize) the neo-Person node into a json string. Now we just do it statically:

public void StoreNode(Person person)
{
    var name = person.Name;

    string jsonString = @"{""Name"":""" + name + @"""}";

    var nodeProtocol = new NodeProtocol(uri);

    var nodeId = nodeProtocol.PostNode(jsonString);
    person.Id = nodeId;
}

The interesting part here is the existence of a NodeProtocol implementation. I’ve found it useful working with a remote API like Windows Azure Storage and also with this Neo4j REST API to implement methods in C# code that correlate very closely to the API methods available. In our case we have a PostNode operation on the REST API and consequently we have a .PostNode() method in the C# implementation of that HTTP method call.

public int PostNode(string nodeData)
{
    var uri = HttpHelper.ConcatUri(baseUri, "node");

    HttpWebRequest request =
        Request.CreateWebRequest(HttpWebRequestMethod.POST, uri);

    HttpHelper.AddBody(request, nodeData);

    var response = requestMaker.MakeRequest(request);

    return HttpHelper.FindNodeId(response.Item3);
}

I won’t bore you with the exact details of all of the helper classes. To dig in to those it is better if you just download the code and look at the details your self.

As you can see we create a HttpWebRequest, add some headers to it (in the CreateWebRequest method) specifying things like request.Accept = "application/json"; meaning we accept a json formatted response. Also we specify that the body we send out is formatted as json with request.ContentType = "application/json";. Finally we make the call and return the id of the node we just created.

Again this is a solution bordering on triviality but still it is to the point of making a REST request and receiving a response. Also there is no code to handle any errors present here and the request is done synchronously.

What does the actual HTTP REST and json request and response look like for this simple sample?

Request:

POST http://try.neo4j.org:9999/node HTTP/1.1
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 26
Expect: 100-continue
Connection: Keep-Alive


{"Name":"Thomas Anderson"}

Response (note that the added line breaks in the response body were added by me):

HTTP/1.1 201 Created
server: grizzly/1.8.1
Content-Encoding: UTF-8
Location: http://try.neo4j.org:9999/node/1

Content-Type: application/json
Content-Length: 860
Date: Fri, 16 Apr 2010 08:36:45 GMT

{"incoming typed relationships":

http://try.neo4j.org:9999/node/1/relationships/in/{-list|&|types},

"incoming relationships":http://try.neo4j.org:9999/node/1/relationships/in,

"all relationships":http://try.neo4j.org:9999/node/1/relationships/all,

"create relationship":http://try.neo4j.org:9999/node/1/relationships,

"data":{"Name":"Thomas Anderson"},

"traverse":http://try.neo4j.org:9999/node/1/traverse/{returnType},

"property":http://try.neo4j.org:9999/node/1/properties/{key},

"self":http://try.neo4j.org:9999/node/1,

"properties":http://try.neo4j.org:9999/node/1/properties,

"all typed relationships":

http://try.neo4j.org:9999/node/1/relationships/all/{-list|&|types},

"outgoing typed relationships":

http://try.neo4j.org:9999/node/1/relationships/out/{-list|&|types},

"outgoing relationships":http://try.neo4j.org:9999/node/1/relationships/out}

As you can see the Neo4j REST API is self explanatory. Creating a node gives you back a response with all the things you can now do with this new node over the API. Since this quite a lot of data when the only thing that really matters is in the header (Location: http://try.neo4j.org:9999/node/1) I’ve already given feedback to the Neo team that I’d like a simple Uri switch (like APIDocumentation=0) to shut off the detailed response. But in fact I do like that you get so many Uris back that you may use direct. Depending on your needs this is either good or to verbose a payload to get back in your response. As you can see the node we created got the Id set to 1.

Now we proceed to create all the other nodes and in a very similar fashion the relationships between nodes. Here is the relationship that says that “Neo knows Trinity” (who has a node id of 2). This is a POST operation to /node/1/relationships:

POST http://try.neo4j.org:9999/node/1/relationships HTTP/1.1
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 56
Expect: 100-continue

{"type":"Knows","to":http://try.neo4j.org:9999/node/2}

Once the node graph for the Matrix is created we proceed to make a traversal of the graph. We want to find anyone who knows someone throughout the graph.

The request looks like this in REST (after a similar path through the code as above with a call from business logic to DAL and through to the client lib that makes the actual call using REST):

POST http://try.neo4j.org:9999/node/1/traverse/node HTTP/1.1
Accept: application/json
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 57
Expect: 100-continue
Connection: Keep-Alive

{ "relationships": { "type": "Knows" }, "max depth": 10 }

Again we simplified to a “max depth” of 10 relationships deep into the graph which works fine for our sample.

The result is a list of all the nodes that match. Because of the self explanatory API mentioned above the result also gives you actions to perform on each node that comes back in the response thus making the response body quite long. Rather than pasting the whole thing here into the blog post you can try it out your self (by downloading the application) and you will get a result exactly like this from the app:

Neo knows Trinity and Morpheus, who knows Cypher, who in turn knows Agent Smith.

Difficulties

We had only two real challenges in coding this POC.

The first one was an issue with streams and the optional so called Byte Order Mark (BOM) that you can put at the head of a stream. I’ve posted on this here: Would you like a Byte Order Mark to go with that?.

The other challenge was that I wrote a small app for Windows to show our results. But the guys on the Neo team are certainly not Windows users. The wanted to show the code on Mono. Just to be sure of compatibility I scaled back to a good old Windows Forms demo app in .NET Framework 3.0. This worked great on Mono which was our intent. But the drawback was that the library I’d found to handle object serialization didn’t respond well to that old version of the framework. That’s why I replaced my lib with static string creations (as you can see in the implementation of the .StoreNode() method above).

What have we learned from this?

First of all creating a client library for .NET to speak REST and json over HTTP to Neo4j is very doable; quite easy and straight forward.

Second (and that’s just a word of warning); keep your layers separated. This goes for all applications I guess but if you start to put code to create instances of web requests with web methods and content types and such in your business code you will end up in a very tangled and messy situation very fast which will be a pain and a high cost to maintain.

Finally we have considered looking into using WCF data services (and OData – the Open Data Protocol) and also possibly Linq – Language Integrated Query on the client side to make implementations in the library even more powerful and useful. And we do need a object to json serializer library.

Oh and here is another post on using the Neo4j REST API by the team itself: http://blog.neo4j.org/2010/04/neo4j-rest-server-part1-get-it-going.html

That’s it!

We hope this is inspirational enough for you to get going with your own samples or even client libraries in .NET to speak to Neo4j. If you do; please get back in touch with us!

Cheers,

M.

Our session at TechDays 2010 gave an overview of what storage in the Cloud is, according to Microsoft at present date, and some of the goals for the future in this area. We intended our session to provide the basis for companies to be able to evaluate if and how moving storage to the Cloud can enhance their business for their own specific scenarios.

Speaking about Cloud storage and Microsoft today means you need to give a session divided into two parts; SQL Azure and Windows Azure storage. And so our presentation had two distinct parts.

As consultants working with different clients we see that companies in general are not yet ready to adopt Cloud storage on a broad scale. However we also see a continuous increase in interest in this area and we believe that this is directly related to the richness of the offerings and services now becoming available. The story matures in the Microsoft camp and the pace is rapid. Cloud storage from Microsoft is admittedly still very young. (Youngest in the business?) But now Microsoft has thrown the full weight of the company behind making the cloud story stick. Expect to see a lot of new offerings, services, products, frameworks etc. in the Cloud coming out from Microsoft in the near future.

There are a few common concerns when it comes to storing data in the cloud. A quick (and perhaps not complete) listing follows:

Storage cost in the Cloud

One big concern we hear from clients and in the buzz at conferences that take place around now is a concern related to the cost of storage and data transfer in the Cloud. This is naturally an equally important issue regardless in what Cloud you choose to store your data.

The thing about storing your data in the Cloud is that the cost for this storage is a lot more self evident compared to storing your data in your own servers on premise. This is because you receive a bill each month stating exactly what you have spent for storage in the passed period. What we tend to mitigate is that storage, backups, crash plans and managing all of this on premise, or even hosted, does not exactly come for free either. The big question then becomes what difference in cost there will be for your company for storing and transferring data to a Cloud compared to local hardware operated by your local IT department.

Another aspect of the cost for storage in general is the question of how you build your apps. We believe that many have been a bit oblivious to this issue when they have run their apps and stored our data only on premise or with traditional hosting. Perhaps we have been cost inefficient in this area until now? In the age of Soft Deletes - deleting data does not remove it but rather marks it as deleted – no data is ever actually deleted - managing costs for storing data will become a more and more important question for everyone. Building Cloud applications and storing data in the Cloud has to be done with data storing and management costs in mind. As we hinted to above; the same should have been true for all of your applications up until now. Storing data in the Cloud just makes this point more obvious.

Storage Security when your data in the Cloud

Perhaps the fist fear when it comes to storing data in the Cloud is the fear of security breeches and the perceived loss of control of this security. Your data is no longer at home.

What you need to ask yourself, and here we choose to quote a good colleague Sergio Molero @sergio_molero, is; “What sense of security do you have today? What security is it that you think you will loose when storing your data in the Cloud?”

The same exact thing can be said for hosted data as compared to data stored in the Cloud because the two concepts are very much the same thing in this regard. Data in the Cloud is also hosted data!

One important thing to also keep in mind is that moving to a data storage solution in the Cloud does not mean all of your data must move “out there” somewhere. It is still quite possible to store the most sensitive data on premise and use Cloud storage for more public data only. We believe that the higher focus on security for data in the Cloud will put higher focus on your security efforts compared to your present security on premise. The move to the Cloud could in conclusion prove to heighten the overall security level for your data.

Also look into Windows Identity Foundation (WIF) for insights on how to secure your data in the Cloud.

Trusting someone else with your data in the Cloud

If security from external attacks to your data in the Cloud is a concern you feel somewhat confident you will be able to handle then perhaps trusting someone else with storing your company sensitive data is a concern that causes you to think twice? Trust is a complex issue not to be taken lightly. We will only raise the concern here and not delve too deep into it.

Can you trust a hosted data storage provider today? If you can, then this is no difference compared to trusting data stored in the Cloud.

If you can’t trust anyone else with your data, then the Cloud is not an option for you. Clean and simple.

Legal ramifications for storing data in the Cloud

It feels important to also comment briefly on the potential legal ramifications for storing your data in a Cloud in another country somewhere. That being said we’d just like to point out that we are no lawyers. Let’s be clear on that.

Some data cannot legally be moved outside the borders of your own country. Hospital patient information is a good example of this. Other data must for legal reasons stay within your geo-region, say for instance the EU.

As we hinted to above; not all of your data has this limiting concern. You do have more public data as well as more private data in your business. Only the latter is a potential legal problem. There will be more hybrid solutions for data Storage in the future. Some data for a company will be stored in the Clouds and other data will be stored on premise.

But how can we store data both on premise and in the Cloud without having two separate data storage solutions? One thing that will assist in this endeavor is the rumored upcoming availability of local on-premise, or hosted, Clouds that will be possible to install locally and behave in the same way the big Clouds out there on the Internet do. Right now there does not exist many offerings of this nature. There will in time when companies start to demand them. In fact, given that the storage Clouds today has a REST interface, it is not very difficult to implement that same interface on your own on premise machines! Doing this however will require a good effort on your part and you will end up with a proprietary solution that you will have to evolve over time as the real Clouds shift. Another option is to wait a bit and let the Cloud vendors come up with packaged solutions that you may install simply install in your own data centers. The hybrid solutions that will result from this will play an important role in shaping the solutions for data storage for the coming decade.

SQL Azure

In our SQL Azure part of our talk we pointed out three pros and three cons for attendees to take away from our session.

1. Because Windows Azure takes care of all of the maintenance normally associated with a SQL Server server developers are now allowed to focus on the database rather than performing administrative tasks.
2. The database is always available in the sense that upgrades are seamless as well as failover, backup and crash plan which means that the SQL Azure database is always available.
3. The scaling story is very strong in SQL Azure. If you can shard your database into many pieces you can scale out from one database to many (say 100) databases to meet a peek load and then back down again to just one at will. For now there is no built in support for this in SQL Azure but this will likely be a story further down the road and you can build it your self today.

There are some things to consider when looking into deploying to SQL Azure.

1. Security is locked down compared to SQL Server. The sys-tables are not yet available for instance. This is because your databases are co-located on the same servers as data belonging to some one else.
2. Your data is no longer at home in the sense that it is moved outside your company and even outside your country.
3. The size right now for one database in SQL Azure is 10 GB but this will increase to 50 GB in June.

Windows Azure Storage

With Windows Azure Storage Microsoft addresses a set of data storage stories that do not really fit well into a standard relational database structure.

1. There is a powerful and “infinite” message queuing system that enables messaging between the different machines in your applications. This queuing system does not guarantee 100% ordering of the messages in the queue. This is in case of more than one machine working on the same queue. However the queues require dequeue (GET) and delete (DELETE) in order to complete an actual removal of a message. This method will ensure each message will actually be handled in full in case of some error when decueing and handling the message the first time.
2. There is also a concept in Windows Azure of that which is known as a “Big Table”. Say you want to store 200 million user data sets for Facebook or why not all the messages ever sent to Twitter. For this you need an “infinite” tabular structure that has no end that you will ever hit.
3. The third thing you can store in Windwos Azure are two kinds of blobs. 1) A so called Block Blob. A blob that is handled in blocks when uploaded. A block Blob is not really intended to be changed once uploaded. You can exchange blocks in the blob but that is not the intent. This is typically a video file or a picture or something that is rarely if ever changed. You would use block blobs to build the next youtube where you can stream files down into a player. 2) A Page Blob. This kind of blob is divided into pages rather than blocks. You can real or write pages at will into this kind of blob. This means that you are able to create a Virtual HardDrive (VHD) and upload it as a page blob to Windows Azure. Then you can actually mount this blob to one or more virtual machines in Windows Azure as an extra hard drive. This comes in handy when moving an application to Windows Azure that uses a lot of disk. All of your old File.Open will continue to work.

When considering Windows Azure for your storage think about:

1. You should use only the parts of Windows Azure that you really need. If you need to store backup blob type of data – use only that. There is a good advantage to consider when you have applications to run that will use Windows Azure Storage. If you deploy your application to Windows Azure as well as the data to the same datacenter then the internal data transfer costs between this application and storage is without cost. If you instead run your apps on premise and store data in the cloud the transfer of data to and from the cloud carries a cost.
2. Regarding the cost you should really make a data storage estimate and analysis before you start up deploying. Your application should have a plan for the amount of data stored, transferred and the duration of storage. This is true on premise too, as discussed above, and data storage plans and budgets will become a more and more important part of any application for the future.
3. Windows Azure Storage means learning to use a new API. Perhaps not a dramatic thing but the three storage options also affects the way you build the rest of your application. If you disagree chances are you need to learn more about storing data in the cloud.

Links

Windows Azure SDK

Windows Azure Tools for Visual Studio

Cloud Storage Studio – a SSMS look alike for Windows Azure Storage

Fiddler2 – Web debugging proxy

CloudStorage.API @ CodePlex

Azure Contrib @ CodePlex

SQL Azure Migration Wizard

Downloads

Ziped version of the demo code in the Windows Azure Queue Storage sample:

AzureStorage

Slides from the presentation:

EffektivOchOandligLagringIMolnet.ppsx

EffektivOchOandligLagringIMolnet.pdf

Slides for the demo attached as a zip above

DemoWindowsAzureCloudQueue.ppsx

DemoWindowsAzureCloudQueue.pdf

Final Words

We would like to thank Microsoft for arranging TechDays, we had a blast! Thanks to the audience for listening.

Cheers,

Peter von Lochow @vonlochow and Magnus Mårtensson @noopman

First step is to boot an instance. You need the command line tools for this, as neither the AWS Management Web Console nor ElasticFox currently supports this right now (note to developers - please add!).

ec2-run-instances --region eu-west-1 \
     -K pk-YOURKEYHERE.pem \
     -C cert-YOURCERTHERE.pem \
     -b '/dev/sda1=:100' \
     ami-d7a78ca3 \
     -g my-security-group -g common-security-group \
     -t m1.large \
     -k thekeyhere \
     --disable-api-termination \
     --instance-initiated-shutdown-behavior stop \
     -z eu-west-1a

• The command is ec2-run-instances with an 's' in the end. There is another command in the API without the s, which doesn't work.
• -b is the most important flag here. It says to make the boot disk (/dev/sda is Linux speek for C:) 100 GB, despite what the AMI manifest says (which is 30 GB).
• ami-d7a78ca3 is the Windows image, in this case a 64bit Windows 2008.
• -g security-group you should add as many as you like. Remember to think these through, since you cannot change which groups an instance belongs to.
• The flags --disable-api-termination --instance-initiated-shutdown-behavior stop are really nice, because that make sure you cannot accidentally terminate the instance, making the harddrive disappear.
• Note that I'm running in the EU region here.

Ok, so now you have a running Windows server. Once it has booted, you log on using remote desktop (well, as soon as you have the generated password, which takes 15 min). PLease note that the C: drive will show up as 30 GB at this point. The partition is still only 30 GB, even if the virtual harddrive is 100 GB.

1. Go into the windows management console. Go to disk management.
2. Right-click on the C: drive. Choose Extend volume from the context menu.
3. In the wizard that pops up, just accept the defaults, and click finish.

Done! The change is instant (no reboot or anything), and your system drive C: is now 100 GB.

That said, I must also give credit to DreamHost for always being very helpful and supporting when needed.

If you want to know what we did to gain the speed needed for a lift-of, keep on reading.

Exporting the Existing Database

I really like MySQL, but when it comes to encoding for character sets and collations there seems to be such a complexity in the details that users often get lost and configure it the wrong way. The UTF-8 configured database we used on DreamHost worked just fine together with our WordPress installation but it turned out to be impossible to export a correct UTF-8 version for importing into our new environment. And I tried a lot of things. Really. I'm not going to go into details but to sum it up I had to sit down and do kind of manual replacements of the swedish specific characters in the database export. Lucky me this blog is in english.

Anyway, I had ssh access to the Trafficante server and could use this connection to export the database from the DreamHost MySQL server by issueing:

 
mysqldump --default-character-set=UTF8 -u [user] -p -h [host].dreamhosters.com [database] -r [output].sql
 

This resulted in a database dump later used for importing into the new MySQL instance.

Setting up the Amazon EC2 Instance

To make your Amazon life easier start with installing the Firefox plugins Elasticfox and S3 Organizer. These tools give you a GUI view of the somewhat more complex underlying details of Amazon.

Amazons' cloud solution is based on a user account. There are no start-up fees, you only pay for what you use. So if you still haven't got an account, sign-up now. The rest of this article assumes you have signed up and set up accounts for Amazon EC2, Amazon S3 and Amazon EBS. Start by reading the EC2 Getting Started Guide and don't stop reading until you have created an ssh key pair, a security group and have a server instance running.

Now you should also have a private key and a certificate in your ${EC2_HOME} (preferrably ${HOME}/.ec2/) folder of your local computer. These two files should also be uploaded to your instance /mnt folder to make it possible to bundle and upload your images to your S3 "bucket(s)" (buckets are one level deep folders/directories). Make sure you put them in the /mnt folder to prevent them from being bundled with the image and thereby increasing the risk of having your identity revealed.

Bundling, by the way, should be done when you've made changes to your server instance that you don't want to loose. The result of a bundling and uploading is an image with a specific Amazon Image Id (AMI ID) reflecting the state of your server instance at the moment of bundling. You also have to register the image with the manifest file created for you. If you do that you could then safely close down your instance and have it up and running again by issuing the ec2-run-instances command with the AMI ID as parameter. A way to save money.

Bundling and uploading was made easier for me as I used two scripts created by my colleague Ulrik. I put these scripts in the /root folder.

bundle-myself.sh:
 
#!/bin/sh
prg=`basename $0`
dest=/mnt
image=blog-image
 
USAGE="Usage: $prg <datetime>
 
Example:
% $prg 20090415-1000
Bundling into $dest/$image-20090415-1000.manifest.xml"
 
if [ $# -ne 1 ]; then
  echo "$USAGE"
  exit 1
fi
date=$1
echo "Bundling into $dest/$image-$date.manifest.xml"
ec2-bundle-vol --debug -k /mnt/pk-PRIVATE_KEY.pem -c /mnt/cert-CERTIFICATE.pem -u USER_ACCOUNT_ID -d $dest -p $image-$dat
e -r i386
 

upload-myself.sh:
 
#!/bin/sh
prg=`basename $0`
dest=/mnt
image=blog-image
 
USAGE="Usage: $prg <date>
 
Example:
% $prg 20090415-1000
Uploading bundled AMI parts to https://s3.amazonaws.com:443/[S3_BUCKET] ...
Uploaded blog-image-20090415-1000.part.00 to https://s3.amazonaws.com:443/[S3_BUCKET]/blog-image-20090415-1000.part.00.
..."
 
if [ $# -ne 1 ]; then
  echo "$USAGE"
  exit 1
fi
date=$1
ec2-upload-bundle --retry -m $dest/$image-$date.manifest.xml -b blog-images -a [PUBLIC_KEY] -s [PRIVATE_KEY]
 

Another thing I needed was a static IP address for my instance. To solve this I used Amazons Elastic IP service which doesn't cost anything as long as the generated IP address is in use. So, I generated an Elastic IP address and assigned it to the instance.

Installing Necessary Software

The blog require Apache2, Php5, MySQL and the Php GD graphics package, to start with. This was easily installed using Advanced Packaging Tool, apt-get:

 
# sudo apt-get update
# sudo apt-get install apache2 php5 mysql-server php5-mysql php5-gd
 

Setting up the Database for EBS

EBS volumes are, as you might know, more persistent than EC2 instances and will survive if instances go down. Volumes are attached to the instance and could as well be detached and re-attached to new instances. As EBS volumes are not 100% reliable you should also take snapshots of your database data and have it uploaded to an S3 bucket. All this is nicely described in Eric Hammonds' EBS and MySQL tutorial, so I urge you to read it if you haven't yet.

The only thing I sidestepped in the tutorial was the MySQL configuration to point out the data and log directories. Instead I used symbolic links:

 
# ln -s /vol/lib/mysql/ /var/lib/mysql
# ln -s /vol/log/mysql/ /var/log/mysql
 

The tutorial also walks you through automatic snapshots which requires the EC2 command line tools. Here is a nice tutorial on setting these tools up on Ubuntu.

I had to tweak the environment variable settings in ec2-snapshot-xfs-mysql.pl script from

BEGIN {
  $ENV{PATH} = '/bin:/usr/bin:/usr/local/bin';
}

to

BEGIN {
  $ENV{PATH} = '/bin:/usr/bin:/usr/local/bin:/usr/sbin:/root/ec2-api-tools/bin';
  $ENV{EC2_CERT} = '/mnt/cert-CERTIFICATE.pem';
  $ENV{EC2_PRIVATE_KEY} = '/mnt/pk-PRIVATE_KEY.pem';
  $ENV{EC2_HOME} = '/root/ec2-api-tools';
  $ENV{HOME} = '/root';
  $ENV{JAVA_HOME} = '/usr/lib/jvm/java-1.5.0-sun';
}

to get the snapshotting to fully work. I also wrapped the ec2-snapshot-xfs-mysql.pl script for a more convenient way to handle the input:

/root/snapshot-myself.sh

#!/bin/sh
mountpoint=/vol
volume_id=[VOLUME_ID]

su -c "/root/ec2-snapshot-xfs-mysql.pl ${mountpoint} ${volume_id} > /vol/cron.log"

To automatize the snapshotting I set up a simple crontab job by issueing:

 
# crontab -e
 

and then edited it like this

05 */6 * * * sh /root/snapshot-myself.sh

which makes the job run five minutes past every six hours, every day.

I also had to set the timezone info to apply to our local time. The easiest way I've found out of setting this for a Linux server is like this:

 
# ln -sf /usr/share/zoneinfo/Europe/Stockholm /etc/localtime
 

Configuring MySQL

To configure MySQL for UTF-8 content make sure the /etc/mysql/my.cnf file contains the following:

[client]
default-character-set=utf8

[mysqld]
#
# * Basic Settings
#
init_connect='SET collation_connection = utf8_general_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_general_ci
default-character-set=utf8
skip-character-set-client-handshake

Now, if you haven't yet, go on and start your MySQL server. You need to create your wordpress database. It's as simple as:

 
# mysql -u [user] -p
> CREATE DATABASE [database_name]
 

To create the user with which WordPress will access the database:

 
> GRANT ALL ON [database_name].* TO '[user]'@'localhost' IDENTIFIED BY '[password]';
 

And to import your database dump into the newly created database:

 
# mysql -u [user] -p [database_name] < [output].sql
 

You shouldn't need to specify any character set or collation when importing thanks to the configuration done in /etc/mysql/my.cnf.

In order for us to stick with our domain name and have no downtime we needed to use the instances IP address until we were finished testing. This means I had to replace two properties in the database to prevent the blog from redirecting to the original host; the 'siteurl' and 'blogurl' properties residing in the option_value column of the wp_options table. The commands

 
# mysql -u [user] -p
> UPDATE wp_options SET option_value='http://[ip_address]/wordpress/' WHERE option_name='siteurl'
> UPDATE wp_options SET option_value='http://[ip_address]/wordpress/' WHERE option_name='blogurl'
 

fixed this. As soon as the DNS info was propagated throughout the Internet this was reset to "http://blog.jayway.com/wordpress" from within the WordPress admin interface.

Apache Configuration

Our WordPress installation makes use of permalinks and redirects to FeedBurner for RSS statistics. This is based on .htaccess redirects (at least that's how we've solved it) which in turn is dependent on the Apache redirect module. So, in order to make this to work I had to enable that.

 
# sudo a2enmod rewrite
 

Easy as that! One more thing on redirections. I had to configure Apache to allow redirection directives specified in the .htaccess file by changing the /etc/apache2/sites-available/default configuration of the element from

 
	<Directory /var/www/>
		Options Indexes FollowSymLinks MultiViews
		AllowOverride None
		Order allow,deny
		allow from all
	</Directory>
 

to

 
	<Directory /var/www/>
		Options Indexes FollowSymLinks MultiViews
		AllowOverride All <----------
		Order allow,deny
		allow from all
	</Directory>
 

Securing the WordPress Admin Interface

Again, I choose to refer to detailed and well written sources. Here are the two articles I followed to create a self-signed certificate and to configure Apache to use it.

• Creating a self-signed certificate
• Configuring Apache for SSL on Ubuntu

To enable https access from the "outside world" I also edited the group policy to open up port 443. This could easily be done via the Elasticfox Firefox plugin mentioned earlier.

Installing and Configuring WordPress

The default location for Apache httpd server for reading content is /var/www/. If you, as I did, have an existing WordPress installation you can simply copy this to the new server. I ended up having it located at /var/www/wordpress/ on the Amazon instance.

Once the database connection properties in the wp-config.php wordpress configuration file was adapted to the new environment I was able to access the WordPress installation via a browser and the Amazon instances' IP address. Sweet!

And as we have our WordPress installation in a separate "/wordpress" folder instead of in the document root of Apache we need to specify, in WordPress admin settings, that the 'Site URL' should be "${host}/wordpress" but the 'Blog URL' should simply say "${host}" to prevent the resulting URL from containing "/wordpress/...". This also requires that we move the .htaccess and index.php files from within our "/wordpress" folder and up one level to "/var/www/", in our case.

In the index.php file, make sure to change from

 
/** Loads the WordPress Environment and Template */
require('./wp-blog-header.php');
 

to

 
/** Loads the WordPress Environment and Template */
require('./wordpress/wp-blog-header.php');
 

If or when specifying the permalinks, WordPress must have write access to the .htaccess file, so 'chmod' that file.

 
# chmod 755 .htaccess
 

Otherwise it will complain and simply print out the .htaccess modification it tried to modify with and you are left to change it manually, instead.

Sending Mail from WordPress

Mailing is used in WordPress when generating new passwords to users, among others. We also use it to send data backups to the administator of the blog.

Setting up a mail server isn't a trivial task. I gave it a thought but quickly put it aside in favour of the simple WordPress plugin WP Mail SMTP as I realized I could make use of our existing corporate mail server. It was a breeze to configure the plugin with the necessary mail user credentials.

So, here we are. Sailing these fluffy clouds.

Prerequisites

In order to keep this reasonably brief I'll have to refer to the getting started guide for information on how to get going with Amazon EC2.

A FactoryBean to Launch EC2 Instances

What we want to achieve here is to launch an EC2 instance transparently and independently of the actual test code. Ideally, the test code should be oblivious of the target server and we want to externalize those details to external configuration. We will use Spring's excellent JUnit test support to automatically wire the integration test setup and make sure that the target server is up and running before the actual test code is running.

A powerful way to transparently perform complex initialization logic when using Spring is the FactoryBean concept. We will create a FactoryBean implementation to launch the EC2 image and set up our target resource:

AbstractEc2InstanceLaunchingFactoryBean.java
public abstract class AbstractEc2InstanceLaunchingFactoryBean extends AbstractFactoryBean {
  private static final int INSTANCE_START_SLEEP_TIME = 1000;
  private static final long DEFAULT_PREPARATION_SLEEP_TIME = 30000;
  private static final Log log = LogFactory.getLog(AbstractEc2InstanceLaunchingFactoryBean.class);
  private String imageName;
  private String awsKey;
  private String awsSecretKey;
  private String keypairName;
  private String groupName;
  private Instance instance;
  private long preparationSleepTime = DEFAULT_PREPARATION_SLEEP_TIME;
  public void setImageName(String imageName) {
      this.imageName = imageName;
  }
  public void setAwsKey(String awsKey) {
      this.awsKey = awsKey;
  }
  public void setAwsSecretKey(String awsSecretKey) {
      this.awsSecretKey = awsSecretKey;
  }
  public void setKeypairName(String keypairName) {
      this.keypairName = keypairName;
  }
  public void setGroupName(String groupName) {
      this.groupName = groupName;
  }
  public void setPreparationSleepTime(long preparationSleepTime) {
    this.preparationSleepTime = preparationSleepTime;
  }
  @Override
  protected final Object createInstance() throws Exception {
      Assert.hasLength(imageName, "ImageName must be set");
      Assert.hasLength(awsKey, "AwsKey must be set");
      Assert.hasLength(awsSecretKey, "AwsSecretKey must be set");
      Assert.hasLength(keypairName, "KeyName must be set");
      Assert.hasLength(groupName, "GroupName must be set");
 
      log.info("Launching EC2 instance for image: " + imageName);
 
      Jec2 jec2 = new Jec2(awsKey, awsSecretKey);
      LaunchConfiguration launchConfiguration = new LaunchConfiguration(imageName);
      launchConfiguration.setKeyName(keypairName);
      launchConfiguration.setSecurityGroup(Collections.singletonList(groupName));
 
      ReservationDescription reservationDescription = jec2.runInstances(launchConfiguration);
      instance = reservationDescription.getInstances().get(0);
      while (!instance.isRunning() && !instance.isTerminated()) {
          log.info("Instance still starting up; sleeping " + INSTANCE_START_SLEEP_TIME + "ms");
          Thread.sleep(INSTANCE_START_SLEEP_TIME);
          reservationDescription = jec2.describeInstances(Collections.singletonList(instance.getInstanceId())).get(0);
          instance = reservationDescription.getInstances().get(0);
      }
 
      if (instance.isRunning()) {
          log.info("EC2 instance is now running");
          if (preparationSleepTime > 0) {
              log.info("Sleeping " + preparationSleepTime + "ms allowing instance services to start up properly.");
              Thread.sleep(preparationSleepTime);
              log.info("Instance prepared - proceeding");
          }
          return doCreateInstance(instance.getDnsName());
      } else {
          throw new IllegalStateException("Failed to start a new instance");
      }
   }
 
  protected abstract Object doCreateInstance(String ip) throws Exception;
 
  @Override
  protected void destroyInstance(Object ignored) throws Exception {
    if (this.instance != null) {
      log.info("Shutting down instance");
      Jec2 jec2 = new Jec2(awsKey, awsSecretKey);
      jec2.terminateInstances(Collections.singletonList(this.instance.getInstanceId()));
    }
  }
}
 

This superclass makes use of the typica library (also available in maven) to launch the EC2 instance and delegates to doCreateInstance for creating the target resource that we will use in our test case. In our setup we want to create a ContextSource, which is the Spring LDAP equivalent of a DataSource:

ContextSourceEc2InstanceLaunchingFactoryBean.java
public class ContextSourceEc2InstanceLaunchingFactoryBean extends AbstractEc2InstanceLaunchingFactoryBean {
  private String base;
  private String userDn;
  private String password;
  @Override
  public final Class getObjectType() {
    return ContextSource.class;
  }
  @Override
  protected final Object doCreateInstance(final String dnsName) throws Exception {
    Assert.hasText(userDn);
    LdapContextSource instance = new LdapContextSource();
    instance.setUrl("ldap://" + dnsName);
    instance.setUserDn(userDn);
    instance.setPassword(password);
    instance.setBase(base);
 
    instance.afterPropertiesSet();
    return instance;
  }
  public void setBase(String base) {
    this.base = base;
  }
  public void setUserDn(String userDn) {
    this.userDn = userDn;
  }
  public void setPassword(String password) {
    this.password = password;
  }
}
 

Note that the AbstractEc2InstanceLaunchingFactoryBean will wait for the instance to start up properly. It will also wait an additional period of time once the instance is up and running to allow for all relevant services to start up properly. Finally it will automatically close down the launched instance once it is properly shut down (which it will be when executing using the Spring automated test support).

Test Case Configuration

We now have the infrastructure to have the EC2 instance launched transparently from a Spring Application Context. All we need to do is configure it:

testContext.xml
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="ldap.properties" />
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
</bean>
 
<bean id="contextSource" class="org.springframework.ldap.ContextSourceEc2InstanceLaunchingFactoryBean">
<property name="awsKey" value="${aws.key}" />
<property name="awsSecretKey" value="${aws.secret.key}" />
<property name="imageName" value="${aws.ami}" />
<property name="groupName" value="${aws.security.group}" />
<property name="keypairName" value="${aws.keypair}" />
<property name="base" value="dc=jayway,dc=se" />
<property name="userDn" value="${userDn}" />
<property name="password" value="${password}" />
</bean>
 
<bean id="ldapTemplate"
  class="org.springframework.ldap.core.LdapTemplate">
  <constructor-arg ref="contextSource" />
</bean>
 

We are referring to ldap.properties for the actual settings:

userDn=cn=admin,dc=jayway,dc=se
password=secret
base=dc=jayway,dc=se
aws.ami=ami-56ec083f
aws.keypair=spring-ldap-keypair
aws.security.group=spring-ldap

The referenced AMI is a public image specifically set up for our test cases. It has OpenLDAP installed and has been pre-populated with the test data that our test cases expect.

Note that we are not specifying any property value for aws.key and aws.secret.key, as this is the Amazon account access information. That will be the account settings of the individual developer and should be supplied at runtime using system properties (e.g. mvn -Daws.key=xxxxxxx -Daws.secret.key=xxxxxx test).

Performing the Test

Now all we need to do is use Spring's automated test support to start our test:

SomeLdapITest.java
@ContextConfiguration(locations = { "testContext.xml" })
public class SomeLdapITest extends AbstractJUnit4SpringContextTests {
  @Autowired
  private LdapTemplate tested;
 
  @Test
  public void testSomething() {
    // Use the automatically injected LdapTemplate instance to perform a test.
  }
}
 

Set to Go

With the above you should be all set to go to start using this powerful approach to integration testing. The actual code is available with the 1.3.0.RC1 release of Spring LDAP; links to downloads and documentation available here.

In the perfect world, each developer would at all times have access to a server park with all of these different server versions installed. Reality is however almost always a different question: Being an Open Source project with limited funding and developers located throughout the world this has been impossible for us, up until now. You're probably already guessing: The answer is in the Clouds.

Having used Amazon EC2 in a couple of customer projects, we suddenly realized this is the answer to our problems. For those of you not completely familiar with the concept, the general idea of Amazon EC2 is that you configure one or several machine images, installing whatever software and data you might need on each image. You will then start an instance from one of these pre-configured images, which will in effect start up a virtual computer somewhere in the amazon cloud, a computer that will then be yours to play around with in any way you might like until you're finished with it and just shut the instance down. All of this is available at a very reasonable pricing - you pay only (a very modest amount) for the time that your instances have been running.

The concept suits us perfectly: we want a number of isolated environments, each with a known start state where we can run our test suite against a particular server setup. So that's what we've started to do: we have now created an EC2 image with an Open LDAP installation, pre-loaded with the data that our test cases expect. We then devised support classes to have the JUnit test cases automatically start an appropriate instance (using the excellent typica library), run the tests against the started virtual machines, and shut down the instance after the tests are finished (regardless of the outcome - we don't want any stray instances running around costing us money ). We still have some more work to do with setting up more images for other server setups, but we're hoping to get that work going as we move on...

I'm planning to dig a little bit deeper into the details of the setup and the code involved in a later post. In the meantime: if you're really interested, feel free to check the code out from the svn repo and have a look at what we've done so far. The relevant code for this is in the test/integration-tests-openldap directory.

I used the Jungle Disk software to provide the mapping from the S3 web service interface to a network drive. You give Jungle Disk your S3 account information, and it provides a network disk which you can use as the backup device. I first tried my existing backup software (Synchronize! Pro X) and it happily began creating directories and copying files. I then tried the backup feature built into Jungle Disk, and it works fine too. It's not fast compared to a local FireWire disk. It's limited to your upload bandwidth, which often is substantially smaller than the download bandwidth.

Anyway, it has been tugging along today storing 5 GB of by digital photos. It feels pretty good to know that my photos are now stored encrypted and with 99.99% availability somewhere far far away, costing me only 75 cents per month after the initial $2 for the transfer.