Thankfully, this can be done using a small ruby script. First, install the needed ruby gems:

>sudo gem install right_aws

Then, paste this into your text editor, and edit before you run:

 
#!/usr/bin/env ruby -w
 
require 'rubygems'
require 'right_aws'
 
s3 = RightAws::S3.new(aws_access_key='your-access-key-here', aws_secret_access_key='your-secret-access-key-here')
 
s3.buckets.each do |bucket|
 
  keys = bucket.keys('max-keys'=>1000 )
 
  while keys.length > 0 do
    keys.each do |key|
      puts key.full_name
      key.delete
    end
    keys = bucket.keys('max-keys'=>1000 )
  end
 
  bucket.delete
 
end
 

Note that I only get a 1000 keys at a time. Having a lot of files in a bucket will break most GUI tools, because they try to do to much - like listing all of my files...

Happy deleting!

Tip: Run this for an instance on Amazon, not from your own computer. Running inside Amazon is much faster.

First step is to boot an instance. You need the command line tools for this, as neither the AWS Management Web Console nor ElasticFox currently supports this right now (note to developers - please add!).

ec2-run-instances --region eu-west-1 \
     -K pk-YOURKEYHERE.pem \
     -C cert-YOURCERTHERE.pem \
     -b '/dev/sda1=:100' \
     ami-d7a78ca3 \
     -g my-security-group -g common-security-group \
     -t m1.large \
     -k thekeyhere \
     --disable-api-termination \
     --instance-initiated-shutdown-behavior stop \
     -z eu-west-1a

• The command is ec2-run-instances with an 's' in the end. There is another command in the API without the s, which doesn't work.
• -b is the most important flag here. It says to make the boot disk (/dev/sda is Linux speek for C:) 100 GB, despite what the AMI manifest says (which is 30 GB).
• ami-d7a78ca3 is the Windows image, in this case a 64bit Windows 2008.
• -g security-group you should add as many as you like. Remember to think these through, since you cannot change which groups an instance belongs to.
• The flags --disable-api-termination --instance-initiated-shutdown-behavior stop are really nice, because that make sure you cannot accidentally terminate the instance, making the harddrive disappear.
• Note that I'm running in the EU region here.

Ok, so now you have a running Windows server. Once it has booted, you log on using remote desktop (well, as soon as you have the generated password, which takes 15 min). PLease note that the C: drive will show up as 30 GB at this point. The partition is still only 30 GB, even if the virtual harddrive is 100 GB.

1. Go into the windows management console. Go to disk management.
2. Right-click on the C: drive. Choose Extend volume from the context menu.
3. In the wizard that pops up, just accept the defaults, and click finish.

Done! The change is instant (no reboot or anything), and your system drive C: is now 100 GB.

That said, I must also give credit to DreamHost for always being very helpful and supporting when needed.

If you want to know what we did to gain the speed needed for a lift-of, keep on reading.

Exporting the Existing Database

I really like MySQL, but when it comes to encoding for character sets and collations there seems to be such a complexity in the details that users often get lost and configure it the wrong way. The UTF-8 configured database we used on DreamHost worked just fine together with our WordPress installation but it turned out to be impossible to export a correct UTF-8 version for importing into our new environment. And I tried a lot of things. Really. I'm not going to go into details but to sum it up I had to sit down and do kind of manual replacements of the swedish specific characters in the database export. Lucky me this blog is in english.

Anyway, I had ssh access to the Trafficante server and could use this connection to export the database from the DreamHost MySQL server by issueing:

 
mysqldump --default-character-set=UTF8 -u [user] -p -h [host].dreamhosters.com [database] -r [output].sql
 

This resulted in a database dump later used for importing into the new MySQL instance.

Setting up the Amazon EC2 Instance

To make your Amazon life easier start with installing the Firefox plugins Elasticfox and S3 Organizer. These tools give you a GUI view of the somewhat more complex underlying details of Amazon.

Amazons' cloud solution is based on a user account. There are no start-up fees, you only pay for what you use. So if you still haven't got an account, sign-up now. The rest of this article assumes you have signed up and set up accounts for Amazon EC2, Amazon S3 and Amazon EBS. Start by reading the EC2 Getting Started Guide and don't stop reading until you have created an ssh key pair, a security group and have a server instance running.

Now you should also have a private key and a certificate in your ${EC2_HOME} (preferrably ${HOME}/.ec2/) folder of your local computer. These two files should also be uploaded to your instance /mnt folder to make it possible to bundle and upload your images to your S3 "bucket(s)" (buckets are one level deep folders/directories). Make sure you put them in the /mnt folder to prevent them from being bundled with the image and thereby increasing the risk of having your identity revealed.

Bundling, by the way, should be done when you've made changes to your server instance that you don't want to loose. The result of a bundling and uploading is an image with a specific Amazon Image Id (AMI ID) reflecting the state of your server instance at the moment of bundling. You also have to register the image with the manifest file created for you. If you do that you could then safely close down your instance and have it up and running again by issuing the ec2-run-instances command with the AMI ID as parameter. A way to save money.

Bundling and uploading was made easier for me as I used two scripts created by my colleague Ulrik. I put these scripts in the /root folder.

bundle-myself.sh:
 
#!/bin/sh
prg=`basename $0`
dest=/mnt
image=blog-image
 
USAGE="Usage: $prg <datetime>
 
Example:
% $prg 20090415-1000
Bundling into $dest/$image-20090415-1000.manifest.xml"
 
if [ $# -ne 1 ]; then
  echo "$USAGE"
  exit 1
fi
date=$1
echo "Bundling into $dest/$image-$date.manifest.xml"
ec2-bundle-vol --debug -k /mnt/pk-PRIVATE_KEY.pem -c /mnt/cert-CERTIFICATE.pem -u USER_ACCOUNT_ID -d $dest -p $image-$dat
e -r i386
 

upload-myself.sh:
 
#!/bin/sh
prg=`basename $0`
dest=/mnt
image=blog-image
 
USAGE="Usage: $prg <date>
 
Example:
% $prg 20090415-1000
Uploading bundled AMI parts to https://s3.amazonaws.com:443/[S3_BUCKET] ...
Uploaded blog-image-20090415-1000.part.00 to https://s3.amazonaws.com:443/[S3_BUCKET]/blog-image-20090415-1000.part.00.
..."
 
if [ $# -ne 1 ]; then
  echo "$USAGE"
  exit 1
fi
date=$1
ec2-upload-bundle --retry -m $dest/$image-$date.manifest.xml -b blog-images -a [PUBLIC_KEY] -s [PRIVATE_KEY]
 

Another thing I needed was a static IP address for my instance. To solve this I used Amazons Elastic IP service which doesn't cost anything as long as the generated IP address is in use. So, I generated an Elastic IP address and assigned it to the instance.

Installing Necessary Software

The blog require Apache2, Php5, MySQL and the Php GD graphics package, to start with. This was easily installed using Advanced Packaging Tool, apt-get:

 
# sudo apt-get update
# sudo apt-get install apache2 php5 mysql-server php5-mysql php5-gd
 

Setting up the Database for EBS

EBS volumes are, as you might know, more persistent than EC2 instances and will survive if instances go down. Volumes are attached to the instance and could as well be detached and re-attached to new instances. As EBS volumes are not 100% reliable you should also take snapshots of your database data and have it uploaded to an S3 bucket. All this is nicely described in Eric Hammonds' EBS and MySQL tutorial, so I urge you to read it if you haven't yet.

The only thing I sidestepped in the tutorial was the MySQL configuration to point out the data and log directories. Instead I used symbolic links:

 
# ln -s /vol/lib/mysql/ /var/lib/mysql
# ln -s /vol/log/mysql/ /var/log/mysql
 

The tutorial also walks you through automatic snapshots which requires the EC2 command line tools. Here is a nice tutorial on setting these tools up on Ubuntu.

I had to tweak the environment variable settings in ec2-snapshot-xfs-mysql.pl script from

BEGIN {
  $ENV{PATH} = '/bin:/usr/bin:/usr/local/bin';
}

to

BEGIN {
  $ENV{PATH} = '/bin:/usr/bin:/usr/local/bin:/usr/sbin:/root/ec2-api-tools/bin';
  $ENV{EC2_CERT} = '/mnt/cert-CERTIFICATE.pem';
  $ENV{EC2_PRIVATE_KEY} = '/mnt/pk-PRIVATE_KEY.pem';
  $ENV{EC2_HOME} = '/root/ec2-api-tools';
  $ENV{HOME} = '/root';
  $ENV{JAVA_HOME} = '/usr/lib/jvm/java-1.5.0-sun';
}

to get the snapshotting to fully work. I also wrapped the ec2-snapshot-xfs-mysql.pl script for a more convenient way to handle the input:

/root/snapshot-myself.sh

#!/bin/sh
mountpoint=/vol
volume_id=[VOLUME_ID]

su -c "/root/ec2-snapshot-xfs-mysql.pl ${mountpoint} ${volume_id} > /vol/cron.log"

To automatize the snapshotting I set up a simple crontab job by issueing:

 
# crontab -e
 

and then edited it like this

05 */6 * * * sh /root/snapshot-myself.sh

which makes the job run five minutes past every six hours, every day.

I also had to set the timezone info to apply to our local time. The easiest way I've found out of setting this for a Linux server is like this:

 
# ln -sf /usr/share/zoneinfo/Europe/Stockholm /etc/localtime
 

Configuring MySQL

To configure MySQL for UTF-8 content make sure the /etc/mysql/my.cnf file contains the following:

[client]
default-character-set=utf8

[mysqld]
#
# * Basic Settings
#
init_connect='SET collation_connection = utf8_general_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_general_ci
default-character-set=utf8
skip-character-set-client-handshake

Now, if you haven't yet, go on and start your MySQL server. You need to create your wordpress database. It's as simple as:

 
# mysql -u [user] -p
> CREATE DATABASE [database_name]
 

To create the user with which WordPress will access the database:

 
> GRANT ALL ON [database_name].* TO '[user]'@'localhost' IDENTIFIED BY '[password]';
 

And to import your database dump into the newly created database:

 
# mysql -u [user] -p [database_name] < [output].sql
 

You shouldn't need to specify any character set or collation when importing thanks to the configuration done in /etc/mysql/my.cnf.

In order for us to stick with our domain name and have no downtime we needed to use the instances IP address until we were finished testing. This means I had to replace two properties in the database to prevent the blog from redirecting to the original host; the 'siteurl' and 'blogurl' properties residing in the option_value column of the wp_options table. The commands

 
# mysql -u [user] -p
> UPDATE wp_options SET option_value='http://[ip_address]/wordpress/' WHERE option_name='siteurl'
> UPDATE wp_options SET option_value='http://[ip_address]/wordpress/' WHERE option_name='blogurl'
 

fixed this. As soon as the DNS info was propagated throughout the Internet this was reset to "http://blog.jayway.com/wordpress" from within the WordPress admin interface.

Apache Configuration

Our WordPress installation makes use of permalinks and redirects to FeedBurner for RSS statistics. This is based on .htaccess redirects (at least that's how we've solved it) which in turn is dependent on the Apache redirect module. So, in order to make this to work I had to enable that.

 
# sudo a2enmod rewrite
 

Easy as that! One more thing on redirections. I had to configure Apache to allow redirection directives specified in the .htaccess file by changing the /etc/apache2/sites-available/default configuration of the element from

 
	<Directory /var/www/>
		Options Indexes FollowSymLinks MultiViews
		AllowOverride None
		Order allow,deny
		allow from all
	</Directory>
 

to

 
	<Directory /var/www/>
		Options Indexes FollowSymLinks MultiViews
		AllowOverride All <----------
		Order allow,deny
		allow from all
	</Directory>
 

Securing the WordPress Admin Interface

Again, I choose to refer to detailed and well written sources. Here are the two articles I followed to create a self-signed certificate and to configure Apache to use it.

• Creating a self-signed certificate
• Configuring Apache for SSL on Ubuntu

To enable https access from the "outside world" I also edited the group policy to open up port 443. This could easily be done via the Elasticfox Firefox plugin mentioned earlier.

Installing and Configuring WordPress

The default location for Apache httpd server for reading content is /var/www/. If you, as I did, have an existing WordPress installation you can simply copy this to the new server. I ended up having it located at /var/www/wordpress/ on the Amazon instance.

Once the database connection properties in the wp-config.php wordpress configuration file was adapted to the new environment I was able to access the WordPress installation via a browser and the Amazon instances' IP address. Sweet!

And as we have our WordPress installation in a separate "/wordpress" folder instead of in the document root of Apache we need to specify, in WordPress admin settings, that the 'Site URL' should be "${host}/wordpress" but the 'Blog URL' should simply say "${host}" to prevent the resulting URL from containing "/wordpress/...". This also requires that we move the .htaccess and index.php files from within our "/wordpress" folder and up one level to "/var/www/", in our case.

In the index.php file, make sure to change from

 
/** Loads the WordPress Environment and Template */
require('./wp-blog-header.php');
 

to

 
/** Loads the WordPress Environment and Template */
require('./wordpress/wp-blog-header.php');
 

If or when specifying the permalinks, WordPress must have write access to the .htaccess file, so 'chmod' that file.

 
# chmod 755 .htaccess
 

Otherwise it will complain and simply print out the .htaccess modification it tried to modify with and you are left to change it manually, instead.

Sending Mail from WordPress

Mailing is used in WordPress when generating new passwords to users, among others. We also use it to send data backups to the administator of the blog.

Setting up a mail server isn't a trivial task. I gave it a thought but quickly put it aside in favour of the simple WordPress plugin WP Mail SMTP as I realized I could make use of our existing corporate mail server. It was a breeze to configure the plugin with the necessary mail user credentials.

So, here we are. Sailing these fluffy clouds.

I used the Jungle Disk software to provide the mapping from the S3 web service interface to a network drive. You give Jungle Disk your S3 account information, and it provides a network disk which you can use as the backup device. I first tried my existing backup software (Synchronize! Pro X) and it happily began creating directories and copying files. I then tried the backup feature built into Jungle Disk, and it works fine too. It's not fast compared to a local FireWire disk. It's limited to your upload bandwidth, which often is substantially smaller than the download bandwidth.

Anyway, it has been tugging along today storing 5 GB of by digital photos. It feels pretty good to know that my photos are now stored encrypted and with 99.99% availability somewhere far far away, costing me only 75 cents per month after the initial $2 for the transfer.