Layered Cloudformation stacks

Defining information service infrastructures using AWS Cloudformation can sometimes feel like a double-edged sword. On the one hand it provides us with a high degree of control regarding consistency, on the other hand once a stack is deployed it can be challenging to modify and update.

Cloudformation does provide us with stack updating capabilities. In practice, however, updating a stack in situ comes with a large set of caveats. Some stack resources can be updated cleanly, some conditionally, and some resources do not support updating at all. So how should we manage information services and server infrastructure in order to encourage development and growth while reducing risk of technical debt? One way is to use a technique we call stack layering.

Modularity and abstraction layering are widely used concepts in software engineering. As are the mutually complementary design strategies we call “high cohesion” and “loose coupling”. Let’s look at how we could employ all of these with Cloudformation and what effects their usage might yield. We’ll use two templates, base and top: Base will describe an AWS VPC with no dependencies, and top will describe a PostgreSQL instance that depends on a VPC. We’ll use the VPC template described in my previous post as base, and bring in a new top layer:

A few notes on the above:

    We need a few input params, a few of which we get from base layer resources
    We’re allowing direct internet access for easy demoing purposes. Never do this for a production system

Let’s call the Postgres template “top.template” and the VPC one “base.template”. Now we’ll instantiate them, bottom up, using the AWS CLI. (See this post for an easy way to install and manage AWS CLI installations)

This creates the VPC and takes a couple of minutes to complete. Once it’s done we can check the resource details by running:

From the list we get out of the above command we can pick out the “PhysicalResourceId” for each of the resources we need to instantiate the top layer. Let’s note those down and move on to instantiating the top layer. We’ll do it essentially the same way as we did for the base layer, with the addition of a few input parameters. The way the AWS CLI takes input params is a bit ugly, so let’s do a multi-line command. Note that every line ending in “\” is immediately followed by a return keypress.

Starting up this Postgres instance will take 5-10 minutes, we can monitor the creation with:

Which will eventually yield something like:

We should now be able to log in like this:

And there we are! We’ve successfully layered one stack on top of another. The immediate gains from using this approach are, amongst other:

    Absolute consistency. Because we are not relying on the standard VPC, our Postgres deployment is completely unaffected by any potential changes to the standard VPC.
    Repeatability. This method will work on any common AWS account. We can also adjust any of the templates used and expect consistent results over deploys.

As to shutting down these two stacks, we want to do it top-down. Should we tear down the VPC before postgres we’re sure to run in to some interesting problems.

2 Comments

  1. Jan Kronquist

    You may also want to checkout the stackDependency custom resource so you can get rid of the command line parameters:

    http://blog.jayway.com/2015/07/04/extending-cloudformation-with-lambda-backed-custom-resources/

Leave a Reply