Simple Clustering with Docker Swarm and Nginx

docker-swarm

Bringing up your own cluster has never been easier. The recent 1.0 release of Docker Swarm signals that the Docker team feel that Swarm is ready for production.

I've been running a bunch of applications on Docker for a while now, but I have managed the containers on the single machine level instead of as a cluster. With the release of Swarm 1.0, I believe it is time to start clustering my machines.

Spinning Up the Swarm

How to spin up a Swarm for development is described well in the Docker documentation and I'm not going to describe it in depth here. I'll settle for the commands and extra documentation when I feel that it may be called for.

I'm using the Swarm for development with VirtualBox here, but it is simple to substitute any of the supported docker-machine providers.

Create a Token

Create a token with the Docker Hub discovery service. When running this in production you should probably setup an alternate discovery backend to avoid the external dependency.

Create a Swarm Manager

The swarm manager will be used to control the swarm. It should be protected from access from anyone but you. I'll simulate this here by setting --engine-label public=no. This is just a tag and you would have to make sure that you setup the manager protected from public access. It is possible to use multiple labels to tag the engine with all the qualities of this machine.

Create a Publicly Accessible Machine

In this demo I'm only spinning up another VirtualBox machine and I'm giving it the --engine-label public=yes to allow me to discover this box in the swarm.

Create a Couple of Additional Non-Public Machines

Here I start a couple of machines with an additional --engine-label. One with model=high-memory and one with model=large-disk

List the Swarm

the-swarm-machines

Connect to the Swarm

Configure the docker client to connect to it.

Starting the Containers

Now it is time to start the containers. The plan is to bring up two database containers, Postgres and Redis, two counter web-services, and one proxy to front the whole cluster, like this.

the-cluster

Alright, let's start some containers!

Databases

According to the picture above I want to put the Redis container on the machine named backend1, but I don't want to address it by name, instead I'm going to target it by its labels.

I also want to start a Postgres container on a machine with a constraint:model==large-disk.

Starting Redis

In this case, constraint:public!=yes is not needed but I like to add it to avoid mistakes.

Starting Postgres

If this was not a VirtualBox machine I would also mount a volume, -v /var/pgdata:/var/lib/postgresql/data, for the database, but this does not work with VirtualBox.

OK, let's see what we have.

Nice, two running databases on the designated machines.

Starting the Reverse Proxy

nginx

Nginx is one of my favorite building blocks when it comes to building reliable web services. Nginx provides an official Docker image, but in this case, when I want to automatically configure Nginx when new containers are started, I prefer to use an alternative image called nginx-proxy.

A container started from the nginx-proxy image, listens to events generated by the docker engine. The engine generates events for all kinds of events but all we care about here is when a container is started and stopped. If you want to see what events are triggered from the CLI, run docker events in one terminal and start and stop a few containers in another.

When nginx-proxy receives an event that a container has been started it checks if the container has any ports EXPOSEd, if it does it also checks for a VIRTUAL_HOST environment variable. If both these conditions are fulfilled nginx-proxy re-configures its Nginx server and reloads the configuration.

When you now access the VIRTUAL_HOST, Nginx proxies the connection to your web service. Cool!

Naturally, you will have to configure your DNS to point to your Nginx server. The easiest way to do this is to configure all your services to point to it with a wildcard record. Something like this:

In this case, we are using VirtualBox and we can settle for changing the /etc/hosts file with the IP-number of our frontend.

What is even more cool is that events works with Swarm and it is possible to use the nginx-proxy to listen to services that are started on different machines. All we have to do is configure it correctly.

Starting Nginx-Proxy

nginx-proxy is started with configuration read from the docker client environment variables. All the environments variables were automatically configured when you configured the docker client to access the Swarm, above.

OK, we are almost done. Now it is time to start the web services.

Starting Web Services

As a web service I'm going to use a simple counter image since it can use both Postgres and Redis as backend. I want to start the web services on the same server as the databases since this allows me to use --link to connect to the container and it will speed up the data access. To do this I can use an affinity constraint: --env affinity:container==*redis*.

The affinity constraint is not really necessary since affinity constraints are automatically generated by Swarm when --link is present as you can see when we start the postgres-counter.

Browse to http://redis-counter.docker or http:/postgres-counter.docker and you should see your services up and running.

redis-counter

postgres-counter

Summary

Here's an illustration of our current setup:

swarm-complete

And here is a listing of all the containers on their respective machines.

May the Swarm be with you! :D

10 Comments

  1. Jolubax

    Hi,
    Nice tutorial. Although I’m struggling getting it to work wit containers on different machines.
    You wrote “it is possible to use the nginx-proxy to listen to services that are started on different machines”
    What should be done so that nginx-proxy forwards the http queries to containers running on different machines?

    Thanks for your help!
    Jolubax

  2. Anders Janmyr

    @Jolubax,Nginx should configure itself automatically for the servers that you are running in the swarm.
    If you need to check out the configuration you can exec into the container with


    docker exec -it nginx bash
    # Inside the container you can check what the configuration contains
    cat /etc/nginx/conf.d/default.conf

    You can also check the nginx-log with

    docker logs nginx

  3. Michal

    Hi, great post.

    Isn’t it an error on the first picture, under “List the Swarm”? swarm-aster is labeled with public=yes on the pic but before in config files public=no was set, correct ?

  4. Jian

    docker run -d –name nginx \
    -v $DOCKER_CERT_PATH:$DOCKER_CERT_PATH \
    -p “80:80” \
    –env constraint:public==yes \
    –env DOCKER_HOST \
    –env DOCKER_CERT_PATH \
    –env DOCKER_TLS_VERIFY \
    jwilder/nginx-proxy
    Run this command through docker-machine will not mount the DOCKER_CERT_PATH because this dir actually not exist on target node (frontend).

  5. Hi Anders ,

    We are not currently using docker swarm but we are using nginx in frontend with few backend containers , the frontend nginx has already a customized configuration setup by us . Now we just was to add few nodes dynamically using docker-compose scaling ability .

    What’s the easiest way to add nodes dynamically into our custom nginx configuration ?

  6. Puneet Pandey

    I get this error when I try to run Nginx container:

    “Error creating generator: Unable to create docker client: TLS verification was requested, but CA cert does not exist”

    • Anders Janmyr

      Puneet, make sure you have the correct path to the certificate in DOCKER_CERT_PATH. The path I use is only valid for my local machine.


      export DOCKER_CERT_PATH="/Users/andersjanmyr/.docker/machine/machines/swarm-master"

Leave a Reply