I’ve been searching for a way to host Jenkins in a Docker container and inside this container also be able to run integration tests inside other Docker containers. This approach is called Docker in Docker (or DIND) and requires a bit of Docker knowledge and tweaks to get working properly. To understand this blog you’ll need some basic knowledge about Docker and Supervisord already.

Building a Jenkins image with a Docker daemon

First of all Jenkins have an official Jenkins Docker Image that we’d like to use. This image defines an EntryPoint that points to a script starting Jenkins. However the image doesn’t contain Docker so we have to create our own Dockerfile that inherits from this image and adds Docker. This can be achieved like this:

FROM jenkins
MAINTAINER johan.haleby@jayway.com

# Switch user to root so that we can install apps (jenkins image switches to user "jenkins" in the end)
USER root

# Install Docker prerequisites
RUN apt-get update -qq && apt-get install -qqy \
    apt-transport-https \
    apparmor \
    ca-certificates \
	lxc \
	supervisor

# Install Docker from Docker Inc. repositories.
RUN echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list \
  && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 \
  && apt-get update -qq \
  && apt-get install -qqy lxc-docker

# Switch user back to Jenkins
USER jenkins

This will install the latest version of Docker when building the image, for example:

docker build -t johanhaleby/jenkins .

However the Docker daemon is not started automatically when we start a container based on this image. This is because Docker ignores initialization daemons such as upstart, systemd etc so we need to handle this ourselves. Since Docker is designed to run one process per container we need to use a process control system. Supervisor is very popular for this use case in the Docker world and this is what we’re going to use as well. This means that we need to update our Dockerfile to also install supervisor (we’ll see a complete Dockerfile later so don’t worry). Supervisor needs a configuration file in order to know what processes to provision. This is an example of a working supervisor config file that starts Jenkins and the Docker daemon:

[supervisord]
user=root
nodaemon=true

[program:docker]
user=root
autostart=true
autorestart=true
command=/usr/bin/docker -d
redirect_stderr=true
stdout_logfile=/var/log/docker/%(program_name)s.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10

[program:jenkins]
user=jenkins
autostart=true
autorestart=true
command=/usr/local/bin/jenkins.sh
redirect_stderr=true
stdout_logfile=/var/log/jenkins/%(program_name)s.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10
environment = JENKINS_HOME="/var/jenkins_home",HOME="/var/jenkins_home",USER="jenkins",COPY_REFERENCE_FILE_LOG="/var/log/copy_reference_file.log"

To start supervisor we must copy the supervisord.conf from our file system into Docker and add a command to our Dockerfile that starts it:

# Create log folder for supervisor, jenkins and docker
RUN mkdir -p /var/log/supervisor
RUN mkdir -p /var/log/docker
RUN mkdir -p /var/log/jenkins
# Copy the supervisor.conf file into Docker
COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
..
# Start supervisord when running the container
CMD /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

Since we’re starting supervisord when the container is started the Jenkins EntryPoint (that previously ran /usr/local/bin/jenkins.sh) is now ignored. This is why we also start Jenkins from supervisord.

It’s very important to set the environment parameter for Jenkins. Supervisor doesn’t include environment variables declared for another user (jenkins in this case) so we need to pass them along ourselves. The most important part is probably that we need to set JENKINS_HOME to /var/jenkins_home otherwise the volume defined by the Jenkins image will not work as expected and Jenkins will assume the home folder to be /var/jenkins_home/.jenkins. Also note that Jenkins must be executed as user “jenkins” and not root.

Here’s an example of the Dockerfile so far:

FROM jenkins
MAINTAINER johan.haleby@jayway.com

# Switch user to root so that we can install apps (jenkins image switches to user "jenkins" in the end)
USER root

# Install Docker prerequisites
RUN apt-get update -qq && apt-get install -qqy \
    apt-transport-https \
    ca-certificates \
	lxc \
	supervisor

# Create log folder for supervisor, jenkins and docker
RUN mkdir -p /var/log/supervisor
RUN mkdir -p /var/log/docker
RUN mkdir -p /var/log/jenkins

COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf

# Install Docker from Docker Inc. repositories.
RUN echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list \
  && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 \
  && apt-get update -qq \
  && apt-get install -qqy lxc-docker

CMD /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

Starting the container

Alright, we’ve managed to build an image (docker build -t johanhaleby/jenkins .) but there are some things we need to think about when starting a container based on our image. First of all if we’re going to use Docker in Docker we need run our container in so called privileged mode. This is done by using the --privileged flag when starting the container. Another thing we may need to add is another DNS provider than our default one. This is because it’s common to have /etc/resolv.conf point to 127.0.0.1 on your host but this doesn’t work from the Docker container. A workaround is to use the --dns flag and point to, for example, Googles public DNS service using 8.8.8.8. Here’s an example:

$ docker run --privileged --dns 8.8.8.8 -d --name jenkins -v /home/johanhaleby/jenkins_data:/var/jenkins_home -p 8080:8080 -u root johanhaleby/jenkins

This particular script mounts the Jenkins volume (/var/jenkins_home) into folder /home/johanhaleby/jenkins_data so that the data is persisted accross restarts. Note that you could also use the data volume container pattern if you need to make the data portable between different servers. We also start the container using the root user but this is not something that’s recommended in a production environment. Another solution would be to install sudo and then configure the jenkins user to be “sudoer” (preferably with a password) but this is out of scope for this blog post.

Now that the container is started let’s login:

docker exec -it jenkins bash

If you try running something like:

docker run -it ubuntu bash

you’ll run into the following error:

FATA[0006] Error response from daemon: Cannot start container df7d5f605f5c1b6750614c6a04889e34aa9b96a4de98dcfc91b8f38f9d445aad: failed to find the cgroup root

After some help from stackoverflow things started to clear up. We cannot only start the Docker daemon process as we did in our supervisord config (command=/usr/bin/docker -d). The Docker daemon process requires other processes to be running (such as cgroup) prior to starting it. Luckily there’s a project called DIND that has solved this issue for us. DIND contains a magic wrapper script that takes care of all the prerequisites needed to start Docker inside another Docker container. We must download this script, copy it to our image from our Dockerfile and update our supervisor config to call this script instead of /usr/bin/docker -d.

Once this is done we are able to run Docker images from within the Jenkins container. But there’s still one more caveat. Jenkins itself is running as user jenkins and it doesn’t have access to use Docker without using sudo (which is not installed). Thus we need to allow the jenkins user to run Docker without the need for sudo. We can do this by adding the jenkins user to the docker group:

RUN gpasswd -a jenkins docker

Once this is done Jenkins should be able to run Docker images from inside jobs while Jenkins itself is also running in Docker. How to actually create Jenkins jobs taking advantage of this is the subject of another blog post.

Stopping and restarting

It’s obviously nice to be able to stop the container and restart it with everything working as it did prior to the restart. However when starting a stopped Jenkins container using:

docker start jenkins

you can run into errors such as:

Service 'my_jenkins_job' failed to build: Can't set task name /dev/mapper/docker-253:4-408036-pool

when running a Docker job inside Jenkins. The workaround is to run:

dmsetup mknodes

before starting Docker. dmsetup manages logical devices that use the device-mapper driver and this command ensures that all nodes in /dev/mapper correspond to mapped devices currently loaded by the device-mapper kernel driver. This is not something that is included in the wrapdocker script from DIND so what I’ve done is to update it by adding dmsetup mknodes as the first command in the script.

After doing this and rebuilding our Docker image it’s possible to restart the container for example after a system reboot.

Update 2015-03-16: I committed a pull request to the DIND project (which was accepted) that adds dmsetup mknodes to the wrapdocker script so this workaround should no longer be necessary.

What we lose

What we lose with this approach is that we’re no longer able to pass in command-line arguments to Jenkins when starting a container (since we’re starting Jenkins from supervisor). For example running the official Jenkins image allows us to do things like:

docker run -p 8080:8080 jenkins -- 

I.e. we lose the ability to specify <jenkins_arguments> when starting the container. If this is required then the workaround is to append the arguments to the command in the supervisor config:

command=/usr/local/bin/jenkins.sh -- 

Summary

As we’ve seen there are a couple of obstacles we need to pass in order to get Docker in Docker working with Jenkins. Here’s a complete Dockerfile for reference:

FROM jenkins
MAINTAINER johan.haleby@jayway.com

# Switch user to root so that we can install apps (jenkins image switches to user "jenkins" in the end)
USER root

# Install Docker prerequisites
RUN apt-get update -qq && apt-get install -qqy \
    apt-transport-https \
    ca-certificates \
	lxc \
	supervisor

# Create log folder for supervisor, jenkins and docker
RUN mkdir -p /var/log/supervisor
RUN mkdir -p /var/log/docker
RUN mkdir -p /var/log/jenkins

COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf

# Install Docker from Docker Inc. repositories.
RUN echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list \
  && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 \
  && apt-get update -qq \
  && apt-get install -qqy lxc-docker

# Add jenkins user to the docker groups so that the jenkins user can run docker without sudo
RUN gpasswd -a jenkins docker

# Install the magic wrapper
ADD ./wrapdocker /usr/local/bin/wrapdocker
RUN chmod +x /usr/local/bin/wrapdocker

CMD /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

and here’s the supervisord config:

[supervisord]
user=root
nodaemon=true

[program:docker]
user=root
autostart=true
autorestart=true
command=/usr/local/bin/wrapdocker
redirect_stderr=true
stdout_logfile=/var/log/docker/%(program_name)s.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10

[program:jenkins]
user=jenkins
autostart=true
autorestart=true
command=/usr/local/bin/jenkins.sh
redirect_stderr=true
stdout_logfile=/var/log/jenkins/%(program_name)s.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10
environment = JENKINS_HOME="/var/jenkins_home",HOME="/var/jenkins_home",USER="jenkins",COPY_REFERENCE_FILE_LOG="/var/log/copy_reference_file.log"

To build the image run:

docker build -t johanhaleby/jenkins .

and to start it run:

$ docker run --privileged --dns 8.8.8.8 -d --name jenkins -v :/var/jenkins_home -p 8080:8080 -u root johanhaleby/jenkins

Todo

When restarting the Jenkins container (using docker run) all downloaded images are lost (don’t worry, they will be downloaded automatically by Docker so you won’t lose anything except that it takes longer run our Jenkins jobs that are using Docker for the first time). A better approach would be to mount the folder where Docker keeps its images as a volume (or data volume container if it needs to be portable).