The Deployment Frustration

For the last while I have been involved with various projects in the general area of Continuous Deployment. Since we work within the Windows and .NET realm, this involves problems that are unnecessarily hard to deal with. Luckily, my colleagues in the Architecture Team at Jayway have brilliant ideas on how to solve this problem in a reliable and standardized fashion.

There are quite a few things to consider when building software, as compiling all the components that need it, and sorting out all dependencies and collecting all output files and setting up everything in such a fashion that the system can be reliably installed for use. There used to be the issue of client installation, which was a tortuous nightmare straight out of some Absinth-laced naturalistic drama from the 19th century due to the unpredictability of the environment in which you were putting your software with unknown shared libraries of unknown versions all conspiring to impede the successful execution of your software. These days only lesser nightmares, but nightmares still, remain.

Versioning

When you build your system it is nice to have a version number of the product and have it propagated to all version resources and binaries in the system, in order to be able to easily confirm that the binaries on your various servers are actually of the intended version.

There  are two main solutions to this problem. One is to keep the source tree updated with the latest version built, which means auto-updating the AssemblyInfo.cs files  and comitting those updated versions back to the version contol system.  This is counter-intuitive and feels wrong as any changes to source should require human interaction, however there are other instances of auto-generated code, but those changes are usually made in code where developers are warned not to make changes, and not in code where humans do –and should – make changes, such as AssemblyInfo.cs files.

If you still, which many do, choose to go  with this approach, there are modules available for most build systems to help you with this, but it still requires a surprisingly large amount of adaptation when dealing with TeamBuild in TFS, for instance.

The other approach is much cleaner in its delimitation between developer and build system, and uses settings and build system controlled versioning to generate the assemblyInfo.cs files from scratch during build. The draw-back is that developers don’t have a checked-in version of each AssemblyInfo.cs that is the same as the one currently built. This may or may not be a problem.

Packaging

With MSDeploy, Microsoft have addressed packaging websites in a way that means you get a deployment mechanism directly to an IIS server using SSL or to a configurable package that can be imported manually on a remote IIS and configured in a very friendly wizard. This is awesome, albeit with a cumbersome MS style verbose command-line tool, and the fact that if you need to also package a Windows Service or anything unrelated to the IIS, you are out of luck.

Deployment Pt I – COPYING A FILE

In the simple text-book case, aside from the dream textbook case in which MSDeploy can directly access the remote web server, you just copy the file to another server using an NTFS file share. However, in reality, though, you will have an ops man with a large firewall in between your integration server and your target servers, probably not even on the same domain. There are options for file shares where you can configure the same username and passwords on both servers so that compatibility across the network will exist, but the NTLM hashes are notoriously weak so it is undesirable to be sending them across the wire. You would be better off initiating a HTTPS download from the web server node which you are intending to populate to get the file down.

DEPLOYMENT PT II – COMPELLING THE REMOTE SERVER TO CONFIGURE ITSELF

If you had access to an SSH server in Windows, which you can have, you’ll just have to buy it separately, you could easily solve this point and the previous paragraph by SCP:ing the file onto the remote server( s) and then executing the necessary scripts to extract files out of the compressed archive, copy files into the correct folders, install windows services, apply registry patches, run SQL scripts et cetera. But, if you are running a vanilla Windows Server installation with the aforementioned realistic firewall restrictions, you will have problems.

But what are we doing here?

Let us take a step back for a moment and consider what we want to do here. We want to repeatably, automatically, install a server, yes most likely a VM, and apply configuration changes to it – alternatively just applying changes to a running virtual server without disrupting the current running state in an undue way.

Wait a minute – isn’t that a solved problem? Why – yes it is. There are two popular ways of doing this – Puppet and Chef. Puppet is a declarative way of describing a state on a server. A puppet agent on the server instance then is in charge of “making it so”,  as in modifying the current state of the server until it matches the desired state regardless of what that means in terms of  copying files, installing services/daemons (yes, it is a tool from the Dark Side) or changing permission settings, while Chef is a more imperative system that allows you to state outright the various steps that need to be taken, you create a recipe – get it?

So what kind of voodoo are you suggesting?

Well, a repeatable, standardized way of setting up one or more virtual machines to run a system consisting of one or more websites and a few windows services backed by a SQL Server, and doing this for test or production could be set up in the following way.

Server bootstrap

Bootstrap a Windows Server Core to install IIS, .NET and ruby using Windows Pre-Install Environment.

System basics

To set up the basics for a system, be it a single server or a cluster, you will need the following services (so, yes this will not be necessary on all  VMs if you are setting up multiple servers):

  1. Install a DNS Server
    Multiple services will need a DNS server to find itself and its components
  2. Install a DHCP Server, if necessary.
    When you are provisioning virtual machines using Puppet, you can use  DHCP to assign IPs, but for example, Amazon VM Images, AMIs come preconfigured with an IP-address so no such provisioning is necessary.
  3. Install Puppet
    This also gives you a CA, either with a default self-signed certificate, or if you like you can configure it to use a proper certificate  that you have bought. This CA module is thereafter used to secure the entire system, but unless you want to have others trust your certificate and that it looks fancy in a certificate manager, there is no significant loss of security to use your self-signed certifiicate within your own system as long as you pay attention.
  4. Install DNSSEC
    Leverage the CA to apply security to the DNS-requests and prevent DNS poisoning.
  5. Install an LDAP-server for user account management –or configure the system to use an existing LDAP-server such as an Active Directory Domain Controller. Using LDAP provides you with kerberos tokens which adds a large amount of security to the communication between servers. You can forego that and just use some entity that provides you with user authentication, but Kerberos is a Good Thing™.
  6. Install nginx as a reverse proxy, directing traffic to the various IIS:es involved providing load-balancing and SSL-termination.
    This helps ensuring that SSL is used reliably while relieving the website code of managing URL rewrites and dealing with other things than their actual domain purpose. It also helps manage DOS-attacks.

Again, the above settings, up until you have puppet up and running are made in a batch script called from the bootstrap
, and the rest are managed using Puppet manifests. There are no manual steps involved in the process.

Node basics

On the nodes where your websites or windows services will run, you will only need a puppet agent installed which will be part of the bootstrap, and then puppet will be able to set up the computer  – with one important caveat – you will need to have access to the various media, as in, the data has to be fetched onto the node computers. Here there are a couple of options, but we think – for now – that a package manager called Chocolatey can be of service.

Conclusion

While  building a large systems with many external dependencies or stakeholders, it would behoove you to set up various test environments, probably different ones for different stakeholders, thereby giving yourself a headache making sure the various test- and acceptance environments are running the correct versions of the software and that all components are correctly configured for the environment they are in. This, surely, is not work fit for man, it should be delegated to a computer. The concept of automating system install and deployment using manifests and scripts, Puppet and Chocolatey may seem complicated, but remember that what you get is a predictable way to set up your system in any environment. That is, once you have tweaked your Puppet manifests, you can easily set up a new development environment or runtime environment and update existing ones.  The idea here is to be able to, as developers, assume responsibility for the installability of our software where it matters – in production.

There are of course great practical concerns here that we will explore. For now, these are the seneral ideas. Here are some relevant links:

http://puppetlabs.com/ 
Download software and a VM for playing around with puppet.

http://www.microsoft.com/downloads/details.aspx?familyid=696DD665-9F76-4177-A811-39C26D3B3B34&displaylang=en
Windows Automatic Installation Kit – featuring Windows PE which will allow you to bootstrap a windows server.

http://theforeman.org/
Integrates with Puppet and acts as a web front-end to it.

This Post Has 3 Comments

  1. Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.

  2. I’m currently using MSDeploy quite heavily and I’m evaluating switching to Puppet/Chocolatey for deployments. Have you found they are working well?

    Also, it is very, very, very cumbersome, but you can package a standard windows service with MSDeploy. This is a terrifyingly complicated command we run from powershell to transfer and install a windows service on a remote machine:

    cmd.exe /C $(“C:MsDeploymsdeploy.exe -verb:sync -source:dirPath=C:localserver -dest:dirPath=C:PathOnDestination,computerName=https://destination.com:8172/MSDEPLOYAGENTSERVICE -preSync:runCommand="net stop DestinationService & timeout 10 & %windir%Microsoft.NETFramework64v4.0.30319InstallUtil.exe /u C:PathOnDestinationservice.exe“,waitInterval=10000,waitAttempts=10 -postSync:runCommand="%windir%Microsoft.NETFramework64v4.0.30319InstallUtil.exe C:PathOnDestinationservice.exe & net start DestinationService“,waitInterval=10000,waitAttempts=10” -f $wsComputerName);

  3. I really think this specific posting , “The
    Deployment Frustration – Jayway”, quite pleasurable not to mention the
    blog post ended up being a great read. Thanks for the post,Trena

Leave a Reply

Close Menu