Continuos Integration is the practice of integrating changes from many people as often as possible. Instead of merging changes once a month and spending time handling merge errors you try integrate every day, perhaps even every hour. Each integration is built and tested on a server. If there are build errors or test failures, you and your team will be notified right away.
This is the second part of the blog post I wrote about TDD in XCode. Make sure you’ve followed that tutorial before continuing here, or at least have a project of your own with OCUnit tests ready.
The server which we’ll use is called Hudson and is very popular in the Java community. It does not readily support XCode projects but it does allow shell commands, so we’ll use that to make our Calculator project compatible. We also want to fetch new versions of the source code as soon as it’s checked in. We’ll be a bit fancy and use Git as source control software. Here’s a summary of what we want to do:
- Download and install the Hudson continuos integration server
- Download, install and setup the Git source control software as well as the Git plugin for Hudson
- Download and install a script that converts OCUnit test results to JUnit format
- Setup a new job in Hudson that checks out our XCode project and builds and tests it
- Set up a notification and try it out
Downloading and Installing Hudson
In order to keep the instructions in this tutorial simple, we will have the source code repository and the build server itself on localhost. In real life you’ll probably want to setup a dedicated machine.
Head over to the Hudson web site and download your copy of the server. Follow their included installation instructions, it’s quite simple. Once you have Hudson up and running you can visit http://localhost:8080 to see the welcome screen. By the way, when you’re ready to install Hudson on a real remote server I recommend running it as a Servlet inside Apache Tomcat instead.
Downloading and Setting Up Git
Next we want to install Git. There are several ways to install it. The simplest might be to use the OS X installer, but I personally prefer MacPorts. Either way, go ahead and install it and make sure that you can use the “git” command from Terminal.app when you’re done.
In the Terminal, change directory to your XCode project folder and type:
git init
You now have a working source code repository in your project folder. Neat. Before you continue create a text file called .gitignore and let it contain:
build/* *.pbxuser *.mode1v3 .DS_Store test-reports/*
That’s a listing of the files and folders that should not be version controlled, since they are temporary or just used for XCode. Feel free to add any other files that you think only should be present locally. Now add all files (that aren’t ignored) using:
git add *
and finally commit them to a first version using:
git commit -m "Initial import"
Fetching the Script
We need to fetch one more thing, ocunit2junit.rb, which is a little Ruby script that I wrote. Since you now have git, you can also download it using git (or clone the repo, which is the correct term). Change directory to /tmp and then type like this:
git clone git://github.com/ciryon/OCUnit2JUnit.git
Copy the script from /tmp/OCUnit2JUnit/ anywhere, like to /usr/local/bin/. Make sure it’s executable by typing chmod +x ocunit2junit.rb.
The script will parse output from xcodebuild (the console command for building your XCode project) and look for output from OCUnit. The test results (success, failure, time passed etc) will be saved in the “test-reports” folder in the same XML format that JUnit uses. This also happens to be a format that Hudson understands.
Setting Up Hudson
Now head back to your Hudson welcome screen. Select Manage Hudson, Manage plugins and check “Hudson GIT plugin” under Available. Hudson will automatically download and install the plugin. Afterwards just restart Hudson.
We’re ready to create a job in Hudson for our project. From the main screen select “New job”, Build a free-style software project and call it “Calculator”. There are some settings which you should fill in:
-
Select to use git as source code management tool. The URL should be the file path to your project, like:
/Users/youruser/XCodeProjects/Calculator/
-
Select to poll SCM with Schedule:
* * * * *
That means it’ll check every minute if there are changes. If you want to check less often, do that.
-
Add new build step and select to execute shell. Use this command (make sure to point to the version of the iPhone SDK you want to use):
xcodebuild -target "UnitTests" -sdk /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.2.sdk/ -configuration "Debug" | /usr/local/bin/ocunit2junit.rb
-
Select to also publish a JUnit test result report and the Test report XMLs reside at
test-reports/*.xml
Trying it out
This is the moment of truth. Try to change your CalculatorTest.m so that one of the test cases fail. Type:
git -a commit CalculatorTest.m "Broke the test, on purpose"
and see what happens. If all goes well, and you have entered everything correctly, Hudson should automatically notice there’s a new version available, build it and try to run the test suite. Check the results and how it reports the test failure. Nice, eh? Fix the error and commit the test class again to see how Hudson reacts.
There are many settings, plugins and cool things to try out in Hudson. Go into Manage Hudson-> Configure system and setup your email settings, to allow Hudson to send mail. Then go back to configure your Calculator job and at the bottom select E-mail Notification. Enter your own email address and see how it works after you commits a file that breaks the build. If you don’t want a mail, you can get an instant message, or perhaps have the server play a loud annoying sound? Everyone in your team should be aware of when a build fails, so it can be fixed right away. This is one of the great advantages of having a continuos integration server.
this is so cool! works perfekt! great for bigger projects.
xcodebuild -target UnitTests -sdk /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.2.sdk/ -configuration Debug | /usr/local/bin/ocunit2junit.rb
> just the last is too much and cost me some minutes :)
thanks and best wishes from stockholm
Whops, fixed! :)
Thank you so much for your ocunit2junit.rb script. It works very well. I use it for another CI system than Hudson and he works also, as expected.
my 2 cents: the first word of your article has a typo :-)
Can this be done using Subversion instead of Git? I think the answer is yes but I want to confirm this.
Absolutely, it works in the same manner using Subversion instead.
Thanks Christian. I’ll try to play with this this weekend. I’m very new to all of this. I have Hudson installed but I’m using Subversion. Unit tests in Xcode is also new to me…….
Christian. Question on Hudson. I just about have everything working except in Hudson when I type in test-reports/*.xml I get a message from Hudson saying ‘test-reports/*.xml’ doesn’t match anything:even ‘test-reports’ doesn’t exist.
Any suggestions? I’m new at this whole thing.
This worked great for me.
Mark, try referencing your test reports as ‘**/test-reports/*.xml’ it seems ** = $WORKSPACE.
Just tried your suggestion Chad and still nothing for me. I’m running Hudson on a Ubuntu server and the test builds are being executed via a RemoteSlave node since Xcode only runs a Mac. I hope this helps..
Message in Hudson
‘**/test-reports/*.xml’ doesn’t match anything: ‘**’ exists but not ‘**/test-reports/*.xml’
Its strange it creates a folder named test-reports for me and I see xml, but when I make my test pass the build passes, but the last line in the output says it failed. Snippet from the Output
/Developer/Tools/RunPlatformUnitTests.include:446: note: Passed tests for architecture ‘i386’ (GC OFF)
/Developer/Tools/RunPlatformUnitTests.include:462: note: Completed tests for architectures ‘i386’
** BUILD SUCCEEDED **
Recording test results
No test report files were found. Configuration error?
Finished: FAILURE
The first time you enter test-reports/*.xml Hudson will complain that the folder doesn’t exist, but as long as your XCode build actually produces some unit test output the folder will be created and all will go well. I created a new project setup just yesterday so I’m pretty sure it works as described in the guide.
I’m very curious how it’s working for you under Linux with remote xcodebuild binary. Perhaps it’s something related to that which is causing your problem?
Thanks Christian. I’ll try this without using Ubuntu tonight when I return from work.
Do I have to install JUnit? Everything works just fine if I don’t click on “Publish JUnit test result report” test-reports/*.xml still generates errors either running Hudson locally java -jar hudson.war or Remote(Ubuntu) Any other suggestions? Here is the command I’m executing
cd ~/SourceControl/Calculator
xcodebuild -target “UnitTests” -sdk /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.3.sdk/ -configuration “Debug” | /usr/local/bin/ocunit2junit.rb
Got it to work. Using a remote build is a little tricky. I just had to figure out where the build was really being compiled from, so I wrote a little build script and attached it to my project. This is a really cool script. Thanks everyone and especially you Christian for putting together this tutorial. I’ve learned a lot.
Pingback: itemprop="name">Cocoa, Hudson CI and Xcode - go green | manicwave.com
Pingback: itemprop="name">Continuous iPhone Application Integration with my buddy Hudson and Xcode… and then some « just another developer
Christian
Have you tried to get Code Coverage working with your setup? I have Unit testing working fine, but following stuff I’ve read, I haven’t been able to get Code Coverage to work. I keep getting “Skipping Cobertura coverage report as build was not SUCCESS or better”
Thanks for a nice tutorial!
I’m having problems with my test reports, though. To be more precise, I’m not getting any reports at all.
I’ve got a simple project set up with a single test. It runs smoothly in Xcode, giving me the test results in the Build result panel. But nothing gets printed to the debugger console – is that a requirement for the Hudson report to work?
I’m getting the following error (same as Mark mentions above):
…
setenv YACC /Developer/usr/bin/yacc
/bin/sh -c /opt/local/var/db/hudson/jobs/VCC/workspace/build/VCC.build/Debug-iphoneos/LogicTests.build/Script-3D431BAF1271C6EA00CC02E7.sh
** BUILD SUCCEEDED **
Recording test results
No test report files were found. Configuration error?
Finished: FAILURE
I’ve Googled for a while now, but can’t find any way to get the tests output the results in any other way…
(I’m using the OCTest framework.)
Very grateful for any reply!
I think I got it working! I had set up the Hudson config according to another tutorial (very similar to yours), and the SDK setting in the build script was missing.
Got it working by setting it to 4.0, while 3.2 failed due to some weird bug (see http://stackoverflow.com/questions/1364456/xcode-3-2-in-snow-leopard-hangs-running-unit-tests).
Thanks!
Pingback: itemprop="name">iOS: Running Unit Tests with Code Coverage in Hudson « Software Noise
This was VERY useful, thanks so much. I also confirmed this works fine with SVN. I get an autobuild to run when the SVN archive is updated and the script correctly strips out my unit test errors/passes. Awesome job!
Hi,
This is really a perfect tutorial, especially the tool ocunit2junit.rb works like a charme. I use it directly in XCode to beautify my test-results. In XCode 4 it’s sometimes a little bit ugly to find the real reason for a unit test failure when working with OCMockObjects and Google Toolbox. You always have to leave your sources and go to the latest log. Now I just open a browser window and show the test results after converting your test-results with ant->junitreport.
One great improvement of your script would be to also add the next lines of the log to the error message until a line starts with “tests_errors[” When working with OCMock in these lines the methods are printed to the log, which were not called although they were expected to be called. Now I unfortunately only see the information, that two expected methods were not called.
But (thumbs up)! :)
Guido
Xcode definitely has its pluses and minuses like anything else. Online apps like the IT calculator seem to be more popular because they are easy to code and functional.
Pingback: itemprop="name">CI with Jenkins for iOS apps + build distribution via testflightapp tutorial | Shine Technologies Blog
I get error in installation itself . File not found lib. I am on OS X 10.7.4
pp$ gem install ocunit2junit
Successfully installed ocunit2junit-1.1
1 gem installed
Installing ri documentation for ocunit2junit-1.1…
File not found: lib
I have same problem from first comment, someone has the solution?
”
gem install ocunit2junit
Successfully installed ocunit2junit-1.1
1 gem installed
Installing ri documentation for ocunit2junit-1.1…
File not found: lib
“
Hi and sorry for the late response (I get a ping when someone comments to this thread but easy to miss)
I’ll investigate what’s causing this problem. Meanwhile perhaps you can install the gem from source instead? http://github.com/ciryon/OCUnit2JUnit
It works as I test it here with Ruby 1.9.3. Which version are you using? It would be splendid if you created an issue over at GitHub for this.
Thank you for putting this together.
When running from the command line, I get this error:
/Applications/Xcode.app/Contents/Developer/Tools/RunUnitTests:68: note: RunUnitTests exited without running tests because TEST_AFTER_BUILD was set to NO.
and then after changing to YES, I get this error:
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/Tools/RunPlatformUnitTests:81: warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).
Have you seen these errors? What is the way around this? Do I have something set up incorrectly?
This is a problem with Apple’s way of handling unit tests. There are work-arounds, the one I’m using is the following: http://stackoverflow.com/a/10823483/22012
You should however also file a bug at Apple so they do something about this annoying problem.
I have followed a raywanderlich tutorial on CI, in which your ocunit2junit.rb file is used. I added it to my jenkins script to allow for easier interpretation of the any failed tests should they occur.
I am having a problem at the moment where everything succeeds but at the very end no reports are found. Any help would be much appreciated. Please see below.
+ xcodebuild -target ApplicationTests -sdk iphonesimulator -configuration Debug TEST_AFTER_BUILD=YES clean build
+ /Users/admin/bin/OCUnit2JUnit-master/bin/ocunit2junit.rb
Recording test results
No test report files were found. Configuration error?
Build step ‘Publish JUnit test result report’ changed build result to FAILURE
Finished: FAILURE
It looks as you’re missing the pipe character |