My friend and colleague Christian Hedin gave me tips on iCuke. Cucumber has been widely used for testing web applications, but now it’s also possible to test iOS (iPhone and iPad) apps with help of the iCuke library (https://github.com/unboxed/icuke). iCuke uses AppleScript to drive XCode in order to launch your application into the iOS Simulator. A preloaded library is used to inject a small HTTP server into your application. The HTTP server allows you to see an XML representation of the iOS device screen and to emulate input, such as taps, swipes and pinch gestures.

You can read more about it here:
http://www.unboxedconsulting.com/blog/cucumber-iphone-icuke
http://pragprog.com/magazines/2010-07/bdd-on-iphone-icuke

Really excited about Cucumber on iPhone we decided to give it a try. After installing iCuke and doing some test runs it was clear that some challenges had to be overcome to make this a truly useful tool for Behaviour Driven Development and automated testing of iOS apps.

Challenge 1: Screen returns the previous screen's xml

In our test, we wanted to tap a button, come to another screen and expected to see a text. When the test runs the iPhone is driven to the correct screen, e can see the expected text but the test fails anyway. After little debugging we realized that the method screen.xml returns old xml directly after changing the screen.

Solution:

We needed to refresh screen before checking if the expected text is on the new screen. iCuke has a method to refresh screen but it is private and we could not use it. So we just added a new method to the existing ICukeWorld class.

class ICukeWorld
 def refresh_screen
   refresh
   screen
 end
end

Calling this method before checking for the presence of the text solved this problem.

Later on, I forked iCuke and added this and some other methods.

git://github.com/DavorC/icuke.git

Challenge 2: Timing

Everybody knows that using sleep and delays in code is not so flexible.

sleep 3 # wait 3 seconds

We want to check something on the screen and give it 3 seconds to finish its loading.

Is it enough? Maybe, maybe not. Screen content loading could take 0.1 second or 5 seconds or... - You know what I mean.

In the first case loading is finished quickly and we unnecessarily spend 3 seconds for doing nothing . If we have a lot of delays in our code then our tests would waste a lot of precious time.

In the second case the delay is not long enough and the test fails.

Solution:

We need to write some help functions to wait for different items which are expected: some text, a button, downloading spinner etc.

Example with wait for text:

def wait_for_text(text, timeout = @@timeout)
 puts "#{method_name}(#{text}, #{timeout})" if @@debug
 refresh_screen
 start_time = Time.now
 until(screen.exists?(text)) do
   if Time.now - start_time > timeout
     flunk("#{method_name}: Timed out after #{timeout} seconds")
   end
   sleep 0.1
   refresh_screen
 end
end

As you can see the method is waiting for the text to appear and does checking every 0.1 second. As soon as the text is found the test continues. If, after given timeout, the text is still not found, the test fails.

I prefer to use unit test assertions in my tests (flunk is an assert which always fails). To use assertions with cucumber you need to add assertions to the Cucumber World:

require 'test/unit/assertions'
World(Test::Unit::Assertions)

Challenge 3: Different tappable object on screen can have the same text label.

Identifying objects on the screen only by text is not enough. By default, the first tappable object is tapped. What if we want to tap the second one?

Solution:

I created a set of help functions for:

returning all objects that satisfy some criteria
returning a specific object
waiting for a specific object
checking if a specific object exists

Objects are described in xml by: type, label, traits and index.

Here is an example of getting an array of all elements satisfying given attribute values.

def get_all_elements_by_type_label_and_traits(type, label, traits)
 puts "#{method_name}(#{type}, #{label}, #{traits})" if @@debug
 refresh_screen
 doc = REXML::Document.new(screen.xml.to_s)
 elements = REXML::XPath.match(doc, "//#{type}[@label=#{label.inspect}][@traits=#{traits.inspect}]")
 elements
end

Observe how it is easy to parse xml using ruby's REXML library.

Of course I could write more generic methods and decrease number of code lines - something like:

get_all_elements(type, options = {})

but I like readability so I wrote a set of help functions with more specific naming:

get_element_by_type(type, index = 0)
get_element_by_type_and_label(type, label, index = 0)
get_element_by_type_and_traits(type, traits, index = 0)
get_element_by_type_label_and_traits(type, label, traits, index = 0)
get_all_elements_by_type(type)
get_all_elements_by_type_and_label(type, label)
get_all_elements_by_type_and_traits(type, traits)
get_all_elements_by_type_label_and_traits(type, label, traits)
get_all_static_texts()
get_all_labels_by_type(type)
get_all_labels_by_traits(traits)
element_by_type_exists?(type, index = 0)
element_by_type_and_label_exists?(type, label, index = 0)
element_by_type_and_traits_exists?(type, traits, index = 0)
element_by_type_label_and_traits_exists?(type, label, traits, index = 0)
text_exists?(text)
wait_for_element_by_type(type, index = 0, timeout = @@timeout)
wait_for_element_by_type_and_label(type, label, index = 0, timeout = @@timeout)
wait_for_element_by_type_and_traits(type, traits, index = 0, timeout = @@timeout)
wait_for_element_by_type_label_and_traits(type, label, traits, index = 0, timeout = @@timeout)
wait_for_text(text, timeout = @@timeout)
get_center_of_the_element(element)
tap_coordinates(x, y)
double_tap_coordinates(x, y)
tap_element(element)
double_tap_element(element)
tap_text(text)
double_tap_text(text)

In order to use these functions you need to use iCuke from:

git://github.com/DavorC/icuke.git

Don't forget to use –recursive flag when you clone it:

git clone --recursive git://github.com/DavorC/icuke.git

After building and installing the iCuke gem you need to

require 'icuke/cucumber_ext'

instead of:

require 'icuke/cucumber'

Example of usage

In your feature-file:

Background:
 Given "myApp.xcodeproj" is loaded in the simulator

Scenario Outline: User try to login with different invalid credentials with valid signs
 When I am in "Account" section
  And I paste in username "<user>"
  And I paste in password "<pass>"
  And I tap Login button
 Then I will see alert dialog

 Examples:
 | user            | pass      |
 | test@test.com   | test      |
 | 123456          | qwertyui  |
 | ..@..com        | . . . €<> |

step definitions:

When /I am in "(.*)" section/ do |section|
 wait_for_text(section)
 tap(section)
 wait_for_element_by_type_and_label("UINavigationItemView", section)
end

When /I paste in username "(.*)"/ do |user|
 label = "E-mail"
 refresh_screen
 assert(get_all_static_texts().include?(label), "No text field with label #{label} was found")
 write_to_mac_clipboard(user)
 paste_clipboard_to_text_field("UITextFieldLabel", label)
end

When "I tap Login button" do
 wait_for_text("Login")
 tap("Login")
 wait_for_download_indicator_finish
end

Then "I will see alert dialog" do
 wait_for_element_by_type("UIAlertView")
end

Testing both iPhone and iPad
If you’re testing a universal app that runs on both iPhone and iPad I recommend writing different scenarios for the platforms. iPad in landscape mode is most likely to reuse most of code you have written for iPhone. Place them in different feature files and tag them with e.g. @iphone respective @ipad tags.

In your env.rb file:

$PLATFORM = "iphone"

Before('@ipad') do
 $PLATFORM = "ipad"
end

Before('@iphone') do
 $PLATFORM = "iphone"
end

Use in your feature files:

Given I have started application

Implementation:

Given "I have started the application" do
 # ... some code
 Given "\"myApp\" from \"myApp.xcodeproj\" is loaded in the #{$PLATFORM} simulator"
 # ... more code
end

If you wants to run iPad simulator in landscape mode:

def switch_ipad_to_landscape
 if(get_ipad_orientation == PORTRAIT)
   rotate_simulator_left
 end
end

where get_ipad_orientation is some application specific method to decide if the simulator is in portrait or landscape mode.

Running iCuke tests on Hudson server

It’s really nice to be able to run your test suite at given intervals, or when you commit to the source code repository. To run your iCuke tests on Hudson (which is a popular continuous integration build server) you must start your iPhone simulator from a terminal window. This is easiest to do by launching an AppleScript from Hudson.

Below are two scripts, one AppleScript and one shell script, that I used to run my iCuke tests from Hudson.

- run_cuke.scpt

-- run_cuke.scpt
tell application "Finder"
 set my_folder_path to container of (path to me) as text
 set posixPath to POSIX path of file my_folder_path
 set scriptPath to posixPath & "cuke.sh"
end tell
tell application "Terminal"
 activate
 do script scriptPath
end tell
delay 1200 -- it should be enough to finish all tests
set logscript to "grep -c FAILED " & posixPath & "cuke.log" & " | cat"
set cuke_failed to do shell script logscript
if cuke_failed > 0 then
 error "Cucumber tests failed"
end if
try -- do not leave terminals after test run
 do shell script "killall 'Terminal'"
end try

- cuke.sh

#!/bin/bash
scriptpath=$(cd ${0%/*} && echo $PWD/${0##*/})
SCRIPTFOLDER=`dirname "$scriptpath"`
echo Cucumber script is run in: $SCRIPTFOLDER
cd $SCRIPTFOLDER
cucumber $SCRIPTFOLDER/features --format=html --out $SCRIPTFOLDER/cuke_results.html > $SCRIPTFOLDER/cuke.log

Add these two scripts to your project.
Then, add this Cucumber hook to your env.rb file:

After do |s|
 if(s.failed?)
   puts "Scenario FAILED: <#{s.name}>"
   puts "More info about failure: SCREEN XML"
   puts screen.xml
 else
   puts "Scenario PASS: <#{s.name}>"
 end
end

On Hudson, create a new job and copy the configuration from your project's existing job.
Configure job:
add to description:

<a href="/hudson/job/Helios_Cuke/ws/cuke_results.html">Cucumber Results</a>
<b> | </b>
<a href="/hudson/job/Helios_Cuke/ws/cuke.log">Debug.log</a>

add build step (execute shell):

osascript ${WORKSPACE}/run_cuke.scpt

Of course you can change all scripts according your needs.
At the end you should have your iCuke tests running on Hudson and you will get both a nice test report in HTML and the debug output as a plain text.
There is a lot of potential for doing automated feature tests for iOS using Cucumber and with iCuke and the additions above you’ll hopefully be well on your way for doing BDD in your next iOS project!

First, to enable FindBugs reporting in Maven, just add report section to your pom files, something like example below, but of course, you can do your own configuration.

<reporting>
  <plugins>
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>findbugs-maven-plugin</artifactId>
      <version>2.3</version>
      <configuration>
        <excludeFilterFile>exclude.xml</excludeFilterFile>
        <findbugsXmlOutput>true</findbugsXmlOutput>
        <findbugsXmlWithMessages>true</findbugsXmlWithMessages>
        <threshold>Low</threshold>
        <effort>Max</effort>
        <xmlOutput>true</xmlOutput>
      </configuration>
    </plugin>
  </plugins>
</reporting>

Then you can just use command mvn site if you want to generate more comprehensive project information or mvn findbugs:findbugs for only FindBugs reports.

Finally use mvn dashboard:dashboard to generate charts.

Voila. We have got nice graphic reports.
Well, not so nice. We have bugs we need to fix.