In my latest project I was responsible for test and quality of a hybrid mobile application for Android, iOS and Windows Phone. The plan was to have a simple native “wrapper” for the different platforms and then a webview running all the “real” code that could then be shared between all platforms. However, due to different circumstances, it became more complex and instead of one webview we had multiple simultaneously running webviews and the native wrapper included navigation, socket implementation and a messagebus for sending messages between native and web and syncing the different webviews.
When it comes to testing the biggest challenge was all the different variants.
First we had the different platforms:
- iOS
- Windows Phone
- Android
For android we also had two different build variants, one supporting ICS – Jellybean and one supporting KitKat – Lollipop (I will not go in to the reason behind this)
Then for all platforms I needed to test different:
- OS-version
For iOS we supported from version 7.1, for Android from version 4.0 and for Windows Phone from version 8.1 - Devices/Screen resolutions
We sometimes found issues connected to one specific device or manufacturer and for the styling of the app it needed to be verified on “all” different screen resolutions - Language/Country settings
Not just for string translation but the application also had some different functionality depending on country and language setting - Network condition
The application was very network dependent so this needed extensive testing on the different platforms - Test environment/User accounts
We had multiple environments to test on, supporting different functionality, and for the different environments there were multiple user accounts with different data
One of the advantages with hybrid applications are that development and maintenance can be easier since one fix in web could mend an issue in all three platforms. But for test it is not easier at all. Because even if a fix or new developed feature is just in web code you cannot take for granted that it will work the same in all browsers and webviews on the different platforms. We saw many cases where e.g. a fix would work on Android and iOS but not on Windows Phone. Or worked on iOS 8.1 but not on iOS 7.1. So for each small fix in web it still needed to be tested on a large amount of devices.
Because of this I started to look at some automation frameworks and came across 4 different solutions:
- Pure web testing using e.g. Selenium
This could maybe be a good solution if you have a very simple native “wrapper” and then all other code in web. But since we had quite much logic in the native layers as well this was really not an option in our case. - Using OS specific frameworks (with webview support) for the different platforms.
This will demand three different frameworks (with different programming languages) and three different tests for each scenario. This is far too much work in development and maintenance so this was also not an option. - “Cross-platform” frameworks
There are (to my knowledge) no cross-platform test framework that have support for Android, iOS and Windows Phone. There are however frameworks like Appium (http://appium.io/) that support Android and iOS. But when I started this project Appium was very new and had some issues which made me abandon this solution. - Using one OS specific framework
This was the solution I started to implement by using Robotium (https://code.google.com/p/robotium/) for Android. With this my tests covered native Android, web and the communication between native and web but would not cover native iOS and Windows Phone or web and web/native communication specific issues on those platforms.
The conclusion is that there are no good cross-platform test framework that will work on all these three platforms. The frameworks that title themselves “cross-platform” are often targeting Android and iOS. Also it’s not that easy to write a generic test that will work on both platforms native components (for the webviews it’s no problem). You need to synchronize development of the applications for the different platforms so that id’s, names, class-names, x-paths etc. are the same for the components you need to use in your test. And even then I think it’s hard to get the tests 100% generic since there are a lot of things differing the two platforms.
But if I should start this today I would probably anyway use Appium instead since this seems to have matured and become more stable now and also seems to be under constant development to get even better. This together with cloud services like e.g. Sauce Labs (https://saucelabs.com/) have potential of “solving” the hazard with the large amount of device variants.
Ever considered Calabash?
Yes I have done some small experiment with Calabash but never used it in a real project. From my limited experience I don’t see mush benefit of Calabash compared to Appium. Both are cross-platform (Android and iOS) but I think it is as mush work in both solutions to actually get one test to work on both platforms. Both could be used in cloud services, SauceLabs for Appium and Xamarin Test Cloud for Calabash. But I think that Appium has the advantage in that you can select from multiple programming languages to implement your test while Calabash only supports Ruby. And for Appium you can test the application you actually release to the stores while in Calabash iOS you will have to create a special build with a test library for it to work.
But as I said I have only done some small experiments with it so maybe there are other advantages with Calabash that I don’t know about.
A superb product, which wasn’t mentioned in the blog, for cross-platform development specifically, is the “Xamarin Test Cloud”:
http://xamarin.com/test-cloud
The services offers your tests to be executed on a multitude of different devices with different settings and form factors.
As Xamarin mention themselves:
“[…] that can automate and test any iOS or Android app, native or hybrid, from the UI level down”
Hope this is at least worth your consideration.
// David
Yes I have read a little on the Xamarin Test Cloud but have never used it. It uses Calabash which I think have some disadvantages compared to Appium (see previous comment). But other than that it seems promising.
Thomas,
With this 1 year .. have you finalized or using a good tool available in the market? What do you use?
I have played around a little more with Appium (see https://blog.jayway.com/2016/06/30/cross-platform-testing-example-android-ios-using-appium/) and I think this would be my first choice of automation framework for cross platform projects today. It seems to have become more stable and mature since I first tested it and it is still under constant development. It’s also widely used (both Appium itself and Selenium Webdriver that Appium is based on) so it’s easy to find help in case you get stuck.
package Alchemy.Alchemy;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
import io.appium.java_client.MobileElement;
import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.remote.AndroidMobileCapabilityType;
import io.appium.java_client.remote.MobileCapabilityType;
import io.appium.java_client.remote.MobilePlatform;
import org.testng.annotations.AfterTest;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Test;
public class SmokeTestAlchemy
{
public AndroidDriver driver;
@BeforeTest
public void setup() throws MalformedURLException, InterruptedException
{
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME,MobilePlatform.ANDROID);
capabilities.setCapability(MobileCapabilityType.UDID,”ZY223R9XDP”);//Give Device ID of your mobile phone
capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION,”6.0.1″);
capabilities.setCapability(MobileCapabilityType.DEVICE_NAME,”Moto G4″);
capabilities.setCapability(“app”, “G://android-armv7-debug.apk”);
//capabilities.setCapability(AndroidMobileCapabilityType.APP_PACKAGE,”com.ecifm.otgservices”);
// capabilities.setCapability(AndroidMobileCapabilityType.APP_ACTIVITY,”com.ecifm.otgservices.MainActivity”);
capabilities.setCapability(“fullReset”,”true”);
capabilities.setCapability(“autowebview”, true);
capabilities.setCapability(AndroidMobileCapabilityType.RECREATE_CHROME_DRIVER_SESSIONS, true);
capabilities.setCapability(“newCommandTimeout”, “120”);
capabilities.setCapability(“chromedriverExecutable”, “C://Users//vijender.kumar//chromedriver.exe”);
driver =new AndroidDriver(new URL(“http://127.0.0.1:4723/wd/hub”), capabilities);
driver.manage().timeouts().implicitlyWait(30L, TimeUnit.SECONDS);
// WebView.setWebContentsDebuggingEnabled(true);
/*WebDriverWait some_element = new WebDriverWait(driver,100);
some_element.until(ExpectedConditions.visibilityOfElementLocated(By.id(“pageTitle”)));*/
}
@Test
public void operation()
{
System.out.println(“operation() :: driver.start() executed”);
try{
By webView = By.className(“android.webkit.WebView”);
String title=driver.findElement(By.partialLinkText(“Service”)).getText();
WebDriverWait wait = new WebDriverWait(driver,300);
System.out.println(“Title is” +title);
Set availableContexts1 = driver.getContextHandles();
System.out.println(“Total No of Context Found Before reaching WebView = “+ availableContexts1.size());
System.out.println(“Context Name is “+ availableContexts1);
driver.findElement(By.xpath(“//input[@type=’text’]”)).click();
//4.1 Navigate to a portion of your app where a web view is active
//driver.findElement(By.id(“com.mkyong.android:id/buttonUrl”)).click();
// wait.until(ExpectedConditions.visibilityOfElementLocated(webView));
Set availableContexts = driver.getContextHandles();
System.out.println(“Total No of Context Found After we reach to WebView = “+ availableContexts.size());
/*for(String context : availableContexts) {
if(context.toString().toUpperCase().contains(“WEBVIEW”)){
System.out.println(“Context Name is ” + context);
// 4.3 Call context() method with the id of the context you want to access and change it to WEBVIEW_1
//(This puts Appium session into a mode where all commands are interpreted as being intended for automating the web view)
driver.context(context);
//driver.context(context);
driver.findElement(By.xpath(“//input[@type=’text’]”)).click();
}
}*/
}catch(Exception ex){
ex.printStackTrace();
}
}
//driver.findElement(By.xpath(“//input[@type=’text’]”)).sendKeys(“bworking”);
/* @AfterTest(alwaysRun= true)
public void teardown() {
driver.quit();
}*/
}
//hybrid app automation testing
getting error chrome driver not starting
chrome driver:2.35
chrome version :63 both
ANDorid:3.0.1
appium versoin:1.7.2
Hi Thomas , while searching for a single open source framework to test the hybrid app (built using the Vue.Js) i came across this article. I have been doing a lots of searching around which one i can use for testing my hybrid app running on Android,IOS and web platform. I wanted to go for Appium, but found that with appium you cannot test the IOS app using windows machine .If i do not have the Mac machine, i cannot. From your experience and research.Could you please suggest which one i can use. I came across this Test Project (https://testproject.io/) and Katalon( https://www.katalon.com/) . I am working on some side project, with my colleagues to build a free app to Non Profit organization. I do not want to spend more time on preparing the boiler plate so i was searching something half ready so that i can focus more on testing.
Thanks for your article. I hope you have come up something by something now with the lots of tools and frameworks out there.
Hi.
I have a Mac myself so I don’t have any experience on running Appium on Windows. But if you find a workaround to run Appium test on iOS from Windows (or if you bye a Mac) I have written another blog post about Appium that also includes an example project. Maybe you can used is as an initial setup for your test. You can find it here: https://blog.jayway.com/2016/06/30/cross-platform-testing-example-android-ios-using-appium/
Thank you for posting this type of great content … I was looking for something like this.