In my latest project I was responsible for test and quality of a hybrid mobile application for Android, iOS and Windows Phone. The plan was to have a simple native “wrapper” for the different platforms and then a webview running all the “real” code that could then be shared between all platforms. However, due to different circumstances, it became more complex and instead of one webview we had multiple simultaneously running webviews and the native wrapper included navigation, socket implementation and a messagebus for sending messages between native and web and syncing the different webviews.
When it comes to testing the biggest challenge was all the different variants.
First we had the different platforms:
- Windows Phone
For android we also had two different build variants, one supporting ICS – Jellybean and one supporting KitKat – Lollipop (I will not go in to the reason behind this)
Then for all platforms I needed to test different:
For iOS we supported from version 7.1, for Android from version 4.0 and for Windows Phone from version 8.1
- Devices/Screen resolutions
We sometimes found issues connected to one specific device or manufacturer and for the styling of the app it needed to be verified on “all” different screen resolutions
- Language/Country settings
Not just for string translation but the application also had some different functionality depending on country and language setting
- Network condition
The application was very network dependent so this needed extensive testing on the different platforms
- Test environment/User accounts
We had multiple environments to test on, supporting different functionality, and for the different environments there were multiple user accounts with different data
One of the advantages with hybrid applications are that development and maintenance can be easier since one fix in web could mend an issue in all three platforms. But for test it is not easier at all. Because even if a fix or new developed feature is just in web code you cannot take for granted that it will work the same in all browsers and webviews on the different platforms. We saw many cases where e.g. a fix would work on Android and iOS but not on Windows Phone. Or worked on iOS 8.1 but not on iOS 7.1. So for each small fix in web it still needed to be tested on a large amount of devices.
Because of this I started to look at some automation frameworks and came across 4 different solutions:
- Pure web testing using e.g. Selenium
This could maybe be a good solution if you have a very simple native “wrapper” and then all other code in web. But since we had quite much logic in the native layers as well this was really not an option in our case.
- Using OS specific frameworks (with webview support) for the different platforms.
This will demand three different frameworks (with different programming languages) and three different tests for each scenario. This is far too much work in development and maintenance so this was also not an option.
- “Cross-platform” frameworks
There are (to my knowledge) no cross-platform test framework that have support for Android, iOS and Windows Phone. There are however frameworks like Appium (http://appium.io/) that support Android and iOS. But when I started this project Appium was very new and had some issues which made me abandon this solution.
- Using one OS specific framework
This was the solution I started to implement by using Robotium (https://code.google.com/p/robotium/) for Android. With this my tests covered native Android, web and the communication between native and web but would not cover native iOS and Windows Phone or web and web/native communication specific issues on those platforms.
The conclusion is that there are no good cross-platform test framework that will work on all these three platforms. The frameworks that title themselves “cross-platform” are often targeting Android and iOS. Also it’s not that easy to write a generic test that will work on both platforms native components (for the webviews it’s no problem). You need to synchronize development of the applications for the different platforms so that id’s, names, class-names, x-paths etc. are the same for the components you need to use in your test. And even then I think it’s hard to get the tests 100% generic since there are a lot of things differing the two platforms.
But if I should start this today I would probably anyway use Appium instead since this seems to have matured and become more stable now and also seems to be under constant development to get even better. This together with cloud services like e.g. Sauce Labs (https://saucelabs.com/) have potential of “solving” the hazard with the large amount of device variants.