iOS App Testing

iOS app testing is critical to the overall mobile app testing strategy. Most users are busy exploring their options and the tools available to decide what’s right for their testing needs. This page will put the general issues relating to the iOS testing scene in context and throw light on the specifics of Bitbar’s offering. In short, we offer several things…

What “iOS App Testing” Means on Different iOS Configurations?

iOS testing is normally made up of some or all of the following parts. To learn more about each part, and what Testdroid provides, please click the links.

Manual testing – you do this by running a Testdroid Interactive, or by installing and testing the app on a real iPhone or iPad. Testdroid can help you perform manual testing on multiple iOS configurations without having to set up the complex device lab yourself. Testdroid Interactive will be available later 2015. Please check the status on Testdroid Cloud -> Interactive.

Test automation – for any iOS app with some degree of complexity, inevitably, you will have to write scripted tests and automate their running. There are two main types of automated tests: end-to-end tests and unit tests.

XCTest and XCUITest for Unlimited Number of Tests

system tests – in this type of full system test, a real user is simulated. The test checks how all components of the system work together. This form of testing involves launching the app on real device, then automating interaction with the user interface to perform certain user actions (such as touching, swiping), and then checking the test result. For example, you can run an end-to-end test that 1) logs into the app and 2) tries to upgrade from a free to paid subscription. This process is a highly sensitive operation and one that must be tested from end-to-end. Because system tests are “costly” in terms of computing resources and also more complex to run, you always need to prioritize and test only the most important workflows in your application. Another challenge is that system tests run differently on different iOS configurations (for example, on an iPad vs. on an iPhone).

That means you have to run each test several times – once for each configuration you intend to support. We see three scenarios for running end-to-end tests:

Running XCTest / XCUITest yourself – you can use your own real devices o run scripted end-to-end tests. But the number of tests you can possibly run, and the different iOS configurations you are able to test will be limited by the number of machines and mobile devices you actually own. In addition, you must install and manage the infrastructure on your own.

Running XCTest / XCUITest in the cloud – Bitbar products provide the most versatile selection of different iOS devices in the cloud and allow you to run end-to-end tests on the provider’s servers in the cloud — you need only supply your app and a test script. The number of servers at the cloud provider far exceeds the number anyone tester can set up. Also, Testdroid has already set up hundreds of iOS version-device combinations, so you cover more options and run more tests more quickly. But there’s a price to all this versatility- you’ll need to pay the cloud provider, usually based on the number of tests you perform and in accordance with test complexity.

Testing, End Tests and System Tests in Agile Development Process

Selenium automation – the open-source Selenium project has already become the de-facto standard for scripting and running end-to-end tests in the non-mobile automated testing world. Testdroid strongly believes in leveraging this standardization effort to include mobile testing, too. Our mobile testing offering supports Selenium and provides a painless ramp-up for existing Selenium users.

Unit testing – a unit test involves coding to test a single component of an app, taken in isolation. The test typically passes to the component a series of variables, and checks that the component returns the correct values. Running unit tests is faster and far less “costly” than running full system tests because you don’t have to initiate a full instance of iOS for each test. Instead, you can run a large number of tests together on a local machine. Apple’s XCode development environment lets you create and run unit tests for iOS apps. It is often used together with Kiwi, an Objective-C unit testing framework, which aims to improve the readability of tests and make test running and error reporting more seamless.

Testing frameworks – XCTest is nowadays the standard framework with XCode. However, if you consider doing unit tests or full-blown system tests, and to make it all work together, you are going to need a testing framework. If you are using older Xcode versions (<= Xcode 7) the most basic solution is to learn Apple’s Javascript-based UI Automation framework, which lets you write test scripts and run them through the XCode development environment. If you are using the latest Xcode (Xcode 8 >=) XCTest and XCUITest are readily available for your app testing needs. Outside of that, there are several open source frameworks that try to make automated iOS testing easier. Among them are Jasmine, Frank, Calabash-iOS, iOS-Driver, Keep It Functional (KIF), and Appium. There are important differences between these frameworks.

Continuous integration – if you perform automated testing on a large scale, you probably use a CI system such as Jenkins or Travis. Those systems typically request and execute all these tests, record their results, and feed them into an agile development process. Of the iOS testing frameworks, some, including Appium, enable you to connect to a CI system and run automatic iOS tests in the context of your agile development process. That’s OK, but be conscious that mobile CI is in its infancy and does not yet work as smoothly as CI with traditional testing tools. We’ll shortly provide further details here about our mobile CI offering.