Our guest blog is back. Today’s guest is Tom Whipple, a Mobile Engineer and Android Lead for card.io. card.io makes an easy to use SDK for visually reading credit card numbers, and was acquired by PayPal this summer. He would like to share some of his experiences about how Testdroid helps his team in development.
Using Testdroid Cloud to test a camera SDK
Developing an SDK that makes heavy use of the camera is different from typical Android app development. First, developing an SDK means that we have to think about ease of integration with other apps. For example, things like XML layouts and
R.drawable are not available to us, since we don’t want any unnecessary integration steps. Second, we are doing on-device computer vision, so we have a lot of hightly optimized C and even some hand written assembly which must be compiled with the NDK and linked via JNI. And, if you’ve been fortunate to work with the Android native layer, you know that the toolchain is not quite as stable as the Java SDK. Finally, we’re doing frame by frame processing with the camera. Obviously, the camera is a piece of hardware that has been implmented differently by each manufacturer, so there are inevitably some differences in behavior if it is accessed in a way that is not quite as intended.
Importance of testing
As others have explained, there are a huge number of device/OS combinations out there. NDK developers face the additional challenge that there are four ABI types that are supported by Android: ARM, ARMv7a, x86 and MIPS. Additionally, some versions of ARMv7a include the NEON vector instruction set, while others do not. Card.io only supports ARMv7a, but we do have a manual entry fallback mode. (ARM 5/6 are not powerful enough for vision, and x86/MIPS don’t have enough market share to prioritize).
So clearly, testing is critical. How do we go about it? There are a few options:
- Buy a bunch of test devices. A few test devices are essential for development, but testing quickly becomes unmanageable. Plus, it isn’t always clear that the phone you are buying will exhibit the problems your customers are experiancing. Not all phones are available in all regions, even within the US.
- Hire an Android test engineer. A very expensive option for a startup. And you still have to deal with #1.
- Test as best you can, then ship. This is the most common option and the one we used for our short-lived consumer app. Very quickly we found that a few users with modern phones were giving us bad ratings because the app crashed. We couldn’t reporoduce the problem on any phone we could get our hands on, so blew it off. Finally, we were able to convince a user or two (thanks to Spence in KC!) to install the Android SDK and send in a logcat. It took 2+ months to understand that the problem was with non-NEON phones and buy one. I don’t recommend this approach.
- Use Testdroid Cloud. This is the simplest, most effective option. I highly recommend it. I wish I’d know about it when we shipped our first version.
Our goal with cloud based testing is not to exactly replicate the user experiance, since that experiance involves putting a credit card in front of the phone. Instead, we can take advantage of the fact that most of the device specfic bugs and crashes occur when the camera is opened during the creation of the scan activity, or as the camera is being closed when the activity is finishing. As a result, the mere fact that the scan activity opens and is rendered without a crash demonstrates tremendous value.
Our second test strategy is a little more involved. We’ve built a special test app that calls
onPreviewFrame() with an appropriately constructed YCbCr image buffer, in almost the same way that Android does. (I say almost, because the pixels are not reandered onto the preview surface, and this approach only works on Android 4.0 and above at the moment. We haven’t taken the time to figure what’s different about earlier versions.)
The result of these two strategies is that we cover a good portion of our workflow on all of the devices supported by Testdroid Cloud. This automated testing is not a substitute for manual testing, but it reduces the manual test burden to a manageable level.