One of the most interesting areas in mobile has been the quick evolution of Augmented Reality (AR) platforms and applications utilizing mobile device’s camera in overlaying UI elements over the live video stream in real time. These applications and games create very engaging user experiences but are very difficult to test for the following reasons:
The input for the apps comes from the live video feed generated by the device’s camera and as well as the information generated by the inbuilt accelerometer. Instead of generating clicks on the screen, the device needs to be moved around to provide input to test the application.
Updating for User Experience
The user experience is very sensitive to any delay, distortion and noise in the input and, as such, testing involves significant qualitative component in addition to more traditional quantitative fail/pass test results.
Real-time processing of moving video and overlaying 3D object on top of the video stream is very resource intensive and the majority of the processing is done on a device’s dedicated hardware accelerated chips. Therefore using real devices is a must and the test results vary significantly from one device (or hardware platform) to another.
When Google approached Bitbar with their requirements of automating the AR application testing process it was clear from the beginning that this was not going to be a standard test automation project and therefore the solution had to be a heavily customized private device cloud that takes these requirements into account. It was also clear that all this could not be achieved by custom software alone but required innovative hardware arrangements to do the job.
Building in the Physical for the Digital
The first task was to create a physical arrangement where all the devices were positioned on a plane the same distance from a three-dimensional ‘table’ about 2 feet (60cm) from the devices. Additionally the table needed to have different textures so the Augmented Reality platform was able to align the superimposed 3D objects above the table surface. Finally we needed to ensure even lighting conditions on and around the table so that shadows or dark areas did not create interference to the testing process. This was achieved by attaching wide angle LED-lights pointing at the table from four directions creating an even light blanket.
The second requirement was much more challenging: To be able to calibrate the AR application the automation has to move the devices for some time to calibrate the video and accelerometer input together. This has to be done every time the application is re-installed (in this case every time). To make the setup more complex, the devices have to be stationary when the actual tests are executed. To solve this we mounted the devices on oscillating planes and used a custom built controller to oscillate the planes over API calls. This way the customer was able to start and stop the oscillation directly from their test scripts and have full control of the whole automated test flow.
On with the Tests
The tests themselves were standard Server-Side Appium test executed simultaneously on all (Android and iOS) devices to ensure comparable test results. The output of the tests was traditional pass/fail information, detailed video of the execution as well as detailed metrics of the usage profile of critical hardware metrics (CPU, GPU, Memory).
The sky’s the limit when it comes to the customization of Bitbar Private Cloud setups. We have helped automate testing of mobile applications that interact with wearables, wifi access points, fire alarms, intelligent fridges, automotive systems, etc. And we’ve always been able to find creative solutions to bring our customers’ test automation to the next level.