Implementing a reliable and scalable game test automation solution on real mobile devices

The client

A prominent company in the card game development industry, our client is offering applications for both Android and iOS platforms. The testing team has already integrated and used the AltTester® Unity SDK within their development cycle.

Our solution

We started with a 4-week proof of concept project where we addressed some of their needs. Initially, we updated the current testing framework to incorporate the most recent version of AltTester® (v2.0.2). One of the first steps in establishing a CI pipeline to run tests on real devices was to adapt existing methods to create instrumented builds on both Android and iOS platforms.

Run tests both from Unity Editor and as part of an NUnit test project

The need to integrate Appium and Allure into the existing testing framework led to the requirement of running tests from an NUnit test project, using dotnet cli, outside the Assets folder. At the same time we had to keep the tests in Assets for the early stages of development checks.

One of the challenges to achieve this was raised by the fact that the tests had been written using an API library that only worked from within Unity Runtime so we could not call it when we ran the tests without Unity.

We created a script which copied all the tests outside the Assets folder and we ran them as a separate NUnit test project. We used HTTPClient API instead of the Unity dependent library. Furthermore, in the test code, we needed to handle various cases regarding what can be executed in the Unity Editor and what cannot.

When running tests on real devices, it was necessary for all packages and libraries used in the Assets folder to be available both inside the Assets folder and outside it. In order to achieve that, we saved relevant .dll files as artifacts from the Jenkins job building the game then referenced them in the new NUnit project, in .csproj.

Deal with connectivity between AltDriver and AltTester® Desktop

Jenkins was set up on a machine where one Android and one iOS device were connected through USB. To make sure that AltTester® Desktop differentiates the builds, we adapted the instrumentation job to set custom app names for each platform.

To facilitate communication between AltDriver from the game and AltServer on Android, using adb for reverse port forwarding and employing a build with the host set to localhost is enough.

The situation is different for iOS devices since IProxy lacks the capability for setting up reverse port forwarding. To establish the connection between the game and the AltServer, we used the IP address of the machine where the AltServer was running.

When integrating the execution of tests on a cloud service, we utilized BrowserStack Local tunnel. This tool simulates the device being on the same network as the machine running AltServer. For this setup, we employed an instrumented build for iOS with localhost.

Use Appium to create tests which interact with native elements

Nowadays, applications integrate with social network apps, necessitating the testing of situations where users exit the application and connect to other apps. Moreover, we’ve created tests using Appium for scenarios involving In-App Purchases.

Successfully overcoming the challenge of transitioning control between AltDriver and Appium driver, we managed to construct more complex test scenarios.

Save test results in a report with screenshots & logs from Unity

In addition to saving test results in a report, the Allure framework provides the option to append screenshots and log files to the report. In order to provide additional information to developers when debugging, we looked for solutions to save the Unity exceptions triggered while executing tests on the real devices. Besides filtering device logs captured with tools like adb logcat or idevicesyslog, we employed AltDriver’s method: AddNotificationListener.