Implementing a reliable and scalable game test automation solution on real mobile devices
Game Test Automation Case Study
- Integrate Appium and enable automated testing on real devices
- Integrate Allure Framework and improve test reporting
- Be able to run tests both from Unity Editor and as part of an NUnit test project
- Enable running tests on a cloud service provider: BrowserStack
Types of testing
- Game Test Automation
- Mobile Application Testing
- End-to-End Test Automation
- UI Test Automation
Tools & technologies
- AltTester® Unity SDK
- Game development
- Mobile Apps
A prominent company in the card game development industry, our client is offering applications for both Android and iOS platforms. The testing team has already integrated and used the AltTester® Unity SDK within their development cycle.
The needs and challenges
They approached us to help them improve their testing processes. The main goals were to start running tests on real devices in a CI environment, upgrade test reporting by saving test results after each execution and enable running tests on devices from a cloud service provider.
We started with a 4-week proof of concept project where we addressed some of their needs. Initially, we updated the current testing framework to incorporate the most recent version of AltTester® (v2.0.2). One of the first steps in establishing a CI pipeline to run tests on real devices was to adapt existing methods to create instrumented builds on both Android and iOS platforms.
Run tests both from Unity Editor and as part of an NUnit test project
The need to integrate Appium and Allure into the existing testing framework led to the requirement of running tests from an NUnit test project, using dotnet cli, outside the Assets folder. At the same time we had to keep the tests in Assets for the early stages of development checks.
One of the challenges to achieve this was raised by the fact that the tests had been written using an API library that only worked from within Unity Runtime so we could not call it when we ran the tests without Unity.
We created a script which copied all the tests outside the Assets folder and we ran them as a separate NUnit test project. We used HTTPClient API instead of the Unity dependent library. Furthermore, in the test code, we needed to handle various cases regarding what can be executed in the Unity Editor and what cannot.
When running tests on real devices, it was necessary for all packages and libraries used in the Assets folder to be available both inside the Assets folder and outside it. In order to achieve that, we saved relevant .dll files as artifacts from the Jenkins job building the game then referenced them in the new NUnit project, in .csproj.
Deal with connectivity between AltDriver and AltTester® Desktop
Jenkins was set up on a machine where one Android and one iOS device were connected through USB. To make sure that AltTester® Desktop differentiates the builds, we adapted the instrumentation job to set custom app names for each platform.
To facilitate communication between AltDriver from the game and AltServer on Android, using adb for reverse port forwarding and employing a build with the host set to localhost is enough.
The situation is different for iOS devices since IProxy lacks the capability for setting up reverse port forwarding. To establish the connection between the game and the AltServer, we used the IP address of the machine where the AltServer was running.
When integrating the execution of tests on a cloud service, we utilized BrowserStack Local tunnel. This tool simulates the device being on the same network as the machine running AltServer. For this setup, we employed an instrumented build for iOS with localhost.
Use Appium to create tests which interact with native elements
Nowadays, applications integrate with social network apps, necessitating the testing of situations where users exit the application and connect to other apps. Moreover, we’ve created tests using Appium for scenarios involving In-App Purchases.
Successfully overcoming the challenge of transitioning control between AltDriver and Appium driver, we managed to construct more complex test scenarios.
Save test results in a report with screenshots & logs from Unity
In addition to saving test results in a report, the Allure framework provides the option to append screenshots and log files to the report. In order to provide additional information to developers when debugging, we looked for solutions to save the Unity exceptions triggered while executing tests on the real devices. Besides filtering device logs captured with tools like adb logcat or idevicesyslog, we employed AltDriver’s method: AddNotificationListener.
We paved the way to a more reliable and scalable game test automation solution.
Our collaboration with the client was a journey of empowering their testing capabilities, helping them achieve the proposed milestones and contributing to the advancement of their testing stage within the development cycle. More specifically:
✔️ We successfully upgraded the version of AltTester® framework
✔️ We seamlessly integrated Appium for automated testing on real devices within a CI environment
✔️ We incorporated the Allure Framework to enhance test reporting
✔️ We enabled the flexibility of running tests on a cloud service provider like BrowserStack.
Through these strategic measures, we not only achieved our objectives, but also paved the way for a more robust and efficient testing infrastructure, ultimately contributing to the client’s success in delivering high-quality software.
Struggling with game test automation?
We can find a solution suitable for you!
or call +40 371 426 297