Once Gamescom wrapped, on the train back to the airport, we started the usual: what went well, what was different from last year, what we could have done better, what to do differently next year. The conversation stayed with me longer than the trip did. In the days after, I kept coming back to a handful of points, trying to go deeper into their meaning. In this article, I want to touch on those aspects and why I think they matter.
Faster to build, not simpler to plan
In this new AI era, in our case with the AI Extension and CLI, it’s much faster and more accessible to do test development. This was noticeable in the Gamescom discussions too. We are approached by more and more QA companies that want to start doing test automation. Studios without a dedicated QA team, or with QA stretched thin against a fast release schedule, are feeling more confident about allocating resources to this. But speed and accessibility come with a catch worth saying out loud: making it easier to write tests doesn’t necessarily fix our issues. Our critical thinking skills are now practised more extensively when reviewing strategies, test plans, and test automation approaches. Each output suggested by an AI has to be questioned so we don’t end up with lots of tests but only a few actually helpful.
Why determinism matters in test automation
Using LLMs at each testing step, or using AI to write automation scripts, are both valid approaches, and each has its own merits. However, for repetitive tests (sanity checks, smoke suites, regression tests), we believe a clear, deterministic output, one where the clean passing result is the same each day, is what makes them trustworthy, without needing extra output checking or AI review on top. It also means the people running the tests can spend their attention on the failures that matter instead of chasing false positives, which is one of the biggest advantages of test automation to begin with.
Where AI actually helps, and where it costs us
Player backlash against AI in gaming is increasingly visible. My guess is that it’s mostly aimed at what players can actually see: AI-generated art, voice, dialogue. But during Gamescom Dev, which Ru attended, she noted that some studios are going AI-free specifically because of this backlash. This got us thinking about how far that caution should extend behind the scenes, in coding, testing, and so on, which of course leads to the never-ending question of how much AI helps versus when it no longer does, across just about every industry.
What do we gain from using AI in our work, or even in day-to-day life (faster game releases, for instance), versus what we lose? I find this a genuinely useful question. If we can answer it, even partially, it opens up a better follow-up question: how do we keep that loss as small as possible? We touched on this briefly already, faster test implementation versus the need for careful review to preserve quality, but the list of such tradeoffs only gets longer. Coming back to games specifically, each studio is best placed to judge which way that balances tips for them, and how to best protect the quality of their games while doing so.
For studios that choose to go entirely AI-free, the good news is that AltTester’s core testing functionality never depends on AI to run, the AI Extension is optional.
More genres are opening up to test automation
Each year the range of studios being open to test automation gets wider and wider. If in the past, some game genres were thought (by studios) not to be a fit for test automation, they now seem more and more open to try it. They start to see the value they bring, and the discussions are turning from “it doesn’t apply to our context” to “okay, how could we test this and that scenario?” Open world RPGs, live multiplayer titles, casual and puzzle games, each one raises different challenges which turn the conversation mood into problem-solving quests rather than trying to convince anyone that test automation is valuable (no matter the tooling you try).
AltTester® team, off the clock
What this trip means to me, beyond Gamescom itself, each year (not just this August), is the time itself. Besides the fact that we get to spend time with Ru in person, since she’s remote from the core team, the time between meetings is where we end up catching up informally, in bits and pieces, eventually talking about what could work better in the tool, how we run things, and what we could do better or differently. Those are the kinds of discussions that happen spontaneously when you spend time together, discussions that would otherwise need to be scheduled and timeboxed during a regular working week.
