AI QA agents: the gains, the risks and the human judgment

AI agents can help a QA team explore more scenarios, prepare tests and investigate failures. I would keep experienced QAs responsible for interpreting the evidence and assessing release risk. Giving a system more tasks to execute also gives the team more results to validate.

Here, I use agent to mean a system that can pursue a testing goal through several steps: choose actions, use tools, inspect results and adjust its next move. A model drafting a test and a bot exploring a game are different systems. Evidence about one does not establish what the other can do.

The World Quality Report 2025-26 found that 89% of responding organizations were piloting or deploying generative AI in quality engineering, while 15% had implemented it across the enterprise. Those are survey responses from organizations in that report, rather than a census of every QA team.

For me, that gap points to the practical work after a successful demo: integrating tools, deciding what counts as a useful test, protecting data and keeping maintenance manageable. A convincing run on one build needs to become a repeatable process.

The useful question is where an agent can remove repetitive effort while leaving the team with stronger evidence. I would begin with a narrow task, a known baseline and a clear definition of failure. That makes the benefits and the remaining work visible.

Meta's TestGen-LLM offers a concrete example. Its engineers reported that 75% of generated tests built correctly, 57% passed reliably and 25% increased coverage in an evaluation on Instagram Reels and Stories. At separate test-a-thons, engineers accepted 73% of the tool's recommendations for production.

Those percentages describe different stages and evaluations. The 73% acceptance rate applies to recommendations after filtering. Tests had to build, run reliably and improve coverage before engineers reviewed them. It is a company-authored industrial study, with a deliberately constrained task.

I would borrow that design principle: generate candidates cheaply, reject unusable ones automatically and spend human attention on the survivors. Coverage can guide that filtering, although it still takes judgment to decide whether a test checks something meaningful.

Games also have examples beyond language models. EA SEED's 2023 research describes agents trained through reinforcement and imitation learning to navigate game environments and discover glitches. These techniques differ from an LLM agent writing tests, and the results do not transfer automatically to another game.

For a mobile team, I would consider drafting scenario variations, organizing failure evidence and exploring repeatable paths as starting points. Each output needs a check: does this scenario represent a player risk, can the failure be reproduced, and does the report contain enough evidence to act on?

The hardest part of testing is often deciding what should happen. That decision is sometimes called the test oracle. An agent that reads the implementation can mistake the current behavior for the intended behavior and write an assertion that preserves a bug.

A research preprint studying 24 Java repositories found that LLM-generated oracles mainly captured actual program behavior, which made bug detection difficult. That is a finding about those methods and repositories. I take it as a reason to review expected results carefully.

Consider a fictional mobile game: a purchase succeeds, but the player receives the reward twice. A test generated from the existing code might assert that two rewards appear. The test passes while protecting the defect. A QA who checks the purchase rule has an independent basis for rejecting that result.

In Stack Overflow's 2025 developer survey, 46% distrusted AI output accuracy and 3% highly trusted it. This is a developer survey, rather than a game-QA survey. It helps explain why plausible output needs verification.

I would review changes to assertions as closely as changes to production code. A repaired test that stops checking the reward amount needs an explanation. The same applies to skipped scenarios, unexplained retries and failures relabeled as expected.

There are operational costs too. I would budget for execution, review and maintenance, and decide which builds, logs or player data may reach an external service. Saving time on generation can lose its value if the team spends longer resolving unreliable results.

An agent can traverse a level and collect signals. Assessing whether its difficulty fits the intended audience involves designers, player feedback and QA judgment. The sources reviewed here do not establish that an agent independently validates fun or game feel. That is a limit of this evidence, rather than proof that such capabilities will never improve.

I would ask a QA to examine what the numbers leave out: whether feedback makes sense after a failed move, whether touch controls feel responsive, whether a tutorial explains the next action and whether interruptions leave the player in a recoverable state.

Real-device performance can also be measured through automation. The requirement is to run on relevant hardware and inspect the measurements. A simulated playthrough by itself does not tell me about sustained frame pacing, heat, memory pressure or a failure specific to one phone.

Google Play defines a bad-behavior threshold when at least 8% of daily active users on a single device model experience a user-perceived crash. Exceeding that threshold can reduce discoverability on affected devices and lead to a store-listing warning. The overall threshold across models is 1.09%.

I would therefore choose devices using the audience and observed risks, then combine automated runs with targeted human sessions. Include updates, network interruptions, backgrounding and purchase recovery where relevant. Record the build, device, conditions and evidence so another person can reproduce the result.

Experienced QAs connect a technical symptom to player impact and help decide what deserves attention first. That responsibility becomes especially useful when agents increase the volume of findings.

At Kakunyn, we take an AI-first approach in several areas: we consider early where AI can make a workflow more effective. Our experience also tells us to keep experienced people involved in validation. I see AI as a way to expand the team's capacity and shorten feedback cycles.

We have achieved gains of up to 100% in the speed of specific pipeline steps using AI. That describes our experience in particular situations, rather than a general benchmark. Doubling a step's throughput does not mean doubling the whole pipeline or cutting every cost in half. The aim is to create more room for useful testing and reduce avoidable rework and waste.

Measurement matters. In METR's early-2025 experiment, 16 experienced developers completed tasks 19% more slowly with AI while believing afterward that it had made them 20% faster. In February 2026, METR said newer tools likely provided greater speedups, but selection effects made its follow-up data unreliable. I would measure the complete workflow, including review.

We also use our own tool across several stages of the QA workflow, with an experienced QA validating the results. How that tool supports the process deserves a separate post.

Our mobile game QA services cover functional testing, compatibility and performance. If you are deciding where agents fit in your testing, I would start by mapping the repetitive work, the player risks and the decisions that need accountable review. That gives automation a useful scope and keeps the QA team focused on quality.

If you want to discuss that balance for your game, talk to us through the site's contact page.

My starting checklist would be: