AI Agents Learn to Write Acceptance Tests That Actually Catch Software Bugs

Software requirements are notoriously slippery creatures. A sentence like “the system should respond quickly” can mean entirely different things to the product manager who wrote it, the developer who reads it, and the tester who must verify it. When large language models are asked to turn such ambiguous requirements into executable acceptance tests, they can produce test suites that look impressive but quietly encode the wrong interpretation of what the software is supposed to do. A new study published in Frontiers in Artificial Intelligence by Yunjiao Shi of The University of Hong Kong and Rui Song of the University of Cambridge confronts this problem head-on, presenting an agent-based workflow called ReqTestAgent that grounds every generated test in actual execution rather than in the model’s confident prose.

The core insight behind the research is a distinction that the authors argue is often blurred in evaluations of AI-generated tests: behavioral compatibility is not the same thing as fault-detection strength. A test suite might pass against a reference implementation, meaning it is compatible with accepted behavior, yet still be too weak to catch real defects. Conversely, a suite that fails the canonical implementation might be doing so because its assertions conflict with the accepted behavior rather than because it has detected a genuine fault. ReqTestAgent treats canonical compatibility as a prerequisite: only after generated tests pass against a reference implementation does the system interpret mutation-based measures of test strength. This ordering matters, because a mutation score computed on incompatible tests is essentially meaningless noise.

ReqTestAgent is an execution-grounded workflow that stitches together four components: requirement interpretation, retrieval of testing knowledge, test generation, and feedback from canonical execution and mutation testing. When a requirement arrives, the agent first produces an explicit interpretation record, documenting what it believes the requirement means. It then retrieves relevant testing knowledge to inform how the behavior should be probed, generates candidate test suites, and finally submits those suites to a feedback loop in which they are executed against a canonical implementation and evaluated with mutation testing. The interpretation records are linked to executable checks, creating an auditable chain from the model’s reading of a requirement to the assertions that verify it.

The evaluation design is unusually careful for this kind of study. The authors built 100 requirement instances from 25 tasks drawn from the Orchid and BigCodeBench benchmarks, and for each task they created four variants: lexical, semantic, syntactic, and vagueness. This structure lets them probe how different types of requirement ambiguity affect test generation. Crucially, the evaluator had access to a reference implementation, so the ground truth of what the software should do was available for checking. The comparison set included three generation baselines, their verifier-enhanced counterparts, and component ablations of ReqTestAgent itself, allowing the team to isolate which parts of the workflow actually contribute to performance.

The headline numbers are striking. The three generation baselines failed the canonical implementation on 51 to 68 percent of instances, meaning that more than half of the time, tests produced by straightforward language-model generation simply did not match accepted behavior. All verifier-enhanced methods, by contrast, achieved a 100 percent canonical pass rate, demonstrating that adding a verification step is sufficient to fix compatibility. But compatibility alone is not enough. ReqTestAgent achieved 92 percent qualified suites and a macro-averaged effective mutation score of 0.678, compared with 0.655 to 0.670 for the verifier-enhanced baselines. That gap is modest but consistent, and it points to something beyond verification that ReqTestAgent contributes to test quality.

The ablation results make the contribution of verification unmistakable. When the verification component was removed from ReqTestAgent, its effective mutation score collapsed to 0.216, a dramatic drop that shows execution feedback is the principal validity improvement in this controlled setting. Across all instance-mutant evaluations, ReqTestAgent killed 492 of 756 mutants, a pooled rate of 0.651 that weights instances by their mutant counts. Mutation testing works by introducing small artificial defects into the reference implementation and checking whether the test suite detects them, so a higher kill rate means the tests are genuinely probing behavior rather than merely executing code paths without meaningful assertions.

Perhaps the most illuminating part of the study is a comparative execution trace showing how explicit domain assertions detect faults that the verifier-enhanced baselines miss. The baselines, even with verification, tend to produce tests that pass against the canonical implementation but lack the specificity to distinguish subtly different behaviors. ReqTestAgent’s interpretation records, by contrast, push the model to commit to a particular reading of an ambiguous requirement and then encode that reading as concrete assertions. When those assertions are wrong, the canonical execution feedback reveals the mismatch; when they are right, they become powerful probes that catch mutants the baselines let slip through.

The implications reach beyond the benchmark. Acceptance testing is the point in software development where human intent meets executable code, and it is precisely where requirement ambiguity does the most damage. If language-model agents are to be trusted with generating tests from natural-language requirements, they need mechanisms that expose their interpretations and subject them to execution-based scrutiny. ReqTestAgent’s workflow links interpretation records to executable checks, creating a traceable path from the model’s reading of a requirement to the assertions that verify it. This kind of grounding could become a template for how AI agents handle other tasks where ambiguity is endemic, from legal document analysis to scientific data interpretation.

The study also establishes a methodological point that the authors argue should reshape how AI-generated tests are evaluated: canonical compatibility must be interpreted as a prerequisite for reading mutation-based test strength. A mutation score computed on a test suite that fails the canonical implementation conflates behavioral mismatch with fault detection, producing numbers that cannot be meaningfully compared across systems. By ordering evaluation so that compatibility is checked first and strength is measured only on compatible suites, the evaluation framework avoids this conflation. The authors suggest that this ordering should become standard practice in studies of automated test generation, where the temptation to report a single aggregate score can mask fundamental validity problems.

As language-model agents move deeper into software engineering workflows, the boundary between what a model believes a requirement means and what the software actually does becomes the critical fault line. ReqTestAgent demonstrates that execution feedback, rather than model self-assessment, provides the principal validity improvement in this setting. The workflow’s combination of explicit interpretation records, testing-knowledge retrieval, and canonical execution feedback offers a concrete answer to a question that will only grow more urgent as AI systems take on responsibility for verifying software behavior: how do you know that an agent’s test actually tests what you meant? The answer, this research suggests, is to make the agent commit to its interpretation, encode it as executable assertions, and then let the code itself have the final word.

Subject of Research: An execution-grounded language-model agent workflow for generating acceptance tests from ambiguous software requirements, evaluated against canonical implementations and mutation testing benchmarks.

Article Title: ReqTestAgent: execution-grounded language-model agents for acceptance testing under requirement ambiguity

Article References: Shi, Y., & Song, R. (2026). ReqTestAgent: execution-grounded language-model agents for acceptance testing under requirement ambiguity. Frontiers in Artificial Intelligence, 9, Article 1960601. https://doi.org/10.3389/frai.2026.1960601

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1960601

Keywords: large language models, acceptance testing, automated test generation, mutation testing, requirement ambiguity, software engineering, ReqTestAgent, execution-grounded, language-model, agents, acceptance, testing