AI Agents Are Being Graded Wrong: Landmark Audit Finds No Benchmark Controls All Key Threats

Large language models no longer simply answer questions. They plan multi-step tasks, call external tools, browse environments, and act autonomously across long chains of decisions. Yet according to a sweeping new systematic survey published in Artificial Intelligence Review, the benchmarks used to grade these AI agents produce scores that are far less meaningful than the field assumes. The study, led by Vinoth Nageshwaran of the University of the Cumberlands with colleagues at Indiana University of Pennsylvania, Ton Duc Thang University, and Elizabeth City State University, argues that a benchmark number is not self-interpreting: what it means depends entirely on which capability it measures, how that capability is scored, and where the agent is tested when it is measured.

The research team built their analysis on a reproducible systematic mapping review conducted under the PRISMA-ScR standard, the accepted protocol for systematic reviews in health and computer science research. After a rigorous screening pipeline, 259 primary studies formed the core analytical corpus, with 294 studies in the released living-review corpus. The authors are unusually candid about the limits of their own method: primary-arm records were screened by a single audited automated pass, while human double-screening with an inter-rater agreement of kappa equal to 0.65 was applied only to a supplementary arm. They explicitly describe broad-map attribute shares as provisional heuristic estimates and stress that the corpus is a carefully constructed mapping sample, not a census of the entire field. That kind of methodological honesty is rare in a literature often criticized for overclaiming.

The survey’s first major contribution is a crisp conceptual boundary called the dependent-step test. The test demarcates genuine autonomous agentic evaluation from ordinary static natural-language-processing evaluation and from prompt engineering. In essence, an evaluation qualifies as agentic only when later steps in a task depend on the outcomes of earlier steps, so that errors compound and the agent must recover, replan, or abandon a strategy mid-trajectory. A model that answers a thousand independent trivia questions is being tested very differently from one that must book a flight, notice the payment failed, diagnose why, and retry with a corrected form. The dependent-step test gives researchers a principled way to decide whether a benchmark is actually measuring agency or merely repackaging static question answering.

The second contribution is a meta-taxonomy that places every agent benchmark in a three-pillar coordinate system: capability, meaning what the benchmark measures; scoring paradigm, meaning how performance is judged; and environment topology, meaning where the agent operates. This framework matters because two benchmarks with identical headline scores may probe entirely different faculties. One may reward factual recall in a sandboxed text environment, while another measures tool orchestration in a live operating system where a single mistyped command can cascade into failure. By locating each benchmark along these three axes, the taxonomy lets researchers compare evaluations like coordinates on a map rather than as isolated leaderboard entries, exposing blind spots where entire capability regions remain untested.

Third, the authors paired an evidence map over the full corpus with a purposive, hand-verified, venue-verified landscape matrix of seventeen prominent benchmarks, achieving an inter-rater reliability of kappa equal to 0.77, a level conventionally regarded as substantial agreement. This matrix functions as a deep-dive companion to the broad map: where the corpus-wide analysis offers breadth with provisional estimates, the seventeen-benchmark matrix offers verified depth on the evaluations that most actively shape the field’s self-image. The contrast between the two instruments is itself instructive, showing how much confidence a reader should place in each kind of claim.

The survey’s most striking finding emerges from its critical analysis of that verified matrix. The authors identify a structural trilemma facing agent evaluation: data contamination, non-determinism, and execution cost. Data contamination occurs when benchmark tasks leak into training data, inflating scores without genuine capability. Non-determinism arises because agentic trajectories involve stochastic model behavior and environment interactions, so the same agent can score differently across runs. Execution cost reflects the computational expense of running agents through long, tool-using trajectories, which discourages the repeated trials needed to tame non-determinism. Among the seventeen prominent, verified benchmarks, the study found that none reports evidence that all three threats are jointly controlled, and none reports a complete standardized run-cost record, a 0 out of 17 result that should unsettle anyone quoting leaderboard numbers.

The implications of that 0 out of 17 finding ripple outward. When contamination is uncontrolled, a high score may reflect memorization rather than reasoning. When non-determinism is unquantified, a single reported run may be a lucky draw rather than a stable estimate of ability. When cost is unrecorded, results cannot be reproduced at reasonable expense, and the community cannot even assess whether an evaluation is practical to repeat. Together these gaps mean that many celebrated comparisons between competing agents may be statistically fragile, and that the field’s rapid narrative of progress rests on measurements whose error bars are largely unknown. The trilemma is structural because addressing any one threat tends to worsen another: more repetitions to handle non-determinism raise cost, and larger task sets that resist contamination raise it further.

The transparency of the review process itself deserves attention as a model for the field. The authors released a companion repository containing their pipeline, including a build script that reproduces every reported count against a committed HTTP response cache, yielding byte-identical, MD5-verified outputs with zero live API calls. They document per-source query status, flagging databases where only counts could be verified and one major index that remained unresolved for lack of credentials. They even ran a capture-recapture diagnostic between OpenAlex and Crossref, then declined to treat its output as a recall estimate because the two indices are heterogeneous targeted searches rather than independent random samples, a violation of the estimator’s core assumptions. A probe-set validation instead confirmed that canonical benchmarks were captured completely, with Scopus corroborating the result. This is bibliometrics done with unusual care.

Looking forward, the survey proposes a five-direction roadmap that reframes progress not as the invention of yet more individual benchmarks but as a shift toward standardized, reliability-first, cost-aware evaluation infrastructure. In practice that means community-agreed protocols for reporting contamination checks, variance across repeated runs, and the compute cost of every evaluation, so that a benchmark score arrives with the same kind of measurement metadata that a physics experiment or clinical trial would demand. The roadmap effectively asks the field to grow up: to treat agent evaluation as a measurement science with known instruments, calibrated error, and reproducible procedures, rather than as a proliferation of leaderboards each with its own unstated assumptions.

For an industry pouring billions into agentic AI products, the stakes could hardly be higher. Autonomous agents are beginning to write code, manage workflows, and interact with real systems on behalf of users, and deployment decisions increasingly cite benchmark performance as evidence of safety and competence. If, as this survey shows, no prominent benchmark currently demonstrates joint control over contamination, non-determinism, and cost, then the numbers guiding those decisions are weaker than they appear. The study does not claim agents are failing; it claims we cannot yet reliably tell. Turning evaluation from a competitive spectacle into trustworthy infrastructure, the authors argue, is the prerequisite for knowing what today’s agents can actually do, and the open repository accompanying the paper offers the community a concrete place to start.

Subject of Research: Evaluation and benchmarking of large language model agents

Article Title: Large language model agent evaluation and benchmarking: a systematic survey, meta-taxonomy, and critical research roadmap

Article References: Nageshwaran, V., Ezekiel, S., Tran, T. T., & Lakshmi Narasimhan, V. (2026). Large language model agent evaluation and benchmarking: a systematic survey, meta-taxonomy, and critical research roadmap. Artificial Intelligence Review. https://doi.org/10.1007/s10462-026-11678-4

Image Credits: AI Generated

DOI: 10.1007/s10462-026-11678-4

Keywords: large language models, AI agents, benchmarking, agent evaluation, meta-taxonomy, systematic review, data contamination, non-determinism, execution cost, PRISMA-ScR, evaluation infrastructure, Artificial Intelligence Review