The authors frame their argument against the backdrop of an extraordinary acceleration in medical AI capabilities. Large language models trained on vast corpora of biomedical text have demonstrated expert-level performance on medical licensing examination-style question sets, prompting sweeping public claims about their potential to transform diagnosis, triage and clinical decision support. Foundational work published in Nature, including studies on generalist medical AI and the demonstration that large language models encode substantial clinical knowledge, has fueled a narrative of imminent transformation. Yet, as the commentary emphasizes, answering multiple-choice questions correctly is a fundamentally different task from managing a complex patient across an entire episode of care. Benchmarks typically evaluate isolated, static problems with a single correct answer, while clinical practice involves ambiguous presentations, incomplete data, longitudinal reasoning, ethical judgment and accountability for outcomes.

One of the central technical concerns raised in the paper is the problem of construct validity: whether a benchmark actually measures the capability it purports to represent. A model that scores highly on a question-answering dataset may be exploiting statistical shortcuts, memorized test items or surface linguistic patterns rather than genuine clinical reasoning. Contamination, in which benchmark questions leak into the training data of frontier models, further inflates reported scores and makes year-over-year comparisons meaningless. The authors also highlight that most benchmarks lack the ecological validity of real clinical environments. Recent efforts to address this shortcoming, such as virtual electronic health record environments like MedAgentBench, which evaluates the ability of medical language model agents to navigate realistic EHR workflows, and large-scale, continually refreshed initiatives like HealthBench and MedBench v4, represent important steps toward more realistic evaluation. However, the commentary argues that even these sophisticated benchmarks remain proxies, and proxy evidence alone cannot justify claims of clinical readiness.

The paper situates its argument within a broader and increasingly well-documented evidence base. A systematic review of 39 clinical large language model benchmarks, published in the Journal of Medical Internet Research, documented a persistent knowledge-practice performance gap, showing that models excelling on knowledge assessments frequently fail on tasks that approximate real clinical practice. Earlier work in the BMJ, which systematically compared the design and reporting standards of deep learning studies claiming performance parity with or superiority over clinicians, found that many such claims rested on methodologically weak foundations, including non-prospective designs, inappropriate comparators and inadequate handling of dataset shift. Analyses of FDA clearances for AI-enabled medical devices have similarly revealed that most approvals rely on retrospective studies rather than prospective clinical evidence, and investigations of insurance claims data have shown that actual clinical adoption of many cleared devices has lagged far behind their regulatory status. Together these findings suggest that regulatory clearance, benchmark performance and genuine clinical utility form three distinct tiers of evidence that are frequently and dangerously conflated in public discourse.

To address this conflation, the commentary proposes that claims of clinical readiness be treated as governed assertions rather than marketing statements or casual summaries of benchmark scores. Drawing on the emerging documentation infrastructure of the machine learning field, including model cards for transparent reporting of model characteristics and the more recent BenchmarkCards framework for standardized documentation of benchmark datasets, the authors argue that every benchmark result should be accompanied by explicit statements about what the result does and does not license a developer to claim. A high score on a medical question-answering benchmark, under such a framework, would automatically trigger standardized caveats about retrospective, text-only, single-turn evaluation, and would explicitly disclaim readiness for autonomous clinical deployment. This approach shifts the burden of interpretation from end users, who often lack the technical expertise to assess benchmark limitations, to the developers and publishers who generate and disseminate the scores.

The authors also connect their governance proposal to the established ecosystem of clinical reporting guidelines. Frameworks such as DECIDE-AI, developed for the early-stage clinical evaluation of AI-driven decision support systems; CONSORT-AI and SPIRIT-AI, which extend clinical trial reporting and protocol standards to AI interventions; TRIPOD+AI, which governs the reporting of clinical prediction models built with regression or machine learning methods; and FUTURE-AI, an international consensus guideline for trustworthy and deployable healthcare AI, collectively define a pathway from laboratory evaluation to clinical evidence. The commentary argues that benchmark results should be positioned explicitly at the very beginning of this pathway, functioning as justification for investment in prospective evaluation rather than as evidence of deployability in themselves. The transition between these stages, the authors contend, is precisely where governance has been weakest and where overstated claims have caused the most harm to public trust.

A recurring theme in the paper is the principle, articulated in prior work by Youssef and colleagues, that one-time external validation should be replaced with recurring local validation. Because AI models are sensitive to the distributional characteristics of the data on which they are evaluated, performance measured at one institution or on one patient population may not transfer to another. The commentary extends this logic to benchmarks themselves, suggesting that institutions considering deployment should treat published benchmark scores as hypotheses to be locally verified rather than as transferable guarantees. Institutional AI governance committees, modeled on frameworks such as the responsible-use guidelines developed at Mass General Brigham, would play a central role in this process, reviewing vendor benchmark claims, commissioning local silent-mode evaluations, and establishing monitoring protocols that continue for the operational lifetime of the deployed system. Pragmatic trial playbooks for monitoring ambient AI in clinical practice offer one template for how such continuous surveillance can be organized.

The commentary is particularly attentive to the human and organizational dimensions of AI deployment. It cites recent scholarship on protecting clinical value judgment in the age of AI, warning that benchmark-centric narratives risk reducing medicine to an optimization problem and marginalizing the contextual, ethical and preference-sensitive judgments that clinicians make daily. The authors also address the persistent gap between developers and implementers in health AI, noting that the incentives of commercial model development, which reward rapid release and headline performance, are structurally misaligned with the careful, iterative and locally grounded evaluation that clinical integration demands. Bridging this gap requires shared standards for evidence, transparent documentation and clear accountability, which is precisely what a governance framework for readiness claims would provide.

Regulatory context forms another pillar of the argument. The U.S. Food and Drug Administration’s draft guidance on the lifecycle management of AI-enabled device software functions reflects a shift toward total product lifecycle oversight, in which performance must be monitored and re-verified as models are updated and as clinical conditions evolve. The commentary argues that benchmark-based readiness claims should be brought within this lifecycle paradigm: just as a deployed device requires post-market surveillance, a published benchmark claim should carry an obligation to update or retract the claim when the underlying benchmark is shown to be contaminated, outdated or invalid. The rapid obsolescence of medical benchmarks in the era of continuously trained foundation models makes such maintenance an ethical necessity rather than an academic nicety. Related calls for principles to guide clinical AI readiness and move from benchmarks to real-world evaluation, published in Nature Medicine by Azad, Krumholz and Saria, underscore that this concern is now shared across leading institutions internationally.

The significance of the commentary lies less in any single technical recommendation than in its reframing of the problem. The authors do not argue that benchmarks are useless; on the contrary, they view rigorous, transparent, continually maintained benchmarks as indispensable infrastructure for the field, and they point to specialty triage benchmarking and competition initiatives as examples of how structured evaluation can advance medical AI. Their claim is narrower and more actionable: benchmark performance is a necessary but radically insufficient condition for clinical deployment, and the language used to communicate benchmark results must be governed with the same rigor that governs the models themselves. In an era when a single leaderboard result can propagate globally within hours and shape procurement decisions in hospitals that lack the expertise to interrogate it, treating readiness claims as governed, documented, versioned and accountable statements may prove to be one of the most consequential interventions available for ensuring that the promise of medical AI is realized safely. The work was supported by the Clinical Characteristic Specialty Disease Construction Project of the Shanghai Putuo District Health System, and the authors report no competing interests.