Agentic AI in Medicine Needs Clinical Testing Harnesses, Researchers Warn

Artificial intelligence in hospitals has long followed a familiar pattern: a model produces a single output, such as a risk score or a diagnostic suggestion, and a clinician reviews it before anything happens to the patient. A new letter in the Annals of Biomedical Engineering argues that this entire evaluation framework is about to be upended. Ethan Waisberg and Joseph W. Guarnieri, affiliated with the University of Cambridge, the Blue Marble Space Institute of Science, and the Guarnieri Research Group, contend that agentic artificial intelligence systems, which can plan and execute multi-step tasks on their own, break the assumptions underlying how medicine has traditionally validated decision-support technology. Their proposed response is a concept they call the clinical testing harness, and it may determine whether autonomous AI earns a place at the bedside or remains confined to the laboratory.

The core of the problem lies in what the authors describe as agent scaffolding. This is the software infrastructure wrapped around a large language model that transforms passive text generation into active, multi-step execution. With scaffolding in place, an AI system does not merely answer a question. It can retrieve patient records, invoke external tools such as calculators, databases, or imaging analyzers, interpret the results of those tool calls, and then take further actions based on what it finds, all before a human clinician sees anything at all. The relevant unit of clinical risk, the authors argue, therefore shifts from a single inference to an entire trajectory of actions. A model might compose a flawless summary of a chest radiograph and still fail catastrophically if it queries the wrong data source, misconfigures a drug-dosing tool, or chains a series of individually reasonable steps into a collectively dangerous plan.

Waisberg and Guarnieri organize the resulting challenges into four categories, each of which exposes a weakness in current approaches to responsible AI development in medicine. The first is silent error propagation across multi-step tasks. In a single-output system, an error appears in the one thing the clinician reviews. In an agentic system, a mistake made in step two, say a misparsed laboratory value, can silently contaminate steps three through ten, and by the time a conclusion reaches human review, it may look polished and internally consistent even though it rests on corrupted intermediate work. The error is not merely hidden; it is laundered through subsequent stages of reasoning until it becomes difficult to detect.

The second challenge concerns oversight that reviews conclusions rather than processes. Traditional clinical decision support places a single recommendation in front of a clinician, who evaluates it against their own expertise. But agentic systems present the end product of a long chain of retrievals, tool invocations, and intermediate judgments. A clinician who signs off on the final answer has effectively approved a process they never observed. The authors point out that this compounds a well-documented human tendency: automation bias, the inclination to accept machine-generated recommendations without sufficient scrutiny, has been recognized in the medical informatics literature for over a decade. When the machine does far more work invisibly, the bias has more room to operate.

The third challenge is validation that does not survive changes to the model or the tooling. In conventional medical AI, a model is trained, validated on a fixed dataset, and then deployed with its behavior frozen. Regulatory pathways and clinical trust both rest on the assumption that what was tested is what runs. Agentic systems violate this assumption structurally. A developer might swap the underlying language model for a newer version, update a connected tool, or modify the scaffolding logic, and the behavior of the entire system can change in ways that invalidate prior testing. The authors note that medicine has already lived through a cautionary version of this problem with static models: the widely implemented Epic sepsis prediction model, when externally validated by independent researchers, performed far worse than its developers reported, missing the majority of sepsis cases while generating a flood of false alarms. If a frozen, single-purpose model can degrade that badly under real-world conditions, the authors imply, a self-directed system whose components constantly evolve poses a categorically harder validation problem.

The fourth challenge is scope that widens faster than the evidence supporting it. Agentic AI is generality by design. The same scaffolding that lets a system answer a question about diabetic retinopathy can, in principle, let it draft referral letters, order tests, adjust medication lists, or triage messages, and the temptation in deployment is to keep expanding what the agent is allowed to do. But each expansion of scope requires its own evidence base, and the pace of capability demonstration is outrunning the pace of clinical validation. The letter cites recent work including MedAgentBench, a virtual electronic health record environment for benchmarking medical LLM agents, and AgentClinic, a multimodal benchmark for tool-using clinical AI agents, alongside a validated autonomous oncology decision-making agent, as signs of a field whose technical ambitions have sprinted ahead of its safety infrastructure.

The authors’ solution is not a new invention but an adaptation of tools medicine already trusts. They propose the clinical testing harness, a structured evaluation environment with four components. The first is a scenario library built from clinical edge cases, the rare, ambiguous, and dangerous presentations where decision support is most likely to fail and most likely to matter. The second is full-trajectory observability, meaning that evaluators can inspect not just the final output but every retrieval, tool call, and intermediate decision the agent made along the way. This directly answers the silent propagation problem: if every step is logged and examinable, an error introduced in step two can be traced before it metastasizes into a confident, polished, and wrong conclusion.

The third component is explicit escalation testing, which evaluates whether the agent recognizes the limits of its own competence. A safe clinical agent must not only answer questions correctly; it must know when a situation demands a human, and must reliably hand control to a clinician when uncertainty is high, the case is atypical, or the stakes are severe. The fourth component is staged evidence thresholds tied to scope of practice. Instead of a binary approval, an agent would earn capabilities incrementally: broad evidence might support letting it retrieve and summarize information, while higher-stakes actions such as influencing treatment decisions would remain gated until the system had demonstrated reliability at each preceding level. This mirrors how medicine itself grants privileges, and it prevents the scope-expansion problem by making each widening of responsibility contingent on demonstrated performance.

The elegance of the proposal, and arguably its feasibility, comes from its institutional ancestry. Waisberg and Guarnieri observe that medicine already possesses every ingredient of a testing harness in mature form. Simulation-based training subjects clinicians to edge cases before they touch patients. Credentialing and scope-of-practice rules tie permitted actions to demonstrated competence. Morbidity and mortality conferences systematically dissect adverse outcomes to find process failures rather than merely blaming individuals. The clinical testing harness is these institutions translated into software: scenario libraries are simulation, staged thresholds are credentialing, and full-trajectory review is the M and M conference applied to machine reasoning. The letter, published on 23 September 2026 and reviewed under Associate Editor Joel Stitzel, reports no funding and no conflicts of interest, and draws on the authors’ prior work on large language models in medical imaging, concerns about ChatGPT in academia and medicine, and the problem of correlated failure in delegated AI supervision.

Whether the vision takes hold will depend on regulators, developers, and health systems accepting an uncomfortable premise: that the familiar benchmark scores of a language model tell clinicians almost nothing about the safety of an agent built around it. The past decade of clinical AI offers a sobering precedent, from sepsis models that faltered under external validation to the recognition that human oversight is neither automatic nor reliably vigilant. Agentic systems promise genuine gains, potentially extending expert-level reasoning into settings where specialists are scarce, but only if the unit of testing expands to match the unit of risk. A trajectory of actions demands a trajectory of evidence, and the letter’s central claim is that medicine need not invent a new safety culture to provide it. It need only point the tools it already trusts at a new kind of clinician, one that never tires, never second-guesses, and acts before anyone is watching.

Subject of Research: Responsible development and clinical evaluation of agentic artificial intelligence systems in medicine

Article Title: Agentic AI in Medicine: Challenges for Responsible Development and the Case for Clinical Testing Harnesses

Article References: Waisberg, E., & Guarnieri, J. W. (2026). Agentic AI in Medicine: Challenges for Responsible Development and the Case for Clinical Testing Harnesses. Annals of Biomedical Engineering. https://doi.org/10.1007/s10439-026-04381-6

Image Credits: AI Generated

DOI: 10.1007/s10439-026-04381-6

Keywords: agentic AI, clinical decision support, AI safety, large language models, clinical testing, medical AI validation, automation bias, digital health, AI governance, human-in-the-loop, trustworthy AI, biomedical engineering