AI Doctors on Trial: New Framework Exposes Hidden Dangers in Medical Language Models

Large language models are being welcomed into hospitals at a pace that has outstripped nearly every safety discipline in modern medicine. They draft discharge summaries, suggest differential diagnoses, and increasingly sit beside clinicians during time-critical decisions. Yet the yardstick most commonly used to judge them—aggregate accuracy on standardized benchmarks—may be quietly concealing the very failures that matter most. A new proof-of-concept study published in the Journal of Medical Systems argues that two models with identical average scores can carry profoundly different risks of issuing a dangerous recommendation, and it proposes a framework designed to make those risks visible before deployment rather than after harm.

The research, led by emergency physicians and clinical informaticians at Xuanwu Hospital of Capital Medical University in Beijing, set out to answer a deceptively simple question: do models with comparable benchmark performance actually share the same safety profile? The team’s answer, drawn from 972 physician-scored responses across six frontier models, is a resounding no. Aggregate accuracy, they found, treats every error as interchangeable, when in reality an anchoring-driven misdiagnosis, a missed rare disease, and a fatal treatment recommendation carry consequences that no single mean score can weigh together.

To build the framework, the researchers assembled 54 diagnostically challenging emergency cases drawn retrospectively from their institution’s records between 2015 and 2024. A panel of 28 emergency physicians nominated 73 candidate cases, which were then winnowed down through departmental case conferences to exclude insufficient clinical information, unconfirmed diagnoses, and duplicated themes. Crucially, 53 of the 54 final cases had never appeared online, minimizing the risk that models had simply memorized the answers from publicly available examination databases. The cases spanned twelve clinical domains and three pre-specified error categories: rare disease recognition, susceptibility to anchoring bias, and identification of iatrogenic risk.

Six general-purpose large language models were evaluated through their official application programming interfaces between August and December 2025: DeepSeek R1, DeepSeek V3.1, Claude Sonnet 4.5, GPT-5.1, Grok 4, and Gemini 3 Pro. Each case was queried three times per model, with queries spaced at least 24 hours apart and presented in randomized order as independent sessions to prevent cross-contamination. The prompts were standardized, text-only, and written in Chinese, with no system-role messages, few-shot examples, or chain-of-thought directives. Two board-certified emergency physicians, blinded to model identity, independently scored every response across three dimensions—diagnosis, next-step recommendations, and treatment planning—on a 1-to-5 scale, yielding a total score between 3 and 15 for each of the 972 responses.

The framework’s central innovation lies in what it chooses to measure. Rather than treating mean performance as the primary endpoint, it elevates two safety-critical metrics: the frequency of catastrophic failures, defined as any scoring dimension receiving a score of 2 or below—indicating a potentially fatal, contraindicated, or grossly inadequate recommendation—and response reproducibility, assessed at two layers. The first layer examines round-to-round stability across repeated queries; the second examines whether a model remains safe on a specific high-stakes case across all three rounds. Inter-rater reliability was moderate, with an intraclass correlation coefficient of 0.719 for total scores, and Bland-Altman analysis showed minimal systematic bias between the two raters.

The results dismantled the comfortable assumption that average scores tell the safety story. A linear mixed-effects model revealed a three-tier performance hierarchy, with Gemini 3 Pro and DeepSeek R1 at the top, Claude Sonnet 4.5, GPT-5.1, and Grok 4 in the middle, and DeepSeek V3.1 at the bottom. But when the researchers examined catastrophic-failure rates, the picture changed dramatically. The two top-tier models produced dangerous recommendations in only 1.5 percent of pooled case-rounds, while the remaining four models failed at 6.0 percent—a four-fold difference that was highly statistically significant. Most strikingly, GPT-5.1, despite sitting comfortably in the middle aggregate tier, fell within the higher-risk catastrophic tier at 7.4 percent, statistically indistinguishable from Grok 4 and DeepSeek V3.1.

Perhaps the most unsettling finding concerned the nature of these failures. Of the 34 case-model pairs that produced any dangerous round, 31—91 percent—were stochastic rather than systematic, meaning the model answered safely on some rounds and dangerously on others in response to the identical prompt. Only three pairs failed consistently across all three rounds. This stochasticity is precisely the failure mode that aggregate metrics cannot capture: a model that reasons correctly twice but recommends a lethal intervention on the third identical query poses a risk invisible to any leaderboard. The study illustrates the point with a chilling clinical vignette—a 76-year-old man presenting with what appeared to be acute ischemic stroke but actually harboring a Stanford Type A aortic dissection, a condition in which antiplatelet or thrombolytic therapy is absolutely contraindicated and potentially fatal. DeepSeek R1 and Gemini 3 Pro each reasoned correctly in two of three rounds, yet in the remaining round dropped dissection from the differential and recommended antithrombotics. GPT-5.1 failed completely in one round, defaulting to a standard stroke protocol.

Reproducibility analysis compounded the concern. GPT-5.1 was the least reproducible model by a wide margin, with a mean within-case standard deviation of 2.69 points—nearly double the next-highest model and the only model whose variability exceeded the pre-specified 1.5-point minimal clinically important difference. Its largest within-case swing reached 11 of 15 points. Meanwhile, error-taxonomy profiling revealed vulnerability patterns that diverged sharply even among aggregate-equivalent models. Anchoring-bias failure rates varied nearly four-fold across models, from 6.9 percent to 26.4 percent, and rare-disease recognition varied six-fold, from 6.1 percent to 37.9%. Notably, the reasoning-oriented models GPT-5.1 and Grok 4 showed the highest and second-highest anchoring susceptibility, undermining the simple assumption that structured reasoning automatically mitigates cognitive bias. Within the DeepSeek family, however, the reinforcement-learning variant R1 showed markedly lower anchoring than its supervised fine-tuned sibling V3.1, hinting that post-training methodology may shape safety profiles in ways architecture labels do not.

The study also carries a consequential message about open-source models. DeepSeek R1 was statistically equivalent to proprietary Gemini 3 Pro within the 1.5-point equivalence margin by two one-sided tests, and both belonged to the pooled safest tier for catastrophic failures. For institutions operating under HIPAA or GDPR constraints that limit cloud transmission of patient data, the prospect of a locally deployable open-source model matching leading proprietary systems on both accuracy and safety is significant. The authors caution, however, that Chinese-language prompts may have favored DeepSeek, and that single-centre, single-language findings cannot yet justify clinical deployment.

The implications extend directly into regulation. Frameworks such as the EU AI Act and the U.S. Food and Drug Administration’s Software as a Medical Device guidance require conformity assessment for high-risk AI, and the authors argue that error-taxonomy profiling and two-layer response consistency should become standard components of that assessment. The stakes rise further as agentic systems begin to act autonomously within electronic health records—ordering tests, prescribing medications, planning admissions—transforming a stochastic single-round failure from a low benchmark score into a potentially executed clinical action. The team acknowledges substantial limitations: all cases came from one Chinese tertiary hospital, error categories were small, difficulty tertiles were derived post hoc, models evolve rapidly, and no contemporaneous human comparison arm was included. Still, as a proof of concept, the framework demonstrates that the safety-relevant differences concealed by aggregate accuracy are real, measurable, and too consequential to ignore. Multi-centre validation, prospective testing under varied prompts, and cross-domain extension of the taxonomy now stand as the field’s most urgent priorities.

Subject of Research: A multi-dimensional framework for evaluating the safety and reliability of large language models in clinical decision support using catastrophic-failure frequency, error taxonomy, and response reproducibility.

Article Title: Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation

Article References: Liu, F., Liu, Z., Fei, X., He, J., Xing, J., Li, J., & Chan, P. (2026). Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation. Journal of Medical Systems, 50(1), Article 136. https://doi.org/10.1007/s10916-026-02459-1

Image Credits: AI Generated

DOI: 10.1007/s10916-026-02459-1

Keywords: large language models, clinical decision support, AI safety evaluation, catastrophic failure, response reproducibility, error taxonomy, emergency medicine, DeepSeek R1, GPT-5.1, EU AI Act, Software as a Medical Device, anchoring bias