New multi-axis study probes how AI verifiers disagree on clinical reasoning

Artificial intelligence systems are increasingly being asked not just to generate medical conclusions but to judge whether other AI systems reason soundly—a task that, according to new research, they are strikingly bad at agreeing on. A study published in the Journal of Medical Systems has found that when three frontier large language models were asked to independently evaluate the clinical reasoning of other AI models, they disagreed with one another in the vast majority of cases, raising serious questions about one of the most popular strategies for evaluating medical AI at scale.

The research, led by Hyunjung Byun and Beakcheol Jang of Yonsei University in Seoul, together with Dahyoun Lee of Yonsei University’s Department of Biomedical Systems Informatics and Munyoung Jung of the university’s Graduate School of Mechanical Engineering, examined a foundational assumption behind the so-called “LLM-as-a-judge” approach: that independent verifier models, when shown the same piece of clinical reasoning, will broadly agree on whether the stated evidence actually supports the stated conclusion. The answer, the study found, is a resounding no. Using Fleiss’ kappa, a standard statistical measure of agreement among multiple raters, the researchers recorded values ranging from just 0.087 to 0.223 across the verifier panels—figures that fall far below the thresholds typically associated with acceptable reliability. Raw disagreement rates between verifiers ranged from 62.2 percent to 74.3 percent, meaning that in roughly two out of every three cases or more, the AI judges could not concur on whether a given piece of clinical reasoning held together.

To reach these conclusions, the team designed an unusually rigorous multi-axis evaluation framework. Three generator models—HuatuoGPT-o1-8B, a medical-specialized open model; Meta-Llama-3.1-8B-Instruct; and the considerably larger Meta-Llama-3.3-70B-Instruct—were tasked with producing diagnoses and supporting rationales for 1,000 hospital-stay cases drawn from MIMIC-IV, a widely used, de-identified electronic health record dataset developed at the MIT-affiliated Laboratory for Computational Physiology and distributed through PhysioNet under a credentialed data use agreement. Each generated answer was then assessed along four complementary dimensions: medical concept grounding, semantic similarity, semantic uncertainty, and the study’s central innovation, evidence–conclusion coherence.

The first three axes represent relatively established techniques. Medical concept grounding was evaluated by checking whether a model’s output was anchored in recognized biomedical terminology, drawing on resources such as the Unified Medical Language System and clinical-domain embedding models. Semantic similarity was measured using sentence-embedding methods to quantify how closely a generated rationale matched the underlying clinical record. Semantic uncertainty, a technique popularized in recent machine-learning research, captures how much a model’s outputs vary when the same question is asked repeatedly—the idea being that a model that produces wildly different answers across runs is effectively signaling that it does not know. These reference-oriented metrics, the researchers note, have traditionally been the primary tools for judging how well a language model performs on medical text.

But none of these metrics, the authors argue, directly addresses the most clinically consequential question: does the model’s own justification actually entail its own diagnosis? A language model can write a fluent, medically grounded, low-uncertainty rationale and still end up with a conclusion that the rationale itself does not support—the textual equivalent of a chain of reasoning with a hidden broken link. To measure this, the team turned to three frontier verifiers: Claude Sonnet 4.6, Gemini 2.5 Pro, and GPT-5.4 mini. Each verifier independently reviewed the same generated outputs and assigned a categorical coherence label—ENTAILED, INSUFFICIENT, or NOT_ENTAILED—indicating whether the evidence in the rationale supported the conclusion.

The results revealed a dissociation that conventional metrics miss entirely. Generator models could score well on concept grounding, similarity, and uncertainty while still producing rationales whose coherence was judged poor. In other words, a model can look excellent on every standard benchmark dimension while quietly producing justifications that fail to support their own diagnoses. This finding strikes at a persistent vulnerability in how medical AI is evaluated: fluency and surface-level fidelity are poor proxies for logical soundness, and the phenomenon echoes earlier research on unfaithful explanations in chain-of-thought reasoning, where language models have been shown to sometimes produce plausible-sounding justifications that do not reflect the actual computational pathway to their answers.

The coherence judgments themselves, however, proved deeply unreliable. When the three verifiers were shown the same rationale, their verdicts frequently split—sometimes all three disagreeing, assigning three different labels to the identical output. The researchers illustrate this with case-level examples: in one case drawn from HuatuoGPT-o1-8B, the verifiers returned ENTAILED, INSUFFICIENT, and NOT_ENTAILED for the same clinical reasoning passage. In such instances, the effective evaluation outcome would have been determined entirely by the chance of which verifier happened to be queried. This is precisely the scenario that makes the LLM-as-judge approach dangerous in clinical settings, where a not-entailed verdict might flag an unsafe diagnostic rationale for human review, while an entailed verdict might wave it through unchecked.

To anchor the automated judgments in clinical reality, the team conducted a preliminary validation in which an independent physician—Won Hwi Kim, MD, a resident physician in the Department of Psychiatry at Severance Hospital, Yonsei University College of Medicine—adjudicated a 50-case subset of the generated outputs. The physician was blinded to all verifier labels and recorded a one-line clinical rationale for each judgment using the same three-label scheme. The results echoed the statistical findings: agreement between any given verifier and the physician varied substantially across the three models, and, notably, the physician did not consistently side with any single verifier. In one Gemini-divergent case, the clinician’s judgment coincided with the more lenient Google model; in another case with the same divergence pattern, the physician sided with the stricter majority instead. The clinician’s leniency, the authors observe, appeared to depend on whether the stated evidence genuinely supported the specific diagnosis at hand—not on a fixed threshold of the kind a statistical classifier applies.

Taken together, the findings suggest that no single LLM verifier currently possesses sufficient reliability to serve as a stand-alone judge of clinical reasoning at scale. This matters because LLM-as-judge frameworks are rapidly being adopted across healthcare AI evaluation, from screening hospital discharge summaries to assessing note quality across academic medical centers, precisely because human expert review is expensive, slow, and difficult to scale. The study’s results do not imply that automated evaluation is worthless; rather, they imply that treating a single model’s verdict as authoritative is statistically indefensible when verifier agreement is this low. The authors point to a potential middle path: a unanimous-agreement tier, in which automation is permitted only when all verifiers concur, reserving ambiguous cases for structured human oversight. But they caution that even this tier’s clinical reliability remains to be confirmed in larger, multi-clinician adjudication studies, and the current validation rests on a single physician and a modest 50-case subset.

The study also carries broader implications for the field’s evaluation methodology. Reference-oriented semantic metrics—BLEU-style scores, embedding similarities, and the like—measure how closely a model’s output resembles some expected text, but they cannot detect internal logical failure. The Yonsei team’s four-axis framework offers a template for evaluation that treats coherence as a first-class concern, and their finding that models can score well on three axes while failing the fourth is a warning against any single-metric reporting. At the same time, the very tool needed to measure coherence at scale—another LLM—is itself unreliable in this role, creating a methodological catch: the technology needed to audit medical AI appears to require the same human oversight it was meant to replace. The researchers suggest the path forward lies in combining multi-verifier panels, explicit agreement thresholds, and structured clinician adjudication, with automation reserved only for the subset of cases where machines and, ideally, multiple independent judgments converge. The work was supported by the National Research Foundation of Korea, and the analysis code has been made publicly available, while the underlying MIMIC-IV patient data remains governed by PhysioNet’s credentialed access agreement to protect patient-derived content.