The research team, led by Yuntao Wang and colleagues including Mariluz De Ornelas, Matthew T. Studenski, Elizabeth Bossart, Siamak P. Nejad-Davarani and Yunze Yang, drew its test material from the Radiation Oncology Incident Learning System, or RO-ILS, a national reporting database that collects narrative accounts of errors and near-misses from clinics across the country. Rather than feeding the models raw, unstructured text and hoping for the best, the investigators designed a carefully standardized prompt built on the root cause analysis guidelines of the American Association of Physicists in Medicine. Four state-of-the-art models were put through their paces: Gemini 2.5 Pro, GPT-4o, o3 and Grok 3. Each received the Background and Incident Overview sections from nineteen publicly available RO-ILS cases and was instructed to produce three deliverables for every case: the root causes of the incident, the lessons that should be learned from it, and a set of suggested corrective actions.

What makes this study methodologically interesting is the sheer ambition of its evaluation framework. Assessing the quality of an AI-generated causal analysis is not like checking arithmetic; there is no single correct answer key. The team therefore layered three distinct tiers of assessment on top of one another. At the objective level, they used semantic similarity metrics, computing cosine similarity between model outputs and reference analyses with a Sentence Transformer model, a technique that measures how closely two pieces of text mean the same thing even when they use different words. At a semi-subjective level, they calculated precision, recall and F1-scores, along with an expert-adjudicated positive predictive value, a hallucination rate, and performance criteria covering relevance, comprehensiveness, quality of justification and quality of the proposed solutions. Finally, five board-certified medical physicists provided subjective ratings of reasoning quality and overall performance, bringing genuine clinical judgment to bear on every output.

The headline finding is that the models performed satisfactorily across the board. All four demonstrated comparable baseline capabilities in extracting causal factors from incident narratives, and the analyses they produced were judged relevant and accurate, aligned with what the expert reviewers expected from a competent human analyst. Gemini 2.5 Pro emerged with the highest overall performance score among the four systems, though the differences between models were not uniform across every metric. Statistical testing revealed significant differences among the models in expert-adjudicated positive predictive value, in hallucination rate, and in the subjective ratings assigned by the physicist reviewers, with p-values below 0.05. In plain terms, the models were not interchangeable: some were noticeably more trustworthy than others when their outputs were scrutinized by people who do this work for a living.

That last point matters enormously, because the study did not shy away from the technology’s most notorious weakness. Every model exhibited some degree of hallucination, meaning it fabricated or distorted information not supported by the source material, and the rates ranged from a comparatively modest 11 percent to a troubling 61 percent. For a field where patient safety hangs in the balance, a hallucination rate anywhere near the upper end of that spectrum would be disqualifying for unsupervised use. The authors are careful to frame their results as evidence for assistive rather than autonomous deployment. The vision they describe is not a machine that replaces the safety committee but a machine that drafts the first pass of an analysis, surfaces plausible causal threads, and proposes candidate corrective actions that human experts can then verify, refine and own.

The implications for clinical practice could be substantial. Root cause analysis in radiation oncology typically requires convening multidisciplinary teams of physicists, dosimetrists, therapists, physicians and administrators to reconstruct what happened and why. These sessions are time-consuming, and the narrative reports that feed them are often long, ambiguous and written under stressful circumstances. A language model that can rapidly generate a structured preliminary analysis, organized according to established AAPM guidelines, could shorten the path from incident report to corrective action. It could also help smaller clinics that lack the staffing to conduct exhaustive analyses of every near-miss, potentially raising the baseline of safety surveillance across the entire field. In the aggregate, tools like this could strengthen incident learning systems by making the analytical step faster and more consistent.

The study also offers a template for how medical AI evaluations should be conducted more broadly. Rather than relying on a single metric or a single reviewer, the researchers triangulated across objective similarity measures, quantitative classification metrics and human expert judgment, and they applied statistical significance testing to distinguish genuine performance differences from noise. This layered approach acknowledges an uncomfortable truth about language models: text that looks plausible is not necessarily text that is correct, and similarity to a reference answer does not capture whether a proposed corrective action would actually work in a clinic. By combining expert-adjudicated precision with explicit hallucination measurement, the evaluation gets closer to the question that really matters for patient safety: can this system’s output be trusted, and under what supervision?

Caution remains warranted. Nineteen publicly available cases are a small sample, and publicly reported incidents may differ in complexity and completeness from the internal reports a clinic generates behind its own walls. The models were tested on narrative sections only, without access to the full investigative context that a real RCA team would possess. Hallucination rates, even at the low end, mean that every machine-generated statement about an incident would need verification before it informed any corrective decision. There are also governance questions the study does not resolve: how patient confidentiality would be protected when incident narratives are processed by commercial models, who bears responsibility for an AI-suggested action that proves inadequate, and how such tools would be validated and regulated before entering routine safety workflows.

Still, the direction of travel is clear and, for a field built on the principle of learning from error, quietly exciting. Radiation oncology was among the first medical specialties to confront the reality that complex technology fails in complex ways, and its incident learning infrastructure is among the most mature in healthcare. Injecting language models into that infrastructure, as assistive analysts that draft, summarize and propose while humans verify and decide, could compress the feedback loop between error and improvement from weeks to hours. The PLOS Digital Health study is a proof of concept, not a prescription, but it demonstrates that the reasoning capabilities of current frontier models are already close enough to expert expectations to be worth taking seriously. As hallucination rates fall and evaluation standards mature, the safety committee’s newest member may well be an algorithm, one that never gets tired of reading incident reports and never forgets a lesson once it has been written down.

Subject of Research: Large language model-based root cause analysis of radiation oncology patient safety incidents

Article Title: Augmenting patient safety surveillance in radiation oncology with large language model-based root cause analysis: A proof-of-concept study

Article References: Wang, Y., De Ornelas, M., Studenski, M. T., Bossart, E., Nejad-Davarani, S. P., & Yang, Y. (2026). Augmenting patient safety surveillance in radiation oncology with large language model-based root cause analysis: A proof-of-concept study. PLOS Digital Health, 5(9), e0001740. https://doi.org/10.1371/journal.pdig.0001740

Image Credits: AI Generated

DOI: 10.1371/journal.pdig.0001740

Keywords: large language models, radiation oncology, root cause analysis, patient safety, RO-ILS, incident learning, medical physics, hallucination, AAPM guidelines, artificial intelligence, PLOS Digital Health, quality improvement