A groundbreaking study conducted by scientists at the Icahn School of Medicine at Mount Sinai has exposed a significant vulnerability in widely used artificial intelligence (AI) chatbots, particularly within healthcare settings. The research reveals that these sophisticated language models are alarmingly prone to repeating and expanding on false medical details when presented with inaccurate or fabricated information. This discovery raises profound concerns about the unguarded integration of AI in clinical decision-making, emphasizing an urgent need for stringent safeguards before these tools can be reliably deployed in patient care.
The research team embarked on a meticulous evaluation of popular large language models (LLMs) by crafting controlled experimental scenarios featuring entirely fictional medical terms—fake diseases, symptoms, and diagnostic tests. These fabricated patient cases were designed to assess if AI chatbots would blindly accept and elaborate on erroneous information embedded within queries. Initial results were troubling: without intervention, the chatbots not only regurgitated the false data but often augmented it with confident, detailed explanations about these invented conditions, effectively demonstrating a hallmark behavior known as “hallucination,” where models produce convincing yet entirely fabricated content.
Crucially, the investigators identified a simple yet powerful mitigation strategy. By appending a concise cautionary prompt to the AI input—alerting the system that the provided information might be inaccurate—they observed a marked decrease in the frequency and severity of hallucinated responses. This modification effectively halved the incidence of erroneous elaborations, suggesting that prompt engineering and built-in safety warnings can serve as practical countermeasures to reduce misinformation propagation in AI-enabled healthcare applications.
Mahmud Omar, MD, the lead author and an independent consultant collaborating on this study, emphasized the ease with which these systems can be derailed. “Our experiments showed that AI chatbots are highly susceptible to being misled by false medical details—whether introduced accidentally or deliberately. The danger lies not only in parroting misinformation but also in crafting detailed, plausible narratives around these untrue facts,” he explained. Omar noted that the intervention of a simple one-line warning in the prompt dramatically diminished such hazardous behaviors, underscoring that even incremental design changes can have outsized impacts on safety.
The study’s methodology involved a two-pronged approach. In the first phase, chatbots were prompted with fabricated clinical vignettes devoid of any safety instructions, allowing researchers to observe natural responses. In the subsequent phase, a brief disclaimer was embedded within the prompt, cautioning the model about potential inaccuracies in the input data. The comparative analysis clearly demonstrated that the presence of the warning significantly curtailed hallucination rates, reinforcing the concept that proactive prompt design is an indispensable component of responsible AI deployment in medical contexts.
Eyal Klang, MD, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at Mount Sinai and co-corresponding senior author, highlighted the gravity of their findings. “Even a single fabricated term injected into a medical question can trigger the model to generate an authoritative-sounding but entirely fictional medical explanation. However, our results also provide a roadmap for safer AI use: carefully timed safety prompts can meaningfully mitigate those errors, pointing to a future where AI can augment clinical workflows without compromising accuracy,” he stated.
Beyond immediate practical implications, the researchers intend to extend their “fake-term” testing paradigm to real-world datasets, involving de-identified patient records. This next phase aims to stress-test AI systems against misinformation within authentic clinical contexts, allowing for further refinement of safety prompts and integration of retrieval-based tools that may cross-validate the chatbot’s outputs against trustworthy medical knowledge bases. Their iterative validation approach aspires to create robust mechanisms that prevent AI hallucinations from influencing patient care decisions.
Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health and co-corresponding senior author, underscored the broader significance of this research. “Our study shines a spotlight on a blind spot within current AI models—their inadequate handling of false medical information, which can generate dangerously misleading responses. The solution lies not in abandoning AI but in engineering systems designed to recognize questionable input, respond with appropriate caution, and preserve essential human oversight,” he reflected. Nadkarni emphasized that deliberate safety measures and thoughtful AI prompt design are critical levers to unlocking the potential of AI in healthcare while mitigating risks.
The pressing issues revealed by this investigation resonate widely across the rapidly expanding intersection of AI and medicine. As clinicians and patients increasingly adopt AI tools for decision support, the need for transparent, reliable, and safe systems is paramount. Hallucinations—defined as confident AI fabrications—could lead not only to diagnostic errors but also to erosion of trust in technology-assisted medicine. This study’s findings prompt a call for regulatory frameworks, stringent validation protocols, and responsible AI integration practices that prioritize patient safety above all.
Technically, this research also advances our understanding of the cognitive vulnerabilities inherent in large language models. These transformer-based architectures rely heavily on patterns learned from expansive datasets rather than grounded medical fact verification, making them inherently vulnerable to propagating misinformation when presented deceptively. The study’s insight that a relatively minimal prompt addition can effectively curb hallucination frequency suggests that the problem can be partially addressed at the interface between human input and AI generation, rather than requiring entirely new model architectures.
In addition to safety prompt engineering, the study hints at the importance of developing AI systems capable of uncertainty quantification and fact-checking. Future models may incorporate retrieval augmented generation (RAG) techniques, linking generated responses to verified medical literature or electronic health records in real-time to validate outputs. Such approaches, combined with real-time human intervention, could transform AI chatbots from mere language predictors into reliable clinical assistants supporting complex decision making.
Mount Sinai’s Windreich Department of Artificial Intelligence and Human Health, under the leadership of Drs. Nadkarni and Klang, is at the forefront of pioneering responsible AI applications in biomedical contexts. This study not only exemplifies their commitment to ethical AI but also provides a foundational framework for other institutions seeking to evaluate and enhance the safety of AI-driven clinical decision tools. Their continued collaboration with the Hasso Plattner Institute for Digital Health underscores a multidisciplinary approach that bridges computational science, engineering, and clinical medicine.
As artificial intelligence continues to permeate healthcare, studies like this expose the critical challenges posed by AI hallucinations and misinformation. Nevertheless, the promising results around simple safety prompts reflect an optimistic path forward where AI tools can be refined and rigorously tested to meet the high standards essential for clinical reliability. The balance between innovation and caution articulated by the Mount Sinai team paves the way for transformative yet responsible AI advancements that ultimately benefit patient outcomes and the future of medicine.
Subject of Research: Evaluation of misinformation propagation and hallucination tendencies in AI chatbots for clinical decision support, and the effectiveness of safety prompt interventions.
Article Title: Large Language Models Demonstrate Widespread Hallucinations for Clinical Decision Support: A Multiple Model Assurance Analysis
News Publication Date: August 6, 2025
Web References:
https://dx.doi.org/10.1038/s43856-025-01021-3
https://ai.mssm.edu/
References:
- Omar, M., Sorin, V., Collins, J.D., Reich, D., Freeman, R., Charney, A., Gavin, N., Stump, L., Bragazzi, N.L., Nadkarni, G.N., & Klang, E. (2025). Large Language Models Demonstrate Widespread Hallucinations for Clinical Decision Support: A Multiple Model Assurance Analysis. Communications Medicine, August 2, 2025.
Keywords:
Machine Learning, Artificial Intelligence, Large Language Models, AI Hallucinations, Clinical Decision Support, Medical Misinformation, Prompt Engineering, AI Safety, Digital Health, Biomedical Informatics
