The research team embarked on a meticulous evaluation of popular large language models (LLMs) by crafting controlled experimental scenarios featuring entirely fictional medical terms—fake diseases, symptoms, and diagnostic tests. These fabricated patient cases were designed to assess if AI chatbots would blindly accept and elaborate on erroneous information embedded within queries. Initial results were troubling: without intervention, the chatbots not only regurgitated the false data but often augmented it with confident, detailed explanations about these invented conditions, effectively demonstrating a hallmark behavior known as “hallucination,” where models produce convincing yet entirely fabricated content.

Crucially, the investigators identified a simple yet powerful mitigation strategy. By appending a concise cautionary prompt to the AI input—alerting the system that the provided information might be inaccurate—they observed a marked decrease in the frequency and severity of hallucinated responses. This modification effectively halved the incidence of erroneous elaborations, suggesting that prompt engineering and built-in safety warnings can serve as practical countermeasures to reduce misinformation propagation in AI-enabled healthcare applications.

Mahmud Omar, MD, the lead author and an independent consultant collaborating on this study, emphasized the ease with which these systems can be derailed. “Our experiments showed that AI chatbots are highly susceptible to being misled by false medical details—whether introduced accidentally or deliberately. The danger lies not only in parroting misinformation but also in crafting detailed, plausible narratives around these untrue facts,” he explained. Omar noted that the intervention of a simple one-line warning in the prompt dramatically diminished such hazardous behaviors, underscoring that even incremental design changes can have outsized impacts on safety.

The study’s methodology involved a two-pronged approach. In the first phase, chatbots were prompted with fabricated clinical vignettes devoid of any safety instructions, allowing researchers to observe natural responses. In the subsequent phase, a brief disclaimer was embedded within the prompt, cautioning the model about potential inaccuracies in the input data. The comparative analysis clearly demonstrated that the presence of the warning significantly curtailed hallucination rates, reinforcing the concept that proactive prompt design is an indispensable component of responsible AI deployment in medical contexts.

Eyal Klang, MD, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at Mount Sinai and co-corresponding senior author, highlighted the gravity of their findings. “Even a single fabricated term injected into a medical question can trigger the model to generate an authoritative-sounding but entirely fictional medical explanation. However, our results also provide a roadmap for safer AI use: carefully timed safety prompts can meaningfully mitigate those errors, pointing to a future where AI can augment clinical workflows without compromising accuracy,” he stated.

Beyond immediate practical implications, the researchers intend to extend their “fake-term” testing paradigm to real-world datasets, involving de-identified patient records. This next phase aims to stress-test AI systems against misinformation within authentic clinical contexts, allowing for further refinement of safety prompts and integration of retrieval-based tools that may cross-validate the chatbot’s outputs against trustworthy medical knowledge bases. Their iterative validation approach aspires to create robust mechanisms that prevent AI hallucinations from influencing patient care decisions.

Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health and co-corresponding senior author, underscored the broader significance of this research. “Our study shines a spotlight on a blind spot within current AI models—their inadequate handling of false medical information, which can generate dangerously misleading responses. The solution lies not in abandoning AI but in engineering systems designed to recognize questionable input, respond with appropriate caution, and preserve essential human oversight,” he reflected. Nadkarni emphasized that deliberate safety measures and thoughtful AI prompt design are critical levers to unlocking the potential of AI in healthcare while mitigating risks.

The pressing issues revealed by this investigation resonate widely across the rapidly expanding intersection of AI and medicine. As clinicians and patients increasingly adopt AI tools for decision support, the need for transparent, reliable, and safe systems is paramount. Hallucinations—defined as confident AI fabrications—could lead not only to diagnostic errors but also to erosion of trust in technology-assisted medicine. This study’s findings prompt a call for regulatory frameworks, stringent validation protocols, and responsible AI integration practices that prioritize patient safety above all.

Technically, this research also advances our understanding of the cognitive vulnerabilities inherent in large language models. These transformer-based architectures rely heavily on patterns learned from expansive datasets rather than grounded medical fact verification, making them inherently vulnerable to propagating misinformation when presented deceptively. The study’s insight that a relatively minimal prompt addition can effectively curb hallucination frequency suggests that the problem can be partially addressed at the interface between human input and AI generation, rather than requiring entirely new model architectures.

In addition to safety prompt engineering, the study hints at the importance of developing AI systems capable of uncertainty quantification and fact-checking. Future models may incorporate retrieval augmented generation (RAG) techniques, linking generated responses to verified medical literature or electronic health records in real-time to validate outputs. Such approaches, combined with real-time human intervention, could transform AI chatbots from mere language predictors into reliable clinical assistants supporting complex decision making.

Mount Sinai’s Windreich Department of Artificial Intelligence and Human Health, under the leadership of Drs. Nadkarni and Klang, is at the forefront of pioneering responsible AI applications in biomedical contexts. This study not only exemplifies their commitment to ethical AI but also provides a foundational framework for other institutions seeking to evaluate and enhance the safety of AI-driven clinical decision tools. Their continued collaboration with the Hasso Plattner Institute for Digital Health underscores a multidisciplinary approach that bridges computational science, engineering, and clinical medicine.

As artificial intelligence continues to permeate healthcare, studies like this expose the critical challenges posed by AI hallucinations and misinformation. Nevertheless, the promising results around simple safety prompts reflect an optimistic path forward where AI tools can be refined and rigorously tested to meet the high standards essential for clinical reliability. The balance between innovation and caution articulated by the Mount Sinai team paves the way for transformative yet responsible AI advancements that ultimately benefit patient outcomes and the future of medicine.

Subject of Research: Evaluation of misinformation propagation and hallucination tendencies in AI chatbots for clinical decision support, and the effectiveness of safety prompt interventions.

Article Title: Large Language Models Demonstrate Widespread Hallucinations for Clinical Decision Support: A Multiple Model Assurance Analysis

News Publication Date: August 6, 2025

Web References:
https://dx.doi.org/10.1038/s43856-025-01021-3
https://ai.mssm.edu/

References:

• Omar, M., Sorin, V., Collins, J.D., Reich, D., Freeman, R., Charney, A., Gavin, N., Stump, L., Bragazzi, N.L., Nadkarni, G.N., & Klang, E. (2025). Large Language Models Demonstrate Widespread Hallucinations for Clinical Decision Support: A Multiple Model Assurance Analysis. Communications Medicine, August 2, 2025.

Keywords:
Machine Learning, Artificial Intelligence, Large Language Models, AI Hallucinations, Clinical Decision Support, Medical Misinformation, Prompt Engineering, AI Safety, Digital Health, Biomedical Informatics

The crux of the challenge lies in the accelerating integration of AI-driven systems into everyday clinical workflows. Innovations such as machine learning models that can detect early signs of disease from medical imaging, natural language processing applications for electronic health record (EHR) optimization, and decision support systems are no longer futuristic concepts but present-day tools reshaping clinical practice. However, most family medicine training programs have yet to systematically equip trainees with the competencies needed to critically appraise, implement, and ethically navigate these technologies. The AIFM-ed framework directly confronts this educational gap, offering a methodologically sound approach to embedding AI literacy within existing curricula.

Developed through an intricate mixed-methods research process, the framework draws upon a systematic review of prior AI education paradigms and extensive consultations with stakeholders ranging from practicing clinicians to medical residents and AI experts across Canada. This collaborative methodology ensured that the framework reflects both the technological nuances of AI and the practical realities of clinical teaching environments. The resulting structure delineates five foundational pillars: the justification for AI curriculum inclusion, precise learning objectives tailored to family medicine, delineation of essential curriculum content, effective organization of teaching modules, and strategies for curriculum implementation and evaluation.

At the heart of AIFM-ed is the recognition that AI education in medicine cannot adopt a one-size-fits-all approach. Family medicine programs vary widely in terms of resources, institutional priorities, and learner populations. Hence, the framework emphasizes flexibility, enabling educators to tailor content depth and delivery formats based on contextual factors. Whether a program aims for foundational AI understanding or advanced clinical application skills, AIFM-ed provides a versatile template that balances technical proficiency with critical thinking about ethical considerations, data governance, and patient-centered implications of AI use.

The technical dimensions encompassed by the framework extend far beyond basic AI concepts. Physicians in training are guided to grasp algorithmic design principles, such as supervised and unsupervised learning, reinforcement learning, and neural networks, contextualized within medical relevance. Moreover, an emphasis is placed on interpretability and transparency of AI models—often termed explainable AI (XAI)—to foster clinicians’ ability to critically evaluate AI recommendations rather than passively accept them. Understanding biases inherent in training datasets, potential pitfalls like overfitting, and limitations in AI generalizability are embedded in the curriculum to cultivate a discerning clinical mindset.

The AIFM-ed framework also addresses operational integration challenges. Postgraduate learners explore interoperability between AI tools and existing health information systems, gaining familiarity with standards such as Health Level Seven (HL7) Fast Healthcare Interoperability Resources (FHIR), which enable seamless data exchange. Practical training includes navigation of AI-enabled clinical decision support systems, workflow redesign necessitated by AI adoption, and protocols for incident reporting when AI outcomes conflict with clinical judgment. Training also encompasses legal aspects surrounding AI, including data privacy regulations like Canada’s Personal Information Protection and Electronic Documents Act (PIPEDA) and emerging medico-legal frameworks governing AI-assisted diagnoses.

Importantly, the framework champions a reflective, patient-centered orientation toward AI use. Trainees engage with the ethical challenges posed by automated decision-making, such as ensuring informed consent when AI influences clinical recommendations and mitigating potential exacerbation of health disparities due to biased algorithms. Discussions include safeguarding the doctor-patient relationship in an era increasingly mediated by technology and accounting for patients’ perspectives and trust concerning AI. By embedding ethics as a continuous thread, the curriculum fosters physicians equipped not only with technical fluency but with a holistic understanding of AI’s impact on healthcare ecosystems.

Dr. Samira A Rahimi, co-lead of this project and Canada Research Chair in AI and Advanced Digital Primary Health Care at McGill University and the Mila-Quebec AI Institute, emphasizes the urgency of this educational evolution. “Artificial intelligence is no longer an abstract future concept but an active driver of clinical decision-making. Yet, most postgraduate curricula have not caught up with this reality. The AIFM-ed framework is a strategic response that ensures family physicians are clinically competent and technologically agile for the future of care,” she explains. The integration of AI into primary care promises to augment diagnostic accuracy, optimize resource allocation, and personalize treatments, but physicians must first be trained to harness these opportunities responsibly.

Raymond Tolentino, a recent master of science graduate and co-lead on the study, highlights the practical implications for trainee confidence and patient safety. “Our intent is to go beyond imparting new skills. We aim to cultivate a clinical culture where family doctors feel confident evaluating AI outputs, understanding their limitations, and using AI tools to enhance patient care safely. This is essential to translate technological promise into real-world benefits,” he states. The project underscores that technology is a means, not an end—physician judgment and patient welfare remain paramount.

Looking forward, the AIFM-ed team envisions piloting the framework at select Canadian medical institutions to assess adaptability, learner outcomes, and long-term impacts on clinical practice. This iterative implementation phase will enable refinement based on feedback and evolving AI advancements, ensuring the framework remains dynamic. A successful adoption could serve as a scalable model internationally, driving a global shift in medical education norms. Preparing family physicians who are not only clinically adept but also AI literate represents an essential frontier in ensuring equitable, efficient, and ethical healthcare delivery in the digital age.

The emergence of this curriculum framework aligns with broader shifts toward digital transformation in healthcare, echoing initiatives across research, policy, and clinical domains. As AI algorithms gain regulatory approvals and health systems worldwide invest in digital infrastructure, a deficit in healthcare providers prepared to embrace these changes risks creating implementation bottlenecks and unintended patient safety hazards. Frameworks like AIFM-ed bridge this divide by integrating educational innovation with forward-looking workforce development strategies.

In sum, the AIFM-ed curriculum framework positions itself as a decisive step in reconciling the accelerating pace of AI innovation with the imperative to maintain competent, compassionate, and reflective medical practice. By empowering family medicine educators with a structured, evidence-based guide for AI integration, it heralds a future where physicians harness AI not as opaque black boxes but as transparent, trustworthy partners in delivering high-quality patient care. The initiative sets a new benchmark for medical education in the digital era, reinforcing that technological progress must be matched by robust learning paradigms to fully realize AI’s promise in healthcare.

Subject of Research: People
Article Title: AIFM-ed Curriculum Framework for Postgraduate Family Medicine Education on Artificial Intelligence: Mixed Methods Study
News Publication Date: April 28, 2025
Web References:

• JMIR Medical Education: https://mededu.jmir.org/
• Original Study DOI: http://dx.doi.org/10.2196/66828
• Creative Commons License: https://creativecommons.org/licenses/by/4.0/
  Image Credits: JMIR Publications
  Keywords: Artificial intelligence, Educational institutions, Education technology, Clinical research, Education research, Digital publishing, Scientific publishing, Family medicine, Education administration, Digital data, Health care, Tools, Graduate education, Learning processes

The researchers conducted meticulous examinations using nine large language models (LLMs) across a sample of 1,000 emergency department cases. Each case replicated 32 different patient backgrounds, yielding over 1.7 million AI-generated medical recommendations. The aim was to critically assess whether these sophisticated algorithms prioritize clinical accuracy or allow extraneous factors, such as income or ethnicity, to influence treatment paths.

A pivotal finding of the study revealed a worrisome trend: AI models disproportionately escalated care recommendations, especially in mental health evaluations, based largely on demographic factors rather than pressing medical necessity. The data indicated a clear disparity in the recommendation of advanced diagnostic tests between patients characterized as high-income versus their low-income counterparts. High-income patients were more frequently advised to undergo costly diagnostic tests, such as CT scans or MRIs, while low-income patients suffered from a lack of further testing opportunities, often receiving minimal follow-up care. Such discrepancies point to an urgent need for stronger regulatory oversight to ensure medically justified and fair treatment practices.

The implications of this research extend beyond just the medical community. They underline the potential consequences of AI bias on public health outcomes, particularly among vulnerable populations. "Our goal is to establish a framework for AI assurance that empowers developers and healthcare institutions to create fair and reliable AI systems," expressed Dr. Eyal Klang, a co-senior author of the study. Dr. Klang specifically emphasized the importance of training algorithms to differentiate between condition-based recommendations and those driven by patients’ background characteristics.

An alarming aspect of the results showcases how AI’s recommendation patterns can inadvertently perpetuate health disparities rather than mitigate them. The decisions made by AI systems could significantly influence the therapeutic routes offered to patients, treatment access, and overall healthcare equity. Alarmingly, this could reinforce existing inequalities in healthcare delivery, where marginalized groups remain at a disadvantage in receiving necessary medical interventions.

In light of these findings, future research is poised to tackle the challenge of AI biases head-on. The study’s authors are forging alliances with various healthcare institutions to enhance the ethical design of AI tools, ensuring that they consistently adhere to fairness and equity standards. Continuous assurance testing will be vital for evaluating AI models’ performances in real-world scenarios, focusing specifically on how different prompts might subdue biases that could compromise patient care.

Dr. Mahmud Omar, the study’s first author and a physician-scientist, echoed a similar call to action, stating, "As AI technology becomes more enmeshed in clinical settings, scrutinizing its reliability and fairness is imperative. Addressing biases is not merely a suggestion—it is a requisite for the future of patient-centered care." He emphasized the collaborative efforts involved in this research, marking it as a step towards establishing robust protocols for AI assurance that would benefit patients on a global scale.

Furthermore, the study sparks a dialogue about the broader implications of AI advancements in healthcare. While the integration of AI into medical practices promises transformative benefits, it simultaneously presents dilemmas that necessitate diligent scrutiny. AI can augment diagnostic accuracy and streamline decision processes, yet it is crucial for stakeholders to remain vigilant about the unintentional biases that may arise, potentially altering patient welfare trajectories.

Moving forward, investigators plan to simulate complex clinical conversations involving AI models and pilot these models in hospital environments to assess their true impact on healthcare delivery systems. The goal is to develop policies and best practices that facilitate the ethical use of AI, minimizing risks associated with biased decision-making processes in healthcare environments.

This study leaves the audience with profound questions about the ethical deployment of AI in healthcare settings. As AI technologies continue to evolve, the discourse around their societal implications becomes increasingly relevant. Ensuring that AI operates within equitable frameworks will be vital to protect the most vulnerable among us—those who stand to benefit the most from advances in medical technology but may also be sidelined due to inherent biases.

Ultimately, the findings presented in this research herald a vital turning point in the discourse surrounding AI in healthcare, emphasizing the combination of technological advancement and human-centered ethical practices. The intersection of AI and healthcare must foster tools that uphold justice and equality, not ones that inadvertently widen existing gaps in medical care.

As we continue to explore the integration of AI within our healthcare systems, these imperative studies will guide practitioners, developers, and policymakers toward developing solutions that fortify trust and effectiveness in AI-assisted medical decision-making. Achieving this balance will not only enhance the patient experience but also assure quality healthcare delivery in an increasingly digitized medical landscape.

Subject of Research: Socio-Demographic Biases in Medical Decision-Making by Large Language Models
Article Title: Socio-Demographic Biases in Medical Decision-Making by Large Language Models: A Large-Scale Multi-Model Analysis
News Publication Date: April 7, 2025
Web References: DOI: 10.1038/s41591-025-03626-6
References: Nature Medicine
Image Credits: Mahmud Omar, MD
Keywords: Generative AI; Healthcare Disparities; Medical Decision-Making; AI Ethics; Socio-Demographic Biases