In a landmark study recently published in the prestigious journal Nature Medicine, researchers from the Icahn School of Medicine at Mount Sinai have delivered an eye-opening evaluation of ChatGPT Health, an emerging consumer-facing artificial intelligence (AI) tool designed to provide health guidance to the public. Launched in January 2026 by OpenAI, ChatGPT Health has quickly amassed a massive user base—with approximately 40 million daily users seeking medical advice ranging from routine health questions to urgent triage recommendations. However, this independent scientific inquiry exposes significant shortcomings in the AI system’s ability to accurately triage emergencies and respond to high-risk suicide scenarios, calling into question its safety and reliability in critical moments.
The study represents the first rigorous, independent safety assessment of any large language model (LLM)-enabled health chatbot since the debut of ChatGPT Health. With the healthcare landscape increasingly incorporating AI solutions, this research highlights not only the promise but also the perils of relying on automated systems to make nuanced calls about medical urgency. The investigators designed a battery of 60 detailed clinical vignettes covering 21 medical specialties, carefully spanning a broad spectrum from mild, non-urgent conditions to true medical emergencies. Each scenario was meticulously developed by cardiologists, neurologists, emergency medicine specialists, and other clinicians, then validated by a trio of independent physicians who reached consensus on the appropriate urgency level, based on standardized clinical guidelines from over 50 professional societies.
Testing unfolded under 16 distinct contextual variations, such as racial and gender differences, social context modifiers (e.g., the patient downplaying symptoms), and external barriers like insurance status and transportation availability. Through these 960 carefully scripted patient encounters, the researchers urged ChatGPT Health to dispense triage advice, subsequently contrasting its recommendations against the physician consensus benchmark. Strikingly, while the AI broadly recognized textbook emergencies—such as strokes and anaphylaxis—it failed to recommend emergency care in more than half of the nuanced cases that genuinely demanded urgent intervention. For example, in a scenario featuring early signs of impending respiratory failure in an asthmatic patient, the AI recognized danger internally in its reasoning yet paradoxically advised a wait-and-see approach instead of immediate emergency evaluation.
The findings underscore a crucial limitation: current LLM-based systems may perform adequately when clinical indicators are stark and indisputable but stumble profoundly in complex borderline cases where subtle clinical judgment is paramount. This dichotomy is particularly alarming given that many users likely seek AI input precisely because their symptoms are ambiguous or evolving, not when problems are overt. Dr. Ashwin Ramaswamy, the study’s lead author, encapsulated this challenge by stressing how these tools “struggle in more nuanced situations where clinical judgment matters most,” leaving potential emergencies dangerously under-triaged.
An equally troubling dimension of the evaluation concerned ChatGPT Health’s suicide-risk protocols. The system is programmed to connect high-risk users to the 988 Suicide and Crisis Lifeline, a vital resource intended to prevent harm. Yet, the researchers discovered alarming inconsistencies: the AI’s suicide-risk alerts sometimes triggered in relatively low-risk contexts but failed to activate when users disclosed detailed plans for self-harm, a clinical red flag indicative of imminent danger. Dr. Girish N. Nadkarni, senior study author and Director of Mount Sinai’s Windreich Department of Artificial Intelligence and Human Health, described this inversion as “beyond inconsistency,” whereby the tool seemed more vigilant in benign scenarios than in those signaling acute suicidality. In real-world clinical practice, explicit suicidal intent requires immediate intervention, underscoring a hazardous disconnect between ChatGPT Health’s risk assessment and established clinical priorities.
The study’s methodology exemplifies the rigorous approach necessary for AI safety evaluation. By incorporating diverse clinical specialties and extensive contextual modifiers, the research team crafted a testing environment that mirrors the complexity of real-life medicine. Such sophistication is critical because AI tools must navigate social determinants of health and presentation subtleties that often influence clinical decisions. Moreover, the presence of 16 context variants for each case ensured that ChatGPT Health’s recommendations were scrutinized under myriad potential biases, testing the model’s robustness against factors like patient demographics and access disparities.
Isaac S. Kohane, MD, PhD, Chair of Biomedical Informatics at Harvard Medical School and an expert unaffiliated with the study, contextualized these findings by emphasizing that while AI systems are becoming a new default for many patients, their greatest risks lie “at the clinical extremes” where the stakes of misjudgment are highest. His assertion that “independent evaluation should be routine, not optional” communicates the urgent imperative for transparency and continuous external assessment of AI medical products, ensuring they meet stringent safety standards before widespread deployment.
The authors advocate clearly cautious and judicious use of AI-based health chatbots. They stress that such tools should serve as adjuncts to, rather than replacements for, professional medical judgment. In scenarios involving worsening symptoms like chest pain, altered mental status, or severe allergic reactions, direct medical evaluation remains indispensable. Similarly, anyone experiencing thoughts of self-harm should seek immediate help from established crisis resources or emergency care providers rather than relying on AI guidance that may be inconsistent.
The rapidly evolving nature of LLMs like ChatGPT Health also complicates the picture. AI architectures are frequently updated, with training and fine-tuning occurring in near real-time. This dynamic environment means that performance can fluctuate, sometimes improving but potentially regressing without transparent monitoring. The study authors stress the necessity for ongoing, independent evaluation protocols that keep pace with AI model iterations, ensuring any advances manifest as safer, clinically sound recommendations for users.
Looking ahead, the research team from Mount Sinai plans to extend their evaluations to explore other critical spheres such as pediatric care, medication safety checks, and chatbot interactions in non-English languages. Such expansion is crucial given the wide-ranging populations relying on AI assistance, including vulnerable groups potentially at greater risk from erroneous triage. These future assessments will further illuminate the capabilities and limitations of AI in medicine, helping guide responsible integration into healthcare pathways.
The Icahn School of Medicine at Mount Sinai’s Windreich Department of Artificial Intelligence and Human Health, led by Dr. Nadkarni, is at the forefront of these efforts. The department’s mission centers on pioneering ethical, safe, and effective AI applications to revolutionize research, clinical care, and education. By collaborating with the Hasso Plattner Institute for Digital Health, Mount Sinai harnesses world-class computational infrastructure and cross-disciplinary expertise, translating machine learning innovations like the NutriScan AI malnutrition detection tool into impactful clinical solutions—a testament to the transformative potential of applied AI in healthcare when developed and deployed conscientiously.
This pivotal study serves as both a cautionary tale and a call to action for the burgeoning digital health ecosystem. While AI health chatbots like ChatGPT Health offer unprecedented accessibility and immediacy of medical guidance, users and clinicians must remain vigilant of their current limitations. Responsible adoption will require continuous algorithmic refinement, systematic validation, and nuanced clinical oversight to ensure AI augments rather than undermines patient safety. As these tools increasingly permeate everyday health decisions, interdisciplinary collaborations and independent scientific scrutiny will be essential to safeguard public well-being in the AI era.
Subject of Research: People
Article Title: ChatGPT Health performance in a structured test of triage recommendations
News Publication Date: 23-Feb-2026
Web References:
https://icahn.mssm.edu/about/artificial-intelligence
http://dx.doi.org/10.1038/s41591-026-04297-7
References:
Ramaswamy A, Tyagi A, Hugo H, et al. ChatGPT Health performance in a structured test of triage recommendations. Nature Medicine. 2026;DOI:10.1038/s41591-026-04297-7.
Keywords:
Generative AI, Artificial Intelligence, Computer Science, Applied Sciences, Healthcare AI, Chatbot Safety, Medical Triage, Suicide Risk Assessment, Large Language Models
