The core objective of Hu’s study was to interrogate how different versions of ChatGPT handle diagnostic tasks given real-world clinical data. Using de-identified physician notes from 30 emergency department cases, the research team prompted various ChatGPT model iterations—including GPT-3.5, GPT-4, GPT-4o, and the o1 series—to generate their top three diagnostic suggestions. The study’s methodological rigor involved comparing the models’ diagnostic precision and accuracy against actual clinical outcomes to draw a comprehensive performance profile. This approach provides a window into how AI tools can supplement, but not yet replace, human clinical judgment.

One of the profound insights emerging from this investigation is the discrepancy in AI performance between cases with classic, textbook symptoms and those with atypical or “challenging” presentations. For patients exhibiting hallmark signs of disease, ChatGPT models demonstrated promising diagnostic assistance capabilities, supporting physicians by suggesting accurate differential diagnoses. However, when confronted with complex cases lacking traditional symptomatic cues—such as pneumonia cases without accompanying fever—AI’s capacity to correctly identify diagnoses notably diminished. These failures illuminate the inherent difficulty AI models face when operating beyond their training data’s typical patterns, emphasizing the necessity for richer, more diverse datasets.

The researchers note that current AI diagnostic models primarily ingest unstructured text input—in this case, physicians’ notes—without access to multimodal clinical information. Consequently, ChatGPT’s diagnostic reasoning is limited by the breadth and variability of its textual training corpora and the information provided. Hu posits that enhancing future AI frameworks with additional clinical data streams—such as imaging results, laboratory findings, and comprehensive patient histories—could improve the fidelity and robustness of AI-assisted diagnoses in emergency contexts. Integration of these heterogeneous data types would transform AI from a purely linguistic interpreter to a more holistic clinical decision support system.

Analysis of the longitudinal performance of ChatGPT iterations reveals an interesting but cautious trajectory of improvement. While no statistically significant advance was observed when considering the inclusion of AI-generated diagnoses within the top three suggestions, the accuracy of the very top, or primary, diagnosis recommendation improved by approximately 15 to 20 percent in newer models relative to their predecessors. This subtle enhancement suggests iterative refinement in model capabilities but also highlights the persistent challenges in achieving consistently high precision necessary for clinical reliability.

The study underscores a key principle in the deployment of AI-assisted diagnostic tools: the indispensability of human oversight. Given the models’ current inadequate performance on complex cases, physician expertise remains essential to interpret AI outputs critically and corroborate or refute AI-generated hypotheses. This interplay forms a hybrid intelligence paradigm, wherein AI accelerates data synthesis and hypothesis generation while clinicians provide contextual judgment, ensuring that patient care remains both accurate and personalized.

Beyond diagnostic accuracy, Hu envisions AI modalities evolving towards greater transparency and explicability. He stresses the importance of AI systems that do not merely generate results but also reveal their reasoning pathways, enabling clinicians to understand and trust their recommendations. Such “explainable AI” is critical to fostering confidence among healthcare providers, enhancing AI’s integration into clinical workflows, and ultimately improving patient outcomes. Achieving this level of transparency will require methodological innovations in how AI models represent and communicate uncertainty and rationale.

Moreover, Hu’s research team explores imaginative avenues to augment diagnostic reasoning by leveraging multi-agent AI simulations. Drawing on prior work where ChatGPT-4 was deployed in role-playing scenarios—emulating specialists such as physiotherapists, psychologists, and nutritionists engaged in panel discussions—this approach aims to replicate the collaborative diagnostic processes typical in clinical environments. The proposed conversational model suggests that dynamic interactions among diverse AI agents could produce more nuanced, accurate diagnostic assessments, reflecting interdisciplinary integration akin to human medical teams.

Despite these promising strides, the researchers caution that current AI systems, including ChatGPT, do not qualify as certified medical devices and should not be used as standalone diagnostic solutions. In clinical settings where expanded data types, such as imaging, are incorporated, AI models must operate within secure, privacy-compliant hospital clusters as open-source platforms. Compliance with regulatory standards and patient confidentiality laws remains a non-negotiable prerequisite for AI deployment in healthcare institutions.

The study acknowledges support from the National Science Foundation and the National Institutes of Health, emphasizing the significance of federally-funded research in advancing AI applications in medicine. Additional contributors include postdoctoral fellow Jinge Wang, lab volunteer Kenneth Shue, and Li Liu from Arizona State University, reflecting a multidisciplinary collaboration essential to tackling complex problems at the intersection of computer science, bioinformatics, and clinical medicine.

Looking ahead, Hu advocates for future research to focus not only on enhancing AI’s diagnostic performance but also on its capacity to articulate reasoning in clinically meaningful ways. He suggests that improved explainability could facilitate critical emergency department decisions such as triage prioritization and treatment pathway selection, augmenting both efficiency and patient safety.

In summary, the pioneering evaluation of ChatGPT models in emergency diagnostics performed by WVU scientists reveals a nuanced landscape marked by AI’s emerging utility balanced against intrinsic challenges. While encouraging diagnostic accuracy for prototypical cases validates the promise of language models as assistive tools, persistent deficiencies in recognizing atypical disease presentations underscore the imperative for richer data integration, transparent reasoning, and robust human-AI collaboration. This research not only advances scientific understanding of AI capabilities at the clinical frontline but also charts a thoughtful course towards responsible integration of AI in patient-centered care.

Subject of Research:
Evaluation of ChatGPT AI model iterations for diagnostic assistance in emergency department patients using clinical notes.

Article Title:
Preliminary evaluation of ChatGPT model iterations in emergency department diagnostics

News Publication Date:
26-Mar-2025

Web References:

• https://www.wvu.edu/
• https://directory.hsc.wvu.edu/Profile/60888
• https://medicine.wvu.edu/
• https://medicine.wvu.edu/micro/
• https://health.wvu.edu/research-and-graduate-education/research/core-facilities/bioinformatics-core/
• https://www.nature.com/articles/s41598-025-95233-1#citeas
• http://dx.doi.org/10.1038/s41598-025-95233-1
• https://mededu.jmir.org/2024/1/e51157/

References:
Hu, G. M., Wang, J., Shue, K., Liu, L. (2025). Preliminary evaluation of ChatGPT model iterations in emergency department diagnostics. Scientific Reports. DOI: 10.1038/s41598-025-95233-1

Image Credits:
WVU Photo/Greg Ellis

Keywords:
Artificial intelligence, Disease prevention, Clinical medicine, Medical tests, Artificial consciousness, Artificial neural networks, Cognitive robotics, Forward chaining, Generative AI, Genetic algorithms, Logic based AI, Adaptive systems, Cybernetics, Robotics, Computer science

Mental health disorders such as anxiety and depression pose substantial challenges worldwide, particularly among college students who often face immense academic and social pressures. Recognizing symptoms early can significantly improve outcomes, but the demand for accessible, efficient screening tools remains unmet in many settings. Traditional questionnaires like the Patient Health Questionnaire-9 (PHQ-9) and the Generalized Anxiety Disorder Scale-7 (GAD-7) have long served as gold standards in clinical and research settings, yet they rely heavily on self-reporting and require administration by trained personnel.

Enter ChatGPT-4, an advanced iteration of large language models developed by OpenAI, capable of understanding and generating human-like text. Harnessing its natural language processing abilities, the study’s investigators tasked ChatGPT-4 with generating structured interview questionnaires that mirror the content and intention of the PHQ-9 and GAD-7. These AI-generated versions, designated as GPT-PHQ-9 and GPT-GAD-7, offer an innovative approach: transforming static questionnaires into dynamic, conversational assessments that could potentially lower barriers to mental health screening.

The research utilized a cohort of 200 college students who were assessed using both the traditional validated questionnaires and the newly designed ChatGPT-4 adaptations. To ensure rigour, the team applied statistical methods including Spearman correlation analysis and intra-class correlation coefficients (ICC) to gauge reliability and consistency between the two sets of measures. The results revealed promising reliability metrics with Cronbach’s alpha values of 0.75 for GPT-PHQ-9 and 0.76 for GPT-GAD-7, suggesting that the AI-generated instruments maintain internal consistency comparable to their established counterparts.

Intraclass correlation coefficients further supported the concordance between the traditional and AI versions, registering 0.80 for the PHQ-9 and 0.70 for the GAD-7. Spearman’s correlation reflected moderate associations, reinforcing that ChatGPT-4’s dynamically generated questionnaires align well with the clinically validated scales. These correlation values signal that although not perfect, the AI-adapted tools capture core symptoms reliably, laying a foundation for their potential application in broader screening contexts.

Beyond correlation, diagnostic accuracy was scrutinized using Receiver Operating Characteristic (ROC) curve analyses, a standard approach to determine optimal cutoff points that balance sensitivity and specificity. For depressive symptom screening, an AI-generated questionnaire cutoff score of 9.5 achieved high sensitivity and specificity, paralleling the original PHQ-9 performance. Similarly, the GPT-GAD-7 demonstrated an optimal cutoff at 6.5 for detecting anxiety symptoms, endorsing its viability as a screening instrument.

To delve deeper into the nuances of agreement, Bland–Altman plots were employed, visually examining differences between AI-generated and validated questionnaire scores. These graphical assessments confirmed acceptable limits of agreement, further substantiating the AI tool’s potential to approximate human-administered assessments without significant bias or deviation.

The implications of this study are profound. By effectively transforming established psychiatric screening tools into AI-driven conversational formats, ChatGPT-4 could democratize access to mental health evaluation. Such tools may reduce the stigma often associated with clinic visits, offer instant preliminary assessments, and triage students for professional care efficiently. Furthermore, AI’s adaptability allows for continual refinement, potentially tailoring questions to individual responses in real-time, enhancing accuracy and user engagement.

Importantly, while this study focused on college students—a demographic exhibiting heightened vulnerability to mood disorders—the methods and findings hold promise across diverse populations. Future research is encouraged to validate the AI-based questionnaires within various age groups, cultural contexts, and clinical settings to confirm their robustness and generalizability.

However, the study is not without limitations. The cross-sectional design provides a snapshot rather than longitudinal insight into symptom changes over time. Additionally, considerations surrounding data privacy, algorithmic transparency, and ethical deployment of AI in mental health contexts warrant careful navigation to ensure safety and equity.

From a technological perspective, the capacity of large language models like ChatGPT-4 to comprehend nuanced human emotion and psychopathology underscores a new frontier in computational psychiatry. AI’s role could evolve from passive questionnaire administration to more interactive, empathetic supports that aid clinicians and empower patients alike.

In summary, this innovative research articulates a compelling vision where artificial intelligence synthesizes clinical expertise with advanced computational linguistics to redefine mental health screening frameworks. The promising concordance between GPT-generated assessments and validated tools heralds a future wherein mental health support becomes more accessible, personalized, and efficient through AI integration.

As mental health disorders rise globally, the necessity for scalable, effective screening mechanisms has never been greater. The demonstrated reliability and diagnostic precision of ChatGPT-4’s adapted questionnaires serve as an encouraging testament to the transformative potential of AI in psychiatry. Further investigations and technological refinements will be critical in harnessing this potential responsibly, ensuring that AI-enhanced mental health evaluations adhere to the highest standards of care and ethical accountability.

This seminal study not only contributes to academic discourse but also lays groundwork for tangible applications that could revolutionize how mental health services are delivered in educational institutions and beyond. The convergence of AI and psychiatry exemplified here invites a future where early detection and intervention become the norm rather than the exception, ultimately advancing public health outcomes on a global scale.

Subject of Research: Evaluating the validity and agreement of AI-adapted screening questionnaires for anxiety and depression compared to validated clinical tools in college students.

Article Title: Evaluating the agreement between ChatGPT-4 and validated questionnaires in screening for anxiety and depression in college students: a cross-sectional study

Article References:
Liu, J., Gu, J., Tong, M. et al. Evaluating the agreement between ChatGPT-4 and validated questionnaires in screening for anxiety and depression in college students: a cross-sectional study. BMC Psychiatry 25, 359 (2025). https://doi.org/10.1186/s12888-025-06798-0

Image Credits: Scienmag.com

DOI: https://doi.org/10.1186/s12888-025-06798-0