Hospitals nowadays rely heavily on electronic health records (EHRs) – structured datasets that capture patient information, resource availability, and care events. Administrators utilize this data to monitor patient loads, allocate resources, and generate operational reports. Traditionally, these tasks are performed by specialized data analysts deploying programming languages and database queries, a process often fraught with delays when rapid answers are needed for decision-making. The promise of LLMs like GPT-4o and Llama has been to democratize data access by allowing non-technical staff to query these datasets directly using natural language prompts.

In the study, researchers subjected nine leading LLMs to a rigorous battery of tests designed to emulate two foundational administrative functions: counting how many patients meet a specific clinical condition and filtering records based on multiple inclusion criteria simultaneously. The data itself was sourced from a substantial real-world dataset of over 50,000 emergency department visits within the Mount Sinai Health System, grounding the evaluation in practical, messy clinical data rather than synthetic or simplified examples.

The initial experiments employed straightforward prompting techniques, where models were simply asked direct questions such as “How many patients were admitted from this table?” Across the board, all tested LLMs demonstrated subpar accuracy, failing to provide reliable answers when handling these structured queries. This underlines a fundamental disconnect between LLM training—and their practical applicability to real-world numerical and logical operations in healthcare datasets.

To enhance performance, the researchers explored a chain-of-thought prompting approach. This method instructs the model to transparently reason through the problem step-by-step before arriving at the final answer, theoretically enabling more accurate and consistent outputs. However, the results were underwhelming; only modest improvements were observed on smaller tables, and as the size and complexity of the data increased, accuracy declined precipitously. For instance, even GPT-4o, the best performing model under this regime, saw accuracy plummet from approximately 95% on small datasets to below 60% when confronted with larger tables.

Recognizing that prompting alone may not suffice, the research shifted focus to a tool-based model execution approach. Here, LLMs were tasked with generating executable code, such as SQL or Python scripts, to process the data programmatically. This method leverages the LLM’s natural language understanding to translate queries into precise machine-readable commands, which are then run directly against the EHR data for guaranteed accuracy. Impressively, this approach substantially improved results for the most advanced models. GPT-4o and Qwen-2.5-72B demonstrated near-perfect accuracy under these conditions, successfully navigating the intricacies of complex filters and large datasets.

Despite these successes, not all models fared well. LLMs optimized for speed and efficiency, such as distilled variants of DeepSeek, struggled to produce usable outputs even when provided with the ability to generate and run code. Furthermore, the Llama-3.1-8B model encountered major difficulties, failing to produce functional results in the majority of assessments and being ultimately excluded from further analysis. These discrepancies highlight the diverse capabilities within the current LLM ecosystem and caution against broad assumptions regarding their utility in structured data environments.

The study’s findings carry critical implications for the future deployment of LLMs in healthcare administration. Benjamin Glicksberg, one of the authors, emphasized that without integrating tool-based strategies—combining LLM-generated code with actual execution—large language models remain fundamentally unsuitable for standalone use in clinical administrative settings. Clinical workflows frequently involve complex structured data requiring absolute reliability and precision, conditions under which straightforward natural language query processing by LLMs falls short.

Moreover, the requirement for “agentic” approaches is underscored by this work. Agentic AI involves systems that act semi-autonomously, leveraging external tools and code execution capabilities to ensure results remain consistent and verifiable. By integrating LLMs with backend code execution engines, hospitals could dramatically accelerate administrative processes while maintaining data integrity. Such hybrid solutions may bridge the gap between cutting-edge AI capabilities and the stringent accuracy demands of healthcare operations.

This study shines a spotlight on the often-overlooked challenges of applying AI in clinical data environments. While the hype around LLMs centers on their conversational fluency and general knowledge, the ability to perform precise numerical computations and filtered data retrieval within complex EHR systems requires a fundamentally different kind of model reliability. The researchers’ meticulous experimental design and real-world data usage offer a vital reality check for the healthcare sector’s ongoing AI ambitions.

Lastly, the authors note that their work did not receive any external funding, and no competing interests were declared. The open-access publication ensures that the full details, along with extensive methodological descriptions and results, remain available to researchers, clinicians, and AI developers aiming to advance safe and effective AI integration into hospital administration.

Overall, these findings caution healthcare providers and AI developers alike to calibrate expectations around LLMs’ current abilities in administrative contexts. They also highlight the powerful potential unlocked by hybrid human-AI systems that combine natural language understanding with robust programming and execution frameworks. As digital healthcare continues to evolve, researchers and practitioners will need to navigate these complex trade-offs to harness AI’s benefits without compromising accuracy and trustworthiness.

Web References:
https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001326

Subject of Research: Not applicable
Article Title: Large language models are poor clinical administrators: An evaluation of structured queries in real-world electronic health records
News Publication Date: 7-May-2026
References: Klang E, Sorin V, Korfiatis P, Sawant AS, Freeman R, Charney AW, et al. (2026) Large language models are poor clinical administrators: An evaluation of structured queries in real-world electronic health records. PLOS Digit Health 5(5): e0001326. DOI: 10.1371/journal.pdig.0001326

Keywords

Large Language Models, Electronic Health Records, Clinical Administration, Artificial Intelligence, GPT-4o, Tool-based AI, Chain-of-Thought Prompting, Healthcare Data Analytics, AI Reliability, Code Generation, Hospital Resource Management, Clinical Workflow Automation

The core objective of Hu’s study was to interrogate how different versions of ChatGPT handle diagnostic tasks given real-world clinical data. Using de-identified physician notes from 30 emergency department cases, the research team prompted various ChatGPT model iterations—including GPT-3.5, GPT-4, GPT-4o, and the o1 series—to generate their top three diagnostic suggestions. The study’s methodological rigor involved comparing the models’ diagnostic precision and accuracy against actual clinical outcomes to draw a comprehensive performance profile. This approach provides a window into how AI tools can supplement, but not yet replace, human clinical judgment.

One of the profound insights emerging from this investigation is the discrepancy in AI performance between cases with classic, textbook symptoms and those with atypical or “challenging” presentations. For patients exhibiting hallmark signs of disease, ChatGPT models demonstrated promising diagnostic assistance capabilities, supporting physicians by suggesting accurate differential diagnoses. However, when confronted with complex cases lacking traditional symptomatic cues—such as pneumonia cases without accompanying fever—AI’s capacity to correctly identify diagnoses notably diminished. These failures illuminate the inherent difficulty AI models face when operating beyond their training data’s typical patterns, emphasizing the necessity for richer, more diverse datasets.

The researchers note that current AI diagnostic models primarily ingest unstructured text input—in this case, physicians’ notes—without access to multimodal clinical information. Consequently, ChatGPT’s diagnostic reasoning is limited by the breadth and variability of its textual training corpora and the information provided. Hu posits that enhancing future AI frameworks with additional clinical data streams—such as imaging results, laboratory findings, and comprehensive patient histories—could improve the fidelity and robustness of AI-assisted diagnoses in emergency contexts. Integration of these heterogeneous data types would transform AI from a purely linguistic interpreter to a more holistic clinical decision support system.

Analysis of the longitudinal performance of ChatGPT iterations reveals an interesting but cautious trajectory of improvement. While no statistically significant advance was observed when considering the inclusion of AI-generated diagnoses within the top three suggestions, the accuracy of the very top, or primary, diagnosis recommendation improved by approximately 15 to 20 percent in newer models relative to their predecessors. This subtle enhancement suggests iterative refinement in model capabilities but also highlights the persistent challenges in achieving consistently high precision necessary for clinical reliability.

The study underscores a key principle in the deployment of AI-assisted diagnostic tools: the indispensability of human oversight. Given the models’ current inadequate performance on complex cases, physician expertise remains essential to interpret AI outputs critically and corroborate or refute AI-generated hypotheses. This interplay forms a hybrid intelligence paradigm, wherein AI accelerates data synthesis and hypothesis generation while clinicians provide contextual judgment, ensuring that patient care remains both accurate and personalized.

Beyond diagnostic accuracy, Hu envisions AI modalities evolving towards greater transparency and explicability. He stresses the importance of AI systems that do not merely generate results but also reveal their reasoning pathways, enabling clinicians to understand and trust their recommendations. Such “explainable AI” is critical to fostering confidence among healthcare providers, enhancing AI’s integration into clinical workflows, and ultimately improving patient outcomes. Achieving this level of transparency will require methodological innovations in how AI models represent and communicate uncertainty and rationale.

Moreover, Hu’s research team explores imaginative avenues to augment diagnostic reasoning by leveraging multi-agent AI simulations. Drawing on prior work where ChatGPT-4 was deployed in role-playing scenarios—emulating specialists such as physiotherapists, psychologists, and nutritionists engaged in panel discussions—this approach aims to replicate the collaborative diagnostic processes typical in clinical environments. The proposed conversational model suggests that dynamic interactions among diverse AI agents could produce more nuanced, accurate diagnostic assessments, reflecting interdisciplinary integration akin to human medical teams.

Despite these promising strides, the researchers caution that current AI systems, including ChatGPT, do not qualify as certified medical devices and should not be used as standalone diagnostic solutions. In clinical settings where expanded data types, such as imaging, are incorporated, AI models must operate within secure, privacy-compliant hospital clusters as open-source platforms. Compliance with regulatory standards and patient confidentiality laws remains a non-negotiable prerequisite for AI deployment in healthcare institutions.

The study acknowledges support from the National Science Foundation and the National Institutes of Health, emphasizing the significance of federally-funded research in advancing AI applications in medicine. Additional contributors include postdoctoral fellow Jinge Wang, lab volunteer Kenneth Shue, and Li Liu from Arizona State University, reflecting a multidisciplinary collaboration essential to tackling complex problems at the intersection of computer science, bioinformatics, and clinical medicine.

Looking ahead, Hu advocates for future research to focus not only on enhancing AI’s diagnostic performance but also on its capacity to articulate reasoning in clinically meaningful ways. He suggests that improved explainability could facilitate critical emergency department decisions such as triage prioritization and treatment pathway selection, augmenting both efficiency and patient safety.

In summary, the pioneering evaluation of ChatGPT models in emergency diagnostics performed by WVU scientists reveals a nuanced landscape marked by AI’s emerging utility balanced against intrinsic challenges. While encouraging diagnostic accuracy for prototypical cases validates the promise of language models as assistive tools, persistent deficiencies in recognizing atypical disease presentations underscore the imperative for richer data integration, transparent reasoning, and robust human-AI collaboration. This research not only advances scientific understanding of AI capabilities at the clinical frontline but also charts a thoughtful course towards responsible integration of AI in patient-centered care.

Subject of Research:
Evaluation of ChatGPT AI model iterations for diagnostic assistance in emergency department patients using clinical notes.

Article Title:
Preliminary evaluation of ChatGPT model iterations in emergency department diagnostics

News Publication Date:
26-Mar-2025

Web References:

• https://www.wvu.edu/
• https://directory.hsc.wvu.edu/Profile/60888
• https://medicine.wvu.edu/
• https://medicine.wvu.edu/micro/
• https://health.wvu.edu/research-and-graduate-education/research/core-facilities/bioinformatics-core/
• https://www.nature.com/articles/s41598-025-95233-1#citeas
• http://dx.doi.org/10.1038/s41598-025-95233-1
• https://mededu.jmir.org/2024/1/e51157/

References:
Hu, G. M., Wang, J., Shue, K., Liu, L. (2025). Preliminary evaluation of ChatGPT model iterations in emergency department diagnostics. Scientific Reports. DOI: 10.1038/s41598-025-95233-1

Image Credits:
WVU Photo/Greg Ellis

Keywords:
Artificial intelligence, Disease prevention, Clinical medicine, Medical tests, Artificial consciousness, Artificial neural networks, Cognitive robotics, Forward chaining, Generative AI, Genetic algorithms, Logic based AI, Adaptive systems, Cybernetics, Robotics, Computer science