The research uniquely focusses on the perspective of the average internet user, a group that frequently turns to AI as a modern-day symptom checker, reminiscent of how Google was traditionally used for preliminary health information. This user-centered approach is critical because prior studies predominantly examined LLMs from expert or academic lenses, often overlooking practical consumer interactions. By focusing on typical health queries submitted by laypersons, the study offers vital insights into the effectiveness and safety of AI-based medical advice in daily life.

To gather authentic data reflecting real-world usage, the research team organized an innovative event known as the “Diagnose-a-thon” at Penn State. This competition attracted 34 participants spanning faculty, staff, and students across various academic levels. Participants generated a substantial dataset of 212 health-related prompts, encompassing both genuine and hypothetical conditions, crafted from patient and clinician viewpoints. They then queried four distinct state-of-the-art LLMs: ChatGPT-4o, ChatGPT-3.5, Gemini-1.5 Pro, and Llama3-8b. By allowing participants to select their preferred AI model without constraints, the study faithfully replicated the autonomous and diverse usage patterns found in natural settings.

An essential part of the study involved a rigorous evaluation stage where nine board-certified physicians assessed the treatments and information handed back by the LLMs. The evaluation metric was comprehensive, assessing both the clinical accuracy and the potential harm posed by the AI-generated answers, measured on a nuanced six-point scale from very low to very high. This detailed scoring system illuminated how AI diagnostic responses vary across medical specialties and contexts, a level of granularity rarely seen in previous AI investigations.

The findings showed a variable performance landscape across medical disciplines. Obstetrics, gynecology, and otolaryngology yielded the highest levels of correct information with minimal risks, showcasing scenarios where LLMs currently excel. Conversely, fields such as internal medicine, neurology, and dermatology demonstrated more significant challenges for AI systems, where inaccuracies and higher harm potentials were more prevalent. These results underscore an important reality: certain specialized medical domains demand more caution when leveraging AI tools, especially if these tools are employed by untrained individuals.

A fascinating specificity in the study revealed that prompts with a length between 60 and 250 characters tended to produce more accurate AI responses. This suggests that message framing and prompt articulation play crucial roles in steering AI models toward clinically valid outputs. Moreover, highly specialized or narrowly focused questions posed difficulties, suggesting that broad generalist models still face significant hurdles when addressing deeply technical or nuanced medical issues.

Beyond evaluating off-the-shelf AI models, the research team experimented with a novel augmentation approach by retraining the base LLMs using an extensive corpus of medical textbooks, clinical guidelines, and peer-reviewed literature typical of medical school curricula. The goal was to determine whether such domain-specific tuning could enhance clinical validity while reducing harmful outputs. Surprisingly, medical professionals and trainees reviewing these augmented models showed a preference for responses from the original Gemini and Llama bases over the retrained versions. No statistically significant preference was observed regarding ChatGPT’s base versus augmented models. This counterintuitive result suggests that current fine-tuning strategies may not straightforwardly translate into improved clinical communication by AI.

The implications of these findings are profound for the future integration of AI into healthcare delivery. As Dr. Jennifer Kraschnewski, a co-author of the study and a practicing physician, articulates, AI represents a transformative force with the potential to augment clinician capabilities rather than replace human doctors. The challenge lies in harnessing AI tools in ways that bolster medical professionals’ diagnostic processes, reduce cognitive burdens, and improve patient outcomes without exposing patients to the risks of AI errors in unsupervised contexts.

Crucially, the study emphasizes that despite satisfactory accuracy scores in the mid-70s percentage range, the AI models still exhibited an error rate exceeding 20%. This rate is approximately double that of human physicians and highlights the potential for AI to propagate misinformation leading to harm if used uncritically by patients themselves. Such statistical insights counsel for cautious and responsible deployment of AI technologies in healthcare, underscoring the necessity of preserving human clinical oversight.

The study also offers a nuanced view on AI’s evolving role: rather than supplanting the physician’s role, AI could serve as a catalyst to “upskill” clinicians by providing rapid evidence summaries, differential diagnosis suggestions, and decision support, streamlining care processes. The research community is thus encouraged to focus on developing AI systems tailored to professional use, with interfaces and interpretability tuned for clinical environments.

Penn State’s research ecosystem facilitated this multidisciplinary collaboration, bringing together expertise in informatics, intelligent systems, clinical medicine, and AI ethics. Their participatory research design, which mimics user autonomy and real-world interaction dynamics, sets a new methodological standard for evaluating AI systems in societally critical domains. It also expands the discourse on AI accountability and transparency by highlighting the tangible benefits and limitations observed when AI systems engage with health-related content.

Given the inevitable persistence of AI tools in healthcare, public education and digital literacy emerge as pivotal. The study’s co-authors advocate for initiatives that enhance consumer understanding of AI’s strengths and weaknesses in medical diagnosis. Such literacy efforts will empower users to critically appraise AI-generated advice, reducing overreliance and potential misuses.

In summary, this Penn State study, to be presented at the 2026 ACM Fairness, Accountability, and Transparency (FAccT) conference, offers a watershed moment in understanding how large language models intersect with everyday healthcare. Their findings resonate with a dual narrative: AI carries tremendous promise to revolutionize medical diagnostics and patient care when stewarded responsibly, but also harbors non-negligible risks, particularly if accessible without proper clinical guidance. As artificial intelligence advances, the path forward must balance innovation with prudence, ensuring these systems enhance rather than undermine the intricate art of medicine.

Subject of Research: Evaluation of large language models’ accuracy and safety in responding to everyday health-related queries by general users.

Article Title: Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases

News Publication Date: 25-Jun-2026

Web References:
10.48550/arXiv.2506.13805
2026 ACM FAccT Conference

References:
The study data is derived from peer evaluations by board-certified physicians, augmented training on medical textbooks and peer-reviewed articles, and participatory crowdsourced clinical cases generated during the Diagnose-a-thon event hosted by Penn State’s Center for Socially Responsible Artificial Intelligence.

Keywords

Generative AI, Artificial Intelligence, Large Language Models, Healthcare, Medical Diagnosis, Clinical Accuracy, AI Ethics, Doctor-Patient Relationship, AI Safety, Medical Informatics, Healthcare Technology, AI in Medicine

The research revealed that AI systems demonstrate considerable proficiency when addressing factual and procedural medical questions, aligning closely with established expert knowledge in these domains. Nevertheless, these models frequently faltered when confronted with nuanced questions requiring explanatory reasoning—commonly framed as “why” and “how” types—highlighting the persistent gap between algorithmic output and the depth of human clinical insight. This disparity underscores the current boundaries of AI’s interpretive and contextual understanding in complex healthcare scenarios, which remain fundamentally reliant on experiential judgment.

One of the study’s most striking findings concerned the consistency of AI responses. Within a given session, AI systems maintained stable answers, yet when the same queries were posed across multiple sessions, variations emerged. Such discrepancies raise significant concerns, especially when diagnostic accuracy and patient safety hang in the balance. This inconsistency suggests that while AI can be a valuable aid, it cannot yet replace the nuanced and adaptive thinking that human clinicians bring to evolving medical cases. These results call for ongoing refinement of AI algorithms to enhance reliability and foster trust among users.

The research further delves into the qualitative aspects of AI-generated responses, particularly their emotional resonance and communicative style. Unlike human clinicians, whose responses exhibited variable lengths tailored to the complexity of inquiries, AI answers were notably uniform, generally comprising between 400 and 475 words regardless of the question’s nature. Moreover, vocabulary analysis revealed AI’s tendency to employ clinical jargon without adapting its language to patient comprehension or emotional sensitivity. This mechanical delivery often lacked the empathy critical in contexts such as mental health discussions or terminal illness consultations, where human warmth and compassion fundamentally shape therapeutic rapport.

Experts consulting on the study emphasized that medical practice hinges on interpersonal connections unreplicable by AI. Physical presence, nuanced communication, and empathetic engagement form the cornerstone of effective healing, roles that technology cannot supplant. Kelley Strout, associate professor at UMaine’s School of Nursing, highlighted that the true transformative potential lies in synergistic integration—where AI augments clinical judgment and compassion rather than attempting to substitute human care providers. Such integration, however, mandates stringent ethical frameworks and vigilant oversight to preempt errors and unintended consequences.

Contextualizing the study within the broader healthcare landscape reveals a striking urgency propelled by systemic strain, particularly in the U.S. The nation grapples with acute shortages in primary and specialty care providers, exacerbating wait times, inflating costs, and disproportionately affecting rural populations. Projections paint a sobering picture: nonmetropolitan areas alone are expected to face a 42% shortfall in primary care physicians by 2037, intensifying existing healthcare disparities. In parallel, the aging population—projected to increase by more than 50% among those aged 65 and older between 2022 and 2026—further amplifies the demand for effective and accessible health services.

Against this backdrop, AI emerges as a potential ally in alleviating some pressure points. The technology could offer round-the-clock virtual assistance, triaging capabilities, and augment patient-provider communication via portals and remote platforms. Yet, researchers caution that the rapid rollout of AI tools, absent comprehensive regulatory guardrails and ethical safeguards, risks eroding care quality and may exacerbate societal inequities, especially if AI systems are trained on limited, non-representative datasets. Ensuring inclusivity in AI development is paramount to avoiding the reinforcement of existing healthcare biases.

A critical lesson drawn from prior technological adoptions—such as the widespread implementation of electronic health records (EHR)—resonates through the study. Despite the promise of EHRs to streamline workflows and improve outcomes, many systems were originally designed around billing imperatives, not clinical efficacy or user experience, resulting in provider dissatisfaction and compromised patient engagement. The study’s experts urge that AI developers heed these mistakes by centering patient outcomes and provider workflows in system design, thus fostering tools that genuinely enhance care delivery rather than provoke frustration or disengagement.

Moreover, the study highlights the pressing need for addressing accountability and patient privacy in the context of AI’s increasing role in clinical decision-making. Ethical concerns loom large, demanding thoughtful policies tailored to the regulatory and cultural environments of implementation locales. Transparency surrounding AI decision processes and mechanisms for error reporting and correction will be essential for widespread acceptance. Without these foundational pillars, AI’s role risks becoming a source of ambiguity and mistrust rather than clarity.

Despite the challenges, the study supports a growing consensus: AI technologies hold immense potential to optimize healthcare by augmenting rather than replacing human providers. By efficiently sifting through vast datasets and highlighting patterns, AI can expedite diagnosis and recommendation processes in ways previously unattainable. However, the emotional intelligence that human clinicians bring, coupled with their ethical judgment, remains irreplaceable in delivering patient-centered care. Balancing these dimensions stands as the new frontier in digital health evolution.

Future research directions outlined by the study underscore the importance of advancing AI’s interpretive capabilities while concurrently managing ethical risks. Tailoring AI tools to diverse healthcare systems—accounting for differences in regulation, culture, and infrastructure—is critical to ensuring equitable and effective deployment. Such adaptations will enable AI to function as a truly supportive asset, enhancing the humanity and efficiency of medical practice without undermining the clinician-patient relationship.

Technological progress in artificial intelligence is poised to reshape healthcare delivery profoundly. Yet, as C. Matt Graham, author of the study, poignantly states, “Technology should enhance the humanity of medicine, not diminish it.” The crux of innovation lies in designing AI systems to serve as complementary extensions of the clinician’s expertise—intelligent assistants enabling more informed, compassionate, and timely care—rather than autonomous decision-makers disconnected from the nuances that define human-centered medicine.

As healthcare systems worldwide wrestle with growing logistical complexities, workforce shortages, and expanding patient needs, the integration of AI offers both formidable opportunities and daunting challenges. This University of Maine study represents a pivotal stepping stone toward clarifying AI’s role, charting a course that maximizes benefits while safeguarding core humanistic values intrinsic to the art and science of healing.

Subject of Research: Comparative analysis of AI and human clinician diagnostic performance on complex medical queries across different healthcare systems

Article Title: Artificial intelligence vs human clinicians: a comparative analysis of complex medical query handling across the USA and Australia

News Publication Date: 27-May-2025

Web References:

• Journal article: https://www.emerald.com/insight/content/doi/10.1108/jhom-02-2025-0100/full/html
• DOI: http://dx.doi.org/10.1108/JHOM-02-2025-0100

References:

• Health Resources and Services Administration (2024). State of the Primary Care Workforce Report. https://bhw.hrsa.gov/sites/default/files/bureau-health-workforce/state-of-the-primary-care-workforce-report-2024.pdf

Keywords: Artificial intelligence, Generative AI, Machine learning, Computer science, Technology, Health and medicine, Health care, Health care delivery, Clinical medicine