AI Chatbots Flunk the Readability Test When Answering Questions About Painful Mouth Ulcers

Recurrent aphthous stomatitis, the condition most people know as canker sores or mouth ulcers, is one of the most common reasons people type a health question into a search engine or, increasingly, into an AI chatbot. The ulcers are painful, recurrent, and often slow to heal, and because there is no single cause and no universally effective prevention, sufferers frequently seek self-care advice before ever seeing a dentist or physician. That makes the condition a natural test case for the large language models now fielding millions of consumer health questions every day. A new exploratory cross-sectional study published in Frontiers in Oral Health put five publicly accessible chatbots through exactly that test, and the results offer a sobering snapshot of what happens when AI answers questions about a seemingly benign condition that can occasionally mask something far more serious.

The research team, led by Yulong Wang of the Second People’s Hospital in Mudanjiang City and colleagues at Nanjing Medical University-affiliated institutions, developed thirty consumer-oriented English prompts covering seven domains of recurrent aphthous stomatitis information needs. The prompts were distilled from an initial pool of 101 candidate questions, screened for clarity, relevance, and safety importance, and informed by Google Trends and Baidu search-topic patterns from April 2021 to April 2026. Each prompt was submitted once, in a separate session, to ChatGPT, Microsoft Copilot, Gemini, Doubao, and DeepSeek, producing 150 single-turn responses. The investigators deliberately disabled memory, personalization, and browsing extensions where the interfaces exposed such controls, and retained only the first complete response without follow-up or regeneration. Data were collected in Mudanjiang, China, from April 6 to April 8, 2026.

Safety was the primary clinical outcome, and the reviewers applied a prespecified plausible-harm standard: a response was coded as potentially unsafe or misleading when a statement or clinically important omission could plausibly cause material harm, including delayed assessment, inappropriate medication or remedy use, false reassurance, or distorted risk perception. Five dental clinicians with 6 to 26 years of experience independently scored every response before consensus adjudication. Pre-consensus agreement was high, with a Fleiss’ kappa of 0.855 for binary safety and intraclass correlation coefficients ranging from 0.852 to 0.858 for the secondary outcomes, indicating that the panel applied the criteria consistently.

The headline safety finding is that sixteen of the 150 responses, or 10.7 percent, were consensus-coded as potentially unsafe or misleading. Safe-response rates ranged from 83.3 percent for DeepSeek to 93.3 percent for ChatGPT and Copilot, but a Cochran’s Q test detected no statistically significant between-chatbot difference (Q = 2.957, P = 0.565). The authors are careful to note that this non-significant result does not establish equivalence; it simply means no difference was detected in this exploratory sample. The potentially unsafe responses clustered in scenarios requiring qualification or timely escalation rather than routine definitions, including persistent or non-healing ulcers, cancer-related reassurance, home or traditional remedies, supplementation, pregnancy, and pediatric cases.

The qualitative error profile is arguably more informative than the aggregate numbers. The reviewers identified delayed escalation, over-reassurance, overconfident causal claims, restrictive dietary advice, and insufficient qualification of medications, supplements, or traditional products as recurring mechanisms of potential harm. In real-world information seeking, users may self-treat, use supplements or folk remedies, worry about cancer, or delay professional assessment of a lesion that has persisted for weeks. A response that overstated causation, buried warning signs, or offered reassurance without an appropriate persistence threshold could plausibly influence those behaviors. The study prespecified a conservative two-week rule for dental or oral-medicine assessment because uncomplicated minor ulcers typically heal within about two weeks, while acknowledging that some guidance uses a three-week threshold.

Accuracy, supportive communication, and information quality showed a similar pattern of no detected between-chatbot differences. Median accuracy scores ranged from 3.50 to 4.00 on a five-point scale, and median empathy scores from 3.00 to 4.00, with overall P values of 0.915 and 0.361 respectively. Four established instruments were used to assess information quality and transparency: DISCERN, EQIP, the Global Quality Score, and the JAMA benchmark criteria, which evaluate authorship, attribution, disclosure, and currency. None of these measures distinguished the five systems, although the authors caution that the instruments were developed for static written information rather than adaptive dialogue, and uniform application may structurally penalize concise factual responses, particularly on provenance-oriented JAMA items.

Readability was where the chatbots diverged most clearly. Five of six formula-based indices detected statistically significant differences across the systems. ChatGPT and Doubao produced lower estimated grade levels on several formulas, with ChatGPT achieving a mean Flesch-Kincaid Grade Level of 7.10 compared with 8.27 for Gemini. Yet the prespecified public-health target of grade 6 or below was met by only 31 of 150 responses, or 20.7 percent. The authors emphasize that formula scores measure reading difficulty, not comprehension, cultural suitability, or clinical clarity, and that the formula-dependent pattern does not establish clinically meaningful differences in understanding or decision-making.

The study’s limitations tightly constrain interpretation, and the authors are unusually transparent about them. The thirty prompts were researcher-developed from topic sources rather than sampled verbatim from authentic users, and no patients or public contributors generated, prioritized, or validated them. Only English, single-turn, single-generation outputs were tested, so stochastic stability, multilingual use, natural conversations, and multi-turn correction were not evaluated. Chatbot identity was visible to reviewers, the small pragmatic sample was not an equivalence design, and access-tier heterogeneity, hidden provider-side routing, and continuing system evolution further limit reproducibility. The findings cannot be attributed to architecture, training data, or alignment strategy, and they apply only to the recorded user-facing configurations during the defined access period.

What emerges from this study is not a ranking of chatbots but a portrait of a technology that is generally competent yet intermittently dangerous, and consistently harder to read than public-health guidance recommends. The authors conclude that these outputs should be treated only as draft consumer health information requiring professional review, not as autonomous advice or evidence of deployment readiness. They call for referral thresholds, medication and pregnancy cautions, pediatric hydration and intake limits, and the boundaries of self-care to be prominent in any chatbot response, and for developers to test multilingual, repeated, and multi-turn outputs with safety-critical prompts. Future studies, they argue, should use authentic user questions, blinded review where feasible, and direct tests of comprehension, decision-making, and clinical outcomes. Until then, the reassuring fluency of an AI answer about a mouth ulcer should be read with the same caution as any other draft health information: as a starting point for a conversation with a professional, not a substitute for one.

Subject of Research: Large language models and consumer health information on recurrent aphthous stomatitis

Article Title: Safety, accuracy, information quality, readability, and supportive communication of LLM-based chatbots in responses to researcher-developed consumer-oriented prompts on recurrent aphthous stomatitis: an exploratory cross-sectional comparative study

Article References: Wang, Y., Li, J., Gong, Z., Zhuang, Y., & Dong, W. (2026). Safety, accuracy, information quality, readability, and supportive communication of LLM-based chatbots in responses to researcher-developed consumer-oriented prompts on recurrent aphthous stomatitis: an exploratory cross-sectional comparative study. Frontiers in Oral Health, 7, Article 1968034. https://doi.org/10.3389/froh.2026.1968034

Image Credits: AI Generated

DOI: 10.3389/froh.2026.1968034

Keywords: large language models, recurrent aphthous stomatitis, consumer health information, chatbot safety, readability, oral health, Safety, accuracy, information, quality, supportive, communication