AI Chatbots Flunk the Readability Test When Answering Cataract Questions

Millions of people heading into cataract surgery now ask an artificial intelligence chatbot what to expect before they ask their doctor. A new systematic benchmark suggests that this habit deserves caution, because the answers these systems produce can be expertly organized yet surprisingly hard to read. The study, published in Frontiers in Ophthalmology, evaluated five mainstream large language models on their ability to generate patient education materials about cataract, the leading cause of reversible blindness worldwide and a disease concentrated in aging populations.

Researchers from the Eye Hospital and School of Ophthalmology and Optometry at Wenzhou Medical University, working with the National Clinical Research Centre for Ocular Diseases, tested four widely used Chinese domestic models, Doubao, DeepSeek, Wenxin Yiyan, and Tongyi Qianwen, alongside the international benchmark GPT-5. Two clinical experts with more than three years of experience developed twenty frequently asked cataract questions on June 11, 2026, drawing on outpatient consultations, the American Academy of Ophthalmology’s Preferred Practice Pattern for Cataract, national Chinese cataract guidelines, and common patient search queries from a tertiary eye hospital’s online consultation records. The questions spanned five thematic categories covering the entire cataract care pathway: risk and prevention, early screening, preoperative management, surgical diagnosis and treatment, and postoperative management.

Each model received the same standardized prompt template, written in Mandarin Chinese for a general public and patient audience, with instructions to use plain language, structured formatting, and to avoid unnecessary technical terms. Every question was submitted in a fresh chat session to prevent context carryover, and generation parameters such as temperature and maximum output length were held consistent wherever the platforms allowed user control. The result was one hundred outputs, twenty questions across five models, exported as plain text for analysis. Readability was measured with seven established indices, including the Coleman-Liau Index, Linsear Write, Automated Readability Index, SMOG, Gunning Fog Index, Flesch Reading Ease Score, and Flesch-Kincaid Grade Level. Quality and educational suitability were scored with the Global Quality Scale, a one-to-five measure of logical rigor and practical value, and the Chinese version of the Patient Education Material Readability Assessment Tool, a twenty-four-point binary instrument covering intelligibility and practicability.

The evaluators were blinded to which model had produced each response, and inter-rater reliability was substantial, with Cohen’s kappa of 0.82 for the Global Quality Scale and an intraclass correlation coefficient of 0.79 for the assessment tool. Statistical analysis accounted for the paired structure of the data, since every model answered the same twenty questions, using repeated-measures ANOVA and Friedman tests with Bonferroni-corrected post hoc comparisons, and linear mixed-effects models with question identity as a random intercept for content-level analysis.

The headline finding is a striking dissociation between quality and readability. GPT-5 achieved the highest scores on both quality instruments, with a mean Global Quality Scale score of 4.15 out of 5 and a mean assessment tool score of 12.95 out of 24, significantly outperforming Tongyi Qianwen and Wenxin Yiyan on overall quality. Yet GPT-5 also produced some of the most linguistically complex text, with elevated scores on the Automated Readability Index, Gunning Fog Index, Flesch-Kincaid Grade Level, Coleman-Liau Index, and SMOG. Tongyi Qianwen fared worst on readability, posting the highest difficulty scores across nearly every index, including a mean Coleman-Liau Index of 22.27 and a mean Flesch-Kincaid Grade Level of 19.20, while Doubao generated the easiest-to-read material with the highest Flesch Reading Ease Score but did not earn the top quality ratings.

Correlation analysis confirmed that this pattern is not a coincidence. Spearman’s rank correlations between the quality metrics and the readability indices were generally weak, with the assessment tool score showing coefficients close to zero against all traditional readability measures. The Global Quality Scale showed only modest associations, a weak positive correlation with reading ease at 0.22 and weak negative correlations with difficulty indices ranging from -0.24 to -0.27. Meanwhile, the readability indices correlated strongly with one another, with the Automated Readability Index and Flesch-Kincaid Grade Level reaching 0.95, indicating that they function as a coherent cluster measuring text difficulty but exert little direct influence on content quality. In practical terms, a chatbot can produce an answer that experts judge reliable and useful while still demanding a college-level reading burden from an elderly patient.

The content of the question mattered as much as the model answering it. Readability differed significantly across the five thematic categories for most indices. Questions about preoperative management and surgical diagnosis and treatment generated the most complex text, with mean Gunning Fog scores of 20.70 and 19.35 respectively, while postoperative management and risk and prevention topics were markedly easier to read and tended to show better educational suitability. The researchers suggest this is because prevention and aftercare can be expressed as concrete behavioral instructions, whereas surgical topics require more technical explanation. Notably, when the paired data structure was properly modeled, the differences in quality scores across content categories were not statistically significant, suggesting that topic mainly shapes language complexity rather than informational value.

The study has important limitations that the authors acknowledge candidly. The readability formulas were originally developed for English, so their absolute scores may not precisely reflect reading difficulty in Mandarin Chinese, which lacks spaces between characters and has a different syllabic structure; the researchers used them mainly as a relative yardstick for cross-model and cross-topic comparison. Only one response was collected per question per model, making the findings a snapshot rather than a definitive ranking, given the stochastic nature of these systems. The evaluators did not perform an independent systematic fact-check of each response against current clinical guidelines, so high quality scores do not directly establish clinical accuracy, completeness, or safety. And crucially, no real cataract patients participated in the evaluation, leaving actual comprehension, perceived usefulness, and decision-making after reading these texts entirely unknown.

These caveats matter because the intended audience for cataract education is often older adults with variable educational backgrounds, age-related declines in visual function, and heterogeneous health literacy, precisely the population least equipped to parse dense technical prose. The authors conclude that large language models show genuine potential for supporting cataract patient education, but that output quality varies significantly by model and topic. Model selection critically affects quality and educational suitability, while content topic mainly drives language complexity. Their recommendation is a dual-axis evaluation framework that treats content quality, meaning what is said, and readability, meaning how it is said, as complementary rather than interchangeable steps, and they call for expert review, readability optimization, and patient-centered validation before any AI-generated cataract education materials are deployed in clinical settings.

For patients tempted to trust a chatbot’s polished answer over their surgeon’s brief explanation, the message is clear: the most authoritative-sounding response is not necessarily the most readable one, and the easiest-to-read answer is not automatically the most complete. Until these systems undergo rigorous validation with the very populations they aim to serve, AI-generated health information about cataract should be treated as a starting point for conversation with eye care professionals, not a substitute for it.

Subject of Research: Health

Article Title: Performance evaluation of mainstream large language models in cataract science popularization Q&A: a comparative study of readability, quality, and educational suitability

Article References: Hou, Y., Liu, M., Wang, Z., & Jiang, X. (2026). Performance evaluation of mainstream large language models in cataract science popularization Q&A: a comparative study of readability, quality, and educational suitability. Frontiers in Ophthalmology, 6, Article 1943376. https://doi.org/10.3389/fopht.2026.1943376

Image Credits: AI Generated

DOI: 10.3389/fopht.2026.1943376

Keywords: large language models, cataract, patient education, readability, artificial intelligence, ophthalmology, health literacy, Performance, evaluation, mainstream, large, language