AI Chatbots Vary Widely in Shingles Patient-Education Quality, Study Finds

A cross-sectional evaluation of five mainstream large language models has found that the choice of chatbot strongly shapes the quality and readability of health-education text about herpes zoster, commonly known as shingles. The study, conducted by researchers affiliated with hospitals and a technology center in China and accepted for publication in Frontiers in Public Health, generated 100 text responses to standardized shingles questions and scored them with validated assessment instruments. Its central conclusion is that model type is a key determinant of text quality, while readability is shaped independently by both the model and the type of content being explained.

The research team, led by Fengyu Zhang of The Fourth Hospital of Changsha, together with Pengfei Tong of the International Advanced Technology Application Promotion Center in Hefei and Hucai Yang of Ningbo No 2 Hospital, designed twenty herpes zoster-related questions. The questions were developed jointly by a dermatologist and a pharmacist and were sorted into five content themes, a structure that allowed the investigators to separate the influence of what was being asked from the influence of which model was answering. Between July 10 and 15, 2026, each question was entered into five large language models: GPT-5, Doubao, DeepSeek, Qianwen, and Gemini, producing one hundred textual responses in total.

Two complementary instruments were used to judge the outputs. Readability was quantified with seven separate indices, and the statistical design was layered: one-way repeated-measures analysis of variance with Greenhouse–Geisser correction compared readability outcomes, Friedman tests compared the C-PEMAT-P raw totals and the ordinal GQS scores, and post-hoc pairwise comparisons were adjusted using the Holm method to control for multiple testing. The headline result was a consistent and statistically strong separation among the five models.

Significant overall differences were observed for all seven readability indices and for both assessment scores, with all P values below 0.001. Median C-PEMAT-P raw total scores ranked the models in descending order: GPT-5 at 12.50, Doubao at 11.00, DeepSeek at 10.00, and Qianwen and Gemini tied at 9.00. The corresponding median Global Quality Scores were 5.00 for GPT-5, 4.00 for both Doubao and DeepSeek, and 3.00 for both Qianwen and Gemini.

After Holm correction, GPT-5 scored significantly higher than each of the other four models on both measures, making it the only model that stood clearly apart at the top of the field. Beneath that leading position, the differences became more nuanced. No significant difference in C-PEMAT-P raw totals remained between DeepSeek and either Qianwen or Gemini, nor between Doubao and DeepSeek, once the adjustment for multiple comparisons was applied, with all adjusted P values equal to 0.062.

Qianwen and Gemini did not differ significantly from each other either, at an adjusted P of 0.766. In other words, the statistical evidence cleanly separates GPT-5 from the rest, while the remaining four models form overlapping groups in which the observed rank order should not be overinterpreted. The strength of the overall effects was substantial by conventional standards.

Kendall's W, a coefficient of concordance, was 0.611 for the C-PEMAT-P comparisons and 0.779 for the Global Quality Score comparisons, indicating that the models were ranked consistently across the twenty questions rather than trading places from one item to the next. An exploratory pooled Spearman correlation analysis also revealed a strong positive association between the two assessment scores, with ρ equal to 0.853, suggesting that materials judged more suitable for patient education also tended to be judged of higher overall quality. Readability told a different story from quality.

The authors report that readability was independently influenced by both model type and content category, and that the two factors are relatively independent of each other. Because the five content themes were fixed across all five models, the design allowed these two sources of variation to be disentangled rather than confounded. From these findings the authors draw a practical conclusion: in clinical applications, GPT-5, DeepSeek, and Doubao are preferred.

Several boundaries on the findings deserve emphasis. The evaluation covered a single disease topic, herpes zoster, and a fixed set of twenty questions assessed during one week in July 2026, so the results speak to that snapshot of five models rather than to the rapidly evolving systems themselves. Within those limits, the study offers a quantified benchmark: for shingles education, the chatbot a patient or clinician chooses is not a neutral detail but a measurable determinant of how understandable, suitable, and well-rated the resulting information will be.

Subject of Research: Evaluation of large language model chatbots as sources of patient-education information on herpes zoster

Article Title: Artificial intelligence chatbots as information sources for herpes zoster: an evaluation of patient-education suitability, overall quality, and readability

Article References: Zhang, F., Tong, P., & Yang, H. (2026). Artificial Intelligence Chatbots as Information Sources for Herpes Zoster: An Evaluation of Patient-Education Suitability, Overall Quality, and Readability. Frontiers in Public Health, 14, 1961492. https://doi.org/10.3389/fpubh.2026.1961492

Image Credits: AI Generated

DOI: 10.3389/fpubh.2026.1961492

Keywords: herpes zoster, large language models, chatbots, patient education, readability, GPT-5, DeepSeek, Doubao, health communication, C-PEMAT-P, Global Quality Score, cross-sectional study