AI Chatbots Fail Readability and Source Standards in Newborn Jaundice Study

Artificial intelligence chatbots evaluated as supplementary medical sources for neonatal hyperbilirubinemia provided structured health details but failed standard readability thresholds and exhibited significant limitations in source transparency, according to a web-based cross-sectional comparative study published in BMC Pediatrics. Neonatal hyperbilirubinemia represents one of the most common clinical problems encountered during the neonatal period, often prompting parents to seek clear guidance regarding increased bilirubin levels, phototherapy, exchange transfusion protocols, infant feeding, and clinical follow-up. Because modern digital tools increasingly serve as secondary channels for health inquiries, researchers examined whether the underlying quality, transparency, and linguistic complexity of automated responses are sufficient for patient education.

To investigate the clinical utility and educational performance of conversational models, the investigators designed an evaluation framework focused on four mainstream artificial intelligence platforms: ChatGPT 5.5, Gemini 3.5 Flash, Perplexity, and Copilot. The objective centered on evaluating information quality, source transparency, overall educational value, and reading ease, specifically determining whether generated outputs complied with the sixth-grade reading level widely recommended for general patient education materials. By querying each system under standardized conditions, the cross-sectional investigation sought to benchmark how well current artificial intelligence systems convey complex pediatric conditions to non-specialist caregivers.

The study established a rigorous protocol to identify representative clinical queries. Core questions regarding neonatal hyperbilirubinemia were identified using Medical Subject Headings (MeSH) terminology and Google Trends data to reflect actual public search interest and clinical relevance. From an initial pool of 50 candidate search items, the researchers excluded 22 items and selected 28 core questions for systematic analysis across the four chatbots. Each question was systematically entered into the four chatbot interfaces under identical conditions to ensure consistent comparisons across platforms.

Information quality, completeness, transparency, and overall educational value were evaluated using validated instruments, including the DISCERN instrument, the Ensuring Quality Information for Patients (EQIP) tool, the JAMA benchmark criteria, and the Global Quality Scale (GQS). In parallel, text readability was calculated across six standard linguistic indices: the Automated Readability Index (ARI), the Gunning Fog Index (GFI), the Flesch-Kincaid Grade Level (FKGL), the Coleman-Liau Index (CL), the Simple Measure of Gobbledygook (SMOG), and the Flesch Reading Ease Score (FRES). Differences among the four systems were assessed statistically using the Friedman test, followed by paired Wilcoxon signed-rank tests adjusted with Holm correction for significant findings.

The comparative evaluation revealed statistically significant differences among ChatGPT 5.5, Gemini 3.5 Flash, Perplexity, and Copilot across DISCERN, EQIP, JAMA benchmark criteria, and GQS assessments, with all comparisons yielding P values below 0.001. Copilot and Perplexity demonstrated superior comparative performance on DISCERN, EQIP, and JAMA evaluations relative to the other platforms. However, the researchers noted that despite these higher relative scores, the absolute informational quality of the leading chatbots did not reach an excellent standard, leaving room for material improvement in medical accuracy and consistency.

A notable finding concerned source transparency, where substantial divergence appeared among the platforms. ChatGPT and Gemini demonstrated particularly limited source transparency during testing, frequently omitting verifiable references or citations for their clinical explanations. This lack of clear origin tracking poses challenges for users seeking to verify guidance against authoritative clinical guidelines or consult primary medical literature regarding critical pediatric conditions.

Linguistic analysis demonstrated consistent shortcomings in text accessibility across all four artificial intelligence systems. Every evaluated grade-level readability index exceeded the standard sixth-grade threshold recommended for patient education materials. Furthermore, all recorded Flesch Reading Ease Scores remained strictly below 80. These metric outcomes confirm that the generated responses were written at reading levels too advanced and complex for general patient populations, potentially hindering comprehension among parents seeking quick guidance during neonatal care.

The authors concluded that while artificial intelligence chatbots are capable of delivering structured supplementary information on neonatal hyperbilirubinemia, significant between-model discrepancies, deficient source transparency, and excessive reading complexity remain persistent barriers to safe patient education. To mitigate these shortcomings, the study recommended that future platform updates strengthen authoritative source tracing, disclose specific guideline versions, incorporate content completeness verification, and adopt plain-language generation strategies tailored to individuals with lower health literacy.

Subject of Research: Artificial intelligence chatbots as patient information sources for neonatal hyperbilirubinemia

Article Title: Quality, transparency, and readability of artificial intelligence chatbots as sources of information on neonatal hyperbilirubinemia: a web-based cross-sectional comparative study

Article References: Li, R., Hu, M., Xie, Y., Liu, Y., Pang, X., Li, T., Feng, J., Zhang, D., Peng, Y., & Chen, C. (2026). Quality, transparency, and readability of artificial intelligence chatbots as sources of information on neonatal hyperbilirubinemia: a web-based cross-sectional comparative study. BMC Pediatrics. https://doi.org/10.1186/s12887-026-08057-x

Image Credits: AI Generated

DOI: 10.1186/s12887-026-08057-x

Keywords: neonatal hyperbilirubinemia, artificial intelligence, chatbots, ChatGPT, Gemini, Perplexity, Copilot, pediatric health, patient education, readability, DISCERN, EQIP, JAMA benchmark, Global Quality Scale, health literacy, Flesch Reading Ease