The study, published in Frontiers in Public Health, evaluated ChatGPT, DeepSeek, Perplexity, and Microsoft Copilot against 108 standardized heart failure questions, generating 432 responses that were then dissected by a team of clinicians and researchers. Heart failure affects approximately 64 million people worldwide and demands sustained patient education about medications, diet, fluid restriction, and symptom monitoring. The stakes for getting this information right could not be higher, because poor health literacy in heart failure patients is already linked to worse knowledge, poorer self-care, more hospitalizations, and higher mortality.

The research team, led by investigators from the General Hospital of Northern Theater Command in Shenyang, China, and several Chinese medical institutions, designed a rigorous testing protocol. Each of the 108 questions, drawn from a previously validated patient education question set and categorized into basic knowledge, management, and other heart failure-related issues, was submitted to each chatbot using an identical prompt template. The questions were reviewed by two cardiologists and a heart failure specialist nurse to ensure clinical appropriateness. All queries were conducted between August 1 and 5, 2025, from the same workstation, network, and operating system, with ChatGPT and DeepSeek tested with web browsing disabled and Perplexity and Copilot using their default online retrieval modes.

To assess content quality, three independent raters used two established patient-information instruments: the modified DISCERN scale, which measures reliability and clarity of treatment information, and the Ensuring Quality Information for Patients tool, which evaluates organization, understandability, and actionability. The raters were blinded to which chatbot had produced each response, with answers anonymized and presented in randomized order. Inter-rater reliability was strong, with intraclass correlation coefficients of 0.87 for DISCERN and 0.83 for EQIP, and the modified DISCERN instrument showed good internal consistency with a Cronbach’s alpha of 0.87.

The results revealed a clear frontrunner. DeepSeek consistently achieved the highest content quality scores across all three question domains, with mean modified DISCERN scores of 3.73 in the basic knowledge domain, 4.36 in management, and 4.43 in other heart failure-related issues. Perplexity consistently scored lowest, with EQIP percentages falling into the poor quality range in two of three domains. According to the predefined EQIP thresholds, ChatGPT, DeepSeek, and Copilot all achieved good quality with minor deficiencies across all domains, while no system reached the excellent quality category in any domain.

Readability told a more complicated story. In the primary analysis using one calculator, Perplexity generated the most linguistically complex text, with Simple Measure of Gobbledygook scores reaching 11.98 in the basic knowledge domain, well above the sixth to eighth grade reading level generally recommended for patient education materials. However, when the researchers ran a sensitivity analysis using a second calculator with different algorithms for tokenization, syllable counting, and rounding, the rankings shifted, with Copilot showing the highest complexity scores instead. This discrepancy underscores a critical methodological point: readability estimates depend heavily on the specific formula and calculator used, and no single number can capture how easily a patient will actually understand AI-generated health advice.

The most alarming findings came from the hallucination audit. Two independent reviewers with cardiovascular surgical training cross-verified every one of the 432 responses against the 2022 American Heart Association, American College of Cardiology, and Heart Failure Society of America guideline and relevant clinical evidence. A response was classified as hallucinated if it contained at least one clinically meaningful statement that was guideline-inconsistent, contradicted established evidence, was unsupported, was overgeneralized beyond the relevant patient population, or could mislead patient education. The results were sobering: DeepSeek produced hallucinations in 5.56 percent of responses, ChatGPT in 7.41 percent, Copilot in 9.26 percent, and Perplexity in 14.81 percent.

Statistical analysis confirmed these differences were not due to chance. Cochran’s Q test showed a significant overall difference among the four systems, and a generalized estimating equation logistic regression model accounting for question-level clustering found that Perplexity had nearly three times the odds of producing a hallucinated response compared to DeepSeek. Pairwise comparisons after Bonferroni correction showed that the differences between ChatGPT and Perplexity, and between DeepSeek and Perplexity, remained statistically significant. Encouragingly, no high-severity hallucinations were identified; all hallucinated responses were classified as low or moderate severity, meaning none would reasonably encourage unsafe action or delay emergency care, though some could plausibly influence medication use or self-management decisions if followed without clinician review.

The researchers emphasize that their findings should be interpreted as a time-specific evaluation rather than a permanent ranking, given how rapidly these systems evolve. They also note important limitations: the single-turn testing design did not replicate real-world patient interactions, the platforms differed in their retrieval capabilities, and the study did not measure actual patient comprehension or clinical outcomes. Because the internal training data and retrieval mechanisms of these systems were not accessible, the team could not determine whether observed differences reflected model architecture, source selection, or underlying data bias.

What emerges from this study is a clear warning wrapped in cautious optimism. AI chatbots are not going away, and their potential to supplement heart failure education is real, particularly for patients who struggle to access reliable information. But the findings make equally clear that fluency, organization, and apparent completeness are not proxies for accuracy. Every system produced at least some hallucinated content, and even the best performer stumbled in roughly one in eighteen responses. Until these tools are validated in real patients and clinical workflows, the researchers conclude that they should be regarded as information-generating aids rather than trusted sources, with guideline-aligned verification and clinician oversight remaining non-negotiable before any patient-facing deployment.

Subject of Research: Comparative evaluation of four AI chatbots for heart failure patient education quality, readability, and hallucination rates

Article Title: Comparative quality, readability, and factual reliability of four publicly accessible chatbot systems for heart failure patient education: a cross-sectional evaluation

Article References: Comparative quality, readability, and factual reliability of four publicly accessible chatbot systems for heart failure patient education: a cross-sectional evaluation. (n.d.). https://doi.org/10.3389/fpubh.2026.1966612

Image Credits: AI Generated

DOI: 10.3389/fpubh.2026.1966612

Keywords: artificial intelligence, chatbots, heart failure, patient education, hallucination, large language models, health literacy, readability, clinical guidelines, digital health, medical misinformation, DeepSeek