The research team, led by Ayşe Gül Güven of the Division of Adolescent Medicine at Ankara University Faculty of Medicine, together with colleagues from adolescent medicine, child and adolescent psychiatry, public health and clinical nutrition, conducted a cross-sectional content analysis of ChatGPT’s performance. They constructed 34 clinical questions drawn directly from the American Psychiatric Association’s 2023 eating disorders guideline, covering the full arc of care from recognition and diagnosis through nutritional rehabilitation and medical monitoring to psychological and pharmacological treatment. Each question was posed to ChatGPT, and the responses it generated became the dataset for expert evaluation. Because the study analyzed AI-generated text rather than human participants, ethical approval was not required, and the authors note that ChatGPT was used solely as the subject of evaluation, not as a tool in study design, data analysis or manuscript preparation.

To judge the responses, the researchers assembled a panel of nine independent experts spanning three professional groups: physicians specializing in adolescent medicine, child and adolescent psychiatrists, and dietitians. This multidisciplinary design was deliberate. Eating disorders sit at the intersection of pediatrics, psychiatry and nutrition science, and each discipline brings a distinct lens to questions about refeeding protocols, weight restoration targets, family-based treatment and the medical complications of starvation. Every expert rated the same AI-generated answers, allowing the team to measure not only how good the responses were but also how consistently professionals from different backgrounds agreed on that judgment.

Quality and reliability were quantified using two established instruments. The Global Quality Score, or GQS, rates health information on a scale in which higher values indicate better overall quality, taking into account accuracy, completeness and suitability for the intended audience. The modified DISCERN scale, or mDISCERN, evaluates the reliability of published health information by examining whether sources are cited, whether aims are clear, and whether the material distinguishes evidence from speculation. The mean GQS scores pointed to high quality across all three clinical domains tested: diagnosis and assessment earned 4.27 plus or minus 0.39, nutritional management and medical monitoring earned 4.25 plus or minus 0.50, and treatment and clinical interventions earned 4.37 plus or minus 0.33. The corresponding mDISCERN scores of 29.24 plus or minus 4.25, 30.99 plus or minus 3.12 and 29.75 plus or minus 4.66 indicated reasonable reliability, with the treatment domain performing particularly well.

One of the most striking findings was the consistency of evaluation across professional backgrounds. Statistical testing revealed no significant differences in GQS or mDISCERN scores between the adolescent medicine physicians, the psychiatrists and the dietitians, with all comparisons yielding p values above 0.05. In practical terms, this means that a nutrition specialist judging an AI answer about refeeding and a psychiatrist judging an AI answer about psychotherapy converged on similar assessments of quality. Such convergence strengthens confidence that the high scores were not an artifact of one discipline’s perspective but reflected genuine alignment between the chatbot’s output and guideline-based clinical content.

Agreement among raters, however, was not uniform. The researchers calculated Fleiss’ kappa coefficients, a statistical measure of inter-rater agreement that corrects for agreement expected by chance, and found values ranging from 0.28 to 0.56 across disciplines. Kappa statistics in this range span what researchers conventionally label fair to moderate agreement, indicating that while the experts broadly concurred, individual judgments about the reliability of specific responses varied. This variability is itself informative: it suggests that evaluating AI-generated clinical content is not a fully standardized exercise, and that different specialists may weight aspects such as completeness, safety caveats or source transparency differently when reading the same answer.

Readability emerged as the study’s clearest caution. Using multiple standard readability indices, the team found that ChatGPT’s responses to eating disorder questions demanded a reading level spanning from high school to college difficulty. For a topic in which patients are often adolescents and the people searching for answers are frequently worried parents, this is a meaningful barrier. Health communication research has long argued that accessible medical information should target roughly an eighth-grade reading level, far below what the chatbot produced here. An AI system can be factually accurate and still fail a family that cannot comfortably parse its sentences, and the authors highlight this gap as a key limitation of relying on such tools for patient-facing information.

The findings arrive amid explosive growth in the use of large language models for health queries. These models generate fluent, confident-sounding text by predicting likely word sequences based on vast training corpora, and their outputs can look authoritative even when they drift from evidence. That is precisely why guideline-concordance studies matter. By comparing AI answers against a curated, expert-vetted document such as the APA’s 2023 eating disorders guideline, researchers can measure whether the model’s statistical fluency translates into clinically defensible content. In this case, the answer was largely yes: the chatbot demonstrated substantial concordance with the guideline’s recommendations across diagnosis, nutritional management and treatment, a result the authors describe as evidence that AI systems may serve as complementary sources for accessing guideline-based information.

The researchers are careful, however, to frame the result within its limits. Reliability scores, while reasonable, showed variability, and the Fleiss’ kappa range indicates that even trained clinicians did not always agree on how dependable a given response was. Readability levels were high enough to raise concerns about equitable access to the information. And the study evaluated a single AI system’s responses at a single point in time; large language models are updated frequently, and their outputs can differ between sessions, meaning performance measured today may not describe the system a patient encounters tomorrow. The authors conclude that artificial intelligence tools should be used cautiously and should not replace clinical expertise or evidence-based guidelines in clinical decision-making.

For clinicians, the study offers a practical message: AI chatbots are neither a minefield to be banned nor an oracle to be trusted, but a rapidly evolving information layer whose quality can and should be audited against authoritative standards. For families navigating the frightening early days of a suspected eating disorder, the takeaway is more direct. The answers a chatbot provides may indeed reflect what psychiatric guidelines recommend, but the reading level may be challenging, the reliability may vary from question to question, and nothing generated by a language model replaces the assessment of a trained adolescent medicine physician, psychiatrist or dietitian. As AI systems become more embedded in everyday health searches, studies of this kind provide the benchmark data needed to hold them to the standards of the professions whose knowledge they summarize.

Subject of Research: Evaluation of ChatGPT response quality, reliability, and readability against APA guidelines for adolescent eating disorders

Article Title: Consistency of ChatGPT responses with the American Psychiatric Association guidelines in adolescent eating disorders: evaluation of quality, reliability, and readability

Article References: Güven, A. G., Mirioğlu, S., Kurt, B., Temeltürk, R. D., & Aycan, Z. (2026). Consistency of ChatGPT responses with the American Psychiatric Association guidelines in adolescent eating disorders: evaluation of quality, reliability, and readability. Journal of Eating Disorders. https://doi.org/10.1186/s40337-026-01749-w

Image Credits: AI Generated

DOI: 10.1186/s40337-026-01749-w

Keywords: ChatGPT, artificial intelligence, adolescent eating disorders, American Psychiatric Association guidelines, large language models, Global Quality Score, modified DISCERN, readability, anorexia nervosa, clinical decision-making, Ankara University, Journal of Eating Disorders

The study, led by Alexandros Kalkanis and first author Styliani Papadopoulou, employed a structured content-analysis design rather than impressionistic sampling. The researchers selected 127 TikTok videos addressing insomnia management and categorized them by the type of account that produced them: 66 videos came from general users with no stated medical credentials, 20 from physicians, 32 from other healthcare professionals such as nurses, psychologists, or allied health practitioners, and 9 from brand-affiliated accounts, typically promoting commercial sleep products. Each video was then rated with the Global Quality Score, or GQS, a validated five-point instrument widely used in health communication research to grade the educational value of online material. A score of 1 indicates very poor quality with no useful information, while a score of 5 denotes excellent quality grounded in scientific evidence. To capture popularity independently of quality, the team also calculated a Viewing and Engagement Index based on the platforms visible metrics, including views, likes, comments, and shares.

The headline finding is stark. Of the 127 videos analyzed, only 20, or 15.7 percent, demonstrated high educational value, and another 11, or 8.7 percent, reached only moderate quality. In other words, roughly three quarters of the insomnia-related content on the platform fell below the threshold of acceptable educational usefulness. The picture darkens further when the researchers examined specific markers of medical reliability. Just 5 videos in the entire sample were consistent with established clinical guidelines for insomnia, such as those reflected in the European insomnia guideline updated in 2023, which emphasizes cognitive behavioral therapy for insomnia as the first-line treatment. Only 5 videos recommended that viewers consult a physician about their sleep problems, and a mere 2 videos cited any scientific references to support their claims. For a condition affecting a substantial proportion of adults, the near-total absence of guideline-concordant advice is a striking gap between clinical practice and public-facing health information.

Not all content creators performed equally badly, and the gradient of quality tracked closely with professional expertise. Approximately 55 percent of videos produced by physicians achieved a GQS of 3 or higher, indicating at least moderate educational value. That figure dropped to 25 percent for other healthcare professionals, 18 percent for general users, and, perhaps most tellingly, 0 percent for brand-affiliated accounts. Not a single commercially sponsored or brand-associated video in the sample reached acceptable educational quality. The authors also found that sponsored videos were significantly associated with lower-quality content, suggesting that commercial incentives on the platform are actively correlated with degraded information. This pattern echoes a recurring theme in digital health communication research: content designed primarily to sell a product tends to omit the nuance, contraindications, and evidence hierarchy that characterize sound medical advice, while amplifying simplistic claims that lend themselves to engaging short videos.

One of the most consequential findings concerns the disconnect between quality and popularity. The researchers found no correlation between Global Quality Scores and engagement metrics, meaning that videos scoring highly for educational value were no more likely to attract views, likes, or shares than videos rated as poor. This absence of a quality-popularity link matters because TikToks recommendation algorithm is engineered to surface content that generates engagement, not content that is accurate. Prior studies of misinformation have documented similar dynamics, in which emotionally compelling or visually striking material outperforms careful, evidence-based explanations. Applied to insomnia, the implication is that a video promoting an unproven supplement or an anecdotal sleep hack is just as likely, if not more likely, to reach a large audience as a physician summarizing cognitive behavioral therapy, the treatment with the strongest evidence base. The algorithm, in effect, is quality-blind.

The methodological choices of the study deserve attention because they shape how the findings should be interpreted. The Global Quality Score has been validated across numerous platforms and health topics, from YouTube videos on peri-implantitis to online material on psychiatric illness, allowing cross-study comparison. The Viewing and Engagement Index serves as a proxy for real-world reach, though it cannot capture how many viewers act on the advice they see or how long the effect lasts. Because the researchers analyzed publicly released TikTok videos, no ethics review was required, and the dataset reflects what any user searching for insomnia content would plausibly encounter. The authors acknowledge that a snapshot of 127 videos cannot represent the entirety of an ever-shifting platform, where content appears and disappears rapidly and where hashtag rankings vary by region and time. Nevertheless, the consistent pattern across creator categories and the alignment with previous studies of TikTok health content lend the conclusions considerable weight.

The context of prior research makes these results less surprising but more urgent. A 2025 observational study in the Journal of Medical Internet Research documented substantial disinformation about psychiatric illness in TikTok videos, and a cross-sectional analysis in the Journal of Oral Rehabilitation reached similarly pessimistic conclusions about videos on sleep and awake bruxism. Studies of Chinese short-video platforms covering CAR-T cell therapy have likewise shown that popularity does not reflect quality. At the same time, the platform’s influence continues to grow, and the World Health Organization announced a collaboration with TikTok in 2024 aimed at promoting more science-based health information, an acknowledgment by both parties that the information environment on the platform needs deliberate improvement. The new findings suggest that, at least for insomnia, that improvement has not yet materialized at the level of the content users actually watch.

Why does this matter clinically? Insomnia is not a trivial complaint. Chronic insomnia is associated with impaired daytime functioning, increased risk of accidents, worsened mental health outcomes, and elevated healthcare utilization. Clinical guidelines consistently recommend cognitive behavioral therapy for insomnia, or CBT-I, as the first-line intervention, with pharmacological treatment reserved for specific circumstances and generally limited in duration. If people managing insomnia turn first to TikTok and encounter videos that never mention CBT-I, rarely suggest professional evaluation, and frequently promote products without evidence, the platform may delay effective treatment, entreat viewers in unhelpful self-management, or expose them to unnecessary expenditures. The finding that only 2 of 127 videos cited scientific references underscores how rarely creators anchor their claims in the evidence base, leaving viewers with no way to distinguish anecdote from established practice.

The study also carries implications for clinicians and health communicators. The relative advantage of physician-created content suggests that when sleep specialists engage on the platform, they can raise the informational bar, yet their output represented only a small share of the sample and did not dominate engagement metrics. The authors point toward a need for sleep medicine professionals and professional societies to produce accessible, high-quality short-form video content, and for platforms to consider whether engagement-driven ranking should incorporate signals of medical reliability, particularly for health-related topics. Given TikToks behavioral influence on users through algorithmic recommendation, as documented in communication research, the presentation order of content may shape health beliefs as powerfully as the content itself. Absent such interventions, the gap between clinical guidelines and viral advice is likely to persist.

Ultimately, this research letter delivers a quantitative foundation for what sleep clinicians have long suspected anecdotally: TikTok is, at present, a poor source of insomnia management information, and its most-viewed sleep advice cannot be assumed to be its best. For the estimated one in ten adults who meet criteria for an insomnia disorder, and the far larger number who experience occasional sleeplessness, the practical takeaway is caution. Viewers encountering viral sleep tips should treat them as entertainment rather than guidance, look for videos created by identifiable medical professionals, and recognize that none of the brand-promoted content in this study met even minimal educational standards. As short-form video consolidates its role as a first stop for health questions, studies like this one provide the evidence base that policymakers, platforms, and professional societies will need if the information environment is to be repaired rather than merely criticized.

Subject of Research: Medical reliability of insomnia management content on TikTok

Article Title: When sleep advice goes viral: medical reliability of insomnia content on TikTok

Article References: Papadopoulou, S., Karkala, A., Engelshove, E., Testelmans, D., Steiropoulos, P., & Kalkanis, A. (2026). When sleep advice goes viral: medical reliability of insomnia content on TikTok. Journal of Clinical Sleep Medicine, 22(1), Article 162. https://doi.org/10.1007/s44470-026-00163-y

Image Credits: AI Generated

DOI: 10.1007/s44470-026-00163-y

Keywords: insomnia, TikTok, social media, sleep medicine, Global Quality Score, health misinformation, engagement metrics, sponsored content, CBT-I, digital health, sleep disorders, health communication