AI Reads Parent Questionnaires to Flag ADHD, but a Third of Its Reasons Don’t Hold Up

Large language models can read a parent’s description of a child’s attention problems and predict a psychiatrist’s ADHD diagnosis almost as well as a purpose-built statistical model, according to a temporal validation study published in BMC Psychiatry by researchers at Children’s Hospital of Nanjing Medical University. But the same study found a troubling flaw: between roughly 36 and 42 percent of the clinical claims the models offered as justifications were not directly supported by the questionnaire data they had been given. The work, led by Rongqi Jiao and colleagues, is one of the first psychiatric AI evaluations to judge not only whether a model guesses correctly, but whether its stated reasoning actually traces back to the evidence in front of it.

The study’s design deserves close attention because temporal validation is one of the stricter tests a clinical prediction model can face. Rather than splitting a single pool of patients randomly into training and test sets, which risks leaking subtle information across the boundary, the researchers trained their models on 1,268 cases and then evaluated them on 996 entirely separate cases collected at a later time. This matters because clinical populations drift: referral patterns, questionnaire completion habits, and even the mix of symptom severity in a clinic can shift from month to month. A model that looks strong on a random split may quietly degrade when confronted with tomorrow’s patients. By holding out a genuinely future cohort, the study offers a more honest estimate of how these tools would behave in ongoing clinical use, and the fact that performance held up in the 0.80 to 0.82 AUC range under those conditions is a meaningful finding rather than an artifact of optimistic splitting.

The reference standard also shapes how the results should be read. Diagnoses were made by attending psychiatrists following DSM-5 criteria, and the control group was restricted to children with no other primary psychiatric diagnosis. That choice keeps the task clean but also narrows the clinical reality the models were tested against. In everyday practice, children arrive with overlapping concerns such as anxiety, autism spectrum traits, learning disorders, or sleep problems, and distinguishing ADHD from conditions that mimic it is often the hardest part of the diagnostic work. A model validated only against ADHD-versus-no-other-primary-diagnosis cases has demonstrated performance on a simplified version of the problem, and its accuracy on diagnostically tangled presentations remains an open question.

The questionnaire inputs themselves are worth understanding. The Conners Parent Rating Scale is one of the most widely used parent-report instruments in child mental health, capturing items about inattention, hyperactivity, and related behaviors as seen at home, while the Weiss Functional Impairment Rating Scale measures how much difficulty these symptoms cause across domains like school, family life, and social functioning. The researchers fed the models textual descriptions of responses on these instruments rather than raw scores alone, which is what allows a language model to reason over the material at all. The comparison baselines, LASSO and XGBoost, are established supervised machine learning methods that consume structured features and output probabilities without producing any narrative explanation. That contrast is central to the study’s framing: the question was not whether language models can outperform conventional statistics, but whether they can match them while adding something the statistics cannot, namely a written rationale a clinician could inspect.

The groundedness findings are the study’s most distinctive contribution. The authors used a rule-based annotation procedure to check whether each claim in a model’s explanation could be traced to the input data, and then had human experts validate the method on a subset of 80 cases. The two approaches converged closely, with the automated method flagging 36.2 to 41.7 percent of claims as unsupported and expert assessment finding roughly 38 percent on the subset, with substantial agreement between them. This convergence matters because automated audits of generated text are often criticized as unreliable; here the rule-based method appears to have captured what trained clinicians would also flag. An unsupported claim in this context is not necessarily a false statement about ADHD in general. A model might, for instance, assert that a child’s profile indicates executive dysfunction when the questionnaire responses never speak to executive function directly. The narrative reads plausibly, but the specific judgment rests on evidence that was never provided, which is precisely the kind of gap that can mislead a busy clinician who trusts the fluency of the prose.

The finding that reasoning structure did not reliably distinguish correct from incorrect predictions carries practical weight. One hope for auditable AI in medicine is that the shape or quality of an explanation might serve as a signal of trustworthiness, allowing clinicians to spot shaky reasoning before acting on it. If well-structured, coherent-sounding rationales appear just as often behind wrong answers as behind right ones, that safety net does not exist. The study also examined whether models’ language behavior changed when they erred, and found no dependable pattern. In other words, there is no easy tell, no stylistic fingerprint of failure that a reader could learn to detect.

The grey-zone analysis adds a sobering dimension. Using LASSO composite scores, the researchers identified roughly 38 percent of cases as sitting in an ambiguous middle region where classification is inherently uncertain, and they used principal component analysis to probe whether that uncertainty could be reduced. It could not be eliminated. Within this zone, all methods performed poorly, with the language models achieving kappa values between 0.105 and 0.140 and even the LASSO baseline reaching only 0.228. Kappa, which corrects agreement statistics for chance, treats values in this range as slight at best. The implication is that more than a third of real clinical cases fall into a band where no current method, whether a large language model or a transparent regression model, can reliably reproduce the psychiatrist’s judgment. This is not a failure unique to AI; it reflects genuine diagnostic ambiguity in the underlying population. But it does mean that any deployment narrative built on aggregate accuracy figures would obscure the fact that performance collapses precisely where clinical decisions are hardest.

The calibration and threshold questions also deserve mention. An AUC near 0.81 describes discrimination across all possible cutoffs, but clinical use requires choosing a specific threshold, and the balance between false positives and false negatives carries different consequences in child psychiatry. Over-identifying ADHD risks unnecessary intervention and stigma, while missing it delays support during formative years. The study’s reporting of positive and negative predictive values alongside discrimination metrics reflects an awareness that raw accuracy and real-world utility diverge, particularly when the base rate of diagnosis in the tested population differs from the rate in the deployment setting.

The four-dimension framework the authors propose, covering classification performance, explanation groundedness, language behavior under error, and grey-zone boundaries, offers a template that other psychiatric AI evaluations could adopt. Most published evaluations of language models in mental health have stopped at the first dimension, reporting accuracy or AUC and leaving the question of explanation quality unexamined. By formalizing groundedness as a measurable property with both automated and expert validation, and by explicitly mapping where performance breaks down, the framework pushes the field toward evaluations that ask not just whether a model is right but why it claims to be right and when it should not be trusted at all.

Several limitations frame the conclusions. The data came from a single center, the Children’s Hospital of Nanjing Medical University, and both the questionnaire instruments and the diagnostic practices reflect that setting. Whether the same groundedness failures and grey-zone boundaries appear in other clinics, other languages, or with other instruments is unknown. The models tested were four specific large language models at a particular point in their development, and the field moves quickly enough that newer systems might behave differently, though the structural issues the study identifies, such as unsupported claims and undiagnosable grey zones, are unlikely to vanish with incremental model improvements. The authors themselves state plainly that clinical deployment is not supported by these findings, a conclusion that follows directly from the combination of unsupported rationales, unreliable error signals, and poor performance across a large fraction of cases.

What the study ultimately offers is a calibrated picture rather than a verdict. Large language models can extract diagnostic signal from parent questionnaires at a level comparable to purpose-built machine learning, which is itself a notable technical result given that these are general-purpose systems never trained for this task. They can also produce rationales that make their reasoning inspectable in principle. But the audit reveals that inspection is currently necessary rather than optional, because the narratives frequently outrun the evidence, and the models give no external sign of when they are wrong. For clinicians and researchers watching this space, the practical takeaway is that fluency and coherence in an AI explanation are not proxies for correctness, that transparent statistical baselines remain competitive and easier to verify, and that the hardest cases, which constitute a substantial share of real clinical volume, remain unsolved by any automated method tested to date.

Subject of Research: Evaluation of large language models for predicting ADHD clinical labels from parent questionnaires, including explanation groundedness and failure modes

Article Title: Large language models for ADHD clinical label prediction via parent questionnaires: diagnostic performance, explanation groundedness, and failure modes a temporal validation study

Article References: Large language models for ADHD clinical label prediction via parent questionnaires: diagnostic performance, explanation groundedness, and failure modes a temporal validation study. (n.d.). https://doi.org/10.1186/s12888-026-08760-0

Image Credits: AI Generated

DOI: 10.1186/s12888-026-08760-0

Keywords: ADHD, large language models, parent questionnaires, explanation groundedness, temporal validation, machine learning, child psychiatry, grey zone, diagnostic prediction, LASSO, XGBoost, patient safety