AI Chatbots Can Now Count a Child’s Words Almost as Well as Gold-Standard Software

For decades, speech-language pathologists have faced a stubborn paradox. Language sample analysis—recording a child telling a story and then meticulously counting words, morphemes, and clauses—is one of the most ecologically valid ways to assess language ability, endorsed in professional practice patterns and capable of flagging communication difficulties that standardized tests miss. Yet surveys consistently show that clinicians use it far less than recommended, largely because of time demands and enormous caseloads. A new study from the University of Colorado Boulder suggests that the AI chatbots already sitting in clinicians’ email platforms may offer a partial escape route—though with important caveats about where their computational strengths end.

The study, published in Frontiers in Language Sciences, examined the association and agreement between microstructure measures calculated by four large language model chatbots—OpenAI’s ChatGPT, Microsoft’s Copilot, MagicSchool AI, and Google Gemini—and the same measures generated by Systematic Analysis of Language Transcripts, or SALT, the dedicated software that serves as a de facto criterion standard for this kind of analysis. Eighty-four typically developing children participated, including 54 English monolinguals and 30 Japanese-English bilinguals, ranging in age from 2 to 9 years. Each child retold one of the parallel Dog and Cat stories from the Multilingual Assessment Instrument for Narratives, and the resulting narratives were transcribed and coded following SALT conventions by the principal investigator and a trained research assistant, who achieved 94.9 percent item-by-item agreement on the codes.

The technical preparation of the transcripts is central to understanding the results. After transcription, each narrative was separated by communication units—an independent clause together with all subordinate clauses attached to it. Coordinating conjunctions such as “and,” “but,” and “so” mark the start of a new unit, while subordinating conjunctions such as “because” and “when” keep clauses on the same line. Mazes—filled pauses, false starts, and repetitions—were placed in parentheses. Aside from this unit separation and maze marking, nothing else was done to prepare the transcripts, a deliberately minimal workflow designed to reflect what a busy clinician could realistically replicate.

The researchers then entered the randomly ordered transcripts into each of the four chatbots using two prompts. The first was broad and simply requested the microstructure measures for the transcript provided. The second added calculation parameters for mean length of utterance in morphemes and the subordination index, drawn directly from the SALT Standard Convention manual. Data collection ran from September through November 2025, with the free versions of each chatbot kept on standard settings—ChatGPT running GPT-5, Copilot on its Smart setting, MagicSchool’s Raina on the Fastest setting, and Gemini on Fast.

The headline finding is that all chatbot-calculated measures were significantly associated with their SALT counterparts, but the strength of that convergence depended sharply on the computational complexity of the measure. Mean length of utterance in words and number of total words—measures requiring relatively simple word counts—showed standardized coefficients ranging from 0.90 to 0.97 and 0.92 to 0.98 respectively across the four chatbots. Mean length of utterance in morphemes, number of different words, and mazes also yielded strong associations, with coefficients between 0.74 and 0.97. The subordination index, which requires identifying clause boundaries and distinguishing main from subordinate clauses, was the clear outlier, with coefficients ranging only from 0.43 to 0.70.

Intraclass correlation coefficients, which capture agreement rather than mere association, told a broadly similar story but with more variability. MagicSchool generally showed the highest agreement with SALT, achieving intraclass correlations of 0.97 for number of total words, mean length of utterance in words, and number of different words, and 0.93 for mean length of utterance in morphemes. ChatGPT exhibited the strongest agreement for mazes at 0.93. Gemini lagged behind on several measures, with an intraclass correlation of just 0.76 for number of total words. The subordination index again showed the weakest agreement, ranging from 0.39 to 0.65 across chatbots.

One of the most striking results concerned the moderating effects of the children’s characteristics. Age group significantly moderated the association only for Copilot-calculated number of different words, with a stronger association for school-age children than for preschoolers. Language group, however, moderated associations for several measures across all four chatbots. ChatGPT and Copilot showed stronger associations with SALT for the bilingual group on number of different words, MagicSchool showed stronger associations for bilinguals on the subordination index but for monolinguals on mazes, and Gemini showed stronger associations for bilinguals on mean length of utterance in words, in morphemes, and on number of total words. The researchers caution that these patterns reflect differences in coefficient magnitude within this sample rather than differential computational accuracy, and may partly stem from the bilinguals producing shorter samples—an average of 84 total words compared with 105 for monolinguals—which may have reduced opportunities for calculation errors.

Perhaps the most counterintuitive finding was that greater prompt specificity did not strengthen any of the microstructure calculations. The second prompt, which embedded the SALT calculation parameters for mean length of utterance in morphemes and the subordination index, failed to improve associations for any chatbot—and for Copilot and MagicSchool, the coefficients were actually smaller under the more detailed prompt. The authors suggest that the longer prompt may have increased the workload on each model’s tokenizer, the tool that breaks text into discrete units before processing, potentially pushing the models toward hallucinations. This challenges the widely held assumption that prompt engineering reliably improves chatbot output, and raises an open question about whether it is more effective to add context to the prompt or to embed codes within the transcript itself.

The study also documented the practical friction of working with these models. On separate occasions, Copilot, MagicSchool, and Gemini each calculated the subordination index by dividing the number of subordinate clauses by the number of utterances, yielding abnormally small values. SALT calculates the index by dividing the total number of clauses by the total number of utterances. When given a brief corrective follow-up prompt, each chatbot immediately adjusted its calculations. The authors note that this kind of intervention would negate the time-saving purpose of chatbot-assisted analysis if required routinely, and they suggest that current models may simply lack the capability to consistently discern and code different clause types.

The authors are careful to frame these findings as preliminary methodological evidence rather than proof of clinical readiness. The sample consisted of typically developing children, so generalizability to clinical populations remains untested, and the average number of utterances per sample was only about 15—well below the recommended 50 utterances or three-minute sample. The researchers also emphasize the replicability problem inherent to studying commercial chatbots: the models used in late 2025 may have different specifications now, and their opaque internal workings—the so-called black box problem—make it difficult to explain why performance varies across measures or models. Still, the convergence observed for simple word and morpheme counts suggests that some established language measures may be more amenable to chatbot-assisted calculation than others, providing a foundation for future research on AI-assisted language sample analysis while reinforcing that complex syntactic measures still require substantial validation before any clinical use.

Subject of Research: Neuroscience

Article Title: Convergence between AI chatbot-calculated microstructure measures and SALT analysis of child language samples

Article References: Hayes, R. L., & Kan, P. F. (2026). Convergence between AI chatbot-calculated microstructure measures and SALT analysis of child language samples. Frontiers in Language Sciences, 5, Article 1929126. https://doi.org/10.3389/flang.2026.1929126

Image Credits: AI Generated

DOI: 10.3389/flang.2026.1929126

Keywords: artificial intelligence, ChatGPT, language sample analysis, speech-language pathology, child language, large language models, Convergence, between, chatbot-calculated, microstructure, measures, SALT