LLMs Fail to Replace Human Coding of Adolescent Meaning-Making

General-purpose large language models cannot replace formal manual coding of autobiographical reasoning in adolescent narratives, according to a multi-model validation study published in Behavior Research Methods . The research evaluated the capacity of three AI models to automate meaning-making judgments that depend on implicit event–self connections, a task previously reserved for trained human coders. The study found that performance was strongly condition-dependent, with the best-performing configuration achieving only moderate reliability in the initial run.

The investigation collected autobiographical narratives from 437 Chinese high school students to create a standardized dataset for testing. Autobiographical reasoning is defined in the study as drawing self-relevant meaning from personal experiences, a central component of narrative identity research. Manual coding of this construct requires substantial time and trained coders, prompting the researchers to assess whether large language models could handle the nuanced judgments required for meaning-making assessments in a scalable manner.

The researchers tested three distinct large language models: GPT-4o, DeepSeek-V3, and Qwen-Max. To ensure a rigorous evaluation, they employed four different prompting strategies, including zero-shot, chain-of-thought, three-shot, and five-shot methods. Additionally, the models were assessed using three different scoring granularities: binary, ordinal, and continuous.

This multi-model, multi-strategy approach allowed the researchers to isolate the specific conditions under which the AI models performed best and worst. Among the various configurations tested, the combination of the DeepSeek-V3 model with a zero-shot prompting strategy yielded the highest performance. In the first run of this condition, the model achieved a Cohen’s kappa coefficient of .416, a statistical measure of agreement that suggests moderate but not strong reliability.

When the researchers used a majority-vote method across three separate runs, the accuracy for texts lacking autobiographical reasoning was 54.9%, while the accuracy for texts with moderate to high levels of reasoning was 94.4%. Despite these improvements in specific conditions, the overall performance remained highly dependent on the specific model and prompting strategy used. In the fully crossed subset of the data, statistical analysis revealed that the differences between chain-of-thought and zero-shot prompting did not reach statistical significance for any of the three models.

Paired-bootstrap confidence intervals for the differences between these prompting strategies included zero for all three models, indicating that more complex reasoning prompts did not necessarily improve the models' ability to accurately code meaning-making in this context. Error analysis provided further insight into the limitations of the AI systems. The researchers identified two distinct bias patterns in the models' outputs.

First, there was a high concentration of false negatives in texts with low levels of autobiographical reasoning, with an odds ratio of 3.99 and a p-value less than .001. Second, the models exhibited false positives that were consistent with an over-interpretation of emotional expression. The study concludes that under the specific models, prompts, corpus, and scoring rules tested, general-purpose large language models are not yet capable of replacing formal manual coding of autobiographical reasoning.

Subject of Research: Validation of large language models for automated autobiographical reasoning scoring in Chinese adolescent narratives

Article Title: Can large language models code meaning-making? A multi-model validation of automated autobiographical reasoning scoring in Chinese adolescent narratives

Article References: Wang, B., Zhang, C., Cai, Y., & Li, Z. (2026). Can large language models code meaning-making? A multi-model validation of automated autobiographical reasoning scoring in Chinese adolescent narratives. Behavior Research Methods, 58(12), Article 334. https://doi.org/10.3758/s13428-026-03212-9

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03212-9

Keywords: large language models, autobiographical reasoning, narrative identity, automated coding, Chinese adolescents, AI validation, psychological assessment, prompting strategies, large, language, models, code