The investigation collected autobiographical narratives from 437 Chinese high school students to create a standardized dataset for testing. Autobiographical reasoning is defined in the study as drawing self-relevant meaning from personal experiences, a central component of narrative identity research. Manual coding of this construct requires substantial time and trained coders, prompting the researchers to assess whether large language models could handle the nuanced judgments required for meaning-making assessments in a scalable manner.

The researchers tested three distinct large language models: GPT-4o, DeepSeek-V3, and Qwen-Max. To ensure a rigorous evaluation, they employed four different prompting strategies, including zero-shot, chain-of-thought, three-shot, and five-shot methods. Additionally, the models were assessed using three different scoring granularities: binary, ordinal, and continuous.

This multi-model, multi-strategy approach allowed the researchers to isolate the specific conditions under which the AI models performed best and worst. Among the various configurations tested, the combination of the DeepSeek-V3 model with a zero-shot prompting strategy yielded the highest performance. In the first run of this condition, the model achieved a Cohen’s kappa coefficient of .416, a statistical measure of agreement that suggests moderate but not strong reliability.

When the researchers used a majority-vote method across three separate runs, the accuracy for texts lacking autobiographical reasoning was 54.9%, while the accuracy for texts with moderate to high levels of reasoning was 94.4%. Despite these improvements in specific conditions, the overall performance remained highly dependent on the specific model and prompting strategy used. In the fully crossed subset of the data, statistical analysis revealed that the differences between chain-of-thought and zero-shot prompting did not reach statistical significance for any of the three models.

Paired-bootstrap confidence intervals for the differences between these prompting strategies included zero for all three models, indicating that more complex reasoning prompts did not necessarily improve the models' ability to accurately code meaning-making in this context. Error analysis provided further insight into the limitations of the AI systems. The researchers identified two distinct bias patterns in the models' outputs.

First, there was a high concentration of false negatives in texts with low levels of autobiographical reasoning, with an odds ratio of 3.99 and a p-value less than .001. Second, the models exhibited false positives that were consistent with an over-interpretation of emotional expression. The study concludes that under the specific models, prompts, corpus, and scoring rules tested, general-purpose large language models are not yet capable of replacing formal manual coding of autobiographical reasoning.

Subject of Research: Validation of large language models for automated autobiographical reasoning scoring in Chinese adolescent narratives

Article Title: Can large language models code meaning-making? A multi-model validation of automated autobiographical reasoning scoring in Chinese adolescent narratives

Article References: Wang, B., Zhang, C., Cai, Y., & Li, Z. (2026). Can large language models code meaning-making? A multi-model validation of automated autobiographical reasoning scoring in Chinese adolescent narratives. Behavior Research Methods, 58(12), Article 334. https://doi.org/10.3758/s13428-026-03212-9

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03212-9

Keywords: large language models, autobiographical reasoning, narrative identity, automated coding, Chinese adolescents, AI validation, psychological assessment, prompting strategies, large, language, models, code

Researchers defined the outcome as a standard score of at least 53 on the 20-item Zung Self-Rating Depression Scale, or SDS. The predictors were sex, age, scores from the Self-Rating Anxiety Scale, Cognitive Fusion Questionnaire and Acceptance and Action Questionnaire-II, and eight non-depression SCL-90 symptom dimensions plus an additional-items score. These inputs comprised 126 non-SDS questionnaire items, so the method did not reduce questionnaire burden relative to directly administering the SDS.

The grade cohorts were assigned distinct analytical roles: 1,951 first-year students formed the Development cohort, 2,186 second-year students formed the Test cohort, and 2,258 third-year students formed the Holdout cohort. Development was divided into training and validation subsets for fitting the traditional models, selecting XGBoost calibration and choosing supplementary thresholds. Test labels were not used for fitting, calibration or threshold selection, while Holdout evaluated two Test-selected language-model conditions and both structured-data comparators. Because all cohorts came from one university and survey period, Holdout constituted within-institution grade-cohort validation rather than external or temporal validation.

The study compared logistic regression and isotonic-calibrated XGBoost with 12 conditions formed by combining GPT, Claude and Gemini models with four prompt families. These included zero-shot prompts, a rule-informed prompt and two example-augmented conditions containing two or four balanced demonstrations selected from Development-training records. The model calls were made through OpenRouter using fixed model identifiers, prompts and decoding settings, and outputs were required to contain a label, a positive probability and evidence. Hard-label performance was evaluated at 0.5 and at thresholds selected only in Development-validation, alongside discrimination, calibration, structured-output validity, cohort shift, construct overlap, subgroup performance and repeated-call stability.

At the fixed 0.5 threshold on Test, GPT E1, Gemini E2 and XGBoost produced macro-F1 values of 0.732, 0.730 and 0.674, respectively. The confidence intervals for pairwise macro-F1 differences among the three leading language-model conditions included zero, and their differences were only 0.001 to 0.003. Rankings changed when thresholds were selected in Development-validation: Gemini E2 ranked first in 99.9% of paired bootstrap replicates, while its Test macro-F1 remained 0.730 and GPT E1’s fell to 0.702. XGBoost’s Test macro-F1 was 0.660 at its selected threshold of 0.174, with sensitivity rising from 0.316 at 0.5 to 0.871 and specificity falling from 0.979 to 0.622.

Probability quality did not mirror the fixed-threshold ranking. XGBoost had the lowest Brier scores on Test and Holdout, 0.129 and 0.147, and the lowest expected calibration error across the tested binning specifications. Its AUROC was 0.846 on Test and 0.835 on Holdout, compared with 0.818 and 0.808 for GPT E1 and 0.834 and 0.820 for Gemini E2. Calibration intercepts indicated systematic miscalibration for logistic regression and both language-model finalists, so their reported probabilities were not consistently calibrated estimates of risk. In Holdout at 0.5, GPT E1 achieved sensitivity of 0.655, specificity of 0.821 and macro-F1 of 0.729; Gemini E2 achieved 0.723, 0.796 and 0.739, respectively.

Repeated calls exposed a substantial difference in output stability. Across the complete Holdout cohort and 10 calls at temperature 0.1, GPT E1 changed the thresholded prediction for 532 of 2,258 students, an inconsistency rate of 23.6% with a 95% confidence interval of 21.9% to 25.4%. Gemini E2 was inconsistent for 34 cases, or 1.5%, with a 95% confidence interval of 1.1% to 2.1%. GPT E1 instability was concentrated in cases with median probabilities close to 0.5: all 47 cases within 0.05 of the cutoff were inconsistent, compared with 51 of 1,497 cases at least 0.20 from it. In a representative 400-case subset, reducing the temperature to 0 did not clearly remove GPT E1 variability.

The questionnaire predictors also shared substantial measured content. In the combined sample, SDS and SAS scores had a Spearman correlation of 0.60 with a 95% confidence interval of 0.58 to 0.61. Removing SAS reduced the AUROC of logistic regression and XGBoost by approximately 0.025 to 0.029 across Test and Holdout, while removing the non-depression SCL-90 group reduced XGBoost AUROC by 0.019 on Test and 0.018 on Holdout. Models based only on sex and age had AUROCs of 0.531 and 0.538 for logistic regression and 0.498 and 0.521 for XGBoost on Test and Holdout, respectively, compared with 0.844 and 0.832 for full logistic-regression models. These associations describe predictive overlap and are not evidence of causal effects or an independent marker of depression.

Per-1,000 projections further illustrated the operating-point trade-offs. At 0.5 in Holdout, Gemini E2 produced 202 true positives, 147 false positives and 78 missed positives, while XGBoost produced 74 true positives, 17 false positives and 205 missed positives. Lowering XGBoost to its Development-derived threshold increased projected true positives to 245 and false positives to 303, while reducing missed positives to 34. Exploratory sex-stratified results varied by model and metric and did not constitute a comprehensive fairness assessment. Overall, the study establishes an auditable offline benchmark rather than clinical utility: it did not validate diagnoses, external transportability, fairness or reduced questionnaire burden, and it identified repeated-call stability, calibration and threshold choice as separate evaluation dimensions.

Subject of Research: Comparative evaluation of large language model prompting strategies for predicting SDS screen-positive status from structured questionnaire data in university students

Article Title: SDS screen-positive prediction from structured questionnaires in university students: a comparative evaluation of large language model prompting strategies

Article References: Yang, X., Cao, Y., Wang, H., Li, H., Jiang, X., Xu, H., Yao, T., Zhang, X., Chen, Y., Li, J., Zhou, B., Xu, L., Chen, Z., & Zhu, X. (2026). SDS screen-positive prediction from structured questionnaires in university students: a comparative evaluation of large language model prompting strategies. Frontiers in Psychiatry, 17, Article 1942164. https://doi.org/10.3389/fpsyt.2026.1942164

Image Credits: AI Generated

DOI: 10.3389/fpsyt.2026.1942164

Keywords: large language models, mental health screening, structured questionnaires, SDS threshold prediction, prompting strategies, calibration, repeated-output stability, XGBoost, university students, screen-positive, prediction, structured

The study, led by Bingjie Huang of Central China Normal University together with colleagues at Southeast University and Chengdu No. 7 High School, recruited 116 sophomore STEM students enrolled in a calculus-based introductory physics course and randomly assigned them to two groups. Ninety-five students completed all procedures: 50 in the experimental group and 45 in the control group. The experimental group received a structured training session built on Zimmerman’s self-regulated learning framework and cognitive load theory, while the control group received an equally long session on advanced problem-solving strategies with no AI guidance at all. Crucially, both groups then faced the same test: solving items from the Conceptual Survey of Electricity and Magnetism, a standardized 32-question assessment, and then revising their answers after conversing with the Chinese large language model Doubao.

The training itself was organized into three sequential phases. A ten-minute situational introduction contextualized generative AI, contrasting it with the homework-assisted apps that dominate Chinese students’ digital lives. A fifteen-minute strategy-teaching segment demonstrated how to prompt the AI in ways aligned with the three phases of self-regulated learning: forethought, performance, and self-reflection. It also introduced the model’s characteristic weaknesses, including reasoning errors, visual misinterpretation of graphs and figures, and outright hallucinations, along with practical remediation tactics. Finally, a twenty-minute hands-on session let students practice analyzing two mechanics problems with the AI and reflect on the obstacles they encountered. The design deliberately separated the training domain, mechanics, from the assessment domain, electromagnetism, so that any measured benefit would reflect transferable strategy rather than memorized solutions.

The quantitative results were striking. Students who received the AI training achieved significantly higher revision performance than the control group, with an effect size of 0.81, a large effect by conventional standards in educational research. The most dramatic behavioral difference appeared in the quality of revisions. Among trained students, 69.3 percent of revisions converted an incorrect answer into a correct one. In the control group, that figure was only 41.3 percent. Worse, 40 percent of the untrained students’ revisions actually changed correct answers into incorrect ones, a pattern the authors attribute to blind reliance: untrained students treated AI outputs as authoritative and capitulated reactively, even when the model’s reasoning was internally inconsistent.

Chat transcripts revealed exactly how the two groups diverged. The researchers coded six interaction strategies, ranging from the low-level image-only approach, in which students simply upload a photo of the problem, to higher-order behaviors such as asking for definitions, seeking insight into the structure of a problem, and generating new questions for self-assessment. Both groups leaned heavily on image-only prompting, a habit the authors link to the snap-and-solve culture fostered by apps like Zuoyebang and Xiaoyuan, which let students photograph homework and receive instant answers. But the trained group diversified markedly. The proportion of students using the obtaining-insight strategy more than doubled, from 20 percent to 42 percent, and 14 percent of trained students progressed to generating new questions, a behavior entirely absent among controls. Trained students also exchanged significantly more conversational turns with the AI, indicating deeper engagement rather than quick answer extraction.

The qualitative cases are perhaps the most vivid evidence of what the training accomplished. In one exchange, the AI misjudged an electromagnetic induction question, declaring that a coil in one figure formed an unclosed circuit. A trained student pushed back, noting that the coil was in fact closed, which prompted the model to recognize that a contracting closed coil changes its area and therefore its magnetic flux, leading it to the correct answer. Another student caught the AI misreading a graph in a mutual inductance problem and walked it to the right option by inspecting the answer patterns. By contrast, one control student, after the AI correctly graded an item, instructed it to simply show correct answers without analysis going forward, and then failed to catch a visual error the model made on the very next question.

Cognitive load measurements added a theoretically intriguing wrinkle. Overall cognitive load did not differ between the groups, but the trained students reported a marginally higher intrinsic load, the component tied to the inherent difficulty of the task itself. Rather than interpreting this as a cost, the authors propose that it may reflect productive struggle: the heightened working-memory demands that come from systematically decomposing problems and interrogating the AI strategically. They suggest refining cognitive load theory to distinguish task-imposed intrinsic load from productively engaged intrinsic load, arguing that when extra mental effort signals deep metacognitive processing rather than confusion, it may accompany, rather than undermine, better learning outcomes.

The study also probed how three learner characteristics, self-regulated learning skills, prior generative AI experience, and domain knowledge, shaped outcomes. Using robust linear models suited to the skewed distribution of pretest scores, the team found that self-regulated learning skills and prior knowledge both significantly predicted revision performance, while prior AI experience showed only a marginal effect. Notably, the training moderated the role of AI experience: in the control group, prior chatbot experience strongly predicted revision success, but in the trained group that relationship flattened almost to zero. In other words, structured instruction appeared to level the playing field, giving students without extensive AI backgrounds the strategic tools that experienced users had developed informally, if at all.

The authors are careful about the limits of their findings. The assessment was a one-shot revision task conducted on a single day, so the design cannot establish whether the gains reflect durable learning, conceptual change, or simply a temporary surge of attention. Because pretest and posttest used identical items, memory effects cannot be ruled out, and the observed advantage may partly stem from the control group’s lack of any AI guidance rather than from specific training components. The sample, drawn from a single top-tier Chinese university, also limits generalizability. The researchers call for longitudinal studies with delayed posttests, component-level comparisons to isolate the active ingredients of training, and richer measures such as interviews, eye tracking, or electroencephalography to capture students’ cognitive experiences more fully.

Even with those caveats, the implications are hard to ignore. As generative AI becomes embedded in education worldwide, the debate has often been framed as a binary between banning the tools and handing them out. This trial points toward a third path: teaching students explicitly how to collaborate with AI, including how to recognize its failure modes and correct them. The authors propose complementarity recognition, an awareness of the asymmetric capabilities of humans and machines and of when to trust each, as a construct worth measuring and cultivating. A 45-minute session cannot remake a student’s intellectual habits, but it demonstrably shifted behavior in the direction educators want: fewer blind capitulations to the machine, more probing questions, and a strikingly higher rate of turning wrong answers into right ones. The answer machine, it turns out, becomes a genuine partner only when someone teaches students how to work with it.

Subject of Research: The effect of structured generative AI training on students' AI-assisted scientific problem solving in STEM education

Article Title: From answer machine to collaborative partner: impact of structured generative AI training on scientific problem solving in STEM education

Article References: Huang, B., Xie, L., Liu, Y., Liu, X., Tang, H., Feng, X., Qiao, C., Guo, Q., Wu, C., & Bao, L. (2026). From answer machine to collaborative partner: impact of structured generative AI training on scientific problem solving in STEM education. International Journal of STEM Education, 13(1), Article 62. https://doi.org/10.1186/s40594-026-00652-9

Image Credits: AI Generated

DOI: 10.1186/s40594-026-00652-9

Keywords: generative AI, STEM education, self-regulated learning, cognitive load theory, human-AI collaboration, problem solving, metacognitive laziness, physics education, randomized controlled trial, AI literacy, prompting strategies, electromagnetism