Generative AI Boosts Student Learning, But Meta-Analysis Warns Effects Vary Widely

A large new meta-analysis reports that generative artificial intelligence tools such as ChatGPT produce a substantial average improvement in students’ cognitive, learning, and linguistic outcomes, but the same analysis cautions that the headline number may be inflated by small-study effects and that results in any single new setting could plausibly be null or even negative. The study, published in Frontiers in Education, pooled 68 effect sizes drawn from 58 independent primary studies covering a total of roughly 6,941 participants, using a three-level random-effects model with cluster-robust variance estimation to handle the fact that many studies contributed more than one result.

The pooled effect came out at Hedges’ g = 0.864, with a 95% confidence interval of 0.720 to 1.009, a magnitude the authors describe as large under common educational benchmarks. Heterogeneity, however, was very high: the I² statistic was approximately 88%, meaning most of the variation across results reflects real differences between studies rather than sampling noise. More striking still is the 95% prediction interval, which stretched from −0.213 to 1.942. Unlike the confidence interval, which describes the precision of the average, the prediction interval estimates where the true effect of a brand-new, comparable study would be expected to fall. Because that range crosses zero, the data are entirely consistent with a future implementation of generative AI in education showing no benefit at all.

The analysis searched Web of Science and Scopus in May 2026, yielding 3,254 records that were deduplicated and screened down through successive filters—relevance, experimental or quasi-experimental design, English language, peer-reviewed status, full-text availability, and sufficient statistical detail—until 108 primary studies remained. Outcomes were then restricted to three domains: cognitive measures such as memory, attention, reasoning, cognitive load, and problem solving; learning measures such as academic achievement, retention, transfer, and comprehension; and linguistic measures such as writing quality, vocabulary, grammar, pronunciation, speaking, and reading. Studies reporting only affective, motivational, or creativity outcomes were excluded, leaving the final dataset of 68 effect sizes from 58 studies published between 2023 and 2026.

The authors tested seven candidate moderators: outcome domain, educational level, type of generative AI tool, experimental design, cultural orientation, course or subject domain, and comparison-condition type. Only two reached conventional statistical significance before correction for multiple testing. Outcome domain showed the strongest pattern (Qb = 9.45, p = .009): linguistic outcomes produced the largest pooled effect at g = 1.102 across 23 effect sizes, general learning measures yielded g = 0.861 across 29, and cognitive outcomes the smallest at g = 0.535 across 16. Educational level also varied at the unadjusted threshold (Qb = 6.50, p = .039), with primary-school samples showing g = 1.311, higher education g = 0.855, and secondary school g = 0.612. Yet after applying a Holm correction across the seven moderator tests, neither result retained statistical significance, and the primary-school estimate rests on only seven effect sizes.

The most eye-catching raw contrast involved cultural orientation, coded at the country level using the horizontal–vertical individualism–collectivism framework. Samples from vertical-collectivist contexts—such as China, Taiwan, Turkey, and Saudi Arabia—showed a pooled effect of g = 0.926 across 61 effect sizes, compared with g = 0.454 in horizontal-individualist contexts such as Germany, the United States, and Norway, based on just six effect sizes. Because such a two-group comparison can be confounded by other study characteristics that cluster by region, the authors fitted multivariable adjustment models. The cultural contrast remained significant when modeled alongside subject domain and tool type (p = .041) and when a precision term was added to guard against small-study bias (p = .032), but lost significance in a third model constrained by only about 3.5 residual degrees of freedom. A fully joint model could not be fitted because the horizontal-individualist studies contained no primary-school or STEM samples at all.

None of the remaining moderators proved reliable. Tool type (ChatGPT-based versus other or multi-model platforms), experimental design, course domain, and comparison-condition type all failed to reach significance before or after adjustment. There was, however, a suggestive descriptive pattern for comparison conditions: the advantage of generative AI was larger against passive, business-as-usual controls (g = 1.014) than against active alternatives such as human feedback or matched non-AI pedagogies (g = 0.765), hinting that part of the reported benefit may reflect comparison with unenhanced instruction rather than a genuine edge over well-designed teaching.

The publication-bias diagnostics pointed in two directions, and the authors report both rather than forcing agreement. Trim-and-fill analysis identified no missing studies and left the pooled estimate unchanged at g = 0.864. Egger’s regression test, however, detected significant funnel-plot asymmetry (p = .002), and PET-PEESE extrapolation suggested that once small-study effects are corrected for, the true effect could be considerably smaller—somewhere in the region of g = 0.26 to 0.45 depending on the estimator. A three-parameter selection model produced an adjusted effect of g = 0.361 with a confidence interval crossing zero. Because these adjustment methods are known to disagree when genuine heterogeneity, rather than publication bias alone, shapes the funnel—which is almost certainly the case here given the near-88% I²—the authors explicitly decline to treat the corrected figures as replacements for the pooled estimate, but they conclude that g = 0.864 should not be read as a bias-free quantity.

Several robustness checks did support the overall conclusion. Restricting the analysis to the 57 between-groups effect sizes left the estimate essentially unchanged at g = 0.882. Sensitivity analyses varying the assumed pre–post correlation in single-group designs from 0.50 to 0.90 left the pooled estimate invariant. A leave-one-study-out analysis, refitting the model 58 times, produced estimates in a narrow band from 0.834 to 0.888, with no single study capable of overturning the direction or approximate magnitude of the effect. Risk-of-bias category, assessed for the included studies, did not significantly moderate the results.

The authors also flag important limitations. The between-study variance component reached a boundary estimate of zero, meaning most heterogeneity resides across individual effect sizes rather than within studies. Several primary studies contributed multiple comparisons from shared participant pools, introducing some double-counting in the aggregated sample total, and the dataset mixes single-group pre–post designs with between-groups comparisons. The geographic concentration of studies in East Asia and other vertical-collectivist settings, the complete absence of horizontal-individualist samples at the primary-school level or in STEM subjects, the restriction to English-language journal articles in two databases, and the focus on only three outcome domains all constrain generalizability. The cultural findings, though robust to several confound adjustments, were not adjusted for risk-of-bias scores and are described as exploratory.

The practical takeaway is deliberately nuanced. The evidence supports a positive average effect of generative AI on learning-related outcomes, with the strongest and most consistent gains appearing in language-education contexts, but the wide prediction interval and the divergence between observed and bias-adjusted estimates mean the point estimate should be generalized cautiously. The authors recommend treating the cross-cultural pattern as a hypothesis for targeted future research rather than a basis for region-specific policy, and they call for primary studies that combine underrepresented settings—horizontal-individualist, primary-school, and STEM populations—alongside pre-registered publication-bias protocols and fuller reporting of inter-outcome correlations in future syntheses.

Subject of Research: A meta-analysis of the effects of generative artificial intelligence on students' cognitive, learning, and linguistic outcomes

Article Title: A meta-analysis of the effects of generative artificial intelligence on learning, cognitive, and linguistic outcomes

Article References: Khodjaev, B. K., Masalimova, A. R., Chistyakov, A. A., & Ay, Y. (2026). A meta-analysis of the effects of generative artificial intelligence on learning, cognitive, and linguistic outcomes. Frontiers in Education, 11, Article 1971476. https://doi.org/10.3389/feduc.2026.1971476

Image Credits: AI Generated

DOI: 10.3389/feduc.2026.1971476

Keywords: generative AI, meta-analysis, education, ChatGPT, learning outcomes, cognitive outcomes, linguistic outcomes, publication bias, heterogeneity, cultural orientation, Hedges g, Frontiers in Education