Generative AI Boosts Programming Learning, But Big Open Questions Remain

Generative artificial intelligence has swept into computer science classrooms faster than almost any educational technology in memory, and teachers, students, and researchers have been arguing ever since about whether tools like ChatGPT and GitHub Copilot genuinely help people learn to code or simply help them produce code. A new meta-analysis published in Educational Psychology Review offers the most statistically rigorous answer to date. Drawing on 35 empirical studies published between 2022 and 2025, containing 131 separate effect sizes, researchers Mian Wu and Fan Ouyang of Zhejiang University applied a three-level Bayesian meta-analysis to the growing but messy literature. Their central finding is strikingly clear: generative AI produces credible small-to-medium positive effects across every major category of learning outcome measured in programming education, from hands-on coding performance to motivation and emotional well-being.

The technical sophistication of the analysis matters as much as its conclusions. Traditional meta-analyses often struggle with the fact that a single study can report multiple related effect sizes drawn from the same participants, violating the statistical assumption of independence. A three-level model, following the framework popularized by Van den Noortgate and colleagues, explicitly separates variance into three layers: sampling variance within each effect size, between-outcome variance within each study, and between-study variance. This hierarchical structure prevents studies with many measurements from dominating the pooled estimate. The Bayesian approach adds a further layer of rigor. Rather than relying solely on point estimates and p-values, Wu and Ouyang estimated full posterior probability distributions for every effect, using weakly informative priors in the brms package built on Stan, and evaluated models with leave-one-out cross-validation. In a field where the evidence base is young and uneven, Bayesian credible intervals offer a more honest picture of what the data can and cannot support.

To bring order to a heterogeneous literature, the researchers classified learning outcomes into four conceptually distinct categories. AI-assisted programming outcomes, abbreviated AIPO, capture performance when learners work with an AI tool at their side, such as code quality while using Copilot or problem-solving scores with a chatbot available. Independent programming outcomes, or IPO, measure what learners can do on their own once the scaffold is removed, a distinction that has become central to debates about whether AI assistance translates into durable skill. Higher-order skills, HOS, encompass computational thinking, critical thinking, and problem decomposition, the cognitive abilities educators most want programming courses to cultivate. Finally, motivational-emotional outcomes, MEO, include self-efficacy, anxiety, interest, and engagement, which decades of research show are powerful predictors of persistence in computing.

Across all four categories, the pooled posterior estimates landed in the small-to-medium range, and critically, the analysis detected no credible differences among the categories themselves. In other words, the average benefit of generative AI did not statistically favor assisted performance over independent skill, cognitive gains over emotional ones, or any other pairing. That uniformity is itself informative. It suggests that the technology is not merely a crutch that inflates assisted scores while leaving independent ability untouched, at least not on average across the studies conducted so far. Learners using AI tools also reported modestly higher self-efficacy and lower anxiety, outcomes that matter enormously in a discipline notorious for weeding out novices in their first semester.

Perhaps the most consequential, and most sobering, finding concerns the moderator analyses. The researchers tested whether educational context, instructional design, or the technical design of the AI system moderated the effects. Did effects differ between K-12 and higher education? Between flipped classrooms and lectures? Between chatbots and code-completion assistants? Between environments with guardrails and those without? On the evidence available, none of these moderations reached credibility in any outcome category. At first glance this might suggest that generative AI works about equally well everywhere, a convenient conclusion for institutions drafting policy. But the authors are careful, and correctly so, to resist that interpretation.

The problem is statistical power and balance. The corpus of 35 studies is small, and the studies distribute unevenly across moderator levels, with some cells of the design containing very few effect sizes. In Bayesian terms, when the data carry little information about a difference, the posterior remains wide and centered near zero, which the analysis records as an absence of credible moderation. The authors explicitly warn that the null moderation results may reflect limited statistical information rather than genuine equivalence across conditions. A few exploratory pairwise contrasts did emerge as credible for motivational-emotional outcomes under specific educational contexts and strategy-training conditions, hinting that context does matter in ways the field has not yet measured systematically.

This caution echoes a growing body of primary research that complicates the optimistic average. A widely discussed field experiment published in PNAS in 2025 found that high school students given unrestricted access to GPT-4 during math practice performed worse on subsequent exams than students who never used it, a classic case of performance gains masquerading as learning. Related work on metacognitive laziness shows that learners with AI support sometimes engage in shallower self-regulation, offloading the very cognitive work that produces durable knowledge. Cognitive science has long recognized this tension under the banner of cognitive offloading: external aids can free mental resources or can hollow out the skills they were meant to support, depending on how they are deployed. The assistance dilemma, articulated by Koedinger and Aleven in the context of cognitive tutors, is precisely what AI developers and instructors now face in sharper form: when to help, how much, and when to withhold.

What the meta-analysis establishes, then, is a credible average, not a prescription. The pooled effects say that, across the studies conducted between 2022 and 2025, generative AI interventions in programming education did more good than harm on the outcomes measured. They do not say that any deployment will work, that unstructured access to a chatbot during a final project is beneficial, or that particular pedagogical designs outperform others. Those condition-specific questions require larger, better-balanced studies with far more complete reporting of implementation details, such as how the AI was prompted, scaffolded, restricted, or integrated into assessment. The authors call explicitly for this next generation of research, and the field’s rapid growth suggests it will not wait long.

For educators and institutions making decisions now, the practical reading is measured optimism. The evidence supports using generative AI as a positive complement to programming instruction, particularly given its consistent effects on motivation and self-efficacy, outcomes that predict who stays in computing. But the absence of credible moderation findings should be read as an open question, not a blank check. Until studies with adequate statistical power identify which contexts, instructional strategies, and system designs strengthen or weaken learning, the wisest course is deliberate integration, with attention to whether students are genuinely internalizing skills or merely borrowing the machine’s. Wu and Ouyang have given the field its clearest baseline yet, and a well-marked map of what remains unknown. The analysis code and datasets are publicly available through the Open Science Framework, inviting the community to interrogate and extend the evidence as the literature matures.

The study also carries a methodological message for educational research at large. As AI interventions multiply across subjects, the same three-level Bayesian machinery used here can distinguish credible effects from noise in small, rapidly evolving literatures, and can do so transparently, with priors, model comparisons, and posterior distributions open to scrutiny. In a domain where hype and fear both run hot, that kind of careful, quantified uncertainty may be the most valuable outcome of all.

Subject of Research: Effects of generative AI on learning outcomes in programming education, synthesized through a three-level Bayesian meta-analysis

Article Title: How Generative AI Influences Learning Outcomes in Programming Education: A Three-level Bayesian Meta-analysis

Article References: Wu, M., & Ouyang, F. (2026). How Generative AI Influences Learning Outcomes in Programming Education: A Three-level Bayesian Meta-analysis. Educational Psychology Review, 38(1), Article 117. https://doi.org/10.1007/s10648-026-10211-x

Image Credits: AI Generated

DOI: 10.1007/s10648-026-10211-x

Keywords: generative AI, programming education, learning outcomes, meta-analysis, Bayesian statistics, ChatGPT, GitHub Copilot, computational thinking, self-efficacy, educational technology, higher-order skills, motivation