Teacher-governed LLM scaffolding associated with stronger formative support, not significant post-test gain, in quasi-experimental study of 357 online programming learners

A quasi-experimental field evaluation of 357 online introductory programming learners found that a teacher-governed adaptive support system was associated with better formative practice and perceived support, but not with a statistically significant adjusted difference in the primary post-test outcome. The study examined 180 learners in the adaptive condition and 177 in a comparison condition across four course cohorts. Both groups received the same objectives, core materials, assessments, course duration and teacher access, while the adaptive group also received dynamic recommendations, an explainable dashboard and large language model-supported tutoring.

The intervention combined a teacher-authored knowledge-component map, learning-management-system and assessment records, an event-history student model, adaptive recommendation rules, a constrained LLM tutor and explainable analytics. The course map covered four components: variables and data types, conditional statements, loops and basic functions. Each was linked to prerequisites, learning objectives, materials and formative and test items. An independent panel of three informatics-education experts reviewed the map through a Delphi process, after which it was examined with pilot data from 45 learners and revised before the field evaluation.

Rule-based recommendations translated event evidence into four action families: prerequisite review, staged hints, additional practice and reflection or progression. The LLM generated bounded explanations or hints only after support was invoked by the system or learner. It did not assign grades, certify mastery or independently determine progression. The operational channel used gpt-4-1106-preview through the OpenAI API, with a semantic policy requiring diagnostic questioning when needed, progressively explicit hints, references to course resources, appropriate acknowledgement of uncertainty, refusal of complete solutions and escalation to a teacher when the model was uncertain or repeatedly asked for a full solution.

All 1,168 assistant sessions were logged with identifiers, timestamps, query types, duration, hint and escalation flags, and learner usefulness ratings. Mean learner-level usefulness was 4.03 out of 5, and 5.6% of sessions were escalated. However, the anonymised archive did not retain the exact token-level prompt or the executable recommendation thresholds, and the researchers did not independently rate the factual accuracy of every response. These omissions limit exact replay and direct evaluation of the tutor’s accuracy.

The learning management system used stratified block allocation within four cohorts, but the archive lacked the random seed, block size, generated allocation list and allocation-concealment record. The investigators therefore analysed the study quasi-experimentally, and residual selection bias or imbalance cannot be excluded. Baseline comparisons of age, prior programming experience and pre-test score were not statistically significant, although adaptive learners began an average of 3.32 percentage points lower. Course completion was 75.6% in the adaptive group and 76.3% in the comparison group. No AI use was logged in the comparison condition, but unrecorded use of external AI tools could not be ruled out.

After adjustment for pre-test performance, the primary post-test difference between conditions was 3.07 percentage points, with a 95% confidence interval of −0.87 to 7.02, p = .127 and partial eta-squared of 0.007. The study therefore did not establish a statistically significant adjusted difference in the primary academic outcome. A raw-gain comparison showed a small 4.26-point difference favoring the adaptive condition, with a 95% confidence interval of 0.05 to 8.47 and Hedges g = 0.21. Normalised gain and course completion did not differ between groups. The administrative final score was excluded as a separate outcome because it was calculated from the same 12-item post-test total.

Process indicators showed a clearer association with the adaptive condition. After Holm correction within each outcome family, adaptive learners had higher formative correctness and more formative attempts, while total formative time did not differ. Changes in self-efficacy, usability, recommendation clarity and trust also remained significant after within-family correction, with recommendation clarity showing the largest standardised difference. These observations indicate stronger reported support and formative practice in the adaptive condition, but they do not by themselves establish higher final achievement.

For next-response prediction, a regularised logistic event-history model achieved an area under the receiver operating characteristic curve of 0.683, with a 95% cluster-bootstrap confidence interval of 0.667 to 0.698. The logged operational knowledge-tracing probability achieved an AUC of 0.660, with a 95% confidence interval of 0.644 to 0.675. These values indicated moderate discrimination rather than strong separation between correct and incorrect responses. The authors considered them potentially suitable for prioritising low-stakes hints or teacher attention, but not for grading, certification, progression decisions or individual mastery claims.

Qualitative analysis of 42 voluntary follow-up responses included 28 adaptive students, eight comparison students and six teachers, with Cohen’s kappa of 0.82 between two coders. Adaptive students commonly described immediate diagnostic or conceptual support, while trust remained dependent on verification and context retention. Comparison students emphasised waiting and a need for prompt examples or error feedback. Teachers reported fewer routine questions and better allocation of attention, but also described setup costs and concern about reduced visibility into learner difficulties. The voluntary, brief and uneven sample was used to explain quantitative patterns rather than estimate prevalence. Other limitations included moderate test reliability, brief self-reported surveys, possible external AI contamination among comparison learners, one introductory course with four knowledge components, and no long-term or independent programming-performance outcome.

Subject of Research: Evaluation of teacher-governed LLM-supported adaptive scaffolding in an online introductory programming course

Article Title: Adaptive large language model scaffolding supports online programming education through learning analytics and knowledge tracing

Article References: Bartoš, P. (2026). Adaptive large language model scaffolding supports online programming education through learning analytics and knowledge tracing. Discover Education, 5(1), Article 943. https://doi.org/10.1007/s44217-026-02172-8

Image Credits: AI Generated

DOI: 10.1007/s44217-026-02172-8

Keywords: online programming education, large language models, adaptive learning, knowledge tracing, learning analytics, formative feedback, quasi-experimental study, self-regulated learning, teacher governance, explainable AI, Adaptive, large