Metacognitive Prompts Show No Detectable Boost for Statistical Problem-Solving in Small Pilot

A brief intervention designed to make university students plan, monitor, and evaluate their own thinking did not measurably improve their statistical problem-solving skills in a new study — but the researchers behind the work say the result says more about the limits of small classroom experiments than about the value of metacognitive prompting itself. The study, published in Frontiers in Psychology, tested whether self-regulated learning prompts embedded in statistics exercises helped students at Universitas Pamulang in Jakarta, and whether any benefit depended on whether students were driven by mastery goals, defined as developing their own competence, or performance goals, defined as demonstrating competence relative to others. Across every statistical test the authors ran, neither the prompts nor the goal orientations, nor any combination of the two, produced a statistically significant difference in learning gains.

The research team, A. Muhidin of Universitas Pamulang and Shinta Doriza of Universitas Negeri Jakarta, recruited 21 students enrolled in a statistics course and divided them by pre-existing class membership rather than individual random assignment. One intact class of ten students received the intervention; a second intact class of eleven served as the comparison group. Within each class, students were further classified as mastery-oriented or performance-oriented using a questionnaire adapted from Elliot and McGregor’s 2 × 2 achievement goal framework, yielding four groups of five to six students each. Because the classes were intact groups and assignment happened at the classroom level, the authors explicitly characterize the study as a small, single-site pilot comparison rather than a confirmatory experiment.

The intervention itself was deliberately simple. Over two learning sessions of roughly 120 minutes each, the experimental class worked through statistical problem-solving exercises containing brief written cues inserted before, during, and after each item — questions such as “What is this problem asking?”, “Which statistical concept applies here?”, and “Does this answer make sense given the data?” These cues were intended to trigger the three cyclical phases of self-regulated learning described in the educational psychology literature: forethought, performance, and self-reflection. The control class completed exactly the same exercises, taught by the same instructor with the same content and instructional time, but without the prompts. The prompts were the only feature that differed between the two conditions, and the researchers took procedural steps — different class times, instructions not to share materials, distinct worksheets — to reduce contamination between groups.

Learning was measured with a ten-item complex multiple-choice test covering central tendency, variability, z-scores, sampling, correlation, regression interpretation, hypothesis testing, and integrated statistical decision-making. Each item presented several statements of which more than one could be correct, and students earned a point only for selecting exactly the right combination — a format designed to assess higher-order, verification-dependent reasoning rather than single-fact recall. The same test served as pretest and posttest, and the researchers computed each student’s normalized gain, or N-gain, which expresses improvement as a fraction of the score that remained available after the pretest. N-gain values between 0.30 and 0.70 are conventionally categorized as moderate.

The headline statistical analysis, a two-way analysis of variance on N-gain scores with class and goal orientation as factors, found nothing significant. The main effect of the prompt intervention was essentially flat: F(1,17) = 0.02, p = 0.882, with a partial eta-squared of 0.001. The main effect of goal orientation was also nonsignificant, F(1,17) = 1.59, p = 0.224, partial eta-squared = 0.086, and the class-by-orientation interaction — the aptitude-treatment hypothesis at the heart of the study — was likewise null, F(1,17) = 0.08, p = 0.779. An analysis of covariance controlling for pretest scores produced the same nonsignificant pattern, although the authors caution that its key assumption of homogeneous regression slopes was itself violated in the data, so it cannot be treated as an independent confirmation.

The descriptive numbers tell a subtler story. All four class-by-orientation groups achieved moderate mean N-gains ranging from 0.40 to 0.57, meaning every group improved substantially relative to what they had left to learn. Notably, the numerically highest mean N-gain, 0.57, belonged to mastery-oriented students in the control class — students who received no metacognitive prompting at all — followed closely at 0.55 by mastery-oriented students in the prompted class. An exploratory, uncorrected t-test comparing mastery- and performance-oriented students within the control class yielded t(9) = 2.05, p = 0.071, a nominally nonsignificant result the authors present only as a hypothesis-generating observation for future research, not as evidence of a real orientation-related difference.

Why so little detectable effect? The authors point to three converging explanations. First, dosage: most published metacognitive-prompting successes, including a widely cited study showing that prompts improved students’ mental models in hypermedia learning, embedded prompts across multiple extended sessions in adaptive computer-based environments, whereas this intervention consisted of written cues over just two sessions. Second, ceiling compression: several students posted high pretest scores and near-ceiling posttest scores, shrinking the range of possible gain and potentially masking a real treatment effect. Third, and most fundamentally, statistical power. A sensitivity analysis showed that with 21 participants, the design had 80% power to detect only very large effects — partial eta-squared of at least 0.296, equivalent to a two-group Cohen’s d of roughly 1.30. Typical educational interventions produce far smaller effects, which means a genuine but modest benefit of prompting could easily have gone undetected.

The study is equally careful about what its null results do not establish. Equivalence testing against a smallest-effect-size-of-interest of Cohen’s d = 0.50 failed for both the class and orientation comparisons, meaning the data can neither confirm nor rule out effects of that magnitude. Compounding the uncertainty, the two intact classes were not equivalent at baseline: the experimental class started with a significantly higher mean pretest score (5.70 versus 3.91, F(1,17) = 5.40, p = 0.033), a threat to internal validity common in nonequivalent-groups quasi-experiments. Levene’s test also flagged heterogeneous variances across cells, driven by unusually wide variability in the experimental-performance group, though Welch’s ANOVA and a 10,000-permutation randomization test reproduced the same nonsignificant pattern, suggesting the null result is not an artifact of unequal variances.

There is also a structural limitation the authors highlight as the central reason for their cautious framing. Because each condition contained only a single classroom, class is completely confounded with treatment, and students within a class share an instructor, pacing, and environment in a way that can make student-level significance tests appear more precise than the design supports. With only two classrooms, a multilevel or cluster-robust analysis cannot be fitted, since a random class effect is not identifiable from one classroom per condition. The reliability of the adapted goal-orientation questionnaire and the problem-solving test was also not assessed in this sample, leaving measurement error as an unquantified contributor to the null results. The authors additionally note that collapsing the four-category goal framework into a mastery-performance dichotomy may have obscured differences specific to the approach-avoidance dimension.

The authors’ conclusion is deliberately restrained: the hypothesized aptitude-treatment interaction was neither confirmed nor ruled out, and the moderate gains observed in all groups cannot be attributed specifically to the prompt material, since the design lacked a no-instruction comparison group. They call for replication with multiple classrooms randomly assigned to each condition, retention of the full four-category achievement goal classification, sample-specific reliability reporting, longer and more intensive prompting interventions, and process measures such as think-aloud protocols to capture self-regulation more directly than pretest-posttest scores allow. Until such studies arrive, the practical takeaway for statistics instructors is one of honest uncertainty: a single brief set of metacognitive prompts, tested in a single underpowered classroom comparison, has not yet demonstrated — or disproven — a benefit for any group of learners.

Subject of Research: A pilot study testing whether metacognitive prompts and achievement goal orientation affect university students' statistical problem-solving gains.

Article Title: Effects of self-regulated learning prompts on statistical problem-solving by goal orientation

Article References: Muhidin, A., & Doriza, S. (2026). Effects of self-regulated learning prompts on statistical problem-solving by goal orientation. Frontiers in Psychology, 17, Article 1967032. https://doi.org/10.3389/fpsyg.2026.1967032

Image Credits: AI Generated

DOI: 10.3389/fpsyg.2026.1967032

Keywords: metacognitive prompts, self-regulated learning, statistics education, achievement goal orientation, aptitude-treatment interaction, quasi-experimental design, normalized gain, statistical power, undergraduate students, pilot study, problem-solving, Effects