Rigor Replaces RAMsing in STEM Education Meta-Analysis Methods

Meta-analyses of STEM education interventions have been quietly inflating their most celebrated findings, according to a new study that names and quantifies a pervasive reporting practice the authors call RAMSing—reporting after moderators are significant. The research, published in Educational Psychology Review, analyzed nearly 1,800 moderator effect sizes drawn from 89 meta-analyses and found that statistically non-significant moderator results may be only a fraction as likely to appear in the published record as their significant counterparts, distorting the evidence that educators, policymakers, and researchers rely on when deciding what works, for whom, and under what conditions.

Moderator analyses are the workhorses of research synthesis in education. When a meta-analysis pools dozens or hundreds of studies testing an intervention—say, a new instructional model or an educational technology tool—the individual study results rarely agree perfectly. Moderators, tested through subgroup analyses or meta-regressions, are meant to explain that between-study heterogeneity by asking whether effects vary with study characteristics such as instructional approach, sample size, student age, disciplinary focus, or assessment format. In theory, they transform a single average effect into a nuanced map of boundary conditions. In practice, the new study argues, they are vulnerable to a form of selective reporting that operates inside the meta-analysis itself.

The team, led by Mehmet Bicakci of Friedrich Alexander University Erlangen-Nürnberg together with Fabian Heller, Heidrun Stoeger of the University of Regensburg, and Albert Ziegler, defines RAMSing as the practice of running multiple moderator tests but publishing only those that reach conventional significance thresholds such as p < .05, or that exceed familiar effect-size benchmarks like Cohen’s d of 0.20 or 0.50. The practice is distinct from p-hacking and HARKing because it occurs after the analysis, at the reporting stage, and leaves little trace of the broader analytic search that produced the surviving results. During full-text screening of more than 100 STEM education meta-analyses, the researchers repeatedly encountered warning signs: moderators announced in introductions but missing from results sections, inconsistencies in summary tables, and vague statements such as “no other moderators were significant” unaccompanied by any statistics.

To measure the scope of the problem, the team conducted an umbrella review, systematically searching ERIC, PsycINFO, Web of Science, and PSYNDEX for the period 2000 through 2023. After screening 5,103 unique records with inter-rater reliabilities between 0.79 and 0.95, they assembled a final dataset of 1,786 moderator effect sizes from 89 meta-analyses of STEM education interventions. The corpus was dominated by mathematics research, which accounted for nearly half of the corpus, followed by multidisciplinary STEM interventions and science-specific studies. Academic outcomes appeared in over 88 percent of cases, and instructional models plus educational technology were the most common intervention categories.

The centerpiece of the analysis was a pair of bias-robust Bayesian frameworks known as RoBMA-PSMA and RoBMA-Regression. Rather than assuming the published literature is complete, these methods model publication selection explicitly, averaging across dozens of candidate models that weight findings according to their p-values and adjust for small-study effects. The results were stark. The models estimated that moderator effect sizes with p-values between 0.05 and 0.50 were only 43 percent as likely to appear in the published record as highly significant results—a relative visibility deficit of roughly 57 percent. For p-values between 0.50 and 1.00, the estimated selection weight fell to 0.11, meaning clearly non-significant moderator findings were about 89 percent less likely to be visible. The Bayes factor for publication-bias-type selection reached an overwhelming 4.72 × 10³⁶, and the overall visibility gap across non-significant results was summarized at approximately ω ≈ 0.40.

Despite the bias, the moderator effects themselves were not an illusion. The bias-adjusted mean moderator effect was g = 0.27, with a 95 percent credible interval of 0.20 to 0.34, supported by a Bayes factor of nearly three million. But the effects were strikingly heterogeneous, with a between-effect standard deviation of τ = 0.48. When the researchers classified moderators into five broad themes—intervention specifications, sample and context variables, disciplinary focus, study design and quality, and publication metadata—the classification substantially improved model fit with a Bayes factor of 350 and modestly reduced residual heterogeneity to τ = 0.44. Disciplinary focus produced the largest expected effects at g = 0.438, followed by sample and context variables at g = 0.364, while design and quality indicators yielded the smallest at g = 0.197. The finding suggests that collapsing science, technology, engineering, and mathematics into a single undifferentiated category risks obscuring meaningful differences across fields.

The bias also contaminated the yardsticks by which effect sizes are judged. The raw published distribution of moderator effects, right-skewed and leptokurtic with a mean of 0.59, yielded empirical quartiles of 0.29, 0.53, and 0.77. But when the team generated 10,000 posterior-predictive draws from the publication-bias-adjusted model, the quartiles dropped to 0.18, 0.38, and 0.64. Notably, visible discontinuities in the raw histogram clustered around Cohen’s conventional thresholds of 0.20, 0.50, and 0.80, and these jumps disappeared entirely after bias adjustment—raising the possibility that the thresholds function not only as interpretive guides but as implicit reporting incentives. The authors suggest that some moderator findings may be selectively reported not just for statistical significance but for appearing large enough to satisfy magnitude conventions, a secondary form of RAMSing that miscalibrates what a field considers “small,” “typical,” or “large.”

Power analysis added a further sobering dimension. Under a planning scenario using the bias-adjusted median effect of g = 0.38, heterogeneity of τ² = 0.24, and a two-sided test at α = 0.05, detecting a typical moderator effect with 80 percent power requires approximately 14 independent effect sizes per moderator level. Yet about 60 percent of the moderator units in the published corpus fell below that threshold, indicating that underpowered moderator tests are the norm rather than the exception. Low power produces unstable estimates that are especially vulnerable to selective reporting, because apparently interesting findings persist in the literature while null results drift toward the file drawer.

The study’s remedy is a four-component “anti-RAMSing toolkit,” openly available on the Open Science Framework as fully annotated R scripts with a worked example dataset. The workflow begins with structural mapping—a qualitative cataloguing of every intervention-moderator-outcome combination in a literature, visualized as an alluvial diagram—followed by robust Bayesian bias adjustment, bias-aware benchmarking, and prospective power calibration. The authors also propose a 13-item checklist covering moderator justification, preregistration, adequate effect sizes per level, complete reporting of all tested moderators including null results, and sensitivity analyses. Among their practical recommendations: journals could require structured tables listing all prespecified moderators, and reporting guidelines such as PRISMA could add a RAMSing checkpoint demanding disclosure of every moderator test with its accompanying statistics.

The implications reach well beyond STEM education. The authors argue that the vulnerabilities they document—extensive moderator probing, uneven reporting standards, and incentive structures that reward novelty over completeness—are likely wherever meta-analysts probe many study characteristics, including psychology, management research, and the health sciences. Indeed, prior meta-research suggests only about half of all moderator analyses ever reach publication. The deeper problem, the study contends, is the absence of a defensible framework for selecting, testing, and reporting moderators, which leaves the practice partly arbitrary and invites selective disclosure.

The researchers caution that their bias adjustments are model-based corrections rather than direct observations of the complete evidential record, and that they depend on the reporting quality of the published studies themselves—many moderator effects had to be excluded because variances, standard errors, or confidence intervals were missing. Had precision information been consistently reported, the dataset would have been roughly 25 percent larger. Still, sensitivity analyses across narrower and wider prior settings left the substantive conclusions essentially unchanged.

The study’s central message is that the credibility of moderator evidence depends not only on statistical technique but on what the literature chooses to reveal. Selective visibility, the authors conclude, can inflate apparent moderator effects, encourage overconfident claims about what works for whom, and bias the effect-size estimates used for power planning and research investment decisions. A more credible future for meta-analysis, they argue, requires not more moderator tests, but better ones—preregistered where possible, grounded in theory, powered adequately, and reported completely regardless of outcome.