Simulation exposes failure risks in publication-bias tests

A large Monte Carlo simulation found that commonly used tests for publication bias can falsely report selective publication when study characteristics influence both estimated effects and study precision. In the simulated meta-analyses, omitting confounding covariates inflated the Type I error rates of Egger-type regression tests, sometimes reaching 50% to 100% under strong confounding, whereas a likelihood-ratio test based on the three-parameter selection model generally remained near the nominal 5% error level. The results point to complementary strengths: direct covariate-adjusted regression offered strong control of false positives, while the selection model was usually more powerful but required additional care in some low-heterogeneity settings.

The study addressed a recurring problem in meta-analysis: a statistical pattern associated with small studies does not necessarily mean that unfavorable results were withheld. Meta-analyses combine effect estimates from multiple studies, and tests for funnel-plot asymmetry commonly examine whether effect magnitude is associated with precision. Study characteristics can also be associated with both the magnitude of an effect and its standard error, however. Examples described in the research include methodological quality and intervention intensity. If these relationships are omitted, a regression test for small-study effects can mistake an effect-precision association produced by heterogeneity for evidence of selective publication.

To investigate the problem, the researcher generated independent meta-analytic datasets in which each study contributed one standardized mean difference. The data-generating model included a binary moderator, a continuous moderator, residual between-study heterogeneity, and selection linked to statistical significance. Primary-study sample sizes were drawn from an empirical distribution used in earlier psychology research. Both moderators were linked to true study effects and correlated with sample size, allowing their relationships with observed effect estimates and precision to vary across simulation conditions. The generated data used a one-step selection mechanism: favorable results significant at a one-sided p value below .025 were always selected, while other results were retained with probabilities ranging from 1 under no selection to 0.05 under extreme suppression.

The full factorial design comprised 17,496 combinations of meta-analysis size, residual heterogeneity, moderator effects, correlations between moderators and sample size, and the degree of censoring. Meta-analyses contained 20, 50, or 80 studies. Residual heterogeneity took values of 0, 0.2, or 0.5, while the binary and continuous moderators each had slopes of 0.1, 0.2, or 0.5 in the primary simulations. Each condition was examined through 5,000 replications, and the Monte Carlo standard errors of the resulting rejection rates were below 0.008. Performance was evaluated using one-sided Type I error rates under no selection and statistical power when selection was present, with a nominal significance level of 5%.

Three approach families were compared. Two Egger-type regressions related effect estimates to either modified standard error, called Egger-SE, or modified sampling variance, called Egger-V, while allowing for relevant moderators and residual heterogeneity. Residual-based analyses formed a second family by regressing effect-size residuals on measures of precision. The third family was a one-sided likelihood-ratio test from the three-parameter selection model, or 3PSM. Unlike the regression diagnostics, the selection model represented the reporting process directly by comparing a model permitting selective publication with one assuming that all studies were observed. The modified standard errors and sampling variances were used to remove an artificial correlation between an effect estimate and its sampling variance.

When both confounders were omitted, Egger-SE and Egger-V frequently produced far more false positives than the nominal 5% rate. Inflation was most severe when the covariates were negatively correlated with study sample size, which made them positively associated with standard errors as well as true effects. Under strong confounding, false-positive rates for the regression tests ranged from 0.50 to 1.00 when residual heterogeneity was small or absent. The 3PSM test was substantially more robust, generally remaining near the nominal level, although mild inflation appeared when residual heterogeneity, the number of studies, and moderator slopes were all small. When covariates were positively correlated with sample size and therefore negatively associated with standard errors, omission made the one-sided Egger tests conservative rather than liberal.

Partial adjustment did not eliminate every source of distortion. When the binary covariate was included but the continuous covariate was omitted, the regression tests were corrected for the modeled source of confounding but remained vulnerable to the unmodeled one. Their rejection rates rose above nominal levels when the continuous covariate was negatively correlated with sample size and fell below nominal levels under the opposite relationship. The 3PSM test was less sensitive to the missing moderator, with error rates close to or mildly above the nominal level. This robustness was consistent with its direct modeling of the selection process, although the use of the same one-step mechanism in the data generation and the tested model may also have favored it. The study therefore tested a specific scenario rather than the full range of selective-reporting processes.

When both covariates were included, Egger-SE and Egger-V were generally well calibrated across the simulated conditions. The 3PSM test showed some inflation when residual heterogeneity was zero, particularly with few studies and small moderator slopes, and slight inflation could also occur with small but nonzero residual heterogeneity, including τ = 0.2. Its error rates were generally near or below nominal when larger residual heterogeneity was present. Residual analyses that removed only the effect of the covariates on estimated effects tended to be conservative, because residualizing effects alone can underestimate their relationship with precision. A more elaborate residual analysis that also residualized the standard error performed similarly to Egger-SE, although it showed slightly more inflation because of differences in the estimated residual heterogeneity. Overall, the residual approach offered no clear practical advantage over directly including covariates in meta-regression.

Statistical power remained limited for most methods, although 3PSM consistently had the highest power in the fully adjusted analyses. With 50 studies and a continuous-moderator slope of 0.1, the selection-model test exceeded the conventional 80% power benchmark only for strong or extreme selection, corresponding to reporting probabilities of 0.2 or 0.05. Egger-SE, Egger-V, and the residualized-error analysis reached adequate power mainly under extreme censoring with zero residual heterogeneity and a large binary-moderator effect. More studies generally increased power, while stronger confounding, greater residual heterogeneity, and weaker selection reduced it. Comparisons involving conditions with inflated 3PSM error rates require caution because higher rejection rates can reflect false positives as well as genuine detection.

The findings support treating Egger-type tests and 3PSM as complementary analyses rather than declaring either universally superior. Meta-analysts using regression-based diagnostics should include plausible study characteristics as simultaneous predictors, especially when those characteristics may affect both effect magnitude and precision. Direct covariate adjustment provided the strongest control of false-positive rates once the relevant covariates were modeled, while 3PSM usually detected the simulated selection process more efficiently. The conclusions remain bounded by the design: every study contributed one independent effect, selection followed a single one-step rule, covariates were observed without missingness, and the study evaluated detection rather than correction of biased effect estimates. The researcher identified other selection processes, dependent effect sizes, missing moderators, broader questionable research practices, and tests of bias-correction methods as subjects requiring further evaluation.

Subject of Research: Simulation study of publication-bias tests in meta-analyses with confounding covariates

Article Title: Evaluating methods for testing publication bias in meta-analyses with confounding covariates: A simulation study

Article References: Chen, M. (2026). Evaluating methods for testing publication bias in meta-analyses with confounding covariates: A simulation study. Behavior Research Methods, 58(12), Article 330. https://doi.org/10.3758/s13428-026-03206-7

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03206-7

Keywords: publication bias, meta-analysis, Monte Carlo simulation, Egger regression, selection model, covariate confounding, small-study effects, Type I error, statistical power, research methods, Evaluating, methods