New Standardised Tool Aims to Fix Risk-of-Bias Assessment in Environmental Evidence

Environmental evidence synthesis has long lacked a standardised way to judge the reliability of the studies it relies on. A new paper published in the journal Environmental Evidence introduces the Collaboration for Environmental Evidence Critical Appraisal Tool, described by its developers as the first standardised risk-of-bias assessment tool for the environmental sector. The tool is designed for use in quantitative evidence syntheses that address causal questions across any environmental topic, and its formal release follows years of development, testing and revision.

Risk-of-bias assessment, also known as critical appraisal of internal validity, is a required component of systematic reviews. It evaluates how susceptible individual primary studies are to bias at the level of each effect estimate. The results of these assessments feed into data synthesis, where subgroup analyses are conducted to draw overall conclusions about an effect.

According to the authors, recent research has highlighted inadequate practices in risk-of-bias assessment across environmental evidence syntheses. The vast majority of these syntheses either omitted the critical appraisal step entirely or conducted it inappropriately. Compounding the problem, no standardised risk-of-bias assessment tool existed for the environmental sector. Although standardised tools are available in medicine, the authors note that employing these tools in environmental evidence syntheses is not always appropriate, leaving synthesists without a fit-for-purpose framework.

The new tool, developed by Ko Konno, James Gibbons, Ruth Lewis and Andrew S. Pullin, addresses that gap. Its prototype version was released in 2020, followed by real-world testing by environmental evidence synthesists. A survey gathered feedback from users of the prototype, and action items for revision were identified from that process. All of the action items were addressed to produce the formal version that the paper now introduces.

Structurally, the tool provides a standardised framework in which a study’s findings are first assessed within individual risk-of-bias criteria by answering a series of checklist questions. The results of these within-criteria assessments then inform an overall judgement about the risk of bias for the study’s findings.

The scope of the tool is deliberately broad.

The authors frame the release as a step toward higher standards rather than a final solution. They state the hope that the tool’s availability will lead to improved practice in risk-of-bias assessment across environmental evidence syntheses. They also flag future work in two directions: further evaluation of the tool itself, and the creation of an external validity section to help evidence synthesists conduct critical appraisal of external validity objectively, comprehensively and rigorously. The current version focuses on internal validity, leaving external validity as a recognised area for later development.

The paper, published as an open access methodology article on 11 October 2026, arrives as a shared version made available early to provide faster access to peer-reviewed, accepted research. For a field in which most syntheses have historically skipped or mishandled critical appraisal, the introduction of a sector-specific, tested and revised standardised tool marks a concrete attempt to make the assessment of bias in environmental causal studies consistent, documented and comparable across reviews.

Subject of Research: Development and release of the Collaboration for Environmental Evidence Critical Appraisal Tool for assessing risk of bias in environmental studies estimating causal effects

Article Title: Collaboration for Environmental Evidence Critical Appraisal Tool: A tool to assess risk of bias in environmental studies estimating causal effects

Article References: Konno, K., Gibbons, J., Lewis, R., & Pullin, A. S. (2026). Collaboration for Environmental Evidence Critical Appraisal Tool: A tool to assess risk of bias in environmental studies estimating causal effects. Environmental Evidence. https://doi.org/10.1186/s13750-026-00394-9

Image Credits: AI Generated

DOI: 10.1186/s13750-026-00394-9

Keywords: risk of bias, critical appraisal, environmental evidence, systematic review, evidence synthesis, causal effects, internal validity, Collaboration for Environmental Evidence, methodology tool, external validity, open access, Collaboration