Ten simple rules for building a tool to appraise research reporting

Every scientist has met the frustration: a paper that promises a breakthrough but buries its methods in a single vague sentence, omits its exclusion criteria, or never states how many samples were dropped along the way. In an ideal research world, authors would report exactly what they conducted, clearly and transparently. In reality, reporting is often incomplete, and key elements of research conduct are frequently missing from manuscripts. Readers cannot assess what they cannot see, and they cannot see what authors do not report. Poor reporting does not merely annoy reviewers; it can conceal important flaws and biases, leaving editors, reviewers, and readers unable to judge a study’s reliability and robustness. A new article published in PLOS Computational Biology by Kyle Morrison and Malgorzata Lagisz of the Collaboration for Open Science and Synthesis in Ecology and Evolution at the University of Alberta argues that the field’s most trusted fix for this problem—reporting guidelines—is being asked to do a job it was never built for, and that science urgently needs a new class of instruments: reporting appraisal tools.

Reporting guidelines such as PRISMA for systematic reviews and CONSORT for randomised controlled trials are among the most cited publications of all time and are endorsed by many highly reputable journals. They provide practical sets of items that should be reported in a manuscript describing a given type of research, and they have been enormously valuable for authors writing their manuscripts. But Morrison and Lagisz contend that they should not be used to judge the completeness of reporting in manuscripts written by others, as commonly happens during peer review and editorial assessment. The reason is a mismatch of needs. Authors require a comprehensive checklist to guide writing—a detailed list covering all reporting items. Appraisers, by contrast, need a tool that is representative, consistent, and easy to use: concise enough to apply repeatedly, in a format that allows outcomes to be compared across studies. A 30-page checklist designed to help an author draft a paper is simply not operationalised for a reviewer who must decide, quickly and repeatably, whether the minimum essential elements of a study were described.

To close this gap, the authors propose the development of reporting appraisal tools—operationalised tools designed specifically to assess whether a paper has reported the minimum essential elements of a study. Such tools can indicate which predefined key elements of a study were described in a manuscript, at least at a minimum level, and which were not, revealing important gaps in reporting. Despite growing demand for instruments to appraise reporting completeness and research transparency, specifically designed tools remain rare. MATES, a reporting appraisal tool for meta-analyses developed by the same team, is one of the few examples. The new article distils the team’s experience building MATES, together with a methodological analysis of PRISMA extension development processes, into ten simple rules intended to guide anyone who wants to build such a tool from scratch.

The first rule is conceptual and arguably the most important: reporting completeness is distinct from methodological quality. A fundamental first step when creating any tool is to define exactly what you aim to assess. Appraisal tools designed to assess reporting completeness can be used by editors, reviewers, and readers to evaluate whether a manuscript sufficiently describes what was done, without judging the study’s design. This sets them apart from critical appraisal instruments such as AMSTAR2 or ROBIS, which assess methodological quality or risk of bias. A study can be reported perfectly and still be methodologically weak; conversely, an elegant design can be hidden behind incomplete reporting. Conflating the two, the authors warn, undermines the purpose of either kind of tool.

The second rule calls for assembling a steering group to guide development. This core team should include individuals with the methodological expertise, topic knowledge, and practical insight needed to lead the process, and it should reflect a range of perspectives relevant to the tool’s intended use. The steering group defines the scope of the appraisal tool, designs the development process, oversees how feedback is sought and incorporated, organises and synthesises input, documents key decisions, and makes subsequent revisions. Once the tool is finalised, the group also leads preparation of the manuscript and associated materials, ensuring the development process itself is reported transparently enough to support trust, uptake, and future updating.

Rule three addresses efficiency, and it is where the authors see the greatest savings. Historically, developers of reporting guidelines have had to generate reporting items de novo—a process that is often complex, resource-intensive, and time-consuming. Reporting appraisal tools can be developed far more efficiently by drawing on existing reporting guidelines, which already provide curated and conceptually refined sets of reporting items. Developers can specify the field, study type, and study design for which they want to appraise reporting completeness, then search for a closely aligned guideline. A useful starting point is the database collated by the EQUATOR network, which classifies almost 700 existing guidelines based on their scope and methodology. Banks of evaluation items can also be searched. If no guideline closely matches, an existing one may be adapted or extended with the specific items required. Adapting these items into appraisal questions reduces duplication of effort and conserves resources while building on work that has already undergone substantial development.

Rules four and five govern whose voices shape the tool and how those voices are heard. Developers should incorporate diverse expert and community opinion, engaging a combination of methodological experts, relevant stakeholders, and members of the broader research community. Looking beyond disciplinary expertise alone—considering relevant PROGRESS-Plus factors that may shape participants’ perspectives—is essential. Including participants from low- and middle-income countries can surface critical perspectives on the feasibility of mandating resource-intensive reporting practices, which matters because it is counterproductive to appraise the reporting of elements that are rarely conducted in practice. For gathering feedback, the authors recommend a structured, transparent, Delphi-like procedure: iterative rounds of feedback from the research community and methodological experts, combining complementary methods such as consensus meetings and surveys to collect both qualitative and quantitative input. Because broader stakeholder involvement can inflate the number of candidate items as participants advocate for domain-specific additions, the steering group should pre-specify item-retention thresholds before the process begins—for example, retaining only items reaching a defined consensus threshold such as 70 percent agreement—and consider a dedicated prioritisation round, such as forced ranking or a pre-agreed cap on total item count, to filter the candidate pool.

Rule six is the technical heart of the enterprise: designing the structure of the tool itself. Because reporting appraisal tools serve a distinct purpose from reporting guidelines, they require different design considerations. The first key distinction is that items should be presented as direct, answerable questions rather than guideline-style statements instructing authors on what to report. The second is that each question should be accompanied by explicit operational criteria specifying what constitutes a fulfilled item. The authors illustrate with an example drawn from PRISMA: the guideline item

Subject of Research: Ten simple rules for developing a reporting appraisal tool

Article Title: Ten simple rules for developing a reporting appraisal tool

Article References: Morrison, K., & Lagisz, M. (2026). Ten simple rules for developing a reporting appraisal tool. PLOS Computational Biology, 22(10), e1014841. https://doi.org/10.1371/journal.pcbi.1014841

Image Credits: AI Generated

DOI: 10.1371/journal.pcbi.1014841

Keywords: simple, rules, developing, reporting, appraisal, tool, scientific research