AI Reviewers Match Human Experts in Landmark Scoping Review Test

Artificial intelligence has been creeping into the machinery of science for years, but a new benchmark from Johns Hopkins researchers suggests the technology may finally be ready to take on one of academia’s most tedious chores: the scoping review. In a study published in the Journal of Medical Systems, a team led by Frederick H. Kuo of Johns Hopkins All Children’s Hospital built a modular workflow in which large language models performed every labor-intensive stage of a real scoping review, then measured how closely the machines matched the human experts who had done the same work by hand. The results were strikingly close, and they hint at a future in which the bottleneck of evidence synthesis is no longer human patience but computational throughput.

Scoping reviews are the broadest and most unruly members of the evidence-synthesis family. Unlike systematic reviews, which ask a tightly focused clinical question and apply rigid inclusion criteria, scoping reviews map entire research landscapes, charting how concepts are used, where gaps lie, and how a field is evolving. That breadth makes them heterogeneous by design, and it makes them harder to automate. Large language models have already shown human-level performance in the screening phases of systematic reviews, but their behavior on the messier, more open-ended stages of scoping reviews has remained largely unexplored. The new study set out to close that gap with an unusually rigorous head-to-head comparison.

The researchers chose as their reference a genuine scoping review of human digital twins in healthcare, previously published in npj Digital Medicine by several members of the same group. A human digital twin is a computational replica of an individual patient that integrates clinical data to model health states and predict outcomes, a concept that has surged in popularity as medicine becomes increasingly data-driven. The reference review’s keyword search had returned 1,336 records, and its human reviewers had screened, categorized, and synthesized that corpus according to predefined research questions and taxonomies. Those human decisions became the gold standard against which the machines would be judged.

Two general-purpose models, both released by OpenAI in August 2025, were put through the workflow: GPT-5, the flagship commercial system, and gpt-oss-20b, a smaller open-weight model that can run on local hardware. The team supplied each model with prompt sets mirroring the stages of the review, from title and abstract screening through final inclusion decisions and on to data categorization and synthesis. Full texts of the articles were processed with a pipeline of open-source tools, including GROBID for parsing bibliographic structure and pdfplumber for text extraction, so that the models received clean, machine-readable input rather than raw document clutter. Every output was then benchmarked against the human decisions, with disagreements sent to an independent blinded adjudicator.

The statistical machinery of the evaluation was as careful as the workflow itself. Agreement between model outputs and human consensus was quantified with Gwet’s AC1, an interrater reliability coefficient that is less sensitive to prevalence effects than the classic kappa statistic, with values interpreted on the familiar Landis and Koch scale from slight to almost-perfect. For the synthesis stage, where the models had to assign articles to categories rather than make binary decisions, the team turned to total variation distance, a measure of how far two probability distributions diverge, with smaller values indicating closer distribution-level concordance. Disagreements were adjudicated and combined with unanimous decisions into a composite reference standard for the screening endpoints.

In title and abstract screening, both workflows performed at a level that would have been hard to predict even two years ago. GPT-5 achieved substantial agreement with human consensus, with an AC1 of 0.769, while gpt-oss-20b reached 0.722. Reference concordance, the proportion of decisions matching the composite human standard, came in at 90.9 percent for GPT-5 and 92.4 percent for gpt-oss-20b, compared with 89.0 percent for the human reviewers themselves. The machines were not merely keeping pace; on this endpoint, the smaller open-weight model actually edged out the human consensus it was being measured against, a result that reached statistical significance in its favor.

Final inclusion status, the decision that determines which articles actually enter the review, showed even stronger concordance. Interrater agreement was almost-perfect for GPT-5, with an AC1 of 0.948, and gpt-oss-20b reached 0.853. Reference concordance was 95.4 percent for GPT-5 and 92.4 percent for gpt-oss-20b, against 96.1 percent for human reviewers. The differences between the machines and the humans were small enough that they may matter less than the sheer consistency of the models, which never tire, never lose focus on the four-hundredth abstract, and apply the same criteria with identical precision to every record they encounter.

The synthesis stage, where the review’s central finding emerges, proved more demanding but ultimately vindicated the approach. Interrater agreement across data categorization and synthesis ranged from fair to almost-perfect, with AC1 values spanning 0.493 to 0.949 for GPT-5 and 0.342 to 0.929 for gpt-oss-20b, and total variation distances between 1.55 percent and 28.0 percent. Yet the crucial test was whether the machines would reach the same conclusion as the humans, and they did. All three sources, the human reviewers and both model workflows, found comparably low rates of definition-adherent use of the term human digital twin: 12.1 percent for humans, 13.0 percent for GPT-5, and 18.0 percent for gpt-oss-20b. In other words, most studies that invoke the fashionable label are not actually building true digital twins, a finding of real consequence for a field awash in buzzword inflation.

The implications extend well beyond one review. If models can screen thousands of abstracts with concordance matching or exceeding trained human reviewers, and if they can carry a review’s synthesis to the same conclusion, the labor economics of evidence synthesis change fundamentally. Reviews that currently take months of coordinated human effort could be compressed dramatically, allowing clinical guidelines to draw on evidence that is current rather than years out of date, a problem documented repeatedly in the methodological literature. The team suggests that total variation distance may prove useful as a tool for assessing distribution-level concordance in future evaluations, offering a way to quantify how closely a model’s categorical judgments mirror the collective judgment of human experts.

The authors are careful about the limits of what they have shown. This was a single-domain benchmark, drawing on one vendor’s models, one reference review, fixed prompts, and predefined taxonomies, so generalizability, reproducibility, and operational efficiency beyond this setting remain to be established. Broader evaluations across domains and vendors are warranted before anyone declares the human reviewer obsolete. But the direction of travel is unmistakable. In this benchmark, LLM-supported scoping-review workflows demonstrated feasibility for supporting and potentially streamlining selected review stages, with broadly comparable performance on the evaluated endpoints and conclusion-level concordance with the human original. The tedious, error-prone, and exhausting work of mapping the medical literature may soon be shared with tireless machine collaborators, freeing human experts to spend their judgment where it counts most: on the questions worth asking.

Subject of Research: Development and Exploratory Evaluation of a Large Language Model-Supported Scoping Review Workflow

Article Title: Development and Exploratory Evaluation of a Large Language Model-Supported Scoping Review Workflow

Article References: Development and Exploratory Evaluation of a Large Language Model-Supported Scoping Review Workflow. (n.d.). https://doi.org/10.1007/s10916-026-02477-z

Image Credits: AI Generated

DOI: 10.1007/s10916-026-02477-z

Keywords: large language models, scoping reviews, GPT-5, gpt-oss-20b, abstract screening, data extraction, human digital twins, research methodology, artificial intelligence, systematic reviews, interrater agreement, medical literature