New Tool Brings Hard Numbers to the Guesswork of Citation Chasing in Systematic Reviews

Every systematic review lives or dies by the completeness of its literature search, and one of the most trusted techniques for achieving that completeness is also one of the least scientific. Snowballing, the practice of chasing references backward through a paper’s bibliography and forward through the works that cite it, has long been guided by a reviewer’s intuition about when to stop. A new open-source software library called SnowBallSLR, described in the journal SoftwareX, aims to replace that intuition with explicit, reproducible rules, and in doing so exposes just how fragile the traditional approach has been.

The problem the tool addresses is structural. When a reviewer decides that a snowballing search has captured enough new material, that judgement cannot be written into a pre-registered protocol or repeated by another researcher. Live bibliographic APIs such as OpenAlex, Crossref and Semantic Scholar change daily, so the same starting set can yield different results on different days. For a search that forms part of an auditable evidence trail, the developer argues, this is a reproducibility failure rather than a mere caching inconvenience. Existing tools such as citationchaser and paperfetcher perform the chasing itself, but none combines automated iteration with a quantitative stopping criterion, a recall estimate and a replay guarantee.

SnowBallSLR, a Python library of roughly 6,600 lines of code released under the MIT license, organises a search as a resumable state machine. Each iteration expands the frontier of candidate records through references and citing works, normalises and deduplicates them, applies eligibility filters, ranks them for screening, exports the candidates, ingests the screening decisions and then evaluates every configured stopping rule. Screening deliberately happens outside the library, so the software composes with existing reference managers and screening platforms rather than replacing them. Records that cannot be resolved to a persistent identifier are retained and counted but never expanded, preventing an unverifiable title match from grafting an untrustworthy neighbourhood onto the citation graph.

Deduplication is deliberately old-fashioned. Records are keyed by DOI, then by source identifier, then by a signature over normalised title, year and first-author surname, in a fixed cascade of four rules with no machine learning and no learned thresholds. Every merge is explainable and reproducible, and provenance is unioned rather than replaced, so a record keeps every route by which it was found. Every raw API response is written to a content-addressed cache and hashed into a run manifest, which means a completed review can be replayed months later from its own cache without contacting any API, producing byte-identical outputs. A refresh mode refetches against the live databases and classifies what has changed.

The most ambitious feature is recall estimation, borrowed from capture-recapture methods developed in ecology to size animal populations that cannot be fully enumerated. The logic is elegant: if two partially overlapping samples of a population share only a few members, the unseen remainder must be large. In a snowballing run, each bibliographic source acts as a capture occasion, and the overlap between sources allows a statistical estimate of how many relevant studies the search missed. The library implements the Chapman-corrected Lincoln-Petersen estimator for two sources, a log-linear model selected by information criterion for three or more, and Chao’s distribution-free bound, and it refuses to estimate when the data cannot support it.

That refusal discipline proved essential. In simulations with known ground truth, an unguarded recall-based stopping rule fired in every one of 270 simulated reviews at a mean true recall of just 0.450, leaving an average of 74 relevant studies unfound. The failure mechanism is subtle: early in a run, only records within a few citation hops of the start set are reachable at all, so the capture sources agree almost perfectly on a small frame and the estimate collapses onto the observed count, reading near-perfect recall while most of the literature remains undiscovered. The remedy is a stability guard that withholds the stopping decision until the estimated population settles, and with that guard enabled the shortfall against the achievable ceiling fell to zero across all 270 reviews.

The simulations also delivered an uncomfortable finding about arm design. Capture-recapture requires independent capture occasions, but within a single snowballing run almost no natural pairing qualifies. Backward and forward chasing sample near-disjoint strata of a citation graph that is acyclic in time, producing an overlap of exactly zero in the demonstration review. Source-based arms are contaminated because OpenAlex ingests much of its metadata from Crossref, so Crossref captured no record OpenAlex had missed in two of three live corpora, and the library refused the resulting estimate rather than reporting a spurious recall of 1.0. Even splitting the start set at random and chasing each half separately failed, as both chases converge on the same connected component within three iterations.

Three completed reviews on live Scopus-defined corpora, spanning from an adverse case of 260 records with poor DOI coverage to one of more than 14,000 records, put the tool through its paces. The runs identified tens of thousands of candidate records and included 149, 337 and 686 studies respectively. Notably, no saturation-based stopping rule fired in any of them: two hit their screening budgets and one exhausted its screening resource, and the software reported exactly that rather than dressing the outcome up as saturation. A fourth run, revisiting the hardest corpus with a non-binding budget, did reach a saturation stop after ten iterations, and demonstrated frame expansion vividly: the estimated population grew from 6.0 to 643.3, meaning an unguarded rule would have stopped after six included studies with an estimated recall of 1.000 that was wrong by two orders of magnitude.

The live runs also probed the screening layer itself, which used an ensemble of large language models validated against blind positive controls. Term-frequency similarity to the seed set predicted inclusion decisions in every corpus, while citation count and shared venue did not, a pattern that would reverse if the screener were rewarding prestige rather than relevance. Records without abstracts scored significantly lower everywhere, skewing included sets toward well-indexed work, and rewording the scoring rubric shifted the included set by a fifth while leaving the score ordering nearly unchanged, suggesting the fragility lies in the cutoff rather than the screener.

The developers are careful about what the tool does and does not claim. It does not make citation searching find more studies; prior simulation work suggests it excels on precision rather than recall. Its contribution is making the stopping decision explicit, auditable and replayable, and flagging when the available evidence cannot support a completeness claim. Estimated recall, biased optimistically by a median of about 8 percent in coverage studies, is treated as an upper bound and the recall-based rule is advisory by default. For a field in which the TARCiS statement has named automation of citation searching a research priority and PRISMA 2020 reserves a dedicated column for records found this way, a tool that can state in a methods section exactly when and why a search stopped, and replay the entire run on demand, may change what reviewers, editors and readers expect from the humblest and most opaque step of evidence synthesis.

Subject of Research: Saturation-aware, reproducible automated citation searching for systematic literature reviews

Article Title: SnowBallSLR: saturation-aware and reproducible citation searching for systematic reviews

Article References: Matysik, S. (2026). SnowBallSLR: saturation-aware and reproducible citation searching for systematic reviews. SoftwareX, 36, Article 103116. https://doi.org/10.1016/j.softx.2026.103116

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103116

Keywords: systematic reviews, citation searching, snowballing, capture-recapture, recall estimation, reproducibility, OpenAlex, Crossref, stopping rules, PRISMA 2020, open-source software, evidence synthesis