Popular Gene Pathway Scoring Tools Can Silently Erase Disease Signals, Benchmark Warns

Single-cell RNA sequencing has transformed how researchers hunt for the molecular fingerprints of disease, but a new benchmark study suggests that one of the field’s most trusted analytical steps may be quietly discarding exactly the signals scientists are searching for. In a paper published in BMC Bioinformatics, a research team led by Fadhl M. Alakwaa of the University of Michigan introduces PathwayBench, a systematic comparison of five widely used pathway activity scoring methods across eight case-control datasets spanning five human tissues and 682 donors. The study’s central finding is deceptively simple: methods that rely on a fixed top-rank window of genes can go blind or even reverse the direction of a biological effect, a failure mode the authors call rank-window truncation.

Pathway activity scoring is the step in which analysts take a long list of individual genes and compress it into a single number that represents how active a biological pathway, such as immune signaling or tissue scarring, appears to be in a given sample. In the pseudobulk regime that now dominates applied single-cell disease studies, cells from each donor are aggregated into a single profile, and these donor-level profiles are compared between disease and control groups. This approach has become the workhorse of studies of conditions like chronic kidney disease, idiopathic pulmonary fibrosis, and systemic lupus erythematosus, where the key question is how disease alters pathway behavior across groups of patients rather than within individual cells.

Despite how common the technique is, the authors note that method choice has rarely been guided by systematic, multi-criterion evidence in this pseudobulk setting. Previous benchmarks of pathway scoring in single-cell data focused on perturbation ground truth at cell-level resolution, leaving the donor-level case-control regime largely unexplored. That gap matters because the statistical properties of pseudobulk profiles differ sharply from those of individual cells. A pseudobulk profile built from thousands of cells is dense, with many genes expressed at measurable levels, and that density turns out to be precisely the condition under which certain scoring methods begin to fail.

The team evaluated five of the most widely used methods: ssGSEA, GSVA, z-score, AUCell, and UCell. Rather than treating these as a single family, the benchmark reveals that they fall into three statistically distinct groups. The z-score method measures cross-sample standardized magnitude, asking how strongly pathway genes deviate from their typical expression levels across donors. ssGSEA and GSVA are rank-enrichment hybrids that consider the full distribution of gene ranks. AUCell and UCell are top-window truncated rank methods, which look only at the highest-ranked genes in a profile and ask whether pathway genes are concentrated within that privileged window. The authors argue that this three-family taxonomy is more accurate than the conventional two-way split between magnitude-based and rank-based methods, since ssGSEA is itself a rank-based enrichment statistic.

The mechanistic heart of the paper is the identification of rank-window truncation as a two-sided mechanism of signal loss. On one side, if the genes belonging to a pathway fall outside the fixed top-rank window, the method simply cannot see them and returns a null score, no matter how strongly the pathway is actually dysregulated. On the other side, if non-pathway competitor genes invade the window, they crowd out the true signal, attenuating it or, in severe cases, inverting its sign so that an activated pathway appears suppressed and vice versa. Both failure modes are invisible to the analyst, who sees only a plausible-looking number at the end of the pipeline.

To test how broadly this problem extends, the researchers ran a simulation sweep across 49 conditions varying the parameters that govern pathway size, signal strength, and competitor burden. The results were striking. AUCell returned a null score in 33 of the 49 conditions when pathway genes fell below its default window, and both AUCell and UCell inverted the sign of their scores under heavy competitor burden. The effect was not a corner case confined to extreme parameter choices but a broad feature of the parameter grid, suggesting that any pseudobulk analysis using default settings is at substantial risk.

The simulation findings were confirmed on real data through a window-parameter experiment. When the researchers widened the rank window beyond its single-cell default, most of the lost biological signal was recovered at an intermediate setting. This confirms, the authors argue, that the default window is mis-set for dense pseudobulk profiles. The defaults were calibrated for the sparse expression profiles of individual cells, where only a fraction of genes are detectable, and they translate poorly to aggregated donor-level data in which far more genes carry meaningful expression values.

The real-world consequences of these failures are illustrated through extracellular matrix remodeling in chronic kidney disease, a fibrotic process central to disease progression and a pathway of intense therapeutic interest. Using data from the Kidney Precision Medicine Project, the study found that top-window rank methods produced near-zero or wrong-direction effect sizes across three different normalization schemes. In other words, a researcher relying on AUCell or UCell with default settings could conclude that ECM remodeling shows no difference between diseased and healthy kidneys, or even that it runs in the opposite direction from the truth, while whole-distribution methods correctly detect the expected signal. For fibrotic, inflammatory, or otherwise broadly remodeled transcriptomes, where many genes shift together, the authors recommend particular caution in method selection.

PathwayBench evaluated the five methods against five criteria: three covering biological relevance, namely direction accuracy, area under the receiver operating characteristic curve, and effect size, and four robustness axes covering aggregation, outlier, normalization, and sample-size stability. When per-criterion performance was discretized as good, intermediate, or poor, no single method satisfied all five criteria under the primary thresholds. Importantly, the authors acknowledge that this kind of discrete scoreboard is sensitive to the thresholds and weightings chosen, so they report the underlying continuous scores as the primary evidence and present the scoreboard only as a summary aid. This methodological transparency is itself a contribution, resisting the temptation to crown a single winner from inherently multi-dimensional evidence.

The benchmark is delivered as a versioned, extensible resource with per-dataset scores, an interactive advisor application to help analysts choose methods appropriate to their data, and complete reproducibility infrastructure to support community extension. The authors also describe an unusually rigorous verification process: all eight figures were independently regenerated by a separate implementation from the verified data file under strict isolation, with every numerical value checked to agree across both implementations to within display precision. For a field in which pipeline choices can determine whether a disease mechanism is found or missed, PathwayBench offers both a warning and a practical path forward: understand which statistical family your scoring method belongs to, and match it to the density and remodeling breadth of the transcriptome you are studying.

Subject of Research: Benchmarking of pseudobulk pathway activity scoring methods in single-cell RNA-seq and the mechanism of rank-window truncation

Article Title: PathwayBench: a multi-criterion benchmark of pseudobulk pathway activity scoring methods reveals rank-window truncation as a two-sided mechanism of biological signal loss in single-cell RNA-seq

Article References: Alakwaa, F. M., Elbaz, A., Azzam, S. A., & Alshebani, T. (2026). PathwayBench: a multi-criterion benchmark of pseudobulk pathway activity scoring methods reveals rank-window truncation as a two-sided mechanism of biological signal loss in single-cell RNA-seq. BMC Bioinformatics. https://doi.org/10.1186/s12859-026-06632-8

Image Credits: AI Generated

DOI: 10.1186/s12859-026-06632-8

Keywords: single-cell RNA-seq, pathway activity scoring, pseudobulk, benchmarking, rank-window truncation, AUCell, UCell, ssGSEA, GSVA, chronic kidney disease, bioinformatics, gene expression