Perturbational similarity maps are among the most ambitious artifacts in modern biology. The idea is elegantly simple: knock out or repress thousands of genes one at a time, record the molecular or cellular consequences of each perturbation, and then compare those consequence profiles to one another. Genes whose disruptions produce similar effects are inferred to act in the same pathways or protein complexes, turning a mountain of raw screening data into a functional network. If such maps were transferable, a perturbation atlas painstakingly assembled in one system, for example the mouse brain, could be used to interpret screens performed in human cell lines, dramatically multiplying the value of expensive experiments and accelerating the translation of animal findings into human biology.

Testing that assumption required a careful matching infrastructure. Wen began by restricting the comparison to genes with strict one-to-one orthology between mouse and human, a conservative choice that retained 71 shared perturbation targets for comparison with the K562 Perturb-seq dataset, 90 for the RPE1 Perturb-seq dataset, 695 for HeLa morphology profiles from the PERISCOPE resource, and between 273 and 311 for nine L1000 cell-line matrices. Even before any statistics were computed, the attrition was striking: the vast majority of perturbations profiled in any single system simply have no directly comparable counterpart in the others, a practical constraint that any cross-system transfer strategy must confront.

The core of the analysis rested on three complementary endpoints, each designed to probe a different notion of transferability. The first was global pair-rank concordance: if two perturbed genes are ranked as similar neighbors within the mouse brain map, do they receive similar ranks in the human datasets? This was quantified with Spearman rank correlation across all pairs of shared targets. The second was source-defined top-k neighbor preservation, a stricter test asking whether a perturbation’s ten nearest neighbors in the source map remain among its top neighbors in the target dataset. The third was within-validation retrieval against CORUM, a curated database of protein complexes, which checks whether external perturbation matrices show internal structure consistent with known molecular assemblies.

The headline numbers were sobering. Under the primary zero-anchored root-mean-square representation of the response features, the Spearman correlation between mouse and human pair ranks was 0.026 in K562 and 0.035 in RPE1, effectively indistinguishable from zero. The two HeLa morphology conditions yielded correlations of 0.0022 and 0.0055, and across the nine L1000 cell lines the median was 0.0055. In other words, knowing which perturbations resemble each other in the mouse whole-brain atlas provided essentially no information about which perturbations resemble each other in any of these human systems. The neighbor-preservation test fared no better: across all thirteen tested contexts, not a single source-defined top-10 preservation result survived statistical correction for multiple comparisons, with the smallest adjusted q-value reaching only 0.240.

One of the study’s most technically important findings concerns how sensitive these conclusions are to the mathematical representation of the data. Perturbation profiles are high-dimensional vectors, and the choice of how to normalize and scale them can reshape the geometry of the entire map. When Wen switched from the primary zero-anchored RMS representation to across-target standardization in the RPE1 dataset, the correlation rose to 0.102, with a 95 percent confidence interval of 0.027 to 0.201, a weak but statistically detectable signal. Omitting the feature RMS scaling step instead produced a descriptive correlation of negative 0.034. The lesson is double-edged: representation choices can manufacture or destroy apparent transferability, which means that any published claim of cross-system concordance must report and justify its preprocessing pipeline in detail.

The study also probed the reliability of the maps themselves, asking how much of the failure to transfer could be blamed on noise within each dataset. Across fifty random cell-level splits of the K562 and RPE1 Perturb-seq data, the median global correlation between independently derived maps was 0.600 and 0.710 respectively, with top-10 neighbor overlap of roughly 0.50. Across fifty mouse-level splits of the K562- and RPE1-matched source profiles, the median global correlations were 0.731 and 0.703. These figures indicate that the individual maps are internally reproducible to a moderate degree, yet they also reveal substantial split-to-split variability. Crucially, Wen notes that a true cross-species noise ceiling, the maximum correlation one could expect even for perfectly transferable biology given measurement noise, remains unestimated, so the near-zero observed correlations cannot yet be cleanly separated from an unfavorable signal-to-noise regime.

The within-validation CORUM analysis added an important nuance. Several of the external human matrices did show evidence of protein-complex structure when analyzed on their own terms, meaning that perturbations targeting members of the same complex tended to cluster together within those datasets. But Wen is careful to point out that this result does not test preservation of the source map’s neighbor structure; a dataset can be internally biologically coherent while remaining globally discordant with the mouse brain atlas. This distinction matters for the field because within-dataset validation is often presented as evidence of quality, when the more demanding question, whether functional structure survives the journey between systems, is a separate and harder test.

The implications reach well beyond cell culture. In vivo Perturb-seq has recently been used to map transcriptional networks in the developing cortex, to dissect astrocyte function, and to link psychiatric disease risk genes to specific molecular mechanisms, and such whole-brain atlases are increasingly proposed as reference resources for interpreting human genetic data. The new results do not invalidate that enterprise, but they do impose a discipline: conclusions drawn from a perturbation map in one species or modality should not be assumed to hold in another without direct empirical verification, and the choice of feature representation must be treated as a first-class scientific decision rather than a technical afterthought.

Wen is equally explicit about the limits of the negative findings. The results are conditional on the specific representations and feature selections tested, they concern non-neural human systems that differ profoundly from brain tissue, and they do not establish that transferability is absent, particularly for neural systems where matched perturbation data do not yet exist at comparable scale. The analysis code, fixed random seeds, processed inputs, and reviewer-requested analyses have been deposited in a versioned Zenodo archive, and the underlying public datasets from the whole-brain atlas, the Replogle Perturb-seq screens, PERISCOPE, and the Connectivity Map remain freely available. That transparency should make it straightforward for other groups to extend the audit, perhaps to neural organoids, assembloid screens, or emerging spatial Perturb-seq platforms, as those resources mature. For now, the study stands as a rigorous caution: in perturbational genomics, the map is not yet the territory, and every boundary between species, cell systems, and assay modalities must be tested rather than assumed.

Subject of Research: Cross-species and cross-platform transferability of perturbational similarity maps in functional genomics

Article Title: Testing the Transferability of Perturbational Similarity Maps Across Species, Cell Systems, and Assay Modalities

Article References: Wen, J. (2026). Testing the Transferability of Perturbational Similarity Maps Across Species, Cell Systems, and Assay Modalities. Neuroinformatics, 24(4), Article 66. https://doi.org/10.1007/s12021-026-09821-1

Image Credits: AI Generated

DOI: 10.1007/s12021-026-09821-1

Keywords: Perturb-seq, Neuroinformatics, perturbation atlas, cross-species comparison, L1000, Connectivity Map, protein complexes, CORUM, reproducibility, representation transfer, CRISPR screens, whole-brain mapping

The researchers’ central claim is that the metrics most commonly used to score perturbation prediction models — mean squared error (MSE), mean absolute error (MAE), and the control-referenced Pearson correlation, Pearson(Δ_ctrl) — are frequently miscalibrated. A calibrated metric should reward any predictor that captures genuine perturbation-specific signal and penalize predictors that do not. To test whether a metric meets that standard, the team borrowed a concept from experimental biology: controls. The mean baseline serves as an intuitive negative control, since it contains no perturbation-specific information. But benchmarks had lacked a positive control — a predictor that, by construction, contains real signal — leaving an ambiguity whenever models scored poorly. Low scores could mean the model failed, or simply that the metric was too insensitive to notice success.

The team’s earlier work had proposed a technical-duplicate baseline as a positive control: split the cells from each perturbation into two halves, use one half to predict the other, and see how well that works. Because both halves share the same perturbation-specific signal, the duplicate should outperform the mean baseline if a metric is working properly. On the Norman19 dataset, with roughly 102 differentially expressed genes (DEGs) per perturbation, it did. But on the Replogle22 K562 genome-wide Perturb-seq dataset, where perturbations are far weaker — only about 3.25 DEGs per perturbation on average — the technical duplicate actually underperformed the mean baseline on MSE in roughly 95 percent of individual perturbations. The culprit, the researchers found, is an artifact they call signal dilution: when perturbations touch only a handful of genes out of thousands, error metrics are dominated by the vast transcriptomic background, and a better estimate of unperturbed expression — which the mean baseline provides thanks to greater statistical power — looks more accurate than a prediction carrying real but sparse biological signal.

To build a positive control that works even for weak perturbations, the team introduced the interpolated duplicate. This baseline blends the technical duplicate and the mean baseline gene by gene, using an interpolation weight α derived from the adjusted P value of a differential expression analysis. Genes showing strong statistical evidence of being perturbed are weighted toward the technical duplicate; genes that appear unaffected are weighted toward the mean baseline. The result is a predictor that consistently outperforms the mean baseline across datasets, providing the reliable positive control the field had been missing. With negative and positive controls in hand, the researchers could finally ask the question that had gone unanswered: which benchmarking metrics can actually tell the two apart?

Their answer takes the form of a new meta-metric called the dynamic range fraction, or DRF. For any candidate metric, DRF measures how much of the theoretical gap between a perfect prediction and the negative control is actually covered by the empirical gap between the positive control and the negative control. If the positive control comfortably beats the negative control, the metric is well calibrated and DRF approaches one. If DRF hovers near zero, the metric is essentially blind to perturbation-specific signal, no matter how good the model is. Applied per perturbation, DRF turns metric evaluation from a matter of taste into a measurable quantity — and the results were striking.

Across 14 datasets and 18 metrics spanning direct reconstruction error, reference-based delta metrics, and retrieval-based metrics, the workhorse metrics of the field fared poorly. MSE and Pearson(Δ_ctrl) showed low DRF values in most perturbations of the Replogle22 K562 dataset, meaning they could barely distinguish a signal-bearing predictor from an uninformative one. The evidence was not merely statistical. When the team examined SP2, a transcription factor whose inhibition in Replogle22 K562 yields a DRF for MSE near zero, gene set enrichment analysis of the differential expression ranks still recovered SP2’s own binding targets among the top ENCODE/ChEA gene sets — clear biological signal that MSE simply could not see. In contrast, weighted and rank-based metrics, including weighted MSE (WMSE), weighted R²Δ, and the normalized inverse rank (NIR), consistently showed higher calibration. These metrics share a design principle: they upweight the small fraction of genes that actually respond to a perturbation, rather than letting the silent transcriptome drown the signal out.

With well-calibrated metrics in place, the team re-benchmarked nine models on the task of predicting responses to perturbations held out during training: scGPT, GEARS, PRESAGE, scLambda, CellFlow, and four foundation-model embedding probes built on Geneformer, ESM2, scGPT, and GenePT representations. The reversal was dramatic. Under poorly calibrated metrics such as MSE and Pearson(Δ_ctrl), earlier models like scGPT and GEARS showed no advantage over baselines, consistent with prior benchmarking reports. Under well-calibrated metrics, however, even these earlier models mostly outperformed the uninformative baselines — their perturbation-specific signal had been present all along, merely obscured by the choice of ruler. Newer architectures went further: PRESAGE, an attention-based model that encodes biological prior knowledge, and scLambda, a variational autoencoder with language-model-derived embeddings, often beat the baselines even under poorly calibrated metrics. On every metric tested, at least one deep learning model outperformed all baselines, and the findings replicated in the independent Nadig25 HepG2 dataset.

The combination-prediction task, where models must predict the transcriptomic effect of perturbing two genes at once, told a more nuanced story. On Norman19, the field’s favorite benchmark, the additive baseline — which simply sums the effects of the two single perturbations — proved nearly unbeatable, and the team explains why: about 96 percent of effects in that dataset are additive, and the training set covers only 31 of 4,950 possible gene pairs, or 0.63 percent of the combinatorial space, leaving almost nothing to learn about nonadditive interactions. The additive baseline captured roughly 88 percent of the ideal performance gap on the best-calibrated metrics. On Wessels23, a dataset with ten times the combinatorial coverage, the picture changed: multiple models surpassed the additive baseline under WMSE and weighted R²Δ, and PRESAGE beat it on 15 of 18 metrics. The team recommends that future studies stop using Norman19 as the sole test of combinatorial modeling, since it largely measures arithmetic rather than learned genetic interactions.

Importantly, the researchers checked that their conclusions reflect biology rather than metric trivia. Models that scored well under well-calibrated metrics also performed better on two downstream tasks: pathway recovery, measured by the correlation between gene set enrichment results computed on predicted versus true perturbation effects, and neighborhood-structure recovery, measured by the overlap between nearest-neighbor graphs built from predicted and observed perturbation shifts. Pathway recovery rankings correlated strongly with well-calibrated metric rankings (average Spearman ρ of 0.76 in Replogle22 K562 and 0.84 in Wessels23), and neighborhood recovery tracked retrieval-based metrics most closely. Sensitivity analyses varying cell counts, sequencing depth, and the number of highly variable genes confirmed that well-calibrated metrics retain their advantage even when data quality degrades.

The study does not declare victory for deep learning outright. The authors caution against relying on any single metric, note that the unseen-context task — predicting responses in unobserved cell types or donors — remains unaddressed, and acknowledge that any fixed set of models represents only a snapshot of a fast-moving field. But the reframing is significant. Prior benchmarks played a constructive role in exposing modeling limitations, and architectural advances have begun to address them; an independent preprint by Cole and colleagues reached complementary conclusions about the value of prior-knowledge embeddings. What this work establishes is that the debate over whether neural networks can predict genetic perturbations was, in part, a debate about measurement. With calibration-aware evaluation built on proper positive and negative controls, the models look considerably more capable than the field believed — and the path toward reliable in silico perturbation screens looks considerably shorter.

Subject of Research: Metric calibration for benchmarking deep learning models of genetic perturbation responses

Article Title: Deep learning perturbation models can outperform baselines on calibrated metrics

Article References: Miller, H. E., Mejia, G. M., Leblanc, F. J. A., Swain, B., Wang, B., & de Lima Camillo, L. P. (2026). Deep learning perturbation models can outperform baselines on calibrated metrics. Nature Biotechnology. https://doi.org/10.1038/s41587-026-03307-w

Image Credits: AI Generated

DOI: 10.1038/s41587-026-03307-w

Keywords: deep learning, genetic perturbation, Perturb-seq, benchmarking, metric calibration, single-cell RNA-seq, dynamic range fraction, weighted MSE, foundation models, drug discovery, computational biology, Nature Biotechnology

The method, described in the journal BMC Bioinformatics by Jack Kamm, Jake Yeung, and William F. Forrest of Genentech, reframes guide assignment as a problem in classical statistics. Rather than fitting an elaborate probabilistic model to the guide count matrix, the authors propose treating the matrix of guide RNA counts across cells as a contingency table. For every cell-guide pair, they construct a two-by-two table that contrasts the counts involving that cell and that guide with all other counts, and then apply Fisher’s Exact Test, a workhorse of categorical data analysis, to ask whether the cell and guide barcodes are statistically associated. A significant association signals that the guide is genuinely present in the cell rather than appearing by chance through sequencing noise.

The elegance of this formulation lies in what it provides automatically. Because Fisher’s Exact Test conditions on the margins of the contingency table, the resulting p-values implicitly normalize for both cell-specific and guide-specific size factors, the uneven sequencing depth that plagues single-cell data. Cells that received more total reads and guides that were more abundant in the library are handled without any explicit normalization step. The method also produces a p-value for every cell-guide pair, giving users a graded measure of confidence rather than a hard binary call, and enabling downstream analyses such as full precision-recall curves when benchmarking alternative cutoffs.

Two statistical subtleties demanded additional innovation. First, because the method performs one test for every cell-guide combination, large screens can involve tens of millions of hypothesis tests. Naive application of standard multiple testing corrections such as the Benjamini-Hochberg procedure would be inappropriate, because the tests are strongly correlated: a cell that tests positive for one guide is less likely to test positive for others, and counts involving the same guide or the same cell share information. The authors developed a multiple testing correction strategy that accounts for this correlation structure, preserving control of the false discovery rate while avoiding the excessive conservatism that would come from ignoring the dependencies.

The second subtlety is a form of confounding that can flip conclusions in exactly the wrong way. Ambient RNA contamination, sometimes called soup or background noise, introduces guide molecules into cells that never received them. If the contamination is not independent across cells and guides, a phenomenon analogous to Simpson’s paradox can arise: an association that appears significant in the aggregate data disappears, or even reverses, once the noise component is considered separately. Fishash addresses this by testing an adjusted odds ratio that replaces the margins involving the focal cell and guide with their noise-based counterparts. To estimate the unobserved noise counts, the method performs an iterative rank-1 Poisson matrix completion: entries that have already been confidently assigned are masked out, the remaining background structure is fit under a Poisson likelihood by alternating updates of guide and cell abundance parameters, and the imputed noise counts feed back into refined p-values. The procedure repeats until the assignments stabilize.

Recognizing that fair comparison of guide assignment methods has been hampered by a lack of realistic test data, the authors also introduced a simulation framework that generates synthetic guide counts under a detailed model of sequencing noise. The simulator draws on the contamination model popularized by the Cellbender software, incorporating latent cell-level factors such as ambient RNA fractions, capture efficiencies, and dropout rates, along with guide-level expression variation. Crucially, the simulations can vary the number of guides in the library and the multiplicity of infection, the average number of guides delivered per cell, allowing benchmarking across the parameter regimes that matter in practice. The code to reproduce all results is publicly available alongside the method itself.

In benchmarks on both simulated and real datasets, Fishash compared favorably with existing approaches in both accuracy and runtime. On simulations varying the number of guides, the method achieved the top median F1 score in the majority of settings, performing particularly well when the guide library was large or guide RNA expression was low. In simulations varying the multiplicity of infection, Fishash remained competitive at low multiplicity, where most cells carry at most one guide, though other methods such as crispat-NB and dcCLEANSER edged ahead at high multiplicity. The authors emphasize that Fishash was conservative in its error control: its observed precision stayed above its nominal lower bound across the sweep, whereas several competing Bayesian and frequentist methods failed to control the false discovery rate at least once across the scenarios tested.

The stress tests also revealed the method’s limits, which the authors report transparently. When extra overdispersion was injected into the noise counts by replacing the Poisson distribution with a Geometric distribution, one of the most extreme overdispersion regimes available, precision dropped for many methods and Fishash no longer controlled the false discovery rate at its nominal five percent level in every scenario. Even so, its F1 scores remained qualitatively similar to the original benchmarks, indicating a competitive balance between precision and recall even under unusually harsh noise. The authors note that the conditional low-rank Poisson structure assumed for background counts is common to many single-cell contamination models, including Cellbender, SoupX, DecontX, and scAR, so the simulation represents a deliberately challenging departure from the assumptions shared across the field.

Practical accessibility was a design priority. Fishash is distributed as an easy-to-use R package on GitHub, and because it relies on exact tests over contingency tables rather than iterative model fitting, it scales comfortably to screens with tens of thousands of cells and guides, where computationally expensive probabilistic approaches can become prohibitive. The software also outputs its test statistic for every entry of the cell-guide matrix, which allows users to visualize the distribution of scores, select alternative significance cutoffs informed by the data, and compute full precision-recall curves during method benchmarking. The work was carried out entirely within Genentech, with the authors crediting colleagues in the company’s AI Biology and Translation department for discussions that shaped the manuscript.

For the growing community running Perturb-seq experiments, the significance of the method is straightforward: guide assignment is the gatekeeper step that determines whether downstream perturbation effects are real or artifacts of misassignment, and errors made here propagate through every subsequent analysis. By combining a classical, well-understood statistical test with modern corrections for multiplicity, confounding, and background contamination, and by packaging the result in fast, open software, Fishash lowers both the computational and conceptual barriers to reliable pooled single-cell screening. As CRISPR screens continue to expand in scale and ambition, tools that trade heavy model fitting for careful, assumption-aware statistics are likely to play an increasingly central role in turning raw sequencing counts into trustworthy biological conclusions.

Subject of Research: A contingency table and Fisher's Exact Test based method for assigning guide RNAs to cells in single-cell pooled CRISPR screens (Perturb-seq)

Article Title: Fishash: a contingency table approach to Perturb-seq guide assignment

Article References: Kamm, J., Yeung, J., & Forrest, W. F. (2026). Fishash: a contingency table approach to Perturb-seq guide assignment. BMC Bioinformatics. https://doi.org/10.1186/s12859-026-06631-9

Image Credits: AI Generated

DOI: 10.1186/s12859-026-06631-9

Keywords: Perturb-seq, CRISPR screen, single-cell RNA-seq, guide assignment, Fisher's exact test, Simpson's paradox, multiple testing correction, contingency table, false discovery rate, ambient RNA contamination, functional genomics, BMC Bioinformatics