ResolVI cleans up noisy spatial transcriptomics data after segmentation goes wrong

Spatial transcriptomics has transformed how biologists look at tissue. Instead of dissolving a sample into individual cells and losing all sense of place, these technologies measure gene activity directly inside intact tissue slices, revealing which cells sit where, how they talk to their neighbors and how entire tissues organize themselves into functional units. But beneath the dazzling spatial maps lies a stubborn technical problem that has quietly undermined many analyses: a substantial fraction of the RNA molecules being counted are simply assigned to the wrong cells. A new open-source tool called resolVI, published in Nature Methods by Can Ergen of the University of California, Berkeley and the University of Würzburg and Nir Yosef of the Weizmann Institute of Science, tackles this problem head-on with a probabilistic deep learning framework that cleans up the data after the fact, rather than trying to fix segmentation itself.

To understand why misassignment happens, it helps to look at how subcellular-resolution spatial transcriptomics actually works. Technologies such as 10x Xenium, Vizgen MERSCOPE and Nanostring CosMx image individually tagged RNA molecules inside a tissue slice, while sequencing-based platforms such as Stereo-seq capture RNA on densely barcoded spots. In both cases, a crucial computational step called segmentation divides the tissue plane into regions, each meant to approximate a single cell, and the molecules inside each region are tallied to estimate that cell’s gene expression profile. Early segmentation algorithms relied on stained images of nuclei or cell membranes, but dense tissue, weak staining and irregular cell shapes make accurate boundary drawing genuinely difficult. Newer methods fold the RNA molecules themselves into the segmentation process, which helps, yet errors persist.

Even a perfect two-dimensional segmentation cannot fully solve the problem, because molecules can end up inside the wrong cell for reasons that have nothing to do with boundary drawing. The researchers describe a so-called diffusion phenomenon, in which RNA molecules leak from their cell of origin. This can result from tissue handling issues or from the simple fact that cells overlap in the third dimension, perpendicular to the profiled plane, which is typically captured only to a restricted depth of tens of microns. A molecule sitting above one cell may be projected onto its neighbor. On top of this, a nonspecific background signal contaminates every tissue slice. The combined effect is that observed expression profiles become more complex than the true ones, producing biologically impossible patterns such as a microglial cell apparently co-expressing a neuronal marker gene.

ResolVI takes a deliberately different approach. Rather than replacing existing segmentation pipelines, it operates downstream of any segmentation algorithm, taking the initial per-cell gene expression estimates as input and producing corrected, probabilistic representations of both cell states and expression profiles. At its core is a variational autoencoder, a deep generative model trained by optimizing the evidence lower bound. The model represents the observed counts in each cell as the sum of three components: the cell’s true expression, expression that originated from nearby cells and leaked in through diffusion, and a nonspecific background that is constant for each tissue slice. True expression and diffusion are modeled with a negative binomial distribution, while the background uses a Poisson distribution. Cell states are encoded in a low-dimensional latent space governed by a mixture-of-Gaussians prior, a choice that helps separate cell types more cleanly than the standard unimodal approach and allows users to supply known cell-type labels during training.

The intuition behind the correction is elegant. Because the model expects the data to be describable by a low-dimensional encoding, expression profiles inflated by erroneously assigned molecules become harder to reconstruct faithfully. Minimizing the reconstruction loss therefore encourages the model to push apparently unrelated molecules into the background or neighbor-contribution components, leaving a cleaner estimate of what the cell actually expresses. The architecture consists of three neural networks: an expression encoder that infers cell state from observed counts, an expression decoder that generates true expression from that state, and a diffusion encoder that estimates how much of each cell’s signal comes from itself, its neighbors and the background. The decoder receives the batch identifier of each cell as input while the encoder does not, a design that mitigates batch effects. The implementation runs on GPUs, and in the largest test case, covering 1.4 million cells across 52 tissue slices, training took just under six hours, a modest overhead compared with segmentation itself.

The team first tested resolVI on a mouse brain slice profiled with 10x Xenium, covering 248 genes and roughly 130,000 cells. The model estimated a diffusion rate for every cell, and these rates were highest in densely packed regions such as the cortex and thalamus, exactly where segmentation is hardest. The clearest demonstration involved microglia, the brain’s resident immune cells. In the raw data, microglia in the hypothalamus appeared to express Slc17a6, a marker of excitatory neurons, an artifact almost certainly caused by molecules leaking across poorly drawn boundaries. After resolVI correction, this false-positive signal was drastically reduced while genuine microglial markers such as Trem2 were preserved. Quantitatively, using marker gene lists derived from single-cell RNA sequencing references, the corrected data showed far fewer erroneous double-positive cells, while true co-expression patterns were retained at the same rate as in the raw counts.

Importantly, the model does not force the data to conform to preconceived notions of what each cell type should express. In the brain data, resolVI preserved the co-expression of Lyz2, usually considered a myeloid marker, in endothelial cells, a pattern that is biologically real even though it was absent from the single-cell reference dataset. In a liver cancer dataset profiled with Vizgen MERSCOPE, the model retained VEGFC expression in endothelial cells, consistent with the known autocrine signaling of this growth factor in tumors under hypoxic conditions. This ability to distinguish systematic artifacts from genuine biology, rather than simply scrubbing everything that looks unexpected, is what separates resolVI from simpler denoising approaches.

The researchers then stress-tested the tool across segmentation algorithms and platforms. Comparing six segmentation strategies on the liver cancer data, they found that expression-aware methods such as Baysor and ProSeg produced fewer artifacts than image-based approaches, but that residual misassignment remained in every case, and resolVI reduced erroneous co-expression downstream of all of them. On sequencing-based Stereo-seq data with more than 24,000 genes, where expression-aware segmentation is computationally impractical, resolVI processed the full dataset in under 20 minutes and eliminated nearly all false co-expression between the neuronal marker Slc17a7 and the oligodendrocyte marker Mbp, dropping from 14.8 percent of Slc17a7-positive cells to below 0.01 percent. Applied to 10x Visium HD data, it resolved inhibitory neuron subtypes, including VIP, SST and NPY-expressing populations, that were indistinguishable in conventional principal component analysis.

The biological payoff is perhaps most vivid in the disease applications. In human liver cancer samples profiled with Nanostring CosMx, covering 1,000 genes and 1.2 million cells, resolVI-corrected data revealed two distinct tumor niches: one pro-inflammatory, marked by interferon-regulated genes such as MX1, IFI6 and STAT1, and one immunosuppressive and hypoxic, marked by MIF, ENO1, SPP1 and VEGFA. Tumor-associated macrophages expressing SPP1, a known mediator of macrophage-tumor interactions, colocalized with the immunosuppressive tumor region, while pro-inflammatory macrophages and infiltrating T cells occupied the other niche. In a mouse model of inflammatory bowel disease, the tool mapped 1.4 million cells across 52 samples and uncovered incomplete regeneration 35 days after colitis induction: crypt-tip fibroblasts, which normally deliver Bmp signals that support epithelial maturation, had relocated into aggregates in the submucosa, and barrier-promoting genes in colonocytes were reduced, suggesting lingering barrier dysfunction long after visible recovery.

ResolVI is released as open-source software within the widely used scvi-tools framework, integrating into existing Python and R workflows, and the authors provide practical guidance: they recommend initial segmentation with ProSeg, since Baysor, while yielding the cleanest downstream results in their benchmarks, tends to inflate the number of segmented cells. The tool also supports a supervised mode with cell-type labels, transfer learning for mapping new samples onto reference models, and built-in differential expression and spatial niche abundance testing. As spatial transcriptomics moves from technology demonstration toward routine clinical and biomedical use, tools like resolVI address a problem that every laboratory generating these data will recognize, and the demonstration that a model can separate technical ghosts from biological surprises may prove as important as the maps themselves.

Subject of Research: A probabilistic deep learning model that corrects segmentation errors and batch effects in spatial transcriptomics data

Article Title: ResolVI: addressing noise and bias in spatial transcriptomics

Article References: Ergen, C., & Yosef, N. (2026). ResolVI: addressing noise and bias in spatial transcriptomics. Nature Methods, 23(10), 2003-2014. https://doi.org/10.1038/s41592-026-03212-9

Image Credits: AI Generated

DOI: 10.1038/s41592-026-03212-9

Keywords: spatial transcriptomics, ResolVI, variational autoencoder, segmentation errors, batch correction, scvi-tools, single-cell genomics, mouse brain, liver cancer, colitis, machine learning, gene expression