The benchmark covered a deliberately broad methodological spectrum. The evaluated tools included classical image-processing approaches such as Watershed, deep-learning image-only models such as Cellpose, the Segment Anything Model family (SAM, SAM2 and the microscopy-tuned mSAM), transformer-based methods like CelloType and the Swin Transformer, the instance-segmentation framework Cascade Mask R-CNN, and methods that integrate transcriptomic information with imaging, namely BIDCell, SCS and Baysor. The datasets ranged from roughly 700 cells in an ovarian cancer sample to more than 800,000 cells in a prostate cancer dataset, and were produced by Stereo-seq, Xenium, CosMx, STARmap PLUS and MERFISH. Two methods, the Swin Transformer and Cascade Mask R-CNN, were ultimately excluded from quantitative comparisons because they produced biologically implausible, grid-like artifacts dominated by patch boundaries rather than cellular structures.

To make the comparison fair, the team built a unified Nextflow pipeline that applied standardized quality control, consistent preprocessing and a patch-based processing strategy across all methods. Images were converted to grayscale, transcript coordinates were mapped into a common pixel-based space, and when memory constraints made full gene sets impractical for transcript-aware tools, inputs were restricted to the top 1,000 highly variable genes. Segmented outputs were filtered to retain only high-quality cells meeting minimum transcript and gene-count thresholds, mirroring practices established in earlier segmentation studies. The pipeline then realigned the patchified results into the original spatial coordinate space before scoring, ensuring that every method was judged on identical inputs and under identical evaluation rules.

Scoring relied on a suite of 15 biologically informed metrics, collectively termed the CellSPA metrics, which were normalized to a 0-to-1 scale. These measures captured segmentation completeness (the number of high-quality cells and the proportion of transcripts assigned to cells), transcriptomic richness (transcripts and genes per cell), gene-level consistency, cell morphology, cell-type composition similarity against reference annotations, expression purity based on known positive and negative markers, spatial expression diversity, and the coherence of cell-type structure in reduced-dimensional embeddings. The authors grouped these 15 metrics into eight higher-level categories to aid interpretation, while stressing that they did not define an optimal metric subset, since the most relevant criteria likely depend on the dataset and biological question at hand.

Under an equally weighted aggregate score, SAM achieved the highest overall value at 0.58, followed by SAM2 at 0.53, BIDCell at 0.52 and SCS at 0.48. Baysor and mSAM tied at 0.45, Cellpose reached 0.42, CelloType 0.37, and Watershed consistently trailed at 0.27. But the picture changed when the aggregate score was adjusted for coverage, that is, the ratio of segmented cells to reference cells. Under this coverage-adjusted metric, SCS came first at 0.34, with BIDCell and Baysor close behind at 0.33. BIDCell and Baysor also accumulated the most dataset wins, six and five respectively, indicating recurrent top performance across diverse tissues when both quality and yield were considered together.

The metric-level breakdown exposed fundamental trade-offs. SAM and SAM2 delivered the highest transcriptomic richness, with SAM scoring 0.71 for transcripts per cell and 0.73 for genes per cell, yet SAM segmented only a fraction of the reference cells (0.28), and SAM2 even fewer (0.12). BIDCell, by contrast, combined strong coverage (0.62) with high concordance between recovered and reference expression profiles (0.89 for expression correlation). SCS performed well on morphology-related and spatial-diversity measures and achieved the highest silhouette score at 0.75, indicating effective preservation of transcriptional structure. Notably, expression purity showed no correlation with any other metric category across datasets, marking it as an independent and essential dimension of evaluation, and its two constituent measures were themselves negatively correlated, revealing trade-offs even within a single conceptual category.

Perhaps the most consequential findings came from the cross-technology analysis. The team compared three mouse brain datasets generated with Stereo-seq, STARmap PLUS and MERFISH, restricting the analysis to robustly expressed genes and three reliably annotated cell types: astrocytes, endothelial cells and microglia. Stereo-seq reliably captured astrocyte markers across segmentation tools, while MERFISH failed to recover expected astrocytic patterns and instead showed overexpression in unrelated cell types. The trend reversed for microglial markers, where MERFISH and STARmap PLUS recovered the marker Ctss reliably but Stereo-seq did not. STARmap PLUS performed best for endothelial markers. MERFISH, although capable of detecting multiple cell types, exhibited elevated off-target expression, with contamination affecting up to roughly 10 to 15 percent of cells in some conditions, a level of misassignment the authors attribute to segmentation differences rather than background noise alone.

These platform-specific effects propagated directly into downstream biological inference. When the researchers computed ligand–receptor interactions with the Giotto R package for three well-established signaling pairs, Pdgfc–Pdgfra, Prok2–Prokr2 and Vip–Vipr2, STARmap PLUS and Stereo-seq consistently placed interacting partners in the expected cell populations, whereas MERFISH frequently misassigned interactions to incorrect cell types and displayed broader signal dispersion. The authors link these patterns to known technical attributes of the platforms: STARmap PLUS uses highly specific in situ hybridization probes with three-dimensional resolution, Stereo-seq is more susceptible to cross-hybridization and background noise in densely packed regions, and MERFISH showed the highest variability and contamination, which segmentation tools could not fully resolve.

The benchmark also carries practical guidance for researchers designing spatial transcriptomics studies. Image-based methods such as SAM performed well on datasets with visually distinct cells, whereas SCS fared better on tissues with more uniform cellular appearances. Methods integrating transcriptomic information generally required longer execution times than image-only approaches, adding a computational cost to the accuracy-versus-coverage trade-off. The authors provide a decision tree to help researchers match segmentation tools to their tissue architecture, platform and analytical goals, and they emphasize that studies focused on comprehensive spatial maps, rare-cell recovery or cell–cell communication may need coverage-aware evaluation, while analyses demanding high-confidence cellular profiles may benefit from stricter quality-oriented criteria.

The team is candid about limitations. The benchmark does not exhaust all available spatial transcriptomics technologies or parameter settings, and differences in platform design, image quality, gene panels and tissue composition make it difficult to fully separate algorithmic performance from technology-specific effects. The gene-set sensitivity analysis was restricted to a single melanoma skin dataset, where it showed only modest, method-dependent effects, and should be read as a preliminary assessment rather than a general conclusion. Because most publicly available replicates are serial sections rather than true technical reruns of identical tissue, replicate-focused benchmarking remains a dedicated future direction. Even so, the message is clear: as spatial biology moves toward reference atlases, cell–cell communication inference and foundation models, segmentation quality sets the upper bound on what any downstream analysis can achieve, and the standardized Nextflow pipeline released with this study offers a reproducible foundation for evaluating future methods under a common standard.

Subject of Research: Benchmarking of cell segmentation methods for spatial transcriptomics across platforms, tissues and evaluation metrics

Article Title: Benchmarking segmentation methods for spatial transcriptomics

Article References: Martínez-de-Morentin, X., Khan, S. A., Maillo, A., Lehmann, R., Kiani, N. A., Lagani, V., Gomez-Cabrero, D., & Tegner, J. (2026). Benchmarking segmentation methods for spatial transcriptomics. Nature Methods. https://doi.org/10.1038/s41592-026-03272-x

Image Credits: AI Generated

DOI: 10.1038/s41592-026-03272-x

Keywords: spatial transcriptomics, cell segmentation, benchmarking, Nature Methods, SAM, BIDCell, Baysor, SCS, MERFISH, Stereo-seq, Xenium, ligand-receptor inference