Subcellular RNA Patterns Improve Cell Clustering in Spatial Transcriptomics

Researchers have developed a computational framework named SPICEiST that leverages subcellular RNA distribution patterns to enhance the accuracy of cell clustering in imaging-based spatial transcriptomics. Published in Genomics & Informatics, the study presents a graph autoencoder approach designed to address the inherent limitations of current spatial transcriptomics technologies, specifically the small gene panels and challenges in accurate cell segmentation. By integrating subcellular transcript distribution with conventional cell-level gene expression profiles, the method aims to distinguish subtle differences in cell states that are often obscured by standard analytical methods.

Imaging-based spatial transcriptomics allows for the quantification of gene expression at single-cell resolution while preserving the spatial context of the tissue. Technologies such as Xenium, MERSCOPE, and CosMx SMI capture the locations of predefined RNA types and delineate cell boundaries using immunofluorescence images. However, the number of genes that can be targeted simultaneously, known as a plex, typically ranges from a few hundred to a few thousand. This limitation makes these platforms less suitable for unbiased screening of the tissue microenvironment compared to single-cell RNA sequencing, which captures transcriptome-wide expression profiles.

In most studies, imaging-based spatial transcriptomics data is analyzed similarly to single-cell RNA sequencing datasets, starting with a cell-by-gene transcript count matrix. However, transcripts in imaging-based systems are allocated to cells by analyzing images of cell membranes or the expansion of nuclear regions, rather than through cell dissociation. This technique can result in inaccuracies when assigning transcripts to cells, primarily due to the substantial overlap between closely situated cells in a two-dimensional space. Consequently, the combination of a limited gene panel and potential errors in cell segmentation leads to ambiguous distinctions between cell types based on unsupervised cell clustering.

To address these challenges, the researchers utilized a graph autoencoder to extract subcellular gene expression patterns and integrate them with individual cellular gene expression profiles. The framework, SPICEiST, stands for Subcellular Pattern Integration with Cellular Expression in Imaging-based ST. The method constructs a graph for each cell where nodes correspond to unit grids and are featured with both local subcellular-level and global cell-level gene expression data. Using graph convolutional layers, the autoencoder learns a 64-dimensional latent embedding for each cell, guided by a dual-objective loss function that reconstructs fine subcellular details while ensuring the latent space reflects the overall cellular expression profile.

The clustering performance of SPICEiST was systematically evaluated across several cancer datasets and gene panel sizes. The study utilized publicly available datasets from Xenium and CosMx SMI platforms, including human lung and colorectal cancer tissues. For the Xenium lung cancer dataset, the researchers compared the performance of SPICEiST against a non-spatial baseline method (GEX) and a spatial clustering method called SpaceFlow. The GEX method treated the data as a single-cell RNA sequencing dataset, relying solely on cell-level gene expression profiles, while SpaceFlow incorporated the spatial coordinates of cells alongside their gene expression.

Results demonstrated that SPICEiST consistently outperformed conventional cell-level gene expression-based methods in distinguishing subtle differences in cell states. This superiority was measured by the number of cell clusters and clustering indices, such as the Calinski-Harabasz index (CHI) and the Davies-Bouldin index (DBI). The findings indicated that SPICEiST could further enhance performance even with advancements in cell segmentation, particularly for datasets with small gene panels. The improvements in cell clustering indices were more pronounced in datasets with small gene panels of around 300 genes, in contrast to those with large panels containing over a thousand genes.

Notably, SPICEiST revealed more spatially intermixed and less compartmentalized cell clusters, a characteristic that better reflects the complex and heterogeneous nature of tumor microenvironments. This effect was especially evident in datasets with large panels. The assortativity coefficient, which measures spatial coherence, was significantly lower in SPICEiST compared to SpaceFlow and GEX. While this indicates less organized spatial clusters, the researchers suggest that this may more accurately represent the disorganized and heterogeneous nature of tumors compared to the more compartmentalized patterns identified by other models.

To assess the biological significance of the clustering results, the researchers calculated homogeneity and completeness scores by comparing the Louvain clusters generated by SPICEiST and GEX with annotated cell types. A high homogeneity score indicates that clusters are pure, consisting of cells from a single type, while a low completeness score signifies that a single annotated cell type is distributed across multiple clusters. SPICEiST yielded significantly lower completeness scores than GEX, particularly in the Xenium v1 platform. This combination of high homogeneity and low completeness indicates that SPICEiST can separate individual ground-truth cell types into multiple distinct clusters, demonstrating its ability to identify fine-grained cell subtypes.

In the Xenium colorectal cancer dataset, SPICEiST offered a more comprehensive depiction of the tumor microenvironment, showing intricate intermixture between distinct cell types. Marker gene analysis on clusters predominantly consisting of malignant cells revealed distinct functional profiles. Cluster 0 was associated with fibronectin matrix formation and epithelial cell differentiation, Cluster 4 with regulation of vascular wound healing, and Cluster 12 with G1 and S phase-specific transcription and one-carbon metabolic processes. These distinctions highlight the ability of SPICEiST to refine broad cell categories into more specific and biologically relevant subtypes.

The study also evaluated the performance of SPICEiST in the CosMx SMI lung cancer dataset, which has a panel size of 1,001 genes. The observed patterns in CHI, DBI, and assortativity coefficients exhibited congruence with those described in the Xenium datasets. Visual inspection revealed that SPICEiST effectively represented highly intermixed regions of cell infiltration that the GEX model failed to capture. However, many other patches did not show a clear distinction between the two models, suggesting that SPICEiST may be less beneficial in datasets with large panel sizes.

The findings highlight the value of leveraging subcellular transcript patterns to overcome the inherent limitations of imaging-based spatial transcriptomics, particularly for small gene panels. The approach provides new insights into tumor heterogeneity by identifying meaningful cellular subtypes that are not resolved by conventional methods. The researchers noted that the additive effects of integrating subcellular expression patterns are more pronounced in imaging-based ST when a smaller gene panel is utilized.

Limitations of the study include the potential for over-clustering at higher resolutions, which may result in less compact clusters for the SPICEiST model. The trends of relatively lower Average Silhouette Width in SPICEiST at higher resolutions may stem from the elongated distribution patterns of fine clusters within the feature space. Additionally, the enhancement of spatial disorganization in smaller panels may result in a less comprehensive explanation of the spatial intermixture of cells, indicating potential limitations in capturing the full heterogeneous nature of tumors in small panel datasets.

Overall, the study demonstrates that utilizing subcellular gene expression patterns alongside cell-level gene expression profiles through a graph autoencoder can address the limitations of imaging-based spatial transcriptomics. This approach proves especially effective in distinguishing subtle differences in cell states within small gene panel-based platforms, thereby enhancing the understanding of tumor heterogeneity and providing a robust tool for spatial transcriptomics analysis.

Subject of Research: Development of SPICEiST, a graph autoencoder framework for enhancing cell clustering in imaging-based spatial transcriptomics using subcellular RNA patterns.

Article Title: SPICEiST: subcellular RNA pattern enhances cell clustering of imaging-based spatial transcriptomics

Article References: Bae, S., Seong, Y., Lee, D., & Choi, H. (2025). SPICEiST: subcellular RNA pattern enhances cell clustering of imaging-based spatial transcriptomics. Genomics & Informatics, 23(1), Article 23. https://doi.org/10.1186/s44342-025-00056-1

Image Credits: AI Generated

DOI: 10.1186/s44342-025-00056-1

Keywords: spatial transcriptomics, cell clustering, graph autoencoder, subcellular RNA, tumor heterogeneity, bioinformatics, imaging-based ST, Xenium, CosMx SMI, cancer research, SPICEiST, subcellular