The study, led by Vincent Hammer and Stephan Ossowski, tackles a problem that has grown more urgent as spatial transcriptomics has matured. Modern platforms such as Stereo-seq and Visium HD can, in principle, sequence the entire transcriptome at nearly subcellular resolution, revealing which genes are active at which positions within a tissue slice. But most existing analysis tools focus on identifying tissue domains or layers, treating the boundary itself as an afterthought. REBOSEP, by contrast, is boundary-centric: it takes any existing tissue annotation, measures the physical distance of every data point from an annotated border, and represents each gene’s expression as a continuous signal across that border.

The technical challenge is considerable. Although Stereo-seq spots are minuscule, the number of RNA molecules captured in any single spot is too low for meaningful analysis, so spots are merged into bins, typically of 30 by 30 spots, roughly the size of one to four cells. These bins are formed without regard to where cells actually sit, and bins straddling a tissue edge inevitably contain transcripts from both sides. The result is noisy, sparse data in which clustering algorithms assign random labels to isolated bins and even to empty cavities where RNA has diffused during sample preparation. REBOSEP confronts this by letting users filter out small spurious regions, manually correct annotations, and interactively select the exact stretch of boundary they want to study, then reconstructing long continuous borders even across short gaps.

Once a boundary is defined, the workflow computes the average expression of every gene at each distance interval from the border, producing a table that reads like a cross-section of transcriptional activity. Genes whose profiles are too sparse or inconsistent across replicate samples are filtered out using dynamic time warping, a technique borrowed from time-series analysis that aligns curves by locally stretching or compressing them. A restrictive warping window allows the method to tolerate minor positional shifts of about one bin, roughly 30 micrometers, without conflating genuinely different patterns. Hierarchical clustering then groups genes with similar cross-boundary profiles, and gene ontology enrichment analysis reveals the biological processes each group represents.

To demonstrate the approach, the researchers applied REBOSEP to publicly available Stereo-seq data from E16.5 mouse embryos, analyzing five brain boundaries: dorsal pallium-meninges, midbrain-meninges, lower and upper hindbrain-meninges, and subpallium-meninges. Boundary lengths ranged from about 1,740 to 3,667 micrometers, and expression was traced up to 300 micrometers into the brain and 150 micrometers into the surrounding meninges. The resulting atlas captures the expression pattern of every gene the platform detected, including long non-coding RNAs and poorly characterized Riken genes, many of which now carry a boundary-specific expression signature that offers the first clue to their function.

The deepest analysis focused on the dorsal pallium, the embryonic precursor of the neocortex, and its long border with the meninges, the protective membranes that envelop the brain and play active roles in its development. Clustering revealed striking functional homogeneity. Genes expressed constitutively across all pallium layers were overwhelmingly tied to nervous system and neuron development. Genes active throughout the meninges were enriched for extracellular matrix organization, while genes peaking in the meningeal layers closest to the border showed strong enrichment for angiogenesis, the formation of blood vessels, a process closely linked to the developing blood-brain barrier. On the brain side, a distinct cluster of genes sharply upregulated only very near the border was enriched for synaptic signaling and transmission, and another, with a more gradual falloff into deeper pallium layers, was associated with neuron projection and neuron development.

The researchers then examined how specific signaling pathways are spatially regulated at this border. Wnt signaling, a canonical developmental pathway, showed clear boundary-dependent modulation: the receptors Fzd1 and Fzd2 were inhibited at the border, the Wnt-promoting factor Rspo3 peaked around 75 micrometers, and the cell-cycle regulator Ccnd2, a downstream target, was confined to a narrow band within about 20 micrometers of the interface. Notch signaling, essential for cell fate decisions and neurogenesis, was broadly suppressed across the outer brain layers, with Notch receptors highly expressed in the meninges but reduced toward the border. Cell adhesion molecules mediating neuron-neuron and growth cone-axon interactions were largely confined to a thin band of roughly 50 to 150 micrometers within the brain, with genes such as Cntnap2, L1cam, Ncam1, Nrxn1, Nrxn2, and Ptprd gradually decreasing toward deeper layers.

Robustness checks supported the reliability of these patterns. Repeating the clustering a hundred times with random seeds produced vastly higher similarity within clusters than between them, even when artificial noise was added. Comparing Leiden and Louvain clustering algorithms yielded a Spearman correlation of 0.939 between the resulting pattern distances, and varying the bin size had little effect except at the coarsest resolution of 100 bins, well beyond what is typically used. Crucially, the team validated their findings in an independent Visium HD dataset from E15 to E15.5 embryos, a different technology, different specimens, and a slightly earlier developmental stage. The Visium HD profiles closely reproduced the Stereo-seq patterns, preserving the positions of expression peaks, their relative amplitudes, and their rising or decaying trends.

Perhaps the most provocative finding concerns disease. Many of the genes with strong boundary-specific expression, including Apc, Ccnd2, Tle4, Ctnnd2, L1cam, Nrcam, Cadm1, Cntn1, Nlgn1, and Nrxn2, are known players in cancer biology, autism spectrum disorder, or broader neurodevelopmental syndromes. Tumor malignancy is, in essence, the breakdown of a tissue boundary, and boundary roughness is a recognized marker of invasiveness and poor prognosis. The fact that genes governing adhesion and Wnt signaling are so precisely localized at healthy embryonic borders suggests that tumors may co-opt the same molecular machinery to remodel and invade across tissue frontiers, making these boundary profiles a rich source of candidate mechanisms and therapeutic targets.

The authors are candid about limitations. Gene-level interpretations remain exploratory and hypothesis-generating, the quality of boundary definitions depends on upstream tissue segmentation, current capture rates prevent true single-cell resolution, and all analyzed datasets came from female mice, so sex-specific effects could not be assessed. Still, the resource and the accompanying code, both publicly available on GitHub, lower the barrier for any laboratory to interrogate boundaries in their own spatial transcriptomics data. As spatial omics continues to spread through developmental biology, oncology, and neuroscience, tools like REBOSEP signal a shift in perspective: the most interesting biology may lie not within tissues, but precisely at the lines where they meet.

Subject of Research: Spatial transcriptomics analysis of gene expression patterns across tissue boundaries in the developing mouse brain

Article Title: Spatial transcriptomics profiling across tissue boundaries in the developing mouse brain

Article References: Hammer, V., & Ossowski, S. (2026). Spatial transcriptomics profiling across tissue boundaries in the developing mouse brain. iScience, 29(11), Article 117765. https://doi.org/10.1016/j.isci.2026.117765

Image Credits: AI Generated

DOI: 10.1016/j.isci.2026.117765

Keywords: spatial transcriptomics, tissue boundaries, REBOSEP, developing mouse brain, Stereo-seq, Visium HD, meninges, dorsal pallium, Wnt signaling, Notch signaling, cell adhesion, neurodevelopment

The difficulty is geometric. Tissue slices are thin sections cut from three-dimensional structures, and the act of cutting, mounting, and preparing them introduces rotations, shifts, and stretches. One slice of a mouse brain may be tilted a few degrees relative to another; one may be slightly distorted so that its cortical layers sit at different distances from the edge. When researchers overlay two such maps, equivalent regions do not automatically coincide. Comparing gene activity in the wrong regions produces misleading conclusions, so meaningful comparison first requires alignment, a computational task known as registration, in which one digital map is shifted, rotated, and smoothly reshaped until its regions correspond to those of another.

Researchers at Kanazawa University in Japan and Sapienza University of Rome have now introduced a method designed to perform this registration directly on the raw spatial transcriptomics data, without sacrificing the cell-level resolution that makes the technology valuable. The approach, called Domain Elastic Transform, or DET, was developed by Osamu Hirose of Kanazawa University in collaboration with Emanuele Rodolà of Sapienza University of Rome and published online in IEEE Transactions on Pattern Analysis and Machine Intelligence on September 15, 2026. The work addresses a persistent trade-off in the field between preserving fine spatial detail and achieving robust alignment.

That trade-off arises because existing alignment strategies each make compromises. Image-based methods can incorporate gene-activity information, but they generally begin by converting the measurements into a regular grid of pixels, a step that blurs fine structures and reduces cell-level detail. Point-based methods, by contrast, keep the measurements as individual points and align them using their locations alone, but this can confuse regions whose shapes are similar while their gene-expression patterns differ. Other approaches can link cells or regions across samples without estimating how the tissue map should be smoothly reshaped, leaving the underlying geometric distortion unresolved. Each strategy uses only part of the information that spatial transcriptomics provides.

DET takes a different route by treating every measured cell or location as carrying two kinds of evidence at once: where it sits in the tissue and which genes are active there. The method represents each cell by its position and its measured gene-activity profile, then proceeds through an iterative process with two alternating steps. First, it estimates likely matches between cells in the two maps, drawing on both their spatial positions and the similarity of their gene-activity values. Second, it adjusts the positions of cells in one digital map, encouraging neighboring cells to move together so that the transformation remains smooth rather than chaotic. Repeating these steps gradually refines the alignment until it stabilizes.

Two design choices distinguish the method. DET modifies the positions of cells in the digital map but never alters the recorded gene-activity values themselves; the expression data serve only as clues about which regions should match. And the method requires no collection of pre-aligned training examples and no manually identified correspondences, making it training-free and unsupervised. In principle, this means it can be applied to any pair of tissue maps without the laborious preparation of reference alignments, a significant practical advantage in a field where verified ground truth is scarce.

The researchers evaluated DET against competing methods in 90 test cases using mouse brain maps generated with MERFISH, a technique that maps gene activity at single-cell resolution. To make the task deliberately difficult, they digitally rotated the maps by random angles spanning the full range from 0 to 360 degrees and shifted them by large distances. Among the methods tested, DET achieved the highest scores on three measures: how well the tissue maps overlapped after alignment, whether neighboring cells remained neighbors in the aligned result, and how closely gene-activity patterns agreed between corresponding regions. For the gene-activity comparison, the researchers combined expression information from nearby cells before comparing patterns, providing a robust assessment of functional correspondence.

The team then confronted a harder biological scenario: developing embryos, whose shapes change over time along with the types and proportions of cells they contain. Using maps from the MOSTA atlas, a collection of mouse development maps created with Stereo-seq, a technique that records gene activity across tissue, they aligned mouse-embryo maps from two developmental stages one day apart. Each map contained more than 100,000 measurement locations. DET first adjusted the overall position and orientation of the maps, then smoothly reshaped one to refine the alignment. Because no verified reference existed showing exactly which locations should match across these stages, the experiment served primarily to demonstrate that the method could be applied to maps of this size and complexity, a scale at which many alignment approaches struggle.

Scalability was tested further by artificially expanding MERFISH-based data to as many as one million points per tissue slice. Peak memory use averaged less than one gigabyte in every setting tested. The method achieves this efficiency through a landmark strategy: it estimates the alignment using a selected subset of points, called landmarks, and extends the estimated movement to the full map. The number of landmarks, which the researchers call the landmark budget, controls how much computation the reshaping step requires. With the number of landmarks held fixed, peak memory use grew approximately in proportion to the total number of points, while computation time depended mainly on the number of landmarks used in the main alignment step rather than on the full dataset size. The researchers note that exact runtimes depend on the computer and algorithm settings, but the results support applying DET to very large tissue maps.

The implications extend beyond tissue slices. Many scientific datasets combine locations with measurements, and similar alignment challenges arise with data attached to three-dimensional surfaces or anatomical structures. The researchers position DET as a complement to existing alignment methods rather than a universal replacement, designed for tasks that require both fine spatial detail and a smooth transformation of the map rather than only a list of matching locations. The broader aim, as Hirose describes it, is to use both kinds of information that such datasets carry, where something is and what was measured there, when bringing different datasets into alignment, rather than treating shape and measurements separately. For spatial transcriptomics, that could mean more reliable comparisons of corresponding regions across samples, even when their shapes and gene-activity patterns differ, strengthening studies of how organs develop and how disease reorganizes the body’s molecular geography. The research was supported in part by the Japan Science and Technology Agency FOREST Program under grant JPMJFR242V.

Subject of Research: A computational registration method for aligning spatial transcriptomics maps of gene activity across tissue samples

Article Title: Making maps of gene activity easier to compare

Article References: Making maps of gene activity easier to compare. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: spatial transcriptomics, image registration, Domain Elastic Transform, gene expression, MERFISH, Stereo-seq, mouse brain, mouse embryo, MOSTA atlas, computational biology, Kanazawa University, IEEE TPAMI