Segment anything in pathology images with natural language

{“title”:”New AI Model Segments Pathology Images Using Simple Text Descriptions”,”html”:”Researchers have unveiled an artificial intelligence foundation model that can segment virtually any tissue or cellular structure in pathology images simply by being told, in plain natural language, what to look for. The system, called PathSegmentor, and the massive benchmark dataset that underpins it, known as PathSeg, are described in a study published in Nature Computational Science. The work addresses one of the most persistent bottlenecks in computational pathology: the laborious, task-by-task engineering of segmentation models that can recognize only a narrow slice of the biological structures that pathologists encounter every day.

Segmentation, the pixel-by-pixel outlining of objects of interest within an image, is the bedrock of quantitative pathology. Once a tumor region, a gland, a nucleus, or an inflammatory cell cluster has been precisely delineated in a digitized tissue slide, everything else follows: automated cell counting, measurement of invasive fronts, quantification of stromal composition, and the downstream prediction of diagnosis and prognosis. Yet the traditional route to such measurements has demanded a dedicated deep learning model for each structure of interest, each trained on painstakingly annotated data, or else repeated manual prompting in which a user must click points or drag boxes around every single object in every single image. Neither approach scales gracefully to the enormous structural diversity of human histology.

PathSegmentor takes a different route. Instead of relying on spatial prompts, the model accepts a textual description of the target structure, such as a named anatomical region, histological structure, or object type, and produces the corresponding segmentation mask directly. The language interface means that a researcher or clinician does not need to identify and localize every object before the model can work; a single descriptive phrase suffices. In practice, the team reports, these text prompts substantially reduced the need for object-by-object point or box annotations while remaining robust to natural variation in wording, so that slightly different phrasings of the same request still yield reliable segmentations.

The foundation for this flexibility is PathSeg, an aggregated resource assembled from 21 publicly available pathology image segmentation datasets. Together these contribute 275,200 image–mask–label triples, each pairing a pathology image with its expert ground-truth mask and a categorical label. Crucially, the labels are not a flat, idiosyncratic jumble. The team organized them into a three-level hierarchy that moves from anatomical region, through histological structure, down to the specific object type. This structure allows a single naming scheme to span everything from broad tissue classes to individual epithelial cells, smooth muscle cells, leukocytes, red blood cells, lymphocytes, healthy and cancerous prostate glands, and tumor epithelium, across organs including the breast, colon, lung, and prostate.

Trained on this resource, a single PathSegmentor model achieved the highest overall segmentation performance across 16 internal test datasets, encompassing roughly 45,000 evaluated image–mask pairs. The comparisons were rigorous and broad: the authors benchmarked against task-specific state-of-the-art architectures such as nnU-Net and DeepLabV3+, against specialized pathology adaptations like SAM-Path, and against spatial-prompted segmentation foundation models including MedSAM and SAM-Med2D, as well as the multi-modal biomedical model BiomedParse. Statistical comparisons used one-sided Student’s t-tests with Holm-adjusted p-values and bootstrap-estimated 95 percent confidence intervals, and the model’s advantage was consistent across the hierarchy of anatomical regions, histological structures, and object types, with particularly notable gains on intricate, morphologically complex objects where competing approaches struggled.

Internal performance alone proves little in medical imaging, where domain shift, differences in scanners, staining protocols, patient populations, and disease prevalence, can silently degrade a model that looked flawless in development. The researchers therefore validated PathSegmentor on external public datasets and, importantly, on clinical cohorts contributed by collaborating hospitals, including two clinical datasets whose redistribution is restricted by institutional rules. The model generalized across these external settings and across object categories, suggesting that its text-driven, hierarchy-aware design captures representations of histological structure that transfer beyond the specific datasets it was trained on.

Perhaps the most clinically consequential application demonstrated in the study is interpretability. Deep learning classifiers for breast cancer, including models trained on large whole-slide image collections, are famously opaque: they produce a diagnosis or a biomarker call without a transparent account of which image features drove the decision. The team exploited PathSegmentor’s predicted structures as a lens onto these classification models. By systematically perturbing individual segmented objects and by mapping class activation within the boundaries of each predicted structure, they could attribute a breast cancer classifier’s behavior to specific, nameable histological entities, distinguishing the contributions of tumor regions, healthy glands, and other recognized structures in a way that raw saliency maps cannot.

This object-level explanation capability connects to a broader movement in medical artificial intelligence toward models whose reasoning can be audited by the clinicians who must act on their outputs. Instead of explaining a black box with another approximation of a black box, the approach grounds explanations in concrete anatomical and histological units that pathologists already understand. For cancer diagnosis in particular, where lobular versus ductal morphology, lymphocytic infiltration, and gland architecture all carry distinct clinical meaning, the ability to ask not just what the model predicted but which structures it relied upon could accelerate trust, regulatory review, and clinical adoption of computational pathology tools.

The authors note that their results establish a unified framework for flexible pathology segmentation with potential utility for clinically interpretable image analysis, and the practical infrastructure reflects that ambition. The PathSegmentor source code and analysis scripts are publicly released under an MIT License, with the exact release archived on Zenodo, and PathSeg’s dataset sources and harmonization procedures are documented in the paper’s supplementary materials. Remaining challenges are candidly acknowledged in the authors’ own failure analysis: the model can still struggle with complex or ambiguous boundaries and with densely clustered cells, and the clinical datasets underpinning external validation are available only by request through the contributing hospitals. Even so, the combination of a quarter-million annotated triples, a single model spanning organs and scales, and a natural-language interface represents a significant step toward pathology image analysis in which the question you ask, in words, determines what the machine sees.

“,”excerpt”:”A new foundation model called PathSegmentor uses natural language prompts to segment structures across pathology images, outperforming specialized models and enabling interpretable cancer diagnosis.”,”subject”:”A natural-language-driven foundation model for universal segmentation of tissues and cells in digital pathology images.”,”tags”:[“PathSegmentor”,”PathSeg”,”digital pathology”,”image segmentation”,”foundation model”,”natural language prompts”,”deep learning”,”computational pathology”,”breast cancer”,”explainable AI”,”medical imaging”,”machine learning”]}`

The lineage behind PathSegmentor traces back to the Segment Anything Model, the general-purpose vision system introduced in 2023 that demonstrated how a single segmentation model, trained on more than a billion masks, could generalize to unfamiliar images when guided by interactive prompts. That breakthrough quickly inspired medical adaptations, most prominently MedSAM, which showed that the same prompt-driven paradigm could be retuned for radiology, microscopy, and other clinical modalities. What these systems lacked, however, was a way to specify targets semantically. A point or box tells a model where to look, but not what it is looking at, which is precisely the gap that language-prompted approaches such as BiomedParse and PathSegmentor now aim to close in biomedical imaging.

The three-level label hierarchy in PathSeg also reflects a practical truth about pathology annotation: structures in tissue slides are inherently nested. A breast tissue section belongs to an anatomical region, contains histological structures such as glands or stroma, and those structures in turn harbor object types like lymphocytes or tumor epithelium. Encoding this nesting into the training labels gives the model a form of structured supervision that flat label spaces cannot provide, and it mirrors how pathologists themselves are trained to reason about tissue, from organ context down to cellular detail.

The interpretability experiments build on a well-established tension in medical machine learning. Saliency-based explanation techniques, including class activation mapping and randomized input sampling methods, highlight pixels that influence a classifier but produce heatmaps that resist clinical interpretation. Object-level perturbation offers an alternative: by removing or altering a named structure, such as a segmented gland or an infiltrating lymphocyte population, and observing how the classifier’s output changes, the analysis yields attributions expressed in vocabulary that pathologists use daily. This aligns with arguments in the field that high-stakes decisions demand explanations grounded in human-meaningful concepts rather than pixel-level artifacts.

The connection to breast cancer classification is particularly apt given the field’s history. Large molecular studies of breast tumors, including the landmark Cancer Genome Atlas characterization, established that morphological features visible in routine histology correlate with molecular phenotypes, and subsequent work has shown that image-derived features can predict diverse molecular properties of tumors. A segmentation model that reliably delineates the relevant structures provides the missing bridge between such slide-level predictions and the concrete histological evidence behind them.

It is also worth situating the computational approach within the broader methodology of computational pathology. Because whole-slide images are gigapixel-scale and diagnosis often depends on subtle findings scattered across a slide, many classification pipelines rely on multiple-instance learning, in which a slide is treated as a bag of patches with only slide-level labels available. Object-level explanations from a segmentation foundation model could complement these pipelines by revealing which instances within the bag carry diagnostic weight, addressing a long-standing limitation of weakly supervised approaches.

The release of code, an archived version, and documented dataset harmonization procedures follows growing expectations for reproducibility in machine learning for healthcare, allowing independent groups to verify the reported benchmark comparisons and to extend the framework to new organs, stains, and scanning systems as annotated data continue to accumulate.

Subject of Research: Segment anything in pathology images with natural language

Article Title: Segment anything in pathology images with natural language

Article References: Chen, Z., Hou, J., Lin, L., Wang, Y., Bie, Y., Wang, X., Zhou, Y., Li, D., Tan, H., Liang, L., Chan, R. C. K., & Chen, H. (2026). Segment anything in pathology images with natural language. Nature Computational Science. https://doi.org/10.1038/s43588-026-01042-5

Image Credits: AI Generated

DOI: 10.1038/s43588-026-01042-5

Keywords: Segment, anything, pathology, images, natural, language, scientific research