AI Learns to Untangle Overlapping Cervical Cells in Pap Smears

Cervical cancer screening depends on one of medicine’s most labor-intensive visual tasks: a trained cytologist or pathologist examines stained cells scraped from the cervix, hunting for the subtle nuclear and cytoplasmic changes that signal precancerous transformation. The Pap smear has saved countless lives since its introduction, but it is slow, subjective, and vulnerable to fatigue. For decades, researchers have tried to automate parts of this workflow, and for decades they have run into a deceptively simple problem: cervical cells rarely sit politely side by side on the slide. They clump, overlap, and pile on top of one another, blurring the very boundaries a computer needs to trace in order to measure and classify each cell.

A new study published in BMC Medical Imaging by Suxiang Yu, Xiaoqin Yang, Yihe Duan, Dun Hua, Bai Yun, Huimiao Sun, Lingling Zhang, Feihong Wu, Dandan Yang, and Xin Huang takes aim at exactly this problem. The team, spanning pathology departments and engineering schools in China, evaluated and adapted Cellpose-SAM, a modern generalist cell segmentation model, to the specific and stubborn challenge of instance segmentation in overlapping cervical cytology images. Their results, while modest in absolute terms, point to a practical lesson for the field: even powerful foundation models need careful, task-specific tuning before they can be trusted on the messy realities of clinical slides.

Cellpose-SAM itself represents an interesting convergence of two research threads. Cellpose, originally released in 2020, popularized a two-step approach to cell segmentation in which a neural network first predicts the flow of gradients from every pixel toward the center of its parent cell, and an algorithm then follows those flows to group pixels into individual instances. This design elegantly sidesteps the need for the network to label each cell with a unique identifier, which is notoriously difficult to train. The SAM in the name refers to the Segment Anything Model, a vision foundation model trained by Meta AI on an enormous corpus of natural images. By marrying SAM’s pretrained visual backbone to Cellpose’s flow-based decoding, Cellpose-SAM inherits strong generalization to cell types and imaging modalities it has never seen before, including microscopy styles far removed from its training distribution.

But strong generalization is not the same as optimal performance on any given task. As the authors note in the study, the default configuration of Cellpose-SAM may not be suited to densely overlapping cervical cytology images, where cell adhesion, weak cytoplasmic contrast, and ambiguous boundaries conspire to obscure individual cell structures. When two overlapping epithelial cells stain nearly identically and share a barely perceptible junction, the model must decide where one cell ends and the other begins, a judgment that even the underlying flow field may represent poorly without further training on comparable data.

To give the model that training, the researchers assembled a single-center retrospective dataset from the Fourth Central Hospital of Baoding in Hebei Province, China. The dataset consisted of 433 images of overlapping cervical cells, each accompanied by expert manual annotations that served as the ground truth against which the algorithm’s outputs were measured. These images were divided into training, validation, and independent test sets at a ratio of 70 percent, 10 percent, and 20 percent, respectively. The split is a standard but important design choice: the training set teaches the model, the validation set guides tuning decisions, and the test set, held completely apart from both, provides an honest estimate of how the adapted system performs on data it has never encountered during development.

The adaptation strategy had two complementary components. The first was fine-tuning, in which the pretrained Cellpose-SAM weights were updated on the cervical cytology training images, allowing the model’s internal representations to adjust to the specific staining patterns, cell morphologies, and clutter of Pap-smear preparations. The second was inference-parameter optimization, a step that is often overlooked but can be as consequential as retraining itself. Cellpose-SAM exposes several knobs at prediction time: the expected cell diameter, which calibrates the scale at which the network looks for objects; the flow threshold, which determines how strictly the gradient-following algorithm trusts its flow predictions when assembling cell instances; and the cell probability threshold, which decides how readily the model declares a pixel to belong to a cell at all. Rather than accepting defaults, the team systematically tuned these parameters on the validation set, using it as a proxy for the true target distribution and then freezing the chosen values for evaluation.

Performance was assessed on the independent test set using four established segmentation metrics. The Dice coefficient, which measures the harmonic mean of overlap between predicted and annotated regions, reached 0.8737 for the adapted model. Intersection over Union, a stricter spatial-overlap measure, came in at 0.7199. The Aggregated Jaccard Index, a metric designed specifically for instance-level segmentation because it penalizes both missed and spurious instances across a whole image, registered 0.7024. The error rate, which captures instances the model failed to segment correctly, was 0.1551. The authors performed statistical comparisons between the default and adapted configurations using paired image-level analysis, and the adapted model showed improved performance across these measures, suggesting better agreement with the expert manual annotations in the overlapping-cell regions.

The numbers deserve careful interpretation. A Dice coefficient of roughly 0.87 indicates substantial pixel-level agreement, and an IoU above 0.72 is respectable for a task where the ground truth itself is contested by virtue of ambiguous cell boundaries. The AJI of about 0.70, meanwhile, is the more sobering figure, because it aggregates performance at the level of whole instances: every cell the model misses entirely or invents from nothing drags the score down across the entire image. An error rate of about 15.5 percent means that roughly one in six or seven predicted instances does not match the annotation well. In other words, the adapted Cellpose-SAM is a capable assistant rather than a finished autonomous cytologist, and the gap between 0.87 pixel overlap and 0.70 instance accuracy quantifies precisely the residual difficulty of separating cells that touch and overlap.

What makes the study noteworthy methodologically is its insistence that off-the-shelf foundation models are starting points, not endpoints. The cervical cytology community has seen a wave of deep-learning screening tools, many of which treat segmentation as a solved problem and focus instead on classification of abnormalities. Yet if the segmentation stage silently merges overlapping cells or slices one cell into two, every downstream measurement, from nucleus-to-cytoplasm ratio to nuclear area, inherits that distortion. The Baoding team’s finding that simply tuning the cell diameter, flow threshold, and cell probability threshold on a small validation set yielded measurable gains over defaults suggests that laboratories adopting Cellpose-SAM or similar tools should budget time and annotation effort for exactly this kind of calibration, tailored to their staining protocols and scanner optics.

The authors are appropriately measured about the limits of their work. The dataset comprised 433 images from a single center, which raises questions about how well the adapted model would transfer to slides prepared with different stains, scanned on different instruments, or drawn from patient populations with different distributions of inflammation, atrophy, and glandular cells. They state explicitly that further external validation and broader benchmarking are required before clinical generalization can be established. The retrospective study was approved by the hospital’s ethics committee under approval number 2022011, conducted in accordance with the Declaration of Helsinki, and used de-identified, anonymized images, with the informed-consent requirement waived for the retrospective design. The work was funded by the S&T Program of Hebei under project number 22377774D, and the authors declared no competing interests.

Still, the trajectory is easy to read. Foundation models pretrained on millions of natural images are rapidly becoming the default starting point for biomedical image analysis, and the interesting research questions are shifting from how to build such models to how to adapt them efficiently to narrow, high-stakes domains. Cervical screening is a particularly attractive target: it is a globally distributed task with a persistent shortage of trained cytologists, and the cost of a missed abnormality is measured in human lives. If modest fine-tuning and parameter tuning can push a generalist segmentation model to useful accuracy on the notoriously tangled cells of a Pap smear, the path from laboratory benchmark to screening-room assistance becomes considerably more concrete, and the 433 clumped, ambiguous, stubbornly overlapping cells of Baoding become a small but meaningful waypoint on that road.

Subject of Research: Instance segmentation of overlapping cervical cells in Pap-smear images using adapted Cellpose-SAM

Article Title: Adapting Cellpose-SAM for instance segmentation of overlapping cervical cells in Pap-smear images

Article References: Yu, S., Yang, X., Duan, Y., Hua, D., Yun, B., Sun, H., Zhang, L., Wu, F., Yang, D., & Huang, X. (2026). Adapting Cellpose-SAM for instance segmentation of overlapping cervical cells in Pap-smear images. BMC Medical Imaging. https://doi.org/10.1186/s12880-026-02845-8

Image Credits: AI Generated

DOI: 10.1186/s12880-026-02845-8

Keywords: Cellpose-SAM, cervical cytology, Pap smear, instance segmentation, deep learning, fine-tuning, medical imaging, computer vision, cervical cancer screening, parameter adaptation, BMC Medical Imaging, Adapting