A new study published in BMC Medical Imaging by Suxiang Yu, Xiaoqin Yang, Yihe Duan, Dun Hua, Bai Yun, Huimiao Sun, Lingling Zhang, Feihong Wu, Dandan Yang, and Xin Huang takes aim at exactly this problem. The team, spanning pathology departments and engineering schools in China, evaluated and adapted Cellpose-SAM, a modern generalist cell segmentation model, to the specific and stubborn challenge of instance segmentation in overlapping cervical cytology images. Their results, while modest in absolute terms, point to a practical lesson for the field: even powerful foundation models need careful, task-specific tuning before they can be trusted on the messy realities of clinical slides.

Cellpose-SAM itself represents an interesting convergence of two research threads. Cellpose, originally released in 2020, popularized a two-step approach to cell segmentation in which a neural network first predicts the flow of gradients from every pixel toward the center of its parent cell, and an algorithm then follows those flows to group pixels into individual instances. This design elegantly sidesteps the need for the network to label each cell with a unique identifier, which is notoriously difficult to train. The SAM in the name refers to the Segment Anything Model, a vision foundation model trained by Meta AI on an enormous corpus of natural images. By marrying SAM’s pretrained visual backbone to Cellpose’s flow-based decoding, Cellpose-SAM inherits strong generalization to cell types and imaging modalities it has never seen before, including microscopy styles far removed from its training distribution.

But strong generalization is not the same as optimal performance on any given task. As the authors note in the study, the default configuration of Cellpose-SAM may not be suited to densely overlapping cervical cytology images, where cell adhesion, weak cytoplasmic contrast, and ambiguous boundaries conspire to obscure individual cell structures. When two overlapping epithelial cells stain nearly identically and share a barely perceptible junction, the model must decide where one cell ends and the other begins, a judgment that even the underlying flow field may represent poorly without further training on comparable data.

To give the model that training, the researchers assembled a single-center retrospective dataset from the Fourth Central Hospital of Baoding in Hebei Province, China. The dataset consisted of 433 images of overlapping cervical cells, each accompanied by expert manual annotations that served as the ground truth against which the algorithm’s outputs were measured. These images were divided into training, validation, and independent test sets at a ratio of 70 percent, 10 percent, and 20 percent, respectively. The split is a standard but important design choice: the training set teaches the model, the validation set guides tuning decisions, and the test set, held completely apart from both, provides an honest estimate of how the adapted system performs on data it has never encountered during development.

The adaptation strategy had two complementary components. The first was fine-tuning, in which the pretrained Cellpose-SAM weights were updated on the cervical cytology training images, allowing the model’s internal representations to adjust to the specific staining patterns, cell morphologies, and clutter of Pap-smear preparations. The second was inference-parameter optimization, a step that is often overlooked but can be as consequential as retraining itself. Cellpose-SAM exposes several knobs at prediction time: the expected cell diameter, which calibrates the scale at which the network looks for objects; the flow threshold, which determines how strictly the gradient-following algorithm trusts its flow predictions when assembling cell instances; and the cell probability threshold, which decides how readily the model declares a pixel to belong to a cell at all. Rather than accepting defaults, the team systematically tuned these parameters on the validation set, using it as a proxy for the true target distribution and then freezing the chosen values for evaluation.

Performance was assessed on the independent test set using four established segmentation metrics. The Dice coefficient, which measures the harmonic mean of overlap between predicted and annotated regions, reached 0.8737 for the adapted model. Intersection over Union, a stricter spatial-overlap measure, came in at 0.7199. The Aggregated Jaccard Index, a metric designed specifically for instance-level segmentation because it penalizes both missed and spurious instances across a whole image, registered 0.7024. The error rate, which captures instances the model failed to segment correctly, was 0.1551. The authors performed statistical comparisons between the default and adapted configurations using paired image-level analysis, and the adapted model showed improved performance across these measures, suggesting better agreement with the expert manual annotations in the overlapping-cell regions.

The numbers deserve careful interpretation. A Dice coefficient of roughly 0.87 indicates substantial pixel-level agreement, and an IoU above 0.72 is respectable for a task where the ground truth itself is contested by virtue of ambiguous cell boundaries. The AJI of about 0.70, meanwhile, is the more sobering figure, because it aggregates performance at the level of whole instances: every cell the model misses entirely or invents from nothing drags the score down across the entire image. An error rate of about 15.5 percent means that roughly one in six or seven predicted instances does not match the annotation well. In other words, the adapted Cellpose-SAM is a capable assistant rather than a finished autonomous cytologist, and the gap between 0.87 pixel overlap and 0.70 instance accuracy quantifies precisely the residual difficulty of separating cells that touch and overlap.

What makes the study noteworthy methodologically is its insistence that off-the-shelf foundation models are starting points, not endpoints. The cervical cytology community has seen a wave of deep-learning screening tools, many of which treat segmentation as a solved problem and focus instead on classification of abnormalities. Yet if the segmentation stage silently merges overlapping cells or slices one cell into two, every downstream measurement, from nucleus-to-cytoplasm ratio to nuclear area, inherits that distortion. The Baoding team’s finding that simply tuning the cell diameter, flow threshold, and cell probability threshold on a small validation set yielded measurable gains over defaults suggests that laboratories adopting Cellpose-SAM or similar tools should budget time and annotation effort for exactly this kind of calibration, tailored to their staining protocols and scanner optics.

The authors are appropriately measured about the limits of their work. The dataset comprised 433 images from a single center, which raises questions about how well the adapted model would transfer to slides prepared with different stains, scanned on different instruments, or drawn from patient populations with different distributions of inflammation, atrophy, and glandular cells. They state explicitly that further external validation and broader benchmarking are required before clinical generalization can be established. The retrospective study was approved by the hospital’s ethics committee under approval number 2022011, conducted in accordance with the Declaration of Helsinki, and used de-identified, anonymized images, with the informed-consent requirement waived for the retrospective design. The work was funded by the S&T Program of Hebei under project number 22377774D, and the authors declared no competing interests.

Still, the trajectory is easy to read. Foundation models pretrained on millions of natural images are rapidly becoming the default starting point for biomedical image analysis, and the interesting research questions are shifting from how to build such models to how to adapt them efficiently to narrow, high-stakes domains. Cervical screening is a particularly attractive target: it is a globally distributed task with a persistent shortage of trained cytologists, and the cost of a missed abnormality is measured in human lives. If modest fine-tuning and parameter tuning can push a generalist segmentation model to useful accuracy on the notoriously tangled cells of a Pap smear, the path from laboratory benchmark to screening-room assistance becomes considerably more concrete, and the 433 clumped, ambiguous, stubbornly overlapping cells of Baoding become a small but meaningful waypoint on that road.

Subject of Research: Instance segmentation of overlapping cervical cells in Pap-smear images using adapted Cellpose-SAM

Article Title: Adapting Cellpose-SAM for instance segmentation of overlapping cervical cells in Pap-smear images

Article References: Yu, S., Yang, X., Duan, Y., Hua, D., Yun, B., Sun, H., Zhang, L., Wu, F., Yang, D., & Huang, X. (2026). Adapting Cellpose-SAM for instance segmentation of overlapping cervical cells in Pap-smear images. BMC Medical Imaging. https://doi.org/10.1186/s12880-026-02845-8

Image Credits: AI Generated

DOI: 10.1186/s12880-026-02845-8

Keywords: Cellpose-SAM, cervical cytology, Pap smear, instance segmentation, deep learning, fine-tuning, medical imaging, computer vision, cervical cancer screening, parameter adaptation, BMC Medical Imaging, Adapting

The study, published as an open-access research article in Virology Journal, is notable for combining HPV DNA testing with concurrent cervical cytology in a single, large cohort. Women undergoing routine health examinations at participating facilities provided cervical samples for HPV genotyping and ThinPrep cytologic testing. The cohort had a median age of 42 years, with an interquartile range of 36 to 51 years, placing most participants squarely within the age band where cervical cancer screening is most actively recommended. By pairing molecular virology with cytological classification, the researchers were able to map how individual viral genotypes distribute themselves across the spectrum of cervical abnormalities, from benign findings to high-grade lesions.

HPV is a double-stranded DNA virus with more than 200 known genotypes, of which roughly a dozen, designated high-risk types, are established causes of cervical cancer. Persistent infection with HPV16 and HPV18 accounts for the majority of cervical cancer cases worldwide, which is why these two genotypes anchor the bivalent and quadrivalent vaccines, while the nonavalent formulation extends coverage to HPV31, 33, 45, 52 and 58. The Shanxi data confirm that the regional genotype landscape does not perfectly mirror the global one. HPV52 was the single most prevalent high-risk genotype at 2.3 percent, ahead of HPV16 and HPV58, each at 1.7 percent. Combined prevalence of HPV16 and HPV18 was 2.3 percent, while the nine genotypes covered by the nonavalent vaccine collectively accounted for 7.4 percent of infections, meaning roughly two-thirds of high-risk infections in this population fall within the nonavalent vaccine’s protective umbrella.

Beyond genotype frequencies, the researchers used multivariable logistic regression to isolate risk factors associated with high-risk HPV infection. Age emerged as a strong and independent determinant. Compared with women aged 20 to 35 years, those aged 56 to 65 years had roughly twice the odds of carrying a high-risk infection, with an adjusted odds ratio of 2.0 and a 95 percent confidence interval of 1.6 to 2.3. This pattern is consistent with a recognized second peak of HPV prevalence among older women, often attributed to reactivation of previously acquired latent infections, declining immune surveillance after menopause, or cohort effects related to sexual behavior and screening history in older generations. Marital status also carried a signal: married women had about half the odds of high-risk HPV infection compared with unmarried women, an adjusted odds ratio of 0.5 with a confidence interval of 0.4 to 0.7, a finding the authors report without overinterpreting its behavioral or social underpinnings.

Cytological abnormalities were far less common than viral infections overall. Abnormal cytology was detected in 2.1 percent of the 19,384 participants, spanning the full range of the Bethesda classification system, from atypical squamous cells of undetermined significance through low-grade and high-grade squamous intraepithelial lesions. The relatively low rate of abnormal cytology reflects the natural history of HPV disease: most infections are transient and cleared by cell-mediated immunity within one to two years, and only a small fraction of persistent infections progress through cytological change toward malignancy. This biological bottleneck is precisely what makes genotype-specific surveillance valuable, because identifying which viral types are enriched among abnormal cytology results sharpens the predictive value of screening.

The most striking analytical finding concerns how genotype composition shifts with lesion severity. Among HPV-positive women, the prevalence of HPV16 and HPV18 rose significantly as cytological grade increased, with a 3.4-fold higher odds of these two genotypes in atypical squamous cells cannot exclude high-grade lesion or high-grade squamous intraepithelial lesion categories compared with low-grade lesions, at an odds ratio of 3.4 and a confidence interval of 1.0 to 11.5. The genotypes HPV31, 33, 45, 52 and 58 showed a moderate but statistically non-significant increase across severity strata, with an odds ratio of 1.8 and a confidence interval spanning 0.6 to 5.6. Meanwhile, high-risk genotypes outside this seven-type group actually declined in prevalence as lesions became more severe, with an odds ratio of 0.6 and a confidence interval of 0.1 to 2.2. In other words, the viral population underlying mild abnormalities is more heterogeneous, whereas severe lesions concentrate around the genotypes with established oncogenic potency.

From a virological standpoint, this gradient makes mechanistic sense. HPV16, and to a lesser extent HPV18, carry sequence variations in the E6 and E7 oncogenes and in regulatory regions that favor expression of proteins which degrade the p53 and retinoblastoma tumor suppressor pathways. Viruses equipped with these features are more likely to escape cell-cycle control long enough for a lesion to progress, which is why they dominate the high-grade end of the spectrum even when other genotypes are more common among transient infections. HPV52 and HPV58, both classified as high-risk and both notably prevalent in East Asian populations, appear to occupy an intermediate position, contributing substantially to infection burden and to lesions, but with a weaker per-infection association with severe cytology than HPV16/18 in this dataset.

The timing of these findings amplifies their policy relevance. China’s decision to introduce HPV vaccination into its national immunization programme in November 2025 created an urgent need for baseline genotype distribution data across representative provinces. Without such baselines, it would be difficult to distinguish vaccine-driven changes in genotype prevalence from natural fluctuations. The Shanxi results suggest that bivalent and quadrivalent vaccination, which target HPV16 and HPV18 plus HPV6 and 11 in the quadrivalent formulation, will directly address only about one-fifth of the high-risk infections observed here, while nonavalent vaccination would cover roughly two-thirds. At the same time, the significant enrichment of HPV16/18 among high-grade lesions means those two genotypes still account for a disproportionate share of the abnormalities most likely to progress toward cancer, so even narrower vaccines should deliver outsized reductions in cancer incidence.

The authors emphasize that their data support genotype-specific risk assessment as a pillar of future cervical cancer prevention in China. As vaccination scales up, screening programmes may need to adapt, potentially shifting toward HPV-based primary screening with genotyping for HPV16 and HPV18, followed by reflex cytology, a triage algorithm already adopted in many high-income settings. The observed age-related rise in prevalence also cautions against focusing vaccination and screening exclusively on younger cohorts; older women, who are largely beyond catch-up vaccination eligibility, will continue to carry a substantial burden of infection for decades and will remain dependent on effective cytological or molecular screening. The finding that married women had lower infection odds, whatever its underlying behavioral explanation, may help target health education and screening outreach toward demographic groups with higher observed risk.

As with all cross-sectional designs, the study captures a single snapshot in time and cannot establish whether infections preceded cytological changes, nor can it follow individuals to confirm clearance or progression. The participant pool, drawn from women presenting for routine health examinations, may also skew toward populations with better healthcare access than the general provincial population. Nevertheless, with nearly twenty thousand paired HPV and cytology results, the study offers an unusually granular portrait of HPV epidemiology in a region of China where such data had been limited. As the national vaccination programme matures, repeated iterations of this kind of surveillance in Shanxi and comparable provinces will be essential for verifying whether the vaccine is bending the genotype distribution curve, reducing cytological abnormalities, and ultimately accelerating China’s progress toward the World Health Organization’s cervical cancer elimination targets.

Subject of Research: Prevalence, genotype distribution and risk factors of human papillomavirus infection and cervical cytological abnormalities among women in Shanxi, China

Article Title: Prevalence and risk factors of human papillomavirus infection and cervical cytological lesions among women in Shanxi, China: a cross-sectional study

Article References: Ge, S., Gao, H., Wang, J., Guo, S., Chang, Y., Qi, Y., Han, J., Jia, Y., Dong, J., & Wei, F. (2026). Prevalence and risk factors of human papillomavirus infection and cervical cytological lesions among women in Shanxi, China: a cross-sectional study. Virology Journal. https://doi.org/10.1186/s12985-026-03300-4

Image Credits: AI Generated

DOI: 10.1186/s12985-026-03300-4

Keywords: human papillomavirus, HPV prevalence, HPV52, HPV16, HPV58, cervical cytology, cervical cancer, genotype distribution, high-risk HPV, cross-sectional study, Shanxi China, HPV vaccination