Hybrid AI Fuses Deep and Handcrafted Features for Near-Perfect Cervical Cancer Screening

Cervical cancer remains one of the most common cancers affecting women worldwide, and despite being preventable when caught early, it continues to exact a heavy toll, particularly in low- and middle-income countries where regular screening is difficult to access. The frontline screening tool, the Papanicolaou (Pap) smear test, relies on the microscopic examination of cervical cells to identify precancerous and cancerous abnormalities. But manual analysis of these smears is time-consuming, labor-intensive and vulnerable to interobserver variability, meaning different experts can reach different conclusions from the same slide. Now, a new study published in Discover Artificial Intelligence reports that a hybrid machine learning framework, which fuses learned and handcrafted image features, achieved near-perfect classification of Pap smear images, reaching approximately 99.85% test accuracy and a 99.96% F1-score for cancer-class classification using a random forest classifier.

The research, conducted by Nourhan Zayed and Heba A. Elnemr, addresses a persistent dilemma in automated cytology. Deep learning models, particularly convolutional neural networks, have shown remarkable power in medical image analysis by learning hierarchical representations directly from pixels. Yet these models typically demand extensive annotated datasets and considerable computational resources, and their ‘black box’ nature makes it hard for clinicians to understand why a diagnosis was reached. Classical approaches built on handcrafted features, such as texture descriptors, are more transparent and computationally light, but they have historically been less robust to challenges like overlapping cells, staining variability and class imbalance. The new framework attempts to capture the best of both worlds.

At the heart of the study is the publicly available Herlev Pap smear dataset, a collection of 917 annotated single-cell images distributed across seven diagnostic categories, ranging from normal cell types to mild, moderate and severe dysplasia and carcinoma in situ. Each image in the dataset was labeled by two cytotechnicians and a physician. Before analysis, the images underwent preprocessing: histogram equalization enhanced contrast and reduced noise from uneven lighting, normalization mapped intensity values to a consistent range, and resizing ensured uniform inputs for the deep learning components.

The proposed pipeline is deliberately segmentation-free, avoiding an early step that has long been an error-prone bottleneck. Instead of first trying to isolate cell nuclei from the cytoplasm, a task complicated by cell overlap and staining artifacts, the framework extracts three distinct families of features directly from the images. The first branch computes handcrafted texture descriptors, including 14 Haralick features derived from the Gray-Level Co-occurrence Matrix, which quantify spatial relationships between pixel intensities, alongside two-level wavelet packet decomposition using the Daubechies-3 wavelet to capture multi-scale frequency information, yielding 68 features in total.

The second branch relies on a convolutional autoencoder, an unsupervised neural network trained to compress images into a compact 64-dimensional latent representation and then reconstruct them, minimizing pixel-wise reconstruction error with the Adam optimizer and L2 regularization. Because it learns without class labels, the autoencoder captures structural and morphological information on its own. The third branch applies transfer learning with VGG16, a convolutional network pretrained on ImageNet, used as a fixed feature extractor whose fully connected layers were removed, producing 25,088 high-level deep features per image without any fine-tuning on the relatively small Herlev set.

Feature selection and dimensionality reduction then sharpen each branch. For texture features, analysis of variance (ANOVA) identified statistically significant descriptors, removing those with p-values greater than 0.0005 and cutting the set from 68 or 69 features down to as few as 25. For the high-dimensional deep features, standardization was followed by Principal Component Analysis (PCA), which compressed the data to components capturing roughly 95% of total variance, and then Linear Discriminant Analysis (LDA), a supervised technique that projects the data onto axes maximizing class separability. Finally, the complementary features were concatenated into hybrid vectors and fed to six classifiers: logistic regression, k-nearest neighbors (KNN), support vector machines with linear and RBF kernels, decision trees and random forests.

The performance results, validated with stratified five-fold cross-validation in which feature selection and reduction were applied within each fold to prevent information leakage, show a striking hierarchy. Autoencoder-based features achieved near-perfect binary classification of normal versus abnormal cells across all classifiers, with metrics averaging around 100% and ROC area-under-curve values near 1.0. VGG deep features reached approximately 99%, and tree-based classifiers using VGG representations hit essentially 100% accuracy when distinguishing normal cell categories. Texture features alone lagged behind, with KNN managing only 92.15% test accuracy and 83% balanced accuracy on the binary task, though ANOVA-selected significant texture features lifted results to roughly 96-98%.

In the more demanding multi-class tasks, the value of fusion became even clearer. For classifying the four cancer categories, hybrid combinations of autoencoder and significant texture features exceeded 98.5% accuracy, surpassing 99% with random forest and decision tree classifiers, while texture features alone fell to around 52% with KNN. In the full seven-class problem, logistic regression, linear SVM, decision tree and random forest achieved 100% balanced accuracy using autoencoder and hybrid features. Crucially, the authors applied McNemar’s test to pooled predictions across all five folds, confirming that the hybrid model’s improvement over handcrafted features was statistically significant for the binary task, and its lead over VGG16 deep features was significant for multi-class classification (p = 0.042), particularly in reducing confusion between adjacent dysplasia stages.

The authors are candid about the study’s limitations. The Herlev dataset consists of high-quality, pre-segmented single-cell images, a level of separability that raw clinical smears, plagued by overlapping cells, staining artifacts and low-contrast imaging, may not replicate. The dataset is also imbalanced, which is why the team emphasized precision, recall, F1-score, balanced accuracy and normalized confusion matrices rather than overall accuracy alone. Their analysis showed that residual errors clustered between neighboring dysplasia grades and visually similar normal categories, mirroring the subtle morphological transitions that challenge even human experts. The researchers explicitly acknowledge that larger, multi-center external validation on more diverse datasets, such as Mendeley LBC or SIPaKMeD, is required before any clinical deployment.

Even with those caveats, the implications are significant. The hybrid framework offers a transparent and computationally efficient alternative to fully end-to-end deep learning pipelines: after feature reduction, the final classification stage requires far fewer trainable parameters, and the handcrafted component injects clinically meaningful, interpretable descriptors, such as texture uniformity and chromatin-related patterns, that medical experts can actually inspect. The authors suggest that carefully selected handcrafted features may capture most of the discriminative information in some scenarios, potentially reducing reliance on computationally expensive models in low-resource screening settings. Future work, they write, will integrate explainable AI techniques such as SHAP and LIME, explore fully segmentation-free whole-slide image analysis, and pursue cross-dataset evaluation with domain adaptation to account for variations in staining, imaging equipment and population characteristics. If those steps succeed, this fusion of old and new machine learning could bring faster, more consistent and more trustworthy cervical cancer screening closer to the clinic.

Subject of Research: Hybrid deep and handcrafted feature fusion for machine learning-based cervical cancer diagnosis from Pap smear images

Article Title: Synergistic fusion of deep and handcrafted features for accurate and robust machine learning based cervical cancer diagnosis

Article References: Zayed, N., & Elnemr, H. A. (2026). Synergistic fusion of deep and handcrafted features for accurate and robust machine learning based cervical cancer diagnosis. Discover Artificial Intelligence, 6(1), Article 1612. https://doi.org/10.1007/s44163-026-01986-7

Image Credits: AI Generated

DOI: 10.1007/s44163-026-01986-7

Keywords: cervical cancer, Pap smear, machine learning, deep learning, autoencoder, texture features, VGG16, Herlev dataset, computer-aided diagnosis, medical imaging, feature fusion, cross-validation