Head-to-Head Test Shows Deep Learning Beats Classic Models Only on Breast Cancer Images

Breast cancer remains one of the leading causes of cancer-related death worldwide, and even the most experienced radiologists and pathologists can disagree when images are ambiguous or subtle. A new independent evaluation published in PLOS Digital Health has put a wide range of artificial intelligence models through a systematic, side-by-side comparison to determine which algorithms actually perform best for detecting breast cancer, and under what conditions. The study, led by Deepthi, Ashalatha Nayak, and Rani Oomman Panicker, offers one of the more candid assessments to date of how machine learning and deep learning stack up when the data type changes, and its findings carry a sober message for clinics hoping to deploy AI diagnostic tools: the best model depends entirely on the kind of data you feed it.

The research team set out to address a persistent gap in the medical AI literature. Many published studies report impressive accuracy figures for a single algorithm on a single dataset, but few attempt a controlled comparison across both structured clinical data, such as patient records and tumor measurements, and raw medical images, such as mammograms and tissue slides. Because inconclusive imaging results and variability between human observers remain real obstacles to early and accurate detection, the authors argue that complementary computational approaches are needed, but only if those approaches are evaluated honestly and with appropriate statistical rigor. Their study was designed to provide exactly that kind of evaluation, applying multiple algorithm families to multiple data modalities and reporting all performance metrics with 95 percent confidence intervals.

On the structured data side, the researchers drew on three well-known sources: the Wisconsin Breast Cancer Dataset, the CSAW-CC dataset, and the SEER database, which aggregates cancer registry information across large patient populations. Against these tabular datasets they pitted five classical machine learning approaches: Random Forest, Decision Tree, Logistic Regression, Gradient Boosting, and Bayesian classifiers. These algorithms, many of which predate the deep learning revolution by decades, remain workhorses in clinical prediction because they are fast, interpretable to varying degrees, and often surprisingly effective when the input is a curated set of numerical features rather than raw pixels. The question was whether their performance would hold up under a rigorous, confidence-interval-based evaluation.

For imaging data, the team turned to deep learning, specifically convolutional neural networks and several of the most influential architectures of the past decade: ResNet50, VGG16, and DenseNet. These models were applied across three imaging modalities relevant to breast cancer diagnosis: mammography, histopathology, and ultrasound. Transfer learning, in which a network pretrained on a large general image corpus is fine-tuned on medical data, was a central strategy, since medical imaging datasets are typically too small to train deep architectures from scratch without severe overfitting. The inclusion of a from-scratch CNN baseline allowed the researchers to quantify exactly how much benefit the pretrained architectures provide.

The headline result is a clean division of labor. Deep learning models decisively outperformed classical machine learning on image classification tasks, achieving an accuracy of 97.88 percent on histopathological images and 92.00 percent on ultrasound data. This is consistent with the theoretical expectation that convolutional networks excel when the discriminative information is embedded in spatial patterns, such as the architecture of tissue in a biopsy slide or the texture of a lesion on an ultrasound scan, features that classical algorithms cannot easily extract from raw image data without extensive manual feature engineering.

On structured clinical data, however, the picture reversed. Ensemble-based and probabilistic machine learning models achieved predictive accuracy of up to 98 percent, matching or exceeding what the deep networks managed on images. Random Forest and Gradient Boosting, which combine many weaker learners into a single robust predictor, proved particularly well suited to tabular clinical variables. The practical implication is significant: hospitals and screening programs working with registry data, lab results, or demographic risk factors do not necessarily need heavyweight deep learning infrastructure to achieve state-of-the-art prediction, and may in fact be better served by lighter, more interpretable models.

Perhaps the most technically interesting portion of the study goes beyond raw accuracy to examine model calibration, a property that is frequently ignored in AI benchmarks but is critical in clinical practice. A calibrated model is one whose stated confidence matches its actual hit rate: when it says a tumor is malignant with 80 percent probability, it should be right roughly 80 percent of the time. The researchers assessed calibration using the Expected Calibration Error, a numerical measure of the gap between predicted confidence and observed accuracy, together with reliability diagrams that visualize this relationship. An accurate but poorly calibrated model can mislead clinicians by expressing unwarranted certainty, or conversely by hedging on cases it should confidently flag.

The calibration results revealed a second layer of differentiation among the algorithms. Among the structured-data classifiers, Logistic Regression, Gradient Boosting, and Random Forest produced the best-calibrated probability estimates, while Decision Tree and Naive Bayes calibrated comparatively poorly, a known weakness of single decision trees, which produce overconfident probabilities near the leaves, and of Naive Bayes, whose independence assumptions distort its likelihood estimates. On the imaging side, the transfer-learning-based deep convolutional architectures were well calibrated on the histopathology datasets, but calibration weakened noticeably for the from-scratch CNN baseline and for some deep models evaluated on the smaller BUSI ultrasound dataset. The lesson is that dataset size and training strategy matter as much as architecture: models trained on limited data tend to be miscalibrated, and this miscalibration would be invisible in a standard accuracy report.

Taken together, the findings indicate that model performance in breast cancer detection is significantly influenced by data modality, and that no single algorithm family dominates across the board. Deep convolutional networks are the right tool when the signal lives in images, particularly when transfer learning and reasonably large datasets are available, while ensemble and probabilistic machine learning methods remain the strongest and best-calibrated choice for structured clinical records. For developers of clinical decision support systems, this argues against one-size-fits-all procurement and for matching the algorithm to the data pipeline that actually exists in a given hospital. For clinicians, it suggests that AI can serve as a valuable computational tool within the diagnostic workflow, provided its outputs are interpreted with an understanding of how well calibrated they are.

The authors are careful to frame these results as a step rather than a destination. All of the evaluation was retrospective, performed on existing benchmark datasets rather than on patients in a live clinical setting, and the datasets, while diverse, cannot capture the full heterogeneity of real-world populations. The study explicitly calls for further prospective validation using clinical data across diverse patient populations to confirm the robustness, interpretability, and generalizability of the findings. Until such validation is complete, the study stands as a methodological template: report confidence intervals, measure calibration, compare across modalities, and resist the temptation to declare a single winner. In a field where inflated claims have sometimes outpaced clinical reality, that kind of disciplined evaluation may be the most important result of all.

Subject of Research: Comparative evaluation of machine learning and deep learning models for breast cancer detection using structured clinical data and medical imaging

Article Title: Independent evaluation of machine learning and deep learning models for breast cancer detection

Article References: Deepthi, Nayak, A., & Panicker, R. O. (2026). Independent evaluation of machine learning and deep learning models for breast cancer detection. PLOS Digital Health, 5(9), e0001747. https://doi.org/10.1371/journal.pdig.0001747

Image Credits: AI Generated

DOI: 10.1371/journal.pdig.0001747

Keywords: breast cancer, machine learning, deep learning, convolutional neural networks, mammography, histopathology, ultrasound, model calibration, Random Forest, transfer learning, clinical decision support, PLOS Digital Health