AI Faces a Hard Test: Detecting Rare Martensite in Steel Micrographs

Deep learning has transformed how scientists classify materials, but a new study shows just how difficult the task becomes when the most important feature is also the rarest. In research published in the Journal of Materials Science: Metallurgy, Ehsun Saeed evaluated whether a transfer-learning framework could reliably detect Martensite, a hard and brittle phase, in ultra-high carbon steel micrographs when that phase accounted for a mere 3.75 percent of the dataset. The work was conceived as a baseline diagnostic rather than a claim of industrial readiness, and its findings carry a candid message about the limits of artificial intelligence in metallurgical quality assurance.

The dataset consisted of 961 scanning electron microscopy micrographs spanning seven microstructural classes, drawn from the Ultra-High Carbon Steel micrograph collection maintained by the National Institute of Standards and Technology. Spheroidite dominated with 374 images, while Martensite was represented by only 36. This tenfold disparity in representation creates exactly the kind of class imbalance that undermines conventional machine learning, where optimisation naturally favours majority classes and can produce seemingly respectable accuracy while missing the minority class entirely.

Saeed adopted a frozen ResNet50 transfer-learning architecture, using ImageNet pre-trained weights as a fixed feature extractor and retraining only the newly added classification layers. Images were resized to 224 by 224 pixels, normalised, and augmented online with random rotations, flips, and shifts during training. A stratified train-test split of 80 to 20 preserved class proportions, leaving just seven Martensite images in the test partition. Baseline training used the Adam optimiser with categorical cross-entropy loss for 30 epochs, and two reference classifiers, a random predictor and a majority-class predictor, served as lower bounds.

The baseline model achieved a peak validation accuracy of 52.8 percent, well above the random baseline of 14.3 percent and the majority-class baseline of 38.9 percent. Martensite detection, however, remained modest, with precision of 0.600, recall of 0.429, and an F1-score of 0.500. Because overall accuracy can mask poor minority-class recognition, the study also reported balanced accuracy, macro-averaged metrics, and the Matthews Correlation Coefficient, all of which provide a more honest picture under severe imbalance.

To probe robustness, stratified five-fold cross-validation was performed, yielding an average accuracy of 0.412, a macro F1-score of 0.133 plus or minus 0.009, and an MCC of 0.134 plus or minus 0.029. These figures reveal weak multi-class generalisation and suggest that the principal challenge is not instability across partitions but genuine class overlap and insufficient minority-class representation. The gap between the single split and the cross-validated results further indicates that performance estimates are sensitive to dataset composition.

Two imbalance-aware strategies were then tested. Class-weighted loss, with weights inversely proportional to class frequencies, boosted Martensite recall dramatically to 0.857 but crashed overall accuracy to 29.0 percent, illustrating a flood of false positives and an overcompensation for the rarity of the minority class. Focal loss, which down-weights easily classified majority samples, fared better, delivering the strongest overall performance with 56.0 percent accuracy, 36.5 percent balanced accuracy, an MCC of 0.388, and improved Martensite precision of 0.750 while holding recall steady. The results show that loss-function design can shift the balance between sensitivity and precision, but cannot conjure data that does not exist.

A binary Martensite-versus-non-Martensite experiment offered further insight, achieving an AUROC of 0.908 and an MCC of 0.527, substantially stronger than the multi-class formulation. Threshold optimisation raised recall from 28.6 percent to 57.1 percent while preserving precision at 80.0 percent. This suggests that ambiguity among seven overlapping classes compounds the difficulty, and that application-specific threshold tuning can extract meaningful rare-phase discrimination from the same underlying model.

Interpretability and confounding analyses rounded out the study. Grad-CAM visualisations showed that activation maps concentrated on microstructural regions rather than on scale bars or image borders, and border-crop robustness tests confirmed that removing up to 15 percent of the image perimeter left performance largely unchanged. t-SNE projection of the penultimate-layer features, however, revealed that Martensite samples were dispersed throughout the feature space with no distinct cluster, overlapping heavily with Pearlite, Spheroidite, and Network microstructures. A magnification audit exposed substantial variation in imaging scale across classes, from a mean of 89 times for Network to 13,441 times for Pearlite, and magnification-only classification with a Random Forest achieved an AUROC of 0.810, indicating that imaging scale acts as a partial confounder without fully explaining model behaviour.

Handcrafted texture descriptors, long the workhorses of metallographic image analysis, were evaluated for comparison. Features derived from Gray-Level Co-occurrence Matrices and Local Binary Patterns, classified with Support Vector Machine and Random Forest models under stratified cross-validation, achieved performance comparable to the CNN, suggesting that informative texture signals exist in the data and that physically interpretable baselines remain valuable benchmarks in data-scarce settings.

The study positions itself as a reproducible baseline rather than a deployable solution, and the implications are clear. Reliable rare-phase detection in industrial metallography will require larger minority-class datasets, scale-normalised imaging protocols, quantitative microstructural descriptors fused with learned features, and specialised strategies such as few-shot learning, anomaly detection, and metric learning. Until then, the author argues, such models are best regarded as decision-support tools subject to expert review, uncertainty-aware thresholds, and external validation across instruments, compositions, and processing conditions, rather than as autonomous replacements for the trained metallurgist’s eye.

Subject of Research: Rare-phase detection of Martensite in ultra-high carbon steel micrographs using deep learning and handcrafted texture features under severe class imbalance

Article Title: A baseline study of rare-phase detection in ultra-high carbon steel microstructures using deep learning and handcrafted texture features

Article References: Saeed, E. (2026). A baseline study of rare-phase detection in ultra-high carbon steel microstructures using deep learning and handcrafted texture features. Journal of Materials Science: Metallurgy, 1(1), Article 12. https://doi.org/10.1007/s44492-026-00012-2

Image Credits: AI Generated

DOI: 10.1007/s44492-026-00012-2

Keywords: ultra-high carbon steel, Martensite, deep learning, ResNet50, transfer learning, class imbalance, microstructure classification, focal loss, Grad-CAM, t-SNE, GLCM, rare-phase detection