Explainable Skin Lesion Classification Uses Lightweight Multiscale Fusion, Convolutional Transformers, and SegmentAware-TreeSHAP

AI System Combines Super-Resolution and Explainable Transformers to Classify Skin Lesions

A lightweight artificial-intelligence system designed to classify skin lesions has achieved accuracy rates above 94 percent on two major benchmark datasets while producing visual explanations intended to show dermatologists which parts of a lesion influenced its decision. The framework, developed by Madhusmita Priyadarshini Sahoo and Rajeswari Sridhar, combines image enhancement, multiscale feature extraction and an explainability method tailored to the structure of skin lesions. The researchers say the approach could help address two persistent obstacles in automated dermatology: the difficulty of interpreting low-quality clinical images and the “black box” nature of many deep-learning systems. The study reports overall accuracies of 94.61 percent on ISIC 2017 and 94.93 percent on ISIC 2019, as well as 84.86 percent on PAD-UFES-20, a dataset containing clinical images captured with smartphones. The results suggest that compact AI models may be capable of supporting rapid analysis across images collected under substantially different conditions, although the system remains a research tool rather than a replacement for clinical diagnosis.

Skin-lesion classification is a particularly demanding problem for computer vision. Dermoscopic and clinical images can vary in lighting, focus, colour balance, magnification and background. Lesions themselves may differ widely in size, shape, pigmentation and texture, while hair, ruler markings, air bubbles and other imaging artefacts can obscure diagnostically relevant patterns. A model trained primarily on high-quality dermoscopic images may also perform less reliably when presented with smartphone photographs or images from a different population. Deep neural networks can learn subtle visual associations that are difficult for humans to specify in advance, but their predictions are often hard to justify. In medicine, a high score alone is not sufficient: clinicians need to know whether a model is responding to the lesion’s border, pigment network and internal structure, or to irrelevant features such as a dark background or an imaging marker. The new framework was designed around this combination of performance and interpretability.

The first stage uses a hybrid super-resolution preprocessing strategy based on two methods named EdgeSR-MAX and EdgeSR-TM. Super-resolution algorithms attempt to reconstruct a higher-resolution representation from an image with limited spatial detail. They do not recover information that was physically recorded but lost; instead, they infer plausible fine-scale structure from patterns learned during training. For lesion analysis, this can be useful when boundaries and texture are blurred or occupy only a small number of pixels. The researchers’ approach focuses particularly on edges and structural details, features that can influence the apparent asymmetry, border irregularity and internal organization of a lesion. Enhancing those features before classification may give later stages a more stable representation of the image. It also introduces a critical technical consideration: reconstructed details must not be mistaken for genuine biological structures. In any clinical deployment, super-resolution would therefore need to be validated carefully to ensure that it improves recognition without creating misleading visual patterns.

After enhancement, the images are processed by the system’s central classifier, called the Multiscale Feature Fusion Convolutional Transformer, or MFCT. The architecture combines the strengths of convolutional neural networks and vision transformers. Convolutional layers are effective at detecting local patterns, such as edges, small texture changes and compact colour transitions. Transformers, by contrast, use attention mechanisms to model relationships between distant regions of an image. That global context can help a model assess how a lesion’s border relates to its centre, or whether multiple visual features form a coherent pattern rather than isolated marks. The MFCT uses parallel convolutional paths with kernels measuring 3 by 3, 5 by 5 and 7 by 7 pixels. These different receptive-field sizes allow the network to examine fine details and broader structures simultaneously. Their outputs are fused before being passed to a transformer encoder, which models interactions among the resulting features.

This multiscale design is intended to avoid a common trade-off in medical-image analysis. A network focused only on small receptive fields may capture texture while missing the overall lesion geometry. A network using only broad receptive fields may recognize global shape but overlook small regions of pigment or subtle surface irregularity. By processing multiple spatial scales in parallel, the MFCT can retain information from both levels and combine them into a more comprehensive feature representation. The transformer component then applies attention-based global context modelling rather than relying solely on sequential stacks of local convolutions. The researchers also emphasize that the model is lightweight, with reduced complexity and low inference latency compared with larger architectures. That distinction matters for practical use: a system that requires powerful hospital servers may be unsuitable for smaller clinics, mobile devices or point-of-care screening, whereas a compact network could potentially run in resource-constrained environments.

The final component, called SegmentAware-TreeSHAP, is intended to make the model’s reasoning more transparent. Conventional heat maps often assign importance to individual pixels, highlighting areas that appear influential but may be fragmented or difficult to interpret clinically. TreeSHAP, derived from Shapley-value methods in game theory, estimates how much individual features contribute to a prediction by comparing the model’s output with and without those features. In the new system, the attribution process is made segment-aware: instead of treating every pixel as an isolated unit, the method groups image regions into segments that correspond more closely to meaningful lesion structures. The resulting maps are designed to indicate whether the prediction was driven by the lesion’s border, central area or other coherent regions. This approach can make explanations easier to inspect because it connects model attribution to anatomical and morphological organization. It may also reduce the visual noise that can arise when pixel-level explanations are overlaid on complex dermoscopic images.

The researchers evaluated the complete framework on ISIC 2017, ISIC 2019 and PAD-UFES-20, three datasets that represent different imaging conditions and classification challenges. The first two are widely used resources for skin-lesion analysis, while PAD-UFES-20 includes patient information and clinical photographs collected using smartphones. The model achieved overall accuracies of 94.61 percent, 94.93 percent and 84.86 percent on the three datasets, respectively. The lower result on PAD-UFES-20 may reflect the greater variability of ordinary clinical images compared with standardized dermoscopic datasets, as well as differences in class distribution and image quality. The study also reports competitive sensitivity, specificity and F1-scores, measures that capture different aspects of performance. Sensitivity reflects the ability to identify relevant positive cases, specificity measures the rejection of negative cases, and the F1-score balances precision and recall. Together, these metrics provide a more informative assessment than accuracy alone, particularly when disease categories are unevenly represented.

The most important promise of the system may lie not in its headline accuracy but in the combination of speed and explanation. A fast classifier could help prioritize images for specialist review, support preliminary assessment in settings with limited access to dermatologists or provide a second opinion during a consultation. Segment-level attribution maps could allow a clinician to compare the model’s focus with the features they themselves consider important. If the AI highlights an irrelevant artefact, its prediction may be treated cautiously; if it consistently emphasizes clinically meaningful regions, confidence in its use could grow. Yet these explanations should not be confused with proof that the model has reasoned like a dermatologist. An attribution map describes which image regions were associated with the output, not whether those regions reflect causally valid pathology. The system’s performance also comes from benchmark testing, and benchmark results do not automatically establish reliability across hospitals, cameras, skin tones, age groups or rare lesion types.

Before such a framework could influence patient care, it would require prospective testing on independent datasets and carefully designed clinical trials. Researchers would need to examine calibration, false-negative rates and performance across demographic groups, as well as the consequences of image-quality failures. External validation is especially important for systems that combine super-resolution with classification because enhancement can alter the visual signal presented to the network. Human studies could test whether SegmentAware-TreeSHAP explanations genuinely improve diagnostic accuracy, reduce overreliance on automation or simply make incorrect predictions appear more convincing. The authors acknowledge the practical value of explainability as a bridge between high-performing algorithms and clinical trust, while their results indicate that the lightweight MFCT can generalize across heterogeneous data. The work points toward a future in which skin-imaging AI is not merely asked whether a lesion looks suspicious, but is also expected to show the structures behind its answer and operate quickly enough to be useful where medical resources are limited.