Mamba-Powered Adaptation Turns Segment Anything Into an Automated Medical Lesion Segmenter

When Meta AI released the Segment Anything Model (SAM), it redefined what image segmentation could look like. Trained on the SA-1B dataset—11 million high-resolution natural images annotated with 11 billion object masks—SAM can carve out almost any object in a photograph with remarkable precision. Yet the moment it confronts a dermatoscopy slide, an endoscopy frame, or a brain MRI, its performance collapses. The reason is a fundamental domain gap: natural images are colorful, well-lit, and full of high-contrast boundaries, while medical images are grayscale, low in contrast, and riddled with heterogeneity and individual variation. A research team from Tianjin University, Hebei North University, Tianjin Medical University, Fudan University, and Jilin Normal University has now built a bridge across that gap, reporting their framework, SCMamba-SAM, in Frontiers in Neurology.

The problem the researchers set out to solve is one that has frustrated the medical AI community since SAM’s debut. Lesions—whether skin melanomas, gastrointestinal polyps, thyroid nodules, intracranial hemorrhages, or focal cortical dysplasia—are notoriously amorphous. They blur into surrounding tissue, vary wildly in size and shape between patients, and often lack the crisp edges that SAM’s natural-image training rewards. Earlier evaluations, including a large-scale benchmark of 1,050,000 two-dimensional medical images with more than 6 million masks, confirmed that SAM’s zero-shot transfer to medical tasks lags behind specialized state-of-the-art methods, and that its reliance on point, box, text, or mask prompts introduces instability and sensitivity that clinicians cannot tolerate in practice.

SCMamba-SAM attacks the problem on three fronts. First, it keeps SAM’s powerful pre-trained ViT-B image encoder—twelve layers of multi-head self-attention—but inserts Low-Rank Adaptation (LoRA) layers into each self-attention block. LoRA freezes the original Query, Key, and Value weight matrices and learns only small low-rank updates to the Query and Value projections through two trainable linear layers with a reduced projection dimension. This parameter-efficient strategy lets the encoder quietly re-tune itself to the statistics of medical imagery without the cost and overfitting risk of full fine-tuning, which would require updating all of SAM’s parameters on scarce labeled data.

Second, the team bolted on a novel module they call the Encoding Mamba Adapter (EMA). Mamba, a recent architecture built on structured state space models, can model long-range dependencies across a sequence with approximately linear complexity in sequence length—an advantage over the quadratic cost of Transformer self-attention when data are limited and overfitting looms. The EMA is a serial pair of Spatial-Mamba and channel-Mamba blocks: the first captures long-range interdependencies among spatial tokens, while the second, transposed so that channels become the sequence, builds global relationships among feature channels. Each block consists of a squeeze-MLP that compresses the input, a Mamba layer that propagates information through a selective state space, and an expand-MLP that restores the original dimensions. Notably, the authors replaced every linear layer in the original Mamba design with multi-layer perceptrons, a modification their ablation experiments show yields more complex feature extraction. Four EMAs are plugged into the third, sixth, ninth, and twelfth transformer blocks of the encoder, and their adapted outputs are fused with the original features by addition.

Third, SCMamba-SAM discards SAM’s prompt encoder altogether. Instead of asking a radiologist or dermatologist to click points or draw boxes, the framework employs an enhanced mask decoder with deep supervision. Adapted features are progressively up-sampled through four deconvolution layers, each combining up-sampling, ReLU activation, and convolution, to restore spatial resolution and fuse multi-scale information. During training, intermediate decoder outputs at each stage are routed through segmentation heads and supervised against ground-truth annotations using a weighted sum of Focal and Dice losses, so the model learns to attend to lesion regions at every scale rather than only at the final output. At inference time, the pipeline is fully automatic: an image goes in, a lesion mask comes out.

The empirical case for the design is substantial. Across six public lesion segmentation datasets spanning dermatoscopy (ISIC2017 and ISIC2018), endoscopy (Kvasir-SEG), ultrasound (TN-SCUI2020), computed tomography (CT-ICH2020), and magnetic resonance imaging (MR-FCD2023), SCMamba-SAM outperformed CNN-based, Transformer-based, GAN-based, and SAM-based competitors. On ISIC2017 it achieved an IoU of 0.777 and a Dice coefficient of 0.861, edging out every baseline including fine-tuned SAM variants and the adaptation framework MSA. On ISIC2018 it reached an IoU of 0.900 and a Dice of 0.945. On Kvasir-SEG it beat the second-ranked SAM-Med2D by 6 percent in IoU, 3 percent in Dice, and 7 percent in Recall, and it succeeded where fine-tuned SAM failed on small polyps. On thyroid ultrasound it surpassed MedSAM by 10 percent in IoU, and on the particularly difficult focal cortical dysplasia MRI dataset it achieved an IoU of 0.462, six points above the runner-up, correctly segmenting MRI-negative cases that stumped other models.

Ablation studies dissected exactly where the gains come from. Starting from pre-trained SAM alone—IoU 0.550, Dice 0.676—adding LoRA fine-tuning lifted performance to 0.765 IoU and 0.849 Dice, confirming that domain adaptation of the frozen encoder is the single largest contributor. Deep supervision nudged Recall upward from 0.816 to 0.824, and the full EMA module pushed all four metrics to their peaks. Position mattered too: adapters placed at the middle and late transformer layers contributed most, while a shallow-layer adapter actually hurt performance. Replacing the Mamba layers with conventional self-attention degraded Dice by nearly five points, evidence that the state-space formulation, not just extra capacity, drives the improvement. Hyperparameter sweeps identified a LoRA rank of 128 as the sweet spot—larger ranks bought only marginal gains at substantial parameter cost—and settled on scaling factors of 8 for token number and 3 for token dimension inside the EMA.

The framework is not without trade-offs. SCMamba-SAM’s parameter count of 395 million and computational load of 1,564 GFLOPs far exceed those of the lighter CNN and Transformer baselines it outperforms, a cost the authors acknowledge and flag as a target for future lightweight architectures. The current model also processes single-modality images only; for tasks like focal cortical dysplasia, where T1, FLAIR, and PET sequences carry complementary information, multi-modal fusion could further sharpen segmentation. The training setup—PyTorch 2.1.0 on an NVIDIA A800 GPU, 200 epochs, Adam optimizer at a learning rate of 0.0001, and 256-by-256 input resizing—reflects a research-scale pipeline rather than a deployed clinical system.

Even so, the significance of the work lies in its demonstration that foundation models trained on the natural world can be repurposed for medicine with surgical, rather than wholesale, modification. By combining parameter-efficient LoRA tuning, Mamba-based adapters that inject long-range context without the overfitting penalty of full Transformers, and a prompt-free decoder with deep supervision, SCMamba-SAM turns a general-purpose segmentation engine into an automated lesion-delineation tool. For clinicians, the payoff could be faster, more reproducible quantification of lesions—information that directly informs treatment decisions and surgical planning. For the field, it offers a template: the path to medical AI may run not through training ever-larger models from scratch on scarce clinical data, but through clever, efficient adaptation of the massive models the natural-image world has already built.

Subject of Research: Adapting the Segment Anything Model with Mamba-based modules for automated medical lesion segmentation

Article Title: SCMamba-SAM: transferring knowledge from natural images to medical tasks

Article References: Tian, Y., Yuan, Y., Wang, L., Yang, X., Wu, Y., & Xu, X. (2026). SCMamba-SAM: transferring knowledge from natural images to medical tasks. Frontiers in Neurology, 17, Article 1938737. https://doi.org/10.3389/fneur.2026.1938737

Image Credits: AI Generated

DOI: 10.3389/fneur.2026.1938737

Keywords: Segment Anything Model, medical image segmentation, Mamba, state space models, LoRA fine-tuning, deep learning, lesion segmentation, computer-aided diagnosis, Frontiers in Neurology, domain adaptation, Vision Transformer, deep supervision