Research report: Frequency-aware sparse token vision transformer for medical image segmentation

A Frequency-aware Sparse Token Vision Transformer, or FS-ViT, outperformed established segmentation models on five medical image analysis tasks while using a theoretically less demanding alternative to conventional self-attention. The study, published in September 2026, evaluated the system for delineating abdominal organs in CT, cardiac structures in magnetic resonance imaging, breast lesions in ultrasound, skin lesions in photographs and polyps in endoscopy. Across the datasets, FS-ViT achieved the highest overall mean Dice similarity coefficient reported in its comparisons, alongside the lowest reported mean 95% Hausdorff distance in the task summaries.

The findings concern automated image segmentation in research datasets; they do not establish that the system improves diagnosis, prognosis, treatment planning or clinical outcomes. The model was designed around three reported limitations of vision-transformer approaches in medical imaging. Self-attention can connect distant image regions, helping a network represent an organ’s overall shape, but it may also respond strongly to local variations and noise.

Standard attention also becomes computationally expensive as the number of spatial tokens rises, a concern for high-resolution and volumetric scans. In addition, conventional multilayer perceptrons do not explicitly model fine-grained spatial patterns. FS-ViT addresses these issues with frequency-aware sparse self-attention, a frequency-aware multilayer perceptron and a U-shaped encoder–decoder architecture that combines global context with progressively finer spatial resolution.

Its central self-attention module, FS-SA, first projects query and key features into the frequency domain using the Fast Fourier Transform. Rather than constructing the full matrix of pairwise token relationships used by conventional attention, the model multiplies query and key frequency representations element by element and then applies an inverse Fourier transform. Because each Fourier coefficient is influenced by spatial locations across the feature map, this operation can aggregate global context without calculating every pairwise spatial interaction.

A squared ReLU then filters the resulting weights, amplifying positive interaction scores and reducing negative values to zero. Despite the term “sparse token” in the model name, the method does not prune the token sequence or discard spatial tokens. Sparsity is introduced only in the attention-weight matrix, where less informative interactions are suppressed while the token sequence remains intact.

The theoretical efficiency gain comes from replacing quadratic attention with frequency-domain processing. For conventional self-attention, the paper estimates a computational cost of O(N²C) and O(N²) memory consumption for storing the attention matrix, where N is the number of spatial tokens and C is the feature dimension. FS-SA instead uses Fourier transforms costing O(CN log N), followed by element-wise multiplication and sparse attention refinement with O(CN) complexity.

The authors also designed the FA-MLP module to represent fine details more effectively. It uses a partial convolutional layer to reinforce informative features, expands the feature channels, applies a Fourier transform and modulates the frequency coefficients with a learnable matrix. An inverse transform and final convolutional layer return the representation to the spatial domain.

The evaluation covered five publicly available datasets with different anatomical targets and image characteristics. The Synapse collection contained 30 abdominal CT scans and 3,779 axial slices, with manual segmentation annotations for 13 organs; 18 cases were used for training and 12 for testing, while results were reported for eight organs. The ACDC data included 100 cardiac MRI scans from different patients, split into 70 training cases, 10 validation cases and 20 test cases, with masks for the right ventricle, myocardium and left ventricle.

The BUSI dataset contained 780 breast ultrasound images with annotations for tumors, cysts and normal tissue, including 210 malignant, 437 benign and 133 normal images. Two further datasets were used for image-level segmentation. The ISIC 2018 collection contained 2,694 skin images with lesion masks, divided into 1,886 training images and 808 test images.

The Kvasir-SEG dataset contained 1,000 endoscopy images of polyps, with 70% used for training and 30% for testing. All models were implemented in PyTorch and trained on a single NVIDIA V100 GPU with 32 GB of memory, using a batch size of 24. Training lasted 600 epochs for the Synapse and ACDC datasets and 500 epochs for the other three.

Input images were resized or cropped to 224-by-224 patches, and preprocessing included percentile clipping, z-score normalization, random rotation, scaling and mirroring. On the Synapse CT task, FS-ViT reached a mean Dice score of 86.44% and a mean 95% Hausdorff distance of 12.50 millimetres. The previous best model, MERIT, reported 84.90% Dice and 13.22 millimetres, so FS-ViT increased overlap by 1.54 percentage points while reducing the boundary-distance measure by 0.72 millimetres.

Its highest organ-level Dice scores included 95.66% for the liver, 92.24% for the spleen and 85.64% for the stomach; the pancreas score was 80.72%. For cardiac MRI, FS-ViT achieved 92.38% mean Dice, compared with 92.25% for EDLDNet and 91.81% for MERIT. It scored 91.32% for the right ventricle, 95.93% for the left ventricle and 89.88% for myocardium.

The reported improvements over competing methods were significant under a Wilcoxon signed-rank test at p less than 0.01. Similar margins appeared in the three image-based tasks. On BUSI breast ultrasound images, FS-ViT achieved 89.80% mean Dice and 8.23 millimetres mean 95% Hausdorff distance, improving on MSCA-Net’s 88.21% and 9.75 millimetres.

On ISIC 2018 skin lesions, it reached 90.80% Dice and 19.46 millimetres, compared with EDLDNet’s 89.55% and 21.31 millimetres. For Kvasir-SEG endoscopy images, the model produced 94.10% Dice and 2.18 millimetres; the strongest competing Dice was 92.54% for HiFormer, while the second-best reported Hausdorff result was 3.10 millimetres for BGFF-UNet. These gains were larger than one percentage point in Dice on all three datasets, although the magnitude of the Hausdorff-distance reduction varied by task.

Attention and activation visualizations offered an interpretation of the numerical results. Grad-CAM maps for pancreas, right-ventricle and benign-breast-tumor examples showed stronger responses along target boundaries and within relevant anatomy than maps from comparison models. The authors reported more compact attention patterns with reduced activity in background regions.

FS-ViT’s frequency-domain activations were also more localized than those of a plain vision transformer, while retaining responses associated with target structures. In the ablation study, a baseline using conventional self-attention and a conventional multilayer perceptron scored 80.20%, 90.32% and 85.61% Dice on Synapse, ACDC and BUSI. Adding both proposed modules raised those scores to 86.44%, 92.38% and 89.80%, respectively, while mean 95% Hausdorff distances fell from 20.62 to 12.50 millimetres, from 1.58 to 1.16 millimetres and from 14.87 to 8.23 millimetres.

Important limitations remain. The evaluation used relatively small public benchmarks, including only 30 CT volumes for the multi-organ task, and did not test performance across independent hospitals, scanners, acquisition protocols or rare diseases. The authors state that larger multicenter datasets, external cohorts and heterogeneous acquisition protocols would be needed to assess generalizability more comprehensively.

They also note that frequency representations suited to structured CT and MR anatomy may not transfer equally well to severe speckle noise, irregular textures, lower signal-to-noise ratios or modalities such as microscopy. FS-ViT contains no explicit anatomical-prior, topology-constraint or uncertainty-estimation mechanism. Consequently, the reported results support its effectiveness as a research model for automated segmentation, but broader validation is needed before conclusions about robustness or safety in clinical deployment.

Subject of Research: Frequency-domain sparse vision transformers for automated medical image segmentation

Article Title: Frequency-aware sparse token vision transformer for medical image segmentation

Article References: Sun, J., Wang, J., & Liu, J. (2026). Frequency-aware sparse token vision transformer for medical image segmentation. Discover Artificial Intelligence, 6(1), Article 1082. https://doi.org/10.1007/s44163-026-02129-8

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02129-8

Keywords: medical image segmentation, vision transformers, frequency-domain learning, sparse attention, CT, MRI, ultrasound, skin lesions, endoscopy, clinical AI, Frequency-aware, sparse