The obstacle is that the deep learning models behind most automated retinal analysis are supervised. Convolutional neural networks, vision transformers, and hybrid architectures have achieved impressive accuracy in classifying and segmenting retinal pathology, but they depend on large, expertly annotated datasets. In medical imaging, such annotations are expensive, slow to produce, and constrained by ethical and privacy considerations, which leaves many clinical settings, particularly in low-resource regions, unable to build the labeled data foundations that supervised systems require. A research team from Learnify Research Laboratory, Green University of Bangladesh, American International University-Bangladesh, and the University of Victoria has now proposed a way around this bottleneck, publishing their framework in Frontiers in Signal Processing.

The researchers introduce NF-HAE-UDL, short for normalizing flow-guided hybrid autoencoder-based unsupervised deep learning, a framework that learns and clusters retinal image representations from small fundus datasets without requiring pixel-level annotations for clustering. The system uses only image-level disease labels to guide contrastive pair construction and a classification loss during training, a far lighter annotation burden than the dense pixel masks demanded by supervised segmentation models. According to the authors, it is the first framework to jointly combine a hybrid CNN-Transformer autoencoder, a normalizing flow module, and label-guided contrastive learning for retinal disease clustering on genuinely small fundus datasets.

At the heart of the architecture is a dual-branch hybrid encoder. A ResNet-50 convolutional branch, pretrained on ImageNet-1K, extracts local texture features such as microaneurysms, hemorrhages, and vessel morphology, producing a 2048-dimensional feature vector after global average pooling. A Swin-Tiny transformer branch captures global structural context, such as optic disc positioning, through hierarchical window-based self-attention, yielding a 768-dimensional CLS token embedding. These complementary vectors are concatenated and projected through a learned linear transformation with ReLU activation into a unified latent space, so that a single embedding simultaneously encodes fine-grained retinal cues and broader anatomical structure, something single-branch models cannot provide.

The framework then refines this latent representation with a Glow-based normalizing flow, a stack of invertible transformations built from ActNorm, affine coupling, and learned-scale layers. The flow maps the encoded features toward a structured Gaussian distribution, providing explicit likelihood estimation and invertible latent manipulation that standard variational or deterministic autoencoders cannot offer. In practice, the team implemented a computationally efficient simplified variant that retains approximate density estimation while reducing parameter count for small datasets. Training maximizes the log-likelihood of the encoded features under the flow, a regularization signal that complements the contrastive objective and encourages latent codes to occupy a compact, well-organized region of the Gaussian prior.

Contrastive learning supplies the disease-discriminative pressure. Following standard protocols, two augmented views are generated per image using random horizontal flips, rotations within 15 degrees, color jitter, and Gaussian blur, with extreme transformations deliberately excluded because they could obscure diagnostically relevant structures. A supervised contrastive loss in the NT-Xent formulation, computed with a temperature of 0.07 on the flow-refined latent codes, pulls augmented views of the same image together while pushing representations of different images apart, with image-level disease labels guiding which pairs count as positives. The total training objective combines reconstruction loss, contrastive loss, classification loss, and flow-based negative log-likelihood in a unified multi-objective framework, and a two-stage schedule freezes the pretrained backbones for the first 50 epochs before fine-tuning them at a reduced learning rate to prevent catastrophic forgetting on tiny datasets.

After training, the learned latent codes are reduced in dimensionality with UMAP and clustered with DBSCAN, a density-based algorithm that does not require the number of clusters to be specified in advance. Ground-truth labels are introduced only after training completes, solely to compute evaluation metrics, guaranteeing a clean separation between representation learning and assessment. Evaluated across ten independent trials on the STARE dataset, which contains 81 multi-label fundus images spanning 11 disease categories, and a 211-image binary subset of IDRiD focused on diabetic retinopathy, the framework achieved adjusted Rand index scores of 0.58 plus or minus 0.10 on STARE and 0.58 plus or minus 0.06 on IDRiD, with normalized mutual information of 0.70 and 0.63 and Silhouette scores of 0.79 and 0.66 respectively. These results outperformed traditional clustering baselines such as k-means, Gaussian mixture models, spectral clustering, and deep embedded clustering, as well as self-supervised baselines including DINO, BYOL, SimSiam, and MAE, which the authors attribute to the domain gap between natural images and retinal photography and the absence of disease-specific contrastive objectives in general-purpose pretraining.

An ablation study confirmed that the normalizing flow module delivers consistent, statistically significant improvements in clustering metrics, raising NMI by 0.06 and Silhouette by 0.07 on STARE and Silhouette by 0.10 on IDRiD compared with a no-flow baseline, while adding less than 10 percent training time overhead. Replacing the flow with a variational autoencoder produced lower clustering performance on the complex STARE dataset, supporting the choice of normalizing flow over simpler latent regularization. Beyond clustering, the flow enables anomaly detection through likelihood thresholding and invertible latent manipulation such as disease severity interpolation, capabilities the authors argue add interpretability and controllability with minimal computational cost.

To verify that the learned features are clinically meaningful rather than artifacts of the architecture, the team applied a post hoc Grad-CAM interpretability protocol. Because standard Grad-CAM requires supervised class logits, a lightweight linear probe was trained on frozen latent codes to map them to cluster pseudo-labels, and gradients were then backpropagated through the frozen encoder to produce saliency heatmaps. Masking regions with high Grad-CAM activation caused an average confidence drop of 0.42, while masking low-activation regions caused only a 0.07 drop, and heatmaps validated against annotations from an experienced ophthalmologist on 20 randomly selected STARE images achieved an intersection-over-union of 0.52 plus or minus 0.11, indicating moderate spatial agreement with clinical ground truth. For normal retinas, attention was distributed relatively uniformly, while in background diabetic retinopathy the model concentrated on microaneurysms and hemorrhages, and in central retinal vein occlusion it highlighted extensive hemorrhages and cotton wool spots.

Auxiliary classification validation on frozen encoder representations confirmed that the learned embeddings preserve disease-discriminative information, with a linear classifier achieving 88 percent in-sample accuracy on STARE and 98 percent on IDRiD, and micro-F1 scores of 0.95 and 0.99 respectively. The authors are careful to note that these figures represent representation quality assessment rather than diagnostic performance, since the evaluation is in-sample and the datasets are small, single-institution, and collected under controlled imaging conditions. Limitations include the modest statistical power of generalization assessments with only 16 and 42 test images, the possibility that high IDRiD test accuracy exploits dataset-specific characteristics, and the need for external validation on larger multi-center databases such as Messidor, APTOS, and DRHAGIS. Future work will pursue semi-supervised fine-tuning for direct diagnostic benchmarking, stronger alignment between learned attention and clinical reasoning, and interactive visualization tools to support clinical deployment. All experiments ran on a single NVIDIA RTX 3090 GPU, with total training times of 2.1 hours for STARE and 5.8 hours for IDRiD per trial and real-time inference of roughly 22 milliseconds per image, suggesting the framework could fit within resource-constrained screening workflows where large annotated datasets are simply unavailable.

Subject of Research: An unsupervised deep learning framework combining a hybrid autoencoder, normalizing flow, and contrastive learning for clustering retinal fundus images from small datasets

Article Title: Identification of eye diseases with small fundus image datasets using a new unsupervised deep learning approach

Article References: Aurnob, A. R., Tanim, S. A., & Sattar, F. (2026). Identification of eye diseases with small fundus image datasets using a new unsupervised deep learning approach. Frontiers in Signal Processing, 6, Article 1769553. https://doi.org/10.3389/frsip.2026.1769553

Image Credits: AI Generated

DOI: 10.3389/frsip.2026.1769553

Keywords: unsupervised deep learning, retinal fundus images, normalizing flow, contrastive learning, hybrid autoencoder, eye disease clustering, diabetic retinopathy, STARE dataset, IDRiD dataset, Grad-CAM interpretability, small-scale datasets, medical image analysis