Deep Learning System Maps Retinal Blood Vessels Using Geodesic Structural Guidance

Researchers from Queensland University of Technology, Telemedc, and King Edward Medical University have developed an end-to-end deep learning framework termed AV-MapNet to improve the precision of automated retinal vascular tree segmentation and artery-vein discrimination. Quantitative morphological assessment of the retinal microcirculation offers a valuable, non-invasive window into systemic and ocular vascular health, providing biomarkers for diabetic retinopathy, age-related macular degeneration, glaucoma, and hypertension. Calculating diagnostic indicators such as the arteriolar-to-venular diameter ratio relies heavily on identifying whether individual vessels function as arteries or veins. However, manual vascular delineation remains labor-intensive and prone to observer variability, while prior deep learning models often struggle around thin peripheral vessels, crossings, bifurcations, and low-contrast or pathological structures. AV-MapNet addresses these issues by organizing feature extraction along the anatomical contours of the vasculature rather than relying exclusively on fixed rectangular coordinate grids.

The foundational architecture of AV-MapNet operates through a refine-before-classify strategy designed to establish a stable structural vascular representation before executing fine-grained artery and vein classification. Retinal arteries and veins are not isolated objects but interconnected components of a continuous vascular network, meaning segmentation errors in earlier stages can rapidly propagate into erroneous classification outputs. The framework processes standard RGB retinal fundus images using a five-level residual encoder-decoder network. To handle the wide range of vessel calibres across the retina, the model incorporates parallel dilated convolutions at the deepest representation layer, utilizing dilation factors of 1, 2, 4, and 8. An additional dilated context block operates at the quarter-resolution decoder stage with dilation rates of 1, 2, and 3. This dilated design expands the receptive field without sacrificing feature map resolution, ensuring the system captures both broad vessel geometry and narrow terminal branches.

Rather than finalizing the vessel segmentation after a single forward pass, AV-MapNet introduces a sequential residual refinement module that iteratively corrects initial prediction errors. Standard convolutional predictions frequently exhibit fragmented thin branches, inaccurate vessel borders, or spurious false-positive responses. The refinement module receives the intermediate vessel probability alongside context extracted from the full-resolution decoder, predicting an additive residual correction across three sequential steps. By sharing identical refinement parameters across all three iterations, the network recurrently stabilizes vascular boundary estimates with minimal parameter overhead. Supervised training reinforces this iterative behavior through deep supervision, applying auxiliary losses weighted by 0.25 to the initial and intermediate refinement passes. This stabilized vessel probability map establishes an explicit structural foundation that guides subsequent representation learning and classification stages.

The central representation innovation introduced in the study is Geodesic Vessel Tokenization, a mechanism that groups intermediate retinal features according to structural compatibility along predicted vascular pathways. Standard visual tokenization schemes partition images into rigid rectangular patches, which often divide continuous, winding vessels across arbitrary boundaries or merge structurally unrelated background and vascular pixels into the same token. In contrast, the tokenization module predicts three learned structural cues from quarter-resolution vessel features: local vessel orientation, boundary ambiguity, and junction likelihood. A learnable convolutional layer predicts image-dependent spatial seed distributions for sixteen vascular tokens, ensuring token positions dynamically adapt to the underlying vascular anatomy rather than staying locked to predetermined spatial coordinates.

Information propagates from these learned seed distributions through an eight-connected neighbourhood via differentiable geodesic updates across four iterations. The cost of traversing between neighboring pixels is explicitly formulated to reward movement along confident vessel probabilities and aligned vessel orientations, while penalizing movement across background pixels or ambiguous vessel boundaries. Junction likelihood cues reduce traversal costs at branching points, enabling tokens to track diverging vascular paths naturally. Following propagation, soft token assignments are computed using a normalized soft-min distance function, and each token aggregates spatial features into a 64-dimensional embedding. A reliability-aware gating network then evaluates each token’s mean vessel confidence and boundary ambiguity, scaling the token representations so that well-defined vascular regions contribute strongly to global reasoning while ambiguous boundary zones are systematically suppressed.

Once the sixteen vessel-aligned tokens are formed, AV-MapNet applies two topology-aware long-range reasoning blocks equipped with four attention heads. Traditional self-attention determines inter-token relationships based strictly on feature similarity, but the topology-aware attention block injects an explicit structural bias. This bias incorporates the Euclidean distance between normalized token centroids, absolute directional similarity between aggregated orientation vectors, and token junction responses. By favoring tokens that align along continuous vascular trajectories and interact around anatomical branching points, the model contextualizes distant vessel branches without relying solely on local convolutional filters. The contextualized tokens are subsequently projected to eight feature channels and upsampled, reconstructing a dense spatial feature map that provides comprehensive structural guidance for downstream classification.

Vessel classification in AV-MapNet is handled by independent artery and vein encoder-decoder branches conditioned directly on the refined vessel representation. The initial vessel probability map is detached from classification gradients to prevent downstream classification updates from destabilizing the structural vessel tree. The RGB retinal image is multiplied by the refined vessel probability to eliminate irrelevant background pixels, and this vessel-guided appearance is concatenated with the reconstructed token context and the vessel probability map itself. The separated artery and vein branches refine their initial class logits across three class-specific sequential iterations. The resulting logits are then jointly normalized using a two-class softmax, establishing direct competition between artery and vein classes at each pixel. Finally, the normalized probabilities are multiplied by the refined vessel mask, ensuring non-vascular background regions are completely suppressed.

Model training was conducted in PyTorch using an effective batch size of two and the AdamW optimizer over 300 epochs, combining binary cross-entropy and soft Dice loss for vessel segmentation with auxiliary orientation, boundary, and junction losses. AV-MapNet contains approximately 4.06 million trainable parameters, requiring an estimated 68.2 GFLOPs for a 512 by 512 input image and 15.5 megabytes of FP32 parameter storage. Quantitative vessel segmentation evaluations were conducted on the RITE, HRF, and DRIVE datasets, where AV-MapNet achieved Dice similarity coefficients of 83.01 percent, 85.21 percent, and 87.10 percent, respectively, alongside corresponding pixel-wise accuracies of 97.67 percent, 98.01 percent, and 98.52 percent. In controlled benchmark experiments on RITE using identical training splits, 512 by 512 resolutions, and a fixed 0.50 threshold, AV-MapNet outperformed previous models such as MaskVSC, which reached 79.65 percent Dice, and RRWNet, which achieved 74.39 percent Dice.

To evaluate the topological consistency of the segmented vascular networks beyond simple area overlap, the authors evaluated predictions on the 20-image DRIVE test set using centerline Dice and Betti-number errors. Centerline Dice measures alignment between predicted and reference vessel skeletons, while Betti-0 and Betti-1 errors quantify discrepancies in connected foreground components and cyclic vascular loops, respectively. AV-MapNet recorded a centerline Dice score of 0.8490, a Betti-0 error of 0.9071, and a Betti-1 error of 0.2391, yielding a mean combined Betti error of 0.5731. These results reflect lower structural discrepancies compared to the standardized clDice baseline, which recorded a Betti-0 error of 3.080 and Betti-1 error of 0.279, while remaining competitive with specialized topology-oriented architectures.

Direct binary artery-vein classification was assessed across valid ground-truth vessel pixels on the LES-AV, AV-DRIVE, and RITE datasets, excluding ambiguous vessel crossings and background pixels in accordance with standard benchmark protocols. AV-MapNet achieved classification accuracies of 96.01 percent on LES-AV, 96.48 percent on AV-DRIVE, and 97.20 percent on RITE, with corresponding class Dice scores of 88.70 percent, 89.00 percent, and 88.50 percent. Ablation experiments on AV-DRIVE confirmed that adding class-specific sequential refinement raised classification accuracy from 96.07 percent to 96.48 percent, while earlier modules incrementally raised vessel Dice from a baseline score of 84.12 percent to 87.10 percent. Qualitative evaluations on 32 images from the IDRiD dataset also demonstrated that the network maintains clear vascular trajectories when distinguishing true vessels from bright exudates and retinal hemorrhages.

The study highlights specific constraints regarding the structural modeling techniques. The authors clarify that while AV-MapNet is structurally and topologically informed, it does not offer a strict mathematical guarantee of topology preservation. The geodesic tokenization mechanism remains dependent on the initial vessel segmentation; severe vessel fragmentation, missing micro-vessels, or extensive pathological occlusions can supply insufficient evidence for accurate feature propagation. Additionally, the traversal cost weighting coefficients are fixed parameters rather than dynamically learned values. Future investigations will examine learnable or adaptive traversal penalties, multi-center clinical validation, and broader evaluations across severe retinal pathologies.

Subject of Research: Retinal vessel segmentation and artery-vein classification in biomedical image computing

Article Title: AV-MapNet: geodesic vessel tokenization and sequential refinement for retinal vessel segmentation and artery–vein classification

Article References: Ali, O., Xiao, D., Yasin, N., Zhang, J., & Li, Y. (2026). AV-MapNet: geodesic vessel tokenization and sequential refinement for retinal vessel segmentation and artery–vein classification. Frontiers in Radiology, 6, 1913575. https://doi.org/10.3389/fradi.2026.1913575

Image Credits: AI Generated

DOI: 10.3389/fradi.2026.1913575

Keywords: Retinal Imaging, Deep Learning, Vessel Segmentation, Artery-Vein Classification, Geodesic Tokenization, Medical Image Analysis, Biomedical Engineering, Ophthalmology AI, AV-MapNet, geodesic, vessel, tokenization