Improved YOLOv11 with hybrid encoder boosts rotating target detection in remote sensing images

Rotating target detection in remote sensing images has long been hampered by three stubborn problems: small targets lose their fragile features during downsampling, complex backgrounds interfere with global semantic extraction, and arbitrarily oriented objects make precise localization difficult. A new study published in Discover Artificial Intelligence proposes an end-to-end framework, called SR-YOLO-DETR, that integrates an improved You Only Look Once version 11 (YOLOv11) backbone with a Real-Time Detection Transformer (RT-DETR) hybrid encoder to tackle all three challenges at once. On the widely used DOTA and HRSC2016 benchmark datasets, the model achieved mean average precision (mAP) values of 81.68% and 94.45%, respectively, representing improvements of 5.26% and 5.95% over the baseline YOLOv11 model.

The motivation behind the work stems from the unique bird’s-eye perspective of remote sensing imagery. Vehicles, ships and infrastructure appear in arbitrary orientations that change in real time with flight direction and target motion. Horizontal bounding boxes, which must align with the image coordinate axes, cannot tightly enclose such objects, leaving large amounts of irrelevant background inside the box. This weakens the texture discrimination needed for classification, prevents regression branches from capturing true contours and directional parameters, and causes severe false positives during non-maximum suppression in densely arranged scenes. In traffic monitoring, direction perception is the core basis for distinguishing oncoming from ongoing vehicles; in disaster assessment, ignoring true rotation angles can produce severe spatial deviations in rescue route planning and damaged area measurement.

To address the loss of small-target features, the researchers introduced a space-to-depth conversion convolution module, known as SPD-Conv, into the downsampling path of the YOLOv11 backbone. Traditional strided convolutions irreversibly filter out high-frequency texture and geometric details of tiny targets, disconnecting deep semantic information from shallow localization details. SPD-Conv instead slices the input feature map into four sub-feature maps by interval sampling, dispersing pixels from spatial neighborhoods into different sub-maps, and then concatenates them along the channel axis. The spatial resolution is halved while the channel count is quadrupled, so no pixel information is discarded. A subsequent non-strided convolution fuses the expanded channel information, allowing weak target pixels to maintain high activation responses in the deep feature maps P3, P4 and P5 that the backbone outputs after spatial pyramid pooling.

Preserving small features alone does not solve the problem of global context. Convolutional architectures are limited by local receptive fields and struggle to capture long-span dependencies across wide-area backgrounds. The study therefore embeds the efficient hybrid encoder from RT-DETR, which contains two complementary components. The scale-within feature interaction module, AIFI, applies self-attention only to the top-level feature map P5, where semantic information is richest, efficiently establishing long-range context dependencies while reducing computational redundancy. The attention computation follows the standard Transformer encoder paradigm, projecting flattened feature sequences into query, key and value vectors and generating attention maps through scaled dot-product operations.

The second component, the cross-scale feature fusion module CCFM, maps the globally enhanced top-level features back into the multi-resolution feature space. It performs top-down path aggregation by upsampling the AIFI-enhanced features and fusing them with the mid-level and low-level backbone outputs, then compensates for positional attenuation through a bottom-up path that downsamples and re-merges features. RepC3 fusion blocks handle the nonlinear transformations throughout. The result is a feature pyramid containing both rich semantics and fine texture, capable of detecting large ships and tiny buoys in the same scene, and providing multi-scale inputs for the detection heads.

For rotation itself, the framework employs a decoupled detection head with three functionally independent branches for classification, geometric regression and angle prediction. The authors argue that shared detection heads ignore the essential difference in feature requirements between the two tasks: classification needs rotation-invariant discriminative texture, while regression depends on precise edge and direction information. The angle branch is paired with a Gaussian distribution regression strategy that maps each oriented bounding box to a two-dimensional Gaussian distribution. The mean vector corresponds to the box center, while a covariance matrix built from a rotation matrix and a diagonal scale matrix encodes shape and orientation. Angle regression is thereby transformed into measuring the Wasserstein distance between probability distributions, eliminating the boundary discontinuity caused by angle periodicity that destabilizes convergence in direct regression, trigonometric encoding and classification-based approaches.

The ablation experiments, conducted step by step on DOTA and HRSC2016 with the original YOLOv11 as baseline, confirmed the contribution of each module. Adding the SPD-Conv backbone, then the AIFI and CCFM encoder, and finally the decoupled rotation head produced cumulative gains that reached the full 5.26% and 5.95% mAP improvements. The hybrid encoder did increase the parameter count from 20.8 million to 38.2 million and reduced inference speed from 92 to 62 frames per second, but the model retained real-time capability. In fine-grained analysis, the median accuracy in the small-target group of DOTA improved by 16.2% over the baseline, and in the large aspect ratio group of HRSC2016 the median reached 88.5%, an increase of 14.3%, indicating that the modules jointly resolve feature alignment for extreme scales and slender rotating targets.

Comparisons with mainstream algorithms reinforced the picture. Against the Oriented R-CNN two-stage detector, SR-YOLO-DETR achieved precision on DOTA that was 1.23% higher while running 2.8 times faster. Against YOLOv8-OBB, precision improved by 1.86% at similar parameter scale, and under the strict localization threshold of IoU = 0.8 the mAP advantage reached 10.2%, remaining at 58.80% on DOTA and 55.80% on HRSC2016 at IoU = 0.9. In difficult categories, the model scored 80.5% on the dense, tiny small-vehicle class and 58.2% on the highly confusing bridge class, the latter 2.8% above Oriented R-CNN. On HRSC2016 the precision-recall curve enveloped all comparison methods, maintaining 88.6% precision at a recall of 0.9.

The team also tested robustness under degraded imaging by synthesizing haze, low-light and cloud-cover versions of the DOTA test set. All models declined under degradation, but SR-YOLO-DETR consistently held the highest mAP, at 75.46%, 72.18% and 74.55% respectively, with smaller drops than comparable models. The authors attribute this stability to SPD-Conv’s preservation of weak features and to the global context integration of AIFI and CCFM, which partially compensates for information loss caused by occlusion or reduced contrast. The experiments were run on an Ubuntu 22.04 system with an Intel Core i9-13900K processor and an NVIDIA GeForce RTX 3090 graphics card, using PyTorch 2.1.0 and CUDA 11.8, with images scaled to 1024 by 1024 pixels and augmented through mosaic, mixing and random rotation from 0 to 180 degrees.

The authors note limitations alongside the achievements. The current research focuses mainly on general multi-class scenarios and single-class ship detection, and generalization under extreme weather conditions remains insufficiently verified, with future work planned to expand dataset diversity and explore adaptation to haze and night environments. Within its tested scope, however, the study demonstrates that combining a lossless space-to-depth backbone, a Transformer-based hybrid encoder and Gaussian-distribution angle regression can break the traditional trade-off in which high accuracy demands low speed, positioning SR-YOLO-DETR on the Pareto-optimal frontier of the speed-accuracy balance and suggesting practical potential for real-time Earth observation tasks such as urban planning, disaster monitoring and precision agriculture.

Subject of Research: Rotating target detection in remote sensing images using an improved YOLOv11 backbone integrated with an RT-DETR hybrid encoder

Article Title: Rotating target detection in remote sensing images by integrating improved YOLOv11 with an efficient hybrid encoder

Article References: Xie, L. (2026). Rotating target detection in remote sensing images by integrating improved YOLOv11 with an efficient hybrid encoder. Discover Artificial Intelligence, 6(1), Article 1020. https://doi.org/10.1007/s44163-026-02096-0

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02096-0

Keywords: remote sensing, object detection, YOLOv11, RT-DETR, rotating targets, SPD-Conv, DOTA, HRSC2016, oriented bounding box, Gaussian distribution regression, deep learning, computer vision