New AI Tracker Fuses Visible and Thermal Vision to Stay on Target in Any Weather

Tracking a single object through a video stream sounds like a solved problem until the lights go out. When a camera plunges into darkness, smoke fills the frame, or a pedestrian steps behind a car whose hot engine glows in the infrared, even the best visual trackers lose their grip. A research team in China now reports a new approach designed to keep computers locked onto targets under exactly these punishing conditions, by teaching an artificial intelligence to blend what ordinary cameras see with what thermal sensors feel. The work, published in the journal Applied Intelligence, introduces a tracking framework called BFA-HARF that its authors say achieves performance comparable to state-of-the-art methods across four public benchmark datasets.

The challenge the researchers set out to address is known as RGB-T object tracking, where RGB refers to the standard red-green-blue color imagery captured by visible-light cameras and T stands for thermal infrared. Thermal cameras detect heat rather than light, which makes them nearly immune to darkness, glare, and many kinds of visual clutter. Visible cameras, meanwhile, deliver rich texture and color detail that thermal sensors lack. In principle, combining the two should produce a tracker that works around the clock and in almost any weather. In practice, the fusion is far from trivial, because the two modalities carry fundamentally different kinds of information, and naively merging them can inject as much noise as signal.

According to the authors, Can Xu of East China Normal University, Weidai Xia of Central South University, Lingmin Fan of Shenergy Group, and Yue Zhang, also of East China Normal University, existing methods often suffer from insufficient feature representation and redundant cross-modal invalid information. In plain terms, the networks behind many current trackers do not extract rich enough descriptions of the target, and when they combine visible and thermal streams they frequently drag along information from one modality that is useless or misleading in the other. The result is a familiar failure mode in the tracking literature: the bounding box that is supposed to hug the target begins to drift, sometimes sliding onto a nearby distractor or ballooning into an ambiguous region that no longer corresponds to anything in the scene.

The team’s answer is a two-stage design philosophy they describe as align-then-fuse. Rather than throwing the two modalities together at a single point and hoping the network sorts things out, BFA-HARF first makes sure the visible and thermal features are progressively aligned and mutually enhanced, and only then applies a dedicated fusion mechanism. This sequencing, the authors argue, is what allows the tracker to build a comprehensive feature representation instead of a muddled one, and it is the conceptual core of the paper.

The first of the framework’s two synergistic modules is the Bidirectional Feature Adapter, or BFA. Adapters are lightweight neural components inserted into a larger network, a technique that has become popular because it lets researchers adapt powerful pretrained backbones to new tasks without retraining everything from scratch. What distinguishes the BFA is its direction of information flow. Instead of letting the two modalities exchange information only once, at a single fusion layer, the BFA facilitates a continuous bidirectional information flow between the RGB and thermal branches throughout the backbone network. At every stage of feature extraction, each modality receives a steady stream of guidance from its counterpart, so that the visible features gradually absorb thermal cues about where heat signatures lie, and the thermal features gradually absorb visible cues about texture and boundary structure. By the time the features reach the fusion stage, they are no longer two parallel, loosely related descriptions of the scene; they are two mutually refined representations that already share a common frame of reference.

The second module, Hybrid Attention with Receptive Fields, or HARF, takes over once the alignment is done. Its job is to process the aligned features by collaboratively capturing two complementary kinds of structure. On one side, self-attention, the mechanism that powers modern transformer architectures, lets every position in the feature map weigh the relevance of every other position, capturing global dependencies that span the entire search region. This is invaluable when a target is small, distant, or surrounded by context that matters for disambiguation. On the other side, convolutional operations excel at fine-grained local patterns, detecting edges, corners, and textures within a small neighborhood. Convolution is also constrained by its receptive field, the limited window of the input it can see at any given layer, which is precisely the weakness that attention compensates for. By hybridizing the two, the HARF module captures both the forest and the trees: the sweeping global relationships that attention provides and the sharp local detail that convolution preserves.

The practical payoff of this architecture, the authors report, is a measurable reduction in two of the most stubborn failure modes in multimodal tracking. The first is bounding box ambiguity, in which the predicted box becomes uncertain about exactly what it should contain, often because the fused features have blended target and background information indiscriminately. The second is tracking drift, the slow accumulation of error in which a tracker that is slightly off in one frame becomes further off in the next, eventually losing the target entirely. Because the BFA ensures that each modality continuously corrects and enriches the other, and the HARF module fuses only features that have already been aligned, the framework is better equipped to keep the target’s identity stable across frames even when one sensor’s view degrades.

The evidence comes from extensive experiments on four public RGB-T benchmark datasets, the standard proving grounds for this subfield, which include sequences annotated with challenging attributes such as low light, thermal crossover, occlusion, and distractors. On these benchmarks the proposed algorithm achieved performance comparable to state-of-the-art methods, according to the paper, while specifically alleviating the ambiguity and drift problems the design targets. The work builds on a deep lineage of RGB-T research, from early sparse-representation approaches that treated grayscale-thermal fusion as a collaborative coding problem, through Siamese network trackers that learned shared embeddings for both modalities, to recent transformer-based and adapter-based designs such as bi-directional adapters for multimodal tracking and prompt-driven trackers. BFA-HARF’s contribution within that lineage is the insistence that alignment and fusion are distinct problems deserving distinct, staged solutions.

The broader significance of the research lies in its application space. RGB-T tracking underpins technologies where failure is costly: autonomous driving systems that must keep sight of pedestrians at night, surveillance platforms operating through smoke or fog, search-and-rescue drones scanning for body heat in rubble, and traffic monitoring systems that must function in rain and glare. Thermal-infrared object detection for autonomous driving has been an active topic in the same journal, and the new tracker’s emphasis on robustness in complex environments speaks directly to those safety-critical uses. A tracker that resists drift when the visible channel fails could mean the difference between a system that reliably follows a person through a dark parking structure and one that silently loses them.

The authors acknowledge support from the Shanghai Special Program for Promoting High-Quality Industrial Development, and they report no conflicts of interest. The data underlying the study will be made available upon request. For the field, the paper adds a clear architectural lesson: in multimodal perception, the order of operations matters. Aligning features before fusing them, and letting that alignment happen continuously rather than at a single bottleneck, appears to squeeze more value out of each sensor and less noise into the final representation. As cameras and thermal imagers become cheaper and more common on everything from cars to consumer drones, frameworks like BFA-HARF point toward vision systems that do not blink when the lights do.

Subject of Research: Robust RGB-T object tracking via bidirectional feature alignment and hybrid attention fusion of visible and thermal infrared imagery

Article Title: BFA-HARF: Robust RGB-T tracking via bidirectional feature adapter and hybrid attention with receptive fields

Article References: Xu, C., Xia, W., Fan, L., & Zhang, Y. (2026). BFA-HARF: Robust RGB-T tracking via bidirectional feature adapter and hybrid attention with receptive fields. Applied Intelligence, 56(14), Article 416. https://doi.org/10.1007/s10489-026-07466-w

Image Credits: AI Generated

DOI: 10.1007/s10489-026-07466-w

Keywords: RGB-T tracking, thermal infrared, multimodal fusion, cross-modal feature enhancement, bidirectional feature adapter, hybrid attention, receptive fields, self-attention, tracking drift, computer vision, object tracking, Applied Intelligence