New AI Framework Tackles Hidden Identity Mismatches in Infrared Surveillance Tracking

Researchers at Zhejiang University and the Zhejiang University of Water Resources and Electric Power have unveiled a new framework that addresses a subtle but pervasive flaw in unsupervised visible-infrared person re-identification, a technology that lets surveillance systems recognize the same person across ordinary color cameras and thermal infrared cameras without relying on any labeled training data. The study, published in the International Journal of Machine Learning and Cybernetics, is the first to formally define and systematically study what the authors call the identity-misalignment problem, a condition that quietly corrupts many existing approaches and degrades their real-world accuracy.

Person re-identification, often abbreviated as Re-ID, is the task of matching images of the same individual captured by different cameras at different times and locations. In intelligent security applications, matching people across visible-light and infrared modalities is essential because infrared cameras keep working at night, in smoke, and in other conditions where ordinary cameras fail. A cross-modality retrieval model must therefore learn representations of a person that remain stable even when the image shifts from full-color visible detail to the coarse heat signatures of a thermal sensor. Supervised versions of this task depend on expensive annotations that tell the system which images belong to which person, and collecting such labels at scale is prohibitively costly.

Unsupervised visible-infrared person re-identification, or USVI-ReID, removes the need for annotation by letting the model discover identities on its own, typically through clustering algorithms that group visually similar images and assign pseudo-labels. Yet the field has long rested on a hidden assumption. Most existing methods are developed and validated on curated benchmark datasets in which every person appears in both modalities, so when pseudo-labels learned on one modality are transferred to the other, every label finds a valid counterpart. The Zhejiang team argues that this assumption collapses the moment one moves to automatically collected, unlabeled data in the wild.

The reason is geometric and behavioral rather than algorithmic. Infrared cameras and visible cameras occupy different physical positions and capture different fields of view, and pedestrian trajectories are inherently uncertain. A person who walks past a visible camera may never pass within range of the infrared one, or vice versa. As a result, real unlabeled datasets inevitably contain identities that are present in one modality but entirely absent from the other. When a model trained under the fully aligned assumption transfers labels across modalities, these missing identities become sources of what the researchers term label transfer noise, incorrectly binding images of different people or forcing phantom identities onto data that cannot support them. This noise accumulates during training and erodes retrieval accuracy in ways that standard benchmarks fail to reveal.

To confront the problem, the team led by Zhiyong Li, with Wei Jiang as corresponding author, alongside Haojie Liu, Mingyu Wang and Weijie Mao, proposes a Robust Modality Unified Learning framework, abbreviated RMUL. The framework rests on two cooperating components. The first, Robust Cross-modality Label Transfer, or RCLT, changes how pseudo-labels are propagated between modalities. Rather than unifying labels across the entire dataset, RCLT unifies cross-modality labels only for what the authors call modality-shared instances, meaning images whose identities genuinely have counterparts in both the visible and infrared domains. For modality-specific instances, identities that exist in only one modality, the method deliberately retains the original within-modality labels instead of forcing a cross-modality match that does not exist.

This distinction between shared and specific samples flows into the second component, the Modality Unified Learning module, or MUL, which operates during the training stage. Instead of discarding modality-specific samples as noise or treating them identically to shared ones, MUL trains on both populations jointly, allowing the network to extract the shared identity structure that crosses the modality boundary while still learning from the distinctive appearance information carried by modality-specific data. In effect, the framework teaches the model which comparisons are safe to make across the visible-infrared divide and which must remain within a single modality, preventing corrupted associations from poisoning the learned feature space.

A key technical element binding the two components together is a novel objective the authors call the Intra-Cross Prediction Alignment Loss. Memory-based methods, which are the dominant paradigm in unsupervised re-identification, maintain feature memories that store a running representation of each pseudo-identity. In a cross-modality setting there are typically separate memories for each modality alongside a unified cross-modality memory. The new loss aligns the shared cross-modality memory with the intra-modality memories, ensuring that predictions derived from the unified representation remain consistent with predictions drawn from each modality’s own representation. This consistency mechanism acts as a stabilizer, keeping the shared memory anchored to reliable within-modality evidence even when the identity structure is imperfectly aligned across domains.

The experimental evaluation is notable for its breadth. The authors report comprehensive results and component-wise analyses on standard benchmark datasets, including SYSU-MM01, RegDB and LLCM, the three canonical datasets for visible-infrared person re-identification. Crucially, the team evaluated their framework both under the default, fully aligned setting that previous work has used and under artificially induced identity-misalignment settings that simulate the missing-identity conditions of real unlabeled data. Comparisons against state-of-the-art methods show that the proposed approach not only achieves superior performance but also demonstrates markedly greater robustness as the degree of identity misalignment increases, precisely the scenario in which earlier methods deteriorate. Ablation studies, contributed to in part by co-author Mingyu Wang, confirm that both the RCLT and MUL components are necessary for the observed gains, indicating that robust label transfer and joint modality-aware training operate synergistically rather than as interchangeable tricks.

The significance of the work extends beyond a single benchmark. As cities deploy ever denser networks of mixed visible and thermal cameras, the cost of hand-labeling identities grows untenable, making unsupervised approaches the practical path forward for night-capable surveillance, search-and-rescue operations and smart-campus security. But unlabeled data collected automatically will always carry the messy realities of camera placement and human movement, including the partial modality overlap that the Zhejiang team has now formalized. By identifying identity misalignment as a distinct research problem and designing a framework that tolerates it by construction rather than assuming it away, the researchers provide a template for building re-identification systems that survive contact with real deployments.

The study also connects to a broader trend in machine learning research on noisy correspondence, where pairs of data points assumed to be related turn out not to be, and on noise-tolerant training across modalities such as audio-visual recognition and cross-modal retrieval. The authors acknowledge support from the Zhejiang Province Natural Science Foundation of China under Grant LZ24F030004. The datasets used in the study, SYSU-MM01, RegDB and LLCM, are publicly available for academic research, lowering the barrier for other groups to reproduce the identity-misalignment experiments and to extend the robust-by-design philosophy to adjacent problems in biometrics, computer vision and image retrieval.

Subject of Research: Identity misalignment in unsupervised visible-infrared person re-identification using a robust modality unified learning framework.

Article Title: Exploring identity misalignment in unsupervised visible-infrared person re-identification

Article References: Li, Z., Jiang, W., Liu, H., Wang, M., & Mao, W. (2026). Exploring identity misalignment in unsupervised visible-infrared person re-identification. International Journal of Machine Learning and Cybernetics, 17(10), Article 474. https://doi.org/10.1007/s13042-026-03310-2

Image Credits: AI Generated

DOI: 10.1007/s13042-026-03310-2

Keywords: person re-identification, visible-infrared matching, unsupervised learning, identity misalignment, cross-modality retrieval, computer vision, intelligent security, label transfer noise, machine learning, biometrics, image retrieval, robust modality unified learning