The core insight of the new study is that the bottleneck in video search is no longer the accuracy of individual AI models but the lack of a coherent system that stitches them together. Deep multimodal models such as CLIP and its successors have made it possible to align images and text in a shared mathematical space, so that a sentence and a photograph of the same scene end up close together as high-dimensional vectors. Yet, as the authors argue, most research focuses on squeezing extra accuracy out of a single model while ignoring the messy engineering questions: how to chunk hours of raw footage, how to avoid indexing the same person thousands of times, how to keep track of who appeared when and where, and how to serve queries in milliseconds. Their answer is a two-part pipeline that cleanly separates offline indexing from online retrieval.

On the indexing side, the system ingests raw video files and first splits them into roughly ten-minute chunks using FFmpeg, preserving the original encoding so each segment can be processed independently and in parallel. A frame-sampling module then skips redundant frames, since typical surveillance footage recorded at 30 or 60 frames per second offers far more temporal detail than person search actually needs. The sampled frames flow into YOLO11, the latest generation of Ultralytics’ real-time object detector, chosen for its favorable speed-accuracy trade-off, paired with the BoT-SORT tracker, which combines motion cues with appearance features to assign each detected person a persistent identity across frames. On the MOT17 benchmark, BoT-SORT reports 80.5 MOTA and 80.2 IDF1, meaning it rarely confuses one pedestrian with another, a property the authors consider essential for generating temporally coherent metadata.

Detection alone would still flood the database with near-duplicate images, so the pipeline adds a quality-filtering stage that is among the paper’s most practical contributions. Every cropped person image receives a score combining the detector’s confidence with an edge penalty: bounding boxes that touch the frame border, indicating a partially visible subject, are down-weighted. A perceptual hashing step then compares consecutive crops within the same tracked trajectory, discarding any whose hash differs by fewer than a threshold number of bits, a signal that the image is essentially identical to the previous one. What survives is a compact, diverse gallery of high-quality person crops rather than a bloated collection of blurry, truncated, or repetitive snapshots.

Each surviving crop is then transformed into a semantic fingerprint by SigLIP2, a multilingual vision-language encoder that extends the original SigLIP architecture with a sigmoid-based training loss, self-supervised objectives, and careful data curation. Because the model was trained on paired images and text, its visual embeddings live in the same latent space as its text embeddings, which is precisely what allows a typed sentence to retrieve a photograph. The system also computes a second, complementary embedding for each person using InsightFace: the RetinaFace detector localizes the face and its landmarks even under challenging poses and lighting, and the ArcFace recognizer converts the aligned face patch into a highly discriminative identity vector. Both embeddings, along with rich metadata such as source video, timestamps, track identity, and bounding-box coordinates, are stored in a Qdrant vector database, so every search result can be traced back to the exact frame in the original footage.

At query time, the retrieval pipeline mirrors the indexing representation. A user can submit an image, a natural-language description, or both; the same SigLIP2 encoders embed the inputs, guaranteeing that queries and database entries are directly comparable. When both modalities are present, a tunable balance parameter blends the image and text vectors, letting users slide between purely visual similarity and purely linguistic matching. A face-based mode restricts the search to identity embeddings when a face is detectable, and a negative-prompt field excludes unwanted attributes. The back-end runs an approximate nearest-neighbor search with cosine similarity, then consolidates results so that only the single best match per tracked object per video is returned, each accompanied by the retrieved frame, a similarity score, and metadata including camera identifier, first-appearance timestamp, and duration in the scene.

The quantitative centerpiece of the paper is an evaluation on RSTPReid, a benchmark of 20,505 pedestrian images covering 4,101 individuals captured by 15 cameras, each image paired with two text descriptions. The researchers fine-tuned the SigLIP2-SO400M-Patch14-384 checkpoint on a curated mixture of person-centric image-text datasets, including CUHK-PEDES, ICFG-PEDES, RSTPReid itself, an attribute-rich multiview re-identification set, and the video-based TVPReid corpus, totaling roughly 316,000 training pairs. Training used the native contrastive loss on four NVIDIA A100 GPUs on the Cineca Leonardo supercomputer, with early stopping based on validation mean Average Precision. The result: a mean Average Precision of 0.68, comfortably ahead of the best competing value of 0.54 among recent text-based person search methods such as OCDL, UP-Person, RDE, CTGI, WoRA, MARS, MRA, and CONQUER, while Recall@1 remained competitive within one standard deviation of the leaders.

Notably, the gains did not come from inventing new architectures. When the model was fine-tuned only on the three standard datasets used by the comparison methods, it already achieved the best mAP, indicating that the SigLIP2 backbone itself provides a stronger shared embedding space for matching descriptions to pedestrian attributes. Adding the larger, more diverse corpus then improved all metrics further. Ablation experiments on a subset of the Wildtrack multi-camera surveillance dataset underscored the value of the system’s design choices: removing face embeddings dropped Recall@1 from about 51 percent to 33 percent, while replacing quality-scored crops with uniformly sampled ones collapsed mAP from 0.66 to 0.39, showing that both identity cues and careful crop selection matter substantially.

The scalability analysis offers a sober, engineering-minded picture. Processing 128 ten-minute videos, about 256 gigabytes of footage, took roughly 90 minutes on a four-GPU server, with tracking saturating the CPU and encoding bottlenecked by memory traffic and model replication across worker processes rather than raw computation. Extrapolating linearly, the authors estimate the current setup could index about a terabyte of video in six hours, though they are candid that GPU utilization during encoding remains low and that a centralized inference server would be needed for truly efficient large deployments. They also acknowledge limitations: the system is person-centric, does not yet model complex temporal activities or events, and the auxiliary training data included AI-generated descriptions that were only spot-checked rather than fully validated by humans.

Even with those caveats, the study reads as a template for how modern AI systems should be built: not as isolated models chasing leaderboard numbers, but as orchestrated pipelines where detection, tracking, filtering, embedding, metadata, and search reinforce one another. The authors envision extending the framework beyond people to vehicles and other objects, adding temporal and multi-view reasoning, and eventually integrating large language models and knowledge graphs for conversational, explainable video search. If surveillance archives, streaming platforms, and personal devices keep growing at their current pace, systems that turn unwatchable oceans of footage into a simple search box may soon become as unremarkable, and as indispensable, as the web search bar itself.

Subject of Research: A scalable semantic pipeline for video indexing and text-based person retrieval using multimodal vision-language embeddings

Article Title: A scalable and semantic pipeline for efficient video indexing and person retrieval

Article References: Hmaidan, R., Milardo, S., Donato, I., Greco, D., Ingargiola, A., & Reforgiato Recupero, D. (2026). A scalable and semantic pipeline for efficient video indexing and person retrieval. Multimedia Tools and Applications, 85(9), Article 727. https://doi.org/10.1007/s11042-026-21882-7

Image Credits: AI Generated

DOI: 10.1007/s11042-026-21882-7

Keywords: video retrieval, person re-identification, multimodal embeddings, SigLIP2, YOLO11, BoT-SORT, face recognition, vector database, surveillance video, computer vision, text-based person search, RSTPReid

The framework, called CurvBi, short for Curvature-aware Bicomplex, is presented by Thanh Le, Vy Cao, Duong-Khang Au, Trong-Nghia Pham and Bac Le of the University of Science and Vietnam National University in Ho Chi Minh City. Writing in the journal Data Mining and Knowledge Discovery, the authors tackle the task known as multimodal knowledge graph completion: predicting missing links and entities in a graph by exploiting both its symbolic structure and the complementary information carried by textual and visual descriptions of its nodes. Their central insight is that the geometry used to fuse different modalities matters as much as the algebra used to model relationships between them.

To understand why, it helps to look at what existing approaches do. Most multimodal knowledge graph embedding methods treat the problem in two loosely connected stages. First, entity descriptions in each modality are projected into a shared Euclidean vector space, typically through simple concatenation, linear projection or a shallow fusion layer. Then, a relational model such as TransE, RotatE or a quaternion-based embedding scores candidate triples by comparing the vectors. This pipeline is convenient, but it flattens away the very structure that distinguishes modalities from one another. Text embeddings and image embeddings live on manifolds with different intrinsic curvature and different statistical character, and forcing them into a single flat space can blur or destroy the discriminative information they carry.

CurvBi departs from this convention on two fronts simultaneously. On the relational side, it adopts bicomplex embeddings, a representation built in a four-dimensional algebraic space formed by pairing two complex planes into a single number system. Bicomplex numbers are an extension of the complex numbers in which both the real and imaginary components are themselves complex, and their algebra is expressive enough to encode symmetric, asymmetric, inverse and cyclic relationships within one unified scoring function. This matters because real knowledge graphs contain all of these relational patterns at once: a relation like ‘is married to’ is symmetric, ‘is the parent of’ is asymmetric and invertible, and ‘is located in’ often participates in cyclic chains. Prior systems have typically needed separate mechanisms, or careful inductive-bias engineering, to cover this full catalogue of relation types.

The bicomplex formulation builds on a line of work the group has developed over several years, including quaternionic and bicomplex embeddings for temporal knowledge graphs and event prediction. In CurvBi, the four-dimensional algebraic structure plays a second, subtler role: it provides a natural home for the modality embeddings before they are fused. Rather than concatenating a text vector and an image vector end to end, CurvBi treats each modality-specific representation as a structured component within the bicomplex number, preserving the algebraic relationships between components while still allowing the model to learn how much weight each modality deserves for a given entity or relation.

The second front is geometric. Instead of projecting all modality embeddings into flat Euclidean space, CurvBi performs fusion on a Riemannian manifold, measuring the distance between modality-specific representations with the Fisher-Rao distance, a well-studied metric from information geometry that treats probability distributions as points on a curved manifold. Because the Fisher-Rao metric is curvature-aware, similarities between a text embedding and an image embedding are judged in a geometry that reflects their statistical structure rather than their raw coordinate differences. The result, according to the authors, is a fusion step that is geometry-sensitive in a way that Euclidean projection and shallow concatenation simply cannot be, aligning heterogeneous modalities while respecting the distinct manifold each of them inhabits.

This curvature-aware component connects CurvBi to a broader trend in machine learning. Hyperbolic and other non-Euclidean embedding spaces have been shown to represent hierarchies, cycles and scale-free structures far more efficiently than Euclidean space, and methods such as Poincaré embeddings and multicurvature adaptive schemes for temporal knowledge graphs have demonstrated measurable gains. What distinguishes CurvBi is the combination: the same framework unifies the algebraic machinery for relational reasoning and the differential-geometric machinery for modality alignment, so that neither stage of the pipeline is an afterthought bolted onto the other. The manifold structure informs how modalities are brought together, while the bicomplex algebra governs how the fused representation is used to score and complete triples.

The empirical case for the framework rests on four benchmark datasets for multimodal knowledge graph completion. Across all of them, CurvBi achieved consistently competitive results on standard evaluation metrics, including Hits at 1, Hits at 10 and mean reciprocal rank, which measure how often the correct missing entity appears at the top of the model’s ranked predictions. On certain benchmark settings the improvements were substantial: gains of up to 6 percent in Hits at 1 and 4.5 percent in mean reciprocal rank over state-of-the-art baselines. The authors also report that the framework scales well and generalizes across datasets with different sizes, relational vocabularies and modality configurations, an important practical consideration given that real multimodal knowledge graphs vary enormously in how much image and text data they attach to each entity.

The significance of the work extends beyond a leaderboard. Multimodal knowledge graphs are increasingly used in settings where text alone is ambiguous: in biomedical informatics, where molecular structures and microscopy images supplement gene and disease ontologies; in e-commerce, where product images carry information absent from catalog text; and in commonsense reasoning systems, where visual grounding helps language models resolve references. If the fusion geometry is wrong, the extra modalities can actively hurt, a phenomenon earlier studies have documented when visual context proved unhelpful for certain graph completion tasks. CurvBi’s results suggest that a principled geometric treatment of fusion, paired with an algebraically expressive relational model, is one way to make multimodal information reliably additive rather than subtractive.

The research, funded by the Vietnam National Foundation for Science and Technology Development under grant number 102.05-2025.75, also signals a maturing of geometric deep learning as applied to structured knowledge. Rather than treating curvature, complex numbers and manifold optimization as exotic mathematical decorations, the authors deploy them as load-bearing components of a system that demonstrably outperforms flatter alternatives. As knowledge graphs continue to absorb richer and more heterogeneous data streams, the lesson from CurvBi is likely to resonate: the shape of the space in which artificial intelligence reasons is not a technicality, but a first-class design decision that determines what that intelligence can and cannot see.

Subject of Research: Curvature-aware multimodal knowledge graph completion using bicomplex embeddings and Riemannian manifold fusion

Article Title: Learning curvature-aware multimodal representations with bicomplex embeddings

Article References: Le, T., Cao, V., Au, D.-K., Pham, T.-N., & Le, B. (2026). Learning curvature-aware multimodal representations with bicomplex embeddings. Data Mining and Knowledge Discovery, 40(5), Article 89. https://doi.org/10.1007/s10618-026-01257-0

Image Credits: AI Generated

DOI: 10.1007/s10618-026-01257-0

Keywords: knowledge graph completion, multimodal embeddings, bicomplex numbers, Riemannian geometry, Fisher-Rao distance, modality fusion, link prediction, representation learning, curvature-aware learning, quaternion algebra, data mining, artificial intelligence