From Referring to Reasoning: How AI Learns to Segment Video Objects You Describe in Words

A new open-access survey maps one of the most consequential shifts now underway in computer vision: the move from machines that can find what you explicitly describe to machines that can work out what you actually mean. Published in the Springer journal Vicinagearth by Yiqing Shen of the Institute of Artificial Intelligence (TeleAI), China Telecom, and Johns Hopkins University together with Dell Zhang, the review traces the evolution of language-guided video object segmentation from Referring Video Object Segmentation (RVOS) to its far more demanding successor, Reasoning Video Object Segmentation (ReasonVOS). The distinction sounds academic, but it marks the difference between a system that can outline “the person in the red shirt” and one that can outline “the object that will protect you from the rain” — even when that object is an umbrella it has never been shown before.

Video object segmentation itself is a deceptively simple task to state: given a video, produce pixel-level masks that consistently track a target object across every frame. Traditional approaches fall into semi-supervised methods, which propagate a user-provided mask from an initial frame using memory networks or feature matching, and unsupervised methods, which latch onto salient objects through motion analysis and visual attention. The trouble, as the survey makes clear, is that these frameworks rely entirely on visual context. In a surveillance clip containing several people dressed similarly, a conventional system asked to find “the person carrying the red backpack” has no mechanism for understanding the instruction at all — it will simply segment whatever looks most prominent, regardless of intent.

RVOS emerged to close that gap by accepting a natural language expression as an additional input signal. Formally, given a video of T frames and a text query Q, an RVOS model must generate a binary mask for each frame that delineates the referenced instance. Two challenges define the field: cross-modal alignment, meaning the accurate association of words with visual features, and consistent spatio-temporal tracking across appearance changes, occlusions, and complex motion. Architecturally, the field has moved through three phases. Early systems paired convolutional encoders such as ResNet or VGGNet with recurrent text encoders — LSTMs or GRUs — fusing the two streams by broadcasting a fixed sentence embedding across spatial locations. That design struggled with linguistic structure and long-range spatial relationships alike.

The transformer era changed the calculus. MTTR reformulated RVOS as a sequence prediction problem, feeding video frames and the text query jointly through a single multimodal transformer, while ReferFormer introduced the influential “language as queries” paradigm, in which the referring expression generates a small set of object queries that attend selectively to spatio-temporal features and are converted into dynamic convolution kernels for mask generation. More recently, foundation models have entered the picture. The Segment Anything Model 2 (SAM2), with its memory-based architecture providing inherent spatio-temporal objectness, has become a common backbone; frameworks such as SOLA use SAM2’s object token representations as compact video-level features aligned with language embeddings through lightweight selection modules, and SAMWISE adapts SAM2 for streaming, text-driven segmentation while retaining context from past frames.

Yet RVOS has a hard ceiling: it presumes the query describes directly observable attributes. A model built for visual-linguistic matching cannot resolve “the item that caused the accident,” “the eco-friendly vehicle,” or “the cup that was used most frequently.” These demand multi-step inference and external world knowledge — possibly even knowledge absent from the video itself, as when an electric and a gasoline car are visually indistinguishable. The survey organizes the reasoning demanded by ReasonVOS along several dimensions: functional reasoning about affordances, causal reasoning that traces effects back to causes, spatial reasoning involving ordering and counting, temporal reasoning that aggregates behavior across segments, and world knowledge integration that reaches beyond the pixels on screen.

The dominant architectural answer is the multimodal large language model. VISA, the survey’s exemplar of direct fine-tuning, combines an MLLM’s reasoning capacity with a dedicated segmentation decoder: a Text-guided Frame Sampler selects the frames most relevant to the query, the MLLM reasons over them alongside the text, and a special token whose hidden embedding prompts a SAM decoder produces the masks. VideoLISA extends the LISA image-segmentation system to video through sparse-dense sampling — dense frames at full resolution for spatial detail, down-sampled interleaved frames for temporal context — and a One-Token-Seg-All approach built around a token for consistent tracking. Alternative strategies instead integrate frozen foundation models with lightweight adaptation modules: ViLLa adds temporal-aware context aggregation and a video-frame decoder, a just-in-time digital twin framework uses an LLM planner that requests only the specific perceptual information it needs from specialist vision models, and VRS-HQ injects hierarchical frame-level and temporal-level tokens into an MLLM to overcome the spatial limits of single-token designs.

Reasoning itself comes in two flavors. Chain-of-thought methods decompose complex queries into sequential inference steps; ThinkVideo, for example, is a training-free multi-agent system that uses chain-of-thought prompts over closed-source MLLMs to extract object selectivities at keyframes, then bridges to SAM2 for full mask sequences. Multi-step inference frameworks instead dispatch specialized components — agent-based planners, conversational memory, iterative refinement — each handling a different aspect of the reasoning chain. Keeping all of this temporally coherent remains the hardest engineering problem, and the survey catalogs three families of solutions: memory-augmented architectures that maintain persistent feature banks, hierarchical temporal processing that organizes information at multiple time scales, and propagation-based consistency that tracks masks from reasoning-identified keyframes across the remaining frames.

Measurement turns out to be a quiet crisis of its own. Standard RVOS metrics — Region Jaccard (IoU), Boundary F-measure, and their average J&F — can be actively misleading in the reasoning setting. A model asked for “the object that will protect you from the rain” might perfectly outline a hat instead of an umbrella, earning high boundary accuracy while failing the reasoning entirely. Reasoning-aware metrics such as cIoU, which assigns zero when the segmented object belongs to the wrong semantic category, and gIoU, which averages per-sample IoU, address this directly, while GPT-based evaluation uses large models as judges of semantic appropriateness and mask consistency scores quantify temporal stability. The benchmark landscape is evolving in parallel: from small proof-of-concept sets like Ref-DAVIS16 with its 50 clips and 1,200 queries, through the gold-standard Refer-YouTube-VOS with 3,978 videos and roughly 15,000 queries, to motion-centric MeViS, long-video Long-RVOS with clips averaging over 60 seconds, interaction-heavy InterRVOS-8K with 127,314 queries generated partly by GPT-4o and LLaMA-70B, and the reasoning-era benchmarks ReVOS, ReasonVOS, VideoReasonSeg, and GROUNDMORE, whose query types span causal, sequential, counterfactual, and descriptive reasoning.

The survey is candid about the field’s remaining weaknesses. Evaluation still lags behind the task: LLM-as-judge metrics add computational cost and potential bias, and no standardized protocol yet measures spatial accuracy and reasoning correctness together. Benchmarks remain overwhelmingly English-centric, and performance gaps for non-Western cultural contexts and other languages are well documented. Online processing risks error accumulation that compounds into catastrophic tracking failures over long sequences, polysemous queries such as “the player making the shot” confound attention-based fusion when several candidates fit, and the computational overhead of hierarchical temporal processing limits deployment on resource-constrained hardware. The authors point to unified multimodal architectures, language-agnostic reasoning, and dynamic integration of external knowledge sources as the most promising research frontiers.

The application horizon, however, is what makes this survey worth attention well beyond the vision community. Autonomous driving systems that segment objects by function rather than appearance could interpret complex traffic scenes through ordinary language, a safety-critical capability. Surgical guidance and medical imaging could exploit the ability to isolate anatomical structures from functional descriptions in real time. Educational platforms could build interactive environments that respond to conceptual student queries by highlighting exactly the relevant objects on screen. The trajectory the authors describe — multimodal LLMs fusing with foundation segmentation models into systems that both reason and perceive at pixel level — sketches a future in which the boundary between describing a video and understanding it begins, finally, to dissolve.

Subject of Research: Language-guided video object segmentation, from referring expressions to reasoning-based segmentation with multimodal large language models

Article Title: A survey of language-guided video object segmentation: from referring to reasoning

Article References: Shen, Y., & Zhang, D. (2025). A survey of language-guided video object segmentation: from referring to reasoning. Vicinagearth, 2(1), Article 9. https://doi.org/10.1007/s44336-025-00018-9

Image Credits: AI Generated

DOI: 10.1007/s44336-025-00018-9

Keywords: video object segmentation, RVOS, ReasonVOS, multimodal large language models, computer vision, SAM2, cross-modal alignment, chain-of-thought reasoning, benchmark datasets, temporal consistency, visual grounding, world knowledge