Source-linked AI summary
VISA: Reasoning Video Object Segmentation via Large Language Models
Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, Efstratios Gavves
TL;DR
Existing VOS systems generally require explicit categories, masks, or short phrases rather than implicit instructions requiring world knowledge and video-context reasoning. The paper introduces ReasonVOS, VISA, and the ReVOS benchmark, and reports effectiveness across reasoning and referring segmentation in video and image domains.
Problem
Existing VOS systems rely on explicit categories, masks, or short phrases, limiting segmentation from implicit instructions requiring world knowledge and video-context reasoning.
Method
The paper introduces ReasonVOS and VISA, which combine complex text reasoning with long-term video understanding and mask decoding, supported by the ReVOS benchmark.
Results
Experiments on eight datasets show VISA enables reasoning segmentation in video and image domains and generally provides state-of-the-art or competitive performance on referring segmentation.
Takeaways & Limitations
ReasonVOS extends video object segmentation from explicit references to implicit queries requiring reasoning and inference within video contexts.
Takeaways & Limitations
VISA has limited ability to capture very small objects and can miss objects appearing in few frames or requiring extremely long temporal correspondence.
Abstract
from arXiv · showhide
Existing Video Object Segmentation (VOS) relies on explicit user instructions, such as categories, masks, or short phrases, restricting their ability to perform complex video segmentation requiring reasoning with world knowledge. In this paper, we introduce a new task, Reasoning Video Object Segmentation (ReasonVOS). This task aims to generate a sequence of segmentation masks in response to implicit text queries that require complex reasoning abilities based on world knowledge and video contexts, which is crucial for structured environment understanding and object-centric interactions, pivotal in the development of embodied AI. To tackle ReasonVOS, we introduce VISA (Video-based large language Instructed Segmentation Assistant), to leverage the world knowledge reasoning capabilities of multi-modal LLMs while possessing the ability to segment and track objects in videos with a mask decoder. Moreover, we establish a comprehensive benchmark consisting of 35,074 instruction-mask sequence pairs from 1,042 diverse videos, which incorporates complex world knowledge reasoning into segmentation tasks for instruction-tuning and evaluation purposes of ReasonVOS models. Experiments conducted on 8 datasets demonstrate the effectiveness of VISA in tackling complex reasoning segmentation and vanilla referring segmentation in both video and image domains. The code and dataset are available at https://github.com/cilinyan/VISA.
1. Introduction
The paper introduces ReasonVOS for segmenting and tracking objects from implicit, reasoning-intensive video instructions, and proposes VISA to combine multimodal-LLM reasoning with video mask generation. It also contributes ReVOS and reports strong performance across reasoning and referring segmentation tasks.
- Task and motivation: Existing video object segmentation systems primarily use explicit categories, masks, or short descriptive phrases, limiting complex intention and commonsense reasoning.The paper contrasts instructions such as “Find my favorite cup” with explicit appearance-and-location descriptions.
- Task and motivation: ReasonVOS generates binary mask sequences from complex implicit text instructions requiring world knowledge and long-term video understanding.The task targets object localization in dynamic videos for embodied-AI interaction.
- VISA: VISA combines long-term video-feature encoding with spatial-detail preservation, using text-guided frame sampling, multimodal-LLM reasoning, and a mask decoder.Selected frames and text are processed together; a special <SEG> token produces masks for selected frames before propagation.
- Dataset: ReVOS contains 35,074 instruction-mask sequence pairs from 1,042 diverse videos and targets reasoning beyond the explicit phrases in traditional referring-video datasets.The benchmark supports instruction tuning and evaluation for ReasonVOS models.
- Results: Experiments on ReVOS and seven existing datasets show VISA supports reasoning segmentation in video and image domains while remaining competitive on referring segmentation.Figure 2 summarizes performance across reasoning and referring segmentation tasks using J, F, and R metrics.
2. Related Work
Prior VOS methods segment objects using predefined categories, masks, or explicit text, while multimodal-LLM research extends language reasoning to visual tasks. The paper positions ReasonVOS and VISA as an extension toward complex, temporally grounded video instructions.
- Video Object Segmentation: VOS methods are organized around category-based, mask-based, and text-based references for segmenting and tracking objects in videos.Category-based methods use predefined class lists, whereas mask-based methods use segmentation masks.
- Video Object Segmentation: Referring VOS uses short text phrases describing explicit actions, locations, or appearances, but does not handle sentences requiring commonsense reasoning or video-based inference.ReasonVOS extends these references to complex sentences.
- Multi-Modal Large Language Model: Multimodal LLMs have been developed to connect language reasoning with visual inputs through cross-attention, visual encoders, grounding, and instruction-tuned interfaces.The related work includes Flamingo, BLIP-2, mPLUG-OWL, VisionLLM, Kosmos-2, and DetGPT.
- Video Multi-Modal Large Language Model: Video multimodal LLMs face token and sequence-length challenges when representing long videos, motivating VISA’s selection of significant frames without spatial pooling or projection.The selected frames are processed with many visual tokens to preserve information useful for segmentation.
3. Method
VISA defines ReasonVOS as producing binary mask sequences for complex video instructions and combines text-guided frame selection, multimodal reasoning, mask decoding, and tracking. It trains with text-generation and segmentation losses and evaluates on ReVOS and established segmentation datasets.
- 3.1. Task Setting: ReasonVOS maps high-level instructions requiring world knowledge or video inference to binary mask sequences for the described object.Unlike referring VOS, its queries may require common-sense reasoning or inference about upcoming events.
- 3.2. Architecture of VISA: VISA selects a target frame and reference frames with a text-guided sampler, then feeds them with the instruction to a multimodal LLM.The sampler reduces visual-token demands while retaining selected spatially detailed frames for reasoning over long videos.
- 3.2. Architecture of VISA: The LLM’s <SEG> embedding is projected into a mask-decoder prompt that segments the target frame, after which an object tracker propagates the mask bidirectionally across the video.This separates reasoning and target-frame segmentation from sequence-wide mask propagation.
- 3.2. Architecture of VISA: VISA is trained end-to-end with a weighted sum of autoregressive text-generation loss and segmentation mask loss.The mask loss combines per-pixel binary cross-entropy and DICE loss.
- 3.3. ReVOS Dataset: ReVOS contains 35,074 object-instruction pairs from 1,042 videos, including implicit reasoning instructions, explicit referring descriptions, and nonexistent-object cases.The benchmark supports ReasonVOS evaluation, traditional referring-VOS generalization, and hallucination evaluation using J &F and robustness score R.
4. Experiments
VISA is evaluated on reasoning, referring, image, and ablation settings, showing strong performance across segmentation tasks while revealing dependencies on training data, frame selection, and visual-token capacity.
- Comparison: Over 20 J &F improvements are reported for VISA(IT)-7B over traditional methods on ReVOS reasoning segmentation.VISA(IT)-7B also exceeds LISA-7B by 6.0 J &F overall and TrackGPT(IT)-7B by 3.3 J &F overall.
- Comparison: VISA achieves SOTA results on three referring VOS datasets and comparable performance with LISA on three referring image-segmentation datasets.On ReasonSeg, it significantly outperforms traditional methods by over 20%.
- Ablation Studies: Using the TFS-selected target frame ftgt generally outperforms segmenting the first frame f0 by around 2% across settings.The ablation attributes this improvement to selecting significant moments related to the text query.
- Ablation Studies: Global-Local reference-frame sampling slightly outperforms separate Global or Local sampling, and the adopted configuration yields 4.3% J &F improvement on ReVOS.The model adopts Tr=12 reference frames; 256 and 112 visual tokens perform comparably, whereas 52 tokens significantly reduce performance, so L=112 is selected.
- Limitations: VISA remains limited on very small objects and on objects appearing in only a few frames because visual-token capacity and frame selection constrain spatial and temporal information.The paper notes that processing more tokens increases computational burden, while selecting only a few frames can miss long-term temporal correspondences.
5. Conclusion
The paper introduces ReasonVOS and VISA, supported by the ReVOS dataset, and reports broad effectiveness across reasoning and traditional segmentation tasks.
- ReasonVOS generates object mask sequences from text queries requiring complex reasoning and inference within video contexts.
- VISA combines multimodal-LLM world knowledge and reasoning with video object segmentation and tracking capabilities.
- ReVOS contains 35,074 expression-mask pairs from 1,042 videos for instruction tuning and evaluation.
- Experiments on eight datasets show VISA supports reasoning segmentation in videos and generally achieves state-of-the-art performance on traditional video and image segmentation tasks.
A. Visualizations of Annotated Frames
The ReVOS visualizations show annotated videos paired with reasoning-oriented object-selection questions and descriptions.
- Example queries identify targets through comparative physical reasoning, such as which walrus loses the most gravitational potential energy or which dog experiences more external force.
- Other queries use temporal, predictive, behavioral, and semantic criteria, including the last car to appear, likely race winner, hunting readiness, likely death, and trained objects.
- The visualizations present sample annotated videos from the ReVOS dataset.