Source-linked AI summary
The Devil is in Temporal Token: High Quality Video Reasoning Segmentation
Sitong Gong, Yunzhi Zhuge, Lu Zhang, Zongxin Yang, Pingping Zhang, Huchuan Lu
TL;DR
Video reasoning segmentation must capture temporal relations and attributes, but existing methods use a single token with limited temporal context and can rely on inaccurate keyframe detection and decoupled segmentation and propagation. VRS-HQ combines hierarchical MLLM tokens, similarity-weighted fusion, and SAM2-based keyframe selection and mask propagation; it reports state-of-the-art results, including a 9.1% J &F gain over VISA-13B on ReVOS.
Problem
Existing VRS methods rely on a single MLLM segmentation token, limiting temporal context, and can have inaccurate keyframe detection and decoupled segmentation and propagation.
Source: §1 · p. 1Method
VRS-HQ uses frame-level <SEG> and temporal <TAK> tokens, similarity-weighted fusion, token- and occlusion-guided keyframe selection, and SAM2 for mask generation and propagation.
Source: §1 · p. 2Results
VRS-HQ achieves state-of-the-art across video segmentation benchmarks and beats VISA-13B by 9.1% in J &F on ReVOS.
Source: §1 · p. 2Takeaways & Limitations
The reported experiments demonstrate strong capabilities in handling Video Reasoning Segmentation.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Existing methods for Video Reasoning Segmentation rely heavily on a single special token to represent the object in the keyframe or the entire video, inadequately capturing spatial complexity and inter-frame motion. To overcome these challenges, we propose VRS-HQ, an end-to-end video reasoning segmentation approach that leverages Multimodal Large Language Models (MLLMs) to inject rich spatiotemporal features into hierarchical tokens.Our key innovations include a Temporal Dynamic Aggregation (TDA) and a Token-driven Keyframe Selection (TKS). Specifically, we design frame-level <SEG> and temporal-level <TAK> tokens that utilize MLLM's autoregressive learning to effectively capture both local and global information. Subsequently, we apply a similarity-based weighted fusion and frame selection strategy, then utilize SAM2 to perform keyframe segmentation and propagation. To enhance keyframe localization accuracy, the TKS filters keyframes based on SAM2's occlusion scores during inference. VRS-HQ achieves state-of-the-art performance on ReVOS, surpassing VISA by 5.9%/12.5%/9.1% in J&F scores across the three subsets. These results highlight the strong temporal reasoning and segmentation capabilities of our method. Code and model weights will be released at VRS-HQ.
1. Introduction
Reasoning segmentation has advanced for images, while video-level segmentation requires temporal reasoning about object relations and attributes; VRS addresses implicit intent-based expressions. Existing VRS methods still face limited temporal context, inaccurate keyframe detection, and decoupled segmentation and propagation.
1. Introduction
- Video Reasoning Segmentation uses MLLMs’ world knowledge and temporal reasoning to turn implicit intent-based expressions into precise object masklets.Unlike RVOS, it does not rely on explicit descriptive phrases such as “a person skateboarding.”Source: §1 · p. 1
- Existing VRS methods rely on a single MLLM segmentation token, which limits temporal context; VISA also has keyframe-detection errors and decoupled segmentation and propagation.These issues can hinder inter-frame feature capture, keyframe localization, and end-to-end inference.Source: §1 · p. 1
- VRS-HQ uses frame-level <SEG> and temporal-level <TAK> tokens, fusing frame embeddings by cosine similarity to combine spatial features with temporal consistency.The MLLM generates the multi-level tokens autoregressively from sampled video frames and tailored prompts.Source: §1 · p. 2
- During training, VRS-HQ selects the frame whose <SEG> is most similar to <TAK>; at inference, SAM2 occlusion and token-similarity scores guide selection, then SAM2 generates and propagates masks.The authors also fine-tune SAM2 with the temporal token and its memory mechanism for improved mask quality and inference efficiency.Source: §1 · p. 2
- VRS-HQ achieves state-of-the-art across video segmentation benchmarks, beating VISA-13B by 9.1% in J &F on ReVOS and improving J &F by 7.3%/5.6%/6.5% on three standard RVOS datasets.The reported results highlight performance on reasoning and referring video segmentation.Source: §1 · p. 2
2. Related Works
Related work distinguishes referring video object segmentation, which uses explicit descriptions, from reasoning segmentation, which handles complex reasoning and versatile prompts. VISA brings reasoning segmentation to video but is described as limited by single-token representation, keyframe selection, and inference design.
2.1. Referring Video Object Segmentation
2.2. Reasoning Segmentation
- Reasoning segmentation generates masks from complex images and versatile prompts; LISA introduced a new token and embedding-as-mask paradigm, while LISA++ expanded instruction-tuning data for semantic and instance segmentation.The passage also describes PixelLM’s pixel decoder and segmentation codebook approach.Source: §2.2 · p. 3
- VISA extends reasoning segmentation to video by selecting keyframes with an MLLM, segmenting them, and propagating masks with a pre-trained tracker, but its single token and keyframe selection are limitations.The related-work passage also notes that separate pre-trained models prevent end-to-end inference.Source: §2.2 · p. 3
3. Methodology
VRS maps high-level instructions and video clips to frame-by-frame masks, requiring temporal reasoning beyond explicit object descriptions. VRS-HQ combines hierarchical MLLM tokens, token fusion and selection, and SAM2 segmentation and propagation.
3. Methodology
- VRS transforms a video clip and high-level instruction into a binary segmentation mask for each frame, including expressions involving world knowledge or temporal logic.Unlike referring video segmentation, its instructions need not explicitly describe the target object.Source: §3 · p. 3
3.1. Temporal Token Encoding
- The MLLM autoregressively produces frame-level <SEG> tokens and a temporal-level <TAK> token from the prompt and sampled video frames.Their embeddings are extracted and projected into SAM2’s feature space; frame-level embeddings are then aggregated with the video-level embedding.Source: §3.1 · p. 4
3.2. Temporal Dynamic Aggregation
- Temporal Dynamic Aggregation fuses frame-level <SEG> embeddings with the video-level <TAK> embedding using normalized cosine similarities as weights.This fusion combines positional and semantic target information and provides an embedding for keyframe selection and mask generation.Source: §3.2 · p. 4
3.3. Token-driven Keyframe Selection
- Token-driven Keyframe Selection combines token similarity with SAM2 occlusion scores to select a keyframe, whose mask is propagated across frames by SAM2.The occlusion scores indicate confidence in the target’s presence in each sampled frame.Source: §3.3 · pp. 4–5
3.4. Mask Decoding and Propagation
- SAM2 uses the fused temporal embedding with keyframe features to decode a keyframe mask, then propagates it to remaining frames.The method uses SAM2 for segmentation and propagation concurrently.Source: §3.4 · p. 5
3.5. Training Objectives
- End-to-end training combines text-generation and mask losses to optimize the <TAK> and <SEG> embeddings.The mask loss combines binary crossentropy and DICE loss; the text, mask, BCE, and DICE weights are 1, 1, 2, and 0.5, respectively.Source: §3.5 · p. 5
4. Experiments
Experiments compare VRS-HQ with prior methods on video and image segmentation benchmarks, then examine its design choices and qualitative outputs.
4.1. Datasets and Metrics
- Experiments use image and video segmentation datasets plus VideoQA data, and evaluate video with J, F, J&F, and R and images with gIoU and cIoU.Chat-UniVi is LoRA-finetuned for 7,500 iterations on a hybrid image/video dataset; evaluation spans ReVOS, RVOS, and image segmentation benchmarks.Source: §4.1 · p. 5, §4.2 · p. 5, §4.3 · p. 5
4.3. Comparison Results
- VRS-HQ-13B surpasses VISA-13B by 5.9% J&F on ReVOS referring and 12.5% J&F on reasoning subsets.The ReVOS comparison also reports a 4.4% advantage in robustness score R over VISA-13B.Source: §4.3 · pp. 5–6, Table 1 · p. 6
- VRS-HQ-13B improves J&F by 7.3% on Ref-YouTube-VOS and 5.6% on Ref-DAVIS17; on MeViS, gains are 6.2% J, 6.1% F, and 6.4% J&F.The reported comparisons are against VideoLISA and VISA-13B; the authors attribute the improvements to inter-frame perception and keyframe localization.Source: §4.3 · p. 6, Table 2 · p. 6
- On referring image segmentation, VRS-HQ outperforms VISA and is comparable to LISA and VideoLISA; on ReasonSeg, it ranks below only VideoLISA.The authors suggest its relatively lower image-dataset performance may relate to video-focused training and reliance on SAM2's memory mechanism.Source: §4.3 · p. 6, Table 3 · p. 7
4.4. Ablation Studies
- α=0.1 is optimal; occlusion scoring improves J&F by 1.8%/1.6%, while adding CLIP scoring reduces J&F by 0.2%/0.9% on referring/reasoning subsets.Setting α to zero also reduces J&F from 62.1% to 58.8% on referring and from 56.1% to 54.1% on reasoning.Source: §4.4 · p. 7, Table 4 · p. 7, Table 5 · p. 7
- Multi-frame SAM2 training and inference yield the best overall results; CLIP frame sampling beats uniform sampling by 0.5%/0.3% J&F on referring/reasoning subsets.The multi-frame strategy leverages SAM2's memory to store keyframe masks for propagation.Source: §4.4 · pp. 7–8, Table 6 · p. 7, Table 7 · p. 7
4.5. Visualization Analysis
- In qualitative ReVOS examples, VRS-HQ resolves spatial references, identifies a kite rather than a person, and succeeds where VISA misses the target.The figure compares the methods across three scenarios.Source: §4.5 · p. 8, Figure 3 · p. 8
- After similarity-weighted fusion, the <TAK> token's mask embeddings contain richer positional and semantic information and focus more effectively on target objects.The visualization uses PCA to reduce mask embeddings to one channel and blends that representation with the original image.Source: §4.5 · p. 8, Figure 4 · p. 8
5. Conclusion
VRS-HQ combines MLLM temporal reasoning with SAM2 tracking for video reasoning segmentation. Its conclusion highlights hierarchical tokens, temporal dynamic aggregation, and token-driven keyframe selection; the appendix adds implementation details, analyses, and qualitative examples.
5. Conclusion
- Frame-level <SEG> and temporal-level <TAK> tokens capture spatial features and temporal relations, respectively, and are integrated through Temporal Dynamic Aggregation.SAM2 uses <TAK> for keyframe segmentation and mask propagation.Source: §5 · pp. 8–9
- VRS-HQ achieves state-of-the-art performance across various video segmentation benchmarks.The conclusion also reports strong capabilities in handling Video Reasoning Segmentation.Source: §5 · p. 9
- Token-driven Keyframe Selection uses <TAK>-generated occlusion scores to select keyframes.The conclusion describes this as supporting robust keyframe selection.Source: §5 · p. 9
6. Appendix
- The appendix covers dataset ablations, implementation details, SAM2 mask decoding and propagation, failure cases, comparisons with VISA, and in-the-wild visualizations.It presents these materials as additional detail and analysis extending the main paper.Source: §6 · p. 9
A. Datasets Ablation
The dataset ablation finds the full training mixture performs best, with the ReVOS dataset especially important to reasoning segmentation performance.
A. Datasets Ablation
- The full training datasets yield the best performance; excluding image segmentation, VideoQA, or ReVOS individually causes metric degradation.Table 8 presents the training-dataset ablation.Source: A. Datasets Ablation · p. 9, Table 8 · p. 9
- 0.9% lower J &F on both referring and reasoning subsets follows removal of VideoQA, which primarily supports video comprehension rather than segmentation.The passage characterizes this as a minimal impact.Source: A. Datasets Ablation · p. 9
- 4.4% and 7.6% lower J &F follow removal of ReVOS, highlighting its pivotal role in reasoning segmentation for challenging scenarios.Source: A. Datasets Ablation · p. 9
B. Additional Implementation Details
The implementation details specify video-frame sampling and preprocessing, LoRA settings, and SAM2's role in decoding keyframe masks and propagating them through video.
B. Additional Implementation Details
- SAM2 encodes all video frames, uses keyframe features with h′ tak to decode a mask, then propagates it to remaining frames through memory.Source: Figure 5 · p. 9
- For video segmentation datasets, 8–12 frames are uniformly sampled at fixed intervals per video; during inference, CLIP-336 performs global sampling of up to 12 frames per video.Up to three object categories are selected per image or video.Source: B. Additional Implementation Details · p. 9
- Chat-UniVi receives images resized to 224×224, while SAM2 input is augmented and resized to 1024 × 1024.The passage specifies these as separate input-processing steps.Source: B. Additional Implementation Details · p. 9
- LoRA is applied across the MLLM's query and value projection layers with scaling factor 16 and dropout 0.05.The passage describes this as enabling efficient fine-tuning.Source: B. Additional Implementation Details · p. 9
C. More Details of SAM2
SAM2 decodes a keyframe mask from fused temporal and keyframe features, then propagates masks using stored frame memories. The supplementary material also identifies failure cases involving complex world knowledge, temporal reasoning, and negative samples.
C. More Details of SAM2
- SAM2 combines the fused temporal embedding h′ tak with keyframe features in its mask decoder to generate the segmentation mask.All video frames first pass through the image encoder for feature extraction.Source: C. More Details of SAM2 · p. 9
- A FIFO memory bank stores recent predicted masks; subsequent non-keyframes attend to those features to generate masks and propagate segmentation across frames.Source: C. More Details of SAM2 · p. 9
- The accompanying example queries ask which boat moves fastest to the right and which gorilla appears at the video's end.Source: C. More Details of SAM2 · p. 10
- The failure-case visualization identifies challenges with complex world knowledge, temporal reasoning, and negative samples.Source: Figure 6 · p. 10
D. Failure Case Analysis
VRS-HQ exhibits failures in motion-based keyframe localization, brief target appearances, nuanced prompts, and absent-object scenes. The authors propose several possible improvements, leaving their investigation to future work.
D. Failure Case Analysis
- Motion-based queries can challenge keyframe localization, while targets appearing only in the last two frames can be difficult to segment.The examples include finding the fastest-moving boat and a gorilla visible only in the final two frames.Source: D. Failure Case Analysis · p. 10
- Nuanced or implicit prompts can expose comprehension difficulties, and absent-object scenes can produce hallucinated segmentations.The examples include interpreting “high bar” in gymnastics and generating masks for non-existent objects.Source: D. Failure Case Analysis · p. 10
- The authors hypothesize that improving MLLM video comprehension could help interpret complex scenes and queries.They leave a thorough investigation of this potential improvement to future research.Source: D. Failure Case Analysis · p. 10
- Processing more sampled frames might improve sensitivity to subtle temporal changes and short-duration events; specialized empty-mask tokens might address hallucinations.The authors leave investigation of these potential improvements to future research.Source: D. Failure Case Analysis · p. 10
E. More Qualitative Comparison
The qualitative comparisons show VRS-HQ handling color and spatial variation, difficult temporal dynamics, and prompts requiring general or world knowledge in examples where VISA struggles.
E. More Qualitative Comparison
- VRS-HQ captures inter-frame motion and segments more accurately in examples involving color expressions and visually similar objects with complex spatial variation.The comparison describes VISA as less sensitive to color terms such as “white” and “brown” and prone to misidentifying similar objects.Source: E. More Qualitative Comparison · p. 10
- For late-appearing objects, VRS-HQ uses Token-driven Keyframe Selection and SAM2 with temporal tokens to support decoding and tracking.The examples contrast VISA’s difficulty detecting a late airplane and a tiger emerging in the lower-left corner with VRS-HQ’s more reliable tracking.Source: E. More Qualitative Comparison · pp. 10–11
- In examples requiring world knowledge, VRS-HQ identifies nearly all fish and links “dog” with “common household pet,” unlike VISA.The comparison attributes VRS-HQ’s more nuanced semantic understanding to integration of temporal tokens.Source: E. More Qualitative Comparison · p. 11
F. In-the-wild Visualization Results
Qualitative results show VRS-HQ segmenting targets in first-person GTEA videos with implicit prompts and tracking subjects in panoramic PanoVOS videos with intricate prompts.
F. In-the-wild Visualization Results
- On first-person GTEA videos, VRS-HQ segments small targets such as a spoon and watch in cluttered kitchens with similar objects.The examples use implicit prompts and are presented as evidence of generalization capability.Source: F. In-the-wild Visualization Results · p. 11
- On PanoVOS 360-degree videos, VRS-HQ segments people split across distorted video edges and tracks dynamically moving subjects without task-specific optimization.The examples use more intricate prompts.Source: F. In-the-wild Visualization Results · p. 11