MoVISA: Multi-Token Reasoning for Video Object Segmentation

• A single global <SEG> embedding can bottleneck representation of diverse object appearances in long-range tracking or cluttered scenes.Source: §1 · p. 2
• MoVISA assigns distinct <SEG_n,t> tokens to each object-frame pair, enabling independent frame-object reasoning and mask decoding through a modified SAM2.Source: §1 · p. 2
• The token design is integrated into an end-to-end trainable SAM2 and MLLM framework without external VOS or heuristic post-processing modules.Source: §1 · p. 2
• MoVISA consistently outperforms single-token baselines on Ref-YoutubeVOS, MeViS, and ReVOS.Source: §1 · p. 2

2.1 Multi-Modal Large Language Models

• Video-LLaMA injects frame-wise embeddings, while Video-ChatGPT and LLaMA-VID compress video inputs with potential losses of fine detail or spatial precision.Source: §2.1 · pp. 2–3

2.2 Video Object Segmentation

• Sa2VA [Yuan et al., 2025] integrates SAM2 for memory-based propagation, while LISA, VISA, and Video-LLaVA use shared <SEG> tokens across frames.Source: §2.2 · p. 3
• VRS-HQ uses a temporal token (<TAK>) and keyframe selection to enhance temporal abstraction and occlusion-aware propagation, achieving strong long-form video results; VRS-HQ and GLUS both lack fine-grained frame-level segmentation supervision.Source: §2.2 · p. 3
• MoVISA assigns distinct <SEG> tokens across frames to retain detailed object-specific cues over time while preserving language-driven reasoning flexibility.Source: §2.2 · p. 3

• Video object segmentation predicts binary masks of a language-referred object across all frames of an input video.Source: §3.1 · p. 3
• Referring segmentation uses direct appearance or location prompts, whereas reasoning segmentation uses indirect prompts requiring common-sense reasoning.Source: §3.1 · p. 3
• MoVISA assigns distinct segmentation tokens to each object at each frame, enabling temporally dense supervision and explicit frame-level grounding.Source: §3.1 · pp. 3–4
• The unified objective predicts masks from language, visual input, and dynamically constructed learnable segmentation tokens; images are treated as single-frame videos.Source: §3.1 · p. 4

3.2 Framework

• A two-stream MLLM–SAM2 architecture uses dynamic multi-token prompting for temporally grounded mask prediction at object-frame resolution.The framework is designed for referring image and video object segmentation.Source: §3.2 · p. 4

3.2.1 Visual-Language Backbone

• Video frames are encoded as 256 visual tokens each, while image inputs use dynamic-resolution encoding before visual and language tokens enter the LLM.Video inputs typically use five temporally ordered key frames; the tokenizer supports indexed segmentation tokens.Source: §3.2.1 · p. 4

3.2.2 Special Token Design for Multi-Modal Reasoning

• Each queried object receives one indexed segmentation token per sampled frame, producing K = O · T tokens that prompt downstream mask decoding.The dialogue format generates these object-specific token blocks autoregressively.Source: §3.2.2 · p. 5

3.2.3 Memory-Guided Mask Propagation

• SAM2 decodes each predicted token’s hidden state into a key-frame mask, then propagates masks to non-key frames through memory attention.Distinct object- and frame-specific prompts are used for key-frame segmentation and provide a more informative memory representation than reusing the same segmentation across frames.Source: §3.2.3 · p. 5

3.2.4 Unified Representation Across Modalities

• Images are treated as one-frame videos, with one segmentation token per object, allowing co-training across image and video referring-segmentation tasks.The paper describes this as requiring minimal architectural changes.Source: §3.2.4 · p. 5

3.3 Training

• Training jointly uses image and video segmentation samples with natural-language queries and mask supervision across multiple datasets.Table 1 summarizes the training data and model sizes, without specifying values in the supplied passage.Source: §3.3 · p. 5, Table 1 · p. 6
• The language loss is autoregressive cross-entropy for predicting the ground-truth sequence of indexed segmentation tokens.The model’s predicted token sequence is compared with the ground-truth sequence.Source: §3.3 · p. 6
• The mask loss sums pixel-wise binary cross-entropy and Dice losses over predicted and ground-truth masks.The passage identifies the predicted and ground-truth masks as the inputs to these losses.Source: §3.3 · p. 6
• The instruction objective adds the language and mask losses, and end-to-end training propagates gradients through both the LLM and SAM2 decoder.The stated goals are improving segmentation-token prediction and refining spatio-temporal segmentation.Source: §3.3 · p. 6

3.4 Testing

• The visual encoder and LLM process visual inputs with the referring query, after which the LLM autoregressively generates indexed tokens for object-frame pairs.Inputs may be images or videos.Source: §3.4 · p. 6
• A learned adapter maps each token’s hidden state into SAM2’s prompt space, where key-frame masks are generated, encoded into memory, and propagated to non-key frames.Propagation uses SAM2 memory attention.Source: §3.4 · p. 6

4.1 Training Data

• The 4B video-only configuration uses InternVL-2.5 on MeViS, Ref-YouTube-VOS, and ReVOS; the 1B hybrid configuration adds image segmentation data and Ref-SAV.Evaluation covers Ref-YouTube-VOS, MeViS, Ref-DAVIS17, and ReVOS using J&F, combining region similarity and contour accuracy.Source: §4.1 · p. 6, §4.2 · p. 7

4.3 Implementation Details

• Training used six NVIDIA A6000 GPUs; the 1B video-plus-image setup took roughly 24 hours per epoch, while the 4B video-only setup took approximately 7 hours for 2 epochs.Source: §4.3 · p. 8

4.4 Main Results

• 5.5 points over Sa2VA-1b, while MoVISAV(1B) exceeds Sa2VA-4b by 0.9 J&F on MeViS.These are reported referring-VOS comparisons.Source: Table 2 · p. 7, §4.4 · p. 8
• MoVISAVIR exceeds Sa2VA-1b by 0.4 J&F on Ref-YT-VOS and 0.9 J&F on Ref-DAVIS17; on ReVOS, it beats Sa2VA-1B by 4.0 points and Sa2VA-4b by 0.1 points on J&F.The ReVOS comparison uses significantly less training data than Sa2VA-4b, which includes an additional 56K image data and 37K Ref-SAV data.Source: Table 2 · p. 7, Table 3 · p. 7, §4.4 · p. 8

4.5 Ablation Studies

• Without image segmentation and Ref-SAV data, the 1B model's J&F scores drop by 8 on MeViS, 7.3 on Ref-YT-VOS, and 17.2 on Ref-DAVIS17.The authors associate image data with useful visual knowledge and Ref-SAV's descriptive content with fine-grained visual concept understanding.Source: §4.5 · p. 8, Table 4 · p. 8
• Results improve from single tokens to frame tokens to frame-object tokens, which explicitly model both object and frame variations.The single-token alternative uses one token for the video; frame tokens index frames but do not distinguish objects.Source: §4.5 · p. 8, Table 5 · p. 8
• The generation-based approach performs better than decoder-only instruction training, and qualitative examples show multi-token masks handling ambiguous objects, changing positions, and multiple instances.Decoder-only training supplies the segmentation tokens in the prompt and supervises only mask prediction.Source: §4.5 · p. 9, Table 6 · p. 9, Figure 3 · p. 9

5 Conclusion

• The single-token design limits representational capacity for objects that change across frames; MoVISA studies a multi-token design and reports compelling results on challenging datasets.The authors frame this as the paper’s central contribution.Source: §5 · p. 9
• The authors identify possible benefits for autonomous robots and navigation assistance, alongside misuse risks including mass surveillance and unwanted tracking.They say deployments should be implemented with care.Source: §5 · p. 9
• Because of computational limitations, the study focuses on 1B- and 4B-parameter models; assessing improvements for larger models remains beyond its capabilities.The authors consider larger-model assessment valuable.Source: §5 · p. 9

Claims and limitations

• The checklist answers yes that the abstract and introduction reflect the paper’s contributions and scope, and that the paper discusses its limitations.The limitations discussion is identified as Section 5.Source: NeurIPS Paper Checklist · p. 13

Research scope

• The checklist marks theory, human-subject research, and core-method LLM usage as not applicable.The authors state that the paper has no theoretical results and involves neither human subjects nor non-standard LLM components in method development.Source: NeurIPS Paper Checklist · pp. 14, 18, 19

Reproducibility and experimental details

• The authors report sufficient reproducibility details and code release, and say experimental details and compute information are provided.The checklist points to Sections 4.3 for compute details.Source: NeurIPS Paper Checklist · pp. 14–16

Statistical significance

• The authors do not report statistical significance, citing the cost of experiments and consistency with prior work.The checklist asks about error bars or other significance information.Source: NeurIPS Paper Checklist · p. 15

Ethics and impact

• The checklist answers yes on ethics, reports no direct societal impact, and says the paper poses no high-risk release requiring safeguards.These are the authors’ checklist responses.Source: NeurIPS Paper Checklist · pp. 16–17

Assets

• The authors say they cite papers releasing existing assets and that no new assets are released.They note that specific licenses can be obtained from the asset authors.Source: NeurIPS Paper Checklist · pp. 17–18