PaperlayerView exact revision on arXiv

Source-linked AI summary

Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos

Haobo Yuan, Xiangtai Li, Tao Zhang, Yueyi Sun, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, Ming-Hsuan Yang

arXiv:2501.04001v4cs.CV

TL;DR

Sa2VA addresses the absence of a unified model for dense grounded understanding across image and video modalities and diverse tasks. It combines an MLLM with SAM-2 through single-stage instruction tuning and reports strong performance across benchmarks, including a substantial Ref-SAV gain. The framework is also designed to accommodate different modern MLLM backbones.

  • Problem

    Existing models do not unify open-ended language understanding and dense visual grounding across both image and video inputs and diverse tasks.

  • Method

    Sa2VA combines SAM-2 and an MLLM in an end-to-end framework that represents multimodal inputs as tokens and uses [SEG] outputs to generate grounded masks.

  • Results

    Sa2VA achieves strong performance across image and video tasks, including over 15% gain over existing approaches on Ref-SAV in zero-shot testing.

  • Takeaways & Limitations

    The framework provides a unified dense-grounding model that supports multiple tasks and can be extended with modern MLLM backbones.

  • Takeaways & Limitations

    Long videos with complex referring expressions remain challenging because Sa2VA operates online without access to the entire video content.

Abstract

from arXiv · show

This work presents Sa2VA, the first comprehensive, unified model for dense grounded understanding of both images and videos. Unlike existing multi-modal large language models, which are often limited to specific modalities and tasks, Sa2VA supports a wide range of image and video tasks, including referring segmentation and conversation, with minimal single-stage instruction tuning. Sa2VA combines SAM-2, a foundation video segmentation model, with MLLM, advanced vision-language models, and unifies text, image, and video into a shared LLM token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2 in producing precise masks, enabling a grounded, multi-modal understanding of both static and dynamic visual content. Additionally, we introduce Ref-SAV, an auto-labeled dataset containing over 72k object expressions in complex video scenes, designed to boost model performance. We also manually validate 2k video objects in the Ref-SAV datasets to benchmark referring video object segmentation in complex environments. Experiments show that Sa2VA achieves strong performance across multiple tasks, particularly in referring video object segmentation, highlighting its potential for complex real-world applications. In addition, Sa2VA can be easily extended into various MLLMs, including Qwen-VL and Intern-VL, which can be updated with rapid process in current open-sourced MLLMs. Code and models have been provided to the community.

1 Introduction

Sa2VA addresses the lack of a unified model for open-ended language understanding and dense visual grounding across images and videos. It combines SAM-2 and MLLMs through single-stage instruction tuning, introduces Ref-SAV, and reports strong cross-task performance.

  • Motivation: Existing video perception models provide segmentation and tracking but lack open-ended language abilities, while video MLLMs support Video-QA but not perception tasks or visual prompts.These complementary limitations motivate integrating the two model families.
  • Motivation: No prior method successfully unified image and video inputs with diverse conversation, question-answering, grounding, and segmentation tasks.The paper identifies this as the central capability gap.
  • Approach: Sa2VA integrates SAM-2 with MLLMs into an end-to-end model that uses single-stage visual instruction tuning for image and video tasks.The framework treats images, videos, and visual prompts as tokens and connects language outputs to grounded masks.
  • Dataset: Ref-SAV addresses limitations of existing video segmentation datasets through an automatically annotated, challenging benchmark targeting complex video scenes.The authors report that existing datasets are small, short, and contain limited occlusions.
  • Results: 15% performance gain over existing approaches is reported on Ref-SAV under zero-shot testing, with stronger results after training on about 37k Ref-SAV videos.Sa2VA also maintains image and video QA capabilities while performing strongly across six referring segmentation datasets.
  • Results: Sa2VA is designed to support multiple open-sourced MLLM backbones, including InternVL and Qwen-VL, while retaining strong grounded understanding across benchmarks.The paper reports experiments across more than 15 benchmarks spanning image, video, QA, visual prompts, and referring segmentation.

2 Related Work

Related work spans multimodal instruction tuning, unified image-video analysis, referring segmentation, and video grounding. Existing methods generally cover subsets of these capabilities, whereas Sa2VA targets their combination.

  • Multi-modal Large Language Models: Multimodal large language model research has developed fusion methods, visual encoders, instruction tuning, and benchmarks for vision-language tasks.These works establish data and multimodal fusion as important components of MLLM development.
  • Capability Comparison: Table 1 contrasts representative capability sets and indicates that no existing method covers the full spectrum of listed modalities and tasks.Sa2VA is designed to support the complete set shown in the comparison.
  • Multi-modal Large Language Models: Recent models increasingly unify image, video, and multi-image analysis, but representative systems still organize capabilities around particular input sources or task groups.LLaVA-OneVision is cited as a model handling four input sources.
  • Referring Segmentation: Referring segmentation methods predict language-driven image masks or video mask tracks using fusion modules, DETR-like architectures, or LLM-based instruction tuning.Recent work also combines referring segmentation with conversation tasks.
  • Video Segmentation and Grounding: Existing video segmentation systems mainly track pixels in closed or limited-vocabulary settings, while video grounding methods explore open-vocabulary or multimodal understanding with restricted task coverage.The related-work discussion positions Sa2VA as extending these directions toward broader video grounding.

3 Method

Sa2VA unifies multimodal tasks by representing text, images, videos, and visual prompts as tokens processed by an MLLM, whose [SEG] output guides SAM-2 mask generation. Its method also includes SAM-2-based video tracking and a staged pipeline for constructing Ref-SAV.

  • 3.1 Unifying Multi-task Representations: Sa2VA encodes text, visual prompts, images, and videos into tokens, processes them with an LLM, and uses the generated [SEG] token to connect language reasoning with SAM-2 masks.The SAM-2 decoder receives encoder features and the [SEG] prompt; selected modules are trained during single-stage tuning.
  • 3.1 Unifying Multi-task Representations: The unified representation maps task inputs and outputs into token-based forms, allowing chat, segmentation, grounded captioning, and visual-prompt tasks within one instruction-tuning framework.Outputs may include text, image masks, video masklets, or text and masks together.
  • 3.2 Model Architecture: The framework combines a pre-trained MLLM with SAM-2 while keeping SAM-2 outputs outside the LLM, simplifying integration and preserving knowledge inheritance and tracking capabilities.The design is intended to remain compatible with evolving MLLM backbones.
  • 3.2 Model Architecture: For referring video segmentation, Sa2VA predicts masks on key frames and uses SAM-2 memory features to generate masks for remaining frames.This procedure leverages SAM-2’s temporal tracking mechanism after MLLM-guided key-frame segmentation.
  • 3.4 Training and Testing: Joint co-training provides positive cross-task transfer: removing image-segmentation data reduces RefCOCO performance from 77.4 to 20.2 cIoU.The cited ablation supports training one shared model across image/video and chat/segmentation tasks.
  • 3.3 Ref-SAV Dataset and Benchmark: Ref-SAV annotation proceeds through object-level, scene-level, and video-level expression generation, with descriptions refined using image context and sampled video frames.Object captions from InternVL2-76B are consistency-checked by Qwen2-72B before later annotation stages.

4 Experiments

Sa2VA achieves strong cross-task performance by combining a modular SAM-2–MLLM framework with joint training and the challenging Ref-SAV benchmark. Results indicate that it preserves conversational abilities while supporting image and video segmentation, visual-prompt understanding, and grounded generation.

  • Main results: Sa2VA’s framework gains remain consistent across interchangeable MLLM backbones, indicating improvements come from the unified paradigm rather than one particular backbone.The framework also supports replacing SAM-2 with SAM-3.
  • Image segmentation: 81.9, 76.5, and 78.9 cIoU on RefCOCO, RefCOCO+, and RefCOCOg surpass GLaMM-7B by 2.4, 3.9, and 4.7 cIoU, respectively.
  • Image conversation: 2229, 82.4, and 75.5 on MME, MMBench, and SEED-Bench show that Sa2VA maintains strong conversational capabilities alongside referring segmentation.Its image-QA performance is comparable to InternVL2.
  • Video benchmarks: 46.9, 75.2, and 57.6 J&F on MeVIS, Ref-DAVIS17, and ReVOS exceed VISA-13B by 2.4, 4.8, and 6.7 J&F, respectively.Sa2VA-8B also scores 1.34 on MMBench-Video versus InternVL2-8B’s 1.28.
  • Ref-SAV: Sa2VA improves Ref-SAV performance with or without Ref-SAV training, and the training set provides further enhancement on this challenging benchmark.Ref-SAV contains longer descriptions, heavier occlusions, and more diverse annotations than prior benchmarks.
  • Generalist comparison: Sa2VA-26B achieves the strongest results across all five compared referring-segmentation datasets while retaining general-purpose conversational and multimodal capabilities.Its generalist design contrasts with lighter specialist models limited to one modality or task.
  • Visual prompts: 17.3 METEOR surpasses Osprey’s 16.6 on RefCOCOg region captioning, showing strong visual-prompt understanding.

5 Failure Cases and Future Work

Sa2VA has unresolved limitations in long-video alignment and in balancing VQA with referring segmentation. Future work targets broader temporal coverage, stronger memory, and task-specific grounding supervision.

  • Failure Cases: Long videos with difficult referring examples remain challenging because Sa2VA operates online without access to the entire video.The paper identifies broader temporal sampling and stronger long-range memory as potential remedies.
  • Failure Cases: 62.9 versus 58.9 J&F on MeViS val_u shows that uniformly sampling five frames outperforms using the first five frames at equal inference cost.The comparison supports broader temporal coverage for long-video alignment.
  • Future Work: Increasing VQA data can reduce referring-segmentation performance, leaving the balance between language and grounded prediction supervision unresolved.The paper frames this as a representation–objective conflict rather than a capacity limitation.
  • Future Work: Reinforcement learning with verifiable mask or boundary rewards is proposed as future work to mitigate VQA–grounding interference.The suggested rewards include mask IoU and boundary F-score, optionally combined with text-side preference signals.
  • Future Work: Ref-SAV remains challenging for long text, occlusion, camera motion, and longer videos, motivating more robust memory and long-text grounding.The paper identifies these cases as critical benchmark problems.

6 Conclusion

Sa2VA unifies image and video understanding with referring segmentation and conversation through single-stage instruction tuning. Experiments show strong performance across tasks, while Ref-SAV and segmenter modularity support further development of pixel multimodal systems.

  • Conclusion: Sa2VA integrates SAM-2 with MLLMs for dense, grounded understanding of images and video.The framework uses knowledge from both components for mask and language generation.
  • Conclusion: Sa2VA handles referring image/video segmentation and image/video conversation with single-stage instruction tuning.
  • Conclusion: Ref-SAV provides a challenging referring video object segmentation benchmark for evaluating the proposed approach.
  • Conclusion: Sa2VA achieves strong performance across various tasks and can be extended with modern MLLMs.The paper presents this extension as a basis for stronger pixel multimodal baselines.

A Concrete Benefits of the Unified Framework

The unified framework uses one model across image/video and chat/segmentation tasks, preserving conversational ability while enabling cross-task transfer. It also reduces checkpoint maintenance, adds limited segmentation overhead, and uses validated automatic annotations.

  • Positive cross-task transfer: Removing image-segmentation training drops RefCOCO from 77.4 to 20.2 cIoU, while removing video-segmentation data lowers MeViS and Ref-DAVIS17 by 6.4 and 3.3 J&F.These ablations support positive cross-task transfer from joint co-training.
  • Conversational ability: Sa2VA retains base-MLLM conversational performance across MME, MMBench, SEED-Bench, and AI2D while reaching state-of-the-art grounding.
  • Maintenance: A single unified checkpoint achieves comparable or better accuracy than per-dataset fine-tuning, which yields only marginal gains.Per-dataset fine-tuning improves referring sets by 1.5% and RefCOCO by only 0.4.
  • Deployment and inference efficiency: SAM-2 adds approximately 220M parameters and runs at 39.5 FPS, while one weight set serves all tasks without additional visual-prompt or [SEG] token overhead.
  • Annotation quality: InternVL2-76B captions are cross-checked by Qwen2-72B, and inconsistent or unrecognizable cases are discarded.The two-model agreement filter reduces the risk of systematic single-model bias.
  • Annotation quality: Ref-SAV test numbers are anchored to human labels, with a fully human-annotated short-expression set and human-filtered long-expression set.An independent method also improves after training on the automatic labels.

C Generalization to the Newer Segmenter: SAM-3

Sa2VA is designed to accommodate evolving promptable segmentation models through a decoupled [SEG]-token bridge. Replacing SAM-2 with SAM-3 under a fixed pipeline improves performance, especially on video benchmarks, without retraining the MLLM.

  • Design: The decoupled design keeps the segmenter encoder and memory frozen while connecting it to the MLLM through a spatial-temporal [SEG] prompt token.
  • Design: SAM-3 can replace SAM-2 through the existing prompt projection because it retains a promptable visual segmentation interface.
  • Experimental control: The SAM-2 versus SAM-3 comparison uses Qwen3-VL-4B, identical training recipes, and no Ref-SAV data to isolate segmenter effects.
  • Results: MeViS improves from 61.4 to 65.3 J&F and ReVOS from 61.4 to 66.3 J&F after upgrading the segmenter.
  • Results: The framework benefits from stronger segmenters without retraining the MLLM, and SAM-3 integration is released for further research.

D Backbone Fairness: Same-Backbone Comparison

A same-backbone comparison isolates Sa2VA’s unified training paradigm from backbone strength. With LLaVA-1.5-7B held constant, Sa2VA substantially improves referring segmentation and adds referring video segmentation support.

  • Experimental control: The comparison trains all Sa2VA rows from scratch for one epoch with LoRA r = 128 and frozen LLM and visual encoder.Image scores are val cIoU and video scores are J&F.
  • Same-backbone results: Under the same LLaVA-1.5-7B backbone, Sa2VA improves RefCOCO by 5.4, RefCOCO+ by 8.0, and RefCOCOg by 11.3 over LISA-7B.Both methods use identical training conditions without Ref-SAV data, isolating the backbone-independent paradigm effect.
  • Same-backbone results: Sa2VA additionally supports referring video segmentation, whereas LISA-7B does not support that task.

E Disentangling Data and Paradigm

The experiments separate Sa2VA’s gains into effects from its unified paradigm and its targeted Ref-SAV data. Results indicate that both are necessary and complementary: neither the paradigm nor the data alone explains performance.

  • Setup: Sa2VA requires no pre-training and uses one epoch of supervised LoRA instruction tuning, taking approximately 24 hours on 16 NVIDIA H100 80GB GPUs.The reported setup uses LoRA r = 256, bf16, and total batch size 256.
  • Paradigm ablation: Removing any single modality from the co-training data causes large performance drops, showing that the joint paradigm matters with data held fixed.
  • Data versus paradigm: With the same Ref-SAV data, UniRef++ improves overall J&F only from 10.5 to 14.6, remaining far below Sa2VA.This comparison indicates that the unified framework contributes beyond data construction alone.
  • Targeted data: Adding 3M generic VQA samples barely changes segmentation, while adding 37K targeted Ref-SAV samples yields +1.7 J&F on MeViS.The targeted-data benefit appears when the unified paradigm is fixed.
  • Conclusion: Data and paradigm are complementary, and neither factor alone accounts for Sa2VA’s performance.

F The VQA--Grounding Representation Conflict

The paper attributes competition between VQA and grounding to different representational demands in their shared token space. It proposes generative co-training and representation-level controls as possible mitigations, while leaving validation of these directions for future work.

  • Representation conflict: VQA encourages compact holistic semantic representations for autoregressive answers, whereas grounding requires spatially discriminative “[SEG]” features for SAM-2 localization.
  • Generative objectives: Generative objectives such as image or video synthesis and masked reconstruction require spatially grounded representations closer to dense grounding than VQA’s holistic objective.
  • Future direction: The paper conjectures that generative co-training may interfere less with grounding and could act as a mutual regularizer, but proposes empirical validation as future work.Suggested tests include a masked image/video reconstruction head or diffusion-style auxiliary loss.
  • Possible mitigations: Proposed in-paradigm mitigations include task-specific token subspaces with gating, “[SEG]” representation regularization, and reinforcement learning with verifiable segmentation rewards.

G Additional Qualitative Results

Additional visualizations show Sa2VA handling challenging video referring segmentation and generating contextual descriptions from visual prompts. The appendix also presents failures concentrated in long videos with hard, fine-grained expressions.

  • Appendix scope: The qualitative results cover video referring segmentation, visual prompt understanding, and failure cases outside the main text.
  • Video referring segmentation: Sa2VA performs effectively in occlusion scenes and highly dynamic environments in video referring segmentation visualizations.
  • Visual prompt understanding: For visual prompt understanding, the model generates region descriptions whose masks and captions are paired by matching colors.The descriptions are reported to capture details within the visual cues with high contextual awareness.
  • Failure cases: Representative failure cases mainly arise from long videos with hard, fine-grained referring expressions.
Loading 2501.04001v4…