PaperlayerView exact revision on arXiv

Source-linked AI summary

X2SAM: Any Segmentation in Images and Videos

Hao Wang, Limeng Qiao, Chi Zhang, Lin Ma, Guanglu Wan, Xiangyuan Lan, Xiaodan Liang

arXiv:2605.00891v1cs.CVcs.AI

TL;DR

MLLMs still have limited pixel-level perception, while existing segmentation systems do not unify complex textual and visual prompting across images and videos. X2SAM combines an MLLM, Mask Memory, and joint image-video training to provide unified segmentation, achieving strong video results while remaining competitive on image benchmarks. Its scope is bounded by expensive heterogeneous-data training, fixed-size memory for long videos, and possible disadvantages versus specialized models.

  • Problem

    MLLMs have limited dense pixel-level perception, and existing segmentation MLLMs are usually specialized for images or videos without unified textual and visual prompting.

  • Method

    X2SAM unifies image and video segmentation by combining conversational and visual prompting with an MLLM, Mask Memory module, and joint training over image and video data.

  • Results

    X2SAM supports diverse segmentation tasks, achieves strong video and out-of-domain results, remains competitive on image benchmarks, and maintains balanced performance across modalities.

  • Takeaways & Limitations

    X2SAM provides a unified instruction-following framework for pixel-level understanding across images and videos, including temporally consistent video segmentation from visual prompts.

  • Takeaways & Limitations

    Unified training remains computationally expensive, fixed-size FIFO memory may be insufficient for long videos, and the generalist model may lag behind specialized systems on narrow tasks.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated strong image-level visual understanding and reasoning, yet their pixel-level perception across both images and videos remains limited. Foundation segmentation models such as the SAM series produce high-quality masks, but they rely on low-level visual prompts and cannot natively interpret complex conversational instructions. Existing segmentation MLLMs narrow this gap, but are usually specialized for either images or videos and rarely support both textual and visual prompts in one interface. We introduce X2SAM, a unified segmentation MLLM that extends any-segmentation capabilities from images to videos. Given conversational instructions and visual prompts, X2SAM couples an LLM with a Mask Memory module that stores guided vision features for temporally consistent video mask generation. The same formulation supports generic, open-vocabulary, referring, reasoning, grounded conversation generation, interactive, and visual grounded segmentation across image and video inputs. We further introduce the Video Visual Grounded (V-VGD) segmentation benchmark, which evaluates whether a model can segment object tracks in videos from interactive visual prompts. With a unified joint training strategy over heterogeneous image and video datasets, X2SAM delivers strong video segmentation performance, remains competitive on image segmentation benchmarks, and preserves general image and video chat ability.

1 Introduction

X2SAM addresses the limited pixel-level perception of MLLMs and the fragmentation of existing segmentation systems by unifying conversational, textual, and visual prompting across image and video segmentation. Its Mask Memory module and joint training strategy support temporally consistent video masks and broad segmentation coverage.

  • MLLMs excel at global visual understanding but remain limited in generating dense pixel-level outputs for precise spatial and temporal comprehension.
  • Existing systems are fragmented: foundation models rely on low-level visual prompts, while segmentation MLLMs are typically restricted to images or videos and lack unified textual-visual prompting.
  • Mask Memory stores guided vision features and interacts with the Mask Decoder to maintain mask consistency across video sequences.
  • X2SAM supports seven segmentation tasks across images and videos, including generic, open-vocabulary, referring, reasoning, grounded conversation generation, object-centric, and visual grounded segmentation.
  • Unified joint training over image and video datasets yields strong performance across video and image benchmarks while establishing a practical baseline for unified pixel-level spatio-temporal understanding.
  • X2SAM extends any segmentation from images to videos through a unified framework that integrates an MLLM with a Mask Memory module.
  • The V-VGD benchmark evaluates interactive visual prompting for grounding and segmenting instance objects consistently across video frames.

2 Related Work

Prior work separately advances multimodal understanding, image segmentation, and video segmentation, but existing methods remain fragmented across modalities and prompting mechanisms. X2SAM is positioned as a unified architecture addressing these gaps.

  • MLLMs have progressed from task-specific fusion toward generalized instruction-tuned frameworks, but their dense pixel-level output capability remains limited.
  • Image segmentation MLLMs combine language understanding with segmentation models for open-world, unified-task, and language-guided segmentation.
  • Video segmentation MLLMs enable temporal text-to-mask generation but face substantial temporal complexities in dynamic sequences.
  • Existing video-centric MLLMs lack a unified architecture for images and videos, while frame-by-frame decoding struggles to preserve mask consistency and temporal coherence.
  • SAM2 provides memory-based promptable image and video segmentation but mainly relies on low-level visual prompts, whereas X-SAM supports textual and visual prompts but remains image-centric.

3 Method

X2SAM unifies conversational and visually prompted segmentation across images and videos through dual visual processing, an LLM, a redesigned mask decoder, and temporal mask memory. Its training combines category-agnostic segmentor initialization with unified joint optimization over image and video data.

  • 3 Method: X2SAM accepts textual or visual prompts paired with either a single image or a video sequence, producing a language response and corresponding binary segmentation mask.Text specifies the task, while visual cues such as points or boxes designate objects of interest.
  • 3 Method: The architecture combines global visual representations, fine-grained mask features, region-specific embeddings, and tokenized text in an LLM-guided segmentation pipeline.The vision encoder supplies global features, the mask encoder supplies dense features, and the region sampler extracts localized prompt embeddings when needed.
  • 3 Method: The redesigned mask decoder injects LLM semantic token embeddings into spatial features through Query-to-Image Attention and Token-to-Image Attention.The mask encoder is retained from SAM2, while its original decoder is replaced to support parallel mask generation and token-level conditioning.
  • 3 Method: Mask Memory maintains temporal coherence by refining current-frame features with prior guided features, encoding current outputs, and storing them in a FIFO memory bank.The pipeline comprises Memory Attention, Mask Decoder, Memory Encoder, and Memory Bank stages.
  • 3 Method: Training first initializes the mask decoder with category-agnostic dense supervision, then jointly optimizes heterogeneous image and video data under modality-aware batching.The mask encoder remains frozen during initialization, and the mask loss combines binary cross-entropy and Dice objectives with λbce = 5.0 and λdice = 5.0.

4 Experiments

Experiments evaluate X2SAM across 14 image and video segmentation tasks, ablate its decoder, mask memory, training strategy, and memory size, and compare it with specialists and generalists. X2SAM remains competitive on images while achieving strong results across video benchmarks, including reasoning, out-of-domain, and visual grounded segmentation.

  • Tasks, Datasets, and Metrics: X2SAM covers seven image and seven video segmentation tasks driven by textual or visual prompts.The tasks include generic, open-vocabulary, referring, reasoning, grounded conversation generation, object, interactive, and visual grounded segmentation variants.
  • Tasks, Datasets, and Metrics: Training combines class-agnostic mask-decoder pretraining with unified joint training over all 14 segmentation tasks and image/video chat data.The class-agnostic phase uses mask-only SA-1B, while joint training integrates heterogeneous image and video datasets.
  • Ablation Studies: 36.5% lower training cost is achieved by unified joint training, reducing approximately 5.2K GPU hours to 3.3K GPU hours.The simple joint-training method reports 54.4 I-Gen. PQ and 64.6 V-Gen. mIoU while requiring approximately 5.2K GPU hours.
  • Ablation Studies: Zero-initialized T2I attention reaches 83.3/77.8/79.5 on I-Ref and raises V-Ref YT21 from 53.6 to 60.8 J &F.Random initialization disrupts early training and can fall below the baseline on static-image tasks.
  • Ablation Studies: Multi-scale mask memory obtains 45.0 VPQ, 62.5 mIoU, 65.0 J &F on V-Ref. YT21, and 53.5 J &F on V-Rea. All.Mask guidance improves temporal alignment, class guidance improves semantic discrimination, and the full design performs best overall.
  • Ablation Studies: Memory size 6 achieves the best V-OV mAP of 60.2 and V-Rea. All score of 57.5 J &F, whereas size 8 lowers some V-Ref. and V-Rea. results.The study adopts six frames because excessive memory can introduce redundant or noisy temporal cues.
  • Benchmark Results: X2SAM improves V-GCG over VideoGLaMM by +21.5 mIoU, from 54.3 to 75.8, while achieving 65.1 V-Gen. mIoU.It also improves image open-vocabulary segmentation from 20.9 to 31.2 PQ and remains competitive with image-centric X-SAM.
  • Benchmark Results: On video grounded segmentation, X2SAM reaches 74.4 AP on YT-VIS19 and 57.8 AP on VIPSeg with box prompts, versus 54.0 and 40.4 for SAM2-H.Image visual grounded segmentation remains competitive, reaching 45.9 AP with point prompts and 48.5 AP with box prompts on COCO.

5 Discussion

X2SAM unifies image and video segmentation through shared instruction-following, textual and visual prompts, and joint training across heterogeneous datasets. It broadens task coverage while maintaining balanced performance across modalities, but remains limited by training cost, fixed-size memory, and generalist trade-offs.

  • Unified framework: X2SAM supports textual instructions and visual prompts across image and video segmentation within one shared framework.Its formulation covers diverse segmentation tasks and extends image-centric any-segmentation to video sequences.
  • Discussion: X2SAM remains competitive on image segmentation benchmarks, improves many video tasks, and preserves general image and video understanding abilities.The conclusion characterizes this as a balance between task coverage and accuracy.
  • Limitations: Unified heterogeneous training is computationally expensive, fixed-size FIFO memory may be insufficient for difficult long videos, and specialized models can remain better on narrow tasks.The authors identify efficient training, lightweight backbones, and adaptive long-range memory as future directions.
  • Training and evaluation: The training pipeline combines class-agnostic segmentor initialization with unified joint training over heterogeneous image and video datasets.The datasets span generic, referring, reasoning, grounded conversation, object-centric, visual grounded segmentation, and image or video chat.
  • Training and evaluation: X2SAM is evaluated on 14 in-domain image and video segmentation tasks plus out-of-domain datasets for zero-shot or transfer evaluation.The additional datasets cover image referring and open-vocabulary segmentation and video open-vocabulary segmentation.
  • Video visual grounding: The V-VGD datasets convert object tracks into visual-prompt-driven video segmentation examples using point, scribble, box, or mask prompts.Prompts are generated from the first visible annotated frame, while supervision covers the full spatio-temporal mask sequence.

C More Model Details

X2SAM combines SAM2-derived mask encoding, language-conditioned mask decoding, and a recurrent mask memory for temporally refined video segmentation. Its training and evaluation span multiple image, video, segmentation, chat, and grounding tasks, with strong reported results across modalities.

  • Architecture: The SAM2-derived mask encoder extracts dense mask-aware features while retaining fine-grained boundaries, textures, and region-level visual cues.X2SAM discards SAM2’s original decoder and memory and keeps the mask encoder as a lightweight feature extractor.
  • Architecture: The mask decoder transforms the LLM’s <SEG> hidden state into mask queries that attend to visual and memory-enhanced features.This token-to-image attention design produces masks that are spatially accurate and semantically aligned with language instructions.
  • Architecture: The mask memory stores guided vision features from recent video frames and updates a fixed-size FIFO cache after each mask prediction.Memory attention combines cached and current-frame representations to produce temporally refined features while keeping computation bounded.
  • Training and preprocessing: X2SAM uses consecutive frame sampling with length 8 for most video segmentation datasets, 16 frames for video grounded conversation, and 64 frames for video chat.The different sampling strategies support task-specific temporal coverage.
  • Benchmark results: X2SAM obtains 47.3 VPQ and 59.3 VPQ1 on VIPSeg, with the best VPQ1 among compared methods.These results support strong performance on challenging video generic segmentation while preserving image-level understanding.
  • Benchmark results: On YT-VOS19, X2SAM achieves 72.2/75.9/74.0 J/F/J &F scores, extending object-centric segmentation to video beyond image-only MLLM baselines.It remains behind the specialized SAM2-H model, while offering multimodal task versatility without task-specific architectural designs.
  • Video chat: X2SAM achieves 74.4%, 63.1%, 67.1%, and 57.4% on VideoMME, MVBench, MLVU, and LongVideoBench, respectively.It outperforms several chat-centric video MLLMs, while comparison with Qwen3-VL is mixed.
  • Qualitative results: Visualization results cover generic, referring, reasoning, grounded conversation, object-centric, visual grounded, and open-vocabulary segmentation across images and videos.The examples demonstrate accurate and coherent masks under varied prompts, categories, and visual scenarios.

Video Chat

Figure 12 visualizes X2SAM’s visual chat results across both images and videos.

  • Figure 12 presents visual chat examples for images and videos.
Loading 2605.00891v1…