PaperlayerView exact revision on arXiv

Source-linked AI summary

One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos

Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Lei Liu, Zheng Zhang, Mike Zheng Shou

arXiv:2409.19603v1cs.CVcs.AI

TL;DR

Image-based reasoning segmentation methods do not seamlessly transfer to videos, which require temporal understanding and consistent masks across frames. VideoLISA combines sparse-dense sampling with a shared <TRK> token for language-instructed video segmentation. Evaluations span public video and image benchmarks, and the model also generalizes well to image segmentation.

  • Problem

    Video segmentation requires temporal understanding and consistent masks across frames, while complex language instructions can demand reasoning and world knowledge beyond image-based methods’ focus.

    Source: §1 · p. 1

  • Method

    VideoLISA combines sparse-dense sampling for temporal context and spatial detail with a specially trained <TRK> token that segments and tracks objects across frames.

    Source: §1 · p. 2

  • Results

    Evaluations across public benchmarks and ablations support the approach’s effectiveness, and experiments show generalization to image segmentation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VideoLISA is presented as a potential foundation model for unified language-instructed object segmentation across video and image tasks.

    Source: §1 · p. 2

    Despite reducing its LLM to 3.8B parameters, VideoLISA still incurs relatively high computational cost compared with prior video object segmentation work.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce VideoLISA, a video-based multimodal large language model designed to tackle the problem of language-instructed reasoning segmentation in videos. Leveraging the reasoning capabilities and world knowledge of large language models, and augmented by the Segment Anything Model, VideoLISA generates temporally consistent segmentation masks in videos based on language instructions. Existing image-based methods, such as LISA, struggle with video tasks due to the additional temporal dimension, which requires temporal dynamic understanding and consistent segmentation across frames. VideoLISA addresses these challenges by integrating a Sparse Dense Sampling strategy into the video-LLM, which balances temporal context and spatial detail within computational constraints. Additionally, we propose a One-Token-Seg-All approach using a specially designed <TRK> token, enabling the model to segment and track objects across multiple frames. Extensive evaluations on diverse benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate VideoLISA's superior performance in video object segmentation tasks involving complex reasoning, temporal understanding, and object tracking. While optimized for videos, VideoLISA also shows promising generalization to image segmentation, revealing its potential as a unified foundation model for language-instructed object segmentation. Code and model will be available at: https://github.com/showlab/VideoLISA.

1 Introduction

VideoLISA targets language-instructed reasoning segmentation in videos, where models must understand temporal dynamics and produce consistent masks across frames. It combines sparse-dense sampling with a single <TRK> token to address these demands.

1 Introduction

  • Image-based reasoning segmentation methods do not seamlessly transfer to video, which requires temporal understanding and consistent masks across frames.Language instructions can involve intent understanding, reasoning, and world knowledge beyond category names or detailed referring expressions.Source: §1 · pp. 1–2
  • Sparse Dense Sampling preserves full-resolution features for selected frames while down-sampling interleaved frames to retain temporal context under computational constraints.Dense tokens supply segmentation detail; sparse tokens capture motion and change over time.Source: §1 · p. 2
  • One-Token-Seg-All uses a specially trained <TRK> token to segment and track target objects across multiple video frames.The model uses the token’s last hidden embedding to prompt mask decoding, with training designed to segment multiple frames simultaneously.Source: §1 · p. 2
  • Experiments across public benchmarks and ablations support the approach’s effectiveness, and VideoLISA also generalizes well to image segmentation.The evaluations cover video and image referring segmentation, motion-guided video segmentation, and image reasoning segmentation.Source: §1 · p. 2
  • VideoLISA is a video-based multimodal language model, and ReasonVOS is introduced to evaluate complex reasoning, temporal understanding, and object movement.The paper evaluates the model on ReasonVOS and public benchmarks.Source: §1 · p. 2

2 Related Work

Prior video segmentation work adapted referring-image methods and developed video-specific architectures, but motion comprehension and complex reasoning segmentation remain challenges. Video-language models also face token-budget limits when processing multiple frames.

2.1 Video Object Segmentation

  • Video object segmentation studies still identify limited motion comprehension and few methods or benchmarks for complex reasoning-based video segmentation.Existing approaches include per-frame segmentation, DETR-like structures, query propagation, and segment-and-optimize methods.Source: §2.1 · p. 3

2.2 Multimodal Large Language Model

  • Video MLLMs use frame-token concatenation, pooling or merging, and Q-formers to reduce temporal token demands under computational limits.The cited approaches include pooling along spatial and temporal dimensions, similarity-based token merging, and different pooling strengths at a slow-fast pace.Source: §2.2 · p. 3
  • Image-grounding MLLMs can produce fine-grained masks, but video extensions have used external detectors and trackers or iterative token updates; lacking video learning limits temporal perception and reasoning.The paper contrasts these video approaches with its focus on video learning for temporal dynamics.Source: §2.2 · p. 3

3 Method

The method maps a video and free-form language instruction to masks for all frames. Its architecture combines an LLM with a promptable mask decoder, while sampling and a shared token address video-specific temporal and spatial demands.

3 Method

  • VideoLISA takes a video and free-form language expression as input and outputs pixel-level masks for every frame.The expression can emphasize implicit intent reasoning, world knowledge, and video temporal dynamics.Source: §3 · p. 3

3.1 Architecture

  • The architecture feeds sampled video tokens and text tokens into an LLM, then uses a <TRK>-derived prompt embedding and per-frame visual features to generate masks.The visual tokenizer and LLM are initialized from LLaVA, while the vision encoder and mask decoder are initialized from SAM.Source: §3.1 · p. 4

3.2 Sparse Dense Sampling

  • Sparse Dense Sampling keeps selected frames at full resolution and pools the remaining frames to one token each, reducing the total from Tsparse × L to Tsparse + Tdense × L.Dense tokens carry visual detail for nearby sparse frames, while sparse tokens capture temporal dynamics; the paper reports that common video-LLM sampling strategies do not transfer seamlessly to video object segmentation.Source: §3.2 · p. 4

3.3 One Token Seg All

  • One-Token-Seg-All uses one <TRK> token across frames, with video context and multi-frame training intended to support cross-frame object association.The authors attribute LISA’s cross-frame failures to limited temporal input and single-frame training, which can encourage positional shortcuts.Source: §3.3 · p. 5

3.4 Training and Inference

  • Training combines image segmentation with VOS and RVOS data, while inference reuses one <TRK> token to decode masks frame by frame.Training computes segmentation loss on sampled dense frames in parallel and averages it; optional XMem++ post-optimization uses reliable dense-frame masks as references for other frames.Source: §3.4 · pp. 5–6

4 Benchmark

ReasonVOS is designed to evaluate video reasoning segmentation alongside temporal understanding and object movement. It combines existing video and mask annotations with human-checked language instructions for zero-shot evaluation.

4 Benchmark

  • ReasonVOS addresses the lack of a benchmark that comprehensively evaluates reasoning segmentation in videos.Its instructions cover at least one of complex reasoning, world knowledge, or temporal dynamics, and its selected objects prioritize explicit movement.Source: §4 · p. 6
  • ReasonVOS contains 458 video-instruction-mask samples, built from 105 manually annotated seed samples and LLM-rephrased, human-checked expressions.The source datasets include MOSE, MeViS, VIPSeg, and BURST.Source: §4 · p. 6
  • ReasonVOS is intended for zero-shot evaluation because the authors consider reasoning ability to be embedded in the LLM and triggerable by existing image-based reasoning segmentation data.Video and mask annotations are drawn from existing datasets, with language expressions added for the benchmark.Source: §4 · p. 6

5 Experiments

Experiments evaluate VideoLISA on referring, motion-guided, and reasoning video segmentation, image reasoning segmentation, and ablations. The authors report strong results across these tasks and compare temporal-module designs.

5.1 Experimental Setting

  • Training combines image and video segmentation datasets, and video sampling uses 32 sparse and 4 dense frames; video evaluation uses J, F, and their average J&F.The final comparison model is trained for 6,000 iterations; image evaluation uses gIoU and cIoU.Source: §5.1 · pp. 6–7

5.2.1 Referring Video Object Segmentation

  • VideoLISA achieves state-of-the-art performance on Ref-DAVIS-17 and outperforms larger LLM-based methods, while ranking highly against traditional RVOS methods on Refer-YouTube-VOS.The cited comparison includes LISA-13B and TrackGPT-13B; Table 1 reports quantitative results for both benchmarks.Source: §5.2.1 · p. 7, Table 1 · p. 7

5.2.2 Motion-guided Video Object Segmentation

  • VideoLISA achieves state-of-the-art performance on the MeViS motion-guided VOS benchmark, which the authors attribute to temporal-dynamics capture and cross-modal interaction.They connect these capabilities to Sparse Dense Sampling and One-Token-Seg-All.Source: §5.2.2 · p. 7, Table 2 · p. 7

5.2.3 Reasoning Video Object Segmentation

  • VideoLISA shows remarkable performance on ReasonVOS, where most previous methods perform unsatisfactorily on complex reasoning, temporal understanding, and temporally consistent segmentation.The authors identify limitations in traditional RVOS methods' complex-language comprehension and in image-based reasoning methods' temporal modeling.Source: Table 3 · p. 7, §5.2.3 · p. 8

5.3 Evaluation on Image Tasks

  • On image reasoning segmentation, VideoLISA achieves state-of-the-art performance on both validation and test sets, outperforming larger LISA-7B and LISA-13B models.Images are duplicated into multiple frames as pseudo-video during testing; Table 4 reports reasoning-segmentation results.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Ablation Studies

  • Ablations favor Sparse Dense Sampling over the compared video-LLM sampling strategies and One-Token-Seg-All over image-based LISA extensions and One-Token-Seg-One.Pooling strategies lose visual detail, while n-frame comparisons support the need for temporal information; adding a tracker improves the LISA baseline but remains below VideoLISA.Source: Table 5 · p. 8, Table 6 · p. 8, §5.4 · p. 9

6 Limitation and Future Work

The authors identify computational efficiency and video-backbone integration as limitations and directions for future work.

6 Limitation and Future Work

  • Despite its 3.8B parameters, VideoLISA remains relatively costly compared with prior video object segmentation work, reflecting a trade-off between computation and MLLM reasoning ability.The authors identify improving that trade-off as future work.Source: §6 · p. 9
  • Integrating a video-pretrained vision encoder with the LLM and SAM decoder is non-trivial, although such a backbone may benefit temporal tasks like tracking.The authors leave video-backbone integration as a potential future direction.Source: §6 · p. 9
  • This work focuses on adding LLM-based reasoning capabilities to video segmentation rather than integrating a video backbone.The authors describe that integration as a future avenue.Source: §6 · p. 9

7 Conclusion

VideoLISA combines an LLM and SAM with two video-specific designs for language-instructed reasoning segmentation. The authors report strong video segmentation capabilities and image-segmentation performance, positioning it as a potential unified model.

7 Conclusion

  • VideoLISA combines LLM reasoning with SAM masks and introduces Sparse Dense Sampling and One-Token-Seg-All for video segmentation.The conclusion describes these designs as balancing visual detail and temporal context, and supporting temporally consistent masks.Source: §7 · p. 9
  • ReasonVOS validates VideoLISA's capabilities in complex reasoning, temporal understanding, and object tracking.The conclusion also reports notable performance on image segmentation and presents VideoLISA as a potential unified language-instructed object segmentation model.Source: §7 · p. 9
  • The conclusion presents VideoLISA as a video-based LLM for language-instructed reasoning segmentation.It uses an LLM's reasoning capabilities and SAM to produce segmentation masks.Source: §7 · p. 9

A.1 Evaluation on Image Segmentation

VideoLISA performs comparably to image-based LISA on refCOCO and refCOCO+, and achieves state-of-the-art performance on refCOCOg. The accompanying ablation-study setup specifies shared training conditions and three benchmarks for analyzing model capabilities.

A.1 Evaluation on Image Segmentation

  • VideoLISA matches image-based LISA comparably on refCOCO and refCOCO+, and achieves state-of-the-art performance on refCOCOg.The authors describe these results, together with main-paper image reasoning segmentation results, as evidence that VideoLISA is a strong image-segmentation competitor.Source: §A.1 · p. 14, Table 7 · p. 14
  • The image-segmentation evaluation covers three widely adopted referring-image-segmentation benchmarks.The reported results are presented in Table 7.Source: §A.1 · p. 14, Table 7 · p. 14

A.2 Ablation Studies

  • Ablation variants use the same 3k-iteration training setting, data recipe, learning-rate scheduler, and training objective unless specified.ReasonSeg evaluates reasoning, MeViS temporal learning, and Ref-DAVIS-17 general RVOS capability; video metrics use One-Token-Seg-All without post-optimization.Source: §A.2 · p. 14

A.2.1 Temporal Learning Module

The ablations favor VideoLISA's Sparse Dense Sampling and all-frame One-Token-Seg-All supervision over simpler alternatives. Adding post-optimization with XMem++ produces the best reported performance among the compared tracking designs.

A.2.1 Temporal Learning Module

  • Simply fine-tuning LISA on video data does not significantly improve video performance and hurts image reasoning segmentation.The authors attribute the shortfall to a lack of temporal modeling ability from video data, despite improved understanding of temporally related text queries.Source: §A.2.1 · p. 14
  • Concatenating features from n sampled frames substantially outperforms LISA-based methods, but including too many frames is computationally prohibitive.The authors attribute temporal-dynamics perception to multiple frames and describe the One-Token-Seg-All objective as supervising mask association across time.Source: §A.2.1 · pp. 14–15
  • Sparse Dense Sampling is favored over pooling strategies because it preserves dense-frame visual detail while retaining temporal context.The authors report that spatial-temporal pooling loses valuable details and that pooling all frames, even at different strengths, underperforms their strategy.Source: §A.2.1 · p. 15, Table 8 · p. 15

A.2.2 Temporal Association Module

  • The One-Token-Seg-All objective is essential for temporally consistent masks: supervising one frame instead produces a significant performance gap.The One-Token-Seg-One strawman captures video with the temporal learning module but trains on only one frame.Source: Table 9 · p. 15, §A.2.2 · p. 16
  • VideoLISA with XMem++ post-optimization achieves the best performance, while LISA+XMem performs worse than both VideoLISA variants on the benchmarks.XMem++ uses VideoLISA masks on uniformly sampled dense frames as references to infer masks for remaining frames.Source: §A.2.2 · pp. 15–16, Table 9 · p. 15

A.2.3 Ablation on Training Data

Image-only training yields decent reasoning segmentation but unsatisfactory video results, while video-only training improves video performance and sharply reduces reasoning segmentation. Combining image and video segmentation datasets yields strong performance across benchmarks; adding VQA data brings mixed effects.

A.2.3 Ablation on Training Data

  • Combining image and video segmentation datasets yields remarkable performance across various benchmarks.Image-only data supports reasoning segmentation but performs unsatisfactorily on video benchmarks; video-only data improves video results while sharply reducing reasoning-segmentation performance.Source: §A.2.3 · p. 16
  • Adding Image-QA slightly lowers performance across benchmarks, while Video-QA improves reasoning segmentation but has mixed video-benchmark effects.Against segmentation-only training, Video-QA performs slightly better on MeViS offline validation and worse on Ref-DAVIS-17; compatibility among training data and tasks is left for future work.Source: §A.2.3 · p. 16

A.3 Qualitative Results

VideoLISA’s qualitative examples show language referring, world-knowledge reasoning, and temporal reasoning in video segmentation. Its failure cases also expose limits associated with MLLM hallucination, knowledge scope, and mask quality.

A.3 Qualitative Results

  • VideoLISA segments targets through language referring, world-knowledge reasoning, and temporal reasoning, including subtle instruction distinctions without bias toward salient or moving objects.Figure 3 illustrates these capabilities with a representative video.Source: §A.3 · p. 16, Figure 3 · p. 17
  • Additional examples use One-Token-Seg-All inference without special prompting techniques or post-optimization.The red text in the examples is for illustration only.Source: §A.3 · p. 16
  • Figure 5 presents ReasonVOS sample statistics and the sources of its videos.Source: Figure 5 · p. 18

A.4 Failure Cases

  • In one crash-scene case, VideoLISA repeatedly selects the same bottom-left object despite prompt rephrasing, with the authors hypothesizing MLLM hallucination.The model identifies the object inconsistently as a stove, telephone pole, or something else.Source: Figure 4 · p. 17, §A.4 · p. 18
  • A game-rule cue enables the correct-person segmentation when the match result is beyond the MLLM’s knowledge scope, though some frames still have low-quality masks.The authors describe these cases as evidence that VideoLISA’s reasoning is bounded by its underlying MLLM and can be alleviated through prompt engineering.Source: §A.4 · p. 18

B Benchmark

ReasonVOS comprises 91 videos and 458 video-instruction-mask samples, with short object descriptions and longer reasoning instructions. The paper also identifies possible application areas while noting privacy and misuse concerns.

B Benchmark

  • ReasonVOS contains 91 videos and 458 video-instruction-mask samples, including 105 manually annotated seed samples augmented using the Claude 3 API.Videos and mask annotations come from various sources, with additional text descriptions annotated for the benchmark.Source: B Benchmark · p. 18
  • ReasonVOS distinguishes short queries describing specific objects from full-sentence long queries that require reasoning.Source: B Benchmark · p. 18

C Broader Impact

  • The paper identifies surveillance, education, healthcare, and everyday assistance as potential application areas for reasoning-based video segmentation.It also cautions that privacy concerns and potential misuse require responsible and equitable implementation.Source: C Broader Impact · p. 18
  • Responsible deployment should account for privacy concerns and potential misuse of the technology.Source: C Broader Impact · p. 18
Loading 2409.19603v1…