PaperlayerView exact revision on arXiv

Source-linked AI summary

GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation

Lang Lin, Xueyang Yu, Ziqi Pang, Yu-Xiong Wang

arXiv:2504.07962v1cs.CV

TL;DR

RefVOS MLLMs struggle to use limited frames for both global understanding and local tracking, often relying on external modules. GLUS combines context and query frames with an end-to-end VOS memory bank and refinement components in one MLLM. It reports state-of-the-art results on RefVOS benchmarks, including a MeViS improvement over prior MLLM-based methods.

  • Problem

    Limited MLLM frame context makes it difficult to combine global language-based reasoning with local, continuous tracking across a video.

    Source: §1 · p. 2, §3.3 · p. 3

  • Method

    GLUS uses sparse context frames and continuous query frames in one MLLM, trained end-to-end with a pre-trained VOS memory module and supplemented by object contrastive loss and key-frame self-refinement.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    GLUS reports state-of-the-art RefVOS benchmark results; on MeViS, its J &F is over 5% higher than other MLLM-based RefVOS models.

    Source: §5.3 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The authors present GLUS as a RefVOS baseline and hope it will inspire more systematic study.

    Source: §6 · p. 9

    Because Ref-Youtube-VOS relatively lacks multiple complex expressions for the same objects, the contrastive loss is applied only to MeViS, though it benefits Ref-Youtube-VOS as well.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

This paper proposes a novel framework utilizing multi-modal large language models (MLLMs) for referring video object segmentation (RefVOS). Previous MLLM-based methods commonly struggle with the dilemma between "Ref" and "VOS": they either specialize in understanding a few key frames (global reasoning) or tracking objects on continuous frames (local reasoning), and rely on external VOS or frame selectors to mitigate the other end of the challenge. However, our framework GLUS shows that global and local consistency can be unified into a single video segmentation MLLM: a set of sparse "context frames" provides global information, while a stream of continuous "query frames" conducts local object tracking. This is further supported by jointly training the MLLM with a pre-trained VOS memory bank to simultaneously digest short-range and long-range temporal information. To improve the information efficiency within the limited context window of MLLMs, we introduce object contrastive learning to distinguish hard false-positive objects and a self-refined framework to identify crucial frames and perform propagation. By collectively integrating these insights, our GLUS delivers a simple yet effective baseline, achieving new state-of-the-art for MLLMs on the MeViS and Ref-Youtube-VOS benchmark. Our project page is at https://glus-video.github.io/.

1. Introduction

RefVOS requires models to connect language with video content and track objects even when the described action appears in only some frames. GLUS combines global and local reasoning in one MLLM, with memory and refinement components supporting that design.

1. Introduction

  • RefVOS requires locating and consistently tracking objects from language, although the described action may appear in only a subset of frames.Models must understand global object characteristics to propagate the reasoning across the video.Source: §1 · pp. 1–2
  • Prior MLLM approaches face a “Ref” versus “VOS” dilemma: limited frames support either global context or local continuity, often prompting external VOS modules.Those modules add system complexity.Source: §1 · p. 2
  • GLUS assigns sparse, uniformly sampled context frames to global reasoning and continuous query frames to segmentation and local continuity.The MLLM uses the referring expression and preceding query frames as context while predicting.Source: §1 · p. 2
  • An end-to-end trained, pre-trained VOS memory module stores information from prior predictions to support temporal reasoning without calling an external VOS model.GLUS also uses object contrastive loss and a selector trained with frame-quality pseudo-labels for object matching and self-refinement.Source: §1 · p. 2
  • GLUS unifies global and local reasoning in one MLLM and reports a new strong baseline on MeViS and Ref-Youtube-VOS.The paper reports advantages over previous MLLM-based RefVOS methods on both benchmarks.Source: §1 · p. 2

2. Related Work

Related work spans language-guided video segmentation, MLLM adaptations for segmentation, and VOS memory methods. GLUS positions its unified frame strategy and end-to-end memory training against these approaches.

2. Related Work

  • RefVOS methods use language queries to attend to targets or aggregate motion; GLUS unifies global and local reasoning and decouples motion aggregation modules.The cited approaches include [4] [25] [35] [42] [43] [50] for language queries and [9] [13] for motion aggregation.Source: §2 · pp. 2–3
  • Video MLLM approaches adapt LISA [19] with token tracking, global context, context aggregation, or a single tracking token; GLUS aims for motion understanding without additional VOS modules.The approaches discussed are TrackGPT [54], VISA [45], ViLLa [52], and VideoLISA [3].Source: §2 · p. 3
  • VOS memory banks store historical information for temporal reasoning, and GLUS trains a pre-trained bank end-to-end with a RefVOS MLLM; Table 1 contrasts its context-plus-query strategy with prior methods.The cited VOS work explores pixel-level [6] [16] [37] [47] [48] [49] and object-level [1] [2] [8] [21] matching, as well as memory organization [7] [32] [46] [53].Source: §2 · p. 3, Table 1 · p. 3

3. Preliminaries

RefVOS combines language-based target localization with video-wide reasoning and temporal coherence. The preliminaries describe an MLLM segmentation design and the frame-limit and propagation challenges involved in adapting it to video.

3.1. Referring Video Object Segmentation

  • RefVOS predicts binary masks for a language-referred object across a video, requiring global reasoning because the relevant action may occur only in a short clip and consistent tracking across frames.Unlike VOS, RefVOS is offline; unlike referring image segmentation, it requires video-wide reasoning and temporal coherence.Source: §3.1 · p. 3

3.2. LISA: MLLM for Segmentation

  • LISA represents the target object with a dedicated ⟨SEG⟩ token, which a segmentation decoder uses to predict object masks.The MLLM output is simplified to the segmentation token; related work follows this design in many RefVOS MLLMs.Source: §3.2 · p. 3

3.3. Adapting Image MLLMs for RefVOS

  • A multi-image adaptation feeds frames to the MLLM and decodes segmentation tokens into masks, but its frame limit N is commonly smaller than the video length T.Prior approaches often sample key frames and use external VOS propagation, which complicates the system, restricts language-guided propagation, and leaves performance bounded by the key-frame selector.Source: §3.3 · pp. 3–4
  • GLUS’s overview combines context and query frames, a VOS memory-enhanced decoder, object-discriminating token supervision, and a key-frame selector for inference-time refinement.The ⟨SEG⟩ tokens represent the target object and are decoded into masks.Source: Figure 2 · p. 4

4. Methods

GLUS unifies global and local reasoning in one MLLM, pairing sparse context frames with continuous query frames. It adds an end-to-end VOS memory bank, object contrastive loss, and self-refinement for key-frame selection and propagation.

4.1. Global-Local Unified MLLM

  • GLUS uses sparse context frames for video-wide reasoning and continuous query frames for local mask prediction, combining both in one MLLM.Query frames follow context frames so the causal MLLM can use video context while generating segmentation results.Source: §4.1 · pp. 4–5

4.2. End-to-end Memory Banks for MLLMs

  • Jointly optimizing the MLLM and a pre-trained VOS memory bank lets GLUS use historical information while avoiding an external VOS model at inference.The memory bank is integrated into query-frame decoding, and training simulates streaming predictions across query frames.Source: §4.2 · p. 5

4.3. Object Contrastive Loss

  • Object contrastive loss separates ⟨SEG⟩ tokens for different objects, treating expressions for the same object as positives and maintaining a bank of other objects as negatives.The loss is computed only when positive pairs are present; it is applied to MeViS samples because Ref-Youtube-VOS has fewer multiple-expression cases.Source: §4.3 · pp. 5–6

4.4. Self-refined Selection and Propagation

  • GLUS prediction quality supplies pseudo-labels to guide video-QA key-frame selectors, addressing the possibility that sparse context frames miss crucial information.GLUS annotates training frames with IoU scores for fine-tuning the selector.Source: §4.4 · p. 6
  • For propagation, GLUS starts from the selected key frame as the first query frame and performs RefVOS in both forward and backward directions.Unlike prior methods described here, this uses GLUS itself rather than an external VOS model for propagation.Source: §4.4 · p. 6

5. Experiments

Experiments evaluate GLUS on referring and reasoning video segmentation benchmarks and test its core design choices. Results support the global-local strategy, memory bank, contrastive loss, key-frame selection, and internal propagation, while exposing data-scarcity-related overfitting.

5.1. Datasets and Evaluation Metrics

  • Evaluation spans MeViS, Ref-Youtube-VOS, and reasoning datasets; standard SFT uses the first two, while additional SFT adds ReVOS, LV-VIS, and DAVIS-17.The training setup uses an eight-frame context window split into four context frames and four query frames.Source: §5.1 · p. 6, §5.2 · p. 6

5.3. Referring VOS Comparison

  • On MeViS, GLUS exceeds other MLLM-based RefVOS models by over 5% J &F and DsHmp [13] by approximately 4%; it also improves on Ref-Youtube-VOS, ReVOS, and ReasonVOS.The supplied results describe GLUS as outperforming most MLLM-based models on Ref-Youtube-VOS and improving significantly on both reasoning datasets.Source: §5.3 · p. 7, Table 2 · p. 7, Table 3 · p. 7

5.4.1. Global-local Unified Reasoning

  • Global-local unification outperforms either global-only or local-only reasoning in the reported ablation.Table 4 identifies this design as GLU.Source: §5.4.1 · p. 7, Table 4 · p. 7

5.4.2. End-to-end Memory Bank

  • End-to-end training and inference with SAM-2 memory attention improves both RefVOS datasets and enhances mask consistency.The qualitative comparison associates the memory bank with improved temporal consistency.Source: §5.4.2 · p. 8, Figure 3 · p. 8

5.4.3. Object Contrastive Loss

  • Object contrastive loss improves performance despite being trained only on MeViS and corrects misidentified referred objects in qualitative results.The reported design aims to refine fine-grained representations of the segmentation tokens.Source: §5.4.3 · p. 8

5.4.4. Self-refined Key Frame Selection & Propagation

  • The pseudo-label-fine-tuned key-frame selector significantly outperforms off-the-shelf selectors.The comparison includes no selection and off-the-shelf selectors from LLaMA-VID and Chat-UniVi.Source: §5.4.4 · p. 8, Table 5 · p. 8

5.4.5. Propagation without External VOS

  • For propagation from selected key frames, GLUS significantly outperforms Cutie [8] and SAM-2 [32].The authors connect this comparison to using global video information and referring expressions for propagation.Source: §5.4.4 · p. 8, §5.4.5 · p. 9, Table 6 · p. 9

5.4.6. Sampling Ratio of Training Datasets

  • A 1:1 MeViS:Ref-Youtube-VOS sampling ratio balances performance and training efficiency; limited video data is associated with overfitting and final-step validation drops.Object contrastive loss alleviates overfitting, but the models still show a significant drop at the final training steps.Source: §5.4.6 · p. 9, Table 7 · p. 9, §5.4.7 · p. 9

6. Conclusion

GLUS unifies global and local reasoning in one RefVOS MLLM, using context and query frames with an optimized VOS memory module. Object contrastive loss and key-frame pseudo-labeling support object and frame selection within the context window.

6. Conclusion

  • GLUS combines global context frames and local query frames in one MLLM to address both “Ref” and “VOS” challenges.End-to-end optimized VOS memory modules further improve consistency.Source: §6 · p. 9
  • Object contrastive loss and key-frame pseudo-labeling help the MLLM distinguish the correct object and frame within its limited context window.Source: §6 · p. 9
  • GLUS establishes new state-of-the-art results on RefVOS benchmarks.Source: §6 · p. 9

Supplementary Material

The supplementary demo presents six qualitative comparisons of GLUS with DsHmp on MeViS. The examples illustrate motion, global, and vision-language reasoning challenges in RefVOS.

A. Demo Video

  • Six qualitative comparisons evaluate GLUS against the previous state-of-the-art DsHmp using MeViS videos.Source: A. Demo Video · p. 12
  • The examples address motion understanding, global reasoning, and vision-language reasoning in challenging language-guided segmentation cases.Source: A. Demo Video · p. 12
  • The authors report that the six examples demonstrate GLUS effectively tackling challenging RefVOS cases.Source: A. Demo Video · p. 12

B. Implementation Details

The section introduces a detailed account of GLUS’s model architectures and workflow.

B. Implementation Details

B.1. Model Architectures

GLUS processes context and query frames through a vision backbone and projection layer, then uses segmentation-token embeddings and image features to decode query-frame masks. A memory bank stores mask features for later attention.

B.1. Model Architectures

  • The vision backbone processes context and query frames individually, and a vision-to-language projection layer produces the MLLM input embeddings.Source: §B.1 · p. 12
  • The MLLM, projector, and vision backbone are initialized from LISA-7B-v1.Source: §B.1 · p. 12
  • The mask decoder uses a projected hidden embedding from the MLLM as a prompt alongside encoded query-image features to generate a mask.The hidden embedding is taken from the MLLM’s penultimate layer.Source: §B.1 · p. 12
  • Each generated mask is encoded and stored in the memory bank; memory attention uses features from up to m stored masks with the decoded image.The memory attention module and memory encoder follow SAM-2.Source: §B.1 · p. 12

B.2. GLUS Training Details

GLUS training and inference use a memory bank alongside the MLLM, while a separately fine-tuned selector identifies key frames for bidirectional propagation.

B.2. GLUS Training Details

  • Training makes only the LoRA-fine-tuned MLLM, mask decoder, and projection layers trainable, with memory-bank sampling set to every frame.Training uses 3,000 steps and takes approximately 25 hours on four NVIDIA A100 GPUs.Source: §B.2 · p. 12
  • The training objective combines cross-entropy, mask DICE and BCE losses, and contrastive loss, each weighted to compute its respective average.Source: §B.2 · p. 12

B.3. GLUS Inference Details

  • Inference uses four query frames in a sliding window with stride one, carries the last query mask into the next window, and samples memory every three frames.Mask attention uses at most seven masks.Source: §B.3 · p. 12

B.4. Selector training and inference

  • The selector is fine-tuned using frame-IoU pseudo-labels from GLUS masks and text-output supervision, with an L1 score loss and cross-entropy text loss.Pseudo-labels come from an early-stop GLUS model; only half the training set is labeled for selector fine-tuning.Source: §B.4 · p. 13
  • At test time, the selector scores frames for a video-expression pair, and GLUS tracks from the highest-scoring frame in both directions.Predictions are generated iteratively across the full video.Source: §B.4 · p. 13

C. Additional Studies

The additional study reports stable GLUS performance across memory-bank sampling strides and sets stride three as the default.

C. Additional Studies

D. Limitations and Future Works

The authors identify limits in video understanding from fine-tuning, constrained context length, and the amount of available data.

D. Limitations and Future Works

  • Because the work focuses on MLLM fine-tuning, the visual backbone and LLM remain limited in video understanding.The authors suggest starting future work from an MLLM pretrained for video understanding to further enhance motion understanding.Source: D. Limitations and Future Works · pp. 13–14
  • Computational resources constrain context length, requiring downsampled visual features and only four context frames, which may miss critical contexts during fast motion.Source: D. Limitations and Future Works · p. 14
  • The authors identify the amount of data as a bottleneck for video reasoning and point to improving data scale and quality as future work.Source: D. Limitations and Future Works · p. 14
Loading 2504.07962v1…