Source-linked AI summary
MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation
Henghui Ding, Chang Liu, Shuting He, Kaining Ying, Xudong Jiang, Chen Change Loy, Yu-Gang Jiang
TL;DR
Existing referring video benchmarks often rely on static object attributes, leaving motion-centric and complex-scene understanding underexplored. The paper introduces MeViSv2, a large multi-modal dataset and four-task benchmark, and proposes LMPM++; results show existing methods remain limited while LMPM++ addresses identified challenges.
Problem
Existing referring video datasets emphasize salient objects and static attributes, underrepresenting motion in video and language.
Method
The paper constructs MeViSv2 with motion-centric text and audio expressions, annotations, and four supported tasks, then develops LMPM++ using language-conditional object embeddings and temporal reasoning.
Results
Benchmarks across RVOS, AVOS, RMOT, and RMEG show existing methods fall short on motion-expression-guided understanding, while LMPM++ achieves new state-of-the-art results.
Takeaways & Limitations
MeViSv2 provides a platform for developing motion expression-guided video understanding algorithms in complex video scenes.
Takeaways & Limitations
Performance remains challenged by motion expressions, highly dynamic or fast-paced objects, implicit information, and expressions with no actual target.
Abstract
from arXiv · showhide
This paper proposes a large-scale multi-modal dataset for referring motion expression video segmentation, focusing on segmenting and tracking target objects in videos based on language description of objects' motions. Existing referring video segmentation datasets often focus on salient objects and use language expressions rich in static attributes, potentially allowing the target object to be identified in a single frame. Such datasets underemphasize the role of motion in both videos and languages. To explore the feasibility of using motion expressions and motion reasoning clues for pixel-level video understanding, we introduce MeViS, a dataset containing 33,072 human-annotated motion expressions in both text and audio, covering 8,171 objects in 2,006 videos of complex scenarios. We benchmark 15 existing methods across 4 tasks supported by MeViS, including 6 referring video object segmentation (RVOS) methods, 3 audio-guided video object segmentation (AVOS) methods, 2 referring multi-object tracking (RMOT) methods, and 4 video captioning methods for the newly introduced referring motion expression generation (RMEG) task. The results demonstrate weaknesses and limitations of existing methods in addressing motion expression-guided video understanding. We further analyze the challenges and propose an approach LMPM++ for RVOS/AVOS/RMOT that achieves new state-of-the-art results. Our dataset provides a platform that facilitates the development of motion expression-guided video understanding algorithms in complex video scenes. The proposed MeViS dataset and the method's source code are publicly available at https://henghuiding.com/MeViS/
1 INTRODUCTION
MeViS reframes referring video understanding around motion-centric expressions in complex scenes, where static appearance may not identify the target. MeViSv2 expands the dataset’s modalities, annotations, supported tasks, and challenge coverage while benchmarking existing methods and introducing LMPM++.
- Motivation: Existing referring video datasets often use salient objects and static attributes, allowing targets to be identified from a single frame.MeViS instead emphasizes temporal motion characteristics in both videos and language.
- Dataset: 2,006 videos and 8,171 distinct objects form the MeViS foundation for motion-focused referring video understanding.The dataset was designed around complex scenarios with multiple interacting objects in motion.
- Dataset: 33,072 expressions in MeViSv2 add 4,502 challenging examples, including motion-reasoning and no-target expressions.These extensions cover implicit queries, deceptive descriptions, and cases without an actual referred object.
- Tasks and modalities: MeViSv2 adds audio expressions, tracking annotations, and support for RVOS, AVOS, RMOT, and RMEG.The dataset includes more than 150,000 seconds of audio expressions and introduces referring motion expression generation.
- Challenges: Motion expressions can span fleeting movements or long-term actions, requiring frame-level attention and temporal context across the video.The benchmark therefore targets both linguistic and visual challenges in motion understanding.
- Approach and evaluation: LMPM++ uses language-conditional queries, object embeddings, and an LLM to capture and reason over temporal context.Its object-token design is intended to process longer sequences than approaches based on frame features.
2 RELATED WORK
Related work spans referring image and video segmentation, audio-guided segmentation, and referring object tracking. MeViS extends these directions with motion-centric, multi-target, no-target, and pixel-level referring understanding.
- Referring image segmentation: Referring image segmentation identifies objects from language descriptions of properties such as location and color.Transformer-based methods subsequently became prominent in this area.
- Referring video segmentation: Referring video object segmentation tracks and segments an expression-specified object throughout a video.Earlier datasets commonly emphasized single objects and static attributes, whereas MeViS introduced numerous motion expressions.
- Referring video segmentation: Existing RVOS methods commonly adapt semi-supervised video object segmentation by replacing mask references with language references.Later approaches use cross-modal attention, temporal propagation, or Transformers.
- Audio-guided segmentation: Audio-guided video segmentation includes settings where audio identifies sound-emitting objects or speech specifies the target object.These settings distinguish audio-guided object segmentation from referring video segmentation.
- Referring object tracking: Referring object tracking localizes language-specified targets as bounding-box tracklets, with RSOT focusing on one target and RMOT supporting multiple targets.RMOT was introduced to address multi-object and temporally changing scenarios.
- MeViS distinction: MeViS supports single-target, multi-target, and no-target expressions while emphasizing long-term target states and finer-grained pixel-level perception.This broadens referring understanding beyond the narrower settings of earlier RSOT and RMOT datasets.
3 MEVIS DATASET
MeViS is constructed to make motion central to language-guided video understanding, using complex videos, motion-focused expressions, and multimodal annotations. Its scale and generalized expressions support RVOS alongside audio-guided segmentation, multi-object tracking, motion-expression generation, and additional applications.
- Video Collection: MeViS selects videos with multiple moving objects and similar appearances, excluding scenes dominated by one or two salient objects or minimal motion.After reviewing over 4,000 candidates, the authors selected 2,006 videos spanning complex real-world scenarios.
- Dataset Statistics: The dataset contains 2,006 videos, 8,171 objects, 33,072 expressions, and 443k annotation masks, exceeding Refer-YouTube-VOS in objects, expressions, and masks.The corresponding Refer-YouTube-VOS figures are 7,451 objects, 15,009 expressions, and 131k masks.
- Dataset Analysis: MeViS videos average 4.28 objects and 13.16 seconds, while target objects average 10.88 seconds, creating longer and more crowded motion-understanding scenarios.The average video duration is 5.45 seconds in Refer-YouTube-VOS, and MeViS target-object duration exceeds twice that dataset's average.
- Generalized Referring Expressions: Expressions can refer to multiple targets or no target, extending beyond the conventional one-expression-one-object setting.MeViSv2 adds no-target expressions to address sentences that do not match any object and to support robustness studies.
- Tasks Supported by MeViS: MeViS supports RVOS, AVOS, RMOT, and RMEG, with masks, bounding boxes, and motion-expression generation from videos and target-object masks.RMEG requires models to identify relevant objects and generate an expression that distinguishes their motion from other objects.
- Multimodal Expressions: MeViSv2 adds audio recordings for textual expressions, enabling audio-guided video object segmentation while preserving audio cues such as accent, emotion, speed, and noise.The audio includes synthesized and human-recorded speech, with recordings totaling more than 150,000 seconds.
4 LMPM++: A BASELINE APPROACH
LMPM++ combines language-guided object discovery, global temporal reasoning, multimodal inputs, and contrastive learning to generate object mask trajectories for motion-aware video understanding.
- Architecture: LMPM++ generates language-based queries across T frames to identify potential target objects and represent them with object embeddings.Language queries filter irrelevant objects before subsequent processing.
- Architecture: A large language model captures global temporal context from object embeddings and predicts the number of target objects with corresponding <SEG> tokens.The predicted tokens drive subsequent mask generation.
- Multimodal Input: Separate text and audio feature branches enable LMPM++ to process multimodal referring inputs.Text and audio prompts are encoded through distinct extraction pathways before integration with the language model.
- Generalized Referring Expressions: The answer template supports zero, single, and multiple targets by predicting No and exactly No <SEG> tokens.No = 0 represents no target, while No > 1 represents multiple target objects.
- Temporal Motion Perception: Temporal-level contrastive learning aligns language or audio embeddings with <SEG> embeddings having correct temporal order and separates incorrect orders or different objects.Negative samples include embeddings from different objects and disrupted temporal sequences.
- Training Objective: The training objective combines text classification, contrastive, binary cross-entropy, and DICE segmentation losses.The loss terms are balanced by λtxt, λbce, λdice, and λcon.
5 EXPERIMENTS
Experiments show that MeViS is substantially harder than conventional referring-video benchmarks because it requires motion reasoning, temporal understanding, and handling of multiple or absent targets. LMPM++ improves robustness across MeViS and related tracking and segmentation tasks, while remaining computationally efficient.
- Ablation Study of LMPM++: 3.7% and 4.1% J &F gains on MeViSv1 and MeViSv2, respectively, followed the addition of an LLM for temporal context and reasoning.The LLM variant also handled single-object, no-target, and multi-object expressions more effectively.
- Ablation Study of LMPM++: 12.76M additional trainable parameters were introduced by LMPM++ over LMPM, while object tokens made it faster and more parameter-efficient than VISA for long sequences.The trade-off is slower inference speed compared with LMPM.
- MeViS Benchmark Results: 27.8%–31.0% J &F was achieved by previous state-of-the-art methods on MeViSv1, versus usually above 60% J &F on conventional datasets.This gap demonstrates the difficulty of motion-expression-guided segmentation on MeViS.
- MeViS Benchmark Results: 5.6% lower J &F for DsHmp on MeViSv2 than MeViSv1 confirms that no-target and motion-reasoning expressions make the updated benchmark harder.The scores were 40.8% on MeViSv2 and 46.4% on MeViSv1.
- Results on Previous RVOS Datasets: 64.0% J &F on Refer-YouTube-VOS and 65.0% J &F on DAVIS17-RVOS were achieved by LMPM++ with the Video-Swin-Tiny and Video-Swin-Base backbones, respectively.With Video-Swin-Base, LMPM++ reached 67.8% J &F on Refer-YouTube-VOS.
- Referring Multi-Object Tracking Results: 38.1% HOTA*, 28.1% DetA*, and 87.4% T-acc. were achieved by LMPM++ on MeViSv2 RMOT, leading several metrics despite lower N-acc. than TransRMOT.TransRMOT achieved higher N-acc., 56.9% versus 45.7%.
6 CONCLUSION AND DISCUSSION
The paper presents MeViSv2 as a large-scale multi-modal dataset for motion-centric referring video understanding and benchmarks existing methods across four tasks. It identifies substantial limitations in current methods and proposes LMPM++ as a baseline approach, while highlighting several directions for future research.
- MeViSv2 advances referring video understanding through motion-centric language descriptions across diverse and complex scenarios.
- Benchmarks across RVOS, AVOS, RMOT, and RMEG reveal that existing methods struggle to leverage motion expressions effectively.
- LMPM++ is proposed as a baseline approach addressing challenges posed by the MeViS dataset.
- Future Directions: Future work includes stronger motion representations, models for diverse temporal motions and complex scenes, efficient handling of redundant objects, and improved cross-modal fusion.
- Future Directions: Additional directions include transfer learning, domain adaptation, and methods for open-world concepts in visual and linguistic domains.