Source-linked AI summary
MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes
Henghui Ding, Kaining Ying, Chang Liu, Shuting He, Xudong Jiang, Yu-Gang Jiang, Philip H. S. Torr, Song Bai
TL;DR
Existing VOS benchmarks and methods perform strongly on salient, isolated objects but provide limited evidence for realistic complex scenes. This paper introduces MOSEv2, a larger and more challenging dataset, and evaluates VOS and VOT methods across complex conditions. Current methods show consistent performance drops on MOSEv2, while the authors analyze failures and propose practical performance-enhancement tricks.
Problem
Existing VOS datasets mainly contain salient, dominant, isolated objects, leaving realistic complex-scene performance insufficiently represented.
Method
The paper constructs MOSEv2 with intensified and newly introduced real-world challenges, then benchmarks VOS and VOT methods across multiple settings and analyzes failures.
Results
Current methods show consistent performance drops across VOS and VOT; SAM2 falls from 76.4% J&F on MOSEv1 to 50.9% on MOSEv2.
Takeaways & Limitations
MOSEv2 exposes a gap between existing video algorithms and the demands of realistic, complex scenes across segmentation and tracking.
Takeaways & Limitations
Existing methods struggle with re-identification after disappearance or occlusion, densely crowded small targets, and camouflaged or non-physical objects.
Abstract
from arXiv · showhide
Video object segmentation (VOS) aims to segment specified target objects throughout a video. Although state-of-the-art methods have achieved impressive performance (e.g., 90+% J&F) on benchmarks such as DAVIS and YouTube-VOS, these datasets primarily contain salient, dominant, and isolated objects, limiting their generalization to real-world scenarios. To bridge this gap, the coMplex video Object SEgmentation (MOSEv1) dataset was introduced to facilitate VOS research in complex scenes. Building on the foundations and insights of MOSEv1, we present MOSEv2, a significantly more challenging dataset designed to further advance VOS methods under real-world conditions. MOSEv2 consists of 5,024 videos and 701,976 high-quality masks for 10,074 objects across 200 categories. Compared to its predecessor, MOSEv2 introduces much greater scene complexity, including {more frequent object disappearance and reappearance, severe occlusions and crowding, smaller objects, as well as a range of new challenges such as adverse weather (e.g., rain, snow, fog), low-light scenes (e.g., nighttime, underwater), multi-shot sequences, camouflaged objects, non-physical targets (e.g., shadows, reflections), and scenarios requiring external knowledge.} We benchmark 20 representative VOS methods under 5 different settings and observe consistent performance drops on MOSEv2. For example, SAM2 drops from 76.4% on MOSEv1 to only 50.9% on MOSEv2. We further evaluate 9 video object tracking methods and observe similar declines, demonstrating that MOSEv2 poses challenges across tasks. These results highlight that despite strong performance on existing datasets, current VOS methods still fall short under real-world complexities. Based on our analysis of the observed challenges, we further propose several practical tricks that enhance model performance. MOSEv2 is publicly available at https://MOSE.video.
1 INTRODUCTION
MOSEv2 extends complex-scene VOS benchmarking beyond salient, isolated-object datasets by intensifying prior challenges and adding diverse real-world conditions. Its benchmarks show substantial performance degradation across VOS and VOT methods, while analysis motivates practical improvements.
- Dataset motivation and scope: 5,024 videos contain 701,976 high-quality masks for 10,074 objects spanning 200 categories.The dataset expands MOSEv1 with broader scale and category coverage.
- Benchmark findings: 50.9% J&F is achieved by SAM2 on MOSEv2, down from 76.4% on MOSEv1 and 90.7% on DAVIS 2017.Cutie similarly declines from 69.9% on MOSEv1 to 43.9% on MOSEv2.
- Benchmark findings: 36.1% AUC is achieved by SAMURAI on MOSEv2, compared with 74.2% AUC on LaSOT.Nine state-of-the-art video object tracking methods show similar performance drops, extending MOSEv2’s challenge beyond VOS.
- Dataset motivation and scope: MOSEv2 intensifies disappearance-reappearance, occlusion, crowding, and small-object challenges from MOSEv1 while adding adverse weather, low light, multi-shot sequences, camouflage, non-physical targets, and knowledge-dependent scenarios.These conditions include rain, snow, fog, nighttime, underwater scenes, shadows, reflections, and cases requiring high-level reasoning.
- Benchmark findings: MOSEv2 provides comparative analyses and comprehensive benchmarks across multiple VOS settings and video object tracking.The study also analyzes failures and proposes practical tricks intended to enhance performance in complex scenarios.
2 RELATED WORK
Related work covers VOS formulations and methods, neighboring video perception tasks, and complex-scene understanding. The discussion positions MOSEv2 among segmentation, tracking, and occlusion-focused research directions.
- Video Object Segmentation: VOS includes semi-supervised, unsupervised, interactive, and referring settings defined by how the target object is specified.Semi-supervised VOS commonly uses a first-frame mask, while referring VOS uses a text expression.
- Video Object Segmentation: Propagation-based methods use previous-frame masks, whereas matching-based methods compare pixel features with stored target embeddings.These are two broad categories of semi-supervised VOS methods.
- Video Object Segmentation: SAM2 introduced promptable visual segmentation using positive or negative clicks, bounding boxes, or masks on any video frame.Subsequent variants add constrained tree search, distractor-aware memory, motion modeling, and adaptive memory selection.
- Related video tasks: Related video tasks include video instance, semantic, panoptic segmentation, and video object tracking.These tasks address combinations of detection, segmentation, tracking, semantic labeling, and temporal consistency.
- Complex Scene Understanding: Complex-scene understanding has attracted substantial attention in images, including methods for separating overlapping objects and recovering occluded content.The passage contrasts this image-domain progress with comparatively less-developed video-domain occlusion understanding.
3 MOSEV2 DATASET
MOSEv2 is constructed as a larger, more diverse benchmark for video object segmentation in complex scenes, with challenging objects, environments, and temporal dynamics. Its statistics and evaluation protocol target occlusion, long-term tracking, small-object segmentation, and realistic scene complexity.
- Video Collection and Dataset Scale: The collection rules prioritize multiple and crowded objects, occlusions, disappearance and reappearance, diverse scales and visibility, motion, and diversified categories.Longer videos and challenging cases such as camouflage are explicitly encouraged.
- Video Collection and Dataset Scale: MOSEv2 deliberately expands difficult environments and scenarios, including low light, adverse weather, multi-shot changes, and knowledge-dependent tasks.Selection rules include rain, fog, snow, spatial or appearance changes across shots, optical character recognition, spatial reasoning, physical principles, and multi-view understanding.
- Video Annotation: Annotations begin with manually selected targets and first-frame masks, then track and annotate masks across subsequent frames using an interactive tool assisted by SAM2.A verification team reviews all videos after annotation to ensure mask quality.
- Video Collection and Dataset Scale: MOSEv2 combines 2,149 inherited videos with 2,875 newly collected videos, totaling 5,024 videos, 701,976 masks, 10,074 objects, and 200 categories.The new videos come from self-captured real-world scenarios and copyright-free internet sources.
- Dataset Statistics: MOSEv2 includes 183 videos exceeding 300 frames, reaches 7,825 frames, and has a 93.2-frame average length, while its reappearance rate reaches 50.3%.Long videos are designed around richer dynamics such as disappearance, occlusion, scene transitions, and multi-shot clips.
- Dataset Statistics: MOSEv2 expands adverse-scenario coverage and object attributes, including 4,931 occluded objects, 5,067 disappearance-reappearance objects, 609 novel-category objects, 277 multi-shot objects, and 256 knowledge-dependent objects.It also includes 159 rainy, 142 heavy-rain, 73 snowy, 60 foggy, 280 underwater, 255 nighttime, and 50 disaster videos.
- Evaluation Metrics: Evaluation uses region similarity J and contour accuracy F, while an adaptive boundary threshold produces a scale-aware contour metric for small objects.The adaptive metric uses object area A and scaling factor α, with α set to 0.1.
4 EXPERIMENTS
MOSEv2 is evaluated across multiple VOS settings and video object tracking, revealing substantial weaknesses in current methods under complex real-world conditions. The analysis also identifies accuracy–efficiency trade-offs and practical improvements for SAM2.
- Experimental setup: 20 representative VOS methods are benchmarked across semi-supervised, unsupervised, and interactive settings, alongside nine video object tracking methods.The experiments include mask-, box-, and point-initialization settings.
- Semi-supervised VOS: 47.1% J&F is achieved by SAM2-B+ with mask initialization on MOSEv2, versus 74.7% on MOSEv1 and 90.2% on DAVIS17.Reappearance remains especially difficult, with scores ranging from 7.8% to 34.9% across methods.
- Unsupervised and interactive VOS: 36.0% J&F is the highest score for unsupervised VOS, while interactive VOS reaches only 54.1% J&F@60s.These results remain well below the corresponding DAVIS17 scores.
- Challenge-specific analysis: 42.9% J&F is achieved by SAM2Long-B+ overall, but its score falls to 52.9% on MOSEv2’s LD subset versus 84.3% on LVOSv2.Knowledge-dependent scenarios are also difficult, with Cutie-B achieving 30.5% J&F.
- Practical improvements and efficiency: 46.0% to 51.5% J&F is the improvement obtained by applying the proposed tricks to SAM2-B+, a +5.5% gain.SAM2-L improves from 49.7% to 54.4%, a +4.7% increase, while computational trade-offs remain evident: SAM2Long-L runs at 7.1 FPS versus 49.8 FPS for XMem.
5 DISCUSSION AND FUTURE DIRECTIONS
The discussion identifies robust re-identification, occlusion handling, and tracking of small targets in crowds as central challenges for complex VOS. It points toward adaptive, occlusion-aware, and high-resolution approaches.
- Robust Re-identification for Disappearance-Reappearance: Re-identification after disappearance is especially difficult under complex motion, viewpoint changes, and knowledge-dependent cues.Aggressive matching can increase false positives during disappearance and reduce disappearance-handling performance.
- Occlusion Handling: Frequent and complex occlusions motivate models that preserve object identity when targets are partly or fully hidden.Suggested directions include hidden-region attention, multi-scale feature fusion, and temporal modeling.
- Tracking in Crowded and Small-Target Scenarios: Small objects and crowded scenes challenge tracking because limited input resolutions lose fine details.Proposed directions include efficient high-resolution processing, multi-scale architectures, small-object-focused attention, and contrastive learning.
6 CONCLUSION
MOSEv2 expands complex-scene VOS evaluation in both dataset scale and challenge diversity. Its benchmarks expose substantial performance gaps across segmentation and tracking, while proposed tricks improve SAM2 performance.
- Dataset and challenges: 5,024 high-resolution videos and 701,976 object masks across 200 categories comprise MOSEv2.The dataset extends MOSEv1 with more severe existing challenges and new adverse-weather, low-light, multi-shot, camouflage, non-physical-target, and knowledge-dependent cases.
- Benchmark findings: 90.7% to 50.9% J&F is SAM2’s decline from DAVIS 2017 to MOSEv2.Evaluation across multiple VOS and VOT settings reveals significant performance drops for current state-of-the-art methods.
- Practical outcome: +5.5% J&F is the reported gain from the practical improvements applied to SAM2.The authors position MOSEv2 as a resource for advancing robust and generalizable segmentation and tracking in diverse environments.