PaperlayerView exact revision on arXiv

Source-linked AI summary

MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes

Henghui Ding, Kaining Ying, Chang Liu, Shuting He, Xudong Jiang, Yu-Gang Jiang, Philip H. S. Torr, Song Bai

arXiv:2508.05630v2cs.CV

TL;DR

Common VOS benchmarks are nearing saturation, while complex-scene segmentation remains challenging. MOSEv2 extends MOSEv1 with more complex real-world videos; evaluations find performance drops across VOS and VOT, with SAM2’s J&F falling from 76.4% on MOSEv1 to 50.9% on MOSEv2.

  • Problem

    Common VOS benchmarks show near-saturated results, motivating a more challenging dataset for complex-scene VOS.

    Source: §1 · pp. 1, 3

  • Method

    The authors introduce MOSEv2, extending MOSEv1 with inherited and newly collected videos, and benchmark video object segmentation and tracking methods.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    Across VOS and VOT evaluations, current methods show significant performance drops on MOSEv2; SAM2’s J&F falls from 76.4% on MOSEv1 to 50.9% on MOSEv2.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    MOSEv2 highlights a gap between current algorithms and the demands of real-world deployment.

    Source: §6 · p. 14

    Existing VOS models still struggle to re-identify objects after disappearance or handle densely crowded scenes with small, heavily occluded targets.

    Source: §4.6 · p. 11

Abstract

from arXiv · show

Video object segmentation (VOS) aims to segment specified target objects throughout a video. Although state-of-the-art methods have achieved impressive performance (e.g., 90+% J&F) on benchmarks such as DAVIS and YouTube-VOS, these datasets primarily contain salient, dominant, and isolated objects, limiting their generalization to real-world scenarios. To bridge this gap, the coMplex video Object SEgmentation (MOSEv1) dataset was introduced to facilitate VOS research in complex scenes. Building on the foundations and insights of MOSEv1, we present MOSEv2, a significantly more challenging dataset designed to further advance VOS methods under real-world conditions. MOSEv2 consists of 5,024 videos and 701,976 high-quality masks for 10,074 objects across 200 categories. Compared to its predecessor, MOSEv2 introduces much greater scene complexity, including {more frequent object disappearance and reappearance, severe occlusions and crowding, smaller objects, as well as a range of new challenges such as adverse weather (e.g., rain, snow, fog), low-light scenes (e.g., nighttime, underwater), multi-shot sequences, camouflaged objects, non-physical targets (e.g., shadows, reflections), and scenarios requiring external knowledge.} We benchmark 20 representative VOS methods under 5 different settings and observe consistent performance drops on MOSEv2. For example, SAM2 drops from 76.4% on MOSEv1 to only 50.9% on MOSEv2. We further evaluate 9 video object tracking methods and observe similar declines, demonstrating that MOSEv2 poses challenges across tasks. These results highlight that despite strong performance on existing datasets, current VOS methods still fall short under real-world complexities. Based on our analysis of the observed challenges, we further propose several practical tricks that enhance model performance. MOSEv2 is publicly available at https://MOSE.video.

1 INTRODUCTION

MOSEv2 expands complex-scene video object segmentation with a larger dataset and new real-world challenges; benchmarks show substantial performance declines for both VOS and video object tracking methods.

Motivation

  • MOSEv2 builds on MOSEv1, addressing the gap left by standard datasets whose targets are typically salient and isolated despite near-saturation results.On MOSEv1, XMem achieved 57.6%, while subsequent methods such as SAM2 raised performance to 76.4%.Source: §1 · p. 1

MOSEv2 dataset

  • MOSEv2 contains 5,024 videos, 10,074 objects, and 701,976 masks across 200 categories, with intensified existing challenges and new real-world complexities.It features more frequent disappearance and reappearance, severe occlusion and crowding, smaller objects, adverse weather, low light, multi-shot sequences, camouflage, non-physical targets, and knowledge-dependent scenarios. Examples include OCR-dependent object distinction and targets inferred from surrounding motion despite being invisible.Source: §1 · pp. 1–3

Benchmark results

  • SAM2 falls from 76.4% on MOSEv1 to 50.9% on MOSEv2, while Cutie declines from 69.9% to 43.9%.The paper attributes these consistent drops to the more realistic and complex scenarios in MOSEv2.Source: §1 · p. 3
  • SAMURAI achieves 74.2% AUC on LaSOT but 36.1% on MOSEv2, showing that the dataset also challenges video object tracking.The paper benchmarks nine state-of-the-art VOT methods and reports consistent performance drops on MOSEv2.Source: §1 · p. 3

Benchmark scope

  • The evaluation benchmarks 20 representative VOS methods across mask-, box-, and point-initialized semi-supervised VOS, unsupervised VOS, interactive VOS, and video object tracking.The authors also report a comparative analysis of MOSEv2 against existing VOS and VOT datasets.Source: §1 · p. 3

Analysis and practical improvements

  • The authors analyze model performance and failure cases on MOSEv2, then propose practical tricks to improve performance in complex scenarios.They also outline future directions for robust video understanding in the wild.Source: §1 · p. 3

2 RELATED WORK

Related work spans multiple VOS settings and adjacent video segmentation and tracking tasks, while complex-scene understanding in video remains underexplored. MOSEv2 is presented as a dataset to support research on these challenges.

2.1 Video Object Segmentation

  • Semi-supervised VOS uses a first-frame mask and primarily follows propagation-based or matching-based approaches.Propagation methods use prior-frame predictions to guide current-frame segmentation; matching methods compare pixel features with an encoded target. Some methods instead use a first-frame bounding box, as in SiamMask.Source: §2.1 · p. 3
  • SAM2 enables promptable video segmentation, and subsequent variants target error accumulation, distractors, and dynamic scenes.Examples include SAM2Long, DAM4SAM, SAMURAI [25], and MoSAM [59]. Interactive VOS also uses user input to correct predictions and propagate masks across frames; SAM2 supports this setting with flexible interactions.Source: §2.1 · pp. 3–4
  • Referring VOS segments objects from text expressions, while interactive VOS uses user interactions such as clicks or scribbles.Referring methods include bottom-up segmentation and propagation or top-down tracklet selection; recent work explores temporal modeling, multimodal large language models, and expressions across modalities.Source: §2.1 · pp. 3–4
  • Unsupervised VOS automatically segments primary objects without manual input, using local or contextual encoding and, more recently, propagation frameworks.DEVA combines image-level segmentation with class-agnostic temporal propagation; EntitySAM automatically discovers and tracks entities without explicit prompts.Source: §2.1 · p. 4

2.2 Related Video Segmentation and Tracking Tasks

  • Related video tasks include instance, semantic, and panoptic segmentation, plus bounding-box-based object tracking.VOS focuses on segmentation, whereas VOT locates an initially specified target with bounding boxes; MOSEv2 also supports VOT and includes challenging crowds, occlusions, and disappearance-reappearance. VIS concerns predefined-category instances, while VSS emphasizes temporal consistency and efficiency, and VPS tracks foreground instances and background stuff.Source: §2.2 · p. 4

2.3 Complex Scene Understanding

  • Video occlusion understanding remains underexplored, and MOSEv2 is designed to facilitate complex-scene research in VOS and related video tasks.Prior video work cited here includes multi-object tracking approaches to occlusion, while the paper contrasts this limited coverage with broader complex-scene understanding in images.Source: §2.3 · p. 4

3 MOSEV2 DATASET

MOSEv2 combines a large, diverse collection of complex videos with annotation and evaluation procedures designed to test segmentation under real-world challenges. Its statistics show frequent disappearance, crowding, occlusion, small objects, and difficult environments, while its refined metrics address scale bias and target absence or return.

3.1 Video Collection and Annotation

  • MOSEv2 combines 2,149 videos inherited from MOSEv1 with 2,875 newly collected videos selected for diverse, challenging scenes.Selection rules favor multiple or crowded objects, occlusions and repeated disappearance-reappearance, diverse scales, motion, novel categories, adverse environments, multi-shot clips, and knowledge-dependent scenarios.Source: §3.1 · p. 5
  • Annotators track first-frame targets across subsequent frames, leave masks empty during disappearance or full occlusion, and produce annotations at a minimum of 5 FPS.A SAM2-assisted interactive tool supports annotation, and a verification team reviews completed masks.Source: §3.1 · p. 5

3.2 Dataset Statistics

  • MOSEv2 contains 5,024 videos, 701,976 mask annotations, 10,074 objects, and 200 categories, expanding MOSEv1 with 2,875 new videos.Its category set includes rare and non-physical targets such as camouflaged objects and shadows.Source: §3.2 · p. 5, Table 1 · p. 6
  • MOSEv2 records 61.8% disappearance and 50.3% reappearance rates, 13.6 distractors per object, and a 47.0 mean occlusion rate.The reported comparisons include higher disappearance and reappearance rates than MOSEv1, more distractors than MOSEv1, SA-V, LVOSv2, and DiDi, and higher mean occlusion than MOSEv1 and SA-V.Source: §3.2 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
  • MOSEv2 has 50.2% small masks, 183 videos longer than 300 frames, and a maximum video length of 7,825 frames.The average video length is 93.2 frames; the paper notes that long videos were selected for complex dynamics and scenarios, not duration alone.Source: §3.2 · p. 6, Figure 4 · p. 7, Figure 5 · p. 7
  • MOSEv2 expands challenging environment and object-attribute coverage, including 280 underwater videos and 5,067 objects with disappearance-reappearance attributes.It also adds attributes for multi-shot sequences and knowledge dependency, and reports increased counts for occlusion, complex environments, and long duration versus MOSEv1.Source: §3.2 · p. 7, Figure 6 · p. 7, Table 3 · p. 7

3.3 Evaluation Metrics

  • The authors retain J&F evaluation and introduce an adaptive boundary threshold for F to reduce fixed-threshold bias against small objects.The fixed-threshold F can score a non-overlapping 955-pixel chopstick 0.91, whereas the adaptive metric scores it 0; for a 522k-pixel person, both score 0.74.Source: §3.3 · p. 7, Figure 8 · p. 8
  • Dedicated disappearance and reappearance scores evaluate those clips separately, reducing bias from empty-target-frame proportions in video-wide averages.The reappearance score excludes the target’s initial continuous presence and evaluates recovery after it disappears.Source: §3.3 · p. 8, Figure 9 · p. 8

4 EXPERIMENTS

The experiments benchmark mask-, box-, and point-initialized semi-supervised VOS, unsupervised and interactive VOS, and video object tracking on MOSEv2. Models are evaluated on the validation set using standard and challenge-specific metrics, with J&̇F as the primary metric.

4.1 Semi-supervised Video Object Segmentation

  • 51.5% J&̇F: SAM2Long-L leads mask initialization, but reappearance remains difficult, with methods scoring 7.8%–34.9% on J&̇F_r.Cutie-B reaches 42.8% J&̇F and 64.5% J&̇F_d, but its 18.3% J&̇F_r is 4.9% below SAM2-B+.Source: §4.1 · pp. 8–9, Table 4 · p. 9
  • 50.2% J&̇F: SAM2Long-L achieves the best box-initialized result, while all methods fare relatively worse on reappearance than disappearance.The reported MOSEv2 results also show a performance gap from benchmarks such as DAVIS17.Source: Table 5 · p. 9, §4.1 · p. 9
  • 48.5% J&̇F: SAM2Long-L achieves the best point-initialized result using one click; adding clicks from one to five does not consistently improve performance.SAM2Long-L scores 86.1% J&̇F on DAVIS17, compared with its much lower MOSEv2 result.Source: Table 6 · p. 9, §4.1 · pp. 9–10

4.2 Unsupervised Video Object Segmentation

  • 36.0% J&F: DEVA achieves the highest unsupervised VOS score, while reappearance scores fall to 4.1%–12.6% and SAM2Long-L reaches 29.1% J&̇F.DEVA scores 73.4% on DAVIS17; SAM2-based methods use first-frame grid prompts to generate candidate masks, whose incompleteness limits effectiveness.Source: Table 7 · p. 10, §4.2 · p. 10

4.3 Interactive Video Object Segmentation

  • 54.1% J&F@60s: STCN leads the interactive VOS methods, well below its 88.8% score on DAVIS17.The evaluation uses an initial scribble and up to eight rounds of corrective scribbles, with a 30-second limit per object.Source: Table 8 · p. 10, §4.3 · p. 10

4.4 Video Object Tracking

  • 36.1% AUC: SAMURAI-L leads tracking on MOSEv2, while traditional trackers score 23.3%–27.8% AUC.SAMURAI-L scores 60.9% AUC on MOSEv1, 74.2% on LaSOT, and 81.7% on GOT-10k; larger L variants outperform their B+ counterparts.Source: Table 9 · p. 10, §4.4 · p. 10

4.5 Attribute-Based Performance Analysis

  • 42.9% J&̇F: SAM2Long-B+ leads overall in the attribute analysis, while fine-tuning raises SAM2-B+ from 36.8% to 40.7%.Cutie-B performs strongly on disappearance, including 81.8% J&̇F_d on long-duration videos, but scores only 13.5% J&̇F_r there.Source: Table 10 · p. 11, §4.5 · p. 11

4.7 Enhancing SAM2 for Complex Scenarios

  • 54.4% J&̇F: applying all improvements raises SAM2-L from 49.7% to 54.4%, while SAM2-B+ rises from 46.0% to 51.5%.SAM2-L* achieves lower J&̇F_r than SAM2-B+, but the stronger L backbone maintains a better balance between J&̇F_d and J&̇F_r.Source: Table 12 · p. 13, §4.7 · p. 13

5 DISCUSSION AND FUTURE DIRECTIONS

MOSEv2 highlights persistent challenges for video object segmentation in complex real-world conditions. The discussion proposes research directions addressing identity, visibility, scale, category generalization, environmental change, shot transitions, and reasoning.

Robust Re-identification for Disappearance-Reappearance

  • A large drop in J&F_r scores highlights the challenge of re-identifying objects after disappearance and reappearance.Complex motion, viewpoint changes, and knowledge-dependent cues complicate matching; overly aggressive matching can create false positives during disappearance and reduce J&F_d. Future strategies should combine appearance, motion, and semantic reasoning adaptively.Source: §5 · pp. 13–14

Occlusion Handling

  • Frequent, complex occlusions cause current methods to fail when objects are partly or fully hidden.Suggested directions include attention to hidden regions, multi-scale feature fusion, and temporal models that preserve object identity through occlusion.Source: §5 · p. 14

Tracking in Crowded and Small-Target Scenarios

  • Small objects and crowded scenes challenge tracking, while limited input resolutions can erase fine details.The passage cites 480p in Cutie and 1024p in SAM2; suggested approaches include efficient high-resolution processing, multi-scale architectures, small-object-focused attention, and contrastive learning against similar distractors.Source: §5 · p. 14

Generalization to Rare Categories

  • Methods struggle to generalize to rare or unseen categories in MOSEv2’s 200-category long-tail distribution.Shadows and camouflaged objects are examples; limited training data and domain gaps contribute to failures. Proposed directions include test-time adaptation using first-frame cues and stronger instance-level representations.Source: §5 · p. 14

Environmental Robustness

  • Adverse environments—including rain, snow, fog, nighttime, and underwater scenes—severely degrade current VOS performance.Low visibility, illumination changes, and environmental occlusions undermine appearance reliability and temporal consistency; proposed directions include weather-invariant and illumination-robust representations, auxiliary signals, or priors.Source: §5 · p. 14

Multi-Shot Video Handling

  • Scene transitions in multi-shot videos disrupt appearance and position continuity, making identity preservation difficult.Shot-aware strategies are proposed to handle these discontinuities while preserving object identity across shots.Source: §5 · p. 14

Knowledge-Dependent Tracking

  • Methods relying mainly on appearance and position still struggle with tracking that requires external knowledge, spatial reasoning, or common sense.Integrating MLLMs, could improve semantic understanding and high-level reasoning, but must preserve computational efficiency and real-time performance.Source: §5 · p. 14

6 CONCLUSION

MOSEv2 expands MOSEv1 in scale and scenario complexity, while evaluations show substantial performance drops for current methods and practical tricks yield a 5.5% J&F gain.

6 CONCLUSION

  • MOSEv2 comprises 5,024 high-resolution videos and 701,976 object masks across 200 categories, extending MOSEv1 in scale and scenario complexity.It intensifies disappearance-reappearance, occlusion, and crowding challenges and adds adverse weather, low light, multi-shot sequences, camouflage, non-physical targets, and knowledge-dependent cases.Source: §6 · p. 14
  • SAM2 drops from 90.7% J&F on DAVIS 2017 to 50.9% on MOSEv2, while evaluations across multiple VOS and VOT settings reveal significant performance drops.The results highlight a gap between existing algorithms and real-world deployment demands.Source: §6 · p. 14
  • The proposed practical tricks substantially enhance model performance, yielding a 5.5% J&F gain.The authors present MOSEv2 as a resource for advancing robust and generalizable video object segmentation and tracking in diverse, unconstrained environments.Source: §6 · p. 14

Related papers

Loading 2508.05630v2…