PaperlayerView exact revision on arXiv

Source-linked AI summary

MOSE: A New Dataset for Video Object Segmentation in Complex Scenes

Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip H. S. Torr, Song Bai

arXiv:2302.01872v1cs.CV

TL;DR

Existing VOS benchmarks largely use salient, dominant, and isolated objects, leaving segmentation in complex scenes underexplored. The paper introduces MOSE, a large benchmark with complex occlusions, crowds, and disappearance-reappearance, and evaluates existing methods across four settings. Current methods perform substantially worse on MOSE, revealing unresolved challenges in complex-scene VOS.

  • Problem

    VOS under complex scenes has rarely been studied because existing datasets usually contain relatively salient, dominant, and isolated target objects.

  • Method

    The paper constructs MOSE and benchmarks 18 existing VOS methods under four settings covering mask, box, unsupervised, and interactive initialization.

  • Results

    Current VOS algorithms cannot well perceive complex scenes; DeAOT’s J&F drops from 92.9% on DAVIS 2016 to 59.4% on MOSE.

  • Takeaways & Limitations

    Complex scenes expose unresolved challenges in VOS, especially tracking objects that disappear and reappear after occlusion.

  • Takeaways & Limitations

    The BOR measure only roughly reflects occlusion and may misrepresent occlusion degree in MOSE.

Abstract

from arXiv · show

Video object segmentation (VOS) aims at segmenting a particular object throughout the entire video clip sequence. The state-of-the-art VOS methods have achieved excellent performance (e.g., 90+% J&F) on existing datasets. However, since the target objects in these existing datasets are usually relatively salient, dominant, and isolated, VOS under complex scenes has rarely been studied. To revisit VOS and make it more applicable in the real world, we collect a new VOS dataset called coMplex video Object SEgmentation (MOSE) to study the tracking and segmenting objects in complex environments. MOSE contains 2,149 video clips and 5,200 objects from 36 categories, with 431,725 high-quality object segmentation masks. The most notable feature of MOSE dataset is complex scenes with crowded and occluded objects. The target objects in the videos are commonly occluded by others and disappear in some frames. To analyze the proposed MOSE dataset, we benchmark 18 existing VOS methods under 4 different settings on the proposed MOSE dataset and conduct comprehensive comparisons. The experiments show that current VOS algorithms cannot well perceive objects in complex scenes. For example, under the semi-supervised VOS setting, the highest J&F by existing state-of-the-art VOS methods is only 59.4% on MOSE, much lower than their ~90% J&F performance on DAVIS. The results reveal that although excellent performance has been achieved on existing benchmarks, there are unresolved challenges under complex scenes and more efforts are desired to explore these challenges in the future. The proposed MOSE dataset has been released at https://henghuiding.github.io/MOSE.

1. Introduction

MOSE revisits video object segmentation in realistic complex scenes, where occlusion, crowding, small objects, and disappearance-reappearance challenge current methods. The dataset and benchmark expose a substantial gap between performance on conventional datasets and complex-scene VOS.

  • Motivation: Existing VOS benchmarks often contain salient, dominant, and isolated targets, leaving realistic complex-scene segmentation relatively underexplored.The paper motivates MOSE by questioning whether high benchmark performance reflects perception in realistic environments.
  • MOSE Dataset: MOSE contains 2,149 videos, 5,200 objects from 36 categories, and 431,725 high-quality segmentation masks.Its scenes include disappearance-reappearance, small or inconspicuous objects, heavy occlusions, and crowded environments.
  • Benchmark: The authors benchmark 18 existing VOS methods under mask-initialization semi-supervised, box-initialization semi-supervised, unsupervised, and interactive settings.The evaluation covers six mask-based, two box-based, three unsupervised, and seven interactive methods.
  • Results: 92.9% J&F for DeAOT on DAVIS 2016 drops to 59.4% on MOSE, while XMem falls from 92.0% to 57.6%.These results show the difficulty of tracking objects that disappear temporarily because of occlusion.
  • Challenges: Heavy occlusions, crowds, and small objects challenge image-level segmentation, while disappearance-reappearance and flickering increase temporal association difficulty.The paper identifies tracking reappearing objects as a particularly unresolved challenge.

2. Related Work

The related work situates MOSE among VOS variants and neighboring video segmentation tasks. It emphasizes that complex-scene and occlusion understanding remains comparatively underexplored in video, motivating a dedicated VOS benchmark.

  • Video Object Segmentation: VOS includes semi-supervised, unsupervised, interactive, and referring settings, distinguished by how the target object is indicated.Semi-supervised VOS uses an initial clue, unsupervised VOS finds objects automatically, interactive VOS uses user input, and referring VOS uses language.
  • Video Object Segmentation: Bounding boxes are used as first-frame clues in some semi-supervised VOS methods because pixel-wise masks are difficult to obtain.Interactive VOS instead uses clicks or scribbles and propagates corrected masks through the video.
  • Related Video Tasks: Video instance segmentation jointly detects, segments, and tracks predefined-category instances, while related work also studies occluded video instance segmentation.MOSE shares an interest in occluded scenes but targets video object segmentation under complex scenarios.
  • Complex Scene Understanding: Complex-scene understanding is established in image understanding, but video-domain occlusion understanding remains underexplored beyond several multi-object tracking works.Prior examples include separating occluder and occludee layers, de-occlusion, and class-agnostic association.
  • MOSE: MOSE provides a complex video object segmentation dataset to support future research on complex-scene understanding in VOS.Its categories are common in real life and overlap with MS-COCO categories, facilitating use of image-pretrained models.

3. MOSE Dataset

MOSE is a large-scale VOS dataset designed for complex scenes with crowds, occlusions, and objects that disappear and reappear. Its collection, annotation, statistics, and evaluation setup emphasize challenges that existing benchmarks and metrics do not fully capture.

  • Video Collection and Annotation: MOSE combines videos inherited from OVIS with newly captured real-world scenes and uses common categories also found in MS-COCO.The categories include vehicles, animals, and people in activities across settings such as campuses, zoos, indoor spaces, and city streets.
  • Video Collection and Annotation: MOSE collection rules require multiple objects, visible occlusions, disappearance-reappearance events, varied object scales, and sufficient motion.Videos without occlusions or with completely still objects are discarded, while crowded objects of similar appearance are encouraged.
  • Video Collection and Annotation: The annotation process provides first-frame masks and requires tracking target objects across subsequent frames, including blank masks during disappearance or full occlusion.Annotations are produced at least every five frames, with some videos annotated every frame for frame-rate robustness testing.
  • Dataset Statistics: BOR can misrepresent occlusion severity, showing high values for lightly occluded samples and low or zero values for heavily occluded objects.The authors therefore supplement BOR with disappearance statistics to characterize complex temporal scenarios.
  • Dataset Statistics: MOSE has the highest disappearance rate at 28.8%, with 1,553 objects disappearing in at least one frame.The authors use disappearance counts and rates alongside mBOR because disappearance and occlusion are frequent and severe in MOSE.
  • Evaluation Metrics: Evaluation reports region similarity from mask IoU, contour accuracy from contour precision and recall, and J&F as their average.Region and contour scores are averaged over objects before forming the overall J&F measure.

4. Experiments

MOSE evaluates video object segmentation under four settings designed to expose performance in complex scenes. Existing methods perform substantially worse on MOSE than on established benchmarks, particularly with occlusion, crowding, and weak initialization.

  • Experimental Settings: Four VOS settings are benchmarked: mask-initialization semi-supervised, box-initialization semi-supervised, unsupervised, and interactive segmentation.The evaluation uses MOSE training, validation, and testing splits, with masks or boxes for semi-supervised settings and scribbles for interactive segmentation.
  • Experimental Settings: MOSE evaluation replaces previous methods’ YouTube-VOS training data with MOSE and reports J_mean, F_mean, and J&F on its validation set.The implementation follows the methods’ original YouTube-VOS training settings and uses no additional video datasets for pretraining.
  • Mask-initialization: 59.4% J&F is the highest reported MOSE validation performance among six mask-initialization methods, compared with usually above 80% J&F or G on DAVIS 2017 and YouTube-VOS.The MOSE results range from 48.8% to 59.4% J&F, revealing unresolved challenges in complex scenes.
  • Box-initialization: 22.0% and 23.8% J&F are achieved by the two box-initialization methods on MOSE, whereas their DAVIS 2017 results are already above 70% J&F.Heavy occlusions, small objects, and crowded scenes can make the target non-salient within its bounding box and fragment it into disconnected pieces.
  • Unsupervised Video Object Segmentation: Multi-object zero-shot methods are benchmarked on MOSE using videos with exhaustive first-frame annotations, with performance drops attributed mainly to temporal challenges.The evaluated methods include STEm-Seg, AGNN, and RVOS, which rely on image-trained instance segmentation methods for static-image object detection and segmentation.
  • Interactive Video Object Segmentation: Interactive VOS methods predict masks from initial scribbles and may refine them using corrective scribbles on the worst frame for up to eight interactions.Table 6 reports J&F@60s, the performance reached within 60 seconds of interactions.

5. Discussion and Future Directions

MOSE exposes several unresolved challenges for VOS in complex scenes, including reappearing, occluded, small, crowded, and long-duration targets. The discussion identifies stronger association, occlusion understanding, small-object attention, crowd tracking, and efficient long-term segmentation as future directions.

  • Stronger Association to Track Reappearing Objects: Stronger association and re-identification are needed to track objects that disappear and reappear, especially when their appearance changes.Some objects reappear with a different view or appearance, making association difficult.
  • Video Object Segmentation of Occluded Objects: Occlusion-aware VOS remains underexplored because frequent occlusions make tracking and segmenting objects difficult, particularly with box initialization.MOSE provides data for studying occlusion video understanding and emphasizes the challenge of segmenting occluded objects from bounding boxes.
  • Attention on Small & Inconspicuous Objects: Small and inconspicuous objects remain insufficiently studied because many existing VOS methods focus mainly on large and salient targets.MOSE contains many small, inconspicuous objects for evaluating this practical setting.
  • Tracking Objects in Crowd: Crowded scenes challenge both object discrimination and temporal association when multiple objects share similar appearance and motion.The challenge applies to distinguishing one target from similar-looking crowd members across frames.
  • Long-Term Video Segmentation: Long-term VOS requires high-quality segmentation over longer videos while reducing the computation and GPU-memory costs of storing previous-frame features.MOSE’s longer average video length creates additional challenges and opportunities for efficient long-video processing.

6. Conclusion

MOSE is a large-scale dataset and benchmark for evaluating VOS in complex scenes. Its results indicate that tracking and segmenting objects amid crowds, disappearance, occlusion, and inconspicuous appearances remains at an early stage.

  • Dataset and Benchmark: MOSE contains 2,149 high-resolution videos, 431,725 masks, 5,200 objects, and 36 categories for VOS under complex scenes.The dataset includes sufficiently long videos with diverse occlusion, motion, and disappearance-reappearance scenarios.
  • Dataset and Benchmark: The authors benchmark existing VOS methods across semi-supervised, unsupervised, and interactive settings using mask or bounding-box references.The benchmark evaluates six mask-initialized methods, two box-initialized methods, three unsupervised methods, and seven interactive methods.
  • Conclusion: VOS in complex scenes remains at a nascent stage when crowds, disappearance, occlusion, and inconspicuous or small objects occur frequently.This conclusion follows comprehensive evaluation and analysis of existing methods on MOSE.
Loading 2302.01872v1…