Source-linked AI summary
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, Thomas Huang
TL;DR
Video object segmentation is constrained by methods built mainly on static image segmentation and datasets too small for learning long-term spatiotemporal features. This paper introduces YouTube-VOS, a large-scale dataset and benchmark, where S2S with online learning improves overall accuracy by around 6% over OSVOS.
Problem
Existing video object segmentation methods primarily rely on static image segmentation, while available datasets are too small to support learning long-term spatiotemporal features.
Method
The paper constructs YouTube-VOS with 4,453 video clips, 94 object categories, and benchmarks existing video object segmentation algorithms, including unseen-category evaluation.
Results
S2S with online learning achieves around 6% absolute improvement in overall accuracy over the best online-learning method, OSVOS.
Takeaways & Limitations
YouTube-VOS provides a substantially larger benchmark for more comprehensive evaluation of video object segmentation methods.
Abstract
from arXiv · showhide
Learning long-term spatial-temporal features are critical for many video analysis tasks. However, existing video segmentation methods predominantly rely on static image segmentation techniques, and methods capturing temporal dependency for segmentation have to depend on pretrained optical flow models, leading to suboptimal solutions for the problem. End-to-end sequential learning to explore spatialtemporal features for video segmentation is largely limited by the scale of available video segmentation datasets, i.e., even the largest video segmentation dataset only contains 90 short video clips. To solve this problem, we build a new large-scale video object segmentation dataset called YouTube Video Object Segmentation dataset (YouTube-VOS). Our dataset contains 4,453 YouTube video clips and 94 object categories. This is by far the largest video object segmentation dataset to our knowledge and has been released at http://youtube-vos.org. We further evaluate several existing state-of-the-art video object segmentation algorithms on this dataset which aims to establish baselines for the development of new algorithms in the future.
1 Introduction
Effective spatial-temporal learning is important for video analysis, but video object segmentation remains constrained by static-image-based methods and datasets too small for end-to-end sequential learning. YouTube-VOS addresses this limitation with a large-scale benchmark and validation categories for testing generalization to unseen objects.
- Video object segmentation segments a specified object instance throughout a video from its mask in the first frame, supporting tracking, editing, and augmented reality.
- Existing state-of-the-art approaches primarily rely on single-image segmentation frameworks rather than learning long-term video spatial-temporal features directly.
- DAVIS contains only 90 short video clips, making it barely sufficient for training a sequence-to-sequence network from scratch.
- YouTube-VOS is introduced as a large-scale video object segmentation dataset containing 4,453 YouTube clips and 94 object categories.
- The validation set contains 474 videos and 26 categories absent from training, enabling evaluation of existing methods' generalization to unseen categories.
2 Related work
Prior video object segmentation datasets were small and relatively simple, while DAVIS [20] improved video and annotation quality but remained limited in scale. Methodologically, the field progressed from hand-crafted spatial-temporal graph models to deep models largely built on image segmentation networks without sequential modeling.
- Datasets: Existing video object segmentation datasets generally contain only dozens of videos with relatively simple content, including limited occlusion, camera motion, and illumination variation.Some earlier datasets also used low-resolution videos.
- Datasets: DAVIS [20] became a benchmark with 50 single-object videos in 2016 and 90 multiple-object videos in 2017, offering higher-quality video resolutions and annotations than earlier datasets.The 2017 version contains multiple objects per video, whereas the 2016 version contains one foreground object per video.
- Methods: Early methods modeled spatial-temporal graph structures using hand-crafted energy terms based on appearance, boundaries, motion, and optical flow.These methods explicitly incorporated several visual and motion-related feature types.
3 YouTube-VOS
YouTube-VOS is constructed from diverse, activity-rich YouTube videos through category-based retrieval, shot-based clip sampling, and manual verification. The resulting benchmark contains 4,453 clips, 197,272 object annotations, and 94 categories, making it the largest and most comprehensive video object segmentation dataset at the time.
- Category selection: The dataset covers animals, vehicles, accessories, common objects, and humans performing diverse activities and interacting with corresponding objects.Human-related videos are collected using activity tags to increase diversity in motion and behavior.
- Video collection: Candidate videos are retrieved from high-resolution YouTube-8M videos using category annotations, with up to 100 videos collected per segmentation category.YouTube-8M contains millions of videos associated with more than 4,700 visual entities.
- Clip processing: Long retrieved videos are partitioned with shot detection, clips from the first and last 10% are removed, and up to five 3–6-second clips per video are manually verified.The filtering reduces introductory subtitles and credits, while verification ensures sampled clips contain the target content.
- Dataset scale and scope: YouTube-VOS contains 4,453 video clips, 197,272 object annotations, and 94 object categories, making it the largest and most comprehensive video object segmentation dataset to date.It is about 50 times larger than YouTubeObjects and has 15 times more annotations than DAVIS 2017.
4 Experiments
Experiments retrain state-of-the-art video object segmentation methods under a common setting and evaluate them on the publicly available YouTube-VOS validation set. Results compare temporal modeling, generalization to unseen categories, and inference speed.
- Experimental setup: The study retrains existing methods on the YouTube-VOS training set and evaluates them on the validation set under the same training and testing setting to establish baselines.The dataset contains 4,453 videos split into 3,471 training, 474 validation, and 508 test videos; only validation is used because the test set is reserved for the competition period.
- Experimental setup: The comparison includes OSVOS, MaskTrack, OSMN, OnAVOS [35], and S2S, selected as recently proposed methods with state-of-the-art results on earlier small-scale benchmarks.All compared algorithms are retrained and tested under the same setting.
- Overall comparison: S2S without online learning achieves performance comparable to the best online-learning methods, demonstrating the importance of long-term spatial-temporal coherence.S2S uses recurrent neural networks to model long-term spatial-temporal coherence, whereas the first four methods use static image segmentation models; OSVOS, MaskTrack, and OnAVOS require online learning.
- Generalization: All methods perform better on seen than unseen categories, while OSVOS has the smallest discrepancy, possibly because it is pretrained on a large-scale image segmentation dataset.Online learning improves accuracy on unseen categories but does not eliminate the generalization gap.
- Inference speed: OSMN and S2S without online learning have very fast per-frame inference and can operate in real time, offering an advantage for mobile applications despite lower performance than online-learning methods.The speed advantage comes from avoiding online learning.
5 Conclusion
The report introduces YouTube-VOS as the largest video object segmentation dataset to date, exceeding existing datasets in videos and annotations. It enables more comprehensive evaluation of state-of-the-art methods and is intended to foster broader video-based computer vision research.
- YouTube-VOS is presented as the largest video object segmentation dataset to date, with more videos and annotations than existing datasets.
- The dataset enables more comprehensive evaluation of existing state-of-the-art video object segmentation methods.The reported comparisons use the YouTube-VOS validation set and average results across region similarity, contour accuracy, and seen, unseen, and overall category groupings.
- The authors expect YouTube-VOS to foster research in video-based computer vision more broadly.