Source-linked AI summary
YouTube-VOS: Sequence-to-Sequence Video Object Segmentation
Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, Thomas Huang
TL;DR
Video object segmentation lacked both long-term temporal modeling and datasets large enough for end-to-end learning. The paper introduces YouTube-VOS and a ConvLSTM-based sequence-to-sequence model, achieving better or comparable results to state-of-the-art methods on YouTube-VOS and DAVIS 2016.
Problem
Existing methods mainly rely on static image segmentation or optical-flow models, while available datasets are too small for end-to-end learning of long-term spatial-temporal features.
Method
The paper builds YouTube-VOS and uses an end-to-end sequence-to-sequence network with ConvLSTM to model long-term spatial-temporal information for segmentation.
Results
The method achieves better or comparable results than current state-of-the-art approaches on YouTube-VOS and DAVIS 2016.
Takeaways & Limitations
The dataset scale supports explicit end-to-end modeling of long-term spatial-temporal dependency for video segmentation.
Abstract
from arXiv · showhide
Learning long-term spatial-temporal features are critical for many video analysis tasks. However, existing video segmentation methods predominantly rely on static image segmentation techniques, and methods capturing temporal dependency for segmentation have to depend on pretrained optical flow models, leading to suboptimal solutions for the problem. End-to-end sequential learning to explore spatial-temporal features for video segmentation is largely limited by the scale of available video segmentation datasets, i.e., even the largest video segmentation dataset only contains 90 short video clips. To solve this problem, we build a new large-scale video object segmentation dataset called YouTube Video Object Segmentation dataset (YouTube-VOS). Our dataset contains 3,252 YouTube video clips and 78 categories including common objects and human activities. This is by far the largest video object segmentation dataset to our knowledge and we have released it at https://youtube-vos.org. Based on this dataset, we propose a novel sequence-to-sequence network to fully exploit long-term spatial-temporal information in videos for segmentation. We demonstrate that our method is able to achieve the best results on our YouTube-VOS test set and comparable results on DAVIS 2016 compared to the current state-of-the-art methods. Experiments show that the large scale dataset is indeed a key factor to the success of our model.
1 Introduction
Video object segmentation requires temporal reasoning, but existing datasets and methods limit end-to-end learning of long-term spatial-temporal features. YouTube-VOS addresses this gap with a large-scale dataset and a sequence-to-sequence model.
- Video object segmentation supports object tracking, editing, and augmented reality, but leading approaches primarily rely on single-image segmentation frameworks.
- DAVIS contains only 90 short video clips, limiting end-to-end learning of long-term spatial-temporal features from scratch.
- YouTube-VOS contains 3,252 clips across 78 categories, with 3–6-second videos and manually segmented objects.
- The proposed sequence-to-sequence model uses ConvLSTM to learn long-term spatial-temporal information end-to-end without relying on optical-flow or motion-segmentation models.
- The method achieves better or comparable results than current state-of-the-art methods on YouTube-VOS and DAVIS 2016.
2 Related work
Earlier video object segmentation methods used hand-crafted spatial-temporal structures, while later deep-learning methods largely adapted image segmentation networks with limited temporal modeling.
- Existing datasets generally contained only dozens of videos with relatively simple content, while DAVIS introduced benchmark datasets with 50 or 90 videos.
- Early methods formulated video object segmentation with spatial-temporal graphs and hand-crafted terms based on appearance, boundaries, motion, and optical flow.
- Most deep-learning approaches built on image segmentation networks without sequential modeling, often using online learning to improve performance.
- Some methods propagated masks using spatial-temporal consistency or used previous-frame masks as temporal guidance.
3 YouTube-VOS
YouTube-VOS was constructed from diverse YouTube videos and carefully selected clips and object annotations to provide substantially greater scale for video object segmentation.
- The dataset covers animals, vehicles, accessories, common objects, and human activities, with different activities treated as separate categories.
- Candidate videos were retrieved from YouTube-8M using category annotations, providing diverse object appearances and motions.
- Long videos were partitioned into clips, after which clips were filtered for suitable length, category correctness, and usable visual quality.
- Figure 1 shows sample ground-truth annotations with different objects highlighted in different colors.
- 3,252 video clips and 133,886 object annotations make YouTube-VOS the largest video object segmentation dataset described here.
4 Sequence-to-Sequence Video Object Segmentation
The proposed sequence-to-sequence system uses ConvLSTM to accumulate spatial-temporal information across frames and decode segmentation masks from updated hidden states. It is trained end-to-end and can operate offline, with online learning providing further improvement.
- 4.1 Problem formulation: The model predicts later object masks from the initial frame and mask while retaining long-term history to handle appearance variation and rapid motion.
- 4.2 Our Algorithm: An Initializer encodes the initial frame and mask, while an Encoder processes each new frame before ConvLSTM updates its internal states.
- 4.2 Our Algorithm: A Decoder transforms the ConvLSTM output into full-resolution segmentation results, and binary cross-entropy trains the complete model end-to-end.
- 4.2 Our Algorithm: The architecture uses VGG-16-based Initializer and Encoder networks, a convolutional LSTM, and a multi-layer upsampling Decoder.
- 4.3 Implementation Details: Training samples random objects and 5–11 frames, resizing inputs to 256×448 for memory and speed considerations.
- 4.3 Implementation Details: The offline-trained model can segment unseen categories without per-video fine-tuning, achieving comparable or better results while enabling faster inference.
5 Experiments
Experiments evaluate the method on YouTube-VOS and DAVIS 2016, compare online and offline learning, and test dataset scale and model variants. Results show strong segmentation performance, generalization to unseen categories, faster inference without online learning, and a substantial dependence on large-scale video data.
- Benchmark evaluation: The experiments compare the proposed method with prior approaches on YouTube-VOS and DAVIS 2016, including settings with and without online learning.The YouTube-VOS evaluation also separates seen and unseen categories, while the DAVIS evaluation compares methods using different components.
- YouTube-VOS results: Around 8% absolute improvement over OSVOS on J mean is achieved on YouTube-VOS with online learning, alongside larger margins in contour accuracy and decay rate.The model also performs better than previous methods without online learning, although it initially trails online-learning methods during the first few frames.
- YouTube-VOS results: Most methods differ by usually within 2% absolute on seen versus unseen categories, while the proposed offline model avoids the stronger unseen-category degradation observed for OSMN.The authors associate this robustness with spatial-temporal information and attribute other methods’ behavior to online adaptation, prior-mask guidance, or slow-motion examples.
- DAVIS 2016 results: The model without online learning achieves comparable DAVIS 2016 results to online-learning methods without post-processing and runs 60 times faster.Online learning further improves object-boundary accuracy, while OnAVOS inference takes approximately 13 seconds per frame.
- Dataset-scale ablation: 51.3% and 51.9% mean IoU are obtained when training from scratch on 30 and 192 videos, whereas pretrained static-image features produce 45.6%.These settings indicate that existing datasets are insufficient and that spatial-temporal features cannot be trivially transferred from static-image representations.
- Dataset-scale ablation: Using only 25% of YouTube-VOS training videos, approximately 700 videos, drops performance almost 30% from the original model, while performance improves rapidly with more data.The 25% setting also performs much worse on unseen than seen categories, and the 100% data trend has not reached a plateau.
6 Conclusion
The paper introduces YouTube-VOS as the largest video object segmentation dataset to date and uses its scale to support end-to-end modeling of long-term spatial-temporal dependencies. The resulting algorithm achieves better or comparable performance than existing state-of-the-art approaches.
- YouTube-VOS is introduced as the largest video object segmentation dataset to date.
- Its larger number of videos and annotations enables explicit end-to-end modeling of long-term spatial-temporal dependencies for segmentation.
- The proposed algorithm achieves better or comparable results than existing state-of-the-art approaches.