Source-linked AI summary
Learning Video Object Segmentation from Unlabeled Videos
Xiankai Lu, Wenguan Wang, Jianbing Shen, Yu-Wing Tai, David Crandall, Steven C. H. Hoi
TL;DR
VOS commonly depends on extensive pixel-wise annotations, motivating learning from unlabeled videos. MuG provides a unified unsupervised/weakly supervised framework that models video object patterns at multiple granularities, achieves strong results across zero-shot and one-shot settings, and improves with more unlabeled data.
Problem
VOS methods commonly require extensive pixel-wise annotations, which are labor-intensive and expensive to obtain.
Method
MuG is a unified, fully differentiable unsupervised/weakly supervised framework that learns video object patterns from unlabeled videos using multi-granularity cues.
Results
MuG outperforms other unsupervised and weakly supervised methods by a large margin and continuously improves with more unlabeled data.
Takeaways & Limitations
A single MuG model can be applied to object-level zero-shot, instance-level zero-shot, and one-shot VOS settings.
Takeaways & Limitations
The weakly supervised setting assumes annotations are available for a related output domain K that is easier to annotate than the VOS output domain Y.
Abstract
from arXiv · showhide
We propose a new method for video object segmentation (VOS) that addresses object pattern learning from unlabeled videos, unlike most existing methods which rely heavily on extensive annotated data. We introduce a unified unsupervised/weakly supervised learning framework, called MuG, that comprehensively captures intrinsic properties of VOS at multiple granularities. Our approach can help advance understanding of visual patterns in VOS and significantly reduce annotation burden. With a carefully-designed architecture and strong representation learning ability, our learned model can be applied to diverse VOS settings, including object-level zero-shot VOS, instance-level zero-shot VOS, and one-shot VOS. Experiments demonstrate promising performance in these settings, as well as the potential of MuG in leveraging unlabeled data to further improve the segmentation accuracy.
1. Introduction
MuG addresses the annotation burden in VOS by learning video object patterns from unlabeled videos through a unified unsupervised/weakly supervised framework. The learned model supports object-level and instance-level zero-shot VOS as well as one-shot VOS.
- 1. Introduction: Current leading VOS methods rely on extensive pixel-wise annotations, which are labor-intensive and expensive to obtain.This applies to both zero-shot and one-shot VOS settings.
- 1. Introduction: In the unsupervised setting, MuG uses only input videos, while weak supervision additionally provides annotations in a related domain K that is easier to annotate than VOS outputs.Examples include saliency outputs and class-activation maps.
- 1. Introduction: MuG learns video object patterns from unlabeled videos in a unified, fully differentiable unsupervised/weakly supervised framework.The framework uses constraints and supervision signals at multiple granularities.
- 1. Introduction: MuG can be applied across object-level zero-shot VOS, instance-level zero-shot VOS, and one-shot VOS after training.The paper frames this as learning a common video object representation across diverse VOS settings.
- 1. Introduction: MuG outperforms other unsupervised and weakly supervised methods by a large margin and improves continuously as more unlabeled data are added.This is the paper’s reported overall result across its evaluated settings.
2. Related Work
Prior work on VOS with unlabeled videos includes heuristic, weakly supervised, semantic-label, and self-supervised approaches, but the paper positions MuG as a unified multi-granularity alternative. MuG is presented as learning representations applicable to diverse VOS settings after one training process.
- 2. Related Work: VOS models are typically trained with large amounts of elaborately annotated data, whereas the proposed method trains from unlabeled videos.The paper connects this setting to reducing annotation requirements and studying visual recognition in VOS.
- 2. Related Work: Existing unlabeled-video approaches use proxy masks, heuristic masks, weakly supervised location maps, semantic video labels, or self-supervised temporal constraints.These approaches include cross-frame color consistency and temporal cycle-correspondence.
- 2. Related Work: MuG unifies intrinsic video properties and class-agnostic foreground-background knowledge in a multi-granularity framework.The stated goal is a more comprehensive understanding of visual patterns in VOS.
- 2. Related Work: After being trained once, MuG is applied to diverse VOS settings and is presented as connecting the field’s two most influential settings.The paper describes this as evidence of strong video object representation learning ability.
3. Proposed Algorithm
MuG learns VOS representations from unlabeled videos by combining frame, short-term, long-term, and whole-video cues in one differentiable framework. After training, the learned representation supports object-level and instance-level Z-VOS as well as O-VOS with only slight adaptation.
- Multi-Granularity VOS Network: MuG explores frame, short-term, long-term, and whole-video granularities to learn video object patterns.These cues guide the learning of the fully convolutional representation ϕ.
- Frame Granularity Analysis: Frame-level learning uses binarized saliency or CAM masks as initial fore-background supervision.The masks are denoted Qt and guide representation learning through the frame-granularity loss.
- Short-Term Granularity Analysis: Short-term learning tracks patches forward and backward across successive frames to enforce local spatiotemporal correspondence.Training samples short clips and computes a local consistency loss from repeated forward-backward tracking.
- Long-Term Granularity Analysis: Long-term learning matches disordered frames using normalized correlations and learned geometric transformations, then applies a semantic matching loss.The transformation estimator predicts six degrees of freedom, while geometric consistency determines valid correspondences.
- Video Granularity Analysis: Whole-video learning reduces temporal redundancy by sampling one frame per segment, aggregates correlation-based global information, and discriminates surrogate video classes.The resulting representation is trained so same-video instances are closer than instances from other videos.
- Inference Modes: The unified network can be trained once and applied to object-level Z-VOS, instance-level Z-VOS, and O-VOS with slight modifications.Object-level Z-VOS reads segmentation from globally augmented features, while instance-level Z-VOS additionally uses category-agnostic object proposals.
4. Experiment
MuG is evaluated across object-level and instance-level zero-shot VOS and one-shot VOS, with experiments covering benchmark comparisons, ablations, additional training data, runtime, and qualitative results. It performs competitively across these settings and benefits from more unlabeled training videos.
- Diagnostic Experiments: Ablations show that MuG substantially improves low-quality saliency and CAM initialization, with gains of 20.8% and 15.9%.Removing short- or long-term granularity analysis also reduces performance.
- More Training Data: Adding 1,400 LaSOT videos improves performance in both unsupervised and weakly supervised settings.This experiment probes MuG’s potential to leverage additional unlabeled video data.
- Performance for Object-Level Z-VOS: MuG outperforms COSEG and previous heuristic methods on Youtube-Objects object-level Z-VOS, while running at about 0.6s per frame.The runtime is faster than MotionMask at 1.1s and TSN at 0.9s per frame.
- Performance for O-VOS: 54.3 vs 50.3 mean J &F shows MuG outperforming CorrFlow on DAVIS17 O-VOS with the same training data.On DAVIS16, weak supervision with learned CAM further reaches mean J of 65.7.
- Qualitative Results: Qualitative examples show accurate segmentation under view changes, background clutter, fast motion, scale variation, and multiple VOS settings.The examples cover object-level Z-VOS, instance-level Z-VOS, and O-VOS.
5. Conclusion
MuG is an end-to-end unsupervised/weakly supervised approach that learns video object patterns from unlabeled videos across diverse VOS settings.
- MuG models video object patterns from unlabeled videos using supervision signals at different granularities across object-level Z-VOS, instance-level Z-VOS, and O-VOS.The approach is end-to-end trainable and is designed to unify these VOS settings through video object pattern understanding.