Source-linked AI summary
PReMVOS: Proposal-generation, Refinement and Merging for Video Object Segmentation
Jonathon Luiten, Paul Voigtlaender, Bastian Leibe
TL;DR
Semi-supervised video object segmentation requires accurate, consistent object masks from first-frame annotations, especially across multiple objects. PReMVOS generates and refines mask proposals, then merges them into temporally consistent tracks; it reports state-of-the-art performance and challenge wins.
Problem
Semi-supervised video object segmentation must generate accurate, consistent pixel masks and object identities throughout a video from first-frame ground truth.
Method
PReMVOS detects coarse object proposals, refines them into masks, and merges selected proposals over time using objectness, flow, Re-ID, and spatial constraints.
Results
PReMVOS surpasses current state-of-the-art results across DAVIS benchmarks and achieves first place in the 2018 DAVIS and YouTube-VOS challenges.
Takeaways & Limitations
The proposal-based approach is particularly suited to the challenging multi-object video object segmentation task.
Takeaways & Limitations
The Objectness sub-score cannot distinguish different objects of interest, while MaskProp cannot handle heavy occlusion or objects that disappear and later return.
Abstract
from arXiv · showhide
We address semi-supervised video object segmentation, the task of automatically generating accurate and consistent pixel masks for objects in a video sequence, given the first-frame ground truth annotations. Towards this goal, we present the PReMVOS algorithm (Proposal-generation, Refinement and Merging for Video Object Segmentation). Our method separates this problem into two steps, first generating a set of accurate object segmentation mask proposals for each video frame and then selecting and merging these proposals into accurate and temporally consistent pixel-wise object tracks over a video sequence in a way which is designed to specifically tackle the difficult challenges involved with segmenting multiple objects across a video sequence. Our approach surpasses all previous state-of-the-art results on the DAVIS 2017 video object segmentation benchmark with a J & F mean score of 71.6 on the test-dev dataset, and achieves first place in both the DAVIS 2018 Video Object Segmentation Challenge and the YouTube-VOS 1st Large-scale Video Object Segmentation Challenge.
1 Introduction
PReMVOS frames semi-supervised video object segmentation as accurate mask generation followed by temporally consistent object tracking. It addresses this decomposition with a proposal-generation, refinement, and merging algorithm.
- Problem: Video object segmentation estimates pixel masks and assigns consistent object IDs throughout a video sequence.Semi-supervised VOS provides the first-frame ground-truth mask for the target objects.
- Approach: PReMVOS separates VOS into generating mask proposals and selecting and merging them into pixel-wise object tracks.The two-stage design connects instance segmentation with multi-object tracking.
- Approach: The method refines object-detector proposals into accurate masks, then links them over time using objectness, optical flow, Re-ID features, and spatial constraints.Networks are adapted to each target video through first-frame-based augmented images and fine-tuning.
- Results: PReMVOS reports the best results in the 2018 DAVIS and first YouTube-VOS large-scale video object segmentation challenges.The method is designed for accurate and temporally consistent segmentation, including multi-object videos.
2 Related Work
Prior VOS methods are organized around objectness estimation with first-frame adaptation or propagation from previous-frame masks. The paper situates PReMVOS against these established paradigms.
- Existing paradigms: Current state-of-the-art VOS methods fall into objectness estimation with first-frame fine-tuning or previous-frame mask propagation.The first paradigm estimates pixel objectness, while the second propagates masks temporally.
- Objectness estimation: First-frame fine-tuning is used in the objectness-estimation paradigm to estimate objectness for each pixel.This paradigm was extended with semantic segmentation guidance and iterative fine-tuning.
- PReMVOS context: The related-work discussion includes a diagram showing PReMVOS components and their relationships.
3 Approach
PReMVOS generates object proposals independently for each frame, refines them into accurate masks, and greedily links proposals into temporally consistent tracks using multiple cues.
- Approach: PReMVOS detects coarse object proposals, refines their bounding boxes into accurate masks, and merges proposals into object tracks over time.The merging algorithm uses objectness, optical-flow warping, Re-ID embeddings, and spatial constraints.
- Approach: 2500 augmented images generated from each video's first-frame ground truth adapt the networks to the target video domain through fine-tuning.The augmentation removes objects, fills backgrounds, transforms objects and backgrounds, and recombines them into scenes.
- Proposal Generation: A category-agnostic Mask R-CNN with a ResNet101 backbone generates coarse proposals before refinement.The detector maps all object categories to one generic foreground class and is fine-tuned separately for each video.
- Proposal Refinement: The Proposal-Refinement Network crops and resizes each proposal's bounding box, adds a bounding-box mask channel, and predicts an accurate pixel mask.It uses a 385×385 input patch and discards the original coarse mask during refinement.
- Proposal Merging: Greedy merging starts from first-frame ground-truth masks, selects the highest-scoring proposal for each track at each timestep, and resolves overlaps by combined track score.Optical flow is used to warp masks between successive frames for temporal-consistency scoring.
- Proposal Merging: Track scoring combines objectness, Re-ID similarity, mask propagation, inverse Re-ID, and inverse mask propagation scores.Objectness comes from the proposal-generation network; Re-ID compares proposal and first-frame embeddings, while mask propagation compares proposals with optical-flow-warped previous masks.
4 Experiments
PReMVOS is evaluated across DAVIS benchmarks, challenge datasets, proposal refinement, merging components, and runtime. Results show strong mask accuracy and temporal consistency, while ablations identify the contributions and boundaries of refinement and merging.
- Benchmark results: On all DAVIS datasets, PReMVOS outperforms other state-of-the-art methods on F and J &F mean, while achieving best or comparable J scores.The evaluation covers DAVIS 2017 test-dev, DAVIS 2017 val, and DAVIS 2016 val datasets.
- Benchmark results: PReMVOS achieves the best results and first place in the 2018 DAVIS Challenge on the DAVIS test-challenge dataset.Results are reported with and without ensembling against competing methods.
- Proposal refinement: 5.9% IoU and 7.9% boundary-measure improvements result from refinement over Mask R-CNN proposals.The comparison uses oracle merging on the DAVIS 2017 val dataset.
- Proposal refinement: Refined proposals more closely match ground truth, recover lost object regions, and capture boundary contours at higher fidelity than coarse proposals.The refinement network extracts deep features over each object’s area of interest rather than the whole image.
- Proposal merging: The merging algorithm reaches 77.8 J &F mean with equal weights, only 0.4 below optimized weights, indicating robustness to exact weighting.The optimized-weight result is 78.2 J &F mean, while oracle merging provides an 81.2 J &F mean upper bound.
- Proposal merging: ReID is weaker for visually similar objects, MaskProp fails under heavy occlusion or disappearance, and Objectness cannot distinguish multiple target objects.These limitations affect complementary components of the proposal-merging procedure.
- Proposal merging: Removing one merging component lowers accuracy by 0.9 to 2.3 points, while removing both InvReID and InvMaskProp lowers J &F mean by 1.9 points.The results support the importance of all five sub-scores, including spatial-separation components.
- Runtime and large-scale evaluation: Across reported runtime scales, PReMVOS compares to or exceeds other state-of-the-art methods and places first in the YouTube-VOS challenge using its Fast-finetuned version.The large-scale result uses Fast-finetuned PReMVOS rather than the original method.
5 Conclusion
PReMVOS divides video object segmentation into proposal generation and proposal selection and merging, producing accurate, temporally consistent object tracks. The method is reported to outperform prior state-of-the-art semi-supervised VOS methods on DAVIS benchmarks and win the 2018 DAVIS Challenge.
- PReMVOS first generates accurate object segmentation mask proposals for each video frame, then selects and merges them into accurate, temporally consistent pixel-wise object tracks.
- The approach is designed for the difficult multi-object video object segmentation task.
- PReMVOS produces results better than current state-of-the-art semi-supervised video object segmentation methods on DAVIS benchmarks.
- PReMVOS achieved the best score in the 2018 DAVIS Video Object Segmentation Challenge.