Source-linked AI summary
MaskRNN: Instance Level Video Object Segmentation
Yuan-Ting Hu, Jia-Bin Huang, Alexander G. Schwing
TL;DR
Instance-level video object segmentation is challenging in complex scenes and important for applications including video editing and compression. MaskRNN combines recurrent temporal modeling with binary segmentation and localization networks, and achieves state-of-the-art performance on DAVIS-2016, DAVIS-2017, and Segtrack v2.
Problem
Instance-level video object segmentation must handle deforming shapes, fast movements, and occlusions in complex scenes, with applications including video editing and compression.
Method
MaskRNN uses a recurrent neural network and combines binary segmentation masks with predicted bounding boxes for individual object instances.
Results
MaskRNN achieves state-of-the-art performance on DAVIS-2016, DAVIS-2017, and Segtrack v2.
Takeaways & Limitations
The recurrent and localization components let MaskRNN use long-term temporal information and location priors to improve results.
Takeaways & Limitations
The method assumes ground-truth segmentation masks for the object instances are available in the first video frame.
Abstract
from arXiv · showhide
Instance level video object segmentation is an important technique for video editing and compression. To capture the temporal coherence, in this paper, we develop MaskRNN, a recurrent neural net approach which fuses in each frame the output of two deep nets for each object instance -- a binary segmentation net providing a mask and a localization net providing a bounding box. Due to the recurrent component and the localization component, our method is able to take advantage of long-term temporal structures of the video data as well as rejecting outliers. We validate the proposed algorithm on three challenging benchmark datasets, the DAVIS-2016 dataset, the DAVIS-2017 dataset, and the Segtrack v2 dataset, achieving state-of-the-art performance on all of them.
1 Introduction
Instance-level video object segmentation remains challenging because complex scenes involve deforming shapes, fast movements, and occlusions, while existing deep-learning methods primarily address single-object foreground-background segmentation. MaskRNN addresses multiple objects by combining recurrent temporal modeling with per-instance segmentation and localization, achieving state-of-the-art performance on three datasets.
- Complex scenes make instance-level video object segmentation difficult because objects deform, move quickly, and occlude one another.
- Classical methods often assume rigid geometry, while graphical models can propagate labels across videos but are sensitive to parameters.
- Earlier deep-learning methods add temporal cues such as optical flow or mask propagation, but address only single-object foreground-background segmentation.
- MaskRNN uses a bottom-up framework that tracks and segments individual objects before merging them for multi-object instance segmentation.
- MaskRNN captures temporal structure with a recurrent neural network and predicts binary masks within predicted bounding boxes.
- MaskRNN achieves state-of-the-art performance on DAVIS-2016, DAVIS-2017, and Segtrack v2.
2 Related Work
Prior video object segmentation research includes graph-based methods that propagate labels through spatio-temporal structures and deep-learning methods that often achieve state-of-the-art performance. MaskRNN is distinguished by modeling long-term temporal information recurrently and estimating bounding boxes alongside segmentation masks.
- Graph-based methods construct three-dimensional spatio-temporal graphs to propagate foreground-background assignments across video pixels or superpixels.
- Deep-learning video segmentation methods often achieve state-of-the-art performance and commonly fine-tune pretrained networks using the first frame’s ground-truth mask.
- MaskRNN is the only compared method described as using back-propagation through time in a recurrent neural network for long-term temporal information.
- MaskRNN also estimates bounding boxes in addition to segmentation masks, enabling incorporation of a tracked object’s location prior.
3 Instance Level Video Object Segmentation
MaskRNN performs multi-object video segmentation by combining recurrent temporal modeling, per-instance binary masks, and bounding-box localization. It merges the resulting object-specific predictions into a single instance-level segmentation.
- Overview: MaskRNN uses one deep-net pair per object instance: a binary segmentation network predicts a mask, while localization performs bounding-box regression.The approach is organized as a bottom-up pipeline that tracks and segments individual objects before merging their results.
- Temporal modeling: The recurrent formulation lets the previous prediction and current and preceding frames influence the current segmentation through optical flow and mask warping.Optical flow is used both as an input feature and to warp the previous prediction.
- Fusion: The N binary probability maps are fused by assigning each pixel to the object with the largest probability, followed by thresholding the current-frame prediction.The fusion operation produces a single multi-instance mask while preserving separate object-specific predictions until test time.
- Binary segmentation: The binary segmentation network has appearance and flow streams whose outputs are linearly combined to produce each object’s foreground probability map.The appearance stream uses the current frame and warped prior mask; the flow stream uses flow magnitude and the warped prior mask.
- Training: All network operations are differentiable, enabling end-to-end training through back-propagation through time.This training strategy follows from the recurrence connecting predictions across video frames.
- Object localization: Each object’s binary segmentation is restricted by a refined bounding box, which helps avoid outlier predictions far from the object.Pixels outside a bounding box enlarged by a factor of 1.25 are discarded after localization.
4 Implementation Details
The implementation combines dataset-specific training, perturbed-mask offline training, recurrent optimization, and first-frame online fine-tuning. Evaluation uses DAVIS and reports multiple video segmentation metrics.
- Training data: The networks are trained separately on the DAVIS-2016 and DAVIS-2017 training sets, with DAVIS-2016 containing one object per video.The DAVIS datasets provide 30/20 and 60/30 training/testing videos for DAVIS-2016 and DAVIS-2017, respectively.
- Offline training: Offline training first uses static images with randomly perturbed ground-truth masks, then optimizes the segmentation network using recurrence and long-term information.The recurrent stage uses seven frames at a time because of GPU memory limitations.
- Evaluation: DAVIS evaluation reports IoU J, boundary precision F, time stability T, recall, and performance decay over time for J and F.The evaluation is performed on the validation set of the DAVIS dataset.
- Offline training: Optical flow is computed with pre-trained FlowNet2.0, while training uses Adam, a starting learning rate of 10^-5, and standard geometric data augmentation.The augmentation includes resizing, rotation, cropping, and left-right flipping.
- Online fine-tuning: In the semi-supervised setting, both segmentation and localization networks are fine-tuned on the first frame of each test video for 200 iterations at a learning rate of 10^-5.The first-frame ground-truth segmentation mask supplies the supervision for online fine-tuning.
5 Experimental Results
The experiments evaluate MaskRNN with standard segmentation metrics, component ablations, benchmark comparisons, and qualitative examples across three datasets. Results support the value of online finetuning, motion-aware segmentation, localization, and recurrence, while also showing failure cases.
- Ablation study: Online finetuning is important because the network is adjusted to the specific object appearance in the current video.
- Ablation study: The appearance and flow streams outperform the appearance stream alone, while warping the input mask further improves performance by incorporating motion.Optical flow encodes motion boundaries and helps discover moving objects in cluttered backgrounds.
- Ablation study: Adding bounding box regression loss decreases segmentation performance, but applying the bounding box to restrict the mask improves results beyond segmentation alone.
- Ablation study: Recurrence further improves results by producing more consistent segmentation masks over time.
- Quantitative evaluation: 0.6% improvement over other state-of-the-art semi-supervised methods is reported on DAVIS-2016 foreground-background video object segmentation.DAVIS-2016 evaluates foreground-background segmentation using 30 training videos and 20 validation videos, with first-frame finetuning during testing.
- Quantitative evaluation: 5.6% and 4.6% improvements over other state-of-the-art semi-supervised methods are reported on DAVIS-2017 and Segtrack v2, respectively.These datasets evaluate multiple instance-level video object segmentation; Segtrack v2 uses DAVIS-2017 training data because it has no training set.
- Qualitative evaluation: Qualitative results show successful tracking under complex motion, clutter, and occlusion, alongside failures involving similar instance appearances and other conditions.Figure 3 presents results from DAVIS-2016, DAVIS-2017, and Segtrack v2; Figure 4 presents failure cases.
6 Conclusion
The paper proposes MaskRNN as a recurrent neural-network approach to instance-level video object segmentation. Its recurrent component and combination of segmentation and localization nets use long-term temporal information and a location prior to improve results.
- MaskRNN is a recurrent neural-network approach for instance-level video object segmentation.
- The recurrent component and combination of segmentation and localization nets use long-term temporal information and a location prior to improve results.