Source-linked AI summary
Learning Video Object Segmentation from Static Images
Anna Khoreva, Federico Perazzi, Rodrigo Benenson, Bernt Schiele, Alexander Sorkine-Hornung
TL;DR
Video object segmentation requires propagating an object’s identity across frames despite scarce densely annotated video data. The paper introduces guided instance segmentation with a convnet trained on static images, combining offline mask refinement and online instance adaptation. It reports competitive performance across three datasets, including 85% mIoU with one annotation every 10th frame.
Problem
Dense pixel-wise video annotations are difficult and costly to obtain, limiting convnet training for video object segmentation.
Method
MaskTrack performs per-frame instance segmentation using previous-frame mask guidance and combined offline and online learning.
Results
85% mIoU is achieved with only one annotation every 10th frame, while the system reaches competitive performance across three heterogeneous benchmarks.
Takeaways & Limitations
The approach supports bounding-box or segmentation annotations and uses the same model and parameters across videos and datasets.
Takeaways & Limitations
The authors identify future work in architectures with temporal dimensions and global optimization strategies.
Abstract
from arXiv · showhide
Inspired by recent advances of deep learning in instance segmentation and object tracking, we introduce video object segmentation problem as a concept of guided instance segmentation. Our model proceeds on a per-frame basis, guided by the output of the previous frame towards the object of interest in the next frame. We demonstrate that highly accurate object segmentation in videos can be enabled by using a convnet trained with static images only. The key ingredient of our approach is a combination of offline and online learning strategies, where the former serves to produce a refined mask from the previous frame estimate and the latter allows to capture the appearance of the specific object instance. Our method can handle different types of input annotations: bounding boxes and segments, as well as incorporate multiple annotated frames, making the system suitable for diverse applications. We obtain competitive results on three different datasets, independently from the type of input annotation.
1. Introduction
The paper addresses video object segmentation without large densely annotated video datasets by treating it as guided instance segmentation. A convnet uses previous-frame masks and combined offline/online learning to segment objects accurately and flexibly.
- Dense pixel-wise video annotations are usually prohibitive, motivating methods that can learn video segmentation from static images.
- The task is to segment the same annotated object instance throughout the remaining video frames.
- The method feeds each previous-frame mask estimate to a pixel-labelling convnet for per-frame instance segmentation, termed guided instance segmentation.
- The feed-forward system processes one frame at a time, supports bounding-box or segment annotations, and achieves competitive results across heterogeneous benchmarks.
- A combined offline and online learning strategy refines rough masks and specializes the model to the object instance in a new video.
2. Related work
Prior video segmentation methods propagate information through tracking, local or global graphs, and unsupervised motion segmentation. This work instead performs pixel-level, frame-by-frame segmentation with an online fine-tuned convnet.
- Earlier methods combine box tracking with segmentation or propagate first-frame masks through graph-labeling techniques.
- Local-propagation approaches connect neighboring frames using superpixels, object parts, pixels, or bilateral grids.
- Global-propagation methods add long-range frame connections but operate on larger regions such as superpixels or object proposals for speed.
- Unsupervised approaches segment moving image regions and select the space-time tube matching the annotation afterward.
- The proposed approach avoids intermediate boxes, superpixels, and proposals, using an online fine-tuned convnet to propagate masks at full detail.
- Its training strategy draws on offline box regression and online appearance adaptation, but predicts pixel masks rather than boxes.
3. MaskTrack method
MaskTrack formulates video object segmentation as guided instance segmentation: a convnet refines a rough previous-frame mask, while offline and online training provide generalization and instance specialization.
- The model uses a pixel-labelling convnet to produce an instance mask for each frame, guided by the previous frame’s estimate.
- Offline training teaches the network to convert rough masks into accurate masks, while online training specializes it to the target instance.
- The RGB input is expanded with a mask channel encoding the object’s approximate visible area, location, and shape.
- Rough masks can be sufficient because their primary role is to direct the convnet toward the correct object instance, even when supplied as a large bounding box.
- Static-image annotations support training by synthesizing rough masks, avoiding the need for video training data.
- At inference, the previous mask is coarsened and passed to the network for current-frame segmentation; online fine-tuning further improves performance.
- MaskTrack is the paper’s full system using both offline and online training, with variants supporting bounding-box supervision and optical-flow guidance.
4. Network implementation and training
The implementation uses a DeepLabv2-VGG convnet with synthetic mask deformations for offline training and augmented first-frame samples for online adaptation. The base system runs substantially faster than a cited graph-based method.
- The network follows DeepLabv2-VGG training and is initialized from ImageNet-pretrained VGG16 weights.
- Offline input masks are generated with affine and thin-plate-spline deformations, followed by five-pixel-radius dilation.
- Offline optimization uses SGD with minibatches of 10 images, initial learning rate 0.001, momentum 0.9, weight decay 0.0005, and 20k iterations.
- Online adaptation fine-tunes the offline model for 200 iterations using approximately 10^3 augmented samples from the first annotated frame.
- Online fine-tuning updates all convolutional and fully connected layers using the offline learning parameters.
- 12 seconds per frame is the base MaskTrack runtime on DAVIS, versus 2 minutes per frame for ObjFlow.
5. Results
The evaluation shows that MaskTrack achieves competitive video object segmentation across three datasets with fixed parameters, while remaining effective under varied annotation types and amounts. Ablations identify online fine-tuning, static-image offline training, and optional optical-flow/CRF components as important performance factors.
- Ablation study: 74.8 →78.4 mIoU follows from adding optical flow, while CRF post-processing reaches 80% mIoU on DAVIS.These additions are specifically fine-tuned for DAVIS.
- Ablation study: Disabling online fine-tuning causes an approximately 5-IoU-point drop, whereas omitting offline training leaves 57.6 mIoU despite a drastic performance decrease.The results support complementary roles for offline mask refinement and online adaptation to the target object.
- Training data: Reducing training data from 11k to 5k causes only a minor mIoU decrease, while training on small video datasets produces a minor decrease relative to static-image training.The authors attribute the latter to limited video-data diversity and benchmark domain shift, concluding that static images do not reduce performance.
- Cross-dataset evaluation: MaskTrack obtains competitive performance across DAVIS, YoutubeObjects, and SegTrack-v2 using the same model and parameters in a single forward pass.The base results do not use global optimization or optical flow.
- Cross-dataset evaluation: 80.3 mIoU on DAVIS, 72.6 on YoutubeObjects, and 70.3 on SegTrack-v2 are reached with dataset-specific optical-flow and CRF additions.The extended MaskTrack+Flow+CRF configuration improves the results beyond the base system.
- Multiple-frame annotations: With 10% annotated frames, segment annotations reach 0.86 mIoU and a 0.81 20% quantile, while box annotations reach roughly the 0.8 mIoU range and 70% of frames exceed 0.89 IoU.For segment annotations, the baseline reaches only 0.64 mIoU at 10% annotated frames; additional box annotations provide little gain after 10%.
6. Conclusion
The paper concludes that guided instance segmentation with offline and online training enables competitive video object segmentation from image annotations, while remaining flexible to annotation type. It also identifies temporal modeling and global optimization as future directions.
- The method uses frame-by-frame instance segmentation with offline and online training from image annotations only.
- The system achieves competitive performance on three heterogeneous video segmentation benchmarks using the same model and parameters across videos.
- The method remains competitive with bounding boxes instead of segmentation masks as input annotations.
- 85% mIoU is reached with one annotation every 10th frame.
- Future work should explore more sophisticated architectures, temporal dimensions, and global optimization strategies.
Supplementary material
The supplementary material adds quantitative, attribute-based, qualitative, and implementation-related analyses across the evaluated datasets. It also documents dataset-specific tuning and additional visual examples.
- Additional quantitative results are provided for DAVIS, YoutubeObjects, and SegTrack-v2.
- The supplement reports detailed attribute-based evaluation in Section C and Table S6.
- Dataset-specific tuning for additional ingredients is described in Section D.
- Additional qualitative results cover first-frame box and segmentation supervision in Section E and Figure 6.
- Examples document mask generation for the extra input channel and optical-flow magnitude images.
B. Additional quantitative results
This supplementary section provides additional quantitative comparisons across DAVIS, SegTrack-v2, and YoutubeObjects, complementing the main paper's evaluation.
- Additional quantitative results compare methods on DAVIS, SegTrack-v2, and YoutubeObjects.DAVIS and SegTrack-v2 use per-sequence comparisons, while YoutubeObjects uses per-category comparisons.
C. Attribute-based evaluation
The attribute-based evaluation finds MaskTrack robust across varied DAVIS challenges, with optical flow and CRF improving robustness across all categories. Camera-shake is the stated exception to its favorable comparisons.
- Table S6 provides detailed DAVIS attribute evaluation complementing Figure 5 in the main paper.
- MaskTrack compares favorably on DAVIS video-attribute subsets except camera-shake, where ObjFlow marginally outperforms it.
- MaskTrack handles fast-motion, appearance change, and out-of-view cases where competitive methods are failing.
- Over 70% mIoU is reached on every attribute subcategory after adding optical flow and CRF post-processing.The combined MaskTrack+Flow+CRF variant particularly improves discrimination for low resolution, scale variation, and appearance change.
D. Dataset specific tuning
Dataset-specific additions improve MaskTrack performance, but optical flow is brittle across datasets and requires dataset-aware handling. CRF post-processing is also tuned per dataset.
- Dataset-specific tuning: 80.3 mIoU on DAVIS, 72.6 on YoutubeObjects and 70.3 on SegTrackv-2 are reached after adding dataset-specific ingredients such as optical flow and CRF post-processing.The reported gains are obtained with tuning tailored to different datasets.
- Optical flow: Optical-flow magnitude is processed in parallel with RGB inputs, and the two output scores are fused by averaging.
- Optical flow: Optical flow can provide gains but is brittle across datasets because of flow-algorithm failure modes and sensitivity to video data quality.The method therefore uses optical-flow-magnitude training for YoutubeObjects and SegTrackv-2, while DAVIS uses the original RGB-trained model.
- Optical flow: 1∼4% improvement is provided by integrating optical flow on top of MaskTrack on each dataset.
- CRF post-processing: A fully connected CRF is crossvalidated separately for each dataset, with 10 mean-field iterations and a three-frame temporal window.
E. Additional qualitative results
Additional qualitative results show that MaskTrack and MaskTrackBox produce high-quality video segmentations from either first-frame segments or only bounding boxes. Training masks are generated by deforming annotated masks to mimic plausible preceding-frame shapes.
- Annotation types: High-quality segmentations are produced on DAVIS from either first-frame segment annotations or only bounding-box annotations.The system is presented as suitable for applications with diverse annotation types.
- Training-mask generation: Mask deformations are crucial because they make segmentation estimates more robust to noise in the input mask at test time.The deformation ingredient affects quality in both offline and online training.
- Optical-flow guidance: Optical-flow magnitude provides grayscale-like object-shape information that complements RGB inputs.Examples are shown across different datasets.
- Annotation types: MaskTrackBox and MaskTrack generate results comparable to ground truth when propagating first-frame box or segment annotations.
- Training-mask generation: Multiple training masks are generated from one annotated image to mimic plausible object shapes on the preceding frame.