PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Adaptation of Convolutional Neural Networks for Video Object Segmentation

Paul Voigtlaender, Bastian Leibe

arXiv:1706.09364v2cs.CV

TL;DR

Semi-supervised VOS must segment an object from its first-frame mask despite changing appearance, a limitation of OSVOS’s fixed test-time network. OnAVOS adds objectness pretraining and confidence- and spatially selected online updates, achieving 85.7% mIoU on DAVIS.

  • Problem

    OSVOS fine-tunes on only the first frame and cannot adapt to large appearance changes during test-time VOS.

  • Method

    OnAVOS adds PASCAL objectness pretraining and online network updates using confidence-based positive and spatially selected negative examples.

  • Results

    85.7% mIoU on DAVIS, an improvement of almost 6% over OSVOS.

  • Takeaways & Limitations

    Objectness pretraining and online adaptation are highly effective for semi-supervised VOS and generalize to another dataset.

Abstract

from arXiv · show

We tackle the task of semi-supervised video object segmentation, i.e. segmenting the pixels belonging to an object in the video using the ground truth pixel mask for the first frame. We build on the recently introduced one-shot video object segmentation (OSVOS) approach which uses a pretrained network and fine-tunes it on the first frame. While achieving impressive performance, at test time OSVOS uses the fine-tuned network in unchanged form and is not able to adapt to large changes in object appearance. To overcome this limitation, we propose Online Adaptive Video Object Segmentation (OnAVOS) which updates the network online using training examples selected based on the confidence of the network and the spatial configuration. Additionally, we add a pretraining step based on objectness, which is learned on PASCAL. Our experiments show that both extensions are highly effective and improve the state of the art on DAVIS to an intersection-over-union score of 85.7%.

1 Introduction

Semi-supervised VOS segments a generic object from its first-frame ground-truth mask. OnAVOS addresses OSVOS’s inability to adapt to large appearance changes through online updates and selected training examples.

  • Semi-supervised VOS segments pixels belonging to a generic object using the ground-truth pixel mask from the first frame.
  • OSVOS fine-tunes a pretrained convolutional neural network on the first frame but cannot adapt to large appearance changes at test time.
  • Online adaptation helps the network handle viewpoint changes and newly appearing objects in DAVIS validation sequences.
  • OnAVOS updates the network online using pixels selected as confident positive examples and spatially distant negative examples.
  • OnAVOS also adds objectness pretraining and reports significant improvements over the state of the art on two datasets.

2 Related Work

Related work spans classical and neural VOS, temporal-context methods, pixel-objectness pretraining, online tracking adaptation, and fully convolutional segmentation architectures.

  • Video Object Segmentation: Classical VOS methods reduce input granularity with superpixels, patches, or object proposals before optimizing segmentation.
  • Video Object Segmentation: Neural VOS methods include OSVOS, MaskTrack, video propagation networks, and optical-flow-based temporal cues.
  • Pixel Objectness: Pixel objectness pretraining teaches a network to decide whether each pixel belongs to an object-like region and is adopted here for semi-supervised VOS.
  • Video Object Segmentation: LucidTracker uses elaborate data augmentation, whereas the paper reports better performance using only conventional augmentation methods.
  • Online Adaptation: Online adaptation has established precedents in bounding-box tracking, including TLD, online AdaBoost, and MDNet.
  • Fully Convolutional Networks for Semantic Segmentation: Fully convolutional segmentation repurposes classification networks with 1×1 convolutions and skip connections, while later designs use dilated convolutions without skip connections.

3 One-Shot Video Object Segmentation

OnAVOS builds on OSVOS with objectness and domain-specific pretraining, first-frame specialization, and a residual convolutional architecture for one-shot segmentation.

  • OnAVOS adds pixel-objectness pretraining, a newer architecture, and online adaptation to the OSVOS one-shot VOS framework.
  • Pretraining: The base network is pretrained on large datasets to learn an object representation that initializes the VOS model.
  • Pretraining: PASCAL pretraining maps its 20 annotated classes to foreground and other regions to background for pixel-objectness learning.
  • Pretraining: DAVIS fine-tuning adapts the objectness network to target-domain characteristics and produces a domain-specific objectness network.
  • One-Shot Segmentation: Fine-tuning on the first-frame ground-truth mask specializes the pretrained network to the target object’s identity and appearance.
  • Network Architecture: The segmentation model uses a wide 38-layer ResNet variant with roughly 124 million parameters, dilated convolutions, and 0.5 posterior thresholding.

4 Online Adaptation

OnAVOS adapts online to appearance changes and newly appearing objects by selecting spatially configured, confidence-based training pixels while limiting drift with first-frame replay and hard-negative handling.

  • Appearance changes and newly entering background objects motivate online adaptation because objectness pretraining can assign high probability to unseen objects.
  • Pixels with foreground probability above threshold α become positive examples, preserving a memory of the positive class during adaptation.
  • Negative examples are selected spatially from pixels far from the preceding-frame mask because low-probability selection can include false negatives during appearance changes.
  • Pixels outside the selected positive and negative sets receive a “don’t care” label and are ignored during online updates.
  • Mixing the first frame into online updates counters drift caused by naively fine-tuning on current-frame examples.
  • Hard negatives are removed from the next foreground mask so they can be selected again as negatives; stronger hard-negative updates did not improve results further.
  • If erosion leaves an empty foreground mask, online updates stop until the network finds a nonempty mask again.

5 Experiments

Experiments evaluate pretraining, online adaptation, ablations, runtime, post-processing, and cross-dataset performance. OnAVOS reaches 85.7% mIoU on DAVIS and 77.4% on YouTube-Objects.

  • Baseline Systems: 65.2% mIoU without PASCAL or DAVIS pretraining improves to 77.6% with PASCAL and 78.0% with DAVIS pretraining.Using both datasets further improves performance to 80.3% mIoU.
  • Baseline Systems: 72.7% mIoU is obtained in the unsupervised setup, slightly exceeding FusionSeg’s 70.7% mIoU on the DAVIS validation set.FusionSeg uses objectness and optical flow as additional cues.
  • Baseline Systems: 80.3% mIoU with PASCAL and DAVIS pretraining exceeds 77.4% mIoU for OSVOS, while the comparable system without PASCAL reaches 78.0%.The improvement without PASCAL is attributed to the newer network architecture; PASCAL objectness pretraining adds 2.3%.
  • Online Adaptation: 82.8% mIoU with full online adaptation falls to 80.3% without adaptation and to 69.1% without mixing the first frame during updates.The ablation identifies negative examples as more important than positive ones and shows that omitting first-frame mixing causes drift.
  • Runtime: 90 seconds per sequence for initial fine-tuning increases to around 15 minutes with online adaptation on DAVIS.The corresponding runtimes are roughly 1.3 seconds and 13 seconds per frame, respectively.
  • Comparison to State of the Art: 85.7% mIoU on DAVIS and 77.4% mIoU on YouTube-Objects outperform the reported comparison results.The DAVIS result is almost 6% above OSVOS, while YouTube-Objects exceeds LucidTracker’s 76.2% mIoU.

6 Conclusion

The paper proposes OnAVOS by adding objectness pretraining and online adaptation to OSVOS for semi-supervised video object segmentation. The authors report effectiveness, hyperparameter robustness, and generalization to another dataset, while leaving explicit temporal context for future work.

  • Conclusion: OnAVOS builds on OSVOS and combines objectness pretraining with online adaptation for semi-supervised video object segmentation.The paper presents both additions as highly effective.
  • Conclusion: Online adaptation is reported as robust to hyperparameter choices and generalizes to another dataset.The authors expect adaptation schemes to improve robustness against large appearance changes.
  • Conclusion: Explicit incorporation of temporal context information is identified as future work.The current method is described as adapting online rather than explicitly incorporating temporal context.

A More Comprehensive Comparison to Other Methods

Table 4 provides a broader comparison with other methods on DAVIS and YouTube-Objects. The reported comparisons use validation-set results for DAVIS where prior methods otherwise included training sequences.

  • Comparison Scope: Table 4 compares other methods on the DAVIS validation set and the YouTube-Objects dataset.The table includes concurrent work published only on arXiv.
  • Evaluation Protocol: DAVIS results for MaskTrack and LucidTracker are shown on the validation set only, unlike their reports covering all sequences including training data.This distinction is stated to clarify comparability across methods.

B Additional Evaluation Measures for DAVIS

OnAVOS is evaluated on the DAVIS validation set using segmentation overlap, contour accuracy, and temporal stability. The method achieves the best mean results across all three measures.

  • The Jaccard index J measures mean intersection-over-union between predicted foreground masks and ground truth masks.
  • The contour accuracy measure F evaluates agreement between predicted segmentation boundaries and ground truth boundaries.
  • The temporal stability measure T evaluates consistency of predicted masks over time.
  • OnAVOS obtains the best mean results for J, F, and T on the DAVIS validation set.Its temporal stability result exceeds MaskTrack’s despite MaskTrack explicitly propagating masks using temporal context.

C Per-Sequence Results for DAVIS

The DAVIS validation results report mIoU separately for all 20 sequences. OnAVOS achieves either the best or second-best result on 18 sequences, with best and second-best results highlighted.

  • OnAVOS obtains either the best or second-best mIoU result on 18 of the 20 DAVIS validation sequences.
  • The per-sequence DAVIS validation table reports results separately for each sequence.
  • Best and second-best results are highlighted with bold and italic fonts, respectively.

D Hyperparameter Study on DAVIS

OnAVOS is generally robust to hyperparameter choices on DAVIS, although the online learning rate has the strongest influence. The study also indicates that erosion is not helpful and that the same settings generalize to YouTube-Objects.

  • OnAVOS outperforms the un-adapted baseline for every tested hyperparameter configuration.The study changes one hyperparameter at a time from the operating point while holding the others constant.
  • The online learning rate λ is the most important hyperparameter, while β and α have moderate influence.The distance threshold d and the step counts nonline and ncurr produce only minor accuracy changes within a reasonable range.
  • The best erosion setting is size 1, corresponding to no erosion, suggesting that erosion is not helpful on DAVIS.
  • The same hyperparameter settings on YouTube-Objects further confirm OnAVOS’s generalizability and robustness.
Loading 1706.09364v2…