Source-linked AI summary
Learning Fast and Robust Target Models for Video Object Segmentation
Andreas Robinson, Felix Järemo Lawin, Martin Danelljan, Fahad Shahbaz Khan, Michael Felsberg
TL;DR
Video object segmentation must maintain accurate target masks despite appearance changes and similar background objects, while avoiding the low speed and overfitting associated with first-frame fine-tuning. The paper combines an inference-learned lightweight discriminative target model with an offline-trained segmentation network, achieving strong benchmark performance at high frame-rates. Its results support fast VOS with limited training data, though thin structures and nearly identical targets remain difficult.
Problem
VOS requires robust and accurate target pixel classification under appearance changes, occlusions, and similar background objects, while prior fine-tuning methods can be slow and overfit.
Method
A lightweight discriminative target model learns during inference to produce coarse target scores, which a fixed offline-trained segmentation network refines into final masks.
Results
76.7 on DAVIS 2017 and 72.1 on YouTube-VOS are achieved at 22 FPS, while a faster version reaches 41 FPS with slight accuracy degradation.
Takeaways & Limitations
The method provides state-of-the-art YouTube-VOS and competitive DAVIS 2017 results at high frame-rates despite limited training data.
Takeaways & Limitations
The target model struggles with thin or small structures and with separating almost identical targets.
Abstract
from arXiv · showhide
Video object segmentation (VOS) is a highly challenging problem since the initial mask, defining the target object, is only given at test-time. The main difficulty is to effectively handle appearance changes and similar background objects, while maintaining accurate segmentation. Most previous approaches fine-tune segmentation networks on the first frame, resulting in impractical frame-rates and risk of overfitting. More recent methods integrate generative target appearance models, but either achieve limited robustness or require large amounts of training data. We propose a novel VOS architecture consisting of two network components. The target appearance model consists of a light-weight module, which is learned during the inference stage using fast optimization techniques to predict a coarse but robust target segmentation. The segmentation model is exclusively trained offline, designed to process the coarse scores into high quality segmentation masks. Our method is fast, easily trainable and remains highly effective in cases of limited training data. We perform extensive experiments on the challenging YouTube-VOS and DAVIS datasets. Our network achieves favorable performance, while operating at higher frame-rates compared to state-of-the-art. Code and trained models are available at https://github.com/andr345/frtm-vos.
1. Introduction
Video object segmentation must classify target pixels robustly and accurately despite appearance changes, occlusions, and similar background objects. The proposed architecture combines an inference-learned discriminative target model with an offline-trained segmentation network, achieving strong performance at high frame-rates.
- Problem: Video object segmentation predicts pixel-accurate masks for a specified target in every frame, using a ground-truth mask in the first frame.The task is studied here in the semi-supervised setting.
- Problem: Successful VOS requires robust and accurate target pixel classification under appearance changes, occlusions, and similar background objects.
- Limitations of prior methods: First-frame fine-tuning can produce accurate masks but suffers from low frame-rates, overfitting, and reduced performance under drastic changes, occlusions, and distractors.The fine-tuning procedure is also absent from offline training, so the full inference process is not simulated.
- Related approaches: Generative target appearance models support efficient closed-form integration but require large datasets and rely heavily on image and synthetic VOS pre-training.The paper therefore investigates integrating a discriminative target appearance model into VOS.
- Proposed approach: The method combines a lightweight discriminative target model that learns during inference with an offline-trained segmentation network that refines its coarse scores.The segmentation network uses target scores as guidance, adheres to object edges, and suppresses erroneous classifications.
- Results: 76.7 on DAVIS 2017 and 72.1 on YouTube-VOS are achieved at 22 FPS, while a faster version reaches 41 FPS with slight accuracy degradation.The method requires no pre-training on image or synthetic video segmentation data and is trained on VOS data in a single phase.
2. Related work
Related VOS methods commonly fine-tune segmentation networks on initial or later frames, sometimes adding optical flow. This work instead draws on optimization-based learning of a discriminative target appearance model.
- First-frame fine-tuning: Many state-of-the-art VOS methods train segmentation networks offline and fine-tune them on the first frame to learn target-specific appearance.Some methods additionally fine-tune on subsequent video frames or integrate optical flow.
- Discriminative target modeling: The proposed approach takes inspiration from optimization-based learning of a discriminative model to capture target object appearance.
3. Method
The method separates robust target modeling from high-resolution segmentation: a lightweight target model is optimized during inference, while an offline-trained network refines its coarse scores. A memory of past samples supports periodic target-model updates and reduces drift.
- Target model: The target model D learns target-specific parameters from the first-frame mask during inference and produces coarse but robust segmentation scores.It is a lightweight two-convolution-layer model operating on deep features.
- Segmentation network: The segmentation network S combines backbone features with target scores to predict the final high-resolution target mask while remaining target agnostic.Its target segmentation encoder integrates scores and features, followed by refinement blocks in a U-Net-based architecture.
- Online adaptation: During inference, predicted masks and associated features are stored in memory, and D is periodically re-optimized over accumulated samples to reduce model drift.Keeping past samples avoids relying only on the latest frame.
- Target-model optimization: The target model is optimized with an L2 loss using regularization and pixel-weighting masks to balance target and background contributions.The optimization uses a quadratic approximation, Gauss-Newton updates, and Conjugate Gradient descent.
- Offline training: The segmentation network is trained offline with simulated inference episodes, while the feature extractor remains frozen.Training uses a reference frame and validation frames sampled from the same video sequence.
4. Experiments
Experiments on YouTube-VOS and DAVIS show that the proposed components improve segmentation performance while maintaining practical or high frame-rates. Ablations, benchmark comparisons, and qualitative examples identify both the method’s strengths and its failure cases.
- Ablation study: The target model alone matches first-frame fine-tuning while avoiding the severe overfitting associated with fine-tuning an entire segmentation network.Its limited capacity acts as implicit regularization for robustness and generalization.
- Ablation study: On a 300-sequence YouTube-VOS validation split, online updating raises the complete model’s J score to 71.4%.The segmentation network adds 8.3% before online updating, while updating contributes a further 1.6%.
- Comparison to state-of-the-art: On YouTube-VOS, Ours achieves a G-score of 72.1 and outperforms PReMVOS by a relative margin of 5.2%.PReMVOS and other first-frame fine-tuning methods operate below 0.3 FPS, whereas the proposed approach uses a single network with a lightweight target model.
- Comparison to state-of-the-art: Ours-fast reaches a G-score of 65.7 at 41.3 FPS, the highest frame-rate among previous methods reported in the comparison.The method remains effective when trained only on YouTube-VOS video data, where it outperforms earlier methods trained under the same restriction.
- Comparison to state-of-the-art: On DAVIS 2017, Ours obtains a J &F score of 76.7, while Ours-fast obtains 70.2; on DAVIS 2016, the methods perform comparably to fine-tuning-based approaches.When trained only on DAVIS 2017, Ours (DV17) achieves a J &F score of 68.8 and outperforms the compared methods in that setting.
- Qualitative Analysis: Qualitative results show robust segmentation under large target deformations, but failures occur when objects are too thin or appear too similar.The target model commonly produces robust scores, yet struggles with thin or small structures and details.
5. Conclusion
The approach combines a lightweight discriminative target appearance model with a segmentation network, achieving high frame-rates and strong performance on YouTubeVOS and DAVIS 2017 despite limited training data.
- The method integrates a lightweight, highly discriminative target appearance model with a segmentation network.
- A linear discriminative model generates robust target predictions, which the segmentation network converts into high-quality object segmentations.
- The target model is efficiently trained during inference, enabling operation at high frame-rates.
- The method achieves state-of-the-art performance on YouTubeVOS and competitive results on DAVIS 2017 despite limited training data.
Supplementary Material
The supplementary material describes initial sample generation and provides additional quantitative results and parameter ablations.
- The supplement describes how the initial samples are generated for the method's first-frame training process.
- It also reports more detailed quantitative results and analyzes parameter effects through ablation experiments.
6. Initial sample generation
The initial target-model dataset combines the given first-frame mask and features with augmented samples that increase variety while retaining the original sample's greater weight.
- The target model is first trained on an initial dataset built from the target mask and extracted features of the first frame.
- Additional samples are generated by cutting out the target object, restoring the background with fast inpainting, and applying augmentation.
- The unmodified first frame and augmented frames are inserted into M0, with the original sample assigned twice the weight of the other samples.
- Figure 6 illustrates the original first-frame image and mask alongside the resulting augmented image-mask pairs.
7. Detailed Quantitative Results
The supplementary experiments examine data requirements, runtime scaling, and parameter sensitivity, showing competitive performance with reduced training data and approximately linear initialization scaling.
- Training data impact: At 50% of the YouTube-VOS training data, the approach surpasses all compared methods trained on the full training set.
- Training data impact: The target-only variant, requiring no pre-training, obtains a higher G-score than OSVOS, OnAVOS, and RVOS.
- Runtime analysis: The initialization phase accounts for 41 percent of total runtime on DAVIS 2016, ranging from 31 to 60 percent across videos.
- Runtime analysis: Initialization runtime increases approximately linearly with the initial sample-memory size |M0|.
- Runtime analysis: Forward-phase processing is dominated by model-update training and feature extraction, while most short DAVIS videos do not fill the maximum memory.
- Parameter analysis: The method's mean J is relatively insensitive to the memory learning rate η and target-model update interval ts.
- Parameter analysis: YouTubeVOS evaluation is insensitive to the initial dataset size |M0|, while augmentation still gives the best performance in the reported comparison.