Source-linked AI summary
One-Shot Video Object Segmentation
Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, Luc Van Gool
TL;DR
Semi-supervised video object segmentation requires separating a target from the background across a video when only one frame is annotated. OSVOS progressively transfers generic ImageNet knowledge to foreground segmentation and then fine-tunes on the target object, achieving 79.8% versus 68.0% state-of-the-art performance while processing frames independently. The approach also supports improved accuracy with additional annotations and avoids explicit temporal-consistency modeling.
Problem
The paper investigates how much training data is needed to segment an object throughout a video when only one labeled example, such as the first frame, is available.
Method
OSVOS trains a Fully Convolutional Network on varied objects and fine-tunes it for a particular instance using a small number of iterations at test time.
Results
79.8% versus 68.0% state-of-the-art performance is reported on DAVIS and Youtube-Objects, with 86.9% reached using four annotated frames per sequence.
Takeaways & Limitations
Independent frame processing can produce temporally stable segmentation without optical flow, temporal smoothing, sequential processing, or temporal error propagation.
Takeaways & Limitations
OSVOS’s segmentations have room for improvement in contour localization because classification-oriented spatial invariance contrasts with accurate contour localization.
Abstract
from arXiv · showhide
This paper tackles the task of semi-supervised video object segmentation, i.e., the separation of an object from the background in a video, given the mask of the first frame. We present One-Shot Video Object Segmentation (OSVOS), based on a fully-convolutional neural network architecture that is able to successively transfer generic semantic information, learned on ImageNet, to the task of foreground segmentation, and finally to learning the appearance of a single annotated object of the test sequence (hence one-shot). Although all frames are processed independently, the results are temporally coherent and stable. We perform experiments on two annotated video segmentation databases, which show that OSVOS is fast and improves the state of the art by a significant margin (79.8% vs 68.0%).
1. Introduction
OSVOS addresses semi-supervised video object segmentation from a single annotated frame by progressively adapting generic CNN knowledge to a specific object. It processes frames independently while offering temporal stability and adjustable speed–accuracy supervision trade-offs.
- Motivation: OSVOS segments video foreground and background from the manual annotation of one or more frames.The example uses the first-frame mask to produce object masks across 90 frames.
- Method: The method transfers information from ImageNet recognition through video segmentation training to fine-tuning on one target-object frame.This progression moves from generic semantic information to usual object shapes and finally the specific object’s appearance.
- Temporal modeling: OSVOS processes frames independently and obtains temporal consistency without an explicitly imposed temporal constraint.This design avoids sequential processing and can handle occlusions, motion variation, and non-temporally propagated errors.
- Speed and supervision: OSVOS supports speed–accuracy choices, running at 181 ms per frame with 71.5% accuracy or reaching 79.7% at 7.83 s per frame.Additional annotations further improve results to 84.6% with two frames and 86.9% with four.
- Architecture: The architecture uses Fully Convolutional Networks for dense video-segmentation predictions.The paper presents this as the first use of FCNs for video segmentation.
- Results: 79.8% versus 68.0% state-of-the-art performance demonstrates a substantial improvement on two video object segmentation datasets.The experiments use DAVIS and Youtube-Objects.
2. Related Work
Prior semi-supervised video segmentation methods commonly propagate masks through temporal consistency, often using optical flow or sequential processing. OSVOS instead applies an independently processed FCN-based segmentation model trained with limited annotation.
- Video object segmentation and tracking: Most prior methods enforce temporal consistency to propagate an initial mask through subsequent video frames.Approaches may optimize over the full video, a subset of frames, or only neighboring frames.
- Video object segmentation and tracking: Optical flow and related temporal matching can considerably reduce the speed of existing video-segmentation pipelines.The related-work discussion also identifies optical flow or dense trajectories as difficult computations.
- Video object segmentation and tracking: Concurrent deep-learning methods refine masks frame by frame or combine CNN training with bilateral filtering and CRF post-processing.These approaches differ from OSVOS’s independent-frame pipeline.
- Video object segmentation and tracking: Tracking methods using CNNs learn object representations to find similar windows, whereas OSVOS learns one model from frame 1 to segment the remaining frames.The distinction is between bounding-box tracking and pixel-level segmentation from a single object model.
- FCNs for segmentation: FCNs established dense CNN predictions for segmentation, but downsampled activations can produce coarsely localized outputs.Prior work addresses localization with learned upsampling, skip connections, or intermediate-layer supervision.
- FCNs for segmentation: This work explores accurately localized dense prediction with an FCN from a single segmented frame.The paper applies FCN ideas to video segmentation under very limited annotation.
3. One-Shot Deep Learning
OSVOS progressively adapts a fully convolutional network from generic image recognition to foreground segmentation and then to a specific object using one annotated frame. Its architecture balances localization, limited-data training, inference speed, and refinement quality through independent modules and adjustable fine-tuning.
- 3. One-Shot Deep Learning: OSVOS is inspired by progressively refining generic object knowledge into a model of the particular object to segment.The method moves from the prior “It is an object” to “It is this particular object.”
- 3.1. End-to-end trainable foreground FCN: The foreground FCN is trained for binary pixel classification, with a foreground branch producing dense image-sized predictions.The architecture is designed for accurate localization, relatively few trainable parameters, and fast testing.
- 3.1. End-to-end trainable foreground FCN: OSVOS uses pixel-wise cross-entropy and a class-imbalance modification that weights positive and negative labeled pixels.The sigmoid output provides pixel probabilities, while the modified loss addresses imbalance between the two binary classes.
- 3.2. Training details: Offline training adapts an ImageNet-pretrained base network on DAVIS masks, after which online fine-tuning specializes it to the annotated test object.Offline training uses augmented data and stochastic gradient descent for 50,000 iterations; online training then uses the image and mask of the first frame.
- 3.2. Training details: 102 ms per 480 × 854 frame enables fast testing, while longer fine-tuning improves quality and creates an explicit speed–accuracy trade-off.Experiments vary fine-tuning from 10 seconds to 10 minutes per sequence, and Figure 3 illustrates qualitative evolution over time.
- 3.3. Contour snapping: Contour refinement improves localization through edge-aware smoothing or a learned contour branch, with the latter trading 400 ms-per-frame processing for greater accuracy.The contour branch is complementary to the foreground branch, and both refinement modules are optional and modular.
4. Experimental Validation
OSVOS is evaluated on DAVIS and Youtube-Objects through ablations, state-of-the-art comparisons, attribute analyses, timing measurements, and additional-supervision experiments. It generally combines strong segmentation quality with flexible speed and improves further when more annotations are provided.
- Ablation study: Removing parent-network pre-training lowers J by 15.2 points, removing one-shot learning lowers it by 27.3 points, and removing both yields J =17.6%.Boundary snapping adds 2.4 points of improvement and is faster than adding a conventional CRF.
- State-of-the-art comparison: OSVOS outperforms the state of the art on DAVIS, with 11.8 points above the second-best method in region similarity J and 17.2 points in contour accuracy F.It also exceeds an oracle selecting the best object proposal, while remaining 6.7 points below the COB|SP superpixel oracle.
- Attribute-based performance: OSVOS has the best performance across all annotated DAVIS attributes and the smallest performance decrease when each challenge is present.The attribute evaluation compares sequences containing each challenge with sequences without it.
- Training data and refinement: Using only approximately 200 annotated training frames reaches almost the performance of the full DAVIS training split.The method can also refine results by incorporating additional annotated frames, including frames that distinguish visually similar objects.
- Timing and quality trade-off: 74.7 versus 60.0 of BVS at 400 ms, and 79.8 versus 68.0 of OFL at lower speeds, show OSVOS is faster and/or more accurate across timing regimes.Boundary snapping provides better results at added computational cost, so users can choose between speed and quality.
- Cross-dataset evaluation: On Youtube-Objects, OSVOS performs slightly better than the state of the art OFL, which is significantly slower.The evaluation uses pre-computed results from prior work.
5. Conclusions
OSVOS addresses the need to segment video objects from a single annotated example despite deep learning’s usual data demands. It fine-tunes a generically pretrained network for one object, processes frames independently without explicit temporal modeling, and outperforms the DAVIS state of the art.
- Conclusions: Deep learning methods often require large amounts of training data, whereas OSVOS demonstrates one-shot video object segmentation from a single training sample.The method fine-tunes a network pretrained on generic datasets for the target object.
- Conclusions: OSVOS processes each frame independently without explicit optical-flow or temporal-smoothing models, avoiding temporal error propagation while producing temporally consistent segmentations.The conclusion reports this property together with highly accurate results.
- Conclusions: OSVOS outperforms the DAVIS state of the art by 11.8 points.This is the paper’s reported headline comparison.