Source-linked AI summary
Video Object Segmentation Without Temporal Information
Kevis-Kokitsi Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, Luc Van Gool
TL;DR
The paper addresses video object segmentation methods’ dependence on temporal consistency, which can be disrupted by occlusions, lost frames, or abrupt motion. It proposes OSVOSS, which progressively adapts a CNN and combines appearance with semantic instance information while segmenting frames independently. On DAVIS 2016 and Youtube-Objects, it reports state-of-the-art accuracy and speed across operating points, including 75.1 versus 59.4 at 300 miliseconds per frame and 86.5 versus 85.5 at best performance.
Problem
Temporal video segmentation relies on neighboring-frame information, but the paper investigates segmentation from a single annotated frame without temporal information.
Method
OSVOSS progressively adapts an ImageNet-pretrained fully convolutional network from generic information to video segmentation and a specific object, combining semantic instance cues with appearance.
Results
OSVOSS reports state-of-the-art results on DAVIS 2016 and Youtube-Objects, with 75.1 versus 59.4 at 300 miliseconds per frame and 86.5 versus 85.5 at best performance.
Takeaways & Limitations
Independent processing is reported to provide robustness to occlusions and lost frames while maintaining low execution speed, and semantic priors help preserve quality through appearance changes and longer videos.
Takeaways & Limitations
The method’s handling of out-of-vocabulary instances may not transfer to other domains containing uncommon objects or object parts.
Abstract
from arXiv · showhide
Video Object Segmentation, and video processing in general, has been historically dominated by methods that rely on the temporal consistency and redundancy in consecutive video frames. When the temporal smoothness is suddenly broken, such as when an object is occluded, or some frames are missing in a sequence, the result of these methods can deteriorate significantly or they may not even produce any result at all. This paper explores the orthogonal approach of processing each frame independently, i.e disregarding the temporal information. In particular, it tackles the task of semi-supervised video object segmentation: the separation of an object from the background in a video, given its mask in the first frame. We present Semantic One-Shot Video Object Segmentation (OSVOS-S), based on a fully-convolutional neural network architecture that is able to successively transfer generic semantic information, learned on ImageNet, to the task of foreground segmentation, and finally to learning the appearance of a single annotated object of the test sequence (hence one shot). We show that instance level semantic information, when combined effectively, can dramatically improve the results of our previous method, OSVOS. We perform experiments on two recent video segmentation databases, which show that OSVOS-S is both the fastest and most accurate method in the state of the art.
1 INTRODUCTION
The paper reframes semi-supervised video object segmentation as independent per-frame segmentation from one annotated frame, avoiding reliance on temporal consistency. OSVOSS progressively transfers generic, video-object, and instance-specific information to achieve accuracy, speed, and robustness to interruptions.
- Motivation: Temporal redundancy dominates video processing, but optical-flow and matching computations make temporal modeling difficult.Neighboring frames usually carry similar information, yet estimating their correspondences can itself be challenging.
- Motivation: Independent per-frame processing avoids sequence interruptions, supports arbitrary processing rates and parallelization, and prevents errors from propagating temporally.The approach is designed to remain usable when occlusions, lost frames, or abrupt motion disrupt temporal smoothness.
- Method: OSVOSS adapts an ImageNet-pretrained CNN first to video object segmentation and then to a specific object using one annotated frame.This staged refinement moves from generic category information to video-object shapes and finally the appearance of the test object.
- Method: Semantic information is propagated from the first frame by selecting category-consistent instance masks and combining them with the object’s appearance model.The method uses semantic selection followed by semantic propagation to maintain category alignment across frames.
- Results: 75.1% accuracy at 300 miliseconds per frame rises to 86.5% when each 480×854 frame takes 4.5 seconds, exposing a speed–accuracy trade-off.The user chooses the amount of test-time fine-tuning to operate at different points on this trade-off.
- Results: Experiments on DAVIS 2016 and Youtube-Objects report improvements over the state of the art in both accuracy and speed.The paper also evaluates multi-object segmentation on DAVIS 2017 without adapting the method to that new problem.
2 RELATED WORK
Related work centers on temporally consistent propagation, deep fully convolutional segmentation, and semantic instance segmentation. OSVOSS instead segments frames independently and applies instance semantic information to video object segmentation.
- Semi-supervised Video Object Segmentation: Most semi-supervised video segmentation methods propagate an initial mask using temporal consistency, often with optical flow, CNN refinement, or bilateral filtering.Some pipelines also use CRFs, which can considerably reduce processing speed.
- Positioning: OSVOSS differs from temporal approaches by independently segmenting each frame, using a simpler pipeline intended to improve accuracy and speed.Its class-agnostic design is described as able to segment generic objects from one annotated example.
- FCNs for Segmentation: Fully convolutional segmentation methods use ImageNet-pretrained CNNs for dense prediction, while addressing coarse localization caused by downsampled activations.Skip connections, learned upsampling, and intermediate-layer supervision are established strategies for improving localization.
- Semantic Instance Segmentation: Semantic instance segmentation produces a separate segmentation mask for each individual object instance, unlike semantic segmentation or object detection.The reviewed methods include region-proposal, iterative, recurrent, position-sensitive, and Faster-RCNN-based approaches.
- Using Semantic Information to Aid Other Computer Vision Tasks: The paper claims to be the first to apply instance semantic information to video object segmentation.This extends the use of semantic cues beyond related tasks such as reconstruction and monocular depth estimation.
3 ONE-SHOT VIDEO OBJECT SEGMENTATION (OSVOS)
OSVOS gradually adapts a fully convolutional network from generic image recognition to foreground segmentation and finally to a specific object annotated in one frame. A complementary contour branch improves boundary localization, while fine-tuning controls the speed–accuracy trade-off.
- 3.1 End-to-end trainable foreground FCN: OSVOS fine-tunes a parent network on one annotated frame to build an appearance model for the particular video object.The parent network is first trained on DAVIS 2016 masks to learn generic foreground segmentation before object-specific adaptation.
- 3.1 End-to-end trainable foreground FCN: The architecture is designed for localized dense prediction, few trainable parameters, and efficient image-to-image inference.It removes fully connected classification layers and uses a VGG-based convolutional architecture with skip connections.
- 3.1 End-to-end trainable foreground FCN: The FCN uses binary foreground–background classification with pixel-wise cross-entropy, and a modified loss addresses class imbalance.The sigmoid output provides pixel probabilities, while β weights positive and negative pixel sets in the modified objective.
- 3.2 Training details: Fine-tuning produces a direct speed–accuracy trade-off: OSVOSS reaches 75.1% accuracy at 300 milliseconds per frame and 86.5% at 4.5 seconds per frame.Experiments vary fine-tuning from 10 seconds to 10 minutes per sequence, with longer training generally yielding better results but longer waits.
- 3.2 Training details: Ablation results show that both offline generic training and online object-specific training are crucial for good performance.Starting online training directly from ImageNet or omitting object-specific fine-tuning significantly worsens performance.
- 3.3 Contour snapping: A contour-detection CNN complements the foreground branch and refines object boundaries without adding online fine-tuning cost.The two branches use separate computations and losses; contour snapping is modular and can be omitted when speed is prioritized.
4 SEMANTIC GUIDANCE (OSVOSS)
OSVOS-S adds semantic instance guidance to appearance-based foreground segmentation, selecting object semantics in the first frame and propagating them independently across later frames. A conditional classifier combines semantic priors with appearance predictions while retaining softer per-pixel decisions when semantic masks are imperfect.
- Network overview: The network shares a VGG feature extractor across three pixel-wise classifiers: one first-round foreground estimator and two semantic-prior-conditioned classifiers.The first-round estimator produces an appearance-based foreground prediction, while the conditional classifiers process features together with the semantic prior.
- Semantic selection and propagation: The method uses external instance segmentation outputs from MNC, FCIS, or MaskRCNN to estimate semantic information and select masks consistent across the video.The instance segmentation output supplies binary masks, categories, and confidence scores; the method searches for a semantically consistent subset.
- Semantic selection and propagation: Semantic guidance estimates the object's category and instance structure, then propagates the selected semantic prior from the first frame to subsequent frames.Selection uses the given first-frame ground-truth mask; propagation filters instance masks with the first-round OSVOS foreground estimate and selects matching instances.
- Conditional classifier: Two conditional classifiers separately emphasize pixels inside and outside the semantic instance mask, and their weighted predictions are fused into the final output.The conditional formulation incorporates the semantic prior while preserving per-pixel classification for refinement beyond imperfect instance masks.
- Conditional classifier: Unlike feature masking, the conditional classifier makes softer semantic-guided decisions that can recover regions omitted by the semantic prior.The paper gives the dancer's left hand as an example of a region that feature masking would immediately classify as background.
5 EXPERIMENTAL VALIDATION
Experiments show that OSVOSS improves segmentation accuracy, temporal stability, robustness, and speed across datasets and operating regimes. Its semantic propagation reduces false negatives and maintains quality despite appearance changes, while frame-independent processing supports efficient deployment.
- Ablation Study: 12% performance gain follows parent-network pretraining, while adding semantics especially improves temporal stability.The ablation starts from 18.9% with ImageNet weights alone; parent pretraining adds 12%, and semantics substantially improve regions, contours, and temporal stability.
- Semantic Selection and Propagation: Automatic semantic instance selection approaches oracle quality across MNC, FCIS, and MaskRCNN, while the final model surpasses oracle selection.The final result shows that instance segmentation alone is insufficient and that the method is robust to the choice of instance segmentation algorithm.
- DAVIS 2016: OSVOSS is one point above the second-best semi-supervised method and 12.6 points above the best unsupervised method.It is also only 0.3 points below the strong superpixel-based oracle bound.
- DAVIS 2016: OSVOSS exceeds most methods on most DAVIS 2016 sequences, remains above 70% on all but one, and above 80% on all but three.The approach is also robust across challenging attributes such as camera shake and occlusions.
- Temporal Stability: Semantic information reduces false negatives and stabilizes performance as appearance changes, particularly later in sequences.The method processes frames independently, avoiding sequential error propagation and retaining quality throughout the sequence.
- Speed–Accuracy Trade-off: 75.1% accuracy is achieved at 300 milliseconds per frame, while 86.5% is reached with 4.5 seconds of processing per 480×854 image.Across speed regimes, the method is reported as faster and/or more accurate, including 2.5 versus 12 seconds at comparable high quality.
6 CONCLUSIONS
OSVOSS processes video frames independently, using one-shot appearance learning and an instance-level semantic prior to segment objects without temporal information. It achieves robustness to disruptions such as occlusions and lost frames while maintaining state-of-the-art accuracy and speed.
- OSVOSS is a semi-supervised method that processes each video frame independently, ignoring temporal information and redundancy.
- 75.1 versus 59.4 at 300 miliseconds per frame, while the best performance reaches 86.5 versus 85.5 with 4.5 versus 12 seconds per frame.
- The method builds a powerful appearance model from a single segmented frame and fine-tunes a pretrained network on merely one training sample.
- An instance segmentation algorithm supplies a semantic prior that guides the first-frame appearance model and improves robustness to object appearance changes over longer videos.
- The appearance model and semantic prior are combined through a trainable conditional classifier module within a CNN.