Source-linked AI summary
Blazingly Fast Video Object Segmentation with Pixel-Wise Metric Learning
Yuhua Chen, Jordi Pont-Tuset, Alberto Montes, Luc Van Gool
TL;DR
Video object segmentation must combine accurate object tracking with practical processing speed from user annotations. The paper learns a pixel-wise embedding for retrieval-based label transfer, supporting masks and sparse interactions, and reports competitive quality with fast semi-supervised inference and low-interaction interactive segmentation.
Problem
Video object segmentation lacks a method that simultaneously provides state-of-the-art-comparable accuracy and acceptable processing speed.
Method
A fully convolutional network learns an embedding with a modified triplet loss, then transfers labels from annotated reference pixels through nearest-neighbor retrieval.
Results
275 milliseconds per frame and J &F=77.5% on DAVIS 2016 are reported for semi-supervised segmentation, while 10 clicks achieve 74.5% accuracy interactively.
Takeaways & Limitations
The method offers a unified, fast framework for mask- and interaction-based video object segmentation, with embedding computation independent of changing user input.
Takeaways & Limitations
The experiments limit interaction to clicks, although the system also supports scribbles and other input types that are harder to evaluate.
Abstract
from arXiv · showhide
This paper tackles the problem of video object segmentation, given some user annotation which indicates the object of interest. The problem is formulated as pixel-wise retrieval in a learned embedding space: we embed pixels of the same object instance into the vicinity of each other, using a fully convolutional network trained by a modified triplet loss as the embedding model. Then the annotated pixels are set as reference and the rest of the pixels are classified using a nearest-neighbor approach. The proposed method supports different kinds of user input such as segmentation mask in the first frame (semi-supervised scenario), or a sparse set of clicked points (interactive scenario). In the semi-supervised scenario, we achieve results competitive with the state of the art but at a fraction of computation cost (275 milliseconds per frame). In the interactive scenario where the user is able to refine their input iteratively, the proposed method provides instant response to each input, and reaches comparable quality to competing methods with much less interaction.
1. Introduction
The paper targets video object segmentation that is both accurate and fast by casting it as pixel-wise retrieval in a learned embedding space. Its unified framework supports varied user annotations and achieves competitive accuracy with substantially lower computation.
- Motivation: Video object segmentation remains difficult to perform at satisfactory quality and acceptable speed despite its applications.The paper identifies this accuracy–efficiency gap as its main objective.
- Approach: The method learns an embedding space where pixels from the same object instance are close and retrieves labels for other pixels by nearest-neighbor search.A fully convolutional network is trained with a modified triplet loss tailored to video object segmentation.
- Efficiency: A single forward pass and nearest-neighbor search process each frame without test-time fine-tuning.This design makes inference highly efficient.
- Interaction: The framework supports clicked points, scribbles, and segmentation masks, while embedding vectors remain unchanged when user input changes.This supports iterative refinement with immediate feedback after each click.
- Results: 275 milliseconds per frame and J &F=77.5% on DAVIS 2016 provide the reported semi-supervised speed–accuracy trade-off.The paper reports that better-performing algorithms start at 8 seconds per frame, while similarly fast methods reach only 60% accuracy.
2. Related Work
Prior video object segmentation methods address semi-supervised, unsupervised, and interactive settings, but often rely on expensive processing or temporal inputs. This work emphasizes efficient segmentation through a pixel-wise embedding-and-retrieval formulation that supports rapid interaction.
- Video Object Segmentation: Semi-supervised methods use a first-frame segmentation mask, whereas unsupervised methods use only video to identify salient objects.The related work distinguishes these categories by their supervision and input requirements.
- Motivation: The paper focuses on real-world efficiency by avoiding test-time network fine-tuning, optical flow, and previous-frame inputs.These choices contrast with techniques that use deep learning together with online adaptation or temporal inputs.
- Interactive Video Object Segmentation: Interactive methods rely on iterative user input, but prior approaches can require preprocessing and interaction delays ranging from tens of seconds to about an hour.Video Cutout is described as requiring preprocessing plus post-processing on the order of an hour.
- Interactive Video Object Segmentation: 275 milliseconds per frame is reported as the preprocessing time, with almost immediate response to user interaction.The authors present this speed as suitable for real-world use.
- Deep Metric Learning: The proposed formulation learns pixel-level similarity and assigns each pixel the label of its most similar reference pixel in embedding space.The paper positions this as a pixel-wise retrieval formulation related to image retrieval.
3. Proposed Method
The method segments video objects by retrieving nearest reference pixels in a learned embedding space, with user input separated from network computation. Its formulation supports multiple annotation modes and objects while enabling rapid inference and online adaptation to appearance change.
- Pixel-wise Retrieval: Video object segmentation is performed in two stages: embed every pixel, then transfer labels from nearest reference pixels.The embedding network produces pixel representations, and per-pixel retrieval assigns labels according to the nearest reference pixel.
- User Interaction: User input is disentangled from network computation, so changing annotations requires only a fast nearest-neighbor search after one forward pass.This avoids test-time fine-tuning and avoids recomputing embeddings when user input changes.
- Embedding Model: The fully convolutional embedding model incorporates pixel coordinates and frame number to encode spatial and temporal information.The embedding is represented as e_j,i = f(x_j,i, i, j), counteracting the loss of such information from convolutional translation invariance.
- Online Adaptation: Online reference-pool updates address appearance changes by adding samples whose k = 5 nearest neighbors agree on the label.The paper interprets appearance change as distribution shift in the embedding space and updates reference samples accordingly.
- Generalization: The framework accepts clicked points, scribbles, or other labeled pixel sets, without requiring exhaustive first-frame annotations, and supports multiple objects.Object labels can identify one of K objects, generalizing the method beyond single-object segmentation.
- Training: The modified triplet loss avoids forcing all positive pixels together, instead separating the closest negative beyond the closest positive with a margin.This design targets intra-object variation, where different object parts may have substantially different appearances.
4. Experimental Validation
Experiments on DAVIS 2016 and 2017 evaluate the method across semi-supervised and interactive settings, emphasizing quality–speed trade-offs and annotation efficiency. Ablations and qualitative results examine the contributions of the loss, online adaptation, spatio-temporal awareness, and robustness to challenging video conditions.
- Evaluation setup: The evaluation uses DAVIS 2016 validation videos and DAVIS 2017 sequences with multiple objects, comparing against recent video object segmentation methods.DAVIS 2016 contains 50 annotated videos, with 30 training and 20 validation videos; DAVIS 2017 extends this to 150 sequences.
- Semi-supervised VOS: PML is comparable to MSK in global J &F and especially competitive in boundary accuracy F, despite lacking an explicit refinement or smoothing step.The comparison uses the best-performing, slowest regime reported for each technique.
- Semi-supervised VOS: +18 points over BVS while processing frames 100 milliseconds faster, and only 5 points below OnAVOS while processing each frame 43× faster.Figure 5 compares J &F against mean processing time per frame at 480p resolution.
- Semi-supervised VOS: The method is close to the best performance on most DAVIS validation sequences, achieves the best result on some, and has a worst performance of 0.5.These per-sequence results indicate robustness across varied challenges and scenarios.
- Ablation study: The modified triplet loss outperforms alternative contrastive-loss and original-triplet-loss models in the ablation comparison.The alternative-loss models keep the other experimental settings unchanged.
- Ablation study: Combining online adaptation with spatial and temporal information achieves 75.5% overlap, an overall +3.5% improvement at nearly no extra cost.Online adaptation alone adds +1.2% in J, while spatial and temporal information adds +2.3% in J and +4.5% in F.
- Interactive VOS: Interactive segmentation reaches J = 80% with 0.55 clicks per frame and J = 83% with 2 clicks per frame in the simulated interaction study.The method matches the full-first-frame-mask result of J = 75.5% using only 0.15 clicks per frame, with variance 0.1 at 1 click per frame.
- Interactive VOS: Real users achieve J = 77.7% with 0.17 clicks per frame, equivalent to 11 clicks per video and approximately 24 seconds of annotation.This quality is comparable with most state-of-the-art semi-supervised methods at a fraction of annotation and running cost.
5. Conclusions
The method casts video object segmentation as pixel-wise retrieval in a learned embedding space, classifying pixels by nearest neighbors to annotated references. It achieves state-of-the-art-comparable quality with much lower computation and supports near-instant interactive refinement.
- The method learns an embedding space with a video-specific modified triplet loss, then classifies unannotated pixels by nearest-neighbor retrieval.Annotated pixels from clicks, scribbles, or masks serve as reference samples.
- The method matches state-of-the-art quality in semi-supervised segmentation while operating significantly faster.It processes frames using a single learned embedding and nearest-neighbor classification.
- User-input-independent embeddings enable almost instantaneous feedback during interactive segmentation.The method reaches semi-supervised quality with 0.15 clicks per frame and generalizes to multiple objects.