Source-linked AI summary
FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation
Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, Liang-Chieh Chen
TL;DR
Video object segmentation methods can be slow, complex, or dependent on first-frame fine-tuning, limiting practical usability. FEELVOS uses pixel-wise embeddings with global first-frame and local previous-frame matching as internal guidance for an end-to-end segmentation network. It achieves a 71.5% J &F score on DAVIS 2017 validation without fine-tuning.
Problem
Many successful VOS methods rely on first-frame fine-tuning, high runtime, or extensive engineering, limiting their practical usability.
Method
FEELVOS uses semantic pixel-wise embeddings with global first-frame and local previous-frame matching as soft internal guidance for a convolutional segmentation network trained end-to-end.
Results
71.5% J &F mean score is achieved on the DAVIS 2017 validation dataset without fine-tuning, establishing a new state of the art for non-fine-tuning VOS.
Takeaways & Limitations
FEELVOS is presented as a fast and practical method for multi-object VOS that combines global first-frame and local previous-frame matching.
Takeaways & Limitations
Global matching distance maps can be noisy and contain false-positive small distances, so they are used as segmentation-head inputs rather than directly producing segmentations.
Abstract
from arXiv · showhide
Many of the recent successful methods for video object segmentation (VOS) are overly complicated, heavily rely on fine-tuning on the first frame, and/or are slow, and are hence of limited practical use. In this work, we propose FEELVOS as a simple and fast method which does not rely on fine-tuning. In order to segment a video, for each frame FEELVOS uses a semantic pixel-wise embedding together with a global and a local matching mechanism to transfer information from the first frame and from the previous frame of the video to the current frame. In contrast to previous work, our embedding is only used as an internal guidance of a convolutional network. Our novel dynamic segmentation head allows us to train the network, including the embedding, end-to-end for the multiple object segmentation task with a cross entropy loss. We achieve a new state of the art in video object segmentation without fine-tuning with a J&F measure of 71.5% on the DAVIS 2017 validation set. We make our code and models available at https://github.com/tensorflow/models/tree/master/research/feelvos.
1. Introduction
FEELVOS targets practical semi-supervised video object segmentation by combining simplicity, speed, end-to-end multi-object learning, and strong performance without first-frame fine-tuning. It uses embedding-based matching as internal guidance for a segmentation network and achieves a 71.5% J &F mean score on DAVIS 2017 validation.
- Motivation: Semi-supervised VOS estimates masks for given objects throughout a video from ground-truth masks provided only for the first frame.
- Motivation: Many successful VOS methods rely on first-frame fine-tuning, high runtimes, or extensive engineering that limits practical usability.PReMVOS, for example, uses four neural networks, fine-tuning, and merging, with a runtime of around 38 seconds per video frame.
- Design goals: FEELVOS is designed to use a single neural network, avoid simulated data and first-frame fine-tuning, run quickly, and train multi-object segmentation end-to-end.
- Approach: FEELVOS extracts pixel-wise embeddings, globally matches the current frame to the first frame, locally matches it to the previous frame, and feeds both distance maps to a dynamic segmentation head.The head combines these maps with backbone features and previous-frame predictions to produce the final segmentation.
- Results: 71.5% J &F mean score is achieved on the DAVIS 2017 validation dataset without fine-tuning, establishing a new state of the art for the stated setting.
2. Related Work
Related VOS methods trade off accuracy, speed, fine-tuning requirements, simplicity, or end-to-end multi-object handling. FEELVOS builds on embedding-based matching while aiming to improve results without first-frame fine-tuning.
- Fine-tuning methods: Many semi-supervised VOS methods use first-frame fine-tuning, including OSVOS, OnAVOS, OSVOS-S, and PReMVOS.These approaches achieve strong results but do not meet the fast-design goal because they rely on first-frame fine-tuning.
- Embedding-based methods: VideoMatch uses a soft matching layer that considers the closest k neighbors in the first-frame embedding space for each current-frame pixel.
- Methods without fine-tuning: OSMN, FAVOS, PML, and VideoMatch avoid fine-tuning and achieve high speed, but FEELVOS is reported to produce significantly better results.
- Multi-object handling: RGMP uses two shared encoder streams and refinement modules, but requires per-object network execution, heuristic merging, and an elaborate training procedure with synthetic data and backpropagation through time.FEELVOS is contrasted with RGMP as simpler, using no simulated data and a simpler training procedure.
3. Method
FEELVOS uses embedding-based global and local matching as soft guidance for a dynamic, end-to-end segmentation network. Its design supports efficient multi-object video segmentation by combining first-frame information, previous-frame predictions, and shared computation.
- Overview: FEELVOS extracts shared backbone features and embedding vectors, then computes object-specific matching cues for the current frame.The architecture uses DeepLabv3+ features with stride 4 and a separate embedding layer at the same stride.
- Global Matching: Global matching compares current-frame embeddings with first-frame pixels belonging to each object and assigns nearest-neighbor distances as soft object cues.The first-frame object sets provide the reference pixels, while background is handled as another object.
- Global Matching: Global distance maps can be noisy, so FEELVOS feeds them into a segmentation head rather than using them directly for final segmentation.The visualization contains false-positive small distances in the water, motivating refinement by the segmentation head.
- Local Previous Frame Matching: Local matching transfers information from the previous frame by restricting nearest-neighbor searches to a spatial window, reducing computation for small inter-frame motion.The local distance map can be computed efficiently using cross-correlation and was found more effective than global previous-frame matching.
- Dynamic Segmentation Head: The dynamic segmentation head is instantiated once per object, stacks one-dimensional object logits, applies softmax, and trains with cross entropy.Shared backbone computation lets the method scale to multiple objects and supports end-to-end training with a variable number of objects.
- Previous Frame Predictions: Previous-frame predictions provide both the masks used for local matching and an additional posterior-probability feature cue.If an object’s predicted previous-frame pixel set is empty, its matching distance is defined as 1.
4. Experiments
Experiments evaluate FEELVOS across multiple video object segmentation benchmarks, compare its accuracy and runtime with prior methods, and analyze the contribution of its matching components.
- Main Results: FEELVOS achieves a J &F score of 71.5% on the DAVIS 2017 validation set, establishing a new state of the art among non-fine-tuning methods.This is 4.8% higher than RGMP and 2.4% higher without YouTube-VOS training data.
- Main Results: FEELVOS achieves a J &F score of 57.8% on DAVIS 2017 test-dev, 4.9% higher than RGMP.Its runtime is almost independent of the number of objects, and runtime is nearly identical to RGMP on this set.
- Runtime: 0.51 seconds per frame: FEELVOS provides a strong speed/accuracy trade-off on DAVIS 2017.Figure 4 compares quality and timing across methods with and without fine-tuning.
- Additional Benchmarks: 81.7% J &F on DAVIS 2016 is comparable to RGMP’s 81.8%, while FEELVOS requires neither simulated training data nor multiple neural networks.Without simulated data, RGMP achieves only 68.8% J &F.
- Ablation Study: 69.1% J &F is obtained by the full ablation setup using first-frame global matching, previous-frame local matching, and previous-frame predictions.Replacing local with global previous-frame matching reduces performance to 64.2%, while disabling previous-frame matching reduces it to 54.9%.
- Ablation Study: Disabling previous-frame predictions reduces results to 52.6%, whereas retaining local matching without those predictions yields 63.3%.The comparison indicates that previous-frame local matching contributes more in this ablation than previous-frame predictions alone.
DAVIS 2017
FEELVOS benefits from combining first-frame global matching with previous-frame local matching, but qualitative results reveal failures when object appearances are similar or unseen initially.
- Combining first-frame global matching with previous-frame local matching is most effective because the two information-transfer mechanisms are complementary.
- Previous-frame local matching transfers information more effectively than using previous-frame predictions as features.
- Disabling first-frame global matching deteriorates the result by 13%, underscoring its importance for accurate segmentation.
- FEELVOS can lose track of some fish when many objects have similar appearances.
- FEELVOS initially misses parts of a cat's back when that texture is absent from the first frame, but later recovers from the error.
5. Conclusion
The paper concludes that FEELVOS addresses practical usability in video object segmentation through fast, simple, end-to-end embedding learning. Its global first-frame and local previous-frame matching components are experimentally effective, supporting strong results without fine-tuning.
- FEELVOS targets practical video object segmentation with a fast and simple design that achieves strong results.
- FEELVOS learns a semantic embedding for multi-object segmentation in an end-to-end way.
- Global matching to the first frame and local matching to the previous frame are the method's key components.
- Experiments show that each component is highly effective and that FEELVOS achieves state-of-the-art VOS results on DAVIS 2017 without fine-tuning.