Source-linked AI summary
Learning Video Object Segmentation with Visual Memory
Pavel Tokmakov, Karteek Alahari, Cordelia Schmid
TL;DR
The paper addresses unsupervised segmentation of moving objects in unconstrained videos without manually annotated test frames. It combines appearance and motion streams with bidirectional ConvGRU visual memory, achieving top performance on DAVIS and FBMS.
Problem
Video object segmentation must identify objects that move in at least one frame, but prior approaches depend on manual annotations, flow, or lack learned temporal memory.
Method
A two-stream network combines appearance and motion features with a bidirectional convolutional GRU that encodes visual memory over video frames.
Results
The method achieves top performance on the DAVIS and FBMS video object segmentation benchmarks.
Takeaways & Limitations
The visual memory framework supports segmentation when objects become static or motion prediction fails, without manually annotated input frames.
Takeaways & Limitations
Training on DAVIS alone biases ConvGRU toward uninterrupted motion, so stop-and-go robustness requires augmented training data.
Abstract
from arXiv · showhide
This paper addresses the task of segmenting moving objects in unconstrained videos. We introduce a novel two-stream neural network with an explicit memory module to achieve this. The two streams of the network encode spatial and temporal features in a video sequence respectively, while the memory module captures the evolution of objects over time. The module to build a "visual memory" in video, i.e., a joint representation of all the video frames, is realized with a convolutional recurrent unit learned from a small number of training video sequences. Given a video frame as input, our approach assigns each pixel an object or background label based on the learned spatio-temporal features as well as the "visual memory" specific to the video, acquired automatically without any manually-annotated frames. The visual memory is implemented with convolutional gated recurrent units, which allows to propagate spatial information over time. We evaluate our method extensively on two benchmarks, DAVIS and Freiburg-Berkeley motion segmentation datasets, and show state-of-the-art results. For example, our approach outperforms the top method on the DAVIS dataset by nearly 6%. We also provide an extensive ablative analysis to investigate the influence of each component in the proposed framework.
1. Introduction
The paper targets unsupervised video object segmentation with a two-stream network and explicit visual memory, addressing limitations of hand-crafted, flow-only, and frame-supervised approaches.
- Video object segmentation extracts spatio-temporal regions corresponding to objects moving in at least one video frame.
- Prior methods rely on hand-crafted features, manually segmented frames, or optical flow, limiting robustness when flow fails or objects stop moving.
- The proposed two-stream network combines appearance and temporal features with a convolutional GRU that encodes object evolution.
- For each frame, the network uses memory aggregated from previously seen frames to compute segmentation, then updates memory with the current features.
- The approach requires no manually annotated frames and reports state-of-the-art results on DAVIS and FBMS, alongside extensive ablation studies.
2. Related work
Related work spans trajectory-based, foreground-background, motion-based, and CNN-based segmentation, while this paper uses recurrent convolutional memory to model video over time.
- Trajectory-based methods segment videos using spatio-temporal motion features but lack robustness when feature matching fails.
- Foreground-background methods estimate object regions, build appearance models, and integrate cues such as saliency, shape, or pairwise constraints.
- Motion-segmentation methods use optical flow, but heuristic initialization and absent motion can reduce robustness or prevent object extraction.
- CNN-based video methods fine-tune image-segmentation networks on the first frame, requiring manual annotation and omitting temporal modeling.
- ConvGRU extends recurrent models with spatially structured states and is used here bidirectionally to encode object evolution across the video.
3. Approach
The approach combines appearance and motion networks with a ConvGRU visual memory that refines segmentation and retains object information across frames.
- The model takes video frames and estimated optical flow as input and outputs binary segmentations of moving objects.
- The appearance network encodes RGB-frame semantic content, while the motion network supplies motion cues for complementary object representations.
- The ConvGRU candidate hidden state is computed using two convolutional operations and a tanh nonlinearity.
- The ConvGRU memory module combines the two streams, refines initial estimates, and memorizes object appearance and location.
- Visual memory helps segment objects when they are static or when motion prediction fails.
4. Visual memory module
The visual memory module uses ConvGRU states to encode evolving spatio-temporal object patterns from two-stream features. Bidirectional processing aggregates information across the entire video and addresses stop-and-go motion, while training requires sequence-unrolling memory and augmented data.
- ConvGRU memory: ConvGRU updates a visual-memory state from the current two-stream representation and the previous state, using convolutional gates to characterize spatio-temporal patterns.The state, update gate, reset gate, and candidate memory are learned with convolutional operators and nonlinearities.
- ConvGRU memory: The update gate controls how much candidate memory enters the new state, while the reset gate controls how much previous state influences that candidate.A near-zero update gate ignores candidate memory; a near-zero reset gate makes the unit forget its previous state.
- Bidirectional processing: Bidirectional ConvGRU runs identical-weight instances forward and backward, concatenating their activations at each frame to aggregate information over the entire video.The resulting representation is passed through a 3×3 convolution.
- Bidirectional processing: Nearly 3%: bidirectional ConvGRU improves segmentation performance on the DAVIS dataset and helps correct motion-prediction errors.It is used during both training and testing, with especially prominent influence in cases involving later object motion.
- Training: Training memory grows with sequence length because backpropagation through time retains intermediate activations, making whole-network training infeasible under the described setup.A sequence of length n corresponds to a 6n-layer CNN for unidirectional ConvGRU or 12n for bidirectional ConvGRU during training.
- Training: DAVIS training contains motion in every frame, biasing ConvGRU toward uninterrupted motion; stop-and-go augmentation is used to improve robustness when objects stop moving.The augmentation duplicates the final five frames and removes motion input for those frames.
5. Experiments
Experiments on DAVIS, FBMS, and SegTrack-v2 evaluate the model’s components and compare it with prior methods. Results show strong benchmark performance, while revealing limitations on long static sequences and low-resolution videos.
- Datasets and evaluation: The study evaluates DAVIS for training and testing, FBMS and SegTrack-v2 for testing, and FT3D for training a model variant.DAVIS uses a 30/20 training/validation split, while FBMS contains multiple moving objects and annotations on only a subset of frames.
- Ablation study: Removing the appearance stream causes a 26.6% drop in performance relative to the full model using fc6 appearance features.Adding raw RGB frames improves performance by 14.8% over the motion-only model, while learned appearance representations improve it further.
- Ablation study: ImageNet-only appearance pretraining reduces performance by 6%, while removing the motion stream leaves performance 10.5% below the complete method.The motion-free variant still outperforms most motion-based DAVIS approaches but fails in more challenging cases.
- Comparison to the state-of-the-art: On DAVIS, the method outperforms MP-Net-Frame by 5.9% on IoU and improves temporal stability by 30.8%.It is also 20.1% better than the next-best method, while the full method with CRF is nearly 6% better than MP-Net-Frame in the MP-Net comparison.
- Comparison to the state-of-the-art: On FBMS, the method exceeds MP-Net-Frame by 15.6% on the test set and 14.2% on the training set in F-measure, but some methods achieve higher recall.The recall limitation is attributed to very long static sequences, while SegTrack-v2 performance reaches an IoU of 57.3 and is affected by low-resolution videos.
6. Conclusion
The paper combines appearance and motion with a bidirectional ConvGRU visual memory to encode object evolution and improve video motion segmentation. It reports top performance on DAVIS and FBMS.
- A bidirectional ConvGRU visual memory encodes the spatio-temporal evolution of objects and uses it to improve motion segmentation.
- The approach achieves top performance on the DAVIS and FBMS datasets.
- Figure 7 visualizes ConvGRU gate activations alongside motion-stream outputs and final segmentation results for DAVIS validation sequences.White denotes high activation in the grayscale heat maps.