Source-linked AI summary
Video Object Segmentation with Episodic Graph Memory Networks
Xiankai Lu, Wenguan Wang, Martin Danelljan, Tianfei Zhou, Jianbing Shen, Luc Van Gool
TL;DR
Video object segmentation needs efficient adaptation to specific videos and changing target appearances. The paper introduces a graph memory network that learns to update segmentation through fixed-size episodic graph memory, and reports favorable results across one-shot and zero-shot benchmarks.
Problem
Existing matching-based VOS methods do not fully exploit target-specific information, support-set context, or online updating for appearance variation.
Method
The model uses a fixed-size, fully connected episodic graph memory to store and reason over target-specific representations for updating segmentation.
Results
The method performs favorably against state-of-the-art O-VOS methods and outperforms competitors on Z-VOS datasets.
Takeaways & Limitations
Learnable memory controllers enable online model updating without extensive parameter optimization, while the framework generalizes to zero-shot segmentation.
Abstract
from arXiv · showhide
How to make a segmentation model efficiently adapt to a specific video and to online target appearance variations are fundamentally crucial issues in the field of video object segmentation. In this work, a graph memory network is developed to address the novel idea of "learning to update the segmentation model". Specifically, we exploit an episodic memory network, organized as a fully connected graph, to store frames as nodes and capture cross-frame correlations by edges. Further, learnable controllers are embedded to ease memory reading and writing, as well as maintain a fixed memory scale. The structured, external memory design enables our model to comprehensively mine and quickly store new knowledge, even with limited visual information, and the differentiable memory controllers slowly learn an abstract method for storing useful representations in the memory and how to later use these representations for prediction, via gradient descent. In addition, the proposed graph memory network yields a neat yet principled framework, which can generalize well both one-shot and zero-shot video object segmentation tasks. Extensive experiments on four challenging benchmark datasets verify that our graph memory network is able to facilitate the adaptation of the segmentation network for case-by-case video object segmentation.
1 Introduction
Video object segmentation must adapt efficiently to video-specific targets and changing appearances. The proposed graph memory network learns to update segmentation through external episodic memory, supporting both one-shot and zero-shot settings.
- O-VOS identifies annotated objects from the first frame in subsequent video frames, while Z-VOS infers primary objects without test-time first-frame annotations.
- Earlier O-VOS methods finetune the network for each annotated object, whereas matching-based methods directly assign query labels using pixel-wise similarity.Finetuning is effective but time-consuming.
- Matching-based methods use generic matching networks, cannot fully exploit target-specific first-frame information, and neglect context among support frames.They also lack efficient adaptation to the input video and online updating for appearance variation.
- The graph memory network treats O-VOS as episodic memory reasoning, rapidly fusing first-frame information while learning how to cache and recall target representations.Memory updating supports online adaptation without extensive parameter optimization.
- The fixed-size graph memory performs message passing without increasing memory consumption and provides one framework for O-VOS and Z-VOS.Experiments report favorable performance against state-of-the-art methods on O-VOS and stronger results than competitors on Z-VOS.
2 Related Work
One-shot video object segmentation tracks first-frame annotations through subsequent frames at pixel level. Related methods include segmentation-by-detection, mask propagation, and matching-based strategies.
- O-VOS tracks the first-frame annotation to subsequent frames at pixel level.
- Segmentation-by-detection learns a video-specific representation of annotated objects before pixel-wise detection in later frames.
- Propagation-based pipelines propagate segmented masks to fit objects in upcoming frames.
- Matching-based strategies use prototypical Siamese networks to match pixels or feature embeddings between reference and query frames for label assignment.Some matching-based methods employ internal or external memory.
3 Proposed Algorithm
The proposed model treats video object segmentation as episodic graph-memory reasoning: support frames initialize a fixed fully connected memory, while learnable controllers read, update, and write representations for query-frame prediction. Iterative reasoning mines support context, incorporates query information, and produces segmentation masks without increasing memory size.
- Prediction and Memory Efficiency: After K reasoning steps, the decoder predicts the segmentation mask from the final episodic feature and query embedding.Graph updating embeds neighbor information and progressively incorporates new query information without increasing memory size or requiring a separate memory-writing strategy.
- Graph Memory Initialization: The graph memory is a size-fixed, fully connected graph whose nodes store support-frame representations and whose edges encode relations between memory cells.The memory is initialized from N sampled support frames, with embeddings generated from support frames and segmentation-mask information.
- Graph Memory Reading: A query encoder extracts visual features, and a learnable read controller initializes its state before repeatedly retrieving relevant content from the external graph memory.At each reasoning step, query–memory-node similarity determines the read weights used for memory summarization.
- Graph Memory Reading: The read controller combines query and memory information into a hidden state that encodes the representations needed for segmentation prediction.The controller uses gated updates, with the update gate controlling how much of the previous hidden state is retained.
- Graph Memory Updating: A learnable write controller updates every memory cell using its prior state, current read-controller content, and information aggregated from neighboring cells.Neighbor relations are computed from feature-matrix similarities, normalized, and used to propagate contextual information across the graph.
- Full Network Architecture: The end-to-end fully convolutional implementation uses ConvGRU controllers, ResNet50 encoders, a skip-connected decoder, and shuffled instance labels during training.Label shuffling discourages memorizing fixed relations between target instances and one-hot labels.
4 Experiments
Experiments evaluate the graph memory network across one-shot and zero-shot video object segmentation on four challenging datasets. The method achieves strong benchmark performance, handles difficult appearance changes, and benefits from graph-memory updating and iterative reasoning.
- Experimental Setup: The model is evaluated on two O-VOS and two Z-VOS datasets using region, boundary, stability, and aggregate metrics.O-VOS experiments use DAVIS17 and Youtube-VOS; Z-VOS experiments use DAVIS16 and Youtube-Objects.
- O-VOS Results: 82.8% mean J &F, 80.2% mean J, and 85.2% mean F establish state-of-the-art performance on DAVIS17.The evaluation also reports speed alongside region similarity, contour accuracy, and their average.
- O-VOS Results: 80.2% overall on Youtube-VOS outperforms S2S at 64.4% and performs favorably on both seen and unseen categories.The method improves over time-consuming online-learning baselines without invoking online finetuning.
- Z-VOS Results: 71.4% mean J on Youtube-Objects surpasses competing methods, while qualitative results show robustness to fast motion, occlusion, and view changes.The Z-VOS evaluation also includes DAVIS16 with region similarity, boundary accuracy, and time stability metrics.
- Diagnostic Experiments: The graph memory network improves mean J and mean F over the backbone by +6.5%, +7.5%, +9.3%, and +8.7% across VOS settings.A 3-node memory is sufficient for good performance, and iterative updates improve results through K = 3 before performance nearly saturates at K = 4.
5 Conclusion
The paper concludes that graph memory enables efficient adaptation of segmentation networks to specific videos without catastrophic inference or finetuning. Episodic reasoning and learnable memory controllers produce video-specific summaries, and experiments across four datasets support the method’s performance and extension to Z-VOS.
- Graph memory efficiently adapts segmentation networks to specific videos without catastrophic inference or finetuning.
- Episodic reasoning generates video-specific memory summaries that benefit final segmentation prediction, while learnable controllers implement online model updating.
- Experiments on four challenging datasets demonstrate promising performance and support extending the framework to zero-shot video object segmentation.