Source-linked AI summary
Video Object Segmentation using Space-Time Memory Networks
Seoung Wug Oh, Joon-Young Lee, Ning Xu, Seon Joo Kim
TL;DR
Semi-supervised video object segmentation must track objects despite appearance changes, occlusions, and limited guidance. The paper proposes a space-time memory network that adaptively reads mask information from past frames, achieving strong accuracy and speed across video segmentation benchmarks.
Problem
Semi-supervised video object segmentation estimates object masks after the first frame, despite drastic appearance changes, occlusions, and drift; using more frames offers richer guidance.
Method
A space-time memory network stores past frames and masks externally, densely matches them with the query frame, and adaptively updates memory with intermediate probability maps.
Results
The method achieves the best reported accuracy among fast methods without online learning and significantly outperforms competing methods on Youtube-VOS and DAVIS benchmarks.
Takeaways & Limitations
The framework is robust to occlusions and complex motions while combining high accuracy with fast inference, and may extend to other pixel-level estimation problems.
Takeaways & Limitations
Matching prior methods’ training and evaluation settings is not easy for a large-scale benchmark.
Abstract
from arXiv · showhide
We propose a novel solution for semi-supervised video object segmentation. By the nature of the problem, available cues (e.g. video frame(s) with object masks) become richer with the intermediate predictions. However, the existing methods are unable to fully exploit this rich source of information. We resolve the issue by leveraging memory networks and learn to read relevant information from all available sources. In our framework, the past frames with object masks form an external memory, and the current frame as the query is segmented using the mask information in the memory. Specifically, the query and the memory are densely matched in the feature space, covering all the space-time pixel locations in a feed-forward fashion. Contrast to the previous approaches, the abundant use of the guidance information allows us to better handle the challenges such as appearance changes and occlussions. We validate our method on the latest benchmark sets and achieved the state-of-the-art performance (overall score of 79.4 on Youtube-VOS val set, J of 88.7 and 79.2 on DAVIS 2016/2017 val set respectively) while having a fast runtime (0.16 second/frame on DAVIS 2016 val set).
1. Introduction
Semi-supervised video object segmentation estimates masks after receiving the first frame’s ground-truth mask, despite appearance changes, occlusions, and drifts. The paper proposes using multiple frames through a memory-network architecture for more accurate and efficient segmentation.
- Task and challenges: Video object segmentation separates foreground and background pixels across every frame of a video.In the semi-supervised setting, the first frame supplies the target mask and masks for later frames must be estimated.
- Task and challenges: Appearance changes, occlusions, and drifts make estimating the target mask throughout a video challenging.
- Motivation: Using more frames, potentially every frame, is a natural extension because two-frame guidance benefits video segmentation.
- Proposed approach: The proposed memory-network system computes spatio-temporal attention over pixels in multiple frames to classify each query pixel.Frames with masks form memory, while the current frame is processed as the query; new information can be added to memory without restricting its size.
- Proposed approach: The framework uses rich reference information to address appearance changes, occlusions, and drifts, while reporting improvements in speed and accuracy over existing methods.
2. Related Work
Prior video segmentation methods propagate masks, detect objects from the first frame, or combine both strategies, with online learning often improving accuracy at substantial computational cost. The proposed offline memory framework stores intermediate outputs and adaptively retrieves guidance during runtime.
- Existing approaches: Propagation methods refine masks toward target objects, while detection methods classify pixels using first-frame appearance or feature-space templates.
- Existing approaches: Hybrid methods combine first-frame visual guidance with previous-frame spatial priors, while other methods exploit longer-term video information.
- Online and offline learning: Online learning can improve accuracy but is computationally expensive, motivating offline methods that use guidance information at test time.
- Memory-network formulation: The proposed framework stores intermediate outputs in external memory instead of fixing the guidance frames, then adaptively selects information during runtime.
- Memory networks: Memory networks provide external memory that neural networks can write to and read from, typically separating addressing keys from returned value features.
- Memory-network formulation: In this framework, masked frames enter memory, the current frame is the query, and dynamic updates incorporate newly predicted masks.The paper states that this helps address appearance changes, occlusions, and error accumulation without online learning.
3. Space-Time Memory Networks (STM)
The space-time memory network encodes masked memory frames and an unmasked query frame into keys and values, densely matches them across space and time, and decodes retrieved information into the query mask. Training uses image-based pre-training, video fine-tuning, and dynamically updated probabilistic memory.
- Framework overview: Past frames with masks serve as memory, while the current unmasked frame serves as the query during sequential video processing.The first frame uses its ground-truth annotation; later memory masks may be estimated outputs.
- Space-time memory read: Keys address relevant memory values by dense matching of every query pixel against memory locations across the video’s spatio-temporal space.
- Key and value embedding: Dedicated encoders produce key and value maps for both memory and query frames, but only the memory encoder receives an object mask.The memory mask is a probability map, including softmax outputs for estimated masks.
- Key and value embedding: Memory key and value maps from multiple frames are stacked along the temporal dimension, producing 3D representations indexed by the number of memory frames.
- Space-time memory read: The memory read computes soft similarity weights, retrieves memory values by weighted summation, concatenates them with query values, and supplies the result to the decoder.
- Decoder: The decoder reconstructs the current frame’s object mask from the memory-read output through compression, residual processing, and progressive upscaling.
- Multi-object extension: For multiple objects, the model runs independently per object and merges the resulting probability maps using soft aggregation.
- Training: Training combines image-based pre-training, real-video fine-tuning, and dynamic memory updates using unthresholded foreground probability maps.The dynamic memory models uncertainty in previous estimates.
4. Evaluation
The method is evaluated on Youtube-VOS and DAVIS, including single- and multi-object settings, with comparisons emphasizing accuracy, runtime, training data, and qualitative robustness. It achieves strong performance without online learning and benefits substantially from additional Youtube-VOS training data.
- Youtube-VOS: On Youtube-VOS, the method significantly outperforms all compared methods across every evaluation metric.The validation set includes seen and unseen object categories, supporting evaluation of generalization performance.
- DAVIS-2016: The method achieves the best DAVIS-2016 accuracy among competing methods without online learning while maintaining a fraction of the runtime.It remains competitive with top-performing online-learning methods, and additional Youtube-VOS data increases performance by a large margin.
- DAVIS-2017: The method shows the best performance among fast, non-online-learning methods on DAVIS-2017 multi-object segmentation.With additional Youtube-VOS data, it largely outperforms previous state-of-the-art methods, including the 2018 DAVIS challenge winner.
- Training data analysis: Additional training data produces a large performance leap, indicating that DAVIS is too small to train a generalizable network without over-fitting.The passage also links this dataset-size issue to the weaker performance of online-learning methods on large-scale Youtube-VOS.
- Qualitative results: Qualitative examples sampled before and after occlusions show robustness to occlusions and complex motions.The examples come from challenging Youtube-VOS and DAVIS validation videos.
5. Analysis
The analysis shows that both training strategy and memory management materially affect performance. Pre-training improves generalization, while intermediate memories mainly help on difficult frames.
- Training Data: Pre-training only outperforms main-training only and all other methods on YouTube-VOS, but combining both strategies achieves maximum performance.The pre-training-only model does so without real video data.
- Training Data: 11.2 points: removing pre-training reduces the YouTube-VOS Overall score.The authors attribute this gap to insufficient video data and suggest diverse pre-training objects improve generalization and prevent over-fitting.
- Training Data: Training on YouTube-VOS transfers well to DAVIS and outperforms other methods, whereas training on DAVIS transfers poorly to YouTube-VOS.The authors relate the asymmetry to DAVIS being too small for broad generalization.
- Memory Management: First and previous frames provide state-of-the-art accuracy, while adding intermediate memories further improves extremely challenging cases.The final model saves an intermediate frame every 5 frames in addition to the first and previous frames.
- Memory Management: Additional memory frames produce similar accuracy on successful cases but a clear advantage below the 30th percentile of Jaccard scores.The largest gap appears between the 10th and 30th percentiles, indicating better handling of difficult scenes.
- Memory Analysis: The memory visualization shows that the read operation accurately matches corresponding pixels between the query and memory frames.Similarity scores are computed for query pixels and converted into normalized soft weights over memory frames.
6. Conclusion
The paper presents a space-time memory network for semi-supervised video object segmentation. It reports leading accuracy and speed, and identifies broader pixel-level estimation tasks as future applications.
- The paper introduces a space-time memory network for semi-supervised video object segmentation.
- The method performs best among existing methods in both accuracy and speed.
- Future applications include object tracking, interactive image/video segmentation, and inpainting.