Source-linked AI summary
Kernelized Memory Network for Video Object Segmentation
Hongje Seong, Junhyuk Hyun, Euntai Kim
TL;DR
Semi-supervised VOS requires predicting target masks from a first-frame annotation, while STM’s non-local matching conflicts with VOS’s predominantly local structure. KMN kernelizes memory reading and uses Hide-and-Seek pre-training, achieving state-of-the-art benchmark performance.
Problem
STM applies non-local matching to a predominantly local VOS problem, allowing incorrect matches and overlooking prior target locations.
Method
KMN uses Gaussian-kernelized memory reading and Hide-and-Seek pre-training on static images to address localization, occlusion, and boundary challenges.
Results
KMN achieves state-of-the-art performance across DAVIS 2016, DAVIS 2017, and Youtube-VOS benchmarks, including online-learning methods.
Takeaways & Limitations
The kernel approach addresses a central memory-network problem in VOS, while Hide-and-Seek improves VOS pre-training effectiveness.
Abstract
from arXiv · showhide
Semi-supervised video object segmentation (VOS) is a task that involves predicting a target object in a video when the ground truth segmentation mask of the target object is given in the first frame. Recently, space-time memory networks (STM) have received significant attention as a promising solution for semi-supervised VOS. However, an important point is overlooked when applying STM to VOS. The solution (STM) is non-local, but the problem (VOS) is predominantly local. To solve the mismatch between STM and VOS, we propose a kernelized memory network (KMN). Before being trained on real videos, our KMN is pre-trained on static images, as in previous works. Unlike in previous works, we use the Hide-and-Seek strategy in pre-training to obtain the best possible results in handling occlusions and segment boundary extraction. The proposed KMN surpasses the state-of-the-art on standard benchmarks by a significant margin (+5% on DAVIS 2017 test-dev set). In addition, the runtime of KMN is 0.12 seconds per frame on the DAVIS 2016 validation set, and the KMN rarely requires extra computation, when compared with STM.
1 Introduction
Semi-supervised VOS predicts target-object masks after receiving the first-frame mask, but STM’s non-local matching mismatches the predominantly local nature of the task. KMN addresses this mismatch with Gaussian-kernelized memory and Hide-and-Seek pre-training.
- Semi-supervised VOS predicts target-object masks in subsequent video frames from a ground-truth mask provided in the first frame.
- STM uses non-local query-to-memory matching even though VOS targets usually appear near their previous memory locations.
- KMN uses a Gaussian kernel to reduce STM’s non-locality and improve memory-network effectiveness for VOS.
- Hide-and-Seek pre-training is intended to improve robustness to occlusion and refine object-segment boundaries using static images before real-video training.
- KMN significantly outperforms previous methods on DAVIS 2016, DAVIS 2017, and Youtube-VOS 2018, including online-learning approaches.
- KMN is developed to reduce STM non-locality and uses Hide-and-Seek to pre-train on static images.
2 Related Work
Related work covers semi-supervised VOS, memory-network matching, kernel soft argmax, Hide-and-Seek localization, and boundary-focused segmentation methods. KMN follows an STM-like architecture while introducing kernelized memory reading for VOS.
- Semi-supervised VOS predicts target objects throughout a video using target information supplied in the first frame.
- Memory networks use query, key, and value features to generate query-memory correlations and retrieve weighted memory values.
- Kernel soft argmax applies Gaussian kernels to memory-side search results to create a gradient-propagable argmax for semantic correspondence.
- Hide-and-Seek hides random object patches during training to force localization of less salient object parts and can improve VOS pre-training.
- Boundary-focused segmentation methods address inaccurate object boundaries through boundary-specific training or separate treatment of boundary and non-boundary pixels.
3 Kernelized Memory Network
KMN retains STM’s query-memory architecture but replaces its memory read with bidirectional matching and Gaussian-kernelized local retrieval. This design addresses ambiguity from one-way matching and the mismatch between STM’s non-local operation and predominantly local VOS.
- Architecture: KMN uses the current frame as query and past frames with predicted masks as memory, extracting key and value features for both.The memory mask is concatenated with RGB channels before key and value embedding.
- Architecture: The correlation map is computed from inner products between all query and memory key-feature combinations.The resulting map supports retrieval of highly matched memory pixels.
- Kernelized Memory Read: KMN’s main innovation is bidirectional memory reading: Query-to-Memory matching is combined with Memory-to-Query matching.STM uses only Query-to-Memory matching, allowing multiple query objects to match the same memory target.
- Kernelized Memory Read: For each memory position, KMN finds the best-matched query position and centers a 2D Gaussian kernel there.The kernel’s standard deviation is σ.
- Kernelized Memory Read: Gaussian kernels retrieve memory values locally, after which the retrieved values are concatenated with the query values and passed to the decoder.The decoder contains a residual block and two stacks of refinement modules.
4 Pre-training by Hide-and-Seek
KMN is pre-trained on synthetic videos generated from static images, with Hide-and-Seek added to simulate occlusion and improve boundary quality. This pre-training substantially improves VOS performance over earlier pre-training methods.
- Synthetic Video Pre-training: KMN is pre-trained on static image datasets with foreground masks before training on real videos.Synthetic videos are generated by applying random affine transforms to images and corresponding masks.
- Hide-and-Seek: Hide-and-Seek hides random object patches during pre-training to expose the model to occlusion-like conditions absent from affine-transform-only synthetic videos.The strategy is used to address the common VOS problem of target-object occlusion.
- Hide-and-Seek: Hide-and-Seek produces cleaner and more accurate object boundaries during pre-training.The method is motivated partly by inaccurate ground-truth masks near object boundaries in segmentation datasets.
- Results: Hide-and-Seek pre-training significantly improves VOS pre-training performance over previous methods.Its performance is reported as high as the full-training performance of some previous methods.
5 Experiments
Experiments evaluate KMN across DAVIS and Youtube-VOS, using static-image and video pre-training, benchmark comparisons, ablations, and qualitative analyses. Results support improvements from Hide-and-Seek pre-training and kernelized memory in difficult segmentation cases.
- Training Details: Training uses separate static-image pre-training and main VOS training phases, with augmentations, Hide-and-Seek masking, temporal frame sampling, dynamic memory, and pixel-level cross-entropy.Static-image pre-training generates three transformed frames and hides grid cells with probability increasing from 0 to 0.5; main training samples three temporally ordered video frames.
- Quantitative Results: Hide-and-Seek pre-training substantially improves static-image and simulated-video pre-training performance, reaching levels comparable to some prior full-training results.The paper attributes this to more accurate object boundaries produced during pre-training.
- Quantitative Results: KMN achieves state-of-the-art performance on DAVIS 2017 test-dev and Youtube-VOS 2018, including a reported significant improvement over previous studies.The DAVIS test-dev evaluation uses unavailable ground truths, while Youtube-VOS validation ground truths are also unavailable.
- Qualitative Results: Qualitative comparisons show accurate predictions under fast deformation, similar background objects, multiple similar objects, and severe occlusion.Comparisons are made against released state-of-the-art methods and STM, with DAVIS comparisons using no additional VOS training data in the main comparison.
- Analysis: Ablation experiments report performance improvements from the proposed approaches, while boundary visualizations show that Hide-and-Seek helps the network learn cleaner mask boundaries.Runtime measurements use a 1080Ti GPU system, and the Gaussian kernel is omitted during training because argmax errors cannot be backpropagated.
6 Conclusion
The work introduces a new memory read operation and static-image method for handling occlusion and obtaining accurate boundaries. Evaluations across three VOS benchmarks achieve state-of-the-art performance, including against online-learning methods.
- A new memory read operation and static-image method address occlusion handling and accurate boundary extraction.
- KMN achieves state-of-the-art performance on DAVIS 2016, DAVIS 2017, and Youtube-VOS benchmarks, including against online-learning methods.
- Ablation results show the kernel approach addresses the main problem of memory networks in VOS.
- Hide-and-Seek-based approaches show effectiveness for VOS and support reproducible improvements with potential relevance to other segmentation-related fields.