Source-linked AI summary
Efficient Regional Memory Network for Video Object Segmentation
Haozhe Xie, Hongxun Yao, Shangchen Zhou, Shengping Zhang, Wenxiu Sun
TL;DR
Semi-supervised VOS methods based on global-to-global memory matching can confuse similar objects and incur high computational cost. RMNet instead performs local-to-local matching over tracked target regions, and experiments on DAVIS and YouTube-VOS report favorable performance against state-of-the-art methods with faster running speed.
Problem
Global-to-global matching in Space-Time Memory-based semi-supervised VOS can mismatch similar objects and has high computational complexity.
Method
RMNet memorizes target-containing regions, tracks current query regions using optical-flow-warped masks, and matches memory and query regions locally.
Results
RMNet outperforms state-of-the-art methods on DAVIS and YouTube-VOS with much faster running speed.
Takeaways & Limitations
Regional Memory Reader reduces ambiguity from similar objects while making feature matching more computationally efficient.
Abstract
from arXiv · showhide
Recently, several Space-Time Memory based networks have shown that the object cues (e.g. video frames as well as the segmented object masks) from the past frames are useful for segmenting objects in the current frame. However, these methods exploit the information from the memory by global-to-global matching between the current and past frames, which lead to mismatching to similar objects and high computational complexity. To address these problems, we propose a novel local-to-local matching solution for semi-supervised VOS, namely Regional Memory Network (RMNet). In RMNet, the precise regional memory is constructed by memorizing local regions where the target objects appear in the past frames. For the current query frame, the query regions are tracked and predicted based on the optical flow estimated from the previous frame. The proposed local-to-local matching effectively alleviates the ambiguity of similar objects in both memory and query frames, which allows the information to be passed from the regional memory to the query region efficiently and effectively. Experimental results indicate that the proposed RMNet performs favorably against state-of-the-art methods on the DAVIS and YouTube-VOS datasets.
1. Introduction
Semi-supervised VOS uses first-frame object masks but remains difficult under deformation, occlusion, appearance changes, and confusion among similar objects. RMNet addresses global matching errors by restricting memory and query processing to target-object regions.
- Semi-supervised VOS requires target-object masks in the first frame, yet remains challenging under deformation, occlusion, appearance variation, and similar-object confusion.
- Global-to-global matching can associate current targets with wrong past objects or past targets with wrong current objects because it also matches background regions.
- RMNet memorizes features only in past-frame target regions and uses optical-flow-warped previous masks to estimate target regions in the current frame.
- Regional Memory Reader performs local-to-local matching between target-containing regions, reducing similar-object ambiguity and computational complexity.
- RMNet outperforms state-of-the-art methods on DAVIS and YouTube-VOS while running substantially faster.
2. Related Work
Prior propagation-based methods addressed video object segmentation through temporal label propagation, while RMNet uses regional memory and query embeddings with local-to-local matching around target objects.
- Propagation-based Methods: Propagation-based methods formulate video object segmentation as a temporal label propagation problem.
- Propagation-based Methods: ObjectFlow, SegFlow, and DVSNet jointly consider video segmentation and optical flow estimation.
- RMNet extracts target-object regions for regional memory and query embeddings, then performs efficient local-to-local matching to reduce similar-object mismatches.
3. Regional Memory Network
RMNet replaces global-to-global memory reading with regional embeddings and local-to-local matching for target-object regions. Optical-flow tracking supplies query regions, while occlusion handling expands matching when objects disappear; the regional reader also reduces computational cost.
- 3. Regional Memory Network: RMNet embeds only target-object regions in memory and query frames, unlike STM’s global embeddings from all regions.Regional attention maps are applied to feature embeddings to construct the memory and query representations.
- 3.1.1 Regional Memory Embedding: Regional memory keys and values are formed by applying regional attention maps to global memory features for the target object.The attention maps are derived from object masks and retain features within the relevant memory regions.
- 3.1.2 Regional Query Embedding: Optical flow warps the previous object mask to predict the current query region, while a low-pixel condition triggers global matching when the object disappears.The matching region expands to the whole image during disappearance and shrinks after reappearance, improving robustness to occlusion.
- 3.2. Regional Memory Reader: Local-to-local matching between regional memory and query embeddings alleviates mismatching to similar objects in both memory and query frames.The reader restricts correspondence to regions containing target objects rather than comparing all image locations.
4. Experiments
RMNet is evaluated on DAVIS and YouTube-VOS using region- and contour-based metrics, with experiments covering datasets, implementation, benchmark comparisons, and ablations of regional matching and query-region prediction.
- Evaluation and Setup: The experiments use region similarity J and contour accuracy F, and report training and evaluation settings for DAVIS and YouTube-VOS.The implementation uses two-phase training, Adam optimization, and evaluation on DAVIS and YouTube-VOS datasets.
- Video Object Segmentation: On DAVIS 2016 single-object segmentation, RMNet is comparable to competitive methods but has faster inference speed.The evaluation uses the DAVIS 2016 validation set, with some results trained using additional YouTube-VOS data.
- Video Object Segmentation: RMNet outperforms competitive methods on DAVIS 2017 validation and test-dev, while achieving a 0.815 average score on YouTube-VOS validation.With additional YouTube-VOS training data, RMNet also achieves better DAVIS 2017 validation accuracy and outperforms state-of-the-art methods.
- Ablation Study: Local-to-local matching is about 5 times faster and around 25 times smaller in FLOPS than global-to-global matching.Target-object regions usually occupy less than 20% of the image, and the regional reader avoids mismatching outside target regions.
- Ablation Study: Flow-based query-region prediction outperforms previous-region and best-match-region alternatives for segmentation.Previous-region prediction is vulnerable to occlusion and drifting, while best-match regions can be affected by lighting and similar objects.
- Ablation Study: Replacing TinyFlowNet with FlowNet2-CSS or RAFT yields nearly identical segmentation accuracy, while TinyFlowNet is 6 and 16 times faster, respectively.The comparison evaluates optical-flow estimators for query-region prediction.
5. Conclusion
RMNet memorizes and tracks target-object regions to reduce similar-object ambiguity and feature-matching complexity, outperforming state-of-the-art methods with faster running speed on DAVIS and YouTube-VOS.
- RMNet memorizes and tracks regions containing target objects, reducing similar-object ambiguity and feature-matching computational complexity.