Source-linked AI summary
Video Object Segmentation with Joint Re-identification and Attention-Aware Mask Propagation
Xiaoxiao Li, Chen Change Loy
TL;DR
Multiple-instance video object segmentation must handle occlusion, appearance changes, and distractors while maintaining temporal continuity and recovering missing targets. DyeNet unifies re-identification with attention-based recurrent mask propagation in an end-to-end framework. It achieves a 68.2 global mean on DAVIS 2017 test-dev, compared with 66.1 for the winning solution.
Problem
Multiple-instance video object segmentation is challenged by occlusion, scale and pose variation, and distractors that can disrupt tracking and propagation.
Method
DyeNet combines iterative template-expanding re-identification with bidirectional attention-based recurrent mask propagation in one end-to-end network.
Results
68.2 global mean on DAVIS 2017 test-dev outperforms the winning solution's 66.1 on the same partition.
Takeaways & Limitations
Template expansion reduces reliance on first-frame masks, while attention-based propagation improves resilience to distractors during multi-instance segmentation.
Takeaways & Limitations
Temporal-continuity approaches remain prone to tracking errors under severe occlusion and distractors, motivating the proposed re-identification and attention mechanisms.
Abstract
from arXiv · showhide
The problem of video object segmentation can become extremely challenging when multiple instances co-exist. While each instance may exhibit large scale and pose variations, the problem is compounded when instances occlude each other causing failures in tracking. In this study, we formulate a deep recurrent network that is capable of segmenting and tracking objects in video simultaneously by their temporal continuity, yet able to re-identify them when they re-appear after a prolonged occlusion. We combine both temporal propagation and re-identification functionalities into a single framework that can be trained end-to-end. In particular, we present a re-identification module with template expansion to retrieve missing objects despite their large appearance changes. In addition, we contribute a new attention-based recurrent mask propagation approach that is robust to distractors not belonging to the target segment. Our approach achieves a new state-of-the-art global mean (Region Jaccard and Boundary F measure) of 68.2 on the challenging DAVIS 2017 benchmark (test-dev set), outperforming the winning solution which achieves a global mean of 66.1 on the same partition.
1. Introduction
Video object segmentation with multiple instances must handle occlusion, scale and pose variation, while preserving temporal continuity and recovering targets after disappearance. DyeNet unifies re-identification and bidirectional recurrent mask propagation, achieving 68.2 global mean on DAVIS 2017 test-dev versus 66.1 for the challenge winner.
- Multiple instances create occlusions and substantial scale and pose variations that make video object segmentation difficult.
- Fixed first-frame templates may not sufficiently cover target pose and scale variations.
- Temporal-continuity methods can lose targets after occlusion and be distracted by backgrounds or other object segments.
- DyeNet combines re-identification and recurrent mask propagation to establish starting points, retrieve missing segments, and propagate masks bidirectionally through the video.
- Iterative template expansion retrieves reappearing objects across different poses and scales, while attention-based propagation disregards distractors.
- 68.2 global mean on DAVIS 2017 test-dev exceeds the winning solution's 66.1 on the same partition.
2. Related Work
Related work addresses video object segmentation through generic representations, template-based re-identification, and temporal continuity. DyeNet is positioned against these approaches through template expansion and a unified recurrent framework.
- Video object segmentation differs from semantic and instance segmentation because object classes are undefined and masks must be propagated object-agnostically.
- Prior work exploits temporal information to establish spatiotemporal correlations across video frames.
- VS-ReID uses a generic object re-identification model with two-stream mask propagation but relies only on first-frame templates, making it more susceptible to pose variations.
3. Methodology
DyeNet unifies re-identification and recurrent mask propagation in an iterative network that retrieves missing instances and propagates masks bidirectionally. Template expansion, flow-guided recurrent propagation, and region attention address appearance variation, occlusion, and distractors.
- Overall architecture: DyeNet combines a re-identification module with recurrent mask propagation in one end-to-end network.Re-ID establishes starting points and retrieves missing segments, while Re-MP propagates their masks.
- Re-identification: Iterative inference expands the template set with high-confidence predictions, enabling retrieval of missing objects across changing poses and scales.Object proposals at multiple anchor sizes support scale variation, while newly selected templates reduce reliance on first-frame masks.
- Re-identification: The Re-ID module compares proposal-derived masks with templates and retains candidates whose cosine similarity exceeds a high threshold as propagation starting points.Candidate features are resized with RoIAlign, then processed by mask and re-identification sub-networks.
- Recurrent mask propagation: Re-MP propagates retrieved masks bidirectionally into tracklets using an RNN that combines current-region features with flow-warped historical memory.Optical flow warps both the preceding mask for localization and the hidden state for spatial consistency.
- Recurrent mask propagation: An attention distribution derived from the warped hidden state reweights the current hidden state so propagation concentrates on the tracked object.The enhanced hidden state produces the next mask, and propagation aborts when the object becomes too small.
- Tracklet linking: A greedy linking procedure orders tracklets by cosine similarity between their starting points and templates, then assigns and merges them into consistent mask tubes.
4. Experiments
Experiments evaluate DyeNet through component ablations, attribute-based analysis, benchmark comparisons, and speed measurements. Results show complementary benefits from attention-aware propagation, template-expanding re-identification, and online-training choices.
- Ablation Study: Template expansion increases G-mean across Re-ID iterations, with the best performance at ρreid = 0.7.Lower thresholds retrieve more instances early but introduce imprecise starting points, whereas stricter thresholds improve recall as templates expand.
- Benchmark: 68.2 G-mean is achieved by full DyeNet on DAVIS17 test-dev, while the offline model reaches 62.5.The benchmark reports use online fine-tuning for the 68.2 result and identify 62.5 as the competitive result without online training.
- Ablation Study: Re-MP tracks small objects and avoids distractors during partial occlusion, while Re-ID retrieves instances lost under heavy occlusion.Template expansion further supports re-identification under large pose variations.
- Attribute Analysis: Object size and occlusion affect performance most, and scale variation has more influence than pose variation.The analysis categorizes instances by size, scale variation, occlusion, and pose variation.
- Benchmark: DyeNet achieves state-of-the-art performance across DAVIS16, DAVIS17, SegTrackv2, and YouTubeObjects.The model trained on DAVIS17 also performs strongly on SegTrackv2 and YouTubeObjects without fine-tuning.
- Speed Analysis: 84.7 mIoU is obtained offline at 2.4 FPS, compared with 86.2 mIoU at 0.43 FPS after 2k per-dataset online-training iterations.The speed analysis links online training with higher accuracy and slower inference.
5. Conclusion
DyeNet unifies re-identification with attention-based recurrent temporal propagation for challenging multi-instance video object segmentation. It uses template expansion and attention to address pose variation, occlusion, distractors, and background clutter.
- DyeNet unifies re-identification and attention-based recurrent temporal propagation for multi-instance video object segmentation.
- Template expansion reduces reliance on first-frame templates to handle target pose variations.
- Attention-based recurrent temporal propagation improves robustness against distractors and background clutter.
- DyeNet requires no online training to achieve competitive accuracy and can operate faster than many existing methods.
- With online training, DyeNet achieves state-of-the-art performance across standard benchmarks including DAVIS, SegTrackv2, and YouTubeObjects.