Source-linked AI summary
Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion
Ho Kei Cheng, Yu-Wing Tai, Chi-Keung Tang
TL;DR
Interactive VOS seeks easier ways to segment objects across videos than full pixel-wise annotation, while existing methods couple interaction handling with propagation. MiVOS separates interaction-to-mask, propagation, and difference-aware fusion, preserving user corrections across frames and supporting varied interactions. The paper reports improved accuracy with fewer frame interactions, alongside a 4.8M-frame synthetic VOS dataset for future research.
Problem
Interactive VOS needs efficient video segmentation from lightweight interactions, but coupled methods restrict interaction types and make training difficult.
Method
MiVOS decouples interaction-to-mask, temporal propagation, and difference-aware fusion, using mask changes to preserve user intent during propagation.
Results
MiVOS achieves better performance with fewer frame interactions and generalizes across interaction types in qualitative and quantitative DAVIS evaluations.
Takeaways & Limitations
The modular design makes interactive VOS versatile across user inputs while retaining accuracy and efficiency.
Abstract
from arXiv · showhide
We present Modular interactive VOS (MiVOS) framework which decouples interaction-to-mask and mask propagation, allowing for higher generalizability and better performance. Trained separately, the interaction module converts user interactions to an object mask, which is then temporally propagated by our propagation module using a novel top-$k$ filtering strategy in reading the space-time memory. To effectively take the user's intent into account, a novel difference-aware module is proposed to learn how to properly fuse the masks before and after each interaction, which are aligned with the target frames by employing the space-time memory. We evaluate our method both qualitatively and quantitatively with different forms of user interactions (e.g., scribbles, clicks) on DAVIS to show that our method outperforms current state-of-the-art algorithms while requiring fewer frame interactions, with the additional advantage in generalizing to different types of user interactions. We contribute a large-scale synthetic VOS dataset with pixel-accurate segmentation of 4.8M frames to accompany our source codes to facilitate future research.
1. Introduction
Interactive VOS reduces the burden of video annotation by using iterative interactions, but tightly coupled interaction and propagation limit interaction flexibility and complicate training. MiVOS decouples these tasks while preserving user intent through difference-aware fusion, enabling broader interaction support and fewer interacted frames.
- Motivation: Interactive VOS uses scribbles or clicks that users can iteratively refine, avoiding the full first-frame annotation required by semi-supervised VOS.These interactions are easier to provide than specifying a complete pixel-wise mask.
- Problem: Current methods jointly perform interaction understanding and temporal propagation, which limits interaction forms and makes training difficult.Decoupled attempts had not reached state-of-the-art accuracy because user intent was inadequately preserved.
- Motivation: Users can focus on correcting one frame multiple times before sparse propagation, reducing interacted frames while targeting the same or better accuracy.Single-frame interaction is nearly instantaneous, whereas propagation is more time-consuming.
- Method: MiVOS decouples interaction-to-mask from propagation and models mask differences before and after each interaction to preserve user intent during propagation.The mask difference is presented as interaction-type independent, supporting different user inputs.
- Contributions: The framework contributes a lightweight top-k memory filtering scheme, a difference-aware fusion module, and a synthetic VOS dataset containing 4.8M frames.The contributions target propagation efficiency, faithful user-intent capture, reduced interaction, and future research support.
2. Related Works
Interactive VOS builds on semi-supervised propagation and interactive image segmentation, but existing deep feature-fusion methods tie propagation to particular user-input forms. MiVOS instead represents user intent through mask differences and swaps the interaction-to-mask component to generalize across interaction types.
- Existing VOS: Semi-supervised VOS segments an object throughout a video from a fully annotated first-frame mask, while interactive VOS uses user corrections such as scribbles or clicks.Interactive segmentation has also explored bounding boxes, clicks, and extreme points.
- Existing iVOS: Most deep interactive VOS methods fuse user interactions through interconnected encoders or interaction-feature memories tied to mask propagation.This coupling links the propagation process to the particular form of user input and makes training difficult.
- MiVOS: MiVOS represents user intent as the mask difference before and after an interaction round, rather than directly carrying raw interactions into propagation.The approach is designed to avoid ambiguity from interaction types while preserving the correction signal.
- Generalization: The modular design adapts to bounding boxes, clicks, extreme points, or other inputs by replacing the corresponding interaction-to-mask algorithm.This separates interaction handling from the propagation process.
3. Method
MiVOS separates interaction-to-mask, temporal propagation, and difference-aware fusion so users can refine one frame while preserving corrections across the video.
- Framework: MiVOS first converts user interactions into a mask, then independently propagates the corrected mask and fuses it with previous results.The interaction module provides instant feedback, while propagation is more time-consuming and operates independently of the previous round’s mask.
- Interaction-to-Mask: The interaction module supports interchangeable segmentation tools, including scribbles, clicks, freehand drawing, and local control.The evaluated Scribble-to-Mask network uses RGB, an existing mask, and positive/negative scribble maps, while the broader framework is not restricted to one interaction type.
- Interaction-to-Mask: Local control restricts interaction processing to a user-specified region, preventing global deterioration when only a small refinement is needed.The approach is intended for cases where the global shape is correct but a local area, such as the ears, needs adjustment.
- Temporal Propagation: Top-k filtering retains only the strongest memory affinities, removing noise that grows with memory size and improving robustness and computational efficiency.The filtering can be applied before softmax, reducing expensive exponential operations; it improves propagation performance and stability, especially over long temporal distances.
- Temporal Propagation: The propagation decoder combines memory-read features with query features to produce an upsampled object mask, achieving 11.2 FPS for multi-object propagation on a 2080Ti.The decoder uses skip-connections from the query encoder and processes multiple objects individually before soft aggregation.
- Difference-Aware Fusion: Difference-aware fusion models positive and negative mask changes, aligns them to target frames through space-time memory, and preserves user corrections during fusion.This addresses the failure of correction-agnostic linear blending, where user corrections can disappear between the interacted and target frames.
4. Dataset: BL30K
BL30K is introduced as a large-scale synthetic VOS dataset designed to provide pixel-accurate segmentation at a scale unavailable in existing datasets. Its rendered videos contain long, high-frame-rate sequences suitable for VOS, and experiments report improved real-world performance despite synthetic origins.
- Dataset motivation: BL30K addresses the trade-off between DAVIS quality and YouTubeVOS scale by providing large-scale, pixel-accurate synthetic VOS segmentations.The paper compares BL30K with DAVIS and YouTubeVOS, which respectively lack quantity or have moderate annotation quality.
- Dataset construction: 51,300 three-dimensional models from ShapeNet were animated and rendered into corresponding RGB images and segmentations.The dataset uses Blender with a two-pass rendering scheme.
- Dataset properties: Each BL30K video contains 160 frames at 768 × 512 resolution, with higher frame rate and longer sequences than FlythingThings3D.The paper states that these properties make BL30K suitable for VOS, unlike FlythingThings3D.
- Observed utility: Despite being synthetic, BL30K improves real-world performance in ablation experiments, whereas extended training on YouTubeVOS and DAVIS caused severe overfitting.The reported gain is therefore not attributed simply to more training iterations.
5. Implementation Details
MiVOS is trained modularly across synthetic and real video data using a staged propagation procedure and curriculum learning. The implementation uses limited hardware, requires about two weeks to train, and reports sparse propagation for fast runtime.
- Training procedure: The propagation module is first trained on synthetic sequences from static images, then transferred to BL30K and DAVIS.The training description also notes that YouTubeVOS is excluded because of less accurate annotation.
- Training procedure: Each training iteration samples three random frames, while the maximum frame distance increases gradually from 5 to 25 through curriculum learning.The supplied passage truncates the later training schedule, but explicitly states the frame sampling and distance curriculum.
- Resources: All three modules can be trained with two 11GB GPUs using the Adam optimizer.The paper reports this hardware and optimizer configuration for efficient modular training.
- Resources: Training all modules takes about two weeks with two GPUs.Detailed hyperparameters are deferred to the open-sourced code.
- Runtime: Propagation is performed sparsely, keeping MiVOS the fastest among the compared competitors in the reported runtime analysis.Runtime is measured on the 480p DAVIS 2017 validation set, with propagation time amortized.
6. Experiments
MiVOS outperforms competing interactive VOS methods on DAVIS while using interactions on only three frames, and user studies show higher accuracy with less interaction time. Its modular components and ablations support transferability, synthetic pretraining benefits, and difference-aware fusion.
- DAVIS Interactive Track: 3 frames instead of 8: MiVOS outperforms all competitors on the DAVIS interactive validation set.The evaluation constrains interactions to three frames, compared with eight for competing methods.
- DAVIS Interactive Track: ∼12 seconds: MiVOS reaches better J&F performance on average than methods requiring 55+ seconds or 37 seconds.Figure 11 compares the performance-versus-time behavior on the DAVIS validation set.
- Ablation Study: Difference-aware fusion improves interactive VOS accuracy by preserving user intent when interaction and propagation are decoupled.The module compares masks before and after interaction and supports their alignment at target frames through space-time memory.
- Ablation Study: Top-k memory filtering, BL30K pretraining, and difference-aware fusion each further improve the already strong decoupled baseline.The ablation study also reports that difference-aware fusion outperforms naïve linear blending and difference-agnostic learnable fusion with the same network architecture.
- User Study: Higher accuracy with less interaction time: users achieved better results using MiVOS, with additional modes enabling faster convergence and higher final accuracy.The user study involved 10 volunteers labeling five DAVIS 2017 validation videos within a two-minute wall-clock limit.
- User Study: Higher final accuracy and AUC: both MiVOS variants outperform ATNet in mean IoU versus user time.Ours-Free additionally supports f-BRS, free-hand drawing, and local control, allowing experienced users to improve final accuracy with more time.
7. Conclusion
MiVOS decouples interaction-to-mask, propagation, and difference-aware fusion into a modular framework. The design preserves user intent during propagation while supporting versatile interactions and accurate, efficient interactive VOS.
- 7. Conclusion: MiVOS consists of three modules: Interaction-to-Mask, Propagation, and Difference-Aware Fusion.The framework separates interaction processing from temporal propagation and uses fusion to reconcile them.
- 7. Conclusion: Decoupling interaction from propagation makes MiVOS versatile rather than limited to a particular interaction type.The modular design allows the interaction-to-mask component to be replaced for different user interactions.
- 7. Conclusion: Difference-aware fusion captures user intent and mitigates information loss introduced by decoupling, enabling accurate and efficient iVOS.The module reconciles the interaction and propagation stages through the mask differences before and after interaction.