Source-linked AI summary
VideoMaMa: Mask-Guided Video Matting via Generative Prior
Sangbeom Lim, Seoung Wug Oh, Jiahui Huang, Heeji Yoon, Seungryong Kim, Joon-Young Lee
TL;DR
Video matting struggles to generalize to real-world footage because high-quality, diverse labels are scarce and synthetic compositions create a domain gap. VideoMaMa uses pretrained video diffusion priors to convert segmentation masks into alpha mattes, then enables large-scale pseudo-labeling through MA-V; SAM2-Matte trained on this dataset shows stronger in-the-wild robustness than comparable training alternatives.
Problem
Real-world video matting is limited by scarce, low-diversity annotations and a domain gap from synthetic compositions.
Method
VideoMaMa converts binary segmentation masks into continuous alpha mattes using pretrained video diffusion priors and spatial-temporal training.
Results
VideoMaMa robustly generates high-quality mattes from diverse mask sources, enabling scalable pseudo-labeling and MA-V annotations for over 50K real-world videos.
Takeaways & Limitations
SAM2-Matte trained on MA-V is more robust on in-the-wild videos than SAM2 trained on existing matting datasets and other video matting methods.
Takeaways & Limitations
Binary masks can cause copy-paste behavior for simple shapes or masks containing fine-grained details.
Abstract
from arXiv · showhide
Generalizing video matting models to real-world videos remains a significant challenge due to the scarcity of labeled data. To address this, we present Video Mask-to-Matte Model (VideoMaMa) that converts coarse segmentation masks into pixel accurate alpha mattes, by leveraging pretrained video diffusion models. VideoMaMa demonstrates strong zero-shot generalization to real-world footage, even though it is trained solely on synthetic data. Building on this capability, we develop a scalable pseudo-labeling pipeline for large-scale video matting and construct the Matting Anything in Video (MA-V) dataset, which offers high-quality matting annotations for more than 50K real-world videos spanning diverse scenes and motions. To validate the effectiveness of this dataset, we fine-tune the SAM2 model on MA-V to obtain SAM2-Matte, which outperforms the same model trained on existing matting datasets in terms of robustness on in-the-wild videos. These findings emphasize the importance of large-scale pseudo-labeled video matting and showcase how generative priors and accessible segmentation cues can drive scalable progress in video matting research.
1. Introduction
VideoMaMa addresses scarce, low-diversity annotations and the synthetic-to-real gap by converting segmentation masks into alpha mattes with pretrained video diffusion priors. The resulting model supports scalable pseudo-labeling, enabling MA-V and stronger robust matting with SAM2-Matte.
- Video matting supports video editing applications including background replacement, visual composition, and relighting.
- Scarce high-quality annotations and synthetic-video artifacts make robust generalization to diverse real-world footage difficult.Existing annotations are largely captured in controlled environments, while synthetic compositions can distort lighting, motion blur, and temporal coherence.
- VideoMaMa converts binary segmentation masks into continuous alpha mattes using pretrained video diffusion models and a two-stage spatial-temporal training strategy.The method also injects semantic knowledge through DINOv3 features while preserving generative capabilities.
- VideoMaMa consistently produces high-quality mattes from diverse mask sources, supporting its use as a pseudo-labeler for large-scale annotation.The evaluation includes binary masks produced by existing video segmentation models.
- MA-V provides high-quality pseudo-matting annotations for over 50K real-world videos spanning diverse scenes, objects, and motion dynamics.The dataset is built by converting segmentation labels from the SA-V dataset through a scalable pipeline.
- SAM2-Matte, fine-tuned on MA-V without architectural modifications, is more robust on in-the-wild videos than SAM2 trained on existing matting datasets and other methods.
2. Related Work
Prior video matting approaches and datasets are constrained by portrait-centric settings, manual auxiliary inputs, scarce real annotations, and synthetic compositions. VideoMaMa instead exploits pretrained diffusion priors to address these limitations in real-world video matting.
- Auxiliary-free methods typically focus on portrait matting, while trimap-guided methods require manual annotations that limit zero-shot inference.
- Existing video matting datasets contain only hundreds of videos, predominantly human subjects captured in controlled settings or annotated manually.
- No large-scale real video matting dataset exists with naturally co-occurring foregrounds and backgrounds; existing composition-based datasets create artificial scenes.
- Diffusion models encode priors about natural scenes, motion dynamics, and temporal coherence, motivating their use for video matting.
3. Video Mask-to-Matte Model
VideoMaMa converts RGB video and binary guide masks into pixel-accurate alpha mattes using a pretrained video diffusion model. Its design combines latent-space single-step generation with mask augmentation and semantic feature injection to improve efficiency, detail generation, and object understanding.
- 3.1. Problem Formulation: Video matting represents foreground opacity continuously, capturing details such as hair strands, motion blur, and intricate regions that binary masks cannot express.
- 3.1. Problem Formulation: Binary masks provide object shape while allowing VideoMaMa to focus on generating fine-grained matting details from RGB video.
- 3.2. Architecture Design: VideoMaMa adapts Stable Video Diffusion with mask conditioning, semantic knowledge injection, and single-step inference for mask-guided video matte generation.
- 3.2. Architecture Design: Video frames, masks, and alpha mattes share a VAE latent space, reducing memory requirements while preserving spatial correspondence.
- 3.2. Architecture Design: The model predicts clean alpha-matte latents directly from concatenated video latents, mask latents, and Gaussian noise, then decodes them through the VAE.
- 3.3. Training Recipe: Mask augmentation removes fine-grained input details through polygon and downsampling degradation, discouraging copy-paste behavior and encouraging RGB-based matte inference.
- 3.3. Training Recipe: Pixel-level supervision optimizes visual quality, while DINO feature alignment improves semantic boundary understanding and tracking for overlapping or articulated objects.
4. Matte Anything in Videos (MA-V) Dataset
MA-V addresses the scarcity and limited diversity of real-world video matting data by converting diverse SA-V segmentation masks into continuous alpha annotations. The resulting dataset contains over 50K natural videos and supports fine-tuning SAM2 for matting.
- Existing video matting datasets contain at most hundreds of videos and predominantly focus on humans captured in controlled environments or manually annotated.
- MA-V contains 50,541 naturally captured videos, nearly 50× larger than existing real-video datasets.It covers diverse object categories and scales, from small objects to full scenes.
- The dataset construction pipeline applies VideoMaMa to SA-V segmentation annotations to produce continuous matting labels while preserving natural scene context.The resulting mattes include semi-transparent regions, motion blur, and intricate boundary structures.
- SAM2-Matte is obtained by fine-tuning SAM2 on MA-V, using a sigmoid after mask logits to produce continuous alpha values in [0,1].Its quality is assessed against models fine-tuned on existing video matting datasets using standard benchmarks.
5. Experiments
Experiments evaluate VideoMaMa and SAM2-Matte across mask-guided settings, datasets, frame counts, and training configurations. Results support robust mask refinement, temporal generalization, and benefits from MA-V and the proposed training components.
- Experimental Details: VideoMaMa is trained with a two-stage procedure combining high-resolution spatial learning, temporal modeling, and semantic feature injection.The training objective combines L1 loss for pixel accuracy with Laplacian loss for edge and boundary detail.
- Quantitative Evaluations: Table 2 compares VideoMaMa with MaGGIe and MGM on V-HIM60 and YouTubeMatte using degraded and SAM2-generated masks.The evaluation uses 12-frame sequences and reports MAD and Gradient error.
- Quantitative Evaluations: VideoMaMa consistently outperforms existing mask-guided matting models across synthetically degraded and model-generated masks.The results indicate robustness to different input-mask styles.
- Quantitative Evaluations: SAM2-Matte surpasses MatAnyone on challenging V-HIM60 Hard and YouTubeMatte benchmarks.The result supports the quality of MA-V for learning matting quality and temporal consistency.
- Ablation Studies: VideoMaMa handles inference with frame counts ranging from single frames to 24 frames despite training on at most 3 frames.Performance remains consistent across the evaluated frame counts.
- Ablation Studies: Using both training stages produces the best results, while adding DINO semantic features further improves object understanding and boundary localization.The ablation identifies the two components as necessary and complementary for optimal performance.
- Ablation Studies: Combining existing matting datasets with MA-V achieves the best overall matting quality on V-HIM60 Hard.MA-V alone substantially improves over existing datasets, while the authors hypothesize that shared synthetic content contributes to the combined result.
- Ablation Studies: Adding existing synthetic datasets to MA-V degrades DAVIS tracking performance compared with using MA-V alone.The authors suggest that synthetic datasets introduce domain biases that reduce in-the-wild tracking robustness.
6. Conclusion
The paper introduces VideoMaMa and MA-V as a generative-prior model and scalable real-world video matting dataset. Experiments show state-of-the-art performance for VideoMaMa and MA-V-trained models across diverse video matting benchmarks.
- VideoMaMa uses generative priors to perform robust video matting.
- MA-V is a large-scale pseudo video matting dataset built from real-world videos by converting SA-V segmentation masks.
- VideoMaMa and MA-V-trained models, including SAM2-Matte, achieve state-of-the-art performance across diverse video matting benchmarks.
Appendix
The supplementary material adds implementation, metric, and qualitative details omitted from the main paper. It is organized into sections covering SAM2-Matte and VideoMaMa configurations, MAD-T, and comparisons with binary SAM2.
- The supplementary material provides implementation, evaluation-metric, and qualitative-result details excluded from the main paper due to space constraints.
- Section A: Section A covers SAM2-Matte training configurations and VideoMaMa’s feature-injection architecture.
- Sections B–C: Section B defines MAD-T and gives its frame-by-frame transition-region evaluation algorithm, while Section C compares SAM2-Matte with binary SAM2.
A. Additional Implementation Details
VideoMaMa obtains framewise input masks by propagating SAM2 prompts or initial ground-truth masks, while its architecture injects semantic features into an SVD upsampling block. SAM2-Matte uses the SAM2 Base-Plus architecture and mixed video-image training.
- For in-the-wild videos, a first-frame point prompt is propagated across the sequence using SAM2 tracking to supply VideoMaMa input masks.
- For benchmark evaluation, the first-frame ground-truth mask is binarized and propagated with SAM2 to generate VideoMaMa’s input mask sequence.
- VideoMaMa extracts semantic features with DINOv2 and injects them into SVD’s first upsampling block through a two-layer MLP projection.
- SAM2-Matte is based on SAM2 Base-Plus, uses official pretrained weights, and trains with a mixed strategy combining video and image samples.
- The existing datasets used to train SAM2-Matte are summarized in Table 7.
B. Evaluation Metric Details
MAD-T evaluates matting accuracy specifically in uncertain transition regions rather than averaging errors over broad certain regions. The unknown region is generated per frame by eroding binary foreground and background masks derived from ground-truth alpha.
- MAD-T restricts Mean Absolute Difference to the unknown trimap region to measure transition-region accuracy.This targets fine details such as hair and motion blur that standard metrics can dilute.
- The unknown region is generated dynamically for each frame because video datasets rarely provide ground-truth trimaps.
C. Comparison with SAM2
SAM2-Matte adapts SAM2 for alpha matte generation by retaining sigmoid outputs instead of thresholding binary masks, with evaluation including MAD-T frame-wise calculation and comparison against SAM2.
- SAM2-Matte adaptation: SAM2-Matte maps SAM2 mask logits to continuous alpha values by removing thresholding and retaining the sigmoid activation.This adaptation forms the basis for comparing SAM2-Matte with the original SAM2.
- Evaluation: MAD-T computes frame-level error over an unknown region formed by excluding eroded certain foreground and background regions.The procedure uses a structuring element with kernel size 10 and evaluates absolute alpha differences in the unknown region.
- Comparison: Figure 7 compares SAM2-Matte with SAM2 on a real video, using a sigmoid applied to mask logits to simulate alpha matte generation for SAM2.The comparison also identifies whether results use an existing dataset.
D. Limitation
VideoMaMa can refine rough or imperfect guidance masks, but its mask-guided design limits performance when the input selects the wrong object instance or contains substantial errors.
- VideoMaMa: The model cannot generate correct mattes when the mask captures a completely wrong instance, such as a different object.It can refine missing fine-grained details but propagates fundamentally incorrect target selection.
- VideoMaMa: VideoMaMa struggles to refine mattes when the input guidance mask contains significant errors.The limitation is illustrated by red arrows in Figure 8.
- SAM2-Matte: SAM2’s 64 × 64 mask decoder output is upsampled to input resolution, causing loss of high-frequency details needed for precise alpha mattes.The limitation affects details such as hair strands and object boundaries.