VideoMaMa++: Temporally Consistent Video Matting via Preserve-and-Refine Token

1 Introduction

• Video matting supports background replacement, visual composition, and relighting, where convincing composites require fine structures and temporal consistency.Important details include hair strands and motion-blurred edges.Source: §1 · p. 2
• Diffusion backbones process only 6–24 frames at a time, so independently matting chunks from coarse masks can create flickering edges and discontinuous transparency.A binary mask permits multiple plausible boundary refinements, and separate chunks may settle on different ones.Source: §1 · p. 2
• RGB frames and coarse masks remain separate token streams, allowing full spatio-temporal self-attention to combine texture and foreground layout adaptively.The model can rely on RGB at uncertain boundaries and on masks in the foreground interior.Source: §1 · p. 2
• Learnable preserve and refine tokens enable overlap-based matte propagation across chunks, while a globally shared reference anchor guards against long-sequence error accumulation without additional training.The preserve token marks trusted mattes; the refine token marks frames to generate from coarse masks.Source: §1 · pp. 2–3
• VideoMaMa++ outperforms prior methods on VideoMatte, YoutubeMatte, and V-HIM60 in fine-detail accuracy and temporal consistency, and achieves state-of-the-art long-video matting.Prior diffusion methods fail to maintain consistency across chunk boundaries.Source: §1 · pp. 2–3

Video matting

• Auxiliary-free methods largely focus on portraits, while auxiliary-guided methods use extra conditioning; trimap methods need manual annotations and mask-guided methods reduce that burden.MaGGIe struggles with in-the-wild generalization, and MatAnyone propagates target-assigned mattes using a memory-augmented architecture trained on dedicated datasets.Source: §2 · p. 3

Diffusion priors for perception

• Diffusion priors from internet-scale pretraining have been transferred from depth estimation to several video perception tasks, including work using DiT backbones.Marigold adapted Stable Diffusion for depth estimation.Source: §2 · p. 3

Diffusion long-video inference

• Long-video diffusion methods use overlapping sliding windows or autoregressive propagation, but hidden-state propagation does not naturally compose with explicit guide masks.VideoMaMa++ instead passes fully formed reference mattes through preserve tokens while retaining each chunk’s user-supplied mask guidance.Source: §2 · p. 3

Diffusion-based video matting

• GVM and VideoMaMa show that video diffusion priors can generalize to non-human and in-the-wild matting, but inherit chunk-wise inference limits.VideoMaMa also fuses RGB, mask, and noise by channel concatenation, which entangles modalities and limits fine-detail recovery; VideoMaMa++ separates guidance streams and adds preserve/refine tokens.Source: §2 · p. 3

3.1 Problem Formulation and Overview

• The pipeline propagates a first-frame user prompt into a mask track, generates alpha mattes from masks and RGB, then processes long sequences in chunks with preserve/refine tokens.Matting seeks per-pixel opacity values that retain fine details such as hair and motion blur, unlike binary masks.Source: §3.1 · pp. 4–5

3.2 Repurposing Diffusion Transformer for Video Matting

• Video, guide masks, and training mattes are encoded by a shared frozen VAE, and separate token streams let the DiT preserve modality-specific information.The model is built on a video DiT with an internet-scale generative prior; decoded predictions are supervised in pixel space, and one denoising step suffices at inference.Source: §3.2 · p. 5

3.3 Preserve and Refine Tokens

• Independent chunk refinement can drift across boundaries, so preserve tokens retain supplied mattes while refine tokens generate fine mattes from coarse guidance.Joint training lets frames in the two modes interact through full self-attention.Source: §3.3 · p. 5

3.4 Temporally Consistent Inference

• Overlapping chunks use previous predictions as preserve inputs and remaining masks as refine inputs, allowing preserved frames to condition generation across the chunk.A globally reliable reference matte is supplied to each chunk to guard against errors cascading from corrupted overlap frames.Source: §3.4 · p. 6
• The reference anchor can be selected by the user; the first chunk’s matte is the default, but a later, clearer frame can be chosen when the first is unreliable.Examples of first-frame unreliability include heavy occlusion, motion blur, or partial visibility.Source: §3.4 · p. 6

4.1 Implementation Details

• Training uses 3-frame clips, while inference uses 6-frame chunks with 2-frame overlaps; evaluations give mask-guided methods identical SAM3-derived guidance.The guidance masks are propagated from a binarized first-frame ground-truth matte, so no method receives a privileged mask.Source: §4.1 · p. 6, §4.2 · pp. 6–7

4.2 Quantitative Evaluations

• VideoMaMa++ achieves the best performance across all metrics on every benchmark, with particularly pronounced gains in Gradient and dtSSD.Gradient measures spatial boundary sharpness, while dtSSD measures temporal smoothness.Source: §4.2 · p. 7

4.3 Qualitative Evaluations

• On long in-the-wild videos, VideoMaMa++ retains fine boundary detail and avoids the chunk-boundary discontinuities seen with VideoMaMa.The comparison also shows stable transitions while fine-grained details emerge in later chunks.Source: §4.3 · p. 7, Figure 3 · p. 7
• Selective refinement re-tags the highest-error frame as refine and its neighbors as preserve, improving that frame without adding a guide signal.The model attends to the preserved frames as anchors when regenerating the selected frame.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 Ablation Studies

• Adding the global reference anchor improves all six metrics on both benchmarks, while chunk-index MAD stays flatter than without the anchor.Alternative first-chunk anchors and periodic refresh perform nearly unchanged; even a fully occluded anchor causes only marginal degradation.Source: §4.4 · pp. 8–9, Table 2 · p. 8, Table 3 · p. 8
• Preserving overlap mattes reduces both MAD and Gradient discrepancy versus regenerating them from coarse masks, transferring information across chunk boundaries.The targeted comparison disables the reference anchor in both settings.Source: §4.4 · p. 9, Table 4 · p. 9
• Performance remains consistent across mask generators; token-wise inputs beat channel concatenation, while SAM-logit gains do not transfer to in-the-wild videos.The channel-concatenation variant costs less attention compute but underperforms across metrics.Source: §4.4 · pp. 9–10, Table 5 · p. 9, Table 6 · p. 10
• One denoising step is the default; 6-frame chunks outperform 3- and 12-frame settings, and LoRA rank 512 leads across all six metrics.Additional denoising steps over-refine the matte; the 6-frame setting balances temporal context with preserve-frame influence.Source: §4.4 · p. 10, Table 7 · p. 10, Table 8 · p. 10

5 Conclusion

• VideoMaMa++ reports state-of-the-art performance across diverse benchmarks and fine-grained, temporally consistent results on long in-the-wild videos.An annotation-free analysis finds no measurable chunk-boundary discontinuity, unlike VideoMaMa processing chunks independently.Source: §5 · p. 10
• The method remains more computationally demanding than feed-forward models and depends on the upstream mask track; real-footage temporal-smoothness benchmarking is also limited.The conclusion identifies reducing DiT inference cost and benchmarking temporal smoothness on real footage as future directions.Source: §5 · p. 10

Appendix Overview

• The supplementary material covers implementation, qualitative and zero-shot evaluations, annotation noise, guidance protocols, chunk-boundary tests, runtime, anchor robustness, limitations, and societal impacts.Source: Appendix Overview · p. 15

• Predicted latents are decoded with the VAE decoder, and the resulting alpha mattes are supervised in pixel space.Source: A Additional Implementation Details · p. 15
• Training uses an effective batch size of 16, a learning rate of 1 × 10^-4, AdamW, and 8 NVIDIA H100 GPUs for approximately 2.5 days.Table 10 is identified as listing the training datasets.Source: A Additional Implementation Details · p. 15, Table 10 · p. 15
• The objective combines pixel-wise L1 accuracy, spatial-gradient edge sharpness, and Laplacian emphasis on high-frequency boundaries and hair details.The weights are λL1 = 1.0, λgrad = 5.0, and λlap = 2.0.Source: A Additional Implementation Details · p. 15

• Figure 4a presents qualitative examples, while Table 11 reports quantitative results for interactive refinement.Source: B Interactive Refinement Qualitative… · p. 15, Table 11 · p. 15
• Across iterations, interactive refinement recovers regions missed initially and sharpens fine boundary details without modifying the input mask guidance.Source: B Interactive Refinement Qualitative… · pp. 15–16
• The marked frame draws on adjacent-frame context through DiT self-attention, supporting cross-frame reasoning rather than simple smoothing.Source: B Interactive Refinement Qualitative… · p. 16

C Image benchmark

• VideoMaMa++ achieves state-of-the-art performance against prior portrait matting methods despite not being designed specifically for portrait matting.On AM-2K, annotation errors can inflate MAD even when predictions recover details missing from the ground truth.Source: C Image benchmark · p. 16, Table 12 · p. 16

D AM-2K Justification

• On AM-2K, MAD-T indicates state-of-the-art performance against comparison methods because it evaluates transition-region detail while avoiding bias from missing ground-truth detail.MAD averages error over the whole image, so annotation gaps can penalize visually superior predictions.Source: D AM-2K Justification · p. 16, Figure 6 · p. 16

E Guidance Mask Protocol and YouTubeMatte SAM3 Mask

• On YouTubeMatte static_0017, SAM3 was initialized at frame 60 so its guidance track included a paper entering mid-video, and VideoMaMa++ remained best under both initialization settings.All mask-guided methods degrade with first-frame initialization because the paper is absent from their guidance; MatAnyone is unaffected because it uses no external mask track.Source: E Guidance Mask Protocol and… · pp. 17–18, Table 13 · p. 18

Test design

• The annotation-free test detrends frame-to-frame matte change and checks whether spikes recur at chunk seams using seam-period autocorrelation and seam-versus-interior comparisons.Both models use the same cadence: a 6-frame first chunk followed by 4 new frames per chunk.Source: F Annotation-free Temporal Consistency · p. 18

Validation and results

• In 62–64 of 64 YouTubeMatte clips, VideoMaMa’s seam contrast exceeded VideoMaMa++’s; in 16 in-the-wild videos, only VideoMaMa showed seam-period structure.Ground-truth mattes showed no seam elevation, while VideoMaMa++’s seam-elevated counts were consistent with chance in both settings.Source: F Annotation-free Temporal Consistency · p. 19, Table 14 · p. 19

Robustness and scope

• The test measures chunk-boundary discontinuities only, not per-frame accuracy or within-chunk flicker, and applies only to chunk-wise methods.Its conclusions were unchanged across alternative difference statistics and detrending windows of 3 to 11 frames.Source: F Annotation-free Temporal Consistency · p. 19

Throughput and memory

• At 1024² on one H100, VideoMaMa++ runs at 1.17 FPS with 11.3 GB peak memory, versus VideoMaMa at 3.41 FPS and 7.9 GB.Feed-forward baselines run at 15–60 FPS; reported VideoMaMa++ throughput amortizes each 6-frame window over its 4 new frames.Source: G Computational Cost · p. 19, Table 15 · p. 20

Stage costs

• At 1024², the DiT forward pass takes 616 ms of the 853 ms per output frame, compared with 139 ms for the VAE encoder and 98 ms for the decoder.SAM3 mask propagation adds 45.4 ms per frame regardless of matting resolution.Source: G Computational Cost · p. 19, Table 16 · p. 20

Inference implications

• Because each additional denoising step adds a full DiT evaluation per window, single-step inference is essential for keeping the computational cost practical.End-to-end throughput is 1.11 FPS at 1024² and 9.4 FPS at 256².Source: G Computational Cost · p. 19, Table 16 · p. 20

Anchor choice and refresh

• On V-HIM60, replacing the first-frame anchor with a random frame or the last frame of the first chunk leaves performance nearly unchanged across metrics.Refreshing the anchor every three chunks also yields nearly identical quality despite substantial pose, scale, and appearance variation in the sequences.Source: H Reference Anchor Analysis Details · p. 20

Corruption protocol

• The corruption experiment degrades the first-frame input that serves as the anchor, then excludes that frame from evaluation to measure how the degraded anchor propagates to later video frames.Tested inputs include eroded or dilated guidance masks, motion-blurred RGB, and an occluded subject.Source: H Reference Anchor Analysis Details · p. 20

Corruption results

• Across all tested corruptions, performance degrades only marginally from the clean baseline, with full occlusion most severe, followed by motion blur and poor guidance masks.Larger corruption kernels generally cause slightly greater degradation, and even a fully occluded anchor has limited effect on the rest of the video.Source: H Reference Anchor Analysis Details · pp. 20–21

I Limitation

• A wrong-object track lasting an entire sequence makes the matte follow that object, and this dependency is shared by mask-guided matting methods.Frame-local tracking errors may be corrected through neighboring-frame attention or interactive refinement.Source: I Limitation · p. 21
• Real-video temporal smoothness lacks a rigorous benchmark; the seam test measures chunk-boundary discontinuities but not per-frame accuracy or within-chunk flicker.The test applies only to chunk-processing methods, so it cannot compare arbitrary methods' overall temporal behavior.Source: I Limitation · p. 21

J Societal Impact

• The framework may reduce manual matting effort and support creative work, education, communication, and accessibility-oriented applications.The paper specifically mentions background replacement in video conferencing and virtual classrooms.Source: J Societal Impact · p. 21
• Improved matting may also make deepfakes and manipulated media more convincing, contributing to misinformation and reduced public trust in video evidence.Source: J Societal Impact · p. 21