PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM2Matting: Generalized Image and Video Matting

Ruiqi Shen, Guangquan Jie, Chang Liu, Henghui Ding

arXiv:2606.27339v1cs.CV

TL;DR

Video matting must reconcile robust temporal tracking with fine-grained alpha estimation without relying on costly, narrowly scoped video datasets. SAM2Matting decouples tracking from matting using pretrained VOS trackers and dedicated image-trained components, achieving state-of-the-art zero-shot video matting with robust generalization.

  • Problem

    Video matting requires both high-level tracking for temporal consistency and low-level matting for fine-grained details, while existing datasets are costly and narrowly scoped.

  • Method

    SAM2Matting preserves established VOS trackers for temporal consistency and adds ROI detection with progressive alpha refinement trained on diverse image matting data.

  • Results

    SAM2Matting achieves state-of-the-art video matting performance in a zero-shot setting while generalizing to human-centric and in-the-wild scenarios.

  • Takeaways & Limitations

    The framework supports diverse prompts, maintains temporal consistency over extended videos, and offers real-time efficiency for practical deployment.

Abstract

from arXiv · show

Despite impressive advances in image matting, video matting remains challenging due to the inherent gap between high-level tracking, which requires frame-wise understanding, and low-level matting, which focuses on extremely fine-grained details. Existing methods attempt this with expensive and narrowly-scoped video matting datasets, which may limit out-of-domain generalization and compromise tracking robustness. We rethink the paradigm with SAM2Matting, a tracker-to-matting framework that advances VOS trackers to high-fidelity video matting. Specifically, it decouples the task by enhancing a foundational tracker (e.g., SAM2, SAM3) with a region-proposal bridge and dedicated matting heads, enabling the uncompromised tracker to handle temporal consistency while the matting components resolve fine-grained details. Notably, despite being trained only on images, SAM2Matting establishes new state-of-the-art performance on video matting, supports diverse prompt types, maintains strong temporal consistency, and demonstrates robust generalization across both human-centric and in-the-wild scenarios.

1 Introduction

SAM2Matting addresses video matting’s tension between robust high-level tracking and fine-grained alpha estimation by decoupling the task into tracker-based temporal consistency and image-matting-based detail refinement. It achieves zero-shot video-matting SOTA with broad generalization, diverse prompts, and efficient tracker integration.

  • Motivation: Video matting requires semantic understanding for consistent target tracking and fine-grained perception for intricate, semi-transparent details.Existing video-matting datasets are limited in scale and domain, while training from scratch or fine-tuning VOS models can weaken tracking robustness.
  • Method: SAM2Matting decouples video matting into high-level tracking and low-level matting, leveraging established VOS trackers and diverse image-matting datasets.The framework integrates the two sub-tasks through a tracker-to-matting design.
  • Method: An ROI Detector identifies matting-critical regions from image-level cues and mask-level tracking priors while correcting tracking inaccuracies.These regions contain fine-grained details or semi-transparency, and the framework applies progressive alpha flows for cascaded refinement.
  • Results: SAM2Matting achieves new SOTA performance on image and strictly zero-shot video matting, generalizing to human-centric and in-the-wild scenarios.In-the-wild results include rapid motion, complex backgrounds, and target attachments.
  • Results: The framework supports mask, point, box, and text prompts across SAM2.1-Tiny, SAM2.1-Base+, and SAM3 variants with minimal FPS and VRAM overhead.SAM2.1-Tiny runs at 40 FPS on a 200-frame 1080p video using less than 5GB GPU memory and maintains stable performance on challenging extended videos.

2 Related Work

Related work distinguishes automatic image matting from target-specified approaches and shows video matting evolving from target-agnostic estimation toward explicit target specification. Image matting predicts pixel-level alpha mattes separating foreground from background.

  • Image Matting: Image matting separates a foreground target from its background by predicting a pixel-level alpha matte α.The image is modeled as I = αF + (1 −α)B.
  • Image Matting: Automatic image matting directly mats all objects without additional input beyond the image itself.This paradigm is also described as auxiliary-free matting.
  • Video Matting: Early video matting methods estimated alpha mattes for all visible objects, but this target-agnostic setting becomes ambiguous when targets enter and exit scenes.Recent methods therefore require explicit target specification.

3 Method

SAM2Matting decouples temporally consistent VOS tracking from fine-grained matting through an ROI Detector and a Progressive Alpha Predictor. It freezes the tracker and trains only the matting components on image data, preserving tracking robustness while refining alpha mattes.

  • Framework overview: A VOS tracker supplies a temporally consistent mask, while dedicated matting components resolve fine-grained details and semi-transparency.The framework first identifies matting-critical regions and then refines the matte through a coarse-to-fine process.
  • ROI Detector: The ROI Detector performs pixel-wise binary classification using the VOS mask, current frame, and multi-scale image features.Scale-specific ROI heads predict logits at each resolution, which a hierarchical convolutional network aggregates using global context and structural details.
  • Pseudo-trimap construction: The predicted ROI becomes the unknown region of a pseudo-trimap, while the VOS mask defines definite foreground and background for subsequent alpha estimation.The pseudo-trimap uses labels 1 for foreground, 0 for background, and 0.5 for unknown regions.
  • Progressive Alpha Predictor: The Progressive Alpha Predictor sequentially refines mattes from coarse to fine, using image features, frames, pseudo-trimaps, and preceding-scale matte guidance.The finest-scale matte is upsampled to the original resolution to produce the final alpha matte.
  • Training design: The VOS tracker is frozen, and only the matting components are trained on high-quality image matting data to preserve temporal consistency and tracking robustness.Training therefore enables fine-grained alpha refinement without updating the tracker.

4 Experiments · 4.1 Experimental Setup · 4.2 Quantitative Evaluation on Image Matting

SAM2Matting is evaluated using multiple tracker variants, standardized training and matting metrics, and three image-matting benchmarks. All variants outperform prior baselines, including an 11.48 lower MAD than MAM on P3M-500-NP for SAM2.1-Tiny.

  • 4.1 Experimental Setup: Training uses eight image matting datasets, including I-HIM50K, P3M-10k, CelebAHairMask-HQ, AIM-500, Distinctions-646, AM-2K, UHRIM, and RefMatte.
  • 4.1 Experimental Setup: Variants use training datasets and trackers aligned with the baselines for fair comparisons.
  • 4.1 Experimental Setup: Three variants use SAM2.1-Tiny, SAM2.1-Base+, or SAM3 as VOS trackers.
  • 4.1 Experimental Setup: The trackers remain frozen while only matting components are optimized during training.Training lasts 5 epochs on 4 NVIDIA A6000 GPUs with batch size 32 and AdamW; hyperparameters are selected by grid search.
  • 4.1 Experimental Setup: Evaluation uses MAD, MSE, Grad, Conn, and dtSSD, with lower values indicating better performance.dtSSD is used only for video matting.
  • 4.2 Quantitative Evaluation on Image Matting: Image matting is evaluated on P3M-500-NP, AM-2K, and PPM-100 benchmarks.
  • 4.2 Quantitative Evaluation on Image Matting: 11.48 lower MAD than MAM is achieved by SAM2.1-Tiny on P3M-500-NP, while all three variants consistently outperform previous baselines across metrics.The reported gains are attributed mainly to the proposed matting design rather than only larger training data or a stronger tracker backbone.

4.3 Quantitative Evaluation on Video Matting

SAM2Matting achieves strong zero-shot video matting performance on V-HIM60 and VideoMatte, outperforming video-supervised baselines. Its lowest dtSSD further indicates strong temporal consistency from the VOS tracker, validating the decoupled tracking-and-matting design.

  • Benchmark and setting: It consistently outperforms recent video-supervised approaches, including MatAnyone2, MatAnyone, MaGGIe, FTP-VM, and RVM.The compared methods include recent state-of-the-art approaches and RVM.
  • Benchmark and setting: SAM2Matting is evaluated zero-shot on the V-HIM60 and VideoMatte video matting benchmarks.The evaluation compares against baselines supervised on video matting datasets.
  • Qualitative evidence: Qualitative comparisons show superior resolution of fine-grained hair strands and semi-transparencies on human matting.The comparison is against MatAnyone and RVM.
  • Temporal consistency: SAM2Matting achieves the lowest dtSSD, indicating strong temporal consistency inherited from the VOS tracker.The temporal-consistency result supports the tracker’s role in the decoupled framework.
  • Design validation: The results validate decoupling tracking and matting so both components specialize independently while cooperating effectively for robust video matting.The tracker handles temporal consistency while the matting components address fine-grained details.

4.4 Qualitative Evaluation · 4.5 Ablation Studies

SAM2Matting qualitatively preserves fine details, tracking robustness, and matte-mask consistency across human, in-the-wild, and distracted-target scenarios. Ablations show that its gains persist under controlled comparisons and arise from architectural, ROI, and supervision designs, while video fine-tuning improves in-domain performance but harms generalization.

  • 4.4 Qualitative Evaluation: SAM2Matting outperforms RVM and MatAnyone on intricate hair-level details and semi-transparencies, preserving fine details under complex lighting while suppressing foreground occlusions.Figure 4 highlights suppression of undesired occlusion parts such as green bars in front of a human.
  • 4.4 Qualitative Evaluation: SAM2Matting generalizes better than domain-specific video matting methods to in-the-wild sequences with fast-moving, non-human, and semi-transparent targets.MatAnyone2 and MaGGIe struggle with rapidly growing roots, semi-transparent butterflies, and rapidly dripping water, whereas SAM2Matting preserves robust tracking and recovers intricate details.
  • 4.4 Qualitative Evaluation: SAM2Matting handles attached target objects such as bicycles and ski poles while suppressing nearby background distractions.The qualitative examples attribute this behavior to matte-mask consistency supervision.
  • 4.5 Ablation Studies: SAM2Matting performs consistently better than MAM and Matte Anything with aligned training data and with the same SAM2.1-B+ tracker, indicating gains from architecture and supervision rather than data or backbone scale.The default ablation model is SAM2Matting (SAM2.1-B+).
  • 4.5 Ablation Studies: The ROI Detector outperforms Morphological and Mask-only ROI strategies across all evaluated settings on V-HIM60-Hard.Morphological uses mask dilation and erosion, while Mask-only directly uses the raw VOS mask for matte prediction.
  • 4.5 Ablation Studies: Multi-scale cascade refinement progressively recovers fine structures from coarse to fine, while matte-mask consistency and related supervision designs improve matte quality.The consistency penalty fills foreground holes, with examples including a hollow chair and flying hair.
  • 4.5 Ablation Studies: Video fine-tuning improves V-HIM60-Hard but worsens AM-2K generalization, barely changes VideoMatte and PPM-100, and degrades original tracking robustness.The results indicate overfitting because V-HIM2K5 covers relatively simple scenarios within a narrow human-centric domain.

4.6 Resistance to Tracking Inaccuracies

SAM2Matting mitigates VOS tracker inaccuracies by treating tracker masks as one cue among several and fusing them with image-level appearance priors for ROI detection. The resulting ROI guidance helps recover tracking-missed foreground details while preserving robustness compared with video fine-tuning.

  • Robustness to tracker inaccuracies: The ROI detector fuses VOS masks with image-level appearance priors instead of treating masks as hard constraints, suppressing tracker-induced errors before alpha prediction.This provides reliable ROI guidance for the subsequent alpha predictor.
  • Robustness to tracker inaccuracies: The ROI Detector identifies instance-specific matting-critical regions missed by morphological operations or raw masks, including flying hair, thin leaves, and limb gaps.These regions represent fine foreground structures that can be overlooked by coarse tracking masks.
  • Robustness to tracker inaccuracies: Video fine-tuning noticeably weakens the original tracking robustness, whereas the original decoupled model retains stronger tracking robustness.“Original” denotes the original decoupled model, while “Video-FT” denotes the variant fine-tuned on video matting data.

4.7 FPS and VRAM Efficiency

SAM2Matting maintains stable processing speed and modest VRAM usage across input resolutions on VideoMatte, with SAM2.1-T and SAM2.1-B+ exceeding 30 FPS for real-time video matting.

  • 4.7 FPS and VRAM Efficiency: On VideoMatte, all three SAM2Matting variants maintain stable FPS and modest VRAM usage across different input resolutions.Efficiency is evaluated using a single NVIDIA A6000 GPU.

4.8 Flexible Prompting.

SAM2Matting enables interactive video matting through flexible prompts inherited from VOS trackers, with SAM3 additionally supporting text prompts.

  • Flexible Prompting: SAM2Matting supports initial-frame masks, points, and boxes, while the SAM3 variant additionally supports text prompts for interactive video matting.A selfie-video example shows high-quality results across different prompts.

5 Discussion

SAM2Matting maintains temporally stable mattes during rapid motion and across long sequences with repeated target disappearances and reappearances. It consistently tracks and mattes targets across 500 frames, contrasting with baselines that show stronger temporal fluctuations and visible flickering artifacts.

  • Flickering effect: SAM2Matting maintains stable mattes under rapid target motion, preserving coherent foregrounds over time.Baselines exhibit more pronounced temporal fluctuations and visible flickering artifacts.
  • Performance on long videos: 500 frames: SAM2Matting consistently tracks and mattes through repeated target disappearances and reappearances.The demonstration addresses long-sequence requirements in applications such as film post-production and e-commerce live streaming.
  • Performance on long videos: Existing public video-matting benchmarks mostly contain short clips, motivating evaluation on a challenging long video.The long-video evaluation tests target entry, exit, disappearance, and reappearance across an extended sequence.

6 Conclusion

SAM2Matting decouples high-level tracking from low-level matting in a generalized image and video matting framework. It achieves state-of-the-art video matting without costly video matting datasets while offering broad generalization, real-time efficiency, diverse prompts, and strong long-term temporal consistency.

  • 6 Conclusion: SAM2Matting decouples high-level tracking from low-level matting.This design forms the framework’s central contribution.
  • 6 Conclusion: SAM2Matting achieves state-of-the-art video matting without relying on costly video matting datasets.The framework is presented as generalized across image and video matting.
  • 6 Conclusion: SAM2Matting generalizes robustly to both human-centric and in-the-wild scenarios.This supports its applicability across varied video matting settings.
  • 6 Conclusion: SAM2Matting offers real-time efficiency, supports diverse prompt types, and maintains strong temporal consistency over extended videos.These properties are highlighted as facilitating real-world deployment and future research.
Loading 2606.27339v1…