Source-linked AI summary
Mask Guided Matting via Progressive Refinement Network
Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang, Zhe Lin, Ning Xu, Yutong Bai, Alan Yuille
TL;DR
MG Matting addresses the burden and limited generalization of trimap-guided matting by using versatile coarse masks with progressive self-guided refinement. Its PRN, guidance perturbations, and foreground-color training strategy support robust performance across guidance types, while the paper reports state-of-the-art results on synthetic and real-world benchmarks.
Problem
Most matting methods require burdensome trimap annotations, while trimap-free methods face generalization challenges on complex real-world images.
Method
MG Matting combines coarse-mask guidance, a Progressive Refinement Network with self-guided decoding, guidance perturbation training, and Random Alpha Blending for foreground-color estimation.
Results
MG Matting achieves state-of-the-art performance on synthetic and real-world datasets and remains robust to multiple guidance-mask types.
Takeaways & Limitations
The framework supports matting pipelines using trimaps, rough segmentation masks, or low-quality alpha mattes while improving foreground-color prediction.
Abstract
from arXiv · showhide
We propose Mask Guided (MG) Matting, a robust matting framework that takes a general coarse mask as guidance. MG Matting leverages a network (PRN) design which encourages the matting model to provide self-guidance to progressively refine the uncertain regions through the decoding process. A series of guidance mask perturbation operations are also introduced in the training to further enhance its robustness to external guidance. We show that PRN can generalize to unseen types of guidance masks such as trimap and low-quality alpha matte, making it suitable for various application pipelines. In addition, we revisit the foreground color prediction problem for matting and propose a surprisingly simple improvement to address the dataset issue. Evaluation on real and synthetic benchmarks shows that MG Matting achieves state-of-the-art performance using various types of guidance inputs. Code and models are available at https://github.com/yucornetto/MGMatting.
1 The Johns Hopkins University 2 Adobe
Figure 1 visually compares MG with other matting methods and Photoshop, showing each method’s guidance input where applicable.
- Figure 1 compares MG with other matting methods, including Photoshop’s commercial matting method.Guidance inputs appear at the bottom-left of each image; BSHM uses an internal segmentation network instead of an external mask.
1. Introduction
The paper addresses the burden and limited applicability of trimap-guided matting by introducing Mask Guided Matting, a robust coarse-mask framework with progressive refinement and improved foreground-color training. It also releases a real-image benchmark and reports state-of-the-art performance on synthetic and real-world datasets.
- 1. Introduction: Mask Guided Matting uses coarse masks of varied quality and form, including trimaps, rough binary masks, and low-quality alpha mattes.The framework is designed to support guidance inputs from different application pipelines.
- 1. Introduction: PRN provides self-guidance that progressively refines uncertain regions during decoding, while perturbation operations improve robustness to external guidance.Training perturbations include random binarization, morphological operations, and CutMask.
- 1. Introduction: Random Alpha Blending addresses noisy and limited foreground-color labels by generating synthetic training data from random alpha mattes and images.The paper reports improved foreground-color prediction without additional manual annotations and more visually plausible compositions.
- 1. Introduction: MG Matting achieves new state-of-the-art performance on synthetic and real-world datasets.
- 1. Introduction: The authors release a high-quality real-image matting benchmark for evaluating real-world model performance.
2. Related Work
Prior work includes trimap-based sampling and propagation methods, trimap-free matting, and foreground-color prediction. MG Matting instead accepts general guidance masks and uses Random Alpha Blending to address foreground-label limitations.
- Trimap-based Image Matting: Trimap-based methods estimate alpha values in unknown regions using foreground/background sampling or affinity-based propagation.
- Trimap-free Image Matting: Trimap-free methods remove external trimaps but face a generalization gap on complex real-world images or rely on specialized automated pipelines.
- Foreground Color Decontamination: Foreground-color methods can suffer from limited training samples and inaccurate, noisy color labels, especially near boundaries.
- MG Matting accepts coarse masks of different qualities and types, allowing use as either trimap-based or trimap-free matting depending on available guidance.
- Random Alpha Blending introduces diverse training samples while avoiding bias from inaccurate boundary color labels.
3. MG Matting
MG Matting replaces the strong trimap assumption with coarse-mask guidance and progressively refines uncertain regions through self-guided decoding. It also addresses foreground-color supervision with Random Alpha Blending and trains for robustness to perturbed guidance.
- 3. MG Matting: MG Matting accepts coarse masks of varying quality and uses PRN to progressively refine uncertain regions during decoding.The framework can use predicted or manually selected masks, while PRN coordinates semantic and low-level feature learning through coarse-to-fine self-guidance.
- 3.1. Progressive Refinement Network: Confident regions are preserved while each decoding level focuses on refining the remaining uncertain region.The previous matte is upsampled, transformed into self-guidance, and selectively fused with the current output; three feature levels progressively shrink uncertainty.
- 3.1. Progressive Refinement Network: The PRN is trained end-to-end with L1, composition, and Laplacian losses, while level-specific losses emphasize the unknown regions.The reported loss combines the three terms, and the implementation uses level weights w0 : w1 : w2 = 1 : 2 : 3 with masks derived from preceding outputs.
- 3.1. Progressive Refinement Network: Guidance perturbations train robustness to noisy external masks through random thresholding, morphology, localized CutMask corruption, and self-guidance dilation.The training pipeline includes random binarization, dilation or erosion, patch overwriting, and dilation of internal self-guidance masks.
- 3.2. Foreground Color Estimation: Random Alpha Blending addresses limited and noisy foreground-color labels by generating diverse composites with accurate, unbiased transparent-region supervision.The method can supervise the whole image and produce smoother foreground-color predictions, while a separate encoder-decoder estimates color from the image and alpha matte.
4. Experiments on Synthetic Datasets
On synthetic benchmarks, MG Matting remains robust across guidance types and improves existing mattes, while PRN and RAB provide complementary gains.
- Experimental setup: The experiments evaluate matting with SAD, MSE, Gradient, and Connectivity errors under synthetic data settings and multiple guidance configurations.The synthetic evaluation includes traditional benchmark testing and refinement of mattes produced by other methods.
- Synthetic benchmark results: The model surpasses previous state-of-the-art methods despite not being trained with trimaps, and remains consistent between trimap and trimapFG guidance.Its foreground color predictions also reduce errors and produce smoother results for noisy alpha mattes.
- Distinction-646: MG Matting consistently improves state-of-the-art methods when used to refine their predicted mattes on Distinction-646.The comparison uses predicted alpha mattes as guidance and evaluates refinement without assuming the guidance type is known.
- Ablation studies: PRN coordinates semantic and low-level detail refinement better than linear side-output fusion, while CutMask further improves performance and robustness.The ablation compares a backbone, deep supervision, fusion convolutions, PRN, and guidance perturbation.
- Foreground color prediction: RAB reduces foreground-color MSE from 0.00623 to 0.00321 and SAD from 82.30 to 62.01 over foreground regions.The improvement is evaluated on Composition-1k using foreground-color errors.
5. Experiments on Real-world Portrait Dataset
On the real-world portrait benchmark, MG Matting is evaluated across whole-image and detail regions and shows strong performance, including under perturbed guidance.
- Dataset: The released benchmark contains 637 diverse, high-resolution images with expert alpha annotations and detail masks for hair and other soft tissues.The detail masks support quantitative evaluation of boundary-detail capture in real images.
- Implementation: Training uses Composition-1k with image degradations including re-JPEGing, Gaussian blur, and Gaussian noise to improve adaptation to real-world noise.The implementation also removes transparent objects from training data to reduce the semantic gap between datasets.
- Real-world portrait results: MG Matting achieves superior real-world portrait performance, especially on hair and soft-detail regions, while LFM performs poorly.The evaluation reports errors separately for the whole image and manually labeled detail regions.
- Real-world portrait results: On a 100-image subset, MG Matting improves over BSHM from MSE 0.0155 to 0.0095 and SAD 10.66 to 8.01 for whole-image evaluation.The comparison is performed after downsampling images to a 720-pixel long side because BSHM supports only low-resolution inputs.
- Guidance robustness: Under mask erosion by 10, 20, and 30 pixels, SAD rises only from 26.8 to 27.1, 27.2, and 27.4, indicating consistent predictions across perturbed guidance.The experiment also tests dilation at the same kernel sizes.
6. Conclusion
The paper presents MG Matting as a general framework for versatile guidance masks, supported by progressive refinement, improved foreground-color learning, and a new real-world dataset.
- Conclusion: MG Matting handles trimaps, rough segmentation masks, and low-quality alpha mattes rather than targeting one specific guidance format.Its robustness comes from self-guidance that progressively refines uncertain regions during decoding.
- Conclusion: The paper combines PRN with Random Rendering for foreground-color estimation and releases a high-quality real-world matting dataset for quantitative evaluation.The dataset is intended to support better benchmarking of real-image matting models.