PaperlayerView exact revision on arXiv

Source-linked AI summary

MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition

Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan, Rynson W. H. Lau

arXiv:2011.11961v4cs.CV

TL;DR

Portrait matting needs precise alpha mattes, but existing trimap-free methods often depend on costly auxiliary inputs or computationally expensive multi-stage pipelines and may generalize poorly to real-world data. MODNet addresses this with a lightweight single-image network that jointly optimizes decomposed sub-objectives using explicit constraints, e-ASPP, and SOC. It runs at 67 frames per second and outperforms prior trimap-free methods on Adobe Matting and PPM-100 benchmarks.

  • Problem

    Existing trimap-free portrait matting methods require costly auxiliary inputs or multiple computationally expensive stages, and often suffer domain shift on real-world data.

  • Method

    MODNet uses a single RGB image, jointly optimizes semantic, detail, and fusion sub-objectives with explicit constraints, and adds e-ASPP and self-supervised SOC.

  • Results

    MODNet runs at 67 frames per second on a GTX 1080Ti and achieves state-of-the-art results on the Adobe Matting and PPM-100 benchmarks.

  • Takeaways & Limitations

    MODNet provides real-time trimap-free portrait matting with one RGB image and is reported to generalize better to new data domains through SOC.

Abstract

from arXiv · show

Existing portrait matting methods either require auxiliary inputs that are costly to obtain or involve multiple stages that are computationally expensive, making them less suitable for real-time applications. In this work, we present a light-weight matting objective decomposition network (MODNet) for portrait matting in real-time with a single input image. The key idea behind our efficient design is by optimizing a series of sub-objectives simultaneously via explicit constraints. In addition, MODNet includes two novel techniques for improving model efficiency and robustness. First, an Efficient Atrous Spatial Pyramid Pooling (e-ASPP) module is introduced to fuse multi-scale features for semantic estimation. Second, a self-supervised sub-objectives consistency (SOC) strategy is proposed to adapt MODNet to real-world data to address the domain shift problem common to trimap-free methods. MODNet is easy to be trained in an end-to-end manner. It is much faster than contemporaneous methods and runs at 67 frames per second on a 1080Ti GPU. Experiments show that MODNet outperforms prior trimap-free methods by a large margin on both Adobe Matting Dataset and a carefully designed photographic portrait matting (PPM-100) benchmark proposed by us. Further, MODNet achieves remarkable results on daily photos and videos. Our code and models are available at https://github.com/ZHKKKe/MODNet, and the PPM-100 benchmark is released at https://github.com/ZHKKKe/PPM.

Introduction

MODNet targets real-time trimap-free portrait matting with one RGB image, avoiding costly auxiliary inputs and sequential multi-stage pipelines. It combines objective decomposition, efficient feature fusion, and self-supervised consistency, achieving fast inference and strong benchmark performance.

  • Introduction: Existing trimap-free methods are less suitable for real-time use because they require auxiliary inputs or multiple sequential stages.Background-image approaches require capturing and aligning two photos, while multi-stage pipelines increase inference time.
  • Introduction: MODNet uses one RGB image and explicit constraints to optimize matting sub-objectives simultaneously in a single stage.The design shares intermediate representations rather than accumulating errors across sequential stages.
  • Introduction: e-ASPP provides efficient multi-scale feature fusion, while SOC improves robustness to real-world domain shift.SOC addresses limited generalization caused by insufficient labeled training data.
  • Introduction: 67 frames per second is achieved on a GTX 1080Ti with 512 × 512 input, including data loading.MODNet is reported to be much faster than previous trimap-free methods.
  • Introduction: MODNet achieves state-of-the-art results on the Adobe Matting benchmark and the proposed PPM-100 benchmark.The authors evaluate methods under a shared training and validation environment to support fair comparison.
  • Introduction: The authors release MODNet code, a pretrained model, and the PPM-100 benchmark.The benchmark is designed for portrait matting evaluation, while the implementation and models are publicly available.

Related Works

Image matting estimates per-pixel foreground opacity from an image, but the problem is ill-posed without additional information. Prior trimap-free systems use extra inputs or multiple models, whereas MODNet decomposes the objective into jointly optimized semantic, detail, and fusion sub-objectives.

  • Image Matting: Image matting predicts a per-pixel alpha value to extract a desired foreground from an image.The image is modeled using foreground, background, and alpha components.
  • Image Matting: The matting problem is ill-posed because the foreground, background, and alpha variables are all unknown.Trimaps constrain estimation by marking absolute foreground, absolute background, and an unknown region.
  • Related Works: Traditional methods rely on low-level color cues through sampling or propagation, which often fail in complex scenes.These approaches use image evidence to determine or propagate alpha values.
  • Trimap-Free Portrait Matting: Trimap-free methods avoid costly user-created trimaps but may use background images or multiple stages, limiting interactive applications.Background changes across frames and multi-stage computation make such pipelines unsuitable for interactive use.
  • Trimap-Free Portrait Matting: MODNet divides matting into semantic estimation, detail prediction, and semantic-detail fusion, optimized simultaneously through three branches.The branches are interdependent and support end-to-end optimization.
  • Other Techniques: Unlike consistency methods using multiple models, MODNet imposes consistency among sub-objectives within one model.This design supports consistency constraints across the model’s internal predictions.

MODNet

MODNet decomposes trimap-free portrait matting into semantic estimation, detail prediction, and semantic-detail fusion, optimizing the three sub-objectives simultaneously in one end-to-end model. Its lightweight design uses MobileNetV2 and e-ASPP to support real-time inference while preserving matting performance.

  • MODNet: MODNet divides matting into semantic estimation, detail prediction, and semantic-detail fusion optimized simultaneously through three branches.The model trains end-to-end using the summed semantic, detail, and alpha losses.
  • Semantic Estimation: MobileNetV2 provides efficient high-level semantic features that support subsequent branches and joint optimization.The semantic branch uses an encoder without a separate decoder, reducing computational requirements.
  • Semantic Estimation: e-ASPP extracts and fuses multi-scale features depth-wise, followed by inter-channel fusion, for efficient semantic processing.It modifies standard ASPP through depth-wise atrous convolutions, point-wise convolutions, and channel compression.
  • Semantic Estimation: 1% of the original ASPP parameters and 1% of its computational overhead are used by e-ASPP, with comparable performance in MODNet experiments.The comparison is against the original ASPP.
  • Detail Prediction: The detail branch predicts a boundary detail matte from the image, semantic features, and low-level features, focusing its L1 supervision on portrait boundaries.The boundary mask is generated by dilation and erosion of the ground-truth alpha matte.
  • Semantic-Detail Fusion: The fusion branch upsamples semantic features, concatenates them with detail features, and predicts the final alpha matte using compositional supervision.The compositional loss measures the absolute difference between the input and an image composited from predicted alpha, foreground, and background.

SOC for Real-World Data

MODNet addresses domain shift in real-world portrait matting with self-supervised sub-objectives consistency. The method is motivated by limited and difficult portrait-matting annotations and is evaluated with PPM-100, a diverse benchmark using original backgrounds.

  • Motivation: Portrait-matting datasets are usually small because accurate hair-region labeling on natural images requires experienced annotators and substantial time.Even images with monochromatic or blurred backgrounds remain difficult to annotate precisely.
  • SOC: SOC adapts MODNet to unseen data distributions by imposing consistency constraints among predictions of its semantic, detail, and fusion sub-objectives without ground-truth labels.The semantic content of the alpha prediction should agree with the semantic prediction, while its details should agree with the detail prediction.
  • Benchmark: PPM-100 contains 100 finely annotated portrait images with varied original backgrounds and balanced foreground conditions.Its diversity includes full-body coverage, background blur, and additional objects held by the person.
  • SOC: Because direct consistency losses can smooth alpha boundaries, SOC duplicates MODNet with a frozen copy and adds regularization to preserve fine details.The combined consistency and detail-preservation losses are optimized during SOC.

Experiments

Experiments evaluate MODNet against trimap-free and trimap-based methods on synthetic and photographic benchmarks, then assess efficiency, ablations, and real-world video adaptation. MODNet performs strongly on photographic data and efficiency while remaining vulnerable to difficult poses, costumes, and motion-related video artifacts.

  • PPM-100 benchmark: PPM-100 contains 100 finely annotated portraits with diverse backgrounds, whole-body coverage, background blur, held objects, natural backgrounds, and richer postures.
  • PPM-100 results: MODNet outperforms other trimap-free methods on both MSE and MAD on PPM-100, but does not outperform trimap-based methods there.With trimaps supplied during training and testing, MODNet outperforms the compared trimap-based methods.
  • AMD results: Trimap-free models approach trimap-based performance on synthetic AMD composites only when foreground-background fusion is unnatural or semantics are mismatched.
  • Efficiency: 14.9 ms (67 fps) is MODNet’s inference time on a GTX 1080Ti with 512×512 inputs, twice the fastest comparison method’s 31 fps, while using the fewest trimap-free parameters.
  • Ablation: Ablations show that semantic and boundary constraints improve performance, while e-ASPP effectively fuses multilevel feature maps with lower overhead.SPS pre-training is optional for MODNet but important for other trimap-free methods.
  • Real-world video: SOC adapts MODNet using approximately 400 unlabeled video clips totaling about 50,000 frames, and OFD further removes boundary flickers by using adjacent frames.

Conclusion

MODNet performs real-time portrait matting from a single RGB image through objective decomposition and concurrent explicit supervision. Its e-ASPP and SOC techniques support efficient feature fusion and domain adaptation, while strong motion blur remains a limitation for video matting.

  • MODNet predicts high-quality alpha mattes in real time from one RGB image through objective decomposition and concurrent optimization with explicit supervisions.
  • The method introduces e-ASPP for efficient multi-scale feature fusion and SOC for handling domain shift.
  • Experiments report better performance than existing trimap-free methods on AMD, PPM-100, and various real-world data.
  • MODNet may fail on videos with strong motion blur because it lacks temporal information.

Appendix A: Analysis of e-ASPP

The appendix analyzes standard ASPP and e-ASPP through their parameter and FLOPs formulations. With k = 3 and m = 5, e-ASPP is reported to retain comparable performance while substantially reducing computational cost.

  • Standard ASPP: Standard ASPP independently extracts multi-scale features, concatenates them, and applies a point-wise convolution for fusion.Its parameter and FLOPs calculations use the input feature dimensions, atrous kernel size, and number of atrous layers.
  • Efficient ASPP: e-ASPP uses Channel Reduction, Multi-Scale Feature Extraction, Multi-Scale Feature Fusion, and Inter-Channel Feature Fusion.The total parameter count and FLOPs are the sum of these four operations.
  • Comparison setup: The appendix sets k = 3 and m = 5 for comparing e-ASPP with standard ASPP.It notes that c ≥ 256 is usually used in most networks.
  • Efficiency comparison: 1% of the parameters and 1% of the computational overhead are required by e-ASPP relative to standard ASPP.When Channel Reduction is disabled, e-ASPP still uses only 2% of the parameters and 2% of the computational overhead.
  • Efficiency comparison: MODNet experiments show that e-ASPP achieves performance comparable to standard ASPP.The reported efficiency comparison concerns both parameters and computational overhead.

Appendix B: Results on CRGNN-R and D646

The appendix reports evaluations on CRGNN-R and D646 and identifies the compared methods and datasets. It also references visual comparisons with existing trimap-free methods on PHM-100.

  • CRGNN-R: Table 4 reports quantitative results on the CRGNN-R video matting dataset proposed by Wang et al. 2021.The table is used to show the effectiveness of the proposed SOC strategy.
  • D646: Table 5 compares MODNet with previous state-of-the-art methods on the D646 dataset proposed by Qiao et al. 2020.The passage identifies the dataset and comparison purpose but provides no numerical values.
  • PHM-100: Figure 8 compares MODNet with DIM, FDMPA, LFM, SHM, HAtt, and BSHM on PHM-100.DIM is evaluated without trimaps and is pre-trained on the SPS dataset.

Appendix C: Visual Results on PHM-100

The appendix points to additional visual comparisons of MODNet and existing trimap-free methods on PHM-100. It also lists D646 results in Table 5.

  • PHM-100 visual comparisons: Figure 8 provides more visual comparisons of MODNet and existing trimap-free methods on PHM-100.The cited passage introduces the figure but does not report a specific visual outcome.
  • PHM-100 visual comparisons: The PHM-100 comparison is framed around MODNet and existing trimap-free methods.No individual method names or ranking are stated in the supplied passages for this section.
  • D646 results: Table 5 presents results on the D646 dataset proposed by Qiao et al. 2020.The supplied passage provides the table caption but no numerical or qualitative result.

Appendix D: Comparison with BM

The appendix compares MODNet with background matting in fixed-camera scenes because BM does not support dynamic backgrounds. MODNet is reported to remain robust when a moving object enters the background.

  • Evaluation setting: BM does not support dynamic backgrounds, so the comparison uses fixed-camera scenes from Sengupta et al. 2020.BM relies on a static background image for its operation.
  • Visual comparison: MODNet outperforms BM when a car enters the background in the fixed-camera comparison.The entering car is identified as the red region in Figure 9.
  • Background matting assumption: BM implicitly assumes that pixels changing across frames belong to the foreground.This assumption follows from its reliance on a static background image.
  • Visual comparison: MODNet is robust to disturbances caused by a moving object suddenly appearing in the background.The corresponding BM result is affected in the same situation.
Loading 2011.11961v4…