PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Human Matting

Quan Chen, Tiezheng Ge, Yanyu Xu, Zhiqiang Zhang, Xinxin Yang, Kun Gai

arXiv:1809.01354v2cs.CVcs.GRcs.LGstat.ML

TL;DR

Automatic human matting remains difficult because its decomposition is severely under-constrained and typically requires user constraints. SHM jointly learns semantic information and fine details through a fusion-based network and a large annotated dataset, achieving results comparable to state-of-the-art interactive matting methods. The method also generalizes to natural human images.

  • Problem

    Human matting is severely under-constrained, and existing methods commonly require user trimaps or scribbles, limiting large-scale and time-sensitive use.

  • Method

    SHM integrates a semantic segmentation module, a deep matting module, and a fusion strategy that combines coarse semantics with fine details for automatic alpha estimation.

  • Results

    SHM achieves comparable results with state-of-the-art interactive matting methods and generalizes to natural human images.

  • Takeaways & Limitations

    A learned semantic constraint can support automatic human matting while retaining high-quality detail extraction across benchmark and natural images.

  • Takeaways & Limitations

    The fusion loss remains under-constrained, so SHM adds a trimap classification loss to obtain a stable solution.

Abstract

from arXiv · show

Human matting, high quality extraction of humans from natural images, is crucial for a wide variety of applications. Since the matting problem is severely under-constrained, most previous methods require user interactions to take user designated trimaps or scribbles as constraints. This user-in-the-loop nature makes them difficult to be applied to large scale data or time-sensitive scenarios. In this paper, instead of using explicit user input constraints, we employ implicit semantic constraints learned from data and propose an automatic human matting algorithm (SHM). SHM is the first algorithm that learns to jointly fit both semantic information and high quality details with deep networks. In practice, simultaneously learning both coarse semantics and fine details is challenging. We propose a novel fusion strategy which naturally gives a probabilistic estimation of the alpha matte. We also construct a very large dataset with high quality annotations consisting of 35,513 unique foregrounds to facilitate the learning and evaluation of human matting. Extensive experiments on this dataset and plenty of real images show that SHM achieves comparable results with state-of-the-art interactive matting methods.

1 INTRODUCTION

SHM addresses the under-constrained nature of automatic human matting by combining semantic information with fine details and avoiding user-specified constraints. It introduces a fusion strategy, a large annotated dataset, and achieves results comparable to interactive matting methods.

  • Motivation: Human matting is under-constrained, so conventional methods often require user-provided trimaps or scribbles.Semantic segmentation alone blurs structural details, while interactive matting is unsuitable for data-intensive or time-sensitive scenarios.
  • Approach: SHM integrates semantic segmentation with deep matting to automatically extract human alpha mattes.Learned semantic information acts as an implicit constraint for detailed alpha estimation.
  • Approach: A fusion strategy adaptively combines coarse semantic and fine-detail predictions per pixel for probabilistic alpha estimation and end-to-end training.The strategy combines high- and low-level results while allocating training error to coarse and fine outputs.
  • Dataset: 35,513 distinct human foregrounds were collected with fine matte annotations, producing 52,511 training and 1,400 testing images.The dataset expands the volume and diversity available for human matting training and evaluation.
  • Results: SHM achieves comparable results to state-of-the-art interactive matting methods and generalizes to natural human images.The authors report extensive experiments on the constructed dataset and real images.

2 RELATED WORKS

Prior matting methods either emphasize coarse semantic segmentation or fine-detail extraction, while automatic systems combine these capabilities with varying limitations. SHM-related work uses learned semantic constraints and a fusion strategy to support end-to-end detail-aware matting.

  • Semantic segmentation captures coarse pixel-level categories but can blur structural details important for human matting.
  • General matting methods recover alpha mattes through sampling, propagation, or CNNs using color and low-level features.
  • Automatic matting systems reduce user interaction, but prior approaches target portrait images or rely on traditional matting modules.
  • The related-work figures and tables introduce existing matting datasets and compositional examples used for comparison.
  • SHM uses semantic information as an implicit constraint for a deep matting network, with fusion explicitly combining coarse semantics and fine details.

3 HUMAN MATTING DATASET

The human matting dataset addresses limited and narrow prior data by combining carefully annotated human foregrounds with composited backgrounds. Its scale, coverage, and manual quality control support training and evaluation of SHM.

  • The dataset contains human foregrounds with accessories, carefully annotated alpha mattes, and 52,511 composited images.
  • Fashion Model data contribute more than 188k collected images, manually inspected annotations, and 35,311 selected images after quality review.
  • The dataset compares existing matting datasets through a configuration table covering its human-matting data construction.
  • With 35,513 different foregrounds, the dataset is substantially larger than prior datasets and includes human objects beyond portrait-only coverage.
  • All 35,513 foreground objects are manually annotated and carefully inspected, supporting high-quality alpha mattes and semantic integrity.

4 OUR METHOD

SHM automatically extracts human alpha mattes by jointly learning coarse semantic regions and fine details. Its T-Net, M-Net, and Fusion Module are trained together to combine semantic probabilities with detail predictions, while auxiliary losses and staged pre-training stabilize learning.

  • Overall pipeline: SHM takes an RGB image and directly outputs a same-sized one-channel alpha matte without trimaps or scribbles.The method targets humans as a specific semantic pattern.
  • T-Net: T-Net predicts per-pixel foreground, background, and unknown-region probabilities to provide a semantic hint for matting.The implementation uses PSPNet-50, while the T-Net can generally use semantic segmentation networks.
  • M-Net: M-Net receives the RGB image and T-Net's 3-channel segmentation output as a 6-channel input, then generates a raw alpha matte emphasizing structural and textural details.It is a deep convolutional encoder-decoder with 13 encoder convolutional layers, 4 max-pooling layers, 6 decoder convolutional layers, and 4 unpooling layers.
  • Fusion Module: The Fusion Module combines semantic foreground probability with the M-Net result, weighting the detail prediction according to the unknown-region probability.When unknown-region probability is high, the fused output approaches the M-Net prediction; when it is low, it approaches the semantic foreground estimate.
  • Training objective: The loss combines alpha-prediction and compositional errors over the entire image with a classification loss that keeps the trimap meaningful.The alpha/compositional weighting parameter γ is 0.5, while the decomposition constraint uses a small λ such as 0.01.
  • Training procedure: Training first pre-trains T-Net and M-Net separately, then fine-tunes the complete network end to end on the human matting dataset.The training procedure uses crop, resize, rotation, flipping, and trimap-width augmentation to improve robustness and detail learning.

5 EXPERIMENTS

Experiments evaluate SHM on the human matting dataset against segmentation, regression, and interactive matting baselines, then analyze its components and real-image behavior. SHM benefits from end-to-end training and fusion of coarse semantic predictions with fine matting details.

  • Experimental Setup: The evaluation uses four alpha-matte metrics on a dataset containing 52,511 training images and 1,400 testing images.The metrics are SAD, MSE, Gradient error, and Connectivity error.
  • Experimental Setup: Interactive baselines receive trimaps predicted by the pretrained T-Net, while PSP50 Seg and PSP50 Reg provide segmentation and regression comparisons.Compared interactive methods include CF, KNN, DCNN, IFM, and DIM.
  • Performance Comparison: SHM outperforms the designed baselines because probabilistic fusion coordinates coarse semantics and fine details for end-to-end training.Binary segmentation and regression perform poorly, while DIM is stronger among the interactive methods using the same predicted trimaps.
  • Performance Comparison: Visual comparisons show that SHM recovers sharper details, including hair, while producing fewer semantic errors than other methods.These comparisons are shown on the semantic human matting testing dataset.
  • Automatic Method vs. Interactive Methods: Using only RGB images, SHM is slightly inferior to TrimapGT+DIM in test loss but visually comparable, despite that baseline receiving groundtruth trimaps.The T-Net estimates coarse human structure, which M-Net complements with matting details.
  • Evaluation and Analysis of Different Components: End-to-end training and the fusion module improve performance, while the trimap constraint makes the trimap more meaningful and the decomposition more stable.The fusion module combines coarse T-Net estimates with fine M-Net predictions; removing it or end-to-end training reduces performance relative to SHM.
  • Real-Image Generalization: On real images with complicated backgrounds, SHM recovers hair and fingers more effectively than other methods and supports visually high-quality compositions.Intermediate visualizations show T-Net predictions, M-Net alpha mattes, and their fusion result.

6 CONCLUSION

The conclusion presents SHM as an automatic human matting system that combines semantic information and local details through a trimap network, a matting network, and a fusion module. It reports comparable results with state-of-the-art interactive matting methods.

  • 6 CONCLUSION: SHM cascades a trimap network and a matting network with a fusion module to automatically generate human alpha mattes.The model structure is designed to capture global semantic information and local details.
  • 6 CONCLUSION: SHM achieves results comparable to state-of-the-art interactive matting methods.
Loading 1809.01354v2…