PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Composite and Real: Towards End-to-end Deep Image Matting

Jizhizi Li, Jing Zhang, Stephen J. Maybank, Dacheng Tao

arXiv:2010.16188v3cs.CVcs.LGeess.IV

TL;DR

Natural image matting must recover semantic foregrounds and fine details despite limited real-world annotations and the need for extra priors. The paper proposes GFM for collaborative semantic segmentation and detail matting, and RSSN with BG-20k plus real-world benchmarks to address composite-to-real domain gaps. GFM outperforms state-of-the-art models, while RSSN reduces generalization error by 60%.

  • Problem

    Natural image matting remains difficult for furry, diverse foregrounds and has limited real-world data, while composite training can generalize poorly to natural images.

  • Method

    GFM uses a shared encoder and separate Glance and Focus decoders for collaborative semantic segmentation and detail matting, while RSSN and BG-20k reduce composition discrepancies.

  • Results

    GFM outperforms state-of-the-art matting models, and RSSN with BG-20k reduces generalization error by 60%.

  • Takeaways & Limitations

    The benchmark and composition route support evaluating and training matting models with stronger generalization to real-world images.

  • Takeaways & Limitations

    Transition areas remain substantially less accurate than foreground and background regions, and composite-based models still have room to match models trained on original images.

Abstract

from arXiv · show

Extracting accurate foregrounds from natural images benefits many downstream applications such as film production and augmented reality. However, the furry characteristics and various appearance of the foregrounds, e.g., animal and portrait, challenge existing matting methods, which usually require extra user inputs such as trimap or scribbles. To resolve these problems, we study the distinct roles of semantics and details for image matting and decompose the task into two parallel sub-tasks: high-level semantic segmentation and low-level details matting. Specifically, we propose a novel Glance and Focus Matting network (GFM), which employs a shared encoder and two separate decoders to learn both tasks in a collaborative manner for end-to-end natural image matting. Besides, due to the limitation of available natural images in the matting task, previous methods typically adopt composite images for training and evaluation, which result in limited generalization ability on real-world images. In this paper, we investigate the domain gap issue between composite images and real-world images systematically by conducting comprehensive analyses of various discrepancies between the foreground and background images. We find that a carefully designed composition route RSSN that aims to reduce the discrepancies can lead to a better model with remarkable generalization ability. Furthermore, we provide a benchmark containing 2,000 high-resolution real-world animal images and 10,000 portrait images along with their manually labeled alpha mattes to serve as a test bed for evaluating matting model's generalization ability on real-world images. Comprehensive empirical studies have demonstrated that GFM outperforms state-of-the-art methods and effectively reduces the generalization error. The code and the datasets will be released at https://github.com/JizhiziLi/GFM.

1 Introduction

Natural image matting must identify semantic foregrounds and recover fine transition details without user-provided trimaps or scribbles. The paper addresses this with GFM, a unified semantic-and-detail model, and RSSN plus new benchmarks to improve real-world generalization.

  • Natural image matting requires both hard foreground/background labels and soft transition-area labels, making furry and visually diverse subjects difficult to extract without trimaps or scribbles.
  • Existing end-to-end methods either perform global segmentation before local matting or provide global guidance, but sequential errors may persist and joint foreground/background-transition prediction remains challenging.
  • GFM uses a shared encoder with separate Glance and Focus decoders to learn semantic segmentation and detail matting collaboratively in one end-to-end network.
  • Limited real-world annotations motivate composite training, but mismatches between foregrounds and backgrounds can create domain gaps and poor natural-image generalization.
  • RSSN and BG-20k reduce composition discrepancies, while experiments report a 60% reduction in generalization error and performance comparable to models trained on original natural images.
  • The benchmark contains 2,000 high-resolution animal images and 10,000 portrait images with manually labeled alpha mattes for evaluating real-world generalization.

2 Related Work

Prior matting methods commonly rely on auxiliary inputs or composited training data, while available real-world datasets remain small or low quality. GFM instead targets end-to-end matting without a coarse trimap through a shared encoder and two decoders.

  • Classical methods use trimaps or scribbles, while CNN-based approaches learn discriminative features for matting.These methods typically propagate labels into unknown regions or predict from learned representations.
  • Existing end-to-end methods commonly perform global segmentation and local matting sequentially.Global segmentation generates a trimap or foreground/background estimate that guides the later matting stage.
  • GFM differs from AdaMatting by addressing segmentation and matting simultaneously without requiring a coarse trimap.The supplied passage identifies GFM as a multi-task model with a shared encoder and two decoders.
  • Existing datasets contain few annotated alpha mattes, while DAPM provides fixed-size, low-resolution portrait images.Examples include 27 training and 8 test images in alphamatting, 431 and 50 in Comp-1k, and 596 and 50 in HAttMatting.
  • Image composition is used to generate synthetic matting data, and prior work improves realism through color, lighting, texture, and geometric compatibility.The passage frames composition as important for image editing and synthetic dataset generation.

3 GFM: Glance and Focus Matting Network

GFM decomposes natural image matting into semantic segmentation and detail matting, using a shared encoder with separate glance and focus decoders. Their predictions are combined through collaborative matting to produce the final alpha matte.

  • Architecture: GFM follows a glance-then-focus formulation that separates rough semantic recognition from transition-area detail extraction.The formulation mirrors recognizing salient foreground/background regions before resolving ambiguous transition areas.
  • Architecture: A shared encoder processes one image through five resolution-reducing blocks and feeds both decoders.The encoder can use DenseNet-121, ResNet-34, or ResNet-101 backbones pretrained on ImageNet.
  • Glance Decoder: The glance decoder uses a large receptive field and pyramid pooling to learn high-level semantics for easy regions and unknown-area identification.Its output is a two- or three-channel class-probability map determined by the semantic-transition representation.
  • Focus Decoder: The focus decoder uses dilated convolutions and encoder-decoder shortcuts to preserve local structural details in transition areas.Its training loss combines transition-region alpha prediction and Laplacian terms.
  • Focus Decoder: The focus alpha loss measures absolute differences between ground-truth and predicted alpha mattes in unknown transition regions.The transition-region indicator identifies which pixels receive this supervised loss, with ε = 10^-6 added for computational stability.
  • RoSTa: GFM links its decoders through three semantic-transition representations: trimap-based GFM-TT, foreground-based GFM-FT, and background-based GFM-BT.These variants define the glance supervision and transition area differently using dilation, erosion, or foreground/background masks.
  • Collaborative Matting: Collaborative matting combines glance and focus predictions, with replacement, addition, or subtraction rules selected by the RoSTa variant.The glance decoder handles rough foreground/background recognition, while the focus decoder handles details in unknown areas.
  • Collaborative Matting: The final training objective sums the glance-decoder loss, focus-decoder loss, and collaborative-matting loss.The collaborative loss includes alpha-prediction, Laplacian, and composition components.

4 RSSN: A Novel Composition Route

RSSN is a composition route designed to reduce resolution, semantic, sharpness, and noise discrepancies between foreground and background images, improving the realism and domain alignment of composites.

  • Composite images can contain artifacts because foregrounds and backgrounds are sampled from different distributions, creating a domain gap and encouraging overfitting to cheap composition cues.
  • MS COCO and PASCAL VOC backgrounds introduce resolution discrepancy and semantic ambiguity because they are low-resolution and may contain salient objects.
  • BG-20k provides high-resolution backgrounds without salient foreground objects to reduce resolution discrepancy and semantic ambiguity during composition.Images are selected with shortest sides of at least 1080 pixels, and salient-object images are removed.
  • RSSN computes foregrounds using the method of Levin et al. before alpha blending, avoiding the visible color artifacts produced by directly multiplying the alpha matte with the original image.
  • RSSN simulates shallow depth of field by randomly blurring backgrounds, while denoising foregrounds and backgrounds and adding Gaussian noise to composites to reduce sharpness and noise discrepancies.Background blurring uses averaging filters with randomly selected kernel sizes and is applied with probability 0.5.
  • The RSSN pipeline samples K = 5 BG-20k backgrounds per foreground and outputs composite images for training.

5.1 Benchmark for Real-world Image Matting

The benchmark evaluates matting on real-world animal and portrait images and separately measures performance when models are trained on composite images and tested on natural images.

  • The benchmark addresses the domain gap by evaluating models trained and tested on real-world images and models trained on composites but tested on real-world images.
  • AM-2k: AM-2k contains 2,000 high-resolution real-world animal images spanning 20 categories, with manually annotated alpha mattes and diverse appearances and backgrounds.
  • PM-10k: PM-10k contains 10,000 high-resolution portrait images with multiple postures and diverse backgrounds, processed to ensure clear and salient humans.
  • Evaluation tracks: ORI-Track is the primary benchmark for end-to-end matting on original real-world images.
  • Evaluation tracks: COMP-Track investigates whether reducing composition-induced domain gaps improves generalization from composite training images to natural images.

5.2 Evaluation Metrics and Implementation Details

The evaluation uses pixel-wise, detail-sensitive, semantic-region, and model-complexity measures, with standardized augmentation and implementations for compared methods.

  • Evaluation metrics: MSE, SAD, gradient, and connectivity are the major metrics, with MSE and SAD measuring pixel-wise differences while gradient and connectivity favor clear details.
  • Evaluation metrics: MAD, SAD-TRAN, SAD-FG, and SAD-BG provide auxiliary measures for overall error and errors in transition, foreground, and background regions.
  • Evaluation metrics: Model complexity is compared using parameter count, computational complexity, and inference time.
  • Implementation details: Training uses randomly selected 640 × 640, 960 × 960, or 1280 × 1280 crops resized to 320 × 320, with random horizontal flipping.
  • Implementation details: Baseline methods use official implementations when available and paper-based reimplementations otherwise, with ResNet-34 used for SHMC for fair comparison.

5.3 Quantitative and Subjective Evaluation

GFM consistently outperforms prior matting methods on real-image evaluation, while RSSN substantially improves composite-image generalization by reducing composition discrepancies.

  • Results on the ORI-Track: GFM achieves the best performance across RoSTa representations and encoder backbones by jointly segmenting foregrounds, backgrounds, and transition areas.The model outperforms prior methods on both AM-2k and PM-10k ORI-Track evaluations.
  • Results on the ORI-Track: GFM reduces SAD in the transition area to 8.24 versus 10.26 on AM-2k and 7.70 versus 8.53 on PM-10k against SHM.It also achieves lower foreground and background SAD values than SHM on both datasets.
  • Results on the ORI-Track: The unified model preserves semantic recognition and transition details better than stage-wise and globally guided methods in subjective comparisons.The cited comparisons report missed foreground parts for several baselines and unclear transition details for globally guided methods.
  • Results on the COMP-Track: BG-20k improves composite training by reducing resolution discrepancy and semantic ambiguity in background regions.The reported reductions are from 182.70 to 52.36 for AM-2k and from 168.75 to 34.06 for PM-10k.
  • Results on the COMP-Track: RSSN further reduces SAD for SHM from 52.36 to 23.94 on AM-2k and from 34.06 to 22.02 on PM-10k.Its composition route simulates large-aperture blur, removes source noise, and adds noise to composites.
  • Results on the COMP-Track: Trimap-based methods follow the same performance trends across original-image and composite-image tracks, with gaps depending on the method.The evaluation compares training with MS COCO backgrounds against training with original dataset backgrounds.
  • GFM with Trimap-based Matting Method: GFM remains stronger than the GFM+DIM variant in the transition area, with SAD 8.24 versus 9.54 for GFM(d).This supports the effectiveness of GFM’s foreground-detail decoder and collaboration module.

5.4 Ablation Study

Ablation studies show that GFM’s two-decoder decomposition and semantic-context components are important, while RSSN’s composition techniques work jointly to improve performance.

  • Results on the ORI-Track: Replacing GFM’s two decoders with a single decoder increases SAD from 10.26 to 13.79 for GFM-TT(d) and from 10.89 to 15.50 for GFM-TT(r).This supports decomposing end-to-end matting into collaborative semantic and detail sub-tasks.
  • Results on the ORI-Track: Removing the pyramid pooling module increases SAD from 10.26 to 10.86 for GFM-TT(d) and from 10.89 to 11.90 for GFM-TT(r).The study attributes this to the value of larger-receptive-field global context for semantic segmentation.
  • Results on the ORI-Track: The r2b encoder variant improves all metrics over GFM-TT(r) but uses more parameters and computations.The variant enlarges early feature maps through altered convolution, pooling, and additional encoder layers.
  • Results on the COMP-Track: RSSN combines blur simulation, source denoising, and composite-image noise addition as complementary techniques.The ablation compares each technique individually with their full combination using BG-20k backgrounds.

5.5 RoSTa Integration and Hybrid-resolution Test

RoSTa integration improves matting quality, while hybrid-resolution testing balances global-context and detail processing against inference speed.

  • RoSTa Integration: Ensembling RoSTa outputs with the median improves all evaluation metrics over individual representations but substantially slows testing.The reported SAD comparison is 10.62 for EN-median versus 11.28, 11.76, and 12.26 for individual RoSTa variants.
  • RoSTa Integration: GFM-RIM achieves better results than each individual RoSTa on four metrics and reaches MSE 0.0025 versus 0.0026 for EN-median.It has comparable test speed to individual RoSTa models and is faster than EN-median.
  • Hybrid-resolution Test: Hybrid-resolution testing performs best with d1 = 3 and d2 = 2, but slower inference is the trade-off.A larger d1 benefits the glance decoder’s receptive field, while a smaller d2 benefits the focus decoder.
  • Hybrid-resolution Test: Using the same ratio for both decoders gives the best performance at d1 = d2 = 2.6.This setting balances the two decoder resolutions while resizing the test image accordingly.

5.6 Model Complexity Analysis

The complexity analysis compares model size, computational cost, and inference time on 800 × 800 inputs to assess efficiency against existing methods.

  • Model Complexity Analysis: GFM variants with DenseNet-121, ResNet-34, or ResNet-101 are evaluated against prior methods using parameters, GMac, and inference time.All methods are measured on the same hardware and image resolution.

6 Conclusion and Future Works

The paper presents GFM as an end-to-end matting model that jointly learns semantic recognition and detail extraction, while identifying remaining limitations in transition-area accuracy, composite-to-natural domain transfer, and dataset coverage.

  • Contributions: GFM uses Glance and Focus decoders with a shared encoder to jointly recognize diverse foregrounds and extract details from ambiguous backgrounds.The decoders are trained jointly to accomplish end-to-end natural image matting.
  • Limitations: Although GFM outperforms state-of-the-art methods, transition-area errors remain larger than foreground and background errors.Reported errors are 8.24 versus 2.01 or 7.80 versus 4.09 for the compared regions.
  • Limitations: Composite-based models still have room to match models trained on original images, motivating domain adaptation because composite datasets are easier to generate.The paper frames this as an unresolved domain discrepancy issue rather than a completed solution.
  • Future work: Future work includes improving Focus Decoder design, structure-aware and perceptual losses, decoder interaction, and adaptation to different matting image types.The paper also suggests investigating new Glance Decoder representations for explicit transition areas.
  • Future work: Current natural-image datasets focus on salient human or animal foregrounds with small transition areas, leaving semi-transparent and long-range transitions as an open challenge.Examples include plastic bags and raindrops.
Loading 2010.16188v3…