PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Automatic Natural Image Matting

Jizhizi Li, Jing Zhang, Dacheng Tao

arXiv:2107.07235v1cs.CVcs.AI

TL;DR

Automatic image matting seeks a soft foreground from an image without auxiliary input, but existing methods are limited beyond salient opaque objects. This paper proposes a network using generalized trimap representations and attention, and introduces AIM-500 to evaluate diverse natural images. The authors report that their method performs best across the evaluated metrics and performs well both objectively and subjectively.

  • Problem

    Prior automatic image matting methods are limited to salient opaque foregrounds, leaving transparent/meticulous and non-salient natural images difficult to handle.

    Source: §1 · pp. 1–2

  • Method

    The end-to-end network uses a generalized trimap as unified semantic representation and attention to guide matting toward transition areas, alongside AIM-500, a manually labeled 500-image natural-image test set.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    The authors report that their model achieves the best results across all metrics among AIM methods and generally performs best both objectively and subjectively.

    Source: §4.3 · pp. 5–6

  • Takeaways & Limitations

    AIM-500 covers three natural-image types and many categories, providing a test bed to benchmark AIM models and facilitate future research.

    Source: §1 · p. 2, §5 · p. 6

Abstract

from arXiv · show

Automatic image matting (AIM) refers to estimating the soft foreground from an arbitrary natural image without any auxiliary input like trimap, which is useful for image editing. Prior methods try to learn semantic features to aid the matting process while being limited to images with salient opaque foregrounds such as humans and animals. In this paper, we investigate the difficulties when extending them to natural images with salient transparent/meticulous foregrounds or non-salient foregrounds. To address the problem, a novel end-to-end matting network is proposed, which can predict a generalized trimap for any image of the above types as a unified semantic representation. Simultaneously, the learned semantic features guide the matting network to focus on the transition areas via an attention mechanism. We also construct a test set AIM-500 that contains 500 diverse natural images covering all types along with manually labeled alpha mattes, making it feasible to benchmark the generalization ability of AIM models. Results of the experiments demonstrate that our network trained on available composite matting datasets outperforms existing methods both objectively and subjectively. The source code and dataset are available at https://github.com/JizhiziLi/AIM.

1 Introduction

Automatic image matting aims to extract a soft foreground without user-provided constraints, but prior methods struggle beyond salient opaque objects. The paper proposes a network and a natural-image benchmark to address matting across opaque, transparent/meticulous, and non-salient foregrounds.

1 Introduction

  • The paper frames natural image matting as soft-foreground estimation and notes that traditional methods use pixel sampling or affinity propagation, while deep methods learn representations from labeled data.Auxiliary inputs such as trimaps, scribbles, or background images constrain this ill-posed task.Source: §1 · p. 1
  • Prior automatic matting methods are limited to salient opaque foregrounds, making transparent/meticulous and non-salient images difficult to handle.The paper distinguishes salient opaque (SO), salient transparent/meticulous (STM), and non-salient (NS) foreground types.Source: §1 · pp. 1–2
  • The proposed end-to-end network uses a generalized trimap as unified semantic representation and attention to guide matting toward transition areas.Its customized designs include semantic-decoder SE attention and spatial attention between the semantic and matting decoders.Source: §1 · p. 2
  • AIM-500 provides 500 diverse natural images spanning all three types and categories, with manually labeled alpha mattes for evaluation.The benchmark addresses the lack of a test bed covering different types of natural images.Source: §1 · p. 2
  • The paper’s stated contributions are investigating AIM across image types, designing customized network components, and establishing AIM-500 as a test bed for future research.The network builds on a reference architecture and is intended for images spanning all three foreground types.Source: §1 · p. 2

2 Rethinking the Difficulties of AIM

AIM is challenging because it must estimate foregrounds without auxiliary constraints, generalize from synthetic data to natural images, and handle foregrounds beyond salient opaque objects.

Matting without auxiliary inputs

  • Without trimaps, scribbles, or background images, AIM must infer the holistic foreground-background partition, which may vary across image types.Auxiliary inputs constrain the solution space and let matting models focus on transition areas and local details.Source: §2 · p. 2

Matting on natural images

  • Synthetic training images can contain artifacts and semantic ambiguity, so models may overfit to those cues and generalize poorly to natural images.The text notes the limited supply of available foregrounds and the use of synthesized composites to build more training images.Source: §2 · p. 2

Matting on all types of images

  • Existing AIM methods mainly target salient opaque foregrounds, and extending them to transparent or non-salient images is not straightforward.These image types differ in foreground characteristics, making semantic features difficult to learn without explicit supervision.Source: §2 · pp. 2–3
  • In AIM, “semantic” refers more to foreground-background distinction than to the category of the foreground object.The distinction is difficult to learn for transparent objects and non-salient foregrounds without explicit supervisory signals.Source: §2 · p. 3

3 Proposed Methods

The proposed matting network adapts its backbone and semantic representation for different natural-image foreground types, then uses semantic features to guide matting in transition areas.

3.1 Improved Backbone for Matting

  • The customized ResNet-34 preserves spatial detail while retaining receptive field and outperforms vanilla ResNet-34 on objective metrics and visual quality.The changes keep feature-map dimensions higher and retain pooling indices for the local matting decoder.Source: §3.1 · p. 3

3.2 Unified Semantic Representation

  • The unified representation uses a trimap for salient opaque images, a background-and-transition duomap for transparent or meticulous images, and a whole-image transition unimap for non-salient images.These pure duomap and unimap representations avoid trivial foreground and background pixels that are difficult for the semantic decoder to predict.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3 Guided Matting Process

  • Semantic features generate spatial attention over transition areas to guide the matting decoder; the final alpha combines unified representation U with guided result M and is supervised by alpha, Laplacian, and composition losses.The matting decoder uses a U-Net structure and reversed max-unpooling indices to recover fine structural details.Source: §3.3 · p. 4, Figure 3 · p. 4

4 Experiment

On AIM-500, the proposed model performs best among AIM methods and outperforms DIM* on most metrics, including transition areas and STM/NS images. The experiments also describe the benchmark, training setup, and ablations of the model’s design choices.

4.1 Benchmark: Automatic Image Matting-500

  • AIM-500 contains 500 high-resolution natural images across three foreground types and many categories, with manually labeled alpha mattes.The images include portraits, animals, transparent objects, plants, furniture, toys, and fruit; Figure 4 shows examples.Source: §4.1 · pp. 4–5, Figure 4 · p. 5

4.2 Implementation Details

  • Training combines Comp-1k, HAtt, and AM-2k composites; NS backgrounds receive blur augmentation, and DUTS is used for pretraining before matting-data finetuning.The training set contains 8,710 images.Source: §4.2 · p. 5
  • The model uses multi-scale crops for finetuning and a hybrid test strategy with scale factors 1/3 and 1/4.Training takes about 1.5 days; the supplied passage gives the optimizer and learning rates.Source: §4.2 · p. 5

4.3 Objective and Subjective Results

  • 43.92 vs 52.66 SAD for our model and GFM, respectively; GFM remains the strongest AIM baseline but lags especially on STM and NS images.The authors also report U2NET failures on STM and NS, while DIM can struggle when the trimap region is large.Source: §4.3 · p. 5
  • Our model leads all AIM methods on every metric and beats DIM* on most, including transition areas and STM/NS; overall results are best objectively and subjectively.DIM* uses ground-truth trimaps as an extra input.Source: §4.3 · pp. 5–6, Table 2 · p. 6

4.4 Ablation Study

  • SAD falls 81.08→76.90 with unified representations, 75.90→52.66 with transfer learning, and 52.66→43.92 with all customized designs.The authors report that max-pooling improves visual detail despite higher MSE than the version without it.Source: §4.4 · p. 6, Table 3 · p. 6

5 Conclusion

The conclusion presents the paper’s contributions as a study of AIM across natural-image types, a customized automatic matting network, and the AIM-500 test set.

5 Conclusion

  • The paper investigates AIM without auxiliary inputs, on natural images, and across all image types.The conclusion identifies these as difficulties addressed by the study.Source: §5 · p. 6
  • It introduces duomap and unimap as a unified semantic representation and reports that they are useful.Source: §5 · p. 6
  • AIM-500 is a natural-image test set established to benchmark AIM models and facilitate future research.The conclusion describes it as the first natural-image test set for this purpose.Source: §5 · p. 6

(Supplementary Material)

The supplementary material adds details about the network, AIM-500, visual comparisons, and the ablation analysis.

Supplementary Material

1 More details about our model

The network structure description reports model size, computational complexity, and processing time for an 800×800 image.

1 More details about our model

  • The network has 55.3M parameters.Source: §1.1 · p. 8
  • Its computational complexity is 191.52GMac.Source: §1.1 · p. 8
  • Processing an 800×800 image takes about 0.1633 seconds.Source: §1.1 · p. 8

2 More details about AIM-500

AIM-500 is presented as a natural-image alternative to synthetic validation, which can bias evaluation; it contains natural images with manually labeled alpha mattes.

2.1 Compare with synthetic validation set

  • Synthetic validation can bias evaluation through unreasonable labels when salient objects are pasted over existing ones and through composite artifacts that limit real-world generalization.Earlier synthetic test sets paste high-resolution foregrounds onto backgrounds, including from low-resolution COCO or PASCAL VOC images.Source: §2.1 · p. 8
  • AIM-500 contains natural images with manually labeled alpha mattes and is proposed as a test bed for benchmarking AIM models.Source: §2.1 · p. 8

2.2 More examples of AIM-500

  • AIM-500 spans seven categories, with diverse foregrounds and backgrounds; Figure 1 provides additional examples.Source: §2.2 · p. 8, Figure 1 · p. 8

3 More results of Experiment

On AIM-500, the proposed method achieves the best visual results across all three image types; SE blocks also improve unified semantic representation prediction across those types.

3.1 More visual results on AIM-500

  • The proposed method achieves the best visual results across all three AIM-500 image types, outperforming U2Net, SHM, LF, GFM, and DIM with ground-truth trimaps.U2Net and GFM miss fine detail, while SHM and LF make semantic foreground or background errors; DIM performs worse especially on SO images with large trimap regions.Source: §3.1 · p. 8

3.2 More of ablation study

  • SE blocks benefit unified semantic representation prediction across all three image types.The semantic decoder uses five SE attention modules to recalibrate features and select informative features; Table 1 reports IoU and accuracy, and Figure 3 shows visual results.Source: §3.2 · pp. 8–9, Table 1 · p. 9, Figure 3 · p. 9

Related papers

Loading 2107.07235v1…