Source-linked AI summary
Deep Image Matting
Ning Xu, Brian Price, Scott Cohen, Thomas Huang
TL;DR
Image matting remains difficult in unconstrained scenes because prior methods rely heavily on low-level color cues and limited datasets. This paper learns alpha mattes directly from images and trimaps with a two-stage deep network and trains it on a large composited dataset. The method outperforms prior approaches on standard and synthetic evaluations and generalizes better to real images, although benchmark rankings may not track human judgments.
Problem
Current matting approaches generalize poorly to everyday scenes because the problem is underconstrained, color-overlap cases are difficult, and standard datasets are small and controlled.
Method
A two-stage network directly predicts alpha from an image and trimap, then refines the prediction, while training uses a large composited dataset and composition loss.
Results
The method outperforms prior methods on the standard and synthetic datasets, achieves first place on alphamatting.com, and generalizes significantly better to real images.
Takeaways & Limitations
Learning structural and semantic features enables improved matting results beyond methods centered primarily on color cues.
Takeaways & Limitations
Standard alpha-matting metrics may not accurately measure human perceptual judgments, and methods may overfit the alphamatting.com dataset.
Abstract
from arXiv · showhide
Image matting is a fundamental computer vision problem and has many applications. Previous algorithms have poor performance when an image has similar foreground and background colors or complicated textures. The main reasons are prior methods 1) only use low-level features and 2) lack high-level context. In this paper, we propose a novel deep learning based algorithm that can tackle both these problems. Our deep model has two parts. The first part is a deep convolutional encoder-decoder network that takes an image and the corresponding trimap as inputs and predict the alpha matte of the image. The second part is a small convolutional network that refines the alpha matte predictions of the first network to have more accurate alpha values and sharper edges. In addition, we also create a large-scale image matting dataset including 49300 training images and 1000 testing images. We evaluate our algorithm on the image matting benchmark, our testing set, and a wide variety of real images. Experimental results clearly demonstrate the superiority of our algorithm over previous methods.
1. Introduction
Image matting is practically important but remains difficult to generalize because it is underconstrained, color-dependent, and trained on narrow datasets. The paper addresses these limitations with end-to-end deep learning and a large composited dataset, reporting stronger benchmark and real-image performance.
- Motivation: Matting estimates foreground appearance in images and videos, supporting image editing and film-production workflows in unconstrained scenes.The paper emphasizes the need for methods that handle real-world images rather than controlled settings.
- Problem formulation: The problem has 7 unknown values per pixel but only 3 known RGB values, leaving foreground color, background color, and alpha unknown.The image formation equation represents each observed pixel as a combination of foreground and background colors.
- Limitations: Existing methods rely primarily on color and positional cues, making them sensitive to overlapping foreground-background colors and prone to smearing or chunky artifacts.Prior approaches use sampling, propagation, or hybrids based largely on the matting equation.
- Limitations: The standard alphamatting.com dataset contains only 27 training and 8 test images, with limited diversity that can encourage overfitting and poor real-scene generalization.Its controlled lab scenes exclude broad variation such as humans, animals, and unconstrained lighting.
- Approach: The proposed approach directly predicts alpha from an image and trimap using learned structural features, with an encoder-decoder, residual refinement, and composition loss.The method targets patterns in hair, fur, blur, object edges, and semitransparent regions that low-level features may miss.
- Dataset: 49300 training images and 1000 test images are created by compositing extracted objects onto new backgrounds to support natural-image generalization.The authors use carefully extracted objects and synthetic compositions as training data.
- Results: The method achieves first place on the alphamatting.com challenge, outperforms prior methods on the synthetic test set, and receives strong user preference on 31 natural images.The study includes humans, animals, and other objects under varied scenes and lighting; it also finds robustness to trimap placement.
2. Related works
Earlier matting methods primarily use color-based sampling or propagation, while recent deep methods still depend on existing matting algorithms. The paper instead presents dataset compositing as a route to broader training coverage.
- Prior approaches: Traditional matting methods rely mainly on color and positional features through sampling, propagation, or combinations of both.These methods distinguish foreground and background using observed regions, color metrics, or propagated alpha values.
- Sampling methods: Sampling methods select candidate foreground and background colors from boundaries, rays, full boundaries, or color clusters before choosing a combination.The cited variants differ primarily in how candidate colors are gathered.
- Propagation methods: Propagation methods reformulate the matting equation to spread alpha values into unknown regions, with Closed-form Matting solving a sparse linear system under local smoothness assumptions.Closed-form Matting is often used as a post-process after sampling.
- Deep learning methods: Recent deep methods do not directly learn alpha from image and trimap alone, instead relying on existing algorithms to compute or initialize matting results.This dependence leaves them vulnerable to the same problems as earlier methods.
- Dataset construction: The paper's compositing procedure manually extracts alpha and foreground colors from simple-background objects, then composites them onto varied backgrounds as ground truth.Figure 2 depicts the input, computed matte, computed foreground colors, and resulting composites.
3. New matting dataset
The authors create a larger composited matting dataset to address the benchmark’s limited scale and diversity, while testing whether synthetic composition introduces bias.
- Motivation: The new dataset addresses the prior benchmark’s 27 training images, 8 testing images, static objects, and small-scale laboratory scenes.The earlier dataset’s controlled ground-truth collection also restricts diversity and excludes typical dynamic content.
- Dataset construction: The dataset composites real foreground objects onto new backgrounds after manually creating alpha mattes and pure foreground colors.Objects are selected from simple-background images, including prior benchmark images and video frames.
- Dataset construction: 49,300 training images and 1,000 testing images are constructed from 493 and 50 unique foreground objects, respectively.Training uses N = 100 backgrounds per object, while testing uses N = 20; trimaps are randomly dilated from ground-truth alpha mattes.
- Dataset characteristics: Composited examples include similar foreground and background colors and complex background textures, making the dataset more challenging and practical.These properties target difficult conditions in natural image matting.
- Bias assessment: Experiments found far superior natural-image results than prior methods, addressing concern that compositing could bias learned lighting or noise differences.The stated comparison is to prior methods on natural images.
4. Our method
The method combines a deep encoder-decoder that predicts alpha mattes from image patches and trimaps with a refinement network that sharpens the predictions.
- Architecture: The network has two stages: a deep convolutional encoder-decoder predicts an alpha matte, and a small convolutional network refines it.The first stage uses image patches and trimaps; the second stage targets more accurate alpha values and sharper edges.
- 4.1. Matting encoder-decoder stage: The encoder-decoder receives a 4-channel concatenation of an image patch and trimap, then transforms features through convolution, pooling, unpooling, and convolutional decoding.Its design is motivated by encoder-decoder success in segmentation, boundary prediction, and hole filling.
- Losses: Training uses alpha-prediction and compositional losses, with the latter constraining predicted composites to follow the compositing operation.The overall loss weights the two terms equally in the reported experiment and emphasizes unknown trimap regions.
- 4.2. Matting refinement stage: The refinement stage takes the image patch and first-stage alpha prediction as a 4-channel input and outputs the corresponding ground-truth alpha matte.It uses four convolutional layers, no downsampling, and a skip-model structure to preserve subtle structures.
- 4.2. Matting refinement stage: The refinement stage makes limited large-scale changes while refining and sharpening alpha values that can otherwise become overly smooth.Figure 4 compares the input, encoder-decoder result, and refinement result.
- Training and inference: The two stages are trained sequentially before jointly fine-tuning the complete network, and inference passes the initial alpha prediction into refinement.The refinement training uses only the alpha-prediction loss; Adam optimization uses a constant learning rate of 10^-5.
5. Experimental results
The method is evaluated on benchmark, synthetic, and natural-image datasets, showing strong quantitative performance, robustness to trimap placement, and favorable human judgments. Results attribute these gains to learned structure and semantics, with refinement improving encoder-decoder outputs.
- Evaluation setup: The evaluation spans alphamatting.com, Composition-1k, and a 31-image natural dataset covering varied foregrounds and scenes.The benchmark includes three trimap types, while Composition-1k broadens object and background diversity.
- Benchmark results: The method ranks first by SAD and second by both MSE and Gradient on the alphamatting.com benchmark.It also has the smallest SAD errors for five images across all three trimaps.
- Benchmark results: Learned structure and semantics help resolve difficult cases involving similar foreground-background colors and complex textures.The Troll and Doll examples show fewer mistakes and more detail than prior methods in these regions.
- Composition-1k results: All approach variants outperform prior methods on Composition-1k across SAD, MSE, Gradient, and Connectivity errors.The compositional loss improves results, while refinement provides the clearest visual and quantitative improvement over the encoder-decoder output.
- Robustness and visual results: The method maintains low, stable SAD error as trimap dilation increases and handles hair, holes, semi-transparency, and objects without pure foreground pixels.Sampling- and propagation-based methods require known foreground and background pixels for the latter case.
- Real-image results: In the real-image user study, users preferred the method over prior methods in nearly four out of five comparisons.The study involved 392 users, with each method pair evaluated by 5 to 6 unique users per image.
6. Conclusion
The work shows that neural networks can use structural and semantic features to improve matting and generalize better to real images.
- Neural networks can capture high-order structural and semantic features for improved matting results.
- The method outperforms prior methods on the standard dataset and generalizes significantly better to real images.