Source-linked AI summary
AlphaGAN: Generative adversarial networks for natural image matting
Sebastian Lutz, Konstantinos Amplianitis, Aljosa Smolic
TL;DR
Natural image matting must estimate foreground opacity despite unknown foreground and background colors, making it severely ill-posed and important for compositing applications. AlphaGAN introduces a GAN with a generator trained on alpha, compositional, and adversarial objectives, plus dilated convolutions and a PatchGAN discriminator. It reports first-place overall gradient performance on the alphamatting.com benchmark and visually appealing results, while relying on synthetic compositing because the available foreground dataset is limited.
Problem
Natural image matting estimates foreground opacity despite seven unknown values and only three known values, creating a severely ill-posed problem for compositing applications.
Method
AlphaGAN combines an encoder-decoder generator using dilated convolutions with a PatchGAN discriminator trained on real and predicted-alpha composites.
Results
AlphaGAN achieves first place overall on the alphamatting.com benchmark’s gradient evaluation metric and reports visually appealing results.
Takeaways & Limitations
Adversarial supervision helps the generator learn alpha predictions that produce visually appealing compositions, including sharper structures.
Takeaways & Limitations
The available training dataset contains only 431 unique foreground objects, so the authors augment it with random backgrounds and compositing.
Abstract
from arXiv · showhide
We present the first generative adversarial network (GAN) for natural image matting. Our novel generator network is trained to predict visually appealing alphas with the addition of the adversarial loss from the discriminator that is trained to classify well-composited images. Further, we improve existing encoder-decoder architectures to better deal with the spatial localization issues inherited in convolutional neural networks (CNN) by using dilated convolutions to capture global context information without downscaling feature maps and losing spatial information. We present state-of-the-art results on the alphamatting online benchmark for the gradient error and give comparable results in others. Our method is particularly well suited for fine structures like hair, which is of great importance in practical matting applications, e.g. in film/TV production.
1 Introduction
Natural image matting estimates foreground opacity from images containing foreground and background, but the unknown foreground and background colors make it severely ill-posed. AlphaGAN addresses this challenge with adversarial training and architectural changes aimed at preserving spatial detail and producing visually appealing compositions.
- Natural image matting estimates each pixel’s foreground opacity from an image containing foreground and background.
- The problem is severely ill-posed because each pixel has seven unknown values but only three known values.
- Existing methods often rely on color-based sampling or affinity propagation, which can create artifacts when foreground and background color distributions overlap.
- AlphaGAN combines a generator with adversarial supervision and uses dilated convolutions to capture global context without downscaling feature maps.
2 Previous Work
Earlier natural image matting methods primarily used sampling or propagation, while newer deep-learning approaches learned alpha mattes with convolutional networks. These approaches established encoder-decoder, refinement, and auxiliary-feature strategies that AlphaGAN builds upon.
- Traditional matting methods infer unknown alpha values from user-defined foreground and background samples using sampling or propagation.
- Deep-learning matting introduced convolutional systems that predict alpha mattes from images and trimaps or operate automatically on portrait images.
- Xu et al. used an encoder-decoder stage followed by a refinement network that sharpens predicted alpha edges.
- Other approaches combined RGB images with outputs from local and non-local matting algorithms or used learned mappings to reconstruct alpha mattes.
3 Our Approach
AlphaGAN uses an encoder-decoder generator and PatchGAN discriminator to predict alpha mattes from composited images and trimaps. Its objective combines alpha, compositional, and adversarial losses, while dilated convolutions and skip connections preserve context and spatial detail.
- Generator: The generator is a convolutional encoder-decoder trained with ground-truth alpha supervision and adversarial loss.
- Training data: Training data are expanded by compositing foreground objects with random MSCOCO backgrounds because the available 431 foreground images are insufficient alone.
- Generator: Dilated convolutions capture broader context without reducing feature-map resolution, addressing CNN spatial-localization issues.
- Generator: Skip connections and staged upsampling combine encoder features with decoder features at matching resolutions.
- Discriminator: PatchGAN classifies local image patches and uses a four-channel input containing a composite image and trimap to guide sharper alpha predictions.
- Network objectives: The full objective combines alpha-prediction, compositional, and adversarial losses, with the generator minimizing against a discriminator that maximizes the adversarial objective.
4 Experimental Results
The method is evaluated on the alphamatting.com benchmark and Composition-1k, using several error metrics and architecture comparisons. It performs strongly overall, including first place on gradient error and strong results for fine-structure images.
- Datasets and metrics: The evaluation uses alphamatting.com and Composition-1k, reporting SAD, MSE, gradient, and connectivity errors.Composition-1k contains 1000 test images, while alphamatting.com provides trimaps of three sizes.
- Network architecture: The generator-architecture comparison varies output stride, dilated-convolution design, skip connections, and adversarial loss.Table 1 identifies ASPP, multi-grid dilation, skip connections, and GAN loss as compared design choices.
- Composition-1k results: On Composition-1k, AlphaGAN outperforms all tested methods except for the gradient error of the comprehensive sampling approach.Comparisons exclude other methods’ failed cases, whereas AlphaGAN succeeds on every image.
- Alphamatting.com results: On alphamatting.com, AlphaGAN achieves state-of-the-art SAD and MSE results for Troll and Doll and ranks first overall on gradient error.For the remaining images, its SAD and MSE values are numerically close to the top-performing results.
- Qualitative results: The strongest qualitative results capture fine hair structures, while Net is reported as the worst-performing image.The paper attributes sharper structures to the discriminator’s ability to distinguish overly smooth predictions from ground-truth compositions.
5 Conclusion
The paper introduces a GAN architecture for natural image matting, combining alpha prediction with adversarial discrimination of composited images. It reports first place on the perceptual gradient metric, comparable results on other metrics, and especially strong hair-structure predictions.
- Method: The paper presents a GAN in which a generator predicts alpha mattes and a discriminator distinguishes ground-truth from predicted-alpha compositions.The discriminator supplies adversarial training feedback for the generator.
- Method: The architecture adds generator enhancements intended to improve natural-image matting performance.The conclusion links these changes to visually appealing compositions.
- Results: AlphaGAN ranks first on the benchmark’s gradient metric and achieves comparable results on the other metrics.It is also first in SAD and MSE for the Troll and Doll images.
- Takeaway: The Troll and Doll results capture high-frequency hair structures, which the paper suggests may be attributed to adversarial loss.This claim is presented as an attribution rather than a definitive causal finding.
10 LUTZ, AMPLIANITIS, SMOLI ´C: GAN FOR NATURAL IMAGE MATTING Acknowledgements
The acknowledgements recognize hardware, dataset, evaluation-code, and financial support for the research.
- Acknowledgements: The authors acknowledge NVIDIA for donating the Titan Xp GPU used in the research.They also thank the Deep Image Matting authors for training data and evaluation code, and acknowledge Science Foundation Ireland funding.
LUTZ, AMPLIANITIS, SMOLI ´C: GAN FOR NATURAL IMAGE MATTING 15 6 Supplementary material
The proposed generator combines a ResNet50-based encoder with an ASPP module and a compact decoder to produce 320 × 320 alpha mattes while retaining localized information.
- The encoder uses ResNet50 with its last two layers removed and an ASPP module producing 256 40 × 40 feature maps.
- The decoder upsamples feature maps to 320 × 320 using bilinear interpolation, unpooling, and fractionally-strided convolution.
- Max-pooling indices are saved in the encoder and reused during unpooling to support decoder upsampling.
- Batch normalization and ReLU follow all convolutional layers except the final layer, which uses a sigmoid to scale outputs between 0 and 1.
- Skip connections are added to retain localized information in the generator architecture.