Source-linked AI summary
Natural Image Matting via Guided Contextual Attention
Yaoyi Li, Hongtao Lu
TL;DR
Natural image matting is ill-defined because foreground color, background color, and opacity are unknown at mixed pixels. This paper introduces guided contextual attention to propagate opacity information using appearance guidance, and reports superiority on Composition-1k and alphamatting.com evaluations.
Problem
At mixed pixels, foreground color, background color, and alpha value are unknown, making alpha matting ill-defined.
Method
The method uses an end-to-end neural network with guided contextual attention that propagates alpha features under guidance from low-level appearance features.
Results
The proposed method outperforms state-of-the-art approaches on the Composition-1k testing dataset and alphamatting.com benchmark.
Takeaways & Limitations
Guided contextual attention imitates affinity-based opacity propagation in a fully convolutional framework while using features learned by a neural network.
Takeaways & Limitations
The approach relies on the affinity-based inductive bias that locally similar appearances should have similar opacity.
Abstract
from arXiv · showhide
Over the last few years, deep learning based approaches have achieved outstanding improvements in natural image matting. Many of these methods can generate visually plausible alpha estimations, but typically yield blurry structures or textures in the semitransparent area. This is due to the local ambiguity of transparent objects. One possible solution is to leverage the far-surrounding information to estimate the local opacity. Traditional affinity-based methods often suffer from the high computational complexity, which are not suitable for high resolution alpha estimation. Inspired by affinity-based method and the successes of contextual attention in inpainting, we develop a novel end-to-end approach for natural image matting with a guided contextual attention module, which is specifically designed for image matting. Guided contextual attention module directly propagates high-level opacity information globally based on the learned low-level affinity. The proposed method can mimic information flow of affinity-based methods and utilize rich features learned by deep neural networks simultaneously. Experiment results on Composition-1k testing set and alphamatting.com benchmark dataset demonstrate that our method outperforms state-of-the-art approaches in natural image matting. Code and models are available at https://github.com/Yaoyi-Li/GCA-Matting.
Introduction
Natural image matting estimates per-pixel foreground opacity despite an ill-defined decomposition, motivating methods that propagate contextual information through learned image features. This paper introduces guided contextual attention to transmit opacity information directly within a neural network.
- Natural image matting separates foreground from background while estimating each pixel’s foreground opacity for editing, compositing, and film post-production.
- The matting equation is ill-defined because foreground color, background color, and opacity are all unknown at each pixel.
- Affinity-based methods borrow opacity from similar known patches, whereas sampling-based methods use foreground–background sample pairs to estimate unknown pixels.
- Recent learning-based methods use network priors to predict alpha mattes, while SampleNet combines learning with foreground and background inpainting.
- The proposed method propagates opacity through a guided contextual attention module that uses low-level image features to guide high-level alpha feature transmission.
- The framework can be interpreted both as affinity-based alpha propagation with a network prior and as guided inpainting of the alpha image.
Related Work
Natural image matting research includes sampling-based, propagation-based, and learning-based approaches. These methods differ in how they estimate alpha values from foreground, background, affinities, or learned network representations.
- Natural image matting methods are classified into sampling-based, propagation-based, and learning-based categories.
- Sampling-based methods solve the matting combination equation by selecting foreground and background colors for pixels in the unknown region.
- Propagation methods estimate alpha mattes by transferring alpha values from foreground and background regions to pixels in the unknown area.
- Information-flow Matting combines local and nonlocal color affinities with spatial smoothness through a color-mixture flow.
- Learning-based methods became dominant with deep convolutional networks, including fully neural, generative-adversarial, sampling, and index-based matting architectures.
Baseline Network for Deep Image Matting
The model uses a customized U-Net baseline for matting, with feature-flow, normalization, loss, and augmentation choices designed to preserve detail and stabilize training.
- Baseline architecture: The customized U-Net baseline is an encoder-decoder with stacked residual blocks that takes a cropped image patch and one-hot trimap as a six-channel input.It predicts the corresponding alpha matte and replaces GCA blocks with identity layers while omitting the image feature block.
- Feature fusion: The decoder combines encoder features before upsampling, aligns their channels with a two-layer shortcut block, and forwards lower-level features for fusion.These choices target detailed texture retention while avoiding additional convolutions on encoder features.
- Normalization: Spectral normalization is added to each convolutional layer alongside batch normalization to constrain the network's Lipschitz constant and stabilize training.The framework overview identifies spectral normalization and batch normalization as architectural components.
- Loss function: The network uses one alpha prediction loss: absolute predicted–ground-truth alpha differences averaged over the trimap's unknown region.The unknown region is denoted U, with predicted and ground-truth alpha values defined at each position.
- Data augmentation: Data augmentation combines foreground objects, applies resizing and affine transformations, generates trimaps by dilation and erosion, crops 512×512 unknown-centered patches, and applies HSV jitter.The dataset contains only 431 foreground objects for training, motivating extensive augmentation.
Guided Contextual Attention Module
The guided contextual attention module uses low-level appearance affinity to propagate high-level opacity information between known and unknown regions. It combines this propagation with neural-network features while controlling computational cost at a 64 × 64 feature stage.
- Feature Flows: Guided contextual attention uses low-level image features to guide information flow on alpha features containing opacity information.The framework separates alpha feature flow from image feature flow, analogous to constructing an affinity graph from opacity and appearance information.
- Patch Correlation: The module extracts 3 × 3 image-feature patches, reshapes them as convolutional kernels, and measures unknown-to-image-patch correlation with normalized inner products.Each patch represents appearance information at a specific position.
- Attention Scores: A scaled softmax converts patch correlations into guided attention scores along the candidate-patch dimension.These scores define the affinity used for subsequent propagation.
- Region Weighting: Unknown regions borrow features from both known and unknown patches because trimaps may contain overwhelming unknown areas and few known pixels.The method assigns different weights to known and unknown patches to accommodate this imbalance.
- Opacity Propagation: The guided attention scores propagate information on alpha features through deconvolution, reconstructing alpha features in the unknown area.This implements affinity-style opacity propagation within the neural network.
- Architecture and Cost: The attention block has complexity O(c(hw)^2), so two blocks are appended at the 64 × 64 feature stage rather than higher-resolution stages.Higher-resolution features provide more detail but increase computational cost.
Results
The proposed method performs strongly across Composition-1k and alphamatting.com, with particular advantages in semitransparent regions, larger unknown areas, detailed textures, and contextual error cases.
- Composition-1k Testing Dataset: The method outperforms state-of-the-art approaches on the Composition-1k testing set, and its baseline also surpasses some leading methods.Comparisons include three deep matting methods and three conventional algorithms.
- Composition-1k Testing Dataset: The approach achieves better qualitative performance on different foreground objects, especially in semitransparent regions and when the unknown region is larger.The authors attribute this performance to information flow between feature patches with similar appearance features.
- Composition-1k Testing Dataset: A single Nvidia GTX 1080 with 8GB memory evaluates each Composition-1k image whole without scaling, while higher-resolution attention blocks improve detailed texture performance.The attention blocks operate on feature maps at resolutions much higher than 64×64 during training.
- Alphamatting.com Benchmark dataset: GCA Matting ranks first under Gradient Error on the alphamatting.com benchmark and is generally one of its top-performing methods.Its results under the large and user trimaps are reported as much better than other top approaches, while its overall MSE ranking is nearly the same as SampleNet.
- Alphamatting.com Benchmark dataset: On the Plastic bag example, the method correctly predicts pixels around an iron wire that most previous methods misclassify.The authors connect this result to contextual information from surrounding background patches.
- Visualization of Attention Map: The attention map shows no information flow in known regions, while unknown-region patches borrow information from visually similar patches.The visualization identifies where opacity information is propagated in the input image.
Conclusions
The paper frames image matting as opacity information propagation in an end-to-end neural network. Its guided contextual attention module imitates affinity-based propagation by transmitting opacity information between alpha features using appearance guidance, and evaluations on two datasets show superiority.
- The method solves image matting through opacity information propagation in an end-to-end neural network.
- The guided contextual attention module imitates affinity-based propagation in a fully convolutional manner.
- The module transmits opacity information between alpha features under the guidance of appearance information.
- Evaluations on the Composition-1k testing dataset and alphamatting.com dataset show the superiority of the proposed method.