Source-linked AI summary
$F$, $B$, Alpha Matting
Marco Forte, François Pitié
TL;DR
Alpha matting requires estimating an opacity mask, but most networks predict only α while recovering foreground and background colours requires post-processing or computationally expensive joint-prediction architectures. This paper proposes a low-cost joint prediction approach and studies training and loss choices. It reports state-of-the-art alpha matte and composite colour quality on Adobe Composition-1k, as well as the best current result on alphamatting.com.
Problem
Most alpha-matting networks predict only α and leave foreground and background colour recovery to post-processing, while recent joint-prediction methods carry significant computational and memory cost.
Source: §1 · p. 2Method
The paper proposes an architecture that jointly estimates α, F, and B and studies training regimes and losses for these predictions.
Source: §1 · p. 3, §5 · p. 14Results
The method achieves state-of-the-art alpha matte and composite colour quality on Adobe Composition-1k and is the current best-performing method on alphamatting.com.
Source: §1 · p. 3, §5 · p. 14Takeaways & Limitations
The proposed simultaneous α, F, and B prediction achieves state-of-the-art performance without increased computational or memory cost.
Source: §5 · p. 14Experiments used a single training run, so precise confidence intervals are not known.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Cutting out an object and estimating its opacity mask, known as image matting, is a key task in many image editing applications. Deep learning approaches have made significant progress by adapting the encoder-decoder architecture of segmentation networks. However, most of the existing networks only predict the alpha matte and post-processing methods must then be used to recover the original foreground and background colours in the transparent regions. Recently, two methods have shown improved results by also estimating the foreground colours, but at a significant computational and memory cost. In this paper, we propose a low-cost modification to alpha matting networks to also predict the foreground and background colours. We study variations of the training regime and explore a wide range of existing and novel loss functions for the joint prediction. Our method achieves the state of the art performance on the Adobe Composition-1k dataset for alpha matte and composite colour quality. It is also the current best performing method on the alphamatting.com online evaluation.
1 Introduction
The paper addresses the need for foreground and background colours alongside alpha mattes, which most existing networks leave to post-processing. It proposes jointly predicting all three and studies training and loss choices; the resulting network reports state-of-the-art performance on Adobe Composition-1k and alphamatting.com.
1 Introduction
- Alpha matting estimates an object's opacity mask, and the Compositing Equation models each observed pixel colour as a mixture of foreground and background colours weighted by α.The paper notes the problem is ill-posed, with three nonlinear equations for seven unknowns and no well-defined spatial dependency.Source: §1 · p. 1
- Most matting networks predict only α, leaving foreground and background colour recovery to post-processing, while recent joint-prediction architectures are computationally expensive.The paper proposes a novel architecture for jointly estimating α, F, and B.Source: §1 · p. 2
- The authors report that batch size 1 in had a critical performance impact and could by itself explain reported difficulties reproducing Deep Image Matting.They frame training and loss choices as important alongside core architecture design.Source: §1 · p. 3
- The study explores 17 experiments spanning min-batch versus stochastic gradient descent, batch normalization versus group normalization, alpha-matte losses, and losses for predicting F and B.The examined alpha losses include L1, gradient, Laplacian pyramid, and compositing losses; foreground/background losses include L1 and exclusion losses.Source: §1 · p. 3
- The resulting network achieves state-of-the-art alpha matte and composite colour quality on Adobe Composition-1k and is the current best performer on alphamatting.com.The section reports these benchmark outcomes without giving metric values.Source: §1 · p. 3
2 Related Works
Prior matting work spans spatial priors, segmentation-derived network architectures, and losses for both alpha mattes and compositing. The paper situates joint foreground/background prediction against sequential CNN approaches and discusses exclusion loss as a way to separate predicted colours.
2 Related Works
- Earlier matting methods used sampling strategies, while Levin et al. introduced a spatial prior model later widely adopted; CNNs enabled more expressive spatial priors.Deep matting networks adapted segmentation architectures, with later work changing encoder-decoder depth, pooling, fusion, or trimap use.Source: §2 · pp. 3–4
- Alpha losses range from Huber and L1 to gradient, Laplacian-pyramid, and discriminative losses, reflecting the matte's binary-like values and continuous structure.The cited work notes that strand-shape errors can be more noticeable than slight errors in overall opacity.Source: §2 · p. 4
- Because α, F, and B are interdependent, Deep Image Matting combined alpha Huber loss with a compositing reconstruction loss using ground-truth foreground and background colours.Later works also used this compositing loss.Source: §2 · p. 4
- CNN approaches to joint colour prediction include sequential F-and-α decoding and three separate encoder-decoders for B, F, and α, with the latter being very deep and large.The prior works use L1 losses on predicted F and B; also uses a VGG16 feature loss, while introduces a composition loss.Source: §2 · p. 4
- The paper investigates exclusion loss, previously introduced in similar form for reflection removal, to separate F and B and avoid structures from the original image leaking into both.The supplied passage presents exclusion as an alternative loss to investigate.Source: §2 · p. 5
3 Proposed Approach
The proposed approach jointly predicts the alpha matte and foreground and background colours with a low-overhead encoder-decoder, alongside changes to training, losses, fusion, and test-time augmentation. It modifies the training data to support foreground prediction and uses a fusion step that accounts for the compositing model.
3.1 Network Architecture
- A single U-Net-style encoder-decoder predicts α, F, and B through seven output channels, adding minimal parameters rather than chaining predictions sequentially.It uses ResNet-50, blurred trimap inputs, dilated encoder layers, and a pyramid-pooling decoder with skip connections; hardtanh clamps α while sigmoid bounds F and B.Source: §3.1 · p. 5
3.2 Batch Normalisation vs. Group Normalisation
- Batch size one motivates replacing batch normalisation with Group Normalisation and Weight Standardisation in the ResNet-50 encoder.The paper reports that batch size one can greatly increase accuracy in the original Deep Image Matting method, while ResNet-50 batch normalisation assumes batches larger than one.Source: §3.2 · p. 6
3.3 F, B, α Losses
- Training sums alpha L1, composition, gradient, and multiscale Laplacian losses with foreground-background L1, Laplacian, composition, and gradient-exclusion losses.The passages identify these terms as the loss functions used for the respective predictions.Source: §3.3 · p. 6
3.4 ˆF, ˆB, ˆα Fusion
- The fusion mechanism uses a simplified maximum-likelihood model and iterative block updates to combine α, F, and B predictions while accounting for the compositing equation.It assumes independent prediction errors, ignores spatial dependence and spatial losses, replaces L1 with L2, and uses one update iteration in practice.Source: §3.4 · p. 7
3.5 Training Dataset and Modifications
- Re-estimating foreground colours across the training images addresses invalid colours that augmentation could otherwise spill into regions used for foreground prediction.The original foreground images are valid only where α is nonzero; re-estimation extends them across the picture.Source: §3.5 · pp. 7–8
3.6 Training Details
- Training uses randomly cropped patches, varied trimaps and augmentations, scale sampling, and a learning-rate schedule that drops at 40 epochs before five more epochs of fine-tuning.The reported crop sizes are 640 × 640, 480 × 480, and 320 × 320; training took 16 days on one 1080ti GPU.Source: §3.6 · p. 8
3.7 Test Time Augmentation (TTA)
- Test-time augmentation combines rotation, flipping, and scaling of the input image and trimap.The paper reports that results for this comprehensive augmentation are shown in Tables 4 and 3.Source: §3.7 · p. 8
4 Experimental Results
Experiments evaluate alpha mattes and foreground composites on Composition-1k, with ablations of losses, batch size and normalization, and joint F, B, α prediction. The final OursFBα TTA leads the reported comparisons on Composition-1k and alphamatting.com.
4 Experimental Results
- The evaluation uses 1,000 Composition-1k test images and measures alpha-matte quality with SAD, MSE, GRAD, and CONN, plus composite MSE and SAD.The reported experiments evaluate both alpha mattes and the combined predicted alpha-foreground composite.Source: §4 · p. 9
- Because each experiment was run only once, precise confidence intervals are unavailable.Each training run took two to four weeks; earlier experiments suggested most observations were consistent.Source: §4 · p. 9
4.1 Evaluating the α Loss Functions
- The Laplacian loss significantly reduces errors across all metrics, compositing loss reduces errors except for gradient, and gradient loss increases errors across metrics.Despite the gradient-loss result, it was included in subsequent models because the finding came late; further examination was left to future work.Source: §4.1 · pp. 9–10
- Clipping activation outperforms sigmoid for alpha prediction, so subsequent models use clipping.The authors note that sigmoid reaches 0 and 1 only for infinite-valued inputs, whereas alpha mattes are mostly 0 or 1.Source: §4.1 · p. 10
4.2 Batch-Size and BatchNorm vs. GroupNorm
- A batch size of 1 significantly reduces error relative to larger-batch models, while GroupNorm offers no hidden advantage over BatchNorm at batch size 6.The single-image minibatch model uses GroupNorm with Weight Standardisation because BatchNorm is incompatible with batch size 1.Source: §4.2 · p. 10
4.3 Evaluating the Impact of Jointly Estimating F, B, α
- Foreground-background losses preserve alpha accuracy while enabling foreground prediction, and full-image foreground supervision improves composite results; test-time F, B, α fusion improves both composite and alpha quality.The extended foreground colours also address colour spill from resizing undefined foreground colours during augmentation.Source: Figure 2 · p. 9, §4.3 · p. 11
4.4 Comparison with Other Works
- OursFBα TTA reduces Composition-1k error by an average of 34% against the best-performing method for each metric, and the models outperform prior methods on all four alpha-matte metrics.The final model has improved gradient and connectivity measures but worse SAD than OursFBα without test-time augmentation.Source: Table 4 · p. 12, The Composition-1k dataset · p. 12
The Composition-1k dataset
- OursFBα TTA has the lowest average rank among the top four prior methods on alphamatting.com and ranks first overall.It leads in connectivity, while OursFBα TTA performs best in all reported metrics except gradient when compared with Oursα TTA.Source: Table 5 · p. 14, The alphamatting.com Dataset · p. 14
5 Conclusion
The paper jointly predicts the foreground, background, and alpha map, and reports state-of-the-art performance without increased computational or memory cost. Its training study finds that batch size one, longer training, and test-time augmentation have a bigger impact than loss choices.
5 Conclusion
- The method simultaneously estimates the foreground, background, and alpha map from a natural image and trimap.The primary training-regime contribution is batch size one, combined with 45 epochs and test-time augmentation.Source: §5 · p. 14
- Batch size one, 45 epochs, and test-time augmentation have a bigger impact than the choices of loss functions.Source: §5 · p. 14
- The proposed joint α, F, B prediction achieves state-of-the-art performance for all predictions without increased computational or memory cost.The reworked foreground dataset, new exclusion loss, and fusion mechanism improve final composite quality and can be incorporated in future work.Source: §5 · p. 14