Source-linked AI summary
Semantic Image Matting
Yanan Sun, Chi-Keung Tang, Yu-Wing Tai
TL;DR
Prior deep matting methods do not adequately consider how foreground semantics relate to different matting patterns. This paper incorporates semantic classification through 20 classes, a semantic trimap, and semantic-level regularization, and reports state-of-the-art performance across multiple benchmarks.
Problem
Prior deep methods use semantic information at the data level but do not adequately consider how different foreground semantics underlie matting patterns.
Source: §1 · p. 2Method
The method classifies 20 matting patterns, automatically generates a semantic trimap, and uses a multi-class discriminator and content-sensitive gradient constraints to regularize alpha predictions.
Source: §1 · p. 2Results
The method reports state-of-the-art performance on the Semantic Image Matting Dataset and Composition-1K, and ranks first on three alphamatting.com metrics.
Source: §5.2 · p. 7Takeaways & Limitations
The class-balanced Semantic Image Matting Dataset is intended to benefit future matting research and support analysis of algorithm performance across matting classes.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Natural image matting separates the foreground from background in fractional occupancy which can be caused by highly transparent objects, complex foreground (e.g., net or tree), and/or objects containing very fine details (e.g., hairs). Although conventional matting formulation can be applied to all of the above cases, no previous work has attempted to reason the underlying causes of matting due to various foreground semantics. We show how to obtain better alpha mattes by incorporating into our framework semantic classification of matting regions. Specifically, we consider and learn 20 classes of matting patterns, and propose to extend the conventional trimap to semantic trimap. The proposed semantic trimap can be obtained automatically through patch structure analysis within trimap regions. Meanwhile, we learn a multi-class discriminator to regularize the alpha prediction at semantic level, and content-sensitive weights to balance different regularization losses. Experiments on multiple benchmarks show that our method outperforms other methods and has achieved the most competitive state-of-the-art performance. Finally, we contribute a large-scale Semantic Image Matting Dataset with careful consideration of data balancing across different semantic classes.
1. Introduction
Natural image matting is ill-posed and must handle foregrounds with diverse objects and local patterns, while prior deep methods do not adequately consider the underlying causes of matting due to different object semantics. The paper introduces semantic classification of matting regions, a semantic trimap, a multi-class discriminator, and content-sensitive weights, alongside a class-balanced dataset.
1. Introduction
- Foregrounds range from humans and furry animals to transparent glass and complex shapes such as nets or trees, producing varied matting patterns.Human portraits can combine fuzzy hair with sharp body boundaries.Source: §1 · pp. 1–2
- Because matting is ill-posed, existing methods commonly use user-supplied clues such as scribbles or trimaps.Source: §1 · p. 1
- Prior deep methods use semantic information at the data level but do not adequately consider how different object semantics underlie matting patterns.A portrait, for example, can contain both fuzzy hair and sharp boundaries.Source: §1 · p. 2
- The framework classifies 20 matting patterns, extends the conventional trimap to an automatically generated semantic trimap, and uses a multi-class discriminator for semantic regularization.Gradient constraints with content-sensitive weights balance different regularization losses.Source: §1 · p. 2
- The authors report that semantic classification of matting patterns is the first such approach in a natural image matting framework.Source: §1 · p. 2
- The paper contributes a large-scale, class-balanced Semantic Image Matting Dataset intended to benefit future matting research and enable analysis across matting classes.Source: §1 · p. 2
2. Related Work
Earlier matting approaches include low-level traditional methods, general deep-learning methods, and methods tailored to specific object classes. The paper identifies a lack of principled semantic matting across classes and introduces 20 diverse matting classes.
2. Related Work
- Traditional matting methods use sampling, propagation, or both, relying on color similarities or propagating alpha from known regions.They are based on the linear-combination matting equation, apart from.Source: §2 · p. 2
- Deep-learning methods include general-object architectures and approaches that relax trimap input or jointly estimate foreground, background, and alpha.Examples include the two-stage encoder-decoder of Xu et al. [43] and FBA-net of Forte and Pitié [15].Source: §2 · p. 2
- Human-specific methods can perform well on portraits but fall short on general objects, and their performance drops when the human detector fails.Some approaches eliminate the trimap because the foreground is known to be human.Source: §2 · p. 2
- Other class-specific methods target motion-blurred objects, out-of-focus regions, or transparent objects with refraction.These approaches use task-specific motion modeling, segmentation, or refractive-flow estimation.Source: §2 · p. 2
- The paper proposes 20 matting classes with diverse appearances, covering almost all scenarios in existing matting datasets.Source: Figure 2 · p. 3
- The authors characterize prior methods as lacking a principled approach to semantic image matting or as limited to one or a few classes.Source: §2 · p. 3
3. Dataset
The Semantic Image Matting Dataset addresses class bias in existing datasets with 20 matting classes and balanced representation. Its class analysis shows that alpha patterns range from smooth textures to fine, sharp structures.
3. Dataset
- The 20 proposed matting classes cover almost all scenarios in existing datasets, while other real-world cases are described as quite rare.The classes are based on distinct alpha-matte patterns.Source: §3 · p. 3
- The dataset contains 726 training foregrounds and 89 testing foregrounds across 20 classes.Foregrounds were selected from Adobe Image Matting Dataset and collected clean images with corresponding patterns; their mattes were extracted with Photoshop.Source: §3 · p. 3
- Existing large-scale matting datasets are biased toward common objects, while Semantic Image Matting Dataset is class-balanced across 20 matting classes.Figure 4 compares its class distribution with Adobe Image Matting Dataset and Distinctions-646.Source: Figure 4 · p. 3, §3 · p. 3
- Fire, smoke, and silk have smooth textures and small alpha gradients, whereas net and spider web show fine features and sharp structures.Figure 5 plots class-specific alpha ratios and mean alpha-gradient magnitudes within unknown regions.Source: Figure 5 · p. 3, §3 · p. 4
4. Method
The framework combines an automatically generated semantic trimap with an encoder-decoder to predict alpha mattes. A multi-class discriminator and learnable content-sensitive weights provide semantic and gradient-related regularization.
4.1. Framework
- The semantic trimap appends class-confidence maps for unknown pixels to the conventional trimap, and patch analysis generates these maps for the RGB-image input.At inference, multi-scale overlapping patches are analyzed and their score maps are stitched together.Source: §4.1 · p. 4
- A multi-class discriminator regularizes alpha predictions against class-specific structures using classification and feature-reconstruction losses.The classification loss uses groundtruth-patch class probabilities as labels, while feature reconstruction compares predicted and groundtruth alpha features.Source: §4.1 · p. 5, §4.2 · p. 6
- Learnable 2D content-sensitive weights balance foreground, background, and alpha contributions in the gradient constraint, linking gradient contributions with semantic representation.The motivation is that classes such as Hair and Fire have different gradient distributions.Source: §4.1 · p. 5
4.2. Losses
- The objective combines alpha and foreground/background reconstruction, semantic classification and feature reconstruction, gradient-related losses, and an exclusion loss.The total loss weights foreground/background reconstruction, feature reconstruction, gradient, and exclusion losses by 0.2, and classification loss by 0.1, alongside alpha reconstruction loss.Source: §4.2 · pp. 5–6
5. Experiments
The method reports state-of-the-art performance on the Semantic Image Matting and Composition-1K datasets, and ranks first on three metrics on alphamatting.com. The analysis examines how semantic trimaps, a multi-class discriminator, and content-sensitive weights contribute to the framework.
5.1. Implementation Details
- Training composites foreground objects with COCO backgrounds, using image augmentations, merged foregrounds, and dilated or eroded alpha mattes to generate trimaps.Foreground crops use sizes 320, 480, or 640; trimap generation uses random kernel sizes from 2 to 9 and iterations from 5 to 15.Source: §5.1 · p. 6
5.2. Main Results
- The method reports state-of-the-art performance on the Semantic Image Matting and Composition-1K datasets and ranks first on three alphamatting.com metrics.It also performs better than other methods on every matting class in the Semantic Image Matting Dataset.Source: §5.2 · p. 7, Table 2 · p. 7, Table 3 · p. 8
- Real-world images with user-labeled trimaps provide qualitative evidence of the framework’s effectiveness and generalization ability.Figure 9 shows qualitative results on these images.Source: §5.2 · p. 7, Figure 9 · p. 7
5.3. Analysis
- Semantic score maps provide class priors for unknown pixels, addressing cases where coarse conventional trimaps provide insufficient clues.Figure 10 visualizes two score-map channels, hair hard and sharp.Source: §5.3 · p. 7, Figure 10 · p. 8
- The multi-class discriminator supplements pixelwise distance by learning class-specific alpha-matte statistics through classification and feature-reconstruction losses.The authors state that these losses preserve pattern statistics and consequently improve performance.Source: §5.3 · pp. 7–8
- Learnable content-sensitive weights improve SAD by 1.97 while weighting gradient constraints for different classes.For semi-transparent smooth textures such as silk, the alpha matte and background contribute most to the image gradient; Figure 11 visualizes sample weights, and Table 5 presents ablations.Source: §5.3 · p. 8, Figure 11 · p. 8, Table 5 · p. 8
6. Conclusion
The conclusion presents semantic classification of alpha patterns as the basis for a matting framework with semantic trimaps, a multi-class discriminator, and learnable gradient constraints. Experiments across three datasets and real-world images show advantages over existing methods.
6. Conclusion
- The method clusters 20 regional alpha-pattern classes and automatically generates semantic trimaps to provide class information for predictions.It also uses a multi-class discriminator to regularize class-specific alpha statistics and learnable content-sensitive weights for alpha-gradient constraints.Source: §6 · p. 8
- The framework combines semantic trimaps, a multi-class discriminator, and learnable content-sensitive weights to regularize alpha prediction and optimization.The discriminator is designed to preserve class-specific alpha statistics, while the weights are used for alpha-gradient constraints.Source: §6 · p. 8
- Experiments on the Semantic Image Matting Dataset, Composition-1K, and real-world images show quantitative and qualitative advantages over existing methods.Source: §6 · p. 8