Source-linked AI summary
Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation
Yukun Su, Ruizhou Sun, Guosheng Lin, Qingyao Wu
TL;DR
WSSS with image-level labels can confuse objects with correlated contextual backgrounds, limiting object-focused segmentation. CDA relocates self-derived object instances into new scenes and trains with original images pairwise. It improves multiple WSSS methods, reaching state-of-the-art mIoU results on PASCAL VOC 2012 and COCO.
Problem
Image-level-label WSSS can couple foreground objects with correlated backgrounds, causing CAM-based methods to include contextual regions.
Method
CDA uses self-provided segmentation masks to copy and paste object instances into new contexts, with pairwise online augmentation training.
Results
CDA boosts various WSSS methods to state-of-the-art, reaching 66.1% validation and 66.8% test mIoU on PASCAL VOC 2012, and 33.7% validation mIoU on COCO.
Takeaways & Limitations
Changing object-context arrangements provides a generally applicable WSSS augmentation strategy without additional data.
Takeaways & Limitations
Random blending can produce noisy examples, although the authors state such hard cases are uncommon and do not affect learning.
Abstract
from arXiv · showhide
Data augmentation is vital for deep learning neural networks. By providing massive training samples, it helps to improve the generalization ability of the model. Weakly supervised semantic segmentation (WSSS) is a challenging problem that has been deeply studied in recent years, conventional data augmentation approaches for WSSS usually employ geometrical transformations, random cropping and color jittering. However, merely increasing the same contextual semantic data does not bring much gain to the networks to distinguish the objects, e.g., the correct image-level classification of "aeroplane" may be not only due to the recognition of the object itself, but also its co-occurrence context like "sky", which will cause the model to focus less on the object features. To this end, we present a Context Decoupling Augmentation (CDA) method, to change the inherent context in which the objects appear and thus drive the network to remove the dependence between object instances and contextual information. To validate the effectiveness of the proposed method, extensive experiments on PASCAL VOC 2012 dataset with several alternative network architectures demonstrate that CDA can boost various popular WSSS methods to the new state-of-the-art by a large margin.
1. Introduction
CDA addresses WSSS’s reliance on contextual co-occurrences by relocating object instances into different scenes, improving object-focused learning without additional data. Experiments report substantial gains across WSSS methods and datasets.
- CDA randomly pastes object instances into new scenes to decouple foreground objects from their inherent contextual positions.
- The method targets image-level-label WSSS, which reduces annotation effort but can cause models to associate foreground classes with correlated backgrounds.
- Traditional geometric and color transformations preserve contextual semantic relations, limiting their ability to help WSSS discover object regions.
- 66.1% mIoU on PASCAL VOC 2012 validation, 66.8% on its test set, and 33.7% on COCO validation establish new state-of-the-art results.
- CDA is presented as a generally applicable augmentation approach that does not require additional data and can focus networks on object regions.
2. Related Work
Prior WSSS methods rely heavily on CAM-derived object seeds, while conventional augmentation changes images without breaking object-context correlations. CDA extends copy-and-paste augmentation to WSSS without pixel-wise labels.
- CAM-based WSSS can mistakenly include correlated backgrounds such as water, sky, or tracks because only image-level labels are available.
- Conventional augmentation increases sample diversity through operations such as rotation, flipping, noise, synthesis, and style transfer.
- The proposed scheme first obtains simple object instances from off-the-shelf WSSS methods, then pastes them into raw images for pairwise online training.
- CDA applies copy-and-paste augmentation in WSSS without requiring pixel-wise instance labels or other auxiliary approaches.
3. Framework
CDA builds a two-stage WSSS augmentation framework by collecting qualified object instances from initial predictions, then randomly pasting them into training images online and pairwise. It decouples objects from their original contexts without additional pixel-wise labels, using varied pasting and occlusion to improve object-feature learning.
- 3.1. Object Instances Collecting: CDA first collects qualified object instances from off-the-shelf WSSS predictions, selecting simple single-class images and masks whose foreground area falls within threshold bounds.The foreground-to-image area ratio uses thresholds ϵ1 and ϵ2; overly large masks may include background, while overly small masks may miss foreground pixels.
- 3.1. Object Instances Collecting: The collected objects are pasted only into images where their class is absent, with random rescaling, rotation, and optional Gaussian smoothing to diversify scenes and blend boundaries.Gaussian smoothing makes added object boundaries appear more natural.
- 3.2. Online Augmentation Training: CDA treats partial occlusion as useful hard augmentation because covering discriminative regions can force the network to discover more complete object regions.Extreme cases that fully cover an original object may confuse classification, but the framework argues they are uncommon because random pasting favors off-center positions.
- 3.2. Online Augmentation Training: Pairwise training compares images with and without newly blended objects, helping the classifier identify discriminative object features despite duplicated background context.The paper motivates this comparison as analogous to finding differences in human visual perception.
- 3.3. Discussion: Unlike copy-and-paste methods requiring pixel-wise labels, CDA uses self-provided masks and online combinations to decouple foreground objects from context while enabling exponential-level augmentation diversity.The changing combinations of object instances and natural images can vary across training rounds.
4. Experiments
Experiments evaluate CDA through augmentation ablations, qualitative CAM and pseudo-mask analyses, baseline integration, and comparisons with state-of-the-art WSSS methods. CDA improves object-focused representations and segmentation performance across multiple architectures and benchmarks.
- Ablation Studies: Random rescale pasting achieves 49.8% mIoU among compared augmentation methods, while combining rescale with rotation reaches 50.8% mIoU on the PASCAL VOC training set.Gaussian smoothing does not improve performance, so subsequent experiments use random rescale combined with rotation.
- Ablation Studies: Qualitative CAM comparisons show that CDA suppresses background activation, focuses on target objects, and expands coverage beyond only the most discriminative regions.Examples contrast table-related chair activation, train-related track activation, and incomplete cat localization with CDA results.
- Ablation Studies: Pairwise training with original and augmented images outperforms training on augmented images alone, while pasting objects from different categories helps decouple contextual dependence.The pairwise strategy helps the classifier learn more discriminative features.
- Analysis of pseudo labels and Segmentation masks: CDA improves different WSSS baselines, with SEAM achieving the best segmentation-mask performance on both PASCAL VOC validation and testing sets.The study reports more accurate and complete pseudo-masks covering object areas.
- Comparison with State-of-the-arts: On PASCAL VOC, SEAM with CDA outperforms other state-of-the-art methods by a large margin, while IRNet with CDA ranks second; on COCO, CDA-IRNet reaches 33.7% mIoU, 1.1% above the previous best.The comparison uses matching DeepLab architectures for fairness and includes validation and testing sets where reported.
5. Conclusion
The paper concludes that Context Decoupling Augmentation improves weakly supervised semantic segmentation by copying and pasting network-provided object instances into images. With pairwise training, CDA boosts multiple WSSS methods to state-of-the-art performance.
- 5. Conclusion: CDA uses two-stage training to copy and paste network-provided object instances into input images for weakly supervised semantic segmentation.The method is designed to narrow the gap with fully supervised segmentation.
- 5. Conclusion: Pairwise training with augmented and non-augmented images helps the classifier distinguish more discriminative object features.The conclusion identifies pairwise training as an additional mechanism for improving object-feature learning.
- 5. Conclusion: Experiments show that CDA boosts various WSSS methods to new state-of-the-art performance.The conclusion summarizes the reported gains across the evaluated methods.