Source-linked AI summary
Complementary Patch for Weakly Supervised Semantic Segmentation
Fei Zhang, Chaochen Gu, Chenyue Zhang, Yuchao Dai
TL;DR
Image-level WSSS suffers because CAMs identify seeds mainly in a few highly discriminative regions, leaving incomplete foreground pseudo-masks. The paper uses complementary patched image pairs, a triplet CP Network, and PRCM to enrich and refine CAMs, reporting 67.8% validation and 68.5% test performance on PASCAL VOC 2012. The authors conclude that the method achieves new state-of-the-art WSSS performance, while its benefit depends on patch size.
Problem
CAM-based image-level WSSS discovers seeds from limited highly discriminative regions, which may leave foreground pseudo-masks incomplete.
Method
The method combines Complementary Patch representations, a triplet CP Network with three regularization functions, and PRCM for contextual CAM refinement.
Results
67.8% and 68.5% mIoU are reported on the PASCAL VOC 2012 validation and test sets, respectively, establishing new state-of-the-art performance in WSSS.
Takeaways & Limitations
The results support using complementary-patch information and CP Network refinement to obtain more informative CAMs for image-level WSSS.
Takeaways & Limitations
The information gain and performance depend on patch size, with extreme patch sizes unable to discover new seed areas effectively.
Abstract
from arXiv · showhide
Weakly Supervised Semantic Segmentation (WSSS) based on image-level labels has been greatly advanced by exploiting the outputs of Class Activation Map (CAM) to generate the pseudo labels for semantic segmentation. However, CAM merely discovers seeds from a small number of regions, which may be insufficient to serve as pseudo masks for semantic segmentation. In this paper, we formulate the expansion of object regions in CAM as an increase in information. From the perspective of information theory, we propose a novel Complementary Patch (CP) Representation and prove that the information of the sum of the CAMs by a pair of input images with complementary hidden (patched) parts, namely CP Pair, is greater than or equal to the information of the baseline CAM. Therefore, a CAM with more information related to object seeds can be obtained by narrowing down the gap between the sum of CAMs generated by the CP Pair and the original CAM. We propose a CP Network (CPN) implemented by a triplet network and three regularization functions. To further improve the quality of the CAMs, we propose a Pixel-Region Correlation Module (PRCM) to augment the contextual information by using object-region relations between the feature maps and the CAMs. Experimental results on the PASCAL VOC 2012 datasets show that our proposed method achieves a new state-of-the-art in WSSS, validating the effectiveness of our CP Representation and CPN.
1. Introduction
The paper addresses incomplete CAM foreground seeds in image-level WSSS by treating seed expansion as an information-increase problem. It introduces Complementary Patch representations, a triplet CP Network, and PRCM, achieving state-of-the-art PASCAL VOC 2012 performance.
- Motivation: Image-level WSSS relies on CAMs, but their focus on highly discriminative regions leaves many foreground areas incomplete.Region-erasing methods can lose image regions during training, while iterative hiding introduces additional computation.
- Core idea: Complementary Patch representation combines CAMs from complementary patched image pairs to incorporate more foreground information than the original CAM.The information increase depends on patch size and disappears in extreme cases where the patch is too large or too small.
- Method: CPN uses a triplet network with Triplet CP and CP Cross Regularization losses to narrow the gap between the original CAM and the summed CP-pair CAMs.Grid Patch and Super-pixel Patch templates generate the complementary pairs.
- Results: 67.8% and 68.5% mIoU are reported on the PASCAL VOC 2012 validation and test sets, respectively, establishing new state-of-the-art performance in WSSS.The reported performance also varies with patch size, consistent with the CP representation analysis.
- Method: PRCM further refines CAMs by incorporating contextual information through pixel-region relationships.It is presented as an additional module beyond the CP representation and CPN regularization.
2. Related Work
Related work in image-level WSSS primarily improves CAM-derived seeds through pooling changes, pixel or semantic relations, extra resources, and region erasing or mining. The framework is also connected to self-attention methods that use contextual feature relationships for segmentation-mask refinement.
- Weakly supervised semantic segmentation: Image-level WSSS methods modify CAM seeds through pooling-based approaches, inter-pixel or semantic relations, extra resources, and region erasing or mining.Examples include super-pixel pooling, affinity-based random walks, web images, videos, and saliency maps.
- Self-attention model: Self-attention segmentation methods refine feature maps with contextual relationships, including non-local, channel-attention, spatial-attention, and object-contextual representations.These methods motivate using relationships among spatial points or regions to improve segmentation masks.
- Method context: The proposed framework is illustrated as a triplet architecture that processes the original image and complementary-patch pair alongside PCM and PRCM refinement.Its outputs are constrained by classification, triplet-CP, and CP-cross-regularization losses.
3. Proposed Methods
The method expands CAM object seeds using complementary hidden-image pairs and information-based regularization, then refines CAMs with pixel-region context. CPN combines these components through a triplet network and complementary patch strategies.
- 3.1. Complementary Patch Representation: CAM represents object c through class-specific activation maps derived from feature maps and classifier weights.The CAM for class c is computed as the classifier weight transpose applied to the feature map.
- 3.1. Complementary Patch Representation: A Complementary Patch Pair consists of two images with complementary hidden regions whose sum reconstructs the original image.The two inputs are denoted Ih and I_h̄, with Ih + I_h̄ = I.
- 3.1. Complementary Patch Representation: The CP representation treats seed expansion as increased information: summing CP-Pair CAMs preserves baseline seeds and can add newly discovered object regions.Except under two extreme cases, the summed CAMs contain more foreground seeds than the baseline CAM.
- 3.1. Complementary Patch Representation: CPN narrows the gap between the original CAM and the weighted sum of CP-Pair CAMs using a shared-weights triplet network and CP regularization.The weighting depends on the quantity of uncovered pixels, and gradients from the CP-Pair outputs are stopped while updating the original branch.
- 3.2. Complementary Patch Strategies: CP pairs are generated with Grid Patch or Super-pixel Patch strategies, differing in whether fixed-size grids or SLIC superpixels provide the hidden regions.Grid Patch hides each patch with probability ph = 0.5, while Super-pixel Patch uses a predefined segment number.
- 3.3. Modules in CPN: PRCM augments CAM context by modeling pixel-region relations, and the final refined CAM sums the PCM and PRCM outputs.PRCM treats CAMs as soft object regions and strengthens them using relations between feature-map regions and pixels.
4. Experiments
Experiments on PASCAL VOC 2012 evaluate CPN’s CAM improvements, component effectiveness, foreground localization, patch settings, and final segmentation performance. CPN improves CAM quality and achieves strong results, while patch size and hidden probability materially affect performance.
- CAM Improvements: CPN improves CAM mIoU over the baseline across single- and multi-scale tests, with a nearly 10% multi-scale improvement over the baseline.The PRCM-enhanced SEAM* also outperforms original SEAM across tested scales.
- CAM Improvements: 57.43% multi-scale CAM mIoU is reported for CPN, compared with SEAM at the cited multi-scale result.
- Foreground Localization: CPN finds more foreground regions than the baseline and SEAM, reaching 56.13% foreground mIoU and exceeding them by 9.75% and 2.07%, respectively.Visualizations show more seeds in low-response areas, although small objects can be over-segmented.
- Patch Size: Super-pixel patching peaks at 57.43% mIoU with SN = 200, while excessive or insufficient patch sizes reduce CAM quality.At SN = 8000, performance reaches 55.79%.
- Patch Size: Grid Patching peaks at 57.07% mIoU with K = {56, 112}, while extreme settings reach 55.67% and 55.80%.Super-pixel Patching performs 0.43% better on average than Grid Patching.
- Hidden Probability: The hidden probability ph = 0.5 achieves the best performance at 57.43% mIoU, whereas ph = 0.1 reaches 55.52%.
- Final Segmentation: After Random Walk refinement, CPN pseudo-label mIoU reaches 67.79%, followed by final-mask evaluation with DeepLab and post-CRF refinement.
5. Conclusion
The paper concludes that complementary patches can mine additional foreground seeds from image-level supervision. CPN combines this representation with regularization and PRCM, achieving new state-of-the-art performance on PASCAL VOC 2012.
- Conclusion: The proposed pipeline targets weakly supervised semantic segmentation using only image-level labels.
- Conclusion: CP Representation uses complementary-patch image pairs whose summed CAMs mine more foreground seeds than the original CAM.
- Conclusion: CPN uses regularization to improve CAMs, while PRCM adds contextual information through pixel-region relations.
- Conclusion: Experiments on PASCAL VOC 2012 show that CPN achieves new state-of-the-art performance.