Source-linked AI summary
Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation
Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, Xilin Chen
TL;DR
Image-level supervision leaves CAM-based segmentation incomplete and inconsistent because it lacks the equivariant constraint of pixel-level segmentation. SEAM adds transformed-image consistency regularization and a pixel correlation module, producing more consistent CAMs and state-of-the-art PASCAL VOC 2012 performance with image-level annotations.
Problem
Image-level CAM supervision often fails to provide complete, clean, transformation-consistent object masks because of the gap between weak and fully supervised segmentation.
Method
SEAM uses equivariant consistency regularization across transformed images and a pixel correlation module that refines CAMs using inter-pixel similarity.
Results
SEAM achieves state-of-the-art performance on PASCAL VOC 2012 using only image-level annotations.
Takeaways & Limitations
The method provides additional self-supervision that makes CAMs more consistent across transformed inputs and better aligned with object shapes.
Abstract
from arXiv · showhide
Image-level weakly supervised semantic segmentation is a challenging problem that has been deeply studied in recent years. Most of advanced solutions exploit class activation map (CAM). However, CAMs can hardly serve as the object mask due to the gap between full and weak supervisions. In this paper, we propose a self-supervised equivariant attention mechanism (SEAM) to discover additional supervision and narrow the gap. Our method is based on the observation that equivariance is an implicit constraint in fully supervised semantic segmentation, whose pixel-level labels take the same spatial transformation as the input images during data augmentation. However, this constraint is lost on the CAMs trained by image-level supervision. Therefore, we propose consistency regularization on predicted CAMs from various transformed images to provide self-supervision for network learning. Moreover, we propose a pixel correlation module (PCM), which exploits context appearance information and refines the prediction of current pixel by its similar neighbors, leading to further improvement on CAMs consistency. Extensive experiments on PASCAL VOC 2012 dataset demonstrate our method outperforms state-of-the-art methods using the same level of supervision. The code is released online.
1. Introduction
Image-level weak supervision makes semantic segmentation cheaper but leaves CAMs incomplete, over-activated, and inconsistent under transformations. SEAM adds equivariant self-supervision and pixel correlation to improve CAM quality, achieving state-of-the-art performance on PASCAL VOC 2012 with image-level annotations.
- Semantic segmentation requires expensive pixel-level labels, motivating weakly supervised approaches based here on image-level classification labels.
- CAMs often cover only discriminative object parts, activate background regions, and change inconsistently under affine transformations.These shortcomings reflect the supervision gap between fully and weakly supervised semantic segmentation.
- SEAM applies consistency regularization to CAMs from transformed images, using self-supervision to narrow the supervision gap.
- PCM captures context appearance information and revises CAMs through learned affinity attention maps, while ECR couples PCM and self-supervision in a siamese network.
- SEAM achieves state-of-the-art performance on PASCAL VOC 2012 using only image-level annotations.
2. Related Work
Related work mainly refines CAMs generated from image-level labels, while self-supervised learning creates labels through annotation-free pretext tasks. SEAM combines these directions by using segmentation equivariance as additional supervision.
- Weakly supervised semantic segmentation uses labels such as bounding boxes, scribbles, or image-level classifications instead of massive pixel annotations.
- Many image-level WSSS methods refine CAMs to approximate segmentation masks, using strategies such as seed-expand-constrain, adversarial erasing, and pixel-affinity transitions.
- Self-supervised learning designs pretext tasks that generate labels without additional manual annotations, including spatial transformation prediction and image inpainting.
- Because image-level labels are too weak to fit object boundaries, the paper uses equivariance as a pretext task to provide additional self-supervision.
3. Approach
SEAM narrows the supervision gap in image-level WSSS by enforcing CAM equivariance across affine transformations and refining CAMs with pixel-context correlations. Its siamese architecture combines equivariant regularization, PCM, and specialized losses to improve consistency and object coverage.
- SEAM overview: SEAM integrates equivariant regularization and a pixel correlation module within a shared-weight siamese network.The two branches process original and transformed images, coupling CAM consistency with context-based refinement.
- Pixel correlation refinement: Self-attention refines each pixel by aggregating information from other pixels according to learned feature similarity.The formulation uses embedded features and similarity weights to combine contextual information with the current signal.
- Pixel correlation refinement: PCM refines original CAMs with low-level pixel features and cosine-based affinities, producing weighted combinations with normalized similarities.ReLU suppresses negative similarities, while the module is designed to preserve the original CAM activation intensity.
- Equivariant regularization: Classification supervision localizes objects, while equivariant regularization constrains CAMs to remain consistent under affine transformations.The transformation branch warps the input before feedforward, whereas the other branch transforms the network output for regularization.
- Loss design: The ECR loss regularizes PCM outputs against original CAMs from the other siamese branch to avoid CAM degeneration during refinement.Directly regularizing revised CAMs led to a local minimum in which all pixels received the same class, motivating cross-branch supervision.
- Loss design: Background scores are added because zero foreground CAM vectors at background positions cannot provide gradients for learning background pixel affinities.Foreground activations are normalized after suppressing non-maximum responses, and a thresholded background score is used during inference.
4. Experiments
Experiments on PASCAL VOC 2012 show that SEAM improves CAM quality and pseudo-label accuracy by combining equivariant regularization with PCM, outperforming baseline and alternative localization refinements.
- Experimental Setup: The evaluation uses PASCAL VOC 2012 with image-level classification labels during network training.The augmented training set contains 10,582 images, while the official split has 1,464 training, 1,449 validation, and 1,456 test images.
- Comparison with Baseline: SEAM’s ablation reaches 55.41% mIoU on PASCAL VOC train pseudo labels after applying OHEM, compared with 52.40% for baseline CAMs refined by dense CRF.The full SEAM configuration further reaches 56.83% mIoU after aggregating dense CRF as post-processing.
- Affine Transformation: Rescaling with a 0.3 down-sampling rate improves pseudo-label mIoU from 47.43% to 55.41%, whereas combining additional transformations yields little further benefit.The tested alternatives include rotation, translation, horizontal flip, and rescaling.
- Affine Transformation: Increasing the baseline rescaling range alone does not improve pseudo-label accuracy, indicating that gains come from PCM combined with equivariant regularization rather than augmentation range.This comparison isolates the contribution of the proposed modules from a simple data-augmentation change.
- CAM Quality: SEAM achieves lower mFN and mFP than baseline, indicating more complete activation coverage and fewer over-activated pixels across image scales.Its curves are also more consistent across scales, supporting the operation of equivariance regularization during learning.
- Comparison with State-of-the-arts: The final synthesized pseudo labels achieve 63.61% mIoU, and SEAM significantly improves validation and test performance over baseline under the same training setting.The paper reports state-of-the-art performance on the PASCAL VOC 2012 test set using only image-level labels; qualitative results cover both large and small objects.
5. Conclusion
SEAM narrows the supervision gap in weakly supervised semantic segmentation by enforcing equivariant consistency and refining CAMs with pixel similarity. Its pseudo-labels produce state-of-the-art performance on PASCAL VOC 2012.
- SEAM narrows the supervision gap by adding equivariant regularization that enforces consistency across transformed inputs.The method uses a siamese network with efficient regularization losses.
- PCM refines CAMs by learning inter-pixel similarity, improving their consistency and fit to ground-truth mask shapes.
- The generated CAMs remain consistent across transformed inputs and better fit the shapes of ground-truth masks.
- Retraining the segmentation network with synthesized pixel-level pseudo-labels achieves state-of-the-art performance on PASCAL VOC 2012.