Source-linked AI summary
Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation
Jungbeom Lee, Eunji Kim, Sungroh Yoon
TL;DR
Weakly supervised segmentation often localizes only small discriminative object regions, while pixel-level annotation is expensive. AdvCAM expands attribution maps by anti-adversarially increasing the target classification score and regularizing irrelevant or already-high attributions. On PASCAL VOC 2012 test images, it achieves mIoUs of 68.0 and 76.9 for weakly and semi-supervised segmentation, respectively.
Problem
Classifier attribution maps used for weakly supervised segmentation often cover only small discriminative regions instead of the whole target object.
Method
AdvCAM iteratively perturbs image pixels along gradients that increase the target classification score, with regularization suppressing unrelated regions and limiting already-high attributions.
Results
68.0 and 76.9 mIoU are achieved on PASCAL VOC 2012 test images for weakly and semi-supervised semantic segmentation, respectively.
Takeaways & Limitations
AdvCAM is a post-hoc analysis that can improve existing segmentation methods without modifying or retraining their classifiers.
Takeaways & Limitations
Too much repetitive adversarial manipulation can remain problematic even when the adversarial-climbing loss surface is reasonably flat.
Abstract
from arXiv · showhide
Weakly supervised semantic segmentation produces a pixel-level localization from a classifier, but it is likely to restrict its focus to a small discriminative region of the target object. AdvCAM is an attribution map of an image that is manipulated to increase the classification score. This manipulation is realized in an anti-adversarial manner, which perturbs the images along pixel gradients in the opposite direction from those used in an adversarial attack. It forces regions initially considered not to be discriminative to become involved in subsequent classifications, and produces attribution maps that successively identify more regions of the target object. In addition, we introduce a new regularization procedure that inhibits the incorrect attribution of regions unrelated to the target object and limits the attributions of the regions that already have high scores. On PASCAL VOC 2012 test images, we achieve mIoUs of 68.0 and 76.9 for weakly and semi-supervised semantic segmentation respectively, which represent a new state-of-the-art.
1. Introduction
Weakly supervised semantic segmentation reduces annotation costs but classifier attribution maps often cover only small discriminative object regions. AdvCAM expands these regions through anti-adversarial image manipulation and regularization, improving existing methods without network modification or retraining.
- Motivation: Pixel-level semantic segmentation labels every image pixel, but pixel-level annotation is costly, taking about 4 minutes per image on average and over 90 minutes for some large scenes.These costs motivate weaker forms of supervision.
- Motivation: Class-label supervision is cheaper than pixel-level labeling and is therefore the most popular form of weak supervision.Class labels can take about 20 seconds to obtain, and many public datasets already include them.
- Problem: Classifier attribution maps usually identify small discriminative regions rather than the whole target object, making them unsuitable as direct training targets.Existing methods extend these regions by manipulating images or feature maps.
- Existing approaches: Erasure finds new object regions after removing discriminative ones, but it requires network changes or extra training and may cross a decision boundary to produce erroneous maps.FickleNet instead diversifies maps using random feature-map dropout and aggregation.
- AdvCAM: AdvCAM perturbs pixels along gradients that increase the target classification score, moving away from the decision boundary so relevant nondiscriminative regions become involved.Iterative manipulation produces successive CAMs that identify more of the target object.
- AdvCAM: Regularization suppresses other-class scores and limits attribution scores in already-high-scoring regions, reducing irrelevant activations and excessive emphasis.The resulting maps serve as pseudo ground-truth masks for weakly and semi-supervised training.
- Contribution: AdvCAM is a post-hoc analysis that improves existing weakly supervised segmentation methods without modifying or retraining their networks.The paper reports new state-of-the-art performance on PASCAL VOC 2012 in both weakly and semi-supervised settings.
2. Related Work
Related work improves weakly supervised segmentation by refining classifier seeds and growing object regions, while semi-supervised methods combine limited pixel labels with weak or absent annotations. Adversarial methods can instead manipulate inputs or models to produce deceptive attribution maps.
- Weakly supervised segmentation: Weakly supervised segmentation methods improve classifier-derived seeds and grow them to approximate the full region occupied by each target object.The two goals are improving the initial seed and expanding it during segmentation.
- Obtaining a High Quality Seed: Seed-refinement methods use equivariance regularization or latent semantic classes to improve attribution-map quality and feature learning.Latent semantic classes are subcategories of annotated parent classes that can be pseudo-labeled.
- Growing the Object Region: Region-growing methods propagate confident semantic labels to ambiguous neighboring pixels, including during segmentation-network training.SEC and DSRG begin with CAM seeds containing ambiguous regions and allocate pseudo-labels to them.
- Semi-supervised segmentation: Semi-supervised segmentation trains with a small number of pixel-annotated images alongside a much larger number of weakly annotated or unlabeled images.CCT enforces prediction invariance across perturbations, while other methods use synthesized images or adversarial training.
- Adversarial methods: Adversarial attack methods manipulate inputs to deceive DNNs, and can produce spurious attribution maps through image or model-parameter manipulation.Such attacks aim to replace an attribution map with one highlighting another image location.
3. Proposed Method
AdvCAM extends classifier-based localization by iteratively increasing the target-class score, causing additional relevant object regions to enter the CAM, while regularization suppresses unrelated activations and limits already-high attributions.
- Motivation: CAMs usually cover only small discriminative regions, leaving them inadequate for representing the target object's full extent.This motivates methods that extend attribution regions for weakly supervised segmentation.
- Adversarial Climbing: AdvCAM reverses adversarial attack by iteratively perturbing the image along the target-class score gradient, increasing the classification score and involving more object regions.At step t, the manipulated image is updated from the gradient of the class-c logit at the previous step.
- Adversarial Climbing: The localization map A aggregates CAMs from successive manipulated images so that previously non-discriminative yet relevant features contribute to the final attribution.The aggregation combines CAMs obtained at each adversarial-climbing iteration.
- Regularization: The loss landscape analysis compares adversarial climbing and attack using interpolations of a normal vector and a random vector around the original image.The paper uses this visualization to examine the curvature associated with the two image-manipulation directions.
- Regularization: Regularization suppresses logits for other classes and restricts high attributions so background or other-object regions are inhibited and existing strong regions remain near their original scores.The restricting mask M identifies regions whose current CAM attribution exceeds threshold τ, while λ controls the masking regularization strength.
- Adversarial Climbing: Adversarial climbing enhances non-discriminative features more than discriminative ones, with regularization increasing the difference to > 2.5× versus < 2× without it.This produces denser CAMs and more effective discovery of new target-object regions.
4. Experiments
Experiments on PASCAL VOC 2012 show that AdvCAM improves seed and pseudo-ground-truth quality and achieves strong weakly and semi-supervised segmentation performance under class-label supervision.
- Experimental setup: Experiments use PASCAL VOC 2012 images with image-level annotations for weak supervision and mixed pixel-level and class-label annotations for semi-supervision.Evaluation uses mIoU on validation and test images.
- Mask quality: Our initial seeds are 6.8% better than the original CAM baseline and outperform other compared methods.The comparison includes CRF-refined seeds and pseudo-ground-truth masks.
- Mask quality: AdvCAM outperforms competing pseudo-ground-truth masks regardless of whether PSA or IRN seed refinement is used.The comparison evaluates both refinement techniques for fairness.
- Weakly supervised semantic segmentation: 68.0 mIoU is achieved on PASCAL VOC 2012 test images with image-level annotations alone, exceeding the IRN baseline by 4.6% on validation images.The validation mIoU is 68.1, while the test mIoU is 68.0.
- Weakly supervised semantic segmentation: The method outperforms methods using salient-object masks or extra web images and is comparable to methods using bounding-box supervision.These comparisons involve stronger or additional supervision than image-level labels alone.
- Semi-supervised semantic segmentation: 76.9 mIoU is achieved on PASCAL VOC 2012 test images in the semi-supervised setting, outperforming methods under the same supervision and even a method using bounding-box labels.The validation mIoU is 77.8, and the validation result is 4.6% above the CCT baseline.
5. Discussion
Iterative adversarial climbing initially improves seed quality, while regularization enables continued localization with less noise and makes the method effective across seed generators.
- Iterative adversarial climbing: Initial-seed mIoU rises steeply with or without regularization, but the unregularized curve peaks around iteration 8.The analysis evaluates seed quality across adversarial iterations.
- Iterative adversarial climbing: Regularization keeps noise much lower than in the initial CAM and enables discovery of new target-object regions through as many as 30 adversarial steps.Without regularization, noise rises steeply after several iterations as background regions become activated.
- Hyper-parameter sensitivity: The masking regularization coefficient improves seed mIoU from 50.43 at λ = 0 to 55.55 at λ = 7, a gain exceeding 5%.The performance curve flattens after λ = 5.
- Hyper-parameter sensitivity: Varying the masking threshold τ between 0.3 and 0.7 changes seed mIoU by less than 1%.The threshold controls how many pixels retain attribution values similar to the original CAM.
- Generality of the method: AdvCAM improves the quality of initial seeds by an average of over 4% when combined with multiple seed-generation methods without modifying or retraining them.The evaluated generators include Chang et al., SEAM, and IRN.
- Feature-level analysis: Feature-level visualizations use t-SNE to show trajectories for correctly classified cat and bird images and their manipulated versions.The feature set combines class images with 20 manipulated images for representative cat and bird examples.
6. Conclusion
AdvCAM expands small discriminative regions through pixel-level image manipulation, improving target-object localization without modifying or retraining the classifier. It integrates with weakly supervised segmentation networks and achieves state-of-the-art performance in weakly and semi-supervised settings.
- AdvCAM uses classifier-gradient pixel perturbations to increase the perturbed image’s classification score and expand target-object attribution maps.The method manipulates images using gradients of the classifier output with respect to the input image.
- The approach is post-hoc, so it requires no classifier modification or retraining and can be integrated into existing methods.
- AdvCAM combined with recent weakly supervised semantic segmentation networks achieves new state-of-the-art performance in both weakly and semi-supervised semantic segmentation.
A.1. Implementation Details
The implementation independently manipulates eight multi-scale, flipped images for adversarial climbing and uses multi-scale testing with max-pooled segmentation outputs.
- Adversarial climbing independently manipulates eight images formed by flipping and resizing one image at scales 0.5, 1.0, 1.5, and 2.0.CAMs from the eight images are otherwise aggregated by pixel-wise sum pooling.
- Semantic segmentation uses DeepLab-v2-ResNet101 with multi-scale inference at scales 0.5, 0.75, 1.0, and 1.25.The independently generated outputs are aggregated by pixel-wise max pooling into the final segmentation map.
A.2. Additional Analysis
Additional analyses select thresholds for initial seeds, evaluate regularization across adversarial iterations, compare per-class scores, and provide further mask and localization examples.
- Threshold analysis: Threshold analysis evaluates initial-seed mIoU across thresholds for CAM, AdvCAM without regularization, and AdvCAM with regularization.The analysis separates foreground and background in map A using threshold values.
- Threshold analysis: t = 8 is selected for AdvCAM without regularization, while t = 27 is selected for AdvCAM with regularization.These are the best threshold values for the respective settings according to the referenced analysis.
- Effects of suppressing other classes: Suppressing other classes provides better adversarial manipulation across adversarial iterations than omitting that regularization.The comparison evaluates initial-seed mIoU with and without the regularization while excluding masking in all experiments.
- Comparison of per-class mIoU scores: Table A1 compares per-class mIoU scores for the proposed method and recently produced methods.
- Additional examples: Figures A2 and A3 provide additional semantic-mask examples and successive attribution maps from iterative adversarial climbing.The mask examples include FickleNet, IRN, CCT, and the proposed method.