Source-linked AI summary
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation
Jiwoon Ahn, Suha Kwak
TL;DR
Semantic segmentation needs costly pixel-level labels, while image-level supervision lacks object location and shape information. The paper uses AffinityNet to learn local semantic affinities from incomplete CAMs, propagates activations with random walk, and generates synthetic segmentation labels. On PASCAL VOC 2012, models trained with these labels outperform prior methods using the same supervision and compete with methods using stronger supervision or extra data.
Problem
Pixel-level segmentation suffers from expensive annotation, while image-level labels indicate object presence without the location and shape information needed for segmentation.
Method
AffinityNet predicts affinities between adjacent coordinates from CAM-derived supervision, and random walk uses them to revise CAMs and generate synthetic labels for training a segmentation model.
Results
On PASCAL VOC 2012, the method substantially outperforms previous state-of-the-art approaches with the same supervision and is competitive with methods using stronger supervision or extra data.
Takeaways & Limitations
Image-level class labels can support competitive semantic segmentation when learned pixel-level affinity propagation converts incomplete CAMs into training labels.
Takeaways & Limitations
AffinityNet relies on CAMs as incomplete supervision, which can be inaccurate even though carefully manipulated CAMs provide reliable affinity supervision.
Abstract
from arXiv · showhide
The deficiency of segmentation labels is one of the main obstacles to semantic segmentation in the wild. To alleviate this issue, we present a novel framework that generates segmentation labels of images given their image-level class labels. In this weakly supervised setting, trained models have been known to segment local discriminative parts rather than the entire object area. Our solution is to propagate such local responses to nearby areas which belong to the same semantic entity. To this end, we propose a Deep Neural Network (DNN) called AffinityNet that predicts semantic affinity between a pair of adjacent image coordinates. The semantic propagation is then realized by random walk with the affinities predicted by AffinityNet. More importantly, the supervision employed to train AffinityNet is given by the initial discriminative part segmentation, which is incomplete as a segmentation annotation but sufficient for learning semantic affinities within small image areas. Thus the entire framework relies only on image-level class labels and does not require any extra data or annotations. On the PASCAL VOC 2012 dataset, a DNN learned with segmentation labels generated by our method outperforms previous models trained with the same level of supervision, and is even as competitive as those relying on stronger supervision.
1. Introduction
The paper addresses the ill-posedness and data cost of pixel-level semantic segmentation using only image-level labels. AffinityNet learns local semantic affinities from incomplete CAM supervision, then random walk propagates responses to generate labels for training a segmentation model.
- Pixel-level segmentation is limited by expensive annotations, scarce examples, and insufficient class diversity in existing datasets.
- Image-level labels are scalable but reveal object presence without the location and shape information required for segmentation.
- AffinityNet predicts semantic affinities between adjacent image coordinates and uses them to guide random-walk propagation of CAM activations.
- CAMs provide incomplete but locally reliable supervision for learning semantic affinities within small image areas.
- The framework computes affinity labels from CAMs, revises CAMs through random walk, synthesizes segmentation labels, and trains a final segmentation network.
- The approach trains AffinityNet with image-level class labels only and avoids heavy reliance on off-the-shelf techniques.
- On PASCAL VOC 2012, the method achieves state-of-the-art performance under equal supervision and remains competitive with stronger supervision or external data.
2. Related Work
Related work uses weaker annotations, localization cues, affinity learning, or synthetic labels to reduce the need for pixel-level segmentation supervision. The paper distinguishes its approach through learned propagation with an end-to-end trainable AffinityNet.
- Bounding boxes, scribbles, and points reduce annotation demands but still require human intervention and remain costly at scale.
- Image-level labels require little or no human intervention, but their coarseness makes direct segmentation training unsatisfactory.
- Prior methods combine discriminative seeds with superpixels, segmentation, motion, or salient regions to recover object shape, often requiring extra data, supervision, or preprocessing.
- AffinityNet learns from data how to propagate local activations across an object, unlike methods using fixed optimization or off-the-shelf preprocessing.
- Pixel-level affinity matrices have been learned with segmentation labels and combined with random walk or architectural mechanisms for refinement.
3. Our Framework
The framework synthesizes pixel-level labels from image-level class labels using CAMs, AffinityNet, and a segmentation model. AffinityNet learns local, class-agnostic semantic affinities from carefully selected CAM regions and uses them to propagate activations while addressing training-pair imbalance.
- Framework overview: The framework first synthesizes pixel-level segmentation labels from image-level class labels, then trains a segmentation model with those generated annotations.It uses three DNNs: a CAM network, AffinityNet, and the final segmentation model.
- Computing CAMs: CAMs provide segmentation seeds and supervision for AffinityNet, despite typically highlighting only local salient object parts.CAMs are computed using a classification network with global average pooling and a fully connected layer; irrelevant classes are suppressed and background activations are estimated separately.
- AffinityNet: AffinityNet predicts class-agnostic semantic affinity between adjacent coordinates using L1 distances between features in its convolutional feature map.Its multi-level backbone features are reduced, concatenated, and adapted through 1×1 convolutions to incorporate multiple fields of view.
- Semantic propagation: Random walk uses AffinityNet affinities to propagate CAM activations toward nearby semantically identical areas and away from other classes.The resulting revised CAMs provide the synthesized supervision for the downstream segmentation model.
- AffinityNet training: Affinity labels are generated by identifying confident object and background regions from refined CAMs, assigning 1 to same-class coordinate pairs and 0 to different-class pairs.Pairs involving neutral coordinates are ignored, and only pairs within a search radius are used because distant affinities are difficult to predict and expensive to compute.
- AffinityNet training: The training loss separates positive object, positive background, and negative pairs to reduce severe class imbalance in locally sampled coordinate pairs.The loss is class-agnostic, allowing one representation to be shared across object classes and background.
4. Network Architectures
The framework uses a shared modified ResNet38 backbone for three DNNs: CAM generation, AffinityNet, and segmentation. AffinityNet aggregates multi-level features, while the segmentation model adds atrous convolutions for its output.
- Backbone: All three DNNs use a modified ResNet38 backbone with 38 convolution layers and wide channels.The original global average pooling and fully connected layers are removed, and later convolutional levels use atrous convolutions.
- CAM network: The CAM network adds a 3×3 convolution, global average pooling, and a fully connected classification layer to the backbone.
- AffinityNet: AffinityNet aggregates multi-level backbone feature maps to exploit semantic information from different fields of view.
- Segmentation model: The segmentation model adds two atrous convolution layers with dilation rate 12 and channel sizes 512 and 21, producing Ours-ResNet38.
5. Experiments
Experiments evaluate synthesized labels and segmentation models on PASCAL VOC 2012 using IoU-based comparisons. Random walk with AffinityNet substantially improves label quality, and models trained with the resulting labels outperform prior image-level methods while remaining competitive with stronger supervision.
- Evaluation: Intersection-over-Union (IoU) is used to evaluate segmentation performance on PASCAL VOC 2012.
- Dataset: The benchmark uses 10,582 training images and 1,449 validation images after adding segmentation annotations from prior work.
- Label synthesis: Random walk with learned AffinityNet substantially improves synthesized-label quality over CAM, while dCRF provides a further slight improvement.The final dCRF-refined labels are used to train the segmentation network.
- Label synthesis: AffinityNet detects semantic boundaries from image-level supervision, helping random walk handle CAM false positives and missing object areas.
- Comparisons to Previous Work: Compared with CrawlSeg, Ours-ResNet38 captures larger object areas, misses fewer objects, and produces smoother boundaries without dCRF at final inference.
- Comparisons to Previous Work: Both Ours-ResNet38 and Ours-DeepLab outperform the prior image-level state of the art on PASCAL VOC 2012 validation and test sets.Ours-ResNet38 is slightly better than Ours-DeepLab because of ResNet38's stronger representation.
- Comparisons to Previous Work: The models substantially outperform approaches using the same supervision plus extra data or annotations and remain competitive with scribble- and bounding-box-supervised methods.Ours-ResNet38 also outperforms the early fully supervised FCN model.
- Comparisons to Previous Work: Ours-DeepLab recovers 86% of its bound, while Ours-ResNet38 achieves 77%.
6. Conclusion
The paper proposes AffinityNet-based label synthesis using image-level class labels only. On PASCAL VOC 2012, models trained with these labels outperform prior methods with equivalent supervision and compete with methods using stronger supervision or extra data.
- Conclusion: The framework generates accurate training segmentation labels from image-level class labels only, addressing the lack of annotated segmentation data.
- Conclusion: On PASCAL VOC 2012, models trained with the generated labels substantially outperform prior methods with the same supervision level.
- Conclusion: The resulting models are competitive with methods requiring stronger supervision or extra data.
A. Appendix
The appendix supplies omitted technical details, deeper quantitative analyses, additional qualitative results, and brief future-work remarks.
- Appendix: Sections A.1 and A.2 provide technical details of the proposed framework.
- Appendix: Sections A.3 and A.4 analyze the quantitative results in depth, while Section A.5 provides additional qualitative results.
- Appendix: The appendix concludes with brief remarks about future work.
A.1. Architecture Details of Our Networks
The framework uses a ResNet38-based backbone with atrous convolutions and an output stride of 8. Its CAM network adds adaptation, global average pooling, and fully connected layers for image classification.
- Backbone network: ResNet38 forms the backbone, with its last three convolutional levels converted to atrous convolutions.The L5 and L6 dilation rates are doubled and quadrupled, respectively.
- Backbone network: The modified backbone produces a final convolutional feature map with output stride 8.This stride is four times smaller than the original referenced configuration.
- CAM network: The CAM network adds a 3x3 adaptation convolution, global average pooling, and a fully connected layer above the backbone.These layers support learning the network using a single classification criterion.
A.2. Practical Details for Inference
Inference applies the segmentation model to ten flipped and rescaled versions of each test image. This test-time jittering slightly improves validation performance, while the section also documents the relevant architecture and evaluation artifacts.
- Test-time augmentation: Each test image is horizontally flipped and rescaled by five predefined ratios, producing 10 input versions.The ratios are 1/2, 3/4, 1, 5/4, and 3/2.
- Architecture reference: Figure 7 illustrates the detailed network architectures used in the framework.The supplied figure references identify this as the architecture overview.
- Test-time augmentation: Test-time jittering improves the model’s PASCAL VOC 2012 validation performance by less than 2% in mIoU.The improvement is described as slight.
- Evaluation reference: Table 5 reports synthesized-label and final-model accuracy in mIoU on the VOC 2012 validation set.The table is the evaluation artifact for comparing the two outputs.
A.3. Analysis on Effects of the Hyper-parameters
The synthesized-label method is relatively insensitive to its hyper-parameters over the analyzed settings. However, parameter values were selected without ground-truth segmentation labels, so the reported settings are not guaranteed to be optimal.
- Hyper-parameter sensitivity: Synthetic-label quality fluctuates within only 1.0 mIoU when β ≈ 2 log t − 8.This result is reported for the analyzed β and t relationship.
- Hyper-parameter sensitivity: Synthetic-label accuracy saturates when α is greater than 16.The analysis uses this saturation to characterize the method’s robustness.
- Selection constraint: The paper’s parameter values are not optimal because hyper-parameter analyses using ground-truth segmentation labels were unavailable in the weakly supervised setting.The authors instead qualitatively evaluated a small training-image subset.
- Selection constraint: For α, β, and t, tuning relied on qualitative evaluation rather than ground-truth segmentation labels.The default dCRF parameter values were taken from the original code for the same reason.
A.4. Justification of Learning the Fully Supervised Segmentation Network
The final segmentation network improves substantially over the synthesized labels used to train it. Qualitative results are provided for both label synthesis and final segmentation, while future work considers transfer learning and boundary detection.
- Quantitative comparison: 58.7 mIoU for CAM+RW+dCRF rises to 61.7 mIoU for the final segmentation network on VOC 2012 validation images.The reported gap supports training a separate segmentation model with the synthesized labels.
- Quantitative comparison: The CAM+RW+dCRF comparison has an advantage because it uses ground-truth image-level labels to filter irrelevant CAM classes.Without those labels, its score would be lower according to the passage.
- Future work: Future work includes using source-domain ground-truth segmentation labels to train AffinityNet for transfer to exclusive target-domain classes.The passage also proposes weakly supervised semantic boundary detection.
- Qualitative results: Figure 9 presents qualitative examples of the segmentation-label synthesis procedure on VOC 2012 training images.These examples are supplementary to the quantitative comparison.
- Qualitative results: Figure 10 presents additional qualitative segmentation results for the final Ours-ResNet38 model on VOC 2012 validation images.The figure is identified as semantic segmentation results on the validation set.