Source-linked AI summary
Discovering Class-Specific Pixels for Weakly-Supervised Semantic Segmentation
Arslan Chaudhry, Puneet K. Dokania, Philip H. S. Torr
TL;DR
Pixel-level annotations are laborious, motivating weakly supervised semantic segmentation from image labels. The paper combines class-specific fully convolutional attention maps with hierarchical class-agnostic saliency to form pixel-level training cues. It reports 60.8% and 61.9% mIoU on PASCAL VOC 2012 val and test sets, respectively, with gains over the published state of the art.
Problem
Pixel-level annotations are laborious, while image-level labels are easier to collect and can support training on many downloaded images.
Source: §1 · p. 1Method
The method combines class-specific fully convolutional attention maps with hierarchical class-agnostic saliency to create approximate pixel-level training cues.
Source: §1 · p. 2Results
60.8% mIoU on PASCAL VOC 2012 val and 61.9% on test set; gains over the published state-of-the-art results are 5.1% and 5.2%, respectively.
Source: §1 · p. 2, §5.2 · p. 8Takeaways & Limitations
Combining attention and hierarchical saliency enabled reliable pixel-level class-specific cues and new state-of-the-art results on PASCAL VOC 2012.
Source: §6 · p. 10Weakly supervised methods have difficulty detecting object boundaries under severe occlusion because their training cues are weak.
Source: §6 · p. 10
Abstract
from arXiv · showhide
We propose an approach to discover class-specific pixels for the weakly-supervised semantic segmentation task. We show that properly combining saliency and attention maps allows us to obtain reliable cues capable of significantly boosting the performance. First, we propose a simple yet powerful hierarchical approach to discover the class-agnostic salient regions, obtained using a salient object detector, which otherwise would be ignored. Second, we use fully convolutional attention maps to reliably localize the class-specific regions in a given image. We combine these two cues to discover class-specific pixels which are then used as an approximate ground truth for training a CNN. While solving the weakly supervised semantic segmentation task, we ensure that the image-level classification task is also solved in order to enforce the CNN to assign at least one pixel to each object present in the image. Experimentally, on the PASCAL VOC12 val and test sets, we obtain the mIoU of 60.8% and 61.9%, achieving the performance gains of 5.1% and 5.2% compared to the published state-of-the-art results. The code is made publicly available.
1 Introduction
Pixel-level annotations are costly and impede scaling segmentation to new categories, motivating supervision from easier-to-collect image labels. The paper combines class-specific attention with class-agnostic saliency to generate training cues and reports new PASCAL VOC 2012 results.
1 Introduction
- Pixel-level annotation takes almost four minutes per image on average and impedes scaling segmentation networks to new object categories.Source: §1 · p. 1
- Image-level labels are easier to collect—almost one second per class or object category—and support webly supervised learning with downloaded images.Source: §1 · p. 1
- Attention maps identify class-specific discriminative pixels, while hierarchical saliency discovers additional foreground regions missed by a salient object detector.The combined cues provide approximate pixel-level ground truth for training a segmentation network.Source: §1 · p. 2
- Training combines segmentation and auxiliary classification losses, with pixel-level cues updated as training progresses.The stated intuition is that pixel-wise cross-entropy improves localization cues.Source: §1 · p. 2
- 60.8% mIoU on PASCAL VOC 2012 val and 61.9% on test set new state-of-the-art results for image-label supervision.Source: §1 · p. 2
2 Related Works
Prior weakly supervised segmentation work uses varied supervision, including image labels, saliency masks, scribbles, points, and object proposals. This work distinguishes itself through hierarchical saliency erasing and a shared, end-to-end network for attention and segmentation.
2 Related Works
- Earlier approaches used bounding boxes and image labels, saliency masks, scribbles, annotated points, or image labels alone as supervision.Other methods used simple-to-complex training, proposal selection, or combinations of losses.Source: §2 · p. 2
- Unlike [34], this method erases salient regions to find new ones while keeping the attention network intact; Figure 1 shows later maps revealing more salient objects and reducing false activations.Source: §2 · p. 3, Figure 1 · p. 3
- Unlike [34], the method uses one network for attention and segmentation and trains it end-to-end; unlike [12], it extends attention-and-saliency cues beyond single-category images.Source: §2 · p. 3
3 Preliminaries
The paper uses salient-object masks for class-agnostic foreground cues and attention maps for class-specific pixel scores. Its weakly supervised setting learns pixel labels from images paired with object tags rather than pixel-level annotations.
3 Preliminaries
- Saliency here means binary masks for visually noticeable foreground objects, using salient-object detection rather than eye-fixation prediction.Source: §3 · p. 3
- Attention maps assign each pixel a score in [0,1] for its likelihood of belonging to an object category.The paper defines attention in terms of pixels to which the CNN is most sensitive when classifying an image.Source: §3 · pp. 3–4
- The schematic uses the same network for image classification and segmentation, yielding fully convolutional attention maps without training a new classification network.Green-headed arrows indicate the backward pass.Source: Figure 2 · p. 4
- Weakly supervised segmentation learns pixel labels from images with relatively easy annotations such as object tags, rather than pixel-level class annotations.Source: §3 · p. 4
4 Discovering Class-Specific Pixels for Weakly-Supervised Semantic Segmentation
The approach builds approximate pixel labels by combining class-specific attention with class-agnostic saliency, then uses them to train a segmentation network. It adds multi-object attention, hierarchical salient-region discovery, and iterative cue updates to this process.
4 Discovering Class-Specific Pixels for Weakly-Supervised Semantic Segmentation
- The method builds weak pixel labels from multi-object attention localization, hierarchical salient-region mining, and combined cues for network training.Source: §4 · p. 4
4.1 Fully Convolutional Attention Network
- The FCAN uses class-specific convolutional filters before global average pooling to produce multi-object attention maps within the segmentation network.Its attention can remain spatially limited and sometimes spill onto background pixels, motivating combination with saliency cues.Source: §4.1 · pp. 5–6
4.2 Hierarchical Saliency For Multiple Salient Objects
- Hierarchical saliency erases detected regions and reruns the detector to uncover additional objects; final cues combine class attention and saliency by harmonic mean and thresholding.Source: §4.2 · p. 6
4.3 Training
- Training combines pixel-wise segmentation and auxiliary image-level classification losses, then periodically refreshes localization cues from the network’s pixel predictions.Source: §4.3 · p. 7
5 Experimental Results, Comparisons and Analysis
The experiments compare DCSP with weakly and semi-supervised methods on PASCAL VOC12 and analyze hierarchical saliency, cue adaptation, and joint training. The reported results include gains over prior approaches and improvements from these components.
5.1 Dataset and Experimental Setup
- The experiments use PASCAL VOC12 with 10,582 training images, 1,449 validation images, and 1,456 test images, without additional data in the train/test pipeline.DHSNet supplies saliency masks; the setup uses two erasing thresholds, and the network is trained, cue-adapted, then trained further.Source: §5.1 · pp. 7–8
5.2 Comparison with State-of-the-arts
- 5.1% and 5.2% gains over Joon et al. [20] on PASCAL VOC12 val and test, respectively, accompany DCSP’s reported state-of-the-art comparison.The authors also report that their VGG16 variant maintains state-of-the-art performance and that DCSP remains competitive without CRF post-processing (60.8% to 59.5%).Source: §5.2 · pp. 8–9, Table 1 · p. 9
- DCSP outperforms compared baselines even when some use stronger supervision or additional data, and its saliency network needs no retraining for new object categories.The comparison includes methods using scribbles, bounding boxes, MCG, spots, or additional images; the table caption notes several such dependencies.Source: §5.2 · p. 8, Table 1 · p. 9
5.3 Analysis
- 4.5% gain in mIOU is reported for hierarchical saliency (S3) versus not using it (S1), and Table 3 illustrates recovery of otherwise-missed multiple objects.The authors also report that adapting localization cues removes many false positives; the qualitative figure notes failures under severe occlusion.Source: §5.3 · p. 9, Table 2 · p. 9, Table 3 · p. 9
- 11.2% gain in mIOU is reported for jointly training saliency and attention cues in a unified segmentation network.The authors attribute the benefit to learning parameters and a shared feature space for the combined segmentation task rather than separately mapped cues.Source: Table 4 · p. 9, §5.3 · p. 10
6 Conclusion and Future Work
The paper proposes discovering class-specific pixels by combining FCAN attention cues with hierarchical saliency masks. It reports state-of-the-art results on PASCAL VOC12 and identifies severe occlusion as a limitation of weakly supervised methods.
Contribution
- The proposed method combines class-specific attention cues from FCAN with class-agnostic hierarchical saliency masks to obtain pixel-level cues for weakly supervised segmentation.Source: §6 · p. 10
Results
- The experiments report new state-of-the-art results on the PASCAL VOC 2012 dataset.Source: §6 · p. 10
Limitation and Future Work
- Weakly supervised methods are limited in detecting object boundaries under severe occlusion, which the authors attribute to weak training cues.They suggest exploring edge- and shape-based priors as future work.Source: §6 · p. 10
Supplementary Material
The supplementary analysis examines performance across erasing steps and the effect of adapting localization cues during training.
Supplementary analysis
- The supplementary analysis compares network performance at different erasing steps and examines the effect of adapting localization cues during training.Source: Supplementary Material · p. 11
Performance Analysis at Different Erasing Steps
The first erasing step produces a significant performance boost, while a second yields only a small gain and performance remains consistent. Accuracy remains poor for elongated or disjointed categories such as chairs, which the authors link to localization challenges.
Performance Analysis at Different Erasing Steps
- The first erasing step (S2) brings a significant performance boost, while the second (S3) yields only a small gain and performance remains consistent.The authors suggest S2 is sufficient in most cases because the average number of salient objects in PASCAL remains small.Source: Performance Analysis at Different… · p. 11
- Table 6 reports chair IoUs of 14.9%, 20.9%, and 20.8% after S1, S2, and S3, respectively, versus aeroplane IoUs of 73.7%, 75.6%, and 79.4%.With full pixel-level supervision, chair IoU is 30.7% compared with 84.4% for aeroplane [23].Source: Performance Analysis at Different… · p. 11
- The authors suspect elongated shapes and occlusion contribute to poor results, since combining attention and saliency may produce non-contiguous cues that fail on disjointed objects.They propose edge- and shape-based priors as a possible way to improve such categories.Source: Performance Analysis at Different… · p. 11
Adaptive Training
Adaptive training updates localization cues using network outputs constrained by image-level labels. The qualitative examples show fewer false-positive activations after the adapt step.
Adaptive Training
- Constraining the adapt step with image-level labels removes many false-positive activations, including background pixels assigned to foreground and extra predicted classes.The examples show background corrected for person and plane, and extra classes removed; the authors say this produces more refined training cues.Source: Adaptive Training · pp. 11–12
- Tables 5 and 6 compare segmentation accuracies by object category across S1, S2, and S3, the saliency maps from the original image and after successive erasing steps.The captions identify Table 5 as PASCAL VOC 2012 val and Table 6 as test.Source: Table 5 · p. 12, Table 6 · p. 13
- Figures 4 and 5 provide qualitative comparisons of adaptive training on PASCAL VOC12 val images, with and without CRF post-processing.The captions identify whether CRF was applied but do not state a comparative outcome.Source: Figure 4 · p. 14, Figure 5 · p. 14