Source-linked AI summary
Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi- Supervised Semantic Segmentation
Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng, Thomas S. Huang
TL;DR
Weakly supervised segmentation is limited by the difficulty of producing high-quality dense object localization from image-level supervision. The paper equips a classification network with convolutional blocks using different dilation rates to transfer discriminative information to non-discriminative regions. It reports new state-of-the-art Pascal VOC results in both weakly- and semi-supervised settings.
Problem
Weakly supervised segmentation lacks reliable methods for densely localizing object regions from image-level supervision, limiting the quality of object cues for segmentation training.
Method
The approach augments a standard classification network with multiple dilated convolutional blocks whose varying rates transfer discriminative information to adjacent object regions.
Results
60.8% and 67.6% mIoU are achieved on the Pascal VOC test set in weakly- and semi-supervised settings, respectively, as new state-of-the-art results.
Takeaways & Limitations
Dense localization maps generated by the classification network can be used to learn semantic segmentation models in both weakly- and semi-supervised settings.
Takeaways & Limitations
A failure case occurs for large objects when discriminative regions are sparsely highlighted at one end, making transfer to the other end difficult with small dilation rates.
Abstract
from arXiv · showhide
Despite the remarkable progress, weakly supervised segmentation approaches are still inferior to their fully supervised counterparts. We obverse the performance gap mainly comes from their limitation on learning to produce high-quality dense object localization maps from image-level supervision. To mitigate such a gap, we revisit the dilated convolution [1] and reveal how it can be utilized in a novel way to effectively overcome this critical limitation of weakly supervised segmentation approaches. Specifically, we find that varying dilation rates can effectively enlarge the receptive fields of convolutional kernels and more importantly transfer the surrounding discriminative information to non-discriminative object regions, promoting the emergence of these regions in the object localization maps. Then, we design a generic classification network equipped with convolutional blocks of different dilated rates. It can produce dense and reliable object localization maps and effectively benefit both weakly- and semi- supervised semantic segmentation. Despite the apparent simplicity, our proposed approach obtains superior performance over state-of-the-arts. In particular, it achieves 60.8% and 67.6% mIoU scores on Pascal VOC 2012 test set in weakly- (only image-level labels are available) and semi- (1,464 segmentation masks are available) supervised settings, which are the new state-of-the-arts.
1. Introduction
The paper targets the difficulty of densely localizing object regions from image-level supervision, proposing multi-rate dilated convolution to transfer discriminative information across objects. The resulting localization maps support weakly- and semi-supervised segmentation, achieving state-of-the-art Pascal VOC performance.
- Dense object localization from image-level labels remains a critical challenge for weakly supervised semantic segmentation.
- The approach uses a standard classification network to generate dense localization maps that train semantic segmentation models in weakly- and semi-supervised settings.
- CAM often identifies only sparse discriminative parts, especially on large objects, rather than dense object regions required for segmentation.
- Multiple dilated convolutional blocks with varying rates enlarge receptive fields and transfer high-response context to adjacent low-response object regions.
- 60.8% and 67.6% mIoU are achieved on the Pascal VOC test set in weakly- and semi-supervised settings, respectively, as new state-of-the-art results.
2. Related Work
Prior work reduces segmentation annotation demands through coarse or image-level supervision, but image-level approaches still struggle to obtain sufficiently informative object regions. The paper motivates transferring information across regions using dilated convolution.
- Segmentation with Coarse Annotations: Pixel-level annotation is labor intensive, motivating segmentation with bounding boxes, scribbles, instance points, or image-level labels.
- Segmentation with Image-level Annotations: Image-level supervision is attractive because it is easy to obtain, and prior methods include multiple instance learning, expectation-maximization, proposals, and additional networks.
- Motivation: Varying dilated rates transfers information from initially discriminative regions to other regions, with corresponding localization maps shown in Figure 2.
- Segmentation with Image-level Annotations: SEC obtains small and sparse object-related seeds, which cannot provide enough information for learning reliable segmentation models.
3. The Proposed Approach
The approach augments classification networks with multiple dilated convolutional blocks to transfer discriminative information into non-discriminative object regions and produce dense localization maps for segmentation training.
- 3.1. Revisiting Dilated Convolution: Dilated convolution enlarges receptive fields, enabling discriminative information to transfer from highlighted object parts to adjacent non-discriminative regions.Varying dilation rates supports information propagation across object regions at multiple scales.
- 3.2. Multi-dilated Convolution for Localization: Using dilation rates d = 3, 6, 9 complements standard convolution maps by highlighting low-response object regions that d = 1 often misses.Localization maps generated at different rates are complementary and are integrated for denser coverage.
- 3.1. Revisiting Dilated Convolution: The proposed network appends convolutional blocks with multiple dilation rates to a VGG16-based classification model for object localization.The network removes fully connected layers and one pooling layer before appending multi-rate blocks.
- 3.2. Multi-dilated Convolution for Localization: The anti-noise fusion strategy averages maps from dilated blocks and adds the result to the standard-convolution map to form the final localization map.This preserves accurately mined regions while suppressing unrelated regions.
- 3.2. Multi-dilated Convolution for Localization: Pixels above threshold δ in the final map provide foreground cues, while low-saliency pixels provide background cues for segmentation training.The resulting dense localization maps are applied to weakly and semi-supervised segmentation models.
- 3.3.1 Weakly-supervised Learning: Weakly supervised training combines pseudo masks from dense localization maps with online predicted masks as supervision for an FCN.The framework extracts confidence maps for ground-truth image-level labels to infer online masks.
- 3.3.2 Semi-supervised Learning: Semi-supervised training combines strongly annotated images and weakly annotated images in a shared segmentation network.The two supervision sources are integrated by sharing network parameters.
4.1. Dataset and Settings
The approach is evaluated on the PASCAL VOC 2012 segmentation benchmark using pixel mIoU averaged over 21 categories.
- Dataset and Evaluation Metrics: PASCAL VOC 2012 contains one background category and 20 object categories with segmentation annotations.The augmented training set contains 10,582 images, while validation and test subsets contain 1,449 and 1,456 images.
- Dataset and Evaluation Metrics: Performance is measured by pixel mIoU averaged across 21 categories.The experiments use image-level supervision only for the stated training setup.
4.2. Comparison with State-of-the-arts
The method outperforms image-level weakly supervised baselines and achieves better semi-supervised results than WSSL under the same setting on PASCAL VOC 2012.
- Weakly-supervised Semantic Segmentation: 2.3% higher validation mIoU than Hong et al. is achieved using only 10K training images, despite Hong et al. using additional video data.The authors attribute the competing method’s improvement mainly to temporal information from video.
- Weakly-supervised Semantic Segmentation: More than 2.1% higher test-set mIoU than other methods establishes a new state-of-the-art for the weakly supervised benchmark.The approach uses only image-level labels and outperforms all compared image-level methods.
- Weakly-supervised Semantic Segmentation: The weakly supervised results approach scribble- and box-based methods and exceed spot-based approaches by more than 8.8% without pixel-level supervision.Some comparison methods use additional images or implicit pixel-level supervision through proposals.
- Semi-supervised Semantic Segmentation: 65.7% versus 64.6% validation mIoU and 67.6% versus 66.2% test mIoU are obtained against WSSL under the same semi-supervised setting.The strong/weak split contains 1.4K strongly annotated and 9K weakly annotated images.
- Semi-supervised Semantic Segmentation: The method achieves competitive or better mIoU than approaches using bounding boxes, despite using weaker image-level supervision for the weakly annotated data.This comparison evaluates both validation and test sets.
4.3. Ablation Analysis
The ablation analysis shows that multi-dilated localization maps complement one another: larger dilation rates recover missed object regions but can introduce false positives, which anti-noise fusion suppresses. Dense fused maps improve segmentation performance and support reliable thresholding across object scales, while a documented failure remains for large objects with sparse cues at one end.
- Dense object localization: Multi-dilated blocks produce complementary localization maps, with d = 1 offering high precision but low recall and larger rates recovering additional object regions.Blocks with d = 3, 6, and 9 highlight regions missed by d = 1, including parts of cats and motorbikes.
- Dense object localization: Anti-noise fusion averages maps from enlarged dilation rates and sums them with the d = 1 map, retaining shared true positives while reducing rate-specific false positives.True positive regions are usually shared across maps, whereas false positives vary with dilation rate.
- Dense object localization: A fixed threshold can extract most object regions accurately from dense localization maps regardless of object scale, unlike the d = 1 maps alone.The d = 1 maps require lower thresholds for large objects and higher thresholds for small objects.
- Dense object localization: One failure case occurs when a large object has discriminative regions sparsely highlighted at one end, preventing small dilation rates from transferring knowledge across its full extent.The authors suggest adversarial erasing as a possible way to address this case.
- Ablation results: 57.1% mIoU is achieved with the anti-noise fusion strategy, improving over 50.3% to 54.4% obtained as dilation rates increase.Averaging all convolutional blocks, including d = 1, lowers mIoU by almost 1% relative to the current fusion strategy.
- Segmentation results: 60.4% and 60.8% mIoU are achieved on the validation and test sets, respectively, outperforming other weakly supervised methods.In semi-supervised learning, performance drops only 0.9% when strong images decrease from 1.4K to 500; 2.9K strong images yield 68.5% mIoU.
5. Conclusion
The paper proposes using multiple convolutional blocks with different dilation rates to generate dense object localization maps for weakly and semi-supervised segmentation. It reports new state-of-the-art mIoU scores on both tasks and identifies failure-case handling and larger-scale evaluation as future work.
- Conclusion: The approach uses multiple convolutional blocks with different dilation rates to generate dense object localization maps.The maps are produced with a classification network and used to learn semantic segmentation networks.
- Conclusion: The generated localization maps support semantic segmentation learning in both weakly and semi-supervised settings.The paper describes the approach as easy to implement and generic across these settings.
- Conclusion: The method achieves new state-of-the-art mIoU scores on the two segmentation tasks.The conclusion presents this as the paper’s main empirical outcome.
- Conclusion: Future work includes extending discriminative regions across failure cases and evaluating the method on large-scale datasets such as MS COCO and ImageNet.The paper specifically notes cases where discriminative regions must be extended from one end of an object to the other.