Source-linked AI summary
Attention to Scale: Scale-aware Semantic Image Segmentation
Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, Alan L. Yuille
TL;DR
Semantic segmentation systems use multi-scale features, but conventional merging treats scales equally or selects them sparsely. This paper learns pixel-level attention over scale-specific features, jointly trained with extra per-scale supervision, and consistently improves strong baselines while providing diagnostic visualizations.
Problem
Multi-scale segmentation methods commonly merge features with average- or max-pooling, limiting adaptive weighting across image positions and scales.
Method
The method jointly learns pixel-level attention to softly weight shared-network features across input scales, with extra supervision at each scale.
Results
Across three datasets, the method consistently improves strong baselines, including a 6.4% improvement over DeepLab-LargeFOV on PASCAL VOC 2012.
Takeaways & Limitations
Attention improves multi-scale feature merging over average- and max-pooling while enabling diagnostic visualization, and extra per-scale supervision is essential for excellent performance.
Takeaways & Limitations
The models do not outperform current best methods that jointly train conditional random fields with fully convolutional networks.
Abstract
from arXiv · showhide
Incorporating multi-scale features in fully convolutional neural networks (FCNs) has been a key element to achieving state-of-the-art performance on semantic image segmentation. One common way to extract multi-scale features is to feed multiple resized input images to a shared deep network and then merge the resulting features for pixelwise classification. In this work, we propose an attention mechanism that learns to softly weight the multi-scale features at each pixel location. We adapt a state-of-the-art semantic image segmentation model, which we jointly train with multi-scale input images and the attention model. The proposed attention model not only outperforms average- and max-pooling, but allows us to diagnostically visualize the importance of features at different positions and scales. Moreover, we show that adding extra supervision to the output at each scale is essential to achieving excellent performance when merging multi-scale features. We demonstrate the effectiveness of our model with extensive experiments on three challenging datasets, including PASCAL-Person-Part, PASCAL VOC 2012 and a subset of MS-COCO 2014.
1. Introduction
Semantic image segmentation assigns semantic labels to every pixel and benefits substantially from multi-scale features in FCN-based models. This work introduces scale-dimension soft attention for share-nets, jointly trained with per-scale supervision and evaluated across three challenging datasets.
- Semantic image segmentation assigns labels such as “person” or “dog” to every pixel and supports applications including image editing, augmented reality, and self-driving vehicles.
- FCN-based segmentation methods achieve strong benchmark results, with multi-scale features identified as a key element of successful performance.
- Share-nets process several resized inputs through a shared deep network and typically fuse scales using average- or max-pooling, treating features as equally important or sparsely selected.
- The proposed method explores attention in the scale dimension, using soft weights to focus on the most relevant multi-scale features for each object and pixel location.The model adapts a state-of-the-art segmentation network to a share-net and learns weights according to object scale.
- Extra supervision at every scale is essential for better performance, and the attention model and multi-scale networks are trained jointly across PASCAL-Person-Part, PASCAL VOC 2012, and a subset of MS-COCO 2014.The experiments report consistent improvements over strong baselines.
2. Related Work
The work builds on deep networks, multi-scale features for semantic segmentation, and attention models. It specifically extends DeepLab with share-net multi-scale inputs and attention that learns pixelwise scale importance.
- Deep networks: DCNNs achieve state-of-the-art results in image classification and object detection, while FCN variants lead semantic image segmentation.The proposed method builds upon the state-of-the-art DeepLab model.
- Multi-scale features: Multi-scale segmentation methods include skip-nets, which fuse features from different network levels, and share-nets, which process resized inputs through a shared network.Examples include FCN-8s, hypercolumns, zoom-out features, and multi-scale DeepLab variants.
- Attention models for deep networks: Attention models have been used for image classification, object detection, image generation, machine translation, image captioning, and video captioning.Prior methods apply attention to image regions, words, or spatial and temporal dimensions.
- Attention to scale: Unlike prior captioning and translation methods that attend over spatial, temporal, or linguistic dimensions, this work uses attention to identify relevant input scales.The approach is motivated by attention models that softly weight input importance.
- Attention to scale: Average-pooling and max-pooling are common ways to merge multi-scale predictions, whereas the proposed model jointly learns pixelwise weights over scale-specific features.Its final output is a weighted sum of score maps across all input scales.
3. Model
The model applies a shared-weight DeepLab FCN to images at multiple scales and merges their resized score maps using learned, position- and scale-specific attention weights. The attention module is trained end-to-end with the FCN, while extra supervision is added at each scale alongside final-output supervision.
- DeepLab backbone: DeepLab converts VGG-16 into a fully convolutional network that produces dense feature maps for semantic segmentation.Its original fully connected layers become convolutional layers, including a final 1×1 spatial convolutional kernel.
- Attention-based merging: The attention model learns soft weights for each spatial position and scale to compute a weighted merge of the multi-scale score maps.The weights are shared across channels, and their maps can visualize feature importance at different positions and scales.
- Multi-scale processing: A shared-weight DeepLab processes resized inputs at multiple scales, producing one score map per scale.The score maps are then resized to a common resolution using bilinear interpolation.
- Attention-based merging: Average pooling and max pooling over scales are special cases of the proposed attention formulation.Average pooling uses weights of 1/S, whereas max pooling uses a max operation with ws_i = 1 for every scale and position.
- Training: End-to-end joint training backpropagates through the attention model and FCN, avoiding pixel-level ground-truth scale annotations.The final prediction applies softmax to the merged score maps and is optimized with averaged cross-entropy using SGD.
- Training: Extra supervision is added to the FCN output at each scale in addition to supervision on the final merged output.This encourages the model to merge discriminative multi-scale features for final classification.
4. Experimental Evaluations
Experiments across PASCAL-Person-Part, PASCAL VOC 2012, and a subset of MS-COCO 2014 evaluate multi-scale inputs, feature-merging methods, and extra supervision. Across datasets, two scales generally outperform one, while three-scale attention performs best when paired with extra supervision and produces interpretable scale-specific weight maps.
- Experimental design: Experiments vary one to three input scales, average-, max-pooling, or attention-based feature merging, and training with or without extra supervision.The input scales are s ∈ {1, 0.75, 0.5}.
- PASCAL-Person-Part: On PASCAL-Person-Part, two input scales outperform one and slightly exceed three-scale average-pooling or attention, while max-pooling is robust to three-scale fusion.The authors hypothesize that three-scale fusion requires sufficiently discriminative features because direct fusion can degrade performance.
- Qualitative analysis: Attention weight maps are interpretable: scale 1 focuses on small objects, scale 0.75 on middle-scale objects, and scale 0.5 on large objects or background.Failure cases arise from extremely difficult human poses and confusion between clothing and person parts.
- PASCAL VOC 2012: On PASCAL VOC 2012, three scales with attention and extra supervision achieve a 6.8% improvement over the DeepLab-LargeFOV baseline and outperform DeepLab-MSc-LargeFOV by 4.69%.Extra supervision is necessary for effective three-scale merging, especially with average-pooling and attention.
- PASCAL VOC 2012: On the VOC 2012 test set, attention improves over average pooling by 1%, over DeepLab-LargeFOV by 6.4%, and over DeepLab-MSc-LargeFOV by 4.5%.Randomly scaling training images from 0.6 to 1.4 adds another 0.6% improvement.
- MS-COCO 2014: On MS-COCO, multi-scale inputs, attention, and extra supervision improve the DeepLab-LargeFOV baseline by 4.6% and DeepLab-MSc-LargeFOV by 4.17%.The DeepLab-LargeFOV baseline has mean IOU 31.22% because of MS-COCO’s object-scale variance and larger number of classes.
5. Conclusion
The paper adapts DeepLab-LargeFOV for multi-scale semantic segmentation, showing that multi-scale inputs improve performance and attention-based feature merging surpasses pooling while enabling diagnostic visualization.
- 5. Conclusion: Multi-scale inputs yield better semantic-segmentation performance than single-scale input across experiments on three datasets.The adapted model is DeepLab-LargeFOV.
- 5. Conclusion: The proposed attention model improves multi-scale feature merging over average- and max-pooling baselines while enabling diagnostic visualization of feature importance across positions and scales.The attention mechanism is evaluated within the adapted DeepLab-LargeFOV model.
A. More qualitative results
The paper presents additional qualitative segmentation results across PASCAL-Person-Part, PASCAL VOC 2012, and a subset of MS-COCO 2014.
- A. More qualitative results: Additional qualitative results are shown on PASCAL-Person-Part in Fig. 8.
- A. More qualitative results: Additional qualitative results are shown on PASCAL VOC 2012 in Fig. 9.
- A. More qualitative results: Additional qualitative results are shown on a subset of MS-COCO 2014 in Fig. 10.