Source-linked AI summary
Squeeze-and-Attention Networks for Semantic Segmentation
Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Zhifeng Li, Wei-Shi Zheng, Jonathan Li, Alexander Wong
TL;DR
Semantic segmentation methods often overlook pixel grouping and remain constrained by convolutional grid structure. The paper proposes SANets with SA modules that jointly model pixel-group attention and pixel-wise prediction through multistage feature integration. SANets achieve 54.4% mIoU on PASCAL Context and promising performance on two challenging benchmarks.
Problem
Semantic segmentation methods largely ignore pixel grouping and are constrained by the grid structure of convolution kernels, despite requiring both grouping and pixel-wise prediction.
Method
SANet uses squeeze-and-attention modules with an additional partially down-sampled convolutional channel for pixel-group attention, alongside four multistage heads for multi-scale prediction.
Results
54.4% mIoU is achieved by the EfficientNet SANet on PASCAL Context, setting a state-of-the-art result for that dataset.
Takeaways & Limitations
The authors conclude that SA modules and SANets provide promising segmentation performance while addressing pixel grouping and dense prediction together.
Abstract
from arXiv · showhide
The recent integration of attention mechanisms into segmentation networks improves their representational capabilities through a great emphasis on more informative features. However, these attention mechanisms ignore an implicit sub-task of semantic segmentation and are constrained by the grid structure of convolution kernels. In this paper, we propose a novel squeeze-and-attention network (SANet) architecture that leverages an effective squeeze-and-attention (SA) module to account for two distinctive characteristics of segmentation: i) pixel-group attention, and ii) pixel-wise prediction. Specifically, the proposed SA modules impose pixel-group attention on conventional convolution by introducing an 'attention' convolutional channel, thus taking into account spatial-channel inter-dependencies in an efficient manner. The final segmentation results are produced by merging outputs from four hierarchical stages of a SANet to integrate multi-scale contexts for obtaining an enhanced pixel-wise prediction. Empirical experiments on two challenging public datasets validate the effectiveness of the proposed SANets, which achieves 83.2% mIoU (without COCO pre-training) on PASCAL VOC and a state-of-the-art mIoU of 54.4% on PASCAL Context.
1. Introduction
The paper reframes semantic segmentation as both pixel-wise prediction and pixel grouping, then introduces SA modules and SANets to address both dimensions with multi-scale features.
- Motivation: Prior segmentation methods mainly improve pixel-level performance while largely ignoring the implicit task of pixel grouping.Existing context aggregation improves pixel-wise labeling but does not directly encourage same-class pixels to group together.
- Motivation: Grid-structured convolution kernels restrict learned spatial-feature shapes, while feature aggregation leaves global image understanding underexploited.The paper motivates reducing these local spatial constraints through pixel grouping without spatial limitation.
- Proposed approach: SANets use four SA modules on multistage backbone outputs to learn multi-scale spatial and non-local spectral features.The architecture integrates multistage features for improved object boundaries and scene parsing outcomes.
- Motivation: Semantic segmentation is disentangled into pixel-wise dense prediction and pixel grouping, which emphasizes connections among pixels.Pixel-wise prediction labels each pixel, while pixel grouping connects pixels belonging to the same class.
- Proposed approach: The squeeze-and-attention module accounts for multi-scale dense prediction and spatial attention of pixel groups.Its design targets both segmentation sub-tasks rather than pixel-wise prediction alone.
2. Related Works
Prior work improves segmentation through multi-scale context and channel-wise attention, while SANet adds pixel-group attention through a partially down-sampled convolutional channel.
- Multi-scale contexts: Multi-scale contextual features are combined through pyramid structures, multi-path refinement, encoder-decoder fusion, and pooling-based approaches.These methods aim to improve discriminative feature extraction and segmentation outputs across feature levels.
- Multi-scale contexts: Gather-excite reduces local convolution constraints by gathering long-range contextual features, while SANet merges outputs from different backbone stages.The related approaches motivate combining broader context with multistage feature integration.
- Channel-wise attention: Channel-wise attention reweights feature-map channels to increase representational power, as exemplified by squeeze-and-excitation modules.SE-based approaches are used for image classification and categorical recognition in segmentation-related models.
- Pixel-group attention: SA replaces fully squeezed channel attention with a down-sampled convolutional channel that produces categorical-specific soft attention masks for pixel grouping.The same channel also adds scaled spatial features to conventional convolution channels for pixel-level prediction.
- Pixel-group attention: The SA approach addresses pixel-level dense prediction and the largely ignored pixel-group attention.It adapts channel reweighting toward segmentation-specific spatial and grouping requirements.
3. Framework
SANet combines squeeze-and-attention modules with residual backbones to model pixel-group attention alongside dense pixel-wise prediction. Four hierarchical SA heads integrate multi-scale features, while auxiliary masks and losses guide segmentation outputs.
- Squeeze-and-attention module: The SA module augments convolution with a re-weighting channel that preserves spatial information for pixel-group attention.Its attention channel is down-sampled rather than fully squeezed, then up-sampled to match the main convolution channel.
- Squeeze-and-attention network: Class-wise masks from hierarchical SA heads are supervised by categorical regions and guide the FCN head's pixel-wise prediction.This combines pixel-group attention extraction with multi-scale contextual features in the final segmentation process.
- Squeeze-and-attention network: SANet uses SA heads on four backbone stages to extract multi-scale spatial and non-local spectral features.Dilated ResNet and EfficientNet backbones provide the recognition features consumed by the hierarchical SA modules.
- Squeeze-and-attention network: SANet training combines dense prediction, mask, and categorical cross-entropy losses, with α weighting categorical loss and β weighting auxiliary loss.The reported ablation fixes β = 0.8 when selecting α and fixes α = 0.2 when selecting β.
- Implementation: The implementation reduces input channels to one-fourth in both main and attention channels and uses an attention-channel downsample and upsample ratio of 8.The SA heads are adapted to the class number of each segmentation dataset.
- Results: SANet produces more accurate sample segmentations than an FCN baseline, particularly around object boundaries, while complex contexts can still yield failures.The comparison uses a dilated ResNet50 FCN and SANet with the same backbone.
4. Experimental Results
Experiments evaluate SANets through ablations, comparisons with SE modules and baselines, qualitative visualizations, and PASCAL VOC performance under different pretraining settings. Results show improved segmentation accuracy, clearer boundaries, and favorable efficiency relative to a heavier model.
- PASCAL Context results: 54.4% mIoU is achieved by the EfficientNet SANet on PASCAL Context, establishing a state-of-the-art result.The ResNet101 SANet achieves 53.0% mIoU, while complex images remain challenging for both SANet and the FCN baseline.
- SA module comparison: 4.1% and 4.5% mIoU increases are obtained by SANets over corresponding FCN baselines with SE modules.The comparison includes ResNet50 and ResNet101 backbones; FCN-SE models barely improve segmentation results.
- Qualitative analysis: Attention channels have broad low-level receptive views but focus around selected points at high levels, while main channels preserve local boundaries or extract semantic features.The visualization compares SA head1 and head4 feature maps, including main channel, attention channel, and output.
- PASCAL VOC results: 83.2% mIoU is achieved on PASCAL VOC without COCO pretraining, exceeding competing models and leading several classes.The evaluation excludes models using extra datasets such as JFT for fairness.
- PASCAL VOC results: 86.1% mIoU is achieved on PASCAL VOC with COCO pretraining, comparable to top-ranking models and better than RefineNet with ResNet152.The SA module is reported as more computationally efficient than EncNet’s encoding module.
- Efficiency analysis: SANet surpasses the reimplemented Deeplab V3+ by 1.7% mIoU without COCO pretraining while using 55.5M parameters, less than one-fourth of SDN’s 238.5M.Both models use ResNet101 backbones for the comparison.
5. Conclusion
The paper reframes semantic segmentation as pixel-wise prediction plus pixel grouping and introduces SA modules and SANets to address both dimensions. SANets achieve promising results on two challenging benchmarks.
- Conclusion: Semantic segmentation is reconsidered as two independent dimensions: pixel-wise prediction and pixel grouping.Pixel grouping accounts for connections among pixels that belong to the same class.
- Conclusion: SA modules address the implicit pixel-grouping sub-task while enhancing pixel-wise dense prediction.The conclusion presents SA modules as the core mechanism underlying the proposed architecture.
- Conclusion: SANets achieve promising segmentation performance on two challenging benchmarks.The paper positions the architecture as a simple yet effective approach intended to facilitate further segmentation research.