Source-linked AI summary
Pyramid Attention Network for Semantic Segmentation
Hanchao Li, Pengfei Xiong, Jie An, Lingxue Wang
TL;DR
Semantic segmentation must preserve localization while modeling objects at multiple scales, but existing contextual and decoder designs have limitations. PAN combines Feature Pyramid Attention with Global Attention Upsample to fuse multiscale context and guide low-level detail recovery. It achieves 84.0% mIoU on PASCAL VOC 2012 without MS-COCO training or Dense-CRF post-processing.
Problem
Semantic segmentation must handle objects at multiple scales while preserving pixel localization lost through spatial-resolution reduction and existing pyramid operations.
Method
PAN combines FPA, which fuses multiscale context and global attention for high-level features, with GAU, which uses high-level global context to guide low-level localization details.
Results
84.0% mIoU was achieved on the PASCAL VOC 2012 test set without MS-COCO training or Dense-CRF post-processing.
Takeaways & Limitations
FPA and GAU together provide a semantic-segmentation architecture that combines pixel-level attention, expanded receptive fields, and guided localization recovery.
Abstract
from arXiv · showhide
A Pyramid Attention Network(PAN) is proposed to exploit the impact of global contextual information in semantic segmentation. Different from most existing works, we combine attention mechanism and spatial pyramid to extract precise dense features for pixel labeling instead of complicated dilated convolution and artificially designed decoder networks. Specifically, we introduce a Feature Pyramid Attention module to perform spatial pyramid attention structure on high-level output and combining global pooling to learn a better feature representation, and a Global Attention Upsample module on each decoder layer to provide global context as a guidance of low-level features to select category localization details. The proposed approach achieves state-of-the-art performance on PASCAL VOC 2012 and Cityscapes benchmarks with a new record of mIoU accuracy 84.0% on PASCAL VOC 2012, while training without COCO dataset.
1 Introduction
The paper identifies two segmentation challenges: recognizing objects across multiple scales and recovering precise pixel localization after encoding reduces spatial resolution. PAN addresses them with FPA for receptive-field expansion and GAU for guided detail recovery.
- Challenges: Encoding high-dimensional representations loses spatial resolution, causing FCN to miss small object parts and misclassify details.The examples include a missing bicycle handle and an incorrectly categorized sheep.
- Challenges: Objects at multiple scales make category classification difficult, while spatial pyramid and dilated-convolution approaches have limitations.The cited limitations include grid artifacts from sparse dilated convolution and possible localization loss in pyramid pooling.
- Contributions: Feature Pyramid Attention increases receptive field and classifies small objects by extracting precise pixel-level attention for high-level features.FPA is presented as the response to scale variation and localization limitations.
- Challenges: High-level features support category classification but are weak at reconstructing original-resolution binary predictions, motivating a more efficient decoder.Existing U-shaped decoder designs use low-level information to recover image details but are described as time consuming.
- Contributions: Global Attention Upsample provides global context as guidance for weighting low-level features and recovering category localization details.The module is introduced as an effective decoder for the second challenge.
- Contributions: Combining FPA and GAU, PAN is reported to achieve state-of-the-art accuracy on VOC2012 and Cityscapes benchmarks.The introduction lists FPA, GAU, and their combined PAN architecture as the paper’s three main contributions.
2 Related Work
Related work largely builds on FCN-based encoder-decoder architectures and contextual-information modeling. These methods have also influenced architectures used across several other computer-vision tasks.
- Contextual modeling: FCN-based models have substantially improved semantic segmentation and motivated research into network structures that better use contextual information.The related-work discussion frames contextual modeling as a major research direction.
- Encoder-decoder architectures: Encoder-decoder networks are widely used in state-of-the-art segmentation frameworks and have also been applied to pose estimation, detection, stylization, matting, and super-resolution.The passage lists these applications as examples of broader encoder-decoder use.
3 Method
PAN combines a spatial pyramid attention module for high-level context with a global-attention decoder that guides low-level localization features. Its design targets scale variation, pixel-level attention, and efficient detail recovery.
- Network overview: PAN uses ResNet-101 to extract dense features, then applies FPA and GAU to produce precise pixel predictions and localization details.The overview identifies downsampling and upsampling operators within the encoder-decoder pipeline.
- Motivation: Existing pyramid methods can lose local information through dilated convolution artifacts or multiscale pooling, motivating PAN’s attention-based design.The stated issue is preservation of local consistency and pixel localization during contextual aggregation.
- Feature Pyramid Attention: FPA fuses context from three pyramid scales using a U-shaped structure with 3×3, 5×5, and 7×7 convolutions.The high-level feature resolution makes the larger kernels computationally manageable, and the pyramid integrates neighboring scales step by step.
- Feature Pyramid Attention: FPA multiplies the original CNN feature map pixel-wise by pyramid attention features and incorporates global context to form high-level attention.This combines multiscale spatial information with channel-wise global context.
- Feature Pyramid Attention: Unlike PSPNet and ASPP, FPA multiplies context with the original feature map pixel-wise instead of concatenating pyramid features before channel reduction.The paper states that this design does not introduce too much computation.
- Global Attention Upsample: GAU uses global average pooling from high-level features to guide low-level features, then combines weighted low-level information with high-level features during gradual upsampling.The guidance selects category localization details while deploying multiscale feature maps in a simple decoder.
- Complete architecture: PAN uses FPA as the encoder-decoder center block and GAU as a fast decoder that guides low-level information with high-level features.The complete architecture uses a ResNet-101 backbone with dilated convolution and progressively combines precise features during decoding.
4 Experimental Results
Experiments evaluate FPA and GAU through ablations and comparisons on PASCAL VOC 2012, then assess the complete PAN on VOC and Cityscapes. The modules improve segmentation performance, with PAN reaching 84.0% on the VOC 2012 test set without COCO pretraining or Dense-CRF post-processing.
- Experimental setup: The experiments evaluate PAN on PASCAL VOC 2012 and Cityscapes, including ablations of FPA and GAU and comparisons with existing methods.The VOC experiments use validation and test sets; Cityscapes evaluation excludes coarse annotations.
- Feature Pyramid Attention: 77.54% mIoU results from the FPA C333-AVE setting, compared with 72.60% for the baseline on the VOC validation set.The C333 setting uses 3×3 convolutions, while AVE denotes average pooling.
- Feature Pyramid Attention: FPA outperforms DeepLabv3 by almost 1.2% under the same output stride.The comparison uses direct upsampling and output stride 16.
- Global Attention Upsample: 77.84% performance follows the addition of global pooling to GAU, up from 73.56% without the global context attention branch.The decoder ablation compares low-level features alone, channel reduction variants, and global context attention.
- Decoder comparison: 77.84% is achieved without COCO training data, compared with 77.50% for Global Convolution Network trained with extra COCO data.The proposed decoder also outperforms the RRB decoder by 1.2%.
- Complete PAN evaluation: 84.0% is achieved by PAN on the PASCAL VOC 2012 test set without MS-COCO pretraining or Dense-CRF post-processing.Evaluation uses multi-scale inputs, left-right flipping, and fine-tuning on the VOC trainval set.
5 Conclusion
The paper concludes that PAN combines FPA and GAU to provide pixel-level attention, enlarged receptive fields, and guided recovery of pixel localization. Its experimental results are comparable with other state-of-the-art models on PASCAL VOC 2012.
- Conclusion: PAN combines Feature Pyramid Attention and Global Attention Upsample for semantic segmentation.FPA provides pixel-level attention and expands the receptive field, while GAU uses high-level features to guide low-level localization recovery.
- Conclusion: PAN achieves comparable performance with other state-of-the-art models on the PASCAL VOC 2012 semantic image segmentation benchmark.