Source-linked AI summary
Pyramid Scene Parsing Network
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia
TL;DR
Scene parsing must handle large, unrestricted vocabularies and diverse scenes, while existing FCN models lack suitable global scene-context integration. PSPNet combines pixel-level features with pyramid-pooled global context and achieves state-of-the-art performance across datasets, including first-place finishes on ImageNet, PASCAL VOC 2012, and Cityscapes.
Problem
Scene parsing remains difficult because datasets contain diverse scenes, many labels, and unrestricted open vocabularies, while FCN models lack suitable global scene-context integration.
Method
PSPNet extends an FCN-based pixel-prediction framework with pyramid-pooled global and local context features, using deeply supervised loss for optimization.
Results
PSPNet achieves state-of-the-art performance across available datasets, winning ImageNet scene parsing 2016, PASCAL VOC 2012, and Cityscapes benchmarks.
Takeaways & Limitations
PSPNet provides a promising direction for pixel-level prediction tasks through global context integration.
Abstract
from arXiv · showhide
Scene parsing is challenging for unrestricted open vocabulary and diverse scenes. In this paper, we exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling module together with the proposed pyramid scene parsing network (PSPNet). Our global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction tasks. The proposed approach achieves state-of-the-art performance on various datasets. It came first in ImageNet scene parsing challenge 2016, PASCAL VOC 2012 benchmark and Cityscapes benchmark. A single PSPNet yields new record of mIoU accuracy 85.4% on PASCAL VOC 2012 and accuracy 80.2% on Cityscapes.
1. Introduction
Scene parsing assigns category labels to every pixel but remains difficult because scenes and labels are diverse, especially under unrestricted vocabularies. PSPNet addresses this limitation by combining local pixel features with global pyramid-pooled context and achieves state-of-the-art results across major benchmarks.
- Introduction: Scene parsing assigns each pixel a category label to provide complete scene understanding, including each element’s label, location, and shape.The task has applications including automatic driving and robot sensing.
- Introduction: Scene parsing becomes more difficult as scene and label variety increase, with ADE20K presenting a large, unrestricted open vocabulary and more scene classes.Earlier datasets included 33 scenes or semantically similar labels such as chair and sofa or horse and cow.
- Introduction: Current FCN-based methods struggle with diverse scenes and unrestricted vocabularies because similar-looking objects can be misclassified without global scene context.A boat may be mistaken for a car even though the surrounding boathouse-and-river context supports the correct prediction.
- Introduction: PSPNet extends a dilated FCN with specially designed global pyramid pooling, combining local and global clues to make pixel predictions more reliable.The approach embeds difficult scenery context features in an FCN-based pixel prediction framework.
- Introduction: The paper develops an optimization strategy using deeply supervised loss for deep ResNet-based models and provides implementation details, code, and trained models.The authors describe these details as important to their performance.
- Introduction: PSPNet achieves state-of-the-art performance across available datasets, winning the ImageNet scene parsing challenge 2016 and placing first on PASCAL VOC 2012 and Cityscapes.The authors suggest the framework may also benefit pixel-level tasks such as stereo matching, optical flow, and depth estimation.
2. Related Work
Prior scene parsing and semantic segmentation work enlarges receptive fields, combines multi-scale features, and refines structure, but complex scenes still require stronger global context representations. PSPNet addresses this gap through different-region-based context aggregation rather than global pooling alone.
- Dilated convolutions enlarge neural-network receptive fields, while coarse-to-fine structures also advance pixel-level prediction tasks.
- Multi-scale feature ensembling combines semantically rich higher-layer features with more spatially informative features to improve performance.
- Structure-prediction methods use CRF post-processing or end-to-end refinement to improve localization and align predicted semantic boundaries with objects.
- Global-context methods extract image-level priors with traditional features or object-detection frameworks, while global average pooling with FCN improves semantic segmentation.
- PSPNet instead aggregates context across different image regions because global descriptors are not representative enough for challenging ADE20K scenes.
3. Pyramid Scene Parsing Network
PSPNet addresses scene-parsing errors caused by mismatched relationships, confusing categories, and inconspicuous objects by constructing a hierarchical global context prior with pyramid pooling. It combines multi-scale context with local features for pixel-level prediction without substantially increasing computational cost.
- Motivation: ADE20K scene parsing exposes errors from mismatched relationships, confusing categories, and inconspicuous objects, motivating global-scene context modeling.The dataset contains 150 stuff/object labels and 1,038 image-level scene descriptors; annotators still make 17.60% pixel error.
- Pyramid pooling module: The pyramid pooling module constructs a hierarchical global prior to reduce context-information loss across different scales and sub-regions.It addresses the limitations of directly fusing global average-pooled pixels, which can lose spatial relations and cause ambiguity.
- Pyramid pooling module: The module fuses four pyramid levels with bin sizes of 1×1, 2×2, 3×3 and 6×6, using varying-size pooling kernels over different sub-regions.A 1×1 convolution follows each pyramid level to maintain the weight of global features.
- PSPNet architecture: PSPNet uses dilated ResNet features at 1/8 input resolution, gathers whole-, half-, and small-image context, concatenates it with the original feature map, and predicts pixels convolutionally.The network extracts the final feature map with a pretrained ResNet and dilated network strategy before applying pyramid pooling.
- PSPNet architecture: PSPNet provides a more representative global contextual prior than global pooling, adds little computational cost over dilated FCN, and jointly optimizes global and local features.End-to-end learning simultaneously optimizes the global pyramid pooling module and local FCN feature.
4. Deep Supervision for ResNet-Based FCN
The paper introduces deep supervision for ResNet-based FCN training by using an auxiliary loss to produce initial results before learning residuals with the final loss. The auxiliary branch improves optimization during training, while only the master branch is used for final prediction.
- ResNet motivation: ResNet uses skip connections to address optimization difficulty from increasing network depth, with later layers learning residuals from earlier ones.This residual structure motivates the proposed supervision strategy.
- Deep supervision strategy: Deep supervision decomposes optimization into initial-result generation with an additional loss and subsequent residual learning with the final loss.The two optimization steps are each simpler to solve.
- Deep supervision strategy: In ResNet101, an auxiliary classifier follows the fourth stage while the main branch trains the final classifier with softmax loss.Both loss functions propagate through all preceding layers, unlike relay backpropagation, which blocks the auxiliary loss from several shallow layers.
- Inference and generality: During testing, the auxiliary branch is discarded and the optimized master branch alone produces final predictions.The training strategy works with pretrained ResNet models under different experimental settings.
5. Experiments
Experiments evaluate PSPNet on ImageNet scene parsing, PASCAL VOC 2012, and Cityscapes, using ablations and benchmark comparisons. The method consistently improves over baselines and achieves strong results across these datasets.
- Datasets and Evaluation Metrics: PSPNet is evaluated on ImageNet scene parsing 2016, PASCAL VOC 2012 semantic segmentation, and Cityscapes.ADE20K contains up to 150 classes and diverse scenes, requiring parsing of both objects and stuff.
- Ablation Study for PSPNet: Average pooling outperforms max pooling, pyramid parsing outperforms global pooling, and dimension reduction further enhances performance.These comparisons form the main PSPNet ablation study against a ResNet50-based FCN with a dilated network.
- Ablation Study for Auxiliary Loss: 1.41/0.94 improvement in Mean IoU and Pixel Acc. (%) is achieved over the baseline with auxiliary loss weight α = 0.4.The auxiliary loss helps optimize learning without influencing the master branch; deeper networks are expected to benefit further.
- Ablation Study for Pre-trained Model: 1.49 absolute improvement raises (Mean IoU + Pixel Acc.) / 2 (%) from 60.86 to 62.35 when ResNet depth increases from 50 to 269.The study tests ResNet depths of 50, 101, 152, and 269.
- Results in Challenge: 55.38% is obtained by the single-model submission in ImageNet scene parsing challenge 2016, while the ensemble reaches 57.21% and the team places 1st.The single-model score exceeds several other multi-model ensemble submissions, although testing performance is lower than validation performance.
6. Concluding Remarks
The paper proposes an effective pyramid scene parsing network for complex scene understanding, combining global pyramid pooling with deep supervision to provide additional context and improve optimization. The authors also release implementation details to support adoption in scene parsing and semantic segmentation.
- The proposed pyramid scene parsing network targets complex scene understanding.
- Global pyramid pooling provides additional contextual information for scene parsing.
- Deeply supervised optimization is provided for ResNet-based FCN networks.
- Publicly available implementation details can help the community adopt these strategies for scene parsing and semantic segmentation.